> ## Documentation Index > Fetch the complete documentation index at: https://docs.hightouch.com/llms.txt > Use this file to discover all available pages before exploring further. # SFTP > Transfer data at scale from your warehouse to your apps through SFTP ## Supported syncing | Object Type | Description | Supported Sync Modes | | - | - | - | | Any data set | Sync data from a source to SFTP as CSV, JSON, NDJSON, XML, or Parquet files | Insert, All, Diff | * **All**: All mode creates one file with all the rows in the query results, every time the sync runs. * **Insert**: Insert mode creates one file with the rows that were added since the last sync. * **Diff**: Diff mode creates three files, one for rows added, one for rows changed, and another for rows removed since the last sync. For more information about sync modes, refer to the [sync modes](https://docs.hightouch.com/syncs/types-and-modes.md#sync-modes) docs. > [!NOTE] > > The order of rows and columns in uploaded files may differ from how they appear in your model. This is expected behavior and applies to all sync modes. [Learn more about row and column ordering](https://docs.hightouch.com/models/creating-models.md#row-and-column-ordering). ## Connect to SFTP Go to the [**Destinations** overview page](https://app.hightouch.com/destinations?from=docs) and click the **Add destination** button. Select **SFTP** and click **Continue**. You can then authenticate Hightouch to **SFTP** by entering the following required fields into Hightouch: * **Hostname** * **Port** * Credentials (see the [supported authentication methods](#supported-authentication-methods) below) > [!WARNING] > > The user needs to have access to the target directory and file with **write > privileges**. A `permission denied` error message during a sync indicates the > user may not have write permissions for both the directory and the file. > [!NOTE] > > You may need to allowlist Hightouch's IP addresses to let our systems connect to your SFTP server. Reference our [networking docs](https://docs.hightouch.com/security/networking#ip-addresses) to determine which IP addresses you need to allowlist. ### Supported authentication methods We support three authentication methods: * Username + password * Username + SSH private key * Username + SSH private key + passphrase We support two formats for the SSH private key: * OpenSSH * RSA Provide the full private key for the user, including the header and footer. For example: ```bash theme={null} -----BEGIN OPENSSH PRIVATE KEY----- -----END OPENSSH PRIVATE KEY----- ``` #### Password and SSH key together > [!NOTE] > > Password and SSH key authentication for SFTP must be enabled by Hightouch. [Contact us](#pylon-chat) to turn it on. Some SFTP servers require more than one authentication method on the same login, such as an SSH key followed by the account password. OpenSSH servers set this up with the `AuthenticationMethods` setting, for example `AuthenticationMethods publickey,password`. To connect to one of these servers, enter both the **Password** and the **Private Key**, plus the **Private Key Passphrase** if the key has one. Hightouch completes each step with the method the server asks for next, in whatever order the server requires. If your server collects the password through a keyboard-interactive prompt, which is common on PAM-backed servers (`publickey,keyboard-interactive`), Hightouch answers the prompt with the **Password** you entered. If your server accepts either credential on its own, you can still enter both. Hightouch tries the password first, then the private key. ### PGP encryption The SFTP destination supports PGP encryption. Given a **PGP Public Key**, all files transferred to the destination will be encrypted. To use PGP encryption, provide your public key in the **PGP Public Key** text area including the header and footer in the following format: ```bash theme={null} -----BEGIN PGP PUBLIC KEY BLOCK----- -----END PGP PUBLIC KEY BLOCK----- ``` > [!NOTE] > > The **PGP Private Key** is required to decrypt the files synced to your > destination. You don't need to insert this value in Hightouch. ### Password-protected zip The SFTP destination can encrypt output files at rest with a password-protected AES-256 `.zip`. Unlike SFTP, which only protects data in transit, this keeps a file encrypted after it's delivered to the destination server. To use it, enter a **Zip password** in the destination configuration, then set a sync's [compression format](#compress-output-files) to **Zip**. Hightouch encrypts that sync's output with the password, and the recipient extracts it using the same password with any zip tool that supports AES-256 (for example, 7-Zip, WinZip, or PeaZip). Syncs that use a different compression format ignore the password. We recommend adding `.zip` to your [filename](#enter-filename) in the sync configuration. > [!NOTE] > > A zip password can't be combined with a PGP public key. Both encrypt files at > rest, so choose the one that fits your use case. ## Sync configuration Once you've set up your SFTP destination and have a [model](https://docs.hightouch.com/getting-started/concepts.md#models) to pull data from, you can set up your sync configuration to begin syncing data. Go to the [**Syncs** overview page](https://app.hightouch.com/syncs?from=docs) and click the **Add sync** button to begin. Then, select the relevant model and the SFTP destination you want to sync to. ### Select file format Hightouch supports syncing CSV, JSON, NDJSON, XML, and Parquet files via SFTP. ### Enter filename The filename field lets you specify the parent directory and the name of the file you want to use for your results. You can include timestamp variables in the filename, surrounding each with `{}`. Hightouch supports these timestamp variables: * `YYYY`: Represents the full year in four digits. * `YY`: The last two digits of the year. * `MM`: Two-digit month format (01-12). * `DD`: Two-digit day format (01-31). * `HH`: Two-digit hour format in 24-hour clock (00-23). * `mm`: Two-digit minute format (00-59). * `ss`: Two-digit second format (00-59). * `ms`: Three-digit millisecond format. * `X`: Unix timestamp in seconds. * `x`: Unix timestamp in milliseconds. All dates and times are UTC. For example, you could enter `upload/{YYYY}-{MM}-{DD}-{HH}-{mm}-result.json` to dynamically include the year, month, date, hour, and minute in each uploaded file. Hightouch would insert each file in the `upload` directory, which would need to already exist in your bucket. You can also use other [variable values](https://docs.hightouch.com/syncs/mapping-data.md#variable-values) to include sync metadata in the filename: * `{model.id}` * `{model.name}` * `{sync.id}` * `{sync.run.id}` > [!NOTE] > > If a file already exists at the path you entered at the time of a sync, > Hightouch overwrites it. > [!TIP] > > If you are using an audience and would like to include the audience name, you > will still use `{model.name}`. ### Set filename offset By default, Hightouch uses the timestamp of the sync run to fill in timestamp variables. You can optionally include an offset in seconds. For example, if you want the filename's date to be 24 hours *before* the sync takes place, enter `-86400` (`24 hours * 60 minutes * 60 seconds`). If you want the filename's data to be one hour after the sync takes place, you would enter `3600` (`60 minutes * 60 seconds`). ### CSV options If you're syncing to a CSV file, you have additional configuration options: * **Delimiter**: Your options are comma (`,`), semicolon (`;`), pipe (`|`), tilde (`~`), and tab * (Optional) Whether to **include a CSV header row** in the exported files * (Optional) Whether to **include a [byte order mark (BOM)](https://en.wikipedia.org/wiki/Byte_order_mark)** in the exported files, the BOM is `` ### Columns to sync You can export all columns exactly as your model returns them or choose to export specific ones. If you need to rename or transform any column values you're syncing, you can use the [advanced mapper](https://docs.hightouch.com/syncs/mapping-data.md#advanced-mapper) to do so. If you choose this option, Hightouch only syncs the fields you explicitly map. ![Field mapping in the Hightouch UI](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-sftp-add-mappings.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=6239339e1298e28226f180e0418d7984) The preceding example shows how to selectively export the `id`, `email` and `location`. These columns are mapped to new fields in the destination file as `user_id`, `user_email` and `user_location`. Hightouch exports these fields to the new fields in the file and ignores all other columns from your results. > [!TIP] > > `Boolean` values that appear as `True` and `False` in your model appear as `1` > and blank respectively in the file created during the sync. If you'd like for > these values to appear in another format in your downstream file, you can use > [template mapping](https://docs.hightouch.com/syncs/mapping-data.md#template-mapping) to achieve your > desired format. ### Batch size By default, and depending on your [sync mode](#supported-syncing), Hightouch sends one file for each export. (In diff mode, Hightouch sends three files.) By enabling batching, you allow Hightouch to send multiple files if your model query results are large. This can help avoid file size errors. ### Empty file results You can select how Hightouch should handle empty results files. Empty result files can occur if your model's query results haven't changed since the last sync. You can select whether to **skip empty files**. If you skip empty files, it means Hightouch won't export any files if your model's query results haven't changed since your last sync. ### Compress output files You can compress each output file. In the sync configuration, set the compression format to one of: * **None**: Hightouch delivers the file uncompressed. * **Gzip**: Hightouch compresses the file with gzip. * **Zip**: Hightouch packages the file in a `.zip` archive. If the destination has a [zip password](#password-protected-zip), the archive is AES-256 encrypted at rest; otherwise it's a plain zip. We recommend adding the matching extension (`.gz` or `.zip`) to your [filename](#enter-filename), though it isn't required. ## Encoding By default, Hightouch saves the file in ASCII encoding. If there are any non-ASCII characters in your results, the file will be saved with utf-8 encoding. If you've enabled [including a BOM](#csv-options), the file will be saved with utf-8 encoding with BOM. ## Tips and troubleshooting To see the exact API requests and responses from each sync run, use the [live debugger](https://docs.hightouch.com/syncs/debugger.md). To get notified by Slack, PagerDuty, SMS, or email when a sync fails, set up [sync alerts](https://docs.hightouch.com/syncs/alerting.md). If you run into an issue this page doesn't cover, [contact support](#pylon-chat). > [!NOTE] > > For Hightouch platform error codes related to destinations, see [Error > codes](https://docs.hightouch.com/developer-tools/error-codes.md#other-destinations). ### Validate your setup The **Test connection** button on the destination configuration page tries to connect to your configured server, but it can only detect whether your credentials are completely invalid. To fully test user permissions, create and run a one-row test sync to verify that your data syncs to your SFTP server. ## Common errors ### All configured authentication methods failed ``` All configured authentication methods failed ``` **Cause:** The SFTP server rejected every credential Hightouch tried. Common reasons: * The username, password, private key, or passphrase is incorrect. * The server requires a password and an SSH key on the same login. See [Password and SSH key together](#password-and-ssh-key-together). **Solution:** 1. Test the same credentials with a standalone SFTP client. Running `sftp -v` shows which authentication methods the server accepts at each step. 2. If the server requires a password and a key together, follow [Password and SSH key together](#password-and-ssh-key-together). 3. Otherwise, correct the credentials in the destination configuration. If you're not sure which method your server uses, check with whoever manages your SFTP server. ### Renaming temporary file to target file failed **Cause:** During an SFTP sync, Hightouch writes data to a temporary file on your server, then renames it to the final filename from your sync configuration. This avoids partial writes if the network drops. The error means a downstream process on the SFTP server consumed or deleted the temporary file before the sync finished, or the [credentials you're using](#connect-to-sftp) lack permission to rename the temporary file or modify its directory. **Solution:** Adjust directory listeners to ignore temporary files, which are prefixed with a period (`.`), or set a different temporary file directory in your sync configuration. Also confirm the SFTP user has permission to rename files in the target directory. ### Connection timed out/waiting for handshake **Cause:** Hightouch's IP addresses aren't allowlisted and may be blocked by your SFTP firewall. **Solution:** Allowlist Hightouch's [IP addresses](https://docs.hightouch.com/security/networking#ip-addresses) on your SFTP firewall. If you're unsure whether the IPs are allowlisted, ask your network or IT team, or your third-party SFTP vendor, to verify.