Supported syncing
For more information about sync modes, refer to the sync modes docs.
This destination is for Snowflake-managed Iceberg tables. If you want to sync to standard Snowflake tables, use the Snowflake destination instead.
Before you begin
Before setting up the Snowflake Iceberg destination, you must have an external volume configured in Snowflake. Iceberg tables require an external volume that specifies where the table data and metadata are stored.Snowflake credential setup
To allow Hightouch access to Snowflake for Iceberg tables, it’s best to create a user specifically provisioned with access to the required tables, schemas, and external volumes. You can also use a personal Snowflake login for your credentials, as long as it has the correct permissions, specifically:- Add, update, and delete (if applicable) rows in your sync’s Iceberg table
- View the
INFORMATION_SCHEMA.COLUMNStable which is used for gathering metadata to set up the sync USAGEprivilege on the external volume used for Iceberg tables
Service account setup
You can use the following SQL template to create a service account with the necessary roles and permissions:Connect to Snowflake Iceberg
Go to the Destinations overview page and click the Add destination button. Select Snowflake Iceberg and click Continue. If you’re not using a tunnel, you can then authenticate Hightouch to Snowflake by entering the following fields:- Account identifier - You can find this identifier at the beginning of your Snowflake URL, for example, (
https://ACCOUNT_IDENTIFIER.snowflakecomputing.com). - Warehouse - This specifies the warehouse to use when Hightouch executes queries in Snowflake.
- Database - This specifies the database to use when Hightouch executes queries in Snowflake.
Account identifier format may differ based on Snowflake account age. For example, older Snowflake accounts often have identifiers that look like
ACCOUNT_LOCATOR.CLOUD_REGION_ID.CLOUD, whereas newer Snowflake accounts have identifiers that look like ORGNAME-ACCOUNT_NAME.
For more details, visit Snowflake’s account identifier docs.Iceberg configuration
- External volume - The Snowflake external volume that provides storage for Iceberg tables. This must be configured in your Snowflake account before setting up the destination.
- Base location (optional) - A prefix path within the external volume where Iceberg tables will be stored.
Credentials
- Username - This can be your personal Snowflake login or a dedicated user for Hightouch. Note: Use the “Login name”, which may differ from the Snowflake username.
- Role (optional) - Use this field to specify the role Hightouch should use when executing queries in Snowflake. If left blank, Hightouch uses the user’s default role.
- RSA key pair (recommended)
- Password
Snowflake is deprecating password authentication in favor of RSA key pair authentication or multi-factor authentication. Hightouch connections configured with password authentication continue to work only if Snowflake permits it. For details, see Snowflake’s MFA rollout guide. Check your Snowflake account for the rollout timeline.
RSA authentication
Generate a private key and public key by running the following commands in your terminal:ALTER USER command to assign the public key to your Hightouch Snowflake user:
SSH tunneling
Hightouch can connect directly to Snowflake over the public internet or via an SSH tunnel. Since data is encrypted in transit via TLS, a direct connection is suitable for most use cases. You may need to set up a tunnel if your Snowflake instance is on a private network or virtual private cloud (VPC). Hightouch supports both standard and reverse SSH tunnels. To learn more about SSH tunneling, refer to Hightouch’s tunneling documentation.Sync configuration
Once you’ve set up your Snowflake Iceberg destination and have a model to pull data from, you can set up your sync configuration to begin syncing data. Go to the Syncs overview page and click the Add sync button to begin. Then, select the relevant model and the Snowflake Iceberg destination you want to sync to.Table selection
Select the schema and Iceberg table you want to sync to. You can either select an existing Iceberg table or enable Automatically create and update Iceberg table to have Hightouch create a new Iceberg table using your configured external volume. If auto-create is enabled and the table already exists, Hightouch alters its definition to add new columns as needed.Sync mode
Hightouch supports Upsert mode using theMERGE INTO statement, with the option to delete removed rows, Insert mode using the COPY INTO statement, and Mirror mode which will truncate the table and then insert all rows from the model.
Record matching
To match rows from your model to rows in the Iceberg table, Hightouch requires you to select a unique identifier in the table you are syncing to. The Snowflake column you select must be set as aUNIQUE, PRIMARY KEY column within Snowflake.
You can see columns that fit this criteria as available options in records matching section.
If there are no fields in the dropdown, you must add a unique type column to your Iceberg table. Then, click the refresh icon to access the newly created column.
In Insert mode, Snowflake automatically generates an identifier for every new record synced, so there is no need to match an existing record.
Column types
Iceberg tables have some data type limitations compared to standard Snowflake tables:TIMESTAMP_LTZ is the recommended timestamp type for Iceberg tables. It internally stores UTC time with a specified precision. All operations are performed in the current session’s time zone, controlled by the TIMEZONE session parameter. Note that TIMESTAMP_TZ is not supported by Iceberg tables.Field mapping
You can sync columns from your model to the columns in your Iceberg table. When auto-create is enabled, you can create new columns by mapping to column names that don’t exist yet.Batch size
You can tune the number of rows Hightouch upserts or inserts per query based on your needs and database threshold. The default is 50,000 rows per batch. If you want to increase the sync’s speed, you can increase the batch size. Keep in mind that in upsert mode, Snowflake fails the entire batch of rows if it detects any erroneous row. If you suspect that you will have many bad rows, don’t use a high batch size. To avoid locks, ensure you account for your database’s capacity when increasing the batch size.Delete behavior
The delete behavior you select dictates what to do when a row no longer appears in your model’s query results. You have the following options:Tips and troubleshooting
To see the exact API requests and responses from each sync run, use the live debugger. To get notified by Slack, PagerDuty, SMS, or email when a sync fails, set up sync alerts. If you run into an issue this page doesn’t cover, .External volume errors
If you see errors related to the external volume during connection testing, verify:- The external volume exists in your Snowflake account
- Your user has
USAGEprivilege on the external volume - The external volume name is spelled correctly in the destination configuration