> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hightouch.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Amazon SQS

> Connect to your internal microservices via a message queue

## Overview

Hightouch integrates directly with Amazon SQS to support high-throughput, distributed, or asynchronous workloads, letting you build a custom connector to your internal systems.

This destination was designed to be as flexible as possible. You can publish into different queues including standard and FIFO queues for each message trigger and define custom ordering keys and metadata fields to enrich your messages.

## Connect to Amazon SQS

### Connect to your AWS account

When setting up the Amazon SQS destination for the first time, you need to enter your **AWS Credentials** to give Hightouch access to your AWS account. Hightouch needs permission to send messages to Amazon SQS on your behalf.

![Entering AWS Credentials in Hightouch](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-amazon-sqs-set-up.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=af27ff558573de2364bcd6c68855f8bc)

Hightouch believes in the principle of least privilege. We ask for no more permissions than necessary. For the Amazon SQS destination, Hightouch requires `sqs:SendMessage` and `sqs:ListQueues`.

When configuring your credentials in Hightouch, you have two options:

* Setting up a cross-account role (recommended)
* Providing an access key

#### Set up cross-account role

Cross-account roles are the most secure method for granting Hightouch access to Amazon SQS in your AWS account.

You need the **Account ID** and **External ID** in the Hightouch UI to set up a cross-account role.

![Account ID and External ID in Hightouch](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-aws-cross-account-role.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=5ba3e60149d6a8bdb9a0deb10eaa217b)

1. In the **AWS Console**, navigate to **IAM** → **Roles**.
2. Click **Create Role**.
3. Under **Trusted entity type**, choose **AWS account**.
4. Select **Another AWS account** and also click **Require external ID**.

   ![AWS account configuration](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-aws-iam-create-role.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=c98a232f06d4b6643471afa2621571f6)
5. Copy/paste the **Account ID** from Hightouch into the **Account ID** field in AWS.
6. Copy/paste the **External ID** from Hightouch into the **External ID** field in AWS.
7. On the **Add permissions** screen, proceed to attach permissions to the role using any policy that includes at least `sqs:SendMessage` and `sqs:ListQueues`. Then create the role.

   ![AWS ARN](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-amazon-sqs-copy-role-arn.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=01706e571dc06b97c1c1211feed623b8)
8. Copy/paste the **Role ARN** from AWS into Hightouch. Click **Create** to finish setting up your cross-account role.

   ![Pasting the AWS ARN in Hightouch](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-aws-cross-account-role-arn.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=1ba20486b86ba17688134663f97a4cca)

#### Provide access key

If you don't want to create a cross-account role, you can create a regular IAM user and share a **Access Key ID** and **Secret Access Key** with Hightouch.

![Entering Access Key ID and Secret Access Key into Hightouch](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-aws-credentials-access-key.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=fca46a94fc68d7537af1ced9eacded6d)

Be sure to attach the user to a permission policy that includes `sqs:SendMessage` and `sqs:ListQueues` for the specific resource you want to use via a Hightouch sync.

An example policy looks like this:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["sqs:SendMessage", "sqs:ListQueues"],
      "Resource": ["YOUR_SQS_ARN_1", "YOUR_SQS_ARN_2", "YOUR_SQS_ARN_3"]
    }
  ]
}
```

### Select your AWS region

In this step, you specify which region has the resources you want to sync to.

## Syncing data

Once you've authenticated your AWS account in Hightouch and selected an AWS Region, you've completed setup for a Amazon SQS destination in your Hightouch workspace. The next step is to configure a sync that send messages whenever rows are added, changed, or removed in your model.

### Configure your events trigger

Hightouch monitors your data model for added, changed, and removed rows. In this step, you specify which of these events should trigger message publishing.

![Events to trigger message publishing](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-apache-kafka-events-trigger.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=28854166e227ae9a2b4a5ef059e2c5e1)

### Choose your queue

In this step, you choose to send the messages to either Standard or FIFO queues. Hightouch allows you to sync to existing queues that are already in your Amazon SQS.

![Toggle use column to send message to multiple queues](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-amazon-sqs-choose-queue.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=c05ae2969ebb222b504d12dfa2337636)

### Customize your message

![JSON message options in the Hightouch UI](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-apache-kafka-customize-message-data.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=0c1b4ab9a4c632c12b92dd202d69638a)

In this step, you tell Hightouch which column in your model contains the unique message ID and how to build the JSON message data object using data from your model.

#### Configure unique message ID

This unique ID field is sent along with the message data and is an identifier for a message in this batch used to communicate the result.

![Configure unique message ID](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-amazon-sqs-choose-message-id.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=ffba1130f8fddb61ab1e1b71780db6ed)

#### Configure message group ID (for FIFO queues only)

If you are syncing to a FIFO queue, you need to provide a message group ID to specify that a message belongs to a specific message group. Messages that belong to the same message group are processed in a FIFO manner however, messages in different message groups might be processed out of order.
You can toggle between a column in your model or a static value to set your message group ID.

![Configure unique message group ID](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-amazon-sqs-choose-message-group-id.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=7cf8d1f81993829ffe95ebdf5556faa0)

This destination offers three methods of composing a JSON object:

* [Using a JSON editor](#use-json-editor)
* [Using one column from a model](#use-one-column-from-model)
* [Using multiple columns from a model](#use-multiple-columns-from-model)

#### Use JSON editor

![Selecting the JSON editor method in the Hightouch UI](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-apache-kafka-json-editor.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=6830e78d7d8182901468cda59e8a7d2f)

With the JSON editor, you can compose any JSON object using the [Liquid template language](https://shopify.github.io/liquid/). This is particularly useful for complex message data bodies containing nested objects and arrays, which can sometimes be difficult to model entirely in SQL.

Suppose your data model looks like this:

| full\_name | age | email\_address | phone\_number |
| - | - | - | - |
| John Doe | 30 | `john@example.com` | +14158675309 |

And you want your message data like this:

```json theme={null}
{
  "name": "John Doe",
  "age": 30,
  "contact_info": [
    {
      "type": "email",
      "value": "john@example.com"
    },
    {
      "type": "phone",
      "value": "+14158675309"
    }
  ]
}
```

Your Liquid template should look like this:

```txt theme={null}
{
  "name": "{{row.full_name}}",
  "age": {{row.age}},
  "contact_info": [
    {
      "type": "email",
      "value": "{{row.email_address}}"
    },
    {
      "type": "phone",
      "value": "{{row.phone_number}}"
    }
  ]
}
```

This makes it so you can reference any column using the syntax `{{row.column_name}}`. You can also use advanced Liquid features to incorporate control flow and loops into your dynamic message data.

> [!NOTE]
>
> When injecting strings into your JSON request body, be sure to surround the
> Liquid tag in double quotes.

#### Use one column from model

![Selecting using one column from the model as the JSON construction method in the Hightouch UI](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-apache-kafka-one-column-from-model.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=c66e2810e7ecfc25e2b2850a2dc23d25)

If you're already storing JSON data in your source, or if you have the ability to construct a JSON object using SQL, you can select one column in your model that already contains the full message data.

This setting is commonly used when syncing web events that have already been collected and stored as JSON objects in your database.

#### Use multiple columns from model

![Selecting using multiple columns from the model as the JSON construction method in the Hightouch UI](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-apache-kafka-multiple-columns-from-model.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=382ba61008ce45c38571d0baab75e043)

For the simplest use cases, Hightouch can construct a JSON object with key/value pairs based on multiple columns in your model.

Suppose your model looks like this:

| email | first\_name | last\_name |
| - | - | - |
| `alice.doe@example.com` | Alice | Doe |
| `bob.doe@example.com` | Bob | Doe |
| `carol.doe@example.com` | Carol | Doe |

The field mapping in the preceding screenshot would generate the following message data for the first row:

```json theme={null}
{
  "customer_first_name": "Alice",
  "customer_last_name": "Doe",
  "customer_email": "alice.doe@example.com"
}
```

> [!TIP]
>
> You can use the field mapper to rename fields. For example, `first_name` can
> be mapped to `customer_first_name`.

### Configure optional message properties

Along with your row data in JSON format, you can optionally include ordering keys to configure the order your queue receives your message and metadata fields as headers.

##### `DelaySeconds` (for standard queues only)

A number field that postpone the delivery of new messages to consumers for a number of seconds. Any messages that you send to the queue will remain invisible to consumers for the duration of the delay period. The maximum length of time you can delay for is 15 minutes.

##### `MessageDeduplicationId` (for FIFO queues only)

This is a string field. If a message with a particular Message Deduplication ID is sent successfully, subsequent messages with the same Message Deduplication ID are accepted successfully but aren't delivered. If you enabled Content-based deduplication when configuring your FIFO queue then you may optionally embed a deduplication message ID for your message. If you turned Content-based deduplication off, then this section is required for the messages to be published successfully.

##### `MessageAttributes`

This is an object containing key/value pairs of custom mapping fields. Values can be text or byte strings.

### Configure initial sync behavior

In this step, you tell Hightouch how to handle rows present in your model results during the first sync run.

Certain workflows may require performing a backfill of all rows during the initial sync. For other use cases, you might only want to send messages in response to future data changes.

![Declaring how to handle initial sync behavior in Hightouch](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/destinations/destination-apache-kafka-initial-sync-behaviour.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=4a41e51ef0ff4e7c1152c87b50407a79)

## Tips and troubleshooting

To see the exact API requests and responses from each sync run, use the [live debugger](https://docs.hightouch.com/syncs/debugger.md). To get notified by Slack, PagerDuty, SMS, or email when a sync fails, set up [sync alerts](https://docs.hightouch.com/syncs/alerting.md). If you run into an issue this page doesn't cover, [contact support](#pylon-chat).