> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hightouch.com/llms.txt
> Use this file to discover all available pages before exploring further.

# S3

> Securely sync CSV or JSON data from any S3 bucket

> [!TIP]
>
> Need to join, transform, or filter your S3 data? Consider using the [AWS
> Athena](https://docs.hightouch.com/sources/amazon-athena.md) source to directly query your S3 data using
> SQL.

## Overview

Hightouch lets you pull data from CSV and JSON files stored in an Amazon S3 bucket and push them to downstream destinations. To get started, you need an S3 bucket and AWS credentials.

## AWS credential setup

See the guide for [configuring AWS credentials](https://docs.hightouch.com/security/aws.md) to learn how you can set up an IAM principal and use its credentials. The IAM principal whose credentials you use must have programmatic access enabled and permission to read from the S3 path you want to use.

Hightouch needs the following IAM actions to retrieve items from your bucket:

| Action | Details |
| :- | :- |
| `s3:GetObject` | Grants permission to retrieve objects from Amazon S3 |
| `s3:ListBucket` | Grants permission to list some or all the objects in an Amazon S3 bucket (up to 1000) |
| `sts:GetFederationToken` | (Optional) Grants permission to generate short-lived AWS credentials for faster warehouse reads |

> [!TIP]
>
> If you use an **access key** credential in Hightouch, we recommend adding
> `sts:GetFederationToken`. This lets Hightouch generate short-lived credentials
> scoped to the S3 prefix being written, which enables syncs to export data from
> your warehouse to S3 (faster than reading rows via streaming). If you don't
> grant this permission, Hightouch can still work, but syncs will fall back to
> the slower streaming method instead. If you use a **cross-account role**,
> `sts:GetFederationToken` isn't required.

You can use the following JSON sample to create your IAM policy:

```yaml theme={null}
{
  "Version": "2012-10-17",
  "Statement":
    [
      {
        "Effect": "Allow",
        "Action": ["s3:GetObject", "s3:ListBucket"],
        "Resource":
          ["arn:aws:s3:::${bucketName}/*", "arn:aws:s3:::${bucketName}"],
      },
    ],
}
```

If you want to add the optional permission, include the following statement:

```json theme={null}
{
  "Sid": "AllowFederationTokenForS3",
  "Effect": "Allow",
  "Action": ["sts:GetFederationToken"],
  "Resource": "*"
}
```

## Connection configuration

To get started, go to the [**Sources** overview page](https://app.hightouch.com/sources?from=docs) and click the **Add source** button. Select **Amazon S3** and follow the steps below.

### Configure your credentials

Select the credentials you previously created or click **Create new** to set them up now.

### Configure your source

Enter your S3 bucket's **Region** and the **Bucket name**. The bucket name should be just the name of the bucket, not a URL.

## Test your connection

When setting up Amazon S3 as a source for the first time, Hightouch validates your AWS credentials and access to your S3 bucket. Once the test passes, click **Continue** to finish setup.

## Create models

Once your source configuration has passed the necessary validation, your source setup is complete. Next, you can set up [models](https://docs.hightouch.com/getting-started/concepts.md#models) to define which data you want to pull from your S3 bucket. The file must have either a `.csv` or `.json` extension.

### CSV requirements

CSV files must meet the following requirements:

* They must have a header row. The values in the header row are automatically available as column names when you set up a sync.
* They must use comma-separated values; tabs and other delimiters aren't supported.
* They must use double quotes (`"`) for quoted values.

### JSON requirements

JSON files must meet the following requirements:

* The input file must contain an array at the top level.
* Each element in the input file must be an object with the same keys.

The keys of the array elements are automatically available as column names when you set up a sync.

### Model setup

Hightouch supports syncing data from a specific file in your S3 bucket as well
as from the last modified file that matches a given prefix.

1. In Hightouch, go to the [**Models** overview page](https://app.hightouch.com/models?from=docs).
2. Click **Add model.**
3. Select the Amazon S3 source you previously created.
4. Enter the **relative path** to the CSV or JSON file that you want to sync
   data from, like `path/prefix/file.csv`. The path shouldn't contain the bucket
   name.
5. To sync the last modified file with a given prefix, check the **Use last
   modified** box and enter a prefix like `path/prefix` in the path field. See
   Amazon's [S3 docs on
   prefixes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html)
   for examples of how prefixes match objects. To sync the last modified file in
   the entire bucket, check the **Use last modified** box and leave the path field
   blank.
6. Preview your model's query results.
7. Click **Continue**.
8. Name your model and select its [**primary key**](https://docs.hightouch.com/models/creating-models.md#unique-primary-key-requirement). Hightouch uses the primary key to determine which rows have been added, changed, or removed since the last sync.

## Tips and troubleshooting

If you run into an issue this page doesn't cover, [contact support](#pylon-chat).