- Supports dbt version
0.19.0 - Supports Seeds
- Correctly detects views and their columns
- Experimental support for incremental models
- Only inserts
- Does not support the use of
unique_key
- Only supports Athena engine 2
pip install git+https://github.com/Tomme/dbt-athena.git
A dbt profile can be configured to run against AWS Athena using the following configuration:
| Option | Description | Required? | Example |
|---|---|---|---|
| s3_staging_dir | S3 location to store Athena query results and metadata | Required | s3://bucket/dbt/ |
| region_name | AWS region of your Athena instance | Required | eu-west-1 |
| schema | Specify the schema (Athena database) to build models into (lowercase only) | Required | dbt |
| database | Specify the database (Data catalog) to build models into (lowercase only) | Required | awsdatacatalog |
Example profiles.yml entry:
athena:
target: dev
outputs:
dev:
type: athena
s3_staging_dir: s3://athena-query-results/dbt/
region_name: eu-west-1
schema: dbt
database: awsdatacatalogAdditional information
threadsis supporteddatabaseandcatalogcan be used interchangeably
external_location(default=none)- The location where Athena saves your table in Amazon S3
- If
nonethen it will default to{s3_staging_dir}/tables - Note If you are using a static value, when your table is recreated Athena will not remove the underlying redundant
data causing a
HIVE_PATH_ALREADY_EXISTSerror. Ideally you should set a dynamic value to avoid this issue. Here are some example of dynamic values.external_location="s3://dbt-tables/example-directory/" + run_started_at.isoformat()external_location="s3://dbt-tables/example-directory/" + invocation_id
partitioned_by(default=none)- An array list of columns by which the table will be partitioned
- Limited to creation of 100 partitions (currently)
bucketed_by(default=none)- An array list of columns to bucket data
bucket_count(default=none)- The number of buckets for bucketing your data
format(default='parquet')- The data format for the table
- Supports
ORC,PARQUET,AVRO,JSON, orTEXTFILE
field_delimiter(default=none)- Custom field delimiter, for when format is set to
TEXTFILE
- Custom field delimiter, for when format is set to
More information: CREATE TABLE AS
Experimental support for incremental models.
- Only inserts
- Does not support the use of
unique_key
Due to the nature of AWS Athena, not all core dbt functionality is supported. The following features of dbt are not implemented on Athena:
- Snapshots
- Quoting is not currently supported
- Tables, schemas and database should only be lowercase
- Only supports Athena engine 2
First, install the adapter and its dependencies using make (see Makefile):
make install_depsNext, configure the environment variables in dev.env to match your Athena development environment. Finally, run the tests using make:
make run_tests