S3 Parquet Export

To write data to S3, set up the credentials. Note that the region parameter should match the region of the bucket you want to access.

Query

CREATE SECRET (    TYPE s3,    KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩',    SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩',    REGION '⟨us-east-1⟩');

Tip

If you get an IO Error (Connection error for HTTP HEAD), configure the endpoint explicitly via ENDPOINT 's3.⟨your-region⟩.amazonaws.com'.

Alternatively, retrieve the credentials automatically from the AWS credential chain:

Query

CREATE SECRET (    TYPE s3,    PROVIDER credential_chain);

Once the S3 credentials are configured, Parquet files can be written to S3 using the following command:

Query

CREATE TABLE table_name (id INTEGER);
COPY table_name TO 's3://s3-bucket/filename.parquet';

Result

db error: ERROR: Unable to connect to URL s3://s3-bucket/filename.parquet: (varies)

Similarly, Google Cloud Storage (GCS) is supported through the Interoperability API. You need to create HMAC keys and provide the credentials as follows:

Query

CREATE SECRET (    TYPE gcs,    KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩',    SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩');

After setting up the GCS credentials, you can export using:

Query

COPY table_name TO 'gs://gcs_bucket/filename.parquet';

Result

db error: ERROR: Unable to connect to URL gs://gcs_bucket/filename.parquet: (varies)