Secrets Manager

The Secrets manager provides a unified user interface for secrets across all backends that use them. Secrets can be scoped, so different storage prefixes can have different secrets, allowing for example to join data across organizations in a single query. Secrets can also be persisted, so that they do not need to be specified every time SereneDB is launched.

Attention

Persistent secrets are stored in an unencrypted format.

Types of Secrets

Secrets are typed, their type identifies which service they are for. Currently, the following secret types are available:

Secret type	Service / protocol
http	HTTP and HTTPS
huggingface	Hugging Face
iceberg	Iceberg REST Catalog
postgres	PostgreSQL
s3	AWS S3

For each type, there are one or more “secret providers” that specify how the secret is created. Secrets can also have an optional scope, which is a file path prefix that the secret applies to. When fetching a secret for a path, the secret scopes are compared to the path, returning the matching secret for the path. In the case of multiple matching secrets, the longest prefix is chosen.

Creating a Secret

Secrets can be created using the CREATE SECRET SQL statement. Secrets can be temporary or persistent. Temporary secrets are used by default and live in memory for the lifespan of the SereneDB instance. Persistent secrets are stored in an unencrypted format and are automatically loaded when SereneDB starts.

Secret Providers

To create a secret, a Secret Provider needs to be used. A Secret Provider is a mechanism through which a secret is generated. To illustrate this, for the S3 secret type, SereneDB currently supports two providers: CONFIG and credential_chain. The CONFIG provider requires the user to pass all configuration information into the CREATE SECRET, whereas the credential_chain provider will automatically try to fetch credentials. When no Secret Provider is specified, the CONFIG provider is used.

Temporary Secrets

To create a temporary unscoped secret to access S3, we can now use the following:

Query

CREATE SECRET my_secret (    TYPE s3,    KEY_ID 'my_secret_key',    SECRET 'my_secret_value',    REGION 'my_region');

Note that we implicitly use the default CONFIG secret provider here.

Persistent Secrets

In order to persist secrets between SereneDB database instances, we can now use the CREATE PERSISTENT SECRET command, e.g.:

Query

CREATE PERSISTENT SECRET my_persistent_secret (    TYPE s3,    KEY_ID 'my_secret_key',    SECRET 'my_secret_value');

This writes the secret (unencrypted) so that it is available again the next time SereneDB starts.

Deleting Secrets

Secrets can be deleted using the DROP SECRET statement, e.g.:

Query

DROP PERSISTENT SECRET my_persistent_secret;

Creating Multiple Secrets for the Same Service Type

If two secrets exist for a service type, the scope can be used to decide which one should be used. For example:

Query

CREATE SECRET secret1 (    TYPE s3,    KEY_ID 'my_secret_key1',    SECRET 'my_secret_value1',    SCOPE 's3://⟨my-bucket⟩');

Query

CREATE SECRET secret2 (    TYPE s3,    KEY_ID 'my_secret_key2',    SECRET 'my_secret_value2',    SCOPE 's3://⟨my-other-bucket⟩');

Now, if the user queries something from s3://⟨my-other-bucket⟩/something, secret secret2 will be chosen automatically for that request. To see which secret is being used, the which_secret scalar function can be used, which takes a path and a secret type as parameters:

Query

FROM which_secret('s3://⟨my-other-bucket⟩/file.parquet', 's3');

Listing Secrets

Secrets can be listed using the built-in table-producing function, e.g., by using the duckdb_secrets() table function:

Query

FROM duckdb_secrets();

Sensitive information will be redacted.