Logging

SereneDB implements a logging mechanism that provides users with detailed information about events such as query execution, performance metrics and system events.

Basics

The SereneDB logging mechanism can be enabled or disabled using a special function, enable_logging. Logs are exposed through a table function named duckdb_logs, which can be queried like any standard table.

Example:

Query

CALL enable_logging();
-- Run some queries...SELECT * FROM duckdb_logs;

Result

success

To disable logging, run

Query

CALL disable_logging();

Result

success

To clear the current log, run

Query

CALL truncate_duckdb_logs();

Result

success

Log Level

SereneDB supports different logging levels that control the verbosity of the logs:

• ERROR: Only logs error messages
• WARN: Logs warnings and errors
• INFO: Logs general information, warnings and errors (default)
• DEBUG: Logs detailed debugging information
• TRACE: Logs very detailed tracing information

The log level can be set using:

Query

CALL enable_logging(level = 'debug');

Result

success

Log Types

In SereneDB, log messages can have an associated log type. Log types allow two main things:

• Fine-grained control over log message generation
• Support for structured logging

Logging-Specific Types

To log only messages of a specific type:

Query

CALL enable_logging('HTTP');

Result

success

The above function will automatically set the correct log level and will add the HTTP type to the enabled_log_types settings. This ensures only log messages of the 'HTTP' type will be written to the log.

To enable multiple log types, simply pass:

Query

CALL enable_logging(['HTTP', 'QueryLog']);

Result

success

Structured Logging

Some log types like HTTP will have an associated message schema. To make SereneDB automatically parse the message, use the duckdb_logs_parsed() macro. For example:

Query

SELECT request.headers FROM duckdb_logs_parsed('HTTP');

To view the schema of each structure log type simply run:

Query

DESCRIBE FROM duckdb_logs_parsed('HTTP');

List of Available Log Types

This is a (non-exhaustive) list of the available log types in SereneDB.

Log Type	Description	Structured
QueryLog	Logs which queries are executed in SereneDB	No
FileSystem	Logs all FileSystem interaction with SereneDB's Filesystem	Yes
HTTP	Logs all HTTP traffic from SereneDB's internal HTTP client	Yes

SereneDB Server Log Types

Beyond the core types above, the SereneDB server (serened) emits its own log types from its subsystems. These types are not accepted as a filter argument to enable_logging (which only recognizes the core types listed earlier). Instead, enable logging without a type filter and select the subsystem you are interested in through the type column of duckdb_logs — for example SELECT * FROM duckdb_logs WHERE type = 'Search' to see only search-engine activity.

Log Type	Description
Startup	Server startup and initialization: endpoints, role creation, readiness
Search	Inverted-index (search engine) activity: background refresh and compaction, index maintenance
IResearch	Low-level events from the underlying IResearch engine
Storage	Storage-engine events
SSL	TLS/SSL configuration and connection events
HTTP	HTTP traffic (shared with the core HTTP type above)

Server log messages that are not assigned a type fall into the default (empty) type, so they appear in duckdb_logs with an empty type and are always shown when logging is enabled without a type filter.

Log Storages

By default, SereneDB logs to an in-memory log storage (memory). SereneDB supports different types of log storage. Currently, the following log storage types are implemented in core SereneDB.

Log Storage	Description
memory	(default) Log to an in-memory buffer
stdout	Log to the stdout of the current process (in CSV format)
file	Log to (a) csv file(s)

Note that the duckdb_logs table function is automatically updated to target the currently active log storage. This means that switching the log storage may influence what is returned by the duckdb_logs function.

Logging to stdout

Query

CALL enable_logging(storage = 'stdout');

Result

success

Logging to File

Query

CALL enable_logging(storage = 'file', storage_config = {'path': 'path/to/store/logs'});

Result

success

or using the equivalent shorthand:

Query

CALL enable_logging(storage_path = 'path/to/store/logs');

Result

success

Advanced Usage

Normalized vs. Denormalized Logging

SereneDB's log storages can log in two ways: normalized vs. denormalized.

In denormalized logging, the log context information is appended directly to each log entry, while in normalized logging the log entries are stored separately with context_ids referencing the context information.

Log Storage	Normalized
memory	yes
file	configurable
stdout	no

For file storage, you can switch between normalized and denormalized by providing a path ending in .csv (for normalized) or without .csv (for denormalized). For file logging, denormalized is generally recommended since this increases performance and reduces the total size of the logs. To configure normalization of file log storage:

Query

-- normalized: creates `${__TEST_DIR__}/duckdb_log_contexts.csv` and `${__TEST_DIR__}/duckdb_log_entries.csv`CALL enable_logging(storage_path = '${__TEST_DIR__}');
-- denormalized: creates `${__TEST_DIR__}/logs.csv`CALL enable_logging(storage_path = '${__TEST_DIR__}/logs.csv');

Result

success

Note that the difference between normalized and denormalized is typically hidden from users through the 'duckdb_logs' function, which automatically joins normalized tables into a single unified result. To illustrate, both configurations above will be queryable using FROM duckdb_logs; and will produce identical results.

Buffer Size

The log storage in SereneDB implements a buffering mechanism to optimize logging performance. This implementation introduces a potential delay between message logging and storage writing. This delay can obscure the actual message writing time, which is particularly problematic when debugging crashes, as messages generated immediately before a crash might not be written. To address this, the buffer size can be configured as follows:

Query

CALL enable_logging(storage_config = {'buffer_size': 0});

Result

success

or using the equivalent shorthand:

Query

CALL enable_logging(storage_buffer_size = 0);

Result

success

Note that the default buffer size is different for different log storages:

Log Storage	Default buffer size
memory	STANDARD_VECTOR_SIZE (2048)
file	STANDARD_VECTOR_SIZE (2048)
stdout	Disabled (0)

So for example, if you want to increase your stdout logging performance, simply enable buffering to greatly (>10x) speed up your logging:

Query

CALL enable_logging(storage = 'stdout', storage_buffer_size = 2048);

Result

success

Or imagine you are debugging a crash in SereneDB and you want to use the file logger to understand what's going on: Simply disable the buffering using:

Query

CALL enable_logging(storage_path = '${__TEST_DIR__}/mylogs', storage_buffer_size = 0);

Result

success

Syntactic Sugar

SereneDB contains some syntactic sugar to make common paths easier. For example, the following statements are all equal:

Query

-- regular invocationCALL enable_logging(storage = 'file', storage_config = {'path': 'path/to/store/logs'});
-- using shorthand for common path storage config paramCALL enable_logging(storage = 'file', storage_path = 'path/to/store/logs');
-- omitting `storage = 'file'` -> is implied from presence of `storage_config`CALL enable_logging(storage_config = {'path': 'path/to/store/logs'});

Result

success
success
success