QuackDB (quackdb v0.7.0)

Copy Markdown View Source

DuckDB for Elixir applications: the DBConnection client.

Queries, streams, transactions, and native appends over DuckDB's Quack protocol (version 3, served by DuckDB 2.0). QuackDB.Server supervises a local DuckDB process, and Ecto.Adapters.QuackDB puts Ecto on top of this client.

Connection options

  • :uri and :token locate and authenticate the Quack server.
  • :heartbeat_timeout is the connection lease in seconds the client asks for (default 60). The server drops a connection that sends no request for that long, so each connection sends a heartbeat every third of the lease over a separate HTTP connection. Long queries, slow streams, and long transactions keep their connection. A connection whose lease expires anyway reconnects.
  • :client_id identifies this client in the server's logs.
  • :acknowledge_results confirms every fully received result, so a server with quack_enable_reconnects drops its cached copy right away instead of waiting for quack_result_ttl.
  • :fetch_retries is how often a result batch is requested again after its HTTP connection broke (default 1). Batches are numbered, so a retry never skips or repeats rows.
  • :copy_binaries copies decoded strings and blobs off the HTTP response (default false); it can also be passed per query. Without it, a decoded string of 64 bytes or more is a slice of the response it arrived in, so keeping it keeps the whole response, up to one result batch, in memory. Copying about doubles decode time for string columns and costs nothing for other columns. Use it when you keep a few values from large results, for example in a cache or a long-lived process.
  • :connect_timeout, :receive_timeout, :shutdown_timeout, and :mint_options tune the HTTP transport.

Query options include :inline_rows, the number of rows the server may return with the first response before the rest is fetched in batches.

Summary

Functions

Interrupts the query running on the remote connection connection_id.

Streams query results as column-oriented batches.

Runs a query and returns its result as a column-oriented map.

Appends column-oriented values to a DuckDB table by streaming binary vectors into an INSERT.

Appends row-oriented values to a DuckDB table by streaming binary vectors into an INSERT.

Appends an enumerable of row maps/keywords in batches through native append.

Appends any Table.Reader compatible tabular data through native append.

Types

append_type()

@type append_type() :: QuackDB.Type.spec()

Native append column type specs.

insert_column()

@type insert_column() :: {atom() | String.t(), [term()]}

insert_row()

@type insert_row() :: map() | Keyword.t()

start_option()

@type start_option() ::
  {:uri, String.t()}
  | {:token, String.t()}
  | {:name, GenServer.name()}
  | {:connect_timeout, timeout()}
  | {:receive_timeout, timeout()}
  | {:shutdown_timeout, timeout()}
  | {:mint_options, keyword()}
  | {:heartbeat_timeout, pos_integer()}
  | {:client_id, String.t()}
  | {:acknowledge_results, boolean()}
  | {:fetch_retries, non_neg_integer()}
  | {:copy_binaries, boolean()}

Functions

cancel(uri, connection_id, options \\ [])

@spec cancel(String.t() | URI.t(), String.t(), Keyword.t()) ::
  :ok | {:error, Exception.t()}

Interrupts the query running on the remote connection connection_id.

Run it from another process than the one waiting on the query. The connection id appears in every QuackDB.Result, QuackDB.Error, and [:quackdb, :query, :start] telemetry event, including those of Ecto queries, and a query interrupted this way fails with a :query_cancelled error:

:telemetry.attach("report-connection", [:quackdb, :query, :start], fn _event, _measurements, metadata, parent ->
  send(parent, {:running, metadata.connection_id})
end, self())

task = Task.async(fn -> MyApp.AnalyticsRepo.all(yearly_report) end)

receive do
  {:running, connection_id} -> QuackDB.cancel("http://[::1]:9494", connection_id)
end

Task.await(task)
#=> ** (QuackDB.Error) Interrupted: query was cancelled

Pass query_uuid: uuid to cancel only that query. Transport timeouts and disconnects already cancel their query on the server.

child_spec(options)

@spec child_spec([start_option()]) :: Supervisor.child_spec()

column_batches(connection, statement, params \\ [], options \\ [])

@spec column_batches(DBConnection.conn(), iodata(), [term()], Keyword.t()) ::
  Enumerable.t()

Streams query results as column-oriented batches.

Each item is a map from disambiguated column names to the values in that fetch batch. This keeps large analytical results vector-shaped without materializing the whole result set. Prefer columnar_batches/4 when you also need batch metadata.

columnar(connection, statement, params \\ [], options \\ [])

@spec columnar(DBConnection.conn(), iodata(), [term()], Keyword.t()) ::
  {:ok, QuackDB.Columns.t()} | {:error, Exception.t()}

Runs a query and returns a QuackDB.Columns struct.

This preserves column order, original names, row count, and result metadata in addition to the column vectors.

columnar!(connection, statement, params \\ [], options \\ [])

@spec columnar!(DBConnection.conn(), iodata(), [term()], Keyword.t()) ::
  QuackDB.Columns.t()

columnar_batches(connection, statement, params \\ [], options \\ [])

@spec columnar_batches(DBConnection.conn(), iodata(), [term()], Keyword.t()) ::
  Enumerable.t()

Streams query results as QuackDB.Columns batches.

This uses a columnar cursor path so large analytical results can stay vector-shaped instead of being materialized as row lists first.

columns(connection, statement, params \\ [], options \\ [])

@spec columns(DBConnection.conn(), iodata(), [term()], Keyword.t()) ::
  {:ok, %{required(String.t()) => [term()]}} | {:error, Exception.t()}

Runs a query and returns its result as a column-oriented map.

Duplicate column names are disambiguated with suffixes such as _2 and _3. Prefer columnar/4 when you also need column order and result metadata.

columns!(connection, statement, params \\ [], options \\ [])

@spec columns!(DBConnection.conn(), iodata(), [term()], Keyword.t()) :: %{
  required(String.t()) => [term()]
}

insert_columns(connection, table, columns, options \\ [])

@spec insert_columns(
  DBConnection.conn(),
  String.t() | atom(),
  [insert_column()],
  Keyword.t()
) ::
  {:ok, QuackDB.Result.t()} | {:error, Exception.t()}

Appends column-oriented values to a DuckDB table by streaming binary vectors into an INSERT.

Column values are provided as {name, values} pairs. All columns must have the same row count. Pass :columns with type specs when values are empty, contain only nils, or need a specific nested DuckDB type. Explicit MAP columns accept ordinary Elixir maps and DuckDB-style key/value entries.

insert_columns!(connection, table, columns, options \\ [])

@spec insert_columns!(
  DBConnection.conn(),
  String.t() | atom(),
  [insert_column()],
  Keyword.t()
) ::
  QuackDB.Result.t()

insert_rows(connection, table, rows, options \\ [])

@spec insert_rows(
  DBConnection.conn(),
  String.t() | atom(),
  [insert_row()],
  Keyword.t()
) ::
  {:ok, QuackDB.Result.t()} | {:error, Exception.t()}

Appends row-oriented values to a DuckDB table by streaming binary vectors into an INSERT.

Rows are maps or keywords. Without :columns, every row's keys name the columns and every value takes part in a column's type: integers widen to BIGINT or HUGEINT, integers and floats make DOUBLE, decimals and integers make one DECIMAL, and maps make a STRUCT of all their keys. Any other mix, such as integers and strings, fails with :mixed_append_types instead of truncating or dropping values. Pass :columns with type specs when values are empty, contain only nils, mix shapes (:variant keeps them), or need a specific DuckDB type.

Plain Elixir maps infer as DuckDB STRUCT values. For explicit {:map, key_type, value_type} columns, ordinary Elixir maps are encoded as DuckDB MAP values:

QuackDB.insert_rows!(conn, "events", [[labels: %{env: "prod"}]],
  columns: [labels: {:map, :varchar, :varchar}]
)

DuckDB-style key/value entries are also accepted for explicit MAP columns:

QuackDB.insert_rows!(conn, "events", [[labels: [%{key: "env", value: "prod"}]]],
  columns: [labels: {:map, :varchar, :varchar}]
)

Duplicate MAP keys decode with the later entry winning, matching Map.put/3. Keys and values are encoded through the declared DuckDB key/value types; for example, atom keys in {:map, :varchar, :varchar} columns become strings.

Declare VARIANT columns as :variant, so each value keeps its own shape, and JSON columns as :json.

Options:

  • :columns - [{name, type}] column types, also naming the inserted columns. Omitted table columns get their defaults.
  • :schema - the table's schema, or a nested schema path as a list.
  • :batch_size - rows per request; requests carry vectors of at most 2048 rows.
  • :on_conflict - :raise (default), :nothing, or {:nothing, targets}.
  • :returning - columns to return; the result then carries those rows.

insert_rows!(connection, table, rows, options \\ [])

@spec insert_rows!(
  DBConnection.conn(),
  String.t() | atom(),
  [insert_row()],
  Keyword.t()
) ::
  QuackDB.Result.t()

insert_stream(connection, table, rows, options \\ [])

@spec insert_stream(
  DBConnection.conn(),
  String.t() | atom(),
  Enumerable.t(),
  Keyword.t()
) ::
  {:ok, QuackDB.Result.t()} | {:error, Exception.t()}

Appends an enumerable of row maps/keywords in batches through native append.

insert_stream!(connection, table, rows, options \\ [])

@spec insert_stream!(
  DBConnection.conn(),
  String.t() | atom(),
  Enumerable.t(),
  Keyword.t()
) ::
  QuackDB.Result.t() | nil

insert_table(connection, table, tabular, options \\ [])

Appends any Table.Reader compatible tabular data through native append.

insert_table!(connection, table, tabular, options \\ [])

maps(connection, statement, params \\ [], options \\ [])

@spec maps(DBConnection.conn(), iodata(), [term()], Keyword.t()) :: Enumerable.t()

ping(connection, options \\ [])

@spec ping(DBConnection.conn(), Keyword.t()) :: :ok | {:error, Exception.t()}

prepare(connection, statement, options \\ [])

@spec prepare(DBConnection.conn(), iodata(), Keyword.t()) ::
  {:ok, QuackDB.Query.t()} | {:error, Exception.t()}

prepare_execute(connection, statement, params \\ [], options \\ [])

@spec prepare_execute(DBConnection.conn(), iodata(), [term()], Keyword.t()) ::
  {:ok, QuackDB.Query.t(), QuackDB.Result.t()} | {:error, Exception.t()}

query(connection, statement, params \\ [], options \\ [])

@spec query(DBConnection.conn(), iodata(), [term()], Keyword.t()) ::
  {:ok, QuackDB.Result.t()} | {:error, Exception.t()}

query!(connection, statement, params \\ [], options \\ [])

@spec query!(DBConnection.conn(), iodata(), [term()], Keyword.t()) ::
  QuackDB.Result.t()

rows(connection, statement, params \\ [], options \\ [])

@spec rows(DBConnection.conn(), iodata(), [term()], Keyword.t()) :: Enumerable.t()

start_link(options)

@spec start_link([start_option()]) :: GenServer.on_start()

stream(connection, statement, params \\ [], options \\ [])

@spec stream(DBConnection.conn(), iodata(), [term()], Keyword.t()) ::
  QuackDB.Stream.t()