DuckDB
Overview
DuckDB is an in-process SQL OLAP database management system and this destination is meant to use locally if you have multiple smaller sources such as GitHub repos, some social media and local CSVs or files you want to run analytics workloads on. This destination writes data to the MotherDuck service, or to a file on the local filesystem on the host running Airbyte.
Use with MotherDuck
This DuckDB destination is compatible with MotherDuck.
Specifying a MotherDuck Database
To specify a MotherDuck-hosted database as your
destination, simply provide your database uri with
the normal md: database prefix in the
destination_path configuration option.
We do not recommend providing your API token in
the md: connection string, as this
may cause your token to be printed to execution
logs. Please use the
MotherDuck API Key setting instead.
Authenticating to MotherDuck
For authentication, you can can provide your
MotherDuck Service Credential
as the motherduck_api_key configuration
option.
Sync Overview
Output schema
Each table will contain 3 columns:
-
_airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed. -
_airbyte_emitted_at: a timestamp representing when the event was pulled from the data source. -
_airbyte_data: a json blob representing with the event data.
Features
| Feature | Supported | |
|---|---|---|
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Append + Deduped | No | |
| Namespaces | No |
Performance consideration
This integration will be constrained by the speed at which your filesystem accepts writes.
Troubleshooting
Error message
Request failed: (UNAVAILABLE, RPC
'GET_WELCOME_PACK')
This error may indicate that you are connecting with
a 0.10.x DuckDB client (as per DuckDB
Destination connector versions
>=0.4.0) and your database has not
yet been upgraded to a version
>=0.10.x. To resolve this,
you'll need to manually upgrade your database
or revert to a previous version of the DuckDB
Destination connector.