Packages and recommended libraries

A Flight runs your Python with the packages you list in requirements.txt. This page covers how to declare dependencies, how to choose a loading pattern for Flight ingestion, and the two libraries we recommend for the most common workloads.

requirements.txt is plain pip syntax

Pass package specifications one per line, the same as a regular pip requirements file:

duckdb==1.5.3
dlt==1.27.0
httpx==0.28.1
pandas==2.2.3

You can use any version specifier pip supports: ==, >=, ~=, extras (some-package[extra]), and so on.

The one dependency worth special attention is DuckDB: pin it to the version MotherDuck's server ships. Find that version in the MotherDuck release notes, or run a quick query against MotherDuck:

Check the MotherDuck DuckDB version

SELECT
  version();

SQL Editor loading...

Login to connect

The runtime environment

Before main() runs, the runtime installs the packages from requirements.txt into the Flight's Python environment. A few properties of that environment are worth knowing up front:

• Declare every dependency in requirements.txt. Dependencies are installed once, before main() starts; there's no interactive pip step inside the run. To run a tool's command-line interface (dbt, dlt), call its console script with subprocess — for example subprocess.run(["dbt", "build"], check=True). The console scripts are on PATH after install, so you don't need python -m.
• Most system binaries aren't preinstalled. The runtime is a base Debian image with git available, so git+https:// requirements work without extra setup. Other tools, such as ffmpeg or Playwright, aren't present until you install them with apt-get at the start of main() (see Beyond Python).

Runtime limits

A Flight is sized for orchestration and basic processing, not for crunching large tables in the runtime memory. Two limits commonly bite first:

• Definition size. The Flight's source_code is capped at 200 KB, and requirements.txt at 20 KB. Don't embed reference data or large fixtures in the source — load them from object storage or an external URL at run time instead.
• Memory. The runtime has a fixed memory ceiling of 16 GB. Heavy in-memory work can be OOM-killed, often with little in the log. Keep heavy compute in SQL so MotherDuck does the work, process in bounded chunks, and when running dbt lower --threads to cap peak memory. See Monitoring and debugging for the OOM symptom and fix.
• Maximum runtime per run. A single run can execute for up to 1 hour on Lite, or up to 8 hours by default on Business and Enterprise plans. See Availability and plan limits.

warning

CAST(timestamptz AS VARCHAR) renders in the session time zone. The same row hashed on a laptop (local time zone) and in a Flight (UTC) produces different strings, so md5 or row-hash recipes built on string-cast timestamps disagree across environments and can trigger a false full re-import. Pin the session time zone (SET TimeZone = 'UTC';) wherever determinism matters, or hash an epoch value (epoch_ms(ts)) instead of a string cast.

Choose a loading pattern

Flights often start with Python variables: API responses, scraped rows, JSON objects, or files written under /tmp. The slow path is to send one row at a time to MotherDuck. Pick a bulk pattern before the data grows.

Source shape	Use this pattern	Why
A few hundred control rows	Direct INSERT or executemany is acceptable.	The code stays simple and the round-trip overhead is small enough.
API pages already in Python memory	Build batches with PyArrow, Polars, or Pandas, then INSERT INTO ... SELECT from the registered table.	Keeps the load as a bulk operation. PyArrow and Polars give better type control than plain Python objects.
Larger scrape or API pull without cloud storage	Write CSV, Parquet, or a local DuckDB file under /tmp, then load in chunks.	Keeps memory bounded. Parquet is typed and compressed; CSV is easy when you control both write and read. Clean up /tmp at the end of the run.
Files already in S3, or data you want to replay and backfill	Write Parquet to S3 and load with read_parquet() or INSERT INTO ... SELECT.	Best fit for large, partitioned, or shared datasets. It requires cloud credentials, but gives you durable staging and easier retries.
Schema-evolving API or app data	Use dlt[motherduck] and make the loader format explicit with loader_file_format="parquet".	dlt handles state, schema evolution, and merge logic while avoiding row-wise remote inserts.

As a rough rule, direct inserts are only for tiny control tables. For Flight ingestion, aim to flush batches rather than individual rows. Batches in the 10-100 MB range are usually easier to reason about than one huge load, and they leave room for retries, logging, and memory headroom.

tip

If you already have files in object storage, keep them there and let MotherDuck read them. If the data exists only inside the Flight process, batch it locally first; only write to S3 when you need durable staging, replay, backfills, or larger parallel reads.

Recommended libraries

Two libraries cover most of what teams build with Flights.

dltHub's dlt for ingest

dlt, from dltHub, is the recommended Python library for moving data into MotherDuck. It handles schema evolution, incremental loading, retries, and state tracking, and it ships a MotherDuck destination out of the box.

duckdb==1.5.3
dlt[motherduck]==1.27.0

A minimal ingest from a REST API into MotherDuck:

import dlt
import httpx

def main():
    pipeline = dlt.pipeline(
        pipeline_name="github_stars",
        destination="motherduck",
        dataset_name="github",
    )

    response = httpx.get("https://api.github.com/repos/duckdb/duckdb", timeout=30)
    response.raise_for_status()
    pipeline.run(
        [response.json()],
        table_name="repo_stats",
        loader_file_format="parquet",
    )

if __name__ == "__main__":
    main()

Use the MotherDuck destination, not the generic DuckDB destination pointed at md:, for remote MotherDuck loads. The MotherDuck destination uses Parquet and COPY for data loading; the generic DuckDB destination has different defaults. Passing loader_file_format="parquet" in Flight examples makes the intended loading path explicit. See the dlt MotherDuck destination docs for the full setup.

dbt for transformation

dbt with the dbt-duckdb adapter is the recommended way to run transformation graphs against MotherDuck data.

duckdb==1.5.3
dbt-duckdb==1.10.1

Run a dbt project from a Flight:

import os
import subprocess

def main():
    cwd = os.path.dirname(os.path.abspath(__file__))
    subprocess.run(["dbt", "build", "--target", "prod"], cwd=cwd, check=True)

if __name__ == "__main__":
    main()

If your dbt project pulls in dbt packages from git (for example, dbt-utils declared in packages.yml), call dbt deps before dbt build. git is available in the runtime, so no apt-get step is needed:

import subprocess

def main():
    subprocess.run(["dbt", "deps"], check=True)
    subprocess.run(["dbt", "build"], check=True)

if __name__ == "__main__":
    main()

Use a dlt verified source

dlt's verified sources cover many SaaS applications, including Stripe, HubSpot, and Shopify. The usual way to get one is dlt init <source> motherduck, which copies the connector into a local project so you can edit it. That scaffold spans several modules, and a Flight's source_code is a single file, so you can't paste it in.

Install the connector as a dependency instead. The verified-sources repository is a buildable Python package, so pip can install it straight from a GitHub archive:

dlt[motherduck]==1.30.0
dlt-verified-sources @ https://github.com/dlt-hub/verified-sources/archive/3957506893a7da821dbcc6acd51c7ca4475d1f53.tar.gz

The connectors then import from the sources namespace:

from sources.hubspot import hubspot
from sources.shopify_dlt import shopify_source
from sources.stripe_analytics import stripe_source

Four things to know before you rely on this:

• Pin a commit SHA. The package isn't published to PyPI and its tags trail the default branch, so a SHA in the archive URL is the only meaningful pin. master.tar.gz also works, but a Flight reinstalls its dependencies on every run, so an unpinned URL can change the connector between two runs of a Flight you haven't touched. Move the pin forward deliberately instead, and rerun the Flight once on demand after you do.
• Add each source's extra dependencies yourself. The repository declares them in [dependency-groups], which pip doesn't install. Stripe needs stripe, for example, or sources.stripe_analytics fails with ModuleNotFoundError: No module named 'stripe'. HubSpot and Shopify need nothing beyond dlt.
• Expect a heavy install. The package depends on dlt[bigquery, duckdb], which pulls in the Google Cloud libraries on every run. It doesn't include the MotherDuck destination, so declare dlt[motherduck] yourself.
• You get upstream behavior only. A dlt init scaffold exists so you can modify the connector. As a dependency you can't, so if you need to change how a source extracts data, keep it in a local project and run it elsewhere.

The repository's sources/*_pipeline.py demo scripts import as from hubspot import hubspot, which assumes the dlt init layout. Read them for the resource names and arguments, then write your own main() against the sources. import path.

Pass credentials with Flight secrets, and create the secret before the Flight that references it: MD_CREATE_FLIGHT rejects an unknown name with user_secret not found. Because a secret's keys are injected under their bare names with case preserved, naming a key after dlt's own config variable lets dlt resolve it with no glue code in your Flight. For a source's non-secret settings, use the Flight's config map with the same naming:

SOURCES__HUBSPOT__API_KEY
SOURCES__STRIPE_ANALYTICS__STRIPE_SECRET_KEY
SOURCES__SHOPIFY_DLT__PRIVATE_APP_PASSWORD

That bare-name trick is for values dlt itself reads. For anything your own code reads, prefer the namespaced <secret_name>_<KEY> form, which is unambiguous when two secrets define the same key.

dlt attaches its destination database but never creates it, so run CREATE DATABASE IF NOT EXISTS once at the start of main() or the first load fails with Catalog with name "<db>" does not exist.

See the Stripe, HubSpot, and Shopify integration pages for complete Flights built this way.