Dataform CLI Reference

This document is a complete reference for the dataform command line interface. The command list, usage lines and argument/option tables are generated from the CLI source in this repository (cli/), so they match dataform help. It also covers practical example use cases.

For conceptual documentation (project structure, SQLX syntax, the JavaScript API, etc.) see the root readme and the hosted Dataform CLI guide.

Installation

npm i -g @dataform/cli

Invocation

dataform <command> [project-dir] [options]

project-dir is optional for every command and defaults to . (the current directory). It is resolved with tilde-expansion, so ~/my-project works. project-dir can be passed either positionally (dataform compile my-project) or as a named flag (dataform compile --project-dir=my-project).

Run dataform with no arguments, or dataform help, to print the list of commands. Run dataform help <command> or dataform <command> --help for a command's full flag list.

Global behavior

• --help — shows help for the current command.
• --version — prints the installed @dataform/cli version number.
• Strict parsing — unknown commands or flags are rejected with an error, and the CLI suggests the closest matching command name for typos.
• Colored output — output is colorized by default. Set NO_COLOR=1 (or true/yes) to disable it.
• Lineage debug logging — set DATAFORM_LINEAGE_DEBUG=1 to enable verbose logging from the OpenLineage emitter used by run --emit-lineage.

Commands

• init
• install
• init-creds
• compile
• test
• run
• format

init

dataform init [project-dir] [default-database] [default-location]

Creates a new Dataform project (workflow_settings.yaml, .gitignore, definitions/, and includes/) in project-dir. Fails if workflow_settings.yaml, package.json, or dataform.json already exists in project-dir.

Positional arguments

project-dir is the directory to create the project in. All three positional arguments can also be passed as named flags (--project-dir, --default-database, --default-location), e.g. dataform init my_project --default-database=my-gcp-project-id --default-location=US.

Options

When --iceberg is set, init prompts interactively for an Iceberg bucket name, table folder root/subpath, and connection.

install

dataform install [project-dir]

Installs a project's NPM dependencies (via npm i --ignore-scripts) as declared in the project's package.json.

Positional arguments

Options

None.

Note: for projects that specify dataformCoreVersion in workflow_settings.yaml, @dataform/core is installed automatically in an isolated temporary directory at compile/run time, and running dataform install throws an error (No installation is needed when using workflow_settings.yaml, as packages are installed at runtime.). Use dataform install only when managing @dataform/core and external dependencies via package.json (without dataformCoreVersion in workflow_settings.yaml).

init-creds

dataform init-creds [project-dir]

Interactively creates a .df-credentials.json file in project-dir that Dataform uses to connect to BigQuery. Only BigQuery is currently supported.

Prompts for:

1. Dataset location (US, EU, a custom region name, or auto). Choosing auto leaves location out of the credentials file so BigQuery chooses the location dynamically.
2. Authentication method — Application Default Credentials (ADC) or a JSON service account key file.
   • ADC: prompts for your billing project ID.
   • JSON key: prompts for the path to a downloaded service account key file.
3. A service account to impersonate (optional, press Enter to skip). It's saved as impersonateServiceAccount, see Credentials file.

Positional arguments

Options

compile

dataform compile [project-dir]

Compiles the project and prints a summary (or JSON / Graphviz DOT representation) of the compiled dependency graph without executing it against your warehouse. Useful for validating a project and inspecting its structure.

Positional arguments

Options

Notes

• --json outputs the compiled graph and can't be combined with --dot.
• --watch ignores node_modules, recompiles with a debounce of 500ms, and runs until interrupted with Ctrl+C.
• --output-actions, --output-tags, --output-include-deps and --output-include-dependents only prune what's printed; the whole project still compiles. --output-actions entries are matched exactly (no * wildcard expansion).

test

dataform test [project-dir]

Compiles the project and runs its unit tests (type: "test" actions) against BigQuery.

Positional arguments

Options

--json outputs the test results as JSON.

Exits with code 1 if any test fails, if the project fails to compile, or if no unit tests are found. Exits 0 if all tests pass.

run

dataform run [project-dir]

Compiles and executes the project against BigQuery (or validates it with --dry-run).

Positional arguments

Options

Notes

• --actions entries are matched exactly: * wildcards are not expanded, and a bare action name that matches several actions is an error.
• --full-refresh only affects incremental tables; regular tables and views are always rebuilt.
• --json outputs the run result as JSON and is only supported together with --dry-run; plain run --json is rejected.
• --timeout bounds compilation only, not execution. If it's set without --execution-timeout, run prints a warning reminding you of that.
• --job-labels override defaultJobLabels, action jobLabels and table labels that use the same key.
• --emit-lineage requires the Data Lineage API (datalineage.googleapis.com) to be enabled and the roles/datalineage.producer role; for configuration and troubleshooting, see the Data Lineage guide.

Exit codes: 0 on success, 1 if compilation fails, a BigQuery job would get more than 64 labels (checked before any action or unit test runs), unit tests fail (with --run-tests), the run fails, times out, or is cancelled (Ctrl+C).

format

dataform format [project-dir]

Formats .sqlx and .js files in the project using Dataform's built-in formatter.

Positional arguments

Options

Notes

• Despite the shared flag name, --actions here takes file globs relative to project-dir, not action names. With --ignore-js-files, the default becomes {definitions,includes}/**/*.sqlx.
• --check exits with code 1 if any file would be reformatted.

Shared project-config flags

compile, test, and run all accept the following flags, which override the equivalent settings in the project's workflow_settings.yaml for that invocation only:

Credentials file

init-creds writes a .df-credentials.json file (filename fixed, not configurable) into the project directory. This file stores the connection details Dataform needs to reach your warehouse:

• ADC-based: { "projectId": ..., "location": ... }
• JSON-key-based: { "projectId": ..., "credentials": "<service account key JSON as a string>", "location": ... }

Additional details:

• Location is optional in both of the above (init-creds omits it if auto is chosen)
• Service account impersonation (optional, with either of the above): add "impersonateServiceAccount": "<service account email>" to run BigQuery queries (and lineage emission) as that service account. Your ADC identity, or the key above, is used to impersonate it and needs roles/iam.serviceAccountTokenCreator on that service account. init-creds asks for it as an optional last step.

Only BigQuery is supported by init-creds and the bundled BigQuery adapter in this codebase. Treat .df-credentials.json as a secret — dataform init adds it to .gitignore automatically; never commit it.

Example use cases

First-time project setup

dataform init my_project my-gcp-project-id US
cd my_project
dataform init-creds
dataform run

Compiling and inspecting a project without touching the warehouse

# Human-readable summary
dataform compile

# Full compiled graph as JSON, e.g. to pipe into jq
dataform compile --json | jq '.tables[].target'

# Visualize the dependency graph
dataform compile --dot > graph.dot && dot -Tpng graph.dot -o graph.png

# Only show a subset of the graph (by unambiguous action name or full database.schema.name target)
dataform compile --output-actions=my_table --output-include-deps
dataform compile --output-actions=my-gcp-project-id.my_schema.my_table --output-include-deps

Running unit tests before deploying changes

dataform test
# ...or fail a run early if tests don't pass:
dataform run --run-tests

Running only part of a pipeline

# Run a specific table (by bare name or full database.schema.name) and anything downstream of it
dataform run --actions=my_table --include-dependents
dataform run --actions=my-gcp-project-id.my_schema.my_table --include-dependents

# Run everything tagged "daily", including their dependencies
dataform run --tags=daily --include-deps

# Force a full rebuild of the "my_incremental_table" incremental table
dataform run --actions=my_incremental_table --full-refresh

Validating a run without applying changes

dataform run --dry-run --json > dry-run-result.json

Bounding how long a run is allowed to take

# Compilation must finish within 30s; the whole run (compile + execute) within 15 minutes
dataform run --timeout=30s --execution-timeout=15m

# Cap each JiT-compiled model's own compilation step at 1 minute
dataform run --execution-timeout=15m --jit-timeout=1m

Targeting a different environment for one invocation

dataform run \
  --default-database=my-staging-project \
  --schema-suffix=staging \
  --vars=env=staging,region=eu

Iterating on SQLX during development

dataform compile --watch

Checking formatting without modifying files

dataform format --check

Tracking and labeling jobs from a run

dataform run --job-prefix=nightly --job-labels=team=analytics,pipeline=nightly-etl

Emitting lineage events for a run

DATAFORM_LINEAGE_DEBUG=1 dataform run --emit-lineage

Lineage events are emitted for every action that runs a BigQuery job: tables (including views and incremental tables), operations, assertions and property graphs. On COMPLETE events, inputs and outputs are populated for actions that materialize a BigQuery object (table, view, incremental, assertion, propertyGraph, and operations with hasOutput: true). Assertions appear as a process for the assertion view written to the assertions dataset, with the assertion's dependencies reported as inputs. Property graph inputs are populated from the action's dependencies (dependencyTargets), so node and edge tables referenced via ref are reported as inputs while raw table paths in the graph definition are not. Declarations are not emitted. The action type is reported in the gcp_bq_pipelines_job.actionType attribute as TABLE, OPERATIONS, ASSERTION or PROPERTY_GRAPH, matching the values used by Dataform in Google Cloud. For prerequisites, workflow_settings.yaml configuration, and troubleshooting, see the Data Lineage guide.

Lineage events are sent to the location set by defaultLocation in workflow_settings.yaml or, if that is not set, by location in .df-credentials.json. If neither is set, lineage emission is skipped with a warning and the run proceeds normally.