Skip to content
Pipelines

SQL pipelines that run on a schedule

Write sequential SQL. Declare a schedule in the same file. Commit to Git and DeltaForge discovers the pipeline, extracts lineage, and runs it on cron. No DAGs. No YAML. No orchestration code.

Scheduling, retries, SLAs, and approval gates declared inline
Git-backed: one workspace links to one Git repository
SSMS-style execution: highlight a statement, press Run, see results inline
daily_etl.sql PIPELINE my_etl SCHEDULE '0 6 * * *' TIMEZONE 'UTC' RETRIES 3 FAIL_FAST true ; git push Git repo HEAD scan DeltaForge discovers pipeline, extracts lineage schedules on compute worker Delta tables written SSMS-style execution SELECT * FROM bronze.sales 3 rows (42ms) -- results inline highlight + Run, no full pipeline needed

One SQL file is one pipeline

The scheduling contract lives at the top; the work lives below

Declared inline

Pipelines carry their schedule, timeouts, retries, SLA target, fail-fast behavior, and notification routing in the same SQL file as the work. No separate YAML, no orchestrator config to keep in sync.

Approval gates

Production runs can require an explicit approval. The gate resets the moment the source file changes, so a review is always against the version that will run.

Sequential SQL

The body is just SQL. CREATE, INSERT, MERGE, UPDATE, DELETE, procedure calls, whatever the engine understands. The pipeline runs them in order.

Git-backed workspaces

Each workspace links to one Git repository; pipelines are SQL files in that repo

Discovery

DeltaForge scans the linked repository for .sql files that contain a PIPELINE declaration. No manifest, no registration step.

Version history

Pipeline changes are commit history. Each scheduled run records the Git SHA it executed so results can be traced to a specific version.

IDE integration

Open the workspace in VS Code or the built-in Pipeline IDE. Commit, push, pull, and branch without leaving the editor.

Lineage

Table-level lineage is derived from the SQL statements. No annotation needed. See what each pipeline reads and writes from the catalog.

Pipelines are just SQL on a schedule

Write the SQL you know, declare when to run it, commit to Git.