Write sequential SQL. Declare a schedule in the same file. Commit to Git and DeltaForge discovers the pipeline, extracts lineage, and runs it on cron. No DAGs. No YAML. No orchestration code.
The scheduling contract lives at the top; the work lives below
Pipelines carry their schedule, timeouts, retries, SLA target, fail-fast behavior, and notification routing in the same SQL file as the work. No separate YAML, no orchestrator config to keep in sync.
Production runs can require an explicit approval. The gate resets the moment the source file changes, so a review is always against the version that will run.
The body is just SQL. CREATE, INSERT, MERGE, UPDATE, DELETE, procedure calls, whatever the engine understands. The pipeline runs them in order. The house style is idempotent SQL: a retry or a re-run lands the same result as the first run.
Each workspace links to one Git repository; pipelines are SQL files in that repo
DeltaForge scans the linked repository for .sql files that contain a PIPELINE declaration. No manifest, no registration step.
Pipeline changes are commit history. Each scheduled run records the Git SHA it executed so results can be traced to a specific version.
Open the workspace in VS Code or the built-in Pipeline IDE. Commit, push, pull, and branch without leaving the editor.
Table-level lineage is derived from the SQL statements. No annotation needed. See what each pipeline reads and writes from the catalog.
Git-first SQL pipelines, in practice
No. Pipelines are sequential SQL, and the scheduling contract lives in the same file as the work. Where notebook-based orchestration scatters logic across cells and a separate scheduler, here a code review of one SQL file shows exactly what runs and when.
Each workspace links to one Git repository. DeltaForge scans the linked repo for .sql files containing a PIPELINE declaration. Commit to Git and the pipeline becomes visible on the next scan; no manifest, no registration step.
A pipeline can require an explicit approval before a production run. The gate resets the moment the source file changes, so the review is always against the exact version that will execute.
Typical pipeline workloads on this engine: SCD Type 2 on Delta Lake in pure SQL, incremental processing with Change Data Feed, and scheduled OPTIMIZE, VACUUM, and Z-ORDER maintenance.
Write the SQL you know, declare when to run it, commit to Git.