Each query runs inside a native execution engine that actually executes SQL, not a metadata library you still have to pair with a cluster. Arrow columnar batches flow through a vectorized operator pipeline using all available CPU cores, and the same binary scales from a laptop to a pool of workers. No cluster spin-up, no JVM warmup, no idle overhead.
From SQL text to result set, one coherent native pipeline
SQL is parsed to an AST, type-checked, and optimized: predicate pushdown, partition pruning, projection pruning, join reordering based on collected statistics.
The physical plan is broken into morsels (chunks of partitions). A task scheduler assigns morsels to worker threads, balancing load across cores throughout the query.
Each operator (scan, filter, hash join, hash aggregation, sort) processes a full columnar batch per call. Column-at-a-time evaluation keeps CPU caches hot and enables SIMD-friendly memory layouts.
Sort and hash aggregation spill to disk when data exceeds available memory. Spill is compressed and async so other operators continue in parallel.
Delta Lake-aware cost-based optimizer with file-level statistics
Filters on partition columns eliminate entire directory scans. File-level min/max statistics skip Parquet files that cannot contain matching rows.
The optimizer picks join order based on estimated cardinality from table statistics, reducing intermediate result size.
EXISTS and NOT EXISTS subqueries are converted to semi-joins and anti-joins with early termination, avoiding unnecessary full-table evaluation.
Constant expressions are folded at plan time. Dead predicates (1=1, WHERE false) are eliminated before any data is read.
Single-worker by default; explicit opt-in for workloads that need cross-worker parallelism
Every query runs on one worker. That worker uses all its CPU cores via the morsel scheduler. Most analytical workloads fit comfortably inside a single large VM.
For workloads where a single node is a genuine bottleneck, you can opt a query into distributed execution across multiple workers. Workers are stateless: each one processes its assigned partitions independently and results are gathered by a coordinator phase, so adding or removing a worker never requires data rebalancing. Compute autoscaling grows and shrinks the worker pool to match demand.
Quick answers about single-node execution and when to scale out
Yes. DeltaForge executes Delta Lake queries inside a native single-node engine: Arrow columnar batches flow through a vectorized, morsel-scheduled operator pipeline that uses every CPU core. There is no JVM and no cluster to provision, and the same binary scales out to multiple workers when you opt a query into distributed execution.
Morsel-driven scheduling breaks the physical plan into small chunks of partitions called morsels and hands them to a pool of worker threads. The scheduler keeps load balanced across cores throughout the query, so a slow file or skewed partition does not stall the rest of the pipeline.
Most analytical workloads fit comfortably inside one large VM. A single worker that uses every core via the morsel scheduler removes coordinator chatter, cross-node shuffles, and cluster spin-up. Distributed execution is offered as opt-in.
Partition predicates eliminate entire directory scans. File-level min/max statistics from the Delta transaction log skip Parquet files that cannot contain matching rows. Join order is picked from estimated cardinality.
Yes. Sort and hash aggregation spill to disk when data exceeds available memory. Spill is compressed and async, so other operators in the pipeline keep running in parallel.
Opt in when a single large VM is provably the bottleneck: scan volumes that exceed a single node IO ceiling, or joins whose intermediate results exceed local memory after spill.
Hands-on guides that run on this engine
Load raw CSV into a Delta table with plain SQL, executed end to end by this engine on a single node.
Row-level writes against Delta tables through the same vectorized pipeline, with no cluster to provision.
Keep file sizes and clustering healthy so the statistics-based pruning described above stays effective.
A native execution engine that keeps all your cores busy on every query.