# DeltaForge

> DeltaForge is a native SQL engine for Delta Lake and Apache Iceberg. It runs standard PostgreSQL-compatible SQL (CREATE TABLE, INSERT, UPDATE, DELETE, MERGE, time travel) directly on open table formats stored in object storage, without Spark, without Hadoop, and without a JVM. It ships as a single binary control plane plus stateless Arrow-vectorised compute workers that run on a laptop, in Kubernetes, or on AWS, Azure, GCP, or on-premise object storage. Data remains in open Parquet tables and transaction metadata on storage you control.

This file is auto-generated by `cargo run -p delta-forge-docs -- generate` from the page registry in `delta-forge-docs/src/gen/discoverability.rs`. It is an index for AI agents and search crawlers: links use canonical https URLs, and descriptions are curated from the marketing page registry. The structured per-command reference (SQL commands, functions, operators, keywords, data types, table properties, CLI flags) lives on the documentation site as machine-readable JSON; the JSON endpoints are listed in the Reference Data section below.

## Product overview

- [Home](https://deltaforge.org/): Top-level positioning, headline capabilities, primary call to action. Start here for a one-page summary.
- [Platform Features: SQL on Delta Lake and Iceberg](https://deltaforge.org/features): Single index of every shipped capability across SQL, storage, compute, governance, and connectivity.
- [Delta Lake Without Spark: SQL on Delta Tables](https://deltaforge.org/delta-lake-without-spark): Pillar guide to running full Delta Lake operations as plain SQL without Spark or Databricks: MERGE/UPDATE/DELETE, CSV ingest, change data feed, time travel, SCD2, GDPR deletes, and OPTIMIZE/VACUUM/Z-ORDER, each linked to a runnable tutorial.
- [Lakehouse Architecture Without a JVM](https://deltaforge.org/architecture): How the control plane, compute workers, and storage layer fit together; deployment topologies for laptop, Docker Compose, Kubernetes, and managed cloud.
- [Self-Hosted Data Platform Pricing | DeltaForge Lakehouse Engine](https://deltaforge.org/pricing): Tier comparison with included DFCU compute units, support level, and per-tier limits.
- [Install DeltaForge | Desktop App, CLI, MCP Server, Cloud Deployment](https://deltaforge.org/install): Install paths for macOS (Homebrew), Windows (winget, Scoop), and Linux. Releases include GPG signatures and SHA256 manifests for manual verification.

## Core engine

- [Delta Lake SQL Engine: Full DML, No Spark](https://deltaforge.org/sql-engine): PostgreSQL-style SQL surface, PL/pgSQL-style procedures, 200+ built-in functions, window functions with frame specifications, cost-based optimiser.
- [Single Node Query Engine for Delta Lake](https://deltaforge.org/compute-engine): Apache Arrow vectorised execution, SIMD acceleration, parallel task scheduler, spill-to-disk memory management.
- [Lakehouse Control Plane: Catalog, RBAC, One Binary](https://deltaforge.org/control-plane): Catalog, RBAC, credential vault, resource pools, scheduled jobs, audit log, multi-tenant isolation.
- [Lakehouse Compute Autoscaling: Stateless Workers](https://deltaforge.org/compute-autoscaling): Automatic compute scaling that adjusts capacity to match query demand. Scale to zero when idle, keep a warm baseline for instant response, pay only for compute that runs, on your own cloud.
- [One Engine That Reads and Writes Delta and Iceberg](https://deltaforge.org/table-format): Delta Lake reader V1-V3 / writer V2-V7 coverage, deletion vectors, time travel, schema evolution, change data feed, OPTIMIZE, VACUUM, Z-ORDER.
- [Apache Iceberg SQL Engine: No Spark Needed](https://deltaforge.org/iceberg): Native Apache Iceberg read and write through the same SQL surface; UniForm interoperability with Delta tables.

## Specialised capabilities

- [Graph Analytics on the Data Lake: Cypher on Delta Tables](https://deltaforge.org/graph): 14 graph algorithms (PageRank, community detection, shortest path, centrality, similarity) callable as SQL, results returned as columnar Arrow data.
- [GDPR Pseudonymisation in SQL on Delta Tables](https://deltaforge.org/pseudonymisation): GDPR-aligned pseudonymisation controls with keyed hashing, AES encryption, redaction, masking, and generalisation; reversible options where configured; selected fields can be transformed before lake writes.
- [H3 Geospatial SQL on Delta Lake, No Databricks](https://deltaforge.org/h3-geospatial): Uber H3 hexagonal indexing plus PostGIS-compatible spatial functions exposed as SQL.
- [Indexes on Delta Lake Tables: PGM, B+ Tree, Bloom](https://deltaforge.org/indexing): Optional PGM, B+ tree, and bloom-filter indexes layered on top of unmodified Delta tables; planner uses them for file skipping and point lookups.
- [CREATE CHART: SVG Charts Straight from SQL](https://deltaforge.org/charting): CREATE CHART renders SVG visualisations directly from a SQL query: ten chart types (BAR, HBAR, LINE, AREA, SCATTER, PIE, HISTOGRAM, HEATMAP, RADAR, CANDLESTICK).
- [SQL UDFs and PL/pgSQL Procedures on Delta Lake](https://deltaforge.org/udf-system): Scalar, aggregate, and table-valued user-defined functions for extending the engine.
- [Query CSV, Excel, XML, EDI & HL7 Files with SQL](https://deltaforge.org/connectors): CREATE ENDPOINT lands REST API responses as managed Delta tables; 20+ source connectors with predicate pushdown.
- [MCP Server for Delta Lake and Iceberg](https://deltaforge.org/mcp-server): Native Model Context Protocol server so Claude, Cursor, Copilot, and any MCP client get typed tools for catalog, lineage, SQL execution, and pipelines under existing RBAC and audit logging.
- [ODBC Driver for Delta Lake and Iceberg | No Databricks](https://deltaforge.org/odbc): Connects Power BI, Tableau, Excel, DBeaver, and Python directly to Delta tables without copying data into a separate SQL database.
- [ADBC Driver for Delta Lake + Power BI Desktop](https://deltaforge.org/adbc): ADBC driver for Delta Lake with a bundled Power Query connector for Power BI Desktop. Arrow record batches handed to the BI tool by reference; measured 13.7x faster than .NET ODBC on a 1M x 22-column scan.
- [ADBC vs ODBC Performance Benchmark for Power BI](https://deltaforge.org/benchmarks-bi-drivers): Reproducible benchmark of the DeltaForge ADBC and ODBC drivers on a 1M-row, 22-column Delta fact table. Per-phase wall-time numbers, one-command reproduction on Linux x86_64.
- [Known issue: Power BI relationship joins on the ADBC connector](https://deltaforge.org/power-bi-adbc-relationship-folding): Known issue and fix: Power BI DirectQuery relationship joins can fail to fold on the ADBC connector (reported as 'different data sources'). This is a Power BI Adbc.DataSource fold-path limitation, not a DeltaForge engine bug; the fix is to route both tables through one shared source query, or use the ODBC connector, which folds the same model.

## Industry formats

- [Healthcare Lakehouse: HL7, FHIR, X12 in SQL](https://deltaforge.org/healthcare): Healthcare landing page covering HL7v2 and FHIR ingestion, transformation, and SQL query.
- [FHIR to SQL: Query FHIR Resources as Delta Tables](https://deltaforge.org/fhir): Fast Healthcare Interoperability Resources support for modern healthcare data exchange.
- [EDI X12 to SQL: 837, 835, 850, 856 in Delta](https://deltaforge.org/edi): X12, EDIFACT, and TRADACOMS parsing and SQL transformation for supply chain and B2B transactions.

## Pipelines, security, developer experience

- [Git-First SQL Pipelines on a Schedule](https://deltaforge.org/pipeline-workflow): Git-first SQL pipelines, cron scheduling, approval gates, statement-level execution history, lineage tracking.
- [Row-Level Security, RBAC and Masking for Delta Lake](https://deltaforge.org/security): RBAC with row-level security and column masking, centralised credential vault, and audit logging designed to support SOC 2, HIPAA, and GDPR evidence.
- [Why DeltaForge | The Lakehouse Without Spark](https://deltaforge.org/why): Why DeltaForge exists: most lakehouse jobs do not need distributed coordination. Native agents run directly on Delta Lake and Apache Iceberg, scaling across workers only when a job actually needs it.
- [DeltaForge for Developers | A Lakehouse on Your Laptop](https://deltaforge.org/why-developers): DeltaForge for developers: single binary, standard SQL, ODBC, JDBC, Python, R, and .NET drivers, MCP server for Claude, Cursor, and Copilot, same engine on laptop and in production.
- [DeltaForge for Executives | Data Sovereignty and Cost Control](https://deltaforge.org/why-executives): DeltaForge for executives: direct lakehouse access for BI tools, active-compute pricing instead of cluster uptime, public conformance against Apache Spark, customer-installed in your cloud or on-prem.

## Compatibility evidence

- [Spark Conformance Suite for Delta Lake and Iceberg](https://deltaforge.org/conformance): Round-trip conformance suite against Apache Spark across Delta Lake and Apache Iceberg. DeltaForge writes, Spark reads; Spark writes, DeltaForge reads. Verdicts refreshed each release.
- [Read Spark-Written Delta and Iceberg Tables](https://deltaforge.org/conformance-read): Read-direction conformance: Apache Spark writes Delta Lake or native Iceberg, DeltaForge reads. Thousands of hand-written scripts assert row count, named cell values, aggregates, and schema shape.
- [Delta and Iceberg Writes Spark Can Read](https://deltaforge.org/conformance-write): Write-direction conformance: DeltaForge writes Delta Lake and Iceberg UniForm, Apache Spark reads. Row count, content hash, and schema hash must all agree for a test to pass.
- [Concurrent Delta and Iceberg Chaos Test Plan](https://deltaforge.org/conformance-chaos): Chaos-engineering plan: SIGKILL mid-commit, concurrent writers, network partitions, disk-full, clock skew, checkpoint corruption. Atomicity, isolation, and durability invariants verified by both engines.

## Articles

- [Delta Lake and Iceberg Tutorials, No Spark Required](https://deltaforge.org/blog/): Step-by-step tutorials for Delta Lake and Apache Iceberg in SQL: DML, MERGE, maintenance, EDI, HL7, FHIR, and graph analytics. Each tutorial ships with a runnable demo.
- [B+ Tree Indexes for Fast Delta Lake Updates](https://deltaforge.org/blog/btree-index-delta-lake-fast-updates): Create a B+ tree index on a Delta Lake table so targeted UPDATE, DELETE and MERGE statements can locate rows without scanning the table.
- [Cannot Time Travel Delta Table to Version X: The Fix](https://deltaforge.org/blog/cannot-time-travel-delta-table-version-fix): Seeing 'cannot time travel Delta table to version X'? Learn how VACUUM and log retention erase old versions, the retention math, and how to fix it with SQL.
- [Community Detection in SQL: Louvain on Delta Lake](https://deltaforge.org/blog/community-detection-in-sql-louvain-on-delta-lake): Run community detection in SQL with DeltaForge: Louvain, label propagation, and Leiden over Delta Lake tables, no graph database or Spark cluster required.
- [Composite PGM Indexes for Delta Lake](https://deltaforge.org/blog/composite-pgm-index-delta-lake): Create a compact learned PGM index across multiple Delta Lake columns for fast equality and range lookups on the indexed prefix.
- [CSV to Delta Lake Without Spark: Two SQL Statements](https://deltaforge.org/blog/csv-to-delta-lake-without-spark): Convert CSV to Delta Lake without Spark: one CREATE EXTERNAL TABLE with schema inference, one CTAS. Pure SQL on DeltaForge, plus a CSV to Iceberg path.
- [Read and Write Delta Lake and Iceberg from One Engine](https://deltaforge.org/blog/delta-and-iceberg-one-engine): Use one self-hosted SQL engine to read and write Delta Lake and Apache Iceberg tables without Spark, a JVM, or separate query stacks.
- [Delta Lake Change Data Feed in SQL: table_changes() End to End](https://deltaforge.org/blog/delta-lake-change-data-feed-sql-tutorial): Delta Lake change data feed tutorial in plain SQL: enable CDF, query table_changes(), read _change_type rows, and build an incremental pipeline without Spark.
- [GDPR Right to Be Forgotten on Delta Lake, in SQL](https://deltaforge.org/blog/delta-lake-gdpr-right-to-be-forgotten-sql): Implement the Delta Lake GDPR right to be forgotten in SQL: erase PII with UPDATE, audit via DESCRIBE HISTORY, then VACUUM so time travel cannot recover it.
- [Reduce Delta Lake Cloud Metadata Overhead with Meta Store](https://deltaforge.org/blog/delta-lake-metastore-cloud-metadata): Mirror Delta Lake logs, file metadata, row groups and Parquet statistics into PostgreSQL so cloud queries can plan without repeatedly walking object storage.
- [OPTIMIZE, VACUUM and Z-ORDER Without Spark](https://deltaforge.org/blog/delta-lake-optimize-vacuum-zorder-maintenance-guide): Run OPTIMIZE and VACUUM on a Delta table without Spark. A SQL runbook covering ordering, retention, Z-ORDER outside Databricks, and the small files problem.
- [Delta Lake vs Iceberg: A Neutral Comparison](https://deltaforge.org/blog/delta-lake-vs-iceberg): Delta Lake vs Iceberg: log vs metadata tree, hidden partitioning, deletion vectors, Z-ORDER, catalogs, merge-on-read, and a clear guide on how to choose.
- [Can DeltaForge Handle Extremely Large Tables?](https://deltaforge.org/blog/deltaforge-at-scale-table-size-performance): Yes, with caveats. DeltaForge streams from object storage with no single-machine memory ceiling. The real challenges at extreme scale are file count and Delta log maintenance, not the engine itself.
- [FHIR Analytics Without Spark: A Lighter Stack for Health Data](https://deltaforge.org/blog/fhir-analytics-without-spark): FHIR analytics without Spark: query bulk export NDJSON, join Patient and Observation resources in SQL, and land curated results in Delta Lake tables.
- [GPS Fleet Tracking on Delta Lake with H3 SQL](https://deltaforge.org/blog/gps-fleet-tracking-delta-lake-h3-sql): Index GPS pings with H3, map them to service regions, and query fleet activity directly on Delta Lake with SQL.
- [HL7 to SQL: Query HL7v2 Without an Interface Engine](https://deltaforge.org/blog/hl7-to-sql-without-an-interface-engine): HL7 to SQL without an interface engine: create external tables over raw HL7v2 ADT and ORU files and query patient, visit, and lab data with plain SQL.
- [Iceberg INSERT, UPDATE, DELETE and MERGE in SQL, No Spark](https://deltaforge.org/blog/iceberg-insert-update-delete-merge-sql-without-spark): Run Iceberg UPDATE, DELETE and MERGE without Spark. DeltaForge executes full Iceberg DML in plain SQL, on-prem or any cloud, with a worked MERGE INTO example.
- [MERGE, UPDATE and DELETE on Delta Lake Without Spark](https://deltaforge.org/blog/merge-update-delete-delta-lake-without-spark): Delta Lake MERGE, UPDATE and DELETE without Spark: validated SQL upserts, copy-on-write internals, and a runnable demo, with no JVM cluster or Python glue.
- [How to Connect Power BI to Delta Lake Without Databricks or Spark](https://deltaforge.org/blog/power-bi-delta-lake-without-databricks): Connect Power BI to Delta Lake without Databricks or Spark. Use Power Query for small imports, or native ODBC for DirectQuery, pushdown, and transaction-log-correct reads.
- [Serve Power BI Slicers from Memory with a DIM Index](https://deltaforge.org/blog/power-bi-dim-index-delta-lake): Create a DIM index on Delta Lake columns so Power BI DirectQuery slicers and distinct-value probes can be served from memory without scanning Parquet files.
- [Query Delta Lake with SQL, ODBC and ADBC](https://deltaforge.org/blog/query-delta-lake-with-sql-odbc-adbc): Query Delta Lake without Spark through SQL, then connect Power BI, Tableau, Excel, Python, and other clients over ODBC or ADBC.
- [Yes, You Can Query EDI Files with SQL: Here Is How](https://deltaforge.org/blog/query-edi-files-with-sql): Query EDI files with SQL directly from object storage. DeltaForge external tables read X12, EDIFACT, and TRADACOMS: CREATE EXTERNAL TABLE, then SELECT.
- [Query Excel Files with SQL, No OPENROWSET, No Notebook](https://deltaforge.org/blog/query-excel-files-with-sql): Query Excel files with SQL straight from S3 or ADLS: create an external table over xlsx, join across sheets, and load Delta Lake with one CTAS statement.
- [Query a Folder of Files with One SQL Command](https://deltaforge.org/blog/query-file-folder-with-discover-sql): Point the DeltaForge DISCOVER command at a file or folder to detect its format, generate table options, and make it queryable with SQL.
- [Build a Property Graph on Delta Lake Tables](https://deltaforge.org/blog/run-cypher-on-parquet-and-delta-tables-without-neo4j): Build a property graph over Delta Lake tables, query it with Cypher, and join graph results back to SQL without copying data into Neo4j.
- [SCD Type 2 on Delta Lake in Pure SQL: MERGE Tutorial](https://deltaforge.org/blog/scd-type-2-delta-lake-pure-sql): A complete SCD Type 2 Delta Lake SQL example: expire changed dimension rows and insert new versions with MERGE, in pure SQL with no Spark cluster required.
- [Self-Hosted Databricks Alternatives for Open Lakehouse Tables](https://deltaforge.org/blog/self-hosted-databricks-alternatives): Compare self-hosted ways to query and modify Delta Lake and Apache Iceberg tables, including DuckDB, Trino, delta-rs, and DeltaForge.

## Optional

- [Delta Lake SQL Examples: Runnable Demo Gallery](https://deltaforge.org/demos): Hands-on SQL demos covering Delta, Iceberg, graph, healthcare, EDI, geospatial, and 10+ file formats.
- [Contact DeltaForge | Request a Demo, Evaluation, or Support](https://deltaforge.org/contact): Sales, enterprise, and technical support contact form.
- [Partner Resources | DeltaForge Marketing Materials for CSP Partners](https://deltaforge.org/partners): Marketing materials for resellers and Microsoft CSP program partners: approved product descriptions, logos, screenshots, product videos, key selling points, and partner contacts.

## Documentation (HTML reference + machine-readable JSON)

DeltaForge publishes reference pages for SQL commands, functions, operators, keywords, data types, table properties, CLI flags, ODBC settings, GUI actions, graph algorithms, PL/pgSQL constructs, pipeline configuration options, data sources, spatial functions, and bundled demos. Documented entries are available as static HTML pages (SEO-indexed, individually canonical, with TechArticle JSON-LD) and as JSON documents. AI agents should prefer the JSON endpoints; search engines should follow the HTML pages from the docs sitemap.

- [Documentation site root](https://docs.deltaforge.org/): Top-level portal. Interactive React SPA backed by the same content as the static HTML pages.
- [Documentation llms.txt](https://docs.deltaforge.org/llms.txt): The companion AI-agent index for the docs subdomain. Lists every section, every consolidated reference page, and every JSON endpoint.
- [Documentation sitemap](https://docs.deltaforge.org/sitemap.xml): Lists ~1,000 individually crawlable documentation pages.
- [SQL Commands index](https://docs.deltaforge.org/commands/): All 199 SQL commands grouped by category (DDL, DML, catalog, security, pipeline, graph, etc.). Each command links to a dedicated page with syntax, parameters, examples, and pitfalls.
- [Built-in Functions index](https://docs.deltaforge.org/functions/): All 412 functions across math, string, datetime, JSON, regex, geospatial, H3, hash, encoding, and aggregate categories.
- [SQL Keywords reference](https://docs.deltaforge.org/keywords.html): All 470 reserved and non-reserved keywords on a single page.
- [Data Types reference](https://docs.deltaforge.org/data-types.html): All 58 supported types (numeric, string, temporal, complex, binary, boolean).
- [Operators reference](https://docs.deltaforge.org/operators.html): All 40 operators (arithmetic, comparison, logical, JSON, string, array, bitwise).
- [Table Properties reference](https://docs.deltaforge.org/table-properties.html): All 104 Delta table properties (Auto-Optimize, Deletion Vectors, Statistics, UniForm, Constraints, etc.).

Machine-readable JSON (preferred for AI agents):

- [Section navigation](https://docs.deltaforge.org/data/nav.json): Lists every section with categories and entry counts.
- [Search index](https://docs.deltaforge.org/data/search.json): Single JSON array with id, name, section, category, summary for every entry. Use to resolve a name to an id.
- [Per-item JSON example: SELECT](https://docs.deltaforge.org/data/doc/SELECT.json): Each item lives at `https://docs.deltaforge.org/data/doc/<ID>.json`. Fields: id, name, category, summary, syntax, parameters, examples, description, see_also.

## Last Updated

2026-07-17