Databricks is one platform where every service reads and writes the same open tables (Delta Lake or Iceberg) in your own cloud storage, governed by one catalog (Unity Catalog). Ingestion, ETL, SQL warehousing, BI, ML, AI agents, apps and an operational Postgres database are separate services on top of that one copy of data, each running on Databricks-managed or serverless compute. Names below reflect the 2026 renames (Delta Sharing is now OpenSharing, Databricks Assistant is now Genie Code, Genie Spaces are now Genie Agents).
At a glance
Because every service sits on the same catalog and tables, a pipeline's output is immediately queryable in SQL, usable for model training and servable to an app, with no copies in between.
Foundation
| Service | What it does | How it works |
|---|---|---|
| Control plane vs compute plane | Splits Databricks' own backend from where your data is processed. | The control plane (web UI, notebooks, job scheduler, cluster manager) runs in Databricks' account. Compute runs either as VMs in your cloud account (classic) or in Databricks' account (serverless). Data stays in your object storage. |
| Workspace | The environment where a team works: notebooks, jobs, dashboards, permissions. | Several workspaces sit under one Databricks account; all of them can attach to the same Unity Catalog metastore so they see the same tables. |
| Delta Lake | The default table format: ACID transactions on files in S3/ADLS/GCS. | Data is stored as Parquet files; a _delta_log folder of JSON commits records which files make up each version. That log gives you transactions, time travel (VERSION AS OF), schema enforcement and MERGE. |
| Apache Iceberg support | Lets the same tables be read by Iceberg engines (Snowflake, Trino, etc.). | Managed Iceberg tables are native; Delta tables can expose Iceberg metadata (UniForm) so either format's readers work on one copy. |
| Databricks Runtime | The software image every cluster runs. | Apache Spark plus Databricks optimisations, libraries and connectors; versioned (e.g., Runtime 19), with ML variants that bundle PyTorch, XGBoost, etc. |
| Predictive optimization | Keeps tables fast without manual tuning. | Runs OPTIMIZE (compaction), VACUUM and statistics collection on managed tables automatically, based on observed usage. |
| Liquid clustering | Replaces partitioning and Z-ordering. | You name clustering keys; Databricks incrementally reorganises files around them so queries skip irrelevant files, and keys can change without rewriting the table. |
Compute
| Service | What it does | How it works |
|---|---|---|
| All-purpose clusters | Interactive compute for notebooks and exploration. | A driver node plus worker VMs running Spark; you choose instance types, autoscaling range and auto-termination. Shared by several users. |
| Job clusters | Compute for one scheduled job run. | Created when the job starts and destroyed when it ends, so you pay only for the run and no one else's work interferes. Cheaper per DBU than all-purpose. |
| Serverless compute | Notebooks, jobs and pipelines with no cluster to configure. | Databricks keeps warm capacity in its own account and attaches it in seconds; it autoscales and you pay per use. |
| SQL warehouses | Compute dedicated to SQL queries and BI tools. | Classic, Pro or Serverless clusters tuned for SQL, with Photon on, result caching and query queuing; scale out by adding clusters as concurrency grows. |
| Photon | Faster query engine under Spark and SQL. | A C++ vectorised engine that replaces parts of Spark's JVM execution for scans, joins and aggregations; no code changes needed. |
| Serverless GPUs / AI Runtime | GPU compute for model training and fine-tuning. | Request GPUs from a notebook or job without provisioning a GPU cluster; billed only while running. |
| Lakehouse//RT | Millisecond-latency, high-concurrency serving on lakehouse tables. | A real-time query tier over your existing open tables, autoscaling to zero when idle, still governed by Unity Catalog, so no separate real-time database is needed. |
| Instance pools & policies | Faster start-up and cost guardrails for classic compute. | Pools keep idle VMs ready to hand to clusters; compute policies restrict what users can create (sizes, tags, runtimes). |
Data engineering (Lakeflow)
| Service | What it does | How it works |
|---|---|---|
| Lakeflow Connect | Managed ingestion from databases and SaaS apps (100+ connectors). | Pick a source (Salesforce, SQL Server, Workday, etc.); Databricks runs a gateway/CDC process that lands changes incrementally into Delta tables, handling schema changes and retries. |
| Zerobus Ingest | Push events directly into tables. | Apps send records to a Databricks endpoint, which writes them straight into Unity Catalog tables without a Kafka hop in between. |
| Auto Loader | Incrementally load new files arriving in cloud storage. | cloudFiles streaming source; tracks which files it has processed (via directory listing or cloud notifications) so each file is ingested exactly once, and infers and evolves schema. |
| Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) | Build batch or streaming ETL by declaring tables, not steps. | You write CREATE STREAMING TABLE / MATERIALIZED VIEW (SQL or Python); the engine works out the dependency graph, runs it in order, applies incremental refresh, enforces data-quality expectations and retries failures. A real-time mode targets millisecond latency. |
| Lakeflow Jobs (formerly Workflows) | Orchestrate notebooks, SQL, pipelines, dbt, Python and more. | A job is a DAG of tasks with dependencies, retries, conditionals and loops; triggered by schedule, file arrival, table update or API. |
| Lakeflow Designer | No-code visual pipeline builder for analysts. | Drag-and-drop or describe the transformation in plain English; it generates a governed pipeline that runs on the same engine as coded pipelines. |
| Structured Streaming | Spark's streaming engine, used under the services above. | Treats a stream as an unbounded table processed in micro-batches (or continuous mode), with checkpoints for exactly-once results. |
| Genie ZeroOps (preview) | Agent that keeps pipelines healthy. | Monitors pipelines and jobs, diagnoses failures and proposes or applies fixes within the permissions you grant. |
Data warehousing & BI
| Service | What it does | How it works |
|---|---|---|
| Databricks SQL | The data warehouse: SQL editor, queries, alerts, BI connectivity. | Runs ANSI SQL on SQL warehouses against Unity Catalog tables; Tableau, Power BI and others connect over JDBC/ODBC. |
| AI/BI Dashboards | Built-in dashboards. | Each visual is backed by a SQL dataset run on a warehouse; dashboards can be published with embedded credentials and scheduled as email snapshots. |
| Genie Agents (formerly Genie Spaces) | Ask questions of your data in plain English. | A curated space of tables, instructions and example queries; an LLM turns the question into SQL, runs it under the asker's permissions and returns a table or chart. |
| Genie One | The business user's front door: a chat "coworker" across data and apps. | One account-level Genie (formerly Databricks One) on web, mobile, Slack, Teams, Excel and Sheets; routes each question to the right Genie Agent, dashboard or document, grounded by the Genie Ontology. |
| Unity Catalog metric views | A governed semantic layer. | Define measures and dimensions once in YAML/SQL; dashboards, Genie and external BI tools all query the same definitions. |
| Query federation (Lakehouse Federation) | Query external databases without copying data. | Register a connection to MySQL, PostgreSQL, Snowflake, BigQuery, etc. as a foreign catalog; queries are pushed down to the source, governed by Unity Catalog. |
Governance & sharing
| Service | What it does | How it works |
|---|---|---|
| Unity Catalog | One place to govern tables, files, models, functions and agents across all workspaces. | A three-level namespace catalog.schema.object; permissions are SQL GRANTs checked by every engine on every access. It also records column-level lineage, audit logs, tags and row/column masks. |
| Volumes | Govern non-tabular files (PDFs, images, CSVs). | A Unity Catalog object pointing to a storage path; accessed as /Volumes/catalog/schema/volume/... with the same permissions as tables. |
| Catalog federation | Govern tables registered in other catalogs (Hive metastore, AWS Glue, Snowflake Horizon). | Mirrors the external catalog's metadata into Unity Catalog so Databricks can query and govern those tables in place. |
| OpenSharing (formerly Delta Sharing) | Share live data and AI assets with anyone, on any platform, without copying. | An open protocol (now a Linux Foundation project): the provider grants a share; the recipient's client gets short-lived signed URLs to read the underlying files directly. Now also covers models, agent skills and Genie Agents, and Iceberg clients. |
| Databricks Marketplace | Find, buy and sell datasets, models, notebooks and apps. | Listings are delivered through OpenSharing, so a purchased dataset appears as a catalog in your workspace. |
| Clean Rooms | Joint analysis with a partner without exposing raw data. | Both parties share tables into an isolated environment; only approved notebooks run there and only their outputs leave. |
| Data quality monitoring | Detect stale, drifting or broken tables. | Profiles tables over time, tracks freshness and statistics, and raises anomalies and alerts. |
| Lakewatch (preview) | Security SIEM built on the lakehouse. | Lands security logs into governed tables and runs AI agents for detection, triage and response. |
Machine learning & AI
| Service | What it does | How it works |
|---|---|---|
| Managed MLflow | Track experiments, register models, trace and evaluate GenAI apps. | Each training run logs parameters, metrics and artifacts; models are registered in Unity Catalog with versions and aliases; MLflow Tracing records every step of an agent call for debugging and evaluation. |
| Feature engineering / feature store | Reuse features consistently in training and serving. | Features are Delta tables with primary keys in Unity Catalog; they are joined automatically at training time and served from online stores (backed by Lakebase) at inference. |
| AutoML | Quick baseline models for classification, regression, forecasting. | Tries several algorithms and hyperparameters, then generates editable notebooks for the best runs. |
| Model Serving | Deploy models as REST endpoints. | Serverless endpoints that autoscale (including to zero); serve custom models, external models (OpenAI, Anthropic) or Foundation Model APIs (Llama, Claude, etc.) behind one interface. |
| Foundation Model APIs | Call hosted LLMs from SQL or Python. | Pay-per-token or provisioned-throughput endpoints; ai_query() and other AI functions call them row by row inside SQL. |
| Vector Search | Similarity search for RAG. | Builds an index from a Delta table, embeds the text, and keeps the index synced automatically as the table changes; queried by endpoint. |
| Agent Bricks | Build, evaluate and run production AI agents. | Choose a task (extraction, knowledge assistant, multi-agent) or bring your own framework (LangGraph, CrewAI, Claude Code SDK, OpenAI Agents SDK); Databricks supplies tools, managed memory on Lakebase, sandboxes for running code, and automatic evaluation/optimisation. |
| Unity AI Gateway | Govern every LLM and agent call. | A proxy in front of model endpoints and tools: rate limits, guardrails (PII, safety), usage tracking and payload logging into tables. |
| AI Playground | Try models and prompts in the browser. | Chat UI to compare LLMs side by side and prototype tool-calling agents, then export the code. |
| AI Functions | LLM tasks as SQL functions. | ai_classify, ai_extract, ai_parse_document, ai_query etc. run an LLM over each row in a query or pipeline. |
Apps & operational data
| Service | What it does | How it works |
|---|---|---|
| Lakebase | Fully managed, serverless Postgres for apps and agents (OLTP). | Postgres with storage separated from compute: instances autoscale to zero, and branching/snapshots take about a second, so dev and AI tools can work on safe copies. Synced tables keep Postgres and lakehouse tables in step; Lakebase Search adds vector and full-text search. |
| LTAP | Transactions and analytics on one copy of data. | Lakebase handles writes while the lakehouse engines read the same open-format data, removing the ETL step between the app database and analytics tables. Rolling out to existing Lakebase customers. |
| Databricks Apps | Host internal web apps and data tools next to your data. | Deploy a Streamlit, Dash, Gradio, Flask, React or Node app; it runs on serverless containers, signs users in with Databricks identity and calls warehouses, Lakebase, Model Serving and Genie with governed permissions. Apps can also be listed on Marketplace. |
Developer tooling & operations
| Service | What it does | How it works |
|---|---|---|
| Notebooks | Interactive Python, SQL, Scala and R. | Cells execute on attached compute; supports real-time co-editing, widgets for parameters, and calling other notebooks. |
| Genie Code (formerly Databricks Assistant) | AI agent for data work inside the workspace. | Lives in notebooks, the SQL editor, the pipelines editor and dashboards; plans and runs multi-step tasks (write ETL, debug a failed job, build a dashboard) using workspace context, within Unity Catalog permissions. |
| Git folders | Version control for notebooks and code. | Clone a GitHub/GitLab/Azure DevOps repo into the workspace; commit, branch and pull from the UI. |
| Databricks Asset Bundles | Deploy jobs, pipelines and apps as code (CI/CD). | A databricks.yml file describes resources per environment (dev/staging/prod); databricks bundle deploy creates or updates them. |
| CLI, SDKs, REST API, Terraform | Automate everything. | Every UI action is a REST call; the CLI, Python/Java/Go SDKs and the Terraform provider wrap it. |
| Databricks Connect | Run code from your own IDE against Databricks compute. | Your local Spark session sends the query plan to a remote cluster or serverless compute and gets results back. |
| System tables | Observability on usage, cost, lineage and audit. | Read-only tables under system.* (e.g., system.billing.usage, system.access.audit) that you query with SQL. |
| Budgets & cost controls | Track and cap spend. | Budgets on tagged usage with email alerts; per-user limits for products such as Genie One. |
| Free Edition | Learn and build at no cost. | A serverless-only workspace for individuals; now includes Genie Code, serverless GPUs, Lakebase, Agent Bricks and Lakeflow Designer. |
Sources
Current product names and 2026 launches were checked against these pages; the "how it works" descriptions are from general knowledge of the platform.