Snowflake vs Databricks

Nexa runs on either Databricks or Snowflake, chosen by the customer at install time. This page is for engineers who maintain both: it explains which repos are duplicated per data platform, how the runtime execution model differs, and what the shared UI/backend layer hides. The guiding design is parallel implementations behind a common backend — nexa-web and nexa-backend are shared, and everything below the backend has a Databricks and a Snowflake variant.

What is shared vs split

Layer	Shared across platforms?	Notes
UI (nexa-web)	Shared	Exposes both Databricks and Snowflake screens; backend decides which is live
Gateway (nexa-backend)	Shared	Distinct /api/* (Databricks) vs /api/snowflake/* route families; swaps proxy target URL
Platform API	Split	nexa-databricks-api vs nexa-snowflake-api
Agent/automation control plane	Split	nexa-agents-api vs nexa-snowflake-agents-api
Automation runtime	Split	nexa-workflow-automations vs nexa-snowflake-automations
Mapping engine (actg-ai)	Shared (Databricks-oriented)	Currently uses Databricks Vector Search / model serving
Metadata store (Postgres)	Shared	Same nexa-db; platform metadata in platform vs sf_platform schema

The backend abstracts the split: it exposes /api/pipelines for Databricks and /api/snowflake/pipelines for Snowflake, and points NEXA_DATABRICKS_API_URL or SNOWFLAKE_API_URL (and AGENT_API_HOST) at the variant that is deployed.

The three variant pairs

Each pair shares route/module shape and API contracts, differing only in the data-platform SDK and execution model.

Concern	Databricks	Snowflake
Platform data/control API	nexa-databricks-api (Poetry, databricks-sdk)	nexa-snowflake-api (uv, snowflake-connector-python)
Agent/automation control plane	nexa-agents-api (Poetry, Delta)	nexa-snowflake-agents-api (uv, Snowpark/Cortex)
Automation runtime	nexa-workflow-automations (Asset Bundle jobs)	nexa-snowflake-automations (Tasks + stored procs)

Caution

The Snowflake agents/automations repos are near-literal ports of the Databricks originals — matching route filenames, and the nexa-snowflake-agents-api README still self-titles “snowflake_v2”. When you change a contract or fix a bug on one side, check whether the twin needs the same change; drift between the pairs is a recurring source of bugs.

Runtime execution model

This is the deepest difference. Both control planes materialize artifacts to Git, but what CI deploys and how it runs diverges.

Databricks

nexa-workflow-automations is a Databricks Asset Bundle (databricks.yml, bundle: nexa-workflow-automations). Each automation lives in resources/<alias>/ with an automation.yaml, metadata.json, a generated job resource, and templates. The control plane (nexa-agents-api) writes these artifacts into the repo; CI deploys the bundle to Databricks; the automation runs as a Databricks job (notebooks/Python). Execution writes results to Delta, and nexa-agents-api background workers project those Delta rows back into Postgres (schema agents).

Data lives in Unity Catalog / Delta. The platform API uses the databricks-sdk and databricks-sql-connector.

Snowflake

nexa-snowflake-automations deploys automations as Snowflake Tasks backed by Python stored procedures. The control plane writes automation.yaml and renders a deploy.sql into resources/<alias>/; a Git push triggers GitHub Actions, whose deploy_changed_automations.py runs the deploy.sql (CREATE PROCEDURE + CREATE TASK), optionally EXECUTE TASK, then callbacks to the API. Shared stored-procedure tools include TOOL_AGENT_QUERY (Cortex Agent), TOOL_EMAIL_SEND, TOOL_SLACK_SEND, and TOOL_DOCUMENT_GENERATE. Branch→env: develop→dev, release/*→uat, main→prod.

Data lives in Snowflake. The platform API uses snowflake-connector-python / snowflake-sqlalchemy, and the agents API uses Snowpark and Cortex.

Toolchain and build differences

The Databricks and Snowflake services also differ in Python tooling, which matters when you set up a repo locally or read its CI:

Aspect	Databricks side	Snowflake side
Dependency manager	Poetry	uv (uv.lock)
Test/lint gate in CI	ruff + Bandit (no active pytest)	ruff + active pytest with coverage
Image registry	ECR (build-push-ecr.yml)	Snowflake SPCS image registry (build-deploy.yml)
Image tag strategy	branch-per-env (develop-<sha>)	git sha + latest
Security scans in CI	Gitleaks + Trivy + Cosign + SBOM	ruff + pytest only (no Trivy/Gitleaks/Cosign)
Auth in CI	AWS OIDC role	Snowflake key-pair JWT

See CI/CD for the full pipeline breakdown. The Snowflake API path is deliberately the leaner pipeline because it publishes to SPCS rather than ECR.

Infrastructure provisioning

Where the data platform itself comes from depends on the cloud, not just the data-platform choice:

• Databricks on Azure — the workspace is created by nexa-azure-app-iac (stacks/06-databricks); governance (Unity Catalog, warehouses) is applied by nexa-azure-databricks-iac. Storage credentials use the Azure access connector → ADLS; secret scopes use Key Vault.
• Databricks on AWS — the entire E2/MWS workspace is built by drai-customer-databricks-iac (SRA-based): S3 + KMS, PrivateLink, metastore, VPC peering. Storage credentials use IAM role ARNs → S3.
• Snowflake — the data platform is an external SaaS account; there is no in-cloud workspace to provision. The Snowflake API and automations authenticate to the account directly (key-pair JWT) and deploy to SPCS/Tasks.

For the cloud axis of this matrix, see AWS vs Azure.