GitOps & Deployments

This page documents how built images become running workloads. The nexa-deployments repo is the single GitOps source of truth for the DataReadyAI-operated clusters: ArgoCD watches it, Kustomize renders per-environment manifests, and argocd-image-updater writes new image tags back into git. It picks up where CI/CD leaves off (an image pushed to ECR).

The stack is AWS EKS + ArgoCD + Kustomize + Envoy Gateway (Gateway API) + cert-manager + Secrets Store CSI. The ECR registry is 930816733209.dkr.ecr.ap-southeast-1.amazonaws.com.

Repository layout

nexa-deployments is a Kustomize base/overlays repository:

• Directorynexa-deployments/

  • Directorybase/ one dir per service (deployment, service, httproute, policies)

    • …
  • Directoryoverlays/

    • Directorydev/ develop branch → dev environment

      • …
    • Directorytest/ release branch → test environment

      • …
    • Directoryprod/ main branch → prod environment

      • …
  • Directoryargocd/

    • Directorybootstrap/ app-of-apps parents (per env)

      • …
    • Directoryapplications/ one Application per service+env

      • …
    • Directoryinfrastructure-applications/

      • …
  • Directoryinfrastructure/ envoy-gateway, cert-manager, image-updater, secrets-store-csi

    • …

Each base/<service>/ holds deployment.yaml, service.yaml, httproute.yaml, an Envoy security-policy.yaml (SecurityPolicy) and traffic-policy.yaml (BackendTrafficPolicy), plus a kustomization.yaml. Overlays patch these per environment and set the image via a kustomize images: block. Base services: nexa-web, nexa-backend, nexa-databricks-api, nexa-agents-api, nexa-orchestration-service, nexa-slackbot-service, nexa-stt-service, nexa-teams-ingress-bridge, and actg-ai. bifrost (the AI gateway) is deployed from an upstream Helm chart, not from base/.

Environments and the branch contract

There are three overlays, each tied to a git branch that ArgoCD tracks as targetRevision:

Overlay	Git branch (targetRevision)	Namespace suffix	Image tag pattern	Sync
dev	develop	-dev	develop-<sha>	automated (prune + selfHeal)
test	release	-test	release-<sha>	automated (prune + selfHeal)
prod	main	-prod	semver v<x.y.z>	manual (prune: false, selfHeal: false)

Namespaces follow a per-service, per-env pattern — web-dev, backend-dev, databricks-dev/databricks-test/databricks-prod, agents-dev, slack-dev, stt-dev, orchestration-dev, and so on. Overlays apply a namePrefix (e.g. dev-) so resource names carry the environment.

Caution

Coverage is not uniform across environments. dev has all services (plus bifrost); test has a subset; prod currently has only nexa-databricks-api and an actg-ai overlay. Production sync is deliberately manual — nexa-databricks-api-prod has automated disabled, so a human triggers the sync.

ArgoCD app-of-apps

Deployment uses the classic app-of-apps pattern (not an ApplicationSet). Under argocd/bootstrap/ a parent Application per environment (app-of-apps-dev.yaml → nexa-apps-dev, -test, -prod) points at the argocd/applications/ directory and uses a glob include to select that environment’s child Applications (e.g. dev includes *-dev.yaml). Each child is a per-service, per-env Application such as nexa-web-dev, nexa-databricks-api-prod, actg-ai-test.

1. Install the parent app-of-apps for the environment (argocd/bootstrap/app-of-apps-dev.yaml).

2. The parent syncs the directory of child Application manifests in argocd/applications/.

3. Each child Application syncs its overlay path (overlays/<env>/<service>) into its destination namespace, creating the namespace if needed (CreateNamespace=true).

4. Envoy Gateway core is installed by separate Applications under argocd/infrastructure-applications/<env>/ with an early sync-wave and ServerSideApply=true.

Dev/test Applications set automated: {prune: true, selfHeal: true} with PrunePropagationPolicy=foreground and PruneLast=true; prod leaves automation off for a manual gate.

Image promotion (argocd-image-updater, git write-back)

New images reach a cluster through argocd-image-updater, configured with annotations on each Application. It watches ECR, picks the newest matching tag, and writes the tag back into git — the write-back method is git, not the ArgoCD API, so every promotion is an auditable commit.

Example annotations (nexa-web-dev):

argocd-image-updater.argoproj.io/image-list: nexa-web=930816733209.dkr.ecr.ap-southeast-1.amazonaws.com/aws-iac-nexa-web
argocd-image-updater.argoproj.io/nexa-web.update-strategy: newest-build
argocd-image-updater.argoproj.io/nexa-web.allow-tags: regexp:^develop-.*$
argocd-image-updater.argoproj.io/nexa-web.kustomize.image-name: nexa-web
argocd-image-updater.argoproj.io/write-back-method: git

• Update strategy is newest-build everywhere it is enabled.
• allow-tags enforces the branch-per-env contract: dev tracks ^develop-.*$, test tracks ^release-.*$.
• The write-back target is a per-app .argocd-source-<app>-<env>.yaml file in the overlay (e.g. overlays/dev/nexa-web/.argocd-source-nexa-web-dev.yaml), which pins kustomize.images.
• Production image-updater is disabled — the nexa-databricks-api-prod annotations are commented out, with a commented-out semver alternative (update-strategy: semver, allow-tags: regexp:^v\d+\.\d+\.\d+$). Prod image bumps are intentional git edits, not automatic.

Image-updater runs via Helm (infrastructure/image-updater/), authenticates to ECR through an ecr-login.sh script using EKS Pod Identity (AWS_REGION: ap-southeast-1, creds cached ~10h), and talks to argocd-server.argocd.svc.cluster.local over gRPC-web.

Ingress: Envoy Gateway

Ingress uses the Kubernetes Gateway API via Envoy Gateway. A single Gateway named nexa-gateway in namespace envoy-gateway-system (fronted by an AWS NLB) has three listeners: http (:80), https (:443, TLS terminate), and a wildcard https-dev-aws (:443 for *.dev.aws.nexa-drai.com). An HTTP→HTTPS RequestRedirect route sends :80 traffic to :443.

Each service attaches an HTTPRoute to nexa-gateway with a path prefix and backend port:

Service	Path prefix	Port
nexa-web	/	3000
nexa-backend	/api	4000
nexa-databricks-api	/	8000
nexa-agents-api	/	8000
nexa-orchestration-service	/orchestration	8003
nexa-stt-service	/stt	8010
nexa-teams-ingress-bridge	/teamsbot	8000
actg-ai	/	9000
bifrost	/	8080

Hostnames in use include dev.nexa-drai.com, test.nexa-drai.com, and the wildcard *.dev.aws.nexa-drai.com. The nexa-web route additionally carries a scanner-block rule set (Exact/PathPrefix matches for /.env, /.git, /wp-, etc.) routed to a direct-response filter.

TLS: cert-manager

Certificates come from cert-manager via ACME / Let’s Encrypt using DNS-01 over Route53 (no HTTP-01 in the current path). Two ClusterIssuers exist — letsencrypt-staging-dns01 and letsencrypt-prod-dns01 — both authenticating to Route53 through EKS Pod Identity (region: ap-southeast-1, a fixed hostedZoneID). The gateway TLS secret nexa-gateway-tls-dev is a wildcard *.nexa-drai.com certificate (issuer letsencrypt-prod-dns01, 90-day duration, rotationPolicy: Always), so it auto-renews. See rotate a certificate for the operational check.

Secrets: Secrets Store CSI + AWS Secrets Manager

Secrets are delivered by the Secrets Store CSI driver (AWS provider). Each service+env overlay has a SecretProviderClass that reads an AWS Secrets Manager secret (via Pod Identity, usePodIdentity: 'true', region: ap-southeast-1), extracts keys, and syncs them into a Kubernetes Opaque Secret that the Deployment consumes with envFrom.

The shared secret nexa-shared-secret-<env> backs nexa-web (keys like PG_*, DATABRICKS_*, NEXA_AUTH_*, GITHUB_TOKEN), while some services read a dedicated secret (e.g. nexa-databricks-api-<env>). To add a key, you add it in both the AWS Secrets Manager secret and the SecretProviderClass objects/secretObjects — there is no cross-namespace secret sharing.