Common Runbooks

Recurring operational procedures for the DataReadyAI platform team running the Nexa clusters. Each runbook assumes you have kubectl access to the target EKS cluster, ArgoCD access, and write access to the relevant repo. These cover the DataReadyAI-operated environments; they are distinct from the customer-facing operations guide.

Background you should have first: images are built and pushed by CI/CD with branch-per-env tags, and rolled out by ArgoCD from nexa-deployments. Most of these procedures are git edits, not kubectl apply — the cluster is reconciled from git.

Deploy a new service

Adding a new containerized service means wiring it into both the build pipeline and the GitOps flow.

1. Add the build pipeline. In the service repo, copy the build-push-ecr.yml template from an existing service (e.g. nexa-slackbot-service). Ensure the ECR repository aws-iac-<service> exists — it is declared in aws-iac (modules/ecr/), so add it there and apply if missing.

2. Grant image pull / cloud identity. Add a Pod Identity association (AWS) or Workload Identity federated credential (Azure) for the service’s ServiceAccount in the IaC repo, scoped to its Secrets Manager / Key Vault secret. See AWS vs Azure.

3. Create the base manifests. In nexa-deployments/base/<service>/, add deployment.yaml, service.yaml, httproute.yaml, security-policy.yaml, traffic-policy.yaml, and kustomization.yaml. Give the HTTPRoute a unique path prefix and the correct backend port.

4. Create the overlay. In nexa-deployments/overlays/dev/<service>/, add kustomization.yaml (with namePrefix: dev- and the ECR images: block), namespace.yaml, service-account.yaml, secret-provider-class.yaml, and the patch files.

5. Register the ArgoCD Application. Add argocd/applications/<service>-dev.yaml with targetRevision: develop, the destination namespace, automated sync, and the argocd-image-updater annotations (update-strategy: newest-build, allow-tags: regexp:^develop-.*$, write-back-method: git). The dev app-of-apps (*-dev.yaml glob) picks it up automatically.

6. Merge and verify. Merge to develop. Confirm the child Application appears and syncs Healthy in ArgoCD, and that the route resolves through the gateway.

Tip

Promote to test/prod by adding the corresponding overlay and argocd/applications/<service>-<env>.yaml. Remember prod sync is manual (automated off) and its image-updater is disabled — prod image bumps are intentional git edits.

Roll back an image

Because image tags are pinned in git (via argocd-image-updater git write-back), a rollback is a git revert, not a kubectl edit.

1. Find the last-good tag. In the overlay, open overlays/<env>/<service>/.argocd-source-<service>-<env>.yaml (or the kustomization.yaml images: block) and look at the git history for the previous develop-<sha> / release-<sha> tag.

2. Pin the previous tag. Edit the newTag (or the .argocd-source kustomize.images entry) back to the last-good tag and commit to the environment’s branch. Do not just delete the newer image.

3. Pause auto-update if needed. If image-updater would immediately re-bump to the bad newest build, temporarily set the app’s allow-tags to pin the specific tag, or disable the updater annotation, until the bad image is removed from ECR.

4. Sync. For dev/test, ArgoCD self-heals to the pinned tag. For prod, trigger the manual sync in ArgoCD.

5. Verify the pods are running the expected digest (kubectl -n <ns> get pods -o jsonpath on the image field) and the service is healthy.

Verify or rotate the TLS certificate

The gateway certificate is a cert-manager-issued wildcard *.nexa-drai.com from Let’s Encrypt via DNS-01 over Route53. It auto-renews (rotationPolicy: Always, ~90-day duration, renews ~30 days before expiry), so routine rotation is automatic — this runbook is for verifying that, or forcing a reissue.

1. Check certificate status:

   kubectl -n envoy-gateway-system get certificate nexa-gateway-tls-dev
   kubectl -n envoy-gateway-system describe certificate nexa-gateway-tls-dev

   READY: True and a future Not After mean it is healthy; no action needed.

2. If issuance is stuck, inspect the CertificateRequest and Order/Challenge:

   kubectl -n envoy-gateway-system get certificaterequest,order,challenge
   kubectl -n envoy-gateway-system describe challenge <name>

   DNS-01 failures usually mean the Route53 permission (Pod Identity for cert-manager) or the hosted zone ID is wrong — check the ClusterIssuer letsencrypt-prod-dns01.

3. Force a reissue (only if you must rotate early) by deleting the Secret so cert-manager re-requests:

   kubectl -n envoy-gateway-system delete secret nexa-gateway-tls-dev

   cert-manager reissues into the same Secret name; the Gateway certificateRefs picks it up.

Caution

Use the letsencrypt-staging-dns01 ClusterIssuer when testing issuance changes — Let’s Encrypt production has rate limits, and a burst of failed prod orders can lock you out. Only switch a Certificate to letsencrypt-prod-dns01 once staging succeeds.

Add a secret key to a service

Secrets are delivered by the Secrets Store CSI driver from AWS Secrets Manager (or Key Vault on Azure). A new environment variable must be added in two places or the pod will not see it.

AWS

1. Add the key/value to the backing AWS Secrets Manager secret (nexa-shared-secret-<env> for shared keys, or the service’s own <service>-<env> secret). Do not commit secret values to git.

2. In the service’s overlay secret-provider-class.yaml, add the key under objects (the JMESPath extraction) and under secretObjects (so it syncs into the k8s Secret consumed via envFrom).

3. Commit the overlay change, let ArgoCD sync, and restart the deployment so the CSI volume remounts:

   kubectl -n <ns> rollout restart deployment/<name>

Azure

1. Add the secret to the service’s Key Vault.

2. Update the overlay SecretProviderClass (Azure provider) objects and secretObjects to include the new key.

3. Commit, let ArgoCD sync, and kubectl rollout restart the deployment so the mounted secret refreshes.

There is no cross-namespace secret sharing — each service reads its own synced Secret. See GitOps & Deployments for the SecretProviderClass structure.

Restart or inspect a workload

Quick reference for the day-to-day checks:

# List a service's pods and their image
kubectl -n <ns> get pods -o wide

# Tail logs
kubectl -n <ns> logs deploy/<name> -f

# Restart (picks up new secret mounts / forces reschedule)
kubectl -n <ns> rollout restart deploy/<name>

# Check the gateway routes
kubectl -n envoy-gateway-system get gateway nexa-gateway
kubectl get httproute -A

Namespaces follow the <service>-<env> pattern — web-dev, backend-dev, databricks-prod, agents-dev, and so on. If a service is unreachable but pods are Healthy, check its HTTPRoute path prefix and the nexa-gateway listener before touching the deployment.