Troubleshooting & Diagnostics

Use this page when a Nexa service is unhealthy or a request fails. Start with the symptom table to narrow the cause, confirm it with the kubectl command in the “Confirm” column, then apply the fix. If you still can’t resolve it, collect the diagnostic bundle and open a case with support.

Namespaces follow <service>-<env> (e.g. databricks-dev); Deployments are <env>-<service>.

Symptom → cause → confirm → fix

Symptom	Likely cause	How to confirm	Fix
Pod in CrashLoopBackOff	Missing/invalid env var or secret at startup	kubectl logs -n <ns> deploy/<env>-<svc> --previous	Fix the value in Secrets Manager / overlay, reconcile the CSI secret, rollout restart
Pod stuck ContainerCreating with secret mount error	CSI SecretProviderClass can’t fetch the AWS secret	kubectl describe pod -n <ns> <pod> (look for secrets-store / FailedMount)	Confirm the AWS secret exists and the Pod Identity role can read it; re-apply the SecretProviderClass
Pod Pending	No node capacity or unschedulable	kubectl describe pod -n <ns> <pod> (Events)	Raise node_desired_size/node_max_size and terraform apply; check taints
Pod OOMKilled / restarts under load	Memory limit too low (esp. STT, orchestration)	kubectl describe pod → Last State OOMKilled	Raise memory limits in the overlay; sync
503/404 from the gateway for one path	HTTPRoute misconfigured or backend not ready	kubectl get httproute -A; check the target pod is Ready	Fix the path/backend in the service’s httproute patch; sync
TLS error / expired cert in browser	Certificate not Ready or DNS-01 challenge failing	kubectl get certificate -n envoy-gateway-system	See certificate rotation; check Route53 access for the issuer
All ingress down, gateway has no address	NLB not provisioned or gateway not Programmed	kubectl get gateway -n envoy-gateway-system -o wide	Check the Envoy Gateway controller pods and the load-balancer annotations
401/403 between services	Service auth token rotated on one side only	Compare NEXA_SERVICE_AUTH_TOKEN across caller and callee	Set the new token everywhere, keep the old in NEXA_SERVICE_AUTH_TOKEN_PREVIOUS, roll
API returns DB connection errors	Wrong PG_HOST, exhausted connections, or SSL required	kubectl logs for the API; RDS connection metric in CloudWatch	Verify PG_* in the secret; check max_connections; align rds_force_ssl with the client
Request blocked / 403 from an allowed client	Envoy SecurityPolicy IP allowlist too strict	kubectl get securitypolicy -A -o yaml	Add the client CIDR to the allowlist patch; sync
Automations not progressing	Orchestration worker scaled to 0 and not waking	kubectl get scaledobject -n orchestration-<env>	Confirm KEDA can reach Postgres (DATABASE_URL) and work_items has pending rows
ArgoCD app OutOfSync/Degraded	Bad manifest or image tag, or drift	ArgoCD UI or kubectl get applications -n argocd	Inspect the app’s conditions; fix the overlay in Git; sync
New image not deployed	Image Updater didn’t match the tag	Check the app’s .argocd-source write-back and allow-tags regex	Push a tag matching the branch/regex; force an Image Updater run

Reading logs

Nexa services log JSON to stdout. Key fields: level, timestamp, message, and (where present) a request/correlation id you can grep across services.

• Live: kubectl logs -n <ns> deploy/<env>-<svc> -f
• After a crash: kubectl logs -n <ns> deploy/<env>-<svc> --previous
• Raise detail temporarily: set LOG_LEVEL=debug in the overlay and roll; revert when done.

Common log messages and what they mean:

Message pattern	Meaning	Action
secretKeyRef ... not found / mount denied	CSI secret not synced or key missing	Reconcile the SecretProviderClass; verify the key name
connection refused to an internal *.svc.cluster.local host	Target service down or wrong URL env var	Check the callee pod and the *_URL variable
SSL connection is required (Postgres)	rds.force_ssl on but client not using TLS	Enable SSL in the client or align rds_force_ssl
401 unauthorized on service call	Service auth token mismatch	Rotate consistently with the _PREVIOUS overlap
403 at gateway before reaching a pod	SecurityPolicy denied the source IP	Update the IP allowlist

Tip

Reproduce a failing route without the browser to isolate gateway vs. app: kubectl port-forward -n <ns> deploy/<env>-<svc> 8080:<port> then curl -i localhost:8080/<health-path>. A 200 here but a failure through the domain points at ingress, not the service.

Diagnostic bundle

When escalating, capture this bundle so support has cluster state, logs, config (with secrets redacted), and versions. Run against the affected environment.

1. Set the target and create an output directory:

   ENV=dev
   OUT="nexa-diag-$ENV-$(date +%Y%m%d-%H%M)"; mkdir -p "$OUT"

2. Capture cluster and workload state:

   kubectl get pods,svc,httproute,gateway,certificate -A -o wide > "$OUT/cluster-state.txt"
   kubectl get applications -n argocd -o wide            > "$OUT/argocd-apps.txt"
   kubectl get scaledobject -A                           > "$OUT/keda.txt"
   kubectl get events -A --sort-by=.lastTimestamp        > "$OUT/events.txt"

3. Describe and log the unhealthy service (set NS and SVC):

   NS=databricks-$ENV; SVC=$ENV-nexa-databricks-api
   kubectl describe deploy/$SVC -n "$NS"      > "$OUT/$SVC-describe.txt"
   kubectl logs deploy/$SVC -n "$NS" --tail=2000            > "$OUT/$SVC-logs.txt"
   kubectl logs deploy/$SVC -n "$NS" --previous --tail=2000 > "$OUT/$SVC-logs-prev.txt" 2>/dev/null

4. Capture ingress and cert-manager detail:

   kubectl describe gateway -n envoy-gateway-system      > "$OUT/gateway.txt"
   kubectl get challenges,orders -A -o wide              > "$OUT/acme.txt"

5. Record versions (never include secret values):

   kubectl version --short                > "$OUT/versions.txt"
   kubectl get deploy -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].image}{"\n"}{end}' > "$OUT/images.txt"

6. Redact and package. Do not include Secret objects or env values.

   tar czf "$OUT.tar.gz" "$OUT" && echo "Attach $OUT.tar.gz"

Caution

Never run kubectl get secret -o yaml into the bundle — it exposes credentials. Capture only names and mount status (describe pod), never values. Scrub any host, token, or IP you don’t want to share before sending.

Attach nexa-diag-<env>-<timestamp>.tar.gz to your support case, with the exact symptom, the time it started, and what changed just before. See the FAQ for questions that don’t need a case.