Operations Guide (Day 2)

This guide covers the recurring work of running Nexa in production after it is installed: watching health, reading logs, backing up and restoring the database, planning for disaster recovery, scaling, and rotating certificates and secrets. It assumes the reference architecture — EKS, Postgres, Envoy Gateway, cert-manager, and AWS Secrets Manager via the CSI driver. For the settings referenced here, see the configuration reference.

Throughout, <env> is dev, test, or prod, and namespaces follow the <service>-<env> pattern (e.g. databricks-dev).

Health & monitoring

Health endpoints

Every service exposes an HTTP health endpoint that its Kubernetes liveness and readiness probes already call. Use the same paths for external monitoring.

Service	Namespace	Port	Path
nexa-web	web-<env>	3000	/
nexa-backend	backend-<env>	4000	/health-check
nexa-databricks-api / nexa-snowflake-api	databricks-<env>	8000	/health
nexa-agents-api	agents-<env>	8000	/api/v1/health
nexa-orchestration-service	orchestration-<env>	8003	/readyz
nexa-stt-service	stt-<env>	8010	/readyz
nexa-slackbot-service	slack-<env>	8000	/api/v1/health

Check readiness across a namespace:

kubectl get pods -n databricks-<env> -o wide
kubectl get httproute -A          # ingress routes and their attached gateway
kubectl get gateway -n envoy-gateway-system

Metrics to watch

The reference IaC does not ship Prometheus/Grafana; add them at the cluster layer and scrape the signals below. Until then, the sources column tells you where each is visible today (CloudWatch or kubectl).

Metric	Why it matters	Threshold	Source
Pod restarts / CrashLoopBackOff	Repeated crashes = bad config, missing secret, or OOM	Any restart loop → investigate	kubectl get pods
Container memory vs. limit	STT and orchestration have large limits; OOMKills drop work	Alert at 85% of limit	metrics-server / CloudWatch
CPU throttling	Throttled API pods raise latency	Sustained > 25% throttle	metrics-server
Readiness probe failures	Pod is up but not serving	Any sustained failure	kubectl describe pod
work_items pending depth	Backlog the KEDA worker must drain	Growing for > 5 min → scale/investigate	Postgres query
KEDA worker replicas	Confirms autoscaling is reacting	Pinned at max 5 while backlog grows → bottleneck	kubectl get scaledobject
Postgres connections	Connection exhaustion blocks all services	Alert at 80% of max_connections	RDS CloudWatch
RDS free storage / CPU	Full disk or saturated DB stalls the platform	Alert at 80% storage, 80% CPU	RDS CloudWatch
Certificate days-to-expiry	Expired TLS breaks all ingress	Alert at < 21 days	kubectl get certificate
5xx rate at the gateway	User-facing failures	Alert on any sustained rise	Envoy access logs

Logging

Nexa services log to stdout as structured JSON when LOG_FORMAT=json. There is no bundled log shipper — forward container logs to CloudWatch Logs or your SIEM at the cluster layer.

• Live tail: kubectl logs -n <ns> deploy/<env>-<service> -f
• Previous crash: kubectl logs -n <ns> deploy/<env>-<service> --previous
• Control-plane / audit: EKS API and audit logs stream to CloudWatch (all five log types are enabled).
• Database: Postgres logs export to CloudWatch under the RDS log group.

Set verbosity with LOG_LEVEL (info default; debug for investigation) and keep DEBUG=false in production to avoid leaking stack traces.

Backup & restore

The platform’s durable state is Postgres (metadata) and S3 (raw data lake and Terraform state). Pods are stateless and rebuilt from container images by ArgoCD.

What is backed up

Store	Mechanism	Retention
Postgres	RDS automated backups + snapshots	7 days (default)
S3 raw data lake	Bucket versioning	until lifecycle expiry
Terraform state (S3)	Versioning + access logging	90 days noncurrent
Secrets Manager	Deletion recovery window	7 days

Caution

7 days of RDS retention is the module default. For production, raise backup_retention_period, enable Multi-AZ, and take an on-demand snapshot before every upgrade.

Take an on-demand backup

1. Snapshot Postgres before risky changes:

   aws rds create-db-snapshot \
     --db-instance-identifier <project>-<env> \
     --db-snapshot-identifier <project>-<env>-preupgrade-$(date +%Y%m%d)

2. Confirm it reached available:

   aws rds describe-db-snapshots --db-snapshot-identifier <project>-<env>-preupgrade-$(date +%Y%m%d)

Restore Postgres

1. Restore the snapshot into a new instance (never overwrite the live one):

   aws rds restore-db-instance-from-db-snapshot \
     --db-instance-identifier <project>-<env>-restore \
     --db-snapshot-identifier <snapshot-id>

2. Repoint the platform at the restored host by updating PG_HOST in AWS Secrets Manager (nexa-shared-secret-<env>), then reconcile the CSI-synced Kubernetes secret.
3. Roll the services that read Postgres so they pick up the new host:

   kubectl rollout restart deploy -n databricks-<env>
   kubectl rollout restart deploy -n backend-<env>

4. Verify health endpoints return 200 and the app loads.

Disaster recovery

The blast radius is one environment = one region. Recovery is a rebuild-from-IaC plus a database restore.

Objective	Target	Basis
RPO (max data loss)	≤ 24 h	Daily RDS automated backup; tighter with Multi-AZ + PITR
RTO (time to restore)	1–3 h	terraform apply to rebuild infra + ArgoCD sync + DB restore

1. Re-provision infrastructure from IaC: terraform apply against the target account/region with the environment’s .tfvars.
2. Let ArgoCD reconcile the app-of-apps; it redeploys every service from ECR images.
3. Restore Postgres from the latest snapshot (see above) and repoint PG_HOST.
4. Confirm cert-manager re-issued the wildcard TLS certificate and the gateway is serving 443.
5. Run health checks across all namespaces before returning traffic.

To shrink RPO/RTO for production: enable RDS Multi-AZ (automatic failover), turn on point-in-time recovery, and pre-stage the standby region’s Terraform state.

Scaling

• Nodes: raise node_desired_size / node_max_size (see the configuration reference) when nodes sit above ~80% CPU/memory or pods are Pending for lack of capacity.
• Service replicas: edit replicas in the service’s overlay kustomization.yaml and sync. There is no HPA — replica counts are explicit per service.
• Orchestration workers: the KEDA ScaledObject scales 0→5 off pending work_items. If the backlog stays high at max replicas, the bottleneck is downstream (warehouse or DB), not the worker count.
• Database: vertical only — raise rds_instance_class; expect a brief failover.

Signals to scale down: sustained low CPU/memory, empty work_items backlog, off-hours. The repo ships node pause/resume scripts for non-prod cost control.

Routine maintenance

Certificate rotation

TLS is a Let’s Encrypt wildcard issued by cert-manager over the DNS-01 challenge (Route53), with rotationPolicy: Always and renewBefore 30 days — renewal is automatic. To verify or force it:

1. Check status and expiry:

   kubectl get certificate -n envoy-gateway-system
   kubectl describe certificate nexa-gateway-tls-<env> -n envoy-gateway-system

2. If a cert is not Ready, inspect the challenge and issuer:

   kubectl get challenges,orders -A
   kubectl describe clusterissuer letsencrypt-prod-dns01

3. Force reissue only if needed by deleting the secret; cert-manager recreates it:

   kubectl delete secret nexa-gateway-tls-<env> -n envoy-gateway-system

Secret rotation

Secrets live in AWS Secrets Manager and reach pods through the CSI driver, synced into a Kubernetes Secret.

1. Update the value in Secrets Manager (nexa-shared-secret-<env> or the per-service secret).
2. Reconcile the CSI-synced Kubernetes secret (the repo provides a reconcile script), or delete the synced secret so the CSI driver rewrites it.
3. Roll the consuming Deployments so they re-read the value: kubectl rollout restart deploy -n <ns>.

For the service-to-service token, set the new value in NEXA_SERVICE_AUTH_TOKEN and move the old value to NEXA_SERVICE_AUTH_TOKEN_PREVIOUS first — services accept both during the rollout, giving zero-downtime rotation. Then remove the previous value on the next cycle.

Housekeeping

• Reconcile drift: keep ArgoCD selfHeal on so manual cluster edits revert to Git.
• Prune old images: ECR lifecycle keeps the last 10 tagged images and expires untagged ones after 7 days.
• Before upgrades, snapshot Postgres and read the upgrade guide; when something breaks, use troubleshooting.