# Runbook

## Site shows fallback content

1. Check `docker compose ps` and the backend health status.
2. Request `http://azionelab.org:8080/api/site/home/` through NGINX.
3. Inspect `docker compose logs backend db`.
4. Verify `PUBLIC_CMS_API_URL` resolves from the frontend container.
5. Restart the frontend after recovery so its static page is rebuilt from CMS data.

## Wagtail does not start

1. Confirm PostgreSQL is healthy.
2. Check `DATABASE_URL` and allowed hosts without printing secret values in shared logs.
3. Run `docker compose exec backend python manage.py migrate`.
4. Review backend logs for a specific migration or configuration error.

## Images are missing

1. Verify the `media_data` volume is mounted at `/app/media`.
2. Check `WAGTAILADMIN_BASE_URL` uses a URL reachable by the browser.
3. Confirm the image still exists in Wagtail and has not been removed from the volume.

## A service is unhealthy

1. Run `docker compose ps` and inspect `docker compose logs --tail=200 SERVICE`.
2. For PostgreSQL, check volume capacity and database/user configuration.
3. For the backend, check `/health/`, migrations, and database connectivity.
4. For the frontend, request port `4321`, then check the API separately because
   fallback content can mask a CMS outage.

## NGINX returns 404 or 502

1. Confirm the request host is exactly `azionelab.org`; unknown hosts intentionally
   receive 404.
2. Run `docker compose exec proxy nginx -t`.
3. Check `docker compose ps` and verify both backend and frontend are healthy.
4. Inspect `docker compose logs --tail=200 proxy backend frontend`.
5. Verify local DNS or `/etc/hosts` maps `azionelab.org` to the proxy address.

## Let's Encrypt does not issue a certificate

1. Confirm `LETSENCRYPT_ENABLED=1` and that `docker compose ps -a certbot proxy` shows
   both containers running and the proxy healthy.
2. Check that the domain's public A/AAAA records point to this host. Remove an AAAA
   record if IPv6 does not actually reach it.
3. Verify inbound TCP port 80 reaches NGINX; HTTP-01 cannot use only port 443.
4. Request `http://azionelab.org/.well-known/acme-challenge/missing`: a 404 from NGINX
   confirms the challenge route is reachable, while a timeout or another server does
   not.
5. Inspect `docker compose logs --tail=200 certbot proxy` for ACME validation or rate
   limit errors. Do not repeatedly retry the production CA; use staging while fixing
   connectivity.

## HTTPS is not activated after issuance

1. Run `docker compose run --rm --no-deps --entrypoint certbot certbot certificates`.
2. Confirm `LETSENCRYPT_DOMAIN` exactly matches the certificate name used by both
   services.
3. Wait for `TLS_RELOAD_INTERVAL_SECONDS`, then inspect proxy logs for `nginx -t` or
   reload errors.
4. Run `docker compose exec proxy nginx -t` and check HTTPS locally with an explicit
   DNS override.

## Rollback

Disable Certbot with `LETSENCRYPT_ENABLED=0` if TLS is moving to a load balancer, then
recreate the affected services. Revert the application commit and rebuild containers
for a full rollback. Preserve database, media, and certificate volumes. Before
reversing migrations or deleting volumes, make and validate coordinated backups.