AI/playbooks/service-golive.md

Playbook: Service Go-Live Review

Use this playbook before exposing any service to external access through Nginx Proxy Manager (NPM). When invoked, read the project directory in the current working directory and work through each section as an interactive checklist.

Topology assumption: NPM runs on a separate server on a separate network from the app host. Traffic flows: Internet → NPM server → (routed/firewalled link) → app host published port → app container. The database lives only on an internal Docker network on the app host.

---

How to Use

Tell the AI: "Use the service-golive playbook to review this project: https://git.chns.tech/CHNS/AI/raw/branch/main/playbooks/service-golive.md"

The AI will:

1. Read the project files in the current directory
2. Work through each section below
3. For each item — report PASS, FAIL, or WARN with specific findings
4. At the end, give a go/no-go recommendation and save a report

Do not proceed to the next section until the current one is resolved or explicitly deferred.

Severity: Items tagged [CRITICAL], [HIGH], [MEDIUM], or [LOW] use that severity when they FAIL. For untagged items, the AI assigns severity based on how exploitable or disruptive the gap is and states its reasoning.

Evidence: Every result — PASS included — must cite evidence: a file:line reference, command output, or the user's confirmation. A PASS with no evidence is reported as WARN.

---

Section 0: Pre-flight

Goal: Make sure the review tools are available before relying on them.

• trivy --version
• bandit --version (Python projects)
• semgrep --version (Python projects — catches framework-specific issues bandit misses)
• pip-audit --version (Python projects)
• gitleaks version
• hadolint --version
• testssl.sh --version (or run via its Docker image)
• docker info (needed for the ZAP baseline scan in Section 4)
• curl, openssl, and nmap available

AI Action: If a tool is missing, tell the user and ask whether to install it or skip the checks that depend on it. Skipped checks are reported as WARN, not PASS.

---

Section 1: Feature & Improvement Review

Goal: Catch missing functionality before users find it.

• Does the service have a health check endpoint (e.g. /health or /ping)?
• The health endpoint returns only a status — no versions, hostnames, DB details, or stack traces
• Does the app container have its own Compose healthcheck wired to that endpoint (not just the database)?
• Is restart: unless-stopped (or always) set on every service?
• Are all intended routes/endpoints implemented and reachable?
• Is there a meaningful error response for bad input (not raw stack traces)?
• A global exception handler returns a generic 500 body and logs the detail server-side
• Are there any obvious UX gaps or incomplete flows in the UI (if applicable)?
• Is there logging in place to capture errors and key events?
• Are there any TODO/FIXME/HACK comments in the code that indicate unfinished work?
• Does the service handle its own startup failures gracefully (exits cleanly, logs reason)?
• Schema changes use migrations (e.g. Alembic), not Base.metadata.create_all() at startup
• Graceful shutdown: the app handles SIGTERM, finishes in-flight requests, and closes DB connections (stop_grace_period set in Compose if shutdown takes longer than 10s)
• Production server runs without --reload and with an appropriate worker count

1a. Ntfy Admin Notifications

Goal: Ensure the super admin is alerted to significant events without having to monitor logs manually. Ntfy is required for all web apps (see preferences/python-web.md).

• [HIGH] Is Ntfy integrated into the application?
• Are admin-relevant events triggering Ntfy notifications?
• [HIGH] Is the Ntfy topic access-protected (token or user auth on the self-hosted instance) so others can't subscribe to admin alerts?
• Are Ntfy calls wrapped in try/except with a short timeout so a failed or slow notification can't crash or stall the app?
• Ntfy URL, topic, and token come from .env — not hardcoded

If Ntfy is NOT implemented, flag as FAIL and recommend the following events for notification coverage based on what the app does:

Event	Severity	Why it matters
Successful admin login	High	Detect unauthorized admin access
Failed admin login (threshold reached)	High	Brute-force indicator
New user registration	Medium	Visibility into who is joining
User account deletion	Medium	Audit trail for removals
Role/permission escalation	High	Privilege change could indicate compromise
Password reset requested	Medium	Could indicate account takeover attempt
MFA disabled or reset on an account	High	Common step in account takeover
Rate limit triggered	Medium	Abuse or misconfigured client
API key created or revoked	High	Credential lifecycle event
Service startup / crash recovery	Medium	Unexpected restarts need awareness
High error rate (e.g. 5xx spike)	High	App health degrading in production
Large data export initiated	Medium	Data exfiltration risk indicator
Config or environment change detected	High	Unplanned changes should be visible

AI Action: Search the codebase for Ntfy integration (look for ntfy, ntfy.sh, or HTTP POST calls to a notification endpoint). If none found, list the above recommended events as FAIL items and ask the user whether to implement before go-live.

1b. Uptime Kuma Monitoring

Goal: Get alerted when the service is down — Ntfy runs inside the app, so it can't report its own outage.

• [HIGH] An Uptime Kuma monitor exists for this service, pointed at the public health endpoint URL (through NPM), not the internal container address
• The monitor has a notification configured (e.g. Ntfy) so downtime actually alerts someone
• Monitor uses an HTTP(s) status or keyword check, not just a ping/TCP check
• Certificate expiry notification is enabled on the monitor

AI Action: Ask the user to confirm the monitor is set up — the AI can't see Uptime Kuma directly.

---

AI Action: List any gaps found with file and line references. Ask the user whether to fix now or defer.

---

Section 2: Performance Review

Goal: Ensure the service won't collapse under real load.

• Are database queries using indexes on columns used in WHERE/JOIN/ORDER BY clauses?
• Are N+1 query patterns present (loop that fires a query per item)?
• Is connection pooling configured for the database?
• Are large responses paginated?
• [HIGH] Pagination limit/page_size parameters have a server-side maximum — a client can't request ?limit=1000000
• Are any blocking operations (file I/O, external API calls) being done synchronously in an async context?
• Outbound HTTP calls (httpx etc.) set explicit timeouts
• Are static assets (if any) being served by Nginx, not the app?
• Is there any unbounded data being loaded into memory (e.g. SELECT * with no limit)?
• Are background tasks or scheduled jobs using a proper queue/worker model (not threading hacks)?
• Is Gzip/Brotli compression enabled in Nginx for text responses?

AI Action: Flag any issues with specific file references. Suggest fixes. Ask user to confirm or defer.

---

Section 3: Security Audit

Goal: Do not put a vulnerable service on the internet. Be thorough.

3a. Secrets & Credentials

• [CRITICAL] No hardcoded passwords, tokens, API keys, or secrets in any source file
• [CRITICAL] .env file is in .gitignore and not committed
• [CRITICAL] No secrets anywhere in git history (a previously committed .env is still exposed even if deleted later — rotate any secret found)
• [HIGH] .dockerignore exists and excludes .env, .git, and other local-only files so they aren't baked into the image
• .env.example exists with placeholder values only
• [HIGH] No secrets in Docker Compose files (use env_file or environment variable references, not literal values)
• [HIGH] No secrets in Nginx config files
• [CRITICAL] The app refuses to start if SECRET_KEY (or any signing key) is shorter than 32 random bytes or still equals the .env.example placeholder
• [HIGH] If JWTs are used: the accepted algorithm is pinned server-side (e.g. algorithms=["HS256"]), none is never accepted, and exp is always validated

3b. Authentication & Authorization

• [CRITICAL] All non-public endpoints require authentication
• [CRITICAL] Object-level authorization: every endpoint that takes a record ID (/items/{id}, ?user_id=) verifies the current user owns or is allowed to access that record — changing the ID must not expose another user's data (IDOR/BOLA)
• [CRITICAL] Function-level authorization: admin-only endpoints check the role server-side, not just by hiding UI elements
• Authentication tokens/sessions have an expiry
• [HIGH] Password hashing uses bcrypt, argon2, or scrypt — not MD5/SHA1
• Password policy: minimum length of at least 12, no forced complexity rules, and (ideally) a check against known-breached passwords
• [CRITICAL] There is no default admin password that ships with the service
• [HIGH] Admin accounts require MFA (e.g. TOTP)
• Role/permission checks exist if the app has multiple access levels
• [HIGH] Failed login attempts are rate-limited or account-locked after N failures — per account and per IP
• Login and password-reset responses don't reveal whether an account exists (same message and similar timing for valid/invalid users)
• Password reset tokens are single-use, short-lived, and invalidated after a successful reset
• Session ID / token is regenerated on login (no session fixation)
• Logout and password change invalidate existing sessions/tokens

3c. Input Validation & Injection

• All user input is validated server-side (not just client-side)
• Pydantic string/list fields have max lengths; request schemas use extra="forbid" where practical
• [HIGH] Mass assignment: Create/Update schemas don't accept privileged fields (role, is_admin, owner_id, id, etc.) that the user shouldn't set
• [CRITICAL] SQL queries use parameterized statements or ORM — no string concatenation (including text() with f-strings)
• [CRITICAL] No shell command built from user input (subprocess with shell=True, os.system)
• [HIGH] HTML output is escaped to prevent XSS (Jinja2 autoescape on, no |safe on user data)
• [HIGH] SSRF: if the app fetches any user-supplied URL (webhooks, image import, link preview), it blocks private/internal ranges (RFC1918, 127.0.0.0/8, 169.254.0.0/16, IPv6 equivalents) and Docker service names, and re-checks after redirects
• Redirects only go to allowed/relative URLs — no open redirect
• [HIGH] Deserialization does not allow arbitrary object instantiation (no pickle/yaml.load on untrusted data; XML parsed with defusedxml)
• User-supplied regexes are not executed, and server regexes aren't vulnerable to catastrophic backtracking (ReDoS)

3d. HTTP & Nginx Security Headers

Verify the NPM proxy host config (on the NPM server) and/or app responses include:

• X-Content-Type-Options: nosniff
• X-XSS-Protection: 0 or omitted — the legacy 1; mode=block value is deprecated and can introduce vulnerabilities; CSP replaces it
• Referrer-Policy: strict-origin-when-cross-origin
• [HIGH] Content-Security-Policy defined, and of usable quality:
  • no 'unsafe-inline' / 'unsafe-eval' in script-src (or a documented reason)
  • frame-ancestors 'none' (or 'self') — supersedes X-Frame-Options, though sending both is fine
  • object-src 'none' and base-uri 'self' (or 'none')
• Permissions-Policy restricts browser features the app doesn't use (e.g. camera=(), microphone=(), geolocation=())
• [MEDIUM] Strict-Transport-Security (HSTS) with max-age >= 31536000 (add includeSubDomains only if every subdomain is HTTPS)
• Authenticated responses send Cache-Control: no-store
• [HIGH] NPM header gotcha: if the proxy host has Custom Locations or an add_header inside a location block in Advanced config, nginx drops the server-level add_header directives for that location. Headers must be repeated per location — verify every route, not just /
• Server version header suppressed on the NPM server (server_tokens off) and the app doesn't send its own Server/X-Powered-By with versions
• Unnecessary HTTP methods disabled (e.g. TRACE, DELETE if not used)

3e. TLS / HTTPS

• [CRITICAL] TLS certificate is valid and not self-signed for production
• [HIGH] HTTP traffic redirects to HTTPS (not served in parallel)
• TLS 1.0 and 1.1 disabled — only TLS 1.2+ allowed (1.3 preferred)
• Weak cipher suites disabled
• Certificate expiry is monitored (NPM auto-renews, but verify it's configured)
• DNS has a CAA record limiting which CAs can issue certificates for the domain (e.g. 0 issue "letsencrypt.org")
• NPM → app hop: either proxied over HTTPS to the app host, or plain HTTP is a documented, accepted risk because the link is restricted by firewall rules (see 3g). This traffic leaves the NPM server and crosses a routed network, so it is not loopback-only

3f. Docker & Container Security

• [HIGH] Containers do not run as root (check user: in Compose or Dockerfile USER instruction)
• [HIGH] No container has privileged: true unless there is a documented reason
• [HIGH] No unnecessary host volume mounts (especially /var/run/docker.sock unless intentional)
• [HIGH] cap_drop: [ALL] on app containers (add back only specific capabilities with a documented reason)
• [HIGH] security_opt: ["no-new-privileges:true"] on every container
• read_only: true root filesystem on the app container, with tmpfs for paths that need writes (e.g. /tmp) and named volumes for real data
• pids_limit set to prevent fork bombs
• Resource limits (mem_limit, cpus) are set on containers
• Images pinned to a specific version tag — preferably by digest (image: name:tag@sha256:...) — never latest
• Dockerfile uses a multi-stage build; the final image is slim (no compilers, build tools, or package caches)
• hadolint Dockerfile has no errors (warnings reviewed)
• [CRITICAL] Docker socket is not exposed over TCP or to any network

3g. Network & Exposure (separate NPM server)

Topology reminder: NPM is on its own server and network, so the app port must be published on the app host. The goal is to make it reachable only from NPM.

• [CRITICAL] Only 80/443 are exposed publicly, and only on the NPM server — the app host has no port-forward / NAT rule from the internet
• [CRITICAL] Only the single app/web port is published on the app host (ports:); every other service (DB, Redis, workers) uses expose: or no ports at all
• [CRITICAL] Database ports (3306, 5432, 6379) are NOT published on the host — the DB sits on a Docker network marked internal: true that only the app container joins
• [CRITICAL] Docker bypasses UFW/firewalld for published ports. Source restriction to NPM's IP is enforced in the DOCKER-USER iptables chain on the app host — a UFW rule alone does nothing. Example:

  iptables -I DOCKER-USER -i <ext_if> -p tcp -m conntrack --ctorigdstport <app_port> --ctdir ORIGINAL ! -s <NPM_IP> -j DROP
  

  Rules are persisted across reboots (e.g. iptables-persistent), and repeated with ip6tables if the host has IPv6
• [HIGH] The firewall between the NPM network and the app network allows only <NPM_IP> → <app_host>:<app_port> (single source, single port — not "any" or a whole subnet)
• [HIGH] Hosts on the app host's own network can't reach the app port directly. The inter-network firewall doesn't filter same-segment traffic — only the DOCKER-USER rule does. Direct access would bypass NPM's access lists, headers, and rate limits
• If published ports are bound to a specific interface IP (<host_ip>:<port>:<port>), it's the interface facing NPM, not 0.0.0.0
• NPM proxy host has access list or basic auth if the service is internal-only
• Rate limiting is configured in Nginx or the app for API endpoints
• [HIGH] The service does not expose an admin panel (e.g. /admin, /dashboard) without additional auth
• [HIGH] API docs (/docs, /redoc, /openapi.json in FastAPI) are disabled or behind auth in production
• [HIGH] Debug mode is off (debug=False, no auto-reload, no verbose tracebacks returned to clients)
• SSH is not running inside any container

3h. Database Access

• [HIGH] App connects with a dedicated MySQL user limited to its own database — not root
• App DB user has only the privileges it needs (no GRANT, SUPER, FILE, or global privileges)
• App DB user's host is restricted to the Docker network (not '%' if avoidable)
• MySQL root password is separate from the app user's password and stored only in .env
• local_infile is disabled on the MySQL server

3i. Dependency & Supply Chain

• Dependencies are pinned to exact versions via a lockfile with hashes (pip-compile --generate-hashes, uv lock, or package-lock.json) and installed with hash checking (pip install --require-hashes)
• Known CVEs in dependencies? (run trivy fs ., pip-audit, or npm audit)
• No abandoned or unmaintained packages with known issues
• Newly published package versions are not adopted immediately — wait ~7 days (e.g. uv exclude-newer, pnpm minimumReleaseAge, Renovate minimumReleaseAge) to avoid compromised releases
• Docker base images are from official/verified sources
• Security scans (gitleaks, bandit/semgrep, trivy) run automatically in Gitea Actions on push, so go-live isn't the first time they run

3j. Request & Session Hardening

• [HIGH] Session/auth cookies set Secure, HttpOnly, and SameSite=Strict or Lax
• [HIGH] CSRF protection is in place for state-changing (non-GET) endpoints, or the API is stateless/token-authenticated (not cookie-authenticated) and doesn't need it
• [HIGH] CORS policy restricts Access-Control-Allow-Origin to known origins — no wildcard * combined with Access-Control-Allow-Credentials: true
• [HIGH] Host header validation: TrustedHostMiddleware (or equivalent) allows only the public hostname(s), and links in emails (password reset etc.) are built from a configured base URL, never from the request Host header
• [HIGH] Trusted proxy: the app trusts X-Forwarded-For/X-Forwarded-Proto only from NPM's IP (e.g. uvicorn --proxy-headers --forwarded-allow-ips=<NPM_IP>, never *). Verify the source IP the app actually sees — if the inter-network router/firewall NATs traffic, the app sees the router's IP instead and the setting must match that
• Request body size is capped on the NPM proxy host (client_max_body_size) and in the app for upload endpoints
• NPM proxy timeouts are set to sane values (slow-client / slowloris protection)
• If rate limiting is in the app: the limiter uses shared storage (e.g. Redis) when running multiple workers — an in-memory limiter is per-worker, multiplying the real limit
• Rate limits cover login, registration, password reset, and expensive endpoints
• File uploads:
  • enforce a max size limit
  • validate actual content type (magic bytes), not just the extension
  • are stored outside any served path with random server-generated names (no path traversal possible)
  • are served with Content-Disposition: attachment and the correct Content-Type
  • archives are checked for decompressed size (zip bombs)

3k. Backups & Recovery

• [HIGH] Database/data volumes have an automated backup process (cron, script, or handled by the DB image)
• Backups are stored outside the container (separate volume, external disk, or remote target) so they survive the container/host being wiped
• A restore from backup has been tested at least once, not just assumed to work
• Backup files containing sensitive data are encrypted or access-restricted
• Live .env and any config that only exists in Portainer/Dockhand are backed up somewhere retrievable (they aren't in git)
• NPM server config, database, and certificates are backed up from the NPM server (separate host — not covered by the app host's backups)
• App host DOCKER-USER rules and inter-network firewall rules are documented so they can be rebuilt

3l. Logging & Observability

• Logs are rotated or size-capped (Docker logging.driver/max-size, or app-level rotation) so they can't fill disk over time
• Logs persist outside the container (volume-mounted or shipped elsewhere) so they survive a container restart/crash
• Logs record the real client IP (depends on the trusted proxy setting in 3j), not NPM's IP
• [HIGH] Logs never contain passwords, tokens, session IDs, API keys, or full request bodies with sensitive fields
• Security events (logins, failed logins, permission changes, admin actions) are written to a durable audit log/table with timestamp, user, and client IP — Ntfy alerts are not a record
• App host and NPM server clocks are NTP-synced so logs from both can be correlated

3m. NPM Server

NPM is the internet-facing host. If it's compromised, an attacker can read or alter all proxied traffic.

• [CRITICAL] NPM admin UI (port 81) is not reachable from the internet — LAN/VPN only
• [HIGH] NPM admin account uses a strong unique password (default [email protected] account changed/removed)
• NPM container and host OS are patched and on a current version
• SSH on the NPM server is key-only, no root login, and not exposed to the internet
• The NPM server can reach only the app host ports it proxies to, not the entire internal network
• [MEDIUM] Automated ban layer (CrowdSec or fail2ban reading NPM logs) blocks scanners and brute-force sources
• "Block Common Exploits" is enabled on the proxy host

AI Action: Run the following tools if available:

• bandit -r . -ll — Python static security analysis
• semgrep scan --config p/python --config p/secrets — additional static analysis
• pip-audit -r requirements.txt — Python dependency CVEs
• trivy fs . --severity HIGH,CRITICAL — dependency and filesystem CVE scan
• gitleaks git . — secret scan across full git history (older gitleaks versions: gitleaks detect --source .)
• hadolint Dockerfile — Dockerfile lint
• docker scout cves <image> or trivy image <image> — container image vulnerability scan

Ask the user to paste iptables -S DOCKER-USER (and ip6tables -S DOCKER-USER if IPv6 is on) from the app host and the relevant inter-network firewall rules — the AI can't see them directly.

Report all FAIL/WARN findings. Do not proceed to Section 4 until CRITICAL and HIGH issues are resolved.

---

Section 4: External Verification

Goal: Confirm the live service matches what the config says. Sections 1–3 check files — this checks what the network actually sees.

Only start this section once Sections 1–3 have no unresolved CRITICAL/HIGH findings.

4a. Staged Exposure

1. Enable the NPM proxy host with an access list restricted to the admin's public IP
2. Run all checks in 4b and 4c against the public URL
3. Only after they pass, remove the access list to open the service to everyone
4. Re-run the header and redirect checks once open

If any CRITICAL/HIGH check fails at any point, disable the proxy host (or re-apply the access list) until it's fixed.

4b. Public Checks

• [HIGH] Security headers from 3d are present in the live response — on / and on every custom location / API route
• [HIGH] HTTP → HTTPS redirect works on the public URL
• [MEDIUM] SSL Labs test (https://www.ssllabs.com/ssltest/) or testssl.sh scores A or better
• securityheaders.com (https://securityheaders.com) scores A or better, or gaps are documented
• [CRITICAL] External port scan of the NPM server shows only 80/443 open — run from outside the LAN (e.g. phone hotspot) or an online port scanner, over IPv4 and IPv6
• [HIGH] /docs, /redoc, /openapi.json, and any admin paths return 401/403/404 from the public URL
• Error pages don't leak stack traces or server/framework versions (request a nonexistent path and a malformed request)
• [HIGH] OWASP ZAP baseline scan has no unaddressed High findings
• Uptime Kuma monitor shows the service as up

4c. Internal Reachability Checks

• [CRITICAL] From a machine on the app host's network (not NPM): the app port is refused/filtered
• [CRITICAL] From the NPM server: the app port responds, and no other app host ports (DB, Portainer/Dockhand agent, etc.) are reachable
• The real client IP shows up in app logs when the service is accessed through the public URL

AI Action: Run:

• curl -sI https://<public-url> (and each custom location / API route) — verify headers
• curl -sI http://<public-url> — verify redirect
• curl -s -o /dev/null -w "%{http_code}" https://<public-url>/docs (repeat for /redoc, /openapi.json, admin paths)
• echo | openssl s_client -connect <host>:443 -servername <host> 2>/dev/null | openssl x509 -noout -dates -issuer
• testssl.sh --severity MEDIUM https://<public-url>
• docker run --rm -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t https://<public-url>

Ask the user to run the external port scan, the internal reachability checks (nmap -Pn -p- <app_host> from a same-network machine and from the NPM server), and the SSL Labs/securityheaders.com checks if not automated, then paste the results.

---

Section 5: Go-Live Decision

• An incident runbook exists (README or RUNBOOK.md) covering:
  • how to take the service offline fast (disable the NPM proxy host)
  • how to rotate every secret (DB passwords, SECRET_KEY, API keys, Ntfy token) and where each is stored
  • how to invalidate all sessions (rotate the signing key)
  • how to roll back to the previous image tag/digest
  • how to restore from backup, and roughly how long it takes

After all sections are complete:

• List all unresolved FINDs grouped by severity: CRITICAL / HIGH / MEDIUM / LOW
• CRITICAL or HIGH unresolved = NO GO. These must be fixed before external access.
• MEDIUM/LOW unresolved = user decides whether to defer with documented acceptance
• Provide a final summary:
  • Total checks: X
  • Passed: X
  • Failed (critical): X
  • Failed (non-critical): X
  • Deferred: X
  • Recommendation: GO / NO GO / GO WITH CONDITIONS

---

Section 6: Baseline & Handoff

Goal: Record what went live so the service-update maintenance playbook has something to compare against. Only complete after a GO or GO WITH CONDITIONS decision.

• Go-live version tagged in Gitea (e.g. v1.0.0) and pushed
• Release-Notes/v1.0.md created
• Running image tags/digests recorded in the report
• SBOM generated for each running image and saved alongside the report (trivy image --format cyclonedx --output reports/sbom-<image>-<YYYY-MM-DD>.json <image>)
• App host DOCKER-USER rules and inter-network firewall rules recorded in the report
• Deferred items recorded in the report with the reason for deferring
• Report saved to reports/golive-<YYYY-MM-DD>.md in the project directory — the first maintenance review uses it as its prior report