updated golive playbook based on newer version of claude

playbooks/service-golive.md

@@ -3,6 +3,8 @@
 Use this playbook before exposing any service to external access through Nginx Proxy Manager (NPM).
 When invoked, read the project directory in the current working directory and work through each section as an interactive checklist.
 
+**Topology assumption:** NPM runs on a **separate server on a separate network** from the app host. Traffic flows: Internet → NPM server → (routed/firewalled link) → app host published port → app container. The database lives only on an internal Docker network on the app host.
+
 ---
 
 ## How to Use
@@ -19,6 +21,8 @@ Do not proceed to the next section until the current one is resolved or explicit
 
 **Severity:** Items tagged `[CRITICAL]`, `[HIGH]`, `[MEDIUM]`, or `[LOW]` use that severity when they FAIL. For untagged items, the AI assigns severity based on how exploitable or disruptive the gap is and states its reasoning.
 
+**Evidence:** Every result — PASS included — must cite evidence: a `file:line` reference, command output, or the user's confirmation. A PASS with no evidence is reported as WARN.
+
 ---
 
 ## Section 0: Pre-flight
@@ -27,8 +31,13 @@ Goal: Make sure the review tools are available before relying on them.
 
 - [ ] `trivy --version`
 - [ ] `bandit --version` (Python projects)
+- [ ] `semgrep --version` (Python projects — catches framework-specific issues bandit misses)
+- [ ] `pip-audit --version` (Python projects)
 - [ ] `gitleaks version`
-- [ ] `curl` and `openssl` available
+- [ ] `hadolint --version`
+- [ ] `testssl.sh --version` (or run via its Docker image)
+- [ ] `docker info` (needed for the ZAP baseline scan in Section 4)
+- [ ] `curl`, `openssl`, and `nmap` available
 
 **AI Action:** If a tool is missing, tell the user and ask whether to install it or skip the checks that depend on it. Skipped checks are reported as WARN, not PASS.
 
@@ -39,14 +48,19 @@ Goal: Make sure the review tools are available before relying on them.
 Goal: Catch missing functionality before users find it.
 
 - [ ] Does the service have a health check endpoint (e.g. `/health` or `/ping`)?
+- [ ] The health endpoint returns only a status — no versions, hostnames, DB details, or stack traces
 - [ ] Does the app container have its own Compose `healthcheck` wired to that endpoint (not just the database)?
 - [ ] Is `restart: unless-stopped` (or `always`) set on every service?
 - [ ] Are all intended routes/endpoints implemented and reachable?
 - [ ] Is there a meaningful error response for bad input (not raw stack traces)?
+- [ ] A global exception handler returns a generic 500 body and logs the detail server-side
 - [ ] Are there any obvious UX gaps or incomplete flows in the UI (if applicable)?
 - [ ] Is there logging in place to capture errors and key events?
 - [ ] Are there any TODO/FIXME/HACK comments in the code that indicate unfinished work?
 - [ ] Does the service handle its own startup failures gracefully (exits cleanly, logs reason)?
+- [ ] Schema changes use migrations (e.g. Alembic), not `Base.metadata.create_all()` at startup
+- [ ] Graceful shutdown: the app handles `SIGTERM`, finishes in-flight requests, and closes DB connections (`stop_grace_period` set in Compose if shutdown takes longer than 10s)
+- [ ] Production server runs without `--reload` and with an appropriate worker count
 
 ### 1a. Ntfy Admin Notifications
 
@@ -55,7 +69,7 @@ Goal: Ensure the super admin is alerted to significant events without having to
 - [ ] `[HIGH]` Is Ntfy integrated into the application?
 - [ ] Are admin-relevant events triggering Ntfy notifications?
 - [ ] `[HIGH]` Is the Ntfy topic access-protected (token or user auth on the self-hosted instance) so others can't subscribe to admin alerts?
-- [ ] Are Ntfy calls wrapped in `try/except` so a failed notification can't crash the app?
+- [ ] Are Ntfy calls wrapped in `try/except` with a short timeout so a failed or slow notification can't crash or stall the app?
 - [ ] Ntfy URL, topic, and token come from `.env` — not hardcoded
 
 **If Ntfy is NOT implemented**, flag as FAIL and recommend the following events for notification coverage based on what the app does:
@@ -68,6 +82,7 @@ Goal: Ensure the super admin is alerted to significant events without having to
 | User account deletion | Medium | Audit trail for removals |
 | Role/permission escalation | High | Privilege change could indicate compromise |
 | Password reset requested | Medium | Could indicate account takeover attempt |
+| MFA disabled or reset on an account | High | Common step in account takeover |
 | Rate limit triggered | Medium | Abuse or misconfigured client |
 | API key created or revoked | High | Credential lifecycle event |
 | Service startup / crash recovery | Medium | Unexpected restarts need awareness |
@@ -102,8 +117,10 @@ Goal: Ensure the service won't collapse under real load.
 - [ ] Are N+1 query patterns present (loop that fires a query per item)?
 - [ ] Is connection pooling configured for the database?
 - [ ] Are large responses paginated?
+- [ ] `[HIGH]` Pagination `limit`/`page_size` parameters have a server-side maximum — a client can't request `?limit=1000000`
 - [ ] Are any blocking operations (file I/O, external API calls) being done synchronously in an async context?
-- [ ] Are static assets (if any) being served through Nginx, not the app?
+- [ ] Outbound HTTP calls (httpx etc.) set explicit timeouts
+- [ ] Are static assets (if any) being served by Nginx, not the app?
 - [ ] Is there any unbounded data being loaded into memory (e.g. `SELECT *` with no limit)?
 - [ ] Are background tasks or scheduled jobs using a proper queue/worker model (not threading hacks)?
 - [ ] Is Gzip/Brotli compression enabled in Nginx for text responses?
@@ -124,130 +141,222 @@ Goal: Do not put a vulnerable service on the internet. Be thorough.
 - [ ] `.env.example` exists with placeholder values only
 - [ ] `[HIGH]` No secrets in Docker Compose files (use `env_file` or environment variable references, not literal values)
 - [ ] `[HIGH]` No secrets in Nginx config files
+- [ ] `[CRITICAL]` The app refuses to start if `SECRET_KEY` (or any signing key) is shorter than 32 random bytes or still equals the `.env.example` placeholder
+- [ ] `[HIGH]` If JWTs are used: the accepted algorithm is pinned server-side (e.g. `algorithms=["HS256"]`), `none` is never accepted, and `exp` is always validated
 
 ### 3b. Authentication & Authorization
 - [ ] `[CRITICAL]` All non-public endpoints require authentication
+- [ ] `[CRITICAL]` Object-level authorization: every endpoint that takes a record ID (`/items/{id}`, `?user_id=`) verifies the current user owns or is allowed to access that record — changing the ID must not expose another user's data (IDOR/BOLA)
+- [ ] `[CRITICAL]` Function-level authorization: admin-only endpoints check the role server-side, not just by hiding UI elements
 - [ ] Authentication tokens/sessions have an expiry
 - [ ] `[HIGH]` Password hashing uses bcrypt, argon2, or scrypt — not MD5/SHA1
+- [ ] Password policy: minimum length of at least 12, no forced complexity rules, and (ideally) a check against known-breached passwords
 - [ ] `[CRITICAL]` There is no default admin password that ships with the service
+- [ ] `[HIGH]` Admin accounts require MFA (e.g. TOTP)
 - [ ] Role/permission checks exist if the app has multiple access levels
-- [ ] `[HIGH]` Failed login attempts are rate-limited or account-locked after N failures
+- [ ] `[HIGH]` Failed login attempts are rate-limited or account-locked after N failures — per account and per IP
 - [ ] Login and password-reset responses don't reveal whether an account exists (same message and similar timing for valid/invalid users)
+- [ ] Password reset tokens are single-use, short-lived, and invalidated after a successful reset
+- [ ] Session ID / token is regenerated on login (no session fixation)
 - [ ] Logout and password change invalidate existing sessions/tokens
 
 ### 3c. Input Validation & Injection
 - [ ] All user input is validated server-side (not just client-side)
-- [ ] `[CRITICAL]` SQL queries use parameterized statements or ORM — no string concatenation
-- [ ] `[HIGH]` File upload paths are sanitized — no path traversal possible
-- [ ] `[HIGH]` HTML output is escaped to prevent XSS (or a framework handles this automatically)
+- [ ] Pydantic string/list fields have max lengths; request schemas use `extra="forbid"` where practical
+- [ ] `[HIGH]` Mass assignment: Create/Update schemas don't accept privileged fields (`role`, `is_admin`, `owner_id`, `id`, etc.) that the user shouldn't set
+- [ ] `[CRITICAL]` SQL queries use parameterized statements or ORM — no string concatenation (including `text()` with f-strings)
+- [ ] `[CRITICAL]` No shell command built from user input (`subprocess` with `shell=True`, `os.system`)
+- [ ] `[HIGH]` HTML output is escaped to prevent XSS (Jinja2 autoescape on, no `|safe` on user data)
+- [ ] `[HIGH]` SSRF: if the app fetches any user-supplied URL (webhooks, image import, link preview), it blocks private/internal ranges (RFC1918, `127.0.0.0/8`, `169.254.0.0/16`, IPv6 equivalents) and Docker service names, and re-checks after redirects
 - [ ] Redirects only go to allowed/relative URLs — no open redirect
-- [ ] `[HIGH]` JSON deserialization does not allow arbitrary object instantiation
+- [ ] `[HIGH]` Deserialization does not allow arbitrary object instantiation (no `pickle`/`yaml.load` on untrusted data; XML parsed with `defusedxml`)
+- [ ] User-supplied regexes are not executed, and server regexes aren't vulnerable to catastrophic backtracking (ReDoS)
 
 ### 3d. HTTP & Nginx Security Headers
-Verify the Nginx config for the proxy host includes:
-- [ ] `X-Frame-Options: DENY` or `SAMEORIGIN`
+Verify the NPM proxy host config (on the NPM server) and/or app responses include:
 - [ ] `X-Content-Type-Options: nosniff`
 - [ ] `X-XSS-Protection: 0` or omitted — the legacy `1; mode=block` value is deprecated and can introduce vulnerabilities; CSP replaces it
 - [ ] `Referrer-Policy: strict-origin-when-cross-origin`
-- [ ] `Content-Security-Policy` header defined (even if broad to start)
+- [ ] `[HIGH]` `Content-Security-Policy` defined, and of usable quality:
+  - no `'unsafe-inline'` / `'unsafe-eval'` in `script-src` (or a documented reason)
+  - `frame-ancestors 'none'` (or `'self'`) — supersedes `X-Frame-Options`, though sending both is fine
+  - `object-src 'none'` and `base-uri 'self'` (or `'none'`)
 - [ ] `Permissions-Policy` restricts browser features the app doesn't use (e.g. `camera=(), microphone=(), geolocation=()`)
-- [ ] `[MEDIUM]` `Strict-Transport-Security` (HSTS) with `max-age` >= 31536000
-- [ ] Server version header suppressed (`server_tokens off`)
+- [ ] `[MEDIUM]` `Strict-Transport-Security` (HSTS) with `max-age` >= 31536000 (add `includeSubDomains` only if every subdomain is HTTPS)
+- [ ] Authenticated responses send `Cache-Control: no-store`
+- [ ] `[HIGH]` NPM header gotcha: if the proxy host has **Custom Locations** or an `add_header` inside a `location` block in Advanced config, nginx drops the server-level `add_header` directives for that location. Headers must be repeated per location — verify every route, not just `/`
+- [ ] Server version header suppressed on the NPM server (`server_tokens off`) and the app doesn't send its own `Server`/`X-Powered-By` with versions
 - [ ] Unnecessary HTTP methods disabled (e.g. TRACE, DELETE if not used)
 
 ### 3e. TLS / HTTPS
 - [ ] `[CRITICAL]` TLS certificate is valid and not self-signed for production
 - [ ] `[HIGH]` HTTP traffic redirects to HTTPS (not served in parallel)
-- [ ] TLS 1.0 and 1.1 disabled — only TLS 1.2+ allowed
+- [ ] TLS 1.0 and 1.1 disabled — only TLS 1.2+ allowed (1.3 preferred)
 - [ ] Weak cipher suites disabled
 - [ ] Certificate expiry is monitored (NPM auto-renews, but verify it's configured)
+- [ ] DNS has a `CAA` record limiting which CAs can issue certificates for the domain (e.g. `0 issue "letsencrypt.org"`)
+- [ ] NPM → app hop: either proxied over HTTPS to the app host, **or** plain HTTP is a documented, accepted risk because the link is restricted by firewall rules (see 3g). This traffic leaves the NPM server and crosses a routed network, so it is not loopback-only
 
 ### 3f. Docker & Container Security
 - [ ] `[HIGH]` Containers do not run as root (check `user:` in Compose or Dockerfile `USER` instruction)
 - [ ] `[HIGH]` No container has `privileged: true` unless there is a documented reason
 - [ ] `[HIGH]` No unnecessary host volume mounts (especially `/var/run/docker.sock` unless intentional)
-- [ ] Container images are not using `latest` tag in production
-- [ ] `[CRITICAL]` Docker socket is not exposed to the external network
+- [ ] `[HIGH]` `cap_drop: [ALL]` on app containers (add back only specific capabilities with a documented reason)
+- [ ] `[HIGH]` `security_opt: ["no-new-privileges:true"]` on every container
+- [ ] `read_only: true` root filesystem on the app container, with `tmpfs` for paths that need writes (e.g. `/tmp`) and named volumes for real data
+- [ ] `pids_limit` set to prevent fork bombs
 - [ ] Resource limits (`mem_limit`, `cpus`) are set on containers
+- [ ] Images pinned to a specific version tag — preferably by digest (`image: name:tag@sha256:...`) — never `latest`
+- [ ] Dockerfile uses a multi-stage build; the final image is slim (no compilers, build tools, or package caches)
+- [ ] `hadolint Dockerfile` has no errors (warnings reviewed)
+- [ ] `[CRITICAL]` Docker socket is not exposed over TCP or to any network
 
-### 3g. Network & Exposure
-- [ ] `[CRITICAL]` Only port 80/443 are exposed publicly — no app ports (e.g. 8000, 3000) directly open to internet
+### 3g. Network & Exposure (separate NPM server)
+Topology reminder: NPM is on its own server and network, so the app port **must** be published on the app host. The goal is to make it reachable **only** from NPM.
+
+- [ ] `[CRITICAL]` Only 80/443 are exposed publicly, and only on the NPM server — the app host has no port-forward / NAT rule from the internet
+- [ ] `[CRITICAL]` Only the single app/web port is published on the app host (`ports:`); every other service (DB, Redis, workers) uses `expose:` or no ports at all
+- [ ] `[CRITICAL]` Database ports (3306, 5432, 6379) are NOT published on the host — the DB sits on a Docker network marked `internal: true` that only the app container joins
+- [ ] `[CRITICAL]` Docker bypasses UFW/firewalld for published ports. Source restriction to NPM's IP is enforced in the **`DOCKER-USER`** iptables chain on the app host — a UFW rule alone does nothing. Example:
+  ```bash
+  iptables -I DOCKER-USER -i <ext_if> -p tcp -m conntrack --ctorigdstport <app_port> --ctdir ORIGINAL ! -s <NPM_IP> -j DROP
+  ```
+  Rules are persisted across reboots (e.g. `iptables-persistent`), and repeated with `ip6tables` if the host has IPv6
+- [ ] `[HIGH]` The firewall between the NPM network and the app network allows only `<NPM_IP>` → `<app_host>:<app_port>` (single source, single port — not "any" or a whole subnet)
+- [ ] `[HIGH]` Hosts on the app host's **own** network can't reach the app port directly. The inter-network firewall doesn't filter same-segment traffic — only the `DOCKER-USER` rule does. Direct access would bypass NPM's access lists, headers, and rate limits
+- [ ] If published ports are bound to a specific interface IP (`<host_ip>:<port>:<port>`), it's the interface facing NPM, not `0.0.0.0`
 - [ ] NPM proxy host has access list or basic auth if the service is internal-only
 - [ ] Rate limiting is configured in Nginx or the app for API endpoints
 - [ ] `[HIGH]` The service does not expose an admin panel (e.g. `/admin`, `/dashboard`) without additional auth
 - [ ] `[HIGH]` API docs (`/docs`, `/redoc`, `/openapi.json` in FastAPI) are disabled or behind auth in production
 - [ ] `[HIGH]` Debug mode is off (`debug=False`, no auto-reload, no verbose tracebacks returned to clients)
-- [ ] `[CRITICAL]` Database ports (3306, 5432, 6379) are NOT exposed beyond the Docker network
 - [ ] SSH is not running inside any container
 
 ### 3h. Database Access
 - [ ] `[HIGH]` App connects with a dedicated MySQL user limited to its own database — not `root`
-- [ ] App DB user has only the privileges it needs (no `GRANT`, `SUPER`, or global privileges)
+- [ ] App DB user has only the privileges it needs (no `GRANT`, `SUPER`, `FILE`, or global privileges)
+- [ ] App DB user's host is restricted to the Docker network (not `'%'` if avoidable)
 - [ ] MySQL root password is separate from the app user's password and stored only in `.env`
+- [ ] `local_infile` is disabled on the MySQL server
 
 ### 3i. Dependency & Supply Chain
-- [ ] Dependencies are pinned to specific versions (not `*` or `latest`)
-- [ ] Known CVEs in dependencies? (run `trivy fs .` or `pip-audit` / `npm audit`)
+- [ ] Dependencies are pinned to exact versions via a lockfile with hashes (`pip-compile --generate-hashes`, `uv lock`, or `package-lock.json`) and installed with hash checking (`pip install --require-hashes`)
+- [ ] Known CVEs in dependencies? (run `trivy fs .`, `pip-audit`, or `npm audit`)
 - [ ] No abandoned or unmaintained packages with known issues
+- [ ] Newly published package versions are not adopted immediately — wait ~7 days (e.g. uv `exclude-newer`, pnpm `minimumReleaseAge`, Renovate `minimumReleaseAge`) to avoid compromised releases
 - [ ] Docker base images are from official/verified sources
+- [ ] Security scans (gitleaks, bandit/semgrep, trivy) run automatically in Gitea Actions on push, so go-live isn't the first time they run
 
 ### 3j. Request & Session Hardening
 - [ ] `[HIGH]` Session/auth cookies set `Secure`, `HttpOnly`, and `SameSite=Strict` or `Lax`
 - [ ] `[HIGH]` CSRF protection is in place for state-changing (non-GET) endpoints, or the API is stateless/token-authenticated (not cookie-authenticated) and doesn't need it
 - [ ] `[HIGH]` CORS policy restricts `Access-Control-Allow-Origin` to known origins — no wildcard `*` combined with `Access-Control-Allow-Credentials: true`
-- [ ] File uploads enforce a max size limit and restrict allowed file types/extensions
+- [ ] `[HIGH]` Host header validation: `TrustedHostMiddleware` (or equivalent) allows only the public hostname(s), and links in emails (password reset etc.) are built from a configured base URL, never from the request `Host` header
+- [ ] `[HIGH]` Trusted proxy: the app trusts `X-Forwarded-For`/`X-Forwarded-Proto` **only** from NPM's IP (e.g. uvicorn `--proxy-headers --forwarded-allow-ips=<NPM_IP>`, never `*`). Verify the source IP the app actually sees — if the inter-network router/firewall NATs traffic, the app sees the router's IP instead and the setting must match that
+- [ ] Request body size is capped on the NPM proxy host (`client_max_body_size`) and in the app for upload endpoints
+- [ ] NPM proxy timeouts are set to sane values (slow-client / slowloris protection)
+- [ ] If rate limiting is in the app: the limiter uses shared storage (e.g. Redis) when running multiple workers — an in-memory limiter is per-worker, multiplying the real limit
+- [ ] Rate limits cover login, registration, password reset, and expensive endpoints
+- [ ] File uploads:
+  - enforce a max size limit
+  - validate actual content type (magic bytes), not just the extension
+  - are stored outside any served path with random server-generated names (no path traversal possible)
+  - are served with `Content-Disposition: attachment` and the correct `Content-Type`
+  - archives are checked for decompressed size (zip bombs)
 
 ### 3k. Backups & Recovery
 - [ ] `[HIGH]` Database/data volumes have an automated backup process (cron, script, or handled by the DB image)
 - [ ] Backups are stored outside the container (separate volume, external disk, or remote target) so they survive the container/host being wiped
 - [ ] A restore from backup has been tested at least once, not just assumed to work
 - [ ] Backup files containing sensitive data are encrypted or access-restricted
-- [ ] Live `.env` and any config that only exists in Portainer/Dockhand or NPM are backed up somewhere retrievable (they aren't in git)
+- [ ] Live `.env` and any config that only exists in Portainer/Dockhand are backed up somewhere retrievable (they aren't in git)
+- [ ] NPM server config, database, and certificates are backed up from the NPM server (separate host — not covered by the app host's backups)
+- [ ] App host `DOCKER-USER` rules and inter-network firewall rules are documented so they can be rebuilt
 
 ### 3l. Logging & Observability
 - [ ] Logs are rotated or size-capped (Docker `logging.driver`/`max-size`, or app-level rotation) so they can't fill disk over time
 - [ ] Logs persist outside the container (volume-mounted or shipped elsewhere) so they survive a container restart/crash
-- [ ] The app trusts `X-Forwarded-For`/`X-Real-IP` from the proxy correctly, so rate limiting and audit logs record the real client IP, not NPM's
+- [ ] Logs record the real client IP (depends on the trusted proxy setting in 3j), not NPM's IP
 - [ ] `[HIGH]` Logs never contain passwords, tokens, session IDs, API keys, or full request bodies with sensitive fields
+- [ ] Security events (logins, failed logins, permission changes, admin actions) are written to a durable audit log/table with timestamp, user, and client IP — Ntfy alerts are not a record
+- [ ] App host and NPM server clocks are NTP-synced so logs from both can be correlated
+
+### 3m. NPM Server
+NPM is the internet-facing host. If it's compromised, an attacker can read or alter all proxied traffic.
+
+- [ ] `[CRITICAL]` NPM admin UI (port 81) is not reachable from the internet — LAN/VPN only
+- [ ] `[HIGH]` NPM admin account uses a strong unique password (default `admin@example.com` account changed/removed)
+- [ ] NPM container and host OS are patched and on a current version
+- [ ] SSH on the NPM server is key-only, no root login, and not exposed to the internet
+- [ ] The NPM server can reach only the app host ports it proxies to, not the entire internal network
+- [ ] `[MEDIUM]` Automated ban layer (CrowdSec or fail2ban reading NPM logs) blocks scanners and brute-force sources
+- [ ] "Block Common Exploits" is enabled on the proxy host
 
 **AI Action:** Run the following tools if available:
 - `bandit -r . -ll` — Python static security analysis
+- `semgrep scan --config p/python --config p/secrets` — additional static analysis
+- `pip-audit -r requirements.txt` — Python dependency CVEs
 - `trivy fs . --severity HIGH,CRITICAL` — dependency and filesystem CVE scan
 - `gitleaks git .` — secret scan across full git history (older gitleaks versions: `gitleaks detect --source .`)
+- `hadolint Dockerfile` — Dockerfile lint
 - `docker scout cves <image>` or `trivy image <image>` — container image vulnerability scan
 
+Ask the user to paste `iptables -S DOCKER-USER` (and `ip6tables -S DOCKER-USER` if IPv6 is on) from the app host and the relevant inter-network firewall rules — the AI can't see them directly.
+
 Report all FAIL/WARN findings. Do not proceed to Section 4 until CRITICAL and HIGH issues are resolved.
 
 ---
 
 ## Section 4: External Verification
 
-Goal: Confirm the live service matches what the config says. Sections 1–3 check files — this checks what the internet actually sees.
+Goal: Confirm the live service matches what the config says. Sections 1–3 check files — this checks what the network actually sees.
 
-Only start this section once Sections 1–3 have no unresolved CRITICAL/HIGH findings. Enable the NPM proxy host, then run these checks immediately. If any CRITICAL/HIGH check fails, disable the proxy host until it's fixed.
+Only start this section once Sections 1–3 have no unresolved CRITICAL/HIGH findings.
 
-- [ ] `[HIGH]` Security headers from 3d are present in the live response
+### 4a. Staged Exposure
+1. Enable the NPM proxy host **with an access list restricted to the admin's public IP**
+2. Run all checks in 4b and 4c against the public URL
+3. Only after they pass, remove the access list to open the service to everyone
+4. Re-run the header and redirect checks once open
+
+If any CRITICAL/HIGH check fails at any point, disable the proxy host (or re-apply the access list) until it's fixed.
+
+### 4b. Public Checks
+- [ ] `[HIGH]` Security headers from 3d are present in the live response — on `/` **and** on every custom location / API route
 - [ ] `[HIGH]` HTTP → HTTPS redirect works on the public URL
-- [ ] `[MEDIUM]` SSL Labs test (https://www.ssllabs.com/ssltest/) scores A or better
+- [ ] `[MEDIUM]` SSL Labs test (https://www.ssllabs.com/ssltest/) or `testssl.sh` scores A or better
 - [ ] securityheaders.com (https://securityheaders.com) scores A or better, or gaps are documented
-- [ ] `[CRITICAL]` External port scan shows only 80/443 open — run from outside the LAN (e.g. phone hotspot) or an online port scanner, not from inside the network
+- [ ] `[CRITICAL]` External port scan of the NPM server shows only 80/443 open — run from outside the LAN (e.g. phone hotspot) or an online port scanner, over **IPv4 and IPv6**
 - [ ] `[HIGH]` `/docs`, `/redoc`, `/openapi.json`, and any admin paths return 401/403/404 from the public URL
 - [ ] Error pages don't leak stack traces or server/framework versions (request a nonexistent path and a malformed request)
+- [ ] `[HIGH]` OWASP ZAP baseline scan has no unaddressed High findings
 - [ ] Uptime Kuma monitor shows the service as up
 
+### 4c. Internal Reachability Checks
+- [ ] `[CRITICAL]` From a machine on the **app host's network** (not NPM): the app port is refused/filtered
+- [ ] `[CRITICAL]` From the NPM server: the app port responds, and no other app host ports (DB, Portainer/Dockhand agent, etc.) are reachable
+- [ ] The real client IP shows up in app logs when the service is accessed through the public URL
+
 **AI Action:** Run:
-- `curl -sI https://<public-url>` — verify headers
+- `curl -sI https://<public-url>` (and each custom location / API route) — verify headers
 - `curl -sI http://<public-url>` — verify redirect
 - `curl -s -o /dev/null -w "%{http_code}" https://<public-url>/docs` (repeat for `/redoc`, `/openapi.json`, admin paths)
 - `echo | openssl s_client -connect <host>:443 -servername <host> 2>/dev/null | openssl x509 -noout -dates -issuer`
+- `testssl.sh --severity MEDIUM https://<public-url>`
+- `docker run --rm -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t https://<public-url>`
 
-Ask the user to run the SSL Labs, securityheaders.com, and external port scan checks and paste the results.
+Ask the user to run the external port scan, the internal reachability checks (`nmap -Pn -p- <app_host>` from a same-network machine and from the NPM server), and the SSL Labs/securityheaders.com checks if not automated, then paste the results.
 
 ---
 
 ## Section 5: Go-Live Decision
 
-- [ ] Is a rollback or "what to do if this breaks/is compromised" step documented (README or runbook) for this service?
+- [ ] An incident runbook exists (README or `RUNBOOK.md`) covering:
+  - how to take the service offline fast (disable the NPM proxy host)
+  - how to rotate every secret (DB passwords, `SECRET_KEY`, API keys, Ntfy token) and where each is stored
+  - how to invalidate all sessions (rotate the signing key)
+  - how to roll back to the previous image tag/digest
+  - how to restore from backup, and roughly how long it takes
 
 After all sections are complete:
 
@@ -271,5 +380,7 @@ Goal: Record what went live so the `service-update` maintenance playbook has som
 - [ ] Go-live version tagged in Gitea (e.g. `v1.0.0`) and pushed
 - [ ] `Release-Notes/v1.0.md` created
 - [ ] Running image tags/digests recorded in the report
+- [ ] SBOM generated for each running image and saved alongside the report (`trivy image --format cyclonedx --output reports/sbom-<image>-<YYYY-MM-DD>.json <image>`)
+- [ ] App host `DOCKER-USER` rules and inter-network firewall rules recorded in the report
 - [ ] Deferred items recorded in the report with the reason for deferring
 - [ ] Report saved to `reports/golive-<YYYY-MM-DD>.md` in the project directory — the first maintenance review uses it as its prior report

preferences/python-web.md

@@ -138,33 +138,46 @@ services:
 ## Ntfy Notifications
 
 - All web apps must integrate Ntfy push notifications for admin-relevant events
-- Use a self-hosted Ntfy instance; URL and topic stored in `.env` — never hardcoded
+- Use a self-hosted Ntfy instance; URL, topic, and access token stored in `.env` — never hardcoded
+- The topic must be access-protected on the Ntfy server so only the admin can subscribe; the app publishes with a token
 - Send notifications via HTTP POST using `httpx` (async) — do not add a separate Ntfy library
 
 ```python
 # app/utils/notify.py
+import logging
+
 import httpx
 from app.config import settings
 
+logger = logging.getLogger(__name__)
+
 async def notify(title: str, message: str, priority: str = "default", tags: list[str] | None = None):
+    """Send an admin notification. Never raises — a failed notification must not break the request."""
     headers = {
         "Title": title,
         "Priority": priority,
+        "Authorization": f"Bearer {settings.ntfy_token}",
     }
     if tags:
         headers["Tags"] = ",".join(tags)
-    async with httpx.AsyncClient() as client:
-        await client.post(
-            f"{settings.ntfy_url}/{settings.ntfy_topic}",
-            content=message,
-            headers=headers,
-        )
+    try:
+        # Short timeout so a slow/unreachable Ntfy server can't stall the app
+        async with httpx.AsyncClient(timeout=5.0) as client:
+            response = await client.post(
+                f"{settings.ntfy_url}/{settings.ntfy_topic}",
+                content=message,
+                headers=headers,
+            )
+            response.raise_for_status()
+    except Exception as exc:
+        logger.error("Ntfy notification failed: %s", exc)
 ```
 
 **Required `.env` variables:**
 ```
 NTFY_URL=https://ntfy.example.com
 NTFY_TOPIC=your-topic
+NTFY_TOKEN=tk_your-access-token
 ```
 
 **Standard events to notify on** (adapt to what the app actually does):