homeschool/reports/maintenance-2026-09-24.md

# Maintenance Review — 2026-09-24

- **Service:** Homeschool Dashboard (`homeschool.chns.tech`)
- **Host:** `docker-20-3` (app), NPM on `Docker-22-1` (172.16.22.21) → `docker-20-3:8054`
- **Repo HEAD:** `8e92ae6` (main, clean, in sync with origin). No git tags exist.
- **Prior report:** none. This is the first maintenance review, and no go-live report exists.
- **Playbook:** service-update.md

## Rollback point (running images)

| Container | Image | Image ID | Built |
|---|---|---|---|
| homeschool_db | mysql:8.0.40 | sha256:6c55ddbef969… | 2024-10-14 |
| homeschool_backend | homeschool-backend | sha256:f72d140376b1… | 2026-03-23 08:43 |
| homeschool_frontend | homeschool-frontend | sha256:406c3c0faba3… | 2026-03-23 08:29 |

App images are only tagged `homeschool-*:latest` (local build). A rebuild overwrites them, so re-tag them before any update: `docker tag homeschool-backend homeschool-backend:rollback-20260924`, and the same for the frontend.

---

## Section 1: Baseline & Drift

| Check | Result | Notes |
|---|---|---|
| Tree clean and pushed | PASS | |
| Running version matches `main` | **FAIL** | The backend code matches HEAD. The **frontend bundle predates `8e92ae6`**: the live JS still has the old `>=-15` meeting catch-up window, not `>=-300`. The fix from Mar 31 was never deployed. |
| Live stack matches repo compose | USER | Containers are named per the repo compose. Confirm no edits were made in the Portainer UI. |
| NPM proxy config documented | USER | README has an NPM section. Confirm the live proxy host matches it. |
| `.env` vs `.env.example` | WARN | All variables are set. `DOCS_ENABLED` is in both files but is **not passed to the container** in `docker-compose.yml`, so it has no effect. This fails safe (docs stay off), but README says setting it enables docs. |
| Rollback digests recorded | PASS | See table above. No release tags. |

## Section 2: Runtime Health

| Check | Result | Notes |
|---|---|---|
| All containers healthy | WARN | Only `db` has a healthcheck. `backend` and `frontend` show only `Up`. |
| Restart count | WARN | backend RestartCount=4. It crash-loops about 4 times after **every host reboot** (09-06, 09-08, 09-13, 09-17): `Can't connect to MySQL server on 'db'` → startup failed. It recovers on its own. On reboot, the `restart:` policy ignores `depends_on: service_healthy`. |
| Public health endpoint | PASS | `https://homeschool.chns.tech/api/health` → 200 |
| Logs (30 days) | WARN | Only the reboot crash loops above. 11× 429 on `/ws/…` (see the rate-limit finding in 5b). ~2.7k 404/405 from WordPress scanners (harmless). |
| CPU/memory headroom | WARN | db 421 MiB / 512 MiB (**82%**). backend 14%, frontend 6%. |
| Disk space | PASS | `/` 45% used (51 GB free) |
| Log rotation | **FAIL** | No `logging:` in compose and no `/etc/docker/daemon.json`. json-file logs grow without limit. |
| Reclaimable Docker space | WARN | Host-wide: 5.1 GB build cache and 6.6 GB unused images can be reclaimed. |

## Section 3: Functional Check (user to confirm)

The app has no scheduled/cron jobs. Its background work is WebSocket broadcasts and startup migrations only.

- [ ] Login and logout. The access token expires after 30 minutes and should auto-refresh through the cookie.
- [ ] Dashboard: start a day session, then start, pause, resume and complete a block.
- [ ] TV view `/tv/<token>` updates live over WebSocket.
- [ ] Schedules: create/edit a template. Admin: add a child or subject.
- [ ] Logs view loads.
- [ ] Super admin login at `/super-admin/login` → user list. The ntfy alert arrives **and shows the real client IP**.
- [ ] DB connectivity: PASS. The only DB errors are the startup race on reboot.

## Section 4: Backups & Recovery — **CRITICAL**

| Check | Result | Notes |
|---|---|---|
| Recent DB backup | **FAIL (CRITICAL)** | No backup job found: no user crontab, no cron.d entry, no systemd timer, no backup container, no dump files. The only copy of the data is the `homeschool_mysql_data` volume on this host. **USER:** confirm whether VM-level snapshots or backups exist outside this host. |
| Backup size / integrity | FAIL | No backups exist. |
| Stored off-host | FAIL | |
| Upload volumes | N/A | No upload volumes. The DB is the only state. |
| `.env` backed up | USER | |
| Retention | FAIL | |
| Test restore in last 6 months | FAIL | |

## Section 5: Security Posture

### 5a. TLS & Headers
| Check | Result | Notes |
|---|---|---|
| Cert valid >14 days | PASS | Let's Encrypt, expires 2026-12-19 (86 days). Confirm auto-renew is on in NPM. |
| HTTP → HTTPS redirect | USER | Port 80 timed out from inside the LAN (hairpin/split DNS). Run `curl -sI http://homeschool.chns.tech` from outside. |
| Security headers | PASS | XFO, XCTO, Referrer-Policy, CSP and HSTS are all present. |
| Server version suppressed | PASS | `server: openresty` (no version). |

### 5b. Exposure
| Check | Result | Notes |
|---|---|---|
| Only 80/443 public | WARN | `8054` is published on `0.0.0.0` of `docker-20-3` (needed for the NPM host). Confirm a firewall limits it to 172.16.22.21 so the LAN can't bypass TLS. DB/backend ports are not published. |
| **Rate limiting per client** | **FAIL (HIGH)** | Frontend nginx sees **every** request as coming from the NPM host (20,450 of 20,451 log lines = 172.16.22.21). There is no `real_ip` config, so `limit_req` keyed on `$binary_remote_addr` is **global**. The login limit (5/min) applies to all users combined, and one brute-forcer can lock everyone out. The TV/WS limit (10/min) is shared too, which caused today's 429s on `/ws/128699`. Fix: add `set_real_ip_from 172.16.22.21; real_ip_header X-Real-IP;` (or X-Forwarded-For). |
| Spoofable client IP in alerts | WARN | `auth.py:62,86` and `admin.py:26` take the **first** `X-Forwarded-For` entry, which the client controls. An attacker can forge the IP shown in ntfy alerts. Use the entry added by the trusted proxy (the last one) or `X-Real-IP`. |
| No unreviewed containers/ports | PASS | 3 services, same as compose. |

### 5c. Accounts & Secrets
| Check | Result | Notes |
|---|---|---|
| Account review | USER | The DB query was not run: the tool permission policy blocked it. Run the query below. |
| Secret rotation | WARN | `.env` last modified 2026-03-22 (about 6 months). Secrets have never been rotated since then. |
| Failed-login spikes | PASS | 0 auth rate-limit hits. 54× 401 are almost all `/api/auth/refresh` (expired sessions). **But** the per-IP limits are broken (5b), so this signal is unreliable. |

```
docker exec -it homeschool_db sh -c 'mysql -u"$MYSQL_USER" -p"$MYSQL_PASSWORD" "$MYSQL_DATABASE" -e "SELECT id, email, is_active, created_at FROM users ORDER BY id;"'
```

### 5d. Container Hardening
| Check | Result | Notes |
|---|---|---|
| Non-root / no privileged / no sock | PASS / WARN | backend runs as `appuser`. The frontend nginx master runs as root (stock image; consider `nginxinc/nginx-unprivileged`). None are privileged, and there is no docker.sock mount. |
| Resource limits | PASS | mem_limit and cpus are set on all three. |

## Section 6: Vulnerabilities & Currency

Scanned with trivy image, HIGH/CRITICAL only.

| Image | CRIT | HIGH | Key fixable items |
|---|---|---|---|
| homeschool-backend (OS, debian 13.4) | 19 | 716 | openssl/libssl3 3.5.5-1~deb13u1 → deb13u2+, perl-base, gzip, util-linux, linux-libc-dev. Rebuild with an updated base and `apt-get upgrade`. Drop `libmariadb-dev`/`gcc` from runtime (aiomysql is pure Python). That removes the no-fix mariadb CRITs. |
| homeschool-backend (Python) | 1 | 11 | **anyio 4.12.1→4.14.2 (CRIT)**, starlette 0.38.6 (via fastapi 0.115.0), PyJWT 2.12.0→2.13.0, cryptography 46.0.5→48.0.1+, python-multipart 0.0.22→0.0.27+, Mako 1.3.10→1.3.12 (via alembic) |
| homeschool-frontend (alpine 3.23.3) | 2 | 50 | libssl3/libcrypto3 3.5.5→3.5.8, curl, libexpat, musl, zlib. Rebuild on the latest nginx alpine. |
| mysql:8.0.40 | 0 | 130 | Many el9 package fixes |

| Check | Result | Notes |
|---|---|---|
| Image CVEs | **FAIL (HIGH)** | See above. A rebuild fixes most of them. |
| App dependency CVEs | **FAIL (HIGH)** | Python list above. Frontend: **no `package-lock.json`**, so `npm audit` can't run and each build resolves transitive deps fresh. Known: axios 1.7.7 has advisories fixed in ≥1.12. vite 5.4.10 is build-only. |
| Runtimes EOL | **FAIL** | **MySQL 8.0 reached EOL in April 2026.** Move to 8.4 LTS. **Node 20 reached EOL on 2026-04-30** (build stage only). Move to Node 22/24 LTS. Python 3.12 is supported. nginx 1.29 mainline is superseded by 1.30.x stable. |
| Abandoned deps | WARN | `passlib` 1.7.4 is unmaintained (last release 2020) and forces `bcrypt==3.2.2`. Consider using `bcrypt` directly. |
| Pinning | PASS / WARN | Images and top-level deps are pinned. There is no npm lockfile (above). |

## Section 7: Applying Updates — HIGH findings resolved

This section was first blocked on backups. The user then asked for the HIGH findings to be fixed the same day, so a one-off pre-update backup was taken.

| Check | Result | Notes |
|---|---|---|
| Fresh backup before update | PASS | `mysqldump` → `~/backups/homeschool/homeschool-pre-mysql84-20260924-2327.sql.gz` (14 tables). Volume copy → `homeschool_mysql_data_pre84_20260924` (187 MB). **Both are on-host only.** |
| Rollback point recorded | PASS | `homeschool-backend:rollback-20260924`, `homeschool-frontend:rollback-20260924`, plus the table at the top |
| Major-version release notes | PASS | MySQL 8.0 → 8.4 in-place upgrade is supported. The app user already uses caching_sha2. |
| Pinned versions | PASS | All bumped versions are pinned exactly. |
| Tested against a copy of real data | PASS | Dump restored into a throwaway `mysql:8.4.11`. The new backend was smoke-tested there: register, login, children, subjects, schedules, logs, super admin, forged/`alg=none` JWT rejected, refresh. **Caught a regression:** unpinned PyMySQL floated to 1.2.3 and broke SQLAlchemy 2.0.35 ping. It is now pinned to 1.1.2. |
| Real-IP rate limiting tested | PASS | Test nginx: client A was limited after its burst, client B was unaffected, and a forged XFF prefix resolved to the true client. |
| Deploy | PASS* | The MySQL data upgrade (80040 → 80411) took about 90s, longer than the healthcheck window, so compose aborted the dependent containers. The user started them manually. Fixed by adding `start_period: 180s` to the db healthcheck. |
| Sections 2/3 re-run | PASS (partial) | All containers up, backend logs clean, public `/api/health` 200. Live bundle now contains `8e92ae6`. nginx logs real client IPs. The user confirmed the site is back up. |
| Version tag / release notes | PASS | `v1.1.0`, `Release-Notes/v1.1.md` |
| Old images cleanup | DEFERRED | Keep the rollback tags and volume copy until the update has been stable for about a week. Then remove them: `docker rmi homeschool-{backend,frontend}:rollback-20260924 mysql:8.0.40` and `docker volume rm homeschool_mysql_data_pre84_20260924`. |

### Changes applied

- `frontend/nginx.conf`: `set_real_ip_from` the NPM host and Cloudflare ranges, `real_ip_header X-Forwarded-For`, `real_ip_recursive on`
- `backend/requirements.txt`:
  - fastapi 0.115.0 → 0.141.1, starlette → 1.7.0 (pinned)
  - anyio 4.15.1 (pinned), PyJWT 2.12.0 → 2.15.0, cryptography 46.0.5 → 50.0.1
  - python-multipart 0.0.22 → 0.0.32, alembic 1.13.3 → 1.20.0, Mako 1.4.3 (pinned), PyMySQL 1.1.2 (pinned)
- `backend/Dockerfile`: python 3.12.13 → 3.12.14-slim. Dropped gcc / libmysqlclient-dev / pkg-config (not needed). Added `apt-get upgrade`.
- `frontend/Dockerfile`: nginx 1.29.6 → 1.30.5-alpine (stable) plus `apk upgrade`
- `docker-compose.yml`: mysql 8.0.40 → 8.4.11. db healthcheck `start_period: 180s`.

### New running images

| Container | Image | Image ID |
|---|---|---|
| homeschool_db | mysql:8.4.11 | sha256:ee241324a55f… |
| homeschool_backend | homeschool-backend (v1.1.0) | sha256:1eb66dc84e67… |
| homeschool_frontend | homeschool-frontend (v1.1.0) | sha256:50f1de90c447… |

### Post-deploy regression

After the HIGH deploy, `POST /api/sessions/{id}/timer` returned **500 on every call**: 8 failures, 0 successes, starting 06:43 UTC. FastAPI 0.141 raises on a failed lazy-load of `current_block.subject.options` during response serialization, where 0.115 silently skipped it. The first smoke test didn't cover timer actions. Fixed in v1.1.0 by eager-loading `Subject.options` in `sessions.py` and `dashboard.py`. The smoke test now covers the full timer lifecycle, the TV dashboard with an active block, schedule/subject writes, and every parameterless GET route (from the OpenAPI spec).

### v1.1.0 deploy verification

All 3 containers are healthy with 0 restarts, and log rotation is active (10m × 3). Public `/api/health` returns 200. Production timer actions return 200 (4 of 4 right after the deploy). Tagged `v1.1.0` and pushed. The pre-review rollback images (`rollback-20260924`) were pruned by the user; `rollback-20260924b` (the HIGH-fix build) remains.

### Post-update scan (trivy HIGH/CRITICAL)

| Image | Before | After |
|---|---|---|
| backend OS | 19 C / 716 H | 0 C / 44 H (no fixes available) |
| backend Python | 1 C / 11 H | 0 |
| frontend | 2 C / 50 H | 0 |
| mysql | 0 C / 130 H | 8.4.11: 4 H curl (OS), 6 H in bundled Python tools, 1 C / 21 H Go stdlib in `gosu`. All are upstream in Oracle's latest image and accepted until the next mysql 8.4.x release. |

## Section 8: Documentation
| Check | Result | Notes |
|---|---|---|
| README accurate | WARN | `DOCS_ENABLED` is documented but not wired into compose (Section 1). |
| Rollback runbook | **FAIL** | README has "Stopping and Restarting" but no backup, restore or rollback procedure. |
| Deferred items recorded | PASS | This report. |

---

## Section 9: Summary

### CRITICAL
1. **No database backups.** No backup job, no dump files, and nothing stored off-host has been found.

### HIGH (all resolved 2026-09-24; see Section 7)
2. ~~Rate limiting is global, not per client.~~ Fixed with real_ip.
3. ~~Fixable HIGH/CRITICAL CVEs in all images and Python deps.~~ Fixed by the rebuild and dependency bumps.
4. ~~MySQL 8.0 is EOL.~~ Upgraded to 8.4.11 LTS.

### MEDIUM (all resolved 2026-09-24, v1.1.0)
5. ~~The frontend is not deployed at HEAD.~~ Resolved by the rebuild.
6. ~~No Docker log rotation.~~ Added `x-logging` anchor (json-file, 10m × 3) to all services.
7. ~~The backend crash-loops on reboot, and there are no backend/frontend healthchecks.~~ `_wait_for_db()` in `main.py` retries for up to 180s. Tested: the backend started before its DB, logged retries, and came up with 0 restarts. Healthchecks were added to both services.
8. ~~Node 20 EOL and no lockfile.~~ Now node:24.21.0-alpine with `package-lock.json` and `npm ci`. axios → 1.20.0, vite → 5.4.21. Remaining npm audit items (vite ≤6.4.2 / esbuild) are dev-server-only and accepted; the fix needs Vite 6.4+.
9. ~~No rollback/restore runbook or release tags.~~ README now has "Backup, Restore & Rollback" and documents real-IP proxy trust. Added `Release-Notes/v1.1.md` and tagged `v1.1.0`.
10. ~~Client IP in ntfy alerts can be spoofed.~~ New `app/utils/client_ip.py` reads `X-Real-IP`, which nginx sets after real_ip resolution. Tested: a forged XFF is ignored.

### LOW
11. db memory is at 82% of its 512 MiB limit.
12. `DOCS_ENABLED` is not passed to the container (README mismatch).
13. `passlib` is unmaintained. The frontend nginx runs as root.
14. Secrets are about 6 months old and have never been rotated.
15. About 11 GB of Docker space can be reclaimed host-wide.

### Pending user input
Portainer/NPM drift, functional smoke tests, ntfy test, off-host/VM backups, HTTP→HTTPS redirect from outside, firewall on 8054, NPM auto-renew, and the user account review.

### Counts
- Total checks: 58
- Passed: 19
- Failed (critical/high): 7. That is backups (counted once, CRITICAL), rate limiting, image CVEs, dependency CVEs, runtime EOL, log rotation and frontend drift.
- Failed (non-critical) / WARN: 18
- Pending user confirmation: 14
- Deferred: 1 (old image cleanup)
- Updates applied: 9 (HIGH: real-IP rate limiting, Python deps, base images, MySQL 8.4. MEDIUM: log rotation, DB-wait + healthchecks, Node 24 + lockfile, runbook + release tag, trusted client IP)

**Overall: NEEDS ATTENTION.** All HIGH and MEDIUM findings are resolved. The CRITICAL no-backups finding is still open: the pre-update dump and volume copy are on-host only, with no schedule.