PH6.6 — Observability and production compose profile #47

Closed
opened 2026-09-11 09:02:20 +00:00 by bart · 2 comments
Collaborator

Parent

Row PH6.6 in docs/12-implementation-plan.md (branch, owned paths and doc section are defined there).

What to build

Operators get meaningful metrics, structured log fields and OpenTelemetry hooks, and a production compose profile with TLS and a persistent or external directory per Q21.

Acceptance criteria

  • /metrics exposes request, task and solver metrics; logs carry request id, user and task id
  • OTel exporter can be enabled by config
  • Prod profile starts with TLS on nginx and LDAP per Q21; Q21 resolved in docs

Blocked by

  • #3 — Compose base: db, LDAP, Keycloak realm, nginx, OSRM
  • #4 — API skeleton: core, router discovery, healthz, db + audit hook, alembic 0001, CLI
## Parent Row **PH6.6** in `docs/12-implementation-plan.md` (branch, owned paths and doc section are defined there). ## What to build Operators get meaningful metrics, structured log fields and OpenTelemetry hooks, and a production compose profile with TLS and a persistent or external directory per Q21. ## Acceptance criteria - [x] `/metrics` exposes request, task and solver metrics; logs carry request id, user and task id - [x] OTel exporter can be enabled by config - [x] Prod profile starts with TLS on nginx and LDAP per Q21; Q21 resolved in docs ## Blocked by - #3 — Compose base: db, LDAP, Keycloak realm, nginx, OSRM - #4 — API skeleton: core, router discovery, healthz, db + audit hook, alembic 0001, CLI
Author
Collaborator

Starting PH6.6 on branch feature/infra-observability.

Owned paths for this session: infra/** (unsealed for this row), backend/src/polaris/core/logging.py, backend/src/polaris/api/metrics.py. Not touching anything else under backend/src/polaris/**, .github/**/.forgejo/** (parallel #8 CI session), or frontend/**.

Plan: extend /metrics with a request-latency histogram, task-queue-depth/claim-latency gauges, and best-effort solver_run metrics (guarded so a fresh clone without the not-yet-merged solver_run migration still serves clean output). Extend logging.py with service/environment/user/task-id fields plus OTel trace/span-id correlation (all via structlog contextvars, matching the existing request_id pattern). OTel tracing wired in as an optional, lazily-imported dependency (no pyproject.toml change) enabled via POLARIS_OTEL_ENABLED/standard OTEL_* env vars, since opentelemetry-* isn't in the dependency list yet -- will flag that as a follow-up seam request rather than touching the sealed pyproject.toml. New infra/compose/docker-compose.prod.yml overlay for TLS on nginx + OpenLDAP, resolving Q21 (persistent storage + TLS as the compose-bundled OpenLDAP's prod default, matching the existing 07/11 doc lean toward shipped OpenLDAP rather than an external directory).

Starting PH6.6 on branch `feature/infra-observability`. Owned paths for this session: `infra/**` (unsealed for this row), `backend/src/polaris/core/logging.py`, `backend/src/polaris/api/metrics.py`. Not touching anything else under `backend/src/polaris/**`, `.github/**`/`.forgejo/**` (parallel #8 CI session), or `frontend/**`. Plan: extend `/metrics` with a request-latency histogram, task-queue-depth/claim-latency gauges, and best-effort solver_run metrics (guarded so a fresh clone without the not-yet-merged solver_run migration still serves clean output). Extend `logging.py` with service/environment/user/task-id fields plus OTel trace/span-id correlation (all via structlog contextvars, matching the existing request_id pattern). OTel tracing wired in as an optional, lazily-imported dependency (no pyproject.toml change) enabled via `POLARIS_OTEL_ENABLED`/standard `OTEL_*` env vars, since `opentelemetry-*` isn't in the dependency list yet -- will flag that as a follow-up seam request rather than touching the sealed `pyproject.toml`. New `infra/compose/docker-compose.prod.yml` overlay for TLS on nginx + OpenLDAP, resolving Q21 (persistent storage + TLS as the compose-bundled OpenLDAP's prod default, matching the existing 07/11 doc lean toward shipped OpenLDAP rather than an external directory).
Author
Collaborator

PR opened: #62 (feature/infra-observability -> main).

All three acceptance criteria checked. Notes/discovered follow-ups (none block this PR, flagging for the record):

  1. task_id in logs: core/logging.py now has bind_task_id/clear_task_id (same bind/unbind pattern api/middleware.py uses for request_id) and it's unit-tested, but nothing calls it yet — the worker's task-claim loop (backend/src/polaris/worker/**, PH0.3, outside this ticket's owned paths) is the natural caller. A one-line addition there (bind_task_id(task.id) around the handler call, clear_task_id() after) would finish wiring it end-to-end.
  2. polaris_http_requests_total (pre-existing, PH0.2) still labels by raw request path, not the route template the new latency histogram uses — unifying them needs a small edit to the sealed api/main.py. Left as-is; flagging as a possible future seam request rather than touching that file.
  3. Real OpenTelemetry packages (opentelemetry-api/sdk/instrumentation-fastapi/exporter-otlp-proto-http) are intentionally not added to pyproject.toml (sealed for this row) — the tracing hook is lazily imported and opt-in (POLARIS_OTEL_ENABLED), so it's a real no-op today. Someone with pyproject.toml access can add the four packages when there's an actual collector to point at.
  4. Q21 resolved as: shipped OpenLDAP is the prod default (persistent storage was already the base compose default; this PR adds TLS via docker-compose.prod.yml). Docs updated in 07-architecture.md and 11-roadmap.md.

docker compose config parses cleanly for base, dev overlay and the new prod overlay; ruff/mypy --strict/pytest (unit + integration against a real testcontainers Postgres) all green on the backend files touched.

PR opened: #62 (`feature/infra-observability` -> `main`). All three acceptance criteria checked. Notes/discovered follow-ups (none block this PR, flagging for the record): 1. `task_id` in logs: `core/logging.py` now has `bind_task_id`/`clear_task_id` (same bind/unbind pattern `api/middleware.py` uses for `request_id`) and it's unit-tested, but nothing calls it yet — the worker's task-claim loop (`backend/src/polaris/worker/**`, PH0.3, outside this ticket's owned paths) is the natural caller. A one-line addition there (`bind_task_id(task.id)` around the handler call, `clear_task_id()` after) would finish wiring it end-to-end. 2. `polaris_http_requests_total` (pre-existing, PH0.2) still labels by raw request path, not the route template the new latency histogram uses — unifying them needs a small edit to the sealed `api/main.py`. Left as-is; flagging as a possible future seam request rather than touching that file. 3. Real OpenTelemetry packages (`opentelemetry-api`/`sdk`/`instrumentation-fastapi`/`exporter-otlp-proto-http`) are intentionally *not* added to `pyproject.toml` (sealed for this row) — the tracing hook is lazily imported and opt-in (`POLARIS_OTEL_ENABLED`), so it's a real no-op today. Someone with `pyproject.toml` access can add the four packages when there's an actual collector to point at. 4. Q21 resolved as: shipped OpenLDAP is the prod default (persistent storage was already the base compose default; this PR adds TLS via `docker-compose.prod.yml`). Docs updated in `07-architecture.md` and `11-roadmap.md`. `docker compose config` parses cleanly for base, dev overlay and the new prod overlay; `ruff`/`mypy --strict`/`pytest` (unit + integration against a real testcontainers Postgres) all green on the backend files touched.
bart closed this issue 2026-09-11 11:15:27 +00:00
Sign in to join this conversation.
No labels
ready-for-agent
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
patrick/Polaris#47
No description provided.