patrick/Polaris
Watch
1
0
Fork
You've already forked Polaris
 0
Code Issues 35 Pull requests 3 Projects Releases Packages Wiki Activity Actions 3

PH6.6 — Observability and production compose profile #62

Merged
bart merged 4 commits from feature/infra-observability into main 2026-09-11 11:15:27 +00:00
Conversation 0 Commits 4 Files changed 14 +983 -13
bart commented 2026-09-11 11:07:50 +00:00
Collaborator
Copy link

Summary

  • Extends /metrics (backend/src/polaris/api/metrics.py) with a request-latency histogram keyed by method+route-template, task queue-depth/claim-latency gauges, and best-effort solver_run gauges. solver_run isn't migrated on main yet (PH1.2/T-E are still open PRs), so it's reflected for at scrape time rather than imported as a model — the endpoint just renders nothing extra until that table exists, instead of erroring.
  • Extends core/logging.py with service/environment/pid on every log line, an explicit user alias for the existing actor_sub, and bind_task_id/clear_task_id helpers (same bind/unbind-around-the-work pattern api/middleware.py already uses for request_id) for the worker's task-claim loop to adopt.
  • Adds optional OpenTelemetry tracing (install_otel, called from install_metrics): lazily imported and opt-in via POLARIS_OTEL_ENABLED (plus the standard OTEL_* env vars), since opentelemetry-* isn't declared in pyproject.toml (sealed for this row) — disabled or package-missing both degrade to a clean no-op with a warning log line, never an error. trace_id/span_id get stitched into structlog output when a span is current.
  • Adds infra/compose/docker-compose.prod.yml: TLS termination at nginx (infra/nginx/prod.conf, HTTP redirects to HTTPS except the healthcheck) and TLS on the compose-bundled OpenLDAP (LDAPS only, plaintext 389 dropped from the prod stack). Certificates are self-signed for now, generated once per host by infra/nginx/generate-self-signed-cert.sh / infra/ldap/generate-self-signed-cert.sh into git-ignored certs/ directories.
  • Resolves Q21: shipped OpenLDAP is the production directory by default — persistent storage (already the base compose file's default) + TLS (this overlay). An external corporate directory remains a config-only swap, not required. Updated the open-questions tables in docs/07-architecture.md and docs/11-roadmap.md per CLAUDE.md's rule for resolved Qn assumptions.

Scope notes / seam requests for a future ticket

  • api/main.py is sealed for this row, so the new latency middleware is installed from within api/metrics.py:install_metrics (already called by main.py) rather than touching that file. The existing polaris_http_requests_total counter (populated by main.py's own middleware) still uses the raw request path; unifying it onto the same route-template label as the new histogram needs a small main.py change — flagged as a follow-up seam request rather than widening this ticket.
  • bind_task_id/clear_task_id are ready for the worker's task-claim loop (backend/src/polaris/worker/**, owned by PH0.3, outside this ticket's owned paths) to call around a handler invocation; not wired up there in this PR.
  • Real OpenTelemetry dependencies (opentelemetry-api, opentelemetry-sdk, opentelemetry-instrumentation-fastapi, opentelemetry-exporter-otlp-proto-http) aren't added to pyproject.toml (sealed for this row) — the hooks are fully optional/lazily-imported so this isn't a hard dependency; adding the packages for real use is a separate seam request.
  • Keycloak's own TLS/hostname hardening is intentionally out of scope (ticket's bar is "TLS on nginx at minimum"; the browser reaches Keycloak directly for the OIDC redirect, not through nginx).

Test plan

  • docker compose -f infra/compose/docker-compose.yml config parses cleanly
  • docker compose -f infra/compose/docker-compose.yml -f infra/compose/docker-compose.prod.yml --profile app config parses cleanly
  • docker compose -f infra/compose/docker-compose.yml -f infra/compose/docker-compose.dev.yml config parses cleanly (untouched by this PR, checked for regressions)
  • infra/nginx/prod.conf validated with nginx -t against a self-signed cert from the new generator script
  • ruff check / ruff format --check on all touched backend files
  • mypy --strict (uv run mypy, project's own files = ["src"] scope) — clean
  • pytest tests/unit -q — 37 passed
  • pytest tests/integration -q -m integration (real Postgres via testcontainers) — 11 passed, including new task-queue-depth/claim-latency/solver_run-absence integration tests against a running Postgres

Closes #47

🤖 Generated with Claude Code

https://claude.ai/code/session_013YioTVKBPoE6thZqbnTtnM

## Summary - Extends `/metrics` (`backend/src/polaris/api/metrics.py`) with a request-latency histogram keyed by method+route-template, `task` queue-depth/claim-latency gauges, and best-effort `solver_run` gauges. `solver_run` isn't migrated on `main` yet (PH1.2/T-E are still open PRs), so it's reflected for at scrape time rather than imported as a model — the endpoint just renders nothing extra until that table exists, instead of erroring. - Extends `core/logging.py` with `service`/`environment`/`pid` on every log line, an explicit `user` alias for the existing `actor_sub`, and `bind_task_id`/`clear_task_id` helpers (same bind/unbind-around-the-work pattern `api/middleware.py` already uses for `request_id`) for the worker's task-claim loop to adopt. - Adds optional OpenTelemetry tracing (`install_otel`, called from `install_metrics`): lazily imported and opt-in via `POLARIS_OTEL_ENABLED` (plus the standard `OTEL_*` env vars), since `opentelemetry-*` isn't declared in `pyproject.toml` (sealed for this row) — disabled or package-missing both degrade to a clean no-op with a warning log line, never an error. `trace_id`/`span_id` get stitched into structlog output when a span is current. - Adds `infra/compose/docker-compose.prod.yml`: TLS termination at nginx (`infra/nginx/prod.conf`, HTTP redirects to HTTPS except the healthcheck) and TLS on the compose-bundled OpenLDAP (LDAPS only, plaintext `389` dropped from the prod stack). Certificates are self-signed for now, generated once per host by `infra/nginx/generate-self-signed-cert.sh` / `infra/ldap/generate-self-signed-cert.sh` into git-ignored `certs/` directories. - Resolves **Q21**: shipped OpenLDAP is the production directory by default — persistent storage (already the base compose file's default) + TLS (this overlay). An external corporate directory remains a config-only swap, not required. Updated the open-questions tables in `docs/07-architecture.md` and `docs/11-roadmap.md` per `CLAUDE.md`'s rule for resolved `Qn` assumptions. ## Scope notes / seam requests for a future ticket - `api/main.py` is sealed for this row, so the new latency middleware is installed from within `api/metrics.py:install_metrics` (already called by `main.py`) rather than touching that file. The existing `polaris_http_requests_total` counter (populated by `main.py`'s own middleware) still uses the raw request path; unifying it onto the same route-template label as the new histogram needs a small `main.py` change — flagged as a follow-up seam request rather than widening this ticket. - `bind_task_id`/`clear_task_id` are ready for the worker's task-claim loop (`backend/src/polaris/worker/**`, owned by PH0.3, outside this ticket's owned paths) to call around a handler invocation; not wired up there in this PR. - Real OpenTelemetry dependencies (`opentelemetry-api`, `opentelemetry-sdk`, `opentelemetry-instrumentation-fastapi`, `opentelemetry-exporter-otlp-proto-http`) aren't added to `pyproject.toml` (sealed for this row) — the hooks are fully optional/lazily-imported so this isn't a hard dependency; adding the packages for real use is a separate seam request. - Keycloak's own TLS/hostname hardening is intentionally out of scope (ticket's bar is "TLS on nginx at minimum"; the browser reaches Keycloak directly for the OIDC redirect, not through nginx). ## Test plan - [x] `docker compose -f infra/compose/docker-compose.yml config` parses cleanly - [x] `docker compose -f infra/compose/docker-compose.yml -f infra/compose/docker-compose.prod.yml --profile app config` parses cleanly - [x] `docker compose -f infra/compose/docker-compose.yml -f infra/compose/docker-compose.dev.yml config` parses cleanly (untouched by this PR, checked for regressions) - [x] `infra/nginx/prod.conf` validated with `nginx -t` against a self-signed cert from the new generator script - [x] `ruff check` / `ruff format --check` on all touched backend files - [x] `mypy --strict` (`uv run mypy`, project's own `files = ["src"]` scope) — clean - [x] `pytest tests/unit -q` — 37 passed - [x] `pytest tests/integration -q -m integration` (real Postgres via testcontainers) — 11 passed, including new task-queue-depth/claim-latency/solver_run-absence integration tests against a running Postgres Closes #47 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_013YioTVKBPoE6thZqbnTtnM
Add a request-duration histogram (route-templated, timed by a second
middleware installed in api/metrics.py rather than editing the sealed
api/main.py), task-queue depth/claim-latency gauges from the task
table, and best-effort solver_run gauges. solver_run isn't migrated on
main yet (PH1.2/T-E are still open PRs), so it's reflected for at
scrape time instead of imported as a model, and simply renders nothing
until that table exists.

Closes #47 (PH6.6, in progress).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013YioTVKBPoE6thZqbnTtnM
Add service/environment/pid to every structlog line (service from
POLARIS_SERVICE_NAME, a plain env var rather than a Settings field
since settings.py is sealed for this row), an explicit "user" alias
for the existing actor_sub, and bind_task_id/clear_task_id helpers
using the same structlog.contextvars bind/unbind pattern the
request-id middleware already uses. trace_id/span_id are added only
when OpenTelemetry tracing is active (api/metrics.py:install_otel) and
a span is current, via a cached, lazily-imported opentelemetry.trace
lookup that no-ops when the package isn't installed.

Refs #47 (PH6.6, in progress).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013YioTVKBPoE6thZqbnTtnM
Add docker-compose.prod.yml: TLS termination at nginx
(infra/nginx/prod.conf, HTTP redirects to HTTPS except the
healthcheck) and TLS on the compose-bundled OpenLDAP (LDAPS only
published, plaintext 389 dropped from the prod stack). Certificates
are self-signed for now, generated once per host by
infra/nginx/generate-self-signed-cert.sh and
infra/ldap/generate-self-signed-cert.sh into git-ignored certs/
directories; real certificate management is out of scope.

Persistent storage for OpenLDAP was already the base file's default
in every environment (ldap-data/ldap-config are named volumes); this
overlay adds TLS on top, resolving Q21 (docs updated separately).

Also sets POLARIS_SERVICE_NAME per service in the base compose file
(api/worker) and POLARIS_ENVIRONMENT=prod in the overlay, so
core/logging.py's new service/environment log fields are populated
out of the box rather than only in theory.

Uses the Compose Specification's `!override` merge tag on a few
list-type fields (nginx's volumes/ports, openldap's ports): compose
concatenates lists across -f files by default, which would otherwise
mount both nginx confs to the same path and publish LDAP's plaintext
389 alongside 636.

Refs #47 (PH6.6, in progress).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013YioTVKBPoE6thZqbnTtnM
Q21 (shipped OpenLDAP vs. an external corporate directory in prod) is
resolved: shipped OpenLDAP by default, with persistent storage
(already the base compose default) plus TLS
(docker-compose.prod.yml). An external directory remains a
config-only swap, not required. Updates the open-questions tables in
both 07-architecture.md and 11-roadmap.md per CLAUDE.md's rule for
resolved Qn assumptions.

Also documents the new /metrics series, the extended log fields, the
optional OTel hooks, and the prod compose overlay in
07-architecture.md's Observability and Docker Compose sections.

Refs #47 (PH6.6).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013YioTVKBPoE6thZqbnTtnM
bart merged commit 789b6b2a29 into main 2026-09-11 11:15:27 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
ready-for-agent
Self-contained ticket an agent can pick up

No labels
ready-for-agent
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
patrick/Polaris!62
No description provided.