feat(observability): o11y dev-stack cleanup + wire into make #121

Problem

PR #116 added deployments/signoz/ — a self-contained Docker Compose stack for running SigNoz locally (ClickHouse + query service + OTel collector + one-shot migrator). The stack works but has a handful of smells from initial authoring, and crucially is not wired into the dev workflow — there's no make target that brings it up, and nothing makes the dev binary point at it. As a result, "dev with observability so I can watch traces in SigNoz" is a multi-step manual process today.

The goal of this issue: harden the stack and add the integration so make dev-obs is a one-shot "dev mode + SigNoz running + WaveHouse already pointed at it, dashboards visible" experience that mirrors how make dev brings up ClickHouse + WaveHouse with hot-reload today.

Current state — review notes

Files: deployments/signoz/docker-compose.yaml, deployments/signoz/otel-collector-config.yaml, deployments/signoz/.env.example.

Real problems (would block a clean reproducible setup):

1. :latest image tags everywhere — signoz/query-service:latest, signoz/signoz-otel-collector:latest, clickhouse/clickhouse-server:24.1.2-alpine (only ClickHouse is pinned). A backwards-incompatible upstream change can break the stack overnight without a code change. Pin to specific known-good versions and document the bump procedure.
2. container_name: set on every service — prevents running multiple instances and breaks the docker-compose project-prefix convention. If anyone runs docker compose -p test up they collide. Drop the container_name: lines and let compose handle uniqueness via the project prefix.
3. No healthchecks on signoz (UI/query) or otel-collector — only ClickHouse has one. That makes docker compose up -d --wait useless for the question we actually care about: "is the collector ready to receive OTLP, and is the UI ready to query?" Without these, the new make signoz-up target can't safely chain on success.
4. Port 4318 (OTLP HTTP) isn't exposed — the collector listens on it (otel-collector-config.yaml lines 13-14: http: endpoint: 0.0.0.0:4318) but the compose file only publishes 4317 (gRPC). Either drop the HTTP receiver from the collector config or add - "4318:4318" to the otel-collector service ports. Probably expose, since it's free and useful for curl-based debugging.
5. LOW_CARDINAL_EXCEPTION_GROUPING env var referenced in collector config (line 72) but never set anywhere — evaluates to empty string at runtime. Either set it explicitly (the SigNoz default is false) or drop the ${env:...} reference and inline the value.

Style / hygiene (worth fixing while in the area):

6. No header comment in either YAML explaining what this stack is, what it's for, what's NOT included (auth, persistence guarantees, prod posture). Future operators will read these cold.
7. No restart policy on long-running services. Add restart: unless-stopped on clickhouse, signoz, otel-collector. The migrator stays as-is (one-shot).
8. No memory/CPU limits — SigNoz can be hungry. Not a blocker for dev; a comment ("for prod, set deploy.resources.limits") is enough here.
9. File named docker-compose.yaml — modern Compose v2 convention is compose.yaml. Cosmetic; rename or leave.
10. Explicit network name wavehouse-obs-net — fine but unnecessary. Project-default works and avoids the noun-pile. Drop the networks: block entirely.
11. volume-fixer init container — works (sets ownership on named volumes for ClickHouse UID 101 + SigNoz UID 1000) but it's an unusual pattern and there's no comment explaining why it exists. Add a one-line comment so future readers don't think it's leftover scaffolding.

Proposed Solution

Stack hardening

Apply 1-11 above. Net result is a smaller, better-documented compose file with pinned versions and proper healthchecks. Roughly a single PR's worth of changes.

Dev integration — new Make targets

Mirror the existing deps-* family (which manages the dev ClickHouse), then add a top-level dev-obs target that composes everything:

SIGNOZ_COMPOSE_FILE := deployments/signoz/docker-compose.yaml
SIGNOZ_COMPOSE      := docker compose -f $(SIGNOZ_COMPOSE_FILE)

.PHONY: signoz-up
signoz-up: ## Start SigNoz stack (idempotent; blocks until collector + UI healthy)
	@echo "$(CYAN)==> Starting SigNoz stack...$(RESET)"
	@$(SIGNOZ_COMPOSE) up -d --wait

.PHONY: signoz-down
signoz-down: ## Stop SigNoz stack (preserves volumes — UI history kept)
	@echo "$(YELLOW)==> Stopping SigNoz...$(RESET)"
	@$(SIGNOZ_COMPOSE) down

.PHONY: signoz-logs
signoz-logs: ## Tail SigNoz collector + UI logs
	@$(SIGNOZ_COMPOSE) logs -f signoz otel-collector

.PHONY: signoz-wipe
signoz-wipe: ## Stop SigNoz AND destroy its volumes (DESTRUCTIVE)
	@echo "$(RED)==> Wiping SigNoz (containers + volumes)...$(RESET)"
	@$(SIGNOZ_COMPOSE) down -v --remove-orphans

.PHONY: dev-obs
dev-obs: deps-up signoz-up $(AIR) ## Hot-reload dev server with SigNoz observability
	@echo "$(CYAN)==> Starting WaveHouse with SigNoz observability$(RESET)"
	@echo "    WaveHouse:  $(GREEN)http://localhost:8080$(RESET)"
	@echo "    SigNoz UI:  $(GREEN)http://localhost:3301$(RESET)  (first-run: create local admin account)"
	@WH_OTEL_ENABLED=true \
	 WH_OTEL_ADDR=127.0.0.1:4317 \
	 WH_OTEL_METRICS_PROMETHEUS_ENABLED=true \
	 $(AIR) -c .air.toml

clean-all extended to also tear down SigNoz volumes:

clean-all: clean clean-test clean-tools
	@rm -rf data/
	@$(DEV_COMPOSE) down -v --remove-orphans 2>/dev/null || true
	@$(SIGNOZ_COMPOSE) down -v --remove-orphans 2>/dev/null || true   # NEW
	@docker compose -f tests/e2e/compose.yaml down -v --remove-orphans 2>/dev/null || true

The integration is loose-coupled: make dev is unchanged (still works with zero SigNoz dependency); dev-obs adds the SigNoz layer + env flags on top. Ops who want SigNoz running but don't want to use dev-obs can just make signoz-up standalone.

Documentation

• docs/development.md — short section on dev-obs flow with the SigNoz first-run account-creation note (SigNoz pops a "create local admin" page on first visit; not a misconfigured deployment).
• docs/deployment.md — already has a SigNoz section from feat(observability): add otel pipeline and system tracing #116; minor update to reference the new Make targets.
• README quick-start — one bullet under the dev-loop section linking to the new flow.

Test plan

• make signoz-up brings up the stack, healthchecks all green within ~60s.
• make signoz-down stops cleanly; make signoz-up again restarts with prior data intact (UI history preserved).
• make signoz-wipe + make signoz-up = clean slate (admin account creation again).
• make dev-obs brings up everything in the right order, WaveHouse boots with otel pipeline established log, traces from a sample POST /v1/ingest/... show up in the SigNoz UI within ~10s.
• make clean-all tears down both deps and signoz stacks cleanly, no leaked volumes (docker volume ls | grep -i wavehouse empty).
• Run twice in a row: second make dev-obs starts as fast as a no-op since both stacks are already up.

Alternatives Considered

• OBS=1 make dev flag instead of dev-obs target — env-flag pattern doesn't exist anywhere else in this Makefile, hurts discoverability via make help.
• Pin to :stable / SigNoz-published rolling tag — depends on whether upstream maintains one; if not, pin to specific versions. Same outcome either way: predictable from the operator's side.
• Run SigNoz in the same compose project as deps-up — tighter coupling, but then everyone running make dev pays the SigNoz startup cost. Better to keep separate.
• Skip the integration entirely; document the manual flow — what we have today; the friction is real enough that "dev with observability" effectively never happens.

Out of Scope

• TLS / direct-to-cloud OTLP support (tracked in feat(observability): direct-to-cloud OTLP — TLS + auth headers (no sidecar) #97).
• Production-grade SigNoz deployment (resource limits, persistent volume policy, auth) — this issue is dev-only.
• Replacing SigNoz with another backend (Jaeger, Tempo, etc.) — anyone wanting that can mirror this pattern with their own compose.

Additional Context

Surfaced during PR #116's late review while reviewing the SigNoz files added by an earlier contributor. The SigNoz stack itself is functional and correctly wired for OTLP receive — this issue is about taking it from "files exist, runs if you know how" to "first-class dev workflow."