Skip to content

refactor: cleaner abstractions across MQ / KV / ClickHouse (incl. CH connection pooling) #246

Closed

Description

@EricAndrechek

Area: architecture — refactor · from Eric's notes

Expected: the external dependencies (MQ, KV, ClickHouse) sit behind clean, swappable interfaces with consistent lifecycle/config/observability — and ClickHouse access uses proper connection pooling.

Actual: these concerns are wired fairly directly throughout the packages; ClickHouse access (structured-query + ingest paths) in particular would benefit from explicit pooling/limits. This is the umbrella over several smaller refactors already in flight.

Scope: define interfaces + lifecycle for MQ / KV / ClickHouse; add ClickHouse connection pooling (in-use/idle/wait visibility pairs with #94). Coordinate with the existing refactors rather than duplicating them.

Related: executeCHQuery → CH HTTP API #171; split internal/api #146; extract app.App #140.


From Eric's notes scratchpad; triaged 2026-06-04.

Activity

  1. added
    enhancementNew feature or request
    area/apiHTTP handlers, routing, middleware
    area/infraCI, build, deploy, Docker, release
    on Jun 4, 2026
  2. coderabbitai commented on Jun 4, 2026

    @coderabbitai
    🔗 Related PRs

    #164 - refactor(ingest)!: insert-only pipeline; mutations via /v1/query [merged]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  3. EricAndrechek commented on Oct 8, 2026

    @EricAndrechek
    MemberAuthor

    Closing: the goals of this umbrella have landed. MQ, coordination, cache and dedupe sit behind interfaces (mq.Broker, coord.Coordinator, cache.Cache, dedupe.Deduplicator), with each implementation chosen in one place in internal/app/wire.go (#618). ClickHouse access is pooled per connection tuple, with per-pool sizes and a clickhouse.max_total_conns ceiling (internal/chconn, #603), and the process lifecycle is owned by internal/app (#585). Pool in-use/idle/wait metrics are item 16 of #94, and the executeCHQuery and internal/api refactors continue on #171 and #146.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiHTTP handlers, routing, middlewarearea/cacheLocal / shared / tiered cachingarea/infraCI, build, deploy, Docker, releasearea/ingestIngest pipeline (Bento, batching, DLQ)area/observabilityMetrics, logs, traces, health, profilingarea/pipesNamed query pipesarea/queryStructured query AST, SQL builderenhancementNew feature or request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions