build: share the Go build cache across worktrees with -trimpath

CHANGELOG.md

@@ -51,6 +51,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ### Changed
 
+- **Worktrees share the Go build cache** (`Makefile`, `scripts/build.sh`, `internal/mq/natstest/natstest.go`, `tests/integration/roles_test.go`, `docs/src/content/docs/development.md`): closing [#763](https://github.com/Wave-RF/WaveHouse/issues/763). Go keys this module's own packages in the build cache by their absolute directory, so every worktree built and cached its own copy of each variant (normal, test, `-race`, `-cover`), and Go evicts only entries unused for five days. The Makefile now exports `GOFLAGS=-trimpath` (a caller's `GOFLAGS` is kept), which makes the entries path-independent, and CI inherits it through the same targets. The two tests that located repository files with `runtime.Caller` (whose path is module-relative under `-trimpath`) now find the repository root from the working directory. `make build`'s debug binary is exempt (`scripts/build.sh` passes `-trimpath=false`) and keeps absolute source paths for debuggers; trimmed binaries need a source-path substitution, and `GOFLAGS=-trimpath=false make …` opts out. Release builds are separate ([#537](https://github.com/Wave-RF/WaveHouse/issues/537)).
 - **One escaping for composite keys, `-` kept; dead-letter counts per table** (`internal/keyenc` (new, + tests), `internal/mq/{mq,subject,embedded,deadletter}.go` (+ tests), `internal/query/ident.go` (+ tests), `AGENTS.md`, `docs/src/content/docs/{architecture,development}.md`): part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). NATS subject tokens and the cache's namespace tokens each carried a copy of the same encoder; both now call `internal/keyenc`, which the dedupe keys use too, and subjects are built with its `Join` (each field escaped, with a separator the escaping never writes between them). The escaping now keeps `-` as well as ASCII letters, digits and `_` — exactly the tenant-id grammar — so a table or scope such as `my-table` is `my-table` in a subject rather than `my%2Dtable`; every other byte is escaped as before (pinned by golden tests, and against v0.1.0's encoder for every other byte value). Upgrading from v0.1.0 notices nothing further, since its queue is deleted at boot (below). A queue an unreleased build since [#612](https://github.com/Wave-RF/WaveHouse/pull/612) wrote still reads, because decoding is `url.PathUnescape` as it was: `%2D` decodes to `-`, and a dead-letter count merges both forms. Only a `/v1/stream` client resuming across such an upgrade (`Last-Event-ID` or `since`) on a table whose name holds `-` misses that table's events queued before it, since the replay filters on the table's exact subject. `GET /v1/ops/dlq/stats` now counts every scope of a table under the table itself, and `?table=` keeps all of its scopes; a scoped message used to count under `table.scope`, a name a dotted table could share. Scope is always empty today, so the response is unchanged.
 
 - **Each tenant has a message queue of its own** (`internal/mq/{mq,subject,embedded}.go` (+ tests), `internal/ingest/{sweeper,worker}.go` (+ tests), `internal/api/{dlq,ingest}.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `internal/stream/{subscriber,hub}.go`, `internal/settings/{settings,store,registry}.go` (+ tests), `internal/testutil/{mocks,testutil}.go`, `clients/ts/src/{dlq,types}.ts` (+ tests), `docs/src/content/docs/{deployment,api,architecture,ingest-pipeline,durability,why-wavehouse}.md`, `docs/src/content/docs/sdk/{admin,reference,streaming}.md`, `docs/src/content/docs/{settings-directory,configuration}.mdx`, `AGENTS.md`): story 5b of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). The embedded NATS server keeps each tenant's events on a pair of JetStream streams of its own — `INGEST_<tenant>` (`ingest.<tenant>.>`, `DiscardNew`) at the tenant's own `mq.max_bytes_gb`, and `DLQ_<tenant>` (`dlq.<tenant>.>`, `DiscardOld`) at a tenth of it — opened when the tenant is first served and kept, at the budget it last had, when its folder is rejected or removed; subjects are unchanged, and nothing outside `internal/mq` names a stream. A tenant at its budget gets `503` while the others keep publishing, and the ingest worker's and the hub bridge's durables are held on every tenant's stream, each with its own ack floor and `MaxAckPending`, so one tenant's backlog holds back neither another's delivery nor its purge; the worker's prefetch and the hub bridge's fetch-ahead are each shared across the tenants' streams. A removed or rejected tenant's stream is still consumed, so its queued rows reach the worker and are parked on its own dead-letter queue. The sweeper purges each tenant's stream at that tenant's own `stream.gap_window_minutes` — a rejected tenant's as its folder last had it (`settings.Registry.Known` now yields each tenant's last adopted store), and all of its history if the folder has been rejected since boot, so its clients resume once the folder is fixed — keeping no acknowledged history for a removed tenant, and every served tenant's `mq.max_bytes_gb` is applied after each reload rather than tenant `0`'s alone (the boot warning about a nested directory with no tenant `0` is gone with it, and so is the tracking of tenant `0`'s last adopted store: a flat directory's ops gate, its one reader left, reads tenant `0` through the registry). One tenant's failed purge holds up no other tenant's, and the sweep logs it at `ERROR` unless every failure in it is a buffer consumer not created yet. A reload that shrinks a budget no longer deletes dead letters: a dead-letter stream holding more than a tenth of the new budget keeps what it holds, and that is logged — the interim guard of [#532](https://github.com/Wave-RF/WaveHouse/issues/532). JetStream's own check of the streams' caps against the disk, which made them fit 75% of the free disk at boot together, is lifted: a budget is a cap and never a reservation, what the budgets add up to against the disk is [#138](https://github.com/Wave-RF/WaveHouse/issues/138)'s, and a flat directory whose budget exceeds three quarters of the free disk now boots where it used to be refused. A queue that cannot be opened refuses a flat boot like any other store and, over a nested directory, costs its tenant alone — its ingest answers `503`, each reload trying again, and a publish at most once every five seconds, so its clients retrying never hold up another tenant's queue. `GET /v1/ops/dlq/stats` reads one tenant's dead-letter queue — the one `?tenant=` names, parsed strictly like the other admin reads, and tenant `0`'s without it, no longer the sum across tenants — answering for a rejected or removed tenant too and `404` for a tenant with no queue; the SDK's `wh.dlq.list()` and `.table()` take a `tenant` option. The streams an earlier build kept for every tenant together (`WAVEHOUSE`, `WAVEHOUSE_DLQ`) overlap every tenant's subjects and are deleted at boot, what they held with them, and a subject with no tenant token no longer reads as tenant `0`'s.

Makefile

@@ -225,6 +225,17 @@ GO_TOOLCHAIN := $(shell scripts/ci/go-toolchain.sh)
 $(if $(GO_TOOLCHAIN),,$(error cannot derive the Go toolchain from go.mod, see above))
 export GOTOOLCHAIN := $(GO_TOOLCHAIN)
 
+# -trimpath makes compiled packages independent of the checkout's directory,
+# so every worktree shares one Go build cache entry per package and variant
+# instead of building its own. A caller's GOFLAGS is kept, and one that already
+# names -trimpath (`-trimpath=false` to opt out) is left alone, which also
+# stops sub-makes from prepending it again.
+GOFLAGS ?=
+ifeq ($(findstring -trimpath,$(GOFLAGS)),)
+override GOFLAGS := -trimpath $(GOFLAGS)
+endif
+export GOFLAGS
+
 # ==============================================================================
 # Targets
 # ==============================================================================

docs/src/content/docs/development.md

@@ -338,6 +338,8 @@ Each test target writes `covdata` to `tmp/coverage/<suite>/data/`, renders a tex
 
 **Note on timing**: gotestsum's `DONE ... in X.XXXs` reports pure test execution time. The total wall time includes Go compiling all packages — the first run compiles everything (~15s), subsequent runs use the build cache (~1s).
 
+**Build cache across worktrees**: `make` exports `GOFLAGS=-trimpath` (your own `GOFLAGS` are kept), so compiled packages don't embed the checkout's path and every worktree of the same source shares one build cache entry per package and variant, instead of building its own. Opt out with `GOFLAGS=-trimpath=false make …`. `make build` is the exception: its debug binary keeps absolute source paths, so a debugger finds the files. Release and coverage binaries and test binaries built through `make` are trimmed; their frames read `github.com/Wave-RF/WaveHouse/internal/…` (standard-library frames are plain import paths such as `runtime/proc.go`), so map the module path to your checkout, for example, at the `(dlv)` prompt, `config substitute-path github.com/Wave-RF/WaveHouse /absolute/path/to/your/checkout` (or a `substitute-path` entry with `from` and `to` in `~/.config/dlv/config.yml`, or your editor's `substitutePath`). Code that needs a file from the repository finds it from the working directory, never from `runtime.Caller`, whose paths are module-relative under `-trimpath`.
+
 ### Test Structure
 
 | Category | Location | Docker? | Command |

internal/mq/natstest/natstest.go

@@ -20,7 +20,6 @@ import (
 	"os"
 	"path/filepath"
 	"regexp"
-	"runtime"
 	"sync"
 	"testing"
 	"time"
@@ -43,10 +42,24 @@ const (
 // Secret reference.
 func Password(user string) string { return "pw-" + user }
 
-// repoFile is path under the repository root.
+// repoFile is path under the repository root, found by walking up from the
+// working directory to go.mod. runtime.Caller would not do: -trimpath makes
+// it return module-relative paths.
 func repoFile(path string) string {
-	_, file, _, _ := runtime.Caller(0)
-	return filepath.Join(filepath.Dir(file), "..", "..", "..", path)
+	dir, err := os.Getwd()
+	if err != nil {
+		panic(err)
+	}
+	for {
+		if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {
+			return filepath.Join(dir, path)
+		}
+		parent := filepath.Dir(dir)
+		if parent == dir {
+			panic("natstest: no go.mod above the working directory")
+		}
+		dir = parent
+	}
 }
 
 // ShippedValues and ShippedManifests are the files an operator deploys.

scripts/build.sh

@@ -6,11 +6,14 @@
 #   <variant> debug | release | cover
 #
 # Variant differences:
-#   debug    bin/<name>           DWARF + symbol tables kept (LDFLAGS empty);
-#                                 what `make build` produces.
+#   debug    bin/<name>           DWARF + symbol tables kept (LDFLAGS empty) and
+#                                 absolute source paths (-trimpath=false, which
+#                                 overrides the Makefile's GOFLAGS), so a
+#                                 debugger finds the files; what `make build`
+#                                 produces.
 #   release  bin/<name>-release   `-s -w` applied — DWARF + symbol tables
 #                                 stripped; what `make build-release` produces.
-#   cover    bin/<name>-cov       Same as debug + `-cover` instrumentation;
+#   cover    bin/<name>-cov       Same as debug, but trimmed, + `-cover` instrumentation;
 #                                 used by E2E to capture coverage from a real
 #                                 running binary.
 #
@@ -31,7 +34,7 @@ case "$variant" in
 debug)
 	label=$name
 	ldflags="${LDFLAGS:-} ${VERSION_LDFLAGS:-}"
-	build_flags=()
+	build_flags=(-trimpath=false)
 	output="bin/$name"
 	;;
 release)

tests/integration/roles_test.go

@@ -15,7 +15,6 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
-	"runtime"
 	"strconv"
 	"strings"
 	"sync"
@@ -58,9 +57,9 @@ func buildWavehouseBinary() {
 	if os.Getenv("GOCOVERDIR") != "" {
 		args = append(args, "-cover", "-coverpkg=./...")
 	}
-	_, file, _, _ := runtime.Caller(0)
 	cmd := exec.Command("go", append(args, "./cmd/wavehouse")...) //nolint:gosec // G204: fixed arguments
-	cmd.Dir = filepath.Join(filepath.Dir(file), "..", "..")
+	// go test runs in the package directory, two levels below the module root.
+	cmd.Dir = filepath.Join("..", "..")
 	if out, err := cmd.CombinedOutput(); err != nil {
 		errBinary = fmt.Errorf("go build: %w\n%s", err, out)
 	}