Derived index: a worktrees table mirroring every lane's #97 inventory under inventory:<Workstation>, nudged by every inventory write, and lanes --worktrees to read it with or without --index

docs/README-lanes.md

@@ -3150,6 +3150,10 @@ lanes-edit.sh lane-tree-now <worktree path>
 # <branch> <head> <upstream> <dirty> <unpushed>   — what git says about one tree NOW
 ```
 
+Every lane's records on this workstation, one row per tree, are `lanes
+--worktrees`; the derived index keeps them as its `worktrees` table, which no
+act reads (*The derived index*, below).
+
 The full `head` and the `upstream` are why this is not the prose section one
 more time: `%h` is an abbreviation that lengthens as a repository grows, and
 `0 unpushed` cannot be told from *this branch tracks nothing at all* without the
@@ -3474,15 +3478,93 @@ One versioned schema (`schema_version`); every row carries its `provenance`
 
 **Created empty and written by nothing here**: `swap_states` and
 `job_summaries` — the managed ledger's half, `ledger:<Workstation>`, which the
-001 feature's T058 writes (act 4) — and `worktrees`, **reserved for
-opensoft/openRepoTools#161** (each lane's #97 inventory under
-`inventory:<Workstation>`). Not indexed at all: transcript and handoff bodies,
+001 feature's T058 writes (act 4). `worktrees` is the inventory's, below — the
+same schema 1, whose table was created with #161's columns before a row was
+written into it. Not indexed at all: transcript and handoff bodies,
 credentials, the ledger's claims, fences, releases and exit witnesses.
 
+### The `worktrees` table — every lane's #97 inventory (opensoft/openRepoTools#161)
+
+**What it is for: the next account's first query.** A session that resumes a
+lane — or that sits down at Raven after a day on Eagle — wants every open
+worktree before it decides anything: whose it is, which branch, whether the
+lane stopped `RUNNING` or `SWAPPED`, what was dirty or unpushed, when it was
+last seen and whether a writer is still live. Each lane's handoff already
+records that beside the lane (*The worktree inventory*, above); this table is
+the same records in one place, under the source `inventory:<Workstation>` — one
+per workstation, because the sidecars live on that workstation's disk and
+nowhere else — so a store two workstations sync into answers for both.
+
+```sh
+lanes --worktrees [<lane> | --all]            # this workstation's sidecars
+lanes --index --worktrees [<lane> | --all]    # the table: every workstation it holds
+lanes-index sync      --source inventory      # what every inventory write nudges
+lanes-index reconcile --source inventory [--dry-run]
+```
+
+| column | from |
+|---|---|
+| `workstation`, `lane` | the workstation's name (`lanes-edit.sh workstation`) and the lane's canonical one |
+| `owner` | `legacy`, `managed <owner>` or `unknown` — the read `managed-projection` makes, never a parser of the indexer's |
+| `path`, `branch`, `dirty_count`, `unpushed_count`, `last_seen_utc` | the tree's sidecar: the last observation, `detached <sha>` for a detached head, an `unknown` count as NULL. A sidecar that could not be read is `unreadable sidecar`, one of another schema `unknown schema <n>`, each with the record's id where its path would be |
+| `base`, `last_commit` | two git reads of the RECORDED head, in the tree or, once it is gone, its recorded checkout: its merge-base with `origin/main` (else `main`, else `origin/HEAD`), and `<sha> <subject>` |
+| `lifecycle`, `generation`, `operation` | the lane's snapshot (`lane-state`): `RUNNING`, `SWAPPING`, `SWAPPED` or `CLOSED` for a **legacy** lane; `INDETERMINATE` for a managed or unknown one, whose lifecycle this tooling does not pronounce on, and for a snapshot that is missing, unreadable or of another schema |
+| `writer_live` | the listing's own LIVE — a session record on this workstation naming one of the row's ids — and NULL where those records could not be read or the lane is bound on another host, from where liveness is unknown and never dead (Amendment 18(b)) |
+| `pr_number`, `pr_state` | `gh` at sync, best effort, the newest pull request whose head is the branch; NULL wherever it was not asked (`LANES_NO_GITHUB=1`) or did not answer. A MERGED or CLOSED answer is not asked again while the branch and its last commit stand |
+| `provenance`, `indexed_utc` | the generation the TREE'S OWN record was filed under — behind `generation` exactly when the observation predates the lane's current transition — and the sync |
+
+**Read through the helper, never around it.** The indexer reads `lanes-edit.sh
+worktrees --all`, which lists each lane's sidecars through `lane-trees`' own
+reader, its snapshot through `lane-state`'s and its owner through
+`managed-projection`'s, and walks no directory for a tree no sidecar names —
+that is `lane-reconcile`'s act. Nothing is recomputed: `dirty` and `unpushed`
+are what the handoff saw, and `LAST SEEN` says when.
+
+**Rows follow the inventory.** Keyed by (source, `<lane>:<path>`) and versioned
+by a digest of the row and its provenance, so a replay writes nothing; a tree
+the inventory no longer names — removed at landing, or by a sweep's disposition
+— leaves in the same sync. The provenance moves only forward **per lane**: a
+`sync` that reads a lane's lifecycle generation below the one its rows were
+written at (a snapshot moved aside and begun again) writes nothing for that lane
+and says so, and `reconcile --source inventory` is the act that takes the lane
+as it now stands. A lane whose `trees` directory is there and cannot be listed
+keeps its rows — it is not a lane with no worktrees. The source's own row
+carries a fingerprint of the whole table as its compare-and-swap token and a
+sync count that only goes up. A container with no `LANES_WORKSTATION` syncs
+nothing (exit 2): a source named for a container's own id is one no restart
+finds again.
+
+**The nudge.** Every inventory write — `set-lane-state`, `set-lane-tree`, the
+lifecycle follow after a `STARTED`, `RESUMED`, `ENDED` or `RETIRED`, and a
+rename's move of the control root — starts the same detached `lanes-index sync`
+a landed push does, as `sync --source inventory`, skipped where `lanes-index` is
+not on `PATH` or `LANES_INDEX=off`. One sync at a time per workstation: a nudge
+that finds the lock held leaves a mark NAMING ITS SOURCE, and the holder syncs
+every source a mark names before it lets go.
+
+**The read.** `lanes --worktrees` reads this workstation's sidecars; with
+`--index` it reads the table — every `inventory:*` source the store holds, so
+Raven sees Eagle's leftovers — and stderr says `read: index (<store>) at
+inventory:<ws> synced <UTC>`, or `read: sources (index <why>)` and the sidecars
+answer. Where the table is current, this workstation's rows are the same bytes
+either way — and a store only this workstation syncs into (SQLite, or a
+Postgres nobody else writes) prints exactly the source read, while a shared one
+adds the other workstations' rows among them. Every row is a last observation: the current truth of a lane is `lanes-edit.sh lane-reconcile
+<lane>`, which reads the disk, and the footer says so.
+
+**What never reads it.** No act: not `lane-reconcile` — there is no
+`lane-reconcile --index`, by Amendment 14(b) — not `lane-start`, `lane-end`,
+`lane-handoff` or the worktree sweep (opensoft/openRepoTools#162). Each of them
+reads the sidecars and the disk, exactly as before this table existed; an
+inventory index that is missing, stale, unreachable, wiped or wrong changes no
+answer of theirs.
+
 ### The tests' promise
 
 `tests/test_lanes_index.py`, and a section of `tests/test_lane_helpers.sh` for
-the acts only that suite drives, hold the index to Amendment 14's eight:
+the acts only that suite drives, hold the index — the inventory's table
+included, whose writers and `lane-reconcile` are among the acts — to Amendment
+14's eight:
 **offline no-read** (every act byte-identical against an unreachable store),
 **poisoned index** (wrong rows change no act's answer), **wiped-index
 reconcile** (the rebuild equals a synced index row for row), **replay**,

lanes

@@ -10,6 +10,11 @@
 #   lanes --here     only this workstation's lanes
 #   lanes --fetch    any of the above, after refreshing from origin
 #   lanes --index    any of the above, out of the DERIVED INDEX (Amendment 14)
+#   lanes --worktrees [<lane>|--all]
+#                    every open worktree the #97 inventory records - owner,
+#                    branch, lifecycle, dirty/unpushed, last seen, writer
+#                    (opensoft/openRepoTools#161); with --index, out of the
+#                    index's `worktrees` table, every workstation it holds
 #
 # THE DEFAULT IS EVERY LANE, AND THE NARROWING IS BY CHECKOUT (A11 Addendum 4
 # ruling 6, ratified "a11 addendum 4 yes" with ruling 6 answered NO, over the
@@ -105,6 +110,7 @@ usage() {
 lanes [--fetch] [--index] [--all] [--here] [--closed] [--repo <owner/repo>]
       [--dir <path>] [--prefix <repo>] [--ws <workstation>]
       [--rename <old> <new>]
+lanes --worktrees [<lane> | --all] [--index] [--fetch]
 
 Inside a lane checkout: THAT REPOSITORY's lanes, ending with the next free
 position and the exact `lane-start <repo> <n>` that takes it. Outside one: every
@@ -147,6 +153,19 @@ is running is never started a second time.
                    home repository and the checkout it sits in are not the same
                    fact.
   --ws <name>      another workstation's lanes.
+  --worktrees [<lane> | --all]
+                   every open worktree of one lane, or of every lane: the #97
+                   inventory each lane's handoff recorded beside it — owner,
+                   branch, lifecycle, dirty and unpushed counts, when it was
+                   last seen and whether the lane's writer is live
+                   (opensoft/openRepoTools#161). Every field is that LAST
+                   OBSERVATION and nothing is recomputed: the current truth of a
+                   lane is `lanes-edit.sh lane-reconcile <lane>`, which reads the
+                   disk. Without --index it reads this workstation's sidecars;
+                   WITH it, the derived index's `worktrees` table — every
+                   workstation syncing into that store, so Raven sees Eagle's
+                   leftovers — and stderr says which, as for the listing. It
+                   takes none of the listing's narrowing flags.
   --rename <old> <new>
                    REFUSED, and it names the word that does it:
                    `lane-rename <old> <new>`. This command writes nothing
@@ -191,11 +210,22 @@ all_given=0         # 1 once `--all` has been seen, wherever in the line it was
 # NEXT FREE POSITION footer.
 here_repo=""
 args=()
+# `--worktrees` (opensoft/openRepoTools#161) TAKES ONE POSITIONAL, THE LANE, and
+# the listing takes none — so whether this line asks for worktrees is known
+# before the loop below meets a positional, wherever the flag stands in it.
+wt_mode=0
+wt_lane=""
+# `${1+"$@"}` AND NOT A BARE `"$@"`: bash 3.2 under `set -u` calls the empty
+# `"$@"` of a bare `lanes` an unbound variable and exits (tests/run.sh says why).
+for wt_a in ${1+"$@"}; do
+	case "$wt_a" in --worktrees) wt_mode=1 ;; esac
+done
 while [ $# -gt 0 ]; do
 	case "$1" in
 	-h | --help) usage; exit 0 ;;
 	--fetch) do_fetch=1; shift ;;
 	--index) idx_args=(--index); shift ;;
+	--worktrees) shift ;;
 	# `--all` CLEARS THE LABEL AS WELL AS THE FILTERS. The helper's `--all` drops
 	# `--repo`, `--dir` and `--prefix`, so the ROWS were right — but `here_repo`
 	# is this wrapper's own, and `lanes --all --prefix repoA11` (either order)
@@ -243,10 +273,102 @@ while [ $# -gt 0 ]; do
     (lane-collision-protocol Amendment 16)." 64 ;;
 	--) shift ;;
 	-*) die "unknown option '$1' — see $prog --help" 64 ;;
-	*) die "lanes takes no positional arguments; did you mean \`lane $1\`? — see $prog --help" 64 ;;
+	*)
+		if [ "$wt_mode" = 1 ] && [ -z "$wt_lane" ]; then
+			wt_lane="$1"; shift
+		elif [ "$wt_mode" = 1 ]; then
+			die "--worktrees takes ONE lane, or --all — see $prog --help" 64
+		else
+			die "lanes takes no positional arguments; did you mean \`lane $1\`? — see $prog --help" 64
+		fi ;;
 	esac
 done
 
+# ------------------------------------------------ `--worktrees` (#161)
+#
+# EVERY OPEN WORKTREE, rendered from `lanes-edit.sh worktrees`' rows — the #97
+# inventory as each lane's handoff last recorded it, read from this
+# workstation's sidecars, or with `--index` from the derived index's
+# `worktrees` table, which holds every workstation that syncs into it. Where
+# the index is current, this workstation's rows are the same bytes either way,
+# and a store no other workstation syncs into prints exactly the source read;
+# stderr says which was read. It recomputes nothing and it decides nothing: the act
+# that reads the disk is `lanes-edit.sh lane-reconcile <lane>`, and the footer
+# says so.
+US="$(printf '\037')"
+if [ "$wt_mode" = 1 ]; then
+	for wt_a in ${args[@]+"${args[@]}"}; do
+		[ "$wt_a" = --all ] && continue
+		die "--worktrees takes ONE lane or --all, and none of the listing's narrowing flags ('$wt_a' was given) — see $prog --help" 64
+	done
+	[ "$closed" = 0 ] || die "--worktrees takes ONE lane or --all, not --closed: a closed lane's worktrees are listed with every other's — see $prog --help" 64
+	[ "$all_given" = 0 ] || [ -z "$wt_lane" ] || die "--worktrees takes ONE lane OR --all, not both — see $prog --help" 64
+	wt_args=()
+	if [ -n "$wt_lane" ]; then wt_args+=("$wt_lane"); else wt_args+=(--all); fi
+	[ "$do_fetch" = 1 ] && wt_args+=(--fetch)
+	rc=0
+	wt_err=""
+	wt_err_file="$(mktemp "${TMPDIR:-/tmp}/lanes-worktrees.XXXXXX" 2>/dev/null || printf '')"
+	if [ -n "$wt_err_file" ]; then
+		out="$("$LANES_EDIT" worktrees ${idx_args[@]+"${idx_args[@]}"} "${wt_args[@]}" 2>"$wt_err_file")" || rc=$?
+		wt_err="$(cat -- "$wt_err_file" 2>/dev/null || printf '')"
+		# `-e` AND NO `--` AFTER THE PATTERN: BSD's `grep` stops at its first
+		# operand, so a `--` there is a FILE it cannot open on macOS.
+		command grep -v -e 'not fetching; reading origin/' "$wt_err_file" >&2 || :
+		rm -f -- "$wt_err_file"
+	else
+		out="$("$LANES_EDIT" worktrees ${idx_args[@]+"${idx_args[@]}"} "${wt_args[@]}")" || rc=$?
+	fi
+	case "$rc" in
+	0) : ;;
+	8)
+		say "no worktree is inventoried${wt_lane:+ for lane $wt_lane}."
+		say "  each handoff records its lane's trees (lanes-edit.sh set-lane-tree); the disk itself is read by"
+		say "  lanes-edit.sh lane-reconcile <lane>"
+		exit 8 ;;
+	64) die "bad arguments for the worktrees read — see $prog --help" 64 ;;
+	2)
+		case "$wt_err" in
+		*"unknown subcommand 'worktrees'"*)
+			die "the \`lanes-edit.sh\` on this workstation has no \`worktrees\` read, so it
+    predates opensoft/openRepoTools#161. Nothing is wrong with the inventory —
+    take the install:  openRepoTools --install" 2 ;;
+		esac
+		exit 2 ;;
+	*) die "$LANES_EDIT worktrees failed (exit $rc). That is NOT 'no worktree is inventoried', and
+    this command will not print an empty listing on an answer it did not get
+    (Amendment 7(d)). Run it by hand to see what it says." 1 ;;
+	esac
+	printf '%-11s %-26s %-16s %-13s %-7s %-34s %5s %8s  %-20s  %s\n' \
+		WORKSTATION LANE OWNER LIFECYCLE WRITER BRANCH DIRTY UNPUSHED "LAST SEEN" PATH
+	n=0; nl=0; nw=0; wt_unread=0; wt_lanes=" "; wt_wss=" "
+	while IFS="$US" read -r w_kind w_ws w_lane w_owner w_path w_branch w_base w_life w_gen w_op w_live w_seen w_last w_dirty w_unp w_prov; do
+		case "${w_kind:-}" in
+		TREE)
+			case "${w_live:-}" in 1) w_wr=live ;; 0) w_wr=none ;; *) w_wr=unknown ;; esac
+			printf '%-11s %-26s %-16s %-13s %-7s %-34s %5s %8s  %-20s  %s\n' \
+				"$w_ws" "$w_lane" "${w_owner:-unknown}" "${w_life:-INDETERMINATE}" "$w_wr" \
+				"${w_branch:-unknown}" "${w_dirty:-?}" "${w_unp:-?}" "${w_seen:-never}" "$w_path"
+			n=$((n + 1))
+			case "$wt_lanes" in *" $w_ws/$w_lane "*) : ;; *) wt_lanes="$wt_lanes$w_ws/$w_lane "; nl=$((nl + 1)) ;; esac
+			case "$wt_wss" in *" $w_ws "*) : ;; *) wt_wss="$wt_wss$w_ws "; nw=$((nw + 1)) ;; esac ;;
+		UNREAD)
+			# `UNREAD <ws> <lane> <why>`: the third field is the reason.
+			printf '%-11s %-26s inventory NOT READ: %s\n' "$w_ws" "$w_lane" "$w_owner"
+			wt_unread=$((wt_unread + 1)) ;;
+		esac
+	done <<EOF
+$out
+EOF
+	say ""
+	say "$n worktree(s) in $nl lane(s) on $nw workstation(s), each as its lane's inventory last recorded it (LAST SEEN); nothing here is recomputed."
+	if [ "$wt_unread" != 0 ]; then
+		say "$wt_unread lane(s) above have an inventory that is there and could not be read: that is NOT a lane with no worktrees."
+	fi
+	say "  the current truth of one lane:  lanes-edit.sh lane-reconcile <lane>   (it reads the disk; this listing decides nothing)"
+	exit 0
+fi
+
 # `--all` WINS WHEREVER IT APPEARS IN THE LINE, so the label is cleared after the
 # loop and not inside the arm: `lanes --all --prefix repoA11` and
 # `lanes --prefix repoA11 --all` are the same command and must print the same