Benchmarks Website V3: Admin and Auto-Deploy

[connortsui20]

Summary

http://ec2-18-219-54-101.us-east-2.compute.amazonaws.com:3000/

Hopefully the Benchmarks V3 finale. The website is one Rust binary + one DuckDB file managed by
systemd: vortex-bench-deploy.timer polls origin/develop every 60s, rebuilds + atomic-swaps the
binary on website-touching commits, /health-verifies, rolls back on failure.
vortex-bench-backup.timer fires hourly, calls a new loopback-only POST /api/admin/snapshot
(per-table Vortex export under one READ ONLY transaction), tar czfs the result, and uploads to
s3://vortex-benchmark-results-database/v3-backups/<ts>.tar.gz. Admin endpoints (snapshot +
read-only POST /api/admin/sql) sit on a separate ADMIN_BEARER_TOKEN-gated listener that refuses
to start on a non-loopback bind. v2 keeps shipping unchanged via the existing publish workflow until
DNS flips.

Operator runbook (one-time install on a fresh EC2 host, day-to-day, recovery):
benchmarks-website/ops/README.md. On a clean box, git clone
the repo, edit INGEST_BEARER_TOKEN + ADMIN_BEARER_TOKEN in /etc/vortex-bench.env,
./benchmarks-website/ops/install.sh.

Admin cheat sheet (generated by claude)

Status — "what's the state right now?"

systemctl status vortex-bench-server                    # main service status + recent log lines
systemctl list-timers vortex-bench-deploy.timer vortex-bench-backup.timer
                                                         # next + last fire of each timer
curl -fsS http://127.0.0.1:3000/health | jq             # live build_sha, row counts, db path
cat /var/lib/vortex-bench/last-deployed-sha             # SHA the deploy timer thinks is live
readlink /var/lib/vortex-bench/bin/vortex-bench-server  # symlink target = versioned binary

Manual triggers — "do something now"

/var/lib/vortex-bench/ops/restart.sh                    # restart binary in place, show before/after
/var/lib/vortex-bench/ops/force-rebuild.sh              # rebuild origin/$DEPLOY_BRANCH + restart
sudo systemctl start vortex-bench-deploy.service        # run deploy IF origin moved (no-op otherwise)
sudo systemctl start vortex-bench-backup.service        # snapshot to S3 right now
/var/lib/vortex-bench/ops/migrate.sh run --output \
    /var/lib/vortex-bench/bench.duckdb                  # re-run v2→v3 migration
/var/lib/vortex-bench/ops/inspect.sh "SELECT COUNT(*) FROM commits;"  # read-only SQL

Logs — "what happened?"

journalctl -fu vortex-bench-server                      # live server logs
journalctl -fu vortex-bench-deploy.service              # watch a deploy in progress
journalctl -u vortex-bench-deploy.service --since '15 min ago' --no-pager
journalctl -u vortex-bench-backup.service --since '4 hours ago' --no-pager
journalctl -u vortex-bench-server --since '5 min ago' --no-pager

Pause / resume the autopilot — "stop fiddling for a minute"

sudo systemctl stop vortex-bench-deploy.timer           # pause auto-deploys
sudo systemctl start vortex-bench-deploy.timer          # resume

sudo systemctl stop vortex-bench-backup.timer           # pause hourly snapshots
sudo systemctl start vortex-bench-backup.timer          # resume

sudo systemctl stop vortex-bench-server                 # take the site down
sudo systemctl start vortex-bench-server                # bring it back

Config changes — "edit settings"

sudo $EDITOR /etc/vortex-bench.env                      # tokens, DEPLOY_BRANCH, S3 prefix, etc.
sudo systemctl restart vortex-bench-server              # pick up server-side env changes
                                                         # (no rebuild — restart.sh shows new pid)
/var/lib/vortex-bench/ops/force-rebuild.sh              # pick up build-time env changes

Recover from S3 — "restore a backup"

aws s3 ls s3://vortex-benchmark-results-database/v3-backups/ | tail
LATEST=$(aws s3 ls s3://vortex-benchmark-results-database/v3-backups/ | tail -1 | awk '{print $4}')
aws s3 cp "s3://vortex-benchmark-results-database/v3-backups/$LATEST" /tmp/
cd /tmp && tar xzf "$LATEST"
ts="${LATEST%.tar.gz}"
sudo systemctl stop vortex-bench-server
sudo -u ec2-user rm -f /var/lib/vortex-bench/bench.duckdb \
                       /var/lib/vortex-bench/bench.duckdb.wal
duckdb /var/lib/vortex-bench/bench.duckdb <<EOF
INSTALL vortex;
LOAD vortex;
.read /tmp/${ts}/schema.sql
INSERT INTO commits             SELECT * FROM read_vortex('/tmp/${ts}/commits.vortex');
INSERT INTO query_measurements  SELECT * FROM read_vortex('/tmp/${ts}/query_measurements.vortex');
INSERT INTO compression_times   SELECT * FROM read_vortex('/tmp/${ts}/compression_times.vortex');
INSERT INTO compression_sizes   SELECT * FROM read_vortex('/tmp/${ts}/compression_sizes.vortex');
INSERT INTO random_access_times SELECT * FROM read_vortex('/tmp/${ts}/random_access_times.vortex');
INSERT INTO vector_search_runs  SELECT * FROM read_vortex('/tmp/${ts}/vector_search_runs.vortex');
EOF
sudo systemctl start vortex-bench-server

The 5 commands you'll actually use day-to-day: restart.sh, force-rebuild.sh, inspect.sh,
journalctl -fu …, curl /health | jq. Everything else is rare.

[AdamGS]

vortex-k8s @robert3005

[codspeed-hq]

Merging this PR will improve performance by 19.79%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
✅ 1236 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	chunked_varbinview_opt_canonical_into[(1000, 10)]	225.1 µs	187.9 µs	+19.79%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/bench-v3-autopilot (bbc364f) with develop (f852d72)}

[palaska]

You'll need to delete https://github.com/vortex-data/vortex/blob/develop/.github/workflows/publish-benchmarks-website.yml, otherwise it will publish a new image and break the live website

[connortsui20]

I will figure out the performance problems before I merge this