Skip to content

ui-smoke phase 3: screenshots, writable config mirrors, homing settle #4136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
grandixximo wants to merge 19 commits into
base: master
Choose a base branch
from
Open

ui-smoke phase 3: screenshots, writable config mirrors, homing settle #4136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
aacd0c6
test: ui-smoke phase 3, screenshot the GUI on failure and confirm on …
grandixximo Jun 4, 2026
8855a05
test: run gmoccapy ui-smoke against a writable config mirror
grandixximo Jun 5, 2026
9514b7b
ui-smoke: settle after homing before requesting AUTO
grandixximo Jun 5, 2026
9ed19b2
test: ui-smoke compare confirm shot to a committed reference image
grandixximo Jun 12, 2026
d78cb6c
ci: install gdb up front so the ui-smoke crash dump path does not apt…
grandixximo Jun 16, 2026
53d15d5
tests/runtests: export NOSUDO via RUNTESTS_NOSUDO so children can hon…
grandixximo Jun 16, 2026
a43bd74
test: ui-smoke touchy-fit, fail if the window exceeds the screen
grandixximo Jun 5, 2026
a23d4ff
test: bump ui-smoke compare fuzz to 40 % so the metric tracks real drift
grandixximo Jun 16, 2026
1c007bd
test: refresh touchy ui-smoke for the new minimum-height patch
grandixximo Jun 16, 2026
c4de595
test: regenerate axis/gmoccapy/qtdragon references from CI output
grandixximo Jun 16, 2026
f318204
ci: move gdb to control build-deps so package-arch tests get it too
grandixximo Jun 16, 2026
ed7f125
ui-smoke: only use a readable core from this run for the native backt…
grandixximo Jun 16, 2026
ffd8a82
tests: dual red and absolute-difference ui-smoke diffs at 0% fuzz
grandixximo Jun 17, 2026
dbf80a3
test: adopt CI confirm shots as ui-smoke references
grandixximo Jun 17, 2026
31c9888
tests: stabilize qtdragon ui-smoke preview and gcode scroll
grandixximo Jun 17, 2026
bd0bf75
tests: arm the ui-smoke core dump via sudo so CI gets a native backtrace
grandixximo Jun 18, 2026
de967fe
tests: settle the ui-smoke confirm shot until the UI stops changing
grandixximo Jun 18, 2026
a4b2638
test: update axis ui-smoke reference to the settled Max Velocity
grandixximo Jun 19, 2026
8bc3973
tests: make crash dumps opt-in via runtests -d
grandixximo Jun 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 27 additions & 3 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,19 @@ jobs:
- name: Test
run: |
set -x
scripts/rip-environment runtests -p
scripts/rip-environment runtests -p -d
- name: Upload UI smoke screenshots
if: always()
uses: actions/upload-artifact@v7
with:
name: ui-smoke-screenshots-gcc
path: |
tests/ui-smoke/**/screenshot.png
tests/ui-smoke/**/confirm.png
tests/ui-smoke/**/diff.png
tests/ui-smoke/**/diff-abs.png
if-no-files-found: ignore
retention-days: 14
- name: Verify no untracked or modified files after test
run: |
.github/scripts/verify-clean-repo.sh
Expand Down Expand Up @@ -111,7 +123,19 @@ jobs:
- name: Test
run: |
set -x
scripts/rip-environment runtests -p
scripts/rip-environment runtests -p -d
- name: Upload UI smoke screenshots
if: always()
uses: actions/upload-artifact@v7
with:
name: ui-smoke-screenshots-clang
path: |
tests/ui-smoke/**/screenshot.png
tests/ui-smoke/**/confirm.png
tests/ui-smoke/**/diff.png
tests/ui-smoke/**/diff-abs.png
if-no-files-found: ignore
retention-days: 14
- name: Verify no untracked or modified files after test
run: |
.github/scripts/verify-clean-repo.sh
Expand Down Expand Up @@ -237,7 +261,7 @@ jobs:
passwd -d testrunner
adduser testrunner sudo
chmod 0777 $(find tests/ -type d) # make test dirs world-writable for the testrunner
su -c "./scripts/runtests -p ./tests" testrunner
su -c "./scripts/runtests -p -d ./tests" testrunner
- name: Verify no untracked or modified files after test
run: |
.github/scripts/verify-clean-repo.sh ':(exclude)VERSION' ':(exclude)debian/changelog'
Expand Down
3 changes: 3 additions & 0 deletions debian/control.top.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ Build-Depends:
tk@TCLTK_VERSION@-dev,
xvfb <!nocheck>,
x11-xserver-utils <!nocheck>,
x11-utils <!nocheck>,
gdb <!nocheck>,
imagemagick <!nocheck>,
python3-opengl <!nocheck>,
python3-pyqt5 <!nocheck>,
python3-pyqt5.qsci <!nocheck>,
Expand Down
11 changes: 9 additions & 2 deletions scripts/runtests.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -357,6 +357,12 @@ Usage:

$P -v
Show stdout and stderr (normally it's hidden).

$P -d
Enable crash dumps: on a crashing test, point kernel.core_pattern
at a temporary dir via sudo and print a gdb backtrace of the core.
Off by default since it changes a global system setting; CI passes
it. Honors -u (no sudo, no core_pattern change).
EOF
}

Expand All @@ -365,14 +371,15 @@ NOCLEAN=0
NOSUDO=false
STOP=0
PRINT=0
while getopts cnuvsph opt; do
while getopts cnuvsphd opt; do
case "$opt" in
c) CLEAN_ONLY=1 ;;
n) NOCLEAN=1 ;;
u) NOSUDO=true ;;
u) NOSUDO=true; export RUNTESTS_NOSUDO=1 ;;
v) VERBOSE=1 ;;
s) STOP=1 ;;
p) PRINT=1 ;;
d) export ENABLE_CRASHDUMPS=1 ;;
h|?) usage; exit 0 ;;
*) usage; exit 1 ;;
esac
Expand Down
8 changes: 8 additions & 0 deletions tests/ui-smoke/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,11 @@ linuxcnc.err
linuxcnc.pid
ui-smoke.out
ui-smoke.err
# confirm.png is the per-run shot; diff.png and diff-abs.png are its
# comparison to the committed reference.png (which IS tracked, so it is
# not listed here).
screenshot.png
confirm.png
ui-smoke-qt.png
diff.png
diff-abs.png
29 changes: 29 additions & 0 deletions tests/ui-smoke/README
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,35 @@ Shared helpers live in _lib/:
checkresult.sh shared pass/fail predicate
skip-if-missing.sh shared skip predicate

Failure diagnostics (failure path only, no cost on a green run):
crashdump.sh arms a core dump and prints a native backtrace if a GUI
segfaults (the Qt/dbus/GL frames PYTHONFAULTHANDLER misses)
screenshot.sh photographs the Xvfb root window before teardown. On a
failure it captures the cause (a GUI hung on a blocking
modal leaves no core and no traceback); on a clean Phase 2
run it captures confirm.png, the GUI in its post-movement
idle state (final DRO / toolpath) for visual confirmation.
CI uploads both as build artifacts (ui-smoke-screenshots-*
in ci.yml).
compare.sh on a clean run, compares confirm.png to the committed
known-good reference.png (ImageMagick) and writes a
highlighted diff.png, also uploaded as an artifact.

Reference images:
reference.png committed per-GUI known-good shot (in a --run-program test
directory); diff.png is confirm.png compared against it.
The comparison NEVER fails a test: freetype/font versions
differ across distros, so some drift is expected. The diff
is only recorded, not used to gate the test.

Create or refresh the references on a built run-in-place tree:
. scripts/rip-environment
tests/ui-smoke/_lib/make-references.sh # all run-program GUIs
tests/ui-smoke/_lib/make-references.sh axis # just one
This sets UI_SMOKE_UPDATE_REFERENCE=1, which makes compare.sh save each
clean confirm shot as that test's reference.png. Review the PNGs before
committing; they are baselines from the generating machine's fonts.

Skip vs fail policy: the only condition we skip on is xvfb-run absence
(rare local dev env). Python and gi typelib deps the GUIs need are
declared in debian/control under !nocheck so apt-get build-dep
Expand Down
88 changes: 88 additions & 0 deletions tests/ui-smoke/_lib/compare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
#!/bin/bash
# Known-good image comparison for the UI smoke confirm shots. Complements
# screenshot.sh: that grabs confirm.png on a clean run; this compares it to
# a committed reference.png and writes two visual diffs. Like screenshot.sh
# it carries no state and is a logged no-op whenever it cannot run, so it can
# never turn a pass into a fail.
#
# Two diff images, both at 0% fuzz so nothing is hidden:
# diff.png red highlight over a faded copy of the reference (where it
# changed, ImageMagick "compare" style)
# diff-abs.png absolute per-channel difference |b - a|, black where equal
# and bright where it changed (unbiased, magnitude preserved)
#
# Policy: we never fail a test on the image difference. freetype/font
# versions differ across distros, so some drift is expected; the diffs are
# here to record what changed, not to gate. The function always returns 0.
#
# Local "make a known-good image" workflow: run a test with
# UI_SMOKE_UPDATE_REFERENCE=1 (see make-references.sh) and the freshly
# captured shot is saved as the committed reference instead of compared.

# Set IM_COMPARE and IM_CONVERT to the IM7 or IM6 entry points; return 1
# if ImageMagick is absent.
_im_tools() {
if command -v magick >/dev/null 2>&1; then
IM_COMPARE="magick compare"; IM_CONVERT="magick"
elif command -v compare >/dev/null 2>&1; then
IM_COMPARE="compare"; IM_CONVERT="convert"
else
return 1
fi
}

# compare_to_reference <shot> <reference> <diff> <diff_abs>
# Compare the captured shot to the committed reference, writing a red-highlight
# diff and an absolute-difference diff. Always returns 0.
compare_to_reference() {
shot="$1"
reference="$2"
diff="$3"
diff_abs="$4"

if [ ! -s "$shot" ]; then
echo "compare: no shot at $shot, skipping"
return 0
fi

# Update mode: adopt this shot as the new known-good reference.
if [ "${UI_SMOKE_UPDATE_REFERENCE:-}" = "1" ]; then
if cp -f "$shot" "$reference"; then
echo "compare: saved reference $reference (UI_SMOKE_UPDATE_REFERENCE=1)"
else
echo "compare: failed to save reference $reference"
fi
return 0
fi

if [ ! -s "$reference" ]; then
echo "compare: no reference at $reference yet, skipping (run with UI_SMOKE_UPDATE_REFERENCE=1 to create one)"
return 0
fi

_im_tools || {
echo "compare: no ImageMagick available, skipping"
return 0
}

# Red-highlight diff at 0% fuzz: -metric AE counts every differing pixel
# (interpretable), and the same call writes diff.png. compare exits 0
# (identical), 1 (differ) or 2 (error, e.g. the shots are different
# sizes). We log the outcome and always succeed.
metric=$($IM_COMPARE -metric AE -fuzz 0% "$reference" "$shot" "$diff" 2>&1)
rc=$?
case "$rc" in
0) echo "compare: $shot matches $reference (AE=$metric)" ;;
1) echo "compare: $shot differs from $reference (AE=$metric differing pixels); diff at $diff" ;;
*) echo "compare: could not compare $shot to $reference (rc=$rc): $metric" ;;
esac

# Absolute-difference diff |b - a|: black where equal, bright where it
# changed. Unbiased (direction does not matter) and keeps magnitude.
if $IM_CONVERT "$reference" "$shot" -compose difference -composite "$diff_abs" 2>/dev/null; then
echo "compare: absolute-difference diff at $diff_abs"
else
echo "compare: could not write absolute-difference diff $diff_abs"
fi
return 0
}
83 changes: 53 additions & 30 deletions tests/ui-smoke/_lib/crashdump.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,68 @@
#!/bin/bash
# Native crash capture for the UI smoke launchers. A GUI segfault is the
# failure these tests most need to explain, and it lands in C/C++ (Qt,
# dbus, GL) where PYTHONFAULTHANDLER stops at the event-loop frame. Arm a
# core dump before launch; after the run, if the GUI left a core, print a
# native backtrace into the log so CI shows the faulting frame directly.
# Source with LIB_DIR set; runs only on the failure path, so green runs
# pay nothing.
# Native crash capture for the UI smoke launchers. A GUI segfault lands in
# C/C++ (Qt, dbus, GL); PYTHONFAULTHANDLER (set in launch-env.sh) prints a
# Python traceback to linuxcnc.err naming the frame that called in, which
# is the reliable, environment-independent crash signal and is surfaced in
# every failure log. This helper adds a best-effort native backtrace on
# top, but only when runtests is given -d (ENABLE_CRASHDUMPS=1): arm a core
# dump before launch, and after the run, if a readable core from this run is
# present, gdb-print its backtrace. The core only materialises when we can
# point kernel.core_pattern at a writable dir, which needs root, so we do it
# via sudo (CI has passwordless sudo; the suite never runs as root since
# linuxcnc refuses to). It is opt-in because that sudo changes a global
# system setting; runtests -u further skips the sudo. Without -d the native
# backtrace is off and only the Python traceback remains. Source with
# LIB_DIR set; runs only on the failure path, so green runs pay nothing.

crashdump_arm() {
# Off unless runtests was given -d: arming changes a global system
# setting (kernel.core_pattern) via sudo, so it is opt-in. The Python
# faulthandler traceback does not depend on this and is always present.
[ "${ENABLE_CRASHDUMPS:-0}" = "1" ] || return 0
CORE_DIR="$(mktemp -d -t ui-smoke-cores.XXXXXX)"
export CORE_DIR
ulimit -c unlimited 2>/dev/null || true
# core_pattern is global; only set it if already root. Never sudo:
# the suite must run unattended. Non-root falls back to a cwd "core".
if [ "$(id -u)" = 0 ]; then
sysctl -w "kernel.core_pattern=$CORE_DIR/core.%e.%p" >/dev/null 2>&1 || true
# core_pattern is global; point it at our writable dir via sudo so the
# kernel writes a core we can read. Skip under runtests -u to respect a
# user opting out of root tweaks; if sudo is unavailable the core lands
# wherever the system core_pattern says and we fall back to the Python
# traceback. The id -u check is dropped: tests never run as root.
if [ "${RUNTESTS_NOSUDO:-0}" != "1" ]; then
sudo sysctl -w "kernel.core_pattern=$CORE_DIR/core.%e.%p" >/dev/null 2>&1 || true
fi
}

crashdump_report() {
[ "${ENABLE_CRASHDUMPS:-0}" = "1" ] || return 0
[ -n "${CORE_DIR:-}" ] || return 0
local core
# shellcheck disable=SC2012 # mktemp dir, no odd filenames
core=$(ls -t "$CORE_DIR"/core* ./core* /tmp/core* 2>/dev/null | head -1)
if [ -n "$core" ]; then
local c core=""
# Only trust a core we know is from this run and can actually read:
# one the kernel wrote into our fresh CORE_DIR (root path, where we set
# core_pattern), or a relative "core" in the cwd that postdates arming.
# A broad /tmp glob would pick up a stale or foreign core (often root-
# owned), and gdb would just print "Permission denied".
for c in "$CORE_DIR"/core*; do
[ -e "$c" ] && [ -r "$c" ] && { core="$c"; break; }
done
if [ -z "$core" ]; then
for c in ./core*; do
[ -e "$c" ] && [ -r "$c" ] && [ "$c" -nt "$CORE_DIR" ] && { core="$c"; break; }
done
fi
if [ -n "$core" ] && command -v gdb >/dev/null 2>&1; then
echo "=== crash: native backtrace ($core) ==="
# gdb reads the core; pull it in if missing, only when root.
if ! command -v gdb >/dev/null 2>&1 && [ "$(id -u)" = 0 ]; then
apt-get install -y -q gdb >/dev/null 2>&1 || true
fi
if command -v gdb >/dev/null 2>&1; then
# "bt" first: gdb auto-selects the faulting thread on a SIGSEGV
# core. "thread apply all bt" after gives the rest.
gdb -batch -nx \
-ex "bt" \
-ex "echo \n=== all threads ===\n" \
-ex "thread apply all bt" \
"$(command -v python3)" "$core" 2>&1 | head -400
else
echo "(gdb unavailable; core left at $core)"
fi
# "bt" first: gdb auto-selects the faulting thread on a SIGSEGV
# core. "thread apply all bt" after gives the rest.
gdb -batch -nx \
-ex "bt" \
-ex "echo \n=== all threads ===\n" \
-ex "thread apply all bt" \
"$(command -v python3)" "$core" 2>&1 | head -400
else
# No readable core (the common non-root case). The Python
# faulthandler traceback in linuxcnc.err already names the crash
# site; the native backtrace is only a best-effort extra.
echo "=== crash: no readable core dump; see the Python traceback in linuxcnc.err above ==="
fi
rm -rf "$CORE_DIR"
}
12 changes: 12 additions & 0 deletions tests/ui-smoke/_lib/drive.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@
# stability check catches that.
STATE_STABILITY_S = 0.5
STATE_RETRY_BUDGET = 6
# Pause after homing before requesting AUTO. gmoccapy only enables AUTO
# once it has processed the all-homed signal in its own event loop (and
# re-asserts MANUAL itself on that signal). Requesting AUTO before then is
# rejected: it bounces back to MANUAL with an "It is not possible to
# change to Auto Mode" warning. ensure_mode would retry and win, but the
# warning lingers on screen; this settle lets the GUI catch up first.
POST_HOME_SETTLE_S = 2.0

# linuxcnc launcher PID, written to linuxcnc.pid by the launcher and read
# once at startup. The driver watches it so a GUI crash, which tears
Expand Down Expand Up @@ -297,6 +304,11 @@ def run_program(cmd, stat, ngc_path, expect_delta_mm, tol, run_timeout):
if not home_all(cmd, stat, timeout=60.0):
return False

# Let the GUI react to the all-homed transition before requesting AUTO,
# so it does not reject the mode change (see POST_HOME_SETTLE_S).
time.sleep(POST_HOME_SETTLE_S)
stat.poll()

if not ensure_mode(cmd, stat, linuxcnc.MODE_AUTO, "MODE_AUTO"):
return False

Expand Down
39 changes: 39 additions & 0 deletions tests/ui-smoke/_lib/gmoccapy-prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
#!/bin/bash
# Sourced by the gmoccapy ui-smoke tests (smoke and quit) to run gmoccapy
# against a writable copy of its sim config. Sets GMOCCAPY_INI to the
# mirrored ini path; the caller then execs run-gui.sh or quit-launch.sh
# with "$GMOCCAPY_INI". Must be sourced with LIB_DIR already set.
#
# gmoccapy writes its preferences file next to the config: with no
# PREFERENCE_FILE_PATH in the ini, getiniinfo falls back to
# <config-dir>/<MACHINE>.pref. CI mounts the workspace read-only for the
# runtime user, so that write raises PermissionError partway through
# __init__ (during _get_pref_data, before the MDIHistory widget's
# _hal_init runs). gmoccapy pops an error dialog and limps on in a
# half-initialised state: the interp-idle handler then hits a widget with
# no .stat and throws a second dialog. Both vanish once the config dir is
# writable. Mirror it to tmp, same fix qtdragon-prepare.sh uses.

: "${LIB_DIR:?gmoccapy-prepare.sh must be sourced with LIB_DIR set}"

SRC_DIR="$(cd "$LIB_DIR/../../../configs/sim/gmoccapy" && pwd)"

WORK_DIR="$(mktemp -d -t ui-smoke-gmoccapy.XXXXXX)"
trap 'rm -rf "$WORK_DIR"' EXIT
cp -r "$SRC_DIR/." "$WORK_DIR/"

# Seed the preference file (config dir + <MACHINE>.pref; MACHINE=gmoccapy)
# so the first-run "Important change(s)" modal stays hidden. That dialog
# runs a nested gtk loop, so under xvfb it never gets dismissed: it sits
# on top of the UI in the confirm shot and, worse, swallows the SIGTERM
# in the quit test (the loop keeps running after main_quit). A real user
# ticks "Don't show this again" once; hide_startup_messsage replicates
# that. The triple-s key matches gmoccapy's own (sic).
cat >"$WORK_DIR/gmoccapy.pref" <<'PREF'
[DEFAULT]
hide_startup_messsage = 99
PREF

# Consumed by the sourcing test.sh, which execs the launcher with it.
# shellcheck disable=SC2034
GMOCCAPY_INI="$WORK_DIR/gmoccapy.ini"
Loading
Loading