[aaastatsd] Fix event loop and stale RADIUS statistics by Griffin-micas · Pull Request #461 · sonic-net/sonic-host-services

Griffin-micas · 2026-10-15T05:54:20Z

aaastatsd handles every watchdog event, including the open and close events generated when it reads a statistics file. This retriggers handle_update() and causes repeated COUNTERS_DB writes.

Handle only file modification events. Remove the statistics file and COUNTERS_DB entry when a server is deleted or statistics are disabled, and clean stale statistics during daemon startup.

Add unit tests for event filtering, statistics state handling, and stale statistics cleanup.

Why I did it

aaastatsd uses watchdog's on_any_event() callback to monitor RADIUS
statistics files.

When handle_update() reads a statistics file, watchdog generates file open
and close events. These events invoke on_any_event() again, causing the
daemon to repeatedly read the same file and update COUNTERS_DB even though the
statistics content has not changed.

There are also stale-state issues:

Deleting a RADIUS server can recreate or retain its statistics file.
The corresponding RADIUS_SERVER_STATS entry can remain in COUNTERS_DB.
Disabling statistics does not consistently remove all statistics state.
Stale statistics from servers no longer present in CONFIG_DB can remain
after aaastatsd restarts.

How I did it

Replaced on_any_event() with on_modified() so only statistics content
modifications trigger synchronization.
Ignored directory modification events.
Added a common helper for updating and removing COUNTERS_DB statistics.
Removed both the statistics file and COUNTERS_DB entry when a RADIUS server
is deleted.
Removed statistics state when RADIUS statistics collection is disabled.
Cleaned up statistics belonging to servers no longer present in CONFIG_DB
during daemon startup.
Made the statistics enable-state check case-insensitive and default to
disabled when the setting is absent.
Added unit tests covering event filtering, enable-state handling, server
deletion, statistics cleanup, and stale-state removal.

How to verify it

Run the focused unit tests:

python3 -m pytest tests/aaastatsd_test.py -q -o addopts=''

Test Result

master: PASS

Image: SONiC.master.0-dbead5872

Evidence:

Reading a statistics file generated only opened and closed_no_write watchdog events, with no modified event.
Updating a RADIUS statistics file correctly updated RADIUS_SERVER_STATS in COUNTERS_DB.
Reading the file did not retrigger statistics processing.
aaastatsd remained active with the same PID and zero service restarts.
Deleting the test RADIUS server removed both its statistics file and its COUNTERS_DB entry.
No continuous event-processing loop or sustained high CPU usage was observed.

Full test log:

admin@sonic:~$ show version

SONiC Software Version: SONiC.master.0-dbead5872
SONiC OS Version: 13
Distribution: Debian 13.7
Kernel: 6.12.41+deb13-sonic-amd64
Build commit: dbead5872
Build date: Wed Oct 14 09:25:07 UTC 2026
Built by: micasrd@micasrd-PowerEdge-R730

Platform: x86_64-micas_m2-w6940-128qc-r0
HwSKU: M2-W6940-128QC
ASIC: broadcom
ASIC Count: 1
Serial Number: G1BRARN000048
Model Number: 01019D42
Hardware Revision: 150
Uptime: 06:36:00 up 19:25,  3 users,  load average: 1.64, 1.70, 1.77
Date: Thu 15 Oct 2026 06:36:00

Docker images:
REPOSITORY                    TAG                  IMAGE ID       SIZE
docker-gbsyncd-credo          latest               00aebbb71c08   346MB
docker-gbsyncd-credo          master.0-dbead5872   00aebbb71c08   346MB
docker-gnmi-watchdog          latest               d04cfdfb547d   308MB
docker-gnmi-watchdog          master.0-dbead5872   d04cfdfb547d   308MB
docker-sonic-bmp              latest               728dc98603ab   302MB
docker-sonic-bmp              master.0-dbead5872   728dc98603ab   302MB
docker-dhcp-relay             latest               3825388cbab6   343MB
docker-dhcp-relay             master.0-dbead5872   3825388cbab6   343MB
docker-sflow                  latest               da2b797620ca   341MB
docker-sflow                  master.0-dbead5872   da2b797620ca   341MB
docker-lldp                   latest               008bdbbff097   310MB
docker-lldp                   master.0-dbead5872   008bdbbff097   310MB
docker-syncd-brcm             latest               b3ac82d52d45   895MB
docker-syncd-brcm             master.0-dbead5872   b3ac82d52d45   895MB
docker-bmp-watchdog           latest               98a1f425631a   301MB
docker-bmp-watchdog           master.0-dbead5872   98a1f425631a   301MB
docker-sonic-mgmt-framework   latest               c72fe6967f82   369MB
docker-sonic-mgmt-framework   master.0-dbead5872   c72fe6967f82   369MB
docker-orchagent              latest               e16c6651e1d5   355MB
docker-orchagent              master.0-dbead5872   e16c6651e1d5   355MB
docker-sonic-gnmi             latest               9559603ecdfe   458MB
docker-sonic-gnmi             master.0-dbead5872   9559603ecdfe   458MB
docker-snmp                   latest               316e781dff1b   328MB
docker-snmp                   master.0-dbead5872   316e781dff1b   328MB
docker-sonic-p4rt             latest               67f6773b7d8c   403MB
docker-sonic-p4rt             master.0-dbead5872   67f6773b7d8c   403MB
docker-gbsyncd-agera2         latest               179e397b0216   361MB
docker-gbsyncd-agera2         master.0-dbead5872   179e397b0216   361MB
docker-restapi-sidecar        latest               1f308ce935bd   301MB
docker-restapi-sidecar        master.0-dbead5872   1f308ce935bd   301MB
docker-macsec                 latest               a8ba1f4d892a   342MB
docker-macsec                 master.0-dbead5872   a8ba1f4d892a   342MB
docker-dash-ha                latest               4194b6a38e7e   359MB
docker-dash-ha                master.0-dbead5872   4194b6a38e7e   359MB
docker-mux                    latest               f8ee761770ec   313MB
docker-mux                    master.0-dbead5872   f8ee761770ec   313MB
docker-eventd                 latest               91cf95fb0258   301MB
docker-eventd                 master.0-dbead5872   91cf95fb0258   301MB
docker-teamd                  latest               c6c913a4dad5   340MB
docker-teamd                  master.0-dbead5872   c6c913a4dad5   340MB
docker-sonic-otel             latest               dd37f08f6d16   642MB
docker-sonic-otel             master.0-dbead5872   dd37f08f6d16   642MB
docker-fpm-frr                latest               94a602920fb8   406MB
docker-fpm-frr                master.0-dbead5872   94a602920fb8   406MB
docker-platform-monitor       latest               c73ea82a247c   422MB
docker-platform-monitor       master.0-dbead5872   c73ea82a247c   422MB
docker-database               latest               04d636440b87   308MB
docker-database               master.0-dbead5872   04d636440b87   308MB
docker-gnmi-sidecar           latest               d57aa8553be8   301MB
docker-gnmi-sidecar           master.0-dbead5872   d57aa8553be8   301MB
docker-router-advertiser      latest               55b56837e7d7   301MB
docker-router-advertiser      master.0-dbead5872   55b56837e7d7   301MB
docker-sysmgr                 latest               1f82ba97f7f1   306MB
docker-sysmgr                 master.0-dbead5872   1f82ba97f7f1   306MB
docker-gbsyncd-broncos        latest               dcf248f7553c   360MB
docker-gbsyncd-broncos        master.0-dbead5872   dcf248f7553c   360MB
docker-nat                    latest               14005b9a3910   342MB
docker-nat                    master.0-dbead5872   14005b9a3910   342MB

admin@sonic:~$
admin@sonic:~$
admin@sonic:~$ sudo config radius add 172.31.32.50
admin@sonic:~$
admin@sonic:~$
admin@sonic:~$ show radius
RADIUS global auth_type pap (default)
RADIUS global retransmit 3 (default)
RADIUS global timeout 5 (default)
RADIUS global passkey Radius@123
RADIUS global statistics true

RADIUS_SERVER address 172.31.32.50
               auth_port 1812
               priority 1

admin@sonic:~$ redis-cli -n 2
127.0.0.1:6379[2]> keys *RADIUS*
1) "RADIUS_SERVER_STATS:172.31.32.50"
127.0.0.1:6379[2]> hgetall "RADIUS_SERVER_STATS:172.31.32.50"
 1) "access_accepts"
 2) "2"
 3) "access_challenges"
 4) "0"
 5) "access_rejects"
 6) "3"
 7) "access_requests"
 8) "5"
 9) "accounting_requests"
10) "0"
11) "accounting_responses"
12) "0"
13) "bad_authenticators"
14) "0"
15) "counter_0"
16) "0"
17) "counter_10"
18) "0"
19) "counter_12"
20) "0"
21) "counter_13"
22) "0"
23) "counter_14"
24) "0"
25) "counter_15"
26) "0"
27) "counter_16"
28) "0"
29) "counter_18"
30) "0"
31) "counter_19"
32) "0"
33) "counter_21"
34) "0"
35) "counter_22"
36) "0"
37) "counter_23"
38) "0"
39) "counter_24"
40) "0"
41) "counter_25"
42) "0"
43) "counter_26"
44) "0"
45) "counter_28"
46) "0"
47) "counter_29"
48) "0"
49) "counter_30"
50) "0"
51) "counter_31"
52) "0"
53) "counter_35"
54) "0"
55) "counter_6"
56) "0"
57) "counter_7"
58) "0"
59) "counter_8"
60) "0"
61) "counter_9"
62) "0"
63) "invalid_packets"
64) "0"
65) "retried_access_challenges"
66) "0"
67) "retried_access_requests"
68) "0"
69) "retried_accounting_requests"
70) "0"
71) "timeouts"
72) "0"
127.0.0.1:6379[2]> exit
admin@sonic:~$ sudo ls -l /etc/pam_radius_auth.d/statistics/
total 4
-rw-r--r-- 1 root root 71 Oct 15 06:38 172.31.32.50
admin@sonic:~$ sudo config radius delete 172.31.32.50
admin@sonic:~$ show radius
RADIUS global auth_type pap (default)
RADIUS global retransmit 3 (default)
RADIUS global timeout 5 (default)
RADIUS global passkey Radius@123
RADIUS global statistics true

admin@sonic:~$ redis-cli -n 2
127.0.0.1:6379[2]> keys *RADIUS*
(empty array)
127.0.0.1:6379[2]> exit
admin@sonic:~$ sudo ls -l /etc/pam_radius_auth.d/statistics/
total 0
admin@sonic:~$

azure-pipelines · 2026-10-15T05:54:30Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

mssonicbld · 2026-10-15T05:54:39Z

/azp run

azure-pipelines · 2026-10-15T05:54:48Z

Azure Pipelines: Successfully started running 1 pipeline(s).

mssonicbld · 2026-10-15T06:21:01Z

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

_{---Powered by SONiC BuildBot}

aaastatsd handles every watchdog event, including the open and close events generated when it reads a statistics file. This retriggers handle_update() and causes repeated COUNTERS_DB writes. Handle only file modification events. Remove the statistics file and COUNTERS_DB entry when a server is deleted or statistics are disabled, and clean stale statistics during daemon startup. Add unit tests for event filtering, statistics state handling, and stale statistics cleanup. Fixes sonic-net/sonic-buildimage#30148 Signed-off-by: Griffin Gao <griffin@micasnetworks.com>

mssonicbld · 2026-10-15T06:23:16Z

/azp run

azure-pipelines · 2026-10-15T06:23:25Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Griffin-micas force-pushed the fix-aaastatsd-event-loop branch from fb2c89c to 2fcf42b Compare October 15, 2026 06:22

Navigation Menu

Repository navigation

[aaastatsd] Fix event loop and stale RADIUS statistics - #461

[aaastatsd] Fix event loop and stale RADIUS statistics#461
Griffin-micas wants to merge 1 commit into
sonic-net:masterfrom
Griffin-micas:fix-aaastatsd-event-loop

Griffin-micas commented Oct 15, 2026 •

edited

Loading

Uh oh!

azure-pipelines Bot commented Oct 15, 2026

Uh oh!

mssonicbld commented Oct 15, 2026

Uh oh!

azure-pipelines Bot commented Oct 15, 2026

Uh oh!

mssonicbld commented Oct 15, 2026

Uh oh!

mssonicbld commented Oct 15, 2026

Uh oh!

azure-pipelines Bot commented Oct 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Griffin-micas commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Test Result

Uh oh!

azure-pipelines Bot commented Oct 15, 2026

Uh oh!

mssonicbld commented Oct 15, 2026

Uh oh!

azure-pipelines Bot commented Oct 15, 2026

Uh oh!

mssonicbld commented Oct 15, 2026

Uh oh!

mssonicbld commented Oct 15, 2026

Uh oh!

azure-pipelines Bot commented Oct 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Griffin-micas commented Oct 15, 2026 •

edited

Loading