Skip to content

Notification queue: process claim is never applied, and failed sends are discarded #784

Description

@somethingwithproof

thold_notify.php claims rows from the notification queue but never filters on the claim, so concurrent runs can both send the same notification. A failed send is also discarded rather than retried.

The claim is written but not read back

thold_notify.php:111 marks the rows it intends to handle:

UPDATE notification_queue SET process_id = ? WHERE event_processed = 0

Then :148 drains them:

thold_notification_execute();

with no arguments. $pid defaults to 0, so $sql_where stays empty (thold_functions.php:7184, :7229, :7449) and the drain queries select every unprocessed row regardless of which process claimed it.

poller_thold.php:142-150 launches thold_notify.php on every poller cycle with no check that the previous run has finished, so two instances can overlap and both call mailer() on the same rows.

Passing $pid through to thold_notification_execute($pid) restores the intended behaviour.

The claim happens before the running-instance check

The claiming UPDATE at :111 runs before register_process_start() at :121, so a second instance overwrites the first instance's process_id even in the case where it goes on to exit.

Making the claim conditional (WHERE event_processed = 0 AND process_id = 0) and checking db_affected_rows() would make it atomic.

The guard is skipped entirely on non-unix

if ($config['cacti_server_os'] == 'unix') {
    ... register_process_start() ... exit(1);
}

On Windows a false from register_process_start() has no effect and the process continues. Same shape at thold_process.php:119.

A failed send is not retried

When mailer() fails, the row is still marked event_processed = 1 with error_code = 1 (thold_functions.php:7305-7310, :7500-7506). The notification is lost, with no backoff and no operator-visible queue of failures — which matches the symptom reported in #754.

Leaving event_processed = 0, incrementing an attempt counter and backing off would make a transient SMTP outage recoverable.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions