Bug: SIGABRT when two independent polling proxies use the same LoLa event #797

Activity

1. tfa2si

   added the

   Bug

   issue type on Jul 28, 2026
2. github-project-automation

   added this to COM - Communication FTon Jul 28, 2026
3. github-project-automation

   moved this to Backlog in COM - Communication FTon Jul 28, 2026
4. tfa2si commented on Jul 30, 2026

   

   tfa2si

   on Jul 30, 2026

   Last edited by tfa2si

   Author

   More actions

   We hit a different symptom of this same bug: a silent freeze in the notification dispatch when a second subscriber joins a service instance where one subscriber uses pure polling.

   Trigger Conditions

   • Service instance has at least 2 events
   • One subscriber (process A) uses pure polling (GetNewSamples with no SetReceiveHandler) on event 1
   • Another subscriber (process B) calls Subscribe() on event 2 of the same service instance
   • Result: process A's notification handler (for an event on a different service instance) deterministically stops firing

   Our Exact Setup

   • HfcZoneFeedback service instance has 2 events: zone_angle_batch (pure polling by HFC) and zone_stats (event-driven by telemetry)
   • HFC's RT thread polls zone_angle_batch via GetNewSamples — no notification thread involved
   • Telemetry calls zone_stats.Subscribe(64U) — this writes a TransactionLogNode via CAS into SHM
   • zone_adapter (the skeleton process) has a different proxy (HfcZoneCommand on another service instance) with SetReceiveHandler
   • Result: the instant telemetry subscribes, zone_adapter's zone_cmd handler stops firing. However, zone_adapter's own stats thread (which bypasses LoLa notifications and calls zone_stats.Send() directly) continues running fine.

   Why Pure Polling Matters

   With pure polling, the skeleton's notification thread for the polled event (zone_angle_batch) still exists in the LoLa executor thread pool even though SetReceiveHandler was never called. This executor thread shares the same TransactionLogSet SHM region. When Subscribe writes to proxy_transaction_logs_ via CAS for a different event in the same ServiceDataControl, the concurrent iteration by the executor thread reads corrupted state from non-atomic TransactionLogSlot bitfields.

   Key Insight

   The race requires the skeleton process to have an active LoLa executor thread (created by any Subscribe + SetReceiveHandler in that process, not necessarily on the same event) that iterates SHM while a concurrent RegisterProxyElement from another process writes to SHM.

   Workaround Applied

   Moved zone_stats to its own dedicated service instance (HfcZoneStats, serviceId 6666) with separate SHM segment. No shared TransactionLogSet across events = no race.

   Test Results

   With the workaround:

   • zone_angle_batch.Send() — 1623 calls over 15 seconds
   • zone_stats.Send() — 9 calls over 15 seconds
   • Both running concurrently, zero freeze
5. castler

   added

   p0

   on Aug 5, 2026
6. crimson11 commented on Aug 6, 2026

   

   crimson11

   on Aug 6, 2026

   Contributor

   More actions

   @tfa2si
   Trying to understand your issue.
   I focus now on the imho essential part of your report:

   two threads at the same RT priority concurrently access the same event slot via GetNewSamples()

   This can't happen, when you follow our AoUs. I.e. our AoU forbids, that you do concurrent access to score::mw::com provided high-level object instances like Proxy, ProxyEvent, ProxyField, Skeleton, SkeletonEvent, SkeletonField, ...
   See e.g. our AoU ScoreReq.AoU NoneReentrantMethodsPerEventInstance !

   Just for you to know: We have separate Transaction-Logs for reference-transactions per ProxyEvent instance! Thus, I fail to see, how you would get a race-condition on a TransactionLogSlot, when you follow our AoU.
   Can you just explain this specific issue, how a data-race can happen, when our AoUs are followed.

   For the rest of your bug-report:

   • in the summary you state: assumption when Subscribe() spawns executor threads. - we do NOT spawn a thread in Subscribe(). Why do you think we do? I followed your "link" "/message_passing_service_instance.cpp:925-931". There is no executor/thread spawned?
   • it is generally hard to follow (sounds a bit AI generated - apologize, if it is not the case ;) ).
   • The 2nd comment you added: "We hit a different symptom of this same bug: ..." seems unrelated. I propose to either postpone or create another issue.
7. tfa2si commented on Aug 10, 2026

   

   tfa2si

   on Aug 10, 2026

   Last edited by tfa2si

   Author

   More actions

   hi crimson11!
   You are right about two points:

   1. Subscribe() itself does not spawn a thread. My statement that it creates an executor thread was incorrect and I will remove it.
   2. The silent-freeze symptom should be tracked separately from the SIGABRT report. I agree that I should create a separate issue for it.

   The SIGABRT reproduction does not involve concurrent calls on the same C++ ProxyEvent object.

   There are two separate processes, each with its own proxy object and its own polling thread:

   Process A                         Process B
   ---------                         ---------
   Proxy A                            Proxy B
   event.Subscribe()                 event.Subscribe()
   event.GetNewSamples()             event.GetNewSamples()
   

   Both proxy objects refer to the same LoLa service instance and therefore access shared-memory communication state. No Proxy, ProxyEvent, or other high-level communication object is shared between threads or processes.
8. tfa2si

   changed the title [-]Bug: TransactionLog BEGIN/END bitfields race between same-priority threads causes SIGABRT[/-] [+]Bug: SIGABRT when two independent polling proxies use the same LoLa event[/+] on Aug 10, 2026
9. tfa2si commented on Aug 10, 2026

   

   tfa2si

   on Aug 10, 2026

   Author

   More actions

   Thank you for pointing me to ScoreReq.AoU NoneReentrantMethodsPerEventInstance.
   I agree that two application threads must not concurrently access the same
   high-level ProxyEvent instance.

   The reduced reproducer does not do that. It contains two separate client
   processes, and each process creates and uses its own independent
   Proxy/ProxyEvent instance. There is no shared high-level communication
   object between the clients and no concurrent access to one ProxyEvent
   instance by two application threads.

   I have created a minimal three-process reproducer:

   • one LoLa skeleton process;
   • two independent client processes, each with its own Proxy/ProxyEvent
     instance;
   • both clients subscribe to the same zone_angle_batch event;
   • both clients call GetNewSamples() from independent polling loops;
   • no SetReceiveHandler() is used;
   • both clients use the same service instance;
   • both clients run under the normal Linux scheduler (SCHED_OTHER).

   The reproducer does not request or require SCHED_FIFO or any real-time
   scheduling permissions. The latest trace therefore does not involve two
   threads at the same RT priority.

   With score_communication v0.2.1, the latest run reproduced a client abort under plain SCHED_OTHER. The captured backtrace resolves through:

   score::mw::com::impl::lola::TransactionLog::ReferenceTransactionBegin()
   score::mw::com::impl::lola::ProxyEventDataControlLocalView::ReferenceNextEvent()
   score::mw::com::impl::lola::SlotCollector::CollectSlots()
   score::mw::com::impl::lola::SlotCollector::GetNewSamplesSlotIndices()
   score::mw::com::impl::lola::ProxyEvent::GetNewSamples()
   

   In the latest run:

   • Client B aborted and produced the SIGABRT backtrace.
   • Client A continued polling under SCHED_OTHER to 18,000 cycles.
   • An earlier run also reproduced the second observed behavior: one client continued polling but stopped receiving new samples.

   This shows that real-time scheduling is not a prerequisite for the transaction-log failure.

   Could you clarify the cross-process case:

   1. Does ScoreReq.AoU NoneReentrantMethodsPerEventInstance also forbid
      concurrent GetNewSamples() calls when they are made by independent
      ProxyEvent instances in different processes?
   2. If this cross-process use is forbidden, which AoU defines that restriction?
   3. If it is supported, could this assertion indicate a defect in shared-memory
      transaction handling?
   4. Does the separate transaction log per ProxyEvent instance also isolate
      the transaction state when the instances use the same shared-memory service
      instance?

   As a temporary workaround, we avoid multiple independent polling clients on the same service instance. We either use one LoLa consumer and fan out through application-owned shared state, or expose separate service instances for separate consumers.

   The attached archive lola-797-reproducer.tar.gz contains:

   • the minimal skeleton and client sources;
   • the Bazel target definitions;
   • the service configuration;
   • the runbook;
   • the corrected issue draft;
   • the fresh skeleton and client logs from the non-RT run.

   I have attached the archive together with the three trace files:

   lola_trace4_client_a.log

   lola_trace4_client_b.log

   lola_trace4_skeleton.log

   lola-797-reproducer.tar.gz
10. tfa2si commented on Aug 10, 2026

    

    tfa2si

    on Aug 10, 2026

    Last edited by tfa2si

    Author

    More actions

    lola-issue797-evidence.tar.gz

    I performed an additional phase-isolation test to determine whether the failure is caused by Subscribe() or by GetNewSamples().

    With score_communication v0.3.0, process A continuously polled topic_a while process B used the same event:

    • B calling Subscribe() and then waiting did not affect A; A continued receiving
    • B joining three seconds later, calling GetNewSamples() exactly once, and then waiting also did not affect A; the call returned successfully with 32 samples.
    • When both processes continuously called GetNewSamples() on topic_a, A reproducibly aborted with SIGABRT while B continued polling.

    The same-event test used maxSubscribers: 2, and both Subscribe() calls returned successfully. Therefore, the observed failure is not explained by a subscriber-count configuration limit.

    The current evidence does not implicate Subscribe() alone or one GetNewSamples() call. It implicates repeated concurrent polling of the same event from separate processes.

    Different-event comparison

    I also tested two clients polling different events, topic_a and topic_b,
    from the same service instance.

    With score_communication v0.2.1, this configuration reproduced the bug:
    process A polling topic_a aborted through the transaction-log path, while
    process B polling topic_b continued polling but stopped receiving new
    samples.

    With score_communication v0.3.0, the same different-event test completed
    without the transaction-log SIGABRT. We ran three clean trials and one
    60-second trial with both receive counters continuing to advance. This
    indicates that the v0.2.1 cross-event crash is fixed in v0.3.0 in this
    reproducer.

    The complete reproducer and logs are attached for the same-topic subscribe-only, one-poll, continuous-poll, and v0.2.1 comparison runs.

    Could you please clarify whether concurrent polling from separate ProxyEvent instances in different processes on the same event is supported? If it is supported, could the v0.3.0 same-event SIGABRT be investigated as a shared-memory transaction or event-state defect? If it is not supported, which AoU explicitly forbids this cross-process access pattern?
11. crimson11 commented on Sep 1, 2026

    

    crimson11

    on Sep 1, 2026

    Contributor

    More actions

    @tfa2si
    Seems, that both of your consumer/proxy processes are using the very same mw_com_config.json ?
    And there you even did NOT configure a global section with an applicationID property being different for these two proxy/consumer apps?
    Then this error is totally expected. I guess you also would have then ignored the WARN messages in the log for not having configured a applicationID?
    Please read the tutorial: https://eclipse-score.github.io/communication/latest/score/mw/com/doc/tutorial/README.html ... explicitly this section: https://eclipse-score.github.io/communication/latest/score/mw/com/doc/tutorial/chapter_2/README.html#separate-configuration-files-for-provider-and-consumer
12. crimson11

    assigned

    tfa2si

    on Sep 1, 2026
13. tfa2si commented on Sep 1, 2026

    

    tfa2si

    on Sep 1, 2026

    Author

    More actions

    You are right with that, i did not use applicationID in mw_com_config.json, also asked here a dedicated possibility for applicationID configuration to have a common mw_com_config.json for all apllication
14. tfa2si

    closed this as completedon Sep 1, 2026
15. github-project-automation

    moved this from Backlog to Done in COM - Communication FTon Sep 1, 2026