Skip to content

[FR] - Log reason when node switches tip to a fork #6014

Closed

Description

@TerminadaPool

Internal/External
External: Stake pool operator

Area
Other: Logging

Describe the feature you'd like
Please log the reason why the node preferred another fork

Describe alternatives you've considered
The independent tool cncli provides logging of block data including the block VRF value into a sqlite database. This database can be queried for insight into whether the likely reason the node preferred another fork was because the other fork terminal block had a lower block VRF and the tie break rule was applied.

Additional context / screenshots
The node knows the reason for switching tip to a competing fork but this reason isn't logged. Rather the logs only confirm that the tip was switched. But, was the reason due to the competing fork being longer? Or was it due to the other fork's terminal block having a lower VRF causing the tie-break rule to be applied? Or did the node prefer another fork some other reason?

This information would be really helpful when trying to identify what caused your own block to not be adopted.

Activity

  1. github-actions commented on Nov 15, 2024

    @github-actions

    This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

  2. TerminadaPool commented on Nov 15, 2024

    @TerminadaPool
    Author

    Please stay as an open feature request.

  3. rphair commented on Nov 15, 2024

    @rphair

    Yes, please: if this is considered impractical to add I would like to hear why. Our SPO operation believes this is important for quality assurance purposes.

  4. github-actions commented on Dec 16, 2024

    @github-actions

    This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

  5. rphair commented on Dec 16, 2024

    @rphair

    @carbolymer what kind of engagement do the developers expect for this issue not to be Stale? There won't be anything new to report until someone in the relevant team finally responds.

    This log info has been long & vitally requested for diagnostic purposes and the need for it is never going to go away. Those who need it most would be small stake pools with infrequent blocks... and therefore needed more in the near future if & when the K PoS parameter increases (as is constantly suggested in governance conversations).

    If this issue keeps being dismissed, the implication is that QA isn't very important for stake pools and/or chain density isn't that important for Cardano itself. I don't think either of these things are true and I can state from SPO experience dating back to the beginning of the Shelley period that implementing this feature (which we arguably should have had from day 1) would be helpful for SPOs and node infrastructure operators at all levels of endeavour.

  6. carbolymer commented on Dec 27, 2024

    @carbolymer
    Contributor

    There was a slack discussion about this issue some time ago.
    @jasagredo @amesgen would you mind sharing main takeaways here?

  7. amesgen commented on Jan 6, 2025

    @amesgen
    Contributor

    The trace messages already contain enough detail to deduce the reason why a node changes its chain. Concretely, whenever we change our selection, we trace the so called "SelectView" of the tip of both the old and the new chain. This SelectView contains exactly the information that is needed in order to determine whether a chain is preferable to another chain. See PraosChainSelectView for more details and assorted documentation.

    In Conway, the chain order works like this, via a lexicographical combinationi of the following criteria, when deciding whether to switch from the current chain to a candidate chain:

    1. compare chain length, preferring larger values
    2. when (same slot and same issuer) compare operational certificate issue number, preferring larger values
    3. when (slots differ by at most 5) compare VRF, preferring smaller values

    By comparing the old and new SelectView, one can see which of these three rules was responsible for the chain switch:


    Let's look at an example:

    {
      "at": "2024-12-13T13:50:58.295305594Z",
      "ns": "ChainDB.AddBlockEvent.AddedToCurrentChain",
      "data": {
        "kind": "AddedToCurrentChain",
        "newTipSelectView": {
          "chainLength": 11214658,
          "issueNo": 16,
          "issuerHash": "8e0bb9b126acd8e65a023602377d9a7a6f15af394d1f7bdd852182b8",
          "kind": "PraosChainSelectView",
          "slotNo": 142531567,
          "tieBreakVRF": "f8f31aabb555124392485cbd7ba02bded4e4eac8980fc36fde0c9a21d7cf2d51c6fd4ddda8113153639db862781dafacbdd77da4e072452a330c32509506762e"
        },
        "newtip": "17d5b0f21e86503b41851640e9558610ebe9b7a62c96a08873eb4d0c7961a460@142531567",
        "oldTipSelectView": {
          "chainLength": 11214657,
          "issueNo": 20,
          "issuerHash": "c63dab6d780a74cbae2a27696c9723f55b3092b2bd001256df03827f",
          "kind": "PraosChainSelectView",
          "slotNo": 142531554,
          "tieBreakVRF": "905ff37db665e86d60d93dabc92ec86124b1a22165e4968facca226e790d9b7dd3f60502fa84dcd30d95ac28a9278e5e72b8b2eb451759ab59c92fc65909a2b4"
        }
      },
      "sev": "Notice",
      "thread": "76",
      "host": "ramify"
    }

    Here, we can see that the chainLength of newTipSelectView is larger than the one of the chainLength of the oldTipSelectView, so the chain switch happened due to rule 1.

    If these would be equal, one would have to look at rule 2 next (and then rule 3) to see whether it is responsible.


    For convenience/simplicity, we could further enrich the trace message to directly include eg a reason field that says something like "LongerChain" or "BetterTieBreaker".

    cc @mgmeier as you might have already started with this.

  8. TerminadaPool commented on Jan 6, 2025

    @TerminadaPool
    Author

    That sort of log information would absolutely suit my needs. I wasn't aware we could get the "tieBreakVRF" values logged ordinarily.

    What do I need to configure in my node config.json file so that such information is logged in my systemd journal?

  9. amesgen commented on Jan 7, 2025

    @amesgen
    Contributor

    In both the old and new tracing system, you should see this when you enable JSON output and use a severity of at least Notice for ChainDB.AddBlockEvent.AddedToCurrentChain/ChainDB.AddBlockEvent.SwitchedToAFork.

  10. 51 remaining items

  11. rphair commented on Jan 24, 2026

    @rphair

    @mgmeier @jutaro how can this issue be automatically be marked Stale when it's been assigned to somebody & that label has been deliberately removed? How can it be that the CI here really puts such an issue on track for automatic closure? 🤔

  12. github-actions commented on Mar 9, 2026

    @github-actions

    This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

  13. rphair commented on Mar 9, 2026

    @rphair

    🤖

  14. github-actions commented on Apr 13, 2026

    @github-actions

    This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

  15. jasagredo commented on Apr 13, 2026

    @jasagredo
    Contributor

    This feature was implemented in Consensus in IntersectMBO/ouroboros-consensus#1826 and in Cardano-node in #6469.

    Once we have confirmation that the new information is being logged correctly, we can finally close this issue.

    @mgmeier can I ask you to confirm this is the case?

  16. rphair commented on Apr 14, 2026

    @rphair

    thanks @jasagredo @mgmeier — along with this confirmation, please (for the sake of the SPO community) post a snippet of the log message that we should be looking for... in the new tracing system as well as the old tracing system if applicable. 🙏

  17. mgmeier commented on Apr 14, 2026

    @mgmeier
    Contributor

    The old tracing system does not support this new message field.

    In the new tracing system, it's logged under the ChainDB.AddBlockEvent.SwitchedToAFork namespace.
    The message is of severity Notice, so it's only filtered out if you set your config to minimum severity Warning or higher.

  18. mgmeier commented on Apr 14, 2026

    @mgmeier
    Contributor

    This is an example text output:

    [2026-04-14 18:57:00.6869Z][0979accfcade:ChainDB.AddBlockEvent.SwitchedToAFork](Notice,25) Switched to a fork, new tip: 6ca9f1fc17c029a120f662531f1dd93148cdddd35d4c386897fd37be754339db at slot 942
    Reason: candidate is longer (At (BlockNo 29)) than our selection (At (BlockNo 28))
    
  19. mgmeier commented on Apr 14, 2026

    @mgmeier
    Contributor

    This is an example JSON output (I linted it, it's actually a one-liner):

    {
    	"at": "2026-04-14T20:28:13.589952121Z",
    	"ns": "ChainDB.AddBlockEvent.SwitchedToAFork",
    	"data": {
    		"kind": "TraceAddBlockEvent.SwitchedToAFork",
    		"newSuffixSelectView": {
    			"blockNo": 23,
    			"issueNo": 0,
    			"issuerHash": "87569d82e26ad45fb249a162ff5a71f71808746286ae2ecad3ccd34c",
    			"kind": "PraosTiebreakerView",
    			"slotNo": 705,
    			"tieBreakVRF": "6143ed8f0f398121e66cddf0b7e7f1e30daa94492261ba2709d1c286c02edcb2b2a07f994ce9b58235483a667ccc236a82039a286c07a2f97705f5010f04ff10",
    			"weightBoost": 0
    		},
    		"newtip": "6af05875b569a6c3510dae170b1c31507795ce76e75b94e83e32c6026392bb3d@705",
    		"oldSuffixSelectView": {
    			"blockNo": 22,
    			"issueNo": 0,
    			"issuerHash": "842e4a3eb479cec07a5df650f913d6931485f9e466f21247d3cb1fd3",
    			"kind": "PraosTiebreakerView",
    			"slotNo": 902,
    			"tieBreakVRF": "f5198cc0b901b80735fea02da3accc292a14371ae2c0f7c6670161dc3c025b9f78eceeb71e6e2067d2274749733aad6e9e7a6eb024925eda96bddd7ae96884f3",
    			"weightBoost": 0
    		},
    		"reason": {
    			"candidate": "At (BlockNo 23)",
    			"our": "At (BlockNo 22)",
    			"reason": "Longer"
    		}
    	},
    	"sev": "Notice",
    	"thread": "25",
    	"host": "10d0916dc292"
    }
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Staletype: enhancementAn improvement on the existing functionality

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions