Consumer-Tests for docs-as-code #66

🧱 Regression consumer Tests for score_docs_as_code Consumers

🎯 Goal

Ensure that changes to score_docs_as_code do not break core downstream consumers.
We want confidence that minor and patch releases are safe and that any breaking changes are deliberate, documented, and released appropriately.

✅ Current Situation

• score_docs_as_code is one of the most widely used modules in our ecosystem.
• We currently release new versions without automated validation against its main consumers.
• This creates the risk of introducing unintentional regressions, and undermines stability and compatibility guarantees.

🚀 Solution

Introduce regression checks that validate score_docs_as_code against its main consumers using a Python-based test runner executed in CI.

Approach:

• On PR or release, run a test script that:
  • Clones each major consumer into a temporary working directory.
  • Overrides their MODULE.bazel to use the current local version of score_docs_as_code via local_path_override.
  • Executes relevant documentation and test commands.
  • Collects and reports any errors encountered.

Expected outcome:

• Prevent silent breakage.
• Catch compatibility issues early.
• Allow known breaking changes to be flagged explicitly (e.g. in a major release).

🛠️ Concrete Tasks

• Implement script to:
  • Checkout local score_docs_as_code from the current PR branch.
  • Clone the following core consumers:
    • platform
    • process_description
    • module_template
    • (Add more as needed)
  • Override score_docs_as_code in each consumer's MODULE.bazel using local_path_override. Format MODULE.bazel to avoid any formatting-test-failures.
  • Run relevant commands (if they exist. More clever approach later.):
    • docs_incremental_latest
    • docs_incremental_release
    • docs_live_preview_latest
    • docs_live_preview_release
    • docs_docs_latest
    • docs_docs_release
  • Collect and summarize only the errors.
• Integrate the script into a GitHub Actions workflow.
• Document how to add additional consumers or override test behavior.

🔄 Considered Alternatives

Alternative: Manual downstream testing or relying on individual consumers to report issues.

Pros:

• No additional automation effort.

Cons:

• Missed regressions.
• Undermines perceived and actual stability.
• Weakens compatibility guarantees.

🧠 Notes

• Make sure to make it easy to add more modules
• Prioritize easy to read code

📚 References

• Pytest Documentation
• Platform Repo
• Process Repo
• Module Template