Purpose

This issue tracks the arcade-services repository rollout. On top of the Rollout instructions described on the wiki, it provides the person responsible for the rollout checklist of the steps that need to be performed to rollout services in this repository. All relevant information, including the rollout PR, issues encountered during the rollout and steps taken to resolve them should be linked or added to this issue to keep full audit trail of changes rolled out to production.

Process

Build status check (Monday)

• Check the status of the dotnet-arcade-services-weekly pipeline
• Rotate any secrets that need manual rotation
• In case the build is failing, try to fix it and ensure the arcade-services-internal-ci pipeline is green
• Check the Rollout column in the Product Construction board - move any issues rolled-out last week into Done

Rollout preparation (Tuesday)

• Check the vendor prepared the rollout:
  • Thread on the Rollout channel
  • Rollout issue in AzDO
  • Rollout PR in arcade-services
• In case there is a problem with the CI build, notify the Rollout channel
• Link the rollout PR to the Rollout PRs section of this issue
• Double-check that the release notes contain all information
• Merge the already prepared rollout PR
• Link the rollout build to the Rollout build section of this issue
• Verify that Maestro opened a production => main PR in arcade-services with the rollout merge commit (example). The commit in question should be empty. Do not merge the PR yet
• Ensure the build is green and stops at the "Approval" phase

Rollout day (Wednesday)

• Approve the Approval stage of the rollout build (that has been already started the day before)
• Monitor the rollout build for failures.
  • Note: this Maestro exceptions query might help in diagnosing issues.
• Keep track of any issues encountered during the rollout either directly in this issue, or in a dedicated issue linked to this issue
• Update the rollout stats in the Stats section below. The statistics will be available in Kusto a few minutes after the build was finished
• Notify the Rollout channel
• Merge the production => main Maestro PR in arcade-services
• Move rolled-out issues in the Rollout column of the Product Construction board into Done. Add a link in to this rollout issue in the comments before closing them (example)
• Close this issue with closing comment describing a high-level summary of issues encountered during the rollout

Rollout data

Rollout PRs

• The main PR: [Rollout] Production rollout, 2023-08-09 #2812

Rollout build

• Rollout AzDO build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2238280&view=results

Rollout times

Use the following Kusto query to gather data about rollout times:

• Pre-Approval run time: 00:35
• Post-Approval run time: 01:11

Useful links

• AzDO pipelines
  • arcade-services-internal-ci
  • dotnet-arcade-services-weekly
• Rollout channel
• Maestro exceptions
• Deployment Policy