Node API Gateway Deployment Plan

The Deployment Plan

1. A member of Terminus is required to review all PRs before merging into the Node API Gateway.
   
   
2. Terminus will plan to do regular deployments on Tuesday and Thursday mornings in order to keep the releases as small as possible. We’ve noticed when the release has too many PRs we are at a much higher risk for introducing problems.
   • If you need your changes deployed into Development/Staging quicker than our normal cadence, you can kick off a build by clicking on the “Build Now” button in the left panel of the Node API Gateway release pipeline. This will create a new version with your changes that you can watch get deployed in the general Node API Gateway pipeline.
   • Terminus may attempt to isolate PRs into their own release when the PRs are:
     • High risk
     • Hard to verify in a canary.
     • Need very specific deployment timing.
       
       
3. Terminus’s firefighter will act as a “release manager” throughout the process.
   • If the firefighter is unavailable, for instance, because they are in an incident or otherwise actively working a customer issue, they have the ability to delay the release until any issues have been remedied.
     
     
4. Every release of the Node API Gateway will be canaried, at no more than 5% to start.
   1. This will be initiated via a post on #org-nag-terminus, tagging each individual who has a PR that is in the build (including the release manager, if they have a PR). The canary process won’t start until each PR has been accounted for.
      • Example: Is <service name> version <version> ready to be canaried in production? @contributors
      • If an individual with a PR is unavailable, the release manager may at their discretion:
        • Contact their team and request someone to fill in for them.
        • Take responsibility for verifying the changes themselves.
        • Revert the changes and create another build.
   2. Release manager will begin the canary process.
      • If a significant amount of time has passed since requesting permission to start the canary, the release manager may ask contributors to reapprove starting the canary.
        • Example: Is everyone ready for the canary to start? :thumbsup: to approve. @contributors
   3. The release manager will then ask for a second check from the same individuals after they have verified their canaried code (additional checks may be requested by the release manager at a higher canary percentage before moving to production).
      • Example: The canary is running at <canary percentage>%. Indicate you’re verifying your changes with :eyes:. Verify approval for full production with :thumbsup:. @contributors
      • The canary percentage will not be >20% as there are potential resource issues otherwise.
      • If a change is causing issues, the release manager can terminate the canary. At that point the release manager can:
        • Revert the changes and create another build.
        • Wait for the contributor to resolve the issue.
          
          
5. Once all involved contributors have verified their code, the release manager will proceed with a full production release.
   • Each contributor will also need to allocate time to check their code in production and watch it for an appropriate amount of time, in order to ensure there are no issues.