Release Process

• We have four releases per year, usually at the end of each quarter: YY.03, YY.06, YY.09, YY.12.
• There are two product flavors — Platform and Partner Preview Platform (PPP). These run as coexisting deployments in separate namespaces. PPP is usually released one week after Platform. PPP follows the same mechanics in the *-ppp namespace with its own images and snapshots.

Useful tools

• GCP CLI
• Terraform
• Helm, Helm Diff Plugin
• k9s

Overview

A release consist of:

1. Ideally a week before release date, start up staging with the new RC images and data snapshots.
2. Apply any fixes that are required there until everything is ready.

3. On release day, switch staging and production.

   Note

   As staging is still up, if anything is wrong the rollback is instantaneous with helm rollback or switching back.

4. After a couple of days, bring down the old production.

This changes made editing profiles/production-platform.yaml. And applied with make deploy-chart-prod-platform.

Step-by-step

1. Update config

First, configure the idle colour with the upcoming release image tags and data snapshots.

• In the green block, set the new webapp, api and aiapi images, the clickhouse / opensearch snapshots, and the release value.

  profiles/production-platform.yaml
  26

• At the start of the file, Set staging: enabled so the staging deployment is started.

  profiles/production-platform.yaml
  1 2 3 4 5 6 7 8 9 10	gcpProject: open-targets-prod prefix: production product: platform domain: opentargets.org dnsZone: opentargets-org production: blue staging: enabled googleTagManagerId: GTM-WPXWRDV

2. Bring up staging

make deploy-chart-prod-platform

Review the helm diff and confirm. You should see a lot of additions (green lines) as Helm brings up another whole copy of the infra. Green starts on the staging node pool (which scales up from 0) and is served at staging.platform.opentargets.org (plus api. / ai.).

3. Test

First checks you can do:

• Use k9s to check that pods are starting properly. Type :namespaces, select production-platform and check the production-platform-green-* pods by scrolling to them and pressing d.

• Confirm the API reports the expected release and version. For that, go to the Staging API GraphiQL and run the following query:

  API Version query

      {
          meta {
              product
              dataPrefix
              apiVersion {
                  x
                  y
                  z
              }
              dataVersion {
                  year
                  month
                  iteration
              }
          }
      }
  

• Head into the staging platform and check that things look as they should.

4. Promote to production

Before release day:

• Once all tests are confirmed to be passing and things look correct, build final versions of the services and generate final snapshots of the data.
• Swap every -rc / -dev image tag and -rcN snapshot in the green block to its final value (see Final versions).
• Make sure things are still as they should in staging.

On release day:

• Set production: green.
• Re-apply: make deploy-chart-prod-platform. The prod Services now select green pods. Blue keeps running (now the staging colour) as a rollback.

5. Decommission staging

Once the release is confirmed healthy and has been in production for a few days:

• Set staging: disabled.
• Re-apply: make deploy-chart-prod-platform. Blue's workloads and their PVCs are removed and the staging node pool scales back to 0.

Rollback

While the old colour is still up (before step 5), rollback is immediate: set production back to the previous colour and re-apply. After decommission, you must bring the old colour back up as staging first, then promote it.

Final versions

Staging runs release-candidate artifacts; production must run final ones. As the time to release approaches, appropriate tags should be created for ot-ui-apps, platform-api and ot-ai-api. Same with the ClickHouse and OpenSearch Google Cloud Snapshots.

• Image tags: replace …-rc.N with the final release tag (e.g. 26.03.1-rc.4 → 26.03.1).
• Snapshots: replace platform-2603-ch-rcN with the final platform-2603-ch (same for -os).

Checklists

Pre-release

• ot-ui-apps tag present and has run its CI/CD; image tag exists in Artifact Registry.
• platform-api tag present and has run its CI/CD; image tag exists in Artifact Registry.
• ot-ai-api tag present and has run its CI/CD; image tag exists in Artifact Registry.
• ClickHouse and OpenSearch release snapshots created in open-targets-eu-dev.
• Idle colour configured and staging: enabled.

Promotion

• All -rc / -dev image tags swapped to final.
• All -rcN snapshots swapped to final.
• production flipped to the new colour.

Post-release

• Create the release branch/tag in ot-snapshot, pinning the final component versions.
• Staging decommissioned (staging: disabled) after the grace period.
• Production re-verified after teardown of staging.

Troubleshooting