Release Process

We have four releases per year, usually at the end of each quarter: YY.03, YY.06, YY.09, YY.12.
There are two product flavors — Platform and Partner Preview Platform (PPP). These run as coexisting deployments in separate namespaces. PPP is usually released one week after Platform. PPP follows the same mechanics in the *-ppp namespace with its own images and snapshots.

Useful tools

Overview

A release consist of:

Ideally a week before release date, start up staging with the new RC images and data snapshots.
All Teams test staging, apply any fixes that are required there until everything is ready.
The day before release:
- Build final CH/OS images and tarballs.
- Push release to EBI's FTP.
- Push release to Google Cloud Storage open-targets-data-releases bucket.
- Push data to BigQuery dev and BigQuery prod.
On release day, switch staging and production.

Note

As staging is still up, if anything is wrong the rollback is instantaneous with helm rollback or switching back.
After a couple of days, bring down the old production.

This changes made editing profiles/production-platform.yaml. And applied with make deploy-chart-prod-platform.

Step-by-step

Tip

Check out the config file for production.
For this example, we assume blue is live and green is idle.

1. Bring up staging

Update deployment config

First, configure the idle colour with the upcoming release image tags and data snapshots.

In the green block, set the new webapp, api and aiapi images, the clickhouse / opensearch snapshots, and the release value.

profiles/production-platform.yaml
      opensearch:
        image: opensearchproject/opensearch:3.5.0
        snapshot: projects/open-targets-eu-dev/global/snapshots/platform-2606-os
        storageSize: 60Gi

    green:
      release: '26.06'
      webapp:
        image: europe-west1-docker.pkg.dev/open-targets-eu-dev/ot-ui-apps/ot-ui-apps:0.16.11
      api:
        image: europe-west1-docker.pkg.dev/open-targets-eu-dev/platform-api/platform-api:26.6.0
      aiapi:
        image: europe-west1-docker.pkg.dev/open-targets-eu-dev/ot-ai-api/ot-ai-api:0.0.13
        secrets:
          openai: projects/open-targets-prod/locations/europe-west1/secrets/openai-token
      clickhouse:
        image: clickhouse/clickhouse-server:25.8.18.1
        snapshot: projects/open-targets-eu-dev/global/snapshots/platform-2606-ch
        storageSize: 150Gi
      opensearch:
        image: opensearchproject/opensearch:3.5.0
        snapshot: projects/open-targets-eu-dev/global/snapshots/platform-2606-os
        storageSize: 60Gi

At the start of the file, Set staging: enabled so the staging deployment is started.

profiles/production-platform.yaml
    gcpProject: open-targets-prod
    prefix: production
    product: platform
    domain: opentargets.org
    dnsZone: opentargets-org

    production: blue
    staging: disabled

    googleTagManagerId: GTM-WPXWRDV

Note

During testing, -rc/-dev image tags and -rcN snapshots are expected here.

Deploy staging

make deploy-chart-prod-platform

Review the helm diff and confirm. You should see a lot of additions (green lines) as Helm brings up another whole copy of the infra. Green starts on the staging node pool (which scales up from 0) and is served at staging.platform.opentargets.org (plus api. / ai.).

2. Test

First checks you can do:

Use k9s to check that pods are starting properly. Type :namespaces, select production-platform and check the production-platform-green-* pods by scrolling to them and pressing d.

Confirm the API reports the expected release and version. For that, go to the Staging API GraphiQL and run the following query:

API Version query

    {
        meta {
            product
            dataPrefix
            apiVersion {
                x
                y
                z
            }
            dataVersion {
                year
                month
                iteration
            }
        }
    }

Head into the staging platform and check that things look as they should.

3. Day before release: data publishing

Build images and tarballs

Once all tests are confirmed to be passing and things look correct, build final versions of the services and generate final snapshots of the data.
Swap every -rc / -dev image tag and -rcN snapshot in the green block to its final value (see Final versions).
Make sure things are still as they should in staging.

[WIP]

Push to EBI's FTP

[WIP]

Push to GCP Bucket

[WIP]

Push to BigQuery

[WIP]

4. Release day: Promote staging to production

Set production: green.
Deploy: make deploy-chart-prod-platform. The prod Services now select green pods. Blue keeps running (now the staging colour) as a rollback.
Celebrate the release!

5. Decommission staging

Once the release is confirmed healthy and has been in production for a few days:

Set staging: disabled.
Deploy: make deploy-chart-prod-platform. Blue's workloads and their PVCs are removed and the staging node pool scales back to 0.

Rollback

While the old colour is still up (before step 5), rollback is immediate: set production back to the previous colour and deploy (or helm rollback). To rollback after having decommisioned the staging env, you must bring the old colour back up as staging first, then promote it.

Final versions

Staging runs release-candidate artifacts; production must run final ones. As the time to release approaches, appropriate tags should be created for ot-ui-apps, platform-api and ot-ai-api. Same with the ClickHouse and OpenSearch Google Cloud Snapshots.

Image tags: replace …-rc.N with the final release tag (e.g. 26.03.1-rc.4 → 26.03.1).
Snapshots: replace platform-2603-ch-rcN with the final platform-2603-ch (same for -os).

Warning

Final versions should be running in staging before release day!
Changing a ClickHouse or OpenSearch snapshot recreates its StatefulSet — the snapshot name is part of the StatefulSet name — and triggers a full reload from the new snapshot. It is not an in-place update, so be sure to do this before the actual release.

Checklists

Pre-release (One week before release)

ot-ui-apps tag present and has run its CI/CD; image tag exists in Artifact Registry.
platform-api tag present and has run its CI/CD; image tag exists in Artifact Registry.
ot-ai-api tag present and has run its CI/CD; image tag exists in Artifact Registry.
ClickHouse and OpenSearch release snapshots created in open-targets-eu-dev.
Idle colour configured and staging: enabled.

Data release (One day before release)

All -rc / -dev image tags swapped to final.
All -rcN snapshots swapped to final.
Images and tarballs built, present in release bucket.
Data pushed to EBI FTP.
Checksums present in the release bucket.
Push to open-targets-data-releases bucket.
Push to BigQuery dev and BigQuery prod.

Release day

production flipped to the new colour.

Post-release

Create the release branch/tag in ot-snapshot, pinning the final component versions.
Staging decommissioned (staging: disabled) after the grace period.
Production re-verified after teardown of staging.

Troubleshooting

Tip

We should write down usual problems and the solution here.