HO Image Override Verification Report
Post-merge verification that the Konflux Snapshot image is correctly applied across all Prow jobs in the ARO HCP release promotion pipeline
CNTRLPLANE-3434 (ARO HCP Release Promotion Pipeline) introduces a Konflux-based nightly pipeline that gates HyperShift Operator image promotion on e2e test results. A key requirement is that the Prow jobs triggered by the pipeline run against the specific HO image extracted from the Konflux Snapshot, not the default CI image.
The initial approach used ci-operator's built-in OVERRIDE_IMAGE_HYPERSHIFT_OPERATOR mechanism to inject the Snapshot image into the Prow jobs via Gangway. However, this mechanism is affected by a race condition in ci-operator: when the override env var is set, ci-operator creates two concurrent steps that both write to the same pipeline:hypershift-operator ImageStream tag:
Both steps launch as concurrent goroutines with no ordering guarantee. The override step is typically faster (it has the pullspec ready), so it writes first. But the base_images step writes second, and since writing an ImageStreamTag to an existing tag is effectively an upsert, it overwrites the override.
Overriding parameter LOCAL_IMAGE_HYPERSHIFT_OPERATOR and the correct tagging lines, which makes it look like the override is working. But the imagestreams.json artifact in build-resources reveals the final resolved digest does not match the override. The override had actually never worked in any pipeline run. This was only discovered when the imagestreams.json artifact was checked directly.
Since ci-operator's OVERRIDE_IMAGE_* mechanism is inherently racy and cannot be fixed without an upstream change, we bypassed it entirely with two PRs on openshift/release:
The pipeline itself was merged as PR #8602 openshift/hypershift
| PipelineRun | ho-release-gate-run-2t6cl |
| Snapshot | hypershift-operator-20260721-172425-000 |
| Date | 2026-07-21 (nightly) |
| Expected Image | quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-operator/hypershift-operator-main@sha256:6d4d6385209209852dd75031d730c457f1a6c0c9ad1bbe60b2bde12debe2c8d3 |
| Expected Commit | aa7db2343a5e5b4f26058ea99caba7e6a6fc9db3 |
Verifies the override was applied during the hypershift-install step.
For each Prow job triggered via Gangway, the hypershift-install step build-log should show:
WARNING: Overriding OPERATOR_IMAGE - the override branch was takenOverride: line containing the expected digest sha256:6d4d63...e2c8d3extract_hcp_cli invoked with the override image - both operator image and CLI binary were overridden| # | Job | Override | Correct Image | CLI Extracted | Log |
|---|---|---|---|---|---|
| 1 | 5.0/e2e-aks | YES | YES | YES | build-log |
| 2 | 5.0/hcm-azure-e2e-aks-upgrade-minor | YES | YES | YES | build-log |
| 3 | 5.0/e2e-aks-multi-x-ax | N/A | N/A | N/A | Job failed during aks-provision |
| 4 | 4.22/e2e-aks-multi-x-ax | N/A | N/A | N/A | Job failed during aks-provision |
| 5 | 4.21/e2e-aks-multi-x-ax | YES | YES | YES | build-log |
| 6 | 4.20/e2e-aks-multi-x-ax | YES | YES | YES | build-log |
| 7 | 4.22/hcm-azure-e2e-aks-upgrade-from-zero | YES | YES | YES | build-log |
| 8 | 4.21/hcm-azure-e2e-aks-upgrade-from-zero | YES | YES | YES | build-log |
| 9 | 4.20/hcm-azure-e2e-aks-upgrade-from-zero | YES | YES | YES | build-log |
| 10 | 5.0/e2e-azure-aks-ovn-conformance | YES | YES | YES | build-log |
Verifies the running HO pod uses the override image.
For each job that reached HO installation, the operator Deployment YAML dumped as a test artifact should show:
image: field matches the expected digesthypershift.openshift.io/install-cli-version annotation contains commit aa7db2343a5e5b4f26058ea99caba7e6a6fc9db3| # | Job | Pod Image | CLI Commit | Deployment YAML |
|---|---|---|---|---|
| 1 | 5.0/e2e-aks | YES | YES | operator.yaml |
| 2 | 5.0/hcm-azure-e2e-aks-upgrade-minor | YES | YES | operator.yaml |
| 3 | 5.0/e2e-aks-multi-x-ax | N/A | N/A | Job failed during aks-provision |
| 4 | 4.22/e2e-aks-multi-x-ax | N/A | N/A | Job failed during aks-provision |
| 5 | 4.21/e2e-aks-multi-x-ax | YES | YES | operator.yaml |
| 6 | 4.20/e2e-aks-multi-x-ax | YES | YES | operator.yaml |
| 7 | 4.22/hcm-azure-e2e-aks-upgrade-from-zero | YES | YES | operator.yaml |
| 8 | 4.21/hcm-azure-e2e-aks-upgrade-from-zero | YES | YES | operator.yaml |
| 9 | 4.20/hcm-azure-e2e-aks-upgrade-from-zero | YES | YES | operator.yaml |
| 10 | 5.0/e2e-azure-aks-ovn-conformance | YES | YES | operator.yaml |
Jobs that failed before HO installation could be attempted.
Jobs 3 and 4 failed during the aks-provision step, before hypershift-install ever ran. The override could not be verified at either layer because HO was never installed.
| Job | Error | Log |
|---|---|---|
| Job 3 (5.0/e2e-aks-multi-x-ax) | ERROR: Timed out waiting for NAP nodes. Only 6/9 ready. |
aks-provision log |
| Job 4 (4.22/e2e-aks-multi-x-ax) | ERROR: Timed out waiting for NAP nodes. Only 3/9 ready. |
aks-provision log |
Both are Azure infrastructure failures (NAP node provisioning timeout), unrelated to the override mechanism. The hypershift-install step is downstream of aks-provision in the workflow DAG, so it was never reached.
| Layer | Verified | N/A (infra failure) |
|---|---|---|
| Layer 1 - Install Log | 8/10 | 2/10 |
| Layer 2 - Deployment YAML | 8/8 | 2/10 |
The image override mechanism works correctly end-to-end, verified at two independent layers:
install-cli-version annotation confirms the CLI was built from the correct commit (aa7db2343a).Both layers were verified across 5 different job types (e2e-aks, upgrade-minor, multi-x-ax, upgrade-from-zero, conformance) and 4 release branches (5.0, 4.22, 4.21, 4.20). All 8 jobs that reached HO installation passed both checks.
false (3 blocking jobs failed: 5.0/e2e-aks, 5.0/e2e-aks-multi-x-ax, 4.22/e2e-aks-multi-x-ax), so the image was not promoted. This is unrelated to the override mechanism: the failures were either test failures (e2e-aks) or infrastructure failures (the two multi-x-ax jobs timed out during aks-provision). The override itself worked as intended in all cases where it could be applied.
Generated 2026-07-22 for CNTRLPLANE-3434 - Alessandro Rossi