CNTRLPLANE-3434

HO Image Override Verification Report

Post-merge verification that the Konflux Snapshot image is correctly applied across all Prow jobs in the ARO HCP release promotion pipeline

Context

Background

CNTRLPLANE-3434 (ARO HCP Release Promotion Pipeline) introduces a Konflux-based nightly pipeline that gates HyperShift Operator image promotion on e2e test results. A key requirement is that the Prow jobs triggered by the pipeline run against the specific HO image extracted from the Konflux Snapshot, not the default CI image.

The Override Problem

The initial approach used ci-operator's built-in OVERRIDE_IMAGE_HYPERSHIFT_OPERATOR mechanism to inject the Snapshot image into the Prow jobs via Gangway. However, this mechanism is affected by a race condition in ci-operator: when the override env var is set, ci-operator creates two concurrent steps that both write to the same pipeline:hypershift-operator ImageStream tag:

  1. The override step, which tags the Konflux snapshot image
  2. The base_images step, which tags hypershift/hypershift-operator:latest (the default)

Both steps launch as concurrent goroutines with no ordering guarantee. The override step is typically faster (it has the pullspec ready), so it writes first. But the base_images step writes second, and since writing an ImageStreamTag to an existing tag is effectively an upsert, it overwrites the override.

Misleading logs: The ci-operator log shows Overriding parameter LOCAL_IMAGE_HYPERSHIFT_OPERATOR and the correct tagging lines, which makes it look like the override is working. But the imagestreams.json artifact in build-resources reveals the final resolved digest does not match the override. The override had actually never worked in any pipeline run. This was only discovered when the imagestreams.json artifact was checked directly.
The Fix

Since ci-operator's OVERRIDE_IMAGE_* mechanism is inherently racy and cannot be fixed without an upstream change, we bypassed it entirely with two PRs on openshift/release:

  • PR #81877 openshift/release
    Added an OVERRIDE_HYPERSHIFT_OPERATOR_IMAGE parameter to the hypershift-install step registry ref, settable via Gangway's MULTISTAGE_PARAM_OVERRIDE_ mechanism. The script reads this variable directly and overrides OPERATOR_IMAGE at the shell level, completely bypassing ci-operator's ImageStream-based override.
  • PR #82061 openshift/release
    Fixed a missing pull secret that prevented extract_hcp_cli from pulling the override image (Konflux images on quay.io/redhat-user-workloads require the CI pull secret).

The pipeline itself was merged as PR #8602 openshift/hypershift

Pipeline Run Under Verification
PipelineRunho-release-gate-run-2t6cl
Snapshothypershift-operator-20260721-172425-000
Date2026-07-21 (nightly)
Expected Imagequay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-operator/hypershift-operator-main@sha256:6d4d6385209209852dd75031d730c457f1a6c0c9ad1bbe60b2bde12debe2c8d3
Expected Commitaa7db2343a5e5b4f26058ea99caba7e6a6fc9db3
Verification Approach: The override is verified at two independent layers. Layer 1 checks the hypershift-install step build-log to confirm the override parameter was detected, the correct image was used, and the CLI binary was extracted. Layer 2 checks the actual operator Deployment YAML dumped as a test artifact to confirm the running HO pod uses the override image and the CLI annotation matches the expected commit. Together, these two layers prove the override was applied (Layer 1) and that it resulted in the correct image actually running in the cluster (Layer 2).

Layer 1: Install Log

Verifies the override was applied during the hypershift-install step.

Criteria

For each Prow job triggered via Gangway, the hypershift-install step build-log should show:

  1. WARNING: Overriding OPERATOR_IMAGE - the override branch was taken
  2. Override: line containing the expected digest sha256:6d4d63...e2c8d3
  3. extract_hcp_cli invoked with the override image - both operator image and CLI binary were overridden
Results
#JobOverrideCorrect ImageCLI ExtractedLog
15.0/e2e-aks YESYESYES build-log
25.0/hcm-azure-e2e-aks-upgrade-minor YESYESYES build-log
35.0/e2e-aks-multi-x-ax N/AN/AN/A Job failed during aks-provision
44.22/e2e-aks-multi-x-ax N/AN/AN/A Job failed during aks-provision
54.21/e2e-aks-multi-x-ax YESYESYES build-log
64.20/e2e-aks-multi-x-ax YESYESYES build-log
74.22/hcm-azure-e2e-aks-upgrade-from-zero YESYESYES build-log
84.21/hcm-azure-e2e-aks-upgrade-from-zero YESYESYES build-log
94.20/hcm-azure-e2e-aks-upgrade-from-zero YESYESYES build-log
105.0/e2e-azure-aks-ovn-conformance YESYESYES build-log

Layer 2: Deployment YAML

Verifies the running HO pod uses the override image.

Criteria

For each job that reached HO installation, the operator Deployment YAML dumped as a test artifact should show:

  1. Pod Image - the container image: field matches the expected digest
  2. CLI Commit - the hypershift.openshift.io/install-cli-version annotation contains commit aa7db2343a5e5b4f26058ea99caba7e6a6fc9db3
Results
#JobPod ImageCLI CommitDeployment YAML
15.0/e2e-aks YESYES operator.yaml
25.0/hcm-azure-e2e-aks-upgrade-minor YESYES operator.yaml
35.0/e2e-aks-multi-x-ax N/AN/A Job failed during aks-provision
44.22/e2e-aks-multi-x-ax N/AN/A Job failed during aks-provision
54.21/e2e-aks-multi-x-ax YESYES operator.yaml
64.20/e2e-aks-multi-x-ax YESYES operator.yaml
74.22/hcm-azure-e2e-aks-upgrade-from-zero YESYES operator.yaml
84.21/hcm-azure-e2e-aks-upgrade-from-zero YESYES operator.yaml
94.20/hcm-azure-e2e-aks-upgrade-from-zero YESYES operator.yaml
105.0/e2e-azure-aks-ovn-conformance YESYES operator.yaml
Note on conformance (#10): The conformance job stores the deployment YAML at a different path compared to other jobs. Other jobs store it under hypershift-azure-run-e2e/artifacts/<TestName>/namespaces/hypershift/apps/deployments/operator.yaml, while the conformance job stores it under dump/artifacts/namespaces/hypershift/apps/deployments/operator.yaml (management cluster must-gather format). Both paths contain the same operator Deployment resource.

Unverifiable Jobs

Jobs that failed before HO installation could be attempted.

Jobs 3 and 4 failed during the aks-provision step, before hypershift-install ever ran. The override could not be verified at either layer because HO was never installed.

JobErrorLog
Job 3 (5.0/e2e-aks-multi-x-ax) ERROR: Timed out waiting for NAP nodes. Only 6/9 ready. aks-provision log
Job 4 (4.22/e2e-aks-multi-x-ax) ERROR: Timed out waiting for NAP nodes. Only 3/9 ready. aks-provision log

Both are Azure infrastructure failures (NAP node provisioning timeout), unrelated to the override mechanism. The hypershift-install step is downstream of aks-provision in the workflow DAG, so it was never reached.

Summary

LayerVerifiedN/A (infra failure)
Layer 1 - Install Log8/102/10
Layer 2 - Deployment YAML8/82/10
Conclusion

The image override mechanism works correctly end-to-end, verified at two independent layers:

  • Layer 1 (install log): The hypershift-install step correctly detects the override parameter, logs the switch from default to override image, and extracts the CLI binary from the override image.
  • Layer 2 (running pod): The actual HO Deployment running in the hypershift namespace uses the override image digest as its container image, and the install-cli-version annotation confirms the CLI was built from the correct commit (aa7db2343a).

Both layers were verified across 5 different job types (e2e-aks, upgrade-minor, multi-x-ax, upgrade-from-zero, conformance) and 4 release branches (5.0, 4.22, 4.21, 4.20). All 8 jobs that reached HO installation passed both checks.

Verification Verdict

PASSED. The HO image override mechanism works correctly. All 8 jobs that reached HO installation used the correct Konflux Snapshot image, verified at both the install log and running pod levels. The 2 unverifiable jobs failed due to Azure infrastructure issues before the override could be applied.
Pipeline gate outcome (for context): The pipeline gate verdict for this run was false (3 blocking jobs failed: 5.0/e2e-aks, 5.0/e2e-aks-multi-x-ax, 4.22/e2e-aks-multi-x-ax), so the image was not promoted. This is unrelated to the override mechanism: the failures were either test failures (e2e-aks) or infrastructure failures (the two multi-x-ax jobs timed out during aks-provision). The override itself worked as intended in all cases where it could be applied.

Generated 2026-07-22 for CNTRLPLANE-3434 - Alessandro Rossi