CNTRLPLANE-3790

CNTRLPLANE-3790 tracks the expansion of the ARO HCP release gating pipeline's blocking test suite. The current pipeline (ITS source) has 9 blocking tests and 1 informing test.

The issue requires two separate investigations, each covered by its own report:

1. Analysis 1 of 2: Should e2e-azure-aks-ovn-conformance be promoted from informing to blocking?
2. This report (2 of 2): Why does e2e-azure-aks-external-oidc not exist for 4.22/5.0, and should it be added?

Current blocking tests (9):

• e2e-aks (5.0)
• hcm-azure-e2e-aks-upgrade-minor (5.0)
• e2e-aks-multi-x-ax (5.0, 4.22, 4.21, 4.20)
• hcm-azure-e2e-aks-upgrade-from-zero (4.22, 4.21, 4.20)

Current informing test (1):

• e2e-azure-aks-ovn-conformance (5.0) - see conformance test report

The investigation must determine:

1. Is the External OIDC feature GA in 4.22+? If so, the non-techpreview test should be re-added.
2. Is the techpreview variant still needed, or has the feature graduated?
3. Should either variant be a candidate for blocking in the promotion pipeline?

Test variants:

The External OIDC feature allows customers to bring their own OIDC provider for direct API server access, replacing OpenShift's built-in OAuth. The feature has a clear Jira lineage:

Verified in the HyperShift codebase (control-plane-operator/featuregates/featuregates.go):

Bryce Palmer confirmed in a CNTRLPLANE-946 comment (2025-11-17) that the ExternalOIDC and ExternalOIDCWithUIDAndExtraClaimMappings feature gates were promoted (graduated to GA) in both 4.21 and 4.20.z.

The config-brancher tool, used to propagate CI configurations from one release branch to the next, runs with the --skip-periodics flag. This means periodic jobs are never automatically propagated to new release branches - they must be manually re-added after each branching.

The test was created for 4.20/4.21 by PR #68741 (XiuJuan Wang, Dec 2025). When the 4.22 and 5.0 branches were created, the periodic jobs were not carried forward, and no follow-up issue was created to re-add them.

QE team issue audit

We audited all related Jira issues under the CNTRLPLANE-883 epic to determine whether any issue had the propagation in scope:

Remaining "time debt" issues (still In Progress)

The External OIDC periodic suite is split across two cloud providers, each using a different OIDC provider setup and a different workflow. Both run the same Go test (TestExternalOIDC) but under different authentication configurations.

Full audit of External OIDC periodic job presence across all active releases. Source: ci-operator/config/openshift/hypershift/openshift-hypershift-release-{version}__periodics.yaml in openshift/release.

Presubmit configuration for the Azure AKS External OIDC test (from openshift-hypershift-release-{version}.yaml):

Figure 1: Sippy Job Health for 4.20, filtered by e2e-azure-aks-external-oidc. Source: Sippy Job Health

Figure 2: Sippy Job Analysis for 4.20. 13 runs, 13 successful. Source: Sippy Job Analysis

Figure 3: Sippy Job Health for 4.21, filtered by e2e-azure-aks-external-oidc. Non-techpreview: 100% (6 runs current, 7 previous). Techpreview: 0% (0 runs current, 1 previous). Source: Sippy Job Health

Figure 4: Sippy Job Analysis for 4.21 non-techpreview. 13 runs, 13 successful. Source: Sippy Job Analysis

Figure 5: Sippy Job Analysis for 4.21 techpreview. 1 run, 0 successful. Source: Sippy Job Analysis

Since Sippy only covers ~2 weeks, we fetched the full job history from Prow by paginating through all available runs. This covers the entire period from the first run (late February 2026) to present (August 5, 2026) - approximately 5.5 months of data.

Failure timeline:

Failure timeline:

Add the missing e2e-azure-aks-external-oidc periodic and presubmit jobs to releases 4.22 and 5.0, mirroring the 4.21 configuration. This includes:

A dedicated Jira issue is required to track this work, as no existing issue covers the propagation gap.

Once the periodic jobs are added to 4.22 and 5.0, include them in the ARO HCP release gating pipeline as informing (not blocking) for an initial observation period.

Why not immediately blocking?

The pass rate data from existing releases is encouraging but does not guarantee the same behavior on newer releases:

The raw pass rate of 84-87% includes a concentrated cluster of 8 infrastructure failures in June 2026 (identical on both 4.20 and 4.21, durations of 0-3 minutes indicating the test never started). Excluding this platform-wide incident, the effective pass rate is higher. The recent 2-week window shows 100% stability.

However, the 4.22 and 5.0 periodic jobs have zero historical data. Before promoting to blocking in the release gating pipeline, we need to:

1. Confirm the test runs cleanly on these releases (new platform versions, new dependencies)
2. Collect enough data points to establish a reliable baseline (target: at least 2-4 weeks of daily runs)
3. Verify no release-specific regressions exist

The techpreview variant (e2e-azure-aks-external-oidc-techpreview) tests ExternalOIDCWithUpstreamParity, which remains TechPreview-only. On 4.21, the periodic had only 1 run with 0% success (scheduled 3x/month via cron: 0 20 9,18,25 * *).

This variant should be propagated to 4.22/5.0 for CI coverage parity, but it is not a candidate for the release gating pipeline given its TechPreview status and lack of reliable data.

Sippy

• Sippy: Job Health 4.20 (filtered by e2e-azure-aks-external-oidc)
• Sippy: Job Analysis 4.20
• Sippy: Job Health 4.21 (filtered by e2e-azure-aks-external-oidc)
• Sippy: Job Analysis 4.21 (non-techpreview)
• Sippy: Job Analysis 4.21 (techpreview)

Prow Job History

• Prow: Job History 4.20 e2e-azure-aks-external-oidc
• Prow: Job History 4.21 e2e-azure-aks-external-oidc

Jira

• CNTRLPLANE-3790 - Add more blocking tests to the ARO HCP promotion pipeline
• CNTRLPLANE-3434 - ARO HCP Release Promotion Pipeline (parent epic)
• CNTRLPLANE-946 - Add payload tests for HyperShift (test origin)
• CNTRLPLANE-883 - Enhancements to BYO External OIDC (feature epic)
• OCPSTRAT-1804 - BYO External OIDC GA (feature request)
• AUTH-530, AUTH-552, AUTH-553, AUTH-554 - QE testing issues (audited for propagation scope)

Code

• openshift/release#68741 - Original PR adding the AKS external OIDC periodic jobs
• control-plane-operator/featuregates/featuregates.go - Feature gate definitions confirming GA status