CNTRLPLANE-3790

CNTRLPLANE-3790 tracks the expansion of the ARO HCP release gating pipeline's blocking test suite. The current pipeline (ITS source) has 9 blocking tests and 1 informing test.

The issue requires two separate investigations, each covered by its own report:

1. This report (1 of 2): Should e2e-azure-aks-ovn-conformance be promoted from informing to blocking?
2. Analysis 2 of 2: Why does e2e-azure-aks-external-oidc not exist for 4.22/5.0, and should it be added?

Current blocking tests (9):

• e2e-aks (5.0)
• hcm-azure-e2e-aks-upgrade-minor (5.0)
• e2e-aks-multi-x-ax (5.0, 4.22, 4.21, 4.20)
• hcm-azure-e2e-aks-upgrade-from-zero (4.22, 4.21, 4.20)

Current informing test (1):

• e2e-azure-aks-ovn-conformance (5.0) ← subject of this analysis

Two conformance test variants exist:

The non-FIPS variant is the primary candidate for blocking due to its broader coverage and higher run frequency. This analysis focuses on determining whether its pass rate is high enough to gate promotion without causing excessive false-positive blocks.

The analysis was performed using Sippy, the OpenShift CI health dashboard. Two views were consulted:

• Job Health (ALL JOBS) -aggregated pass rate and run count over the current reporting period
• Job Analysis -overall success/failure donut chart with job variant metadata

Data was collected on 2026-08-05. The reporting period covers the current and previous Sippy windows (exact dates determined by Sippy's rolling window configuration).

Figure 1: Sippy Job Health for 5.0, filtered by aks-ovn-conformance. Source: Sippy Job Health

Figure 2: Sippy Job Analysis for e2e-azure-aks-ovn-conformance on release 5.0. Source: Sippy Job Analysis

Over a broader window, Sippy reports 509 total runs with 344 successful (67.6%), consistent with the current-period figure of 69.9%.

The FIPS variant (e2e-azure-aks-ovn-conformance-fips) shows 100% pass rate, but with only 1 run in both the current and previous reporting periods. This sample size is far too small to draw any conclusions about reliability.

The FIPS variant runs weekly on Sundays, uses the narrower openshift/conformance/parallel/minimal suite, and skips several test categories. It would need significantly more run data before it could be considered as a blocking candidate.

The e2e-azure-aks-ovn-conformance test is not currently reliable enough to be promoted to a blocking test in the ARO HCP release gating pipeline.

• Pass rate too low: at 69.9% (249 runs), the test would block roughly 1 in 3 nightly promotions, creating significant noise and eroding trust in the pipeline
• Trend is improving (up from 65.4%), but not yet at a level suitable for gating
• FIPS variant has insufficient data (1 run) to evaluate

• Sippy: Job Health for 5.0 (filtered by aks-ovn-conformance)
• Sippy: Job Analysis for e2e-azure-aks-ovn-conformance
• Jira: CNTRLPLANE-3790
• Jira: CNTRLPLANE-3434 (parent epic)