CNTRLPLANE-3921

Job: periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn

Pass rate: ~93.5% (10 failures / 154 runs over 14 days, per Sippy)

Bryan flagged that HyperShift tests were frequently blocking CI release payloads for OCP 5.0. Ahmed identified EnsureGlobalPullSecret as a critical flake and began investigation. Due to time constraints, he handed off with a Claude Code handoff gist. Alessandro continued the deep-dive analysis.

• Bryan's original alert — "FYI we are blocking CI release payloads frequently with our tests"
• Ahmed's handoff + investigation thread

EnsureGlobalPullSecret runs as part of TestCreateCluster / TestNodePool_HostedCluster0 and validates the full lifecycle of the Global Pull Secret feature across 7 phases: in-place pull secret propagation without NodePool rollout, CAPI label checks, additional-pull-secret CRUD, HCCO reconciliation of the global-pull-secret DaemonSet, on-disk kubelet config.json verification via a privileged kubelet-config-verifier DaemonSet, pod image pull verification, and cleanup.

Key code: test/e2e/util/util.go:1958-2257 (test body), test/e2e/util/globalps.go (DaemonSet helpers), waitForDaemonSetReady at util.go:2283 (20-min polling loop, 10s interval).

By analyzing HCP namespace event dumps and node YAML artifacts from run 2082061384283066368, we observed that the network-node-identity pod kill at ~12:07:10 coincides with a broad node-level cordon, suggesting a node drain rather than a targeted pod delete. This has only been analyzed on a single run and needs cross-run validation.

The management cluster for this CI job is itself a HyperShift HostedCluster (773085beb4-mgmt, OCP 4.22.0-ec.5) running on the CI infrastructure. The job runs with REQUEST_SERVING_COMPONENT_TEST=true, which causes test/e2e/util/requestserving.go:SetupReqServingClusterNodePools() to create additional NodePools, including non-request-serving NodePools.

Management cluster node inventory at dump time:

Around 12:07:10, 92 events fired in the HCP namespace. Multiple pods that had been running on ip-10-0-0-27 were replaced and rescheduled to ip-10-0-46-59. The simultaneity suggests a node-level event, but the exact trigger has not been confirmed:

• Critical CP components (kube-apiserver, etcd, oauth-openshift, openshift-apiserver) have PodDisruptionBudgets and were observed running on the original (stable) nodes. network-node-identity has no PDB.
• Its webhook (pod.network-node-identity.openshift.io:9743) is a mutating admission webhook — when down, ALL new pod sandbox creation in the guest cluster is blocked.
• The replacement pod's init container crashes in a loop for ~15 min, turning a brief restart into a prolonged outage.

Observed on 4/4 runs analyzed in depth. When the webhook is unreachable, new pod sandbox creation in the guest cluster fails with failed calling webhook "pod.network-node-identity.openshift.io": connection refused. Multiple guest namespaces affected: openshift-ingress-canary, openshift-insights, openshift-dns.

Observed on 4/4 runs analyzed in depth. The replacement pod's init container hosted-cluster-kubecfg-setup fails 72-73 times over ~15 min with illegal base64 data at input byte 3. This turns what should be a ~30-second restart into a ~20-minute outage. The corrupted Secret and why it takes ~15 min to stabilize is still unknown.

On run 2082061384283066368, we observed that the pod kill at ~12:07:10 coincides with 92 simultaneous events across 8+ deployments, and that the non-request-serving node YAML dumps show spec.unschedulable: true. This is consistent with a node drain, but:

• Node YAML dumps are point-in-time snapshots, not event logs — the exact cordon timestamp is inferred, not directly observed
• The trigger for the cordon (config hash change vs autoscaler vs other) is unknown
• The management cluster dump was scrubbed, preventing verification of the parent cluster operator logs

shouldCheckForStaleCerts() returns false for CPOv2 clusters (hostedcluster_controller.go:2577). OCP 5.0 uses CPOv2. Deployment dump confirms: no hypershift.openshift.io/restart-date annotation present.

In 4/4 runs analyzed, all images reported as "already present on machine". Image pull was not a contributing factor.

Initially hypothesized failures always occur on the same node. Verified across runs that nodes are different ephemeral EC2 instances — the failure is not node-specific but is tied to the non-request-serving NodePool role.

Ahmed investigated a potentially different symptom on the presubmit e2e-aws job: the global-pull-secret-syncer DaemonSet has observedGeneration stuck at 2 while generation is 3, suggesting the DaemonSet controller is not processing the latest spec update.

Hypotheses Ahmed disproved:

1. DaemonSet controller lag (mismatch persists for the full 20 min)
2. Node taint/label issues
3. Resource quota exhaustion
4. PodDisruptionBudget interference
5. Webhook blocking DaemonSet pod creation

Proposed observability improvements:

1. Add resourceVersion to the polling log in waitForDaemonSetReady
2. Ensure full KCM/HCCO log capture (currently truncated to ~3.5 min)
3. Add reconcile result logging in the globalps controller

• PR #9167 — Remove the v1 EnsureGlobalPullSecret test (Cesar Wong)
• PR #9073 — Port test to v2 lifecycle (Dan Mace)
• JUnit lifecycle=informing mechanism to mark flaky tests as non-blocking (Dan Mace, branch)

• Ahmed's observability PR: resourceVersion logging + full KCM/HCCO log capture
• Investigate why non-request-serving nodes are drained ~30 min after creation — is this a NodePool config hash change from CVO rollout?
• Investigate init container base64 error — which Secret, why corrupted during rescheduling
• Consider scheduling constraints to keep network-node-identity on original (stable) management cluster nodes
• Consider adding PDB for network-node-identity in CNO
• Ensure management cluster dumps preserve events (currently scrubbed for sensitive data)

• Prow job page
• network-node-identity Deployment YAML
• HCP namespace events
• PDBs in HCP namespace (no network-node-identity PDB)
• Guest cluster event-filter.html (webhook failures visible)
• Management cluster creation log

Key event YAMLs analyzed: