Software - General
1828486 Members
3397 Online
109978 Solutions
New Discussion

Resolving API Server Timeout During Pod Deployment (Webhook Issue)

 
Pramodmadhavan
HPE Pro

Resolving API Server Timeout During Pod Deployment (Webhook Issue)

Resolving API Server Timeout During Pod Deployment (Webhook Issue)

Issue Summary
While deploying the OpsRamp server, the pod creation fails with a timeout error from the Kubernetes API server.

Root Cause
The timeout is caused by issues in the Webhook configurations, specifically:
- `ValidatingWebhookConfiguration`
- `MutatingWebhookConfiguration`

These webhook calls are either unresponsive or unreachable, causing the API server to hang during pod creation.

Solution
Temporarily disable the admission webhooks by modifying the API server configuration on all control plane nodes, then restart the API server to allow pod creation to succeed.

Step-by-Step Procedure

1. Verify the API Server Service**
On each control plane node, run:

systemctl cat kube-apiserver


2. Edit the API Server Service File**
Open the service file (typically located at `/usr/lib/systemd/system/kube-apiserver.service`):

vi /usr/lib/systemd/system/kube-apiserver.service

Locate the line that starts with `ExecStart=` and append the following flags to disable the webhook admission plugins:

--disable-admission-plugins=ValidatingAdmissionWebhook,MutatingAdmissionWebhook

Repeat this step on all control plane nodes.

3. Reload Systemd and Restart API Server**
After editing the service file, run the following commands:

systemctl daemon-reexec
systemctl daemon-reload
systemctl restart kube-apiserver


4. Retry Pod Deployment**
Once all control plane API servers are restarted, attempt to deploy the OpsRamp pod again:

kubectl apply -f <opsramp-pod.yaml>

Verify that the pod is created successfully.

5. Re-enable Webhooks After Verification**
Once the pod deployment is successful:
- Go back to each control plane node.
- Remove the `--disable-admission-plugins=...` flag from the `kube-apiserver.service` file.
- Restart the API server again:

systemctl daemon-reexec
systemctl daemon-reload
systemctl restart kube-apiserver


Notes
- This approach temporarily disables all dynamic admission controllers, which may bypass security or policy checks.
- It should only be used in controlled or emergency situations.
- Consider investigating and fixing the underlying webhook service (e.g., DNS issue, service down, TLS error).

I'm an HPE employee.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo