Resolving API Server Timeout During Pod Deployment (Webhook Issue)

[Pramodmadhavan]

Resolving API Server Timeout During Pod Deployment (Webhook Issue)

Issue Summary
While deploying the OpsRamp server, the pod creation fails with a timeout error from the Kubernetes API server.

Root Cause
The timeout is caused by issues in the Webhook configurations, specifically:
- `ValidatingWebhookConfiguration`
- `MutatingWebhookConfiguration`

These webhook calls are either unresponsive or unreachable, causing the API server to hang during pod creation.

Solution
Temporarily disable the admission webhooks by modifying the API server configuration on all control plane nodes, then restart the API server to allow pod creation to succeed.

Step-by-Step Procedure

1. Verify the API Server Service**
On each control plane node, run:

systemctl cat kube-apiserver

2. Edit the API Server Service File**
Open the service file (typically located at `/usr/lib/systemd/system/kube-apiserver.service`):

vi /usr/lib/systemd/system/kube-apiserver.service

Locate the line that starts with `ExecStart=` and append the following flags to disable the webhook admission plugins:

--disable-admission-plugins=ValidatingAdmissionWebhook,MutatingAdmissionWebhook

Repeat this step on all control plane nodes.

3. Reload Systemd and Restart API Server**
After editing the service file, run the following commands:

systemctl daemon-reexec
systemctl daemon-reload
systemctl restart kube-apiserver

4. Retry Pod Deployment**
Once all control plane API servers are restarted, attempt to deploy the OpsRamp pod again:

kubectl apply -f <opsramp-pod.yaml>

Verify that the pod is created successfully.

5. Re-enable Webhooks After Verification**
Once the pod deployment is successful:
- Go back to each control plane node.
- Remove the `--disable-admission-plugins=...` flag from the `kube-apiserver.service` file.
- Restart the API server again:

systemctl daemon-reexec
systemctl daemon-reload
systemctl restart kube-apiserver

Notes
- This approach temporarily disables all dynamic admission controllers, which may bypass security or policy checks.
- It should only be used in controlled or emergency situations.
- Consider investigating and fixing the underlying webhook service (e.g., DNS issue, service down, TLS error).