Comparing Policy Engines in Kubernetes
Hi everyone,
As you may know, ValidatingAdmissionPolicy has been generally available since Kubernetes 1.30, and MutatingAdmissionPolicy became generally available in Kubernetes 1.36. This brought a question to my mind: Do I still need Kyverno?
I’ve been using Kyverno very extensively for different reasons, and I have several technical reasons to choose it over Gatekeeper. But now we have a new player in town, and it is natively supported by kube-apiserver (meaning Kubernetes itself)!
What are Admission Policies?
Kubernetes gives us many great features, and I believe one of the best is admission control, which gives us a superpower to mutate or validate API objects across the entire cluster. You can find many great examples here!
This means that if you don’t want to allow a request because it is missing a label, or if you want to add a sidecar, volume, annotation, or almost anything else, admission policies can be your biggest lifesaver!
How does it work?
When you make a request, kube-apiserver first authenticates and authorizes it. During admission, mutating controllers run before validating controllers. If a webhook rule matches, the API server sends an AdmissionReview to the webhook service over HTTPS. The webhook returns a patch, an allow decision, or a rejection.
The native policies take a shorter route. Their CEL expressions are evaluated inside kube-apiserver, so there is no webhook service, TLS hop, or AdmissionReview network round trip. MutatingAdmissionPolicy can return an ApplyConfiguration or JSON Patch, while ValidatingAdmissionPolicy returns a validation decision.
This is a simplified version of the flow:
flowchart TB
C[Client request] --> A[Authentication]
A --> Z[Authorization]
Z --> M{Mutating admission}
M -->|Native policy| MC[CEL inside kube-apiserver]
M -->|Kyverno or Gatekeeper| MW[HTTPS AdmissionReview to webhook]
MC --> S[Object and schema validation]
MW --> S
S --> V{Validating admission}
V -->|Native policy| VC[CEL inside kube-apiserver]
V -->|Kyverno or Gatekeeper| VW[HTTPS AdmissionReview to webhook]
VC --> P[(Persist object)]
VW --> P
One important detail here is that mutating webhooks are called sequentially, while validating webhooks can be called in parallel. This is one reason why adding many mutating webhooks can become expensive. The official Kubernetes admission webhook good practices also recommend using built-in CEL-based admission control when it fits, and using webhooks when you need more complex or external behavior.
What changes?
Actually, almost nothing changes in the idea!
Previously, third-party solutions such as Kyverno and Gatekeeper (and perhaps a custom webhook you developed) handled these jobs very well in different ways. Now kube-apiserver can offer similar validation and mutation for policies that can be expressed with CEL.
But under the hood, plenty changes. Native admission has fewer moving parts and no external webhook hop. Kyverno and Gatekeeper have their own controllers, caches, TLS certificates, webhook configurations, background features, audit capabilities, and much bigger policy ecosystems. Native admission is a new option, not a complete replacement for either project.
What should I choose?
Yes! This is the part that I would like to dive into. I’m interested because I love benchmarks.
Before looking at the numbers, I want to be very clear about what I measured and what I did not measure.
What exactly did I benchmark?
I defined one small policy contract first, and then implemented that contract in Kyverno, Gatekeeper, and Kubernetes-native admission policies.
| Part | Exact contract |
|---|---|
| Resource | Core v1 Pod |
| Operations | CREATE and UPDATE in the tested domain |
| Namespace | policy-bench |
| Subject selector | Label policy-bench-subject: "true" |
| Mutation | Add annotation policy-bench-injected: "true" only when it is absent; never overwrite an existing value |
| Validation | Require annotation policy-bench-team: "platform" |
| Rejection message | policy-bench-team annotation must equal platform |
The policy files cannot be textually identical because Kyverno uses its policy language, Gatekeeper uses Rego plus its mutation CRDs, and native admission uses CEL. The fair target is therefore semantic equivalence: the same resources must receive the same observable mutation and validation result.
Before every load test, I submitted 180 correctness-checking requests. For each request, the benchmark compared the actual Kubernetes admission decision and mutation against a predefined expected result. In the three policy-engine configurations, it expected the same policy behavior; in the no-policy baseline, it expected no policy mutation or rejection.
The machine and load shape
I ran the tests in the following environment:
| Item | Value |
|---|---|
| Host | Ubuntu 24.04.4 LTS, aarch64 |
| CPU | 4 × Neoverse-N1 |
| Host memory | 7.549 GiB |
| Docker Engine | 28.2.2 |
| Kubernetes | 1.36.1, pinned kind node digest |
| kind | 0.32.0 |
| Kyverno | Helm chart 3.8.2, app 1.18.2 |
| Gatekeeper | Helm chart 3.23.0 |
| Topology | One kind control-plane node |
For each measured configuration, I sent exactly 12,000 valid dry-run Pod CREATE requests at an offered rate of 200 requests per second for 60 seconds, with a maximum client concurrency of 128. Before that, I used a 15-second warm-up at 50 requests per second. Every one of the 144,000 measured requests completed successfully.
Why dry-run? Because dryRun=All still executes admission, defaulting, schema validation, and patch processing, but it does not write the object to storage. That removes scheduling, image pulls, Pod startup, object deletion, and most of the noise that has nothing to do with admission. This is useful for this microbenchmark.
How did I measure p50, p95, and p99?
Every request had an absolute scheduled send time. The load generator used a monotonic clock and recorded three values:
- HTTP latency: from the moment a worker started the HTTPS API call until the complete response arrived.
- Client queue delay: from the request’s planned send time until a worker actually started it.
- Scheduled latency: queue delay plus HTTP latency.
This distinction matters a lot! If the system cannot keep up, workers start requests late. Looking only at HTTP latency can hide that overload. Scheduled latency makes the missed schedule visible and reduces coordinated-omission bias.
Pilot results
Here are the pooled end-to-end HTTP results at the same offered load:
| Configuration | Successful requests | p50 | p95 | p99 | Scheduled p95 | Queue p95 | Achieved RPS |
|---|---|---|---|---|---|---|---|
| Baseline | 36,000 | 4.16 ms | 7.60 ms | 14.27 ms | 8.29 ms | 0.87 ms | 199.76 |
| Native | 36,000 | 6.38 ms | 12.77 ms | 24.38 ms | 13.73 ms | 1.35 ms | 199.67 |
| Gatekeeper | 36,000 | 8.76 ms | 19.72 ms | 51.83 ms | 20.23 ms | 1.01 ms | 199.71 |
| Kyverno | 36,000 | 25.03 ms | 1,198.86 ms | 1,892.25 ms | 3,935.95 ms | 3,247.46 ms | 196.05 |
In this environment, native admission added the smallest latency over the baseline. Gatekeeper came next. Kyverno’s tail went into a completely different scale.
But the pooled table hides something very important. These are the HTTP p95 values from each fresh-cluster trial:
| Configuration | Trial 1 | Trial 2 | Trial 3 |
|---|---|---|---|
| Baseline | 5.94 ms | 5.75 ms | 9.89 ms |
| Native | 9.55 ms | 9.03 ms | 17.51 ms |
| Gatekeeper | 21.08 ms | 19.69 ms | 18.29 ms |
| Kyverno | 66.76 ms | 758.36 ms | 1,629.04 ms |
Gatekeeper was quite stable across these three pilot trials. Kyverno was not. Trial 3 reached a saturation state: its client queue p95 was 3.454 seconds, scheduled p95 was 4.385 seconds, and it achieved 188.75 requests per second after drain instead of staying close to the offered 200.
That is why I do not want to hide every trial behind one nice-looking pooled percentile.
Resource consumption
I sampled the whole kind node once per second during idle and load. Whole-node usage is the fairest primary comparison because native policy execution is inside kube-apiserver; there is no separate native policy container whose CPU and memory I can isolate.
| Configuration | Mean node CPU | Mean of trial CPU p95s | Load minus idle CPU | Mean node memory | Mean of trial memory p95s | Load minus idle memory |
|---|---|---|---|---|---|---|
| Baseline | 1.102 cores | 1.200 | 0.894 | 543.70 MiB | 550.16 MiB | 19.40 MiB |
| Native | 1.442 cores | 1.539 | 1.227 | 556.95 MiB | 562.92 MiB | 15.53 MiB |
| Gatekeeper | 2.033 cores | 2.177 | 1.789 | 701.70 MiB | 708.95 MiB | 23.57 MiB |
| Kyverno | 2.660 cores | 3.045 | 2.418 | 1,086.01 MiB | 1,109.29 MiB | 96.69 MiB |
The resource p95 columns are the arithmetic mean of the three fresh-cluster p95 values, not one percentile pooled across all one-second samples.
These are total system costs, not clean policy-only costs. As a secondary view, containers in the Kyverno namespace used a mean of 1.249 CPU cores and 80.94 MiB, while containers in the Gatekeeper namespace used 0.632 cores and 40.20 MiB. The native comparison point would be kube-apiserver, but that includes all API server work, so putting it in the same table as a policy-only process would be misleading.
The benchmark deliberately reduced every engine to the same small validation-and-mutation contract. Gatekeeper audit was disabled. For Kyverno, the reports, background, and cleanup controllers and all admission-report and policy-report generation were explicitly disabled. Its measured overhead therefore came from the admission-only deployment, not from those features. Those features change both what the products can do and what they consume when you enable them.
Feature matrix
| Capability | Kubernetes-native admission | Gatekeeper | Kyverno |
|---|---|---|---|
| Data from other Kubernetes objects | LIMITED! Native policies can access namespaceObject and binding-selected parameter objects, but CEL cannot perform arbitrary GET or LIST calls |
Replicated inventory is available through data.inventory; it is eventually consistent, and both documented synchronization APIs are alpha |
YES! Policies can call the Kubernetes API and read resources such as ConfigMaps |
| External HTTP/API data | NO! Native CEL cannot make outbound HTTP calls | The beta External Data feature calls an approved in-cluster HTTPS provider, which can query registries, LDAP, or other external systems; it supports validation and mutation | YES! Policies can make direct HTTP(S) GET and POST calls through the HTTP CEL library or legacy context.apiCall, and can cache results with GlobalContextEntry |
| Scan existing resources | NO! Native policies run only during admission | YES! Audit periodically evaluates existing resources | YES! Background scans evaluate existing resources and record validation or image-verification results |
| Reporting | Client warnings and Kubernetes audit events or annotations when audit logging captures them; no reconciled compliance inventory | YES! The latest audit results are stored in Constraint .status, with a configurable cap on detailed violations |
YES! PolicyReport and ClusterPolicyReport, plus optional OpenReports support, which is currently alpha |
| Generate, clone, or synchronize supporting resources | NO! Mutation can only change the object currently being admitted | NO! There is no general resource-generation controller | YES! GeneratingPolicy can create, clone, and synchronize resources, including for existing triggers |
| Scheduled cleanup or deletion | NO! | NO! | YES! DeletingPolicy deletes matching resources on a cron schedule |
I didn’t add more features such as testing, attestations, or exceptions because they are outside the scope of this blog post.
What would I choose after looking at the benchmark results and feature matrix?
As I mentioned, I’ve been using Kyverno for a very long time, and I’ve relied heavily on many of its extensive features, such as DeletingPolicy (CleanupPolicy) and API calls (internal and external). Even though API calls can add latency, I’m unable to ditch those requirements. BUT! Not all my policies require API calls!
Based on these benchmark results, I’m leaning toward migrating some policies from Kyverno to native admission. Of course, I would not do it all at once. If you have thousands of Nodes and hundreds of Pods churning every hour, that already adds a lot of load to the API server. Before going full throttle with native admission, I would like to see it battle-tested on a live cluster alongside all the other components!
I’m not an active Gatekeeper user. While collecting evidence and building the Gatekeeper feature matrix, I used AI assistance. So if anything is wrong or missing regarding Gatekeeper, or if you see a problem with the benchmarking methodology, please feel free to reach out!