Kyverno's `generate`: The Architect's Guide to GitOps Automation

A frequent point of confusion, highlighted in Kyverno community issue #13997, is observing a generate rule failure after a validate policy blocks its trigger resource. This is not a bug but a key insight into the asynchronous, event-driven architecture of Kubernetes and Kyverno. While validate rules operate synchronously during the admission request to block a resource, the creation event has already been dispatched. Kyverno's background controller, operating asynchronously, then attempts to process the corresponding generate rule. It fails because the trigger resource was never persisted, resulting in the expected failure log.

This behavior underscores a fundamental design principle. This article provides a deep dive into the generate policy lifecycle to demystify this interaction. We will explain the critical distinction between synchronous admission control and asynchronous background processing that governs Kyverno's controllers. Finally, we will offer concrete best practices for building robust GitOps automation, enabling you to design policies that align with Kubernetes' eventual consistency model and avoid false-positive alerts. Understanding this separation is essential for mastering advanced, reliable policy-as-code implementations.

Kyverno's generate policies are a "perfect fit" for GitOps, as Kelsey Hightower noted, because they extend its declarative model to dynamically managed resources. While GitOps excels at reconciling configurations from a Git repository, it lacks a native mechanism to create resources based on in-cluster events, like the creation of a new Namespace. The generate rule fills this gap by automating resource creation.

The key is the synchronize feature. Setting synchronize=true transforms a generate rule from a one-time trigger into a continuous reconciliation loop. For instance, a policy can automatically generate a default NetworkPolicy for every new Namespace. With synchronize enabled, Kyverno not only creates this policy but also actively monitors it. If the NetworkPolicy is manually modified or deleted, Kyverno's background controller detects this configuration drift and automatically reverts the changes or recreates the resource. This ensures the policy's declared state is enforced continuously, extending the GitOps model of declarative state management to dynamically generated assets and guaranteeing configuration consistency beyond initial creation.

To make this interaction concrete, let's trace the event sequence using an example where a user attempts to create a Namespace named bad-namespace that violates a validate policy.

1. [User Action] A user submits a manifest for the bad-namespace, which lacks a required label enforced by a validate rule.

2. [API Server] The API server receives the request and immediately broadcasts a Namespace creation event to all watchers, including Kyverno. This happens before the request is fully validated.

3. [Background Controller - Asynchronous] Kyverno's background controller sees the creation event and queues a task to run the corresponding generate policy for the bad-namespace.

4. [Admission Controller - Synchronous] The Kyverno admission webhook intercepts the live request. The validate rule executes, detects the missing label, and rejects the bad-namespace creation. The resource is never saved to etcd.

5. [Background Controller - Asynchronous] Later, the queued generate task runs. It queries the API server for its trigger resource, the bad-namespace, to get the data needed to generate a new resource.

6. [Background Controller - Asynchronous] The API server responds with a not found error because the bad-namespace was never created. Kyverno logs this failure, correctly reporting that the trigger resource does not exist.

This "not found" error is not a bug; it is a feature of a healthy, observable system. As observability pioneer Charity Majors advocates, we should build systems that are easy to understand and debug. In an eventually consistent, distributed environment like Kubernetes, this log entry is a valuable piece of the audit trail, providing a transparent record of the system's behavior.

The log is a positive signal that confirms two critical actions executed correctly. First, it verifies that the synchronous validate policy successfully blocked the non-compliant resource at the admission stage. Second, it demonstrates that the corresponding asynchronous generate automation was correctly triggered and then safely halted because its trigger resource rightfully never existed.

This log is a feature of transparency, not a flaw. It is the system providing a clear receipt that your security and automation controls are functioning in concert, even across different timing models. It’s the sound of two distinct mechanisms—admission control and background processing—doing their jobs perfectly. The log isn't noise; it's the system telling you its story.

To build resilient GitOps automation and prevent unintended side effects, integrate the following best practices into your generate policy workflow.

First, align generate triggers with validate conditions using precise match and exclude blocks. If a validate policy requires a Namespace to have a team label, the corresponding generate policy should match only on Namespace resources that possess that label. This ensures the generate rule only attempts to run on resources that successfully pass validation, preventing the "not found" error at the source.

Second, always set synchronize: true. This is the key to aligning with GitOps principles, transforming the rule from a one-time action into a continuous reconciliation controller. Kyverno will not only create the resource but also actively monitor it, automatically reverting unauthorized changes or recreating the resource if it's deleted. This guarantees the generated resource's state remains consistent with your policy, enforcing declarative configuration beyond initial creation.

Third, test policies locally with the Kyverno CLI before committing them to Git. Using commands like kyverno apply, you can simulate how policies will behave against sample resources on your workstation. This practice allows you to validate trigger logic and catch unintended side effects early, providing a critical feedback loop that prevents faulty automation from ever reaching the cluster.

Finally, implement strategic monitoring. Treat the "trigger resource not found" log as an informational signal, not an actionable error. Configure your alerting systems to explicitly ignore this specific log entry. This reduces alert fatigue and allows your team to focus on persistent or different generate failures, which indicate genuine problems. By filtering out this expected noise, you ensure that alerts are meaningful and point to real issues requiring intervention.