Post

What makes you think the CR is more correct than reality? IMO once reconciled the operator is done.

CR is the intended config, so I expect it to overwrite the reality –the question is whether it should try to course-correct periodically. There are some systems (like Prodspec/Annealing) that learn from the current state and bring it back to the configuration SoT, but they're few and rare.

My interpretation is that the CR is the intended state during reconciliation, not forever. If something/someone changes the actual state, they'll be surprised to find it asynchronously revert. I don't think the CR can realistically be a SoT most of the time.

Concrete example: imagine a Bucket resource creates/configures buckets on s3 API. Someone goes to the s3 console and edits bucket configuration. Unless your controller periodically queries s3 API, it won't overwrite the out-of-band modifications until next time Bucket resource is modified.

Ideally an operator would watch the resources it manages and revert drift as soon as possible, to minimize downtime. But in the case where watching is not possible, polling may be acceptable. For scaling, consider adding a component that centralizes the polling and transforms it into watches.

To convert data polled from external API to watches, would you bring data from external API back into KRM APIs? I feel if there's an assumption like CRD will always overwrite the external API, not sure if there's value in reconciling sooner (vs eventually when the resource.generation++).

It’s the old terraform (you’re the only actor) versus kubernetes resource model (reconciliation loops/annealing) conflict. In both cases they seem to have taken a conscious approach.

The problem with the first is that it implies a need for much more ordering (where TF benefits from having a resource graph, which has its own issues), as on a single run you’re expected to completely converge to the model.

I'd expect that they're periodically querying the S3 API in this case. At a high-level, operators are supposed to continuously reconcile the current and desired states (which imply watching out for drift). Unfortunately, most external systems lack APIs to listen for changes, hence the polling.

If your controller is managing 20,000 buckets, you'll make 20k external API calls during controller startup full sync to verify buckets exist and their config is up-to-date (99.99% are) –during which you won't get to processing a new object soon enough, bc the workqueue is still full.