Post

Should Kubernetes controllers+CRDs managing resources in the external world (non-k8s APIs) try to fix drifts (such as out of band modifications made in the external API) periodically –or assume they're the only actor, and wait for the next modification to reconfigure? This makes scaling harder.

Concrete example: imagine a Bucket resource creates/configures buckets on s3 API. Someone goes to the s3 console and edits bucket configuration. Unless your controller periodically queries s3 API, it won't overwrite the out-of-band modifications until next time Bucket resource is modified.

I'd expect that they're periodically querying the S3 API in this case. At a high-level, operators are supposed to continuously reconcile the current and desired states (which imply watching out for drift). Unfortunately, most external systems lack APIs to listen for changes, hence the polling.

If your controller is managing 20,000 buckets, you'll make 20k external API calls during controller startup full sync to verify buckets exist and their config is up-to-date (99.99% are) –during which you won't get to processing a new object soon enough, bc the workqueue is still full.

question: is there a reason a single controller needs to do this work vs partitioning for scale and blast radius containment?