Post

Should Kubernetes controllers+CRDs managing resources in the external world (non-k8s APIs) try to fix drifts (such as out of band modifications made in the external API) periodically –or assume they're the only actor, and wait for the next modification to reconfigure? This makes scaling harder.

Concrete example: imagine a Bucket resource creates/configures buckets on s3 API. Someone goes to the s3 console and edits bucket configuration. Unless your controller periodically queries s3 API, it won't overwrite the out-of-band modifications until next time Bucket resource is modified.

I'd expect that they're periodically querying the S3 API in this case. At a high-level, operators are supposed to continuously reconcile the current and desired states (which imply watching out for drift). Unfortunately, most external systems lack APIs to listen for changes, hence the polling.

question: is there a reason a single controller needs to do this work vs partitioning for scale and blast radius containment?

I'm going off the usual assumption that most people run k8s controllers w/ single active leader + a lot of external APIs have rate limits. Partitioning controllers is not a trivial task (Tim Ebert has a masters thesis on this with a prototype impl.) and nobody I know does this.

Same in k8s with load-balancers. If you have 100 services with LBs, and someone modifies one behind the scenes, we don't know to reconcile, because (AFAICT) none of the clouds have watch APIs and they all have API quota. Polling 100 LBs every 10 seconds = 10 QPS of 99.999% do-nothing traffic.

I'm on this boat exactly –but lots of differing opinions. IMO if you're going behind your own back making out of band edits, don't expect the controller (which assumes it's the sole actor managing this resource, will overwrite any changes) to have this responsibility of promptly fixing the resource.

OTOH, maybe Kubernetes will prompt cloud providers to support some form of WATCH operations, just to alleviate the load of millions of kube clusters polling? I'm an optimist. :)