Post

1/9 If a service commits something to a database and then publishes a message to Kafka, what happens if the database commit succeeds but the Kafka publish fails? This problem has nothing to do with AI. But the mindset behind solving it matters a lot for AI agents.

2/9 In distributed systems, we usually solve this with the Transactional Outbox Pattern. You write the business change and the pending event in the same database transaction. Then another process reliably publishes the event to Kafka.

3/9 When I started moving from software engineering into AI engineering, I thought I needed to deeply understand model training, evaluation, feature engineering, research papers, and ML deployment pipelines before I could build useful agents. I was wrong.

4/9 Learning ML fundamentals is valuable. But most software engineers do not need to become machine learning researchers to build agentic applications. Models are available through APIs now. The harder problem is integrating them into real systems.

5/9 Building AI agents means understanding: How to handle unreliable outputs How to design safe tool calls How to manage state How to recover from partial failures How to protect data and permissions How to observe behavior in production

6/9 This is what people are starting to call harness engineering. Harness engineering is the deterministic system you build around the model. The workflows. The validations. The retries. The logs. The tests. The fallbacks. The guardrails.

7/9 In other words, a lot of agentic AI is still about the same engineering guarantees that have always made distributed systems work. Consistency. Reliability. Security. Observability. Failure handling. Production discipline.

8/9 This does not mean machine learning does not matter. It absolutely does. Machine learning engineers understand model behavior, evaluation, uncertainty, drift, and failure modes. That expertise is critical.