Santiago Viquez

What’s the best way to track the progress of my book, The Little Book of ML Metrics? 1️⃣ Visit the book’s repo: github.com/NannyML/The-... 2️⃣ Download the latest digital WIP version. 3️⃣ Start reading while I keep writing. It gets updated every time I push new changes.

Super proud to work at a place that values open science. Four years ago, at NannyML, we invented the first version of Confidence-Based Performance Estimation. Today, a paper about it was published in JAIR. JAIR: jair.org/index.php/ja... ArXiv: arxiv.org/abs/2407.08649

Big kudos to my colleagues Jakub and Wojtek for their work on this! I’ll be honest, even at the risk of sounding biased. There are many ML monitoring companies out there, but none of them are solving the real problem. Most monitor the data, not the models.

If people don’t think what you do is cringe, then you’re not pushing hard enough. Every person you admire was once considered cringe by someone. A Writer, YouTuber, Founder, Musician, you name it. They all got to where they are because they constantly shared their work with the world. Constantly.

Which ranking metrics am I missing? In the coming weeks, I'll be working on the ranking chapter for "The Little Book of ML Metrics", and I want to make sure I'm not missing any popular ranking/recsys metrics.

Here are the ones I have so far: • MRR: Mean Reciprocal Rank • ARHR@k: Average Reciprocal Hit-Rank at K • nDCG@K: Normalized Discounted Cumulative Gain • Precision@k • Recall@k • F1@K • Average Recall@k • Average Precision@k • MAP: Mean Average Precision

ML Books I'll Be Reading in 2025 📚 1. "AI Engineering: Building Applications with Foundation Models" (Huyen, 2024): amzn.to/4gtQgJo We’ll probably read it in the study group "AI from Scratch."

5. "Fundamentals of Data Engineering: Plan and Build Robust Data Systems" (Reis, Housley, 2022): amzn.to/41TJWq9 Not ML-related, but still relevant.

ML Books I'll Be Reading in 2025 📚 1. "AI Engineering: Building Applications with Foundation Models" (Huyen, 2024): amzn.to/4gtQgJo We’ll probably read it in the study group "AI from Scratch."

3. "Writing for Developers: Blogs That Get Read" (Sarna and Dunlop, 2025): amzn.to/3ZXKmJl Not ML-related, but still relevant.

ML Books I'll Be Reading in 2025 📚 1. "AI Engineering: Building Applications with Foundation Models" (Huyen, 2024): amzn.to/4gtQgJo We’ll probably read it in the study group "AI from Scratch."

During the pandemic—specifically, on May 7, 2020—I wrote some goals on a piece of paper. I folded it, stored it in my wallet, and forgot about it. Today, I found it and realized I’ve accomplished all of them.

4. "Be more present for my family." 5. "Improve my relationship with my friends." 6. "Live abroad with Kathu and Maple."When I wrote this, I didn’t know we’d move to Italy.

During the pandemic—specifically, on May 7, 2020—I wrote some goals on a piece of paper. I folded it, stored it in my wallet, and forgot about it. Today, I found it and realized I’ve accomplished all of them.

Hear me out. Post-deployment data science. The part of data science that focuses on models after they have been deployed. - Checking if the model is delivering value. - Continuously estimating model performance. - Understanding performance issues and fixing them.

Univariate data drift doesn't show the full picture. Take a look at this demo created by my colleague @anopsy.bsky.social There are two univariate distributions (top and right) which remain almost unchanged during the whole process.

We can then use the learned compressor/decompressor to transform any production data and measure its reconstruction error.

Univariate data drift doesn't show the full picture. Take a look at this demo created by my colleague @anopsy.bsky.social There are two univariate distributions (top and right) which remain almost unchanged during the whole process.

In this example, we only have two variables, so it’s easy to see the change. But what if we have a model with 10, 50, or 100 features? For that, we need algorithms capable of learning the relationships between many features and detecting when they change substantially.

Univariate data drift doesn't show the full picture. Take a look at this demo created by my colleague @anopsy.bsky.social There are two univariate distributions (top and right) which remain almost unchanged during the whole process.

We can say that both look like normal distributions. If we run univariate drift detection methods such as Jensen-Shannon, Kolmogorov-Smirnov or any other, we wouldn't be alerted by those minimal changes. But if we look at their joint distribution, the story is different.