Ian Macartney

Rate limiting LLM chat per-user with `useRateLimit` const { status } = useRateLimit(api.rl.getRateLimit) Full code in 🧵 Algorithms: Sending messages: fixed window Token usage: token bucket courtesy of @​convex-dev/rate-limiter + @​convex-dev/agent (components)

Recent talk on agents & agentic workflows, as well as my Convex Agent Component. Takes: • Agentic := prompting + routing • Prompting is input -> LLM -> output • Routing has code at every boundary (...even if an LLM "decides" what to do next) Full 📺🔗->🧵

Streaming LLM text using websockets + client smoothing - no HTTP necessary! Agent v0.2.1 is out! Repo & release notes in 🧵 - Streaming text react hook + server fns - Client-side smoothing hook - Optimistic update helpers

Agent Component: convex.dev/components/a... Example code: github.com/get-convex/a... Changelog: github.com/get-convex/a... It one-ups persistent-text-streaming by syncing down only the deltas, not the full text, so you don't pay for bandwidth except proportional to the total length

Agent Playground for @​convex-dev/agent is live! Investigate threads, messages, tool calls Dial in context params Iterate on prompting, etc. For the @​convex-dev/agent component. Links in 🧵

Not all embeddings are created equal. Some represent more meaningful context. I struggled with AI Town to efficiently do vector search that also included a 1-10 "importance" Last night I figured out a way to prioritize some embeddings over others using a "bias" feature 🧵

My original thought was to just scale all the values, but vector search normalize the vectors for -1:+1 scores. The trick is to add an extra number ("feature") to the embedding. [...1536 numbers, <X>] Then query with [...1536 numbers, 0]. How it works: 🧵

I do RAG via hybrid text/vector search using reciprocal rank fusion (the one-weird-trick of hybrid search imo) for my new Agent framework/component. It's open source and the code is remarkably simple, if you're looking for an example for yourself.

Exciting news for Agent Workflow front: 🪨Durable Workflows🪨: Orchestrate steps async with retries, checkpointing and more, using Inngest-style syntax 🤖 Agent Framework 🤖: Define agents and use threaded memory (can hand off between agents), with hybrid text/vector search. 🧵

Run workflows reliably and asynchronously, using Inngest-style code. Why you need Durable Workflows for agentic systems: stack.convex.dev/durable-work... Workflow component: convex.dev/components/w...

Welp not all experiments work out, but what will outlive all products is the insights you glean along the way. I reimplemented Mastra workflows in Convex last week and I regret it. Article in 🧵

I made some cartoons today for an article I'm writing. This one is me when I'm too deep in a project that's gone off the rails, complete with alt text. Should I ship it or make a "better" version to make it "more professional"?

Thoughts on about safety & reliability in the age of agentic flows and durable workflows: Details in 🧵, but tl;dr: 1️⃣ Isolate unreliable steps to safely retry 2️⃣ Model LLMs as ~idempotent & ~deterministic 3️⃣ Run asynchronously, subscribe to results Bonus: I made a thing

2️⃣ Treat LLM calls as ~idempotent, so long as you don't update memory/ DB state in the same step (see 1️⃣). To do multiple LLM calls in a step, you can journal (cache) LLM outputs (on success) so retrying a multi-prompt flow will be ~deterministic.

Thoughts on about safety & reliability in the age of agentic flows and durable workflows: Details in 🧵, but tl;dr: 1️⃣ Isolate unreliable steps to safely retry 2️⃣ Model LLMs as ~idempotent & ~deterministic 3️⃣ Run asynchronously, subscribe to results Bonus: I made a thing

3️⃣ Kick off workflows asynchronously vs. running them in an HTTP handler. If the user's network blips or they refresh the page, the workflow is in limbo. Persist / return the run identifier instead. Subscriptions also enable multiple users to observe results.

Thoughts on about safety & reliability in the age of agentic flows and durable workflows: Details in 🧵, but tl;dr: 1️⃣ Isolate unreliable steps to safely retry 2️⃣ Model LLMs as ~idempotent & ~deterministic 3️⃣ Run asynchronously, subscribe to results Bonus: I made a thing

Tip: Use the `workpool` component to manage asynchronous workloads. - durable execution w/ onComplete handler - limit parallelism to be a nice neighbor - configure retries (w/ backoff, jitter, max attempts) - backlog & lag monitoring v0.2.1 out now www.convex.dev/components/w...

Local file sharing app (Open Source) Drag your files onto the site Drag them off on another computer git clone github.com/mikecann/our... cd ourfiles/ docker compose up open localhost:5173 by @mikeysee video in 🧵

Self-hosting Convex just got way easier, including running the dashboard. npx degit get-convex/convex-backend/self-hosted/fly fly && cd fly/backend && fly launch Dockerfiles, images, binaries @flydotio and more. Check it out 🧵

Article: stack.convex.dev/self-hosted-... OSS Repo: github.com/get-convex/c...

I used AI to write a script to help me write AI evals to help AI write Convex code. Very meta 💡Each step uses RAG from previous evals Flow: 1. 🧑‍💻oneliner -> 🤖 task description -> 🧑‍💻 audit 2. 🤖 answer ->🧑‍💻 audit 3. 🤖 tests for answer ->🧑‍💻 audit 🔗 to evals 👀 & my script in 🧵