Joshua Crowley

Multimodal models like Flash 2.0 and GPT4o can use audio tokens to natively reason over audio input. Neither can yet provide text and audio tokens that come from the same inference pass, and presumably wont ever. So that presents a flaw in these models no one seems to be discussing.

There’s just so much detail to work through when implementing these experiences, it’s fun but also exhausting because they just pop up!

Multimodal models like Flash 2.0 and GPT4o can use audio tokens to natively reason over audio input. Neither can yet provide text and audio tokens that come from the same inference pass, and presumably wont ever. So that presents a flaw in these models no one seems to be discussing.

How do you transcribe those tokens for the UX of chat? It’s likely a user needs a transcript to understand what’s been understood. Well you have to use another model/call to transcribe the audio input, like whisper. So you have a new problem, does the transcript/audio diverge?

I don’t know what agentic ai really means for users. I can see how an immediate barrier for deeper LLM experiences is access to user data. So will agents just be for integrating your service into a user’s auth/cloud/filesystem? a crutch for interoperability?

Collaboratively plan all your last minute Christmas todos with your significant other and tinytalkingtodos.com. It's a local-first voice assistant with a focus on privacy and families.

I’ll be trying Gemini 2 for code gen soon in my app, specifically to test its visual understanding. I’ve been pretty happy Claude Sonnet 3.5, especially because my current workflow is just to one shot via the users prompt. But excited to see where multiple fast iterations on a workflow gets us.

I’ll be trying Gemini 2 for code gen soon in my app, specifically to test its visual understanding. I’ve been pretty happy Claude Sonnet 3.5, especially because my current workflow is just to one shot via the users prompt. But excited to see where multiple fast iterations on a workflow gets us.

Here's a dev talk I gave on building voice agents in the browser at SydJS. In the talk I try and breakdown the different services you might use, and show how users can BYO keys, for a client based approach. I also demo using OpenAI's realtime websocket approach.

Is there any up to date advice on tool call responses? Seems like a powerful concept, but unclear how to approach it. Atm I have acknowledgment, action_summary, result and next_steps properties – though I can't recall where I found this props.

result shows the new db row { success: true, id: rowId, item: row } and next_steps motivates towards over functions -> "You can now edit this item with updateTodo or remove with deleteTodo." It feels a bit onerous, but may key to getting good alignment with poorer reasoning models like gpt-4o-mini

Is there any up to date advice on tool call responses? Seems like a powerful concept, but unclear how to approach it. Atm I have acknowledgment, action_summary, result and next_steps properties – though I can't recall where I found this props.