Thorin

tmtabor.ioA hallucinated module, a backfiring RAG pipeline and the MCP server that fixed it — Thorin TaborHow an eval suite exposed a RAG pipeline that hurt GenePattern Copilot more than it helped, and what finally fixed it.

15 Sept

With GenePattern Copilot we found that a non-augmented model initially beat our RAG assistant. The failure was naive chunking splitting Q&A pairs. The fix was preprocessing with LLM fact extraction, ensuring every vector DB row was an atomic statement. Read the blog post below. #RAG #AgenticAI

The keynotes at #AGNTCon so far have been really stellar.

The community bash at #AGNTCon is a nice welcome, but it's loud.

Landed in San Jose for #AGNTCon + #MCPCon. I should have packed an umbrella.

Exposing a monolithic API through a single Model Context Protocol (MCP) server degrades model reasoning. Because the protocol lacks progressive disclosure, overwhelming an LLM with excessive tool descriptions dilutes context signal, inflates input tokens and confuses smaller models. #MCP

Finally, the server's state management should minimize context load. Instead of returning massive raw text blocks that crash the context window, it's better to use transient storage, returning a lightweight artifact reference ID that subsequent tools can consume locally. #Backend

What MCP server implementation patterns are working for you? #AgenticAI

For a large API, wrapping every endpoint is rarely good design. Each tool needs a plain-language description of exactly what it does and when to use it -- description quality directly affects performance. An eval framework should benchmark variations. #PromptEngineering

Reliable agentic tools also require smart error handling. Upstream API errors should be transformed into clean semantic instructions. Returning explicit recovery hints creates a feedback loop, letting the model self-correct and retry without stalling. #AIAgents

While implementing the code that calls endpoints is mostly boilerplate, the trickier aspects are tool context and scoping, both of which should be approached as distinct user stories. #SoftwareEngineering

The lessons from building the GenePattern Copilot #RAG assistant: There is no one-size-fits-all chunking strategy. Skipping proper evaluation means spending weeks tuning prompts on a fundamentally broken pipeline. #AgenticAI #LLMs

github.comGitHub - tmtabor/job-agent: Daily job-scanning agent: fetches postings from multiple sources, filters/scores against a candidate profile with an LLM, emails a ranked digest. Runs on GitHub Actions.Daily job-scanning agent: fetches postings from multiple sources, filters/scores against a candidate profile with an LLM, emails a ranked digest. Runs on GitHub Actions. - tmtabor/job-agent

My job hunting agent scores postings against a profile (role, comp, location, dealbreakers) and uses an LLM to evaluate fit and gaps. It also researches companies, caches results and tracks careers pages for promising organizations. BSD-3-Clause licensed. #AgenticAI #LLMs #AIAgents

The gap between a demo agent and a production agent isn't the model. It's everything around it. #AIEngineering

A strong model in a weak harness will consistently underperform a weaker model in a strong one. #Reliability

Build accordingly. #DevTools

The gap between a demo agent and a production agent isn't the model. It's everything around it. #AIEngineering

Don't underestimate state management. Most agent failures aren't model failures. They're state coherence failures. The harness controls what the model sees and when, and that decision matters. #SoftwareArchitecture

You can't improve what you can't see. Tracing, logging, replay: this is where the harness earns its keep. Observability isn't a nice-to-have. It's how you actually get better. #Observability

The gap between a demo agent and a production agent isn't the model. It's everything around it. #AIEngineering

Clean execution environments matter more than people think. Agents need isolated, deterministic contexts to behave reliably. Leftover state, dirty environments, unexpected side effects: these silently corrupt agent behavior in ways that are hard to trace. #Backend

Tool design shapes model behavior. Tools aren't neutral scaffolding. The schema, the naming, the error responses: all of it actively steers what the model does. A poorly designed tool will degrade even a strong model. #PromptEngineering

The gap between a demo agent and a production agent isn't the model. It's everything around it. #AIEngineering

After building agents that have run in production, I've become convinced the harness matters as much as, or maybe more than, the model itself. What separates a brittle demo from something that holds up under real conditions is the quality of everything built around the model. #SoftwareEngineering

So what does that mean in practice? #SystemDesign

The gap between a demo agent and a production agent isn't the model. It's everything around it. #AIEngineering

Here's a framing I keep coming back to: Agent = Model + Harness. #AIAgents

The model gets the attention. The harness does the work. #AgenticAI

Intel markets Gaudi as the open alternative to CUDA. This week, three new open-weight models dropped that I can't run on it. #AIHardware

This is no longer a temporary inconvenience. Intel archived the open-source SynapseAI user-space driver in late 2025, and the ecosystem is mid-transition, pivoting toward vLLM-Gaudi as the long-term inference path, with developers caught in the gap. #OpenSourceAI

The true cost of breaking free from the CUDA monopoly is rarely measured in raw compute power, but rather is paid in software engineering velocity. #CUDA

Intel markets Gaudi as the open alternative to CUDA. This week, three new open-weight models dropped that I can't run on it. #AIHardware

This version mismatch forces a reliance on the optimum-habana library fork rather than vanilla Hugging Face scripts. Testing a new architecture means waiting for downstream teams to manually code, optimize and upstream custom HPU kernels. #MLOps

Trying to bypass this limitation by force-upgrading Python or PyTorch inside Habana's pre-built Docker containers results in dependency hell. #Python

Intel markets Gaudi as the open alternative to CUDA. This week, three new open-weight models dropped that I can't run on it. #AIHardware

Having a compute allocation at SDSC, I want to test them there, but the lag time for new models in the Habana environment makes day-one deployment incredibly difficult. #HPC

That lag is structural. Running on Gaudi requires the specialized Habana stack, locked to specific precompiled versions of Python and PyTorch. The stable Habana release for Gaudi 2 supports only Python 3.10 -- but the latest Transformers updates strictly demand Python 3.11+. #PyTorch

Intel markets Gaudi as the open alternative to CUDA. This week, three new open-weight models dropped that I can't run on it. #AIHardware

Cohere's North Mini Code, Google's DiffusionGemma, and JetBrains' Mellum2 each represent a different frontier: agentic coding, diffusion-based text generation, and efficient MoE inference. All three are on Hugging Face today. None are accessible on the Gaudi nodes at SDSC. #HuggingFace

tmtabor.ioProjects — Thorin TaborProjects by Thorin Tabor — agentic AI systems, research software and open source developer tooling.

Need a local AI to draft content or triage GitHub issues? I've been building tools for exactly that. Check out my updated projects page for open-source Agentic AI tooling, built with Python, Ollama and Pydantic AI. 👇 https://tmtabor.io/projects/ #AgenticAI #OpenSource #Python

Adding a RAG pipeline to GenePattern Copilot actually degraded performance, making the chatbot's answers worse than the baseline frontier model. #RAG

Ultimately, RAG performance is a data engineering problem rather than a retrieval framework problem. Clean data at ingestion outweighs complex orchestration later in the pipeline. #GenAI

What chunking strategies are proving successful for your data sources? #DataScience

Adding a RAG pipeline to GenePattern Copilot actually degraded performance, making the chatbot's answers worse than the baseline frontier model. #RAG

The solution was to restructure the data pipeline. Before vector database ingestion, an LLM preprocesses the source corpora to extract standalone, atomic statements of fact. Each independent fact becomes its own chunk. #PromptEngineering

Isolating facts prevents text within chunks from referencing external, missing context. Resolving these dependencies at the ingestion layer ensures vector matching captures precise semantic intent, drastically improving retrieval quality. #NLP

Adding a RAG pipeline to GenePattern Copilot actually degraded performance, making the chatbot's answers worse than the baseline frontier model. #RAG

However, evals showed that RAG degraded output quality compared to the standalone model. Observability exposed a severe signal-to-noise problem: the retrieval layer was actively corrupting the model's reasoning with irrelevant or fragmented context. #LLM

The root cause was a naive, size-based chunking strategy. Fixed token counts frequently bisect paragraphs, break context windows and separate solutions from their original questions -- especially destructive for forum threads. #MachineLearning

Adding a RAG pipeline to GenePattern Copilot actually degraded performance, making the chatbot's answers worse than the baseline frontier model. #RAG

A decade of messy help forum data had effectively turned the vector database into a high-volume noise generator that corrupted the model's reasoning. Reversing this required abandoning traditional token-count chunking for an LLM-driven atomic fact extraction strategy. #DataEngineering

The platform answers complex bioinformatics and GenePattern system questions. The initial architecture used LangChain, LangGraph and ChromaDB to query a decade of help forum answers, notebooks, platform docs and genomics tool specs. #Bioinformatics

My job-hunting agent pulls daily postings from company career pages (Greenhouse, Lever, Ashby) and aggregators (Adzuna, LinkedIn, Indeed, Glassdoor). The initial filtering for mismatches is done cheaply, requiring no LLM calls. #AgenticAI #MultiAgentSystems #AIAgents

GenePattern Copilot, an #AgenticAI #RAG assistant, initially hallucinated modules. To fix this, we added a tool to fetch the real module list. This evolved into an #MCP server wrapping the GenePattern REST API, turning the Q&A bot into a full agent that could launch jobs and retrieve results.

tmtabor.ioProjects — Thorin TaborProjects by Thorin Tabor — agentic AI systems, research software and open source developer tooling.

I've updated my portfolio with my latest agentic AI, including a Python agent template, a local multi-agent content drafter via Ollama, and an LLM issue triager for GitHub. Check out what I'm building: https://tmtabor.io/projects/ #AgenticAI #Python

Jev is a new model gaining traction. It's fast and cheap, designed to classify unstructured data instead of generating text. This shift back to non-generative methods is important for certain NLP tasks. #LLMs #AgenticAI #NLP

Unlike traditional fixed-label classifiers, its label set is specified at inference time rather than training time. This structure gives it more flexibility. #LLMs #NLP #MachineLearning