Post

@simonwillison.net

This is a neat new variant on RAG - no vectors, not even full-text search, instead showing the model a header hierarchy and giving it a tool to read the relevant sections My notes here: simonwillison.net/2024/Dec/6/r...

simonwillison.netRoaming RAG – make the model find the answersNeat new RAG technique (with a snappy name) from John Berryman: > The big idea of Roaming RAG is to craft a simple LLM application so that the LLM assistant …

John Berryman@jnbrymn.bsky.social

Tired of wrestling with complex RAG pipelines? 🚀 Enter Roaming RAG: a simpler way to make your LLMs find answers in well-structured docs. No vector databases, no headaches—just rich, structured context. 👉 Read how it works: arcturus-labs.com/blog/2024/11...

arcturus-labs.comRoaming RAG – Make the Model Find the Answers - Arcturus LabsRoaming RAG offers a fresh take on Retrieval-Augmented Generation, letting LLMs navigate well-structured documents like a human—exploring outlines and diving into sections to find answers. Forget comp...

6/12/24, 3:04 am20 reposts160 likes

Jerome Etienne #AI

@jerome-etienne.bsky.social

I got similar problem, but my solution is more hierarchic: - tool: fileSearch(semanticQuery) => list of files - tool: fileTableOfContent(filePath) => toc in markdown in list of section - tool: getSection(filePath, sectionName) => section text i would appreciate your opinion. What do you think ?

@simonwillison.net

Seems like that would work well to me

@pleasemyfox.bsky.social

📌

@znerual.bsky.social

Sounds neat. What do you think about a hybrid approach where the heading themselves are embedding and used in a Rag like structure, but can be um collapsed to read in more appropriate content that might not have matched the embeddings

Interesting approach. I'd love to see what the results look like One downside to this is, I don't think it would scale with a large amount of documents but it could be great with smaller datasets

Exactly. Marginal cost of adding a document to pgvector is minuscule, where adding a bunch of tokens for headers to every query will start to add up fast

@mzaks.bsky.social

I +1 this answer. Factually we are moving offline preparation to online evaluation and inference. The online eval can lead to better results as it is specific to the context of the prompt at hand, but it needs to burn through lots of cycles. A good offline strategy (if possible) is more efficient.

Hey, that’s interesting! I have a step in a working model of mine that gives a list of concept documents the LLM can choose for inclusion, but have been wondering if there’s any additional utility there beyond RAG. What you’re describing is a more developed take though

@metaphorician.com

This seems like something an LLM could easily write as well Which would enable a kind of test time learning that persists across sessions Right? Am I misunderstanding something?

I remember some auto-GPT demos doing a basic version of this, but it had a tendency to write a lot of notes of basic and irrelevant stuff and never check on them. I think a big problem is that while its training has made it able to understand this setup, it hasn't been trained to do it effectively.

It doesn't inherently have a good handle on what types of things it should take notes on or how often it should do so or read its notes. Maybe showing it some particularly effective examples of it doing so in the prompt or fine-tuning could help.

@metaphorician.com

Do you think this sort of self-legibility is something that could (in principle) be trained for when training the base models? Maybe hard to do backprop for something so high level? My machine learning knowledge falls short

@simonwillison.net

I don't understand - what do you mean by test time learning?

@metaphorician.com

Must be the wrong term then! I just mean it could write things it learns about the world in the appropriate sections of an enormous nested text structure that it navigates by uncollapsing step by step In other words, it can learn without updating its base params – by conceptual world building kinda

@simonwillison.net

Oh right, so effectively have it update one big "memory" document for itself and then navigate that using the same headings trick as in roaming RAG

@metaphorician.com

I'm imagining a lookup mode with its own temporary context window that is flushed every time it finishes reading section indexes and taking notes of promising sections to uncollapse – each of which may open up new enormous indexes loaded into the temporary context window

@metaphorician.com

With a suitable ontology of uncollapsible headers that fits in the context window, it could read from and write to enormous knowledge bases, like Wikipedia-sized. And context lengths are already pretty big, so it wouldn't necessarily need many nested layers to get to the leaf nodes

@metaphorician.com

I wonder how well it would do at making such suitable ontologies of headers. If it would get in its own way by editing headers to be more narrowly useful than the information their sections contain, or something like that

@davidgasquez.com

Link to the original article seems broken in your post. I think it should be arcturus-labs.com/blog/2024/11.... Interesting approach!

arcturus-labs.comRoaming RAG – RAG without the Vector Database - Arcturus LabsRoaming RAG offers a fresh take on Retrieval-Augmented Generation, letting LLMs navigate well-structured documents like a human—exploring outlines and diving into sections to find answers. Forget comp...

@gravity7.bsky.social

Wouldn't it make sense for standardization and hierarchization of these ontologies within domains? Could LLM agents pass "header" requests to each other, and using standardized ontologies, essentially exchange SEO'd requests/prompts and responses?

@kyle.bricolage.io

heh, we worked our way backwards from the index to the table of contents