Post

If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on?

1. Two metrics that come to mind • Faithfulness: Grounding of answers in document's content. Not to be confused with correctness—an answer can be correct (based on updated information) but not faithful to the document. Sub-metric: Precision of citations

• Evaluate separately: Faithfulness = binary label -> LLM-evaluator; Helpfulness = pairwise comparisons -> reward model

2. How to build robust evals • Use LLMs to generate questions from the text • Evals should evaluate positional robustness (i.e., have questions at the beginning, middle, and end of text)

3. Potential challenges • Open-ended questions may have no single correct answer, making reference-based evals trickly. For example: What is the theme of this novel? • Questions should be representative of prod traffic, with a mix of factual, inferential, summarization, definitional questions

I like this fusion of “usefulness” (which is a better way of describing what “relevance” tries to get at imo) and “completeness.” They both go together but it also sidesteps some challenges with “completeness” as a metric in certain domains (preexisting use of it creates baggage)

thank for for the feedback! it’s an initial attempt at defining helpfulness as a combination of both and can be improved. all ideas appreciated