SourceChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
The paper evaluates Llama-3.1-instruct 8b and 70b on two variants of the Needle-in-a-Haystack test up to 128k context. With the harder needle ('the best thing to do in san francisco is eat a sandwich and sit in dolores park on a sunny day'), neither model achieves reliable retrieval across depths and positions. With the easier passkey needle ('the pass key is 385243. remember it. 385243 is the pass key'), both models pass. This indicates a threshold effect where the models can retrieve short, distinctive tokens but fail on longer, less distinctive sentences.