<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>RAG &#8211; Od Informacji do Wiedzy</title>
	<atom:link href="https://profind.pl/en/tag/rag-en/feed/" rel="self" type="application/rss+xml" />
	<link>https://profind.pl</link>
	<description>Blog o informacjach na temat informacji i wiedzy</description>
	<lastBuildDate>Wed, 05 Nov 2025 12:31:06 +0000</lastBuildDate>
	<language>en-GB</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>
	<item>
		<title>Taming Long Context in LLMs: 3 Problems, 1 Cohesive Set of Strategies</title>
		<link>https://profind.pl/en/2025/11/04/taming-long-context-in-llms-3-problems-1-cohesive-set-of-strategies/</link>
					<comments>https://profind.pl/en/2025/11/04/taming-long-context-in-llms-3-problems-1-cohesive-set-of-strategies/#respond</comments>
		
		<dc:creator><![CDATA[Marcin]]></dc:creator>
		<pubDate>Tue, 04 Nov 2025 19:13:28 +0000</pubDate>
				<category><![CDATA[Research diary]]></category>
		<category><![CDATA[context compression]]></category>
		<category><![CDATA[cross-encoder reranking]]></category>
		<category><![CDATA[LLMLingua / LongLLMLingua]]></category>
		<category><![CDATA[long context (LLM)]]></category>
		<category><![CDATA[LongRoPE]]></category>
		<category><![CDATA[Lost in the Middle]]></category>
		<category><![CDATA[Ms-PoE (Found in the Middle)]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[RULER / OneRULER evaluation]]></category>
		<category><![CDATA[streaming attention (attention sinks)]]></category>
		<guid isPermaLink="false">https://profind.pl/?p=1921</guid>

					<description><![CDATA[In practical LLM systems (QA, analytics, assistants, agentic RAG), three phenomena routinely degrade quality: (1) Lost in the Middle — accuracy drops when the key evidence sits in the middle of a long prompt; (2) Distracting prompts — a few “tempting” sentences derail reasoning; (3) Very large contexts → performance drop — despite advertised 32k+ [&#8230;]]]></description>
										<content:encoded><![CDATA[<article>
<p style="text-align: justify;">In practical LLM systems (QA, analytics, assistants, agentic RAG), three phenomena routinely degrade quality: (1) <em>Lost in the Middle</em> — accuracy drops when the key evidence sits in the middle of a long prompt; (2) <em>Distracting prompts</em> — a few “tempting” sentences derail reasoning; (3) <em>Very large contexts → performance drop</em> — despite advertised 32k+ windows, results and stability degrade. Below: why this happens, what works “right now,” what to implement in the model/pipeline, and how to measure it rigorously.</p>
<hr />
<h2>TL;DR for the impatient</h2>
<ul>
<li style="text-align: justify;"><strong>Instead of stuffing everything into the prompt</strong>: retrieval → cross-encoder reranking → compression → <em>extreme ordering</em> (most important at the beginning <strong>and</strong> end).</li>
<li style="text-align: justify;"><strong>Limit distraction</strong> with a simple instruction + answer format, few-shot with “noise,” <em>self-consistency</em>, and <em>gating/abstention</em> (<em>NO-RESPONSE</em>) at the passage level.</li>
<li style="text-align: justify;"><strong>Stabilise long context</strong> via position scaling (LongRoPE/YaRN), a training regime for long sequences (ProLong), test-time adaptation (LIFT), <em>streaming attention</em> with sink tokens and/or external memory.</li>
<li style="text-align: justify;"><strong>Measure smartly</strong>: not only “needle-in-a-haystack.” Use RULER/ONERULER (also multilingual), multi-needle tests, and real tasks with source citation.</li>
</ul>
</section>
<p><span id="more-1921"></span></p>
<section>
<h2>1. Lost in the Middle — diagnosis and fixes</h2>
<h3>Why it happens</h3>
<p style="text-align: justify;">Models with popular positional schemes (e.g., RoPE and position extrapolation methods) exhibit a <strong>positional bias</strong>: they usually “retain” the beginning and the end of a sequence better, while the middle is often the weakest. When the needle lands in the middle, accuracy drops — even in “long-context” models.</p>
<h3>What works immediately (no training)</h3>
<ol>
<li style="text-align: justify;"><strong>Extreme ordering</strong>: after reranking, place the most important fragments at the <strong>beginning</strong> and <strong>end</strong> of context (use the middle for supporting material).</li>
<li style="text-align: justify;"><strong>Iterative retrieval</strong> (Self-RAG/FLARE): don’t load everything at once — generate in steps and pull missing facts on demand.</li>
<li style="text-align: justify;"><strong>Hierarchical summaries</strong> (map → reduce): condense sections to 1–2 sentences and compose a “map” of arguments.</li>
</ol>
<h3>What works at the model/pipeline level</h3>
<ul>
<li style="text-align: justify;"><strong>Cross-encoder reranking</strong> (cross-encoder/LLM ranker) before arranging context.</li>
<li style="text-align: justify;"><strong>Better positional handling</strong>: LongRoPE/YaRN/PI — more stable windows; and <strong>Ms-PoE</strong> (<em>Found in the Middle</em>) targeted at improving the “middle.”</li>
<li style="text-align: justify;"><strong>SFT with positional debiasing</strong>: rotation/shuffling of order, needles placed at different positions; generally — positional augmentation aligned with the latest long-context guidance.</li>
</ul>
<blockquote><p><strong>Prompt pattern (minimal)</strong></p>
<pre escaped="true"><code>TASK: {question}
RULES: Use only fragments marked [IMPORTANT]. Omit [ADDITIONAL].
FORMAT:
[EVIDENCE] bullet points with a citation --&gt; {passage_id}
[ANSWER] concise conclusion
</code></pre>
</blockquote>
</section>
<hr />
<section>
<h2>2. Distracting prompts and the “tempting” noise</h2>
<h3>Why it happens</h3>
<p style="text-align: justify;">LLMs are sensitive to distractors that are semantically similar to the correct answer. A single irrelevant sentence can steer reasoning off course.</p>
<h3>Tactics without training</h3>
<ol>
<li style="text-align: justify;"><strong>Instruction + answer format</strong>: ask to list <em>evidence/relevant sentences</em> first, then provide the conclusion.</li>
<li style="text-align: justify;"><strong>Few-shot with distractors</strong>: in examples, show the step “filter out irrelevant → solve.”</li>
<li style="text-align: justify;"><strong>Self-Consistency</strong>: several independent reasoning trajectories and majority voting.</li>
<li style="text-align: justify;"><strong>Passage-level gating/abstention</strong>: test each fragment — if it doesn’t contain necessary information, the model should output <code>NO-RESPONSE</code>; apply this filter before final generation.</li>
</ol>
<h3>Noise-robust RAG pipeline</h3>
<p style="text-align: justify;"><em>Retriever → Cross-Encoder Rerank → NLI/entailment filters → Compression (LongLLMLingua/LLMLingua-2) → Extreme ordering → Gating/abstention → Generation with self-consistency.</em></p>
<blockquote><p>Prompt pattern (passage gating)</p>
<pre escaped="true"><code>For each passage do:
1) Does this passage contain information required to answer "{question}"?
2) If NO, return: NO-RESPONSE | {id}
3) If YES, return: EVIDENCE | {id} | {short_quote}
</code></pre>
</blockquote>
</section>
<hr />
<section>
<h2>3. Very large contexts and performance degradation</h2>
<h3>Where degradation comes from</h3>
<ul>
<li style="text-align: justify;"><strong>Positions</strong>: extrapolation beyond the training distribution.</li>
<li style="text-align: justify;"><strong>Optimisation</strong>: long sequences strain the KV cache and attention stability.</li>
<li style="text-align: justify;"><strong>Selection</strong>: the practically useful portion is smaller than the declared window; the rest often is noise.</li>
</ul>
<h3>Layers of defence (from practice to architecture)</h3>
<h4>A. Practice “by tomorrow”</h4>
<ul>
<li style="text-align: justify;"><strong>Context compression</strong> (LongLLMLingua/LLMLingua-2): 2–6× shorter inputs; in many tasks <em>no loss</em>, sometimes <em>improved</em> quality; reported speedups ~1.4–2.9× depending on the task.</li>
<li style="text-align: justify;"><strong>Adaptive retrieval</strong> (Self-RAG/FLARE): “retrieve only when needed,” with self-reflection on sources.</li>
<li style="text-align: justify;"><strong>Positional budget</strong>: aim for ~20–40% of the window for hard evidence; leave the rest for the model’s own reasoning <em>(heuristic — A/B test on your data)</em>.</li>
</ul>
<h4>B. Model and inference</h4>
<ul>
<li style="text-align: justify;"><strong>LongRoPE/YaRN/PI</strong>: stabilisation and extension of 32k+ windows.</li>
<li style="text-align: justify;"><strong>ProLong</strong>: continued pre-training + SFT for long sequences (models up to 512k).</li>
<li style="text-align: justify;"><strong>LIFT (test-time adaptation)</strong>: adapter/learning with long input at test time; improves long-context understanding.</li>
<li style="text-align: justify;"><strong>Streaming + attention sinks</strong>: a fixed pool of anchor tokens + sliding KV window for long streams.</li>
<li style="text-align: justify;"><strong>External memory</strong>: LM-Infinite/InfLLM and related — cheaper than linear prompt growth.</li>
</ul>
<h4>C. Multilinguality</h4>
<ul>
<li style="text-align: justify;"><strong>Language alignment</strong> (instruction ≈ documents) or normalise to a single language.</li>
<li style="text-align: justify;"><strong>Per-language evaluation</strong> — gaps grow with sequence length; use multilingual benchmarks.</li>
</ul>
</section>
<hr />
<section>
<h2>A cohesive production pipeline (sketch)</h2>
<ol>
<li style="text-align: justify;"><strong>Retrieval</strong>: <code>BM25 + bi-encoder (dense)</code>; take top-50.</li>
<li style="text-align: justify;"><strong>Reranking</strong>: cross-encoder/LLM-ranker → top-k (e.g., 8–12).</li>
<li style="text-align: justify;"><strong>Compression</strong>: LongLLMLingua-2/LongLLMLingua targeting ~3–5k tokens.</li>
<li style="text-align: justify;"><strong>Ordering</strong>: <em>extreme ordering</em> — most important at the <strong>beginning and end</strong>; the middle = support.</li>
<li style="text-align: justify;"><strong>Gating</strong>: quick <em>NO-RESPONSE</em> pass over passages.</li>
<li style="text-align: justify;"><strong>Generation</strong>: <em>self-consistency</em> (n=3–7) + critic/reflection over citations.</li>
<li style="text-align: justify;"><strong>64k+ streams</strong>: enable <em>streaming attention</em> with sink tokens and a KV budget.</li>
</ol>
<pre escaped="true"><code class="language-python"># Pseudocode
docs = retriever(q, k=50)
scored = cross_encoder.rank(q, docs)
sel = topk(scored, 10)
compressed = longllmlingua(sel, target_tokens=4000)
ordered = extreme_ordering(compressed)   # front + tail boost
kept = passage_gating(ordered, rule="NO-RESPONSE")
answer = llm.generate(kept, self_consistency=5, cite_sources=True)
</code></pre>
</section>
<hr />
<section>
<h2>How to measure progress rigorously</h2>
<h3>Test suite</h3>
<ul>
<li style="text-align: justify;"><strong>RULER / ONERULER</strong>: lengths 8k→128k, multiple needles, language variants (retrieval, tracking, aggregation).</li>
<li style="text-align: justify;"><strong>“Needle-in-the-Haystack” on steroids</strong>: multiple needles + “hard negatives.”</li>
<li style="text-align: justify;"><strong>Real tasks</strong>: QA/analyses with <em>faithfulness</em> scoring (does it cite the correct sources?).</li>
</ul>
<h3>Metrics</h3>
<ul>
<li style="text-align: justify;"><em>Exact/EM, F1, citation precision/recall</em>, latency, tokens/query cost, % of “abstain” answers.</li>
<li style="text-align: justify;">Curves for <strong>quality vs. length</strong> (8k, 16k, 32k, 64k…) and <strong>quality vs. context budget</strong>.</li>
</ul>
<h3>Simple A/B plan</h3>
<ol>
<li style="text-align: justify;">Dataset of ~200–500 questions (min. 3 domains, 2 languages).</li>
<li style="text-align: justify;">Conditions: baseline, +reranking, +compression, +ordering, +gating, +self-consistency.</li>
<li style="text-align: justify;">Error analysis: separately “middle of context,” “distractors,” “overfill.”</li>
<li style="text-align: justify;">Report: table with cost and quality gains per condition.</li>
</ol>
</section>
<hr />
<section>
<h2>Implementation checklists</h2>
<h3>Lost in the Middle</h3>
<ul>
<li>Cross-encoder reranking before inserting into the prompt.</li>
<li>Extreme ordering (front+tail).</li>
<li>Hierarchical summaries (map→reduce).</li>
<li>If possible: LongRoPE/YaRN/PI + Ms-PoE; positional augmentation during SFT.</li>
</ul>
<h3>Distracting prompts</h3>
<ul>
<li>Instruction “use only necessary information” + format with [EVIDENCE]→[ANS.].</li>
<li>Few-shot with noise and filtering.</li>
<li>Gating/abstention (<code>NO-RESPONSE</code>) at passage level.</li>
<li>Self-consistency (n≥3) for sensitive tasks.</li>
</ul>
<h3>Very large contexts</h3>
<ul>
<li style="text-align: justify;">Compression (LongLLMLingua/LLMLingua-2) before feeding the LLM.</li>
<li style="text-align: justify;">Streaming attention + sink tokens for streams.</li>
<li style="text-align: justify;">External memory instead of “everything in the prompt.”</li>
<li style="text-align: justify;">ProLong/LIFT when you control model/inference.</li>
</ul>
</section>
<hr />
<section>
<h2>Common anti-patterns (and how to fix them)</h2>
<ul>
<li style="text-align: justify;"><strong>“I’ll add more context; it will be better”</strong> → <em>No</em>. First select/compress, then order.</li>
<li style="text-align: justify;"><strong>“We’ll add 20 few-shot examples”</strong> → A shorter, well-curated pattern + format is often more robust.</li>
<li style="text-align: justify;"><strong>“Since the window is 128k, we’ll use 120k”</strong> → Practically effective is often 20–40% of the window. Keep the rest in an index/memory <em>(heuristic → verify via A/B)</em>.</li>
</ul>
</section>
<hr />
<section>
<h2>To wrap up</h2>
<p style="text-align: justify;">Long context is tempting, but not free. The best results come from <em>selection and compression</em>, conscious handling of positional bias (front+tail), and a pipeline that <strong>knows when to be silent</strong> (gating/abstention) and <strong>when to pull more sources</strong> (iterative retrieval). Architectural and training changes can add extra points, but first squeeze the most out of the application layer.</p>
</section>
<section>
<h2 style="text-align: justify;">References and selected sources</h2>
<ol>
<li style="text-align: justify;">Liu, N. F., et al. (2023/2024). <a href="https://arxiv.org/abs/2307.03172" target="_blank" rel="nofollow noopener">Lost in the Middle: How Language Models Use Long Contexts</a> / TACL version: <a href="https://aclanthology.org/2024.tacl-1.9/" target="_blank" rel="nofollow noopener">PDF</a>.</li>
<li style="text-align: justify;">Shi, F., et al. (2023). <a href="https://arxiv.org/abs/2302.00093" target="_blank" rel="nofollow noopener">Large Language Models Can Be Easily Distracted by Irrelevant Context</a> (ICLR). Alt: <a href="https://dl.acm.org/doi/10.5555/3618408.3619699" target="_blank" rel="nofollow noopener">publisher version</a>.</li>
<li style="text-align: justify;">Hsieh, C.-P., et al. (2024). <a href="https://arxiv.org/abs/2404.06654" target="_blank" rel="nofollow noopener">RULER: What’s the Real Context Size of Your Long-Context Language Models?</a> (+ <a href="https://github.com/NVIDIA/RULER" target="_blank" rel="nofollow noopener">repo</a>).</li>
<li style="text-align: justify;">Kim, Y., et al. (2025). <a href="https://arxiv.org/abs/2503.01996" target="_blank" rel="nofollow noopener">OneRULER: Benchmarking multilingual long-context language models</a> (OpenReview: <a href="https://openreview.net/forum?id=3vxxB3Ar9r" target="_blank" rel="nofollow noopener">page</a>).</li>
<li style="text-align: justify;">Hengle, A., et al. (2025). <a href="https://arxiv.org/abs/2504.12845" target="_blank" rel="nofollow noopener">Can LLMs reason over extended multilingual contexts?</a></li>
<li style="text-align: justify;">Ding, Y., et al. (2024). <a href="https://arxiv.org/abs/2402.13753" target="_blank" rel="nofollow noopener">LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens</a>.</li>
<li style="text-align: justify;">Peng, B., et al. (2023). <a href="https://arxiv.org/abs/2309.00071" target="_blank" rel="nofollow noopener">YaRN: Efficient Context Window Extension of Large Language Models</a> (OpenReview: <a href="https://openreview.net/forum?id=wHBfxhZu1u" target="_blank" rel="nofollow noopener">page</a>).</li>
<li style="text-align: justify;">Chen, S., et al. (2023). <a href="https://arxiv.org/abs/2306.15595" target="_blank" rel="nofollow noopener">Position Interpolation (PI)</a>.</li>
<li style="text-align: justify;">Gao, T., Wettig, A., Yen, H., Chen, D. (2024/2025). <a href="https://arxiv.org/abs/2410.02660" target="_blank" rel="nofollow noopener">How to Train Long-Context Language Models (Effectively)</a> (ACL Findings 2025: <a href="https://aclanthology.org/2025.acl-long.366.pdf" target="_blank" rel="nofollow noopener">PDF</a>; ProLong: <a href="https://github.com/princeton-nlp/ProLong" target="_blank" rel="nofollow noopener">repo</a>).</li>
<li style="text-align: justify;">Mao, Y., et al. (2024/2025). <a href="https://arxiv.org/abs/2412.13626" target="_blank" rel="nofollow noopener">LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning</a>.</li>
<li style="text-align: justify;">Xiao, G., et al. (2023/ICLR-2024). <a href="https://arxiv.org/abs/2309.17453" target="_blank" rel="nofollow noopener">Efficient Streaming Language Models with Attention Sinks (StreamingLLM)</a> (GitHub: <a href="https://github.com/mit-han-lab/streaming-llm" target="_blank" rel="nofollow noopener">repo</a>).</li>
<li style="text-align: justify;">Han, C., et al. (2024). <a href="https://aclanthology.org/2024.naacl-long.222/" target="_blank" rel="nofollow noopener">LM-Infinite: Zero-Shot Extreme Length Generalization for LLMs</a>.</li>
<li style="text-align: justify;">Xiao, C., et al. (2024). <a href="https://arxiv.org/abs/2402.04617" target="_blank" rel="nofollow noopener">InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory</a>.</li>
<li style="text-align: justify;">Jiang, H., et al. (2023). <a href="https://arxiv.org/abs/2310.05736" target="_blank" rel="nofollow noopener">LLMLingua</a>; Jiang, H., et al. (2024). <a href="https://arxiv.org/abs/2310.06839" target="_blank" rel="nofollow noopener">LongLLMLingua</a> (ACL-Long 2024: <a href="https://aclanthology.org/2024.acl-long.91.pdf" target="_blank" rel="nofollow noopener">PDF</a>); Pan, Z., et al. (2024). <a href="https://arxiv.org/abs/2403.12968" target="_blank" rel="nofollow noopener">LLMLingua-2</a> (MSR: <a href="https://www.microsoft.com/en-us/research/project/llmlingua/llmlingua-2/" target="_blank" rel="nofollow noopener">page</a>).</li>
<li style="text-align: justify;">Asai, A., et al. (2023). <a href="https://arxiv.org/abs/2310.11511" target="_blank" rel="nofollow noopener">Self-RAG: Learning to Retrieve, Generate, and Critique</a> (site: <a href="https://selfrag.github.io/" target="_blank" rel="nofollow noopener">selfrag.github.io</a>).</li>
<li style="text-align: justify;">Jiang, Z., et al. (2023). <a href="https://arxiv.org/abs/2305.06983" target="_blank" rel="nofollow noopener">FLARE: Active Retrieval Augmented Generation</a> (EMNLP 2023: <a href="https://aclanthology.org/2023.emnlp-main.495/" target="_blank" rel="nofollow noopener">page</a>).</li>
<li style="text-align: justify;">Zhang, Z., et al. (2024). <a href="https://arxiv.org/abs/2403.04797" target="_blank" rel="nofollow noopener">Found in the Middle: Plug-and-Play Positional Encoding (Ms-PoE)</a>.</li>
<li style="text-align: justify;">Amiraz, C., Cuconasu, F., Filice, S., Karnin, Z. (2025). <a href="https://arxiv.org/abs/2505.06914" target="_blank" rel="nofollow noopener">The Distracting Effect: Understanding Irrelevant Passages in RAG</a> (ACL 2025: <a href="https://aclanthology.org/2025.acl-long.892.pdf" target="_blank" rel="nofollow noopener">PDF</a>).</li>
<li style="text-align: justify;">(Reranking) Nogueira, R., Cho, K. (2019). <a href="https://arxiv.org/abs/1901.04085" target="_blank" rel="nofollow noopener">Passage Re-ranking with BERT</a>.</li>
<li style="text-align: justify;">(Dense late interaction) Santhanam, K., et al. (2022). <a href="https://arxiv.org/abs/2112.01488" target="_blank" rel="nofollow noopener">ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction</a> (NAACL 2022: <a href="https://aclanthology.org/2022.naacl-main.272.pdf" target="_blank" rel="nofollow noopener">PDF</a>).</li>
</ol>
</section>
</article>
]]></content:encoded>
					
					<wfw:commentRss>https://profind.pl/en/2025/11/04/taming-long-context-in-llms-3-problems-1-cohesive-set-of-strategies/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
	</channel>
</rss>
