Headroom vs Microsoft LLMLingua: выбор фреймворка сжатия контекста для LLM-agent

Microsoft LLMLingua (EMNLP 2023, LongLLMLingua ACL 2024) — академический эталон сжатия промптов по perplexity—до 20×. Headroom (chopratejas/headroom) — инженерный путь: локальный proxy 8787, обратимый CCR, маршрутизация по типу, интеграции agent включая Runbook OpenClaw + Headroom.

Команды always-on LLM agent выбирают: PromptCompressor в Python или ANTHROPIC_BASE_URL на Headroom. Матрица решений: perplexity pruning vs proxy CCR, headroom proxy --llmlingua, 8-шаговый runbook.

Модели сжатия рядом

LLMLingua family (Microsoft Research)
  Small LM (GPT-2-small / LLaMA-7B class)
    → token perplexity p(token | context)
    → drop low-perplexity tokens (budget controller + iterative passes)
    → optional distribution alignment to target LLM
  LongLLMLingua adds:
    → contrastive perplexity p(question | document)
    → document reorder ("lost in the middle" mitigation)
    → coarse-to-fine compression for RAG stacks

Headroom (engineering stack)
  Incoming messages (tools, logs, JSON, code, chat)
    → CacheAligner (KV-cache-friendly prefixes)
    → ContentRouter
         ├─ SmartCrusher (JSON arrays/objects)
         ├─ CodeCompressor (AST: Py/JS/Go/Rust/…)
         └─ Kompress-base (agentic prose, HF model)
    → CCR stores originals locally; model calls headroom_retrieve
    → Proxy forwards /v1/messages to Anthropic/OpenAI/Bedrock

Цитата: LLMLingua удаляет по SLM perplexity; Headroom маршрутизирует по типу контента с обратимым CCR.

Матрица решений: академия vs инженерия

Сценарий A: RAG по длинным документам

Profile: Legal, support, or internal wiki QA—10–50 PDFs chunked into a single prompt, user question appended.

LLMLingua fit: LongLLMLingua was built for this. Use condition_in_question="after_condition", reorder_context="sort", rate=0.55 per Microsoft's examples. Contrastive perplexity beats vanilla when documents are noisy.

Headroom fit: Strong when chunks mix JSON metadata + prose (ticket exports, CI logs in KB). Proxy mode compresses without rewriting LangChain/LlamaIndex glue.

If X, do Y: If bottleneck is multi-document ordering and lost-in-the-middle, do prototype LongLLMLingua first. If bottleneck is heterogeneous tool+json context in an agent loop, do prototype Headroom proxy first.

Сценарий B: always-on code/ops agent (класс OpenClaw)

Profile: Nightly repo audits, MCP stdio tools, megabyte linter JSON—context grows every turn.

LLMLingua fit: Works as a pre-step if you batch-compress static prompts offline. Per-request compress_prompt() adds SLM inference latency on every gateway call unless cached.

Headroom fit: Designed for this shape—documented OpenClaw plugin, /stats Prometheus metrics, headroom mcp install. See Runbook OpenClaw + Headroom for LaunchAgent wiring.

If X, do Y: If you need drop-in proxy on macOS launchd gateways, do Headroom. If you publish research pipelines with frozen prompts, do LLMLingua in the ingest stage.

Сценарий C: гибридный стек

Headroom supports headroom proxy --llmlingua—Microsoft's perplexity compressor as an optional deeper pass after structural crushers. Trade-off: ~2 GB extra dependencies, 10–30s cold start per Headroom proxy docs.

If X, do Y: If eval shows SmartCrusher leaves >30% fat JSON, do enable --llmlingua on a 24 GB Mac mini M4 only. If latency SLO < 2s p95, do stay on structural crushers + CCR without ML pass.

Рекомендуемый путь

• If you optimize ACL-style RAG benchmarks, do start with LongLLMLingua PromptCompressor and Microsoft rate sweeps.
• If you operate OpenClaw / Claude Code / Cursor fleets, do start with Headroom proxy and measure /stats-history for seven nights.
• If compliance requires verbatim audit trails, do prefer Headroom CCR over lossy perplexity-only pipelines.
• If you need KV-cache compression research, do evaluate LLMLingua-2 and Microsoft's cache line per Microsoft Research.
• If neither hits 40% savings on your traces, do fix prompt design first—compression cannot rescue redundant tool round-trips.

Восьмишаговый runbook оценки

1. Заморозить золотой набор промптов

Capture N≥20 real agent turns: tool JSON, stack traces, instructions. Store SHA-256 per fixture under ~/compression-eval/fixtures/.

2. Базовый подсчёт токенов (несжатый)

Record input tokens from provider dashboard or tiktoken for each fixture.

3. Запустить рукав LLMLingua / LongLLMLingua

pip install llmlingua

from llmlingua import PromptCompressor
pc = PromptCompressor(model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank")
out = pc.compress_prompt(prompt_list, question=question, rate=0.55,
    condition_in_question="after_condition", reorder_context="sort",
    rank_method="longllmlingua")
compressed = out["compressed_prompt"]

Log origin_tokens, compressed_tokens, wall-clock ms.

4. Запустить рукав Headroom proxy

pip install "headroom-ai[proxy]"
headroom proxy --port 8787 --log-file ~/.headroom/eval.jsonl

POST fixtures through /v1/compress or route live agent traffic; read tokens_saved from /stats.

5. Опциональный гибридный рукав

headroom proxy --port 8788 --llmlingua --llmlingua-rate 0.3

Compare p95 latency vs savings uplift.

6. Контроль качества (тот же downstream LLM)

Re-run each compressed fixture through your production model with identical temperature. Score: exact-match for structured fields, LLM-judge for summaries, human spot-check 5%.

7. Регрессионный набор agent

For OpenClaw operators: replay nightly audit job with each arm; compare finding counts and false-negative rate on known seeded bugs.

8. Выбрать победителя по классу нагрузки

Document: RAG ingest → LongLLMLingua, live gateway → Headroom proxy, max compression lab → hybrid—publish internally with token $/month math.

Устранение неполадок

LLMLingua collapsed instruction-following

Симптом: Compressed prompt drops negation or JSON keys.

Исправление: Lower rate (0.55 → 0.75). Use budget controller to exempt instruction block. Compare LLMLingua-2 per Microsoft Research.

Headroom proxy saves tokens but agent misses line numbers

Симптом: Audit agent cites wrong file:line.

Исправление: Instruct model to headroom_retrieve before closing findings; set x-headroom-bypass: true on one repro. Narrow SmartCrusher if schema keys stripped.

Both arms slower than uncompressed

Симптом: p95 latency > 3× baseline.

Исправление: LLMLingua—cache SLM on GPU/MPS, batch offline. Headroom—disable --llmlingua, keep structural crushers only; co-locate proxy on same host as gateway.

Частые вопросы

Headroom — форк LLMLingua?

Нет. Headroom — независимый проект Apache-2.0 с опциональным --llmlingua. По умолчанию SmartCrusher, CodeCompressor и Kompress-base, не только perplexity pruning.

Когда perplexity pruning лучше content-type crushers?

Когда промпты — однородный естественный язык и LongLLMLingua настроен с якорем вопроса. Гетерогенный вывод agent-инструментов обычно выигрывает у роутера Headroom.

LongLLMLingua в OpenClaw без Headroom?

Да—предсжимайте статический контекст в skills через PromptCompressor. Теряется прозрачность proxy и CCR без своего retrieval.

LLMLingua-2 vs LongLLMLingua?

LLMLingua-2 — классификация токенов BERT-энкодером; по отчётам Microsoft в 3–6× быстрее итеративной perplexity. Headroom может наслоить через --llmlingua; скорость vs SLO оценивайте отдельно.

Что одобрить финансам для флота ночных аудитов 20 репозиториев?

Восьмишаговая оценка на одном репо в неделю 1. Доминирует tool JSON → Headroom proxy + OpenClaw; статический doc RAG → LongLLMLingua может выиграть по качеству/доллару.