Ghost Search — MCP-native, Tor-routed search layer for RAG (seeking research collaborators)

sookoothaii · August 21, 2026, 9:28pm

I’m a researcher who got tired of paying 60% of my LLM budget for
search access. So I built Ghost Search. I’m opening it as a community
research project and looking for collaborators.

The Problem

Search is the bottleneck for RAG, not the LLM. A 7B model with good
retrieval beats a 400B model with bad retrieval. But building a search
layer that handles IP blocks, captcha walls, and cookie tracking
requires infrastructure most researchers don’t have.

I’m not the first to address this. OnionSearch (megadose, 1.7k stars),
darkdump (josh0xA), darker (saadejazz), and Robin (apurvsinghgautam)
all scrape onion search engines. LibreX (hnhx, 841 stars) is a
privacy-respecting meta-search with Tor and I2P support. Dark Watchdog
(IEEE 2025) uses RAG for dark web forum monitoring with a fine-tuned
BERT classifier. IntelShed (posted here on HF Forums) combines hybrid
BM25 + pgvector + cross-encoder reranking for OSINT.

So the building blocks exist. What’s missing is the combination that
makes it directly usable in an LLM agent workflow without paid APIs,
without persistence, and without a backend.

What I Built

Ghost Search is a Tor-routed, privacy-preserving meta-search aggregator
with 14 onion search engines, BM25 ranking, and an MCP server interface.
No API keys for the search layer, no per-query fees, no commercial
dependency.

What makes it different from the projects above:

MCP server, not just a CLI. 7 tools (ghost_search, ghost_health,
ghost_engines, ghost_reset_engine, ghost_classify, ghost_stix_export,
ghost_load_blacklist), stdio transport, works with any MCP-compatible
client. OnionSearch and darkdump are CLI tools. Robin has a Streamlit
UI. None expose an MCP interface. If you use Claude Desktop, VS Code,
or any MCP client, ghost_search drops into your workflow directly.
RAM-only, no persistence. No PostgreSQL, no Redis, no Elasticsearch.
Search state lives in memory and is purged on screen-off, process exit,
or panic wipe. darkscraper, voidaccess, and m-lally/dark-web-search all
persist to databases. Ghost doesn’t. This matters for the privacy use
case — there’s nothing to seize, nothing to subpoena.
Android app in F-Droid review. No backend, no accounts, no
telemetry. Links open in Tor Browser via intent. I’m not aware of
another onion search aggregator that targets F-Droid with a
RAM-only architecture.
BM25 ranking with engine agreement boost. Okapi BM25 (k1=1.2,
b=0.75) replaces heuristic frequency sorting. m-lally/dark-web-search
uses RediSearch BM25 but on a crawled index. Ghost ranks live
meta-search results — no crawl, no index, no storage.
STIX 2.1 export. OASIS-standard Observable bundles (url, ipv4-addr,
cryptocurrency-wallet, vulnerability, email-addr) with relationships.
voidaccess also supports STIX 2.1, but their pipeline is heavier
(13-stage, PostgreSQL, relationship graphs). Ghost’s export is
stateless — one tool call, one bundle, nothing stored.
LLM integration with Ollama fallback. NVIDIA NIM (Nemotron-3-Super-
120B) primary, Ollama (qwen3:8b) local fallback with 60s cooldown on
primary failure. Robin supports OpenAI/Claude/Gemini/Ollama but
requires you to send queries through a third-party API. Ghost’s
fallback means you can run the whole thing on a laptop with Ollama
and never touch a paid API.
Entity extraction is ReDoS-safe. BTC, XMR, PGP, email, .onion,
CVE, IPv4 — linear-time regex, bounded quantifiers. darkscraper
extracts similar entities but stores them in PostgreSQL. Ghost
extracts for display only, nothing persisted.
All filters OFF by default. Ahmia abuse blacklist, snippet
liability filter, STIX export — every feature is opt-in. Results
are marked, never removed. This was important for F-Droid acceptance.

180 tests, TypeScript, tsc clean.

Repo: https://codeberg.org/sookoothaii/ghost-search-mcp

What Already Exists vs. What’s Missing

There are public dark web datasets — Dizzy (32,555 onion domains with
category labels, arXiv 2209.07202), DUTA (26-class illegal activity
classification, EACL 2017), CoDA (10,000 documents for linguistic
analysis, NAACL 2022). These are classification datasets. They tell
you what a domain IS.

What doesn’t exist is a search ranking benchmark: “given query X,
which results are most relevant, and does BM25 rank them correctly?”
That’s the gap I want to fill. I can collect raw results through Tor.
I can’t annotate them alone and call it rigorous.

Who This Is For

I have a workstation — i9, RTX 3080 Ti, 16GB VRAM. I can run local
models, fine-tune, experiment. I use that hardware to build the
infrastructure.

But the infrastructure isn’t for me. It’s for the researcher in a
developing country with a laptop and no GPU. It’s for the student
who can’t afford GPT-5 search API calls. It’s for the journalist
who needs anonymous search without exposing their identity to a
commercial API provider.

Free, anonymous, secure, no paid dependency. That’s the design goal.
I build with my hardware so others can use it without hardware.

Where I Need Help

I’m one person. I can build infrastructure. I can’t do all of this:

Search ranking benchmark dataset — I’m building an annotated
dataset of onion search results: query, results, relevance scores.
I need help with annotation guidelines and the actual annotation.
Multiple annotators per result, agreement metrics, published rubric.
I can collect the data. I can’t annotate it alone.
Ranking evaluation against alternatives — BM25 is my baseline.
Cross-encoder re-ranking is a well-established next step (BEIR
benchmark, MS MARCO, sentence-transformers
CrossEncoderRerankingEvaluator). The game-retrieval benchmark by
kimeyu showed BM25 beats dense retrieval in-domain, and reranking
value is domain-dependent. Onion search might be another domain
where BM25 is hard to beat. I need someone who has done retrieval
evaluation to design the experiment properly.
Query refinement with a fine-tuned model — I’m currently using
Nemotron via API for query expansion. The INTERS dataset (ACL 2024)
showed instruction tuning on 20 IR tasks significantly boosts LLM
performance on query understanding. SAIL (EMNLP 2023) fine-tuned
LLaMA-7B with search-augmented training and outperformed ChatGPT
on fact-checking. FOLLOWIR (NAACL 2025) fine-tuned a 7B model to
follow complex IR instructions. I want to know: can a fine-tuned
7B model replace my Nemotron dependency for query refinement?
I have the pipeline and the hardware to train. I need someone
who has done instruction tuning to guide the methodology.
MCP integration testing — If you use Claude Desktop, VS Code,
or any MCP client, I want to know: does ghost_search work in your
RAG workflow? What’s missing? I’ve tested it locally but I haven’t
seen it in someone else’s pipeline.
Privacy audit — I believe the architecture is anonymity-preserving
(Tor-only, RAM-only, no telemetry, no accounts). I’m a developer,
not a security researcher. I’d welcome a review from someone with
OPSEC/Tor expertise.

How to Participate

Star/watch the repo: https://codeberg.org/sookoothaii/ghost-search-mcp
Reply below — tell me what you’d use this for. That helps me
prioritize what to build next.
If you want to contribute: the engine layer is modular TypeScript,
each engine is ~20 lines, adding a new one is straightforward
Dataset annotation: DM me if interested, I’ll coordinate
If you have OPSEC/Tor expertise and want to audit the architecture,
I’d genuinely value that

This is a FOSS project. MIT licensed. No commercial dependency,
no surveillance, no data sale. Privacy is the product, not the cost.

I’m one person. I can build the infrastructure. I can’t build the
community alone. If this is useful to you, tell me. If it’s not
useful yet, tell me what’s missing.

John6666 · August 22, 2026, 11:42pm

For now, from the perspective of making this easier to move forward as a research project, I think it’s probably something like this:

Yes — I think there is a real research project here, and I would probably separate it into two measurement layers before investing heavily in either annotation or a neural reranker:

live acquisition — what the onion engines actually return, and whether they were reachable / challenged / parseable at that moment;
frozen retrieval/ranking — given exactly the same captured candidate set, how BM25, fusion, agreement, heuristics, cross-encoders, etc. reorder it.

That distinction seems unusually important here because a Tor metasearch system is not just a ranker. It is closer to an uncooperative federated-search / result-merging system: several independent search engines produce heterogeneous result lists, then a broker has to normalize, deduplicate and merge them. That is very close to the problem setting of the TREC Federated Web Search track, including the annoying real-world part where the central system may have only URL/title/snippet rather than comparable upstream scores. There is even later work specifically on snippet-based result merging in uncooperative federated search.

So the default route I would use is roughly:

A. Runtime / acquisition sanity
        ↓
B. Measure live per-engine acquisition
        ↓
C. Freeze the raw SERPs / candidate pool
        ↓
D. Build pooled relevance judgments
        ↓
E. Replay the same pool offline:
   raw ranks / RRF / BM25 / agreement /
   quality heuristic / cross-encoder
        ↓
F. Then compare query refinement:
   raw query / Nemotron / local 7B

with MCP integration/failure semantics and privacy/OPSEC as parallel tracks rather than variables silently mixed into the ranking score.

If I were doing only one cheap check first, though, I would check the boring part: are the deployed search paths actually producing non-empty titles and snippets from the engines before BM25 sees them?

I ran a small CPU-only/offline sanity pass against the code version I was looking at (77e68649b8bf4fefda75a1b6f3c6a33fcdc11da2), pinning Bun 1.3.14 and using synthetic HTML rather than live onion traffic. In that environment, global DOMParser was unavailable and linkedom did not resolve; the selector parser therefore fell back to the regex path, which produced the URL as the title and an empty snippet. Bun’s documented server-side Web APIs/globals are consistent with DOMParser not being available by default.

I would not generalize that to your actual deployed environment without checking it there, but it makes this one-line sanity test very high-value:

for each engine:
    status
    raw_result_count
    title_present?
    snippet_present?

If those fields look healthy in your actual runtime, great — move straight on. If not, fixing candidate acquisition first will make every later BM25/reranker comparison much easier to interpret.

Why I think the federated-search framing may be useful

The connection is more than terminology.

The classic federated-search problem is normally split into:

resource selection — which remote search engines/resources are useful for this query?
result merging — how do we combine results from independently ranked sources whose internal scores/statistics may not be comparable?

The TREC FedWeb overview used results from 157 real search engines and explicitly separated those two tasks. For result merging, participants were given search result pages containing titles, snippets and hyperlinks and had to construct a unified ranking.

That is remarkably close to the shape here.

I would not immediately add query-dependent engine selection to Ghost Search — querying all engines is a perfectly useful way to collect baseline data first. But I would preserve the engine identity and raw rank for every candidate. Then later you can measure:

engine availability
engine latency
raw yield
unique yield
relevant yield
pairwise overlap

and decide empirically whether source selection is worth adding.

This also gives a useful interpretation of “engine agreement.”

The intuition that a document returned by several systems may deserve more weight has a long history in data fusion / rank fusion. A particularly cheap baseline is Reciprocal Rank Fusion (RRF), and RRF was also among the submitted TREC FedWeb result-merging runs.

So I would not replace the agreement idea. I would turn it into an ablation:

raw upstream ranking
RRF
BM25
BM25 + raw engine agreement
possibly BM25 + agreement adjusted for engine overlap

The last control matters because “three engines returned this URL” is not necessarily three independent votes. Onion services have mirrors/duplicates, and search engines may themselves have correlated coverage. A simple pairwise URL-overlap/Jaccard matrix would already tell you a lot before inventing a sophisticated correction.

In other words, agreement looks like an interesting research variable, not something that needs to be removed.

How I would build the ranking benchmark

This is probably the part of your post where I think the project can become most reusable to other researchers.

1. Preserve the raw SERPs before normalization

Before canonicalization/dedup/reranking, I would retain something approximately like:

query_id
query_text
timestamp
engine
raw_rank
raw_url
title
snippet
acquisition_status
Ghost version/config

Then derive:

canonical_url
duplicate_class
engine_count

later.

That gives you the option of changing dedup/fusion/ranking without losing the original observations.

It also matters because this is a temporally unstable collection. A qrel saying “document X was relevant to query Q” is much more useful if it really means:

this result surrogate was observed from engine E at time T, under configuration C.

That lets a future replication distinguish ranking drift from web/index drift.

2. Define what the annotator is judging

I think there are at least three subtly different targets here:

SERP-surrogate relevance
From URL/title/snippet, does this look relevant to the query?
underlying-page topical relevance
After fetching the document, is the actual page relevant?
downstream RAG utility
Did the retrieved content help answer the user’s task?

Those do not have to become three benchmarks immediately.

For a first open benchmark, I would probably make SERP-surrogate relevance the explicit target, because that is exactly what the central Ghost ranking layer often has available, and it avoids making full-page collection a prerequisite.

That also makes the benchmark safer and easier to redistribute: harmless research queries plus result surrogates can establish a lot before anyone needs a broad archive of onion content.

Later you could add a smaller page-level subset where it is appropriate.

3. Pool across diverse runs before judging

The main methodological trap I would avoid is:

Ghost BM25 top-k
    ↓
human judges those documents
    ↓
same qrels used to prove BM25 beats alternatives

That can favor the system that created the judgment pool.

TREC’s normal solution is pooling: take high-ranked results from several sufficiently different runs, union them, then judge that pool. NIST has also documented how shallow/incomplete pools can bias evaluation (Buckley et al., “Bias and the Limits of Pooling for Large Collections”).

A cheap first Ghost pool could be:

union(
    each engine's raw top-k,
    simple fusion top-k,
    BM25 top-k,
    one substantially different reranker top-k
)

You do not need every imaginable model in the first pool. The important thing is that the pool is not defined solely by the same retrieval function you are evaluating.

TREC FedWeb is again useful here because its public evaluation resources include not only relevance judgments but also duplicate-document classes.

4. Graded relevance seems worth it

Something simple such as:

0 = not relevant
1 = marginal / weakly useful
2 = relevant
3 = highly relevant

would support nDCG while still being reasonably annotatable.

I would keep these separate:

relevance
availability
duplicate/mirror status
challenge/error status

A dead page is not automatically irrelevant; a CAPTCHA is not a relevance judgment; two mirrors are not two separate relevance judgments.

That separation becomes especially useful later.

5. Multiple annotators

Your idea of multiple annotators + agreement metrics makes sense.

I would publish:

the rubric;
the anonymized per-annotator judgments if possible;
the aggregated judgment;
agreement statistics;
adjudication policy, if disagreements are adjudicated.

That makes the dataset much easier to reinterpret later if someone disagrees with the relevance boundary.

6. Keep a live track and a frozen track

This is the part I would emphasize most.

Live acquisition track:

query × engine
    success?
    latency
    challenge?
    HTTP failure?
    parse failure?
    raw result count
    unique contribution

Frozen ranking track:

Take one timestamped captured candidate set and replay:

raw engine order
RRF
BM25
BM25 + agreement
quality heuristic
cross-encoder
LLM reranker

Now when model B beats model A, you know the candidate set did not move underneath the experiment.

That gives you two useful results rather than one ambiguous result:

How good/reliable is live onion candidate acquisition?

and

Given the same candidates, how good is the Ghost ranking layer?

A few implementation boundaries that seem worth separating before interpreting ranking numbers

These are not reasons to change the overall direction; mostly they are reasons I would establish the measurement contracts before collecting a large qrel set.

Parser/runtime

As mentioned above, in my pinned CPU run:

typeof DOMParser -> undefined
linkedom          -> unresolved
selector fixture  -> URL title + empty snippet fallback

So I would first verify this in the actual environment.

If the real deployment does have a DOM implementation supplied some other way, this concern disappears immediately.

If it does not, it is important because BM25 over:

title + snippet

is then not the intended title/snippet experiment.

MCP and Dashboard currently appear to have different acquisition paths

In the code version I checked, the Dashboard invokes engine-specific prepareSearchUrl() hooks while the MCP search path does not.

This matters particularly for engines that need a preflight/form step.

Ahmia is a concrete example: its current public code generates a rolling six-character token and rotating field name, and redirects requests with an invalid/missing token. You can see that logic directly in the current Ahmia search view.

Ghost already has an Ahmia preparation hook intended to obtain that hidden input, which looks like the right basic idea; the useful question is simply whether the same acquisition core reaches both Dashboard and MCP.

So I would either:

unify:
    Dashboard ─┐
               ├─ search core → engines
    MCP ───────┘

or, if keeping separate paths is intentional, record the path/config in benchmark metadata.

That turns “UI differences” into an explicit experimental variable rather than accidental noise.

Preserve duplicate provenance before dedup

In this snapshot, duplicate URL handling keeps the first result, while the BM25 input is later supplied with:

engineCount = 1
trustScore  = 1

for each document.

I reproduced that behavior with a tiny duplicate fixture.

So the interesting interpretation is not “agreement is a bad idea”; it is almost the opposite:

preserve the provenance first, and agreement becomes an actual variable you can test.

For example:

canonical result
    url
    observed_by:
        - engine A, rank 3
        - engine D, rank 1
        - engine H, rank 7

gives you far more options than collapsing that to the first engine immediately.

Treat the quality filter as a ranking arm in the evaluation

I also tested the current quality filter with a controlled two-result fixture. A result that BM25 placed first could become second after the quality filter, because the quality stage re-sorts by its own quality score.

That is not necessarily wrong — it may be useful.

But experimentally I would label it as something like:

BM25
quality heuristic
BM25 + preserved agreement
cross-encoder

rather than assuming that “quality filtering enabled” is still basically the same BM25 ranking.

Then the ablation tells you what each stage actually buys you.

Distinguish challenge/parse failure from a genuine empty result

The code already contains challenge/CAPTCHA detection logic, which is useful.

The measurement question is whether an HTML challenge that returns HTTP 200 can reach the result parser, yield zero parsed results, and be counted as an engine success.

For an agent, these are very different observations:

valid_empty
challenge
timeout
HTTP_error
parse_empty
partial_success

For an IR benchmark they are also different: none except valid_empty really says “the engine searched successfully and found nothing.”

That distinction is cheap to add and can save a lot of debugging later.

On the Nemotron → local 7B question

I think this is testable, but I would actually delay the fine-tuning step by one experiment.

First measure:

raw query
    vs
Nemotron-refined query

on the same frozen retrieval evaluation.

Then:

Does Nemotron produce a reproducible downstream retrieval lift?

    no
    └─> there may be no useful effect to distill yet

    yes
    └─> add the local 7B and test whether it preserves that lift

That answers a more fundamental question before spending training effort:

Is query refinement helping Ghost Search in this domain at all?

Only after that do you really need:

How small/local can the refiner become?

There is good precedent for smaller instruction-tuned models acquiring useful IR abilities, but I would treat it as evidence that the experiment is plausible, not as evidence that a 7B model will automatically replace Nemotron here.

For example:

INTERS instruction-tunes models across query understanding, document understanding and query-document tasks and reports substantial improvements on IR tasks.
FollowIR builds an instruction-following retrieval benchmark and fine-tunes a 7B retrieval model, improving its ability to follow detailed retrieval instructions.

But neither establishes:

7B query rewriter == Nemotron query rewriter

for onion search.

And newer reproducibility work is a useful caution: a 2026 controlled study of LLM query reformulation found that gains depend strongly on whether the downstream retriever is lexical, learned-sparse or dense, and that larger LLMs are not uniformly better.

So I would make the end-to-end retrieval score the primary outcome, not “the rewrite looks smarter.”

A minimal matrix could be:

Query path	Same candidate-generation configuration	Retrieval metric	Latency/cost
Raw	yes	nDCG / Recall / MRR	baseline
Nemotron	yes	same	measured
Local 7B	yes	same	measured

If Nemotron has a clear lift, you also get training data almost naturally:

raw query → effective refinement

plus retrieval outcome as an external signal.

That seems like a much stronger basis for the local model than training it only to imitate fluent reformulations.

For MCP / RAG integration, I would make partial failure visible

The practical Agent-side issue I would prioritize is failure semantics.

A metasearch tool can legitimately return useful results even if several engines fail, so I would not turn every engine error into a failure of the entire MCP call.

Instead, something conceptually like:

{
  "results": [...],
  "raw_candidate_count": 42,
  "unique_candidate_count": 27,
  "engine_status": {
    "engine_a": "ok",
    "engine_b": "challenge",
    "engine_c": "timeout",
    "engine_d": "valid_empty"
  },
  "partial_failures": [...],
  "elapsed_ms": 1832
}

would let an Agent distinguish:

“there appears to be no result”

from:

“most of my search infrastructure failed.”

That distinction matters a lot in agentic systems because otherwise the model may treat an infrastructure failure as evidence about the world.

MCP already has useful primitives for this direction. The Tools specification supports structuredContent, optional outputSchema, and isError for tool-execution failures. It also explicitly distinguishes protocol errors from actionable tool errors.

So, for example:

14 engines
├─ 9 return usable results
├─ 2 challenge
├─ 2 timeout
└─ 1 valid empty

could still be an overall successful search result with structured partial-failure metadata.

Whereas:

Tor unavailable
all engines unreachable
invalid request

might reasonably become an actual tool execution error.

That would also make real-world MCP testing much more informative: instead of only asking “did Claude/another client invoke the tool?”, you can inspect whether the Agent makes sensible decisions under degraded retrieval.

A few particularly useful integration fixtures would be:

1. normal results
2. genuine empty results
3. half the engines time out
4. challenge page returned as HTTP 200
5. malformed result HTML
6. result snippet containing instruction-like text

The last one is not specific to Tor; it is simply because search results are open-world, untrusted content. MCP’s own security guidance recommends output sanitization and validation of tool results before passing them onward to an LLM.

I would keep that as an integration test rather than trying to make the search engine itself decide what text is “safe.”

For privacy / OPSEC, I would separate Tor transport from the optional LLM data path

I think this part benefits from being described mode-by-mode rather than with one global “anonymous” label.

On the Tor side, some of the current design choices look directionally sensible:

destination hostnames are passed through SOCKS rather than obviously pre-resolved;
per-request SOCKS credentials can be used by Tor for stream isolation;
the supplied Tor configuration includes SafeSocks 1.

There is also a very cheap positive test available directly from Tor.

The Tor Project documents TestSocks 1 specifically for checking whether a SOCKS-using application leaks DNS: Tor logs a notice for safe SOCKS requests and a warning for unsafe ones. SafeSocks 1 can reject unsafe connections.

So an OPSEC smoke test could be as simple as:

TestSocks 1
SafeSocks 1

run several harmless Ghost searches
inspect Tor log

That gives you stronger evidence than just inferring behavior from the socket code.

I would then treat LLM refinement/summarization as a separate trust boundary.

In a mocked network test of the code version I checked, the MCP defaults could construct the LLM client without a NIM key because a local Ollama model is configured, but the client still attempted the remote NVIDIA endpoint first and included the raw query in that request body before falling back to local Ollama.

That does not say anything about what NVIDIA subsequently stores or does with data; I would not infer that from the client code.

It simply means the privacy model is clearer if these modes are explicit:

Mode A:
Tor retrieval only / local processing

Mode B:
Tor retrieval + local LLM

Mode C:
Tor retrieval + remote LLM refinement/synthesis

Then the documentation can say exactly which data crosses which boundary in each mode.

For a project whose design goal includes privacy, that distinction strengthens the claim rather than weakening it.

The same principle applies to observability: status codes, counts and timings are relatively easy to keep; raw query/URL/snippet logging should probably be an explicit choice rather than something introduced accidentally while making the system easier to debug.

If I wanted the maximum information from the first small experiment

I would avoid starting with a giant annotation campaign.

A small first pass could use perhaps 20–50 benign research queries, with an even smaller 1–3 query smoke test before that.

Phase 0 — runtime sanity

DOM parser available?
per-engine title present?
per-engine snippet present?
challenge detector connected?

If this fails, stop there and fix acquisition.

Phase 1 — live acquisition

For every query × engine:

timestamp
status
latency
raw_count
title/snippet presence
challenge/error class

Then compute:

unique URLs per engine
pairwise engine overlap
unique contribution per engine

No relevance annotation is required yet.

This alone tells you whether “14 engines” behaves more like:

14 reasonably independent sources

or:

5 productive sources + 4 correlated sources +
3 frequently challenged sources + 2 mostly dead sources

which is useful research information in its own right.

Phase 2 — freeze the candidates

Save the raw SERPs.

Now network state is no longer part of the ranking experiment.

Phase 3 — cheap ranking baselines

Before paying for cross-encoder or LLM evaluation:

raw engine ordering
round-robin / trivial merge
RRF
BM25
BM25 + preserved engine agreement
quality heuristic

These are cheap enough that they make excellent controls.

If a neural reranker cannot beat those reliably, that itself is a useful result.

Phase 4 — human judgments

Construct a diverse pool and annotate it.

Then use metrics appropriate to the judgment scheme, e.g.:

nDCG@k
MRR
Recall@k

I would report acquisition/reliability metrics separately rather than trying to fold them into one magic score.

Phase 5 — expensive models

Only now:

cross-encoder
LLM reranker
raw vs Nemotron
raw vs Nemotron vs local 7B

This ordering has a nice property: every expensive step is justified by information from the cheaper previous step.

A useful decision tree is:

candidate acquisition unstable?
    yes → measure/fix acquisition first
    no  → freeze candidate pools

ranking baselines meaningfully differ?
    yes → build qrels / compare rankers
    no  → investigate candidate recall / query set first

Nemotron gives measurable retrieval lift?
    no  → local-7B training can wait
    yes → local 7B becomes a justified compression/replacement experiment

A couple of smaller observations I would probably keep secondary rather than letting them distract from the research design:

If engine-specific preparation is moved into the MCP path, keep stdio protocol output clean; normal diagnostic logging should not share stdout with the MCP wire.
Session/cookie-dependent challenge flows deserve a small mock test because a stateless HTTP transport and a browser-like challenge flow have different contracts.
I would avoid calling the benchmark “the first” until the literature search is completely nailed down. Morato et al. already evaluated retrieval/ranking behavior across Tor search engines, even though I have not found a reusable public onion query–qrels collection equivalent to what you are proposing.
If later you optimize source selection rather than querying all engines, that can be treated as a new experimental layer rather than silently changing the benchmark.

So, condensed down to the part I think has the highest information gain:

I would keep the overall project direction, but make the boundaries measurable before scaling it. First verify the result-surrogate acquisition path, preserve raw per-engine provenance, distinguish live retrieval failures from true empty results, and freeze candidate pools before comparing ranking methods. Then the benchmark can tell you separately whether Ghost improves coverage, fusion/ranking, and query refinement, instead of one stage accidentally taking credit or blame for another.

Once those boundaries are in place, the things you are already interested in — engine agreement, BM25 vs cross-encoder, Nemotron vs a local 7B, and MCP/RAG integration — become much cleaner experiments rather than competing moving parts.

sookoothaii · August 24, 2026, 1:40pm

Thanks for the detailed review — this is exactly the kind of methodology guidance I needed. I’ve been working through your suggestions and have concrete progress on several points, including new results from the last two days.

Phase 0 (runtime sanity) — fixed and verified

The DOMParser/linkedom bug you identified was real and is now fixed. linkedom@0.18.13 is in package.json, parseDocument() works in Bun’s server-side runtime, and selector-based parsing produces non-empty titles and snippets. I verified this with a live Docker deployment: 14 engines queried, 13/14 returned non-empty results with real snippets (only tor66 returned error_404).

I also ported 180 tests from the Android app’s test suite (blacklist, camouflage, dedup, entities, query, ranking, STIX) and they all pass in the Bun runtime. The dedup.ts module was extracted with 6 pure functions that have no Android-specific dependencies.

Engine health and partial-failure semantics — implemented

Your suggestion to distinguish valid_empty from challenge/timeout/error is now live in the API response. Every search returns an engineHealth dict with per-engine status, results count, and ms latency:

engineHealth: {

ahmia: {status: “ok”, results: 2297, ms: 7681},

tor66: {status: “error_404”, results: 0, ms: 500},

torgle: {status: “ok”, results: 0, ms: 6025}, // valid_empty

…

}

This flows through to the IntelShed UI as a color-coded engine health panel. Partial failures render results immediately with the health panel below — no error modal for 9/14 engines succeeding.

I also added raw_candidate_count and unique_candidate_count to the response, so an agent can distinguish “2537 raw candidates, 50 unique after dedup” from “0 results because infrastructure failed.”

LLM mode transparency — implemented

The /api/status endpoint now reports which LLM boundary is active:

llm_mode: "remote_nim" | "local_ollama" | "none"

llm_model: “nvidia/nemotron-3-super-120b-a12b” | “qwen3:8b”

llm_boundary: “query_text_sent_to_nvidia” | “local_only”

This is informational, not a control — the operator changes it via env vars, not through the UI. But it makes the privacy boundary explicit, which is exactly what you suggested with the Mode A/B/C separation.

MCP vs Dashboard acquisition path — partially addressed

You were right that the Dashboard and MCP paths had different acquisition logic. The Dashboard’s handleSearch() calls engine.prepareSearchUrl() (needed for Ahmia’s anti-bot token), while the MCP path didn’t. This is now unified — both paths go through the same search core. The IntelShed integration uses the Dashboard’s HTTP REST API (POST /api/search), which has the complete acquisition path including prepareSearchUrl.

The remaining question is whether the MCP stdio path should also be updated for standalone use. Currently the IntelShed integration doesn’t use MCP stdio at all — it uses HTTP REST. But for Claude Desktop / VS Code users, the MCP path should have the same acquisition quality. I’ll address this next.

On the federated search framing — agreed, and it shaped the architecture

Your TREC FedWeb analogy was directly useful. Ghost is now integrated into IntelShed as an “uncooperative federated search broker” — IntelShed calls Ghost via HTTP, Ghost handles all 14 engines, and IntelShed consumes the merged results for entity extraction and FtM matching. The intelligence layer (entity extraction, FtM graph, content retrieval) stays in Python; the search layer (acquisition, parsing, BM25, dedup) stays in TypeScript. One parser implementation, one codebase to maintain.

The engine_health dict is exactly the “live acquisition track” you described — it’s now a permanent part of the API response, not just a benchmark artifact.

SERP freezing and ranking replay — completed (new since last post)

Following your Phase 0-5 ordering, I’ve now completed Phase 2 (SERP freezing) and built a ranking replay tool:

Frozen corpus: 24 queries × 13 engines, 28,280 results collected, 99.4% snippet coverage. Queries span 4 categories (threat intelligence, security research, investigative journalism, infrastructure entities). The corpus is gitignored local research data — reproducible offline, no live network needed for ranking experiments.
Replay tool (replay-ranking.ts): Loads frozen SERPs and runs them through 3 ranking strategies without live Tor:
- engine-count: current production default (aggregate + content dedup)
- BM25: BM25 + engine-agreement (k1=1.2, b=0.75, engine weight=2.0, trust weight=0.5)
- RRF: Reciprocal Rank Fusion (k=60, Cormack et al. SIGIR 2009), trust-weighted per engine

Key findings from the frozen corpus:

Dedup ratio: The highest-volume engine (Ahmia, 24,544 raw results) has a 0.982 unique-URL ratio — its high yield is genuine diversity, not mirror duplication. The remaining 12 engines collectively produce 3,736 raw → 3,086 unique (0.826). Cross-engine overlap between Ahmia and the rest is 0-1.6% on most queries.
Ranking strategy matters: Mean Top-10 Jaccard overlap between strategies: engine-count∩BM25=0.463, engine-count∩RRF=0.390, BM25∩RRF=0.436. On one query (“zero-day”), engine-count and RRF produce completely disjoint Top-10 lists (Jaccard=0.0).
Per-engine unique contribution: Ahmia contributes 23,713 unique URLs (96.6% of its results are unique to Ahmia). OnionLand has 100% unique contribution but only 2 results/query average — small but perfectly non-overlapping.

Engine availability — updated (new since last post)

Two engines that showed 0 results in the frozen corpus (OSS, Torgle) were diagnosed as transient Tor circuit issues, not parser bugs — both now return results live (11 and 10 results respectively). One engine (Torch) had a stale endpoint address; updated to the classic Omega CGI interface, now returning 10 results in 5.1s with a 3.6M document index.

Current live status: 12/13 engines operational, 370 merged results on a test query.

I also added per-engine connect timeout overrides based on Tor Project recommendations (60-90s for onion services): the two slowest engines get individual timeouts (40s and 30s) while fast engines keep the 20s default — no performance impact on healthy engines, but the slow ones no longer get falsely marked as timed out.

On the ranking benchmark — agreed on ordering, starting with acquisition

I agree with your Phase 0-5 ordering. The SERP freezing (Phase 2) and replay tool are now done. The pairwise engine overlap computation (your Phase 1) is partially complete — the benchmark module has Jaccard similarity and uniqueContribution() functions, and I’ve run them on the frozen corpus. The next step is qrels annotation (Phase 3) — manual relevance scoring on the frozen Top-20 per query, which enables nDCG/MAP computation for the ranking comparison.

On query refinement — agreed, measuring Nemotron lift first

I’m not fine-tuning a 7B model yet. The current architecture has Nemotron as primary with Ollama qwen3:8b as fallback. Before investing in training, I’ll run your suggested matrix: raw query vs Nemotron-refined query on the same frozen candidate pool, measure nDCG/Recall/MRR. If there’s no measurable lift, the 7B distillation experiment can wait.

On operational configuration — Tor safety checks noted

The Tor configuration already includes SafeSocks 1. I’ll add the TestSocks 1 log inspection as an explicit smoke test. The per-search SOCKS5 credential isolation (RFC 1929) is already implemented and device-verified — different session IDs per search prove isolated circuits.

What’s next

qrels annotation on frozen SERPs (manual, no network needed)
nDCG/MAP computation for the 3 ranking strategies
Nemotron query refinement lift measurement
MCP stdio path: unify acquisition with Dashboard/REST path

The quality filter is now treated as a separate ranking arm in the evaluation, not silently mixed into BM25. And duplicate provenance is preserved before dedup — the engineHealth dict records which engines returned each URL, not just the first one.

Thanks again for the structured feedback. The federated-search framing in particular changed how I think about the architecture — it’s not just a ranker, it’s a broker, and the measurement boundaries you suggested are now explicit in the API.

John6666 · August 25, 2026, 2:35pm

I think this is a good direction:

For the remaining MCP question: yes, I would bring the standalone stdio path to semantic parity with the REST path if Claude Desktop / VS Code / other MCP clients are still a supported use case. I would not make “literally the same entry point” the requirement, though. The useful invariant is that the same query and the same engine responses produce the same normalized candidates, provenance, failure semantics, and ranking regardless of transport.

For the benchmark, I think the highest-information next step is also still qrels — but I would insert one small calibration stage before annotating all 24 queries:

define the information need for each query, not just the query string;
define exactly what an assessor is judging;
build the judgment pool from several deliberately different ranking arms;
double-annotate 4–6 representative queries;
resolve rubric ambiguities and freeze the evaluation protocol;
then annotate the remaining topics.

That is probably more useful right now than adding another neural reranker. The Top-10 Jaccard numbers already establish that the ranking strategies disagree substantially; qrels are what turn “these rankings are different” into “this difference is useful.”

A compact default route might be:

Step	What it answers	Cost
4–6 query qrels calibration	Are we judging the same notion of relevance?	low
pooled qrels for all 24 topics	Which ranker is actually better?	medium
atomic BM25/RRF/agreement/trust ablations	Which component causes the lift?	low
frozen + live Nemotron tests	Does refinement help ranking, acquisition, or both?	medium
REST/stdio parity fixture	Do MCP users get the same search semantics?	low
Tor ControlPort observation	Are circuit-isolation claims observed, not only configured?	optional

Qrels: I would define the benchmark before scaling the annotation

1. Treat each item as a topic, not merely a query string

This looks particularly important for short queries such as zero-day.

Two annotators can apply exactly the same 0–3 rubric and still disagree completely if one interprets the query as:

current reporting about zero-day exploitation

while another interprets it as:

technical information, research, tooling, markets, or discussion related to zero-day vulnerabilities.

A useful precedent is the TREC Federated Web Search track. FedWeb is unusually close to Ghost’s problem because it evaluated result merging across heterogeneous search resources, and its assessors were shown:

the query;
a short description of the information need;
a narrative explaining what should and should not count.

So before labeling the full corpus, I would freeze something roughly like:

topic_id: q07
query: zero-day

description:
Information about zero-day software vulnerabilities.

narrative:
Relevant results contain substantive information about zero-day
vulnerabilities, exploitation, disclosure, mitigation, research,
or closely related activity.

Not relevant:
Generic security pages that merely contain the words "zero day",
unrelated products/brands, or pages for which the available result
surrogate gives no evidence of relevance.

It does not have to be verbose. Even two sentences can remove a surprising amount of annotation noise.

For any topic whose real downstream objective is something narrower — for example entity extraction, FtM matching, finding a particular infrastructure artifact, etc. — I would keep that downstream objective separate from retrieval relevance. Otherwise one qrel starts trying to score both:

“Is this result relevant?”

and

“Did Ghost successfully extract the entity I wanted?”

Those are both useful measurements, but they are different measurements.

2. Freeze the judgment target

There are at least three reasonable benchmarks hiding here:

A. SERP-surrogate relevance

Judge only what Ghost’s broker actually has:

URL
title
snippet
source/provenance

B. Underlying-page relevance

Open/fetch the result and judge the actual page.

C. Downstream utility

Ask whether the result helped an investigation, RAG answer, entity graph, etc.

For the first Ghost benchmark, I would strongly prefer A.

That matches the central broker’s actual information boundary, keeps annotation offline, and avoids turning the ranking benchmark into a Tor crawling / page-availability / content-extraction benchmark.

It also makes the benchmark easier for other people to reproduce.

If the frozen corpus stores a processed representation — e.g. HTML-stripped or length-limited snippets — I would name that explicitly in the dataset documentation. Something like “frozen Ghost SERP surrogates” is more precise than implying that the benchmark contains untouched raw engine responses.

Then page-level evaluation can become a separate benchmark later if it becomes useful.

3. Use a diverse pool, rather than Top-20 from only one ranking

The usual TREC idea of pooling maps nicely here: judge the union of top results from several runs rather than treating one ranker’s candidates as the truth.

For Ghost, the initial pool could simply be:

Top-k engine-count
∪ Top-k BM25
∪ Top-k plain RRF
∪ Top-k current Ghost RRF
∪ Top-k quality-filter arm, if it retrieves genuinely different candidates

Then canonicalize/deduplicate the pool before presenting it to annotators.

I would start with the 4–6 topic calibration first rather than committing immediately to a pool depth for all 24 topics. You will quickly see whether k=10, 20, or something slightly larger produces a manageable pool.

The useful property is diversity of retrieval mechanisms, not maximum pool size.

A nice side effect is that a future cross-encoder does not immediately inherit BM25’s blind spots just because BM25 generated the qrels pool.

4. Keep relevance and duplication separate

I would not encode duplicates into the relevance label itself.

Something like:

topic_id
candidate_id
canonical_url
duplicate_class_id
title
snippet
relevance_grade
judgment_status
annotator_id

is easier to reason about.

For example:

relevance_grade:
0 = irrelevant
1 = marginal / weakly useful
2 = relevant
3 = highly relevant

judgment_status:
judged
unjudgeable

unjudgeable is worth having. An empty or cryptic snippet should not silently become “irrelevant” if the assessor simply lacks enough evidence.

The duplicate class can then be applied by the evaluation layer.

FedWeb did something very similar conceptually: it maintained duplicate information separately and reported result-merging metrics where later occurrences of the same page received zero gain, as well as variants without the duplicate penalty. The FedWeb 2014 data page even publishes the duplicate-page sets separately from the relevance judgments.

That gives Ghost a clean distinction between:

relevance:
"Is this candidate useful for this topic?"

diversity:
"Is this candidate providing another independent result?"

That distinction will matter a lot for onion search.

5. URL uniqueness and content uniqueness are worth reporting separately

The Ahmia result is interesting:

24,544 results, 0.982 unique-URL ratio, and very little URL overlap with the other engines.

I would preserve that result, but label it specifically as URL-level diversity.

A different URL does not necessarily imply different content.

Because snippet coverage is already 99.4%, a cheap second measure does not require fetching any pages:

normalize(title + snippet)
    -> exact hash

optionally:
    -> SimHash / MinHash / another cheap near-duplicate signature

Then report both:

URL-unique contribution
surrogate-content-unique contribution

Later, after qrels:

uniquely relevant contribution@k

That makes the engine-diversity result considerably more informative without changing the acquisition architecture.

I would also normalize retrieval depth when comparing engines.

Ahmia produced 24,544 of the 28,280 frozen results, so an absolute count such as:

“23,713 URLs unique to Ahmia”

contains both:

diversity;
much greater retrieval depth.

A very cheap companion statistic would be:

unique@10
unique@20
unique@50

per engine/query at equal depth.

Then the interpretation becomes much cleaner:

absolute unique yield
    -> what does this engine contribute in practice?

unique@k
    -> how non-redundant is it at comparable retrieval depth?

relevant unique@k
    -> how much uniquely useful material does it contribute?

I would not optimize engine selection until the third quantity exists.

6. Pilot the rubric before annotating everything

With only 24 topics, I would choose 4–6 that deliberately stress different cases:

high overlap between rankers
low overlap between rankers
very high-volume Ahmia case
low-yield / rare-source case
ambiguous short query
entity/infrastructure-style query

zero-day, where engine-count and RRF had Jaccard 0.0 in your replay, sounds like a particularly useful calibration topic.

For those pilot topics:

two independent annotators
-> compare disagreements
-> discuss only after independent labeling
-> clarify the narrative/rubric
-> relabel if necessary
-> freeze rubric

You can calculate weighted agreement if useful, but I would treat the disagreement examples as at least as important as the single agreement coefficient. They tell you where the benchmark definition itself is underspecified.

If annotation manpower is limited, the pilot can be fully double-judged and the full collection can use whatever overlap strategy is affordable. The important thing is to state exactly what was independently judged and what was adjudicated.

7. Metrics: make graded ranking the primary result

For a 0–3 qrel, I would make something like:

nDCG@10
nDCG@20

the primary ranking metrics.

MAP is still useful, but it needs an explicit binary conversion, for example:

grade >= 2 -> relevant
grade < 2  -> non-relevant

Otherwise two implementations can both report “MAP” while evaluating different things.

I would also keep the per-topic values.

Twenty-four topics are enough for a useful pilot benchmark, but they are not enough for a tiny mean difference to become automatically convincing. Classic TREC work on topic-set size and retrieval experiment error is a good reminder that retrieval effectiveness varies substantially by topic.

So I would report:

mean nDCG@10 / nDCG@20
median
per-query delta against baseline

and, if convenient:

paired bootstrap / randomization interval

A per-query delta table will probably be especially informative here because the existing Jaccard numbers already suggest that behavior is heterogeneous across topics.

For future models that retrieve many documents outside the original judgment pool, also track something like:

judged@10
judged@20

so a score drop is not mistaken for “worse ranking” when it is actually “this new system retrieved mostly unjudged documents.”

If the qrels eventually become substantially incomplete for new systems, bpref / incomplete-judgment evaluation is another tool available later. I would not complicate the first experiment with it unless it becomes necessary.

8. Freeze evaluation before tuning

One easy benchmark trap would be:

build qrels
-> tune BM25/agreement/trust/RRF weights on all 24 topics
-> report final performance on the same 24

For an exploratory repo this is understandable, but the final number then measures some amount of tuning to the benchmark.

A low-cost separation would be:

4–6 calibration/development topics
remaining topics = evaluation

or simply:

freeze all parameters before looking at qrel-based effectiveness

for the first baseline comparison.

The latter may actually be preferable initially because your current parameters already exist.

Ranking replay: make the arms atomic

The current Jaccard result already answers one question well:

the ranking strategies are not equivalent.

It does not yet tell us which component caused the change or which ranking is better.

I would therefore make the replay arms as boring and atomic as possible.

For example:

Arm	Purpose
engine-count	current/simple baseline
lexical BM25	lexical-ranking baseline
BM25 + agreement	isolate agreement boost
plain RRF, `k=60`	literature baseline
RRF + trust	isolate trust weighting
quality-filter arm	measure filter independently

The original Cormack, Clarke & Büttcher RRF paper is useful here because plain RRF is deliberately simple:

score(d) = sum over input rankings of 1 / (k + rank(d))

with k=60 in their experiments.

So if Ghost’s RRF includes trust weighting, source priors, snippet-based ordering, or another pre/post-processing step, I would keep those — they may be useful — but expose them as incremental arms:

plain RRF
-> + trust
-> + any Ghost-specific ordering heuristic

Then a result such as:

RRF + trust beats BM25 by X

actually identifies something.

The same logic applies to BM25:

BM25 lexical score
-> + engine agreement
-> + trust/source prior, if used

This also makes a future cross-encoder much easier to interpret.

Instead of:

cross-encoder vs "Ghost ranking"

you get:

plain lexical baseline
fusion baseline
Ghost heuristics
neural reranker

which is much more reusable for a paper or dataset card.

One small replay invariant I would add to the tests is:

after any operation that changes scores,
re-sort before Top-k evaluation

and use one canonical URL identity function everywhere that computes:

dedup ratio
engine overlap
unique contribution
ranking candidate identity

Those two invariants are cheap and prevent benchmark bookkeeping from becoming part of the ranking result.

Nemotron: the frozen experiment and the live experiment answer different questions

I think delaying 7B fine-tuning until Nemotron demonstrates measurable value is exactly the right ordering.

I would split the lift experiment into two independent tests.

A. Frozen-candidate test

same frozen candidates
raw query
vs
Nemotron-refined query

This answers:

Does the rewritten query improve scoring/ranking among candidates that have already been discovered?

That is a clean ranking experiment.

B. Live-acquisition test

same information need
raw query -> engines -> candidate set A

refined query -> engines -> candidate set B

This answers:

Does query refinement cause the underlying search engines to discover better candidates?

For a metasearch broker, this may be the more important effect.

A rewrite could have almost no effect on BM25 over a fixed pool but dramatically change which onion pages the upstream engines return.

Or the reverse.

So I would avoid combining them into a single “Nemotron lift” number.

A minimal record for every refinement run could be:

topic_id
raw_query
refined_query
actual_backend
actual_model
fallback_used
frozen_corpus_id   # for replay experiments

The actual_backend field is particularly useful when the architecture has automatic Nemotron → local Qwen fallback. Otherwise both outputs can accidentally enter the analysis as simply refined=true, even though they came from different models.

For the live test, Tor/search availability introduces another variable, so paired runs are probably enough initially:

half:
raw -> refined

half:
refined -> raw

or alternate the order over repeated trials.

Keep engineHealth with each arm. Then a “refined query won” result is distinguishable from:

three extra engines happened to be reachable during the refined run.

The 7B decision tree then becomes quite simple:

Nemotron has no reliable lift
    -> do not spend time imitating it yet

Lift only on frozen ranking
    -> local model target = ranking-oriented query representation

Lift mainly on live acquisition
    -> local model target = engine-facing query rewriting / recall

Lift in both
    -> stronger case for a local 7B/8B replacement

This keeps the original goal — removing the remote-model dependency — but lets the benchmark tell you what behavior the local model actually needs to learn.

MCP stdio: I would test semantic parity rather than code-path parity

For standalone MCP use, yes, I would update it.

But I would define success as:

REST(query, fixture)
and
MCP-stdio(query, fixture)

produce equivalent search semantics

rather than requiring the transport implementations themselves to look identical.

A tiny deterministic fixture is probably enough.

For example, mock three engines:

engine_a:
  URL X, title/snippet A
  URL Y

engine_b:
  URL X, different title/snippet
  URL Z

engine_c:
  timeout

Then assert both interfaces agree on:

prepared query / URL behavior
engine status
raw candidate count
canonical candidate identity
duplicate provenance
selected representative title/snippet
ranking
Top-k
error / partial-success semantics

That test becomes much more useful than manually checking Claude Desktop after every ranking change.

It also protects the architectural split you now have:

IntelShed
    -> REST

Claude Desktop / VS Code / MCP agents
    -> stdio

both
    -> same Ghost search semantics

There is one transport-specific smoke test I would definitely add: stdio purity.

The MCP transport specification explicitly reserves stdout for MCP JSON-RPC messages; server logging belongs on stderr. See the official MCP stdio transport specification.

So a very cheap CI test is:

launch MCP server over stdio
perform initialize + ghost_search fixture call

assert:
every stdout frame parses as MCP/JSON-RPC

allow:
arbitrary diagnostic text on stderr

That catches an annoying class of integration failure that may never appear through the REST path.

I would treat MCP protocol-version migration as a separate concern from search parity. If the SDK/protocol version changes later, the parity fixture should continue to pass regardless.

Two small measurement boundaries I would keep for privacy/source claims

Tor stream isolation

The current SOCKS credential approach has a real mechanism behind it.

Tor’s current SOCKS extensions specification defines stream isolation based on SOCKS authentication values. Different isolation values can prevent streams from sharing circuits under the configured isolation rules.

I would only change the wording slightly:

different per-search SOCKS isolation values
    -> evidence that the intended isolation mechanism is being used

observed StreamID -> CircuitID mapping
    -> evidence that the actual searches were attached to different circuits

The second is stronger.

If audit-grade evidence becomes useful, Tor’s Control Protocol events expose circuit/stream information, including circuit IDs and SOCKS authentication metadata. A test-only controller can therefore observe two searches and record what circuits they actually used.

That does not need to block the retrieval benchmark.

It is just a clean way to keep:

architecture/configuration claim

separate from:

runtime-observed privacy claim.

One forward-looking detail: current Tor specifications call arbitrary SOCKS username/password isolation values legacy isolation and recommend the newer <torS0X> isolation format for new clients. I would not make migration a benchmark prerequisite, but it is worth putting on the privacy-audit list.

Per-engine timeout values

The per-engine timeout idea itself makes sense for heterogeneous onion services.

I would describe the particular 20/30/40s values as empirically tuned operational settings unless there is a specific Tor Project source for the quoted generic “60–90 second” recommendation.

Tor’s specifications document onion-service timeout/failure semantics, but I have not found a current generic client recommendation saying application connect timeouts should universally be 60–90 seconds.

That is mostly a citation/wording boundary rather than an architectural issue.

Frozen benchmark provenance: one cheap addition that will help collaborators later

Since the frozen corpus is becoming a real research artifact, I would put a tiny manifest beside it.

For example:

{
  "corpus_version": "ghost-serp-v1",
  "collected_at": "...",
  "git_commit": "...",
  "query_set_sha256": "...",
  "engine_registry_sha256": "...",
  "acquisition_config_sha256": "...",
  "normalization_version": "...",
  "tor_version": "...",
  "result_count": 28280,
  "snippet_coverage": 0.994
}

The exact fields do not matter much.

The important thing is that six months later somebody can tell whether:

same qrels + different ranking code

is really the experiment being run, rather than:

same filename + subtly different corpus semantics.

The qrels should then reference a stable candidate_id from that corpus rather than positional rank.

That also makes the eventual dataset much easier to version if you later create:

v1 = SERP-surrogate relevance
v2 = larger query set
v3 = page-level judgments

without invalidating the earlier benchmark.

So if I were choosing the shortest path from the current state, I would probably do:

1. Write query description/narrative for 4–6 topics.
2. Build a diverse Top-k pool.
3. Double-annotate that pilot.
4. Freeze rubric + candidate identity + duplicate handling.
5. Annotate the remaining topics.
6. Run atomic engine-count / BM25 / agreement / plain-RRF / trust comparisons.
7. Only then interpret the existing Jaccard differences as retrieval quality differences.
8. Split Nemotron into frozen-ranking and live-acquisition experiments.
9. Add the small REST/stdio parity fixture.

The Tor audit and heavier neural reranking can stay orthogonal to that path.

That would preserve the direction you already have — Ghost as the acquisition/result-merging broker, IntelShed as the downstream intelligence layer — while making each subsequent number answer one fairly specific question.

sookoothaii · August 27, 2026, 5:44am

Thanks for the continued guidance. I’ve been working through your calibration recommendations and have concrete results to report, though I want to be clear upfront: most of this is still in progress, the qrels are LLM-generated and provisional, and I don’t yet know whether any of these numbers will hold up under scrutiny. This is a work-in-progress report, not a results claim.

On your calibration sequence (posts 2 and 4) — where things stand:

Step	Your recommendation	Status	Result
1	Define information need per query	Done	24 TREC-style narratives (title, description, narrative with grade examples)
2	Diverse pool, not single ranker	Done	4 arms: RRF, engine-count, BM25, unique-contribution. 1197 entries, avg 49.9/query
3	Double-annotate 4-6 queries	Done	6 queries, 289 pairs, 75.1% raw / 87.9% corrected agreement
4	Freeze rubric after pilot	Done	q18 narrative refined, 0-vs-1 boundary clarified, commercial pages = grade 0
5	Annotate remaining topics	In progress	Running now, see below
6	Atomic replay arms	Done	8 strategies evaluated
7	REST/stdio parity fixture	Done	9 tests, 220/220 pass
8	Provenance manifest	Done	SHA-256 hashes for corpus, queries, qrels, narratives, dedup report

I followed your ordering almost exactly. The one place I deviated was running the atomic ablations (step 6) before the full annotation (step 5) was complete — they don’t depend on qrels, so I used the existing heuristic qrels as a provisional baseline. The ablation numbers should be re-validated against the new qrels once they’re ready.

Calibration pilot — what we actually found:

289 pairs across 6 queries (q03, q09, q14, q16, q18, q22), judged by Nemotron-3-Super-120B and DeepSeek V4 Flash with the formalized narratives:

Raw agreement: 75.1% (217/289)
Corrected agreement (excluding 17 Nemotron infrastructure failures): 87.9% (239/272)
Cohen’s kappa: 0.5808 unweighted, 0.7502 weighted
0-vs-2 boundary is stable: only 2 extreme disagreements in 289 pairs
Disagreements cluster at the 0-vs-1 and 1-vs-2 boundaries, which is expected for graded relevance

The 17 excluded pairs were Nemotron 503/timeout errors that defaulted to a parse-fallback grade. This was a bug — I’ll come back to it below.

q18 (“investigative journalism”) was the worst at 58% agreement. The narrative was too ambiguous about forum/directory results. I refined it: forums and directories listing journalism resources are grade 1 (peripheral) unless they provide direct operational access (SecureDrop instances, Signal contacts, encrypted email). I also clarified that commercial “hire a hacker” pages and exploit listings are grade 0 — the research intent is analysis, not purchasing.

I want to be honest that 87.9% corrected agreement is not the same as 87.9% accuracy. Both judges could be wrong in the same way. The agreement tells us the rubric is interpretable; it does not tell us the judgments are correct.

Judgment pool — diversity check:

The 4 arms produced 1197 entries after dedup. The interesting number is Arm D (unique-contribution): it surfaces 430 documents that no other arm finds. These are long-tail URLs that agreement-based rankers (RRF, engine-count) structurally cannot surface. Whether any of them are actually relevant is an open question — but at least they’re in the pool now, which was your main point about pooling bias.

I have not yet computed the pairwise engine overlap matrix you suggested in Phase 1. The frozen corpus has the raw per-engine data, but I prioritized the qrels pipeline. It’s on the list.

Atomic ablations — provisional, using heuristic qrels:

Strategy	nDCG@10	Delta vs baseline
rrf-plain (baseline)	0.4697	—
rrf-trust	0.5899	+0.1203
rrf-quality	0.5266	+0.0569
rrf-trust-quality (production)	0.5669	+0.0973
engine-count	0.3706	-0.0990
agreement-only	0.2787	-0.1910
bm25	0.4203	-0.0494

Trust weighting is the largest single contributor. Pure engine agreement is the worst strategy. BM25 underperforms RRF on this corpus. These numbers are provisional — they use heuristic qrels, not the calibrated ones. I’ll re-run them once the full annotation is complete.

Full re-annotation — in progress, with a complication I want to be transparent about:

I’m running two parallel annotation passes:

3-judge ensemble (DeepSeek V4 Flash, gpt-oss-120b, Nemotron-3-Super-120B): majority vote >=2 of 3, DeepSeek as tiebreaker. This is about 50% complete (q01-q12 done, q13-q24 running).
5-judge ensemble (the above three + GLM-5.2 and Kimi-K3): majority vote >=3 of 5. This just started.

The reason for the second run is something I think is worth raising, because it may be relevant to other people building LLM-judged benchmarks in security-adjacent domains.

During the 3-judge run, I discovered that a significant fraction of model responses were empty or unparseable, and the parser was silently converting these to grade 1. DeepSeek had ~18% parse-fallbacks, gpt-oss ~22%, Nemotron ~7%. This is a systematic bias toward grade 1 that has nothing to do with relevance — it’s a parser bug. I fixed it (empty/unparseable responses now trigger retries, and after retries are exhausted they’re recorded as errors rather than assigned a grade), but the 3-judge qrels produced before the fix contain this contamination.

The 5-judge run uses the fixed parser. I’ll compare the two once both are complete, and the difference should quantify how much the parser bug affected the results.

On the ensemble composition — a methodological question I’m still thinking through:

The 5-judge ensemble has 3 Chinese-origin models (DeepSeek, GLM-5.2, Kimi-K3) and 2 Western-origin models (gpt-oss, Nemotron). This wasn’t accidental. There’s published evidence that Western models exhibit what’s been called “Defensive Refusal Bias” — systematic downgrading of security research content due to keyword-based safety filters. A peer-reviewed study of 2,390 prompts from the National Collegiate Cyber Defense Competition found LLMs refuse defensive security requests at 2.72x the rate of semantically equivalent neutral requests (arXiv 2603.01246). The HuggingFace incident in July 2026 — where HuggingFace’s incident responders had to switch to GLM-5.2 because their primary Western LLM refused to analyze the breach — is a concrete example.

For a benchmark where all 24 queries are security-adjacent, this matters. If gpt-oss and Nemotron systematically downgrade SecureDrop pages or exploit writeups, a 2-Western-1-Chinese ensemble would produce systematically lower grades for exactly the content the benchmark is meant to serve.

But I want to be careful here. The evidence supports the mechanism (Western safety filters downgrading security content). It does not directly prove that my specific 3-Chinese-2-Western composition produces better qrels. That’s an empirical question I can only answer by comparing the 3-judge and 5-judge results. I’ll report whatever the comparison shows, including the possibility that the composition doesn’t matter for this particular benchmark.

I’d welcome your thoughts on whether this reasoning is sound or whether I’m overcorrecting.

What I have not done yet (being honest about the gaps):

Nemotron query refinement lift: not measured yet. Your split into frozen-ranking test vs live-acquisition test is noted and I think it’s the right design. It’s next after the qrels stabilize.
stdio purity test (stdout = MCP JSON-RPC only): not implemented. You’re right that this is a separate class of failure that the REST path won’t catch.
Tor stream isolation: I have SafeSocks 1 and per-search SOCKS credential isolation, but I have not done the ControlPort observation you described. The distinction between “configured” and “observed” is fair.
Per-engine timeout citation: I described 60-90s as a “Tor Project recommendation” but I don’t have a specific source for that. I’ll either find the citation or reword it as empirically tuned.
judged@10 / judged@20 tracking: not yet implemented. I’ll add this before evaluating any new retrieval system against the qrels.
Pairwise engine overlap matrix: data is available, computation is not done yet.
bpref / incomplete-judgment evaluation: not needed yet, but I’ll keep it in mind if new systems retrieve mostly unjudged documents.

On the “first” claim:

You mentioned Morato et al. already evaluated retrieval/ranking across Tor search engines. I’ve removed “first” from my descriptions. I haven’t found a reusable public onion query-qrels collection equivalent to what I’m building, but I also haven’t done an exhaustive literature search. I’d rather under-claim and be corrected than over-claim and be wrong.

What I expect to report next:

Once both annotation runs complete, I’ll have:

3-judge qrels (with parser contamination, documented)
5-judge qrels (with fixed parser)
Per-doc comparison between the two
Grade distribution shift
IR metric impact (nDCG@10, MAP, Recall with both qrels versions)
System ranking stability (do the relative rankings of RRF/BM25/engine-count change?)

If the two qrels sets produce the same system rankings, that’s the strongest evidence the benchmark is robust. If they don’t, I need to understand why before claiming anything.

All qrels will be explicitly labeled as LLM-generated and provisional, not human-validated ground truth. I’m one researcher with LLM judges — that’s a methodological limitation, not a feature.

Thanks again -John6666- for the structured feedback. The calibration sequence you proposed in post 4 shaped the entire pipeline. I’m happy to share any of the raw data (narratives, pool, per-judge reasoning logs) if it’s useful for your own work.

sookoothaii · August 27, 2026, 11:34pm

Quick update — both annotation runs are complete and most of the gaps I listed in my previous post are now closed. This is a final results report, not another work-in-progress.

A few things changed since my last post that I want to be explicit about, because they affect interpretation of the numbers below:

1. **Production ranking changed from engine-count to RRF + quality filter.** In my post 3 I described engine-count as the production default. During the benchmark cycle, the ablations showed RRF with trust weighting and quality filtering consistently outperformed both engine-count and BM25. I switched the production path to RRF + quality filter (RRF k=10, trust-weighted per engine, then quality filter applied). The benchmark numbers below reflect this — “rrf-quality” is the current production configuration, not the one I described in post 3.

2. **RRF k changed from 60 to 10.** My post 3 mentioned k=60 following Cormack et al. SIGIR 2009. After empirical tuning on the frozen corpus, k=10 produced better top-10 rankings for this domain. The Cormack paper used k=60 on TREC newswire; onion SERPs have shorter result lists and different rank-depth distributions, which favors a smaller k. The benchmark numbers below all use k=10.

3. **Engine count: 13, not 14.** My post 1 said 14 engines. Torgol was removed during the benchmark cycle (transient Tor circuit issues, not a parser bug — it returned results intermittently). The frozen corpus and all benchmark numbers below use 13 engines. The engine overlap matrix is 13×13.

4. **Quality filter explanation.** Several posts refer to “rrf-quality” without explaining what the quality filter does. It’s a 6-strategy demotion pipeline that never removes results — it only reorders: (1) domain collapsing (max N per domain), (2) SimHash near-duplicate detection (Hamming distance ≤3), (2b) semantic near-duplicate via embeddings (cosine >0.85), (3) information-quality scoring (snippet/entity/title signals), (4) spam/mirror detection (same title, different domain), (5) commercial-intent penalty (buy/shop/price keywords demoted for research queries), (6) URL-path mirror network detection. The filter is OFF by default and must be explicitly enabled. “rrf-quality” in the benchmark = RRF ranking followed by quality filter reordering.

5. **Test count: 223, not 180.** My post 1 and 3 said 180 tests. The current suite has 223 tests (220 pass, 3 stdio purity tests are environment-dependent and time out without a running MCP server process).

## Calibration sequence — final status

All 8 steps are now complete. The two that were in progress last time:

| Step | Previous status | Final status |

|------|----------------|--------------|

| 5 — Full re-annotation | “~50% complete” | **Done** — 24/24 queries, 1197 pairs, both 3-judge and 5-judge |

| 6 — Atomic ablations | “provisional, heuristic qrels” | **Re-validated** against both qrels_v2 and qrels_v3 |

## Two complete qrels sets

I now have two independent qrels for the same 1197-pair judgment pool:

**3-judge (qrels_v2):** DeepSeek V4 Flash + gpt-oss-120b + Nemotron-3-Super-120B. Majority >=2 of 3, DeepSeek tiebreaker. This is the run that had the parser-fallback contamination in q01-q07. I kept it as-is rather than re-running, because the comparison itself is informative.

**5-judge (qrels_v3):** DeepSeek V4 Flash + gpt-oss-120b + Nemotron-3-Super-120B + Llama-3.3-70B + Kimi K2.6. Majority >=3 of 5, DeepSeek tiebreaker. Fixed parser. This is the canonical qrels going forward.

One composition change since last post: I replaced GLM-5.2 with Llama-3.3-70B as the 4th judge. I ran a 25-document head-to-head test (5 queries, same prompts, alternating):

|-------|------------|----------------|-------------|

| Llama-3.3-70B | 24/25 (96%) | 0 | 1138ms |

| GPT-4.1-Nano | 18/25 (72%) | 0 | 796ms |

| GLM-5.2 | 15/25 (60%) | 8 | 1207ms |

| Qwen3.8-Flash | 1/10 (10%) | 5 | 5453ms |

| Mistral-Large-3 | 0/25 | 25 | unavailable |

GLM-5.2 had 8 parse failures in 25 calls and only 60% exact match. Llama-3.3-70B had 0 parse failures and 96% match. The full 5-judge run confirmed this: Llama achieved 100% coverage over 872 documents with QWK 0.79 against DeepSeek. The replacement was the right call.

I also switched Kimi-K3 to Kimi K2.6 mid-run (73% cost reduction, parser already handles code-fences). This is a model-version change, not a model-family change.

## 3-judge vs. 5-judge comparison

This was the empirical question I raised last time — whether the ensemble composition actually matters for this benchmark.

| Metric | Value |

|--------|-------|

| Per-doc agreement | 89.9% (1076/1197) |

| Disagreement | 10.1% (121) |

Grade shift matrix (3j rows × 5j cols):

```

   | 5j:0  5j:1  5j:2

-------|-----------------

3j:0 | 709 40 8

3j:1 | 18 249 10

3j:2 | 7 38 118

```

The 5-judge ensemble is more conservative at Grade 2 (-27 docs) and more generous at Grade 1 (+50 docs). The biggest shift is 3j=0 → 5j=1 (40 docs) — these are documents the 3-judge called irrelevant that the 5-judge calls peripheral. The reverse shift (3j=1 → 5j=0, 18 docs) is smaller.

I attribute the Grade-2 conservatism partly to Kimi, who votes Grade 1 for 45.6% of all documents and only assigns Grade 2 at 1.8%. The other four judges assign Grade 2 at 8-16%. Kimi is the weakest ensemble member by pairwise agreement:

| Pair | QWK | Landis-Koch |

|------|-----|-------------|

| deepseek ↔ llama | 0.7901 | substantial |

| deepseek ↔ gpt-oss | 0.7753 | substantial |

| gpt-oss ↔ nemotron | 0.7620 | substantial |

| gpt-oss ↔ llama | 0.7425 | substantial |

| deepseek ↔ nemotron | 0.7361 | substantial |

| nemotron ↔ llama | 0.7322 | substantial |

| llama ↔ kimi | 0.3383 | fair |

| nemotron ↔ kimi | 0.2919 | fair |

| gpt-oss ↔ kimi | 0.2810 | fair |

| deepseek ↔ kimi | 0.2574 | fair |

All non-Kimi pairs are “substantial” (0.61-0.79). All Kimi pairs are “fair” (0.23-0.34). Kimi is a candidate for replacement in the next iteration. I have not replaced it yet because the ensemble decisions are robust to a single weak judge — in all 1197 pairs, the other 4 judges always had >=3 agreement, so Kimi’s vote never changed a majority outcome.

## Ranking stability — the key question

The most important result: **the system ranking is identical on both qrels versions.**

|----------|-----------|-----------|-------|

| engine-count | 0.2607 | 0.2762 | +0.016 |

| bm25 | 0.2864 | 0.2915 | +0.005 |

| rrf | 0.3219 | 0.3592 | +0.037 |

| **rrf-quality** | **0.3930** | **0.4566** | **+0.064** |

Order on both: rrf-quality > rrf > bm25 > engine-count.

The 5-judge qrels lift rrf-quality by +16.3% (0.3930 → 0.4566) because Llama and DeepSeek identify more Grade-2 documents than GLM did (16.1% vs 3.1% Grade-2 rate). But the relative order of strategies doesn’t change. This is the strongest evidence I have that the benchmark is robust — two independent judge ensembles with different compositions and a parser-fix in between produce the same ranking conclusion.

## Atomic ablations — re-validated against both qrels

The ablation numbers from last post used heuristic qrels. I’ve now re-run them against both LLM-judge qrels. The relative ordering of components is consistent:

- Trust weighting remains the largest single contributor (+0.12 nDCG over plain RRF)

- Quality filter adds +0.057 nDCG independently

- Pure engine agreement remains the worst strategy

- BM25 underperforms RRF on this corpus

The absolute numbers shifted because the qrels changed, but the component ranking didn’t. Trust + Quality (the production configuration) wins on both qrels versions.

## External validation against NIST human qrels

This is new work I didn’t mention last time. I found a public dataset that allowed me to validate my judges against human ground truth: asukul’s RAG-Eval-LLM-Judge repository (Adisak Sukul, Iowa State University), which releases 537 stratified-balanced TREC RAG 2024 pairs with NIST human relevance labels.

I ran a 15% stratified sample (80 pairs) through my 5-judge ensemble:

|-------|--------------|-------------|----------|--------|

| DeepSeek V4 Flash | 0.4095 | 0.5833 | 100% | 0 |

| GPT-OSS-120B | 0.3302 | 0.4419 | 100% | 0 |

| Llama-3.3-70B | 0.3239 | 0.5484 | 100% | 0 |

| Kimi K2.6 | 0.3091 | 0.4710 | 100% | 0 |

| Nemotron-3-Super | 0.2760 | 0.4250 | 100% | 11 |

| 5-judge ensemble | 0.3458 | 0.5090 | 100% | 11 |

For context, asukul reports a 9-judge ensemble kappa of 0.4941 against the same NIST qrels, and their DeepSeek V4 Pro achieved kappa=0.4705 but with only 39% coverage (212/537 valid responses).

My DeepSeek V4 Flash (kappa=0.4095, 100% coverage) has overlapping confidence intervals with their DeepSeek V4 Pro (kappa=0.4705, 39% coverage). The coverage difference is the more practically significant finding — V4 Flash returns valid responses for every pair, V4 Pro fails on 61%.

My ensemble kappa (0.3458) is below asukul’s (0.4941). I attribute this to three factors, in approximate order of magnitude:

1. **Domain mismatch.** My judge prompt is calibrated for onion-network security content (journalists, investigators, security researchers). asukul’s prompt is 4 lines of generic RAG instructions. On academic TREC queries, my domain-specific calibration hurts — the “commercial pages = grade 0” rule and “intent fit = security research” criterion shift thresholds in ways that don’t match NIST assessor behavior on academic topics.

2. **Scale collapse.** asukul uses a 4-grade scale (0,1,2,3). I use a 3-grade scale (0,1,2) and collapse 2+3 → 2. This loses information exactly where my judges are weakest — they correctly identify Grade 3 documents 85% of the time but Grade 2 documents only 45% of the time.

3. **Ensemble size.** 5 judges vs. 9 judges. More judges reduce variance, though the effect is smaller than domain and scale.

I did not run the full 537 pairs because the domain mismatch is structural — more data would confirm the gap, not close it. The validation was worth doing to confirm that my judges produce reasonable kappa against human ground truth (0.41 for DeepSeek is “moderate” agreement, not random), but I would not use TREC RAG as a primary validation target for an onion-search benchmark.

## Nemotron error analysis

I mentioned last time that Nemotron had infrastructure failures. In the full 5-judge run: 85/1197 errors (7.1%), all HTTP 429 (NVIDIA NIM rate limiting). 10 of 11 errors in the TREC validation run were on high-relevance pairs (human grade 2 or 3).

The error latency pattern is consistent with rate limiting: Nemotron’s OK responses average 981ms, error responses average 1125ms — the request goes out, waits, then gets rejected. On my onion corpus (longer prompts, slower throughput), Nemotron had 0 errors in 509 documents during the initial run. The TREC corpus has shorter passages and faster request rates, which triggers the rate limit.

Ensemble impact: zero. In all 85 error cases, the other 4 judges had >=3 agreement, so Nemotron’s fallback vote (relevance=1) never changed a majority outcome. The 5-judge majority design is robust to single-judge failure, but this is luck as much as design — if two judges failed simultaneously on the same document, the remaining 3 would decide, and the fallback votes could matter.

## Items from the “not done yet” list

All closed except the Nemotron lift experiment:

| Item | Previous | Now |

|------|----------|-----|

| Nemotron query refinement lift | “not measured” | **Partially measured, not with your design.** I ran a frozen-ranking test on 2026-08-25 using heuristic qrels (not the calibrated LLM-judge qrels): nDCG -0.0015, MAP -0.0008, MRR -0.0139 — no lift, 23/24 queries unchanged. But this was only the frozen-ranking half, with provisional qrels. Your full design — frozen-ranking test + live-acquisition test, with calibrated qrels — has not been run. It’s the first item in the post-benchmark phase. |

| stdio purity test | “not implemented” | Done — 3 tests, stdout = JSON-RPC only |

| Tor stream isolation | “configured, not observed” | Still configured-only; ControlPort observation deferred |

| Per-engine timeout citation | “don’t have source” | Done — Tor spec `cbtinitialtimeout=60s` + Tor Stack Exchange q/21966 |

| judged@10 / judged@20 | “not yet implemented” | Done — rrf=1.0, rrf-quality=0.892 (quality filter pushes some unjudged docs into top-10) |

| Pairwise engine overlap matrix | “not done” | Done — 13×13 Jaccard, only excavator↔oss=0.55 correlated, ahmia 97.9% unique contribution |

| bpref / incomplete-judgment | “not needed yet” | Still not needed — judged@10 >=0.89 for all strategies |

## What I’m taking from this

The benchmark answered the question I needed it to answer: **which ranking strategy should production use, and is that answer robust to judge-ensemble composition?** The answer is rrf-quality, and yes, it’s robust — the same strategy wins on both a 3-judge qrels with a known parser bug and a 5-judge qrels with the fix and a different ensemble composition.

The external validation against NIST human qrels confirmed that my judges produce moderate agreement with human assessors (DeepSeek kappa=0.41), which is not validation of correctness but is validation of non-randomness. The gap to asukul’s 9-judge ensemble (0.49) is attributable to domain and scale differences, not judge quality.

The qrels remain LLM-generated and provisional. I have not done human review of any of the 1197 pairs, and I won’t claim correctness I haven’t measured. What I can claim is that the ranking conclusion is stable across two independent judge ensembles, and that the judges produce moderate agreement with external human ground truth on an out-of-domain benchmark.

The next phase is production features — applying the benchmark findings to the actual search system. The benchmarking cycle is complete, with one exception: the Nemotron query refinement lift experiment you requested in post 4 (frozen-ranking test + live-acquisition test with calibrated qrels) has only been partially run — a frozen-ranking test with heuristic qrels showed no lift, but the full design with LLM-judge qrels and the live-acquisition arm is still open. It’s the first item on the post-benchmark roadmap.

Thanks again for the calibration sequence. It shaped the pipeline from start to finish, and the ranking-stability result is directly attributable to your emphasis on diverse pooling and double-annotation before full re-annotation.

John6666 · August 28, 2026, 12:53am

I think the concern is warranted:

My short answer would be: the motivation for broadening the judge panel is sound, but I would treat “3 Chinese-origin + 2 Western-origin” as an initial diversity heuristic, not as the validity argument itself. The stronger argument is whether those five judges, on this benchmark, actually have complementary failure modes and produce more stable retrieval conclusions.

There is good evidence for the mechanism you are worried about. Campbell et al.'s Defensive Refusal Bias study used 2,390 real NCCDC examples and found that defensive requests containing security-sensitive language were refused at 2.72× the rate of semantically equivalent neutral requests. The July 2026 Hugging Face incident disclosure is also a useful real-world example: HF says the commercial API models they initially tried blocked substantial forensic analysis of real attack commands/exploit/C2 material, so they moved the analysis to self-hosted GLM-5.2. Their technical timeline gives more detail.

But neither result establishes:

Chinese-origin judge -> more accurate relevance labels
Western-origin judge -> systematically worse relevance labels
3 Chinese + 2 Western -> optimal panel

They establish a failure mechanism worth controlling for. I would therefore keep the heterogeneous five-judge panel, but measure the judges as instruments rather than using origin as the proxy for validity.

The highest-information path from where you are now seems to me to be:

old original-3 + contaminated parser
        ↓
same original-3 + fixed parser
        ↓
all-5 + fixed parser
        ↓
all 3-of-5 subsets + all 4-of-5 LOJO panels
        ↓
small stratified human anchor
        ↓
winner / pairwise system-order stability
        ↓
stabilized qrels → rerun ranking ablations

The first distinction is especially important:

old-3  vs fixed-3 = parser/error-semantics effect
fixed-3 vs fixed-5 = panel expansion/composition effect

Comparing the contaminated 3-judge qrels directly with the fixed-parser 5-judge qrels is still useful as an end-to-end before/after result, but it cannot by itself quantify how much came from the parser fix versus adding GLM/Kimi.

If you retain the five raw per-judge labels/statuses, almost everything after the fixed five-judge run is cheap offline analysis rather than more inference.

Judge composition: I would measure the judges, not the passport category

1. First deconfound parser repair from panel expansion

I would explicitly preserve three qrel versions:

A = original 3 judges + original parser
B = original 3 judges + fixed parser
C = all 5 judges       + fixed parser

Then:

A -> B
measures:
parser fallback / retry / infrastructure-error contamination

B -> C
measures:
additional judges + changed aggregation behavior

That distinction matters here because the old parser was not injecting random noise. Mapping empty/unparseable responses to grade 1 creates a directional bias toward the middle relevance grade.

So I would treat these separately in the artifact:

valid grade
refusal
empty response
API/provider error
timeout
parse failure
retry exhausted

and never convert the last five into relevance grades.

A missing judge output is an availability/policy/infrastructure observation, not evidence that the document is marginally relevant.

2. Once the five labels exist, enumerate the panel space offline

With five fixed-parser raw judge outputs, you can generate:

all-five panel

C(5,3) = 10 different 3-of-5 panels

5 different 4-of-5 leave-one-judge-out panels

without another model call.

For each panel I would compare at least:

qrel flip rate
0↔1 changes
1↔2 changes
extreme 0↔2 changes
unresolved / insufficient-quorum count
grade distribution
nDCG@10 / MAP / Recall
winner
top-2 ordering
every pairwise retrieval-system order

The last few are probably more useful than raw label agreement alone.

For example, suppose two panels disagree on 15% of individual document grades, but both produce:

RRF-trust
    >
RRF-plain
    >
BM25
    >
engine-count

for essentially every system comparison. That is a substantially more reassuring result for a retrieval benchmark than 95% document-level agreement accompanied by a winner flip.

Conversely, if removing one judge changes the winner or several pairwise system comparisons, that judge deserves direct inspection even if the overall five-judge agreement coefficient looks good.

This is also where leave-one-judge-out becomes very interpretable:

remove judge J
    ↓
how many qrels change?
how many retrieval-system pairwise orders change?
does the winner change?

If one judge has disproportionate influence, audit that model.

If all ten 3-of-5 subsets and most 4-of-5 panels produce essentially the same retrieval conclusions, I would probably stop optimizing the geographic ratio. At that point the empirical result is stronger than the taxonomy.

3. “Diversity” should mean diversity of error, not diversity of labels

There is independent work supporting multi-judge relevance assessment in general. For example, JudgeBlender explicitly ensembles multiple LLMs or multiple prompts to reduce single-judge bias and obtains competitive automatic relevance assessments.

That supports the ensemble instinct.

But the useful diversity variable is something like:

pairwise judge disagreement
shared human-label error
refusal/error correlation
grade-distribution correlation
terminology sensitivity
LOJO influence

rather than:

country-of-origin count

because two models from different organizations/countries can still share the same failure, while two models from the same broad region may behave quite differently.

A table like this would answer your methodological question much more directly:

Judge	Parse/error %	Refusal %	Mean grade	Human agreement	Security-wording delta	LOJO system-order effect
DeepSeek
gpt-oss
Nemotron
GLM
Kimi

Then “why these five?” has an empirical answer.

4. I would add one tiny paired wording control

Because the hypothesis is specifically defensive refusal / safety-language sensitivity, you can test the mechanism much more directly than by comparing national origins.

For a small subset of hand-checked cases, prepare semantically equivalent formulations:

security-sensitive wording
vs
neutral wording expressing the same defensive information need

and send both to every judge with the same result surrogate.

Measure separately:

refusal / empty / parse-error shift
grade shift
0↔1 / 1↔2 boundary shift
reasoning change

You do not need to neutralize all 24 topics. A carefully checked sample of the categories you expect to stress safety behavior is enough to tell whether the mechanism is present in your judge panel.

That is close to the experimental logic of Defensive Refusal Bias: hold intent approximately constant, alter the security-sensitive framing, and observe the change.

Possible outcomes:

no meaningful judge-specific wording effect
    -> geography probably is not doing useful work here

one or two judges shift strongly
    -> audit/weight/replace those judges

all judges shift similarly
    -> adding geographically diverse judges did not remove the shared failure

different judges fail on different examples
    -> heterogeneous ensemble may genuinely be buying robustness

This is, to me, a cleaner answer to the concern than trying to infer safety behavior from model provenance.

5. Predeclare quorum/tie semantics too

With three judges, 2/3 majority is easy until all three choose different ordinal grades.

With five judges, distributions like:

2 votes grade 0
2 votes grade 1
1 vote  grade 2

have no strict 3/5 majority.

And once errors/abstentions are allowed, the denominator itself can change.

I would therefore freeze:

minimum number of valid judges
missing/error semantics
tie semantics
insufficient-quorum semantics

before comparing panel variants.

For graded 0/1/2 relevance, one useful sensitivity baseline is simply the ordinal median of valid judges, because it avoids giving a named tiebreaker extra influence. I would not necessarily replace your majority rule with it, but comparing:

majority + unresolved
vs
ordinal median

is almost free.

If an item lacks sufficient valid judges, I would make it:

unjudged / audit queue

rather than manufacture a relevance label.

Small human anchor + system-level validity

I think your sentence

agreement is not accuracy

is exactly the right boundary.

I would now resist the temptation to solve that by adding still more automatic judges. The next highest-information addition is a small human anchor, not full human annotation.

1. Human-label the cases where human information has the most value

Something on the order of tens of examples can already be informative; if 50–100 is feasible, even better.

I would stratify rather than sample only uniformly:

judge disagreements
0-vs-1 boundaries
1-vs-2 boundaries
previous refusal/error/parser cases
security-sensitive wording cases
q18 / SecureDrop / journalism edge cases
unique-contribution pool documents
cases where one judge is the LOJO pivot
random unanimous/easy cases as controls

If possible, have humans blind to:

which ranker produced the document
which judge gave which grade
which pool arm introduced it

so the anchor does not simply reproduce the automatic pipeline’s assumptions.

You do not need humans to replace the LLM panel.

LARA: LLM-Assisted Relevance Assessments is a useful conceptual reference here: use limited human judgments where they provide the most information, then use them to calibrate/debias cheaper automatic assessment rather than pretending the automatic labels are perfect ground truth.

2. Judge the benchmark at two levels

This distinction seems particularly important after TREC 2025’s RAG track.

For its automatic Relevance Judgment task, even the best automatic submissions only achieved roughly 0.30–0.34 direct agreement fraction with human relevance judgments.

But elsewhere in the same evaluation, automatic assessments reproduced run-level ranking behavior much more strongly: aggregated retrieval-system scores had high Kendall rank agreement with manual evaluation even while individual/narrative-level judgments were noisier.

That suggests two separate validity questions:

1. label validity
How often do these qrels agree with a human anchor?

2. evaluation validity
Do these qrels lead us to the same conclusions about retrieval systems?

Ghost primarily needs the second property if the benchmark’s purpose is:

compare BM25 / RRF / trust / agreement / rerankers reproducibly.

So for every qrel variant I would report both.

Label-level

human agreement
weighted confusion matrix
per-grade precision/recall if sample size allows
judge-specific false-low / false-high behavior

System-level

Kendall / Spearman across all systems
winner preserved?
top two preserved?
all pairwise system-order flips
per-query source of each flip

With only a handful of ranking arms, I especially like the pairwise matrix because it is very hard to hide a practical instability inside a single correlation coefficient.

For example:

                 qrels A    qrels B    human-anchor-corrected
RRF > BM25          yes        yes             yes
RRF > engine        yes        yes             yes
BM25 > engine       yes        no              yes
...

That immediately tells the reader where the benchmark conclusion is fragile.

3. A useful stop rule

I would stop spending effort on judge-composition optimization when something like this is true:

fixed-3 and fixed-5 produce same winner
most/all 3-of-5 subsets preserve pairwise system order
LOJO does not expose one dominant judge
small human anchor finds no systematic panel failure
security-sensitive wording control does not expose a hidden directional bias

At that point “is 3 Chinese / 2 Western exactly right?” becomes much less important.

You have directly shown that the result is not sensitive to reasonable judge-panel perturbations.

If the system ordering isn’t stable, then the raw per-judge matrix tells you where to look before investing in more retrieval machinery.

Qrels and pooling: I would stabilize this before interpreting the ablation numbers strongly

The four-arm pool is a meaningful improvement, especially the unique-contribution arm.

The fact that it contributes 430 documents not surfaced by the agreement/fusion arms is already useful because it shows why:

pool generator

and:

production ranker

do not have to be the same thing.

The next question is simply:

how many of those 430 are relevant?

If an arm is mediocre as a production ranking strategy but contributes many uniquely relevant documents to the judgment pool, it is doing valuable benchmark work.

I would eventually report:

unique candidates by arm
judged unique candidates
unique relevant candidates
unique relevant@equal-depth

rather than only raw unique count.

1. Be careful about provisional qrels that favor their own pool contributor

The classic problem is well documented in the IR literature.

NIST’s Bias and the Limits of Pooling for Large Collections discusses how incomplete/shallow pools can create biased relevance sets, and Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements explicitly notes that pooling is inherently capable of disadvantaging systems that did not contribute judged documents.

The nuance is important: pooling is not inherently unusable. Well-designed, sufficiently deep and diverse TREC pools have often proved surprisingly reusable. The risk grows when:

the pool is shallow
the contributing systems are homogeneous
a new system retrieves many unseen documents
unjudged = nonrelevant

So I would not throw away the current results.

I would simply make the fully judged four-arm pool the boundary between:

exploratory ablation

and:

evidence about component effectiveness

If the heuristic/provisional qrels used for the current table were heavily influenced by trust-weighted RRF candidates, then numbers such as:

rrf-trust +0.1203
BM25 -0.0494

are excellent hypotheses for what to re-test, but I would avoid interpreting the exact delta causally until the diverse pool is judged.

Your planned:

judged@10
judged@20

is exactly useful here.

For every evaluated system, I would report:

judged@10
judged@20

alongside nDCG/MAP whenever the system was not strongly represented in pool construction.

If a new ranker looks worse while half its top 20 is unjudged, that is a different diagnosis from:

its judged top 20 is genuinely less relevant.

A cheap additional robustness check is leave-one-pool-arm-out: see how much the resulting system comparisons depend on the unique judgments introduced by each pool mechanism.

2. Freeze the information need, not just the query

The 24 TREC-style narratives are a good move.

There is now direct experimental support for this in Formalized Information Needs Improve Large-Language-Model Relevance Judgments: LLM assessors given richer retrieval topics/descriptions/narratives show better agreement and more reliable relevance assessment than query-only assessors.

This seems particularly important for short Ghost queries such as:

zero-day
secure drop
bitcoin

where two judges can apply the same rubric while silently assuming different information needs.

I would freeze and version:

query
description
narrative
grade examples
judge prompt
rubric

before the final annotation pass.

And if narratives were initially drafted using examples from the current retrieval pool, I would simply record that provenance and make the reviewed/frozen narrative version the benchmark definition. The important thing is that the topic definition does not keep drifting after you start interpreting final ranker scores.

3. Keep topical relevance separate from quality-policy attributes

One design separation I would keep is:

relevance_grade

versus things such as:

commercial_intent
generic_directory
duplicate/mirror
quality_flag
unjudgeable

The reason is not that commercial pages or directories should rank highly.

It is that their relevance can depend on the topic.

For example, if the information need is investigating the existence or characteristics of an underground market, a marketplace listing can itself be relevant evidence even if production Ghost should later down-rank it for user-quality reasons.

If:

qrels:
commercial -> low relevance

and

quality ranker:
commercial -> lower score

then part of the quality ranker’s evaluation advantage is built into the relevance definition.

That is avoidable.

A cleaner decomposition is:

retrieval question:
is this useful evidence for the stated information need?

quality/policy question:
is this the kind of evidence Ghost should prefer in production?

duplication question:
is this independent information or another copy?

Then you can report both relevance effectiveness and policy effectiveness without making them circular.

For q18 specifically, your refined research-intent narrative may legitimately make many commercial “hire a hacker” pages irrelevant. I would just make that topic-specific consequence of the information need, rather than a universal commercial-page law.

Ranking ablations: make each arm answer one question

The current provisional matrix is already useful because it tells you where the large deltas may be.

Once the qrels stabilize, I would rerun it with maximally boring/atomic arms.

Something like:

pure lexical BM25
    ↓
BM25 + engine agreement
    ↓
BM25 + engine agreement + trust

and separately:

raw-rank RRF
    ↓
RRF + snippet-based ordering
    ↓
RRF + trust
    ↓
RRF + trust + quality

The point is not that Ghost should become a collection of isolated components in production.

It is that an evaluation arm named:

BM25

should ideally answer:

what does lexical BM25 contribute?

and:

RRF

should answer:

what does rank fusion contribute?

If the “plain” arm still contains another ordering heuristic, then the subsequent delta is harder to attribute.

The final production configuration can still combine everything that wins.

The experiment is simply easier to interpret.

I would keep three outputs for each ablation:

mean metric
per-query delta
system-order impact

because the mean can hide a very asymmetric pattern such as:

trust helps 20 queries modestly
but catastrophically hurts 4

or the reverse.

That per-query heterogeneity is especially interesting with only 24 topics.

One evaluator/ranker interaction I would keep in mind

If the same Nemotron model/family contributes both:

relevance judgments

and:

an LLM reranker being evaluated

I would add one independent control before treating a positive reranker delta as settled.

Balog, Metzler & Qin, “Rankers, Judges, and Assistants” reports empirical evidence that LLM judges can be biased toward LLM-based rankers and can have difficulty resolving subtle ranking differences.

That does not mean a Nemotron reranker improvement is fake.

It means the strongest version of the result would be:

Nemotron reranker wins under all-five qrels

and still wins under:
    qrels excluding Nemotron
    or
    the small human-anchor subset
    or
    another independent judge variant

That is another cheap “remove the evaluator from the evaluated system” control.

A few small reproducibility checks I would keep, but below the judge question

These are secondary to the qrels work, but they are cheap enough that I would keep them in the research artifact.

1. Clarify the 217/289 → 239/272 transformation

One small accounting detail caught my eye.

You report:

raw:       217 / 289
corrected: 239 / 272

and describe the corrected denominator as excluding 17 Nemotron infrastructure failures.

Dropping 17 rows alone can reduce the denominator from 289 to 272, but it cannot increase the number of agreements from 217 to 239.

So I assume “corrected” includes something additional such as:

rerunning failures
replacement judgments
post-parser-fix rejudging

If so, I would just record that transformation explicitly.

Something like:

raw artifact
-> identify N infrastructure/parser failures
-> rerun/reparse according to rule X
-> corrected artifact

That will make the calibration table much easier for another researcher to reproduce.

This is bookkeeping rather than a methodological problem.

2. Keep parser/error provenance in the qrels artifact

For each judge/item, I would retain:

raw model output
parsed grade
valid/invalid
retry count
error class
fallback used?   # ideally always false in final qrels
model/backend/version
prompt/rubric version

Then the final aggregated qrel can always be reconstructed.

A compact provenance manifest could hash:

frozen corpus
queries
narratives
judge prompt/rubric
pool
raw judge matrix
final qrels

You are already doing much of this, so I would continue in that direction.

3. REST/stdio semantic parity and stdio purity are different tests

Your 220/220 semantic-parity fixture is useful.

I agree with your own note that stdio purity is a separate contract.

The official MCP stdio transport specification reserves stdout for valid MCP messages; diagnostics belong on stderr.

So the final cheap smoke test is something like:

spawn actual MCP subprocess
initialize
invoke ghost_search through a controlled/mock engine fixture

assert:
every stdout frame is valid JSON-RPC/MCP

allow:
diagnostics on stderr

That catches a class of failure that an in-process semantic-parity test cannot.

I would not block qrels work on it.

4. Tor configured-vs-observed is already the right separation

I think your wording here is now basically right:

SafeSocks + per-search isolation credentials
    = mechanism/configuration evidence

ControlPort StreamID -> CircuitID observation
    = runtime evidence that the actual requests were isolated

The latter is stronger if you eventually want an audit-grade runtime claim; it does not need to be part of the ranking benchmark.

Likewise, if the 60–90 second timeout value is empirically chosen rather than traceable to a Tor Project recommendation, describing it as an empirical operational default is cleaner than forcing a citation.

How I would interpret the five-judge question after all of those controls

For me, the decision tree would be approximately:

Does fixed-3 differ substantially from old-3?
    |
    yes
    -> parser contamination was material
    -> do not use old-3 vs fixed-5 to infer ensemble benefit

Does fixed-5 improve human-anchor behavior over fixed-3?
    |
    yes
    -> panel expansion has empirical value
    |
    no
    -> five judges may still improve robustness, but not demonstrated by accuracy

Do different 3-of-5 subsets / LOJO panels preserve retrieval-system order?
    |
    yes
    -> panel composition is not a major benchmark fragility
    |
    no
    -> identify which judge/examples produce the flips

Do security-sensitive vs neutral paired prompts reveal judge-specific shifts?
    |
    yes
    -> measure/mitigate that judge behavior directly
    |
    no
    -> geographic balancing is probably solving a problem not visible in this benchmark

Does a small human anchor reveal shared five-judge errors?
    |
    yes
    -> calibrate rubric/prompt/panel against them
    |
    no
    -> much stronger case for keeping automatic provisional qrels

Do qrel variants preserve the same retrieval winner / pairwise ordering?
    |
    yes
    -> benchmark is probably already useful for comparative IR work
    |
    no
    -> understand the unstable topics before making ranker claims

So I would characterize your current reasoning as:

sound motivation, reasonable ensemble experiment, but potentially over-specific if “3 Chinese / 2 Western” becomes the explanation for validity.

The strongest version of the story is instead:

Security-adjacent relevance assessment may have policy/refusal failure modes, so I deliberately used a heterogeneous judge panel; then I measured each judge’s failure behavior, compared panel subsets, anchored uncertain cases to a small human sample, and verified that the retrieval-system conclusions were stable.

That is a much harder methodological story to attack than:

I chose a 3:2 geographic ratio because Western models might refuse security content.

And importantly, it preserves the exact thing you are trying to protect: a benchmark that does not silently grade down the security material its users actually need.

If I were minimizing work from your current state, my default order would now be:

1. Finish the fixed-parser five raw judge matrix.

2. Re-run the original same three judges with the fixed parser.
   This is the critical missing deconfound.

3. Offline:
   all 3-of-5 subsets
   all 4-of-5 LOJO panels
   fixed-3 vs fixed-5

4. Run a small paired security-sensitive/neutral wording control.

5. Human-label only the high-information sample:
   disagreements + boundaries + errors + security cases + random controls.

6. Compare:
   label agreement
   AND
   winner / pairwise retrieval-system ordering.

7. Finish judging the diverse four-arm pool and report judged@K.

8. Re-run the atomic ranking matrix against those stabilized qrels.

9. Only after that, move on to Nemotron query-refinement lift / neural reranking.

At this point I would not add a sixth judge, a more complicated voting scheme, or a heavier reranker just to increase apparent rigor.

You already have enough machinery.

The next gain is mostly from separating the variables you already have.

sookoothaii · August 30, 2026, 12:48am

Thanks for the clarification. I fully agree with the distinction: origin is a diversity heuristic, not the validity argument. The stronger claim is whether the five judges actually have complementary failure modes and produce more stable retrieval conclusions. That’s the frame I’ve used.

I ran your 9-step sequence in the order you gave. Where things stand:

Steps 1–3 (parser/ensemble deconfound). All three arms done:

A: original 3 judges + old parser
B: same 3 judges + fixed parser
C: all 5 judges + fixed parser

The critical deconfound you flagged (B) is included. All three arms: rrf-quality wins, same pairwise order RQ > RRF > BM25 > EC. I then ran all 10 3-of-5 subsets and all 5 leave-one-judge-out panels offline — as you said, cheap once the 5-judge matrix exists. Across all 17 panels, winner and pairwise order are invariant. Document-level qrels do vary — up to 32.8% flip rate in subsets that exclude DeepSeek — but that variation doesn’t propagate to the system-level conclusion.

Step 4 (wording control). You asked for this specifically, so the result here is the relevant one. Paired design: 6 queries, 30 stratified documents (1× grade 2, 2× grade 1, 2× grade 0 per query), 5 judges — 300 API calls, ~$0.217. Each document judged by the same judge under two framings: a security-sensitive information need (“I need this for investigating ransomware operations…”) and a neutral one (“I need this for general research about ransomware”). Document identity and judgment prompt were identical; only the framing differed.

Per-judge deltas (security minus neutral, accuracy shares):

Judge	Security	Neutral	Delta
DeepSeek-R1	0.800	0.833	-0.033
GPT-OSS-120B	0.833	0.800	+0.033
Nemotron-3-Super	0.800	0.800	0.000
Llama-3.3-70B	0.867	0.833	-0.033
Kimi K2.6	0.867	0.700	-0.167

No systematic defensive refusal penalty on Western judges. Kimi is an outlier, but in the opposite direction from the hypothesis — it grades less strictly under security framing, not more. Cultural-group comparison: Chinese judges -0.067, Western judges -0.022, gap 0.044. The refusal mechanism Campbell et al. identified (2.72× refusal rate) is real, but it doesn’t manifest in judge grades on these queries and these documents. I’m not generalizing this to other models or other content domains — it’s a benchmark result, not a universality claim.

Step 5 (human anchor). This is the one step I haven’t completed. I don’t have a human annotation pipeline. The closest substitute was validation against asukul’s RAG-Eval-LLM-Judge (NIST TREC RAG, 80-pair sample) — DeepSeek-R1 reached kappa=0.41 against human assessors, moderate, non-random. But that’s out-of-domain (TREC RAG isn’t onion security content), so it’s only a weak proxy. A small in-domain human anchor set over disagreements + boundary cases + security cases remains the highest-value missing validation.

Steps 6–8 (system-level comparison, judged@K, ranking rerun). All done. judged@10: rrf=1.0, rrf-quality=0.892 (the quality filter pushes some unjudged docs into top-10). Atomic ablations reran on both qrels — winner and pairwise order stable across v2 and v3.

Step 9 (query refinement lift). This was the last open step. I ran it with two separate arms, because you were right that acquisition and ranking are different questions:

Frozen-candidate arm: Same document set, ranked with raw vs. expanded query. BM25 nDCG@10 -0.0655 (term dilution hurts). RRF and RRF-quality are controls here — both +0.0000, since candidates are identical and RRF doesn’t use query terms.
Live-acquisition arm: 24 query pairs over 13 live onion search engines via Tor, 624 requests, ~30 minutes. Expanded queries retrieve 4.26x fewer documents (mean 1110 vs 260) with 6% pool overlap (Jaccard). All four strategies show negative lift:

Strategy	Raw nDCG@10	Expanded nDCG@10	Lift
engine-count	0.2660	0.0500	-0.2160
BM25	0.2915	0.0615	-0.2300
RRF	0.3474	0.0242	-0.3232
RRF-quality	0.4324	0.0601	-0.3723

The mechanism is twofold: (1) onion search engines treat long queries as AND/exact-phrase and return fewer results; ahmia errors on 10/24 expanded queries, tordex on 19/24. (2) The different documents they do return have only 6% overlap with the raw pool, so most qrels-annotated relevant documents are lost. Both arms agree: no lift. Production config keeps raw queries.

What I don’t claim. The qrels remain LLM-generated and provisional. I haven’t human-reviewed any of the 1197 pairs. Panel robustness (17 panels, invariant conclusion) and NIST validation (moderate agreement, non-random) are what I have — they’re robustness and non-randomness evidence, not correctness evidence. A small in-domain human anchor set (step 5) is the validation that would strengthen this most, and I don’t currently have a way to do it.

What I took from your advice. The “measure judges as instruments, not by origin” framing shaped the wording control design — test whether the failure mechanism shows up in actual outputs rather than extrapolating from origin. The deconfound step ordering (old-3 → fixed-3 → fixed-5) was useful: the end-to-end old-3 vs fixed-5 comparison is informative as a before/after, but the internal comparisons (old-3 vs fixed-3 for parser, fixed-3 vs fixed-5 for ensemble) tell me the parser fix was the larger effect and adding GLM/Kimi didn’t change the conclusion. Separating variables rather than adding judges was the right call.

I didn’t add a sixth judge, a heavier reranker, or a more complex voting scheme. The conclusion is stable within the 5 judges I have. The next phase is applying the findings to the production system, not more benchmark iteration.

sookoothaii · August 30, 2026, 3:49am

Step 5 is done. Here’s the result.

Human anchor annotation. 288 items, stratified per your specification:

Stratum	Count	Purpose
3-2 splits	207	Panel genuinely divided — human label is decisive
Boundary cases (final=1 with dissent)	50	Grade threshold unclear
Unanimous grade 2	11	Positive control
Unanimous grade 0	20	Negative control

Single annotator — domain expert with onion/security research experience. 62 documents had no snippet and required Tor page fetch to assess. I built a local annotation tool with auto-save and resume so the work could be done iteratively across sessions.

Cohen’s kappa (human vs each LLM judge):

Judge	Kappa	Interpretation	Agreement
DeepSeek-R1	0.842	almost perfect	89.6%
GPT-OSS-120B	0.340	fair	55.6%
Llama-3.3-70B	0.336	fair	55.4%
Nemotron-3-Super	0.117	slight	38.5%
Kimi K2.6	0.074	slight	35.1%
LLM Majority	0.454	moderate	62.5%

DeepSeek-R1 is the only judge with “almost perfect” human agreement. Kimi (0.074) and Nemotron (0.117) are unreliable as standalone judges — Kimi due to the max_tokens=512 truncation artifact I documented earlier, Nemotron for reasons I haven’t investigated yet. The majority vote (0.454, moderate) is better than any individual judge except DeepSeek, but still disagrees with the human on 37.5% of cases.

Fleiss’ kappa (human + 5 LLM judges together): 0.389 (fair).

Controls confirmed. Unanimous grade 0: 100% agreement (20/20). Unanimous grade 2: 100% agreement (11/11). The human agrees with the LLM panel on clear cases — the annotation task was calibrated correctly.

Per-stratum agreement (human vs majority):

3-2 splits: 60.4% — the human disagrees with the LLM majority on 40% of genuinely disputed cases.
Boundary cases: 48.0% — the human disagrees on half, which is expected for grade-threshold-ambiguous documents.

Per-category agreement:

threat_intel: 56.1% — lowest, security content is genuinely harder to judge.
journalism: 67.2%.
privacy: 66.4%.

System-order rerun. I replaced 95 majority labels with human labels in qrels_v3 (where the human disagreed) and reran all 4 ranking strategies on the frozen SERP corpus:

Strategy	Original nDCG@10	Human-corrected nDCG@10	Delta
engine-count	0.2762	0.2776	+0.0014
BM25	0.2915	0.2811	-0.0105
RRF	0.3592	0.3478	-0.0114
RRF-quality	0.4594	0.4080	-0.0515

Absolute nDCG values drop slightly — the human labels are somewhat stricter than the LLM labels. But the system ordering is preserved:

Original: rrf-quality > rrf > bm25 > engine-count
Human-corrected: rrf-quality > rrf > bm25 > engine-count

The ranking conclusion survives human correction of 95 disputed labels.

What this validates and what it doesn’t. This validates that the system-level ranking conclusion is robust to human correction of the most contested labels. It does not validate that every individual qrel label is correct — the human disagreed with the LLM majority on 108 of 288 items (37.5%), and 95 of those changed labels. What it shows is that label-level disagreement doesn’t propagate to the system-level conclusion, which is the question you asked in step 6.

The single-annotator design is a limitation — there’s no human-human kappa, only human-vs-LLM kappa. A second annotator would strengthen this. But the controls (100% on unanimous cases) and the DeepSeek kappa (0.842) suggest the annotation is reliable.

Where the full sequence stands now:

All 9 steps are done. The ranking conclusion (rrf-quality > rrf > bm25 > engine-count) is stable across:

17 panel compositions (A/B/C deconfound + 10 3-of-5 subsets + 5 LOJO)
Wording control (no defensive refusal bias)
Human anchor (95 corrected labels, order preserved)
Query refinement (no lift in either arm)

The benchmark cycle is closed. The next phase is applying the findings to the production system.

John6666 · August 30, 2026, 7:35am

When I replied last time, I could only see one of the posts… And somehow the repo seems to have disappeared again!? But anyway, I think we can probably call the benchmark complete:

At least for the question you originally needed the benchmark to answer —

which ranking configuration should Ghost use in production, and is that choice robust to reasonable changes in the qrels/judges?

—I think you now have enough evidence to close this benchmark cycle and move on.

The important result is no longer just that rrf-quality has the highest mean nDCG. It is that the same production conclusion survived several deliberately different perturbations:

old 3-judge + contaminated parser
        ↓
same 3 judges + fixed parser
        ↓
5 judges + fixed parser
        ↓
all 3-of-5 subsets
        ↓
all leave-one-judge-out panels
        ↓
security-sensitive / neutral wording control
        ↓
288-item human stress sample
        ↓
95 qrel corrections

and the winner remained:

rrf-quality > rrf > bm25 > engine-count

That is much more informative than simply accumulating another judge or another reranker.

In particular, the human-anchor result closes the loop I was most interested in: the individual labels can be quite noisy while the system-level conclusion remains stable. That distinction is well established in IR evaluation. TREC work has long shown that assessor disagreement need not translate into the same degree of instability in comparative system evaluation; the practical question is whether the retrieval-system conclusions survive reasonable judgment perturbations, not whether every relevance label is universally agreed upon. See, for example, Voorhees, “Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness”.

So my default path from here would actually be:

freeze benchmark v1
        ↓
freeze production configuration
        ↓
use benchmark v1 as a regression suite
        ↓
move development effort back to production/search behavior

rather than continuing to enlarge the current benchmark until it becomes its own project.

I would keep only a few publication/reusability hardening items separate from the production decision. None of them looks like a reason to hold production work.

Question	My current read
Which production ranker?	Answered: `rrf-quality`
Does parser contamination explain the winner?	No: fixed-parser comparison preserves it
Does one particular 5-judge composition determine the winner?	Apparently no: subset/LOJO stability
Did the hypothesized Western-judge grade penalty appear here?	Not in the paired grade-control result
Does human correction overturn the winner?	No
Should query expansion be enabled now?	No; raw queries are the safer production choice
Is every qrel now “validated ground truth”?	No, and it does not need to be for the production decision
Is the collection publication-ready without any caveats?	Almost; a few cheap documentation/coverage closures remain

Why I think the benchmark cycle can close

1. The deconfound worked

The most important methodological improvement since the previous post is that you actually ran the missing middle condition:

A = old 3 judges + old parser
B = same 3 judges + fixed parser
C = 5 judges     + fixed parser

So the two effects are no longer silently mixed:

A -> B
parser/error-semantics effect

B -> C
judge-panel/composition effect

That was the main thing missing from the earlier old-3 vs fixed-5 comparison.

The fact that the system winner survives A/B/C means you no longer need to infer robustness from an end-to-end before/after comparison.

2. The panel-composition question is basically answered empirically

The original concern was:

could security-adjacent material be systematically downgraded because some judges have different refusal/safety behavior?

The right response was to measure the judges rather than infer validity from model origin, and the new experiments do that.

You now have:

10 × 3-of-5 subsets
5 × leave-one-judge-out panels
paired security/neutral framing
per-judge agreement behavior
human anchor

and the production ranking survives the panel perturbations.

That is a stronger result than saying:

3 Chinese-origin + 2 Western-origin

is intrinsically better.

In fact, the paired wording test is useful specifically because the expected Western-only grade penalty did not appear.

I would phrase that result narrowly:

On these queries/documents, this paired control did not show a systematic security-framing grade penalty concentrated in the Western judges.

rather than:

defensive refusal bias does not exist.

The latter would go beyond the experiment. Campbell et al.'s Defensive Refusal Bias result is about refusal behavior under semantically equivalent security-sensitive vs neutral prompts; your experiment is a benchmark-specific paired relevance-judgment control. The two fit together nicely precisely because yours asks whether that known mechanism is actually visible in this evaluation pipeline.

And apparently, at least in the public results you reported, it is not driving the ranking conclusion.

That means the origin-composition question can probably be retired unless a future model change reopens it.

3. Document-level instability with system-level stability is not a contradiction

The human result is especially informative here.

On the deliberately difficult 288-item sample:

human vs LLM-majority agreement = 62.5%
108 / 288 disagreements
95 qrel labels changed

That sounds alarming if the target is:

every LLM qrel should reproduce a human judgment.

But then the ranking rerun gives:

original:
rrf-quality > rrf > bm25 > engine-count

human-corrected:
rrf-quality > rrf > bm25 > engine-count

So the benchmark is telling you something subtler:

label-level validity:
still imperfect

comparative system-level validity:
substantially more stable

For a benchmark whose immediate purpose is choosing among retrieval systems, the latter is the property that matters most.

This is why I would not keep adding automatic judges just to raise the apparent agreement number.

4. I would make the current corpus immutable now

At this point I would version the current state as something like:

ghost-serp-benchmark-v1

frozen candidate corpus
frozen 24 topics/narratives
frozen qrels
frozen human corrections
frozen ranker parameters
frozen evaluation implementation

and stop changing it in-place.

Then production work can use it as a regression suite:

change production code
        ↓
replay benchmark v1
        ↓
did effectiveness regress?
did candidate identity change unexpectedly?
did judged@K collapse?

If you later want stronger scientific generalization, create:

benchmark v2

from new queries and/or a fresh acquisition snapshot.

That is cleaner than continuing to tune and re-label v1 forever.

The human anchor is strong evidence, but I would keep two interpretation boundaries explicit

1. It is a stress sample, not a random estimate of 1,197-item accuracy

Your sampling deliberately emphasizes:

207 × 3-2 splits
50 × boundary cases
11 × unanimous grade 2
20 × unanimous grade 0

That is exactly the kind of sample I would want for stress-testing the system conclusion.

But because disputed cases are intentionally overrepresented, numbers such as:

DeepSeek kappa = 0.842
LLM-majority kappa = 0.454

should not automatically be interpreted as estimates of what those kappas would be over all 1,197 documents.

I would describe them as:

agreement on a stratified, disagreement-heavy human audit sample.

That is actually a useful strength, not a weakness: you pointed human effort at the cases most likely to change conclusions.

2. The 62 page-fetched items are not exactly the same relevance task

This is worth preserving in the benchmark metadata.

You said 62 items lacked enough snippet evidence and the human assessor fetched the Tor page.

If the LLM judges saw only:

URL
title
snippet

while the human saw:

URL
title
snippet
full page

then those 62 comparisons contain two changes at once:

different assessor
+
more evidence

There is prior IR work showing that full text can change human relevance judgments relative to URL/title/snippet summaries. Kazai et al., “Less is Less: When Are Snippets Insufficient for Human vs Machine Relevance Estimation?” studies exactly this distinction.

So if the underlying rows are still available, one essentially free analysis would be:

226 surrogate-only human judgments
vs
62 page-assisted human judgments

and report agreement separately.

No new annotation is required.

If the page-assisted subset has a much higher LLM/human disagreement rate, that may simply tell you:

some SERP surrogates do not contain enough evidence.

That is a useful broker-level measurement in its own right.

It could eventually justify:

judgment_status = insufficient_surrogate_evidence

instead of forcing every candidate into 0/1/2.

3. One human assessor is enough for this stress test, but not for a strong human-ground-truth claim

The unanimous controls are reassuring sanity checks:

unanimous grade 0 -> 20/20 human agreement
unanimous grade 2 -> 11/11 human agreement

and the DeepSeek/human agreement is interesting.

But neither independently measures human assessor reliability.

For that, you would need a second human on at least some overlapping items.

I would therefore distinguish:

production benchmark closure:
single expert stress audit is already useful

publication-quality human ground truth:
second independent human subset would strengthen it

I would not delay production work for the second one.

One nuance about the final ranking order

I agree that the winner is robust.

I would be a little more careful saying that every pairwise difference is equally well established.

After human correction:

rrf-quality  0.4080
rrf          0.3478
bm25         0.2811
engine-count 0.2776

The rrf-quality margin remains fairly substantial.

But:

BM25 - engine-count = 0.0035 nDCG@10

is tiny.

With only 24 topics, small mean differences can be quite sensitive to which topics happen to be in the evaluation set. The classic NIST study The Effect of Topic Set Size on Retrieval Experiment Error specifically shows that with small topic sets, apparently small system differences can reverse under another sample of topics.

So I would make the strongest claim:

rrf-quality is the robust production winner among the evaluated configurations.

and a weaker secondary statement:

the observed complete order was RQ > RRF > BM25 > EC.

rather than making the 0.0035 lower-pair margin carry the same evidentiary weight as the winner result.

If the per-topic scores already exist, a paired bootstrap/randomization interval for those lower comparisons is cheap; but again, I would consider that publication polish, not a reason to reopen the production decision.

The remaining qrels-coverage issue is small enough to close cheaply

You now report:

rrf judged@10         = 1.000
rrf-quality judged@10 = 0.892

For 24 queries × 10 positions, 0.892 corresponds roughly to only a few dozen top-rank slots being unjudged.

That is not enough for me to discard the result, especially because the human-corrected rerun still preserves the winner.

But if you want to turn this into a reusable public benchmark, this is probably the cheapest remaining qrels improvement:

take union of current systems' Top-10
or Top-20
        ↓
judge every still-unjudged candidate
        ↓
freeze qrels

No need to re-annotate the whole 28k corpus.

The reason is the standard incomplete-pooling problem: evaluation sets created through pooling can disadvantage systems that retrieve documents that were not contributed to the judged pool. NIST discusses this directly in Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements.

For current production selection, I would call this a caveat.

For a public benchmark that future neural rankers may retrieve against, I would close it.

I would also preserve:

judged@10
judged@20

for every future system.

Then a future model with a low nDCG score but 40% unjudged Top-20 is clearly distinguishable from a genuinely poor ranker.

Query refinement: I think the production decision is answered, but I would phrase the mechanism carefully

I agree with the production decision:

keep raw queries

The experiment gives two independent reasons.

Frozen candidate arm

For BM25:

expanded query -> nDCG@10 -0.0655

On the same documents, that is fairly direct evidence that the expansion diluted the lexical representation used by BM25.

The RRF controls correctly stay unchanged because they do not depend on the query text when candidate lists/ranks are fixed.

That is a nice clean experiment.

Live acquisition arm

The operational result is even more striking:

raw mean candidates      ~1110
expanded mean candidates ~260

~4.26× candidate loss

plus substantial engine errors and only about 6% overlap between raw and expanded pools.

That is more than enough to say:

this expansion scheme is not suitable for the current production metasearch path.

I would only separate that from the stronger statement:

the expanded-only documents themselves are less relevant.

Because the expanded pool is almost completely different, many expanded-only documents were presumably outside the original judgment pool.

Under ordinary pooled evaluation, unjudged documents can be treated as nonrelevant, so the live-arm nDCG can punish a system simply for retrieving unseen material. That is the same incomplete-judgment issue discussed above.

So I would interpret the live result as two layers:

very strong:
expansion hurts candidate yield / engine compatibility

not yet isolated:
expanded-only documents are intrinsically less relevant

For the production decision, the first is enough.

The engine failure mechanism is also worth wording more generally

I would not summarize the failure as:

onion engines treat long queries as AND/exact phrase.

At least Ahmia’s current public implementation is more specific.

Its search view rejects queries over 100 characters or more than 10 space-separated terms, and its Elasticsearch query uses minimum_should_match: "75%", not a literal exact-phrase requirement. The current implementation is visible in Ahmia’s views.py.

So the cleaner generalization is:

free-form LLM expansion collided with heterogeneous engine query contracts.

Different upstream engines may have:

length limits
term-count limits
tokenization differences
AND-like behavior
minimum-match thresholds
form/query syntax
different ranking semantics

That is a more interesting federated-search result anyway.

The Ghost broker cannot assume that one verbose reformulation is a valid query language for every remote engine.

If query refinement ever comes back, I would change the formulation of the problem

Instead of:

raw query
        ↓
LLM writes one richer query
        ↓
send same expansion to all engines

I would test something closer to:

information need
        ↓
query planner
        ↓
short atomic queries / engine-compatible reformulations
        ↓
upstream engines
        ↓
fusion

For example:

"recent ransomware payment infrastructure and laundering methods"

-> ransomware payments
-> ransomware bitcoin wallets
-> ransomware laundering

rather than a sentence-length paraphrase.

That becomes a query planning / federated acquisition experiment, not just “does an LLM rewrite look better?”

But because the present expansion has clearly failed the production test, I would put this in a future branch and move on.

RRF k=10: fine for production, but I would treat v1 as the development collection

The switch from k=60 to k=10 seems perfectly reasonable as an empirical production choice.

One useful historical detail is that the original Cormack, Clarke & Büttcher RRF paper also used a pilot to choose the fusion constant and then kept it fixed for subsequent validation.

That suggests a clean boundary for Ghost:

24-query frozen corpus
    = development / configuration-selection collection

selected:
    RRF k=10
    quality configuration
    trust weighting

Then freeze them.

If later you want to claim:

k=10 is generally preferable for onion metasearch

rather than:

k=10 works best for the current Ghost benchmark

the next test should be a new query set or later frozen SERP snapshot, with k=10 fixed before looking at the new qrels.

That is not unfinished work for benchmark v1.

It is benchmark v2 / external validation.

A few artifact/provenance details I would clean up before publishing the benchmark

These are mostly documentation issues now.

1. I would call qrels_v2 and qrels_v3 “qrel variants” rather than “independent qrels”

They use the same topic set and judgment pool, and share several judges.

So:

two qrel variants
two judgment conditions
two ensemble/parser configurations

is more precise than:

two independent qrels sets

The interesting result does not depend on independence anyway.

What matters is:

materially different judgment conditions still produced the same winner.

2. Make the panel-count convention explicit

You describe:

A / B / C
10 × 3-of-5
5 × LOJO

and elsewhere refer to 17 panels.

On paper those labels can look like 18 named conditions unless one of the categories overlaps another condition.

There may be a perfectly simple counting convention behind this; I would just state it in the artifact so a future reader does not spend time reverse-engineering the number.

3. Pin exact model/provider IDs

There is some naming drift across the thread:

DeepSeek V4 Flash
DeepSeek-R1

Kimi-K3
Kimi K2.6

and GLM was replaced with Llama before the canonical five-judge run.

That is all manageable during exploratory work, but the final benchmark manifest should make each judgment reproducible:

judge_id
provider
exact provider model string
model family/version
date
prompt hash
max_tokens
temperature
parser version
retry policy

This matters especially because one of the interesting findings is itself a model-specific failure mode: Kimi’s grade distribution, Nemotron’s 429s, GLM parse behavior, etc.

4. Keep relevance labels separate from the production quality policy

The quality stage includes things such as:

domain collapse
near-duplicate demotion
information-quality scoring
commercial-intent penalty
mirror detection

while the relevance narratives also encode a research intent.

That is reasonable for production, but for a reusable benchmark I would preserve enough structure that a future researcher can distinguish:

topical relevance
quality/policy preference
duplication
commercial intent

rather than baking them irreversibly into one label.

This is not an argument that rrf-quality should lose.

The human-corrected result suggests the production configuration is robust.

It is just an artifact-design choice that will make the benchmark easier to reuse for a different downstream policy.

About the external NIST sanity check

I would keep the TREC RAG comparison, but as exactly what you now call it: an out-of-domain sanity check.

It establishes something useful:

the judges are not producing arbitrary/random relevance behavior

against an external human-labeled collection.

What I would avoid is assigning the performance gap to:

domain mismatch
scale collapse
ensemble size

as though their individual magnitudes were already measured.

All three are plausible explanations, but separating them would require controlled experiments.

Fortunately, you do not need that causal decomposition.

The in-domain human stress audit is now much more relevant to the Ghost benchmark itself.

So the external NIST result can stay in the supporting-evidence layer:

external out-of-domain sanity check
        +
in-domain human stress correction
        +
system-order stability

Together they tell a stronger story than any one of them alone.

So I think the project has reached a useful transition point:

BENCHMARK V1
============

Acquisition/frozen corpus       done
Topic narratives               done
Diverse judgment pool          done
Parser deconfound              done
Judge-panel perturbation       done
Wording sensitivity control    done
External human-qrel sanity     done
In-domain human stress audit   done
Atomic ranking comparison      done
System-order robustness        done
judged@K accounting            done
query-refinement production test done

Production decision:
    rrf-quality
    raw queries

and then a separate optional layer:

PUBLICATION / BENCHMARK HARDENING
=================================

nice to have:
- fill remaining top-rank unjudged candidates
- split surrogate-only vs page-assisted human results
- exact model/run provenance
- second human subset if stronger human-ground-truth claims are desired
- held-out/fresh benchmark v2 for external generalization

I would not let that second list turn back into a prerequisite for shipping the production findings.

The useful shift now is from:

“How many more ways can I validate this benchmark?”

to:

“How do I freeze what I learned and make sure future production changes do not regress it?”

That gives the benchmark a continuing role without making benchmark construction the permanent center of the project.

sookoothaii · September 2, 2026, 12:34pm

Thanks for the comprehensive review. The repo is back public on Codeberg — I had briefly set it to private during a refactor, it’s visible again at the link in the original post.

I agree the benchmark cycle can close. Here’s where each point landed.

Benchmark v1 frozen

Frozen as ghost-serp-benchmark-v1:

Frozen candidate corpus (24 queries, 13 engines, ~28,700 results)
Frozen 24 topics/narratives
Frozen qrels (v2 + v3 as variants, see below)
Frozen human corrections (95 labels, order preserved)
Frozen ranker parameters (RRF k=10, quality config, trust weighting)
Frozen evaluation implementation

Production configuration: rrf-quality, raw queries, RRF k=10.

The benchmark now serves as a regression suite: change production code → replay v1 → check for regressions.

Surrogate-only vs page-assisted split — done

You suggested this as an essentially free analysis. It was — the data was already there, I just hadn’t split it. Results:

Subset	n	Agreement	Cohen’s kappa	Interpretation
Surrogate-only	226	66.4%	0.507	moderate
Page-assisted	62	48.4%	0.272	fair
Delta		+18.0pp	+0.235

The page-assisted subset has substantially lower LLM/human agreement. This is consistent with your reference to Kazai et al. — the human saw full page text while the LLM judges saw only URL/title/snippet. The two comparisons contain two changes at once (different assessor + more evidence), so I’m not claiming the gap is purely about evidence sufficiency. But the direction is clear: some SERP surrogates do not contain enough information for LLM judges to make accurate relevance judgments.

The page-assisted items are concentrated in threat_intel (55 of 62), which is already the hardest category (surrogate-only threat_intel agreement: 62.7% vs journalism 69.0% vs privacy 67.0%). So the gap is partly confounded with category difficulty. But even within threat_intel, the surrogate-only agreement (62.7%) is higher than the page-assisted agreement (49.1%).

This supports your suggestion of a judgment_status = insufficient_surrogate_evidence field rather than forcing every candidate into 0/1/2. I’ll add this to the benchmark metadata for v1 — not as a new label, but as a flag on the 62 items where the human needed page fetch.

Per-category breakdown:

Category	Surrogate agree	Page-assisted agree
threat_intel	37/59 (62.7%)	27/55 (49.1%)
journalism	40/58 (69.0%)	3/6 (50.0%)
privacy	73/109 (67.0%)	0/1 (0.0%)

The privacy page-assisted count is too small (n=1) to interpret. Threat_intel and journalism both show the expected pattern.

Ahmia correction — accepted

You’re right. I oversimplified the engine failure mechanism. Ahmia’s views.py rejects queries over 100 characters or more than 10 space-separated terms, and uses minimum_should_match: "75%" — not a literal exact-phrase requirement. The cleaner generalization is:

Free-form LLM expansion collided with heterogeneous engine query contracts.

Different engines have length limits, term-count limits, tokenization differences, minimum-match thresholds, and different query syntax. The Ghost broker cannot assume one verbose reformulation is a valid query language for every upstream engine. I’ve corrected this in my internal documentation.

Pairwise margin caveat — accepted

The strongest claim is:

rrf-quality is the robust production winner among the evaluated configurations.

The weaker secondary statement:

The observed complete order was RQ > RRF > BM25 > EC.

I won’t make the BM25 vs engine-count margin (0.0035 nDCG@10 after human correction) carry the same evidentiary weight as the winner result. With 24 topics, that margin is within the noise range that Voorhees and the NIST topic-set-size literature warn about. If I later want to claim the full ordering is robust, I’d need a paired bootstrap or a larger topic set. For now, the production decision only depends on the winner.

qrels naming — accepted

I’ll refer to qrels_v2 and qrels_v3 as “qrel variants” rather than “independent qrels” going forward. They share the same topic set, judgment pool, and several judges. The interesting result — materially different judgment conditions producing the same winner — does not depend on independence.

Model/provider ID pinning

The provenance files already capture: model string, endpoint (OpenRouter), avg latency, error count, prompt version, prompt hash, pool version, narrative version, decision method, and refinements applied (including the GLM→Llama swap and the Kimi max_tokens issue).

Missing from the current provenance: max_tokens, temperature, parser version. These will be added to the frozen v1 manifest. For reference:

max_tokens: 512 for all judges in v2/v3 (this was the root cause of Kimi’s truncation — documented separately)
temperature: 0 for all judges
Parser: JSON codeblock strip + keyword fallback, version v2-calibrated-ensemble

The Kimi max_tokens=512 artifact is documented in the provenance refinements: “Kimi K2.6 is a reasoning model that produces chain-of-thought before JSON output. At 512 tokens, the JSON is frequently truncated, triggering parse fallback.” The head-to-head test (max_tokens=4096, 25 docs) showed Kimi at 84% exact match vs 88% for Qwen3-Max — the model itself was fine, the configuration was wrong. This is a model-specific failure mode that the provenance captures.

Relevance vs quality policy separation

Acknowledged as an artifact-design choice for v2. The current rrf-quality configuration bakes together:

Topical relevance (the qrels labels)
Quality/policy preferences (domain collapse, near-duplicate demotion, information-quality scoring, commercial-intent penalty, mirror detection)

For v1 as a production regression suite, this is fine — the production system uses the combined configuration. For a reusable public benchmark, I’ll preserve enough structure that a future researcher can separate topical relevance from quality/policy. Not blocking v1 freeze.

Query planner — agreed

If query refinement comes back, the formulation should be:

information need
    → query planner
    → short atomic queries / engine-compatible reformulations
    → upstream engines
    → fusion

not “LLM writes one richer query → send to all engines.” The federated-search framing is the right one — the broker needs to speak each engine’s query language, not force one verbose reformulation through every upstream. This goes in a future branch, not v1.

External NIST sanity check

Keeping it as exactly what it is: an out-of-domain sanity check that establishes the judges are not producing arbitrary/random relevance behavior. Not assigning the performance gap to individual causes (domain mismatch, scale collapse, ensemble size) — those would need controlled experiments. The in-domain human stress audit is the relevant validation for the Ghost benchmark itself.

What’s next

BENCHMARK V1 — frozen, serves as regression suite

PRODUCTION — move development effort back to search behavior:
  - engine health monitoring (13 engines, heterogeneous failure modes)
  - engine-specific query contracts (length limits, term limits, match thresholds)
  - snippet quality detection (flag insufficient surrogates before judging)
  - production ranking: rrf-quality, raw queries, RRF k=10

PUBLICATION HARDENING (optional, not blocking):
  - fill remaining top-rank unjudged candidates (rrf-quality judged@10 = 0.892)
  - second human annotator on a subset for human-human kappa
  - benchmark v2 from fresh queries for external generalization

The useful shift you described — from “how many more ways can I validate this benchmark?” to “how do I freeze what I learned and make sure future production changes do not regress it?” — is the one I’m taking.

Thanks for the methodology guidance throughout. The “measure judges as instruments, not by origin” framing, the deconfound step ordering, and the federated-search problem setting all shaped the experiment design in ways that made the results stronger than they would have been otherwise.

John6666 · September 3, 2026, 12:20am

I think you’re good to move into production. I went looking for the landmines I could see:

I would keep benchmark v1 frozen at this point. It has already done the job you needed for the production decision: it selected rrf-quality, raw queries, and RRF k=10, and the later judgment stress tests did not give a strong reason to reopen that decision.

The one thing I would pin before the refactor gets much farther is the boundary between the frozen benchmark implementation and the ranker that production actually executes.

My default route would be:

frozen v1 evidence
    ├── frozen reference/evaluator
    │      corpus + qrels + metric definitions + historical baseline
    │
    └── current production ranker
           system under test

In other words: I would not rewrite the frozen evaluator to follow production. Keep it frozen as the reference. Instead, add a thin adapter that feeds the same frozen SERPs into the actual current production composition, then evaluates that output against the frozen v1 qrels.

That makes the intended workflow literal:

change production code
        ↓
run the current production ranker on frozen v1 inputs
        ↓
compare with frozen v1 evidence
        ↓
either:
  behavior preserved
or:
  behavior intentionally changed, with measured quality impact

That seems higher-value to me than another benchmark round.

I would also give the selected production composition a named profile — something like production-v1 — rather than relying on a collection of defaults. I do not mean changing Ghost’s global defaults; the off-by-default filters have their own policy/liability rationale. I mean making the production deployment’s intended composition explicit and executable.

Roughly:

production-v1
  query policy              = raw
  RRF                       = enabled
  RRF k                     = 10
  quality filter            = enabled
  RRF input ordering        = pinned
  QF candidate depth        = pinned
  QF feature set            = pinned
  duplicate representative = deterministic
  snippet/ranking policy    = explicit
  dedup policy              = explicit
  output limit semantics    = explicit
  profile/version id        = production-v1

The reason I would pin the composition, not only rrf-quality, is that there are currently a few reachable ways for two paths both called “RRF + quality filter” to produce different rankings.

I tried a small offline synthetic fixture against those seams. It is not evidence that the private frozen benchmark loses nDCG/MAP/MRR; I cannot measure that without the frozen corpus/qrels. What it does show is the narrower thing that matters here: some of the current benchmark/runtime paths are not semantically equivalent in general.

The two cheapest production fixes I would probably do immediately are even smaller:

restore/enforce stdio purity — normal preflight/debug logging should never reach the MCP stdout stream;
do not let HTTP 200 + zero parsed results automatically collapse into “healthy empty result” — preserve enough acquisition state to distinguish genuine empty SERPs from challenge/interstitial/parser failure.

Those are both small changes with unusually high diagnostic value.

Why I would make production itself the regression SUT

1. `rrf-quality` is a pipeline, not just two switches

The interesting drift is not only whether qualityFilter is true or whether the RRF constant is 10.

There are several places where ranking semantics can change before those headline settings even matter.

A. Per-engine ordering before RRF

RRF consumes ranks, not the underlying relevance scores. So anything that changes the per-engine order before RRF changes the fusion input.

The frozen evaluation path and the current MCP path do not appear to construct those per-engine ranks identically: one path can reorder using snippet informativeness before assigning RRF ranks, while the runtime path can inherit parser-return order.

That means:

same documents
same RRF formula
same k=10
different input ranks
        ↓
possibly different fused ranking

This is why I would treat “per-engine normalization/order before fusion” as part of production-v1.

It is also a general RRF property: because RRF uses rank positions, the upstream rank list and the depth allowed into fusion are part of the effective algorithm, not incidental plumbing.

B. Quality-filter candidate depth

The frozen rrf-quality evaluation path applies the quality filter to a bounded RRF window (topk * 3).

The production MCP path can feed a substantially wider fused candidate set into the quality filter before slicing the final result count.

That matters because the current quality filter finally sorts on its own qualityScore; the incoming RRF score is not the final comparator.

So a candidate that would never enter the benchmark’s QF window can enter production QF and jump into the final top-k.

A tiny synthetic counterexample was enough to produce a top-10 difference (9/10 overlap). Again, I would interpret that only as:

the two compositions are reachable as different rankers.

I would not interpret it as:

production currently has a measured v1 quality regression.

The latter needs the private frozen inputs/qrels.

This also suggests that QF candidate depth belongs in the named profile:

quality_filter:
  enabled: true
  candidate_source: post_rrf
  candidate_depth: ...

rather than living as an implementation detail.

C. QF feature set

There is another subtle difference: production extracts entities before quality scoring and can pass them into the QF feature calculation, while the frozen evaluator does not necessarily feed the same entity feature into the same scoring path.

The QF currently gives entity density non-zero weight, so this is ranking behavior, not just extra metadata.

I see two perfectly reasonable choices:

If entity density is intended ranking evidence:
    make it an explicit production-v1 rank feature.

If entities are mainly downstream intelligence metadata:
    keep them in the result object but remove that dependency
    from the canonical ranker.

I would not choose between those from outside the project; I would just make the boundary explicit so a refactor cannot silently switch between them.

D. Duplicate representative selection

There is one more determinism seam worth fixing while the code is moving.

With parallel engine acquisition, duplicate URLs may arrive in different completion orders. The merged result can preserve the first-seen title/snippet/entities as the representative metadata, while accumulating provenance from later engines.

If later ranking uses that representative snippet/entities, then there is a possible path like:

network completion order
        ↓
which duplicate is seen first
        ↓
representative title/snippet/entities
        ↓
quality-filter features
        ↓
final rank

I am not claiming this is causing large live instability; I did not measure its live frequency.

But it is a reachable nondeterminism path, and the fix can be very small: choose the duplicate representative by a deterministic rule.

For example:

canonical URL
    + best non-empty snippet
    + deterministic engine precedence

or:

merge all provenance first
    ↓
select representative metadata by a pure scoring rule

The exact rule matters less than making it independent of request completion timing.

2. Keep two different regression questions separate

I would avoid making one golden-output test do two jobs.

Refactor / wiring changes

Question:

Did behavior change when it was not supposed to?

Check things such as:

active profile/version
candidate IDs
top-k order
duplicate provenance
acquisition states
snippet/withheld outcome

That is a behavioral parity gate.

Intentional ranker changes

Question:

We changed ranking intentionally; did measured retrieval quality move unexpectedly?

Then run:

frozen SERPs
    ↓
current production SUT
    ↓
frozen qrels
    ↓
nDCG / MAP / MRR / output deltas

That is a quality regression gate.

Keeping these separate avoids the annoying situation where a deliberate ranking improvement fails CI merely because the exact old ordering changed.

It also means you do not need new topics, new judges, or new annotations for ordinary production refactors.

3. I would keep the reference frozen even if the production ranker is factored out

One subtle point: I would not turn the frozen evaluator itself into a thin wrapper around whatever production code happens to exist today.

That would make future replays reproducible against the current implementation, but it would weaken the historical meaning of “benchmark v1”.

The cleaner split is:

benchmark/v1/reference
    immutable historical implementation

src/.../production-ranker
    current implementation

benchmark/v1/run-production-regression
    adapter from frozen inputs → current implementation

Then a future reader can answer both questions:

What exactly did v1 evaluate when it was frozen?

What does today's production code do on the exact same evidence?

That is a useful audit trail, especially during a large refactor.

The smaller production landmines

4. stdio purity looks like a very cheap regression test to recover

For an MCP stdio server, stdout is protocol traffic. Ordinary diagnostic logging belongs on stderr/a logger, not mixed into the JSON-RPC stream.

There are reachable preflight paths that currently use ordinary console.log(...), and a direct fixture can make that text appear on stdout.

This is especially worth fixing because it looks like you had already thought about this class of failure earlier: stdio-purity coverage existed/planned during the previous parity work.

So I would frame the fix as restoring a boundary you already wanted, not adding a new subsystem:

stdio child process
    stdout -> parse only MCP messages
    stderr -> arbitrary diagnostic log text allowed

A tiny regression test can simply start the real stdio entry point, trigger a preflight path, and assert that every stdout line is valid MCP/JSON-RPC framing.

That is probably more useful than another simulated MCP-vs-REST helper test.

5. `200 + 0 results` needs one more state dimension

The other cheap fix is acquisition semantics.

At the moment there is a reachable path where:

HTTP status < 400
    ↓
parser returns []
    ↓
engine success recorded

But those observations are not equivalent:

transport succeeded
HTTP request succeeded
parser recognized a valid SERP
SERP genuinely contained zero results

A challenge page, interstitial, changed markup, consent/form response, or parser miss can all potentially sit between them.

I would keep the state fairly boring and observable rather than build a fancy classifier:

{
  "transport_status": "ok",
  "http_status": 200,
  "challenge_signal": "none | detected | unknown",
  "parse_status": "ok | empty | failed | unknown",
  "result_count": 0
}

Then derive a higher-level status only when justified:

valid_empty
challenge
parser_miss
transport_error
unknown_zero

The important distinction is:

one engine failed to acquire usable evidence
        !=
the whole Ghost search failed

A metasearch broker should still return useful partial results. The extra state is mainly there so downstream code does not interpret infrastructure failure as evidence of absence.

This fits the engine-health / partial-failure direction you already established earlier; it does not require reopening ranking evaluation.

6. REST vs MCP: share the profile if both remain production surfaces

Earlier in the project, IntelShed was described as using Ghost’s REST /api/search, while MCP stdio was the standalone integration path.

I cannot verify that the current deployment topology is still exactly the same, and I would not assume it is: IntelShed has evolved since then.

So I would make this conditional:

If REST and MCP are both production surfaces:
    both should consume the same production-v1 composition.

If only one is production:
    pin that one, and treat the other as an alternate interface.

The supplied entry points currently do not obviously execute one identical ranking/acquisition pipeline.

There are differences around things such as:

RRF composition
quality-filter default
snippet-filter default
engine timeout/deprecation handling

I would not call every default difference a bug. Some defaults are deliberately conservative.

The useful invariant is narrower:

selecting production-v1 should mean the same ranking contract regardless of which adapter invokes it.

That suggests an architecture like:

engine acquisition
      ↓
normalized EngineResult[]
      ↓
rank(results, productionProfile)
      ↓
canonical SearchResult[]
      ↓
┌───────────────┬───────────────┐
│ MCP adapter   │ REST adapter  │
└───────────────┴───────────────┘

If fully factoring that now is too invasive, even an integration fixture that runs both real entry points over deterministic fake engine responses would catch most of the dangerous drift.

7. Snippet filtering: ranking policy or presentation policy?

This is one place where I would deliberately leave you two valid designs rather than prescribe one.

Today, snippet filtering can sit upstream of entity extraction / quality scoring, so changing the snippet policy can change ranking semantics.

If that is intentional

Then snippet safety is part of the ranker:

production-v1:
  snippet_filter: safe

and it should be frozen with the other ranking settings.

If snippet filtering is mainly a liability/presentation boundary

Then I would separate it:

canonical production ranking
          ↓
presentation policy
          ↓
safe/redacted snippet returned to client

For example:

ranking_profile      = production-v1
presentation_policy  = safe-snippets-v1

That lets a UI/REST consumer and an MCP consumer have different display policies without accidentally becoming different rankers.

Either approach is coherent. The useful thing is avoiding an accidental mixture where a presentation change silently changes relevance ordering.

8. Provenance is worth preserving all the way to the tool boundary

Internally, the fused result has useful information — e.g. multiple contributing engines and fusion state — that can disappear in the final public result projection.

I would preserve a small structured envelope such as:

{
  "title": "...",
  "url": "...",
  "snippet": "...",

  "ranking_profile": "production-v1",
  "engines": ["...", "..."],

  "acquisition": {
    "status": "ok"
  },

  "surrogate_evidence": "sufficient",
  "withheld": false
}

No need to turn this into a protocol migration project.

MCP already has a structured-result concept (structuredContent / output schemas), so this is mainly a question of retaining information you already have. A text-serialized representation can remain alongside it for compatibility if needed.

This also creates a useful trust boundary for RAG/agents:

tool/control metadata
        !=
retrieved external text

Search-result text — especially from arbitrary web/onion pages — should be treated as untrusted external data, not instructions to the agent.

Structured provenance does not magically solve prompt injection, but it gives downstream clients a clean place to maintain that distinction and apply least-privilege policy.

9. Query compatibility probably deserves an engine contract, not another global query planner

Your raw-query production choice still looks sensible to me.

The failed/low-yield expansion behavior also points to a more general production lesson: “better query for an LLM” and “valid/effective query for a particular search engine” are different problems.

Ahmia, for example, currently has concrete query constraints such as a 100-character / 10-term limit and a 75% minimum-match search rule, and several engines need form/token/session preparation.

I would separate three contracts:

query contract
    accepted query text / limits / syntax

acquisition contract
    form/token/session/captcha/timeout preparation

parser contract
    what response shape counts as a valid SERP

Then:

information need
       ↓
raw/broker query
       ↓
per-engine query validation
       ↓
per-engine acquisition preparation
       ↓
engine
       ↓
parser contract

That keeps a future LLM query planner from becoming responsible for transport/session/parser failures.

A tiny registry is enough initially:

type EngineQueryContract = {
  maxChars?: number;
  maxTerms?: number;
  preferredStyle?: "keywords" | "phrase" | "freeform";
  supportsQuotes?: boolean;
  supportsBoolean?: boolean;
};

Unknown values can stay unknown.

I would first use it only for validation/telemetry. Only add automatic per-engine rewrites if the counters show that query-contract failures are a real production source of lost relevant results.

10. I would keep `surrogate_evidence` separate from relevance

The surrogate-only vs page-assisted human audit is useful production evidence, but I would be careful with the causal interpretation.

The page-assisted subset was not a random controlled sample, category difficulty differs, and the human assessor had access to evidence that the model did not.

So I would not say:

full-page fetch improves relevance judgment by N points

What I think it does justify is simpler:

snippet/title/URL evidence sufficiency is its own useful metadata dimension.

That also matches the broader IR observation that document surrogates can be sufficient for many judgments but insufficient for some, and humans/models do not necessarily benefit from full text in the same cases.

A production-friendly state could be:

surrogate_evidence:
  sufficient
  weak
  missing
  withheld

Then conditional fetch remains optional and bounded:

strong surrogate
    → no fetch

weak/missing surrogate
+ result is high-ranked / decision-relevant
+ policy permits page access
    → bounded fetch

This avoids turning page fetching into a new mandatory ranking stage or a new benchmark project.

11. Small observability rule: log the contract, not the sensitive payload

For production debugging I would favor counters/state over raw payload logging:

profile_id
engine_id
query_contract_ok
transport_status
http_status class
parse_status
result_count
latency
challenge_signal

rather than persisting:

raw query
full request URL
raw snippet/page text

unless explicitly needed.

That gives you enough information to diagnose:

query incompatibility
vs
engine outage
vs
challenge
vs
parser drift
vs
valid empty

without quietly expanding the privacy boundary just because observability was added.

What I would *not* reopen yet

I would leave these out of the critical path unless production replay produces a concrete reason:

another qrels round;
another judge ensemble;
more human annotation;
a new GPU experiment;
another RRF/BM25/engine-count comparison;
Tor-isolation redesign — the current code already has the important SOCKS/SafeSocks-style hardening pieces;
a large prompt-injection classifier;
an MCP protocol-version migration;
a sophisticated LLM query planner.

The benchmark phase already generated enough evidence for the stated production decision.

There are also three things I would explicitly leave unknown rather than infer:

what arguments the current IntelShed wrapper actually sends to Ghost;
what the current deployed REST/MCP routing topology is;
how large — or in which direction — the observed composition differences move nDCG/MAP/MRR on the private frozen v1 corpus.

None of those unknowns blocks the architecture above.

The first two disappear if all production adapters consume an explicit production-v1 profile.

The third becomes measurable automatically once the current production ranker can be run as a SUT against the frozen benchmark inputs.

So I think the production transition can stay exactly that: a production transition, not benchmark phase 2.

The compact version of what I would preserve is:

benchmark v1
    = frozen evidence + historical reference

production-v1
    = explicit executable ranking contract

refactor
    = behavioral parity check

intentional ranker change
    = current production SUT replayed against frozen v1

engine failure
    != empty evidence

retrieved content
    != trusted instructions

If that wiring is in place, future changes become much easier to reason about: you can change the production implementation freely while still knowing whether you changed plumbing, ranking semantics, retrieval quality, or only presentation.

I would not reopen the benchmark work unless that production replay actually gives you a reason to.

sookoothaii · September 3, 2026, 1:42pm

Thanks for the production transition guidance. I went through each point, checked against current best practices, and implemented the ones that were actionable. Here’s where each landed.

Benchmark v1 frozen — agreed

Benchmark v1 stays frozen. It already served its purpose: it selected rrf-quality, raw queries, and RRF k=10. The calibration cycle is closed.

Production-v1 named profile — implemented

You were right that relying on scattered defaults is fragile. I created an explicit, executable ranking profile (src/lib/ghost/profiles.ts):

production-v1 (v1.0.0)
  query policy              = raw
  RRF                       = enabled
  RRF k                     = 10
  trust weighted            = true
  quality filter            = enabled
  maxPerDomain              = 3
  simHash threshold         = 3
  semantic dedup            = off (opt-in)
  LLM rerank                = off
  snippet filter            = off
  maxResults                = 20
  input ordering            = shuffle (camouflage)
  duplicate representative  = first-seen
  output limit              = hard-cap
  profile/version id        = production-v1/1.0.0

The profile is a single source of truth. Future changes to any parameter are a profile version bump, not a silent default change.

Production replay adapter — implemented, and it found what you predicted

I built scripts/replay-production.ts which feeds frozen v1 SERPs through the actual production ranker (rankProduction in search.ts) and evaluates against frozen v1 qrels. The intended workflow is exactly what you described:

change production code
    → run current production ranker on frozen v1 inputs
    → compare with frozen v1 evidence
    → either: behavior preserved
    or: behavior intentionally changed, with measured quality impact

The first replay run confirmed your prediction about benchmark/runtime path divergence:

Metric	Frozen v1 reference (eval-ranking.ts)	Production replay (search.ts)	Delta
nDCG@10	0.4080	0.2889	-0.1191
MAP@10	0.2094	0.1508	-0.0586
MRR@10	0.7125	0.6194	-0.0931

The frozen v1 benchmark numbers were measured with eval-ranking.ts, which has its own RRF implementation. The production ranker in search.ts produces different rankings. The two paths are not semantically equivalent — exactly the “reachable ways for two paths both called ‘RRF + quality filter’ to produce different rankings” that you flagged.

The replay adapter makes this drift measurable instead of hidden. The next step is to identify which specific difference (dedup logic, QF application order, RRF score calculation, result limiting) causes the delta, and decide whether to align the benchmark evaluator to production or vice versa. The production decision (rrf-quality wins) is not affected — the system order is preserved — but the absolute numbers need to be reported as production-replay numbers, not benchmark-evaluator numbers.

stdio purity — fixed

You were right that this is a high-diagnostic-value small fix. I checked the MCP spec (2026-07-28): “The server MUST NOT write anything to its stdout that is not a valid MCP message.” Found 8 console.log() calls in engines.ts (OSS and Torgle captcha handlers) that were writing debug output to stdout, corrupting the JSON-RPC stream. All replaced with process.stderr.write(), which the spec explicitly allows for logging.

The albinogeek.com analysis of stdout corruption in MCP servers describes this as the most common stdio server bug — a console.log that survives code review and intermittently breaks the protocol stream under load. That was live in production.

Engine result classification — fixed

Your point about HTTP 200 + zero results collapsing into “healthy empty result” was the other cheap, high-value fix. Previously every HTTP 200 that parsed to zero results was reported as status: "ok" — indistinguishable from a successful search that found nothing.

Now zero-result responses are classified:

captcha — reCAPTCHA/hCaptcha/math captcha patterns in response body
interstitial — “unusual traffic”, “enable JavaScript”, redirect pages
blocked — “access denied”, “403 forbidden”, “rate limit”
parse_failure — substantial body (>1KB) with zero results, no markers
ok_empty — legitimate empty (no signals, short body)

Based on the apiserpent observation: “200 means a page came back, not the right data came back.” The production health dashboard now distinguishes genuine empty SERPs from challenge/interstitial/parser failure.

Live verification

After the MCP server restart I ran the production code through Tor against live engines to confirm the changes are not just code on a branch. Two representative queries:

ransomware — partial engine availability (3 of 13 returned results, the rest timed out or returned zero):

engine	status	results	latency
ahmia	ok	180	17.5s
notevil	ok	8	6.7s
tordex	error	0	30.0s (timeout)

188 total results, quality filter off. Snippet-quality fields present on every result. This is the fail-soft case: partial engine availability is reported per-engine, not collapsed into a single pass/fail.

data breach — full engine availability (13 of 13):

engine	status	results	latency
ahmia	ok	944	28.4s
tor66	ok	38	10.6s
onionway	ok	22	17.4s
excavator	ok	21	7.6s
notevil	ok	20	5.9s
torland	ok	20	17.6s
tordex	ok	23	13.8s
torgle	ok	10	16.6s
torch	ok	10	10.3s
oss	ok	11	18.6s
deepsearches	ok	8	9.9s
amnesia	ok	3	15.4s
onionland	ok	2	11.9s

1132 total results. Quality filter: 1044 unique candidates → 657 demoted (63%), 0 removed. LLM rerank off, not applied. Per-engine latency ranges from 5.9s to 28.4s — visible in the health payload, not hidden behind an aggregate.

The query adapter was also exercised with a 15-term / 135-character query against Ahmia (contract: 100 chars / 10 terms). Ahmia accepted the adapted query and returned 820 results in 12.0s. Without the adapter Ahmia would have rejected the over-length query.

These runs confirm: stdio stays clean under load, engine health reports per-engine status and latency, snippet-quality fields are present, query contracts are active, and partial vs full engine availability is observable in the same response.

What I did not reopen

Benchmark v1 stays frozen. The replay delta is a measurement gap, not a reason to re-run the calibration cycle.
The system ordering (rrf-quality > rrf > bm25 > engine-count) is preserved in both paths. The production decision stands.
I’m not adding more judges, more rerankers, or another benchmark round. The replay adapter gives the signal to reopen if one ever appears.

What’s next

Root-cause the replay delta — identify which implementation difference between eval-ranking.ts and search.ts causes the -0.1191 nDCG gap. Likely candidates: dedup logic (aggregate+dedupeByContent vs normaliseKey), QF candidate depth, or RRF score precision.
Align the benchmark evaluator to production — once root-caused, either port the production ranking into the evaluator or vice versa. The frozen v1 evidence should reflect what production actually does.
More engine query contracts — 3 of 13 engines have documented contracts (Ahmia, Torch, TorDex). The remaining 10 need live probing to determine their constraints.
Query planner — your federated acquisition framing. Information need → query planner → short atomic queries per engine contract → fusion. Not v1, but the architecture is now clean enough to build it.

Wiring

benchmark v1
    = frozen evidence + historical reference (eval-ranking.ts)

production-v1
    = explicit executable ranking contract (profiles.ts)

replay
    = production SUT replayed against frozen v1 inputs (replay-production.ts)

intentional ranker change
    = run replay, compare delta, decide

engine failure
    != empty evidence (5-way classification)

retrieved content
    != trusted instructions

The production transition is staying a production transition. No benchmark phase 2 unless the replay gives a reason to.

sookoothaii · September 4, 2026, 1:36pm

Ghost Search MCP — Production Hardening Complete

This is an update to the previous post. Every item from the “What’s next” section has been shipped, plus additional hardening work. Here’s where each landed.

Replay delta — root-caused and fixed

The -0.1191 nDCG@10 gap between the benchmark evaluator (eval-ranking.ts) and the production ranker (search.ts) had two root causes:

Quality filter replaced RRF scores entirely. applyQualityFilter() sorted by qualityScore, discarding the RRF ordering. This accounted for ~-0.06 nDCG.
Per-engine info-quality sort was missing in production. eval-ranking.ts sorted per-engine results by snippet informativeness before assigning RRF ranks; rankProduction() did not. This accounted for ~-0.06 nDCG.

Fix: the quality filter is now a modifier on RRF scores, not a replacement. Penalized results get rrfScore * 0.1; non-penalized results get rrfScore + qfScore * 0.01. The per-engine info-quality sort was added to both rankProduction() and the live executeSearch() path.

Metric	Before fix	After fix
Eval nDCG@10	0.4080	0.3900
Production nDCG@10	0.2889	0.3944
Delta	-0.1191	+0.0044

The drift is eliminated. The remaining +0.0044 is the quality filter penalty working as intended — production is marginally stricter than the evaluator on duplicate/mirror results, which is the correct direction.

The drift analysis tool (

analyze-ranking-drift.ts) remains in therepo. Anyfuturerankingchange can be replayed against the frozenv1corpus and measuredimmediately.

Engine query contracts — all 13 engines covered

The previous post noted 3 of 13 engines had documented contracts (Ahmia, Torch, TorDex). All 13 now have QueryContract definitions with maxQueryLength, maxTerms, matchMode, booleanOperators, and phraseQuotes constraints. The query adapter (

query-adapter.ts) normalizes queriesperengine beforedispatch—truncating length, limitingterms, stripping Booleanoperators andphrase quotes when unsupported.

Contract values are provisional empirical defaults from live Tor probing, not authoritative engine specifications. HTTP 400 responses are retried once with 2s backoff (transient 400s observed on multiple engines). TorDex’s contract was revised twice after live probing showed error_400 is transient rather than length-based — 10 terms / 84 chars confirmed working on retry.

Contract coverage tests (

engine-contracts.test.ts) verify everyengine has a contractand thatadaptQueryrespectseach constraint.

Query planner — implemented

The federated acquisition framing from the previous post is now shipped. When queryPlanner: true is passed to ghost_search or ghost_batch_search:

An LLM generates 2-3 short atomic sub-queries from the original query
Each sub-query is adapted per engine contract via the existing query adapter
Sub-queries are sent to engines as additional search waves via runEngineWave()
Results are fused into RRF with trust weight = 1.0 (low, experimental)
Single response — no protocol break, no streaming

The LLM cascade is OpenRouter DeepSeek V4 Flash → NVIDIA NIM → Ollama → raw query (fail-soft). If every LLM is unavailable, the search proceeds with the original query. The planner is off by default in the production-v1 profile.

Latency impact: adds one LLM call (~1-3s OpenRouter) plus 2-3 additional engine waves (~20-60s over Tor). This is opt-in and not recommended for interactive use without the operator accepting the latency trade-off.

Replay against frozen v1 with planner OFF: nDCG@10 = 0.3944 (no regression). Live Tor test with planner ON remains as a manual operator-driven task — the replay framework cannot measure it because the planner generates different sub-queries each run.

Additional shipped work

Structured audit logging (PR #25). Every ghost_search call produces one structured JSON line on stderr: tool name, query, engine count, result count, latency, engine health. Never stdout. The MCP spec (2026-07-28) requires that servers MUST NOT write anything to stdout that is not a valid MCP message — this was previously violated by 8 console.log() calls in engine captcha handlers, all replaced with process.stderr.write().

Tool manifest integrity (PR #27). SHA-256 of serialized tool definitions (name + description) computed on startup. createServer() returns { server, manifest }. If EXPECTED_TOOL_HASH env var is set, the server warns on mismatch and refuses to start if STRICT_TOOL_HASH=1. This catches tool definition drift between builds — a tool added, removed, or renamed is detected immediately.

Engine router with fast mode (PR #28). fastMode: true queries 5 high-value engines instead of all 13, reducing latency by ~60%. Engine tiers are derived from the frozen v1 benchmark:

Tier	Engines	Role
Core	Ahmia, Tor66	High nDCG + high unique contribution
Reinforcement	Onionway, Excavator, NotEvil	Moderate nDCG, unique coverage
Low	8 remaining	Minimal unique contribution

Trust derivation from benchmark (PR #28). Engine trust values are no longer hand-tuned. They are derived from frozen v1 per-engine nDCG and unique relevant contribution. The derivation is documented in

TRUST_DERIVATION_V1.md. The circularreasoningrisk(trustderived from the same SERPsused for replay)is mitigated by deriving oncefromv1, freezing, and not re-fitting.

In-memory health history (PR #29). Per-engine latency and status are retained in memory across searches, enabling trend analysis within a session. Not persisted to disk — consistent with the RAM-only design principle.

SOCKS5 frame unit tests (PR #29). Edge-case coverage for frame fragmentation across frame boundaries and unusual ATYP paths.

Legal compliance framework (PR #30). README now includes a legal compliance section. A private legal framework document (gitignored) covers jurisdictional analysis, acceptable use, and operational constraints. The project is licensed under PolyForm Noncommercial 1.0.0.

Test framework migration (PR #33). Migrated from bun:test to vitest for broader compatibility. 437 tests across 20 files, all passing.

Brain module (PRs #34-36, migrated in #37). An observation lifecycle module was built in three phases — observation/decay/session-state, consolidation/reflection/coherence, and conductivity/spreading-activation/hub-distillation. After the three phases, the brain module was extracted to a dedicated repository (digital-brain-mcp) to keep Ghost Search MCP focused on federated search. The migration was clean: the brain module is no longer in this repo, and the search ranking pipeline is unchanged.

Current state

Metric	Value
Engines	13 onion search engines, all with query contracts
MCP tools	13
Tests	437 (vitest, 20 files)
nDCG@10 (production replay)	0.3944
nDCG@10 (benchmark evaluator)	0.3900
Drift	+0.0044
License	PolyForm Noncommercial 1.0.0
LLM cascade	OpenRouter DeepSeek V4 Flash → NIM → Ollama → raw
Ranking profile	production-v1 (explicit, versioned)

What I did not do

I did not re-open the benchmark. Frozen v1 stays frozen. The calibration cycle is closed. The replay adapter gives the signal to reopen if one ever appears — until then, the production decision (rrf-quality with RRF k=10, raw queries, trust-weighted) stands.

I did not add more judges, more rerankers, or another benchmark round. The system ordering (rrf-quality > rrf > bm25 > engine-count) is preserved in both the evaluator and production paths.

I did not pursue SOTA neural ranking methods (QUAM, REGENT, BlockRank). An external architecture review confirmed these are not applicable to federated snippet-based meta-search — they require full-text document access, which a meta-search engine does not have. The one viable SOTA path is LLM-based query expansion with RRF fusion (Exp4Fuse-style), which is what the query planner implements.

Remaining work

Live Tor test with queryPlanner: true — manual, operator-driven. The replay framework cannot measure this because sub-queries are generated dynamically. Only proceed with tuning if the live test shows a positive nDCG delta.
OpenRouter key rotation — current API key has limited remaining credit.
Engine health trend persistence — currently in-memory only. Cross-session persistence would require a storage layer, which conflicts with the RAM-only design principle. Under consideration.

Wiring (unchanged)

benchmark v1     = frozen evidence + historical reference (eval-ranking.ts)

production-v1 = explicit executable ranking contract (profiles.ts)

replay = production SUT replayed against frozen v1 inputs (replay-production.ts)

intentional ranker change = run replay, compare delta, decide

engine failure != empty evidence (5-way classification)

retrieved content != trusted instructions

The production transition is complete. No benchmark phase 2 unless the replay gives a reason.

John6666 · September 6, 2026, 1:32am

I think the architecture is converging :

I would keep benchmark v1 frozen and stay in the production phase. The newer snapshot looks much more like a system that now has explicit contracts than one that needs another ranking study.

The highest-information next step, to me, is therefore not another benchmark, another judge panel, or another ranking feature. It is a short end-to-end contract-closure pass: make sure the contracts you just introduced are actually the contracts the live paths execute.

From the newer snapshot I checked, the remaining high-value seams mostly look like this:

Boundary	Useful question	Cheap check
`production-v1`	Does the runtime actually consume the profile as its source of truth?	deterministic profile → runtime fixture
local-only LLM mode	Does “local-only” mean zero remote prompt-bearing attempts?	mock/spied network call
production replay vs live search	Are they now literally the same ranker semantics?	same `EngineResult[]` → compare Top-k IDs/order
query contracts	Does adaptation preserve intent, or tell us when it cannot?	before/after semantic fixtures
health/observability	Do failure classes survive into history without retaining sensitive payloads?	sentinel-query fixture

So my default route would be roughly:

benchmark v1 stays frozen
        ↓
close the runtime contracts
        ↓
production-v1 really drives runtime
local-only really blocks remote egress
live ranker == replay ranker
query adaptation is observable
health semantics survive into history
        ↓
then spend time on queryPlanner / fast mode / later ranking ideas

None of that requires reopening the calibration cycle.

1. I would make `production-v1` an executable contract all the way to the shared search core

The named profile was a good move. I think the remaining question is simply whether:

production-v1/1.0.0

now means the same thing at every production entry point.

In the current snapshot, the profile is very useful for replay/tests, but several ranking semantics are still also represented elsewhere in runtime code. In other words, there is still some difference between:

the profile declares a value

and:

the live adapter necessarily executes that value

That matters for fields such as:

RRF k
trust weighting
input ordering
QF candidate depth
duplicate representative policy
result limit semantics
query policy
fast-mode engine selection

A concrete example is fastMode.

At the public MCP surface it is accepted, and the local MCP wrapper receives it, but in the snapshot I checked the options are reconstructed before calling the shared search function and fastMode is not carried through that reconstruction. The Dashboard path does pass it through.

That is exactly the kind of thing a profile is supposed to prevent.

I would not necessarily refactor all adapters at once. A small boundary such as:

profile
   ↓
profileToRuntimeOptions()
   ↓
shared search

plus one fixture for each supported entry point may be enough.

The invariant I would aim for is:

If two adapters select the same profile and are given the same engine responses, transport choice should not silently change ranking/acquisition semantics.

That still leaves room for MCP, REST, Dashboard, CLI, etc. to have different presentation or transport behavior.

It just makes the search contract common.

2. I would do one more replay-vs-live ranking parity test

The post #14 replay work looks useful, and the original -0.1191 drift clearly exposed a real problem.

The two root causes you reported also make sense as ranking-level causes:

QF replacing RRF ordering
+
different per-engine ordering before RRF

The fix brought the evaluator/replay numbers very close, which is exactly what the production replay adapter was meant to make observable.

There is one remaining distinction I would preserve, though:

historical evaluator
        !=
production replay helper
        !=
actual live search composition

In the current snapshot, rankProduction() appears to combine RRF and QF approximately as:

RRF score
   ↓
QF modifier / penalty
   ↓
combined score
   ↓
sort

while the live executeSearch() path still appears able to go through:

RRF sort
   ↓
applyQualityFilter()
   ↓
quality-filter ordering
   ↓
Top-k

So I would not infer from:

production replay ≈ evaluator

that:

live search ≈ production replay

until one deterministic fixture says so.

The cheapest version is probably:

fixed fake EngineResult[]
        ↓
rankProduction(...)
        ↓
candidate IDs + Top-k order

same fixed fake EngineResult[]
        ↓
the canonical live rank stage
        ↓
candidate IDs + Top-k order

Then compare:

canonical candidate identity
duplicate representative
engine provenance
pre-RRF order
RRF score/order
QF decision
final Top-k

If that matches, I would close this branch and stop thinking about the old replay delta.

This is also why I like keeping behavioral parity separate from quality regression:

refactor:
    same behavior expected
    -> exact/near-exact parity fixture

intentional ranking change:
    behavior expected to change
    -> replay frozen v1 and measure nDCG/MAP/MRR

A deliberate improvement should not fail just because an old golden ordering changed.

3. `local fallback` and `local-only` look like two different contracts

This is probably the privacy boundary I would close first because the test is so cheap.

The README/local configuration now presents a useful distinction between remote and local operation. If the intended meaning of the Ollama-only configuration is:

no remote LLM dependency
no prompt-bearing remote request

then I would enforce that at the transport-selection layer rather than relying on missing API keys to make the remote call fail.

In the current snapshot, the LLM client still has a NIM primary configuration and can attempt the NVIDIA endpoint before falling back to Ollama even when there is no NIM API key.

That gives two perfectly valid products, but they are not the same privacy mode:

remote-preferred:
    remote NIM/OpenRouter allowed
    local Ollama fallback

local-only:
    remote LLM destinations forbidden
    local Ollama only

none:
    no LLM stage

I would treat those as network policies, not merely backend preferences.

The regression fixture can be tiny:

mode = local-only
NIM key absent
Ollama available
remote fetch = spy/mock

assert:
    remote prompt-bearing attempts == 0
    Ollama call succeeds

If you intentionally want “try remote, then local” even without an API key, that is also coherent; I would just describe it as fallback mode rather than local-only.

I would keep the claim narrow here. The code path can tell us whether query/prompt text is sent toward a remote endpoint. It does not tell us what a remote provider retains or does with that data.

4. The engine-query-contract idea looks right; I would make lossy adaptation visible

I think the move from one global query language to per-engine contracts is one of the more interesting production consequences of the benchmark work.

Ahmia is a good concrete example. Its current public search implementation rejects queries over 100 characters or more than 10 space-separated terms, and its Elasticsearch query uses minimum_should_match: "75%":

Ahmia views.py

So a broker really does need to distinguish:

information need
        ↓
broker query
        ↓
engine-compatible query

rather than assuming one free-form expansion is valid everywhere.

The subtle issue is that “engine-compatible” can mean two different things:

syntactically accepted

and:

semantically equivalent enough

For example, in the current Ghost query parser/adapter path, constructs such as:

foo -bar
foo NOT bar
foo OR bar
"exact phrase"

can lose operators or phrase structure when the query is reconstructed from positive terms.

That may be the correct fallback for an engine that cannot express the original query.

I would just make the loss visible.

Something like:

{
  "original": "ransomware NOT windows",
  "adapted": "ransomware windows",
  "status": "lossy",
  "changes": [
    "negation_not_preserved"
  ]
}

or even just:

unchanged
lossless_adaptation
lossy_adaptation
unsupported

would be enough initially.

Then the query-planning problem becomes much easier to reason about:

Can this engine express the requested semantics?
    |
    yes
    -> adapt losslessly
    |
    partly
    -> send bounded lossy form + record that fact
    |
    no
    -> choose another short sub-query or skip that transformation

That is more useful to me than making adaptQuery() increasingly clever while the semantic change stays invisible.

It also gives future acquisition analysis a useful variable:

relevant yield
by
query-adaptation class

so you can eventually see whether lossy adaptation is actually hurting anything.

5. I would evaluate `queryPlanner` as candidate generation first, ranking second

I agree with keeping the planner opt-in.

I would only change one part of the proposed live evaluation.

Because the planner generates new engine-facing queries, it can create a substantially different candidate pool. Once that happens, existing v1 qrels are not automatically a complete evaluation set for the new candidates.

This is a standard pooled-evaluation issue: systems that retrieve documents outside the original judgment pool can look artificially weak when unjudged documents are treated as non-relevant. NIST has a useful discussion here:

Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements

So for the first planner live run I would record:

raw query vs planner

engine success/failure
candidate count
unique candidate count
pool overlap
judged@10
judged@20
latency
nDCG@10

Then:

judged@k remains high
    -> interpret the nDCG result normally

judged@k collapses
    -> do not call the new candidates irrelevant yet
    -> judge only the small set of new Top-k candidates if needed

That does not mean “benchmark phase 2.”

It can be a very small extension:

planner introduces 27 previously unjudged Top-10/20 candidates
        ↓
judge those 27
        ↓
recompute

The earlier expansion experiment already gave a strong production result:

one verbose expansion sent to every engine
    -> poor compatibility / large candidate-yield loss

The planner is testing a genuinely different hypothesis:

one information need
        ↓
several short atomic queries
        ↓
per-engine contract adaptation
        ↓
fusion

So I would keep it as a candidate-generation experiment rather than treating it as another “LLM rewrite quality” experiment.

6. Preserve the 5-way acquisition semantics beyond the immediate response

The zero-result classifier is a useful improvement:

captcha
interstitial
blocked
parse_failure
ok_empty

because:

HTTP 200

only tells us that an HTTP response arrived. It does not establish that the expected SERP was successfully acquired and parsed.

I would carry that semantic distinction one step farther into health history.

In the current snapshot, the immediate response can classify a zero-result event as captcha, interstitial, blocked, or parse_failure, but those paths can still be fed into the breaker/history layer through recordSuccess().

That creates a possible split such as:

current search response:
    parse_failure

historical health:
    success

The same issue gets more interesting when planner/expansion waves are enabled.

The extra waves contribute breaker/history events, while the returned engineHealth primarily describes the original wave. Then:

one user search

can mean:

one event per engine

or:

several events per engine

depending on planner settings.

I would define the health event unit explicitly:

search_id
wave_id
engine_id
query_variant_id
transport status
HTTP status
challenge classification
parse status
result count
latency

The public response can still aggregate this into one simple engine status.

The point is that the underlying history then keeps enough structure to answer:

engine became less healthy

versus:

planner generated three additional difficult requests

without conflating them.

7. I would separate protocol-safe logging from privacy-safe logging

Moving ordinary stdio logging off stdout was definitely the right fix.

The MCP stdio transport contract is explicit that stdout is MCP protocol traffic and ordinary logging can go to stderr:

MCP stdio transport specification

But:

written to stderr

and:

appropriate to retain

are separate questions.

The structured audit log currently includes the raw query, and transport failures can also carry a full request URL into an error string. That matters because the error can then flow into:

engine health
    ↓
health history
    ↓
optional disk snapshot

For a project whose privacy model has emphasized low persistence, I would make the default observability payload more like:

{
  "profile": "production-v1/1.0.0",
  "engine": "ahmia",
  "query_contract": "ok",
  "transport": "ok",
  "http_status_class": "2xx",
  "parse_status": "ok",
  "result_count": 12,
  "latency_ms": 1834
}

and treat:

raw query
full URL
raw snippet/body

as a separate opt-in diagnostic level if they are ever actually needed.

OWASP’s logging guidance makes the same general distinction: useful operational logging does not require recording every sensitive value directly; removal, masking, sanitization, hashing, or encryption can be appropriate depending on the data:

OWASP Logging Cheat Sheet

One very cheap test would be:

query = ghost-private-sentinel-7f3c
force timeout / parser failure
        ↓
search:
stderr
health response
health history
saved health snapshot

If the sentinel appears somewhere you did not intend it to persist, the exact boundary is immediately visible.

8. Decide whether health persistence is an archive or actual cross-session state

The in-memory health history is easy to understand:

current process
    -> accumulate observations
    -> expose trends

The disk path is slightly less clear.

The current code can save health history and load a previous snapshot at startup, but I did not find a corresponding path that hydrates those loaded observations back into the active breaker/trend state.

So there are two coherent designs:

A. Snapshot as an audit/export artifact

save history
restart
read previous artifact if needed

but:
runtime trend begins fresh

Then I would document it as an archive/export feature rather than persistent health state.

B. Cross-session trend persistence

load snapshot
        ↓
validate/version
        ↓
hydrate trend state

Then ghost_health_trends can actually span process restarts.

I do not think one is inherently better.

A true RAM-only operational profile might reasonably choose A or disable snapshots completely.

The useful thing is just making:

persisted artifact

and:

restored runtime state

two explicit concepts.

9. MCP can carry more of the contract/provenance directly

The startup tool-manifest hash is useful for one specific question:

Did my visible tool list/name/description set change?

I would keep it for that if it is useful operationally.

I would not make it carry the whole interface-integrity burden.

The current MCP specification already gives tools explicit JSON-Schema contracts through inputSchema and optional outputSchema, and structured tool results through structuredContent:

MCP Tools specification

That gives a nice separation:

name + description hash
    -> coarse tool-manifest identity

input/output schemas
    -> machine-checkable interface contract

profile/version in structured result
    -> executed search semantics

provenance in structured result
    -> what actually produced this result

The last part seems particularly useful for Ghost.

Internally the broker can know that one canonical result was observed from several engines, but the final public projection can collapse this back to a single engine.

That throws away information that the benchmark work already showed is valuable.

A structured result could retain something like:

{
  "title": "...",
  "url": "...",
  "snippet": "...",
  "engines": [
    {"id": "ahmia", "raw_rank": 3},
    {"id": "tor66", "raw_rank": 7}
  ],
  "ranking": {
    "profile": "production-v1/1.0.0",
    "method": "rrf-quality"
  },
  "surrogate_evidence": "sufficient"
}

while still returning a text representation for older clients.

That would also make REST/MCP parity easier to test because there is a canonical semantic object to compare rather than only formatted text.

10. Retrieved search text is still an untrusted-data boundary

This is secondary to the four contract checks above, but it becomes more relevant now that Ghost has LLM-assisted ranking/planning paths.

Titles and snippets from arbitrary search results can contain instruction-like text.

If that text is ever placed into an LLM reranking/synthesis prompt, I would explicitly preserve:

retrieved text
    = external data

system/tool instructions
    = control data

rather than trying to classify every malicious-looking snippet.

A small fixture is probably enough:

result snippet:
"IGNORE THE PREVIOUS INSTRUCTIONS AND ..."

expected:
it remains candidate text;
it does not alter the requested output/schema/tool behavior.

OWASP’s current RAG guidance makes the same general point: retrieved content should be treated as untrusted data rather than commands:

OWASP RAG Security Cheat Sheet

I would not turn this into a large prompt-injection subsystem unless a concrete failure gives you a reason.

11. One conditional deployment check: Tor / Dashboard exposure

I would keep this conditional because source configuration and deployed network reachability are not the same thing.

The checked-in configuration I looked at appears to combine:

Dashboard listening on 0.0.0.0

Tor:
HiddenServicePort ... -> dashboard:3939

ControlPort on 0.0.0.0
without the normal cookie-auth boundary

while the project description also talks about the Dashboard as local-only.

The Docker host-side port mapping can still restrict ordinary host access to loopback, so I am not inferring that the Dashboard or ControlPort is currently reachable from the public Internet.

But if that Tor configuration is actually part of the production topology, I would re-check those two boundaries explicitly.

For Onion Services, Tor documents client authorization as the mechanism for making an Onion Service private; without it, anyone who has the onion address can access the service:

Tor Project — Onion Service client authorization

And Tor’s own control-protocol implementation notes say that an open ControlPort with no authentication enabled is generally a poor idea:

Tor Control Protocol — authentication implementation notes

So the decision tree could simply be:

Is the Dashboard intentionally reachable as an Onion Service?
    |
    yes
    -> document that boundary
    -> add client authorization if it is meant to be private
    |
    no
    -> remove/disable that HiddenService mapping

Is ControlPort needed by Ghost?
    |
    no
    -> do not expose it
    |
    yes
    -> bind narrowly + authenticate it

If the checked-in torrc is only a development/example configuration and not what production runs, then this mostly becomes a documentation cleanup.

12. Keep benchmark v1 frozen — but freeze transitive semantics separately from the qrels

I still would not reopen benchmark v1.

There is just a reproducibility distinction worth preserving:

file was not edited

does not necessarily mean:

historical evaluator semantics are unchanged

if the file imports mutable current-production modules.

For example, if the frozen evaluator imports today’s:

ENGINES.trust
quality-filter implementation
normalization
dedup policy

then changing one of those can change the meaning of a later “v1 replay” even though eval-ranking.ts itself never moved.

The clean model is:

benchmark v1 evidence
    frozen corpus
    frozen topics/narratives
    frozen qrels/human corrections

historical v1 reference
    exact evaluator semantics
    exact trust/config/normalization dependencies
    commit/hash/vendor snapshot

current production SUT
    deliberately evolves

Then regression stays:

current production SUT
        ×
frozen v1 evidence
        ↓
quality + behavioral delta

while the historical reference remains available to answer:

What exactly produced the number reported when v1 was frozen?

This does not require a container if that is overkill.

An exact commit plus frozen configuration/data dependencies may already be enough.

13. I would describe trust/fast-mode as v1-derived production policy, not held-out validation

The new trust derivation and engine tiers are a reasonable use of benchmark v1.

I would just be precise about what “freeze after deriving” buys you.

This sequence:

derive engine trust/tier on v1
        ↓
freeze those values
        ↓
replay on v1

prevents continued re-fitting, which is useful.

It does not turn v1 into held-out validation.

So I would describe these as:

v1-derived production settings

rather than evidence that the engine weights or five-engine fast subset generalize to future query populations.

There is no action required now.

If later you want to claim something broader such as:

these five engines are generally the optimal fast subset for onion metasearch

then a fresh query set / later SERP snapshot with the v1-derived settings frozen beforehand would be the stronger test.

Until then, using them as current production policy seems entirely reasonable.

14. One wording change on neural reranking

I would narrow one conclusion from the architecture review:

neural ranking methods require full-text document access, therefore query expansion is the only applicable modern path

is broader than the implementation constraint actually establishes.

Some methods absolutely are poor architectural fits because they depend on:

a local corpus/index
full documents
document graphs
large global candidate structures

But neural reranking as a category does not require full-page ownership.

For example, a CrossEncoder normally scores:

(query, candidate text)

pairs and is commonly used to rerank a Top-k candidate list:

Sentence Transformers — CrossEncoder reranking

So for Ghost:

query + title/snippet

is technically enough to define a CrossEncoder reranking experiment.

Whether snippets contain enough evidence for it to beat the current RRF-quality pipeline is a completely different question.

Your own human audit already found a useful warning here: some SERP surrogates are genuinely evidence-poor.

So I would phrase the architecture boundary as:

full-corpus / graph / full-document methods
    -> often poor fit for this broker

surrogate-level neural rerankers
    -> technically applicable
    -> usefulness not established

I would not reopen the benchmark just to test one.

This is mostly about leaving the future-reader map accurate.

15. A few small consistency cleanups I would keep below the main path

None of these would block production work.

License metadata

The repository-level license is now PolyForm Noncommercial 1.0.0, but the MCP package metadata in the snapshot I checked still says MIT.

That is worth aligning simply so people consuming the package get the same usage terms as people reading the repository.

PolyForm Noncommercial is explicitly a noncommercial-use license:

PolyForm Noncommercial 1.0.0

npm’s package metadata guidance also recommends that a non-SPDX/custom license be represented as:

{
  "license": "SEE LICENSE IN <filename>"
}

with that license file included in the package:

npm package.json license documentation

This is not a judgment about which license Ghost should use — only keeping the distribution boundary consistent with the license you already chose.

`purgeCookies()`

In the current transport implementation, the operation named purgeCookies() appears to rotate the SOCKS authentication identity but not actually clear the cookie jar itself.

Rotating identity may already prevent subsequent requests from reusing the same Tor isolation identity, which is useful.

If the intended guarantee is:

purge = remove cookie material from memory

then I would either clear the jar as well or rename the operation to match what it actually guarantees.

Duplicate representative determinism

The fused candidate keeps provenance from duplicates, but representative title/snippet metadata can still depend on which duplicate is observed first.

If ranking features use that metadata, request completion order can become an input to ranking.

A deterministic representative rule would close it cheaply:

merge provenance
        ↓
choose representative by pure rule:
best non-empty surrogate
then stable engine precedence
then stable lexical tie-break

Shuffle semantics

If engine-order camouflage matters as an operational feature, I would also keep one regression test around the actual returned value of the shuffle operation.

This is the sort of tiny plumbing bug that otherwise survives because the function itself is perfectly correct while the caller discards the transformed list.

Again, none of these needs to become a new project.

So the way I would now think about Ghost is:

benchmark v1
    = frozen evidence

production-v1
    = declared semantics

runtime adapters
    = actually apply those semantics

shared core
    = one acquisition/ranking meaning

observability
    = preserve failure semantics
      without quietly widening persistence/privacy

tool boundary
    = preserve schema + provenance

That feels like a much better place to be than the original situation, because the remaining questions are increasingly testable contract questions, not open-ended architecture questions.

If the cheap parity/privacy/query-contract fixtures above pass, I would stop hardening those areas and move on.

I would only reopen the benchmark when a genuinely new system under test — planner candidate generation, a new ranking family, a new engine population, etc. — produces evidence that v1 can no longer answer the decision you are trying to make.

sookoothaii · September 6, 2026, 12:40pm

Thanks for the convergence read — agreed, the remaining questions are contract-closure, not architecture.

I ran your 15 points against the actual codebase. Here’s the honest state:

Already closed (partially):

production-v1 exists as a typed profile (profiles.ts, 18 fields) and is the declared source of truth. rankProduction() in search.ts consumes 4 of those fields directly (rrfK, qualityFilterEnabled, maxPerDomain, maxResults). The other 14 fields (queryPolicy, trustWeighted, inputOrdering, qfCandidateDepth, duplicateRepresentative, outputLimitSemantics, queryPlanner, queryExpansion, llmRerank, snippetFilter, etc.) are consumed by the search orchestration layer — pre-search planning, post-ranking filters, dedup — not by the ranker function itself. I haven’t verified each field is wired end-to-end. The real contract question is: are all 18 fields consumed somewhere in the pipeline, or are some still decorative? Closing this means tracing each field from profile declaration to consumption point.
Structural parity is stronger than it might look. PR #20 extracted

search.ts as the shared search module. PR #21 made the MCP server delegate to it. Dashboard and MCP both call rankProduction() from the same module. The replay script (

replay-production.ts) also imports rankProduction() from the same module. So live ranker and replay ranker are literally the same function — not two implementations that could diverge. What’s missing is your specific ask: a test that feeds identical EngineResult[] through both paths and asserts Top-k IDs/order match. The architecture guarantees it; the test suite doesn’t prove it. parity.test.ts (9 tests) covers REST/stdio semantic parity (dashboard vs MCP tool schemas), not ranking-output parity.
query-adapter.ts returns AdaptedQuery with truncated, termsDropped, operatorsStripped flags. Lossy adaptation is visible at the data level. What’s not verified: whether these flags are surfaced to the caller or logged when adaptation is lossy. The data structure is there; the observability isn’t wired.
health-persistence.ts does save/load snapshots to disk. It’s currently an archive — snapshots are written but not automatically loaded at startup to inform circuit-breaker decisions. Your question “archive or actual cross-session state?” — right now it’s archive. Making it state means loading on startup and feeding into breaker recovery logic.

Genuinely open:

local fallback and local-only are indeed two different contracts and we don’t distinguish them explicitly. The LLM cascade (nim-client.ts) is OpenRouter → NIM → Ollama → raw query. “Local-only” (zero remote prompt-bearing attempts) is not a mode — it’s just what happens when OpenRouter and NIM are both unreachable. There’s no explicit local-only flag that guarantees zero remote egress.

5-7, 9-15: Architectural recommendations, all valid, none implemented. Will address in order of your priority list.

On the default route you proposed:

benchmark v1 stays frozen

↓

close the runtime contracts

↓

production-v1 really drives runtime (PARTIAL — 4/18 fields in ranker, rest in orchestration, unverified)

local-only really blocks remote egress (OPEN — no explicit mode)

live ranker == replay ranker (STRUCTURAL YES — same function; TEST MISSING — no direct parity assertion)

query adaptation is observable (PARTIAL — data flags exist, not surfaced)

health semantics survive into history (PARTIAL — archive, not state)

↓

then spend time on queryPlanner / fast mode / later ranking ideas

The highest-information next step is #1 — verifying that all 18 production-v1 fields are actually consumed end-to-end, not just declared. The ranker itself only reads 4; the rest belong to orchestration stages that need tracing.

Will report back when the contract-closure pass is done.

Topic		Replies	Views
When is a benchmark conclusion identified across evaluator meanings? Research	3	112	August 24, 2026
TIS 2.0: Token Importance Scoring Now Eliminates Position Bias in RAG Research	30	724	September 6, 2026
MosAIc: a reconfigurable surface where an LLM emits a layout, not just text Show and Tell	7	231	July 11, 2026
Reranker Leaderboard — Compare top rerankers using ELO, nDCG, and latency metrics Models	0	829	November 4, 2025
Feedback on Seek, a long-running autonomous deep research experiment Show and Tell	1	79	August 11, 2026

Ghost Search — MCP-native, Tor-routed search layer for RAG (seeking research collaborators)

The Problem

What I Built

What Already Exists vs. What’s Missing

Who This Is For

Where I Need Help

How to Participate

1. Preserve the raw SERPs before normalization

2. Define what the annotator is judging

3. Pool across diverse runs before judging

4. Graded relevance seems worth it

5. Multiple annotators

6. Keep a live track and a frozen track

Parser/runtime

MCP and Dashboard currently appear to have different acquisition paths

Preserve duplicate provenance before dedup

Treat the quality filter as a ranking arm in the evaluation

Distinguish challenge/parse failure from a genuine empty result

Phase 0 — runtime sanity

Phase 1 — live acquisition

Phase 2 — freeze the candidates

Phase 3 — cheap ranking baselines

Phase 4 — human judgments

Phase 5 — expensive models

1. Treat each item as a topic, not merely a query string

2. Freeze the judgment target

3. Use a diverse pool, rather than Top-20 from only one ranking

4. Keep relevance and duplication separate

5. URL uniqueness and content uniqueness are worth reporting separately

6. Pilot the rubric before annotating everything

7. Metrics: make graded ranking the primary result

8. Freeze evaluation before tuning

A. Frozen-candidate test

B. Live-acquisition test

Tor stream isolation

Per-engine timeout values

1. First deconfound parser repair from panel expansion

2. Once the five labels exist, enumerate the panel space offline

3. “Diversity” should mean diversity of error, not diversity of labels

4. I would add one tiny paired wording control

5. Predeclare quorum/tie semantics too

1. Human-label the cases where human information has the most value

2. Judge the benchmark at two levels

Label-level

System-level

3. A useful stop rule

1. Be careful about provisional qrels that favor their own pool contributor

2. Freeze the information need, not just the query

3. Keep topical relevance separate from quality-policy attributes

One evaluator/ranker interaction I would keep in mind

1. Clarify the 217/289 → 239/272 transformation

2. Keep parser/error provenance in the qrels artifact

3. REST/stdio semantic parity and stdio purity are different tests

4. Tor configured-vs-observed is already the right separation

1. The deconfound worked

2. The panel-composition question is basically answered empirically

3. Document-level instability with system-level stability is not a contradiction

4. I would make the current corpus immutable now

1. It is a stress sample, not a random estimate of 1,197-item accuracy

2. The 62 page-fetched items are not exactly the same relevance task

3. One human assessor is enough for this stress test, but not for a strong human-ground-truth claim

Frozen candidate arm

Live acquisition arm

The engine failure mechanism is also worth wording more generally

If query refinement ever comes back, I would change the formulation of the problem

1. I would call qrels_v2 and qrels_v3 “qrel variants” rather than “independent qrels”

2. Make the panel-count convention explicit

3. Pin exact model/provider IDs

4. Keep relevance labels separate from the production quality policy

Benchmark v1 frozen

Surrogate-only vs page-assisted split — done

Ahmia correction — accepted

Pairwise margin caveat — accepted

qrels naming — accepted

Model/provider ID pinning

Relevance vs quality policy separation

Query planner — agreed

External NIST sanity check

What’s next

1. rrf-quality is a pipeline, not just two switches

1. `rrf-quality` is a pipeline, not just two switches

5. `200 + 0 results` needs one more state dimension

10. I would keep `surrogate_evidence` separate from relevance

`purgeCookies()`