Evaluate retrieval without changing the model

Compare lexical, semantic and hybrid retrieval on the same authorized memories, revisions, query set and externally generated vectors. This workflow records reproducible evidence. It does not promote an experimental ranking method or establish improved agent reasoning from a retrieval score alone.

The 0.5.0 synthetic contract report preserves a negative result: hybrid was not admitted after an exact error-code regression.

What the bundled fixture establishes#

benchmarks/retrieval/contract-fixture.json contains 40 fictional memories, hand-authored four-dimensional vectors, explicit relevance judgments and 18 queries. Nine development queries and nine reserved test queries cover exact identifiers, error codes, file/class names, paraphrases, ambiguous terms, Portuguese, near-duplicate records, noise and unanswerable questions.

Every query explicitly grades every document. Grades mean 0 irrelevant, 1 marginal, 2 useful and 3 directly resolves the information need. Each category has only one query per split. These are author-assigned synthetic judgments, not independent adjudication or evidence of an embedding model's language quality. The vectors exercise ranking and protocol behavior; they were not generated by a language model. Neither weights nor ranking versions are tuned on these queries.

Use an authorized, representative corpus with documented judgments and real external embeddings for a relevance qualification study. Preserve a development split for choosing server-owned ranking versions and reserve a separate test split for final comparisons. Do not repeatedly tune against the test results.

Run a frozen comparison#

Use Python 3.9 or later on Linux or macOS. The evaluator uses the standard library and never calls an embedding provider. Obtain a dedicated platform credential with memory_read and memory_write for an isolated test scope. Store its issuance JSON securely outside the repository. Save the scope as a separate JSON file, for example:

{"project_id":"retrieval-evaluation","mission_id":"trial-v1","agent_id":"evaluator","visibility":"private"}

Then run from the QilbeeDB checkout:

python3 scripts/evaluate_retrieval.py \
  --fixture benchmarks/retrieval/contract-fixture.json \
  --credential-file /secure/path/evaluation-credential.json \
  --scope-file /secure/path/evaluation-scope.json \
  --state /secure/path/evaluation-state.json \
  --report /secure/path/evaluation-report.json \
  --base-url http://127.0.0.1:7474 \
  --container qilbeedb-local \
  --repetitions 3

The credential file's secret is read in memory and is never written into the report. HTTP credentials are allowed over loopback; remote servers require HTTPS. Redirects are rejected rather than forwarding a bearer credential to another URL. The optional --container enables read-only Docker cgroup resource sampling; it does not restart, change or create the container.

The evaluator creates tagged synthetic records and attaches supplied vectors, using deterministic idempotency keys. It writes a manifest containing their server-generated UUIDs, exact revisions and embedding receipts. Reuse that state file and unchanged database for subsequent runs. A new database assigns different UUIDs and may change equal-score ordering; a fixture hash alone does not freeze server IDs. Records remain available for reproducible reruns. Delete them through the memory API when the trial is complete; deletion does not imply receipt erasure.

The tool verifies source content and revisions before and after the trial and requires complete scan/embedding coverage. Changed revisions, unexpected records, duplicate results, partial scans, ranking instability or HTTP failures invalidate the comparison. The JSON report records failures instead of averaging them away; a failed comparison returns a nonzero exit code. Input/model or preparation errors fail before qualification. Keep the state file if preparation is interrupted: identical retries recover the original durable receipts.

Manifest persistence and concurrent runs#

Use a private, pre-existing directory on a local filesystem for state files. A run holds a nonblocking POSIX lock from preparation through final source verification. A second process using the same state path fails before issuing requests. The lock is released by the operating system on exit or a process crash; its hidden .lock file stays in place to avoid races between different lock inodes. Do not remove a lock file while evaluators are running. These are cooperating-process locks, not distributed coordination for network filesystems or different state paths.

Preparation checkpoints at most every 64 newly completed source/binding pairs and at successful completion, avoiding quadratic per-record serialization on larger corpora. A crash can leave up to 64 durable server receipts absent from the local checkpoint. Retry with identical keys recovers those IDs and bindings without duplicating records; an interrupted create/attach pair is completed the same way.

Each state replacement writes a mode-0600 temporary file, syncs its contents, atomically replaces the state and syncs the containing directory. A directory-sync failure is reported as failure even if the replacement is visible; retry with the same state and idempotency keys. Filesystem and hardware guarantees still apply. State and lock symlinks are refused. Before resuming an interrupted preparation, the evaluator checks the saved document set, unique UUIDs, revision and embedding identities, then verifies existing source content before creating further records. It never silently rebuilds a manifest that points at a reset or changed database.

Pin a trial before executing it#

Create a plan without database credentials or HTTP calls:

python3 scripts/evaluate_retrieval.py \
  --fixture benchmarks/retrieval/contract-fixture.json \
  --split development --ranking-version weighted_rrf_v1 \
  --repetitions 3 --seed 20260920 \
  --write-plan /secure/path/development-plan.json

Pass --plan /secure/path/development-plan.json to the run command above and omit --repetitions. A pinned plan cannot be overridden with split, seed, repetition or ranking flags. The plan binds the canonical fixture digest, including corpus, vectors, model identity, judgments and query splits; it also records the exact server profile, final response size and trial budgets. The evaluator rejects a changed fixture or returned hybrid profile. Reports retain the plan and its hash. Keep plans in version control or an independently timestamped experiment registry. Hash matching by itself is not proof of preregistration or independent review.

--split development executes only development queries, including warmup and resource passes. Test summaries have zero queries and null measurements because they were not run. --split test executes only test queries; all keeps the original behavior. Both splits remain in the frozen fixture so the plan detects corpus or judgment edits. Preparing document embeddings does not use query judgments, but operators must separately control access to reserved queries. Previously inspected test queries cannot become fresh holdout evidence by changing a flag or an embedding model. Preserve them as regression or exploratory evidence.

Compare equivalent requests#

All methods use the same corpus tag and authorized scope, queries and final k = 10. Dense and hybrid queries reuse the exact frozen vectors and model space. Requests use up to 10000 source records/bindings and, for lexical/hybrid, up to 67108864 serialized bytes. The existing cosine contract counts embedding bindings; lexical/hybrid count source rows. Requiring complete scans removes page truncation as a relevance confounder. Work counters expose the actual difference.

These are experimental trial budgets, not enterprise production limits. The current implementation has hard request ceilings; this release does not add a tenant quota or admission-control API. Production resource policy and service objectives must be defined separately.

weighted_rrf_v1 fixes method, weights, rank constant and candidate cap on the server. It combines ranks, not raw BM25 and cosine scores. The evaluator records returned ranking versions and the complete hybrid profile, including its experimental flag. Requests cannot introduce ad hoc combination weights.

Read the report#

The command writes JSON evidence and a sibling Markdown report. Development and test results are separate, and reserved-test results are broken down by category.

The nDCG formula follows the graded-ranking treatment in Introduction to Information Retrieval. For incomplete judgments, the evaluator uses zero gain for unjudged results and reports judgment coverage; do not describe those scores as fully judged relevance. Unanswerable queries are excluded from nDCG/recall averages and reported separately.

The trial runs one client, with one warmup pass and a seeded shuffled method/query order on each measured pass. It does not flush RocksDB or OS caches. Resource passes are separate, serial warm passes. Container CPU includes telemetry and any other traffic in that container. memory.current is a point sample; memory.peak is the lifetime container peak, not a proven trial-specific maximum. Use a dedicated server and a separate load study for cold-cache, concurrent or production-capacity claims. Small synthetic category samples cannot support a service-level objective or a representative relevance conclusion.

External embedding latency, generation cost and generation-to-retrieval latency are unmeasured, represented as null with a reason. Frozen-vector reuse is not zero-cost model generation. Record separate provider measurements when running real-model qualification. No provider credential is sent to QilbeeDB.

See external embedding evaluation for an optional local encoder pipeline with pinned artifacts and separate generation measurements.

Qualify hybrid retrieval and agent outcomes separately#

The bundled evaluation always leaves hybrid experimental. Adoption needs useful gains on representative reserved queries, uncertainty estimates, no critical exact-query regressions and passing functional isolation/recovery criteria. If a method loses, preserve that result in the report. No improvement target has already been established by the synthetic fixture.

Next, run Qilbee tasks while holding the model, prompt, tools and acceptance criteria fixed. Compare task completion, observed effects, tool calls, tokens and time. Record evidence and rollback criteria before promoting a learned behavior. This evaluator reports that agent-task stage as not_run; it cannot infer better reasoning or autonomous improvement from retrieval metrics.