Storage architecture

Keynote is a 60,096,016-byte universal Mach-O. The call graph recovered from it is 84,000,308 bytes — larger than the binary it describes. The catalog row that answers is this a Mach-O, is it malicious, who uploaded it measures 1,388 bytes.

Across one corpus of 7,579 binaries, 3,857 of which have a call-graph sidecar on disk:

       110 B   lib/libresolv-0.9.33.2.so   (smallest of 3,857)
    20,045 B   p50
   598,330 B   p90
12,842,687 B   p99
84,000,308 B   Keynote                     (largest)
─────────────
     1,388 B   a catalog row, compressed — one hand-built representative
               row, not a corpus percentile

Five orders of magnitude in the detail, a near-constant in the catalog. That gap is the design: a field you filter across binaries and a field you read for one binary cannot live in the same store without one of them setting the cost of the other.

Every binary is therefore stored under two shapes — a fixed-size row in a SQL catalog for anything filtered across binaries, and files on disk for anything read about one binary — with an in-RAM key-value store for identity, billing and the job queue, and separate vector and full-text indexes for search. One upload writes into six stores, and only three of the artifacts that result cannot be rebuilt.

Catalog rows and detail sidecars

The split is by read pattern: the catalog row carries everything a list, a filter, a facet or a visibility check reads, and the per-binary detail carries everything else, opened only when a request asks for that specific part.

The detail is split further by which request needs it: the call graph opens only when a call-graph node is asked for, the full exports and imports lists only when those are asked for, neither when a binary is merely looked up.

One ingest writes more than the two halves, and everything after the catalog transaction is best-effort: a failed write logs a warning and continues, so a lost card bundle degrades that binary's detail page until the next re-analysis rather than failing the upload.

Reading one detail file is a read plus a parse, and the parse dominates: 28.0 µs for a 12 KB card bundle, 55% of it parse; 2,047 µs for a 4 MB firmware bundle, 89% parse. A 256 MiB in-process cache keyed by scope, SHA and analysis version sits in front of that, so a repeat view skips the syscall and the parse, though a hit still copies the record. Its budget is bytes rather than entries because bundle sizes span three orders of magnitude, and a re-analysis writes a new version, so the key misses rather than returning stale.

Compression, measured on the representative row above and on Keynote's real call graph:

A blob without the zstd frame magic is returned verbatim on read, so rows written before compression still decode. Decompression is bounded — 2 GiB for the full analysis tree, 192 MiB for the strings sidecar — and a decode that reaches its cap is refused as truncated rather than parsed from a partial prefix.

Keys and visibility scopes

The registry row is keyed globally by SHA while the analysis record is keyed by SHA and visibility scope, so the same bytes can carry three isolated analyses at once — three parties analyzed the same file.

lib/libresolv-0.9.33.2.so appears twice in that ledger, under two upload ids and one SHA. The registry row is one; the sightings are many.

Answering when was this hash first seen, and what filenames has it worn is a point-get on a per-SHA rollup in the catalog, folded incrementally on every upload, which keeps the ledger off the request path. On a rollup miss it falls back to scanning the append-only ledger line by line — 14.0 ms p50 over a 100,000-row ledger, 93.8 ms over 650,000 — a scan that grows with total uploads ever made rather than with corpus size, memory bounded by the one SHA's own record count and time by nothing. The rollup is a derived cache, so a dropped write costs one slow read and heals at the next boot reconcile.

Storage engines

Four stores hold the platform's state, chosen by access shape rather than by data size, so a stall in one plane cannot reach another.

Three stores sit beside them:

Matching a 12-component SBOM produced 240 findings in 8.2 ms p50 against a 10,000-CVE index and 8.2 ms against a 100,000-CVE index. The cost is the 12-component fan-out, not the index.

Durability and rebuild

Three things cannot be rebuilt — the accounts write-ahead log, the uploaded bytes, and the upload ledger. Everything else is derived, and a deploy re-derives it.

A re-analysis does not append to the upload ledger. That is what makes the ledger irreplaceable rather than merely inconvenient to lose: the sightings history of every binary is reconstructible only from it.

Catalog scaling: 100K to 1M binaries

Ten times the rows costs 1.7× the resident memory and 7.3× the point-get latency: memory stops being the ceiling and read latency inherits the job.

Measured on one macOS aarch64 box, release build, with catalog rows and vectors synthesized so no analysis time is included. Every latency is single-threaded per-operation unless stated. The catalog runs in WAL journal mode at normal synchronous durability — not an fsync per commit — with a 5-second busy timeout, and bulk import commits every 5,000 rows, the batch the rows-per-second figures amortize over.

The catalog is page-cached SQL rather than heap-resident objects, so resident memory tracks the working set and not the corpus — 10× the rows, 1.7× the memory. The counterweight is the point-get: 17.3 µs to 125.9 µs, 7.3× for 10× the rows, because at 1M the working set no longer sits in page cache.

Concurrency has a knee, not a cliff:

Throughput is bought with latency: across that 100K sweep p99 climbs 24× while reads per second climb under 5×. Writes serialize through a single thread regardless — SQLite admits one writer — but reads run on a pool of eight connections off the same database handle, so readers parallelize while the writer stays alone.

Vector index metric mismatch

The ANN index was trained with the library's default L2 distance while every query asked for cosine, and on a metric mismatch Lance emits a warning and silently falls back to brute force — so the index existed, cost 10.5 seconds to build, and was bypassed by every query.

The 100K and 1M rows are benchmark output; the three 500K rows come from the diagnostic run that identified the metric bug and are recorded in no benchmark artifact.

44.0 ms against 5.19 ms is 8.5× for 10× the rows — the scaling of a linear scan, not of an approximate index. The 13.6 ms flat scan at 500K is the same curve that reads 44 ms at 1M.

Roughly half of the 80 ms end-to-end at 1M is not search at all: 44 ms is the approximate query, ~36 ms the rerank. Each query pulls 500 candidates, then point-gets and deserializes every candidate's content sets to score them by weighted Jaccard — a cost linear in that pool. The pool was widened from 100 to 500 because cosine ordering on the count vector and Jaccard ordering on the content sets diverge, and true top-K matches fell outside the tighter pool.

Deploying the metric fix does not repair an index that already exists: a build that finds one present folds the new fragments in and returns without checking which metric it was trained with, so a store built under L2 keeps flat-scanning until it is wiped or explicitly reindexed.

The 1M benchmark has not been re-run since, so the 44 ms above stands as the last measurement at that scale — the cost of brute-forcing a million 112-dimensional vectors, not of the index that now serves the query.

Vector index parameters and maintenance

The index probes 20 IVF partitions per query, prefilters through scalar BTree indexes, stores the DNA vector zero-padded to 112 dimensions so product quantization divides evenly, and is maintained by incremental fold rather than retrain. The probe count changed alongside the metric and had never been exercised, because it is a no-op on a query that flat-scans.

nprobes — the number of IVF partitions probed per query — is now set explicitly on every vector query. From the same 500K diagnostic run:

Scalar BTree indexes on the visibility and file-kind columns make the prefilter an indexed predicate rather than a pre-scan.

Vector widths decide whether product quantization applies at all:

The pad to 112 is the deliberate part; the index's own partition count follows the data, at roughly the square root of the row count.

The learned embedding gets an index of its own only once 1,024 rows carry a non-null value; below that floor its search flat-scans. The floor exists because the attempt itself is a full-table training-sample scan, which ungated ran on every two-minute maintenance tick even on a corpus holding zero embeddings.

Index maintenance is incremental, so ingest never pays a corpus-wide rebuild:

Both search planes carry a schema revision, and a bump invalidates the plane rather than migrating it:

Undecodable postcard sidecars

Every call-graph sidecar written as postcard could be written and never read: its calls, symbols, data_xrefs and noreturn_vas fields are untyped JSON values, postcard can serialize those but can never deserialize them, and a non-self-describing format cannot answer deserialize_any. Every write succeeded. Every read failed. Every call-graph request silently returned no graph. Two more sidecar classes carried the same untyped lists and the same defect.

All three replacements changed the encoding and the extension, so the dead sidecars are orphaned rather than migrated, and a re-analysis is what restores the graph. The 2.67 GB they still occupy is reclaimed by an operator sweep of the cache roots, safe to run with the server up because no code path can read what it deletes.

Vector upsert data loss

The vector store had the mirror-image failure: its upsert deleted the SHA's row and then appended the replacement, as two commits, so when a row arrived carrying a column the on-disk schema lacked, the delete committed and the append was rejected — re-upserting a binary destroyed its vector. The index drained one row at a time, silently, and only while being refreshed.

It is now a single merge_insert on the SHA key: one commit, matched rows updated, unmatched inserted. A table missing a column the current schema expects is either backfilled from a declared default or refused at open, because refusing to open is the failure mode an operator notices.

Known costs

Five measured paths remain unoptimized.

Not measured

Four gaps bound what the numbers above can be read to claim.

Related briefs

• CVEs & SBOM — the SBOM-to-CVE path these numbers describe.
• Platform — what the engine puts into the catalog in the first place.