Fingerprinting and library identification

Two ELF executables in the public corpus. 02c1abdd… is 239,016 bytes; 0513ba35… is 334,312 bytes. Their sha256, md5, sha1, GNU build-id and TLSH all differ. One still carries its function names (tool_create_output_file, create_dir_hierarchy); the other is stripped down to sub_210e0. Four derived hashes over them come out identical:

                 02c1abdd…  239,016 B    0513ba35…  334,312 B
build-id         2d1b3885e137ff4b…       0ece1ef25bd636ef…
tlsh             T18F343A17A5851CB…      T110646C13F9821C7…
import_md5       0ce141cf4e755e54…       3491189c6aa3f6fc…
symbolhash       181ca76ef0473133…       98202f531322badd…
──── identical ────────────────────────────────────────────
aspecthash       9a6be24b0edba286…       9a6be24b0edba286…
syscallhash      6f6ab0a72af9ab38…       6f6ab0a72af9ab38…
indicator_hash   5eda02c0cbce9a90…       5eda02c0cbce9a90…
hardening_hash   6b2a964b50b008a5…       6b2a964b50b008a5…

Both files are curl, built by different Linux distributions — same glibc 2.34, but minimum-kernel ABI tags of 3.2.0 and 4.3.0. Identification runs in three layers over what no whole-file hash can say:

• Bundled signatures — 1,835,557 mined quad-hash entries, 1,777,840 of them served; 1,015 trained FLIRT pattern blobs holding 1,096,000 entries; 67 curated library signatures; a Mach-O body-hash corpus.
• Corpus-mined names — bodies, semantic anchors and call-graph labels learned from binaries that are already identified, with no pre-built signature involved.
• Family attribution — a byte-free index of 1,130 labelled malware samples across 188 families, compiled into the engine executable.

Byte hashes and derived hashes

What a hash is computed over decides whether it survives a rebuild. The four that agree across the pair above are computed over derived sets rather than over bytes:

Corpus similarity ranking

Three signals score a neighbourhood; only two of them share a ranked list, and only one separates the sibling curl from unrelated software. Scoring 02c1abdd… against the 3,899 other binaries in the similarity index:

rank  name          cosine    weighted-Jaccard   bigram dHash
1     curl          0.98677   0.7104             16
2     libdpv.so.3   0.98458   0.1474             23
3     bsdtar        0.97969   0.1380             13
4     ssh-agent     0.97103   0.0624             15
5     scp           0.98243   0.0527             20

The dHash bands describe the wrong relationship in both directions:

On the byte picture alone bsdtar sits closer to 02c1abdd… than the sibling curl does.

Signature mechanisms

Five mechanisms match code, and each gives up something to survive one of the four losses that stand between a signature and a match: relocation and PIE rewrites of operands, whole-symbol-table stripping, recompilation by a different toolchain and version, and recompilation for a different instruction set. A signature that matches only identical bytes answers a question a sha256 already answered.

When two of them name the same address, a fixed precedence decides:

Quad-hash is the version-bearing tier and the only one that is relocation-aware by construction. Per instruction it splits an opcode prefix from the scalar operand bytes. The loose hash replaces every operand with the sentinel 0xfeeddead, so PIE rewrites, jump-table layout drift and ASLR offsets do not perturb it. The tight hash keeps the literal operand bytes except where the instruction's byte range overlaps a relocation entry, where the sentinel goes back in — so a call site whose immediate the dynamic linker patches hashes the same across distributions. Alignment NOPs are dropped from the hash and from the instruction count; call instructions still hash but are excluded from the count. Bodies under 32 bytes are skipped as noise; the hash streams up to 8,192 instructions.

On a statically linked binary the tight hash cannot bind — the case the bill of materials exists for. A fully static image carries no relocation coverage, so its tight hash is computed over literal operands the signature masked out, and would veto every otherwise-good hit. The match falls back to the loose hash plus the instruction count alone, and a hit clearing only that pair is stamped at the weakest tier above — a guess that never displaces a name. That two-field key is coarse enough to need its own filter: a key appearing under six or more distinct library names is a generic opcode skeleton — a tiny getter, initialiser or wrapper many unrelated projects vendor — and is deleted at corpus-build time, removing 628 keys and 57,717 entries, 3.14% of the mined total, leaving the 1,777,840 served.

The curated tier is the only one that reads meaning rather than shape. 67 signatures across 12 libraries — OpenSSL (20), zlib, SQLite and libcurl (8 each), then zstd, mbedTLS, libpng, xxHash, LZ4, U-Boot, Dropbear, BusyBox. A signature is a size window, the strings the function must reference as data, and — in 20 of the 67 — a prologue hash:

zlib 1.3 · inflate
  size       1024 … 16384 bytes
  strings    "incorrect header check"       ← matched
  algorithmic constants
             0xEDB88320  CRC-32 polynomial  ← recorded, never matched

Sixteen of the 67 record algorithmic constants — CRC polynomials, SHA, MD5 and AES initial values, the zstd frame magic, xxHash primes — and not one is consulted: matching a constant needs the value recovered at IL level, and that leg was never wired. All 67 leave the separate required-constant field empty as well, so in practice a curated match is a size window plus strings, or a size window plus a prologue hash. That costs exactly the eight signatures whose only distinguishing evidence was the constant table. Carrying no string and no prologue, they can never fire:

• zlib — crc32
• OpenSSL — SHA256_Init, SHA1_Init, MD5_Init
• zstd — ZSTD_getFrameContentSize
• xxHash — XXH32, XXH64, XXH3_64bits

A string-only signature clears an evidence bar, because where a string carries the match it carries all of it. A single anchor names a function only if it is at least 12 characters and survives a distinctiveness test that strips printf conversion specifiers before counting letters; anything shorter needs a second anchor. That bar exists because the earlier substring comparison — "%s: %s\n" is a substring of countless real messages — fabricated high-confidence BusyBox matches across roughly 12% of the corpus. The comparison is now exact string equality.

Pattern length is ungated, so "exact" is a statement about the comparison, not about the answer. Of the 1,096,000 entries in the bundled pattern corpus:

libaom's aom__dummy_function sits in all three: the byte C3, an x86 ret, with its CRC16 computed over zero further bytes. It matches every ret-only stub in every x86_64 ELF, and the stripped curl above carries 43 of them at a 16-byte stride, in a binary that links no AV1 codec.

Version coverage

69.6% of the 1,835,557 bundled quad-hash signatures carry a multi-segment version that can be ordered against a CVE range, against 93.2% that carry a version string of any kind. The gap is that a version mined from a soname is often a bare ABI-major integer — 3, from libcrypto.so.3 — which cannot satisfy a range like "before 3.0.14" because it has no minor or patch segment to order.

The 32-point gap on x86_64 is the one that matters, because x86_64 is where most vulnerability questions get asked. Where the version is only an ABI-major, the library is named and the CVE match falls back to the range's own major.

The published coverage figure is not this table. It aggregates three of the six mining corpora — the Linux base, Linux-distribution and BSD bundles — and reports 1,332,153 signatures over 3 architectures and 1,225 library names, while the matcher loads all six. It is 503,404 signatures (27%) short, and silent about exactly the three architectures carrying the embedded coverage: mips32, mips64 and arm.

Architecture and format coverage

Coverage is uneven by tier, and whole (format, arch) cells hold a single library:

On Intel macOS and on 32-bit MIPS the pattern tier is a rounding error.

A third of the pattern corpus is a copy filed under the wrong cell, and 109 further blobs name nothing but .text. Matching stops at the first entry that fits, in corpus iteration order, so where blobs duplicate, sort order rather than evidence decides which library name a hit carries.

Nearly half of each PE cell is import thunks, not function coverage.

Three of the six quad-hash mining corpora carry reach nothing else does — one is the largest, and two are the only ones reaching embedded targets:

Fifteen libraries exist in the pattern corpus on macho/arm64 and nowhere else — libcares, libevent, libgio-2.0, libglib-2.0, libgobject-2.0, libharfbuzz-icu, libmp3lame, libmpdec, libmpg123, libopenjp2, libtheora, libtheoradec, libunistring, libupb, libwebsockets. Pattern matching is gated on the exact (format, arch) pair, so none of them can name the same library statically linked into a Linux or Windows binary.

Apple's userland is the largest hole. The Mach-O body-hash corpus spans 11 kernel images (t8103, t6000, t6020, t8112, t8122, t6030, t6031, t6041, t8132, t8142, vmapple), all from a single OS build — macOS 26.1 (25B78). No Apple system dylib, and no Apple library version, is in any bundled corpus.

Corpus-mined naming

A second layer carries no pre-built signatures at all: it mines names from binaries that are already identified and re-applies them. Nothing in it runs by default — each tier is a no-op unless the deployment points it at a mined database, and the fuzzy and re-anchoring tiers additionally need the token sidecar built beside that database. Writing back is gated separately from reading: a run contributes to the corpus only under a per-job curator authorization, so an ordinary upload reads and adds nothing.

Every accept is gated:

Exact body clustering hashes every recovered function body with relocations normalised away, so the same code collides across binaries. A name propagates only once its body has appeared in two or more distinct binaries — one sighting never seeds a name.

Semantic anchors key a function on the set of API names it calls, its distinctive immediate constants and its distinctive string literals rather than on opcodes. None of those depend on the instruction encoding, so one evidence set matches the same library compiled for MIPS, ARM, PowerPC, RISC-V or x86. A function offering fewer than four anchors is never matched at all.

The fuzzy tier sits behind the exact one to absorb its brittleness. The fast path keys on a hash of the whole joined token set, count included, so one extra inlined literal in a static build changes the hash and the reference .so becomes a total miss. That cannot be tuned out of a hash lookup, so a second index carries it: every token points at the records carrying it, a document-frequency table demotes tokens every binary has (memcpy, malloc) and promotes rare ones (a crypto magic constant, a distinctive error string), and candidates are reranked by IDF-weighted Jaccard. It only runs on near misses of the exact tier, and the best candidate must reach 0.5 on that kernel — a match carried entirely by ubiquitous tokens scores below it.

Re-anchoring is what makes anchors work on a stripped static binary, where every direct call resolves to a placeholder and the call channel collapses. Measured there, constants and strings alone recover about 11% of functions. But the moment a callee is named by its own surviving anchors, its address enters the symbol map, the caller re-extracts and now emits a token naming that callee — which can push the caller over the floor, naming it, which names its callers. The round-0 seed is stricter than steady state because a wrong seed poisons everything downstream. Within a round, naming works off a snapshot taken at the round's start, so the order functions are visited in cannot change the result.

Library-label propagation is measured but not connected. It spreads a library label rather than a name: direct-call targets are decode-derived and PC-relative, so the within-library call subgraph survives static linking and stripping intact — gated on 538 MIPS firmware ELFs, 1.0 million edges, zero panics. A neighbour would inherit the label only when its own anchor evidence independently narrows to exactly that one library; high-fan-out edges are skipped because that fan-out is the signature of two functions merged by faulty boundary inference rather than of a real hub. No ELF, PE or Mach-O analysis path calls it — the only callers are its own tests. The call graph reaches naming through re-anchoring instead.

A library label is only ever attached to a binary that structurally is a library — an ELF carrying DT_SONAME or a Mach-O carrying LC_ID_DYLIB. Executables have neither, so an executable can acquire a library label only by byte-matching code that a real shared object taught. The bill of materials cannot invent a dependency out of an executable that merely resembles one.

Curator names publish to a separate database on a separate rail from the auto-learned one, so a deliberate human assertion is never confused with a cross-binary inference.

Family attribution

The same machinery names a malware family instead of a library, by k-nearest-neighbour vote over a byte-free index of 1,130 labelled samples across 188 families. Each entry is a family label, a sha256, and the sample's DNA set axes — no executable bytes, and not reversible to the malware. The index is compiled into the engine executable, so an air-gapped deployment gets attribution with no runtime fetch and no data directory to populate. The failure mode moves to the build rather than disappearing: a checkout that has not generated the corpus compiles with no index embedded at all, and attribution is then skipped for every binary with a log line and no other symptom.

The vote is its own metric, not the corpus-similarity one: 32 flat-weighted set axes plus embedded formats at 2.0 and hardening posture at 3.0, k of 5, and no corpus discounting — a member every sample carries counts as much here as a rare one. Corpus similarity runs 40 axes, including eight behavioural-signature axes absent here, and carries neither embedded formats nor hardening.

The weights are opinions, and an axis participates only where at least one of the two samples is non-empty — an axis usually empty on both sides drops out of the average rather than scoring zero. Measured against the shipped 1,130 entries, the heaviest axes are the emptiest:

A vote is therefore carried by the populated axes — hardening posture, linked frameworks, ATT&CK techniques, process control, consistency flags, static libraries — not by the entitlement and indicator axes the weights nominate as decisive.

If the single best neighbour scores below 0.55 the classifier returns no family rather than a low-confidence guess. When the sample under test is itself in the corpus, its own entry is held out — otherwise it elects itself at similarity 1.0 and the measurement reports memorisation.

The corpus is heavily skewed, and a family with one labelled sample gets one neighbour:

Known failures and gaps

The one-byte-signature failure is a class, not an incident. aom__dummy_function is the visible end of it. On one real static-musl curl, a single libaom signature (av1_get_third_pass_mi — 3 bytes, 31C0C3) matched 87 distinct functions; whole collision clusters fabricated libaom (342 functions), libldns (133) and libgnutls (114) as components of a binary that links none of them.

Two filters run before anything reaches the bill of materials: any single signature attributed to 8 or more distinct functions in one binary is dropped as degenerate, and a library then has to average at most 2.5 functions per distinct signature to count as evidence at all.

The suppression protects the component list, not the function table — which is exactly why the stripped curl's software bill of materials correctly names zero statically linked libraries while its function listing still shows all 43 aom__dummy_function rows.

Go is invisible to every mechanism above. go build compiles the runtime and the whole standard library into the same image as the author's main, so no library signature ever matches: measured on committed fixtures, 0 of 1,552 functions are named in a Go 1.18 Linux binary and 0 of 2,337 in a Go 1.25 macOS binary. Go identity comes from a different place entirely — the runtime's own function-name table, which survives stripping because the runtime needs it.

That table also supports a distinction none of the signature tiers can make: standard-library code splits into mandatory toolchain code, linked into essentially every Go image and therefore telling you nothing about this one, versus author-elected imports that are present only because someone wrote the import. Over 80 well-recovered Go images spanning 17 families, package prevalence is bimodal — 28 of 259 standard-library packages appear in 95–100% of images, then a nine-package shoulder, then a long tail — and the cut sits at 95%, in the sparse region.

Prevalence is not bimodal at function granularity (9,635 symbols, smooth decay), so a mandatory package over-suppresses its own elective functions:

Rust crates are absent. No Rust crate appears in any bundled corpus, so a statically linked Rust binary's crate-level dependencies are not named by any tier. Its libc is.

MinGW-built PE gets a name and never a version. The FLARE corpus is MSVC-ABI only; the MinGW/GCC ABI is covered by the trained pattern tier, which carries no version at any architecture. The same applies to ppc64, riscv64 and s390x, which the pattern tier reaches and quad-hash does not.

The platform provenance is coarser than it looks. The Linux-distribution bundle is mined from nine container images — Alpine 3.20, Fedora 41, Arch, Ubuntu 24.04 and 22.04, Amazon Linux 2023, Rocky 9, Oracle 9, Debian trixie — but the stored records carry no per-distribution tag, so a hit says "a Linux distribution", never which one. Splitting it requires a change to the miner, not to the query.

The precision gate is a library, not a pipeline. Coverage alone hides the failure that matters — a confident wrong name corrupts both the bill of materials and the set of functions the decompiler skips, which is strictly worse than a blank. The evaluator for that exists: it scores predictions on a stripped binary per function against its unstripped twin's symbol table, computes precision, recall and F1, and accepts a change only when F1 strictly improves and coverage does not regress. It is unit-tested against a synthetic predictor, and a runnable end-to-end harness sits on top of it: mine a reference set of symbol-bearing binaries into one anchor sidecar, then per target capture ground truth, strip a copy, re-analyse it through the production path, and score per binary plus an aggregate — warning when a target also appears in the reference set, because that overlap inflates recall. What is missing is not the harness but the verdict: no build gate consumes it, and no F1 number from a real run is recorded anywhere.

Reference

• Binary anatomy — where whole-file hashes, build IDs and symbol tiers sit in the format.
• Platform — the corpus, the similarity index, and how neighbourhoods are served.
• Malware — what the verdict does with a family hit.
• Fast Library Identification and Recognition Technology — Hex-Rays' description of the pattern-plus-CRC16 scheme every FLIRT tier here implements.
• Ghidra Function ID — the quad-hash design, including the mask sentinel and the full/specific hash split.
• Karta — Check Point's source-to-binary matching, and the origin of the idea that a function's called-API and string set identifies it across architectures.