How one document is checked
A router (SigLIP with a linear probe) first checks the document is a payslip at all. A field detector (YOLO) finds the ten fields — net pay, gross, IBAN, SIRET — so an edit is pinned to a name, not to pixels. A forensic network I designed for the job (ConvNeXt on the image, SRM noise residuals and the JPEG 8×8 grid) paints the edited pixels. Seven payslip rules check that the sums hold: hours × rate = base, gross − contributions = net. And for a PDF, the file itself is questioned: an old amount still hiding under a white box, a different font, a second revision, an online editor as producer.
Each check catches what the others miss. Fusion keeps them honest: any strong signal makes the risk high, weak signals are only noted, and an unreadable value is "unknown", never "fraud" — a blurry photo cannot create an alert.
| Step | Check | What it does |
|---|---|---|
| 1 · Router | SigLIP | Frozen vision-language model + a linear probe: payslip, RIB or invoice. 100% on the test documents (zero-shot: 68%). |
| 2 · Fields | YOLO | Finds the ten fields so an edit is pinned to net pay, not to pixels. mAP50-95 0.975. |
| 3 · Pixels | TamperNet | ConvNeXt on the image, SRM noise residuals and the JPEG 8×8 grid: copy-move, splice and re-typed digits leave traces there. |
| 4 · Sums | Seven payslip rules | Hours × rate = base, base + extras = gross, gross − contributions = net… on OCR, or exactly on a PDF's text layer. |
| 5 · File | PDF forensics | An old amount under a white box, a different font, a second revision, an online editor as producer. |
| 6 · Words | Granite note | A local IBM Granite model writes the analyst's note; it chooses facts and wording, every figure comes from templates. |
Real output, side by side with the truth
Each example shows the submitted page, what docverify returns (red = edited pixels, boxes = detected fields) and the ground-truth mask the model never saw. All documents are synthetic.
Measured, not claimed
On a frozen test split of 793 payslips (390 genuine), the tamper network reaches 0.902 ROC-AUC and catches 72% of forgeries at 5% false alarms, with a pixel F1 of 81% on where the edit is. Rewritten amounts are its hard case — 50% — and that is exactly where the sums help: image plus arithmetic catches 95% of forgeries at 1.7% false alarms, and the rules alone raised no false alarm at all.
| Check | Result |
|---|---|
| Tamper network, recall at 5% false alarms | 72% (v1: 66%) |
| Image + arithmetic, recall | 95% at 1.7% false alarms |
| Arithmetic, false alarms | 0% |
| Edited PDFs flagged by a strong file signal | 100%, 0% of genuine PDFs |
| YOLO fields, mAP50-95 | 0.975 |
| SigLIP router accuracy | 100% (zero-shot 68%) |
An analyst note that cannot invent a number
An optional local LLM — IBM Granite, through Ollama — writes the analyst's note. The first version let it quote figures and checked they existed in the evidence; it passed while swapping "expected" and "found". So the design changed: the model now chooses which facts to cite and how to phrase them, and every number comes from templates. The system flags; an analyst decides.
The same machine, other documents
PDF forensics and the tamper network detect edit traces, not payslips: they apply to any document as-is. The router needs one more class per new document type; fields and rules are written per document type.
Insurance: repair quotes and garage invoices (hours × rate = labour, Σ parts = total), medical and pharmacy bills (price × quantity, act codes vs tariff), accident reports (copy-move and splice on form zones), damage photos (splice, copy-move, near-duplicate search across claims), proofs of purchase and certificates. Elsewhere: whole tenant files for rental agencies, diplomas and transcripts in HR, benefit applications in the public sector, supplier invoices and bank-detail change letters in accounts payable, invoices and certificates of origin in logistics, seller KYC on marketplaces.
| Document | Typical fraud | What catches it |
|---|---|---|
| Payslip (built) | inflated net pay, gross, base salary | pixel edits + seven sum rules + PDF traces |
| Tax notice | income raised, household size changed | pixel edits; tax = scale(income, parts); the 2D-DOC barcode |
| Bank statement | balance raised, salary lines added | opening + credits − debits = closing, line by line |
| RIB / IBAN proof | IBAN swapped to a mule account | IBAN checksum, bank code vs name, font on the IBAN line |
| Proof of address | name or address pasted in | splice detection, PDF producer, consumption × tariff |
| Company accounts | revenue or equity edited | assets = liabilities + equity; totals = sums of lines |
| Invoices (factoring, leasing) | amount or VAT edited, fake supplier | excl. VAT × (1 + VAT) = incl. VAT; SIRET / VAT checksums; duplicate hashing |
From one CPU to bank volumes
Workers are stateless, so capacity is pages per worker times workers. Measured: 540 pages an hour on four CPU threads for the full pipeline, and about 8,200 an hour for the tamper network on a single T4. OCR, not the neural networks, is the bottleneck: native PDFs skip it entirely because their text layer is exact.
| Volume | Peak hour | CPU only (4-vCPU pods) | CPU + GPU |
|---|---|---|---|
| 10k documents / month — a regional lender | ~70 / h | 1 pod | not needed |
| 300k / month — a mid-size bank | ~2,100 / h | ~9 pods at peak, 1–2 at night | 1 T4 + ~6 CPU pods |
| 3M / month — a national group | ~21,000 / h | ~86 pods at peak | 3 T4 + ~57 CPU pods |
Project and weights
Project README · tamper model on Hugging Face · YOLO field detector.
Limits
Synthetic data only: no number here was measured on real payslips. A forger who edits every amount consistently escapes the arithmetic, and an edited bonus breaks no checked total. The PDF forgeries were made by me, so their 100% is optimistic. The analyst note's summaries can be slightly off; the facts it cites, and every number, are always the exact ones.