Free kit · nothing for sale

How often is your document extraction wrong without anyone noticing?

FlowCraft Labs makes a free kit that scores an invoice extractor against known answers and reports its silent error rate. This is for people who have to decide whether extracted data can be trusted.

Status. The kit is available to download now. Nothing is for sale.

Get the free kit How the kit works

Free download

The free eval kit

A small Python package that runs on your machine. It generates synthetic invoices with known ground truth, runs an extractor, applies deterministic checks, and reports per-field accuracy, how many injected defects the checks catch, and the silent error rate.

flowcraft-eval-kit.zip

Download the kit

Size: 61,947 bytes
SHA-256: 66bffaa6f7d5eecea70d1c01e7da8866052cfb4652006e015ad45587bdddad33
Needs: Python 3.12 or newer, and pip install jinja2 weasyprint. The weasyprint package is needed only for PDF output.
Sends: Nothing. The kit makes no network requests of its own. Only the optional model extractor connects anywhere, and only to the address you give it.
Licence: Free to use, no resale, provided as is. Read the license.

Check the download before you run it: sha256sum flowcraft-eval-kit.zip on Linux, shasum -a 256 flowcraft-eval-kit.zip on macOS, or certutil -hashfile flowcraft-eval-kit.zip SHA256 on Windows. The result should match the line above.

unzip flowcraft-eval-kit.zip -d flowcraft-eval-kit
cd flowcraft-eval-kit
pip install jinja2 weasyprint
python -m deadroom_audit.plugins.flowcraft.cli eval --extractor baseline --n 200 --seed 7 --out ./eval --no-pdf
python -m deadroom_audit.plugins.flowcraft.cli synth --n 50 --seed 1 --out ./synthetic
python -m deadroom_audit.plugins.flowcraft.cli verify extracted.jsonl

eval scores the built-in extractor on generated invoices. synth writes sample invoices as PDFs with their ground truth, so it needs weasyprint. verify reads one JSON record per line and prints how many were accepted and how many need a person.

Read this before you quote any number from it

The built-in baseline extractor was written for the generator’s own invoice templates. Its scores on the synthetic invoices say nothing about real documents.
Arithmetic checks cannot catch a wrong vendor name or a wrong invoice number when the record still adds up.
The figure that matters is the silent error rate measured on your own documents. The kit’s command line scores extractors on synthetic invoices only. For your own documents, verify counts accepted and needs-a-person records, but it cannot know which accepted records are wrong. Someone has to compare a sample with the originals. If you write Python, evaluate.score takes your own ground-truth records and extractor outputs.

How it works

Three steps, all on your machine

Generate
The kit writes invoices in three layouts, four date formats and five number formats. Each has a known correct record. Some get one of five injected defects: a wrong total, a duplicate number, a tax mismatch, an impossible date or a line sum that is off.
Extract and check
An extractor reads each invoice and returns a record. The checks below run on that record. A record is accepted only when no check fails. Otherwise it goes to a person.
Report
The kit compares every record with its known answer. You get report.html and report.json, a DISCLOSURE.txt and a MANIFEST.sha256 file you can verify, plus report.pdf unless you pass --no-pdf.

What is checked

The rules, and nothing else

These rules look only at what a record says about itself. Money is compared to the cent. Everything else needs a person.

The checks, what each looks at, and its limit
Check	What it looks at	Limit
Line totals	Each line total equals quantity times unit price, to the cent. Half-up and half-even rounding are both accepted.	Does not know whether the quantity or price was read correctly.
Subtotal	The line totals add up to the subtotal.	A consistent misreading of every line passes.
Total	Subtotal minus discount plus tax equals the total.	Passes when the total is wrong in a way that still adds up.
Tax	Tax equals the tax rate times (subtotal minus discount), in either rounding mode.	Does not check that the rate is the legally right one.
Dates	Issue date and due date are real calendar dates, and the due date is not before the issue date.	A wrong but valid date that keeps the order passes.
Currency	The currency code is on a fixed list of recognised codes.	A wrong but recognised currency passes.
Required fields	Vendor, invoice number, issue date, due date, currency, lines, subtotal, tax rate, tax and total are present.	Present is not the same as correct.
Sanity limits	The total is above zero, each quantity is above zero and under a set ceiling, and no unit price is negative.	Catches impossible values, not plausible wrong ones.
Duplicate numbers	In a batch, the same vendor and invoice number twice is a fail.	The first copy passes, and so does any invoice not in the batch you give it.
Possible duplicates	In a batch, the same vendor, total and issue date under a different number is a warning.	A warning does not stop a record from being accepted.

A record is accepted when it has no failed check and no error. Verification does not show that a document is genuine, that an invoice is owed, or that a tax rate is the right one.

What passes when it is wrong

Vendor names, addresses, descriptions and invoice numbers that are unique.
A wrong but valid date that keeps the due date after the issue date.
A wrong but recognised currency.
A unit price, a quantity and every total all misread in the same consistent way.
The first copy of a duplicated invoice, and any invoice not in the batch given.

This is the reason the silent error rate is worth measuring.

Limits

What this does not do

Prove a field is right

Checks show that a record is consistent with itself. They cannot show that it matches the document.

Predict your results

Scores on synthetic invoices describe those invoices. Your layouts, scans and vendors will differ.

Give professional advice

This is an operational review. It is not legal, medical, tax, accounting, security or compliance advice.

Take regulated data

No protected health information, card numbers, ID numbers, passwords or keys. Not without a separate written agreement.

Data handling

Where your data goes

The kit. Runs on your machine and sends nothing. Its optional model extractor calls only the address you configure.
This site. No cookies, no analytics, no tracking and no third-party requests. Our host, Cloudflare, keeps ordinary request logs.
We do not sell your data.
Software, not AI text. The free kit and its reports are produced by software checks. They contain no AI-written text.

Full detail is in the privacy policy and the terms.

Questions

Plain answers

What does silent error rate mean?

It is the share of records that pass every check while at least one field is wrong. The kit counts records the checks accepted that differ from the known answer, and divides by all records. It also reports that count divided by accepted records alone.

A wrong record that a check sends to a person is not silent. A flaw printed on the document itself is scored as a defect, not as a silent error, when the extractor copied it faithfully.

Why not just trust the model?

Any extractor can misread a field and still produce a record that adds up. Nobody sees the mistake unless something checks it. The checks here catch arithmetic and format problems. They cannot catch a misread vendor name. Measuring a sample tells you how often that happens on your documents.

Where does my data go?

For the kit, nowhere. It runs on your machine and makes no network requests of its own. The optional model extractor calls the address you set in DEADROOM_LLM_URL, so point it at a local server if the documents must stay put.

Who sells this?

Deadroom Ventures LLC, a Nevada limited liability company, operates FlowCraft Labs. Write to hello@deadroomventures.com.