Every vendor selling document reading into this industry will offer you an accuracy figure. Ignore it. Not because they are lying, but because the number cannot mean what you need it to mean, and treating it as the deciding fact leads operations to buy the wrong thing and then trust it in the wrong places.
This is written for the person who runs the intake side of a factoring operation: the one whose team keys invoice numbers, matches rate confirmations, opens remittance advices and finds out on a Thursday that an invoice was funded twice.
What actually arrives
The documents that reach a factoring company are not the documents in a vendor demonstration. A demonstration uses a clean invoice, generated by accounting software, exported as a text bearing PDF, one invoice per file.
What arrives is a photograph taken on a phone at a truck stop, at an angle, with a thumb in the frame. A fax of a scan of a print of a fax. Nine invoices in one PDF with no separator page. A bill of lading with a stamp over the delivery date and a signature over the stamp. A remittance advice that is a screenshot of a table pasted into the body of an email. A rate confirmation from a broker whose template changed last month. An assignment notice returned as a photograph of a screen.
Volume is not the problem. Variance is. A system that reads a clean invoice well and fails silently on the photograph is worse than a person, because the person knows they are guessing.
What breaks, specifically
Segmentation. A single file containing several invoices is the most common cause of the worst outcome, which is the same amount funded twice or an invoice quietly dropped. Deciding where one document ends and the next begins is a different problem from reading the text, and many systems treat it as an afterthought.
Repeated values. A total appears in three places on a typical invoice: a line total, a subtotal and a grand total. A model asked for the amount will return one of them, and it will not tell you which one it chose or how close the alternatives were.
Tables that wrap. Line items running across a page break lose their association with the header row. The extracted result looks complete and is missing the last four lines of a schedule.
Identity. The debtor name on the document is rarely the debtor name in your system. It is an abbreviation, a division, a trading name, or the name of a subsidiary that pays from a shared treasury. Getting this wrong does not produce a visible error. It produces exposure recorded against the wrong obligor, which is invisible until concentration matters.
Ambiguity that has no correct answer. A date written as a sequence of numbers is two different dates depending on the market the document came from. A currency symbol shared by several currencies is a real ambiguity in an operation funding across borders. A model will resolve these confidently. Confidence is not the same as being right.
Near duplicates. The same invoice submitted twice with a different invoice number, or resubmitted with the amount altered. Reading is not what catches this. Comparison across the client and across the portfolio is.
Why the accuracy figure cannot help you
An accuracy figure is a single number produced by averaging across fields and across a corpus. Both of those averages destroy the information you need.
Averaging across fields treats the amount and the address as equally important. They are not. A misread address is a nuisance. A misread amount is money. A misread debtor identity is a concentration error that nobody will notice until it matters.
Averaging across a corpus tells you how the system performs on the documents the vendor tested it on. Your document population is not that corpus. If your book is freight, your population is photographs of bills of lading. If it is staffing, it is timesheets from twelve different systems. The figure does not transfer.
The question that actually predicts operational outcomes is not how often the system is right. It is what happens when it is wrong, and what happens when it does not know.
The two things that matter
A strict schema, enforced outside the model. Extraction output should be required to conform to an explicit structure, and a response that does not conform should be rejected rather than repaired. Repairing a malformed response is where quiet corruption enters, because the repair is a guess made by code that has less context than the model did. Rejection is noisy, and noisy is what you want. It produces an exception a person handles rather than a value nobody questions.
Deterministic revalidation in ordinary code. Whatever the model returns is a proposal. Before it touches your book it should face checks that are arithmetic rather than probabilistic. Line items sum to the stated total. The date parses and falls in a plausible range against the schedule it arrived on. The invoice number does not already exist for this client. The debtor resolves to an entity that exists in your system rather than to a string that looks similar. The amount does not deviate from this client's own median in a way that is worth a second look.
None of those checks are clever. All of them are certain, which is the point. The intelligence proposes and the deterministic layer disposes.
An example
A carrier submits an invoice for a load, with the rate confirmation attached. Extraction reads both. The invoice amount and the rate confirmation amount differ by the price of an accessorial charge that the broker did not agree to.
A system optimised for accuracy reads both documents correctly and funds the invoice, because reading was the job and reading succeeded. A system built around revalidation compares the two figures, finds a difference that no rule permits, and holds the item with the discrepancy stated and both documents attached. The second system may have identical extraction quality. It produces a completely different operational result.
The same pattern applies to a bank account change arriving by email, which is the most expensive message a factoring company receives. The correct behaviour is not to read it accurately and apply it. The correct behaviour is to classify it as requiring human verification and refuse to apply it at all.
A framework for evaluating any extraction claim
Ask three questions, in this order, about every document type you actually handle.
-
What does a wrong value cost here? Sort your document types by the price of a silent error. Amount, invoice number, debtor identity and bank details sit at the top. Almost everything else is a nuisance. Design the controls around the top of that list and stop worrying about the bottom.
-
Which of the checks are deterministic? Ask the vendor which validations are arithmetic and which are the model checking itself. A model verifying its own output is one system with two opinions. Deterministic checks are the only ones whose behaviour you can predict before you buy.
-
What happens when it does not know? Ask to see the refusal. A system that returns a confident value for an unreadable field will do that on your book every day, and you will not find out for months. A system that says it cannot read this field, on this document, and routes it to a person is telling you the truth in a way you can operate on.
Then ask a fourth question about evidence. When an extracted value is later disputed, can you produce the original document, the values that were extracted, and the checks that were run, as they stood at the moment of the decision? If the answer is that the system would re read the document now, you do not have evidence. You have a recreation, and a recreation is worth nothing in an audit.
The document intelligence page describes how FactorFox implements the schema and revalidation approach, and intelligence with evidence covers why a verification run captures its evidence at the time it runs and is never re fetched. For freight specific document handling, including rate confirmations and the double brokering problem, see transportation.