Why Substantive Testing Was So Hard to Automate

For 20 years, audit firms had every reason to automate substantive testing.

They just had no practical way to do it.

The evidence that goes into an audit procedure is different at every client. A report or contract at one company cannot be processed exactly the same way as one at another. Documents look different. Terms are structured differently. Systems are configured differently.

I saw this on my first day on an audit engagement at EY.

You could build automation company by company, but that work rarely transferred. The learning from one client did not reliably apply to the next. So the math was simple: it was faster to do the work manually, faster to offshore it, and more profitable to let staff handle mechanical execution while seniors focused on judgment.

That equation held for decades.

Then large language models changed the shape of the problem.

For the first time, auditors could write instructions in natural language and have software reason over different documents, formats, and accounting setups. The same logic could transfer across clients because the system could interpret the support instead of only matching rigid templates.

This was not a culture problem. Accountants have wanted this solved for years. It was a technology problem.

When GPT-4 reached ChatGPT in March 2023, I prototyped the first version of what became Punchcard within days. Two weeks later, we submitted our Y Combinator application.

The reason we moved quickly was simple: audit automation finally had the missing layer. Software could read, reason, cite, and adapt across the messy reality of client evidence.

That is the core of Punchcard. We are not trying to turn audit judgment into a black box. We are building agents that handle the repeatable execution around judgment, while every answer stays tied to the source a human can inspect.

Substantive testing was hard to automate because the inputs never stayed still. Now the software can move with them.