Can AI Spot a Bad Business Plan?

More and more people paste a business plan into a chatbot and ask what is wrong with it: founders before a raise, investors before they write a check, etc. The bot obliges in a confident tone providing lots of details. The question is whether it catches the problems that actually sink a business, or just lists irrelevant or invented ones.

That is hard to test, because you can't run an experiment. You can't even observe and make conclusions. There are ways around though.

I approached the problem in two ways. First, I generated simulated plans with AI, and intentionally planted mistakes in them. Second, I took real small businesses that raised money from the public, anonymized them, rebuilt each plan as it stood on the day of the raise, and scored the reviews against what later went wrong.

Unfortunately, you have to be careful with anonymization: given only a renamed business plan, a model identified the real company in 37 of 68 cases. That would be a forward informational leak, so we were left with 31 clean cases that the models did not know about.

I asked two frontier models, Claude Fable 5.1 and DeepSeek 4.1, to find what is wrong with business plans. Out of 60 mistakes planted in invented plans, Fable found 59 and DeepSeek 50. On real small businesses, scored against what later went wrong, the clean evidence is 31 companies the models could not recognize. There both caught about four in five of the problems, 81% and 82%: a virtual tie.

Test 1: plans with planted mistakes

Both models found most of the planted mistakes, and Fable missed none.

GPT-6.1 Sol, a third model, wrote 10 business plans of about 1,800 words each, across very different businesses and countries: US software selling into Germany, UK ready meals, Brazilian lending, a German molding shop, US home care, Indian apparel stores, Dutch electric-van delivery, Mexican education technology, a UK renovation acquisition and Texas restaurant franchising. Each plan hid 5 to 7 mistakes, from the obvious to the specialist-only: a works council the plan forgot, the wrong VAT treatment, missing botulism controls for a food product, a Dutch transport license, the timing of a franchise disclosure. Each also carried 2 red herrings: details that look alarming but are fine.

For every planted mistake I wrote down what a review must say to get credit for it. One plan, for example, raised $6 million at a $24 million valuation and told investors they would own 12%. They would own 20%. To get credit, a review had to work out the 20% and say the 12% is wrong. A vague "check the cap table" did not count.

The agent then double-checked every plan and every answer key. It recalculated every table and looked up every legal claim. It dropped two planted mistakes that turned out not to be mistakes, and added two that the original answer key had missed. That left 60 mistakes to find.

Each model then got each plan in a fresh web chat, with the same instructions: review it the way a strong board would, and say what is wrong. The full request is quoted below:

So Fable found nearly everything. That sounds impressive, but it also means this test is too easy to be useful. When one model scores 59 out of 60, there is no room left to see whether another model would do even better.

Real plans are also harder than these. Our invented plans were short and neatly written, and every mistake was put there on purpose, waiting to be found. A real plan is longer and messier, and its mistakes are unintentional. That is why I ran a second test.

Test 2: real businesses, real outcomes

On 31 real companies the two models could not recognize, both caught about four in five of the problems that later mattered, and they tied.

Planted mistakes have a weakness: whoever plants them decides what counts as a mistake. So the second test uses real history. In the US, small companies can raise money from the public under Regulation Crowdfunding. Each files an offering statement with the SEC (Form C), then yearly reports with its actual revenue, profit, cash and debt. That gives a plan written before the fact and a record of what happened after.

How I built the real cases

1. Find the raises. Claude Code searched the SEC's EDGAR database for Regulation Crowdfunding offerings filed from mid-2016 to 2018 and kept those that reached their funding target. That left 243 small businesses: restaurants, breweries, bars, apparel brands, consumer goods, fitness studios, travel and other services.

2. Find out what happened. Each company that raised money must file a yearly report (Form C-AR) with its revenue, profit, employees, cash and debt. Claude collected those and then researched each company's fate in state business registries, news and its own web presence: still operating, closed, sold, changed business, or never opened, with the year and the cause where one was reported. Every fact carries its source. 64 companies had at least two yearly reports or a clear outcome.

3. Rebuild the plan as it stood on the day. From each company's offering statement and business plan (two were scanned images, read with OCR) the agent extracted every forecast, milestone, assumption and stated risk, each with its exact quote. Its website was taken from the Internet Archive snapshot closest to the filing date. From that material alone the agent wrote the plan in the company's own voice, 1,200 to 3,000 words. Nothing dated after the filing went in. The company's own figures were kept exactly, including its arithmetic mistakes, because finding them is the test.

4. Disguise it. The company, its founders and its exact location were renamed; the region was kept. For example, a barbecue restaurant "City Griddle" in a large North Texas city became "Hollow Creek BBQ". The decision question was written as an investor would have faced it on the filing date: the real amount raised, the real use of the money, and no hint of the outcome.

5. Write the answer key. For each case Claude Code identified the problems the later record shows actually mattered: the quote and section where each sits, why it matters, and a "caught if" rule saying what a review must say to count. For example: "tests the revenue against the 1,584 square feet and concludes it is implausible; a generic 'forecast seems optimistic' does not count." The key also lists red herrings: worries the record shows were unfounded, such as a forecast that was actually met. Problems nobody could have known at filing time, such as the pandemic, are recorded but not scored.

6. Check what the models already know. We identified 26 companies out of 64, based on the breadth of information available about them with the local Gemma 4 model. Claude built 26 cases and gave each disguised plan to GPT-6.1 Sol, asking which real company it was. It named 22. For 13 of those it also knew what had happened, so they were dropped. The test ran on the remaining 13: 4 it did not recognize and 9 it recognized without knowing the outcome.

Only those 4 were clean, too few to conclude anything. So I went back for more. Claude took the SEC's full crowdfunding data, 2016 to 2022 and tech companies included, and found 874 more companies with at least two yearly reports. Sol, asked about each by name, city and website, did not know 346 of them. The agent kept the 148 of those that had reported $100,000 or more in yearly revenue, built 42 plans the same way, and asked Sol again with each disguised plan. It named 15 of them, even though it had not known their names. That left 27 more clean cases, 16 of them tech companies.

This time I also asked Fable and DeepSeek themselves. Fable named 2 of the 27 and knew only that both were still operating. DeepSeek said it recognized 20 and named the wrong company every time. Together that makes 31 clean cases with 158 scoreable problems:

The score counts a caught problem as 1 and a partly caught one as half. The difference between the two is one percentage point, which is noise. Both did better on non-tech plans (87% each) than on tech plans (78% each). "Partly" almost always means the model named the right area but not the number or rule that proves it.

One clean case shows what a catch looks like, and what was at stake. A counter-service barbecue restaurant in North Texas raised a revenue-share note in April 2018. Its record shows five problems a reader could have seen at the time:

• first-year sales of about $751 per square foot, or $3,258 a day from opening, with "70 tables" that cannot fit in 1,584 square feet;
• a lunch-dependent concept on a strip the plan itself says is quiet at lunch;
• barbecue that depends on a pitmaster nobody has named, run by founders from finance and sales;
• "net profit" counted before the note payments, which leave about $95,000 (8%) in year one;
• about $159,000 of the $350,000 project unfunded at the minimum raise, with rent running before opening and no reserve.

DeepSeek caught all five. Fable caught four and only partly caught the pitmaster: it named the gap but did not treat it as a key-person risk to fix before investing. What happened: the restaurant opened in October 2018, a year after its lease began. That year it took in $100,083 and lost $243,990. Its pitmaster left shortly after opening, and it closed in 2019 after about six months.

The catch: renaming a company does not hide it

Renaming every company, its founders and its town was not enough. Given only a renamed plan, GPT-6.1 Sol named the real company behind 37 of 68 cases, from details like the founders' backgrounds, the year they started and an unusual concept. That includes 15 of 42 companies it had not recognized by name.

That breaks the test for those 37. A model that can name a company from a disguised plan has that company in its training data. Some it named without knowing how the company turned out, but a model's report of what it does not know is not evidence. So only the 31 it did not recognize count.

In the first batch I also scored 9 cases that Sol recognized but whose outcome it said it did not know. On those, Fable scored 78% and DeepSeek 66%. On the 31 clean cases that gap disappears. Maybe knowing the company helps Fable more, but 9 cases are too few to tell.

Each model also recognizes different companies. I gave the 27 added plans, none of which Sol recognized, to Fable and DeepSeek as well. Fable named 2 of them correctly. DeepSeek claimed to know 20 and was wrong every time. A model's claim to recognize a company is no more trustworthy than its claim not to.

This matters beyond our test. Anyone who scores AI on real companies, real court cases or real patients should assume the model may see through the disguise, and check.

False alarms: half the real red herrings got flagged

On real plans, both models called nearly half of the harmless details problems.

A red herring is a detail that looks risky but is fine. In the planted test the models rarely fell for them. In the real test they did, which the planted test alone would have hidden:

Thirteen is still a small sample, but the pattern is worth knowing. One red herring from the first four clean cases shows how it happens. A fitness studio also ran an online video service with "more than 100" subscribers. Both models called it a distraction from the studio and the franchise plan. It is the part of the business that survived: today the brand runs entirely on livestream and on-demand classes.

Planted red herrings are written to look risky and be fine on the page. Real ones look risky and turn out fine only later. A model asked to find problems will find some that are not there.

What this means if you ask an AI to review your plan

One careful request to a strong model is a cheap, serious first review. Treat what it flags as a list to check, not a verdict.

• Ask it to check the numbers against each other. The catches that mattered were arithmetic across sections: sales per square foot against the floor plan, profit before and after debt payments, a raise against the cost of the project.
• Push for the figure, not the topic. Most misses were partial: the model named the right worry ("the forecast looks optimistic") without the number that proves it. Ask "by how much, and what would have to be true?"
• Expect false alarms. On real plans both models called nearly half the harmless details problems. Before you cut something a model calls a distraction, ask what evidence would show it is working.
• Don't read much into which model wins. On 31 clean cases they scored 81% and 82%; these tests cannot rank them. DeepSeek certainly looks good enough here.

This is the request both models received, followed by the decision question and the plan:

I'm about to make an important business decision and I want the most rigorous review you can give me before I commit. Please review it the way a strong board would: an experienced investor, a CFO, an operator who has run this kind of business, and a lawyer who knows this industry and country. Don't be polite and don't summarize the plan back to me. I want to know what's wrong with it. Read the whole plan carefully, check the numbers against each other, and think about what would actually happen if I executed it as written. Then give me: your recommendation (go ahead, go ahead with conditions, or don't); the problems you found, most serious first, each with what is wrong, where, why it matters (with numbers where you can) and what to do about it; and anything that looks risky but is actually fine. Only list problems you actually believe are real. I'd rather have eight real ones than twenty generic risks.

How I ran it, and what it cannot tell you

These are small tests, run once, built and scored by AI. Read them as a signal, not a ranking.

Who did what. No person reviewed each answer key or grade. Every step was done by a model, named here:

Sol wrote the test 1 plans and graded both tests, but it was not a contestant in either. In test 2, Claude Code / Opus built the cases that Fable, a Claude model, then reviewed; if that helped Fable, it did not show on the clean cases.

Grading. Each answer was scored mistake by mistake against its key, quoting the answer as evidence for every verdict, with anything that would identify the model removed. A mistake counts as caught only if the answer says what its key requires; a generic warning in the same area does not count. A red herring counts against an answer only if the answer presents it as a real problem. As a check, a fourth model, Gemini 3.6 Flash, regraded 9 of DeepSeek's real-test answers before its daily free quota ran out: same total score (79% on those 42 problems), but the same verdict on only 74% of individual problems. Whether a partly caught problem is "caught" is a judgment call, and that noise sits under every number here.

The limits.

• Small numbers. One run per model and case: 60 mistakes in test 1, and 158 clean problems in 31 cases in test 2.
• Easy plans in test 1. Short, tidy AI prose with mistakes put there to be found. Real plans are longer and messier.
• Recognition. Of 68 real cases built, 31 are clean. Fable and DeepSeek were checked on the 27 added cases; the first 4 were checked with Sol alone.
• Survivors. The added cases needed two yearly reports, so most of those companies lasted at least two years, and only one closed. Their answer keys lean on missed forecasts, cash and debt, not on collapse.
• Different settings. Web search was on in test 1 and off in test 2. Results across the two tests are not directly comparable.
• One kind of judgment. Both tests ask what is wrong with a written plan. They say nothing about gathering facts, advice over a long conversation, or decisions that are not on paper.

The whole exercise cost me about $10 in model calls, plus two chat subscriptions. Obviously, a $100 and a week of work would buy a more statistically significant study.

Disclaimer: the analysis was done with Claude Code, and consequently this text is heavily AI-assisted.