Extracting text from product images at scale, accurately

1. Which tool, and when

A sensible default for a seller or agency: an agent that runs vision on every image and writes structured fields, plus a second, cheaper engine (cloud or local OCR) as a cross-check. Wherever the two disagree on a number or a name (after ignoring case and spacing), the agent flags it for a human. That puts human attention on the fields where an error costs the most (numbers, ingredients, claims) instead of on every image. Two engines can agree and both be wrong, so the accuracy test in section 3 still matters.

Vision is not the same as OCR. OCR only reads characters. A model with vision actually looks at the picture, which is why it copes with messy images, and also why it can make human-style misreads. If you want pure character extraction, tell the agent to "use OCR".

2. Build your setup

Answer the questions and get steps and a prompt matched to you. Nothing leaves your browser; this page has no server and stores nothing.

3. How accurate is it? Measure it on your own images

The only accuracy number worth having is one from your own images. Do this before the full batch. The agent does the scoring; the answer key is the slow part. With a sample this size the result is a rough estimate, not a guarantee.

1. Pick up to 50 images: a random sample, plus a separate set of the hard ones (curved bottles, glare, tiny print, dark backgrounds). Score the two sets separately so the hard ones do not hide behind the easy ones. With fewer than 50 images, use them all.
2. Type the right answer yourself for the fields that matter (for example net weight, ingredients, any warning text) into a spreadsheet. This is your answer key. It is slow, and it is the whole point. Start with the one or two fields you cannot afford to get wrong.
3. Tell the agent: "Run the extraction on these 50 images. Compare it to my answer key field by field. Give me a scorecard: percent exact match per field, a list of every miss, and what kind of mistake each one was (misread character, missing line, wrong field, invented text)."
4. Read the misses. If the errors are in fields you can tolerate (a marketing slogan), fine. If they are in fields you cannot (numbers, ingredients, warnings), tighten the prompt, add the cross-check engine, or send those fields to human review.
5. Re-run the same 50 after each change, so you can see whether it actually improved.

What a good extraction looks like (invented example)

A made-up product, to show the shape of the output. The agent would produce one row like this per image.

image:        sample_bottle_front_03.jpg
brand:        Northfield Naturals
product name: Daily Calm Magnesium Gummies
net weight:   60 gummies (120 g)
ingredients:  Magnesium citrate, tapioca syrup, [unclear: ...ctin], natural flavor
warnings:     Keep out of reach of children. Consult a physician before use.
claims:       "Supports relaxation", "Vegan", "No artificial colors"
confidence:   ingredients line partly obscured by glare
flags:        ingredients (unclear text), claims (review)

Notice the three things that make it trustworthy: the source file name is on the row, unreadable text is marked [unclear] instead of guessed, and the row says what it is unsure about.

4. What to do with the text once you have it

Extraction is one step. The value is in what it feeds.

• Structured data. A CSV or JSON row per image, with the file name, ready to load into a spreadsheet, a database, or a listing tool.
• Compare against your listings. Ask the agent to diff the label text against the title, bullets and A+ copy, and list mismatches (a size, a count, a claim that is on one and not the other).
• Competitor teardowns. Run the same extraction on competitors' listing images and compare claims, certifications and sizes side by side. Check the marketplace's terms and copyright rules before you collect or reuse anything.
• Compliance flags, for a human to review. Have the agent flag text that tends to need care before it goes on a listing: health or medical claims, comparative or superlative claims ("best", "#1"), certifications or logos, and anything it cannot read. This is a first pass to point a person at the right lines, not a compliance sign-off. Check the marketplace's current rules yourself.
• Second model as checker. Run a second engine over the same images and have the agent list every disagreement. Disagreements point you at likely errors, but agreement does not prove a reading is right.

5. Is it worth it at your size?

Do not guess; measure. The 5-image test in the prompt gives you the numbers:

• Time per image from the test run, times your image count.
• Usage per image, which you can read off your plan or API dashboard after the test. Multiply by your image count for a batch cost, and check the vendor's current pricing.
• Your current cost: how many minutes a person takes per image today, times their hourly cost.
• Review time: from the accuracy test, what fraction of rows get flagged, and how long a person needs per flagged row. The agent does not remove that work; it shrinks it to the flagged rows.

6. Setting up each tool from scratch

Both tools also come as desktop and editor apps; the terminal is not required. Install commands change from time to time, so if one fails, check the tool's own install page, or paste the error into the agent and ask it to fix it.

The folder layout to use

ocr-project/
  input/        your original images (never edited)
  output/       the extracted fields and text
  logs/         progress log and error list
  truth/        your hand-typed answer key for the 50-image test
  CLAUDE.md or AGENTS.md   the rules below

7. Rules to give your agent

Put these in CLAUDE.md (Claude Code) or AGENTS.md (Codex). The agent reads that file at the start of every session, so you say it once.

• Never modify or delete anything in input/. Outputs go in output/. A rule in a prompt is not a lock, so also keep a separate backup copy of your originals, or make the folder read-only with chmod -R a-w input/.
• Test on 5 files first, show me the results, and wait for my go-ahead.
• Make it resumable. Skip files that already have an output, so a crash or a stopped session loses nothing.
• Log every file to logs/progress.csv: file name, engine used, a confidence note, any error.
• Never guess. If text is unreadable, write [unclear].
• Keep units exactly as printed, and run the cross-check engine on numbers.
• Finish with a report: how many succeeded, how many flagged, and the list of flagged files.

8. Common problems

• The session stopped halfway. Re-run the same prompt. Finished files are skipped.
• Misreads on glare or curved packaging. Ask the agent to try a second pass on those images (crop, contrast, or a different engine) and to flag any that still disagree.
• You hit a usage limit. Run the batch in chunks across days, or use a cheap engine for the first pass and the model only for what it flags.
• Tables or spec panels come out jumbled. Ask for them as their own CSV and have the agent check the row and column counts.
• Install command failed. Paste the exact error into the agent and say "fix this".

9. Keeping it running

A script an agent wrote for you is still a script someone has to own. Decide who that is.

• Keep the project in a git repository and have the agent commit changes, so you can see what changed and roll back.
• Keep the CLAUDE.md/AGENTS.md rules file up to date; it is the instruction manual for whoever opens the project next, human or agent.
• Re-run the 50-image accuracy test whenever you change the prompt, the engine, or the model, and before any large batch.
• If your team will not use a terminal, have the agent wrap the script in a small web page or a drop-folder that people can use without one.

Want this built for your business?

This is the same approach we use at 10ofYou.ai to build working agents for real teams: start small, measure, then scale. If you would like help turning a manual process like this one into an agent that runs for you, get in touch through the site.