Every post comes from something I've done on a real project. AI tools, development approaches, how I actually build things. You're getting a curation of my taste, not takes on stuff I don't use. Subscribers also get extras: things that went wrong, how my thinking about AI is changing, hacky workflows I use every day, and the occasional personal update. Stuff I share with subscribers because it's a little too personal or unpolished to blast across the internet.
unit tests make OCR easier to improve
|
Hey, There's been a ton of valid skepticism around benchmarks recently, but there's a lot of good reason for optimism too! Joe and I are going to discuss one of those in the document space, which is how unit tests are being used in benchmarks. And all of this is applicable to your own product's evals. After all, a benchmark is just another person's evals 😅. This Wednesday, Joe and I are discussing why unit tests make strong OCR benchmarks and how to design them. We’ll work through examples from ParseBench, olmOCR-bench, and OmniParseBench. Join us free from 7:30–8:30 p.m. Eastern on Nov 18. ​https://isaacflath.com/research/unit-test-benchmarks​ Isaac P.S. PDF to Production registration closes this Friday, November 13. It’s $500, including recordings, notebooks, and 30 days of follow-up support: https://isaacflath.com/pdf2production​ |
Isaac Flath
Every post comes from something I've done on a real project. AI tools, development approaches, how I actually build things. You're getting a curation of my taste, not takes on stuff I don't use. Subscribers also get extras: things that went wrong, how my thinking about AI is changing, hacky workflows I use every day, and the occasional personal update. Stuff I share with subscribers because it's a little too personal or unpolished to blast across the internet.
Hey, Registration for PDF to Production closes tomorrow, November 13. Joe Barrow and I teach this Saturday, November 14, from 10 a.m. to 6 p.m. Eastern. It’s $500, including recordings, notebooks, $1200 of vendor credits, and 30 days of follow-up support. See the curriculum and register: https://isaacflath.com/pdf2production If you'd rather stick to free resources and piece things together yourself, check out the Document Roadmap. It goes through an overview of key misconceptions and key...
Hey, I wanted to understand how decision models (like Jev) actually work in detail. I use them a ton, and it's clear they're going to be a key part of software moving forward. Joe and I spent an hour going through all of that, using Perplexity Decider (OSS decision model) to go through the architecture. The talk walks through how this is built: Turning a decoder model into an encoder by changing its full-attention layers. Replacing the language-modeling head with a small head that produces...
LLMs can be made faster with speculative decoding, even while doing more computation (wut???). It does this because the extra compute is spare GPU compute that would be idle while waiting for other parts of the system to finish its tasks. But this only works in specific situations (usually low batch size). It's still useful loads of the time, and you can dynamically adjust based on the situation. This Wednesday, Joe and I are giving a free talk on why this works. We’ll walk through the...