Every post comes from something I've done on a real project. AI tools, development approaches, how I actually build things. You're getting a curation of my taste, not takes on stuff I don't use. Subscribers also get extras: things that went wrong, how my thinking about AI is changing, hacky workflows I use every day, and the occasional personal update. Stuff I share with subscribers because it's a little too personal or unpolished to blast across the internet.
wtf a decision model is and arch details
|
2 new free resources. First, If you've been wondering wtf a decision model actually is, join us. Joe Barrow and I are going to be going into architecture details on the top OSS decision model by Perplexity. It's a free live discussion tomorrow night, Wednesday, on Decision Models like Jev. Second, the written guide on formative and summative evals a few weeks ago was very popular, so I made an 11 minute video walkthrough going through it and all the examples (agent and OCR examples over financial, medical, construction domains). It's about how to do error analysis on document workflows and products. And then how to turn those into fast tests to make your system more robust forever. Hope these are helpful, Isaac |
Isaac Flath
Every post comes from something I've done on a real project. AI tools, development approaches, how I actually build things. You're getting a curation of my taste, not takes on stuff I don't use. Subscribers also get extras: things that went wrong, how my thinking about AI is changing, hacky workflows I use every day, and the occasional personal update. Stuff I share with subscribers because it's a little too personal or unpolished to blast across the internet.
Hey, Registration for PDF to Production closes tomorrow, November 13. Joe Barrow and I teach this Saturday, November 14, from 10 a.m. to 6 p.m. Eastern. It’s $500, including recordings, notebooks, $1200 of vendor credits, and 30 days of follow-up support. See the curriculum and register: https://isaacflath.com/pdf2production If you'd rather stick to free resources and piece things together yourself, check out the Document Roadmap. It goes through an overview of key misconceptions and key...
Hey, There's been a ton of valid skepticism around benchmarks recently, but there's a lot of good reason for optimism too! Joe and I are going to discuss one of those in the document space, which is how unit tests are being used in benchmarks. And all of this is applicable to your own product's evals. After all, a benchmark is just another person's evals 😅. This Wednesday, Joe and I are discussing why unit tests make strong OCR benchmarks and how to design them. We’ll work through examples...
Hey, I wanted to understand how decision models (like Jev) actually work in detail. I use them a ton, and it's clear they're going to be a key part of software moving forward. Joe and I spent an hour going through all of that, using Perplexity Decider (OSS decision model) to go through the architecture. The talk walks through how this is built: Turning a decoder model into an encoder by changing its full-attention layers. Replacing the language-modeling head with a small head that produces...