Every post comes from something I've done on a real project. AI tools, development approaches, how I actually build things. You're getting a curation of my taste, not takes on stuff I don't use. Subscribers also get extras: things that went wrong, how my thinking about AI is changing, hacky workflows I use every day, and the occasional personal update. Stuff I share with subscribers because it's a little too personal or unpolished to blast across the internet.
building demand signal pipeline (scrape, judge, research subagent, UI)
|
Hey, I built a tool to help me figure out what's valuable to build and talk about. It collects questions people ask, clusters repeated problems, runs a research subagent, judges the results, then puts it in a UI for me to annotate and analyze, analyze. It inspires videos, lessons, skills, apps, libraries, product features, or experiments. Here's a bit about it, and a video walkthrough that goes into more detail: The pipeline currently has ~ 6K questions from Maven lessons, GitHub issues, X discussions, and YouTube. This guides me to work on things people value, rather than what I imagine people value (often very different things). Questions people ask in response to content is often much more valuable source than the original content. I want to know what people ask after reading an X article, not just parrot whatever an article says. It starts by scraping the sources, and they're all different. For example, for Maven I need to download and transcribe the videos, and look for Q&A sessions at the end of videos. Then, to get context I need timestamps to be able to watch the person asking the question quickly. Youtube, X, and Github all have their own processes for how people interact. Once I have all the questions in sql, I cluster them (mixedbread embeddings -> cosine clustering) to find groups of questions. Then, I score them. I manually annotated a ton as good, maybe, and bad to create a good LLM as a judge to do this for me. It started with about 30% of questions being good ones for me to work on, and now it's up to about 65%. The questions that pass review go to a Signal Librarian (a subagent, inspired by Amp's Librarian subagent). It writes scripts (like claude workflow, or RLM style). I give it function for SQL (for counts and date searches), semantic search (for related questions), parallel cheap model calls with gemini flash (for comparison and deduplication), and use Codex for final output and orchestration. The final output gives repeated questions, links, disagreements, and possible outputs. One recent cluster contained questions about extracting tables from financial documents, tax forms, and datasheets. That cluster became a plan for ten short demonstrations. Four are published:
I posted a full video walkthrough here We reviewed and discussed the design of this in one of the live sessions AI Engineering Club, where there were several follow-up questions and discussions. The main follow up discussions were around:
We also started talking about agents that make 10–100 calls looking for context before answering and how to evaluate those. For example, an app does a ton of grep calls before starting to build the feature. Missing results in tool call #1 might be OK if tool call #10 finds it and bring it into context. So we considered evaluating the context-building phase as a whole and then tracing failures back to individual calls in the error analysis step. And the table examples led to a comparison of Markdown, HTML, bounding boxes, text extraction, and vision models. If this kind of discussion with other builders sounds interesting at all, check out the free trial of the AI Product Engineering Club. |
Isaac Flath
Every post comes from something I've done on a real project. AI tools, development approaches, how I actually build things. You're getting a curation of my taste, not takes on stuff I don't use. Subscribers also get extras: things that went wrong, how my thinking about AI is changing, hacky workflows I use every day, and the occasional personal update. Stuff I share with subscribers because it's a little too personal or unpolished to blast across the internet.
I’ve been working on extracting data from documents, and I've shared a bunch to of stuff on the topic Hosted talks on OCR and VLMs to get tables, text, images, etc. out of documents Mini examples of which models and pipelines work in specific cases Retrieval on PDF documents to grab specific source data Antaripa Saha worked on #3 as part of her work in the AI Product Engineering Club. She compared keyword search, dense OCR, visual embeddings, multivectors, and reranking on complex financial...
Hey, I often see people struggling because users have different preferences and it's had to get what exactly they want out of them. This leads the agent (and your product) to give an answer that isn't what the user wants. I gave a talk and did a writeup on query disambiguation in the community. It covers patterns from AnkiHub, AI2 Paper Finder, NotebookLM, Deep Research, Censys, Lovable, Spiral, and Codex. Read or watch the post on the web. Isaac Query Disambiguation Read or Watch the talk on...
Hey, How to get data out of documents is the most common thing I get asked about privately, because if the source data extraction is wrong everything in the product will be wrong downstream. There's so many edge cases related to the reading order, tables, images, images with text inside them, scans, handwriting that is a picture embedded in a scanned document, etc. I hosted Joe Barrow for a deep dive on VLMs (vision language models) for OCR. He has processed hundreds of millions of document...