I’ve been working on extracting data from documents, and I've shared a bunch to of stuff on the topic Hosted talks on OCR and VLMs to get tables, text, images, etc. out of documents Mini examples of which models and pipelines work in specific cases Retrieval on PDF documents to grab specific source data Antaripa Saha worked on #3 as part of her work in the AI Product Engineering Club. She compared keyword search, dense OCR, visual embeddings, multivectors, and reranking on complex financial...
about 1 month ago • 9 min read
Hey, I often see people struggling because users have different preferences and it's had to get what exactly they want out of them. This leads the agent (and your product) to give an answer that isn't what the user wants. I gave a talk and did a writeup on query disambiguation in the community. It covers patterns from AnkiHub, AI2 Paper Finder, NotebookLM, Deep Research, Censys, Lovable, Spiral, and Codex. Read or watch the post on the web. Isaac Query Disambiguation Read or Watch the talk on...
about 1 month ago • 6 min read
Hey, How to get data out of documents is the most common thing I get asked about privately, because if the source data extraction is wrong everything in the product will be wrong downstream. There's so many edge cases related to the reading order, tables, images, images with text inside them, scans, handwriting that is a picture embedded in a scanned document, etc. I hosted Joe Barrow for a deep dive on VLMs (vision language models) for OCR. He has processed hundreds of millions of document...
about 1 month ago • 19 min read
Hey, I built a tool to help me figure out what's valuable to build and talk about. It collects questions people ask, clusters repeated problems, runs a research subagent, judges the results, then puts it in a UI for me to annotate and analyze, analyze. It inspires videos, lessons, skills, apps, libraries, product features, or experiments. Here's a bit about it, and a video walkthrough that goes into more detail: The pipeline currently has ~ 6K questions from Maven lessons, GitHub issues, X...
about 2 months ago • 2 min read
Hey this is Isaac, Hamel Husain and I are doing a free public talk on why better RAG starts with late interaction. Most RAG still follows the "embed the chunks, embed the query, compare the vectors” approach. But top retrieval architectures follow a new approach. People do not upgrade because it requires a different setup and they think it’s only an incremental improvement. Companies like Nvidia, AWS, and Cursor have adopted it because it’s a big step up. It’s not just benchmark games. A...
3 months ago • 1 min read
Hey this is Isaac, Hamel Husain and I are doing a free public talk on tool architecture anti-patterns for agents: MCP, skills, CLIs, plugins, and custom tools. Some of the most popular and widely used skills, CLIs, and mcp’s fall into one of these anti-patterns. Is MCP good? Are skills enough? Should this be a CLI? That is the wrong starting point. A tool call is a product surface and should be created with the same care given to any other user facing page. When it’s treated like plumbing to...
4 months ago • 1 min read
Today's post is about a project that I will take concepts from to improve my own work. So I studied the OSS prototype of it to understand how it works. Point and Talk: How Clicky’s AI Interface Works Clicky is an AI buddy made by Farza that lives on your Mac. You press a key, talk, and a glowing blue triangle flies across your screen, points at whatever you asked about, and talks you through the answer. Most apps with an agent should have something like this. Point and talk is closer to how...
4 months ago • 6 min read
Last week I shared my knowledge base tool, agentkb. This week I am OSSing my pi harness that uses it, Harpy. The most interesting part of the harness is that knowledge base <-> harness integration so that's what this week post dives into. It covers what RLM is, how it drives a python loop with state, how Pi (typscript) extension talks to python, and how that allows for smart delegation and fan-out to smaller models to give more accurate search results. How My RLM Tool Works An LLM writes...
5 months ago • 9 min read
Hey, This was a week of context switching 😅. Product release weeks are always odd. The goal is super specific, and yet the work is scattered small tasks (testing, polish, docs, marketing copy, etc.). We shipped the stoa beta, and I kept wishing my agents could remember more. That's why I'm open sourcing `agentkb`, my work-in-progress knowledge base framework today. The Pi harness built around it will be open sourced next. AgentKB: A Local Knowledge Base for Agents My agent should learn from...
5 months ago • 6 min read
