Parsing the world's hardest PDFs @llama_index. cofounder/CEO Careers: https://t.co/EUnMNmbCtx Enterprise: https://t.co/Ht5jwxSrQB
RT Dave Zilberman 🔥🔥🔥🔥🔥
ParseBench-2 hit 30k monthly downloads in 3 days, and is #3 on @huggingface trending benchmarks! 📈 In ParseBench-2, we tried hard to create a comprehensive taxonomy of metrics to evaluate document processing, while creating a severity-based framework to align these metrics with
RT Sebas Beautiful to see ParseBench-2 getting love from the community at this pace
ParseBench-2 hit 30k monthly downloads in 3 days, and is #3 on @huggingface trending benchmarks! 📈 In ParseBench-2, we tried hard to create a comprehensive taxonomy of metrics to evaluate document processing, while creating a severity-based framework to align these metrics with
ParseBench-2 hit 30k monthly downloads in 3 days, and is #3 on @huggingface trending benchmarks! 📈 In ParseBench-2, we tried hard to create a comprehensive taxonomy of metrics to evaluate document processing, while creating a severity-based framework to align these metrics with human judgment. Whether you’re deeply familiar with document OCR and document OCR benchmarks, or just getting started, we encourage you to check out the ParseBench-2 methodology. We’re happy to help you pick whatever subset of metrics you care about if you’re looking to process documents in production. We’ve merged dozens of PRs since the release date, including fixes/metric updates from some of our peers. We’re aggressively fixing any lingering annotation and metric issues. Report: https://www.llamaindex.ai/files/parsebench-v2-severity-aware-evaluation-of-document-parsing.pdf Hugging Face: https://huggingface.co/datasets/llamaindex/ParseBench-2 Github: https://github.com/run-llama/ParseBench/
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Vashista Re Thank you! I take my words back. As an avid consumer of OCR models for my work and startup. Im always out looking for the best I can use and at the start relied on benchmarks to ship quick, as I spent more time learnt a lot and now dont need to rely on the public benches, but theres a lot of users out there who will rely and gotta keep them accurate and honest, thanks for taking a look at them Logan, got a lot more. More than happy to contribute so we can rely on a singular bench. I do beileve rule based systems are needed aswell as LLM as judges too, and parse bench 2 and its methodology covers the first one on a wide range.
RT Aria Spot on by @jerryjliu0 on encoding explicit rubrics for evals! Stanford CS329A · Study Notes 10 (Constitutional AI & Self-Critique) • Naive Self-Critique fails without rubrics • Constitutional AI loops Critique, Revision & RLAIF • Calibrate LLM-as-a-Judge biases
You shouldn’t blindly trust benchmarks - you should be able to encode your preferences relative to your use cases and design eval metrics around that. We’re excited to release ParseBench-2 as not only a comprehensive document parsing benchmark, but also a framework for parsing
View quoted postOne of the hardest parts around document parsing is parsing complex tables. Besides the task itself being difficult, it is also hard to measure what ‘good’ looks like. Ultimately it comes down to intuition around what is a catastrophic error for downstream knowledge work. * Formatting differences (markdown vs. html tags) are harmless * Making the header row a “data” row is also probably harmless (any human/model can interpret it as the header anyways) * Swapping the signs on a number, particular in financial reporting, is much more severe. It is hard to encode what “good” looks like in a quantifiable way. Existing deterministic tests/metrics (OmniDocBench/OlmOCR-Bench/ParseBench-1) use some variant of edit distance, or an average over binary tests, making it hard to distinguish severe errors from normal ones. With our recent release of ParseBench-2, our answer is to tune any deterministic metric by a severity policy, where you can explicitly state how good or bad different errors are. * For instance, in a finance setting, dropping digits/flipping signs is really bad! Our default severity weighting is 10, but you might decide 25 - the policy is meant to be customizable by the user. * Check out the examples below highlighting this. You can compare the impact of “actually bad” or “harmless” perturbations on a few different benchmarks incl. ParseBench-2. We have a taxonomy of 330+ error types, a big subset of which is defined around table errors, categorizing them from 5 severity levels: * most severe: wrong digits, decimals misplaced, values swapped, column missing * medium severe: body rows reordered, one cell split into two, etc. * harmless: formatting errors, html markup More details around table methodology in blog/tech report. Blog: https://www.llamaindex.ai/blog/introducing-parsebench-v2 Tech report: https://www.llamaindex.ai/files/parsebench-v2-severity-aware-evaluation-of-document-parsing.pdf
RT simon Re I think this rule-based vs. model-based eval debate is missing the point we are trying to make (maybe this is valid feedback that we need to write our paper better😅) ultimately, we are trying to say that severity is human preference, and human preference can be quite granular and nuanced. we need evals that reflect that reality. we love rule-based, deterministic evals AND LLM-as-judge evals. with LLM-as-judge, what's in the rubric (i.e. prompt) becomes really important, to properly encode your preference. in fact, we tried to do exactly that, writing down our preference in https://www.parseprotocol.com/ that can be used as rubric for LLM-as-judge. There's no free lunch, you gotta state your preference somewhere. In ParseBench-2. We chose to state the preference with what we call severity policies and optimize / generate deterministic rules to implement those. There's definitely mistakes, as optimization is imperfect, and we have finite resources. But we are making a genuine effort here, taking in feedback, and improving in the public quickly.
There are thousands of OCR tools out there, and ParseBench-2 represents our attempt to not only build a standardized benchmark, but a framework for understanding which tools do better on which dimensions. On day 1, we’ve benchmarked 40 different tools across 6 different dimensions: * specialized OCR tools * frontier models * open-weight solutions We’ve aggressively tried to improve the metrics from ParseBench-1 to reflect real penalties while remaining quantitative. *** For what it’s worth, I would take the overall score with a grain of salt, since it’s a simple composite average. A lot of the insights/nuances in the benchmark come from analyzing each individual dimension and understanding the tagging behind each failure case. It’s fascinating to see where models and other vendors do well - we don’t top the charts in every dimension relative to price point! ✅ frontier models are hillclimbing certain dimensions while ignoring others. e.g. opus 5.5 is quite good at tables, while bad at charts ✅ specialized OCR providers do well on various dimensions, e.g. Extend on forms, Datalab on tables, (we also fixed reducto r-1 chart evals) There has been some feedback on annotations and metrics that we’re continuing to take into account, and will create new versions soon. Also reminder that if you care about certain errors more than others, you're welcome to customize the severities for the use cases you care about. We’re going to be adding a ton more models to the leaderboard, come check it out: https://www.parsebench.ai/ Hugging Face: https://huggingface.co/datasets/llamaindex/ParseBench-2
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT LlamaIndex 🦙 Stop paying 45 credits a page for your hardest PDFs. Agentic Plus is now 25 credits per page. It's the same API call with nothing to migrate. Our new OCRia-1.5 family of models and a tighter agent harness means every page costs us less to process, and we’re passing that straight through. The results: → 44% cheaper → 1.8x the pages on the same credits Your hardest PDFs are still hard (think dense tables, unlabeled charts, crooked scans). But they just got a lot cheaper to get right :)
RT Murtaza Khomusi Re @jerryjliu0 i do philosophically believe that most large orgs want custom intelligence, similar to how how most orgs want fast, accurate and affordable parsing that's well tuned to their docs. evals can get infinitely fine-grained but a framework lets you hillclimb the last mile.
RT Justin Xu LLM-as-a-judge for a discriminative task is such a weird hill to die on
We've released ParseJudge as a way for you to easily do this. You can use a blind LLM judge to decipher between the results, and see for yourself on your own dataset. Reducto doesn't win every result, nor should we. But we believe that the fairest methodology involves your
View quoted postRT Imaan Sultan 🇵🇰🇸🇦 we have always been classy with how we respond which is a testament to how we operate! “it’s an OSS benchmark. submit a PR or do the hard work and make something better” 😌 im happy to see our work lives in minds rent free. we’re doing something right :)
let's be real. it might be cool to shitpost perceived competitors on x or something but why is this even a debate? LLM-as-a-Judge has a place, but you can't argue that carefully curated datasets, annotated pages, and expert-built rules are dead. this is intellectually
View quoted postRT Wey Gu 古思为 🫡
Today, @llama_index is an applied research lab for document OCR and document work. Our core focus is investing in building the best-in-class models+ agents at the pareto frontier of accuracy/cost/latency for document processing. To do this, we have to invest in all the
RT UiPath We see complex documents everywhere, as we work closely with customers to automate business processes. Industry benchmarks like ParseBench-2 from @llama_index are critical for innovation in enterprise-grade parsing and extraction. Check it out:
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Imaan Sultan 🇵🇰🇸🇦 if ur wondering what it's like to work @llama_index btw 😌
rain or snow, we keep shipping
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Imaan Sultan 🇵🇰🇸🇦 if jensen huang’s wife divorces him ill parse the divorce papers live 😭😭😭
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Tim Baer If you're evaluating document AI, ask which errors would actually break your workflow. A missing decimal in a financial table can matter more than a page of formatting differences. That's what I like about our ParseBench-2 release: taking error severity seriously.
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Taqi Jaffri The industry needs benchmarks with real world examples of the most difficult documents (which happen to be the most valuable for the future of work). Great to see ParseBench leveling up! @llama_index @jerryjliu0 @imaanxsultan
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Justin Xu extract evals team getting mogged by parse evals team today gotta start planning ExtractBench Ultra Max++
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Javier Torres ocria-1.5 now runs every LlamaParse mode. the bounding boxes on messy layouts are the part i'd try first
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT berni me: leaves half the form blank on purpose the parser: fills in the blanks, unchecks my checkbox and moves my phone number into the address field glad ParseBench-2 finally made forms its own category
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Niel Pal Proud of the team for shipping ParseBench-2 🚀 4.3k+ pages, a new forms dimension, and severity scores across 330+ error types so the metrics actually match human judgment. Plus OCRia-1.5 now powers every mode in LlamaParse. Go check it out 👇
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Pierre-Loic Doulcet Re As we were developing the metrics, we start by looking how other metric in other benchmark work, and how good they are at spotting designed error type (such as a permutation of 2 digit in a number), and how much they penalize it. We did not find metric that penalized as we expect, so we build our owns. You can explore how different parse benchmark measure/penalize different scenarios here: https://www.parsebench.ai/compare/index.html
RT fred miles de empresas metiendo IA para leer contratos, facturas y nóminas sin saber que puede cambiarte un número sin que nadie se dé cuenta
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Ryan Lee Congrats to everyone on getting this released!
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT matt NO TODOS LOS ERRORES DE UNA IA SON IGUAL DE GRAVES. ParseBench-2 lo mide así: → 330+ tipos de error → Revisados por expertos → Crítico, grave, leve, inofensivo → Sin un LLM juzgando → Mismo resultado siempre Y lo mejor: es todo abierto. → Código de evaluación → Dataset y paper → Cambias tú las severidades Así deberían construirse los benchmarks 👇🏼
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT ares. 🎧 LA MAYORÍA DE IAS NO SABEN LEER UN GRÁFICO. ParseBench-2 ha revisado cada número de 1.025 gráficos, uno por uno. El mejor parser saca un 92. Cinco sacan menos de 4. Lo que pasa en la práctica: → Leen el título → Se saltan los datos → Te responden igual → Con total seguridad Si alguna vez un agente te ha dado un dato raro de un PDF lleno de gráficos, ya sabes por qué 👇🏼
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT natt Most parsing benchmarks barely notice a wrong number, but punish a harmless format change. On ParseBench-2, one wrong number on an otherwise perfect page now costs ten points, while Markdown vs HTML costs nothing. That's how mistakes should be scored.
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Angela Tse Big launch day at LlamaIndex! 🚀 We're introducing ParseBench-2, our most comprehensive document parsing benchmark yet, alongside OCRia-1.5, our newest family of document parsing models. Really exciting to see the team continue pushing the boundaries of what's possible! 🙌🏻
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT zen. The CEO who fired half his team because AI could already read the PDFs, finding out it can't even point to which part of the page each number came from
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT MJ 🦙 Me: It’s better. Them: How do you know? Me: Several researchers have been expecting this question. ParseBench 2 is out. A ridiculous amount of work behind those three words. Major props to our team!
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Pierre-Loic Doulcet Parsebench-2 is the best benchmark out there I think! It is open (you can run it), adaptable (you can change the severity policies to match your case), and overall quite comprehensive. Check it out!
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Sebas We've heard you, ParseBench v1 wasn't perfect, but we have been locked in for the last weeks cooking the fairest, highest quality document parsing benchmark in the world. Check it out!!!
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT simon A really cool part of this work is how we approach automated metric design. Instead of crafting them by hand, we fully acknowledge coding agents are now often better at doing that. What's important for us human to do, is to state our preference and have a scalable method to encode that. This is how we made ParseBench-2 much more granular, robust, and aligned.
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Boyang Zhang Excited to see ParseBench-2 officially launched 🎉 A major step forward in making document parsing benchmarks more granular, reliable, and aligned with human judgment. Proud to have contributed to this work!
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Zhaoqi Li LlamaIndex started as an open-source framework. ParseBench-2 is a good snapshot of where our heads are now: obsessing over whether a document actually got read correctly
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Taibiao Zhao ParseBench-2 from LlamaIndex! Let's go!!!
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Imaan Sultan 🇵🇰🇸🇦 this entire launch video was edited with AI no agency. just me, 2 hours of raw footage, and my secret sauce i’ve been able to cut my editing time by 40%, which meant more time to focus on the execution of this launch (which as a marketing person is the absolute best) if enough people want it I'll make a video showing exactly how I did it :)
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Logan Markewich A second benchmark has arrived: ParseBench-2 is here! it's live, it's open, go run your parser on it! https://huggingface.co/datasets/llamaindex/ParseBench-2
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postRT Murtaza Khomusi i lead marketing at a company that just shipped a benchmark, so you should be skeptical of this post which is why everything is open: dataset, eval code, paper, leaderboard think we rigged it? you can change the weights in the Metric Lab and see for yourself
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
View quoted postToday we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with human judgment. We do this by painstakingly assigning severity scores to 330+ error types, and tuning each metric to match the severity scores. Existing benchmarks either use brittle deterministic metrics (e.g. OmniDocBench) or LLM-as-a-judge (ParseArena), which can help recognize equivalent representations but are coarse/stochastic/subject to bias by the prompt+model. ParseBench-2 also contains: ✅ 4.3k+ document pages (double from ParseBench-1). ✅ An extra dimension (forms) ✅ Much better ground-truth across all dimensions compared to ParseBench-1. This is the next evolution of ParseBench, our hugely popular benchmark that has 50k+ monthly downloads, over 150+ OCR solutions benchmarked, and has been adopted by companies like NVIDIA, Cohere, Aleph Alpha, Perplexity, and others. Alongside ParseBench-2, we are also releasing OCRia-1.5 - a new family of document parsing models that powers all our modes in LlamaParse. They represent a massive step up in capabilities from our latest release, and are state of the art in parsing complex tables, forms, charts, formatting, and assigning bounding boxes to complex layouts. Come check out our blog post, technical report, and website. Blog: https://www.llamaindex.ai/blog/introducing-parsebench-v2?utm_source=socials_jerry_liu&utm_medium=li_social&utm_campaign=launch&utm_content=blog_post Technical Report: https://www.llamaindex.ai/files/parsebench-v2-severity-aware-evaluation-of-document-parsing.pdf?utm_source=socials_jerry_liu&utm_medium=tw_social&utm_campaign=launch&utm_content=technical_report Website: https://www.parsebench.ai/ If you’re interested in trying out our OCRia-1.5 capabilities, come sign up for LlamaParse: https://cloud.llamaindex.ai/
RT Dave Zilberman In other news… Norwest 18 is here, it’s $4B and the largest fund in @NorwestVP’s history. The new fund gives us even more opportunities to back founders building for the emerging generation of enterprise AI, data, and infrastructure.
Today, @llama_index is an applied research lab for document OCR and document work. Our core focus is investing in building the best-in-class models+ agents at the pareto frontier of accuracy/cost/latency for document processing. To do this, we have to invest in all the different components around the research infra stack: 1. Building comprehensive and fair benchmarks. Today we have ParseBench, ExtractBench, and ExtractBench Pro. They're already the most popular benchmarks in parsing and extraction today, and we're just getting started. 2. Investing in data curation and synthetic data generation. 3. Investing in compute and our training pipeline. This has produced models like OCRia-1. 4. Investing in agent optimization - tune everything around the models to further hillclimb cost and accuracy. 5. Closing the loop based on feedback - whether from active usage of our product, implicit/explicit customer feedback, or automated verification. We are accelerating our capabilities at a crazy pace. Today, our work is documented in our research page: https://www.llamaindex.ai/research There is a lot more to come!
RT Imaan Sultan 🇵🇰🇸🇦 ANOTHA ONE 🔑
Nvidia CEO Jensen Huang had another affair with an executive at the company, according to a report from The New York Times. This is his second known affair, following his relationship with an alleged Chinese spy. (https://www.nytimes.com/2026/11/11/technology/jensen-huang-nvidia-affair.html)
RT Daniel Lenton OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience. On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.
RT LlamaIndex 🦙 The hardest document in life insurance isn’t the application. It’s the medical records packet that comes with it. Hundreds of pages of faxed APSs, lab reports and clinic notes from different providers, in no particular order. Underwriters need a handful of facts from it. If you miss one, the policy gets priced on an incomplete picture. On Wednesday, Nov 18 at 9 AM PT, we’ll build the whole workflow live with LlamaParse and show you how to set up your own. It’s free, and you’ll leave with something you can run on your next packet: 1/ Splitting the packet into individual documents by provider and date 2/ Parsing scanned and faxed pages, from handwritten notes to lab tables 3/ Extracting diagnoses, medications and lab values with page citations 4/ Indexing the full record so you can ask follow-up questions Instead of digging through hundreds of pages or rereading the entire file, you only check the cited page and move on. RSVP here: https://watch.getcontrast.io/register/llamaindex-building-a-life-new-business-underwriting-workflow-with-llamaparse?utm_source=twitter&utm_medium=organic
RT Imaan Sultan 🇵🇰🇸🇦 i don’t think you understand what is happening at aws re:invent on dec 1 six ai companies rented an ENTIRE karaoke bar in vegas and hired a live band @llama_index, @turbopuffer, @modal, @cartesia, @p0 - main stage if you’re brave - private booths if you’re not - martinis all night - a custom record you take home - a secret backstage speakeasy, and the only way in is finding one of us on the expo floor rsvp is approval based :) in comments below
RT Imaan Sultan 🇵🇰🇸🇦 can you guys tell my first B in school was in art Imao
Last Thursday we hosted a private dinner for AI leaders in insurance at Niku Steakhouse. The discussion was so good we forgot to take a photo :) We created a photorealistic rendering of what the conversation looked like below. We live in a world where frontier models are
Last Thursday we hosted a private dinner for AI leaders in insurance at Niku Steakhouse. The discussion was so good we forgot to take a photo :) We created a photorealistic rendering of what the conversation looked like below. We live in a world where frontier models are exponentially improving every month. This raises many interesting questions around the opportunity for agentic automation, both within the external products that folks are building (e.g. automating claims, underwriting), and within internal organization and engineering processes. We opened with two questions: 1. How much of your work is AI actually automating today? 2. In an industry this regulated, where does a human stay in the loop? Human in the loop will always be needed for verification and legal purposes in highly regulated environments. But it is fascinating to see that most builders have evolved from building small deterministic workflows into agents that can deliver e2e outcomes. They're also able to build product with a team that's leaner than ever and still multiples more productive than the same time last year. Big thanks to @imaanxsultan for organizing! This is the first dinner for Marginalia, a private dinner series that gathers AI leaders one industry at a time. Thanks to everyone for a great discussion, and we’re hoping to host the next one soon!
RT LlamaIndex 🦙 LlamaParse can run anywhere: SaaS, in your cloud, or on your hardware. Which one fits usually comes down to one question: where are your documents allowed to live and who manages infra? 1/ Our managed API, where we run everything 2/ The full platform in your own cloud account or data center 3/ The platform and storage in your environment, with us running the models 4/ Your existing inference provider, connected to the platform 5/ Your own GPUs, running supported models yourself If you use our inference, we don’t retain your data or train on it. The blog walks through what runs where, what your team owns in each setup, and how to pick from our deployment models: https://www.llamaindex.ai/blog/how-to-flexibly-deploy-llamaparse-wherever-your-data-lives?utm_source=socials&utm_medium=li_channel&utm_campaign=flexible_deploy_llamaparse
RT OpenBMB Great perspective! 💙 Speed, accuracy, and cost all matter for real-world document processing. Excited to see MiniCPM-OCR integrated into LiteParse. Thanks for bringing it into the ecosystem! 🚀
The world needs two types of document OCR: 1. Extremely fast/cheap OCR tools: Each organization might have trillions of documents, and this is oftentimes helpful as a 'first pass' to digitalize each document so that it's accessible to AI agents. Accuracy might not be the
There's a pareto frontier of accuracy and cost for every meaningful task out in the world. Even as frontier models advance raw intelligence, everyone has the opportunity to define and advance the frontier for their own tasks. https://x.com/i/article/2118977493613092864
The world needs two types of document OCR: 1. Extremely fast/cheap OCR tools: Each organization might have trillions of documents, and this is oftentimes helpful as a 'first pass' to digitalize each document so that it's accessible to AI agents. Accuracy might not be the highest, but it's 'good enough' for agentic retrieval and a deeper second pass. This includes free/OSS tools like pypdf, pymupdf. Over the past year, we've invested heavy efforts in this area through `liteparse` - the fastest/most accurate free/OSS parser. It integrates with 5 other open-weight OCR tools, most recently with MiniCPM-OCR from @OpenBMB https://github.com/run-llama/liteparse We're also doing a lot more in this space - stay tuned! 2. Accurate VLM-based OCR: This includes the subset of documents that are a) complex, and b) the organization must parse accurately. The data needs to be accurate because this data is used in business critical workflows across finance, insurance, healthcare, and more. At the same time, even if the tool is not free, any VLM needs to be able to scale to billions of pages in a cost-effective manner. Two weeks ago we launched OCRia-1, the best general purpose model for document parsing. It not only parsers complex text layouts, complex tables, but also visual diagrams and charts. https://www.llamaindex.ai/blog/parse-ocria-1 It powers all our modes within LlamaParse; see the most recent benchmarks on ParseBench: https://www.parsebench.ai/ ------- Both of these document OCR types require rigorous benchmarks, data+training pipelines, and an ambitious/talent-dense team pushing the pareto frontier of accuracy and cost. If you're looking to solve document understanding across either frontier, come talk to us!
RT Imaan Sultan 🇵🇰🇸🇦 ok so this blew up and this is me looking at the rsvp list rn:
we booked out an entire spa in SF for women in growth tuesday nov 17, @llama_index is co-hosting with girls in growth pick one, on us: manicure gua sha facial 30 min massage the spa cannot massage all of SF at once, so spots are limited (and already almost full) DM me 💆♀
RT Imaan Sultan 🇵🇰🇸🇦 the two best times to drink chai ☕️: 1/ diwali weekend 2/ when the sleeve has $500 in LlamaParse credits on it this weekend you get both unindexed ep 4 is a bts of our doc chai launch party at @UseCorgi. we were trending in the bay on both luma and partiful, with 1,000+ RSVPs (!!) i knew our drink had to be my own take on real chai. it’s a simple drink, but it’s such an incredible symbol of bringing people together i designed the sleeve after the bright, maximalist truck art i’d see every time i visited karachi, pakistan 🇵🇰 doc chai is at corgi cafe in fidi until the end of the month. let me know if you try it :)
RT simon Does the model shape the hardware or does the hardware shape the model
Does the land shape the river or does the river shape the land? Does the data shape the network or does the network shape the data?
View quoted postRT simon We will know Opus 4.5 as the start of a new age
1. Benchmarks provide one piece of evidence for acceleration. Since Oct 2025, AECI scores for Anthropic’s models have grown at 1.9x [95%: 1.5–2.8] the earlier rate. Analysis choices affect this estimate, and it remains unclear how benchmark scores map to real-world capabilities.
RT simon well guess this is my acting debut
YOU CAN PARSE EVERYTHING 11.17.26 ocria-1 is already the cheapest, most accurate unified model for doc parsing (not only tables/text, but also charts/diagrams/images/scans) it's only getting better. stay tuned
View quoted postRT Imaan Sultan 🇵🇰🇸🇦 y̵o̵u̵ ̵c̵a̵n̵ ̵s̵e̵e̵ ̵e̵v̵e̵r̵y̵t̵h̵i̵n̵g̵ you can parse everything ✅
YOU CAN PARSE EVERYTHING 11.17.26 ocria-1 is already the cheapest, most accurate unified model for doc parsing (not only tables/text, but also charts/diagrams/images/scans) it's only getting better. stay tuned
View quoted postYOU CAN PARSE EVERYTHING 11.17.26 ocria-1 is already the cheapest, most accurate unified model for doc parsing (not only tables/text, but also charts/diagrams/images/scans) it's only getting better. stay tuned
Today we’re releasing OCRia-1, a frontier VLM for document parsing that sets a new SOTA in accuracy, cost, and latency. It not only sets a new high score on ParseBench (87.4), but also leads in three 3rd-party benchmarks: olmOCR-bench* (83.5 overall, 92.9 on tables!),
View quoted postRT Imaan Sultan 🇵🇰🇸🇦 i was fully going to be a lawyer then i tried growth at a startup and never looked back now i’m on a16z's @cosign awards list for marketing, on the same page as the CMO of @harvey and the head of content at @WisprFlow . surreal is the only word i have i’ve been at @llama_index for just two (!!) months. in that time we’ve: - put ads on SF buses - launched our own drink at corgi cafe - hand-delivered arsicault croissants to early founders - hosted private suites at austin f1 and the us open for customers and so many more co-activations than i can count. i also somehow picked up 1,800 more followers on here, which still feels like a typo if anything i've made has landed for you, you can support me here 💌: http://cosign.co/awards/marketing
RT LlamaIndex 🦙 Going to AWS re:Invent and looking for the best afterparty of the week? We're partnering with our friends at @Braintrust, @Cartesia, @p0, @turbopuffer, and @Modal to host the Agent Soundcheck. Join us for drinks, food, and karaoke at On the Record at the Park MGM. Tue, Dec 1, 6–10 PM. Register here → https://luma.com/agent-soundcheck?utm_source=llamaindex
ayo
Some news: I’m joining OpenAI to lead product management for Core Products! I have a lot of respect for OpenAI for showing the world that scaling works, for bringing AI to over a billion people, and for believing great products are core to the mission. I’m excited to work with
View quoted postOCRia-1 is one of the only VLM OCR models that can uniformly parse any element (tables, charts, images, equations, scans, forms) with a consistently high degree of accuracy. We’ve released a 37 page technical report on ArXiv. It contains comprehensive information about our data and training pipelines and evaluation results. There are a ton of details in the appendix! If you’re looking to parse a massive volume of docs in production with consistently high accuracy, come check it out: https://arxiv.org/pdf/2611.00092v1