Content History
RT Polymarket BREAKING: Jensen Huang reportedly discussing a return as Nvidia CEO with allies just days after the board pushed him out.
RT Vik Paruchuri Re Here's the full guide: how to check any benchmark in 5 minutes, and how to run an eval on your own documents - https://www.datalab.to/blog/how-to-evaluate-ocr .
RT Bryan Bischof fka Dr. Donut How it feels to be in @HamelHusain 's circle of friends:
RT Bryan Bischof fka Dr. Donut I took a stab at visualizing the transitions of weights this year in different categories:
The Artificial Analysis Intelligence Index is currently made up of 9 benchmarks with arbitrary weights. I wanted to know how these weights varied over time. They vary quite a bit!
Love to see error bars on benchmarks It’s very much a chart crime not to show them
Terminal-Bench 4.1 is here with updated verifiers for all tasks. In this release, we focused on: - resisting reward hacking - reducing false negatives - improving reproducibility Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping.
The Artificial Analysis Intelligence Index is currently made up of 9 benchmarks with arbitrary weights. I wanted to know how these weights varied over time. They vary quite a bit!
This comment illustrates the Evals-Industrial complex 1. [The belief that] If you can’t convince people to benchmaxx on your benchmark it is “worthless” 2. If people do infact benchmaxx on them they statistically become worthless cc: @xeophon
Anthropic apparently doesn't give a shit about benchmaxxing Harvey's legal benchmarks. Scores lower than Fable 5.1 and Sonnet, which leads me to believe that this benchmark is dead.
I see some are drawing the wrong inference from this study: they are focusing on this meta benchmark as yet another benchmark for model picking, i.e. open vs closed models, model families etc. That’s too narrow of an interpretation! You need to do your own private evals on your specific tasks to know what’s “better” This should give you healthy skepticism of all public benchmarks. I believe that was the intention of this illustration to begin with.
Nice DeepSWE score! Does it survive a lowercase filename? Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement. Models clustered around 74% on DeepSWE end up 23 pp apart. Good
Preach
jammed with @HamelHusain a bit and since then, i cannot unsee of the state of evals as being similar to the military-industrial complex
RT Florian Brand jammed with @HamelHusain a bit and since then, i cannot unsee of the state of evals as being similar to the military-industrial complex
The industrial-benchmark complex is on thin ice
Nice DeepSWE score! Does it survive a lowercase filename? Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement. Models clustered around 74% on DeepSWE end up 23 pp apart. Good
RT Justin Lo Nice DeepSWE score! Does it survive a lowercase filename? Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement. Models clustered around 74% on DeepSWE end up 23 pp apart. Good at coding, or good at DeepSWE? 🧵
RT rahul Today we're releasing DeepFinance, the first ultra-long horizon benchmark for high finance work. A set of 21 problems that reflect work at top PE and IB firms, DeepFinance is built using real deal data. Problems are constructed by experts with backgrounds from top firms like Evercore, JPM and UBS, and institutions like HBS and Wharton. Problems represent work that would take human professionals 20-40 hours to complete. Can AI really diligence a $100M+ deal? Finance benchmarks today are limited to evaluating agents on QA on financial docs or grading the numerical correctness of simple financial models. However, real world high finance work is full of assumptions and conflicting evidence. We built DeepFinance to realistically evaluate agents on those end-to-end finance workflows. DeepFinance leaderboard: http://juliusresearch.ai/deep-finance
Do your own private evals folks. Don’t rely on public benchmarks. 1. Many benchmarks have material flaws 2. Some models have already memorized the answer key.
Kimi K3.1 recalls instructions it was never given on Terminal Bench. They match the hidden tests. K3.1 while solving: "Instructions likely distilled pitfalls from previous agents." 🤯 When we edit a task to contradict them, K3.1 refuses the new instructions and fails. Other
RT Florian Brand i think about this a lot lately why should i trust any score of an eval run thats not done by me / not done openly? also, why compare two models if one is just blatantly [reward] hacking?
![RT Florian Brand
i think about this a lot lately
why should i trust any score of an eval run thats not done by me / not done openly? also, why compare two models if one is just blatantly [reward] ...](https://pbs.twimg.com/media/HWU83gXacAA3rcY?format=jpg&name=orig)
Public evaluations are foundational for assessing model capabilities. We looked at some popular evaluations and found issues, which we fixed. Sharing our findings:
Amazing BenchmarkGate [1] reveal by @xeophon where he found serious (even comical) problems in popular benchmarks: AutomationBench: "[forces the models to] describe their request bodies in prose rather than in JSON Schema" 😅 He also fixed a number issues with the grader, which when combined with the above result in a near saturation of the benchmark! OfficeQA Pro v2: the grader uses a regex and Python-based extraction, which results in rejecting many valid answers. Additionally, there is evidence of models using web access to retrieve the answer key from HuggingFace (receipts are in the post). With these and other fixes, this again results in near saturation of the benchmark! It's very impt to look at data and not trust metrics from third parties. Better yet, you should rely on your own private evals. [1] BenchmarkGate: the gap between the [un]trustworthiness of benchmarks and the extent to which people rely on them.
Public evaluations are foundational for assessing model capabilities. We looked at some popular evaluations and found issues, which we fixed. Sharing our findings:
this post is such a banger. re: SWE-bench Multimodal "Removing the images barely changes the score ... Most tasks can be solved from the issue text and the code alone"
curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now. https://sankalp.bearblog.dev/swe_multimodal/
RT Radek Sienkiewicz Just got this and starting it today (by @sh_reya and @HamelHusain). I’m preparing to be even more unstoppable now.
RT Shreya Shankar The evals book is here. Hamel is the evals 🐐 and I’m lucky he wanted to write this book with me!! We are grateful for our reviewers, editors, illustrators and readers. Order it on Amazon; or come to our book signing to get a copy (link in replies).
I read the error analysis chapter of Evals for AI Engineers by @HamelHusain and @sh_reya and realised I already had the dataset. Some backstory. I work with coding agents all day: Claude Code, Codex, Hermes, Antigravity. Every session is a trace. What I typed, every tool call,
I've seen a lot of pushback to this, with the following complaints: 1. "The benchmark owners already know about the flaws, and are working on it." 2. "Why are they spending time critiquing and not contributing to the benchmark" OR "critiquing known issues are not helpful". 3. "The flawed verdict is too binary" I find this pushback to be misguided. These benchmarks are used to move millions of dollars of token spend, and many people blindly trust how well these benchmarks generalize to their own products, and don't look any further than the headline numbers. For example, whenever a new model drops, you'll find thousands of youtube videos analyzing the pareto frontier using one of these benchmarks as the primary quality metric without questioning the benchmark. I think we would all be better served by inviting skepticism and encouraging people to develop an independent opinion of how well these benchmarks represent the tasks they care about. Concretely: - For the failing tasks within the benchmark, are they: realistic, have a proper harness, graded properly, sufficiently disambiguated user intent? - Are models and API calls configured properly (ex: inconsistent context windows, reasoning levels). - Are the models equipped with a modern harness and minimal retrieval that is sufficiently realistic? - Does the score even make sense ex: is it just a LLM judge committee that produces a score on a 0-100 scale that isn't aligned with ground truth? It's often the case you'll see many of these issues while looking at traces of a failing benchmark tasks, and it doesn't take long! Often the **very first trace you look at** has a serious issue. I also don't think there is an issue with verified vs. flawed verdict. It's good to have opinions. If you look through enough of what's behind their opinions and you find that you usually agree with their verdict, then they have earned your trust, the same way that a benchmark can lose your trust. I think this eval of benchm...
Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.
I've been looking into public benchmarks and am astounded by the poor quality vs. how much we rely on them as an industry. @xeophon pointed me to this excellent site https://epoch.ai/benchmarks/ which is an independent third party which scrutinizes benchmarks and "looks at the data" 💪 For example, here are some popular benchmarks we all rely on. Before trusting a benchmark its worth running them and inspecting examples the grader failed carefully, to get a sense of the quality. I'm continuing to look more into this, and will write something up eventually once I've collected more data, but I think this is an important subject.
RT Hiten Shah We launched Beam on Monday and hit #1 on Product Hunt. The most requested feature was Intel Mac support. We shipped it three days later. Beam now runs on any Intel Mac with macOS 14 or later, in both directions. Beam its windows or whole desktop to another Mac, or sit at the Intel Mac and open windows from your other Macs. That old Intel Mac gathering dust has a job again. This is a 2018 Intel Mac mini running Beam in Whole Mac mode. https://beam.win
In June, @SamAsante and I launched Typeahead, the first product from our new company. Today we're releasing our next Mac app. It's called Beam, and we're live on Product Hunt. I've got Macs all over my house doing different work. Some run agents and local models. Others handle
View quoted post
RT Riley Goodside Prompt engineering is dead. Prompt however you want. Have fun.
. @sh_reya and I are launching our new book Evals for AI Engineers Nov 12th at @DanOlsen’s @LeanProdMeetup! You can join IRL in Mountain View or on Zoom. Our sponsor @raindrop_ai is providing free copies that we will sign for you if you come in person. Here's the sign up link https://events.ticketleap.com/tickets/dan-olsen/evals
BenchmarkGate is a real thing Don’t blindly trust benchmarks even popular ones We need to be doing evals on our evals
curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now. https://sankalp.bearblog.dev/swe_multimodal/
RT Antaripa Saha can we trust decision models for evals? @HamelHusain and I tested top decision models like Jev, d1-ultra, clef, pplx-decider, openai decisions, etc. on our dataset we explored failure detection, probability calibration, threshold optimization, and the practical tradeoffs of using them for evals. give it a read: https://hamel.dev/blog/posts/decision-model-evals/
RT sankalp curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now. https://sankalp.bearblog.dev/swe_multimodal/
RT Nick This is exactly why evals matter more than prompt or engineering skills alone....because instruction-following is measurable! Without the eval layer, “I wrote a good skill” can become mostly (sometimes, even dangerous) bs. Evals mean you can turn things like a skill's "hard prohibitions" into automated checks that make model output quantifiable and testable; pretty much every flaw listed here in @lex_node 's tweet would qualify. Code-based evals would catch a lot of these deterministically; LLM-as-judge would handle the more semantic ones. So what can you do to learn evals to fix these types of (literal) skill issues? Where do you start? Well, you can take @HamelHusain and @sh_reya 's excellent evals course! But, to give you some appetite for your deep dive, let's drill into some step by step evals basics for beginners:
this is my issue with AI x law. the last thing I want as a lawyer is anything non-deterministic in my documents lol. it doesn't obey the skills even if you wrote the skills to try to make it behave give me a rigid docgen template and an AI that can't do anything but input
BenchGate still exists and is very pervasive. Look at a handful of traces from failing tests in almost any public benchmark you will be shocked.
https://d.erenrich.net/are-you-smarter-than-an-llm/index.html Great way to try out MMLU and get a sense for just what, exactly, we are using to evaluate LLMs! I doubt folks would knife fight for a percent on this benchmark if its contents were realized more broadly.
RT Lenny Rachitsky Moats! Does anyone have one anymore? Yes. Yes, they do. But only a small subset of what people call moats create real long-term value and defensibility. Not actually a moat: 1. Distribution: This does matter more than ever. Building products has gotten easier, but getting them to customers has not. Distribution, however, does not create durable defensibility. It helps create the raw ingredients for one, but it is not itself a moat. 2. Execution: Yes, speed is critical for startups. But a company that has to sprint forever is not a good business. Ultimately, it will misstep, slow down, and competition will catch up. Speed gives a company more shots on goal as it tries to build a great product and a great business, but it must eventually find something more durable. 3. Brand: The idea is that when everyone is building the same thing, the most resonant brand will win. But for startups, brand works more like customer acquisition spend: Put money in, get growth out. It’s not something that will effectively keep others from competing long-term. Brand can become a true moat over time (as with Coke or Nike or Apple), but it just takes too long to really matter for most startups. What you're looking for is compounding advantages: a self-reinforcing flywheel that gets stronger and stronger over time, ultimately making a company very hard to compete with long-term. There are only four compounding advantages that really matter. Every major tech company has at least one: 1. Switching costs: The more deeply embedded a product is in the way a customer works, the harder it is for that customer to switch. This increases retention and revenue per customer, which can be reinvested into the product. 2. Network effects: The more users a product has, the better it gets for everyone. This increases engagement and attracts more users. 3. Scale economies: The more volume a company has, the larger and more efficient its operations can be. This lowers unit costs, ...
RT Pamela Fox Nice post by @isaac_flath on how he evaluates OCR models, with custom annotation apps and agent citation checkers. Great examples of the many ways OCR/VLMs can get things wrong too - like oversimplified tables, swapped digits, and ignored watermarks. https://isaacflath.com/writing/document-evals
Q: How do I make the case for investing in evaluations to my team? A: Review real user interactions and show your team the problems you find. Fix them, then show what improved. https://hamel.dev/blog/posts/evals-faq/how-do-i-make-the-case-for-investing-in-evaluations-to-my-team.html
Q: Can I use the same model for both the main task and evaluation? A: Usually, yes. Test whether the judge agrees with human labels on a held-out test set. https://hamel.dev/blog/posts/evals-faq/can-i-use-the-same-model-for-both-the-main-task-and-evaluation.html
