Content History
Just updated our AI evals FAQ with 5 new questions, 48 questions & answers total! New FAQs just added: - Do I need a reference answer or rubric before annotating data? - How can I do evals when traces contain sensitive data? - How do you review a trace that is really large? - How much context should I give a LLM judge? - What should I do when my “gold” eval dataset becomes stale? It's all here: https://hamel.dev/blog/posts/evals-faq/
RT Quinn Slack Amp is now free to use when you bring your own compute and model subscriptions/keys. No more limits or fees for BYOK. https://ampcode.com/news/free-agent
It's cool that there is more interest in eval tools! Some opportunities for improvement: 1. Right now, this workflow tries to quiz you up front about your recollection of your experience with a plugin. It would be better if it was more "in-situ", meaning you could give feedback on the plugins as you are using them. 2. It goes off and builds datasets and judges automatically based on what you tell it in up front as well as what's documented in the plugin. I would like to see it try to do error discovery with you first to allow you to annotate real traces / session history etc so you can figure out what's not working better. 3. It was clunky to do this against a skill that wasn't a plugin. I had to ask claude to set it up for me and it took 10 minutes to figure that out. 4. I had to stare at the generated HTML for a while before I could understand what everything meant. There is some onboarding experience that is missing but I'm not sure what that is yet. I'm sure this will get better over time and the things above seem doable. It's also positive to see affordances for evals directly in our agents!
New in Claude Code: claude plugin eval See what value your plugin is adding, or if it needs more work. You can create test cases, run your plugin or skill against those test cases, score those runs, then run each case again without the plugin to see the differences.
RT Florian Brand in FrontierSWE, some tasks come with gold outputs so the model has something to work with (which are diff from the tests, so they don't matter tbf) and what do some models (like Grok) do with such things? thats right, hardcode these values so the local tests pass
RT Hugo Bowne-Anderson A lot of my working life over the past year has been preoccupied with harness engineering. When @HamelHusain introduced me to @doesdatmaksense to chat about how evals belong at the center of harness engineering, I immediately wanted her to write a guest post for the VG substack. Check it out to take your agent evals to the next level: https://hugobowne.substack.com/p/how-evals-are-central-to-harness Thanks, AS & HH!
RT Jay Alammar Over 300 original figures! An Illustrated Guide to AI Agents is now OUT on Kindle and other Ebook stores! Presses are rolling, print copies ship in a few weeks. @MaartenGr and I spent the last year and a half making the agent stack legible: memory, tools, planning, evaluation, multi-agent systems, and code agents. We're ecstatic to finally be able to share it with you! Agents are being adopted faster than they're being understood. Our bet is that a small number of concepts will outlast everything else in this field, so we spent the time identifying them and explaining them properly. Concepts and code. You build a working agent yourself along the way. Order here: Amazon: https://www.amazon.com/Illustrated-Guide-AI-Agents-Concepts-ebook/dp/B0H381PNPR/ O'Reilly: https://www.oreilly.com/library/view/an-illustrated-guide/9798341662681/
RT Gergely Orosz What’s the problem w using LLMs to write stuff you want other people to read -emails, articles, posts- and why are people (like me!) so incredibly hostile that they will block you/mute you/block your domain, to never read anything from you again? Bryan spells it out. Read it:
The revolt of the reader https://bcantrill.dtrace.org/2026/09/05/the-revolt-of-the-reader/
View quoted post. @isaac_flath and I are going to see if the slop prompt is good and do some evals. (also first time trying to stream haha). https://x.com/i/broadcasts/1DGLdZNYNNyGm?s=20
RT Ivan Leo thought it was funny i asked @bot to teach me more about post training and it just created fake sft samples and told me to look at the data and do data labelling can't run away from @HamelHusain 's just look at your data haha
RT Cognition GPT-6 Astra is coming to Devin. On FrontierCode 1.1, Astra performs within 0.4 points of Fable 5 at a 64% lower cost. It also sets a new SOTA on our internal testing benchmark, generating more comprehensive tests, clearer reports, and better video evidence.
The suggested prompt to remove slop is itself, pure slop If it works fine, but feels funny to me ngl
Teaching is hard but worth it when you see this 🥰 @sh_reya and I are refreshing the material once again to incorporate the latest eval techniques in our next cohort https://evals.info
Don't fall for this rage bait 😅 DS gives you the tools and judgement to make sense of noisy signals and stochastic outputs For AI -> lets you measure if an AI product is working even though the outputs are text. You have to design experiments and test hypothesis that account for noisy signals. More on this here https://hamel.dev/blog/posts/revenge/
Data science feels so dead now that it's like it never existed at all. AI seems to have killed it
View quoted postRT rahul are we just supposed to forget what they did to windsurf?
Cursor has been a trusted partner of Anthropic since Sonnet 3.5. We’ll continue to increase compute to support Claude models in Cursor and are excited for what comes next with them at SpaceX.
View quoted postReally great use of tokens. Highly recommend. This is an AGI level task b/c have to click forms and such. Codex computer use flying through it
100% of the replies to this are AI bots Super disappointing
Really interesting new blog post from @openai for several reasons: 1) Shows an example of building with WebMCP, meant for when you want agents and and humans to collaborate on using a UI (like co-editing notebook cells). It's different than MCPs or APIs in that its exposed
Really interesting new blog post from @openai for several reasons: 1) Shows an example of building with WebMCP, meant for when you want agents and and humans to collaborate on using a UI (like co-editing notebook cells). It's different than MCPs or APIs in that its exposed directly through the browser. Read the post for discussion of the tradeoffs. 2) They created a new kind of notebook which works with WebMCP that prioritizes meeting people where they are: you bring your own coding agent and files are just markdown. The author uses it to curate runbooks or high quality examples of how to run foundation model evals on their infrastructure. Notebooks are good for this since they require tinkering with state of long running jobs interactively while taking notes inline. And its open source ✨ Blog: https://learn.chatgpt.com/blog/automating-repetitive-work-at-openai-with-codex P.S. this post is authored by Jeremy Lewi who isn't on X but here is his website https://lewi.us/
RT Peter Yang I don’t promote other people’s courses often, but I want to recommend Hamel and Shreya’s top rated AI evals course. Here’s why I think it’s worth taking: 1. A 4.7-star rating across 900 reviews, with students from OpenAI and Google, is almost unheard of on Maven. 2. They’ve completely revamped the course and added a 24/7 AI evals assistant to guide you through the process. 3. They’ve personally helped me improve my evals, and their advice is consistently practical, specific, and grounded in real experience. You can watch my free podcast episode with them to learn more about their eval process before deciding: https://youtu.be/bdMHQLvtVaQ The next cohort of their course starts September 5 and you can get 25% off with this link: https://maven.com/parlance-labs/evals?utm_campaign=peteryang&utm_medium=affiliate&utm_source=maven&promoCode=PETERYANG
I was wondering why Google is in its own special toilet tier in this video 🤣 Summary: it’s expensive as hell compared to everything else
Sneak peek of segments of the new evals course ✨😃
we've been iterating a lot on how DocWriter internally represents a user's writing style! the naive approach is to dump all a user's prior writing in context and pray the AI figures out how to sound like them. this doesn't really work. and the user can't just manually specify
RT Peter Yang In my next episode, I asked @sh_reya and @HamelHusain (taught evals to 4,500+ engineers and PMs) to roast the AI evals that I built for my creator skills live. They also showed me how anyone can use their free Error Discovery skill in Claude Code or Codex to turn real AI failures into reusable evals. 📌 Subscribe to get the full episode tomorrow: https://www.youtube.com/@PeterYangYT?sub_confirmation=1
RT Peter Yang I'm on my way back to Vancouver to be with my mom but wanted to take a moment to celebrate crossing 100K subs on YouTube. Excited to share a lot more practical interviews soon to answer your most burning AI questions: 1. @HamelHusain and @sh_reya (AI evals experts) on how today's AI models have changed evals completely 2. @amoljain_ (Head of Product Engineering at Replit) on which vibe-coded apps have actually become real businesses 3. @ebloch (Product Lead at OpenAI) on how ChatGPT Finance can help you save both time and money 4. @poteto and Roman (SpaceXAI) on how the Grok Bot team uses Grok @bot 📌 Subscribe to get the episodes soon: https://www.youtube.com/@PeterYangYT?sub_confirmation=1
RT knut sat down and actually read @rosmine's research paper on this. https://deftwriting.com/research/distribution-fine-tuning first of all, i was guilty of having a "take" based on this post, pointing out how the copy in the screenshot is not a great example of what "good writing" is, despite scoring 100 on "human written" in @pangram. I recommend actually reading the work before commenting on it - because sloppy takes are as lazy as sloppy texts. (shame on me for joining the band wagon) but sleeping on it, i am grateful that @rosmine took the time to dive into this stuff. AI-slop fatigue is real, and we should support efforts to make agents produce communication that is clear and lucid, and doesn't feel overly synthetic but there are some assumptions in this, however, that is interesting to question when it comes to "what makes for good writing" as far as i understand, Deft is trained to have more diversity/variation in tokens, so less repetition of what we are recognizing as "AI-tells" (certain word and stylistic choices). And i totally agree with @rosmine that: "LLMs are not the cause of slop. Lack of effort/care is. If you spend days researching and planning a blog post, and put all the information into a detailed, well-structured outline, and ask ChatGPT to generate the post based on the outline, then the output will be interesting to read, even if the text has a lot of em-dashes." I argued the same in our eng blog announcement post yesterday: https://www.sanity.io/engineering/announcing-the-sanity-engineering-blog BUT! I still feel that this report (at least somewhat), but especially the various takes on it, conflates something sounding "human" with it being "good." Tricking @pangram doesn't make a text well written. Some reflections: - Making writing sound more "human" by means of adding more variation in word/style choices, doesn't make it better - What makes for a "good" text is highly contextual. If you are writing a recipe or instruction...
Announcing Deft, a new AI lab for better writing, cofounded with @jmrphy See the picture for launch announcement the Deft model wrote for itself Currently, 86% of user queries are fully human according to pangram. This is still a small beta model and it might make mistakes. We
Hire John. I worked with him personally at Airbnb and he’s top 1%
I'm looking for my next role. I'm an AI engineer / PM, currently based in Berlin and open to relocating. Most recently Head of AI & Product at an AI fintech until the startup shut down in June. Before that: data science at Trumid and Airbnb, and Head of Data at Circ. Over the
RT John Enevoldsen I'm looking for my next role. I'm an AI engineer / PM, currently based in Berlin and open to relocating. Most recently Head of AI & Product at an AI fintech until the startup shut down in June. Before that: data science at Trumid and Airbnb, and Head of Data at Circ. Over the past couple of months, I’ve also been getting much closer to the model side: training three LLMs from scratch in PyTorch, post-training them with SFT and GRPO, then building the inference and serving stack, including KV caching, streaming and FastAPI. Everything is open source: models, weights, code, write-ups, and a playground with all nine checkpoints: https://huggingface.co/spaces/JohnEnev/modern-llm-playground I’m looking for an AI engineering or AI PM role, ideally somewhere I can stay close to both the model work and the product. Berlin, remote, or open to relocating. If you know of something, or someone worth speaking to, I’d really appreciate a pointer or repost.
RT Shreya Shankar Interesting to see all the agreement. Check out the DocWriter project's plain-writing skill: https://github.com/docwriter-org/plain-writing-skill We recently added evals for the skill --- comparing a gpt-5.5 agent without the skill to with the skill (using an LLM judge to evaluate each criterion)
Grok 4.6 just ranked #1 on CursorBench 3.2 Outperforming Claude Fable 5, Opus 5 and GPT-5.6 Sol on real-world coding performance And what makes this even crazier is the efficiency....the chart gives CursorBench performance against average cost per task, and Grok 4.6 is sitting
“overcast” describes cloud cover precisely, while “grey” might describe the light, sky, mood,etc. Yes, I care, and it matters.
RT Nick Dobos Anthropic’s response to watermarks confirmed my worst fears. “Our watermarking method doesn’t have any practical impact on the quality or content” That is a lie. They literally say they change the wording in the first paragraph Anyone with an ounce of common sense about how psychology, writing, and legal documents works in the real world, knows this completely changes the meaning and content. This degrades and changes petulance. Look at the example they picked, it’s garbage. “Overcast” vs “grey” are completely different descriptions. That’s their best example, and it changes the sentence subtly yet dramatically. This is unbelievably dangerous This will kill people I am not exaggerating Anthropic just handed the keys to mind control the government They just gave politicians a fully legal back door that can be exploited to influence claude’s thinking to mind control millions of people without anyone ever knowing This is not okay.
We’ve written an FAQ to answer some of the questions we've received about watermarking. In summary: • We’re implementing watermarking to comply with the EU AI Act. Other major model developers have signed the same Code of Practice and will also be implementing watermarking;
View quoted post
Over the summer, @sh_reya and I hosted 13 sessions on AI Engineering topics like retrieval, post-training, inference, and evals. I've summarized all the sessions, organized by theme, with links to the source materials. Warning: I've tried to pull the most important ideas from each talk, so some notes are short (9.5 hours of sessions comes out to about 20 minutes of reading). Enjoy! https://hamel.dev/notes/llm/ai-product-engineering/
RT Joe Barrow An easy but surprisingly useful trick you can learn is napkin math for model training and inference. How much work is done during speculative decoding? How fast should a DETR run at various resolutions? If you can approximate these quickly it will help diagnose slowdowns, reason about new techniques, and maybe even invent a few of your own!
TIL that /visualize is built in codex skill
99% of people don't know you can tell your chief of staff thread to use `/visualize` I have a pinned travel thread that tells me my travel schedule. @PhilippSpiess has done incredible work here
It’s been a long time since I’ve been excited to work through a technical book @rasbt It’s time to bring more ML back into my life
