Sebastian Raschka

Inkling 2 Small took a while... but it's more interesting than I initially thought! Using attention residuals (like Kimi K3), but without the linear-attention hybrid. Food for discussion for the end-of-the-year architecture trends analysis!

Seb's Law, co-signed by another Seb

Wow that escalated quickly. Was gone for a few weeks and now pretty much everyone launched their own decision model copycats: - Sep 23: Together AI, Tev1-4B - Sep 24: Alibaba Cloud, decision-model - Sep 29: Liquid AI, d1 - Sep 30: Databricks, ai_decide - Oct 1: Amazon AWS, Strands Decider 2B - Oct 1: Cloudflare, Clef and Clef-flash (Clef-omni on Oct 9) - Oct 1: Perplexity, Decider (v1.3 on Nov 12) - Oct 6: OpenAI, Decisions API (GPT-6 Luna) - Oct 7: http://H2O.ai, H2O-Lightning-4B - Oct 7: Snowflake, Snowflake Decision - Oct 9: Microsoft, Microsoft-Decision-1 - Oct 15: IBM, Granite Switch 4.1 with Decision adapter - Oct 21: Baseten, Mammoth-1 and Elephant-1 - Nov 6: Salesforce, Koa Decisions - Nov 11: Google DeepMind, Gemini Decision 1 According to a Vercel report, 26% of requests are now routed through decision models 📈

Hoping Nemotron keeps going. Thankfully, open weight LLMs have some alternatives. But their open data and full training recipes would be much harder to replace.

LLM training is hard. Hard to be a jack of all trades.

In reasoning model development, it's always been this interesting tension between training/optimizing chain-of-thoughts (CoTs) vs not touching them to preserve monitorability. CoTs are cool because we can check what the model is up to (e.g., whether it cheats or not) as long as it spells out its thought process. And we risk losing that if we explicitly optimize CoTs. Two caveats: 1) I.e., the catch of CoT and monitorability is that this only works if the written thinking reflects what the model is actually doing. 2) Also, when models become more capable, they spell out less in the CoT (like a cleverer math student needs less note-taking when solving problems during an exam). Nothing new on that front, but an interesting thread below on how to measure "harder to monitor", with some interesting results that confirm the existing intuitions: 1) no-CoT capability (meaning how much reasoning the model can do in a single forward pass without writing anything down) - Better pretraining (lower perplexity) raises this no-CoT capability - Architecture mostly doesn't matter 2) CoT controllability (meaning how well the model can change its written reasoning when asked to add, remove, or change parts of it) - RL alone raises controllability a lot - Controllability is passed on through distillation * Higher no-CoT capability and higher controllability both make monitoring harder. PS: Perhaps the good news is that it seems that process supervision wasn't the main culprit.

RT Brian i've never learned as much as when building things from scratch. currently working through build a large language model by @rasbt, and it's one of the best ways i've found to actually understand how llms work. still struggling to visualize all those dimensions.

Hillclimbing: Make the model better (best intentions). Benchmarking: Measure how much better (best effort). Benchmaxxing: Make the numbers look better (best optics).

Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues. In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids. In the first half of 2026, the trend towards "some full attention" hybrids continued. And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore. The plot legend here means, - Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer. - Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention. - No full attention = Zero dense global layers. The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls. In particular, there are two distinct efficiency gains here: 1) Less attention computation, since sparse attention and the like select tokens or blocks from the context and performs the expensive attention operation from only that subset. (Note that it can still retrieve information from distant positions.) 2) Less memory for retaining context in some architectures, meaning that recurrent layers such as Gated DeltaNet maintain a fixed-size state. Of course, this saves memory, but compressing the history creates challenges for exact retrieval, so they are usually intermixed with traditional attention methods (sparse or full).

What could possibly go wrong?

The complete reasoning-from-scratch video series is now up (and linked) here: https://magazine.sebastianraschka.com/p/building-reasoning-models (Taking a short break but will be back with LLM model and research updates in November!)

Spontaneous morning chat with Juergen Schmidhuber and Jake Newfield philosophizing about AI. Sry my mic somehow disconnected but Juergen had the more interesting points anyways!

RT Sebastian Raschka Re Update: OpenAI just added a Jev clone 😆

A little Ember-1 tl;dr. Seems like a great model! (Was recently asked on a podcast, given $ xx million, what's the best way to develop a frontier LLM today? My recommendation was: start with an existing one and spend that budget on post-training. Great example here.)

RT Out of Context Human Race The golden era.

The main appeal of open-source agent harnesses isn't that they are free, but that we can inspect what they are doing on our computers.

MiMo-V2.6 is "simply" the best (for now). Despite its simple architecture design it's currently No.1 in the open-weight benchmarks (weighted average). With "simple," I mean a classic Grouped Query Attention (GQA) with Sliding Window Attention (SWA) at a tiny 128-token window size. So, that underlines one of the points I've been trying to make in recent months: most of the progress still comes from the data and post-training recipe improvements. Fancy attention variants are just mostly efficiency tweaks. What are some of the training data improvements and recipe improvements? The MiMo team shared a pretty detailed technical report. Lots to carefully digest there, but in short, there are a few things that stood out: 1. An increase in agent tasks; also training across different harnesses (the average DeepSWE pass@1 accuracy on held-out harnesses improved from approximately 50% -> 66%). 2. Better reward signals: they replaced a simple correctness verifier with an agentic grader that looks at the execution traces as well. 3. Large RL batches (1,568 prompts × 16 rollouts = 25,088 trajectories) and 2.7–3.7 billion training tokens per update (unclear, though, what the predecessor used).

It’s easy to dismiss Jev it as “just a classifier”. But people (me included) who have been training encoder-style models for classification for many years know they were usually special-purpose and limited in some way. The breakthrough of Jev is that it generalizes well. And I’d say the secret sauce is probably more in the data than in the training algorithm. (Plus a nice API design on top of it.)

Interesting new insights into the old AdamW vs Muon debate: Muon seems to do better because it reduces memorization.