learning with yacine show

yacine mahdid

a show about untangling the complexity of deep learning and the brain by interviewing smart researchers and builders in the field of intelligence! ideal for scholars, industry practitioners or toddlers. www.yacinemahdid.com

Episodios

  1. 12 ago

    Pretrained LLMs Are Surrounded by Task Experts feat Yulu Gan from MIT

    sometime there is an idea that just makes other weird research results make so much more sense and sort of cascade to explain a whole bunch of phenomenon. neural thicket from the great yulu gan is one such paper that I absolutely loved from start to finish. the old mental model I had for pre-trained neural networked was that the pre-training regime made them land on a single good set of weight that was optimal for many tasks. but after going through the paper "Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights" my view has been changed. the better idea seems to be that the model after pre-training is in a basin where it is surrounded by more specialized expert at varying distance from that set of weights which the authors calls thicket. a bit like the valley surrounded by mountains. now depending on whether the model has big or small capacity this density of thicket will be varying and these expert weights will also have varying level of difficulty to be reached via post-training. this finding elegantly presented by Yulu & co. has far reaching consequence on how we should think about pre-training and it's connection with post-training models (and even distillation / continual learning) very densely packed interview in there and there is a few open threads that are ripe to be picked up if you are interested in this research direction! ps: my favorite bit is the connection with the baldwin effect in biology which is an evolutionary concept proposing that learned behaviors can shape the course of evolution, very fitting with post-training dynamics! Table of Content: 00:00:00 - pre-trained LLM are surrounded by tasks experts 00:05:35 - introduction to the great yulu gan and his beautiful paper 00:08:21 - what are neural thickets? 00:13:25 - relationship between scale and neural thickets? 00:14:56 - why it’s happening for the pretrained base model? 00:18:00 - solution density for small and large model 00:19:30 - do all perturbation help in the same way? and why? 00:25:00 - why does mixed data pretraining create neural thickets? 00:30:20 - would a distill model exhibit a different kind of distribution landscape? 00:32:50 - RandOpt algorithm 00:36:50 - where are the gains coming from with adding random noise? 00:45:45 - what happens to the distribution of the weight when doing the distillation? 00:48:55 - does the useful perturbation of the layers evenly distributed? 00:51:40 - how should we releasing model with a weights distribution perspective? 00:53:56 - does routing work better than distilling you think? 00:56:10 - what do you think will happens at 400B? 00:58:10 - how does this connect to continual learning? 01:02:20 - connection with the baldwin effect from biology? 01:06:30 - discover novelty by random exploration of the weights? 01:09:00 - what’s next for the great yulu gan enjoy folks hope this stuff is useful🌹 Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    Pretrained LLMs Are Surrounded by Task Experts feat Yulu Gan from MIT

  2. 29 jul

    How to Measure Frontier LLM? with Florian Brand from Prime Intellect

    Benchmarking LLM in 2026 is now a high stake task that requires deep expertise on LLM architecture, Prompt Engineering, Harness, Sandbox, Scoring, Hardware, Serving, and more. It’s a much different exercise than in the past that have a big impact on the performance of the final model. Not only that but benchmarking also is starting to have deep cybersecurity concern as frontier models have gained the capabilities of chaining together exploit of various kind (in order to cheat on benchmark). Underelicitation, Safety concern, cost, all plays a role in figuring out if the new models are the right tool for the job and has downstream consequence across the economy. Well in this tutorial we are exploring LLM evaluations with a researcher who spends his days taking benchmarks apart: Florian Brand! he wrote benchmark deep-dives, co-authors the open-models coverage at Nathan Lambert's Interconnects, and now works on evals as a research engineer at Prime Intellect. we'll go through a gentle intro to why evals matter for beginners, dig into why two scores on the "same" benchmark often aren't comparable, and explore where evals are heading in the age of agents + a live walkthrough of the evaluation module in the Prime Intellect lab! Pretty packed session, but highly informative! Enjoy 🌹 Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    How to Measure Frontier LLM? with Florian Brand from Prime Intellect

  3. 21 jul

    How to Learn in the AI Age? with Ludwig & Lazarz

    earlier this year I took two of my friends (ludwig and lazarz) on a big ol' discussion on how to leverage ai for learning purposes in general. what I like about these two lad is that they are their own man. they will say what they think without reserve (hence the many constructive disagreement during the interview). what I also like is that they have quite different learning background with ludwig being an almost pure hustling self-taught learner and lazarz followed an academic path driven mainly by his own curiosity. it was a great conversion which I think might be helpful for the kids out there stressed out of their mind by what to do with their self growth in this ai age that is speeding by. I think a good overall quote from lazarz to elevate your spirit during these emotionally charged time was this one : > "if you have no goals and you think it’s pointless well sorry to break it to you but it has always been pointless".beautiful piece optimistic-nihilism. enjoy! 🌹 sections: 0:00 - man I love learning 2:30 - ludwig wild background 5:50 - “my guy where the hell this drive came from” 8:32 - lazarz inspirational background 12:00 - that kid that understand everything first shot discussion 21:18 - the benefit of formal university education 30:00 - “you don’t need [to know] much to make money” 36:10 - “ma man we need this shipped get it done” plus research is social 42:32 - “getting into an university for networking is a great idea”44:50 - [ludwig] how to decide what is the next thing to learn?49:20 - “in most area that aren’t math the map of knowledge is f****d” 58:48 - “the brain is one hell of a pattern matching machine”1:00:08 - [lazarz] how to decide what is the next thing to learn?1:08:30 - adding value to the world (and yourself) is the way to go get lucky 1:18:39 - how do you deal with the ai doom feeling 1:20:45 - “if you have no goals and you think it’s pointless well sorry to break it to you but it has always been pointless” 1:35:35 - why LLM are fundamentally different from human intelligence? 1:43:10 - non-neuron cells intelligence is super smart 1:48:45 - is neuroscience still useful analogy for creating ai system? 2:02:17 - five years from now what is the one skill you glad you have improved? Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    How to Learn in the AI Age? with Ludwig & Lazarz

  4. 25 may

    Why DAGs Are Taking Over Auto-Research (w/ the Founders of Paradigma)

    auto-research is starting to gain traction as a very viable paradigm for creating useful research discovery.now, that paradigm is still in its infancy and the infrastructure to hold all that trail of context as the agents blaze through experiments isn't well defined (to say the least).on that topic, I had the chance to chat with my boys francesco and giulio from paradigma about what underlying infra is needed to make this paradigm work.the paradigma's paradigm, which involves copious amount of DAGs, make this auto-research paradigm a paradigmatic case of essential infrastructure.a few important links:👉 learn more about Paradigma: https://paradigma.inc👉 giulio twitter: https://x.com/thelokasiffers👉 francesco twitter: https://x.com/tensorqtTable of content:- 0:00 - what is missing from auto-research?- 2:02 - giulio and francesco ai journey- 8:10 - research infra is the bottleneck?- 10:18 - paradigma vision of autonomous research- 13:17 - “important discovery per joules”- 17:15 - why is DAG the unit of research for auto-research?- 20:40 - is paradigma trying to replace the research publication?- 24:50 - how does knowledge is shared between experiments in the DAG?- 27:34 - what is even auto-research lol?- 33:53 - the value of the human mind in this auto-research future.- 37:00 - how do you reconcile hallucination in this auto-research paradigm?- 41:33 - the adoption of auto-research across varied fields?- 47:30 - ✨ introduction to the auto-research infrastructure. ✨- 56:55 - where is the code?- 59:10 - full IDE next?- 1:03:20 - the place of the human in this DAG / code quality? manual node? token spent?- 1:16:02 - who’s the user for auto-research?- 1:18:13 - how to validate bad DAG?- 1:20:18 - ✨ auto-research agent results ✨- 1:22:53 - ✨ how a big research DAG looks like? ✨- 1:25:10 - how to get the canonical DAG for the final result?- 1:27:50 - the auto-research DAG being the new pre-print?- 1:30:05 - what’s next for paradigma and the auto-research infra?- 1:35:00 - what are they excited about research wise?Papers & references to check out:📌 join Paradigma's Discord: https://discord.gg/fsWzmzMBj4📌 learn more about Paradigma: https://paradigma.inc📌 and try out your ideas on Flywheel: https://flywheel.paradigma.inc enjoy! 🌹 also folks if you are a beginner and want to get started with ai do consider checking out scrimba:📌 learn to code from full-stack to AI with Scrimba https://scrimba.com/?via=yacineMahdid (extra 20% off pro with my link, great resource, I love the team) great place to start if you are non-technical to be honest. Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    Why DAGs Are Taking Over Auto-Research (w/ the Founders of Paradigma)

  5. 28 abr

    Why Self-Distillation Is Taking Over LLM Post-Training (w/ the Researchers Behind It)

    I had an awesome time interviewing @IdanShenfeld and @jonashubotter from MIT and ETH Zurich about self-distillation. this very promising post-training paradigm where the model acts as its own teacher by conditioning on environment feedback or demonstrations. we cover the SDPO algo for reinforcement learning with rich feedback and SDFT for continual learning without forgetting along with many applications. we dig into how it works, why it’s simpler and faster than GRPO, and where this is already showing up in production systems. table of content: 0:00 - what is self distillation 2:50 - idan (MIT) and jonas (ETH Zurich) introduction and motivation 18:40 - different perspective of on-policy self-distillation (presentation) 36:00 - metacognition and specificity in self-distillation 37:24 - very long hard task and self-distillation 42:00 - continual learning with self-distillation (presentation) 1:16:50 - what is next in this research direction? 1:20:00 - is there any experience with subjective feedbacks? 1:22:50 - quality vs number of feedbacks? 1:26:40 - what setting would self-distillation struggle vs GRPO? my random thoughts on the paradigm I think this is it. I think this will have the same impact as CoT had when it was first introduced and I would be very surprised given it’s strong performance against SFT that it is not already weaved into the main closed source models. It also is becoming clearer to me that the traditional very rigid boundary between pre-mid-post training are starting to collapse a bit and that the reality is more of a mix that is very dependent on the expected performance of the model. on a more meta note, I think having the model being it’s own teacher make just so much sense. like I was looking at my kids and I realized that each one of them has this bias of looking at the next in term of age for clues about how to do things. they have this innate fascination about HOW the one that is a bit older is doing things, not even the eldest. and I don’t think it’s just a fun quirk of nature. the “policy” of the next in line kid is technically very close to whatever the base kid is at the moment. yes they can learn from me or my wife, but we are so much advanced that it’s a bit hard for them to understand how we are doing things (even though the movement are more precise). it’s much easier to copy the one that is a bit older even with a slightly flawed policy and listen to them if they give feedback because they can just understand them better. it reminded me of the example idan gave where if a smaller model was just listening to the big teacher model in robotics they will just hit point in the data they can’t recover from because they are going to make mistakes that are just not possible for the larger model. anyway, this paradigm has a beautiful kernel of truth in term that is fundamental in my view and is a very exciting angle to get up to speed with! papers the slides were super crisp really cool of them to share! if you are interested in digging more into the literature here is a few papers that are worth checking out: 📌 Reinforcement Learning via Self-Distillation (SDPO): :https://arxiv.org/abs/2601.20802 📌 Self-Distillation Enables Continual Learning (SDFT): https://arxiv.org/abs/2601.19897 📌 Aligning Language Models from User Interactions: https://arxiv.org/abs/2603.12273 📌 RL’s Razor: Why Online Reinforcement Learning Forgets Less: https://arxiv.org/abs/2509.04259 📌 Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs: https://arxiv.org/abs/2410.08020 enjoy my guys 🌹 Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    Why Self-Distillation Is Taking Over LLM Post-Training (w/ the Researchers Behind It)

  6. 15 abr

    The Science of Learning Math (and Anything Else) with Justin Skycak

    hey folks first time posting this sort of interview as a podcast (shout out the three anon on twitter that requested it). now that I think of it the format does make sense since me and the interviewee are mostly chatting and exchanging ideas! will be posting more of these in the future! in this episode I had the pleasure of spending not 1 but 2 days interviewing the learning legend justin skycak. we've talked about his quite impressive self-learning journey (3000h of math in high school) all the way to how he hand curated the initial knowledge graph for math academy to make that process more efficient. I like justin because I understand his inner desire for learning. I had the exact same “awakening” let’s say in my tiny bedroom when I was a teenager. I realized while doing some homework that I loved learning and found an immense sense of peace within it. this realization kind of got lost in the midst of winter depression for a while until it got awoken again with a burning flare that never died down since (more trauma dump here). this understanding that there is a meta-learning-skill is so fun because then the whole world kind of open up for you. you encounter a hard problem you don’t know how to even get started? → great let’s use “learning”. you got an opportunity that you are under prepared for? → great let’s use “learning”. you want to make something seemingly impossible happen? → great let’s use “learning”. what I like about justin’s story is that it is very concrete and touch math which has far reaching usefulness in life. so here is the very lively 3h discussion where we touch these topics:0:00:00 - intro: 0:02:10 - justin background 0:05:45 - 3000h math self study in high school0:11:45 - what a day looked like for that 3000h stretch0:16:10 - meta-learning vs pure math learning0:21:50 - when did you get into cognitive neuro?0:29:55 - how did the fundamental math helped in your research projects0:43:10 - what does the math academy learning system looks like0:47:34 - how did you guys build the 2000 topic knowledge graph1:01:15 - would LLM be useful as an interface to that knowledge graph for the students?1:10:46 - how does the FIRe spaced repetition algorithm works?1:17:34 - does the same knowledge graph structure would work for physics? or other topic?:1:34:05 - how do you understand the subject vs the curiculum1:35:50 - is there a connection between studying math and learning a sport?1:42:00 - do you think in math doing and teaching requires different skills?1:56:25 - could you get understanding without automaticy?2:05:35 - do you see any upside of confusion in learning?2:14:11 - learning math as an adult?2:19:20 - how to fill the motivation gap after learning the fundamental?2:24:10 - how should teaching math for kids and adults balance fundamentals and creativity?2:33:55 - is it ever too late to learn math seriously?2:46:00 - mastery learning vs ultra learning2:51:30 - top-down vs bottom-up2:53:40 - mastery learning for domain without a structured hierarchical structure?2:56:30 - neurodivergence / adhd for structured math learning?3:06:20 - amateur mathematician augmented with technology will be able to contribute to research?3:14:37 - what are you most excited about right now in term of learning enjoy my guys! :) Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    The Science of Learning Math (and Anything Else) with Justin Skycak
  • 6 episodios

Acerca de

a show about untangling the complexity of deep learning and the brain by interviewing smart researchers and builders in the field of intelligence! ideal for scholars, industry practitioners or toddlers. www.yacinemahdid.com

Información

  • Creador
    yacine mahdid
  • Años de actividad
    2026
  • Episodios
    6
  • Clasificación
    Explícito
  • Copyright
    © Yacine Mahdid
  • Mostrar sitio web
    learning with yacine show