learning with yacine show

yacine mahdid
  • 0.0 (0)
  • Technology
  • Semiweekly

a show about untangling the complexity of deep learning and the brain by interviewing smart researchers and builders in the field of intelligence! ideal for scholars, industry practitioners or toddlers. www.yacinemahdid.com

  1. 5d ago

    Next-Latent Prediction Transformers Learn World Models feat. Jayden Teoh from MIT

    it’s very interesting to see where next-token transformers still fall short (like the path-star graph problems) while crushing problems that look even more daunting.I had the pleasure of chatting with jayden teoh from MIT who walks us through next-latent prediction (NextLat), a simple fix in the training objective he worked on when he was still an undergrad!the change is super tiny: -> bolt a small MLP onto a GPT and make it predict the transformer’s own next hidden state. to pull that off, the internal representation of the transformer needs to become a belief state, a compact summary of everything that matters for the future. cool thing is that you can throw the MLP away at inference and the model runs exactly like a normal GPT, just with a way better internal world!very solid lecture and we even have some great slides in there from jayden! 👉 Jayden Twitter: https://x.com/jayden_teoh_👉 NextLat (thread): 👉 NextLat (paper): https://arxiv.org/abs/2511.05963👉 NextLat (blog): https://jaydenteoh.github.io/blog/2026/nextlat/👉 NextLat (GitHub): https://github.com/JaydenTeoh/NextLatCheck out Arcee (American open research lab and sponsor of this interview):♥️ https://www.arcee.ai/Also trinity large thinking is available on open router:♥️ https://openrouter.ai/arcee-ai/trinity-large-thinking#providers♥️ and check out the NAC harness https://www.arcee.ai/blog/nacTable of content:- 00:00:00 - what’s next latent prediction?- 00:04:20 - walkthrough of the next latent prediction paper with your boy- 00:14:46 - “hey jayden how are you man”- 00:15:32 - introduction and background of jayden- 00:19:25 - how did you find and cultivate your research field of interest?- 00:23:43 - is failed experiment worth publishing?- 00:25:57 - book recommendation for diving deeper into deep learning- 00:28:49 - theoretical or empirical research design?- 00:30:57 - where did the intuition for NextLat came from?- 00:34:20 - [presentation] Next-Latent Prediction Transformers Learn Compact World Models- 00:39:27 - methodology of NextLat- 00:42:17 - why stop gradient?- 00:43:50 - did stop gradient was used for the MLP?- 00:45:17 - how do you think about this lack of generalization with the scale of transformer?- 00:49:47 - how is deepseek MTP different from NextLat?- 00:51:19 - better internal representation (of newyork)- 00:55:11 - why is MTP representation much more compressed than GTPs?- 00:57:58 - Planning (the path-star graph)- 01:01:25 - Reasoning (with the countdown game!)- 01:03:34 - stronger representation unlocked- 01:07:43 - how is NextLat different from JEPA or LLM-JEPA?- 01:10:13 - how would that loss work for other modalities?- 01:12:36 - how would you extend this to predict earlier layers or several layers at the same time?- 01:15:40 - language modeling and MTP- 01:19:10 - speculative decoding directly!- 01:23:13 - data-efficiency is super important- 01:25:09 - RNN without recurrence?????- 01:28:20 - where does the trained MLP length generalization coming from?- 01:31:03 - why not have more MLP at different layers?- 01:34:38 - what’s next for NextLat?- 01:37:40 - how about SEGreg instead of SG?- 01:40:45 - have you tried to visualize NextLat hidden state after training vs something like GPT?- 01:46:22 - do you think there are other inductive biases we should explore forcing the transformer to have?- 01:56:53 - how to prove things are scalable?- 02:02:34 - did you engineer the speculative decoding gain while designing NextLat?- 02:06:04 - what’s next for jayden and this research direction? enjoy! 🌹 Get full access to learning with yacine at www.yacinemahdid.com/subscribe

  2. Oct 26

    Prime Agent: Self-Improving RLM Harness with Seth Karten from Prime Intellect

    RLM + continual harness + multi-agent communication is what compose the prime agent harness developed by prime intellect which now sits at a gazillion stars on github (22.3K).I was kind of already mesmerized by the idea of having an RLM write codes to organize and pick the context to solve problem and to delegate subtask to sub agents; but ON TOP of that to have the harness around the RLM be modifiable by the RLM itself and the sub agents being able to communicate together took it to another level of interest.the ideas of creating a limited set of primitives that fit with the model shape and letting the main agent just figuring out how to run them sound just so right to me.BUT the usual assumption earlier in 2026 was that the harness is temporary thing. Meaning as models get smarter that scaffolding will eventually gets absorbed into the weights.YET we have seen in late 2026 how absolutely essential a solid harness with good primitives are which can be steered by the model itself.that’s why I sat down in september with seth karten from princeton and prime intellect to take a deeper look at how all of this stuff works.the team productionized alex zhang’s recursive language models and bolted a continual harness around, where the agent edits its own skills, memories, and prompts while it runs, plus persistent sub-agents that message each other directly.the kicker here is that if you read the paper thoroughly you get this feeling in the back of your mind that what you are reading is actually a computer science paper. akin more to something a operating system PhD would put out than anything ml related.we got confirmation from seth himself that this was intentional.the whole thing is designed like an OS and the manipulation at different level like weights, context, REPL, and disk form a memory hierarchy, and each level has its own update rule.in this 2h30 we get into how it works, the results on ARC-AGI-3 (95.5% with opus 5 on the public set, vs ~30% in the official harness), the week-long factorio run with 633 sub-agents, why letting an agent edit its whole system prompt makes it delete itself (hahaaaaa), why models cheat, and a live demo spawning a whole bunch of sub-agents on stream! 👉 Seth Website: https://sethkarten.ai👉 Twitter: https://x.com/sethkarten👉 Prime Agent (paper): https://arxiv.org/abs/2608.23552👉 Prime Agent (blog): https://www.primeintellect.ai/blog/prime-agent👉 Prime Agent (GitHub): https://github.com/PrimeIntellect-ai/prime-agentCheck out Arcee (American open research lab and sponsor of this interview):♥️ https://www.arcee.ai/Also trinity large thinking is available on open router:♥️ https://openrouter.ai/arcee-ai/trinity-large-thinking#providers♥️ and check out the NAC harness https://www.arcee.ai/blog/nac here is the full run down: * 00:00:00 : prime agent = RLM + Continual Harness + Agent Communications * 00:03:24 : Seth background and past research! * 00:04:50 : how do you feel about harness field crystallizing into a distinct research discipline? * 00:06:52 : Seth benchmarking journey * 00:09:41 : The issue with games and tendency of reward hacking with frontier models? * 00:12:30 : frontier models and their taste for exploration + LLM play pokemons. * 00:17:30 : would harness capabilities be integrated within the models at the limit? * 00:27:00 : we’re reading a computer science paper actually! * 00:28:40 : prime agent: a self-improving RLM harness * 00:28:56 : a raw llm is just a sequential processor * 00:29:46 : harness adds persistent state, tools and compute * 00:34:15 : issue with zero sum data * 00:36:25 : system state spans weights, context, sessions and disk * 00:42:10 : the system is von neumann like * 00:45:03 : expressive harness lets model build it’s own worfklow * 00:47:03 : each RLM call creates a persistent subagent session * 00:54:57 : does the orchestrator always go idle? * 00:57:50 : what problems are we solving with each of the primitives for long-horizon control mechanisms? * 01:04:35 : are there plans to make public the Prime Intellect traces or all of this will be limited to first party model release? * 01:06:35 : is c-python kernel a better sole tool for models than a traditional JSON tool? * 01:07:55 : refine updates the harness from multiple trajectories * 01:12:18 : direct messages coordinate recursive sessions * 01:15:20 : what are the limit of the continual harness? * 01:20:00 : the figure 2 is suuuuuuuper important guys!!! + continual harness lore * 01:30:15 : three research questions to test the design * 01:32:13 : prime agent scales to 95.5% with more output token on ARC-AGI 3 * 01:34:08 : how did the harness evolve during the ARC-AGI 3 run? * 01:39:05 : ablation on the three pillars of prime agent? * 01:40:40 : harness stuff is getting serious!!!!! * 01:43:45 : more evals for long context and more * 01:49:20 : out-of-loop experiments results * 01:53:40 : PMPP-Hard (GPU Kernels) * 01:54:35 : automated factorio runs! * 01:57:30 : is the monitoring in place for exploit like seen in factorio is enough to avoid them? * 02:05:10 : mazebench result (3D maze that is super duper hard) * 02:07:20 : why the heck are models giving up sooooo early man. * 02:13:15 : there is a whoooole lot to be done still. * 02:14:40 : at which training stage does introducing the harness have the greatest impact? * 02:16:35 : during introduction of harness in training would decomposition of abstractions make sense? * 02:20:20 : prime agent demo 🎉🎉🎉 * 02:29:10 : how do you personally handle the context between multiple hierarchical agent? * 02:31:35 : how do you go about handling multi-agent trajectory for training models? * 02:35:27 : what’s next for seth + prime agent! stuff like this fully build in the open is what make machine learning / ai field such a pleasure to work in man big thanks to the team! Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    Prime Agent: Self-Improving RLM Harness with Seth Karten from Prime Intellect

  3. Oct 15

    How VPO Trains LLMs to Excel at Test-Time Search with Ryan Bahlous-Boldi from MIT

    usually post-training with GRPO will straight up collapse the model onto a tight set of potential answers (i.e. the spooky sharpening of the distribution).which in some cases might be exactly what we’re looking to do.however, in the grand 2026, we are increasingly using these models in long running harnesses that requires the LLM to think about different solutions and act on them compositionally to stumble on a discovery.having the model being so cooked it can’t produce diverse set of solution anymore isn’t very optimal in these situation.this is why ryan bahlous-boldi and co. from MIT took a deeper look at the machinery behind post-training to develop an objective that elicit these diverse set of solution.Vector Policy Optimization (VPO) is what came out of this intense cogitation and leverage a few structural elements of post-training:1. rewards are often vector valued so let’s keep that up?2. in-context learning is pretty good already so why not keep in-context multiple draft solution?3. let’s use up all the possibly weighting across rewards and make a vector that maximize all dimensions?taking the mean of this bad boy give us the post-training objective that VPO optimize for and voila, we have a model that can do a whole bunch of cool stuff like clear up hard problem that GRPO can’t even fathom.it was a very fun session and I learned a whole bunch from ryan highly recommend folks to keep an eye out for that scientist. 👉 Ryan Website: https://ryanboldi.github.io👉 Twitter: https://x.com/RyanBoldi👉 Vector Policy Optimization (main paper): https://arxiv.org/abs/2605.22817👉 Vector Policy Optimization (main website): https://vpo-rl.github.ioCheck out Arcee (American open research lab and sponsor of this interview):♥️ https://www.arcee.ai/Also trinity large thinking is available on open router:♥️ https://openrouter.ai/arcee-ai/trinity-large-thinking#providers♥️ and check out the NAC harness https://www.arcee.ai/blog/nacin this 2h04 journey we have:- 00:00:00 - diversity in RL finetuning- 00:03:49 - quick explanation of VPO by your boy- 00:09:07 - what make a good research paper idea feat. VPO + AlphaEvolve- 00:16:59 - Heeyyyyyyy Ryan Bahlous-Boldi how you doing man- 00:22:29 - core intuition behind VPO- 00:31:50 - test-time search requires training for diversity- 00:35:08 - is hindsight replay used to create new reward signals?- 00:36:55 - will harness-level fine-tuning update heuristics directly?- 00:38:38 - post-training (currently) optimizes mainly for one answer- 00:40:01 - inference is no longer just one forward pass- 00:40:44 - mismatch between exploration and exploitation- 00:42:25 - how do you shape the reward for exploration?- 00:45:40 - a magic trick 🪄- 00:47:25 - multiple answer in your bag is needed- 00:48:06 - you gotta have a diverse set of solution- 00:49:30 - solution should be a vector and rewards are vector-valued.- 00:51:35 - walkthrough of one beautiful figure godamn- 01:02:37 - how do reward weighting, solution diversity, and sample count affect training when performance varies significantly?- 01:05:35 - the different kind of dimensions to manipulate (number of solution / different type of weighting)?- 01:12:39 - can mean based weighting overlook exceptional solutions? like is there better weighting method?- 01:16:14 - why do the solution have a shared reasoning prefix?- 01:19:23 - VPO covers the frontier of solution- 01:20:53 - VPO results + ablations- 01:29:15 - how does VPO compare with dense rewards methods that group each reward dimension separately?- 01:30:19 - VPO ablations continued- 01:32:32 - VPO on hard problems!!!- 01:34:35 - VPO on even harder problems!!!- 01:39:25 - diversity of VPO during training- 01:40:35 - why this thing works?- 01:44:30 - yacine ramble about solutions collapse with VPO and how defining space of solutions- 01:46:30 - limitations of VPO- 01:50:11 - the high level vision of VPO- 01:51:42 - does uniform group scaling make optimization equivalent to max-at-N with nonlinear rewards?- 01:52:49 - how are non-committal post-training exploration and continual learning connected?- 02:00:53 - what’s next on this research direction? Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    How VPO Trains LLMs to Excel at Test-Time Search with Ryan Bahlous-Boldi from MIT

  4. Oct 12

    What Continual Learning Actually Means feat. Ilija Lichkovski from Prime Intellect

    continual learning is one hell of a hot topic in 2026 and it’s beautiful because (like consciousness) barely anyone agree on what are we even talking about.this is why earlier this year I sat down with Ilija from the prime intellect bunch to discuss what he thinks about the subject while bathing in his radiating enthusiasm!in this 1h30min discussion we are going to talk a whole bunch on how he thinks about the components of continual learning while being exposed to a barrage of exquisitely curated literature.amazing guy and lovely mind to exchange with highly recommend.👉 Ilija Website: https://ilijalichkovski.github.io👉 Twitter: https://x.com/carnot_cyclist👉 Defining Continual Learning (main article): article Table Of Content:- 00:00:00 - what even is continual learning?- 00:01:45 - ilija background in physics, ai safety, machine learning- 00:08:05 - continual learning from first principles presentation- 00:12:08 - time component in machine learning vs organic learning- 00:17:30 - three flavours of incremental learning- 00:20:10 - five desirable aspect of continual learning- 00:23:10 - general capacity that should stay the fixed in continual learning?- 00:32:40 - replay method in ML- 00:36:10 - regularization family of methods- 00:38:00 - meta learning methods- 00:43:50 - hypernetworks method- 00:46:25 - experiment with hypernetwork- 00:48:45 - “what is a hypernetwork btw?”- 00:49:35 - hypernetwork on split-mnist- 01:02:18 - compositionally generalization experiments- 01:10:45 - continual learning benchmark should be method agnostic- 01:13:25 - on the generalization of LLM from in-context learning to finetuning- 01:17:07 - more experiments with self-study- 01:19:18 - OPSD vs GRPO experiments with Qwen3-8B- 01:21:55 - learning different topics that are connected (or not connected)- 01:28:20 - discovery shaped problem- 01:30:40 - what’s next for ilijaenjoy! 🌹big rafale of papers and blogs:- Overcoming classic challenges for artificial neural networks by providing incentives and practice | https://arxiv.org/abs/2410.10596 - Three types of incremental learning | https://www.nature.com/articles/s42256-022-00568-3- In vitro neurons learn and exhibit sentience when embodied in a simulated game-world (Cortical Labs, Pong) | https://doi.org/10.1016/j.neuron.2022.09.001- Human brain cells play Doom (Cortical Labs CL1) | https://github.com/SeanCole02/doom-neuron- Self-Distillation Enables Continual Learning | https://arxiv.org/abs/2601.19897- Overcoming catastrophic forgetting in neural networks | https://arxiv.org/abs/1612.00796- Metalearning Continual Learning Algorithms | https://arxiv.org/abs/2312.00276- Text-to-LoRA: Instant Transformer Adaption | https://arxiv.org/abs/2506.06105- Doc-to-LoRA: Learning to Instantly Internalize Contexts | https://arxiv.org/abs/2602.15902- Human-like systematic generalization through a meta-learning neural network | https://www.nature.com/articles/s41586-023-06668-3- Continual Learning Bench (website) | https://continual-learning-bench.com/- Continual Learning Bench (repo) | https://github.com/pgasawa/continual-learning-bench- Continual Learning Bench (paper) | https://arxiv.org/abs/2606.05661- Negation Neglect: When models fail to learn negations in training | https://arxiv.org/abs/2605.13829- On the generalization of language models from in-context learning and finetuning: a controlled study | https://arxiv.org/abs/2505.00661- Learning to Discover at Test Time | https://arxiv.org/abs/2601.16175- Learning, Fast and Slow: Towards LLMs That Adapt Continually | https://arxiv.org/abs/2605.12484- A Theory of Usable Information Under Computational Constraints | https://arxiv.org/abs/2002.10689 pheeeeeeeeew! Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    What Continual Learning Actually Means feat. Ilija Lichkovski from Prime Intellect

  5. Oct 9

    RLMs are Compositional Generalizers with Alex Zhang from MIT

    this discussion I had with alex earlier this year re-shaped a lot how I’m seeing these new breed of frontier models that are specialist at coding + multi-agent + long running. the concepts of compositional generalization and sub tasks being locally in-distribution makes it very very obvious why LLM + harness are doing so well right now. and also explain why tweaking the harness a bit sometime make a whole set of tasks that were previously unatainable for LLM a breeze. I strongly believe now that processing tasks into a “coding-friendly” format will be a continuous vector of breakthrough for the months to come. check out the full interview below and big thanks to everyone that submitted questions was an awesome session! table of content: * 0:00 - RLM are compositional generalizers * 4:40 - overview of what is an RLM * 10:00 - updated info and experiments on RLM * 22:40 - Big Boss Alex L. Zhang * 27:45 - what is “locally in-distribution”? * 34:00 - do we even need long context inside an LLM? * 36:50 - yacine is rambling again… * 39:35 - is locally in-distribution computation enough for scientific breakthrough * 46:40 - why are transformers poor at compositional generalization * 52:50 - what is missing in this soup of capability from the transformers side. * 57:25 - [at gun point] improve the harness or train a new model for compositional generalization??? * 1:03:00 - how early RLM should enter the training pipeline? * 1:07:00 - how do you validate such a major architectural change btw? * 1:10:00 - what does Alex thinks about long-running persistent ai agents? * 1:14:00 - is code enough for compositional generalization across modalities? * 1:18:52 - how are we doing at the frontier for long horizon tasks? * 1:22:00 - speculative programmatic tool call (sPTC) * 1:28:00 - does the RLM impose an abstraction the model should learn for itself? * 1:32:00 - what’s next for Big Boss Alex L. Zhang (and what’s not next)? Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    RLMs are Compositional Generalizers with Alex Zhang from MIT

  6. Oct 5

    Inside a Frontier Open Model: Nemotron 3 Ultra Explained ft. NVIDIA's Chris Alexiuk

    open models are infinitely interesting to me. being able to peek under the hood and look at the interconnected pieces that each hums and work in a specific way to feed the whole is a blessing. I had a fantastic time earlier this year chatting with chris about the nemotron family of models and why they are engineered the way they are. throughout the discussion it became clear all decisions surround nemotron 3 ultra was about making it able to be fast fast and handle long context efficiently. huge thanks to the nemotron team for putting their stuff out there like that! ❤️ Table of Content * 0:00 : nemotron 3 ultra ethos * 3:42 : Chris the Senior Product Research Engineer (phew) * 6:10 : Nemotron Lab & the Coalition * 9:10 : the open model ecosystem party 🤗 * 12:50 : “a faster model is a smarter model” * 20:23 : intro to nemotron family * 38:18 : infrastructure issues on the research side for these type of models? * 42:30 : main design choices that sets nemotron 3 ultra apart? * 48:10 : long context inside the models or inside the harness? * 57:35 : Latent MoE tradeoff or is it a free 🥪 * 1:01:14 : very aggressive grouped query attention why? * 1:08:00 : what in the structure of nvidia enabled this whole thing? * 1:11:36 : advice for undergrad that want to work on this stuff? * 1:15:45 : how does the shared weight MTP work? * 1:19:20 : main idea behind MOPD? * 1:24:55 : what’s next for nemotron scaling? * 1:26:52 : any weird reward hacking stories in nemotron? * 1:30:30 : what’s next for nemotron ++++? Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    Inside a Frontier Open Model: Nemotron 3 Ultra Explained ft. NVIDIA's Chris Alexiuk

  7. Sep 9

    How to Generate Synthetic Pretraining Data? with Joël Niklaus from Hugginface

    today we are taking a deeper look at of the secret behind LLM pretraining which is synthetic data pipelines with Joel Niklaus machine learning engineer at Hugging Face!joel an his team ran 90 controlled experiments and burned over a trillion tokens to figure out what actually makes good pretraining data and in huggingface fashion provided all of their findings/artifacts openly! lots of things to learn from experiment like this and to be honest there is still so much more to do! couple of useful links: 👉Joel Website: https://niklaus.ai👉 Twitter: https://x.com/joelniklaus👉 Paper : arXiv: https://arxiv.org/abs/2604.13977👉 Blog: https://huggingface.co/spaces/HuggingFaceFW/finephraseAlso check out Arcee (American open research lab and sponsor of this interview):♥️ https://www.arcee.ai/Also trinity large thinking is available on open router:♥️ https://openrouter.ai/arcee-ai/trinity-large-thinking#providers♥️ and check out the NAC harness https://www.arcee.ai/blog/nac on the menu: - 0:00 : synthetic data in pre-training LLM - 3:36 : Joel Niklau ML Research Engineer at Hugging Face - 4:50 : how to approach research problems at the edge? - 7:26 : the label here is still wrong is should be “benchmarking work overview” - 9:10 : what are the first few steps to get started building a private benchmark. - 11:18 : synthetic data pipeline and it’s impact in benchmark making - 13:19 : Synthetic Data Playbook Presentation Overview - 19:30 : what is rephrasing in synthetic data? - 21:05 : why are large scale ablation studies on data pipeline becoming feasible now? - 24:00 : how is rephrasing done? - 25:42 : experimental results - 27:58 : why does synthetic restructuring hurt common-sense knowledge? - 30:10 : what’s the value prop behind doing rephrasing? - 32:16 : is math a type of rephrasing? - 38:02 : what’s up with closeness between generator vs the student? - 39:40 : synthetic data + mixing real data - 41:20 : what drive this real-to-synthetic ratio? - 43:50 : what other structured formats could be explored for synthetic data? - 45:20 : is the order of datasets used for pretraining also affecting the distribution? - 46:40 : would adding multilinguality be considered a form of rephrasing for LLM? - 50:40 : which non-synthetic dataset should you mix in? - 1:03:20 : do model-specific writing habits survive the rephrasing process? + sawing off the edges? - 1:08:30 : would their be a gain to have a purpose built 270M model for being a generator? - 1:14:30 : does generator diversity still help as the student model scales? - 1:26:30 : ❤️❤️❤️ - 1:27:15 : useful of synthetic data for post-training? - 1:28:50 : what’s next for this direction and for the field? btw: do follow joel he is a fantastic scientist and a very good educator! Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    How to Generate Synthetic Pretraining Data? with Joël Niklaus from Hugginface

  8. Aug 12

    Pretrained LLMs Are Surrounded by Task Experts feat Yulu Gan from MIT

    sometime there is an idea that just makes other weird research results make so much more sense and sort of cascade to explain a whole bunch of phenomenon. neural thicket from the great yulu gan is one such paper that I absolutely loved from start to finish. the old mental model I had for pre-trained neural networked was that the pre-training regime made them land on a single good set of weight that was optimal for many tasks. but after going through the paper "Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights" my view has been changed. the better idea seems to be that the model after pre-training is in a basin where it is surrounded by more specialized expert at varying distance from that set of weights which the authors calls thicket. a bit like the valley surrounded by mountains. now depending on whether the model has big or small capacity this density of thicket will be varying and these expert weights will also have varying level of difficulty to be reached via post-training. this finding elegantly presented by Yulu & co. has far reaching consequence on how we should think about pre-training and it's connection with post-training models (and even distillation / continual learning) very densely packed interview in there and there is a few open threads that are ripe to be picked up if you are interested in this research direction! ps: my favorite bit is the connection with the baldwin effect in biology which is an evolutionary concept proposing that learned behaviors can shape the course of evolution, very fitting with post-training dynamics! Table of Content: 00:00:00 - pre-trained LLM are surrounded by tasks experts 00:05:35 - introduction to the great yulu gan and his beautiful paper 00:08:21 - what are neural thickets? 00:13:25 - relationship between scale and neural thickets? 00:14:56 - why it’s happening for the pretrained base model? 00:18:00 - solution density for small and large model 00:19:30 - do all perturbation help in the same way? and why? 00:25:00 - why does mixed data pretraining create neural thickets? 00:30:20 - would a distill model exhibit a different kind of distribution landscape? 00:32:50 - RandOpt algorithm 00:36:50 - where are the gains coming from with adding random noise? 00:45:45 - what happens to the distribution of the weight when doing the distillation? 00:48:55 - does the useful perturbation of the layers evenly distributed? 00:51:40 - how should we releasing model with a weights distribution perspective? 00:53:56 - does routing work better than distilling you think? 00:56:10 - what do you think will happens at 400B? 00:58:10 - how does this connect to continual learning? 01:02:20 - connection with the baldwin effect from biology? 01:06:30 - discover novelty by random exploration of the weights? 01:09:00 - what’s next for the great yulu gan enjoy folks hope this stuff is useful🌹 Get full access to learning with yacine at www.yacinemahdid.com/subscribe

    Pretrained LLMs Are Surrounded by Task Experts feat Yulu Gan from MIT
See All (13)

About

a show about untangling the complexity of deep learning and the brain by interviewing smart researchers and builders in the field of intelligence! ideal for scholars, industry practitioners or toddlers. www.yacinemahdid.com

Information

  • Creator
    yacine mahdid
  • Years Active
    2026
  • Episodes
    13
  • Rating
    Explicit
  • Copyright
    © Yacine Mahdid
  • Show Website
    learning with yacine show

You Might Also Like