Data Science Decoded

Mike E
  • 5.0 (1)
  • MATHEMATICS
  • UPDATED WEEKLY
Data Science Decoded

We discuss seminal mathematical papers (sometimes really old 😎 ) that have shaped and established the fields of machine learning and data science as we know them today. The goal of the podcast is to introduce you to the evolution of these fields from a mathematical and slightly philosophical perspective. We will discuss the contribution of these papers, not just from pure a math aspect but also how they influenced the discourse in the field, which areas were opened up as a result, and so on. Our podcast episodes are also available on our youtube: https://youtu.be/wThcXx_vXjQ?si=vnMfs

  1. 1 DAY AGO

    Data Science #24 - The Expectation Maximization (EM) algorithm Paper review (1977)

    At the 24th episode we go over the paper titled: Dempster, Arthur P., Nan M. Laird, and Donald B. Rubin. "Maximum likelihood from incomplete data via the EM algorithm." Journal of the royal statistical society: series B (methodological) 39.1 (1977): 1-22. The Expectation-Maximization (EM) algorithm is an iterative method for finding Maximum Likelihood Estimates (MLEs) when data is incomplete or contains latent variables. It alternates between the E-step, where it computes the expected value of the missing data given current parameter estimates, and the M-step, where it maximizes the expected complete-data log-likelihood to update the parameters. This process repeats until convergence, ensuring a monotonic increase in the likelihood function. EM is widely used in statistics and machine learning, especially in Gaussian Mixture Models (GMMs), hidden Markov models (HMMs), and missing data imputation. Its ability to handle incomplete data makes it invaluable for problems in clustering, anomaly detection, and probabilistic modeling. The algorithm guarantees stable convergence, though it may reach local maxima, depending on initialization. In modern data science and AI, EM has had a profound impact, enabling unsupervised learning in natural language processing (NLP), computer vision, and speech recognition. It serves as a foundation for probabilistic graphical models like Bayesian networks and Variational Inference, which power applications such as chatbots, recommendation systems, and deep generative models. Its iterative nature has also inspired optimization techniques in deep learning, such as Expectation-Maximization inspired variational autoencoders (VAEs), demonstrating its ongoing influence in AI advancements.

    33 min

  2. 14 JAN

    Data Science #23- The Markov Chain Monte Carl MCMC Paper review (1953)

    In the 23rd episode we review the The 1953 paper Metropolis, Nicholas, et al. "Equation of state calculations by fast computing machines." The journal of chemical physics 21.6 (1953): 1087-1092 which introduced the Monte Carlo method for simulating molecular systems, particularly focusing on two-dimensional rigid-sphere models. The study used random sampling to compute equilibrium properties like pressure and density, demonstrating a feasible approach for solving analytically intractable statistical mechanics problems. The work pioneered the Metropolis algorithm, a key development in what later became known as Markov Chain Monte Carlo (MCMC) methods. By validating the Monte Carlo technique against free volume theories and virial expansions, the study showcased its accuracy and set the stage for MCMC as a powerful tool for exploring complex probability distributions. This breakthrough has had a profound impact on modern AI and ML, where MCMC methods are now central to probabilistic modeling, Bayesian inference, and optimization. These techniques enable applications like generative models, reinforcement learning, and neural network training, supporting the development of robust, data-driven AI systems. Youtube: https://www.youtube.com/watch?v=gWOawt7hc88&t

    38 min

  3. 7 JAN

    Data Science #22 - The theory of dynamic programming, Paper review 1954

    We review Richard Bellman's "The Theory of Dynamic Programming" paper from 1954 which revolutionized how we approach complex decision-making problems through two key innovations. First, his Principle of Optimality established that optimal solutions have a recursive structure - each sub-decision must be optimal given the state resulting from previous decisions. Second, he introduced the concept of focusing on immediate states rather than complete historical sequences, providing a practical way to tackle what he termed the "curse of dimensionality." These foundational ideas directly shaped modern artificial intelligence, particularly reinforcement learning. The mathematical framework Bellman developed - breaking complex problems into smaller, manageable subproblems and making decisions based on current state - underpins many contemporary AI achievements, from game-playing agents like AlphaGo to autonomous systems and robotics. His work essentially created the theoretical backbone that enables modern AI systems to handle sequential decision-making under uncertainty. The principles established in this 1954 paper continue to influence how we design AI systems today, particularly in reinforcement learning and neural network architectures dealing with sequential decision problems.

    48 min

  4. 25/12/2024

    Data Science #21 - Steps Toward Artificial Intelligence

    In the 1st episode of the second season we review the legendary Marvin Minsky's "Steps Toward Artificial Intelligence" from 1961. Itis a foundational work in the field of AI that outlines the challenges and methodologies for developing intelligent problem-solving systems. The paper categorizes AI challenges into five key areas: Search, Pattern Recognition, Learning, Planning, and Induction. It emphasizes how computers, limited by their ability to perform only programmed actions, can enhance problem-solving efficiency through heuristic methods, learning from patterns, and planning solutions to narrow down possible options. The significance of this work lies in its conceptual framework, which established a systematic approach to AI development. Minsky highlighted the need for machines to mimic cognitive functions like recognizing patterns and learning from experience, which form the basis of modern machine learning algorithms. His emphasis on heuristic methods provided a pathway to make computational processes more efficient and adaptive by reducing exhaustive searches and using past data to refine problem-solving strategies. The paper is pivotal as it set the stage for advancements in AI by introducing the integration of planning, adaptive learning, and pattern recognition into computational systems. Minsky's insights continue to influence AI research and development, including neural networks, reinforcement learning, and autonomous systems, bridging theoretical exploration and practical applications in the quest for artificial intelligence.

    1 hr

  5. 09/12/2024 · BONUS

    Data Science #20 - the Rao-Cramer bound (1945)

    In the 20th episode, we review the seminal paper by Rao which introduced the Cramer Rao bound: Rao, Calyampudi Radakrishna (1945). "Information and the accuracy attainable in the estimation of statistical parameters". Bulletin of the Calcutta Mathematical Society. 37. Calcutta Mathematical Society: 81–89. The Cramér-Rao Bound (CRB) sets a theoretical lower limit on the variance of any unbiased estimator for a parameter. It is derived from the Fisher information, which quantifies how much the data tells us about the parameter. This bound provides a benchmark for assessing the precision of estimators and helps identify efficient estimators that achieve this minimum variance. The CRB connects to key statistical concepts we have covered previously: Consistency: Estimators approach the true parameter as the sample size grows, ensuring they become arbitrarily accurate in the limit. While consistency guarantees convergence, it does not necessarily imply the estimator achieves the CRB in finite samples. Efficiency: An estimator is efficient if it reaches the CRB, minimizing variance while remaining unbiased. Efficiency represents the optimal use of data to achieve the smallest possible estimation error. Sufficiency: Working with sufficient statistics ensures no loss of information about the parameter, increasing the chances of achieving the CRB. Additionally, the CRB relates to KL divergence, as Fisher information reflects the curvature of the likelihood function and the divergence between true and estimated distributions. In modern DD and AI, the CRB plays a foundational role in uncertainty quantification, probabilistic modeling, and optimization. It informs the design of Bayesian inference systems, regularized estimators, and gradient-based methods like natural gradient descent. By highlighting the tradeoffs between bias, variance, and information, the CRB provides theoretical guidance for building efficient and robust machine learning models

    1 hr

  6. 02/12/2024

    Data Science #19 - The Kullback–Leibler divergence paper (1951)

    In this episode with go over the Kullback-Leibler (KL) divergence paper, "On Information and Sufficiency" (1951). It introduced a measure of the difference between two probability distributions, quantifying the cost of assuming one distribution when another is true. This concept, rooted in Shannon's information theory (which we reviewed in previous episodes), became fundamental in hypothesis testing, model evaluation, and statistical inference. KL divergence has profoundly impacted data science and AI, forming the basis for techniques like maximum likelihood estimation, Bayesian inference, and generative models such as variational autoencoders (VAEs). It measures distributional differences, enabling optimization in clustering, density estimation, and natural language processing. In AI, KL divergence ensures models generalize well by aligning training and real-world data distributions. Its role in probabilistic reasoning and adaptive decision-making bridges theoretical information theory and practical machine learning, cementing its relevance in modern technologies.

    53 min

  7. 25/11/2024

    Data Science #18 - The k-nearest neighbors algorithm (1951)

    In the 18th episode we go over the original k-nearest neighbors algorithm; Fix, Evelyn; Hodges, Joseph L. (1951). Discriminatory Analysis. Nonparametric Discrimination: Consistency Properties USAF School of Aviation Medicine, Randolph Field, Texas They introduces a nonparametric method for classifying a new observation 𝑧 z as belonging to one of two distributions, 𝐹 F or 𝐺 G, without assuming specific parametric forms. Using 𝑘 k-nearest neighbor density estimates, the paper implements a likelihood ratio test for classification and rigorously proves the method's consistency. The work is a precursor to the modern 𝑘 k-Nearest Neighbors (KNN) algorithm and established nonparametric approaches as viable alternatives to parametric methods. Its focus on consistency and data-driven learning influenced many modern machine learning techniques, including kernel density estimation and decision trees. This paper's impact on data science is significant, introducing concepts like neighborhood-based learning and flexible discrimination. These ideas underpin algorithms widely used today in healthcare, finance, and artificial intelligence, where robust and interpretable models are critical.

    44 min

  8. 18/11/2024

    Data Science #17 - The Monte Carlo Algorithm (1949)

    We review the original Monte Carlo paper from 1949 by Metropolis, Nicholas, and Stanislaw Ulam. "The monte carlo method." Journal of the American statistical association 44.247 (1949): 335-341. The Monte Carlo method uses random sampling to approximate solutions for problems that are too complex for analytical methods, such as integration, optimization, and simulation. Its power lies in leveraging randomness to solve high-dimensional and nonlinear problems, making it a fundamental tool in computational science. In modern data science and AI, Monte Carlo drives key techniques like Bayesian inference (via MCMC) for probabilistic modeling, reinforcement learning for policy evaluation, and uncertainty quantification in predictions. It is essential for handling intractable computations in machine learning and AI systems. By combining scalability and flexibility, Monte Carlo methods enable breakthroughs in areas like natural language processing, computer vision, and autonomous systems. Its ability to approximate solutions underpins advancements in probabilistic reasoning, decision-making, and optimization in the era of AI and big data.

    38 min
See All (25)

About

We discuss seminal mathematical papers (sometimes really old 😎 ) that have shaped and established the fields of machine learning and data science as we know them today. The goal of the podcast is to introduce you to the evolution of these fields from a mathematical and slightly philosophical perspective. We will discuss the contribution of these papers, not just from pure a math aspect but also how they influenced the discourse in the field, which areas were opened up as a result, and so on. Our podcast episodes are also available on our youtube: https://youtu.be/wThcXx_vXjQ?si=vnMfs

Information

  • Creator
    Mike E
  • Years Active
    2024 - 2025
  • Episodes
    25
  • Rating
    Clean
  • Copyright
    © Mike E
  • Show Website
    Data Science Decoded

You Might Also Like

To listen to explicit episodes, sign in.

Stay up to date with this show

Sign in or sign up to follow shows, save episodes and get the latest updates.
Select a country or region

Africa, Middle East, and India

Asia Pacific

Europe

Latin America and the Caribbean

The United States and Canada