7 episodes

Building better software, one incident at a time. We talk with software engineers about “that time we broke production.”

This is the audio version of the podcast. Watch on YouTube: https://youtube.com/@critical-point

Produced by Complex Systems Group (https://complexsystems.group). Part of Critical Point (https://criticalpoint.tv).

Critical Point War Stories: Software Engineers Talk Breaking Production Critical Point

    • Technology
    • 5.0 • 1 Rating

Building better software, one incident at a time. We talk with software engineers about “that time we broke production.”

This is the audio version of the podcast. Watch on YouTube: https://youtube.com/@critical-point

Produced by Complex Systems Group (https://complexsystems.group). Part of Critical Point (https://criticalpoint.tv).

    Was Akamai the Velvet Underground of SRE? - Abhijit Mehta - Bonus

    Was Akamai the Velvet Underground of SRE? - Abhijit Mehta - Bonus

    Outtake from https://warstories.criticalpoint.tv/episodes/he-broke-turkish-twitter-abhijit-mehta
    Abhijit's Twitter: https://twitter.com/AbhijitCMehtaAbhijit's LinkedIn: https://www.linkedin.com/in/mehtaabhijit/
    To watch the video version of the podcast on YouTube, check out the channel page at https://youtube.com/@critical-point
    Credits:
    Emoji artwork is provided by Emojitwo (https://emojitwo.github.io/), originally released as Emojione 2.2 (https://www.emojione.com/) by Ranks.com (http://www.ranks.com/) with contributions from the Emojitwo community and is licensed under CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode).
    Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
    Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
    Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC.  All Rights Reserved.
    Produced by Complex Systems Group LLC (https://complexsystems.group).

    • 12 min
    He Broke Turkish Twitter - Abhijit Mehta

    He Broke Turkish Twitter - Abhijit Mehta

    Abhijit Mehta tells how one tiny dot made almost made a massive difference to Twitter's Turkish ad revenue.
    Abhijit's Twitter: https://twitter.com/AbhijitCMehtaAbhijit's LinkedIn: https://www.linkedin.com/in/mehtaabhijit/
    Chapters:
    (00:00) - Intro

    (01:25) - How did you find yourself in a place where you could break production?

    (08:53) - Akamai & the 'aha' moments of product development

    (15:30) - Google & the Google culture

    (17:35) - Twitter & what's the difference between brand ads and direct-response ads?

    (23:58) - How Twitter made its money

    (27:58) - Re-architecting the fundamental ad platform at Twitter

    (32:35) - "We had to be really careful!"

    (34:48) - "There's something really weird happening in Turkey"

    (37:49) - The meaning of scale

    (41:02) - "Well we'd thought of all the horses, so we started looking for zebras"

    (43:48) - "So here's what happened"

    (49:30) - "So this will work fine almost everywhere"

    (53:51) - How do you roll things like this out safely?

    (59:56) - "In general I'm an optimist, there's exciting things to come"

    (01:03:52) - Thanks & CTAs
    Credits:Emoji artwork is provided by Emojitwo (https://emojitwo.github.io/), originally released as Emojione 2.2 (https://www.emojione.com/) by Ranks.com (http://www.ranks.com/) with contributions from the Emojitwo community and is licensed under CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode).
    Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
    Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
    Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC.  All Rights Reserved.
    Produced by Complex Systems Group LLC (https://complexsystems.group).

    • 1 hr 5 min
    She Missed the Overnight Push! - Julia Lunetta

    She Missed the Overnight Push! - Julia Lunetta

    Julia Lunetta tells us how an ostensibly minor change caused her employer, a financial institution (Upromise Investments), to miss its overnight reconciliation with its banks.
    Credits:
    Emoji artwork is provided by Emojitwo (https://emojitwo.github.io/), originally released as Emojione 2.2 (https://www.emojione.com/) by Ranks.com (http://www.ranks.com/) with contributions from the Emojitwo community and is licensed under CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode).
    Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
    Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
    Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC.  All Rights Reserved.
    Produced by Complex Systems Group LLC (https://complexsystems.group).

    • 52 min
    She Made Dropbox Give the Internet Free Watermelon? - Kate Rudolph

    She Made Dropbox Give the Internet Free Watermelon? - Kate Rudolph

    Kate Rudolph tells how a rolled back commit on her intern project resulted in Dropbox, her employer, offering free watermelon to the entire Internet.
    Credits:
    Emoji artwork is provided by Emojitwo (https://emojitwo.github.io/), originally released as Emojione 2.2 (https://www.emojione.com/) by Ranks.com (http://www.ranks.com/) with contributions from the Emojitwo community and is licensed under CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode).
    Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
    Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
    Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC.  All Rights Reserved.

    • 54 min
    He Locked Players Out On Launch Day - Zachery Johnson

    He Locked Players Out On Launch Day - Zachery Johnson

    Zachery Johnson tells how an innocent command doomed the database servers and locked players out of Diablo II: Resurrected on Day 2 of launch.
    Blizzard's blog post on the incident: https://us.forums.blizzard.com/en/d2r/t/diablo-ii-resurrected-outages-an-explanation-how-we%E2%80%99ve-been-working-on-it-and-how-we%E2%80%99re-moving-forward/28164
    Credits:
    Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
    Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
    Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC.  All Rights Reserved.

    • 1 hr 24 min
    Twelve Lines of Code Took Down the Whole Company - Patrick O'Doherty

    Twelve Lines of Code Took Down the Whole Company - Patrick O'Doherty

    Patrick O'Doherty (https://patrickod.computer) tells how a seemingly-simple configuration change one Friday morning led to a massive outage.
    Credits:
    Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
    Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
    Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC.  All Rights Reserved.

    • 1 hr 1 min

Customer Reviews

5.0 out of 5
1 Rating

1 Rating

Top Podcasts In Technology

No Priors: Artificial Intelligence | Technology | Startups
Conviction | Pod People
Lex Fridman Podcast
Lex Fridman
All-In with Chamath, Jason, Sacks & Friedberg
All-In Podcast, LLC
Hard Fork
The New York Times
Acquired
Ben Gilbert and David Rosenthal
The Neuron: AI Explained
The Neuron