Critical Point War Stories: Software Engineers Talk Breaking Production Critical Point
-
- Technology
-
Building better software, one incident at a time. We talk with software engineers about “that time we broke production.”
This is the audio version of the podcast. Watch on YouTube: https://youtube.com/@critical-point
Produced by Complex Systems Group (https://complexsystems.group). Part of Critical Point (https://criticalpoint.tv).
-
Was Akamai the Velvet Underground of SRE? - Abhijit Mehta - Bonus
Outtake from https://warstories.criticalpoint.tv/episodes/he-broke-turkish-twitter-abhijit-mehta
Abhijit's Twitter: https://twitter.com/AbhijitCMehtaAbhijit's LinkedIn: https://www.linkedin.com/in/mehtaabhijit/
To watch the video version of the podcast on YouTube, check out the channel page at https://youtube.com/@critical-point
Credits:
Emoji artwork is provided by Emojitwo (https://emojitwo.github.io/), originally released as Emojione 2.2 (https://www.emojione.com/) by Ranks.com (http://www.ranks.com/) with contributions from the Emojitwo community and is licensed under CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode).
Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC. All Rights Reserved.
Produced by Complex Systems Group LLC (https://complexsystems.group). -
He Broke Turkish Twitter - Abhijit Mehta
Abhijit Mehta tells how one tiny dot made almost made a massive difference to Twitter's Turkish ad revenue.
Abhijit's Twitter: https://twitter.com/AbhijitCMehtaAbhijit's LinkedIn: https://www.linkedin.com/in/mehtaabhijit/
Chapters:
(00:00) - Intro
(01:25) - How did you find yourself in a place where you could break production?
(08:53) - Akamai & the 'aha' moments of product development
(15:30) - Google & the Google culture
(17:35) - Twitter & what's the difference between brand ads and direct-response ads?
(23:58) - How Twitter made its money
(27:58) - Re-architecting the fundamental ad platform at Twitter
(32:35) - "We had to be really careful!"
(34:48) - "There's something really weird happening in Turkey"
(37:49) - The meaning of scale
(41:02) - "Well we'd thought of all the horses, so we started looking for zebras"
(43:48) - "So here's what happened"
(49:30) - "So this will work fine almost everywhere"
(53:51) - How do you roll things like this out safely?
(59:56) - "In general I'm an optimist, there's exciting things to come"
(01:03:52) - Thanks & CTAs
Credits:Emoji artwork is provided by Emojitwo (https://emojitwo.github.io/), originally released as Emojione 2.2 (https://www.emojione.com/) by Ranks.com (http://www.ranks.com/) with contributions from the Emojitwo community and is licensed under CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode).
Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC. All Rights Reserved.
Produced by Complex Systems Group LLC (https://complexsystems.group). -
She Missed the Overnight Push! - Julia Lunetta
Julia Lunetta tells us how an ostensibly minor change caused her employer, a financial institution (Upromise Investments), to miss its overnight reconciliation with its banks.
Credits:
Emoji artwork is provided by Emojitwo (https://emojitwo.github.io/), originally released as Emojione 2.2 (https://www.emojione.com/) by Ranks.com (http://www.ranks.com/) with contributions from the Emojitwo community and is licensed under CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode).
Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC. All Rights Reserved.
Produced by Complex Systems Group LLC (https://complexsystems.group). -
She Made Dropbox Give the Internet Free Watermelon? - Kate Rudolph
Kate Rudolph tells how a rolled back commit on her intern project resulted in Dropbox, her employer, offering free watermelon to the entire Internet.
Credits:
Emoji artwork is provided by Emojitwo (https://emojitwo.github.io/), originally released as Emojione 2.2 (https://www.emojione.com/) by Ranks.com (http://www.ranks.com/) with contributions from the Emojitwo community and is licensed under CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode).
Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC. All Rights Reserved. -
He Locked Players Out On Launch Day - Zachery Johnson
Zachery Johnson tells how an innocent command doomed the database servers and locked players out of Diablo II: Resurrected on Day 2 of launch.
Blizzard's blog post on the incident: https://us.forums.blizzard.com/en/d2r/t/diablo-ii-resurrected-outages-an-explanation-how-we%E2%80%99ve-been-working-on-it-and-how-we%E2%80%99re-moving-forward/28164
Credits:
Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC. All Rights Reserved. -
Twelve Lines of Code Took Down the Whole Company - Patrick O'Doherty
Patrick O'Doherty (https://patrickod.computer) tells how a seemingly-simple configuration change one Friday morning led to a massive outage.
Credits:
Intro & outro - "Senpai Funk" by Paul T. Starr. Used with permission.https://soundcloud.com/ptstarr
Critical Point logo and wordmark designed by Timothy James Lopez.https://www.behance.net/lopeztimes
Critical Point logo and wordmark copyright ⓒ 2020 Complex Systems Group, LLC. All Rights Reserved.