Scaling Platform Engineering: Shopify’s Blueprint - OpenObservability Talks S4E08

In this episode, join us as we delve into the intricate world of Platform Engineering with Aparna Subramanian, Director of Production Engineering at Shopify. Discover how Shopify, a powerhouse in e-commerce, masters the art of scaling platform engineering. Gain invaluable insights into their strategies, innovations, and lessons learned while navigating the complexities of sustaining and evolving a robust infrastructure to support millions, even through special peak events like Black Friday and Cyber Monday. If you're keen on understanding the backbone of a thriving online platform, don’t miss out on this episode.

Aparna started her career as a Software Engineer and has spent most part of her almost two decades of technology experience specializing in Infrastructure and Data Platforms. In her current role she leads Shopify’s Cloud Native Production Platform.

Previously, she was Director of Engineering at VMware where she was a founding member of Tanzu on vSphere, a Kubernetes Platform for the hybrid cloud. She also serves as co-chair of the “CNCF End User Developer Experience” SIG and as member of the CNCF End user technical advisory board.

The episode was live-streamed on 11 January 2024 and the video is available at https://www.youtube.com/watch?v=6ShtsTTUizI

OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube.

We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.

https://www.youtube.com/@openobservabilitytalks  

⁠https://www.twitch.tv/openobservability

Show Notes:

00:00 - Show intro & 2023 stats

01:49 - Episode and guest intro

04:15 - Shopify’s scale

06:09 - Shopify’s journey to Platform Engineering

08:56 - Shopify’s platform structure

11:49 - division of responsibility

13:51 - golden path vs flexibility

17:58 - balancing flexibility and abstraction

19:56 - platform group structure

23:28 - handling load spikes

28:55 - FinOps in Platform Engineering

38:38 - avoiding silos and the cultural aspect

41:13 - CNCF end-user SIG and community challenges

49:24 - KubeCon Paris and guest contact 

51:03 - OpenTofu reached GA

53:33 - Isovalent acquired by Cisco

55:00 - year-end summary articles

57:07 - .NET Aspire released preview2

58:58 - Episode and show outro

Resources:

Shopify Engineering Blog https://shopify.engineering/

Performance wins at Shopify: https://www.shopify.com/news/performance%F0%9F%91%86-complexity%F0%9F%91%87-killer-updates-from-shopify-engineering

CNCF End User SIG https://github.com/cncf/enduser-public

OpenTofu has reached GA https://logz.io/blog/terraform-is-no-longer-open-source-is-opentofu-opentf-the-successor/?utm_source=devrel&utm_medium=devrel

Observability in 2024: https://thenewstack.io/observability-in-2024-more-opentelemetry-less-confusion/

OpenTelemetry in 2024: https://www.apmdigest.com/2024-