Relentless Optimism: The Future of Data Engineering with Brad Ediger

Kicking off a new series on all things data, Jerome talks with Brad Ediger, 8th Light's Head of Technology, about the state of data engineering, and takes the opportunity to scratch beneath the surface of Brad’s notorious optimism. They also dive into common challenges businesses face when dealing with data before doing a bit of time-traveling—looking at what’s possible today that was unthinkable 10 years ago, and speculating on where the industry might be a decade from now. 

• (01:59) - Brad’s background
• (03:41) - Optimistic perspective
• (06:41) - Challenge: Legacy System Maintenance
• (08:55) - Challenge: Lack of Standardization
• (11:30) - Challenge: Change Management
• (13:04) - Conway’s Law
• (14:26) - Challenge: Data Provenance
• (15:53) - Advances: Cloud Offerings
• (19:05) - Advances: DevOps Practices
• (20:32) - Future Predictions:  Immutable Data & Abstractions
• (23:57) - Future Predictions: Database Models
• (29:23) - Later Discussions: Compliance & Risk Management
• (30:38) - Later Discussions: Machine Learning

Brad Ediger has been a staple in the Chicago tech scene since 2005. As 8th Light’s Head of Technology, his focus is on growing 8th Light's profile as a software consultancy. He joined 8th Light in 2019 when he merged his independent consultancy with the company.

Brad specializes in connecting subject-matter experts with the tools they need to work more effectively — as exemplified by his success in supporting an e-commerce startup as they grew from five to 150 employees and through multiple rounds of investment. This includes providing wraparound software support, developing decision-support software, and using analytic data to optimize inventory dynamics.

In 2007, early in the life of the Ruby on Rails web framework, Brad authored Advanced Rails, a book that goes in-depth on ways for developers to use Rails to build applications that scale.

Jerome Goodrich leads amazing software teams to design and develop thoughtful solutions to complex problems as a principal software crafter at 8th Light. He loves pairing strenuous hikes with deep conversations and is always trying to see things clearly and with an open heart. Jerome lives much of his life off of the internet, but he occasionally writes on his website.

You can listen to Collaborative Craft via Apple Podcasts, Google Podcasts, Spotify, or wherever you get your podcasts. Don’t forget to subscribe to receive notifications of the latest episodes. And if you enjoy what you’re hearing, please consider leaving a review in Apple Podcasts. It really helps others find the show. 

This podcast was produced in partnership with Dante32.