22 min

95: Data Science Pipeline Testing with Great Expectations - Abe Gong Python Test

    • Technology

Data science and machine learning are affecting more of our lives every day. Decisions based on data science and machine learning are heavily dependent on the quality of the data, and the quality of the data pipeline.


Some of the software in the pipeline can be tested to some extent with traditional testing tools, like pytest.


But what about the data? The data entering the pipeline, and at various stages along the pipeline, should be validated.


That's where pipeline tests come in.


Pipeline tests are applied to data. Pipeline tests help you guard against upstream data changes and monitor data quality.


Abe Gong and Superconductive are building an open source project called Great Expectations. It's a tool to help you build pipeline tests.


This is quite an interesting idea, and I hope it gains traction and takes off.
Special Guest: Abe Gong.
Sponsored By:
Raygun: Detect, diagnose, and destroy Python errors that are affecting your customers. With smart Python error monitoring software from Raygun.com, you can be alerted to issues affecting your users the second they happen.Links:
Great Expectations

Data science and machine learning are affecting more of our lives every day. Decisions based on data science and machine learning are heavily dependent on the quality of the data, and the quality of the data pipeline.


Some of the software in the pipeline can be tested to some extent with traditional testing tools, like pytest.


But what about the data? The data entering the pipeline, and at various stages along the pipeline, should be validated.


That's where pipeline tests come in.


Pipeline tests are applied to data. Pipeline tests help you guard against upstream data changes and monitor data quality.


Abe Gong and Superconductive are building an open source project called Great Expectations. It's a tool to help you build pipeline tests.


This is quite an interesting idea, and I hope it gains traction and takes off.
Special Guest: Abe Gong.
Sponsored By:
Raygun: Detect, diagnose, and destroy Python errors that are affecting your customers. With smart Python error monitoring software from Raygun.com, you can be alerted to issues affecting your users the second they happen.Links:
Great Expectations

22 min

Top Podcasts In Technology

Lex Fridman Podcast
Lex Fridman
All-In with Chamath, Jason, Sacks & Friedberg
All-In Podcast, LLC
No Priors: Artificial Intelligence | Machine Learning | Technology | Startups
Conviction | Pod People
BG2Pod with Brad Gerstner and Bill Gurley
BG2Pod
Acquired
Ben Gilbert and David Rosenthal
Hard Fork
The New York Times