Efficient Deployment of Models at the Edge // Krishna Sridhar // #284

Krishna Sridhar is an experienced engineering leader passionate about building wonderful products powered by machine learning.

Efficient Deployment of Models at the Edge // MLOps Podcast #284 with Krishna Sridhar, Vice President of Qualcomm.

Big shout-out to Qualcomm for sponsoring this episode!

// Abstract

Qualcomm® AI Hub helps to optimize, validate, and deploy machine learning models on-device for vision, audio, and speech use cases. With Qualcomm® AI Hub, you can: Convert trained models from frameworks like PyTorch and ONNX for optimized on-device performance on Qualcomm® devices.

Profile models on-device to obtain detailed metrics, including runtime, load time, and compute unit utilization. Verify numerical correctness by performing on-device inference. Easily deploy models using Qualcomm® AI Engine Direct, TensorFlow Lite, or ONNX Runtime.

The Qualcomm® AI Hub Models repository contains a collection of example models that use Qualcomm® AI Hub to optimize, validate, and deploy models on Qualcomm® devices. Qualcomm® AI Hub automatically handles model translation from source framework to device runtime, applying hardware-aware optimizations, and performs physical performance/numerical validation. The system automatically provisions devices in the cloud for on-device profiling and inference. The following image shows the steps taken to analyze a model using Qualcomm® AI Hub.

// Bio

Krishna Sridhar leads engineering for Qualcomm™ AI Hub, a system used by more than 10,000 AI developers spanning 1,000 companies to run more than 100,000 models on Qualcomm platforms. Prior to joining Qualcomm, he was Co-founder and CEO of Tetra AI, which made it easy to efficiently deploy ML models on mobile/edge hardware. Prior to Tetra AI, Krishna helped design Apple's CoreML, which was a software system mission-critical to running several experiences at Apple, including Camera, Photos, Siri, FaceTime, Watch, and many more across all major Apple device operating systems and all hardware and IP blocks. He has a Ph.D. in computer science from the University of Wisconsin-Madison and a bachelor’s degree in computer science from Birla Institute of Technology and Science, Pilani, India.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

Website: https://www.linkedin.com/in/srikris/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Krishna on LinkedIn: https://www.linkedin.com/in/srikris/

Timestamps:

[00:00] Krishna's preferred coffee

[00:12] Takeaways

[01:27] Please like, share, leave a review, and subscribe to our MLOps channels!

[01:56] AI Entrepreneurship Journey

[04:25] Core ML and Edge AI

[08:44] AI Stack & Workflow Strategy

[11:42] On-device AI Foundations[17:15] Hardware vs Software Optimization

[21:32] On-device AI Challenges

[26:19] Small LLM Orchestration

[28:03] Memory Constraints and Shared Pools

[30:05] Qualcomm AI Hub Edge

[32:53] AI in Unexpected Places

[41:53] Deploying AI on Edge

[45:58] 4X Battery Optimization Tips

[51:00] Wrap up