Developer Voices Kris Jenkins

Every database has to juggle the need to process new data and to query old data. That task falls to any system that "does stuff and remembers stuff". But it's quite hard to really optimise one system for both use cases. There are different constraints on new and old data, and as a system gets larger and larger, those differences multiply to breaking point. That's something Twitter's engineers were figuring out in the 2010s.

One solution that came up in those years was the Lambda Architecture. A two-pronged approach that recognises the divide between new and old data, and works hard to blend the two together seamlessly in userspace. But that seamless blending is easier said than done. It's nearly all bespoke work.

What if you could get it off the shelf? Let someone else do the work of combining two different kinds of database into one neat package? That's the question of the week as we look at the recently open-sourced project Proton, and its attempt to be the Lambda Architecture in a box…

–

Proton Docs: https://docs.timeplus.com/proton

Proton Source: https://github.com/timeplus-io/proton

Timeplus: https://www.timeplus.com/

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#podcast #softwareengineering #databases #dataengineering 

Rust changed the discussion around memory management - this week's guest hopes to push that discussion even further.

This week we're joined by Evan Ovadia, creator of the Vale programming language and collector of memory management techniques from far and wide. He takes us through his most important ones, including linear types, generation references and regions, to see what Evan hopes the future of memory management will look like.

If you've been interested in Rust's borrow-check and want more (or want different!) then Evan has some big ideas for you to sink your teeth into.

–

Vale: https://vale.dev/

The Vale Discord: https://discord.com/invite/SNB8yGH

Evan's Blog: https://verdagon.dev/home

Evan's 7DRL Entry: https://verdagon.dev/blog/higher-raii-7drl

7DRL: https://7drl.com/

https://verdagon.dev/grimoire/grimoire

What Colour Is Your Function?: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/

42, the language: https://forty2.is/

Verona Language: https://www.microsoft.com/en-us/research/project/project-verona/

Austral language: https://austral-lang.org/

Surely You're Joking, Mr Feynman! (book): https://www.goodreads.com/book/show/35167685-surely-you-re-joking-mr-feynman



Evan on Twitter: https://twitter.com/verdagon

Find Evan in the Vale Discord: https://discord.com/invite/SNB8yGH



Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#software #programming #podcast #valelang

The "big data infrastructure" world is dominated by Java, but the data-analysis world is dominated by Python. So if you need to analyse and process huge amounts of data, chances are you're in for a less-than-ideal time. The impedance mismatch will probably make your life hard somehow. 

So there are a lot of projects and companies trying to solve that problem. To bridge those two worlds seamlessly, and many of the popular solutions see SQL as the glue. But this week we're going to look at another solution - ignore Java, treat Kafka as a protocol, and build up all the infrastructure tools you need with a pure Python library. It's a lot of work, but in theory it would make Python the one language for data storage, analysis and processing, at scale. Tempting, but is it feasible? 

Joining me to discuss the pros, cons, and massive scope of that approach is Tomáš Neubauer. He started off doing real time data analysis for the Maclaren's F1 team, and is now deep in the Python mines effectively rewriting Kafka Streams in Python. But how? How much work is actually involved in porting those ideas to Python-land, and how do you even get started? And perhaps most fundamental of all - even if you succeed, will that be enough to make the job easy, or will you still have to scale the mountain of teaching people how to use the new tools you've built? Let's find out.

– 

Quix Streams on Github: https://github.com/quixio/quix-streams

Quix Streams getting started guide: https://quix.io/get-started-with-quix-streams

Quix: https://quix.io/ 

Tomáš on LinkedIn: https://www.linkedin.com/in/tom%C3%A1%C5%A1-neubauer-a10bb144

Tomáš on Twitter: https://twitter.com/TomasNeubauer0

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins  

--

#podcast #softwaredevelopment #datascience #apachekafka #streamprocessing

Erlang wears three hats - it's a language, it's a platform, and it's an approach to making software run reliably once it's in production. Those last two are so interesting I sometimes wonder why those ideas haven't been ported to every language going.  How much work would it be?

This week we're going to dig right down into that question with Leandro Ostera. He's been working on Riot - a project to bring the best of Erlang's runtime system and philosophy to OCaml. But why OCaml? Is it possible to marry together OCaml's type system with Erlang's dynamic dispatch systems? And what is it about the recent release of OCaml5 that makes the whole project easier?

–

Leandro's Blog: https://www.abstractmachines.dev/

Why Typing Erlang is Hard: https://www.abstractmachines.dev/posts/am012-why-typing-erlang-is-hard/

Riot: https://riot.ml/

Riot source: https://github.com/riot-ml/riot

ReasonML: https://reasonml.github.io/

ReScript: https://rescript-lang.org/

Leandro on Twitter: https://twitter.com/leostera

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

--

#podcast #softwaredevelopment #erlang #ocaml #softwaredesign

The likes of LinkedIn and Uber use Pinot to power some astonishingly high-scale queries against realtime data. The numbers alone would make an impressive case-study. But behind the headline lies a fascinating set of architectural decisions and constraints to get there. So how does Pinot work? How does it process queries? How are the various roles split across a cluster? And equally important - what does it *not* try to achieve.

Joining me to go through the nuts and bolts of how Pinot handles SQL queries is Tim Berglund, veteran technology explainer of the realtime-data world. He takes us through Pinot step-by-step, covering the roles of brokers, servers, controllers and minions as we build up the picture of a query engine that's interesting in theory and massively performant in practice.

–

Apache Pinot: https://pinot.apache.org/

Apache Pinot Docs: https://docs.pinot.apache.org/

StarTree: https://startree.ai/
https://startree.ai/Event Driven Design episode with Bobby Calderwood: https://youtu.be/V7vhSHqMxus

Tim on Twitter: https://twitter.com/tlberglund

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#podcast #softwaredevelopment #apachepinot #database #dataengineering #sql

TJ DeVries is a core contributor to Neovim and several of its most interesting sub-projects, and he joins us this week to go in depth into how Neovim got started, how it's structured, and what a truly programmable editor has to offer programmers who want the perfect environment.

Along the way we look at what we can learn from Neovim's successful fork of the 30-year old codebase from Vim, how it still collaborates with the original project, and what putting Lua at the heart of the system has done for casual tinkerers and hardcore plugin writers alike.

Not everyone will come away from this discussion wanting to switch editors, but I'm sure you'll get a newfound appreciation for digging deeper into the developer tools you use everyday.

–

Neovim: https://neovim.io/

Neovim Kickstarter: https://github.com/nvim-lua/kickstart.nvim

Kickstarter walkthrough video: https://www.youtube.com/watch?v=m8C0Cq9Uv9o

A directory of Neovim plugins: https://dotfyle.com/

Vimscript's definition of true and false: https://vimhelp.org/eval.txt.html#Boolean

TJ on Twitter: https://twitter.com/teej_dv

TJ on Twitch: https://www.twitch.tv/teej_dv

TJ on YouTube: https://www.youtube.com/@teej_dv

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#podcast #software #softwareengineering #dx