418: Mental Models For Reduce Functions The Bike Shed

Joël talks about his difficulties optimizing queries in ActiveRecord, especially with complex scopes and unions, resulting in slow queries. He emphasizes the importance of optimizing subqueries in unions to boost performance despite challenges such as query duplication and difficulty reusing scopes. Stephanie discusses upgrading a client's app to Rails 7, highlighting the importance of patience, detailed attention, and the benefits of collaborative work with a fellow developer.


The conversation shifts to Ruby's reduce method (inject), exploring its complexity and various mental models to understand it. Joël and Stephanie discuss when it's preferable to use reduce over other methods like each, map, or loops and the importance of understanding the underlying operation you wish to apply to two elements before scaling up with reduce. The episode also touches on monoids and how they relate to reduce, suggesting that a deep understanding of functional programming concepts can help simplify reduce expressions.



Rails 7 EXPLAIN ANALYZE
Blocks, symbol to proc, and symbols arguments for reduce
Ruby tally
Performance considerations for reduce in JavaScript
Persistant data structures
Avoid passing a block to map and reduce
Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire
monoids
iteration anti-patterns
Joël’s talk on “constructor replacement”


Transcript:


 STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn.


JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way.


STEPHANIE: So, Joël, what's new in your world?


JOËL: I've been doing a bunch of fiddling with query optimization this week, and I've sort of run across an interesting...but maybe it's more of an interesting realization because it's interesting in the sort of annoying way. And that is that, using ActiveRecord scopes with certain more complex query pieces, particularly unions, can lead to queries that are really slow, and you have to rewrite them differently in a way that's not reusable in order to make them fast.


In particular, if you have sort of two other scopes that involve joins and then you combine them using a union, you're unioning two sort of joins. Later on, you want to change some other scope that does some wares or something like that. That can end up being really expensive, particularly if some of the underlying tables being joined are huge. Because your database, in my case, Postgres, will pull a lot of this data into the giant sort of in-memory table as it's, like, building all these things together and to filter them out. And it doesn't have the ability to optimize the way it would on a more traditional relation.


A solution to this is to make sure that the sort of subqueries that are getting unioned are optimized individually. And that can mean moving conditions that are outside the union inside. So, if I'm chaining, I don't know, where active is true on the outer query; on the union itself, I might need to move that inside each of the subqueries. So, now, in the two or three subqueries that I'm unioning, each of them needs to have a 'where active true' chained on it.


STEPHANIE: Interesting. I have heard this about using ActiveRecord scopes before, that if the scopes are quite complex, chaining them might not lead to the most performant query. That is interesting. By optimizing the subqueries, did you kind of change the meaning of them? Was that something that ended up happening?


JOËL: So, the annoying thing is that I have a scope that has the union in it, and it does some things sort of on its own. And it's used in some places. There are also other places that will try to take that scope that has the union on it, chain some other scopes that do other joins and some more filters, and that is horribly inefficient. So, I need to sort of