Why StackOverflow usage is down 50% — with David Hsu of Retool

We are announcing the second edition of our Latent Space demo day event in SF on 2/23: Final Frontiers, a startup and research competition in “The Autonomous Workforce”, ”Beyond Transformers & GPUs”, and “Embodied AI”.
RSVP here! The first one was aimed for 15-20 people and ended up blowing up to >200 and covered in the Information - let’s see what a year of growth (and competition) does to the local events space in 2024.
You can find all Latent Space events here, and of course get in touch with us to host your own AI Engineer meetups like AI Engineering Singapore.
In our December 2023 recap we covered the Four Wars of the AI stack. But how do we know when it’s time to crown a winner? As we kick off 2024, we wanted to do a recap of the State of AI in 2023 to set a baseline of adoption for different products. Retool had a great report at the end of last year which covered a lot of it.
David Hsu, CEO and co-founder of Retool, joined us to go over it together. We also talked about the history of Retool, why they were too embarrassed to present at YC demo day, and how they got to $1M ARR with 3 employees. If you’re a founder, there are a lot of nuggets of advice in here!
Retool AI
In our modeling of the “Software 3.0 Stack”, we have generally left a pretty wide open gap as to the “user interface” equivalent of the AI stack:
Retool AI launched 4 months ago with some nifty features for SQL generation, and its own hosted vector storage service (using pgvector). However, as he explains on the pod, the more interesting potential of Retool is in helping developers build AI infused applications quickly, in combination with its Workflows feature.
This moves Retool down the stack from just the UI for internal tooling to the business logic “piping” as well. There are a bunch of dedicated tools in this space like Respell, BuildShip, Flowise, and Ironclad Rivet.
"We think that practically every internal app is going to be AI infused over the next three years." - David on the pod
RIP StackOverflow?
In July 2023 we talked about the impact of ChatGPT and Copilot:
This was then disputed by StackOverflow, who pointed out (very fairly so) that there were privacy-related changes in their analytics instrumentation in 2022. StackOverflow no longer reports traffic, but based on StackOverflow’s continuing transparency we can see that organic declines have continued throughout 2023.
Retool’s report comes over a year after those changes and has some self reported samples from users:
* 57.6% of people said they have used StackOverflow less; almost all of them replaced it with ChatGPT and Copilot.
* 10.2% said they no longer use StackOverflow.
We also saw a lot more tools being released in the dev tools space such as (one of our oldest pod friends) Codeium (which just raised a $65M Series B), SourceGraph (and their newly released Cody), Codium AI (just released AlphaCodium which was picked up by Karpathy), Phind (which beat GPT-4 with OSS models), and Cursor, one of the most beloved products in the dev community at the moment. Intelligence is getting closer and closer to the IDE, and the trend doesn’t seem to be reverting.
We already said that “You are not too old (to pivot into AI)“, and the advice still stands. When asked to rate “Preference for hiring engineers effective at using ChatGPT/Copilot for coding” on a scale of 1 to 10, where 10 is “Much more likely”, ~40% of companies voted 8-10. Having an AI Engineer skillset is extremely important. 45% of companies between 1,000-4,999 employees said that they increased the difficulty of technical interviews to compensate for these new tools, so the gap between users and non-users will keep widening.
Crossing the AI in Production Chasm
Geoffrey Moore’s “Crossing the Chasm” is one of the most quoted business frameworks. Every market has an initial group of Innovators and Early Adopters, who are willing to suffer through the rough edges of products initially, and eventually crosses into the Early Majority, which expects a full product.
In the AI world, ChatGPT and Midjourney / DALL-E have crossed the chasm in the consumer space. Copilot is probably the only tool that did it in the enterprise, having crossed 1M paid users. ~$50B were invested in AI in 2023, and we still only have <5 breakout products; expect this number to rise in 2024. According to the survey, only 25% of companies had real production usage, but 77.1% said their company is making efforts to adopt more. Closing that gap could triple AI adoption in one year.
The report also broke down adoption by use case. 66% of companies use it internally, while only 43% do so in customer-facing use cases. Internal usage of AI is much more varied than customer-facing one as well:
One point that David made in the podcast is that this number isn’t a knock on AI as a tool, but rather about the demographics of businesses outside of our Silicon Valley bubble:
We all work in Silicon Valley, right? We all work at businesses, basically, that sell software as a business. And that's why all the software engineers that we hire basically work on external facing software, which makes sense with most software companies. But if you look at most companies in the world, most companies in the world are actually not software companies. […] Most of the [work of] software engineers in the world actually goes towards these internal facing applications.
Beyond code models, it’s clear that the big winners of the first wave of AI adoption are vector stores and RAG. Knowledge base Q&A, customer chatbots, recommendation systems, etc are all based on them. Retool even rolled out their own with Retool Vectors. Expect the battlefield to get even hotter in these areas, with Mongo and Chroma leading the charge on a NPS/popularity basis.
It’s also clear that OpenAI won the first campaign in the AI models war, by far. Hopefully Mistral and LLaMA3 will shake up this chart when we look back at it in 2025:
TLDR: We’re really early. If you want to build in AI, there’s a ton of work to be done, and a lot of problems to be solved. You can find the full report here to dive through all the numbers.
Video podcast
Watch along on our snazzy YouTube!
Show Notes
Companies and Projects:
* Retool
* State of AI Report
* Retool AI
* Retool Workflows
* Raising less money at lower valuations
* Paul Graham's "playing house" essay
* Gödel, Escher, Bach (GEB)
Timestamps
* [00:00:00] Introduction
* [00:02:43] Retool's founding story and decision not to present at YC demo day initially
* [00:09:08] Philosophy on fundraising - raising less money at lower valuations
* [00:12:53] Overview of what Retool is
* [00:15:41] Origin story of Retool AI product
* [00:19:59] Decision to use open source vector database PG Vector
* [00:21:29] Most underrated AI use cases
* [00:25:56] Retool's AI UX and workflows
* [00:30:38] Zapier vs Retool
* [00:32:54] Updates from Retool's 2023 State of AI survey
* [00:35:21] Who is adopting AI first?
* [00:37:40] Evolving engineering hiring practices in the age of Copilot/ChatGPT
* [00:40:02] Retool's views on internal vs external AI adoption
* [00:41:50] OSS models vs OpenAI in production
* [00:44:46] Additional survey questions to ask in 2024
* [00:47:04] Balancing enterprise sales vs bottom-up adoption
* [00:51:54] Philosophical thoughts on AGI and intentionality
Transcript
Information
- Show
- FrequencyUpdated Weekly
- PublishedFebruary 1, 2024 at 6:15 PM UTC
- Length58 min
- RatingClean