The Nonlinear Library The Nonlinear Fund

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Visualizing neural network planning, published by Nevan Wichers on May 9, 2024 on The AI Alignment Forum.
TLDR
We develop a technique to try and detect if a NN is doing planning internally. We apply the decoder to the intermediate representations of the network to see if it's representing the states it's planning through internally. We successfully reveal intermediate states in a simple Game of Life model, but find no evidence of planning in an AlphaZero chess model.
We think the idea won't work in its current state for real world NNs because they use higher-level, abstract representations for planning that our current technique cannot decode. Please comment if you have ideas that may work for detecting more abstract ways the NN could be planning.
Idea and motivation
To make safe ML, it's important to know if the network is performing mesa optimization, and if so, what optimization process it's using. In this post, I'll focus on a particular form of mesa optimization: internal planning. This involves the model searching through possible future states and selecting the ones that best satisfy an internal goal. If the network is doing internal planning, then it's important the goal it's planning for is aligned with human values.
An interpretability technique which could identify what states it's searching through would be very useful for safety. If the NN is doing planning it might represent the states it's considering in that plan. For example, if predicting the next move in chess, it may represent possible moves it's considering in its hidden representations.
We assume that NN is given the representation of the environment as input and that the first layer of the NN encodes the information into a hidden representation. Then the network has hidden layers and finally a decoder to compute the final output. The encoder and decoder are trained as an autoencoder, so the decoder can reconstruct the environment state from the encoder output. Language models are an example of this where the encoder is the embedding lookup.
Our hypothesis is that the NN may use the same representation format for states it's considering in its plan as it does for the encoder's output. Our idea is to apply the decoder to the hidden representations at different layers to decode them. If our hypothesis is correct, this will recover the states it considers in its plan. This is similar to the Logit Lens for LLMs, but we're applying it here to investigate mesa-optimization.
A potential pitfall is that the NN uses a slightly different representation for the states it considers during planning than for the encoder output. In this case, the decoder won't be able to reconstruct the environment state it's considering very well. To overcome this, we train the decoder to output realistic looking environment states given the hidden representations by training it like the generator in a GAN.
Note that the decoder isn't trained on ground truth environment states, because we don't know which states the NN is considering in its plan.
Game of Life proof of concept (code)
We consider an NN trained to predict the number of living cells after the Nth time step of the Game of Life (GoL). We chose the GoL because it has simple rules, and the NN will probably have to predict the intermediate states to get the final cell count. This NN won't do planning, but it may represent the intermediate states of the GoL in its hidden states.
We use an LSTM architecture with an encoder to encode the initial GoL state, and a "count cells NN" to output the number of living cells after the final LSTM output. Note that training the NN to predict the number of alive cells at the final state makes this more difficult for our method than training the network to predict the final state since it's less obvious that the network will predict t

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: some thoughts on LessOnline, published by Raemon on May 9, 2024 on LessWrong.
I mostly wrote this for facebook, but it ended up being a whole-ass post so I figured I'd put it here too.
I'm helping run "LessOnline: A Festival of Writers Who Are Wrong On the Internet (But Striving To Be Less So)".
I'm incentivized to say nice things about the event. So, grain of salt and all. But, some thoughts, which roughly breakdown into:
The vibe: preserving cozy/spaciousness of a small retreat at a larger festival
The audience: "Reunion for the The Extended Family Blogosphere, both readers and writers."
Manifest, and Summer Camp
...
I. The Vibe
I've been trying to explain the vibe I expect and it's tricksy. I think the vibe will be something like "CFAR Reunion meets Manifest."
But a lot of people haven't been to a CFAR Reunion or to Manifest.
I might also describe it like "the thing the very first EA Summit (before EA Global) was like, before it became EA Global and got big." But very few people went to that either.
Basically: I think this will do a pretty decent job of having the feel of a smaller (~60 person), cozy retreat, but while being more like 200 - 400 people. Lightcone has run several ~60 person private retreats, which succeeded being a really spacious intellectual environment, with a pretty high hit rate for meeting new people who you might want to end up having a several hour conversation with.
Realistically, with a larger event there'll be at least some loss of "cozy/spaciousness", and a somewhat lower hit rate for people you want to talk to with the open invites. But, I think Lightcone has learned a lot about how to create a really nice vibe. We've built our venue, Lighthaven, with "warm, delightful, focused intellectual conversation" as a primary priority.
Whiteboards everywhere, lots of nooks and a fractal layout that makes it often feel like you're in a seclude private conversation by a firepit, even though hundreds of other people are nearby (often at another secluded private conversation with _their_ own firepit!)
(It's sort of weird that this kind of venue is extremely rare. Many events are hotels, which feel vaguely stifling and corporate. And the nice spacious retreat centers we've used don't score well on the whiteboard front, and surprisingly not even that well on "lots of nooks")
...
Large events tend to use "Swap Card" for causing people to meet each other. I do find Swap Card really good for nailing down a lot of short meetings. But it somehow ends up with a vibe of ruthless efficiency - lots of back-to-back 30 minute meetings, instead of a feeling of organic discovery. The profile feels like a "job fair professional" sort of thing.
Instead we're having a "Names, Faces, and Conversations" document, where people write in a giant google doc about what questions and ideas are currently alive for them. People are encouraged to comment inline if they have thoughts, and +1 if they'd be into chatting about it. Some of this hopefully turns into 1-1 conversations, and if more people are interested it can organically grow into "hey let's hold a small impromptu group discussion about that in the Garden Nook"
...
We'll also have a bunch of stuff that's just plain fun. We're planning a puzzle hunt that spans the event, and a dance concert led by the Fooming Shoggoths, with many songs that didn't make it onto their April 1st album. And the venue itself just lends itself to a feeling of whimsy and discovery.
...
Another thing we're doing is encouraging people to bring their kids, and providing a day care to make that easier. I want this event to feel like something you can bring your whole life/self to. By default these sorts of events tend to not be very kid friendly.
...
...
...
II. The Audience
So that was a lot of words about The Vibe. The second question is "who a

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dating Roundup #3: Third Time's the Charm, published by Zvi on May 9, 2024 on LessWrong.
The first speculated on why you're still single. We failed to settle the issue. A lot of you were indeed still single. So the debate continues.
The second gave more potential reasons, starting with the suspicion that you are not even trying, and also many ways you are likely trying wrong.
The definition of insanity is trying the same thing over again expecting different results. Another definition of insanity is dating in 2024. Can't quit now.
You're Single Because Dating Apps Keep Getting Worse
A guide to taking the perfect dating app photo. This area of your life is important, so if you intend to take dating apps seriously then you should take photo optimization seriously, and of course you can then also use the photos for other things.
I love the 'possibly' evil here.
Misha Gurevich: possibly evil idea: Dating app that trawls social media and websites and creates a database of individuals regardless of if they opt in or not, including as many photos and contact information as can be found.
Obviously this would be kind of a privacy violation and a lot of people would hate it.
but I imagine a solid subset of singles who are lonely but HATE the app experience would be grateful to be found this way.
No big deal, all we are doing is taking all the data about private citizens on the web and presenting it to any stranger who wants it in easy form as if you might want to date them. Or stalk them. Or do anything else, really.
And you thought AI training data was getting out of hand before.
All right, so let's consider the good, or at least not obviously evil, version of this.
There is no need to fill out an intentional profile, or engage in specific actions, other than opting in. We gather all the information off the public web. We use AI to amalgamate all the data, assemble in-depth profiles and models of all the people. If it thinks there is a plausible match, then it sets it up.
Since we are in danger of getting high on the creepiness meter, let's say the woman gets to select who gets contacted first, then if both want to match in succession you put them in contact. Ideally you'd also use AI to facilitate in various other ways, let people say what they actually want in natural language, let the AI ask follow-up questions to find potential matches or do checks first (e.g. 'I would say yes if you can confirm that he…') and so on.
There is definitely not enough deep work being done trying to overturn the system.
Bumble gives up its one weird trick, goes back to men messaging first.
Melissa Chen: The evolution of Bumble:
Sick of men inboxing women ("the patriarchy is so creepy and icky!")
Starts dating app to reverse the natural order (women now make the first move! So empowering! So brave & stunning!)
Women complain it's exhausting
Reinstate the natural law
Hardcore Siege: It's such ridiculous headline. I have never gotten an opener on Bumble besides "hey", women never actually work go start a conversation or have a good opener, they're literally just re-approving the ability of the man to start the conversation.
Outa: Anyone that's used it would tell you that 99% of the time they would just leave a "hey" or "."
Casey Handmer: AFAIK no one has yet made a dating app where the cost of sending messages is increased if you're a creep. This would be technologically easy to do, and would let the market solve the problem.
Several interesting things here.
1. Many 'women never actually initiated the conversation' responses. Women say 'hey' to bypass the requirement almost all the time. That is not obviously useless as a secondary approval, but it presumably is not worth the bother.
2. This was among women who self-selected into the app with mandatory female openers, so yeah, women really real

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Potential Pitfalls in University EA Community Building, published by jessica mccurdy on May 8, 2024 on The Effective Altruism Forum.
TL;DR: This is a written version of a talk given at EAG Bay Area 2023. It claims university EA community building can be incredibly impactful, but there are important pitfalls to avoid, such as being overly zealous, overly open, or overly exclusionary. These pitfalls can turn away talented people and create epistemic issues in the group.
By understanding these failure modes, focusing on truth-seeking discussions, and being intentional about group culture, university groups can expose promising students to important ideas and help them flourish.
Introduction
Community building at universities can be incredibly impactful, but important pitfalls can make this work less effective or even net negative. These pitfalls can turn off the kind of talented people that we want in the EA community, and it's challenging to tell if you're falling into them. This post is based on a talk I gave at EAG Bay Area in early 2023[1]. If you are a new group organizer or interested in becoming one, you might want to check out this advice post.
This talk was made specifically for university groups, but I believe many of these pitfalls transfer to other groups. Note, that I didn't edit this post much and may not be able to respond in-depth to comments now.
I have been in the EA university group ecosystem for almost 7 years now. While I wish I had more rigorous data and a better idea of the effect sizes, this post is based on anecdotes from years of working with group organizers. Over the past years, I think I went from being extremely encouraging of students doing university community building and selling it as a default option for students, to becoming much more aware of risks and concerns and hence writing this talk.
I think I probably over-updated on the risks and concerns, and this led me to be less outwardly enthusiastic about the value of CB over the past year. I think that was a mistake, and I am looking forward to revitalizing the space to a happy medium. But that is a post for another day.
Why University Community Building Can Be Impactful
Before discussing the pitfalls, I want to emphasize that I do think community building at universities can be quite high leverage. University groups can help talented people go on to have effective careers. Students are at a time in their lives when they're thinking about their priorities and how to make a change in the world. They're making lifelong friendships. They have a flexibility that people at other life stages often lack.
There is also some empirical evidence supporting the value of university groups. The longtermist capacity building team at Open Philanthropy ran a mass survey. One of their findings was that a significant portion of people working on projects they're excited about had attributed a lot of value to their university EA groups.
Common Pitfalls in University Group Organizing
While university groups can be impactful, there are several pitfalls that organizers should be aware of. In this section, I'll introduce some fictional characters that illustrate these failure modes. While the examples are simplified, I believe they capture real dynamics that can arise.
Pitfall 1: Being Overly Zealous
One common pitfall is being overly zealous or salesy when trying to convince others of EA ideas. This can come across as not genuinely engaging with people's arguments or concerns. Consider this example:
Skeptical Serena asks, "Can we actually predict the downstream consequences of our actions in the long run? Doesn't that make RCTs not useful?"
Zealous Zack[2] responds confidently, "That's a good point but even 20-year studies show this is working. There's a lot of research that has gone into it. So, it really d

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shrimp Paste and Animal Welfare, published by Aaron Boddy on May 8, 2024 on The Effective Altruism Forum.
Shrimp Welfare Project (SWP) produced this report because we believe it could have significant informational value to the movement, rather than because we anticipate SWP directly working on a shrimp paste intervention in the future. We think a new project focused on shrimp paste could potentially be very high impact and would be excited to collaborate with organizations working in this space.
This report was written by Alethea Faye Cendaña and Trinh Lien-Huong, with feedback provided by Shannon Davis, Aaron Boddy, Michael St Jules, Ren Ryba, and Zuzana Šperlová. We are grateful to all the stakeholders who took the time to offer their thoughts for this study. We would like to thank the external reviewers for their valuable insights and contributions. All errors and shortcomings are our own.
Executive Summary
Acetes shrimps are among the most - if not the most - utilized species for food globally. Harvested from the coastal regions of Asian countries, these shrimps are not only cooked into various dishes but also crucial in the production of shrimp paste: a salty, tangy condiment served in many Southeast Asian dishes.
Despite the enormous scale of production, there's scarce information about the shrimp paste industry and its impact on Acetes shrimp welfare. Thus, it is crucial to explore and understand the industry to identify and implement welfare interventions effectively. This report discusses an overview of the shrimp paste industry and highlights welfare issues and challenges to determine opportunities for welfare interventions.
We take Vietnam, one of the largest producers, as a case study to gain a deeper understanding of the industry
Shrimp paste is an integral condiment for Southeast Asian cuisine, valued for its unique umami flavor and nutritional properties, including vital omega-3 fatty acids. The production process primarily involves sun-drying, grinding, and fermenting Acetes shrimps, which varies in duration and technique across countries.
While deeply rooted in cultural heritage and social practices, shrimp paste is also a significant commercial product, with villages selling locally and countries like the US, UK, Canada, and Australia importing from top-producing countries like Thailand and Indonesia.
The industry provides livelihood opportunities for small coastal communities, where activities are divided among families with roles in catching, trading, and processing. Manufacturing facilities with capital, usually located near coasts, employ manual labor to mix, ferment, and cook the shrimp paste before packaging it for commercial sale. All producers face significant issues in raw material supply due to their reliance on natural shrimp stocks.
These stocks are subject to fluctuations and environmental threats. Moreover, there are additional challenges related to food wastage, as well as food hygiene and safety concerns in both traditional and commercial production methods.
Acetes shrimps are likely to endure significant suffering throughout the capture, retrieval, and processing stages in shrimp paste production. They experience injury, exhaustion, and suffocation during capture, often dying from hypoxia due to overcrowding or suffocation when removed from water. Additionally, during processing, they could suffer from osmotic shock, dehydration, and stress due to salting, grinding, and sun-drying, which are methods used to prepare them for paste production.
Our Vietnam case study explores the fishing practices, the economic and cultural importance of shrimp paste, and the operational dynamics and challenges of small communities and large companies involved in the industry.
We propose interventions to alleviate the suffering of Acetes shrimps, s

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FTX Has Billions More Than Needed to Pay Bankruptcy Victims, published by AnonymousTurtle on May 8, 2024 on The Effective Altruism Forum.
Once it finishes selling all of its assets, the company will have as much as $16.3 billion in cash to distribute, according to a company statement. It owes customers and other non-governmental creditors about $11 billion.
Depending on the type of claim they hold in the case, some creditors could recover as much as 142% of what they are owed. The vast majority of customers, however, will likely get paid 118% of what they had on the FTX platform the day the company entered Chapter 11 bankruptcy.
Earlier this year, the company had about $6.4 billion in cash. The increase is due mostly to a general spike in prices for various cryptocurrencies, including Solana, a token heavily backed by convicted fraudster and FTX founder Sam Bankman-Fried. The company has also sold dozens other assets, including various venture-capital projects like a stake in the artificial-intelligence company Anthropic.
Most FTX account holders will get their money back after bankruptcy
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org