Launching a Digital Zine

While I've been working on systems programming and local-first software, I've resisted diving into building with LLMs for the better part of 8 months. But I think it's too much of a fundamental shift to ignore. I won't be ignoring the groundwork that I've been doing on CRDTs and Prolly Trees, but I will be shifting the thesis and focus a little bit. My larger goal had always been about making programming better, not specifically on databases and syncing.

Part of that shift was working on a digital zine called Forest Friends, where the first issue covers LLM system evals (buy here, preview here). System evals are how you can use qualitative measures of "good" to judge how well an LLM-driven app is doing.

I'm not entirely new to the field of machine learning and data science. However, some of the details were new to me. So Sri (from Technium Podcast) and I decided to team up again for a little side project.

One of our strengths is the ability to synthesize new information about technology and extract the core interesting aspects. While we're proud of the work that we did with the Technium Podcast, we were never able to find a way to monetize it and leverage it in a way that made it sustainable.

Based on some customer development and some guesses, we figured we could do the same work as we did on our podcast, but sell it as a digital zine instead. Our thesis was that:

1. People want to know about system evals because they were building LLM-driven apps.
2. Illustrations will help it stand out and give us the chance to do worldbuilding.  
3. People are willing to pay for digital zines.

We started with the meme that LLMs are more like an alien intelligence--more like a Shoggoth than a human. We just slap some RLHF (Reinforcement Learning from Human Feedback) in front of it to make it palatable to us humans. We thought it would be funny to juxtapose this thing with cute forest animals living in an urban forest as the tried to use it for their daily tasks.

We focused only on two animal friends, Fox and Bear, and how they tried to make a cafe that generated recipes with the LLM Shoggoth, as part of worked examples. The rest of the time, we'd talk about how to build system evals.

We planned to write 30 pages, but ended up with 64+. We planned to use Midjourney to generate the images, but it was too hard to steer at first. Then we tried to do hand-illustrations. That took too long, and finally, after lots of trial-and-error, we figured out how to steer Midjourney.

As our journey as entrepreneurs, we wanted to do a full rep, and this was a way to exercise our muscles. We did a couple things right:

• We talked to some target customers about doing evals, and there was a demonstrated curiosity about it.
• We started selling it with pre-orders to demonstrate some demand for it.
• We set a target dollar amount for sales that seemed slightly impossible.

I'm happy to say that all three aspects of our thesis was true and we hit it in our first week after launch.

So now, we'll mainly try to keep telling people about our work, and look into what our audience wants as a second issue. Until then, check it out, and if you have any feedback, let me know @iamwil.