Year 2024 in Review

Year 2024 in Review

I almost never remember what I did in a year. Having written reviews for 2021 and 2023, I've found it helpful to both have a record to look back on as well as a point to review and reflect. I think this should be a habit every quarter, honestly.

  • Wrote incremental Prolly Tree implementation
  • Did Recurse Center.
  • Gave two talks at Recurse center. "Prolly Trees" and "Functional ideas in various programming languages"
  • Studied Cache-Oblivious Data Structures and Van Emde Boas Order.
  • Collaborated and Published Forest Friends Issue 1 on System Evals with Sri.
  • Contracted with District for a month, building recommendation invites, and other misc features.
  • Study prior art on Reactivity, including React, Use.GPU, Signals, Autotracking and Differential Dataflow.

Back at the beginning of 2024, I was still building a database in the browser, something that I embarked on earlier. I joined the Recurse Center with the expressed purpose of being able to focus on it, and maybe find some people that were interested in the same topics. The experience at Recurse Center was pretty good, but I didn't quite meet my own expectations. One, I spent way longer than I thought I would working on the Prolly Tree, a persistent data structure. Two, I didn't really meet anyone else who was as into local-first and incremental/reactive systems as I was.

For any particular batch, there is overlap with other batches, but there's a wide variety of interests and skill levels. So it's vanishingly rare that you'll find the same people interested in the same thing unless you make a concerted effort to recruit. I did do talks, and lead discussions, but I think it can be an uphill battle. There's an air of what's fashionable at any given moment, and lots of people were into LLMs at the time.

What the Recurse Center is great for, however, is to see the wide variety of things people work on. There are lots of things to poke your head into. Sometimes, this is what people say is so difficult about Recurse–the ability to find focus in the 12 weeks. It goes by fast.

I'm glad that I finished the incremental Prolly Tree, but I didn't give it the proper finish. I didn't pull it out into a repo and I didn't write a blog post about it. To the outside world, all I have is a series of lab notes, which amounts to an illegibility about what I know and what I do.

It's worse than that, actually. I gave a talk at the Recurse Center about Prolly Trees, and another about the core ideas in various less-mainstream languages. Both were not recorded, which I should have, at least to share with others outside of Recurse. In addition, I spent a good month and a half learning about and researching cache-oblivious data structures and van Emde Boas Order traversal, only to not write anything about it.

I spent hard-earned dollars on two courses this year, which I've never done before. One on RAG by Jason Liu, and the other on AI-driven therapy by Dan Shipper and Gena Gorlin.

The RAG course was actually quite helpful in demonstrating a different way of thinking than the usual software methodologies. I was hoping to leverage that into customers for Forest Friends, but I moved too slowly on that for that to be the case.

The AI-driven therapy was less helpful, but I was juggling Recurse Center and a home life at the time and this was low on my list, despite having to pay for it. I should have probably paid more attention to get more out of it. But what I think was valuable was knowing what to ask from a therapy point of view, rather than the actual AI prompting techniques.

After that, for a good part of the second and third quarter was working on Forest Friends, an eZine on setting up and doing system evals for LLM-driven apps. This was a bit out of left field, but I did it for a couple of reasons. First, it was an opportunity to work with my Techium co-host, Sri again. We had a good time working together on the podcast and wanted to collaborate again. Second, it was an opportunity to get a handle on all the LLM stuff. Lastly, we wanted to exercise our entrepreneurial muscles by doing a good rep, from customer dev to getting paid.

We did spend longer than warranted going through the entire process, but it felt good to do the rep. It wasn't clear at first that anyone wanted to do system evals. At the time, lots of people were just doing vibes-based engineering and weren't sure if system evals were just the purview of large companies. We attended a meetup to chat with people about evals and Sri attended an AI conference with a t-shirt he made that said "Ask me about evals". That got lots of people to talk to him, rather than requiring to approach lots of people cold. We've long since learned you won't get a sure-fire answer to any customer-dev situation, but we had a small, but strong enough signal that we decided to bite the bullet.

At first, we collected various writings on the internet, but even then, lots of them were skimpy and felt like AI-generated slop. So we leaned mostly on Sri's experience doing eval at Google for six years, and I did the editing, illustrations, and layout. We had wanted to generate the illustrations at first, and through the initial tests, I found Midjourney wanting, because I wanted very specific images. I needed a specific composition, with consistent characters, in a specific illustrative style, and a consistent interior design or architectural style. So I resorted to drawing things by hand. Though I can draw, I'm not a professional by any means, and I found that it was taking anywhere from six to ten hours for one image, and I wasn't going to make our deadline. So I switched back to trying to generate images. I figured, I just really need to lean into prompting.

Turns out that was the ticket. At first, I was spending 8 hours a day generating just a single image, looking through hundreds of variations, and trying different experiments to see if I could steer Midjourney to give me what I wanted. But after a while, I got the hang of it and could generate the image I wanted in about 1.5 hours. There were some things I could never figure out, however, like how to get aviator goggles on a fox, oddly enough. Lots of people have a skewed view of how powerful an LLM is, because they don't know that the more specific your taste for the acceptability of the results, the more you have to be specific with your prompt.

Our original target for "success" was breaking $1000 in the first month. We blew that target away easily, and it still earns money. Despite entrepreneurial forays before, this felt different. It felt fun to do the entire rep and to make money. Despite that, we had always meant this to be a temporary collaboration and went our separate ways afterward. I have been working on what a second issue would be, on and off, and I sometimes think: "why not just keep doing what works?"

Why not is because I've been circling the drain on the idea of a programming environment that lets me easily construct backend pipelines for a long time. Before doing evals, I saw that there were a bunch of startups and companies doing better on a local-first database than there was when I started. When I took a step back, the goal wasn't to create a database–but to make a functional notebook with effects. The database was a stepping stone.

So I committed to taking another shot at it, and spent the last quarter of the year reloading all the notes and information I read and collected about incremental and reactive systems over the years. Compared to the first time around, lots more things make sense, and having a project to actually work on the ideas and to inspect them in my head deepened an understanding that I didn't have the first time around.

The end of the year left the project on the floor in pieces due to a realization that there was an impedance mismatch between my domain problem and my approach. I spent the holidays reworking how all the pieces fit together and rewrote a part of it now. I hope to make more progress quickly on effects in the first quarter, and I'm eager to get out of this rabbit hole.


Thematically, I wished I didn't take a good chunk of the year doing Forest Friends, as it doesn't line up thematically with what I'm working on before or after it. The way to make a lot of progress over time is to hammer the same nail repeatedly. That said, it was a good foray into the hubbub in AI as well as doing a full entrepreneurial rep. I hope to swing back to this by merging what I've learned here with the reactive notebook I'm working on.

The other lesson is just that the year goes by quickly, and I need to find ways to work faster. I know this intellectually, but my mental habits tend to be more methodical. When I come across a design problem, my instinct is to start doing more research, rather than trying different solutions. I know both have merits, but I tend to lean more heavily on one–the one that takes a bunch of time. Case in point, this last rework for the reactive notebook resulted in a lot of research into signals, autotracking, and a bit of differential dataflow, and not as much code written comparatively.

My other reflective takeaway is to decide which mode I'm in for a time-boxed duration and go for it. If I'm in research mode, then it's ok for me to go down the rabbit hole and dig deep. Once I switch to entrepreneurial mode, I can't use the same mental habits and have to commit to doing, rather than thinking.

I've also been thinking about legibility. I think I'm rather illegible to the outside world. I'm doing all this work, and no one really knows what insights I'm learning or has a concise way to say in a sentence, the work that I'm doing. I need to make myself more legible. I think this probably means committing to getting out some kind of exhaust or artifact in whatever means possible. Originally, I was thinking that I might have to commit to more blog posts. But I do see that the way other people have gotten over this is to just talk into a camera, call unfinished essay posting as "digital gardening", or just only write posts you can finish in a day.

Lastly, I want to be able to finally use what I've been circling about, and I need to do it quickly. My thesis is that a reactive notebook with effects will make building LLM-driven applications much more quickly and pleasant. But that's just a hunch. That's not something verifiable in a book, but needs to be wrought with the friction of users coming up against product. If I don't get there within 2025, I'll be disappointed in myself.

To sum up, for 2025, I wish to change my work habits to do work faster, create digital by-products that make my work more legible, and get a reactive notebook I want to use every day up and running. That's my goal, and my promise to myself a year from now.