Lab notes #018 Changes from category theory to todo lists

The couple of months have been full of changes. It was actually a good time to write down the process, but unfortunately, things changed so quickly that I didn't get a chance to process it all in writing. This is an outline from since the last time.

A couple of weeks ago, I had more time freed up. Hence I took the time to look more deeply into Category Theory. I was looking for a solid foundation to inform an incremental computation implementation for queries. Originally, the motivation was to incrementally compute both a query and the rendering for an interactive app. Differential dataflow appears to be the state-of-the-art, though it relied on some sophisticated counting. Admittedly, I couldn't find any details that let me really understand it. Dida got a good way there at a high level, but it fell a bit short for me. Compiling to categories was an promising direction, but it required compiling to Simply Typed Lambda Calculus first. That made it a non-starter in my particular instance. Eventually, I stumbled upon Datafun [paper][talk], which is a functional query language based on datalog. While promising, and close to what I wanted to do, but according to the repo activity, it looks like it's been in stasis.

But this wasn't what stopped me in my tracks. Recall my ongoing thesis was that state management was a problem for devs making web apps. There shouldn't be so much work shuttling data back and forth over the network boundary. Hence, another line of inquiry was simply what sort of existing solutions existed out there that I hadn't, but should have done. So I started looking for sync databases and compiled a list. Was there something out there that did what I wanted already? There were a couple contenders, and nothing quite fit the bill. Until I looked deeper into both CR-SQLite and Evolu. Both attacked the problem by writing CRDT-based syncing extensions for SQLite, and compiling SQLite into WASM to be run in the browser. Supabase also did an experiment by writing a CRDT-based syncing extensions for Postgresql.

This required me to rethink what I was doing, as well as my goals. Originally, I started down this road, mostly as applied research and learning. However, as I've re-thought my overall goals, this approach of writing SQLite extentions made more sense. It's hard to get people trust new database technologies unless they've been battle-tested for years, or there's some parameter they're 10x on. Was there some parameter what I'm working can 10x in? It would be the incrementalism from data to pixel, but I think it'd also take me far too long to get there.

I've tried both Evolu and CR-SQLite. Evolu worked mostly out of the box, but the documentation is outdated, and the response time for issue inquries was slow. CR-SQLite had quick response times for issue inquiries (which I appreciated!), but I ran into two issues since I was using Next.js, which wasn't well documented. At the end of the day, if a fundamental part of my stack is to be relied upon, it shouldn't be a blocker to my own forward progress.

I think it is indeed possible to write an SQLite extension using Automerge (which CR-SQLite doesn't use), but for now, I'm tabling this direction, and just using Automerge directly.

What would I build if I already had this technology? My original vision was a programming environment that was reponsive and surfaced the data as readily as the code. However, the surface area for that is really large. That was ok while my goal was applied research, but now that they lean more entrepreneurial, I would need to limit the scope.

Something I've wanted was task management for programming projects that I could stick with, but also collaborate with others. I think Steve Krouse was right in that most people simply forget what they're doing and why at every level of doing a todo list.

The task manager would help me do three things.

• It helps me prioritize the important but not urgent over urgent tasks that aligns with the overall goals.
• It leverages a core game loop with mechanics of task completion that helps people feel in flow.
• It can use GPT help complete tasks, starting with specifically, 1/ explaining code, 2/ explaining errors, 3/ committing code.

This seems like a small enough scope to get started and keep going. It's time to start building in public.