Lab notes #017 Surveying Local-first Real-time Databases / Frameworks
One major thing I did in the last two weeks was to take a step back and review the current landscape of real-time databases and syncing frameworks. In my last essay, I came to the conclusion that for app architectures that want to retain state on the client side, we really just need a database. I want an immutable database with reactive and incremental queries that can sync automatically to the server.
I'm prone to diving into the technicals, so I wanted to be more deliberate in the direction I was headed this time. And if it turns out someone is solving this problem, I can just build on that solution instead.
I read over Balaji Srinivasan's Idea Maze again recently. There's a line about PhDs–if they're at all entrepreneurial–who should have done a back-of-the-envelope calculation on the market size of their chosen field of study before they embarked on it. When they do it after graduating, it's often quite depressing. Besides, having a more bird's eye view of the current landscape, and the various decision points to navigate it has been helpful as an exercise.
Surveying the landscape, I was looking for something with the following features, from most important to least important.
- Can be run in the browser
- Is local-first software and works offline
- Can sync and replicate with a server database
- Has reactive or subscription queries
- Uses CRDTs to sync
- Immutable data with a history
- Also can be run on the server
- With incremental queries
- And a datalog or a mango query language
- With access policy
- And can refer to states with URLs
Most of the databases/frameworks I surveyed could fulfill the first four features, but CRDTs weren't pervasive and only software from Clojure were at all concerned with immutable data. And nearly nothing was incremental unless I started looking at streaming systems (which can kinda look like a database if you squint a little).
The open-source projects aren't competitive with startups in this space. Either documentation is lacking, the underlying technical details don't inspire confidence or not enough effort in streamlining the onboarding. It's definitely not something that is picking up the wind.
The startups are much more polished, but they all seem to be searching for the right subset of features that will speak to the users. The words they use to describe themselves are also all over the map. Currently, I have a bias against picking a database solution that's not open source or requires server hosting by the parent company. I don't currently have a good justification for this, other than being conservative about putting such a critical part of your infrastructure in the hands of a proprietary database. Even if it's open source, it's a non-starter if the database can only be hosted by the parent company. If the company goes down, I'd be in trouble.
Part of the view I'm missing is what kind of apps people are building with these real-time databases/frameworks. I think there are certain properties that local-first software can uniquely bring to the table. The obvious class of apps are ones where there's some sort of strong boundary carved around data owned by the user, such as in note-taking, email, and personal finance. However, I think there must be others, as I've run across properties that are not obvious.
I'm also unsure if the winds are shifting back to the server. React Server Components, Rails Hotwire, and Pheonix Livewire are examples of the realization that it might all be much easier to keep all the state on the server, just keep the view as stateless, and simply ship some reactivity and interactivity to the browser instead. That makes the non-obvious properties all the more important as an advantage of the architecture to be a fit for the next generation of apps that people will build on the web.