Lab notes #022 SQL is the new QWERTY

SQL is the new QWERTY. It's a strong enough network effect that we'll have to keep living with it. I don't mind the declarative nature of SQL, but I do wish it was more composable and abstractable. While there is the WITH clause, it's not quite abstraction, because you're never lifted out of the details.

And with the heady flurry of NoSQL back in the mid-2010s, the hung-over consensus is now to just use PostgreSQL. There is a certain wisdom in choosing boring technology, especially data storage. For any innovation venture, there is a quota of innovation tokens you can spend. So if you're pushing the envelop in other areas, the last thing you want is to debug corrupt data.

That said, I often wondered why we have two ways of storing and accessing data. One way as data structures in memory, and one way as relational on disk. That incompatibility and the desire to mash them together results in a lot of complexity in current systems. It may be that at one point, disk and memory had vastly different properties that you needed different approaches to leverage what each is uniquely good at. But I think those days are waning, so we should be able to find a way to simplify our stack.

As I've learned more about databases and what they can do internally, the more I wished it was turned inside out. Those internal tools and data structures should be available natively in the programming language libraries to application developers. Instead, we just build them all over again, because nowadays, web app developers are distributed systems engineers, but we just don't acknowledge it yet.

So any innovation on this front will probably need to trojan horse into the hands of developers through Postgres and Sqlite extensions. As an application dev, I also feel wary of adopting another DB wholesale, so if there was something that piggybacked off of what I already know--well that gives me warmer fuzzies.

This week I used GPT to explain how to write extensions in Postgres and Sqlite to me. I was able to do it in a day, rather than digging through a lot of tutorials and documentation over the course of the better part of a week. I didn't need to trust that the details were fully right and done without bugs, but I needed the mental model to hang my hat upon.

Leveraging GPT to helped get me up to speed a lot faster than I would have otherwise. I've found that I have to be careful not to try and bias its answers with my questions by asking for points and counterpoints.

Again, it's great for basics and clarifications, but it doesn't do insight well, which is the mixing of ideas. You'd think it'd be able to do that because style transfer is demonstrated, but I haven't found this to be the case.