Three ways I learn with AI
"You don’t `learn` from a chatbot. You consume...They’re spewing slop and people are gobbling it up." No. You’re holding it wrong. You can definitely learn from LLMs. It has to be with critical thinking.
You don’t “learn” from a chatbot. You consume. There’s a difference. They’re spewing slop and people are gobbling it up (and eroding their intelligence in the process). Misinformation is everywhere. Please don’t rely on a chatbot for anything important. They can’t replace real human connection
- random 🦋 account
No. You’re holding it wrong. You can definitely learn from LLMs, but with caveats. It has to be with critical thinking where you doubt it every step of the way, on things you can easily verify.
LLMs definitely have a shape to them. Like a knife, you need to find the part of it–the handle–that has the right affordance for human hand. In the case of an intelligence tool like LLMs, you're trying to find the affordance for the human mind. As easy as they are to get started with as a chat conversation, most people haven't figured out how to hold these tools yet.
Here's some ways I notice I consistently and heavily use LLM/AI in three different ways to learn:
- As a patient teaching assistant when reading academic papers.
- To get quickly acclimated and acquainted in a new programming language or framework ecosystem.
- As a sounding partner in software architectural design work.
Reading academic papers
Academic papers are dense with jargon, and rightly so. Their main purpose is to communicate with their own peers to move forward the state-of-the-art. But that also mean it can be inscrutable to outsiders, even other academics outside the field. The effort you needed to come up to speed on not just the jargon, but the requisite concepts in order to understand the paper, required a huge activation energy.
LLMs lower this bar significantly. When I have to read computer science papers, I upload the PDF (or PDF converted to text) and ask it questions of any passages, terms, or concepts I don't understand. I can ask questions about what concepts mean and what its implications are. When I don't understand the abstract concept, I can ask it to write out a concrete example. If I still don't get it? I ask it to explain with metaphors in a domain I know well. I can even ask it to explain it to me like I'm five, and then build it back up to the academic level of understanding from there.
Wil Chung
Importantly, I can spot-check what it's saying, either at random or if it smells fishy, based on my current understanding. Here's some things to do for spot-checking:
- Look at the exact passage in the paper for a claim or fact it's asserting.
- Cross-reference an assertion with a quick search on the web.
- Cross-reference against existing canonical text I know to be authoritative (such as the documentation to a software library)
With this Socratic process, I can read about a paper I'm mildly familiar with in a day, whereas otherwise it would have taken me the better part of a week.
Learning a new language, framework, library, or ecosystem
Lots of career programmers just stick with a single language, framework, and ecosystem for a long time. They only jump ship when they're force by work, or if they feel FOMO from "cool kids" touting a new language or framework. But even for programmers who are curious hackers by nature, it takes a lot of effort, a lot of activation energy to get started with another language and ecosystem.
For one, the only real reason to go to another ecosystem is if it offers something that you can't get anywhere else. For React, it was the one-way data binding and data flow that eased the reasoning burden (compared to frameworks before). For Haskell, it's the pure functional approach. For Lean, it's dependent types. This new capability comes with conceptual ideas that need to be learned.
The LLM can definitely help you with learning the core concepts, but the source of the high activation energy is all the chrome around the novel core concept. This is basic stuff like syntax, conventions, or build tools. It's the locations of the forums, documentations, and package management. These are often arbitrary [^1] and by cultural convention, rather than from any reasoned first principle, so you just have to memorize them.
LLMs are great at this kind of administrivia. For a new language, I can ask the LLM to write me a first draft in the language for a project that I actually care about. It's much more motivating than some arbitrary project that the tutorial writer decided upon. In addition, the documentation in the ecosystem could be dense or poorly written, so this helps sidestep that hump.
For example, I wanted to see how the Isabelle proof system worked, so I had the LLMs generate me the code. That was a year and a half ago, so the models weren't very good at inductive proofs. But I got the hang of the syntax much more easily than if I had to read the docs.
Wil Chung
For new frameworks and libraries, once again, asking it to do a first draft is motivating to see how things work, and if it's a popular ecosystem, I get to see what the conventions are. In most instances, I often have to temper or rewrite the implementation to fit my own standards in my project. However, this is often the quickest way to get something up and running, if the scope of the framework or library is well contained.
In all these instances, I often don't have to check the veracity of the generated response by hand, because it's often code, so there are type checkers, compilers, and build systems to check the output. The value LLMs provide here is to synthesize all the administrivia together to produce a working draft to iterate upon. It's much easier for all this stuff to stick when it's in the context of a project you care about.
Software architecture and design
As someone that works alone, LLMs have been great as a rubber duck that talks back. The value here is to generate options I might not have thought of, and to push back on my thinking and designs. But for this to work, you need to adjust the system prompt to allow the LLM to push back on your ideas to combat their sycophantic tendencies.
As a short list, but not exhaustive list, you should put in the system prompt that:
- Your ideas are often dumb, shallow, and poorly considered. [^2]
- It should ask them pointed questions and help re-direct them toward more insightful lines of thinking.
- Do not be fawning or complimentary.
- It should move them toward more specific, concrete claims backed up by solid evidence.
- It should generate a counter-argument to its assertion, as well as a rebuttal to its counter-argument. [^3]
With that in place, I first lay out the premise of the software design problem, the constraints of the situation, and ask it to generate options with pros and cons. Then after reviewing the options, I critique those options, throw away bad options, or ask it to elaborate its reasons for choosing specific options. From there, we keep iterating. I find that the latest reasoning models, like GPT-5 and Claude Opus are much better at this than previous models. However, they still have weaknesses.
Wil Chung
I find that they tend to revert to the median thinking in software design, which is more in an Object-Orientated tradition in a larger company. This often means a single class for a single concept, when a few functions would do, or an over-complication of relations between subsystems, when a simplification would do. Lots of cover-your-ass chrome, such as health checks and monitoring, when it's not needed for a core design.
Unlike the other two types of learning, this task isn't grounded in easily verifiable facts. What makes the process valuable is the iteration of different scenarios and options, and the articulation of my own tacit knowledge.
First, this process of iteration over different options and different solution spaces makes me better at doing the software design. Instead of asking LLMs for "the best solution", I get lots more practice considering options under different scenarios. When the LLM comes up with a great solution, I ask how it got there, and what it knew or considered. I also examine what sort of faulty assumptions I had that lead me away from a good solution.
Second, the failure modes outlined above are helpful, because it helps me articulate my tacit knowledge of why these are bad solutions. In order to leverage this tacit knowledge as a compounding advantage, I'm starting to record these failures as system eval traces in hopes of doing error analysis and prompt corrections later on. But for now, I'm just writing these corrective guardrails by hand, because asking the LLM to generate them results in subpar generalizations of engineering principles. Either they're too specific for the problem at hand, or they take commonly known engineering aphorisms and apply them to the shallow design problem, instead of the core reason why I rejected the solution.
Don't abdicate your seat at the table
The temptation with LLMs is to absolve yourself from doing critical thinking; to question everything the LLMs says back to you. But to do so is to curb your learning, and to stunt your learning. Despite the hype, I don't think software engineers will go away completely. I think our numbers will greatly diminish, so to stay as one, you have to get much better. In order to do that, you'll have to keep on learning.
The only time you should abdicate your thinking to LLMs is when the task at hand is administrivia: the knowledge that will be outdated in 15 years. If you've ever read a COBOL manual on some general software principle, you'll know what I mean.
[^1]: If you've been around the block enough, some ecosystems do take these arbitrary conventions from each other. The build tool in C is make. In Ruby, it's rake. In Lean, it's lake.
[^2]: To combat the sycophantic tendencies, I use a variation on Maggie Appleton's prompt.
[^3]: Got the argument/counter-argument/rebuttal from Prof Sarah Paine.
