LLMs and Teaching
These are some loose thoughts about ChatGPT and similar Large Language models. I jotted these thoughts down after listening to a talk by Wayne Geerling and Dirk Mateer and hearing their thoughts on the matter
Newer Models
The next generation of Large-Language Model AIs (LLMs) will be even more intelligent and have more diverse capabilities.
- Bing Chat has the ability to search the web for citations, and will soon be available to general audiences.
- GPT4 can interpret visuals, has a much larger context window than GPT3 (meaning it can hold an entire book’s worth of material ‘in its head’), and is being integrated with various 3rd party services.
- Image and voice-generating AIs are also advancing at a rapid pace. Video calls may soon be easy to fake as well.
There are lots of tips and tricks about catching use of ChatGPT. But trying to work around the quirks of the current generation of AI models is a band-aid solution. The entire goal and structure of teaching might have to change. And that’s a daunting thought.
On New Ideas
You mentioned in the talk that LLMs can’t generate new ideas.
And in some sense that’s true. GPT and its ilk are, at their core, text prediction engines. They’re reminiscent of the autocomplete on your phone keyboard, but scaled up to the point where they can ‘predict’ entire sentences/paragraphs/books. LLMs combine and mimic what they have been trained on.
But that’s little comfort for the teacher of an intro or even intermediate course. Because although we sometimes ask students to come up with “new” ideas, we aren’t asking them to advance the frontiers on human understanding. The new ideas we ask students for are typically within the convex hull of existing knowledge. And generating that kind of new idea is well within the capabilities of LLMs.
The Effort of Essays
As college educators, the most spiritually enriching part of our job is helping our students better understand the world. But as economists, we also have to acknowledge that there’s more to it than that. The other service we’re providing - the one that students are willing to pay so much for - is the generation of a credible signal of aptitude and effort.
To generate this signal, we want the ability to pass our classes to be well correlated with the understanding of the subject matter, and with the amount of effort put into understanding the subject matter.
Proctored exams still provide a good signal, but essays at home are on shakier grounds.
I’m actually doubtful that long-form essays have ever been a great tool for imparting understanding. But they’ve been an important teaching tool because they can be used to evaluate understanding, and because even if the essay isn’t perfect, at least the student put in the effort. Performance on essays is/was correlated with aptitude and effort.
LLMs flip the correlation on its head. It’s always been possible to hire someone to write an essay for you. But LLMs are better and cheaper than sketchy essay-writers hired via WeChat. I fully expect going forward that the best lengthy essays will always be, at least in part, generated by LLMs. That kind of homework assignment is no longer useful for generating the signal we want.
In my class, I assign a few essays as HW assignments. This semester, I removed the minimum word requirement, and changed the rubrik to encourage clarity, conciseness, completeness of response, and a focus on personal reflection. I’ve given up on at-home essays as a tool to evaluate effort. If they are to continue to have any use at all, it must be to encourage understanding.