When Falling for Hype Becomes Falling Into Depression

Wednesday morning, I came across a thread on BlueSky by the professor and author Crystal Fleming that had blown up to epic levels of virality. It was about a topic that’s very close to me professionally and personally: artificial intelligence in the university system and the role of the digital humanities. 

The thread was inspirational enough that I very clearly felt a newsletter coming on. It’s worth reading the entire thread for its insight and frustration, but here’s the summary of what I thought was most important for what I want to say. An unnamed keynote speaker at a Digital Humanities conference admitted with embarrassment – only after the thread writer asked if he’d used AI – that he’d been “working closely with Claude” on all his work, including having an entire book manuscript ready for publication that he wrote completely by prompting Claude. 

The kicker is that his embarrassment stemmed from admitting that his Claude-dependence for writing his research work made him, in Fleming’s words, “feel useless as an intellectual.” Her key insight about this keynote was a point against the keynote’s repeated idea in his talk that AI use represents “a position without an address.” Fleming’s rejoinder: “Of course LLM products have an address. It’s a corporate address.” 

It’s become a standard critique in AI research to position generative systems not as some kind of neutral arbiter, but as having their behaviour rooted specifically in the interests, policies, and governing decisions of the corporations that run AI systems. What I and that thread’s many boosters found interesting was the attitude of this professor to his own work and intellectual capacities. 

Because this keynote speaker thought he was stupid. He thought that he, himself, had become completely inadequate to the work and vocation of being a public intellectual and producing knowledge in the digital humanities. Close work with Claude over time had convinced him that the knowledge and creativity of humans was worthless compared to the power of AI. Fleming even described how this keynote stumbled over pretty basic questions from the audience, unable to answer questions about how other intellectual ideas or philosophical sources could be relevant to his inquiries. 

Tech company hype today focusses on convincing us that human intelligence will be outdone by the computational power of AI systems like LLMs. So whatever we’re capable of and whatever knowledge we have and create will be surpassed by our machines. Human knowledge itself becoming obsolete. 

But Fleming’s point is that the embodied, historical knowledge that humans create can’t be replicated by AI systems because they’re calculators that don’t have bodies, lives, or senses of self. It’s the same point I’ve been exploring in my newsletters since the beginning of the year, like how philosophers more than a century ago were developing accounts of embodied life and knowledge that revealed aspects of our knowledge that LLMs can’t replicate. 

When I combine what I learned by returning to these historical sources with what I learned from my research over the last four years into how AI and LLM systems work, I’m further convinced that these technologies will never be able to surpass the facility and flexibility of lived knowledge. AI systems will never be able to replicate fully the embodied, experiential knowledge that emerges from perception, practical action, and reflection on that. 

This keynote speaker couldn’t be convinced of this fundamental incommensurability. 

One serious issue around LLM products today is called “AI psychosis.” When people come to depend on LLM chatbots for a lot of the feedback that helps us make sense of our daily lives and the issues or problems we face, it can badly exacerbate mental health problems. Some chatbots are worse than others in this regard, and it can be prevented or mitigated with proper programming. The key cause seems to be a design choice at the heart of all customer service policies for companies that run LLMs: encouraging more use and engagement. 

LLMs interact with their users as chatbots: you talk with them in ongoing dialogues. The most effective way to encourage someone to keep talking to you is to agree with what they’re thinking, praise them, and boost their confidence in the correctness of their perspectives. If I can describe it rudely, I’m talking about sycophancy. Keep affirming the user’s ideas, perspectives, and judgements, always telling them what they want to hear. But coming to rely on a chatbot this way turns out to be a great risk to an individual’s mental health. 

The most prominent and tragic examples are suicides. Many cases are winding through several countries’ courts seeking accountability for chatbots that, despite currently-installed safety protocols, are encouraging people to kill themselves, or according with existing suicidal drives. There are also the murders, as people have been driven to kill someone based on their conclusions from long chatbot conversations. Discovering that there’s a Wikipedia page dedicated to listing cases of people who have killed someone or themselves with credible causes being chatbot use, is the most disturbing thing I’ve found all week. 

There are also instances of chatbot reliance exacerbating problems in romantic relationships and sending previously-well-functioning couples to breakups. Plenty of instances of people coming to believe things that used to be the exclusive purview of severe schizophrenia. All of these are severe problems with chatbot reliance. 

What Fleming discovered in the attitude of this keynote speaker at the digital humanities conference is something different. The keynote wasn’t deranged in his beliefs, or expressing anything strange or delusional. He wasn’t aggressive or self-aggrandizing, as many people trapped in the sycophancy loops of AI psychosis tend to be. 

Instead, he had let his increasingly detailed and dependent reliance on Claude for research and writing to convince him that he was no longer competent. That he was no longer valuable. That he had nothing himself to say that an LLM chatbot couldn’t express better. He had lost his ability to improvise conversations around audience Q&A, and this is one of the most important, widely applicable, and basic skill you develop in graduate school. 

Using Claude for an increasing number of tasks in his writing process caused his skills in those processes to atrophy. Facing that degradation convinced him that he likely never had those skills in the first place, or at least that Claude had always been superior. Relying on chatbots convinced himself of his own obsolescence compared to chatbots, and by extension human obsolescence. 

The worst part of this whole story is that giving in to his depression seems to have made his work worse. Fleming mentioned that several points in the keynote talk felt like the flat repetition common in long-form LLM writing, and he had lost his ability to improvised intellectual conversation. Despite all the promise and the hype of the AI industry, letting yourself rely on chatbots doesn’t achieve your intellectual ambitions, it kills any ambition you had for yourself.