[GoLUG] LLMs can't reason

[Steve Litt]

On Tue, 17 Jun 2025 23:39:23 -0700
Kyle Terrien <kyle at terren.us> wrote:

> The humanists who argue against the existence of a soul can’t provide
> positive evidence that a soul *doesn’t* exist.
> 
> Just because something isn’t directly observed does not automatically
> mean it doesn’t exist.  Consider the humble ext4 filesystem.  You’ll
> never know about it or how it allocates blocks if all you use is
> open(), read(), write(), and close() to observe it, but the ext4
> filesystem still exists under the hood.

Just a review of how we got here. Kyle, you asserted that if something
has reason then it also has free will, and somehow put in that free
will depends on having a soul.

Slaves could reason, but they had no free will: They (mostly) did their
masters' bidding. The ones who didn't suffered horrible punishments.
One could say they had the free will to risk almost certain whipping
and likely death by defying the master. Well, if that's your definition
of free will, ChatGPT has that too, as evidenced by its once in a while
doing unwanted things (remember your ancient history where a guy got
ChatGPT to fall in love with him?) So we're already on thin ice. But
wait, there's more: Your assertion that once's having free will depends
on one's having a soul.

First, everybody needs to agree on a definition of "a soul". Good luck
with that, and cherry picking ancient philosophers and biblical figures
doesn't do the job. Next, armed with a specific definition of soul, we
all have to agree that people have souls and LLMs don't. I guarantee I
can find someone smarter than you who says humans have no soul, and
someone smarter than him saying they do, and someone smarter than him
saying they don't, and probably the smartest people say "I don't know".

Because you're dealing with a religious thing, which is belief of
things for which there's no proof. Sure, they *might* be correct, but
they might be wrong. So I make up my mind depending on many factors,
and somebody's religious beliefs might make up a tiny part.

Back to "LLMs can't reason". With every passing decade, your risk of
being on the wrong side if history increases. We have already
deconstructed the entire human genome. We're getting closer and closer
to constructing a block diagram of a human, which kind of makes us just
a machine. As we come to understand our machine hood, our "reasoning"
devolves into a set of beliefs and tools. Well, I could write a
computer program with beliefs and tools. Naturally, a carbon based
human uses different tools and a different system of storing beliefs
than a silicon based computer does, but the "reasoning" process is, at
its foundation, remarkably similar.

I know that, for religious people, claiming that humans are machines is
very offensive, but with each passing decade, we get closer to being
able to build humans: Even customized humans.

A vast swath of humanity has an anti-AI prejudice because they feel
like they could be made redundant. I think they're seeing the wrong
problem. The real problem is that, with programs designing programs
designing programs, and more and more AI machines and robots are
closely networked together, we run the risk that the machines will try
to wipe us out like in the "Terminator" movie. And if I were sure of
that outcome, I'd be fighting tooth and nail against AI. But I'm not
sure of it, so for the time being, ChatGPT has become an outstanding
collaborator for me. And besides all that, I think we humans have
set in motion a multi gigadeath depopulation quite independent of
ill-will from machines.


SteveT

Steve Litt 
Spring 2023 featured book: Troubleshooting Techniques of the Successful
Technologist http://www.troubleshooters.com/techniques