[GoLUG] LLMs can't reason

[Kyle Terrien]

On Wed, Jun 18, 2025 at 05:25:19PM -0400, Steve Litt wrote:
> On Tue, 17 Jun 2025 23:39:23 -0700
> Kyle Terrien <kyle at terren.us> wrote:
> 
> > The humanists who argue against the existence of a soul can’t provide
> > positive evidence that a soul *doesn’t* exist.
> > 
> > Just because something isn’t directly observed does not automatically
> > mean it doesn’t exist.  Consider the humble ext4 filesystem.  You’ll
> > never know about it or how it allocates blocks if all you use is
> > open(), read(), write(), and close() to observe it, but the ext4
> > filesystem still exists under the hood.
> 
> Just a review of how we got here. Kyle, you asserted that if something
> has reason then it also has free will, and somehow put in that free
> will depends on having a soul.

I said the former (that reason implies free will), but I don’t think I
made the latter connection (that free will implies a soul).  The
discussion about the nature of a soul was more of an offshoot: the
classical definition of a soul is the life force of a being, and the
human soul is also a spirit, etc.

> Slaves could reason, but they had no free will: They (mostly) did their
> masters' bidding. The ones who didn't suffered horrible punishments.
> One could say they had the free will to risk almost certain whipping
> and likely death by defying the master. Well, if that's your definition
> of free will, ChatGPT has that too, as evidenced by its once in a while
> doing unwanted things (remember your ancient history where a guy got
> ChatGPT to fall in love with him?) So we're already on thin ice. But
> wait, there's more: Your assertion that once's having free will depends
> on one's having a soul.

Slaves certainly had the choice to defy their masters.  That is free
will actualized.  The difference with ChatGPT is that ChatGPT never
has a choice.  It merely follows a mechanical algorithm.

> First, everybody needs to agree on a definition of "a soul". Good luck
> with that, and cherry picking ancient philosophers and biblical figures
> doesn't do the job. Next, armed with a specific definition of soul, we
> all have to agree that people have souls and LLMs don't. I guarantee I
> can find someone smarter than you who says humans have no soul, and
> someone smarter than him saying they do, and someone smarter than him
> saying they don't, and probably the smartest people say "I don't know".
> 
> Because you're dealing with a religious thing, which is belief of
> things for which there's no proof. Sure, they *might* be correct, but
> they might be wrong. So I make up my mind depending on many factors,
> and somebody's religious beliefs might make up a tiny part.

All good points, and this is where some of my personal convictions
show.

> Back to "LLMs can't reason". With every passing decade, your risk of
> being on the wrong side if history increases. We have already
> deconstructed the entire human genome. We're getting closer and closer
> to constructing a block diagram of a human, which kind of makes us just
> a machine. As we come to understand our machine hood, our "reasoning"
> devolves into a set of beliefs and tools. Well, I could write a
> computer program with beliefs and tools. Naturally, a carbon based
> human uses different tools and a different system of storing beliefs
> than a silicon based computer does, but the "reasoning" process is, at
> its foundation, remarkably similar.

We deconstructed the entire human genome, but we still don’t
understand it.

> I know that, for religious people, claiming that humans are machines is
> very offensive, but with each passing decade, we get closer to being
> able to build humans: Even customized humans.

Ah, here are the humanist and evolutionist themes creeping in again.

(And I’m not offended.  This stuff is everywhere, and we are bombarded
with it every day.  “The humans came from apes”, etc.)

The whole “super-human”, demi-god, /homo novus/, or whatever people
want to call it came from a humanist idea, that things tend to evolve
into more powerful things, by nature.  I, however, share the same
philosophy that Tolkien had, which is the same philosophy used by the
Catholic Church for centuries: it’s impossible to give something which
you do not possess.  In other words you can’t make something more
powerful unless you already possess the potential to make it more
powerful.

Darwin proposed an interesting theory in the early 1800s, which is
that certain traits are honed over the course of many generations.
However, humanism was in full swing at the time, and the humanists ran
with it.  According to a humanist, if a little trait such as the shape
of a bird’s beak can change over time, then why can’t big things
change, such as the number of appendages, or the degree of someone’s
intelligence?  Thus, came the idea that men were descended
biologically from monkeys.

But why stop there?  The humanists took it a step further and declared
that we were on a pathway to greatness, a path to define our own
destiny and become super-people.  If the fundamental law that you
can’t give what you don’t have doesn’t apply, then there is nothing
stopping us.

What they don’t tell you is that the story ends with the
“super-humanity” willing its own destruction, like in Paradise Lost or
in any of the dystopian movies you see nowadays.  I suppose the Q
Continuum from Star Trek fall under this category.  There is no doubt
that the Q are superior, but considering how far they stooped to get
there, who in their right mind would want to be Q or to interact with
Q?

Of course, the evolutionism and “super-humanity” narrative caused a
row, because men have something that monkeys do not possess.  You
would call it intelligence.  I call it a human soul, which is also a
spirit, created directly by God in His image.  Now, you can’t give
what you don’t possess, and monkeys certainly do not possess a human
spirit, so there is no way we could get a human spirit from monkeys.
Thus, the conundrum of creationism versus evolutionism.  Some
conjecture that we got the genetic code from the monkeys and human
spirit directly from God.  Some reject evolutionism altogether.

On the contrary, consider Tolkien’s worldview in Middle Earth and
Arda, which putting aside fairy tale aspects actually leans a lot on
Aquinian philosophy as a worldview.  In the Silmarillion, Eru
Illuvatar, the uncreated being, created the Valar who shared in
creation by composing a celestial “song”.  The elves awakened, the
humans came, and then things weakened over time.  Originally, there
were two gigantic towers of light illuminating all of Arda.  Then,
Melchor tore them down.  So, the Valar created two trees of light.
Then Ungoliant devoured them.  The one fruit and one leaf that
remained of the two trees was used to create the Sun and Moon.  You
can’t give what you don’t possess, and things tend toward more
entropy.  We are left to imagine what beholding the trees must have
been like, and we are left further to conjecture what the towers were
like.

By the time of the third age (when the events of Lord of the Rings
took place), Middle Earth was full of ruins left by ancient
civilizations which no race could ever rebuild.  There were Palantir
stones which were once used for communication but no one could ever
build again.  The legendary silmarils forged by elves in the west were
long gone, and what remained of them was in the little bauble which
Galadriel gave Frodo.  Likewise, while technology progresses between
interruptions in Middle Earth (there was certainly no pipeweed or
mithril until someone invented them), the civilizations tend toward
the arrow of entropy over time.

Anyway, why this tangent?  Oh yeah, because the “humans are machines”
is basically an equivalent argument to “humans came from apes”.  While
there are biological processes, which involve mechanical and
computational processes, it is impossible to accurately state that we
are limited to these biological, mechanical, and computational
processes; for the same reason it is inaccurate to say that we are
just apes.

Scientists who forget that we have a human spirit are certainly going
to be taught the hard way time and time again.

> A vast swath of humanity has an anti-AI prejudice because they feel
> like they could be made redundant. I think they're seeing the wrong
> problem. The real problem is that, with programs designing programs
> designing programs, and more and more AI machines and robots are
> closely networked together, we run the risk that the machines will try
> to wipe us out like in the "Terminator" movie. And if I were sure of
> that outcome, I'd be fighting tooth and nail against AI. But I'm not
> sure of it, so for the time being, ChatGPT has become an outstanding
> collaborator for me. And besides all that, I think we humans have
> set in motion a multi gigadeath depopulation quite independent of
> ill-will from machines.

Closing thought: we tend to anthropomorphise--i.e. attach human
attributes to things which are non-human, and I think that applies to
AI.  It is very easy to forget it is still a machine because it
appears the machine talks.

Coincidentally, in Tolkien, the elves (who came before all other
races) after discovering they were the only creature who can talk
thought the ability to talk was so special that they made a word for
it: they called themselves the /Quendi/, meaning “the ones who talk”.
That’s how important language is in the picture of things.

-- 
[*] Kyle Terrien
    Terrenus => from the Earth, to the Cloud
    https://terren.us/

Dilexisti justitiam, et odisti iniquitatem.  -- Psalmus 44:8