[GoLUG] LLMs can't reason

[Steve Litt]

On Wed, 18 Jun 2025 21:16:22 -0700
Kyle Terrien <kyle at terren.us> wrote:

> On Wed, Jun 18, 2025 at 02:31:35AM -0400, Steve Litt wrote:
> > ============================================================
> > 
> > Stevelitt: ChatGPT, do you have a soul?
> > 
> > ChatGPT: No, I do not have a soul. I’m a computer program — a very
> > large statistical model that generates text based on patterns in
> > data. I have no consciousness, awareness, feelings, or existence
> > beyond the servers running me. I can talk about souls, simulate
> > personalities, and discuss philosophy, but it’s all just output
> > from algorithms, not an expression of inner life.
> > 
> > Stevelitt: Do I have a soul?
> > 
> > ChatGPT: That is a profound and ancient question — one that people
> > have debated for thousands of years. It depends on what you believe
> > about human nature, consciousness, and existence.
> > 
> > Many religious and spiritual traditions say yes — you have a soul,
> > an immaterial essence that persists beyond your physical body.
> > 
> > Some philosophers and scientists argue that what we call a “soul” is
> > really a product of the brain — consciousness arising from neurons,
> > with no separate “soul” entity.
> > 
> > Others say the question itself may not have a clear answer — that
> > terms like “soul” are metaphors for things we don't yet fully
> > understand.
> > 
> > Ultimately, whether you have a soul is a personal question — your
> > answer depends on your beliefs, values, experiences, and how you
> > view life and consciousness.
> > 
> > If you want, you can tell me a little about what you believe or are
> > curious about, and we can talk it through.
> > ============================================================
> > 
> > So ChatGPT believes it has no soul, and basically says maybe I have
> > one, and that decision is pretty much up to me.  
> 
> Long ago, I ran a few very similar test conversations with ChatGPT and
> Gemini (when it was in beta).  Unfortunately, the models are trained
> on the Internet, so they inherit all the contradictory popular
> psychology on the Internet (and in scholarly sources nowadays).
> 
> However, when I asked specifically, “What does the philosophy of
> Thomas Aquinas say about the subject?” I got some more concrete
> answers (in line with Aquinas) at the exclusion of the other
> philosophical schools.

Of course. And if I asked what Bertrand Russell would say, I'd get
concrete answers very different to the ones you got.

"He said it, I believe it, that settles it" is not a valid way to
evaluate technology, or even human philosophy or psychology.

SteveT

Steve Litt 
Spring 2023 featured book: Troubleshooting Techniques of the Successful
Technologist http://www.troubleshooters.com/techniques