[GoLUG] LLMs can't reason

[Kyle Terrien]

On Wed, Jun 18, 2025 at 02:31:35AM -0400, Steve Litt wrote:
> ============================================================
> 
> Stevelitt: ChatGPT, do you have a soul?
> 
> ChatGPT: No, I do not have a soul. I’m a computer program — a very
> large statistical model that generates text based on patterns in data.
> I have no consciousness, awareness, feelings, or existence beyond the
> servers running me. I can talk about souls, simulate personalities, and
> discuss philosophy, but it’s all just output from algorithms, not an
> expression of inner life.
> 
> Stevelitt: Do I have a soul?
> 
> ChatGPT: That is a profound and ancient question — one that people have
> debated for thousands of years. It depends on what you believe about
> human nature, consciousness, and existence.
> 
> Many religious and spiritual traditions say yes — you have a soul, an
> immaterial essence that persists beyond your physical body.
> 
> Some philosophers and scientists argue that what we call a “soul” is
> really a product of the brain — consciousness arising from neurons,
> with no separate “soul” entity.
> 
> Others say the question itself may not have a clear answer — that terms
> like “soul” are metaphors for things we don't yet fully understand.
> 
> Ultimately, whether you have a soul is a personal question — your
> answer depends on your beliefs, values, experiences, and how you view
> life and consciousness.
> 
> If you want, you can tell me a little about what you believe or are
> curious about, and we can talk it through.
> ============================================================
> 
> So ChatGPT believes it has no soul, and basically says maybe I have
> one, and that decision is pretty much up to me.

Long ago, I ran a few very similar test conversations with ChatGPT and
Gemini (when it was in beta).  Unfortunately, the models are trained
on the Internet, so they inherit all the contradictory popular
psychology on the Internet (and in scholarly sources nowadays).

However, when I asked specifically, “What does the philosophy of
Thomas Aquinas say about the subject?” I got some more concrete
answers (in line with Aquinas) at the exclusion of the other
philosophical schools.

--Kyle

-- 
[*] Kyle Terrien
    Terrenus => from the Earth, to the Cloud
    https://terren.us/

Dilexisti justitiam, et odisti iniquitatem.  -- Psalmus 44:8