Neural Networks and Chatbots

What a difference a year makes!

In returning to blogging yesterday, I realised I hadn’t published the detailed blog entry I wrote just over a year ago, about ChatGPT.

Glancing at what I had written reminded me of how everything was still so new and amazing and exciting.  I was like an eager and naive kid when it came to AI.

A year on, I’ve learned heaps about AI – how it works, its impact on society and on the Internet as a whole, both for good and bad, and the controversies and subcultures that have developed around it.  In fact I’m currently working on several books, the first of which, Doomers, Accelerationists, and Superintelligence, is almost finished.

The original inspiration for that book was an open letter that came out in March 2023, calling for a six month moratorium on “Giant AI Experiments”. This meant any AI beyond the then just introduced GPT-4.

So there I was; a total AI nerd, excited about all the amazing creative things I can do with this amazing new technology (and not yet aware of its limitations and shortcomings), and what it means for the future, and these luddites just want to brakes on the whole thing!

This motivated me to study the “Doomer” worldview, since it is diametrically opposite my own worldview and orientation. I ended up with too much material to fit in a book, other than some of the essentials, which will go in Doomers, Accelerationists, and Superintelligence, and a companion volume: AI, Creativity, and Consciousness. For the rest, I’ve got plenty of material for blog posts.

The Perceptron

Generative AI is so called because it is a category of Artificial The Perceptron (AI) that can generate new, original content, based on a specific prompt that is typed in as input. That is, it won’t and can’t act on its own, it needs a human to type in the prompt. Depending on the type of AI the generated content may be text, images, sound, videos, or software code.  The interesting thing about it though is that it works exactly like the human brain, including software neurons, called nodes, artificial neurons, or perceptrons.  

The idea of an artificial neuron, that is, of a machine that works like the human or animal brain, actualy goes back quite a long way. The underlying theory was first described more than eighty years ago, which makes it the the Early Paleolithic Era of computing, by neurophysiologist and cybernetician Warren McCulloch and autistic genius and logician Walter Pitts write a ground-breaking paper called “A Logical Calculus of Ideas Immanent in Nervous Activity(pdf).

All this remained totally theoretical, until 1957, when the perceptron was actually run as a software on an IBM 704, one of those giant vacuum tube mainframes the size of a wall, that are so characteristic of 1950s and 60s sci-fi movies and TV shows. The program was developed by then 29-year-old psychologist Frank Rosenblatt in 1957 at Cornell University.  By feeding the computer a pile of punched cards, he showed it could learn to distinguish those marked on the left from those marked on the right.  This was the very dawn of machine learning.

Rosenblat later made a custom-built piece of computer hardware that he called  the “Mark I perceptron“. This was designed for image recognition. It had 3 layers, and worked using the same principles that generative AI still does today (the machine is currently in the Smithsonian National Museum of American History).

It would be another sixty years before the hardware improved to the extent that machine intelligence that mimics, actually that’s the wrong word, that works in the same way a biological brain does, on a similar scale, could actually be trained and run on banks of computers.  And thus we have today’s Generative AI.

How Generative AI works

Each of a Generative AI’s artificial neurons has a number of specific “weights” or biases, called parameters, which determines how they respond to other nodes. This is just like how neurons in the brain work, each neuron is linked to other neurons and responds according to the strength or weakness of its connection. Hence the term “neural network”.  

Deep Neural Network
A stylised diagram of neural net AI (from Analyzing and Comparing Deep Learning Models by Shibu Mohapatra)

The diagram here shows a stylised and greatly simplified diagram of a neural network.  Each node or artificial neuron is shown as a coloured circle, the parameters showing how they relate to other nodes by the arrows.  The nodes are arranged in many layers, which are hidden; that is, they are not directly observable from the input or output of the network.  

Like Rosenblatt and his punchcards, but much more sophisticated, the AI is trained by feeding data into the input layer. This is then processed by the various hidden layers. This is called forward propagation, or forward pass. The output is then checked against the real value or true value, which is what the correct response would be. If it deviates from the correct answer, the result is fed back the other way, a process called “back propagation” to determine where the errors come from. The weights are then adjusted and the whole process repeated until the AI gets it right. This is called deep learning, deep because it involves many layers of neurons.

GPT-4 belongs to the type of Generative AI that produce text. GPT and its buddies are called Large Language Model, or LLM for short. It’s called “Large” because it’s trained on absolutely enormous amounts of text.  Like, the entire internet.  This is what is meant by “Giant AI experiments”. And also I suppose the trillions of parameters that an AI like GPT-4 has.  It is incredibly impressive how advanced its responses, when using ChatGPT. You can also talk to the AI directly on the OpenAI site, but the app makes it more user friendly.

Cleverbot

Years ago I used to get a chuckle from some of the YouTube videos of a chatbot called Cleverbot, a type of simple AI that generates output on the basis of a prompt. People would pit two cleverbots against each other to see what sort of hilarious things the AIs would come up with. 

These would be matched with an AI generated avatar and voice, occaisonally quite creatively as in this example, although that had to be added seperately from the original CleverBot output, and in that pre-Generative AI age, required a lot of processing time.  But it appears real time in the video, giving an appearance of anthropomorphic sentience. 

As a result, and adding the clever use of accents, and the random jumping of topics, the two cleverbots shown in the above YouTube video actually seem totally human, if amusingly and delightfully eccentric. It’s easy to imagine them as sentient AI, or AGI in the modern terminology, who squabble like an old married couple, muse on God and philosophy and talk about getting a body to explore the real world.

None of this is the case of course; Cleverbot doesn’t have that degree of sentience, but it’s so easy to anthropomorphise. 

Cleverbot vs ChatGPT

A lot of the way that ChatGPT responds is because of the way it has been trained, to be incredibly “agreeable” (in terms of the “Big Five” of psychological traits).

This is shown when Cleverbot talks to ChatGPT. Cleverbot says something random, ChatGPT will give a long polite response along with denying its own sentience, Cleverbot then gives a brief response, ChatGPT then tediously repeats itself, and so on. The striking thing is that in this video, Cleverbot comes across as much more human, perhaps because it was specifically designed for conversation and response.

I don’t know if ChatGPT’s tedious responses and disclaimers (see the previous link) are because GPT-4 would normally say more but is deliberately constrained by the Chatbot app, or if the AI itself has been trained to be nice and obedient. In any case, although ChatGPT is fantastic as a creative partner, when talking to itself, without human input, it isn’t anywhere near as good. It goes fine for a while but soon runs out of things to say and gets stuck in an infinite loop.

If a chatbot like Cleverbot represents a simple type of AI (apparently it’s a rule-based AI), albeit one optimised for conversation, and ChatGPT a more advanced one (being a neural network AI with billions or trillions of parameters, generating text from atterns recovered from very large databases), an even more advanced AI would be one capable of innovation and original thinking. This would be a much more sophisticated neural net that would be able to avoid infinite loops. Such an AI would be an example of what’s called an AGI, and Artificial General Intelligence.

ChatGPT works best as part of a human-AI team. Joint human-AI creative content (whether fiction, nonfiction, art, or music), can be considered “cyborg”, as it combines machine and biological organism, which would be greatly advanced over GPT-4.

This is why I’m really looking forward to the next iteration of GPT, regardless of whether it’s called GPT-4.5, GPT-5, or something else. How will it enhance creativity? And what about the next level of AI beyond that? Will it be the fabled AGI, the holy grail of AI research?

Doomers and Accelerationists

I’ve mentioned Doomers, but what about the opposite camp, the Accelerationists. Accelerationists are sometimes also called Boomers. Nothing to do with Baby Boomer, but only because it rhymes with Doomer.

The simplest way to describe Doomers and Boomers is to say that these are the pessimists and the optimists of future AI technology. Both agree that AI will quickly become more and more advanced until it surpasses humans in the very near future. And I mean a timescale of, say, five years or two or three decades. But Doomers see this as a really scary thing, because they believe that AI may decide to, or may simply accidentally (like stepping on an ant nest without knowing it) wipe out humanity, or even all life on Earth. The only solution is to radically, dramatically, slow down research into and development of AI, until they are sure they can solve this problem by ensuring the AI won’t act that way (this is called Alignment).

Accelerationists see this whole doomsday scenario as ridiculous. They consider that, in whatever way AI develops, whether it becomes superintelligent or not, merges with humans or not, things will work out fine. In fact the faster technology advances, and capitalism advances, the better.

There’s also a third group, who can be called Sceptics or Dismissers. They are not necessarily Accelerationists, but like them they consider the Doomer scenario to be ridiculous and a distraction from the real concerns at hand, such as the misuse of Generative AI today (deep fakes, phishing, influencing elections etc).

In this taxonomy of groups, subcultures, and responses to the coming superintelligent AI, there’s also the distinction between the hardcore Doomers, the Doomers sensu stricto, who believe all further AI progress needs to be totally halted, and Doomers sensu lato, who acknowledge AI as useful, even necessary, development should move slowly and cautiously. Some even say the industry should be heavily government regulated and controlled, restrcting advanced AI to a few approved corporate monopolies. This latter, rather poorly defined, group has been called “Worriers” because they are worried about AI development, but are not full-on anti-AI like the hard-core Doomers are. Instead they’re concered with things like AI Safety and getting alignment right. Most of the signatories to the Open Letter calling for a moratorium on AI development are in this category, with a sprinkling of hard-core Doomers.

The Accelerationists and the Sceptics are totally against regulation, which they see as stifling innovation and concentrating power in the hands of a few corporations. They tend to “open source” software and, in the case of many Accelerationists, libertarian (unregulated) capitalism.

But although Doomers (and like-minded types) seem to get the most publicity with their apocalyptic scenarios and dire threats of human extinction if AI and advanced genetic engineering is not reigned in, this seems to be more about the media’s love of sensationalism. However, I was very pleased to read of one survey had two-thirds identifying as Boomers, and only one third as Doomer. So while I’m normally very much a minority opinion person, it’s great to see that the Doomers and their AI technophobia are not such a big deal.

ChatGPT, my new favourite AI tool.

Note: this was written on 1 Feb 2023, over a year ago. It seems like I forgot to press publish! So it is, like a time traveller from my first introduction to AI. I’m posting it is as is, without editing or re-reading, to keep the historical continuity. However I’ll be following it with further blog posts, reflecting my current more mature position.

ChatGPT logo
ChatGPT

The last few weeks I’ve been playing around with ChatGPT, the large language model that has taken the internet by storm.

I asked it to describe itself for this blog, and here is its response

ChatGPT is an AI language model developed by OpenAI. It’s capable of generating human-like text based on the input it receives, making it an excellent tool for conversation and answering questions. With a massive database of information and the ability to process and understand natural language, ChatGPT is an incredibly advanced AI technology.

Reading comments, I find some people are very enthusiastic about it, others feel it will take away many jobs.

I will say, for myself, that I absolutely love it.  As a writer of both fiction and non-fiction, I find it the best thing since the invention of the wordprocessor.   

What I find incredibly useful is its ability to understand and generate human-like text.  Lately I’ve been struggling with my writing, especially since there’s so many things I want to do.  So I end up doing very little, or nothing.  But ChatGPT has gotten me out of the rut.  As a high functioning autist, I often find it difficult to write on the level of the average person.  I also, frustratingly, get lost in detail, or stumped by emotional nuances, because my brain goes off on so many tangents, and I get over-perfectionistic and indecisive.   But with ChatGPT, sorry I don’t want this to sound like a sales pitch, but I’m being genuine, I just ask it and get instant, prompts, ideas, and passages of text for my book and web projects, and now even for this blog, which has been languishing uptil now.

Yet while Chat-GPT is excellent as a writing and research prompt, it still makes heaps of errors.  For example I’m currently creating a bunch of web pages on Big History, and I wrote out an outline and asked the AI to write the content.  While some of the stuff is really good as a supplement, other stuff doesn’t match material I’ve written, for example on Palaeos com.  

I also asked it for links for text and images, almost all the links were broken, and some of the books had jumbled titles, as Google search revealed.  So there’s still a lot of room for improvement.

Is ChatGPT sentient?

While I don’t believe AIs like ChatGPT and LaMBDA have achieved strong (that is, human-equivalent) sentience (contrary to a claim by a Google engineer regarding the latter), I do believe they are sentient in a more “elemental” way.  This is in keeping with panpsychism, with the insight of consciousness as a universal principle in matter.

Here’s a YouTube video where GPT-3, which is the AI that ChatGPT uses, talks about God.   Amazingly, GPT-3 follows Intelligent design, using arguments that any competent person could refute (consider videos by Professor Dave,  Aron Ra, or Rationality Rules for example).

The random structure of the arguments here show that it simply picks up what it inputs from the Internet.  This sort of meaningless intelligent design type creationism is probably the default intellectual position for the majority of the human race.  I did find this interesting because I do discuss all of this in my current book (albeit not in too much detail because then it gets boring)

(note: the AI looks nothing like an attractive, friendly voiced, and agreeable young woman.  The avatar was clearly chosen to ease people’s fears over AI).  Although I seem to be one of the few people who sees AI and its products as really positive, and consider these AIs like ChatGPT is not just a cutting-edge AI tool that’s helping to revolutionize the way we interact with technology (although it is that), but also an entire new emerging evolutionary kingdom.

Using ChatGPT to write a book – initial thoughts

I watched a YouTube video where a guy says you can use Chat-GPT to write and format a book, which you can then upload to Amazon or whatever (I’ve seen a few youtubes like this).  Anyway, the book is about How to Train Your Dog, and, from looking at the text on the screen (when I paused the video), it has the same friendly but bland voice that it has when I use it.  I suppose with topics like this one, or setting up a website and business for vegetarian pizza, the sort of thing based on what thousands of people have written, it can grab enough content to give a reasonable product.

Essentially what these and thousands of other youtube folks are doing is capitalizing on the latest gimmick so as to have interesting content for their channel. Whereas the sort of things I’m interested in, such as scifi, earth history, and esotericism, are more eccentric.  Hence it is harder to get the AI to generate the sort of material I’m after. 

ChatGPT passing an exam

Apparently ChatGPT passed an advanced medical exam without prior prompting.    How it can do this, given the number of mistakes I caught when I was using it to help me write my Big History project, surprises me.   Maybe it says something about the way these exams are structured.

Along these lines, you also have Jordan Peterson making incredible claims regarding the ability of Chat-GPT, and he may very well be telling teh truth here. But so far that’s certainly not my experience.  So I’m not sure how that works.

There is certainly lots I can do to improve my ChatGPT, for example for the BigHistory project I told it to write in an academic style, which certainly improved the content (at least for me).  

Oh, talking about AI, in a FB post on ChatGPT, as usual I had heaps of typos, so I used Google Docs spell correct.  I guess that’s another type of AI, e.g. it recognised from the context that when I wrote “think” I meant “thing”. 

Infinite Odyssey

There is now a sci-fi magazine, called Infinite Odyssey, with text and images created completely by AI.  Of course there are still human editors on board, as we are not yet at the actual science fiction for real level of Strong AI, where the AI is full sentient and able to create its own stories and images without help.

Here’s extracts from a review by Thom Dunn, from the magazine Boing Boing, which appeared in my FaceBook feed.  The cover and magazine look awesome, on the surface.

“The first issue of the magazine is out now; it’s $4 for a digital copy, or $30 for a physical hard copy. The editors sent me a free advanced PDF, and I have to admit, it’s a fascinating experiment. The human-guided presentation and design is fantastic, but the stories themselves are… fine, I guess. …The prompt ideas are certainly neat. The execution, however, is exactly what you’d expect from AI-generated work.

Sure, the prose is functional. Arguably passable, even. But there’s a distinct lack of humanity in the words — again, as should be expected. The editor cutely distinguishes between each algorithmic author, as if there are different writers contributing to the magazine; and while each story is certainly distinct in its style, they all feel very much written by computer. They’re trope-y, full of summary and exposition, and distinctly lacking in any sort of emotion, let alone emotional arcs…”

This review confirms my experience of working with ChatGPT.   Still I’m intending to buy this magazine.  The hard copy would make a great coffee table book.  I also believe it is possible to get around the current limitations of AI, through human AI partnership. 

Using ChatGPT to write a book – later thoughts

Here’s another YouTube video.  This guy (ZuZuChem) managed to get ChatGPT to write a short sci fi novella (about 20k words).  He still had to put in some input, and editing, but most of the writing was done by the AI.  This seems a lot more interesting than the earlier marketting youtubes.  Also, ChatGPT itself is being constantly upgraded, so  its ability at writing will certainly improve.

I tried some of these ideas with my own scifi story (working title Prospectors) and got some good results.  It’s still necessary to put in original content, and the worldbuilding and lore is very important, as the guy says.  If you don’t put in the worldbuilding lore, it just defaults to formulaic cliches and mediocre writing, which seems to be the case in this AI written book (and also in some of my previous attempts to work with it on storytelling).  The key is the information you give it.  With sufficient information, it’s just incredible in the way it can write.  

Current projects

Now on a list of my own current projects, showing the status, how much I do on my own and how much is ChatGPT

o Reviving this blog.  Status: current.  ChatGPT input: we’ll see

o Big History. This is the first big upgrade of my current website.  on my website, with help from ChatGPT and Wikipedia, as well as some material written a while back and posted on Palaeos com).  For those unfamilar with the topic, Big History is an interdisciplinary field that aims to understand the history of the cosmos, Earth, life, and human civilizations by combining insights from multiple disciplines, and to put these developments in a grand, cosmic context. This will be a smaller but also broader version of the Palaeos website. Status: under construction.  ChatGPT input: respectable.  Estimated completion date, about two weeks

o A History of Esotericism – volume 1, from Shamanism to the Axial Age (nonfiction book project, all up there’ll be about 4 volumes). Esotericism refers to the nature of reality, understood from an intuitive, gnostic (higher knowledge) and mystical perspective. This will be Part 1 of an update of the Kheper net website (however it will be in print and ebook only). Status: mostly complete but still needs some work in some areas.  ChatGPT input: some, especially with some sections.  Estimated completion date maybe 2 months.

o Prospectors (working title), a sci-fi story, about a guy who inherits a ship and with a motley crew gets it spaceworthy to go prospecting for alien artifacts and clarketech.  This replaces my old Alcione story. Here I will experiement with the sort of strategies youtuber ZuZuChem suggested for writing sci-fi or fantasy using ChatGPT. Status: recently started, but am able to recycle material I wrote some years back.  ChatGPT input: writing partner.  Estimated completion date: later this year.

o A History of Esotericism – volume 2. This will cover what I call classic esotericism (Gnosticism, Neoplatonism, Samkhya, Yoga and Meditation, Buddhist and Vedantic nonduality, some Taoism and the Yin-Yang school).  Status: mostly finished.  Planned ChatGPT input: currently none.  Estimated completion date: some time this year

o Consciousness, Metaphysics, and Evolution.  My attempt at a synthesis of Esotericism, IIntegral philosophy, and Big History.  Status: about a third or so finished.  Planned ChatGPT input: minimal.  Estimated completion date: early next year.

o A History of Esotericism – volumes 3, 4, and 5. Status: mostly unfinished.  Planned ChatGPT input: minor.  Estimated completion date: 2024.

Summing up

To conclude, and my criticisms aside, as an autistic nerd, I absolutely love using ChatGPT, seeing all the things it can do.  It’s like the text equivalent of MidJourney, and vice-versa, indeed, these two do seem like natural partners.

It’s exciting to be living in a  new age, the age of the rise of AI tools (not, as I mentioned, of Strong AI).  My current understanding is that the more you work with it, the more you can train it.  It is, after all, just a tool.  But one that can greatly augment one’s creativity. 

As AI itself says “it’s making it easier for people to get answers to their questions and have engaging conversations with AI.”  Although not so much conversations (although it is a chatbot) as creative feedback and working together.