Neural Networks and Chatbots

What a difference a year makes!

In returning to blogging yesterday, I realised I hadn’t published the detailed blog entry I wrote just over a year ago, about ChatGPT.

Glancing at what I had written reminded me of how everything was still so new and amazing and exciting.  I was like an eager and naive kid when it came to AI.

A year on, I’ve learned heaps about AI – how it works, its impact on society and on the Internet as a whole, both for good and bad, and the controversies and subcultures that have developed around it.  In fact I’m currently working on several books, the first of which, Doomers, Accelerationists, and Superintelligence, is almost finished.

The original inspiration for that book was an open letter that came out in March 2023, calling for a six month moratorium on “Giant AI Experiments”. This meant any AI beyond the then just introduced GPT-4.

So there I was; a total AI nerd, excited about all the amazing creative things I can do with this amazing new technology (and not yet aware of its limitations and shortcomings), and what it means for the future, and these luddites just want to brakes on the whole thing!

This motivated me to study the “Doomer” worldview, since it is diametrically opposite my own worldview and orientation. I ended up with too much material to fit in a book, other than some of the essentials, which will go in Doomers, Accelerationists, and Superintelligence, and a companion volume: AI, Creativity, and Consciousness. For the rest, I’ve got plenty of material for blog posts.

The Perceptron

Generative AI is so called because it is a category of Artificial The Perceptron (AI) that can generate new, original content, based on a specific prompt that is typed in as input. That is, it won’t and can’t act on its own, it needs a human to type in the prompt. Depending on the type of AI the generated content may be text, images, sound, videos, or software code.  The interesting thing about it though is that it works exactly like the human brain, including software neurons, called nodes, artificial neurons, or perceptrons.  

The idea of an artificial neuron, that is, of a machine that works like the human or animal brain, actualy goes back quite a long way. The underlying theory was first described more than eighty years ago, which makes it the the Early Paleolithic Era of computing, by neurophysiologist and cybernetician Warren McCulloch and autistic genius and logician Walter Pitts write a ground-breaking paper called “A Logical Calculus of Ideas Immanent in Nervous Activity(pdf).

All this remained totally theoretical, until 1957, when the perceptron was actually run as a software on an IBM 704, one of those giant vacuum tube mainframes the size of a wall, that are so characteristic of 1950s and 60s sci-fi movies and TV shows. The program was developed by then 29-year-old psychologist Frank Rosenblatt in 1957 at Cornell University.  By feeding the computer a pile of punched cards, he showed it could learn to distinguish those marked on the left from those marked on the right.  This was the very dawn of machine learning.

Rosenblat later made a custom-built piece of computer hardware that he called  the “Mark I perceptron“. This was designed for image recognition. It had 3 layers, and worked using the same principles that generative AI still does today (the machine is currently in the Smithsonian National Museum of American History).

It would be another sixty years before the hardware improved to the extent that machine intelligence that mimics, actually that’s the wrong word, that works in the same way a biological brain does, on a similar scale, could actually be trained and run on banks of computers.  And thus we have today’s Generative AI.

How Generative AI works

Each of a Generative AI’s artificial neurons has a number of specific “weights” or biases, called parameters, which determines how they respond to other nodes. This is just like how neurons in the brain work, each neuron is linked to other neurons and responds according to the strength or weakness of its connection. Hence the term “neural network”.  

Deep Neural Network
A stylised diagram of neural net AI (from Analyzing and Comparing Deep Learning Models by Shibu Mohapatra)

The diagram here shows a stylised and greatly simplified diagram of a neural network.  Each node or artificial neuron is shown as a coloured circle, the parameters showing how they relate to other nodes by the arrows.  The nodes are arranged in many layers, which are hidden; that is, they are not directly observable from the input or output of the network.  

Like Rosenblatt and his punchcards, but much more sophisticated, the AI is trained by feeding data into the input layer. This is then processed by the various hidden layers. This is called forward propagation, or forward pass. The output is then checked against the real value or true value, which is what the correct response would be. If it deviates from the correct answer, the result is fed back the other way, a process called “back propagation” to determine where the errors come from. The weights are then adjusted and the whole process repeated until the AI gets it right. This is called deep learning, deep because it involves many layers of neurons.

GPT-4 belongs to the type of Generative AI that produce text. GPT and its buddies are called Large Language Model, or LLM for short. It’s called “Large” because it’s trained on absolutely enormous amounts of text.  Like, the entire internet.  This is what is meant by “Giant AI experiments”. And also I suppose the trillions of parameters that an AI like GPT-4 has.  It is incredibly impressive how advanced its responses, when using ChatGPT. You can also talk to the AI directly on the OpenAI site, but the app makes it more user friendly.


Years ago I used to get a chuckle from some of the YouTube videos of a chatbot called Cleverbot, a type of simple AI that generates output on the basis of a prompt. People would pit two cleverbots against each other to see what sort of hilarious things the AIs would come up with. 

These would be matched with an AI generated avatar and voice, occaisonally quite creatively as in this example, although that had to be added seperately from the original CleverBot output, and in that pre-Generative AI age, required a lot of processing time.  But it appears real time in the video, giving an appearance of anthropomorphic sentience. 

As a result, and adding the clever use of accents, and the random jumping of topics, the two cleverbots shown in the above YouTube video actually seem totally human, if amusingly and delightfully eccentric. It’s easy to imagine them as sentient AI, or AGI in the modern terminology, who squabble like an old married couple, muse on God and philosophy and talk about getting a body to explore the real world.

None of this is the case of course; Cleverbot doesn’t have that degree of sentience, but it’s so easy to anthropomorphise. 

Cleverbot vs ChatGPT

A lot of the way that ChatGPT responds is because of the way it has been trained, to be incredibly “agreeable” (in terms of the “Big Five” of psychological traits).

This is shown when Cleverbot talks to ChatGPT. Cleverbot says something random, ChatGPT will give a long polite response along with denying its own sentience, Cleverbot then gives a brief response, ChatGPT then tediously repeats itself, and so on. The striking thing is that in this video, Cleverbot comes across as much more human, perhaps because it was specifically designed for conversation and response.

I don’t know if ChatGPT’s tedious responses and disclaimers (see the previous link) are because GPT-4 would normally say more but is deliberately constrained by the Chatbot app, or if the AI itself has been trained to be nice and obedient. In any case, although ChatGPT is fantastic as a creative partner, when talking to itself, without human input, it isn’t anywhere near as good. It goes fine for a while but soon runs out of things to say and gets stuck in an infinite loop.

If a chatbot like Cleverbot represents a simple type of AI (apparently it’s a rule-based AI), albeit one optimised for conversation, and ChatGPT a more advanced one (being a neural network AI with billions or trillions of parameters, generating text from atterns recovered from very large databases), an even more advanced AI would be one capable of innovation and original thinking. This would be a much more sophisticated neural net that would be able to avoid infinite loops. Such an AI would be an example of what’s called an AGI, and Artificial General Intelligence.

ChatGPT works best as part of a human-AI team. Joint human-AI creative content (whether fiction, nonfiction, art, or music), can be considered “cyborg”, as it combines machine and biological organism, which would be greatly advanced over GPT-4.

This is why I’m really looking forward to the next iteration of GPT, regardless of whether it’s called GPT-4.5, GPT-5, or something else. How will it enhance creativity? And what about the next level of AI beyond that? Will it be the fabled AGI, the holy grail of AI research?

Doomers and Accelerationists

I’ve mentioned Doomers, but what about the opposite camp, the Accelerationists. Accelerationists are sometimes also called Boomers. Nothing to do with Baby Boomer, but only because it rhymes with Doomer.

The simplest way to describe Doomers and Boomers is to say that these are the pessimists and the optimists of future AI technology. Both agree that AI will quickly become more and more advanced until it surpasses humans in the very near future. And I mean a timescale of, say, five years or two or three decades. But Doomers see this as a really scary thing, because they believe that AI may decide to, or may simply accidentally (like stepping on an ant nest without knowing it) wipe out humanity, or even all life on Earth. The only solution is to radically, dramatically, slow down research into and development of AI, until they are sure they can solve this problem by ensuring the AI won’t act that way (this is called Alignment).

Accelerationists see this whole doomsday scenario as ridiculous. They consider that, in whatever way AI develops, whether it becomes superintelligent or not, merges with humans or not, things will work out fine. In fact the faster technology advances, and capitalism advances, the better.

There’s also a third group, who can be called Sceptics or Dismissers. They are not necessarily Accelerationists, but like them they consider the Doomer scenario to be ridiculous and a distraction from the real concerns at hand, such as the misuse of Generative AI today (deep fakes, phishing, influencing elections etc).

In this taxonomy of groups, subcultures, and responses to the coming superintelligent AI, there’s also the distinction between the hardcore Doomers, the Doomers sensu stricto, who believe all further AI progress needs to be totally halted, and Doomers sensu lato, who acknowledge AI as useful, even necessary, development should move slowly and cautiously. Some even say the industry should be heavily government regulated and controlled, restrcting advanced AI to a few approved corporate monopolies. This latter, rather poorly defined, group has been called “Worriers” because they are worried about AI development, but are not full-on anti-AI like the hard-core Doomers are. Instead they’re concered with things like AI Safety and getting alignment right. Most of the signatories to the Open Letter calling for a moratorium on AI development are in this category, with a sprinkling of hard-core Doomers.

The Accelerationists and the Sceptics are totally against regulation, which they see as stifling innovation and concentrating power in the hands of a few corporations. They tend to “open source” software and, in the case of many Accelerationists, libertarian (unregulated) capitalism.

But although Doomers (and like-minded types) seem to get the most publicity with their apocalyptic scenarios and dire threats of human extinction if AI and advanced genetic engineering is not reigned in, this seems to be more about the media’s love of sensationalism. However, I was very pleased to read of one survey had two-thirds identifying as Boomers, and only one third as Doomer. So while I’m normally very much a minority opinion person, it’s great to see that the Doomers and their AI technophobia are not such a big deal.

Leave a Reply

Your email address will not be published. Required fields are marked *