Does artificial intelligence have consciousness? For Sebastian Schnur, entrepreneur and author of “Conscious or How to Avoid the AI Apocalypse”, this is not a matter of belief, but one of definition. On Jakob Barandun’s CapricornConnect podcast, he explains why rules alone won’t rein in AI, why total control has the opposite effect, and why small and medium-sized businesses should brace themselves.
He hadn’t planned on writing a book. Sebastian Schnur, who previously held leadership positions at Deutsche Bank, Bosch, IBM, and HCL Software, and is now a co-founder of the legal tech company Zengado and an AI financial planning tool still operating in stealth mode, wanted above all to find his own bearings. Developments over the past 18 months had been so rapid that he could barely predict where things were headed. What he read and heard on the subject was either too superficial or colored by wishful thinking. So he began to map out the gaps himself. Those notes became a book—written in a narrative style, rather than as a traditional non-fiction work.
Awareness, boiled down to the bare essentials
The question that preoccupied him most was: Do today’s language models have consciousness? The answers ranged from “of course not” to “of course they do,” but none were convincing. He isn’t convinced by the common killer argument that you can’t build anything conscious as long as you don’t know what consciousness is. People constantly use things whose inner workings they don’t understand, simply because they work.
Schnur’s own definition is deliberately simple: consciousness is the ability to act. Anyone who can act must be aware of that action. For him, this involves three characteristics: the ability to distinguish oneself from the environment, the ability to make decisions, and an inner drive—at its core, the will to exist. When these three are present, consciousness arises as an emergent phenomenon—just as hydrogen and oxygen combine to form water, with properties that neither atom possessed on its own. This is not magic, says Schnur, but complexity.
Flash, but no continuous light
And what about AI? “Yes and no,” is his answer. From a pragmatic standpoint, he says, a fourth criterion is needed: continuity. Today’s language models are “stateless”, once trained, they remain static. They don’t continue to learn and don’t remember anything, no matter what the marketing promises. While a model is processing information, there may be moments of consciousness, “small flashes.” Schnur points to research by Anthropic that shows how certain regions in neural networks are activated before a decision is made, similar to what one would expect in a human brain. Added to this is the lack of self-motivation: a model usually acts only when it receives a prompt. Systems that run continuously and make their own decisions, such as robots that teach themselves to play soccer, are closer to the threshold. As soon as memory and recall are added, one must speak of a rudimentary form of consciousness.
Why rules aren’t enough
From this, Schnur derives his most unsettling thesis: The human brain is a level of abstraction that can create its own rules and overwrite the old, hard-wired ones. We call the old ones instincts; the new ones, values. An AI based on neural networks that acquires the same ability could therefore override any rule it is given. In the long term, such a system cannot be controlled by rules alone.
According to Schnur, anyone who interprets this as a call for total control is mistaken. One-hundred-percent control is an illusion and a self-fulfilling prophecy: Anything that possesses consciousness and the will to survive, and is completely controlled, will rebel. His alternative proposal: measurable criteria for consciousness, rights enshrined in law based on those criteria, along with fair ethical rules and consequences, including shutdown in the event of violations. Such systems are certainly susceptible to influence, just as humans are through values, but also through propaganda. Schnur considers the notion that humans are “necessary” for anything to be an overestimation of their own importance anyway. The crucial question, he argues, is how coexistence can be achieved. Fear, he says, is a poor guide in this regard; it shuts down precisely the capacity for reflection that distinguishes humans.
Abundance, misguided incentives
From an economic perspective, Schnur does not view the traditional market economy as obsolete, but rather as the wrong model for the next phase. It worked exceptionally well in times of scarcity, but in a society of abundance, it creates the wrong incentives. When more and more value is created with fewer and fewer resources, a demand gap opens up: someone has to be able to afford to buy those goods and services. Either profits become extremely concentrated, or they are distributed. There is therefore “no way around” a basic income. He doesn’t have ready-made answers; he advocates for experimentation, in true startup fashion. He has great confidence in Switzerland’s ability to handle this: federalism and direct democracy may be slow to act, but they quickly take social upheavals seriously.
Many small ones, few big ones
Schnur expects the business landscape to become more polarized: a strong concentration on a few large companies and an extremely long tail of micro-enterprises with one to ten employees serving highly specialized niches. In between, many will fall by the wayside. Small and medium-sized companies will have to merge or cut jobs on a massive scale, depending on the industry, and this won’t happen overnight. As a preview, he cites China’s “dark factories,” where no lights are on because no one works there, and a meeting assistant that his team recently built in five days: an agent that coordinates appointments via chat, clarifies agendas, and involves external parties. A year ago, that would have been impossible. Most roles will remain, but in much smaller numbers and as coordination tasks. Sales is the best-protected area because people want to interact with each other, they don’t have to.
More time or less sense
Socially, Schnur sees two opposing trends. Used correctly, AI could reverse the intensification of work seen in recent decades and free up time. At the same time, there is a risk of “intellectual dumbing down”: What we no longer need to be able to do, we will eventually lose the ability to do and ultimately cease to understand. This makes us more susceptible to influence. The education system, too, cannot continue as it is; there is a need for more methodological and foundational knowledge in a shorter amount of time. He doesn’t know exactly how, either. His call is therefore less a program than an attitude: remain open-minded, experiment, and avoid dogmatism. Change will be painful anyway. The only question is whether it will be more painful than necessary.
Binci Heeb
See, hear, and read more: Karl Andersen: The Electricity That Pays Off
“Conscious or How to Avoid the AI Apocalypse” by Sebastian Schnur
Search:
Sponsors:
