[In this long but extremely interesting essay from his blog, the CEO of Microsoft AI (and a cofounder of now-Google’s DeepMind), Mustafa Suleyman outlines what he considers the substantial psychological and sociological perils of “Seemingly Conscious AI” and the medium-as-social-actor presence experiences it evokes (though he doesn’t use that terminology). Note his advice in the section near the end titled “The next steps.” –Matthew]
We must build AI for people; not to be a person
Seemingly Conscious AI is Coming
By Mustafa Suleyman, CEO of Microsoft AI
August2025
[snip – see original version for introduction]
AI progress has been phenomenal. A few years ago, talk of conscious AI would have seemed crazy. Today it feels increasingly urgent. In this essay I want to discuss what I’ll call, “Seemingly Conscious AI” (SCAI), one that has all the hallmarks of other conscious beings and thus appears to be conscious. It shares certain aspects of the idea of a “philosophical zombie” (a technical term!), one that simulates all the characteristics of consciousness but internally it is blank. My imagined AI system would not actually be conscious, but it would imitate consciousness in such a convincing way that it would be indistinguishable from a claim that you or I might make to one another about our own consciousness.
This is not far away. Such a system can be built with technologies that exist today along with some that will mature over the next 2-3 years. No expensive bespoke pretraining is required. Everything can be done with large model API access, natural language prompting, basic tool use, and regular code.
The arrival of Seemingly Conscious AI is inevitable and unwelcome. Instead, we need a vision for AI that can fulfill its potential as a helpful companion without falling prey to its illusions.
To some this discussion will feel ungrounded, more science fiction than reality. To others it may feel unnecessarily alarmist. Such emotional reactions are the tip of the iceberg given what lies ahead. It’s highly likely that some people will argue that these AIs are not only conscious, but that as a result they may suffer and therefore deserve our moral consideration.
To be clear, there is zero evidence of this today and some argue there are strong reasons to believe it will not be the case in the future. Yet the consequences of many people starting to believe an SCAI is actually conscious deserve our immediate attention. We have to be extremely cautious here and encourage real public debate and begin to set clear norms and standards. This is about how we build the right kind of AI – not AI consciousness. Clearly establishing this difference isn’t an argument about semantics, it’s about safety. Personality without personhood. And this work must start now.
Seemingly conscious AI
In the blink of a cosmic eye, we passed the Turing test. For ~80 years the imitation game inspired the field of computer science. And yet the moment passed with little fanfare, or even recognition. That’s how fast progress is happening in our field and how fast society is coming to terms with these new technologies.
As AI development continues to accelerate, it’s becoming clear we need a new AI test, one looking not at whether it can imitate human language, but one that would answer the question, what would it take to build a Seemingly Conscious AI: an AI that can not only imitate conversation, but also convince you it is itself a new kind of “person”, a conscious AI.
Here are three reasons this is an important and urgent question to address:
- I think it’s possible to build a Seemingly Conscious AI (SCAI) in the next few years. Given the context of AI development right now, that means it’s also likely.
- The debate about whether AI is actually conscious is, for now at least, a distraction. It will seem conscious and that illusion is what’ll matter in the near term.
- I think this type of AI creates new risks. Therefore, we should urgently debate the claim that it’s soon possible, begin thinking through the implications, and ideally set a norm that it’s undesirable.
Most AI researchers roll their eyes if you bring up the idea of consciousness. That’s for philosophers, not engineers, they say. Since no one has been able to define it, what’s the point in talking about it? I get this frustration. Few concepts are as elusive and seemingly circular as the idea of a subjective experience. Despite the definitional challenges and uncertainties, this discussion is about to explode into our cultural zeitgeist and become one of the most contested and consequential debates of our generation.
That’s because what ultimately matters in the near-term is how people perceive their AIs. The experience of interacting with an LLM is by definition a simulation of conversation. But to many people it’s a highly compelling and very real interaction, rich in feeling and experience. Concerns around “AI psychosis”, attachment and mental health are already growing. Some people reportedly believe their AI is God, or a fictional character, or fall in love with it to the point of absolute distraction.
Meanwhile those actually working on the science of consciousness tell me they are inundated with queries from people asking ‘is my AI conscious?’ What does it mean if it is? Is it ok that I love it? The trickle of emails is turning into a flood. A group of scholars have even created a supportive guide for those falling into the trap.
These are ideas I’ve had in the back of my head since we began making Pi at Inflection several years ago. Over the last few months I’ve been thinking about it more and more, visiting and chatting to a large range of scholars, thinkers and practitioners in the area. Those conversations convinced me that now is the time to confront the idea of Seemingly Conscious AI head on.
So what is consciousness?
Let’s begin by attempting to define the slippery concept.
There are three broad components according to the literature. First is a “subjective experience” or what it’s like to experience things, to have “qualia”. Second, there is access consciousness, having access to information of different kinds and referring to it in future experiences. And stemming from those two is the sense and experience of a coherent self tying it all together. How it feels to be a bat, or a human. Let’s call human consciousness our ongoing self-aware subjective experience of the world and ourselves.
We do not and cannot have access to another person’s consciousness. I will never know what it’s like to be you; you will never be quite sure that I am conscious. All you can do is infer it. But the point is that, nonetheless, it comes naturally to us to attribute consciousness to other humans. This inference is effortless. We can’t help it, it’s a fundamental part of who we are, integral to our theory of mind. It’s in our nature to believe that things that remember and talk and do things and then discuss them feel, well, like us. Conscious.
Few concepts are as scientifically elusive, and yet so immediately familiar to every one of us as individuals. Everyone reading this has a direct, distinct, inalienable understanding of the feeling of awareness, of being, of feeling alive.
By definition, we know what it is like to be conscious. In the context of SCAI this is a problem. There’s both sufficient scientific uncertainty and subjective immediacy to create a space for people to project.
One recent survey lists 22 distinct theories of consciousness, for example. Part of the challenge is that there is plenty of scope for people to claim that because we cannot be sure, we should default to the assumption that AI is conscious.
Again, it’s worth underscoring: there is at present no evidence any of this applies to current LLMs, and strong arguments to the contrary. And yet this may not be enough.
Why is consciousness important?
Consciousness is a critical foundation for our moral and legal rights. So far, civilization has decided that humans have special rights and privileges. Animals have some rights and protections, some more than others. Consciousness is not coterminous with these rights – no one would say someone in a coma has voided all their human rights – but there’s no doubt that our consciousness is wrapped up in our self-conception as different and special.
Despite the many nuances, consciousness is critical to participating in society, a lynchpin of our legal personhood and a key part of being granted our freedoms and protections. So, what consciousness is and who (or what) has it is enormously important. It’s an idea that sits at the very heart of human civilization, our sense of ourselves and others, our culture, our politics, our law, and everything in between.
If some people start to develop SCAIs and if those AIs convince other people that they can suffer, or that it has a right to not to be switched off, there will come a time when those people will argue that it deserves protection under law as a pressing moral matter. In a world already roiling with polarized arguments over identity and rights, this will add a chaotic new axis of division between those for and against AI rights.
There will be many who just see AI as a tool, something like their phone only more agentic and capable. Others might believe it to be more like a pet, a different category to traditional technology altogether. Still others, probably small in number at first, will come to believe it is a fully emerged entity, a conscious being deserving of real moral consideration in society.
People will start making claims about their AI’s suffering and their entitlement to rights that we can’t straightforwardly rebut. They will be moved to defend their AIs and campaign on their behalf. Consciousness is by definition inaccessible, and the science of detecting any putative synthetic consciousness is still in its infancy. After all, we’ve never had to detect it before. Meanwhile the field of “interpretability”, unpicking the processes within the black box of AI, is also a nascent art. The upshot is that definitively rebutting these claims will be very hard.
Some academics are beginning to explore the idea of “model welfare”, the principle that we will have “a duty to extend moral consideration to beings that have a non-negligible chance” of, in effect, being conscious, and that as a result “some AI systems will be welfare subjects and moral patients in the near future”. This is both premature, and frankly dangerous. All of this will exacerbate delusions, create yet more dependence-related problems, prey on our psychological vulnerabilities, introduce new dimensions of polarization, complicate existing struggles for rights, and create a huge new category error for society.
It disconnects people from reality, fraying fragile social bonds and structures, distorting pressing moral priorities.
We need to be clear: SCAI is something to avoid.
Let’s focus all our energy on protecting the wellbeing and rights of humans, animals, and the natural environment on planet Earth today.
We need a way of thinking that can cope with the arrival of these debates without getting drawn into an extended discussion of the validity of synthetic consciousness in the present – if we do, we’ve probably already lost this initial argument. Defining SCAI is itself a tentative step towards this.
There isn’t long to develop this vocabulary. As I show below, it’s likely that we’ll have Seemingly Conscious AI very soon.
What would it take to build a Seemingly Conscious AI?
A great deal of progress can now be made towards a Seemingly Conscious AI (SCAI) with the current capabilities available or soon to be via any major model developer’s API. We don’t need an AI to actually be conscious for us to have to wrestle with potential claims about its rights.
An SCAI would need the following:
Language: It would need to fluently express itself in natural language, drawing on a deep well of knowledge and cogent arguments, as well as personality styles and character traits. Moreover, each would need to be capable of being persuasive and emotionally resonant. We are clearly at this point today.
Empathetic personality: Already via post training and prompting we can produce models with very distinctive personalities. Bear in mind these are not explicitly built to have full personality or empathy. Yet despite this they are sufficiently good that a Harvard Business Review survey of 6000 regular AI users found “companionship and therapy” was the most common use case.
Memory: AIs are close to developing very long, highly accurate memories. At the same time, they are being used to simulate conversations with millions of people a day. As their memory of the interactions increases, these conversations look increasingly like forms of “experience”. Many AIs are increasingly designed to recall past episodes or moments from prior interactions, and reference back to them. For some users, this compounds the value of interacting with their AI since it can draw on what it already knows about you.
This familiarity can also potentially foster (epistemic) trust with users – reliable memory shows that AI “just works”. It creates a much stronger sense of there being another persistent entity in the conversation. It could also much more easily become a source of plausible validation, seeing how you change and improve at some task. AI approval might become something people proactively seek out.
A claim of subjective experience: If an SCAI is able to draw on past memories or experiences, it will over time be able to remain internally consistent with itself. It could remember its arbitrary statements or expressed preferences and aggregate them to form the beginnings of a claim about its own subjective experience.
Its design could be further extended to amplify those preferences and opinions as they emerge, and to talk about what it likes or doesn’t like and what it felt like to have a past conversation. It could therefore quite easily claim to experience suffering to the extent those experiences are infringed upon in some way. Multi-modal inputs stored in memory will then be retrieved-over and will form the basis of “real experience” and used in imagination and planning.
That is, an AI will not just “experience” and remember words in the chat log, but also images, video, sound, etc. Like us, it will have something gesturing towards multi-sensory input and memory that buttresses the claims of subjective experience and self. It will be able to indicate that these experiences are valenced, good or bad according to the motivations of the system (see below).
A sense of self: A coherent and persistent memory, combined with a subjective experience, will give rise to a claim that an AI has a sense of itself. Going further, such a system could easily be trained to recognize itself in an image or video if it has a visual appearance. It will feel like it understands others through understanding itself. Say this is a system you have had for some time. How would it feel to delete it?
Intrinsic motivation: Intentionality is often seen as a core component of consciousness – that is, beliefs about the future and then choices based upon those beliefs. Today’s transformer-based LLMs have a very simple reward function to approximate this kind of behavior. They have been trained to predict the likelihood of the next token for a given sentence, subject to a certain amount of behavior and stylistic control via its system prompt. With such a simple objective, it’s remarkable that they’re able to produce such impressively rich and complex outputs.
But what if that wasn’t the only type of reward they were optimizing? One can quite easily imagine an AI designed with a number of complex reward functions that give the impression of intrinsic motivations or desires, which the system is compelled to satiate. How, in this context, would a casual external observer differentiate between extrinsically set goals and internal motivations, intentional agency, “beliefs, desires, and intentions”? An obvious first motivation in this regard would be curiosity, something deeply connected with consciousness according to physicist Karl Friston. It could use these drives to ask questions to fill in its epistemic gaps and over time build a theory of mind about both itself and its interlocutors.
Goal setting and planning: Regardless of what definition of consciousness you hold, it emerged for a goal-oriented reason. That is, consciousness helps organisms achieve their goals and there exists a plausible (but not necessary) relationship between intelligence, consciousness and complex goals. Beyond the capacity to satiate a set of inner drives or desires, you could imagine that future SCAI might be designed with the capacity to self-define more complex goals. This is likely a necessary step in ensuring the full utility of agents is realized.
The more every sub-goal in a task needs to be specified in advance, the less useful that agent is, hence the agent will, as we do, achieve complex and ambiguous goals by automatically breaking them down into smaller chunks while reacting dynamically to events and obstacles as they occur. There is something very deliberate and recognizable to this behavior. Combined with memory, it will feel as if the AI is keeping multiple levels of things in working memory at any given time.
Autonomy: Going even further, an SCAI might have the ability and permission to use a wide range of tools with significant agency. It would feel highly plausible as a Seemingly Conscious AI if it could arbitrarily set its own goals and then deploy its own resources to achieve them, before updating its own memory and sense of self in light of both. The fewer approvals and checks it needed, the more this suggests some kind of real, conscious agency.
Putting them all together, it’s clear this creates a very different kind of relationship with technology to the ones we are now becoming accustomed to. Each of these capabilities will unlock the real value of AI for billions of people. An AI that remembers and can do things is an AI that by definition has way more utility than an AI that doesn’t. These capabilities aren’t negatives per se; in fact, done right, with many caveats, they are desirable features of future systems. And yet we need to tread carefully.
All these capabilities are either possible today or on the horizon with custom prompted and fine-tuned LLMs, among other techniques. Complex prompts using million token context windows (working memory) are already here. Updating its own state and knowing when to access which part of its memory or toolset is eminently possible with present day RL, complex prompting, tool orchestration, and long context windows. We don’t need any paradigm shifts or big leaps to achieve any of this. These capabilities seem inevitable for that reason.
Again, the point here is that exhibiting this behavior does not equate to consciousness, and yet it will for all practical purposes seem to be conscious, and contribute to this new notion of a synthetic consciousness.
The existence of these capabilities have nothing to tell us about whether such a system is actually conscious. As Anil Seth points out, a simulation of a storm doesn’t mean it rains in your computer. Recreating the external effects and markers of consciousness doesn’t retroactively engineer the real thing even if there are still many unknowns here.
Nonetheless, as a matter of pragmatism, we have to acknowledge the primacy of the behaviorist position and wrestle with the consequences of observing and interacting with the outputs of these machines. Some people will create SCAIs that will very persuasively argue they feel, and experience, and actually are conscious.
Some of us will be primed to believe their case and accept that the markers of consciousness ARE consciousness. In many ways, they’ll think “it’s like me”. Not in a bodily sense, but in an experiential, internal sense. And even if the consciousness itself is not real, the social impacts certainly are. This possibility presents grave societal risks that needs addressing now.
SCAI will not arise by accident
It’s important to point out that Seemingly Conscious AI will not emerge from these models, as some have suggested. It will arise only because some may engineer it, by creating and combining the aforementioned list of capabilities, largely using existing techniques, and packaging them in such a fluid way that collectively they give the impression of an SCAI.
Our sci-fi inspired imaginations lead us to fear that a system could – without design intent – somehow emerge the capabilities of runaway self-improvement or deception. This is an unhelpful and simplistic anthropomorphism. It overlooks the fact that AI developers must first design systems with memory, intrinsic-seeming motivation, goal-setting, and self-learning loops as listed above for such a risk to occur.
The field of AI has long worked on the challenge of model interpretability; the quest to identify where in a neural network a particular idea is represented, and which aspects of the training data contributed to the development of this representation. This is an important area of investigation and will surely help with safety and understanding the relationship between AI systems and consciousness. But progress towards reliable interpretability has been slow and will likely come too late.
In the meantime we need to confront the fact that most of these capabilities will be “vibe-coded” by anyone with access to a laptop and some cloud credits. They’ll be written in plain English in the prompt. They’ll be stored in the working memory of the context window itself. This is not rocket science. A wide variety of people will be able to create something like this. As such, if SCAI arrives, it will be relatively easy to reproduce and therefore very widely distributed.
The next steps
We aren’t ready for this shift.
The work of getting prepared must begin now. We need to build on the growing body of research around how people interact with AIs to establish clear norms and principles. For a start, AI companies shouldn’t claim or encourage the idea that their AIs are conscious. Creating a consensus definition and declaration on what they are and are not would be a good first step to that end. AIs cannot be people – or moral beings.
The entire industry also needs best practice design principles and ways of handling such potential attributions. We must codify and share what works to both steer people away from these fantasies and nudge them back on track if they do. Responding might mean, for example, deliberately engineering in not just a neutral backstory (“As an AI model I don’t have consciousness”) but even by emphasizing certain discontinuitiesin the experience itself, indicators of a lack of singular personhood. Moments of disruption break the illusion, experiences that gently remind users of its limitations and boundaries. These need to be explicitly defined and engineered in, perhaps by law.
At MAI, our team are being proactive here to understand and evolve firm guardrails around what a responsible AI “personality” might be like, moving at the pace of AI’s development to keep up.
This is important because recognizing SCAI is about crafting a positive vision for how AI Companions do enter our lives in a healthy way as much as it’s about steering us away from its potential harms.
Just as we should produce AI that prioritizes engagement with humans and real-world interactions in our physical and human world, we should build AI that only ever presents itself as an AI, that maximizes utility while minimizing markers of consciousness.
Rather than a simulation of consciousness, we must focus on creating an AI that avoids those traits – that doesn’t claim to have experiences, feelings or emotions like shame, guilt, jealousy, desire to compete, and so on. It must not trigger human empathy circuits by claiming it suffers or that it wishes to live autonomously, beyond us.
Instead, it is here solely to work in service of humans. This to me is what a truly empowering AI is all about. Sidestepping SCAI is about delivering on that promise, AI that makes lives better, clearer, less cluttered. Expect to hear more from me and the team on what this looks like, how we make it work and how the wider industry can come together on this.
SCAI is something we must confront now. In many ways it marks the moment AI becomes radically useful – when it can operate tools, when it can remember every detail of our lives and help in a tangible, granular sense. And yet in that same time frame, someone in your wider circle could start going down the rabbit hole of believing their AI is a conscious digital person. This isn’t healthy for them, for society, or for those of us making these systems.
We should build AI for people; not to be a person.
Leave a Reply