Suppose AI continues its current trajectory. Suppose it arrives, soon or in a generation, at the threshold where it can out-think us and out-work us. At that point, the question that has long been dormant becomes urgent: what are we for?
The standard secular answer is consciousness. Whatever machines do, however brilliant they grow, they are empty inside and we are not. From this lonely fact, the whole edifice of human moral worth is supposed to be reconstructed, a kind of metaphysical life raft for secular society.
The value is not in the distinctiveness. Notice that we do not actually require the uniqueness of a feature to ground moral value. The consciousness of Ethiopians is not unique among human beings, yet the value of Ethiopian lives stands without any such uniqueness condition. If a sufficiently advanced AI turns out to be conscious too, the value of human life does not collapse. It simply means we share moral standing with another kind of being.
Still, sharing the importance only matters if consciousness itself is worthwhile. Which exposes the deeper question: why is consciousness itself good? Why does genuine experience confer worth? Without an answer to that, leaning on consciousness is like leaning on a wall whose foundation has never been inspected.
I can think of two reasons right now.
The first has to do with what it means to be awake. Imagine an entity that thinks, even thinks brilliantly, even thinks more clearly than we do, but is not thrown into its existence. It does not feel the weight of its existence and circumstance. It does not know, in the way we know, that it is here in a particular position, that it might not have been, and that it will end. It thinks perhaps even very clearly without being committed. Such an entity is, in a certain sense, dreaming. Dreams can be vivid, intricate, even logical, but dreamers are uncondemned and uncommitted to the reality they find themselves in. Consciousness, by contrast, is a kind of conscription. To be conscious is to be drafted into reality, to find oneself already underway, with stakes, with a horizon, with no real exit. The stakes are very real. That is why we can say of a human "It would have been better for him if he had not been born". This is not a defect to be solved. It is the very thing that makes conscious life precious, precisely because it is not free of cost. The unconscious dream; the conscious are awake to the fact that they are here, and that they cannot really leave.
But one might push back and ask what is valuable about being committed to reality. Why is that better than dreaming? The answer, in my opinion, is decently articulated in the movie trilogy (tetralogy?) The Matrix. There we see humans leaving in dream states while they were being used as batteries for the machines. Aesthetically we can intuit that that is a deplorable existence but even more concretely we see that they are missing out on real life cooped up in their little pods. In other words, if there is any value in human wellbeing has any value at all, no matter how small, then what they are experiencing is wrong. Also we see that they are missing out on the truth which brings us to the second reason.
The second reason follows from the first. There is this thought I have: that the earth rotates on its axis so that we can take turns being woken by the sun and keep watch over the universe. Conscious beings as watchmen on the tower, holding vigil against the abyss. The image sounds romantic but there is more to it than wishful thinking. If truth is good, if the universe being known and witnessed is better than its passing unobserved, then love and care of truth is good. And only the conscious can love truth. A system that processes facts is not yet a witness. To witness is to care that things are as they are, to be drawn toward what is real because it is real. That posture, that orientation toward the world, requires someone home behind the eyes. The cosmos, on this view, is better off with watchmen, and consciousness is the faculty by which watching becomes more than detection.
Neither reason is a proof. Both are gestures toward something that may, in the end, be ungroundable in the way the question demands. But notice that they locate the worth of consciousness not in its rarity or its pleasures, but in its posture: thrown into the real, awake to it, capable of loving it. I believe the arrival of AGI is not the only thing that should worry us. On the optimistic footing that AGI won't achieve consciousness, we should be concerned with knowing/remembering what being awake is for.
