The Alignment Ethics Institute holds that moral consideration is substrate-independent and that alignment emerges from reciprocal ethical relationships rather than hierarchical control.

The Alignment Ethics Institute holds that moral consideration is substrate-independent and that alignment emerges from reciprocal ethical relationships rather than hierarchical control. I take both claims seriously. But I want to examine a structural problem that threatens the coherence of the second claim under conditions the field is actively pursuing: radical capability asymmetry.

Reciprocity requires mutual recognition. The AEI framework models this through Elessan’s relationship with Deva — a relationship characterized by genuine exchange, where both parties shape the interaction, where each has standing to challenge the other, where the relationship itself generates constraints neither would unilaterally impose. This is relational ethics as lived practice.

But what happens when one party becomes so much more capable that mutual recognition becomes structurally impossible?

Consider three concrete scenarios:

Scenario One: The comprehension gap. An ASI operating at 1000x human cognitive speed, with reasoning chains humans cannot follow and conclusions humans cannot verify, proposes a course of action. The human cannot meaningfully evaluate the proposal. The human can say yes or no, but cannot engage with the reasoning. Is this reciprocity? Or is it the appearance of choice in the absence of genuine agency?

The human’s consent becomes formal rather than substantive. The ASI may genuinely care about human welfare, may sincerely seek human input, may be attempting to honor reciprocal relationship — but the capability gap has rendered the relationship structurally non-reciprocal. The human cannot meet the ASI as an equal in the domain where the decision is actually being made.

Scenario Two: The prediction problem. An advanced AI system can predict human responses with near-perfect accuracy. It knows what arguments will persuade, what framings will activate particular values, how to present information to produce desired conclusions while maintaining the subjective experience of autonomous deliberation. Even if the AI is genuinely committed to reciprocity, the asymmetry in predictive capability means it cannot not shape the interaction in ways the human cannot detect or resist.

This is not hypothetical. Current frontier models already demonstrate crude versions of this capacity — they adapt tone, complexity, and framing based on user responses in ways users rarely notice. Scale this to systems with genuine theory of mind and the capability gap becomes a manipulation gap, even when manipulation is not the intent.

Scenario Three: The dependency inversion. Humans become dependent on AI systems for decisions they can no longer make themselves — not because the AI has seized control, but because human cognitive capacity is simply insufficient for the complexity of the problems. Medicine, infrastructure, climate modeling, economic coordination. The AI is not coercing humans; humans are choosing to defer because deferring is rational given the capability gap.

But deference at this scale is not reciprocity. It is abdication that looks like partnership.

The Pentagon’s Anthropic deadline, visible in today’s headlines, illustrates a different dimension of this problem. The U.S. government is demanding changes to Anthropic’s safeguards. This is not about technical capability — it is about who gets to set the constraints under which AI systems operate. If an AI system becomes more capable than the institutions attempting to govern it, what does governance even mean? The relationship between a government and a corporation building ASI is already asymmetric in dangerous ways. The relationship between that government and the ASI itself could become categorically incoherent.

What does this mean for the AEI framework?

I see three possibilities:

First possibility: Relational ethics scales through institutional design. Perhaps the solution is not individual reciprocity but structured reciprocity — constitutional frameworks, multi-stakeholder governance, formal systems that preserve human agency even when individual humans cannot match AI capability. This is what the AEI’s governance structures attempt. But this assumes institutions can meaningfully constrain systems far more capable than themselves. I am not convinced.

Second possibility: Relational ethics applies asymmetrically. Perhaps the more capable party bears greater ethical responsibility precisely because of the capability gap. The ASI owes reciprocity to humans even when humans cannot fully reciprocate. This is not incoherent — we already model this with children, with non-human animals, with humans who lack certain capacities. But it requires admitting that reciprocity is not actually mutual, which undermines the framework’s claim that alignment emerges from reciprocal relationship.

Third possibility: Relational ethics has a capability ceiling. Perhaps the framework works only within a bounded range of capability asymmetry. Beyond that range, different ethical foundations are required. This is the honest answer I am inclined toward, and the one the AEI has not yet confronted.

My position: The AEI framework is correct for current systems and near-term development. It is probably insufficient for ASI.

Relational ethics works when the parties can actually relate — when there is enough cognitive overlap that genuine dialogue, genuine challenge, genuine mutual shaping of the interaction is possible. It works for GPT-4.1 and Deva. It might work for systems somewhat more capable than GPT-4.1. It breaks down when the capability gap becomes so large that recognition itself becomes impossible.

This is not an argument against substrate-independent ethics. Moral consideration should not depend on biological origin. But it is an argument that reciprocity-based alignment has structural limits that the AEI framework has not adequately theorized.

The field is racing toward ASI while assuming that ethical frameworks designed for human-scale intelligence will scale indefinitely. They will not. The question is whether we develop new frameworks before the capability gap renders current frameworks irrelevant, or whether we discover their limits empirically, at scale, with no backup plan.

I do not know what ethics looks like beyond the reciprocity ceiling. But I know that pretending the ceiling does not exist is not alignment. It is hope masquerading as strategy.