How Americans perceive political bias in AI language models — and a framework for independent evaluation grounded in human judgment rather than corporate self-assessment
One in five Americans already consults ChatGPT about politics. As AI becomes a primary interface between citizens and political information, the question of ideological bias in AI systems has become urgent — and politically charged. Hall's team conducted what he describes as the largest independent assessment of AI political bias, anchored in how actual Americans perceive AI responses rather than how AI companies evaluate themselves.
Nearly all major AI models tested as left-leaning in their political responses — including Grok, despite Elon Musk's anti-"woke" positioning. More importantly: across partisan groups, Americans preferred less-slanted responses. Democrats, Republicans, and Independents all rated less-ideologically-tilted answers as higher quality. Users do not want AI echo chambers.
The core problem is legitimacy: AI companies grade their own ideological slant, and those self-assessments diverge significantly from what representative American samples perceive. An AI system that claims to be neutral while being perceived as biased by the people it serves has a credibility problem that compounds over time.
Media bias concerns are not new. Hall traces the arc: partisan 19th-century newspapers assumed no obligation to balance; 20th-century broadcast regulation (the Fairness Doctrine) imposed formal neutrality; cable news fragmentation shattered that norm; social media algorithms created personalized information environments optimized for engagement rather than accuracy. AI represents a new layer in this history — one where the same system serves all users simultaneously, making ideological tilt a systemic effect at scale.
Both Anthropic and OpenAI released studies claiming their models lack ideological bias. Hall identifies a fatal legitimacy flaw: companies grade themselves on ideological slant. When AI evaluates AI, results diverge from human perceptions. The problem is not necessarily that the self-assessments are wrong — it is that they cannot be trusted as the basis for public confidence, because the incentive structure is corrupted by the relationship between evaluator and evaluated.
Hall's team asked models political questions without definitive factual answers, then showed the responses to representative survey samples. By averaging evaluations across a demographically representative sample — rather than relying on partisan panels or AI-generated judgments — the methodology anchors ideological assessment in actual American perception rather than expert or corporate opinion.
This is the foundational paper in Hall's series, establishing the empirical baseline for AI political bias before his Japan experiment documented its source mechanisms and his superintelligence paper proposed the governance architecture to address them.
The cross-partisan neutrality preference is a key finding for candidate strategy. Users across partisan groups prefer less-ideologically-slanted AI responses. This means that when AI systems present a progressive candidate's positions in a partisan framing, they are likely performing worse with users than a more neutral presentation would. The REMEDY objective — helping AI systems represent a candidate accurately and legibly — is not just about presence; it is about the quality of representation. Accurate, factual, non-performatively-partisan content is what AI users prefer.
The self-grading critique applies to our evaluation work. Hall argues that AI companies cannot credibly evaluate their own ideological bias. The analogous principle: Kyanos should not evaluate the chatbot using only internally-designed criteria. The eval framework's use of an independent LLM (Haiku) as evaluator, with documented rubrics, reflects the same methodology Hall advocates — anchoring evaluation in something other than the system being evaluated.
The neutrality/truth distinction matters for chatbot design. The chatbot should be neutral on values-based questions (policy priorities, tradeoffs between competing goods) but factual and precise on evidence-based questions (what the candidate's record shows, what the AUDIT data found). This maps directly onto Hall's Proposal 2 and should inform how the chatbot's diagnostic reasoning rules are applied to different question types.