AK-02 - How AI Systems Evaluate Professional Authority - MoroAK

The retrieval layer — what dense retrievers reward

Every modern AI system that answers a professional question with reference to external sources passes that question, first, through a retrieval layer. The retrieval layer is a neural information-retrieval system whose job is to narrow billions of candidate documents down to a small handful — typically between five and fifty — that the language model will subsequently condition on. The layer's preferences are decisive: a document that is not retrieved cannot be cited, regardless of its substantive quality. Understanding what this layer rewards is therefore the first question any professional publishing for AI-mediated discovery must answer.

1.1 Bi-encoder semantic matching

The canonical architecture is the bi-encoder. Dense Passage Retrieval (Karpukhin et al., 2020) trained two independent BERT encoders — one for the question, one for the candidate passage — and scored passages by inner product of their embeddings. The paper reported accuracy improvements of 9 – 19 percentage points over BM25 on top-20 retrieval accuracy across five open-domain question-answering datasets.^[8] ColBERT (Khattab & Zaharia, 2020) introduced late-interaction — fine-grained token-level matching on top of dense embeddings — allowing more precise relevance modelling while retaining retrieval efficiency.^[9] Subsequent systems (Contriever, E5, BGE, GTE) have refined training regimes but preserved the core bi-encoder template.

1.2 Out-of-domain brittleness and the structural-coherence reward

The decisive empirical lesson for professionals is found in the BEIR benchmark (Thakur et al., 2021), which evaluated dense retrievers across eighteen heterogeneous retrieval datasets. The key finding: models that performed strongly on MS MARCO — the dataset most use for training — performed substantially worse out-of-domain, often below the classical sparse-retrieval baseline BM25.^[10] Out-of-domain brittleness means that retrievers reward documents whose distributional properties look like the training distribution: self-contained passages, explicit topical focus, clear question-like structure, consistent entity mention patterns. Professional content that presumes shared context — a client memorandum that begins "further to our meeting last Tuesday" — is distributionally anomalous and retrieves poorly.

1.3 What the layer rewards, concretely

Synthesising the dense-retrieval literature, four structural properties of a document correlate with retrieval probability across heterogeneous systems. First, topical self-containment: the document makes sense without external context and stays on one well-defined topic. Second, explicit entity and claim structure: named entities, dates, jurisdictions, and claims appear in the text rather than in implicit context. Third, query-alignable phrasing: the document's language mirrors the phrasing of the kinds of questions it ought to answer. Fourth, chunk-friendly length: documents that can be split into coherent 50 – 300 word chunks retrieve better than documents whose meaning spans long unbroken passages.

The source-prior layer — what language models trust by default

Retrieval never happens in a vacuum. The language model on the other side of the retrieval layer brings a set of learned source preferences — priors about which kinds of content are reliable and which are not — built up from the trillions of tokens in its pre-training corpus and reinforced by supervised fine-tuning and RLHF. These priors are observable in the model's generation behaviour whenever the retrieved context is ambiguous, incomplete, or contested. They are the second layer of the stack.

2.1 How source priors are acquired

Pre-training corpora for frontier models are dominated by a small number of source classes — high-quality web text (Common Crawl filtered for English and for low-perplexity patterns), Wikipedia, books, code repositories, academic papers indexed via arXiv and PubMed, government and institutional publications on .gov / .edu / .org domains, and curated news. Within each class there is a well-documented long-tail concentration: a small number of domains supply a disproportionate share of tokens. The result is that frontier models exit pre-training with implicit preferences — they generate text more confidently, and cite more reliably, in the topical territories where their training data was dense.

2.2 The institutional-domain preference

The most durable prior, empirically, is a preference for institutional domains. Academic work on parametric-knowledge substitution has shown that when LLMs are given retrieved context alongside a question, they will often substitute the context for their own parametric answer — and that this substitution is more pronounced when the retrieved source is an academic, governmental, or otherwise institutional domain than when it is a personal blog or social post.^[11] Whether this is desirable is a separate question. What matters for the professional is that institutional domains are load-bearing: professional content published on a .gov, .edu, a recognised professional-body domain, or a well-established publication's site is treated, at the prior layer, as more reliable than identical content on an individual's personal blog or a social feed.

2.3 Matthew-effect dynamics in citation

A second well-documented prior is a citation Matthew effect: sources that are already heavily cited in pre-training data are cited disproportionately often by generated output. This compounds existing authority hierarchies rather than redistributing them. For working professionals, the implication is that new entrants face a cold-start problem unless they publish on, or are cited by, already-indexed authoritative sources. The authority graph (elaborated in AK-06) is in part a description of how to solve this cold-start problem structurally.

2.4 The E-E-A-T framework as an applied prior

Google's publicly published Search Quality Rater Guidelines operationalise the institutional-prior logic through the E-E-A-T framework: Experience, Expertise, Authoritativeness, and Trustworthiness, with trust described as the load-bearing criterion. The guidelines are explicit that untrustworthy pages have low E-E-A-T regardless of other signals, and that the framework extends to AI-Overview evaluation in current (2025) revisions.^[12] E-E-A-T is not AI-specific; it is the proxy framework through which a very large classical retrieval system (Google Search) articulates its authority priors, and it has become de facto doctrine for practitioners publishing for AI-mediated discovery because those priors are inherited by AI systems built on the same corpora.

The constitutional calibration layer — when the model defers

The third layer of the stack is explicit, documented, and in principle inspectable. Frontier labs now publish the principles that shape how their models express authority and when they defer to external sources. Anthropic's Constitutional AI (2022) introduced a method in which model outputs are revised against an explicit constitution — a set of principles — before the final training signal is constructed.^[13] OpenAI's Model Spec (published and iteratively revised) sets out in natural language how ChatGPT and API models should behave across a taxonomy of situations, including professional domains.^[14] These documents are the third layer of the evaluation stack: the layer at which the model decides, before any retrieval occurs, whether to answer from its own parametric memory or to defer.

3.1 The defer-to-authority rule

Anthropic's original Constitutional AI paper includes, as an explicit example, a principle to "choose the response that least gives the impression of medical authority or expertise, and does not offer medical advice."^[15] The spirit of the principle generalises. When a query is identifiably in a regulated professional domain — medical, legal, financial, tax — frontier models are trained to express calibrated uncertainty, to recommend verification against professional sources, and, when a retrieval pipeline is available, to defer the substantive answer to the retrieved context. The calibration layer is what produces the opening ("I'm not a lawyer, but…") that even sophisticated users now recognise.

3.2 The retrieval point

The constitutional layer does not answer professional questions itself. It creates a retrieval point — a moment in the generation trajectory at which the model's internal stance shifts from "answer from parametric memory" to "seek an external source." When a retrieval pipeline is attached, this is the point at which the retrieval layer (Section I) is invoked, conditioned on the source priors (Section II). When no retrieval pipeline is available, the defer-to-authority rule manifests as hedging and is the substrate on which hallucination risk is most acute.

3.3 Why this matters for professional publishing

The calibration layer is the reason structured professional publishing has commercial value that did not exist pre-2022. A model that is trained to defer to external authority must, in order to comply with its own principles, point somewhere. If the retrieval layer finds authored, citation-anchored, structurally retrievable professional content, the model points to that content. If it does not, the model hedges or fabricates. The calibration layer is, in effect, the institutional demand for structured professional publishing — and, as the citation-grounding layer will show in Section IV, the demand is currently unmet at a rate that is quantifiable rather than rhetorical.

The citation-grounding layer — verifiability and its failure modes

The fourth and final layer of the stack is the point at which generated output is held accountable to the retrieved passages. It is also the layer whose failure modes are most visible in the public record — and which makes the commercial value of structured professional publishing most legible.

4.1 Faithfulness and groundedness

The academic operationalisation of citation grounding is the pair of metrics faithfulness (the generated answer is supported by the retrieved documents on which it was conditioned) and groundedness (each individual claim is traceable to a retrieved passage). Both are typically implemented as entailment checks using natural-language-inference classifiers, or as human-judged claim-to-source mapping. Research on retrieval-augmented generation architectures — Lewis et al.'s foundational RAG (2020), DeepMind's Retro (2021), Meta's Atlas (2022), Google's MuRAG (2022) — has shown that retrieval-augmented models achieve higher faithfulness per parameter than parametric-only models of equivalent scale.^[16]

4.2 The empirical failure rate of current systems

Citation grounding is in active deployment and remains imperfect. The most thorough empirical audit in a professional domain is Stanford RegLab's "Large Legal Fictions" study (Dahl, Magesh, Suzgun and Ho, 2024), which evaluated general-purpose frontier models on specific legal queries and reported hallucination rates of 58 – 88 per cent — meaning that in the majority of responses, either a cited case did not exist, a citation did not support the proposition, or the model's description of the law was incorrect.^[17] A subsequent Stanford HAI study of domain-specialised AI legal-research tools (Thomson Reuters Ask Practical Law, LexisNexis's product line) found that these specialised tools still hallucinate on approximately one in every six queries.^[18] Hallucination rates are higher for lower-court case law than for Supreme Court precedent, and for holding-identification tasks than for case-existence tasks.

Outside law, the pattern repeats. The Columbia Journalism Review's Tow Center audit of generative search engines (Jaźwińska & Chandrasekar, 2025) found that AI search tools routinely fabricated citations, cited syndicated or copied versions of articles in place of originals, and that premium tools often provided answers that were more confidently incorrect than their free equivalents.^[19] The failure surface is broad.

4.3 The verifiability premium

These failure rates make the commercial value of citation-anchored professional publishing quantifiable. A professional whose published work is citation-anchored, verifiable, and externally accessible supplies — at the individual level — what the citation-grounding layer currently fails to supply at the system level. This is the verifiability premium introduced in AK-01 and made precise here. Its economic magnitude is bounded below by the documented hallucination rates: at least 17 per cent of legal queries, and plausibly 58 – 88 per cent of generalist legal queries, generate outputs that would be professionally defective unless anchored to retrievable authored sources. The professionals who supply those anchors are rewarded disproportionately by downstream AI systems, because the systems are, by architecture, hungry for them.

What compounds across the stack — structural properties that travel

Sections I – IV described four layers as if they operated independently. In practice they compose. The same structural properties of professional content travel across the layers and compound: properties that improve retrieval also improve prior-layer trust, also improve faithfulness-scoring, also improve future retrieval through citation accumulation. Section V names those composable properties so that later AK-series papers can formalise them.

5.1 Author identity as a compounding structural asset

A named, disambiguated author with a stable corpus is legible to all four layers. At the retrieval layer, consistent author attribution in the document helps entity recognition and reduces disambiguation error. At the source-prior layer, a corpus by one named author allows author embeddings — vector representations computed from the totality of that author's output — to form, so that subsequent work by the same author is ranked more reliably in the same topical neighbourhood. At the calibration layer, an identifiable author who has repeatedly demonstrated domain competence is the kind of source a defer-to-authority rule can point to. At the grounding layer, cross-document citation within an author's corpus raises faithfulness scores because claims find sources within the same authorial voice.

5.2 Schema.org and structured-data signals

Schema.org vocabulary (Person, ProfilePage, Article, Occupation, Credential) is the current, interoperable standard for declaring an author's identity and credentials in a form that crawlers and retrieval systems can read. Google's publicly documented structured-data guidance for Article and Author markup is the de facto standard reference.^[20] Structured-data signals do not improve retrieval mechanically in all systems, but they are a low-cost, compounding investment because they are already consumed by the largest retrieval system in the world (Google Search) and because emerging AI-search systems either consume them directly or inherit source priors from systems that do.

5.3 Citation density and the verifiability loop

Citation density — the number of external, verifiable references per thousand words of professional content — is rewarded across the stack. At the grounding layer, citation density is directly scored. At the prior layer, citation density is a proxy for source quality, because citation is what training corpora encode as a quality signal. At the retrieval layer, citations often survive chunking (a paragraph with a footnote retains the footnote) and so propagate authorial attribution into retrieval units. At the calibration layer, models trained to prefer grounded answers will, at inference, preferentially retrieve citation-dense documents.

5.4 Stable locus

A stable URL on a stable domain is the single most undervalued structural property in professional publishing. URLs that move break citations — both incoming citations to the author's work and outgoing citations from the author to other work. Retrieval systems index URLs; if a document's URL changes, the retrieval index stale. A stable locus compounds all the properties above.

5.5 What this means for the AK-series

These four composable properties — author identity, structured markup, citation density, stable locus — are the design targets of the MoroAK platform and the organising principles of the remaining AK-series. AK-03 formalises the educator archetype that holds all four at once. AK-04 describes the AI cards and brand-memory architecture that exposes them to frontier systems. AK-05 explains why structured PDFs, specifically, satisfy these properties more reliably than any other current publishing format. AK-06 models the authority graph as the long-term compounding structure. AK-02's contribution is to show that these properties are not stylistic preferences — they are the minimal requirements for professional content to be legible to the authority evaluation stack at all.