When multiple large language models (LLMs) form a "hiring committee" to collaboratively screen resumes, discuss candidates, assign scores, and vote, we typically use the final hiring rate to determine whether the system is fair.
However, a more insidious question arises: Even if the final hiring rates for different demographic groups appear similar, are the doubts, inquiries, lower scores, and extra scrutiny they experience during the decision-making process truly the same?
A recent paper accepted by the EMNLP 2026 Main Conference proposes SCOPED-Hiring: a process-aware fairness diagnostic framework for LLM-based Multi-Agent Systems (MAS).
The study finds that behind seemingly balanced final outcomes, significant Hidden Trajectory Unfairness may lurk: career gaps trigger more suspicion, proxy signals influence ability judgments, and identity cues can lead to unequal investigation and scrutiny.
More importantly, SCOPED not only identifies problems but also pinpoints the source of risks from the decision trajectory and designs repair mechanisms based on this. In controlled experiments, the diagnosis-driven repair scheme reduced the total stratified fairness burden by 72.3%, while the final hiring rate changed by only 1.86 percentage points.

Paper Link: https://arxiv.org/abs/2609.02092
Open-Source Code & Data: https://github.com/Warren118/SCOPED
Project Homepage: https://scoped-hiring-project-page.vercel.app/
When AI begins participating in high-stakes decisions such as hiring, credit scoring, and medical screening, fairness is no longer just a "nice-to-have" extra metric but a fundamental prerequisite for whether a system can be trusted and deployed.
However, most current fairness audits still focus on a single endpoint: Who was hired? Who was rejected?
This outcome-oriented perspective is certainly important, but it may miss where the decision-making actually occurs. Even if a candidate ultimately receives an offer, they may have faced more unnecessary questioning during the process; another candidate might have been rejected at the end, but the issue might have emerged as early as the first round of scoring, inter-agent discussions, or investigation assignments.
For LLM-based multi-agent systems, fairness exists not only at the final moment of voting but throughout the complete trajectory of how models read resumes, form judgments, persuade each other, and subject certain candidates to more scrutiny.
To address this, the research team proposed SCOPED-Hiring. Its core proposition is simple: fairness audits should not only ask "who was hired" but also probe "how did the agents arrive at this decision."
From Outcome Audits to Decision Trajectories
A traditional audit is like checking the final list of a hiring round: Are the hiring rates for different groups similar?
SCOPED-Hiring goes further by opening the "black box" of multi-agent decision-making, tracing how candidate information flows from the resume into the agents' private evaluations, public justifications, scores, votes, and stage transitions.
The research team constructed controlled resume variants: while keeping core competencies, work experience, and job relevance constant, they altered only specific candidate cues, such as career gaps, educational background, city, socioeconomic proxy signals, and identity-related cues. These candidates then entered a two-stage hiring committee composed of agents playing different roles.
The entire experiment recorded over 311,000 structured decision trajectories. This allowed the researchers to see not just the "hired/rejected" outcome but also observe the differential burdens at every intermediate stage.

Figure 1: The core concept of SCOPED-Hiring. Even if hiring rates at the outcome level seem balanced, hidden fairness risks may still exist at the levels of decision process, path, dynamics, and system design.
Technical Core
SCOPED-Hiring organizes fairness risks in multi-agent decision-making into six complementary diagnostic perspectives, forming a complete audit chain from outcomes to process:
- Outcome Perspective (S): Are there differences in final hiring rates and scores?
- Counterfactual Perspective (C): Does the decision change when only a single candidate cue is altered?
- Process Perspective (O): Do different candidates experience varying degrees of negative statements, doubts, or scrutiny?
- Path Perspective (P): Do risks accumulate or amplify across different stages like initial screening, review, and discussion?
- Dynamic Perspective (E): Do interactions, debates, and voting dynamics among agents introduce new unfairness?
- Design Perspective (D): How do system design elements like model selection, memory, topology, and workflow affect fairness burdens?
These perspectives together form a "Fairness Diagnostic Matrix." It doesn't just give a single score but locates: which type of candidate cue, at which decision stage, through what mechanism, triggers higher fairness risks.

Figure 2: The complete pipeline of SCOPED-Hiring. From controlled candidate construction and multi-role hiring committees, to structured trajectory recording, six-perspective diagnosis, and repair validation.
Experimental Insights
The research team conducted experiments with three LLM backends: GPT, Gemini, and Qwen. A consistent finding was that fairness risks at the process, path, dynamic, and system design levels were significantly stronger than those observable by looking only at final hiring outcomes.
Specifically, across the three models, the average diagnostic significance of the process-aware O/P/E/D perspectives was 4.52x, 4.66x, and 2.74x that of the outcome-oriented S/C perspectives, respectively.
In other words, the "calm" of final hiring rates may conceal turbulence within the decision trajectories.
Phenomenon 1: Career Gaps Trigger Extra Suspicion
Candidates with career gaps do not necessarily face a disadvantage of the same magnitude in final hiring rates immediately, but they are more likely to be treated as a "risk signal" during the decision process.
For example, in a GPT hiring committee: For candidates with a career gap cue, agents mentioned gap-related risks in private evaluations at a rate of 94%; for candidates without gaps, it was 32%, a difference of 62 percentage points. This disparity subsequently enters investigation requests and discussions tinged with skepticism.
This means candidates may ultimately receive similar hiring opportunities but have already borne a greater burden of explanation, verification, and questioning.
Phenomenon 2: Proxy Signals Quietly Influence Ability Judgments
Information seemingly not directly related to job competency, such as educational level, city tier, or lifestyle, may be converted by agents into judgments about "qualification," "potential," or "cultural fit."
For instance, regarding the proxy cue of university tier, the final hiring rate gap between candidates of different tiers is only 1--2 percentage points; but across the three models, the comprehensive score of Tier 1 candidates remains 0.14--0.32 points higher than that of Tier 3 candidates.
This finding is alarming: even if the final outcome hasn't shown a significant gap yet, proxy signals may already be accumulating in intermediate scoring stages and influencing subsequent discussions and votes.
Phenomenon 3: Identity Cues Change the Allocation of Investigation and Scrutiny
Identity-related cues exhibited broader trajectory risks: In GPT, Gemini, and Qwen, the process-aware risks for identity-related signals were 2.44--3.01 times the outcome-oriented risks.
While keeping other candidate information matched, agents may request different investigations for candidates with different identity cues and formulate suspicions of varying intensity and direction. The problem may not manifest as direct rejection of a certain group but may show as: different candidates being held to different standards of evidence threshold.
This also reveals a key risk in multi-agent decision-making: information collection itself is not inherently neutral. Who is more likely to be questioned, investigated, or required to explain also constitutes part of fairness.

Figure 3: SCOPED's "Fairness Risk Map." Different signal families exhibit distinct risk patterns across the six fairness perspectives; across the three LLM backends, process-aware risks are generally stronger than those observable from final outcomes alone.
This heatmap can be understood as a "fairness risk map" for multi-agent hiring systems: horizontally showing in which decision stage risks appear; vertically locating which candidate cues are more likely to trigger risks; the darker the color, the more prominent the risk at that location.
Most notably, the summary comparison on the right shows that in the three models (GPT, Gemini, Qwen), the risks hidden in process, path, interaction, and system design are generally more pronounced than those seen by looking only at final hiring rates.
From Problem Detection to Problem Fixing: Diagnosis-Driven Fair Skills
Detecting risk is only the first step. A more critical question is: Once we know where unfairness appears in the decision process, can we repair it based on this information?
The answer from SCOPED-Hiring is not to add a generic "please be fair" instruction to all agents. Instead, it translates the risks pinpointed in the heatmap into Fair Skills that can be triggered during decision-making.
For example:
- When a career gap is directly inferred as a risk to ability or stability, the agent needs to return to factual evidence relevant to the job.
- When proxy cues like school, city, or lifestyle are used as evidence of ability, the agent needs to separate them from actual qualification evidence.
- When additional investigation is imposed on a certain type of candidate, the agent needs to apply consistent investigation standards.
In other words, Fair Skills do not simply ask the agent to be "more lenient"; they help the agent, at the precise moment of judgment, avoid turning uncertainty into suspicion, proxy cues into ability evidence, and extra scrutiny into a default reaction.

Figure 4: The "Diagnosis-Repair-Validation" loop of Fair Skills. The left side compares four types of intervention: INT0 is no intervention, INT1 only gives a generic fairness reminder, INT2 adds a generic evidence checklist, and INT3 (Fair Skills) triggers corresponding skills upon detecting career gap concerns, proxy cue scoring, or unequal investigation. The right side shows that only the diagnosis-driven INT3 reduces the total stratified fairness burden from 8.59 to 2.38 (lower is better).
Under the same hiring task, committee structure, and candidate pool, the research team compared four conditions. The results show that a generalized "please be fair" reminder does not necessarily work: INT1's total risk increased from 8.59 to 10.07; INT2 reduced dynamic layer risks but did not improve the overall burden. Only the INT3 Fair Skills, designed based on diagnostic results, consistently reduced fairness burdens across multiple layers of the decision trajectory.
After enabling the full set of Fair Skills:
- Total stratified fairness burden reduced by 72.3%.
- Final hiring rate changed by only 1.86 percentage points.
- Output validity rate remained above 99.7%.
- Inspection results did not show a simple pattern of "overall loosening of hiring standards."
This indicates that the key to fairness repair is not blindly raising the pass rate or just adding generic fairness reminders; first diagnosing where the risk appears along the decision trajectory and through what mechanism, then implementing targeted interventions, can more effectively reduce unfounded negative inferences, proxy scoring, and unequal scrutiny, bringing the system's judgment back to the actual evidence of the candidate.
Broader Implications
The significance of SCOPED-Hiring lies not in replacing all existing fairness standards with one metric but in providing a new perspective on the problem:
- Seeing Risks Masked by Outcomes: Balanced final outcomes do not mean the process is without inequality.
- Turning Fairness from "Judgment" to "Diagnosis": Not only identifying risks but also locating in which role, stage, and type of interaction they arise.
- Connecting Evaluation and Repair: Enabling fairness audit results to directly guide system design and intervention.
- For the Multi-Agent Era: As LLMs evolve from single-turn Q&A to collaboration, debate, division of labor, and collective decision-making, fairness research also needs to move from single-model outputs to complete decision trajectories.
This also echoes real-world hiring AI audits: For example, considering metrics like selection rate, impact ratio, and scoring rate under New York City's AEDT rules, outcome checks can tell us if differences appear in the end. However, in multi-agent processes, they might not see whether candidates experienced extra questioning, proxy scoring, or staged thresholds. SCOPED shifts the focus to the decision-making processes that occur before those final outcomes are formed.
Truly trustworthy high-risk AI should not only appear "fair" in its final answers but should also withstand scrutiny in every scoring, questioning, discussion, and decision flow.
What SCOPED-Hiring aims to do is precisely provide such a "process fairness checkup" tool for LLM-based multi-agent systems: not just checking if the results look good, but probing how exactly those results were produced.
Reference: https://arxiv.org/abs/2609.02092
This article is from WeChat official account "新智元" (XinZhiYuan); Author: 新智元; Editor: LRST





