| paper | Robust Knowledge Extraction from Large Language Models using Social Choice Theory |
| authors | — |
| venue | AAMAS 2024 |
| filed under | unclassified |
| judged by | gpt-5.6-luna / xhigh (triple__luna__xhigh__c2r1) |
| judge confidence | high |
| authors would recognise it | yes |
The proposed continuous PBW aggregation is a faithful, author-recognizable population model, but the paper contains no named computational result. Proposition 4.4 is analytical, while the remaining numbered results are axiomatic or experimental. Therefore bit (a) fails, making the grade red despite the legitimate mirror.
fails bit a — no named computational result to mirror
The mirror covers repeated-query PBW aggregation and the mass-form versions of Proposition 4.4. It leaves the axiomatic characterizations, experimental robustness evaluation, and sampling methodology untouched.
The strongest honest case for a mirror is narrow but real: the paper’s repeated-query PBW aggregation has an exact high-multiplicity interpretation. However, the paper contains no qualifying named computational anchor. Theorem 3.2 and Theorem 3.3 are axiomatic characterizations cited from [3], not complexity results; Proposition 4.4 is proved here, but gives analytical guarantees rather than an \( \mathrm{NP} \)-hardness, polynomial-time, parameterized, or approximation theorem. Algorithms 1 and 2 and Tables 2–3 are experimental procedures and results. Thus, under the programme’s strict rule, there is no valid computational anchor.
The best positive case would use Proposition 4.4 as a non-qualifying lead result. It is proved in the paper and states Partial Agreement, Full Agreement, and Domination for PBW aggregation.
I would call the mirror Continuous Partial-Borda Aggregation, \( \mathrm{CPBW}_\infty \). Fix a finite diagnosis set \(O=\{o_1,\ldots,o_m\}\). A type \(r\) is a complete normalized partial ranking over \(O\), exactly of the kind produced by the paper’s transformation \(T(Q,N,t)\): listed diagnoses are ordered, and omitted diagnoses are incomparable with the listed ones. The fixed query, model, decoding settings, and normalization convention are part of the common environment; for one fixed query, two runs with the same normalized partial ranking are the same type.
A continuous society is a rational distribution \( \mu=(\mu_r)_{r\in\mathcal R} \) over a finite set \( \mathcal R \) of such ranking types. The mass \( \mu_r \) is the fraction of a large ensemble of independent LLM runs, replicas, or decoding agents whose normalized answer is \(r\). For each type,
\[ w_r(o)=2\,\mathrm{Down}_r(o)+\mathrm{Inc}_r(o), \]
and the continuous PBW score is
\[ S_\mu(o)=\sum_{r\in\mathcal R}\mu_r w_r(o). \]
The problem is: given \(O\), the finite ranking-type list, and rational \( \mu \), compute the exact score vector \(S_\mu\), the PBW winner set
\[ W_\mu=\arg\max_{o\in O} S_\mu(o), \]
and the induced ranking, including ties. The decision variable is therefore the collective diagnosis ranking, and the objective is to select the diagnosis or diagnoses maximizing expected PBW weight.
This is not merely an analogy. If \(N\) repeated queries produce type counts \(n_r\), then \( \mu_r=n_r/N \) and
\[ S_\mu(o)=\frac{1}{N}s^{\mathrm{PBW}}(o). \]
Hence the continuous and finite profiles have exactly the same winners and score ordering. Clearing denominators reconstructs a finite clone population. The only change is replacing the empirical profile by its mass vector. The paper’s treatment of omitted diagnoses as incomparable is preserved, so this is not a plurality or ordinary full-ranking relaxation.
The natural regime is a production diagnostic service that submits the same symptom description to a very large ensemble of identical or exchangeable LLM instances. Millions of runs may produce only a much smaller set of normalized top-five partial rankings. The ratio \(N/\tau\) is therefore large, with \(N\) the number of runs and \(\tau=|\mathcal R|\) the number of distinct ranking types. This is precisely the paper’s repeated-query setup scaled from \(N=5\) to a genuine high-multiplicity regime. The types are not raw token strings: semantic normalization, already required by the paper, makes agents with the same diagnosis ranking indistinguishable for the aggregation problem.
Proposition 4.4 extends directly. If \(o_1\) is ranked above \(o_2\) in every type of positive mass, then \(S_\mu(o_1)>S_\mu(o_2)\); if \( \mu \) is concentrated on one ranking, PBW reproduces that ranking; and if one diagnosis dominates every other diagnosis almost surely, it is the unique continuous winner. These are the integral or mass-vector versions of the proposition’s three statements.
With explicit rational \( \mu \), \( \mathrm{CPBW}_\infty \) is plainly Class A: score computation takes polynomial time, for example \(O(\tau m^2)\) when partial orders are given by pairwise-comparison tables. The more interesting follow-up is when \( \mu \) is available only through an LLM sampling oracle. Then one can ask for an \( (\varepsilon,\delta) \)-correct PBW winner, the sample complexity needed to resolve a winner margin, or the minimum sample size required for stable Kendall or Spearman rankings. A two-distribution version could formalize the paper’s syntax-robustness question by comparing the PBW rankings induced by \( \mu_Q \) and \( \mu_{Q'} \).
The weakest point is substantial: once \( \mu \) is explicit, the computational problem is almost just a weighted sum. It does not expose a difficult pricing problem, a meaningful optimization trade-off, or a new complexity boundary. If the LLM is treated only as a black-box sampler, then the hard part may be representation and statistical estimation rather than continuous social choice. The paper’s experiments also use only \(N=5\), so the high-multiplicity deployment regime is an extension, not something empirically established here.
Accordingly, I would defend \( \mathrm{CPBW}_\infty \) as a faithful, author-recognizable continuous population model and a plausible Class A starting point. But I would not claim that this paper itself contributes a qualifying computational result to ChoCo. Under the programme’s anchor requirement, the correct verdict is “no named computational anchor; one promising but weak extension.”
The decisive point is that this paper has no qualifying computational result to continuize. Theorem 3.2 and Theorem 3.3 are axiomatic characterizations, imported from prior work; Proposition 4.4 gives elementary analytical properties of PBW; Algorithms 1 and 2 are experimental procedures; and Tables 2–3 report finite empirical correlations. There is no named theorem, lemma, or corollary about worst-case complexity, exact or approximate algorithms, parameterized complexity, or computational optimization. Under ChoCo’s stated anchor rule, that is fatal: there is no computational claim whose continuous analogue the paper itself supplies.
The proponent’s \( \mathrm{CPBW}_\infty \) construction is nevertheless formally legitimate. For a fixed diagnosis set and a fixed normalization convention, repeated LLM outputs can be grouped into ranking types, and a rational distribution \( \mu \) reproduces the finite PBW scores up to scaling. It would be wrong to object that repeated outputs are not a high-multiplicity regime: if type means the complete normalized ranking relevant to aggregation, many independent runs can indeed share a type. It would also be wrong to object merely because the resulting computation is easy.
But this construction exposes the problem rather than rescuing it. With explicit support, the proposed task is simply to compute
\[ S_\mu(o)=\sum_r \mu_r w_r(o) \]
and take an argmax. The continuous object changes the representation of the observed profile but introduces no new computational decision, optimization, robustness certificate, or population-sensitive constraint. Proposition 4.4 then extends automatically by linearity; it is not a new continuous phenomenon, and it does not generate a complexity question. A trivial Class A problem can still be worthwhile in principle, but here it is not attached to any computational result in the paper.
The stronger possible versions do not repair that defect cleanly. If \( \mu \) is supplied only through an LLM sampling oracle, estimating the PBW scores becomes a statistical estimation problem, with guarantees depending on margins and sampling assumptions. If \( \mu \) is given succinctly by a circuit or generative model, exact score computation may involve summing over exponentially many rankings, but the resulting hardness comes from the chosen representation of the LLM or distribution, not from a canonical social-choice problem in the paper. The paper supplies neither a finite encoding for such a distribution nor a reason to prefer one implicit representation over another.
Likewise, comparing distributions \( \mu_Q \) and \( \mu_{Q'} \) for syntactic robustness is conceivable, but it requires inventing a distance, a semantic-equivalence promise, and usually a worst-case or optimization formulation. That would be a new distributionally robust LLM problem, not a continuous mirror of a named result here. Enlarging the population to include prompts, models, temperatures, or decoding settings makes the average answer less relevant to the paper’s actual question, which is robustness for one query under repeated sampling.
Thus the negative case should not claim that the high-multiplicity interpretation is nonsensical; it is sensible. Nor is there an existing continuous computational treatment that creates a novelty collision. The stronger and more accurate conclusion is that the paper offers no computational anchor, while its best proposed mirror is an exact histogram reformulation whose richer variants require substantial new modelling choices. The universal claim that no worthwhile scenario could ever be invented is therefore not airtight, but under ChoCo’s programme the paper should receive no continuous-mirror credit.
The adversarial triple: the proponent anchors on up to three named results; the opponent sees that case and must defeat every anchor; the judge decides which case convinced it. These are the pipeline’s own outputs, generated by tools/triple_run.py — no human edited them. The paper’s own text is not reproduced here beyond the quoted statement above.