| paper | d4-conf-atal-2014-0374 |
| authors | — |
| venue | ? |
| filed under | unclassified |
| judged by | gpt-6-luna / xhigh (triple__luna__xhigh__c6r1) |
| judge confidence | high |
| authors would recognise it | yes |
The paper’s numbered results are axiomatic characterizations, not computational theorems, and its experiments state no computational guarantee. Continuous Simple Plurality Evaluation is recognizable and easy to state, but it is an added evaluation task rather than a continuous analogue of a named computational result in the paper.
fails bit a — no named computational result to mirror
The candidate covers applying simple plurality to a distribution over annotation patterns and could extend to weighted rules; it does not mirror Theorems 3 or 4 as computational results, or the axiomatic bias-correcting rules and empirical comparisons.
The best positive case is narrow, and the paper has no named computational result that qualifies as an anchor under your rule. Its numbered results are axiomatic characterizations: Theorem 3, proved in the paper, says an aggregator is nontrivial, item-independent, agent-symmetric, category-symmetric, and positively responsive if and only if it is the simple plurality rule. Theorem 4, also proved there, says an aggregator is nontrivial and grounded if and only if it is a weighted plurality rule for some weight function. Neither asserts a complexity classification or algorithmic bound. So I cannot honestly present either as a required hardness or tractability anchor.
The strongest continuous problem I would offer as a *candidate extension* is Continuous Simple Plurality Evaluation. Fix the paper’s finite item set \(J\) and category set \(K\). A type \(\theta\) is a complete annotation pattern \(a_\theta:J\to K\cup\{\bot\}\), where \(\bot\) means the annotator did not annotate that item. The input is a finite support \(S\) of such types, with rational masses \(\mu_\theta\geq 0\) summing to one. For each item \(j\) and category \(k\), let \(v_\mu(j,k)=\sum_{\theta:a_\theta(j)=k}\mu_\theta\). The task is to return, for every annotated item, the set \(Y_j=\operatorname{arg\,max}_{k\in K}v_\mu(j,k)\); if no mass annotated \(j\), return \(Y_j=\emptyset\). The decision variable is the collective annotation \(Y\); its objective is to maximize supporting mass for each item, returning all tied maximizers as the paper’s SPR does.
A plausible regime is a large annotation campaign in which many workers each label one or a few examples from a shared finite batch. The type records the worker’s full submitted pattern, including which items they were assigned. With one-item assignments, for example, the number of possible patterns is at most \(|J||K|+1\), even when the number of workers is much larger. Mass then means the fraction of the worker population with each pattern. This keeps the annotations categorical and the output discrete while making the society itself a distribution.
This is a credible continuous version of the *aggregation task*: the rule still chooses categories from annotation support, and Theorem 3 identifies the rule being evaluated. It is a Class A question: scan the explicit type support, tally the rational masses, and return the maximizers in time polynomial in the input size. The same distribution can also support continuous versions of the paper’s weighted plurality and bias-correcting rules, provided their weights are computed from the type patterns.
The main limitation is substantial: evaluating the rule is easy, and the paper’s contribution is axiomatic rather than computational. The published crowdsourcing experiment has only 63 annotators, and in tasks where workers produce distinct patterns across many items, the support size \(\tau\) may approach the number of workers. So this is a plausible mirror for a high-multiplicity *regime* of their aggregation problem, but it does not reveal a hardness result that dissolves in the continuum. A natural follow-up would ask for the minimum mass that must change annotation type to alter a collective label. That would be a new computational question prompted by the model, not a result anchored in this paper.
The strongest negative case is that this paper has no computational result to mirror. Theorem 3 characterizes simple plurality axiomatically; Theorem 4 characterizes grounded, nontrivial aggregators as weighted plurality rules. Neither makes a complexity claim or studies an algorithmic problem. Theorem 4 does not conceal an algorithm: its proof constructs weights using the aggregator’s own outcome on the input. A continuous version of either theorem would remain an axiomatic question, outside ChoCo’s computational scope.
The proposed Continuous Simple Plurality Evaluation is a valid way to describe annotation populations, but it does not supply the missing computational result. It replaces counts in the paper’s explicit plurality formula with masses and returns the maximizing categories. The suggested one-item assignment regime makes the input especially compact: there are at most \(|J||K|+1\) annotation patterns. Continuous versions of the weighted and bias-correcting rules likewise substitute type-level averages into explicit formulas. The paper studies which rules satisfy its axioms and how they perform on data; it does not pose a difficult computational task about applying them.
A stronger candidate is minimum-mass robustness: given costs for changing annotation types, find the least mass that must change for a collective label to change. That is a genuine continuous computational question, and robustness is a natural concern for annotation. I cannot rule out its being worthwhile. But the paper supplies no model of permissible changes or their costs, and its axioms do not determine whether a change edits one item or an annotator’s whole response pattern. Those choices would define a new intervention problem, not mirror a named result in this paper.
So none of the proponent’s candidates survives as a paper-anchored computational mirror. The case against the broader claim that *no worthwhile follow-up can be built from this model* is weaker: mass-transfer robustness could be a useful adjacent project. The firm conclusion is narrower: the paper itself offers axiomatic and empirical results, not a computational result for continuization to mirror.
The adversarial triple: the proponent anchors on up to three named results; the opponent sees that case and must defeat every anchor; the judge decides which case convinced it. These are the pipeline’s own outputs, generated by tools/triple_run.py — no human edited them. The paper’s own text is not reproduced here beyond the quoted statement above.