| paper | d4-journals-jair-2022-0112 |
| authors | — |
| venue | ? |
| filed under | unclassified |
| judged by | gpt-6-luna / xhigh (triple__luna__xhigh__c6r1) |
| judge confidence | medium |
| authors would recognise it | yes |
Theorem 2 (DRC)
statement extracted from the paper’s text layer
Given a population of \(N\) workers, type masses \(\mu_t\), per-type annotation costs \(c_t\), DRC moment bounds \(V_{t,y}\) and \(D_{t,y}\), and tolerance \(\varepsilon\), choose query rates \(q_t\in[0,1]\) to minimize \(N\sum_t\mu_t c_tq_t\), subject for each label \(y\) to \(\frac{1}{N}\sum_t\mu_t\left(\frac{V_{t,y}}{q_t}+\left(\frac{1}{q_t}-1\right)D_{t,y}\right)\le\varepsilon^2\). Rates are applied independently to workers of each type, and a zero rate is permitted only when the direct estimate is exact for every label.
A society of \(N\) workers with type distribution \(\mu\), where each type fixes annotation and imitation behavior, score functions, and query cost. The decision variables \(q_t\) are per-worker query probabilities; the objective is expected total annotation cost, and the DRC variance constraints scale with \(N\) while retaining independent worker responses.
Theorem 2 takes sampling rates as inputs, while the proposed mirror optimizes them; that design layer is not analyzed in the paper, even though its cost-saving objective and AWS method make it recognizable.
fatal: False
The mirror uses Theorem 2’s DRC variance guarantee to allocate annotation effort across worker types; it leaves Theorems 1 and 3 and the paper’s experimental claims about AIS and AWS outside the optimization.
The strongest honest positive case is a high-multiplicity version of the paper’s worker-sampling problem. The paper’s most useful named anchor is Theorem 2 (DRC), proved here: for each label \(y\), the doubly robust estimator of \(v_x(y)=\frac{1}{m}\sum_i \mathbb{E}[S_i(y,\ell_i(x))]\) is unbiased, with variance \(\frac{1}{m^2}\sum_i\left[\frac{1}{\pi_i}\operatorname{Var}(S_i(y,\ell_i(x)))+(\frac{1}{\pi_i}-1)\mathbb{E}(S_i(y,\ell_i(x))-S_i(y,\hat\ell_i))^2\right]\). That is a result about a computational method’s statistical guarantee, not a complexity classification: the paper has no named theorem saying a problem is in P, NP-hard, or otherwise classified. So Theorem 2 is a plausible algorithmic-statistical anchor, but it is not a strict fit for the complexity-result anchor requirement.
The scenario is a large standing annotation pool handling new images, documents, or other items. For any fixed item, workers fall into a relatively small number of calibrated types: workers in a type have the same annotation cost, scoring behavior, response distribution, and imitation prediction for that item. A pool might have many thousands of available workers but only dozens of such types. The type must include every feature the model uses; the point is that the type counts are high, not that workers within a type have different prices or skills.
Call the mirror DRC-Type Allocation. An instance gives a population scale \(N\), rational type masses \(\mu_t\) summing to one, and \(N\mu_t\) workers of each type \(t\). It also gives the label set \([k]\), type-level scores \(S_t(y,\ell)\), imitation scores \(d_{t,y}\), per-annotation costs \(c_t\), and certified bounds \(V_{t,y}\) and \(D_{t,y}\) on the two corresponding second-moment terms in Theorem 2. Given an error tolerance \(\varepsilon\), choose query rates \(q_t\in[0,1]\) to minimize expected annotation cost \(N\sum_t\mu_t c_tq_t\), subject to the DRC estimate being unbiased and, for every label \(y\), its variance bound being at most \(\varepsilon^2\):
\[
\frac{1}{N}\sum_t\mu_t\left(\frac{V_{t,y}}{q_t}+\left(\frac{1}{q_t}-1\right)D_{t,y}\right)\le\varepsilon^2.
\]
The corresponding solution is the optimal vector of rates, together with the DRC estimates obtained by querying at those rates. A zero rate is allowed only for a type whose direct estimate is already exact for every label; otherwise its rate must be positive. The objective and constraints preserve the paper’s central task—spending fewer annotations to estimate the all-workers score—while replacing named workers with masses of interchangeable workers.
The expected direction is Class A for this allocation question: with the stated bounds, it is a convex optimization problem over \(\tau\) rates. It also creates useful follow-on questions: how closely finite worker counts can implement the fractional allocation; whether type-level rates lose much compared with individualized worker selection; and whether minimum cost subject to a true winner-error probability bound remains tractable when the variance bounds are unavailable. This tractability claim is for the proposed mirror, not a result proved in the paper.
The authors would have a good reason to recognize the mirror: the paper’s ideal-world score already aggregates workers, its target is to reduce annotation cost, and its AWS method already varies worker sampling probabilities. The type masses retain the noisy annotations, imitation baseline, and correction term; they do not replace the paper’s estimation problem with a simpler voting rule. The weakest point is that this allocation problem is a formal optimization layer added to their method, not a complexity result they establish. Their experiments use relatively small worker sets, and their worker models can be individualized, so the many-workers/few-types regime needs its own evidence. That leaves a credible mirror of Theorem 2’s problem, but a modest one for the continuization programme’s complexity agenda.
The strongest objection is that Theorem 2 does not classify a computational problem. It gives an unbiasedness and variance formula for a specified estimator under a specified sampling design. The paper contains no complexity theorem for choosing workers, nor an optimization problem over worker populations. Replacing its finite sum with an integral over worker types would extend the statistical analysis, but it would not expose a complexity question hidden by finite multiplicities.
The proposed DRC-Type Allocation is coherent, but it adds the main decision problem it purports to mirror: choose query rates to meet a variance target at minimum cost. It also supplies type-level costs and certified moment bounds as input. With those data, the resulting optimization is a separable convex allocation problem in the type rates. That may be useful, and the authors might recognize it as related to their goal of saving annotation cost, but it is a new sampling-design layer rather than a continuous version of Theorem 2’s result. Its tractability would say little about the computational landscape of the paper’s estimator.
A stronger mirror could take a probability distribution over worker response types and ask how to estimate the mean score for a fixed item with a finite query budget. This avoids relying on the paper’s particular finite worker panel. But the finite budget is essential: if querying means observing a positive mass of an atomless population, the estimator’s sampling noise collapses; if querying means drawing a finite number of workers, the question is standard sampling and doubly robust estimation over a type distribution. Either way, the key difficulty is statistical sampling, not a computational social choice problem whose structure changes when the population becomes continuous. Adding a winner-error guarantee or adaptive stopping rule could make the design more useful, but would further specify a new problem and require quantities such as score gaps or uncertainty bounds.
There is a real weakness in this negative case: a large annotation pool with many workers sharing response and cost profiles is a plausible high-multiplicity regime, and the paper cares directly about allocating annotation effort. I cannot honestly rule out a useful continuous extension in every such scenario. The stronger claim is narrower: this is a poor fit for ChoCo’s complexity agenda. The paper’s named anchor is a statistical guarantee, and the best population-based reformulations lead to sampling design, not to a new computational landscape for the paper’s core result.
The adversarial triple: the proponent anchors on up to three named results; the opponent sees that case and must defeat every anchor; the judge decides which case convinced it. These are the pipeline’s own outputs, generated by tools/triple_run.py — no human edited them. The paper’s own text is not reproduced here beyond the quoted statement above.