| paper | d4-conf-ijcai-2018-0447 |
| authors | — |
| venue | ? |
| filed under | unclassified |
| judged by | gpt-6-luna / xhigh (triple__luna__xhigh__c6r1) |
| judge confidence | medium |
| authors would recognise it | no |
Theorem 10 gives a quantitative performance guarantee for specified policies, so the claim that the paper has no computational result is too strong. But the only continuous problem actually stated optimizes over all adaptive allocations, while the paper analyzes fixed policies and their error guarantees; that is a plausible follow-up, not a continuous analogue of the named results.
fails bit b-changes — the continuum limit changes the question
The proposed model concerns adaptive annotation allocation and terminal error; it does not continue the paper's named bounds for uniform allocation, uncertainty sampling, or information-gain maximization.
There is no qualifying computational anchor in this paper. The numbered theorems establish error bounds, compare collection policies, or prove a policy equivalence. None classifies a problem by computational complexity. Theorem 10, for example, is proved here and says that for any worker-accuracy distribution \(f_p\) with \(\mathbb{E}\{f_p\}\neq 1/2\), the adaptive policies’ error-exponent efficiency ratio satisfies \(c_{\mathrm{ada}}/c_{\mathrm{uni}}\ge 4\). Here “efficiency” refers to the error exponent. Theorem 4, also proved here, says that with known worker accuracies, uncertainty sampling and information-gain maximisation select the same task, except in a tie. Both are relevant analytical results, but neither is an in-\(\mathrm{P}\), hardness, or parameterized-complexity result. So under your anchor rule, the paper supplies zero anchors.
There is still a plausible continuous-population question motivated by the paper. Consider a very large annotation project with \(M\) exchangeable tasks and \(N\) workers, where \(M,N\) grow while \(N/M=\rho\) stays fixed. Workers fall into a small number \(\tau\) of reliability types, each with a common accuracy \(p\); task difficulty is uniform as in the paper. This is credible for a large standardized corpus annotated by a crowd drawn from a few reliability strata. The high-multiplicity regime is \(M,N\gg\tau\). A worker’s type is their accuracy; each task’s current posterior is its state in the adaptive process.
One possible problem is Continuous Adaptive Annotation Allocation. The input is a finite rational distribution of worker mass over reliability types, an initial unit mass of tasks with prior \(q=1/2\), a fixed worker-arrival mix, and a label budget \(b\) per unit task mass. At each step, the decision is how much arriving worker mass to assign to tasks at each current posterior \(q\). An assignment splits task mass into posterior states
\[
q^+=\frac{qp}{qp+(1-q)(1-p)}
\quad\text{and}\quad
q^-=\frac{q(1-p)}{q(1-p)+(1-q)p},
\]
according to the probabilities of the observed label. The objective is to minimize terminal expected error mass,
\[
\int \min\{q,1-q\}\,d\lambda(q),
\]
where \(\lambda\) is the final distribution of task mass over posterior states. A solution is an allocation policy attaining, or approaching, the minimum; the associated decision question is whether the minimum is at most a given rational threshold \(\varepsilon\).
This is a reasonable continuation of the paper’s task-allocation problem: it preserves adaptive assignment and classification accuracy while making the task and worker populations continuous. It is not anchored by a named computational result in this paper. The authors analyze three specified policies rather than computing an optimal policy over all allocations. The posterior-state distribution can also grow with the number of labels, so I would not yet assign this proposed problem a defensible tractability or hardness class. The mirror’s weakest point is precisely that gap between analyzing fixed policies and optimizing over all adaptive policies; the paper’s strongest positive case is its fit with a large, repeated annotation regime, not a complexity theorem it does not contain.
The strongest case against is that this paper gives ChoCo no computational result to continuize. Theorems 1–3 and 6–9 bound classification error; Theorem 4 proves two policies equivalent under stated conditions; Theorems 5 and 10 compare their error exponents. None classifies the complexity of an algorithmic problem. There are therefore no qualifying result anchors here.
The proponent’s Continuous Adaptive Annotation Allocation is the best replacement candidate, but it changes the question. The paper analyzes three specified collection policies; the proposal asks for an optimal policy over all allocations, minimizing terminal expected error. That may be a worthwhile control problem, but it is not the continuous version of a named result in this paper. And in the paper’s unknown-accuracy setting, task posterior \(q\) alone does not capture what the policy knows: estimates of worker accuracy are updated from workers’ responses across tasks. Capturing that setting requires a richer joint state. The known-accuracy version avoids that complication, but remains a newly formulated optimization problem rather than a continuization of the paper’s results.
So the negative case is strong on the question this assignment sets: this paper has no computational result to mirror, and the proposed candidate is an adjacent problem, not a mirror. It is weaker as a universal claim that no worthwhile continuous model could arise from the paper. A large, standardized annotation project with repeated task and worker types is plausible, and the proposed control problem could interest the authors. If the programme counts such newly formulated problems as mirrors, that is the surviving weakness in the case against.
The adversarial triple: the proponent anchors on up to three named results; the opponent sees that case and must defeat every anchor; the judge decides which case convinced it. These are the pipeline’s own outputs, generated by tools/triple_run.py — no human edited them. The paper’s own text is not reproduced here beyond the quoted statement above.