d4-conf-aaai-2024-0394

Theorem 4.1 establishes a statistical identity, while the paper's runtime discussion is unnumbered and gives no named complexity or algorithmic guarantee. The proposed mass-based stopping problem is precise and related to the paper's cost-quality objective, but it assumes known worker channels and is a new optimization problem rather than a computational result of this paper. The named-result requirement therefore fails.

The objection that survived

The candidate stopping problem assumes worker channels are known, leaving the paper's sparse-data accuracy-estimation contribution outside the mirror; the proponent acknowledges this limitation.

What the mirror covers

The proposed extension covers the online stopping decision and its cost-quality objective. It leaves out OAK and POAK's accuracy estimation, POAKI's IRT model, the statistical theorem, and the empirical comparisons.

Open questions for a prover

The case FOR (proponent)

The strongest positive case is a mirror of the paper’s online decision about when to stop collecting labels. There is a hard qualification: the paper contains no numbered result asserting a computational complexity classification. Its closest named result is Theorem 4.1, proved here. It says that, for fixed label priors \(q\), category \(\ell\), and a worker’s partial confusion-matrix type \(M_i^{(-\ell)}\), there are constants \(\alpha^{(\ell)}\) and \(\beta^{(\ell)}\) such that
\[ \mathbb{E}[\pi_i^{(\ell)}] =\alpha^{(\ell)}c_i^{(\ell)}+\beta^{(\ell)}. \]
Here \(c_i^{(\ell)}\) is the worker’s probability of being correct conditional on reporting \(\ell\), and \(\pi_i^{(\ell)}\) is their average similarity to other workers’ reports of \(\ell\). The theorem supports POAK’s confidence estimates, but it is a statistical identity, not a result that an underlying computational problem is in P, hard, or otherwise classified. The paper’s runtime discussion is unnumbered. Under the requested anchor rule, there is no qualifying anchor.

A plausible *prospective* mirror, though, is Mass-Optimal Online Annotation Stopping. Take a finite categorical label set \(X\), observable item contexts \(U\), and worker types \(W\). A worker type specifies the complete confusion matrix, annotation cost, and assignment eligibility, so workers of the same type are indistinguishable for this problem. Let \(\mu_w\) be the fraction of available workers of type \(w\), and let \(\nu_u\) be the fraction of incoming items with context \(u\). For each context, the instance gives a rational prior over the true label, a rule for how worker types are supplied by the external assignment system, a similarity function, and the aggregation rule. Given a quality target \(Q\) and a maximum of \(r\) annotations per item, find a policy that, after each observed worker type and report, either stops or requests another annotation. The policy must minimize the expected number of annotations per incoming item, weighted by \(\nu\), while achieving expected similarity at least \(Q\). A solution is the policy and its resulting expected cost and quality.

The most credible regime is a large labeling platform with millions of available annotators but a modest number of recurring, genuinely equivalent skill and task profiles, alongside a large stream of items from a modest set of task contexts. The masses are fractions of the workforce and item stream. This fits the paper’s setting: annotators provide labels, and the system decides online whether to pay for more labels. The paper’s focus on the cost-quality trade-off makes that decision recognizable to its authors. A related extension would supply the historical co-label and auditor data as mass distributions and ask how to learn the stopping policy from them.

For fixed \(r\), the stopping problem appears tractable: it has finitely many report histories, and a linear program over policy choices can optimize expected cost subject to the quality constraint. That is a Class A-style direction. The paper’s Theorem 4.1 is relevant supporting evidence: its affine relation between agreement and conditional accuracy extends naturally when the comparison worker is drawn from a distribution over types, by averaging over that distribution. But that extension alone is analytic; it does not establish the complexity of learning from the paper’s partial, noisy training data.

That is the weakest point and the reason this remains a candidate rather than a strong anchor-based case. The proposed stopping problem assumes worker channels and item distributions are supplied. The paper’s contribution includes estimating label accuracy from sparse observations, often without ground truth, and its evaluation uses datasets with only about \(50\)–\(500\) labelers. So the mirror captures the online decision and its cost-quality objective, while the continuous learning problem closest to POAK remains unstated and unclassified.

The case AGAINST (opponent, writing after the proponent)

The strongest objection is that this paper supplies no numbered computational result to mirror. Its only numbered main result, Theorem 4.1, is a statistical identity: under fixed priors and a fixed partial confusion matrix, expected conditional similarity is affine in conditional accuracy. It does not classify an algorithmic problem. Averaging that identity over a distribution of worker types would be a valid extension, but it remains an expectation calculation, not a computational mirror.

The proposed Mass-Optimal Online Annotation Stopping problem is the closest candidate, but its clean formulation assumes the worker channels, item priors, and assignment rule are already known. With finite labels and a fixed annotation limit, its states are just finite histories of observed types and reports; a policy can be found by backward induction or an LP. The worker masses enter only as probabilities for which channel is sampled. Since the paper explicitly leaves worker assignment to another system and imposes no shared-capacity constraint, those masses do not create a population-level decision problem.

A stronger version would optimize a global cost-quality target across item contexts, perhaps with a shared budget for further annotations. That is well-defined, and could use masses of worker and item types. But it changes the paper’s local stopping problem into resource allocation across items. Adding shared worker capacity moves still further into task assignment, which the paper explicitly places out of scope. With a fixed annotation limit, the resulting policy choices remain finite; allowing a growing horizon could make policy computation difficult, but the difficulty would be in the horizon and history space, not in population multiplicity.

Trying instead to preserve the paper’s central contribution—learning label accuracy from sparse, noisy observations—runs into a different mismatch. If channels are supplied by type, the learning problem has been assumed away. If they are unknown, the mirror needs a model of how observations from workers and items reveal those channels, including identifiability and sample requirements. A mass distribution alone does not supply that information. Grouping workers can be sensible when their complete channels really match, but where workers have distinct channels the support may be as large as the workforce; replacing them with a few estimated classes becomes a statistical approximation, not an exact high-multiplicity mirror.

So the anchor-based case against the proposed mirror is strong: Theorem 4.1 is not a computational result, and the natural optimal-stopping formalization either assumes away the paper’s learning problem or adds a resource-allocation problem the paper does not study. The universal claim is weaker. A large platform with recurring, genuinely equivalent worker profiles and a shared annotation budget is a plausible setting in which a mass-based stopping problem could matter, and the paper’s authors might recognize it. I cannot honestly rule out that mirror in every scenario; the negative case is strongest against this paper’s actual computational contribution, not against every possible extension of its cost-quality objective.

The adversarial triple: the proponent anchors on up to three named results; the opponent sees that case and must defeat every anchor; the judge decides which case convinced it. These are the pipeline’s own outputs, generated by tools/triple_run.py — no human edited them. The paper’s own text is not reproduced here beyond the quoted statement above.