| paper | d4-conf-ijcai-2017-0862 |
| authors | — |
| venue | ? |
| filed under | unclassified |
| judged by | gpt-6-luna / xhigh (triple__luna__xhigh__c6r1) |
| judge confidence | high |
| authors would recognise it | unclear |
Propositions 1 and 2 give formal guarantees for majority voting and empirical difficulty estimation, so they pass the computational-result gate. But no continuous question is stated for either result: the only proposal changes per-item estimation into allocating item mass among worker pools under new costs and constraints. That is a plausible follow-on, but the cases do not establish it as a continuous analogue of a named result.
fails bit b-changes — the continuum limit changes the question
The proposed extension is motivated by the paper's easy-versus-hard distinction and recommendation to escalate hard items; it does not mirror the formal guarantees in Propositions 1 and 2 or the feature-learning procedure in Algorithm 1.
The strongest honest case is a plausible continuous model of crowdsourcing, but this paper gives no qualifying named computational result to anchor it. Its numbered results are Proposition 1, which bounds majority-vote error by \(\exp(-2N(1/2-d_x)^2)\), and Proposition 2, which bounds the deviation of empirical difficulty from true difficulty. Both are statistical guarantees, not results that place a computational problem in a complexity class. Algorithm 1 and the optimization in Equation (8) do not supply such an anchor either.
A possible mirror would model a large stream of annotation items by masses over recurring feature types, and the available workforce by masses over complete worker types, including their labeling behavior and costs. The taskmaster would choose what fraction of each item type to send to the current crowd versus specialized workers, maximizing the mass receiving labels that meet a specified quality threshold under a budget. A plausible regime is repeated annotation of document or image templates, with many items but relatively few feature types and worker profiles. This connects to the paper’s distinction between easy items that the current crowd can label and hard items that should be escalated.
That is a candidate extension, not a result this paper establishes. It shifts the focus from learning difficulty from sampled items to optimizing a mass-level routing policy, and a simple version may reduce to an LP. Since the paper contains no named computational result of the required kind, there is no defensible per-anchor continuous problem or complexity prediction to present as the positive case.
The strongest point against is that the paper offers no named computational result to mirror. Propositions 1 and 2 bound statistical error; neither classifies a computational problem or analyzes an algorithm’s complexity. Equation (8) gives a learning objective and Algorithm 1 describes a procedure, but the paper does not establish a complexity result for either. So the proposed routing policy is not a continuous version of a result in the paper.
The best version of that policy would assign mass of recurring item types to worker groups, subject to costs and quality requirements. That is plausible as a follow-on: the paper does recommend sending hard items to specialists. But it adds a resource-allocation problem, with capacities, costs, and worker-group response rates that the paper does not model. If those rates are known and worker groups are identifiable, the task is an allocation problem; if they are unknown, learning difficulty from item features remains the paper’s central challenge. Either way, the mirror rests on a new model and question, not on a computational result established here.
There is also a split in what gets continuized. The paper already represents the worker crowd by a distribution \(D_W\) and defines item difficulty by integrating error over that distribution. Making the worker population explicit as masses therefore adds little to its existing statistical model. Making the items a distribution over types is less direct: exact types would need to preserve the features that determine difficulty, so ordinary image or text datasets may have nearly as many types as items. Grouping merely similar items could lose the item-level distinctions the prediction task is meant to capture. Exact repeats make a coherent mass-allocation problem, but they still do not supply a complexity result from this paper.
That leaves a credible research idea, not a demonstrated mirror. The negative case is weakest here: the paper’s recommendation to escalate hard items gives the allocation question a real connection to its motivation, and a researcher could reasonably pursue it. But under the requested anchor test, it does not rescue the case for a continuous mirror of a named result in this paper.
The adversarial triple: the proponent anchors on up to three named results; the opponent sees that case and must defeat every anchor; the judge decides which case convinced it. These are the pipeline’s own outputs, generated by tools/triple_run.py — no human edited them. The paper’s own text is not reproduced here beyond the quoted statement above.