| paper | d4-conf-aaai-2018-0068 |
| authors | — |
| venue | ? |
| filed under | unclassified |
| judged by | gpt-6-luna / xhigh (triple__luna__xhigh__c6r1) |
| judge confidence | high |
| authors would recognise it | yes |
The only named result the proponent anchors on is an expected-loss bound, not a computational result under the stated criterion. EBM is described as an optimization procedure, but the paper gives no named result establishing its computational properties. The proposed mass allocation is a coherent task-selection extension, but it cannot supply the missing computational anchor.
fails bit a — no named computational result to mirror
The proposed extension covers Proposition 1's statistical error bound and the EBM task-selection criterion; it does not establish a computational result for those tasks or mirror the paper's broader label-aggregation claims.
The strongest positive case I see is a narrow, type-compressed version of selective verification, which I’d call Mass-EBM Selective Verification. There is no named result in the paper establishing a complexity classification such as polynomial-time solvability or hardness. The closest anchor is Proposition 1, proved in this paper (with its proof in the appendix), and the authors compute with the bound it establishes in their EBM strategy.
Proposition 1 says that, for the paper’s semi-supervised estimate \(\hat W_L\), the expected loss error from using \(B\) oracle labels sampled from a verification subset \(L\) is bounded by
\[
O\!\left(\left(1+\frac{\lambda B}{N}\right)
\operatorname{tr}\!\left[
\left(I_X(W^\ast)+\frac{\lambda B}{N}I_L(W^\ast)\right)^{-1}
I_X(W^\ast)\right]\right).
\]
Here \(I_X\) and \(I_L\) are the average Hessians of the negative log-likelihood over all tasks and over the verified subset, respectively. The proposition supplies a computable selection criterion; the EBM section minimizes that criterion to choose tasks. It does not prove a complexity bound for doing so.
For the mirror, consider a very large pool of standardized labeling tasks, such as routine content moderation or product-catalog classification. A task’s type \(x\) is its complete feature and crowd-response signature as used by the model; in the paper’s model, that can be the response vector across workers and labels. Tasks with the same type are indistinguishable to the learner. The instance gives the finite support \(\mathcal X\), rational task masses \(\mu_x\) summing to one, a verification budget \(b\), and the type-level Hessians \(I_x\) at the initial unsupervised estimate. The regime is \(N\) very large with comparatively few distinct supported types \(\tau=|\mathcal X|\), as could arise in a repetitive task stream.
Choose verified mass \(z_x\) from each type, with \(0\le z_x\le\mu_x\) and \(\sum_x z_x=b\). Write \(H_\mu=\sum_x\mu_x I_x\). The mirror asks for an allocation \(z\) minimizing
\[
\operatorname{tr}\!\left[
\left(H_\mu+\lambda\sum_x z_x I_x\right)^{-1}H_\mu
\right].
\]
The fixed factor \(1+\lambda b\) from Proposition 1 does not change the optimizer. A solution is an optimal allocation of verification mass across types; after oracle labels are obtained, the paper’s semi-supervised estimator can be applied to the remaining task mass.
I expect this mirror to be tractable (Class A) under the paper’s positive-definiteness condition, or after its PCA rank reduction. The objective is convex in \(z\): the matrix inside the inverse is affine in \(z\), and the trace-inverse objective with positive semidefinite \(H_\mu\) is convex. So, with finite-precision inputs and suitable conditioning, approximate optimization is a natural continuous-optimization problem. That tractability is a consequence of the mirror’s structure, not a result proved by this paper. Follow-up questions include whether fractional verification offers a meaningful advantage over selecting whole tasks, how large the type support can be before this stops helping, and how sensitive the allocation is to substituting the unsupervised estimate for \(W^\ast\).
I think the authors could recognize this as a high-multiplicity version of their EBM task-selection problem: the signature is already the feature used by their learning objective, and tasks of one signature are interchangeable under that model. The central weakness is that these “population units” are tasks, not agents. This is less direct than a continuous society of workers, and the case depends on treating a large pool of exchangeable data items as the population being continuized. It covers Proposition 1 and the EBM selection objective only; it does not mirror the paper’s broader claims about robust label aggregation.
The strongest objection is that Proposition 1 is statistical, not a computational result of the kind ChoCo is set up to mirror. It bounds expected loss error for an estimator; the EBM section then uses that bound to choose verification tasks by gradient descent. The paper gives no complexity result for that selection problem. The proposed mirror therefore takes a statistical design criterion and asks a new computational question about it.
The proposed mass allocation is coherent. If tasks with the same response vector \(x\) are interchangeable under the model, assigning verified mass \(z_x\) to each type gives a sensible convex design problem. Its subject, though, is the pool of data items: how to allocate an oracle-label budget across task types. The workers are the people in the model, and the paper gives each worker a confusion matrix \(\phi_j\). The EBM selection variable ranges over tasks. This mirror continuizes a data pool and an annotation budget, not a society of agents making choices.
The strongest repair is to use a large pool of repeated, standardized tasks. That makes task types and mass allocation more plausible, and it can make the resulting active-learning problem useful. In the paper, however, \(x_i\) is built from workers’ responses, not a rich description of the task’s semantic content. Tasks sharing \(x_i\) can have different ground truths; they share a model-based posterior and Hessian, not an actual label. Aggregating them is valid for the expected loss criterion, but it makes the mirror an instance of optimal experimental design. Adding enough task detail to distinguish their label-relevant differences risks making most types singletons.
A worker-population version is possible too: group workers by response behavior and represent their shares as a distribution. That changes the model into a random-effects crowd model. It still does not mirror Proposition 1’s selection problem, which chooses tasks for oracle verification. In a worker continuum, response frequencies may simplify inference over the finite task set; the selection unit remains the task.
So the negative case is strong as a recommendation about ChoCo’s scope: this paper offers no clear computational-social-choice result to continuize, and its best mass version is better understood as active learning over a task pool. The universal claim is weaker. If ChoCo includes crowd learning or treats a large task pool as its population, the proponent’s mirror is defensible and may be worthwhile. I would reject it as a ChoCo mirror on scope grounds, while acknowledging that it could be a good project under a broader remit.
The adversarial triple: the proponent anchors on up to three named results; the opponent sees that case and must defeat every anchor; the judge decides which case convinced it. These are the pipeline’s own outputs, generated by tools/triple_run.py — no human edited them. The paper’s own text is not reproduced here beyond the quoted statement above.