| paper | d4-conf-atal-2023-0497 |
| authors | — |
| venue | ? |
| filed under | unclassified |
| judged by | gpt-6-luna / xhigh (triple__luna__xhigh__c6r1) |
| judge confidence | high |
| authors would recognise it | yes |
Lemma 4.2 establishes a structural property of a lower-bound proxy, not a computational result. Algorithm 1 is an efficient procedure, but it is not a numbered theorem, lemma, corollary, or proposition asserting a computational guarantee. A mass analogue can be stated, but that does not supply the missing named result.
fails bit a — no named computational result to mirror
The attempted mirror concerns per-instance human subset selection; it leaves confusion-matrix estimation, probability calibration, deferred prediction, and the experiments aside.
The strongest honest positive case is narrow: a large crowd of interchangeable labelers is a plausible population to continuize, and the paper’s subset-selection objective has a natural mass-valued version. There is an important limit, though: the paper has no numbered result classifying a problem as polynomial-time solvable, NP-hard, or otherwise. Its numbered Lemmas 4.1 and 4.2 are proved here, but they establish accuracy and monotonicity properties, not a complexity class. Algorithm 1 is presented as an efficient procedure for choosing a pseudo-optimal subset; it is not a numbered complexity theorem for the exact maximum-accuracy problem.
The closest named anchor is Lemma 4.2, proved in this paper. For a fixed instance \(x\), it says that
\[
f_S(x)=\prod_{i\in S}
\frac{\phi^{[i]}_{h_i(x),y(x)}}{1-\phi^{[i]}_{h_i(x),y(x)}}
\]
need not be monotone as the selected human set \(S\) grows, even when every human’s overall accuracy exceeds \(0.5\). Adding a human multiplies the expression by that human’s odds factor, which can be below \(1\); more labels therefore need not help. This is a structural result motivating subset selection, not a hardness or tractability result.
My lead mirror is Continuum ComHAI Evidence Selection. Fix a classification instance \(x\) with \(K\) classes and calibrated AI probabilities \(m_j^\theta(x)>0\). Suppose the available labelers have a finite set of confusion-matrix profiles \(a\). For this instance, let \(t=(a,r)\) denote labelers with profile \(a\) who supplied the same hard label \(r\), and let their available population mass be \(N\mu_t\), where the rational proportions \(\mu_t\) sum to \(1\) and \(N\) is the pool size. Their type-specific likelihood for class \(j\) is \(\phi_{t,j}=\phi^a_{rj}>0\).
Choose a mass \(z_t\in[0,N\mu_t]\) of each type to use. The combined posterior is
\[
q_j(z\mid x)=
\frac{m_j^\theta(x)\prod_t\phi_{t,j}^{z_t}}
{\sum_{\ell=1}^{K}m_\ell^\theta(x)\prod_t\phi_{t,\ell}^{z_t}}.
\]
The problem is to return selected masses \(z\) and a predicted class \(j\) maximizing \(q_j(z\mid x)\), to additive accuracy \(\varepsilon\). Since \(q_j\) is the probability that class \(j\) is correct under this model, the objective is expected classification accuracy for the instance. The population being continuized is the human labeler pool; classes and each individual’s label remain discrete. Fractional \(z_t\) is the mass-valued relaxation of selecting individual labelers. The pool scale \(N\) is explicit because the amount of human evidence matters relative to the AI probabilities.
This is recognizably the paper’s problem: it uses the same human labels, confusion matrices, conditional-independence assumption, AI probabilities, and accuracy objective. The co-modification is that interchangeable labelers are selected by mass, and mass contributes proportionally to log-likelihood. The scenario I find plausible is a large crowd-labeling pool with many labelers in a small number of calibration profiles. For a given query, each profile is divided among at most \(K\) reported-label groups, so the support \(\tau\) can be far smaller than the number \(N\) of workers. This is a regime claim about such a pool, not a claim that every human-AI task has high multiplicity.
I would expect this mirror to be tractable approximately, in Class A. For a fixed candidate class \(j\), maximizing \(q_j\) is equivalent to minimizing
\[
\log\sum_{\ell\ne j}
\exp\!\left(
\log\frac{m_\ell^\theta(x)}{m_j^\theta(x)}
+\sum_t z_t\log\frac{\phi_{t,\ell}}{\phi_{t,j}}
\right).
\]
This is a convex log-sum-exp objective over a box in \(\tau\) variables. Solve one such convex problem for each of the \(K\) candidate classes and keep the best result. I would expect polynomial-time additive approximation under the usual convex-optimization assumptions; an exact bit-complexity claim would need to handle conditioning and ties. That is my analysis of the proposed mirror, not a result established by this paper.
The mirror covers the paper’s per-instance choice of human predictions for combination. It does not continuize the learning of confusion matrices, probability calibration, deferred prediction, or the paper’s experiments. It also does not claim that the discrete problem is hard and that the continuum dissolves that hardness: this paper supplies no named hardness anchor for that comparison. A useful next question is whether an \(\varepsilon\)-optimal mass solution can be rounded to an integral subset with a useful accuracy guarantee.
The weakest point is the regime: ordinary panels for one image or case are often small, and workers with the same estimated confusion matrix may still differ in ways relevant to the decision. The mirror is most convincing for large crowd pools where many workers really are interchangeable under the paper’s model, and where the response groups have substantial multiplicity. If those conditions fail, the continuous formulation is much less compelling.
The paper gives no named computational result for a continuous mirror to classify. Lemma 4.2 proves that a particular lower-bound proxy can decrease when another human is added; it does not establish a complexity result, and it does not prove that the combined model’s actual accuracy is non-monotone. Algorithm 1 selects a pseudo-optimal subset for that proxy, rather than solving a stated exact maximum-accuracy problem with a complexity guarantee.
The direct mass version of Lemma 4.2 adds little computational content. Group labelers by confusion profile \(a\) and reported class \(r\), and let \(z_{a,r}\) be the selected mass. For a proposed class \(j\), the proxy becomes
\[
\log f_z(x)=\sum_{a,r} z_{a,r}
\log\!\left(\frac{\phi^a_{rj}}{1-\phi^a_{rj}}\right).
\]
Over independent bounds on each mass, its maximum takes all available mass where the coefficient is positive and none where it is negative. This is the same threshold rule as Algorithm 1, applied to grouped counts. If the workers really are interchangeable, the discrete computation can already use those counts; the continuum does not unlock a harder or newly tractable problem.
The proponent’s stronger posterior objective is a different problem, and its interpretation is shaky. It chooses response-specific mass after seeing the reports, then treats the selected reports as though the selection had not depended on them. That can reward cherry-picking evidence favorable to a chosen class. To optimize expected accuracy cleanly, one could choose a panel before reports arrive, or explicitly model the selection policy and all observed reports. But a precommitted panel loses the paper’s per-instance subset choice; modeling the policy changes the statistical problem. Neither is the computational claim in Lemma 4.2.
There is a plausible nearby question: allocate a budget across worker types before labeling, perhaps with a distribution over tasks. Large crowds can make that population regime sensible, and the paper itself mentions cost-aware selection as future work. But this requires a task distribution, costs, and a new objective. It is best seen as a cost-sensitive crowdsourcing design problem inspired by the paper, not as a continuous mirror of a named result here.
So the case against is not that labelers can never form a high-multiplicity population. In large crowds they can. The weakness is that the paper’s named anchor is a simple proxy property whose mass version reduces to thresholding, while the more ambitious formulations either optimize a selection-biased confidence score or introduce a distinct design problem. The universal claim remains difficult to prove; the honest negative is that this paper supplies little computational substance for ChoCo to continuize.
The adversarial triple: the proponent anchors on up to three named results; the opponent sees that case and must defeat every anchor; the judge decides which case convinced it. These are the pipeline’s own outputs, generated by tools/triple_run.py — no human edited them. The paper’s own text is not reproduced here beyond the quoted statement above.