Appendix H. Calculating Sensitivity and Specificity for Validation

Run algorithm on all N records for which truth is known:

		Algorithm
		+	-
Truth	+	A	B
	-	C	D

Alg sensitivity = A/(A+B)
Alg specificity = D/(C+D)

From the Algorithm-positive charts (A and C), do human review on a sample (a from A and c from C)

The number, a*, from the sample that the human review finds to be positive is a hypergeometric random variable (A, AP, a) = (population size, subpopulation size, sample size)

From the True positive charts, A, a are sampled

		Sampled?		Total
		yes	no
Human Review	+	a*	AP - a*	AP
	-	a-a*	AN-a-a*	AN
Total		a	A-a	A

a* = S_TP (sample true positives)
a-a* = S_FN (sample false negatives)

Similarly for c*
From the True negative charts, C, c are sampled

		Sampled?		Total
		yes	no
Human Review	+	c*	CP - c*	CP
	-	c-c*	CN-c-c*	CN
Total		c	C-c	C

c* = S_FP (sample false positives)
c-c* = S_TN (sample true negatives)

Estimated TP = S_TPA/a
Estimated FN = S_FNA/a
Estimated TN = S_TNC/c
Estimated FP = S_FPC/c

Estimated HR Sensitivity = Estimated TP/(Estimated TP + Estimated FN) = (S_TPA/a)/( S_TPA/a+ S_FNA/a) = … = a*/a

Estimated HR Specificity = Estimated TN/(Estimated TN + Estimated FP) = (S_TNC/c)/( S_TNC/c+ S_FPC/c) = … = (c-c*)/c

Var(c*) = var(c-c*) = (cCP(C-c)CN)/(C²(C-1))
Var(a*) = (aAP(A-a)AN)/(A²(A-1))

So the variance of the estimated HR sensitivity and HR specificity:
Var((c-c*)/c) = (CP(C-c)CN)/(cC²(C-1))
Var(a*/a) = (AP(A-a)AN)/(aA²(A-1))

Overall:
Sensitivity = Alg sensitivity x HR sensitivity
Specificity = Alg specificity + (1 – Alg specificity) x HR specificity

Method 1: Assuming Alg sensitivity and specificity are known constants (because all charts will undergo algorithm and the truth is known).

Var(Sensitivity) = (Alg sensitivity)² x Var(HR sensitivity)

Var(Specificity) = Var(HR specificity) + Var(HR specificity) x (Alg specificity)²

Table for sample size a = c = 50 (so 50 true + and 50 true -)

Algorithm Sensitivity	Algorithm Specificity	Human Review Sensitivity	Human Review Specificity	Sensitivity 95% CI Half-width	Specificity 95% CI Half-width
0.95	0.50	0.80	0.85	0.09	0.11
0.95	0.50	0.80	0.90	0.09	0.09
0.95	0.50	0.80	0.95	0.09	0.07
0.95	0.70	0.80	0.85	0.09	0.12
0.95	0.70	0.80	0.90	0.09	0.10
0.95	0.70	0.80	0.95	0.09	0.07
0.95	0.50	0.85	0.85	0.08	0.11
0.95	0.50	0.85	0.90	0.08	0.09
0.95	0.50	0.85	0.95	0.08	0.07
0.95	0.70	0.85	0.85	0.08	0.12
0.95	0.70	0.85	0.90	0.08	0.10
0.95	0.70	0.85	0.95	0.08	0.07
0.95	0.50	0.90	0.85	0.07	0.11
0.95	0.50	0.90	0.90	0.07	0.09
0.95	0.50	0.90	0.95	0.07	0.07
0.95	0.70	0.90	0.85	0.07	0.12
0.95	0.70	0.90	0.90	0.07	0.10
0.95	0.70	0.90	0.95	0.07	0.07
0.99	0.50	0.80	0.85	0.09	0.11
0.99	0.50	0.80	0.90	0.09	0.09
0.99	0.50	0.80	0.95	0.09	0.07
0.99	0.70	0.80	0.85	0.09	0.12
0.99	0.70	0.80	0.90	0.09	0.10
0.99	0.70	0.80	0.95	0.09	0.07
0.99	0.50	0.85	0.85	0.08	0.11
0.99	0.50	0.85	0.90	0.08	0.09
0.99	0.50	0.85	0.95	0.08	0.07
0.99	0.70	0.85	0.85	0.08	0.12
0.99	0.70	0.85	0.90	0.08	0.10
0.99	0.70	0.85	0.95	0.08	0.07
0.99	0.50	0.90	0.85	0.07	0.11
0.99	0.50	0.90	0.90	0.07	0.09
0.99	0.50	0.90	0.95	0.07	0.07
0.99	0.70	0.90	0.85	0.07	0.12
0.99	0.70	0.90	0.90	0.07	0.10
0.99	0.70	0.90	0.95	0.07	0.07

Method 2: Assuming Alg sensitivity and specificity are not known constants and use Delta Method for variance. The variance for the Alg assumes the A is binomial(A+B, sensitivity) and C is binomial(C+D,1-specificity)

Var(Sensitivity) = (Alg sensitivity)² x Var(HR sensitivity) + (HR sensitivity)² x Var(Alg sensitivity)

Var(Specificity) = Var(HR specificity) + Var(HR specificity) x (Alg specificity)²
                + Var(Alg specificity) x (HR specificity)² + Var(Alg specificity)

There was no perceptible change in CI width. The variance in the estimates were small due to the large sample size (C+D) and high sensitivity of the algorithm.

Return to Contents
Proceed to Next Section