Classical test theory, reliability, and factor analysis
Anchor (Master): Lord & Novick 1968; McDonald — Factor Analysis and Related Methods (Erlbaum, 1985); Mulaik — The Foundations of Factor Analysis (McGraw-Hill, 1972)
Intuition Beginner
Every measurement wobbles. Step on a scale, read 72 kg, step off and on again, and you might see 71.8 or 72.3. The reading is not wrong, but it is not exact either: your true weight hides somewhere inside a small cloud of readings. Classical test theory turns that cloud into arithmetic. It writes each observed score as a true value plus a small error, , and then asks how large the error cloud is relative to the spread of true values across people.
If you ask people twenty questions about anxiety and the answers clump — everyone who is anxious on one question is anxious on the others — then one hidden thing, an anxiety level, is probably driving all twenty answers. Factor analysis is the tool that finds those hidden drivers. It reads a table of how every item agrees with every other item and pulls out a small number of underlying factors that explain the pattern. Each item gets a loading: how strongly it rides on each factor.
This unit takes the reliability and validity ideas of the previous unit and makes them into equations you can compute. The reward is a precise number for how much a single score wobbles (the standard error of measurement) and a structural picture of what a battery of items measures underneath. Both are the quantitative backbone of test construction.
Visual Beginner
The scree plot shows how the decision to keep two factors falls out of the eigenvalue sequence: the first two factors carry most of the shared variance and the rest sit below the cutoff line. The factor diagram shows the same model equationally — one common factor accounts for a share of each item, and the leftover is item-specific noise.
Worked example Beginner
A reasoning test is given to 500 adults. The scores spread across the group with a standard deviation of 10. A separate retest study estimated the test's reliability at 0.84. How tightly does one observed score pin down the person's true reasoning level?
Step 1. Find the error share. Reliability is the fraction of score spread that comes from real differences between people; the rest is measurement wobble. The wobble share is .
Step 2. Convert the wobble share into a standard error of measurement. Take the square root of the wobble share and multiply by the score standard deviation: .
Step 3. Build a band around an observed score. A person scoring 100 has an estimated true score near , that is, roughly 92 to 108. Reporting a single 100 hides a four-point wobble either way.
What this tells us: reliability is not a seal of approval. It converts directly into a confidence band through the standard error of measurement. A reliability of 0.84 gives a band of about plus or minus eight points; a reliability of 0.96 would cut that band in half.
Check your understanding Beginner
Formal definition Intermediate+
Classical test theory [LordNovick1968] opens with the decomposition of an observed score of a fixed person on a fixed test into a true score and an error of measurement:
The defining identity makes the true score an operational quantity — the mean of a hypothetical infinite run of independent parallel administrations — not a hidden Platonic attribute. Four axioms complete the model [LordNovick1968]: (i) , which follows at once from ; (ii) , the true score is uncorrelated with its own error; (iii) errors of distinct persons are uncorrelated; (iv) errors on parallel forms are uncorrelated. Two tests are parallel when they measure the same true score ( as random variables over parallel administrations) and have equal error variances, so and . Under the axioms the variances decompose orthogonally:
The reliability coefficient is the population ratio of true-score variance to observed-score variance,
where is a parallel form. The second equality — that the reliability coefficient equals the correlation between parallel forms — is what makes reliability estimable: although is latent, the correlation between two parallel administrations is observable. The standard error of measurement is the standard deviation of the errors,
Reliability has no single estimator; it is operationalised by the design used to obtain a second observation. Test-retest reliability correlates scores on two administrations of the same test to the same group. Alternate-forms reliability correlates two parallel forms. Split-half reliability correlates two halves of the test and then applies the Spearman-Brown correction (derived in the next section) to project back to full-test length. Internal-consistency estimators — Kuder-Richardson-20 for dichotomous items and Cronbach's alpha [Cronbach1951] for continuous items — extract a reliability estimate from a single administration by treating the items as parallel components.
Validity is treated at length in 29.13.01 and is not re-surveyed here; the three classical categories — content, criterion, construct — remain the targets that reliability enables but cannot guarantee. One quantitative constraint carries over: the largest correlation a test can show with any criterion is , the reliability ceiling proved in the sibling unit.
Factor analysis [Spearman1904] writes each of standardised items as a linear combination of common factors plus unique variance:
The coefficient is the loading of item on factor ; the variance is the uniqueness of item . The implied covariance structure is
with the loading matrix, the factor covariance matrix, and . The proportion of item 's variance explained by the common factors, its communality, is when the factors are orthogonal. The eigenvalues of the reduced correlation matrix rank the factors by variance accounted for; the number to retain is the central modelling decision settled by the criteria of the Advanced results.
Counterexamples to common slips Intermediate+
- High reliability does not imply unidimensionality. A battery of three mutually correlated subtests can yield while fitting a three-factor model better than a one-factor model; alpha rewards inter-item covariance, not single-factor structure.
- Reliability is population-relative. The same instrument shows higher reliability in a heterogeneous sample than in a narrow one, because restricting the range of true scores shrinks and therefore shrinks . A reliability coefficient reported without its sample is incomplete.
- The SEM is a property of the test in a population, not of a score. A score of 100 does not carry a different SEM from a score of 130; both share the same , which depends only on and .
- Parallel forms are an idealisation. Real alternate forms are only approximately parallel (unequal error variances, slightly different true scores), so test-retest and alternate-forms estimates are themselves noisy estimates of .
Key model Intermediate+
Theorem (Spearman-Brown prophecy and the standard error of measurement). Let be parallel measurements of a true score , each with error variance , and let be the reliability of a single measurement. Then (a) the reliability of the composite is
and (b) the standard error of measurement of a test with reliability and observed-score standard deviation is [Brown1910].
Proof. For (a), write with . The errors are mutually uncorrelated with common variance , so the variance of their mean shrinks as :
while the true-score component is unchanged, . Therefore the reliability of is
Dividing numerator and denominator by and substituting gives
Averaging and adding differ only by a scale factor, and reliability is scale-invariant, so the same expression gives the reliability of the summed composite .
For (b), the orthogonal variance decomposition gives . Substituting yields , and taking square roots delivers .
Worked example. A test has reliability . Tripling its length with parallel items gives . Solving the prophecy formula for the factor needed to push up to gives . Each additional decimal of reliability costs progressively more items: the Spearman-Brown curve is a rectangular hyperbola in that flattens toward 1.
Bridge. The Spearman-Brown formula builds toward the reliability ceiling on validity in 29.13.01, and appears again in the common-factor model below, where adding parallel items is replaced by adding items that load the same latent factor. The foundational reason reliability scales this way is that averaging parallel components shrinks error variance as while leaving true-score variance intact — this is exactly the signal-averaging logic that every internal-consistency estimator exploits. Putting these together, the standard error of measurement converts the abstract coefficient into a concrete confidence band on a person's true score, and the bridge is that both Spearman-Brown and the SEM are direct corollaries of the single identity .
Exercises Intermediate+
Advanced results Master
The reliability estimators of the Intermediate tier are members of a larger family, and factor analysis is the structural machinery that turns reliability from a single number into a model of what a test measures. The results below record the quantitative refinements that a psychometrician applies before trusting a coefficient.
Guttman's lambda family. Guttman 1945 [Guttman1945] derived six coefficients , each a computable lower bound on the reliability of a test. Cronbach's alpha is Guttman's ; differs from by ignoring the rescaling; corrects for the skew of inter-item covariances; is the maximal split-half reliability; and isolate the two largest covariance contributions. The greatest lower bound (glb) is the tightest such bound and is always at least as large as alpha. That alpha is a lower bound rather than the reliability itself is the source of the recurring warning that a high alpha can mask a multidimensional scale.
McDonald's omega. McDonald's [McDonald1985] is the model-based reliability coefficient that flows directly from a factor model rather than from the alpha covariance algebra. For a one-factor model with loadings and uniquenesses on standardised items,
The numerator is the common-factor variance of the total score; the denominator is the total observed variance. Because allows unequal loadings, it recovers the reliability that alpha underestimates on a congeneric scale; and coincide precisely under essential tau-equivalence. Modern scale validation reports alongside or instead of .
Correction for attenuation. Unreliability shrinks every observed correlation toward zero. Spearman's correction for attenuation recovers the correlation between true scores from two observed correlations and two reliabilities: , proved in the proof set below. The estimator is unbiased for the true-score correlation but inflates sampling variance, so a disattenuated correlation near 1 often carries a confidence interval too wide to be informative.
Factor indeterminacy and the rotation problem. For factors the loading matrix is identified only up to an orthogonal transformation: if reproduces , so does for any orthogonal , because . This rotational freedom is resolved by a rotation criterion that selects, among the equally fitting solutions, the one whose loadings best approximate Thurstone's simple structure — a few high loadings per item, many loadings near zero. Varimax [Kaiser1960] maximises the variance of squared loadings within each column and yields orthogonal factors; oblimin and promax relax orthogonality and allow correlated factors. The choice between orthogonal and oblique rotation is a substantive modelling decision, not a numerical one.
Eigenvalue criteria for the number of factors. Three rules compete. Kaiser's rule retains any factor with eigenvalue exceeding 1 in the reduced correlation matrix; it is fast but over-extracts as the number of items grows. Cattell's scree test plots eigenvalues in descending order and retains the factors before the "elbow" where the curve flattens; it is graphical and judgement-based. Horn's parallel analysis [Kaiser1960] compares each observed eigenvalue to the distribution of eigenvalues from random uncorrelated data of the same size and retains only factors that exceed the random benchmark; it is the most defensible of the three and is the modern default.
Factor scores and their indeterminacy. Once a factor model is fit, a factor score estimates each person's standing on each factor. The estimates are not unique: the factor score indeterminacy coefficient measures how tightly the observed item responses pin down the latent factor, and it can be substantially below 1 even for a well-fitting model. This is the factor-analytic analogue of the standard error of measurement — the same signal-separability question, asked one layer deeper.
From exploratory to confirmatory analysis. Exploratory factor analysis (EFA) leaves the factor structure free to be discovered from the data; confirmatory factor analysis (CFA), introduced by Jöreskog 1969, fixes the loading pattern in advance from theory and tests whether the implied covariance matrix reproduces the observed one, with model fit judged by chi-square, CFI, RMSEA, and SRMR. Measurement invariance testing extends CFA across groups (configural, metric, scalar invariance) to verify that the same construct is measured in each. The arc from EFA to CFA to invariance is the operational meaning of construct validity: a latent variable is well-measured when its factor structure is specified in advance, reproduces the observed covariances, and is stable across the populations on which the score is used.
Synthesis. Classical test theory and factor analysis are two readings of the same variance decomposition, and the foundational reason they interlock is that every reliability coefficient is a lower bound on the ratio of common-factor variance to total variance. The central insight is that alpha, omega, and a factor model's communalities are three estimators of one quantity — the share of score variance attributable to latent signal — and this is exactly why a high alpha can coexist with multidimensionality while a single-factor model's loadings cannot. Putting these together, the Spearman-Brown law, the standard error of measurement, and the correction for attenuation generalise from a test-level ratio to an item-level structural picture, and the bridge is that exploratory factor analysis turns the reliability question ("how much is signal?") into the validity question ("what is the signal?"). The pattern appears again in 29.13.01 as the reliability-validity distinction and builds toward confirmatory factor analysis and measurement invariance as the modern machinery of construct validation.
Full proof set Master
Proposition (correction for attenuation). Under classical test theory, let and , with each error uncorrelated with both true scores and with the other error. Then the correlation between the true scores satisfies
Proof. Expand the observed covariance. Because is uncorrelated with , , and , and is uncorrelated with and ,
Hence the observed correlation is
while the true-score correlation is
Forming the ratio eliminates the shared covariance:
By the reliability identity and , taking square roots gives and . Substituting,
and therefore .
The proposition quantifies how measurement error attenuates every observed correlation: the observed coefficient is the true coefficient multiplied by the geometric mean of the two reliabilities, a factor strictly less than 1 whenever either test is imperfect.
Proposition (communality and reproduced correlation under the orthogonal one-factor model). Under the model with , , and standardised items (), the communality of item is , and for distinct items the reproduced correlation is .
Proof. For the communality, the variance of item splits additively because and are uncorrelated:
Because the item is standardised, , so the share of variance attributable to the common factor — the communality — is .
For the reproduced correlation, with ,
because by the common-factor assumptions. Standardising divides by , so .
The set of residuals is the misfit of the one-factor model; a model that reproduces the observed correlations to within sampling error is one for which a single latent factor is a defensible account of the item covariances.
Connections Master
Psychometrics foundations
29.13.01. This unit is the quantitative depth of the sibling: it derives the equations behind the reliability and validity concepts introduced there (Spearman-Brown, the standard error of measurement, the common-factor model) and supplies the proofs the sibling stated. Every claim here presupposes the reliability-validity distinction and the Cronbach's alpha definition fixed in the foundational unit.Correlation and regression
26.06.01. The covariance algebra that proves the correction for attenuation, the alpha lower-bound, and the reproduced-correlation proposition is the linear-statistics apparatus of correlation and regression. The factor model's identification up to orthogonal rotation is a problem in the eigendecomposition of positive-semidefinite matrices, and maximum-likelihood factor estimation is a constrained optimisation over loading and uniqueness matrices studied in the regression and statistics strands.Cognition and intelligence
29.05.01. Spearman's extraction of the general factor from a matrix of cognitive-test correlations is the founding application of the common-factor model developed here; the empirical claim that a single factor accounts for most shared variance among diverse ability tests is a psychometric claim about communalities and reproduced correlations, settled by the factor-analytic machinery of this unit.Random variables and expected value
26.03.01. The true score is defined as the expectation , and every variance decomposition in classical test theory is an application of the expectation, variance, and covariance algebra of random variables. The population-relative character of reliability — that the same instrument yields different coefficients in samples with different true-score spread — is a direct consequence of variance being a property of a distribution, not of a measurement in isolation.
Historical & philosophical context Master
Charles Spearman's 1904 paper [Spearman1904] founded factor analysis by extracting a single common factor from the correlation matrix of diverse sensory and cognitive tests, an act that simultaneously invented rank correlation and the common-factor model. The reliability of lengthened tests was placed on quantitative footing by William Brown 1910 [Brown1910] and by Spearman independently, giving the prophecy formula that still bears both names. Louis Guttman's 1945 paper [Guttman1945] showed that Cronbach's alpha is one member (his ) of a six-coefficient family of computable lower bounds on reliability, a result that demoted alpha from "the reliability" to "a lower bound on the reliability" and cleared the ground for the greatest lower bound and for McDonald's model-based [McDonald1985].
The structural side of the discipline matured through Thurstone's multiple-factor analysis and his simple-structure criterion for rotation, was automated by Kaiser's varimax procedure and eigenvalue-greater-than-one rule [Kaiser1960], and was made confirmatory by Jöreskog's 1969 introduction of maximum-likelihood confirmatory factor analysis, which fixed the loading pattern in advance and tested model fit rather than discovering structure. The modern reading of factor analysis — exploratory for discovery, confirmatory for testing, measurement invariance for cross-group generalisation — is the operational content of construct validity as defined in the 2014 Standards for Educational and Psychological Testing: a construct is well-measured when its factor structure is specified in advance, reproduces the observed covariances, and is stable across the populations on which the score is interpreted.
Bibliography Master
@book{lordnovick1968,
author = {Lord, Frederic M. and Novick, Melvin R.},
title = {Statistical Theories of Mental Test Scores},
publisher = {Addison-Wesley},
year = {1968}
}
@book{nunnallybernstein1994,
author = {Nunnally, Jum C. and Bernstein, Ira H.},
title = {Psychometric Theory},
edition = {3},
publisher = {McGraw-Hill},
year = {1994}
}
@article{cronbach1951alpha,
author = {Cronbach, Lee J.},
title = {Coefficient Alpha and the Internal Structure of Tests},
journal = {Psychometrika},
volume = {16},
number = {3},
pages = {297--334},
year = {1951}
}
@article{spearman1904general,
author = {Spearman, Charles},
title = {``General Intelligence,'' Objectively Determined and Measured},
journal = {American Journal of Psychology},
volume = {15},
pages = {201--293},
year = {1904}
}
@article{brown1910experimental,
author = {Brown, William},
title = {Some Experimental Results in the Correlation of Mental Abilities},
journal = {British Journal of Psychology},
volume = {3},
pages = {296--322},
year = {1910}
}
@article{guttman1945basis,
author = {Guttman, Louis},
title = {A Basis for Analyzing Test-Retest Reliability},
journal = {Psychometrika},
volume = {10},
number = {4},
pages = {255--282},
year = {1945}
}
@book{mcdonald1985factor,
author = {McDonald, Roderick P.},
title = {Factor Analysis and Related Methods},
publisher = {Lawrence Erlbaum Associates},
year = {1985}
}
@article{kaiser1960application,
author = {Kaiser, Henry F.},
title = {The Application of Electronic Computers to Factor Analysis},
journal = {Educational and Psychological Measurement},
volume = {20},
number = {1},
pages = {141--151},
year = {1960}
}
@book{thurstone1947multiple,
author = {Thurstone, Louis L.},
title = {Multiple-Factor Analysis},
publisher = {University of Chicago Press},
year = {1947}
}
@article{joreskog1969general,
author = {J{\"o}reskog, Karl G.},
title = {A General Approach to Confirmatory Maximum Likelihood Factor Analysis},
journal = {Psychometrika},
volume = {34},
number = {2},
pages = {183--202},
year = {1969}
}
@article{horn1965rationale,
author = {Horn, John L.},
title = {A Rationale and Test for the Number of Factors in Factor Analysis},
journal = {Psychometrika},
volume = {30},
number = {2},
pages = {179--185},
year = {1965}
}
@book{mulaik1972foundations,
author = {Mulaik, Stanley A.},
title = {The Foundations of Factor Analysis},
publisher = {McGraw-Hill},
year = {1972}
}