Psychometrics — test construction, reliability, and validity

Intuition Beginner

Suppose you step on a bathroom scale, read your weight, step off, and step on again. If the scale gives 72 kg the first time and 68 kg the second time, three minutes apart, something is wrong with the scale. It is not measuring consistently. A measuring tool that disagrees with itself cannot be trusted to tell you anything precise about your actual weight.

That first property — measuring the same thing the same way every time — is called reliability. It is the consistency of a measurement. A reliable bathroom scale gives nearly the same reading each time you step on it, because your weight has not changed in three minutes.

But consistency is not enough. A scale that always reads 10 kg too high is perfectly reliable and perfectly wrong. It measures something consistently — just not your actual weight. The second property — measuring the thing the scale actually claims to measure — is called validity. A valid scale measures weight, not some unrelated quantity that happens to be stable.

The entire field of psychometrics — the science of psychological measurement — is built on these two ideas. Psychologists cannot put "anxiety" or "intelligence" or "extraversion" on a scale and read off a number the way a physicist reads a thermometer. Instead they construct long lists of questions or tasks, collect people's answers, and ask: how consistently does this instrument measure, and what exactly is it measuring?

A psychological test is a measuring instrument. Its raw material is a set of items — individual questions, such as "I feel nervous in crowds: agree / neutral / disagree" on a personality inventory, or "what is 47 × 6?" on a reasoning test. The total score is the sum of the item responses. The whole game of test construction is to design items whose total score behaves the way a good measurement should.

The central question of this unit is the distinction between reliability and validity. A test can be reliable without being valid, and validity is always the harder target. A speedometer that is stuck at 80 km/h is perfectly reliable but tells you nothing about your real speed. The same is true of a psychological test that measures something consistently but measures the wrong thing.

Visual Beginner

The four targets capture the two-by-two: consistency is necessary but not sufficient for measuring the right thing, and a tight cluster off the bullseye is the classic failure mode of a biased test.

Worked example Beginner

Imagine a 3-item "math anxiety" questionnaire given to 200 students. Each student answers three items on a 1-5 scale, and we add the three answers to get a total score from 3 to 15. We want to know: do these three items hang together well enough to count as one measurement?

The internal-consistency check we use is Cronbach's alpha. It rewards items that move together: if students who score high on one item also score high on the others, alpha goes up; if the items measure unrelated things, alpha goes down.

Here is the recipe. Let k be the number of items (here k = 3). Add up the variances of the three items — say item 1 has variance 1, item 2 has variance 2, and item 3 has variance 3, so the total of the item variances is 1 + 2 + 3 = 6. Now find the variance of the total score across all students — say it is 15.

Plug the numbers into the alpha recipe:

alpha = (k / (k - 1)) * (1 - (added item variances / total score variance))
      = (3 / 2)       * (1 - (6 / 15))
      = 1.5           * (1 - 0.4)
      = 1.5           * 0.6
      = 0.90

Alpha runs from 0 to 1. A value of 0.90 means the three items share a great deal of their variation — they behave like three readings of the same underlying quantity. By the rough conventions used in educational and personality testing, alpha above 0.70 is "acceptable" and above 0.80 is "good," so this short scale is unusually coherent.

Notice what alpha does not tell you. It does not say the scale measures math anxiety. It only says the three items move together. They might jointly measure general test anxiety, or simply willingness to agree with questionnaire statements. High alpha is reliability, not validity.

Check your understanding Beginner

Formal definition Intermediate+

Classical test theory (CTT) begins from a single decomposition. Let $X$ be the observed score of a fixed person on a fixed test. Then

$$X=T+E,$$

where $T$ is the true score and $E$ is the error of measurement. The true score is not a hidden Platonic quantity; it is defined operationally as the expected value of the observed score over a hypothetical infinite series of independent administrations of parallel forms:

$$T=\mathbb{E}[X].$$

Consequently $\mathbb{E}[E]=0$. The standard CTT axioms add that the error of person $i$ is uncorrelated with the true score of $i$ and uncorrelated with the true scores and errors of every other person. Under these assumptions the variances decompose cleanly:

$${\sigma }_X^2={\sigma }_T^2+{\sigma }_E^2.$$

The reliability coefficient is the proportion of observed-score variance that is true-score variance:

$${\rho }_{X{X}^{\prime }}=\frac{{\sigma }_T^2}{{\sigma }_X^2}=1-\frac{{\sigma }_E^2}{{\sigma }_X^2}.$$

Reliability is a property of scores in a population, not of a test in the abstract: the same instrument can be highly reliable in a heterogeneous group and unreliable in a narrow one, because restricting the range of true scores shrinks ${\sigma }_T^2$ and therefore shrinks ${\rho }_{X{X}^{\prime }}$.

Validity is the degree to which evidence and theory support the interpretations of test scores for intended uses — the 2014 Standards for Educational and Psychological Testing deliberately treats validity as a unitary judgement supported by several lines of evidence rather than as a set of distinct "types," although three traditional categories remain useful:

• Content validity — the items adequately sample the domain the test claims to cover (a driving test should include both urban and highway items, not only parallel parking).
• Criterion validity — scores correlate with an external criterion of interest, either concurrently or as a predictor (a selection test's scores predicting later job performance).
• Construct validity — scores capture the theoretical construct intended, established by convergent and discriminant correlations, experimental interventions, and factor-analytic structure.

The reliability–validity distinction is asymmetric: reliability constrains validity. A test cannot correlate with a criterion more than the square root of its reliability permits (the reliability-induced ceiling, derived in the proof set), so unreliable measurement caps predictive power. The converse fails — a reliable test may be valid for nothing.

Key model Intermediate+

The workhorse reliability estimator in practice is Cronbach's alpha ^{[Cronbach1951]}. For a test of $k$ items with total-score variance ${\sigma }_X^2$ and item variances ${\sigma }_i^2$,

$$\alpha =\frac{k}{k-1}\left(1-\frac{{\sum }_{i=1}^k{\sigma }_i^2}{{\sigma }_X^2}\right).$$

The intuition is variance accounting. The total variance ${\sigma }_X^2$ splits into item-variance (the diagonal) and item-covariance (the off-diagonal). The numerator ${\sum }_i{\sigma }_i^2$ is the part of the total variance that cannot be explained by items correlating with one another; subtracting its share from 1 leaves the share attributable to shared covariance, which is then rescaled by $k/(k-1)$ to correct for the number of items. When items are uncorrelated, ${\sigma }_X^2={\sum }_i{\sigma }_i^2$ and $\alpha =0$; when items are perfectly correlated, $\alpha =1$.

Alpha matters because it can be computed from a single administration — no retest, no parallel form — which makes it cheap. Its price is a model assumption: alpha equals the reliability coefficient exactly only when the items are essentially tau-equivalent (each item measures the same true score up to an additive constant). Under the weaker condition of congeneric items (each item measures the same latent variable but with different scale factors), alpha underestimates reliability and is best read as a lower bound.

Worked 6-item example. Suppose a 6-item questionnaire produces item variances $2,3,3,4,4,4$ (adding to 20) and a total-score variance of 50. Then

$$\alpha =\frac{6}{5}\left(1-\frac{20}{50}\right)=1.2\times 0.6=\mathrm{0.72.}$$

An alpha of 0.72 sits just above the conventional 0.70 threshold for research use. The interpretation is delicate. Adding more items that correlate with the existing set drives alpha up mechanically — lengthening a scale is the cheapest way to inflate alpha — so alpha rewards length as much as coherence, and a long scale can show alpha above 0.90 while measuring two or three factors at once. A high alpha is consistent with multidimensionality, which is why alpha should be read alongside a factor analysis, never on its own.

The reliability–validity distinction, made concrete through alpha, is the central tension of test construction. Alpha answers "do the items agree with each other?" — a question of internal structure. Validity answers "does the score mean what we say it means?" — a question of external referent. The two questions are different, and the field's recurring scandals (described in the historical section) almost always involve a test that was reliable but whose interpretation outran the validity evidence.

Bridge. Classical test theory builds toward the modern measurement frameworks covered below — item response theory generalises the single error term $E$ into item-specific location and discrimination parameters, and the same variance decomposition appears again in factor analysis, where the true score $T$ becomes a latent factor and the error $E$ becomes unique variance. This is exactly the move from a single number to a structural model: reliability asks how much of ${\sigma }_X^2$ is signal, and the central insight of the rest of this unit is that "signal" can be modeled at the item level rather than only at the total-score level. Putting these together, alpha, factor loadings, and IRT discrimination parameters are three estimators of the same underlying true-score variance. The bridge is that all three are variance-accounting tools, differing in how finely they resolve the structure.

Exercises Intermediate+

Advanced results Master

Classical test theory is a test-level theory: it speaks of the total score $X$, its true score $T$, and a single lump of error $E$. Modern psychometrics refines this picture in three directions, each addressing a weakness of the CTT framework.

Item response theory

Item response theory (IRT) ^[Rasch1960] is a model of the item, not of the total score. For a dichotomously scored item (correct/incorrect, agree/disagree), the Rasch model (also called the 1-parameter logistic model, 1PL) gives the probability that person $j$ with latent ability ${\theta }_j$ answers item $i$ with difficulty $b_i$ correctly as

$$P(X_{ij}=1\mid {\theta }_j,b_i)=\frac{1}{1+e^{-({\theta }_j-b_i)}}.$$

Ability and difficulty live on the same scale, so an item is harder than another exactly when its $b_i$ is larger, and a person is more able exactly when their ${\theta }_j$ is larger. The defining property of the Rasch model is specific objectivity: the comparison of any two persons' abilities is independent of which items are used, and the comparison of any two items' difficulties is independent of which persons are used, provided the items fit the model. This is a strong invariance claim that CTT cannot make — in CTT, a person's score depends on the specific items in the test.

The 2-parameter logistic (2PL) model adds a discrimination parameter $a_i$ that lets the item's logistic curve be steep (sharply discriminating near its difficulty) or shallow; the 3PL model adds a guessing parameter $c_i$ (a lower asymptote, capturing that a multiple-choice item can be answered correctly by chance). Each additional parameter buys flexibility at the price of more data required for stable estimation.

IRT supplies a test information function

$$I(\theta )=\sum _{i=1}^k{I}_i(\theta ),\text{where}{I}_i(\theta )=a_i^2{P}_i(\theta )(1-P_i(\theta )),$$

which says how precisely the test measures at each point of the ability continuum. Unlike CTT reliability (a single number for the whole test), information is local: a test can be highly informative around the average ability and useless in the tails, which is why well-designed achievement tests spread item difficulties across the range of interest. Standard error is the inverse of information: $\mathrm{SE}(\theta )=1/\sqrt{I(\theta )}$.

Factor analysis as the structural tool

Factor analysis ^{[Spearman1904]} is the instrument that turns a matrix of item correlations into a structural claim about latent variables. The common factor model writes each observed item as

$$X_i={\lambda }_{i}F+U_i,$$

where $F$ is a latent common factor, ${\lambda }_i$ is the loading of item $i$ on that factor, and $U_i$ is unique variance (specific variance plus measurement error). With $m$ factors this generalises to $X_i={\sum }_{r=1}^m{\lambda }_{ir}{F}_r+U_i$. Estimation (typically maximum likelihood or minres) finds the loadings and the unique variances that best reproduce the observed covariance matrix.

Factor analysis is how psychometrics answers "what does this test measure?" in a structural rather than correlational way. Spearman's original 1904 observation — that a single factor $g$ accounts for much of the shared variance across diverse cognitive tests — was the founding application. Thurstone's multiple-factor analysis, Cattell's 16-factor model of personality, and the modern five-factor (Big Five) model of personality are the same technique applied to different item batteries. The recurring methodological caveat is that factor structure is underidentified without rotation and without theoretical constraints: the same correlation matrix can be fit by a one-factor model, a three-factor model, or a bifactor model, and the choice between them is a judgement call settled by fit indices, parsimony, and replicability across samples.

Generalizability theory

Generalizability theory (Cronbach, Rajaratnam and Gleser, 1963) decomposes the single CTT error term into multiple facets of measurement — occasions, raters, items, forms — each a source of variance to be estimated through a variance-components (ANOVA-style) model. A generalizability study estimates how much variance each facet accounts for; a decision study then projects how reliability would change under alternative designs (more raters, more occasions, more items). Where CTT offers one reliability coefficient, G-theory offers a generalizability coefficient per intended use, making explicit the design choices that CTT hides inside a single number.

Synthesis. Putting these together, the three pillars of advanced psychometrics — generalizability theory, item response theory, and factor analysis — are not separate disciplines but three parameterisations of the same classical-test-theory identity, that observed variance equals true-score variance plus error. Generalizability theory generalises the single error term of alpha into a variance-components design; item response theory replaces the total-score reliability with item-level curves whose slope, under identification, is this is exactly a rescaled factor loading; and the central insight of factor analysis is that the latent variable underlying a battery of items is dual to the residual structure that reliability lumps together. The foundational reason all three converge is that a good measure must be coherent and unidimensional, and the bridge is that each framework supplies a different diagnostic (generalizability coefficient, information function, loading pattern) for the same underlying requirement: that the signal in the scores be separable from the noise.

Full proof set Master

Proposition (alpha as a lower bound on reliability). Let $X_1,\dots ,X_k$ be $k$ items measuring a common true score, with observed total score $X={\sum }_{i=1}^k{X}_i$. Assume the items are congeneric, meaning each $X_i={\lambda }_{i}T+E_i$ with $\mathbb{E}[E_i]=0$ and the errors mutually uncorrelated and uncorrelated with $T$. Then Cronbach's alpha equals the reliability coefficient ${\rho }_{X{X}^{\prime }}$ when all ${\lambda }_i$ are equal (essential tau-equivalence), and is a strict lower bound on ${\rho }_{X{X}^{\prime }}$ when the ${\lambda }_i$ are not all equal.

Proof. Under congenericity the variance of item $i$ is ${\sigma }_i^2={\lambda }_i^2{\sigma }_T^2+{\sigma }_{E_i}^2$ and the covariance between items $i$ and $j$ ($i\ne j$) is $\mathrm{Cov}(X_i,X_j)={\lambda }_i{\lambda }_j{\sigma }_T^2$. The variance of the total score is

$${\sigma }_X^2=\sum _{i=1}^k{\sigma }_i^2+2\sum _{i<j}{\lambda }_i{\lambda }_j{\sigma }_T^2.$$

The true-score variance of the total is the variance of ${\sum }_i{\lambda }_{i}T=({\sum }_i{\lambda }_i)T$, namely ${\sigma }_T^2({\sum }_i{\lambda }_i{)}^2$. Hence the reliability of the total score is

$${\rho }_{X{X}^{\prime }}=\frac{{\sigma }_T^2({\sum }_i{\lambda }_i{)}^2}{{\sigma }_X^2}.$$

Now examine alpha. By definition,

$$\alpha =\frac{k}{k-1}\left(1-\frac{{\sum }_i{\sigma }_i^2}{{\sigma }_X^2}\right)=\frac{k}{k-1}\cdot \frac{{\sigma }_X^2-{\sum }_i{\sigma }_i^2}{{\sigma }_X^2}=\frac{k}{k-1}\cdot \frac{2{\sum }_{i<j}{\lambda }_i{\lambda }_j{\sigma }_T^2}{{\sigma }_X^2}.$$

So alpha and ${\rho }_{X{X}^{\prime }}$ agree exactly when $k\cdot 2{\sum }_{i<j}{\lambda }_i{\lambda }_j=(k-1)({\sum }_i{\lambda }_i{)}^2$, i.e. when

$$\frac{2}{k(k-1)}\sum _{i<j}{\lambda }_i{\lambda }_j={\left(\frac{{\sum }_i{\lambda }_i}{k}\right)}^2.$$

The left-hand side is the average product ${\lambda }_i{\lambda }_j$ over pairs; the right-hand side is the square of the average loading. By the identity between the variance of the loadings and the gap between these two quantities,

$${\left(\frac{{\sum }_i{\lambda }_i}{k}\right)}^2-\frac{2}{k(k-1)}\sum _{i<j}{\lambda }_i{\lambda }_j=\frac{1}{k-1}\cdot \frac{1}{k}\sum _i({\lambda }_i-\bar{\lambda }{)}^2\ge 0,$$

with equality precisely when all ${\lambda }_i$ are equal. Therefore the average loading squared is at least the average product, the right-hand side of the alpha formula is at most the numerator of ${\rho }_{X{X}^{\prime }}$, and

$$\alpha \le {\rho }_{X{X}^{\prime }},$$

with equality if and only if the items are essentially tau-equivalent. $\square$

Corollary (the reliability ceiling on validity). If a test has reliability ${\rho }_{X{X}^{\prime }}$ and the criterion has reliability 1, then the maximum observable correlation between test and criterion is $\sqrt{{\rho }_{X{X}^{\prime }}}$. Proof sketch. Write the observed test score as $X=T+E$ with $\mathrm{Corr}(E,T)=0$. For any criterion $Y$ driven perfectly by $T$, $\mathrm{Corr}(X,Y)=\mathrm{Cov}(T+E,T)/({\sigma }_X{\sigma }_T)={\sigma }_T^2/({\sigma }_X{\sigma }_T)={\sigma }_T/{\sigma }_X=\sqrt{{\rho }_{X{X}^{\prime }}}$. Measurement error can only attenuate, never amplify, the correlation.

Connections Master

• This unit is the measurement layer beneath the IQ and intelligence-testing material in 29.05.01; every claim there about test-retest reliability, the structure of the IQ score, and the meaning of $g$ depends on the reliability and factor-analytic tools defined here, and the contested race-and-IQ debate cannot be read responsibly without separating reliability of the instrument from validity of its cross-group interpretation.

• The trait-theories unit 29.08.02 pending uses factor analysis as its central structural tool: the Big Five factors (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) are recovered by factor-analyzing large batteries of personality items, and the replicability of that five-factor structure across languages and cultures is itself a psychometric claim about construct validity.

• The reliability and validity coefficients defined here are special cases of the correlation and regression apparatus covered in 29.01.03 pending; the proof that alpha lower-bounds reliability is pure covariance algebra, and the maximum-likelihood estimation underlying factor analysis and IRT belongs to [45.x mathematical statistics]. Readers wanting the distribution theory behind these estimators should cross to the statistics strand.

Historical & philosophical context Master

Psychometrics is unusual among quantitative sciences in that its central coefficient is named after a person who warned against its misuse. Charles Spearman's 1904 paper ^{[Spearman1904]} introduced both the rank-correlation coefficient and the factor-analytic extraction of $g$, founding two of the three pillars of the field in a single article. Spearman himself regarded $g$ as a statistical regularity awaiting a biological explanation, not as a fixed biological property, and he revised his views repeatedly as data accumulated.

The axiomatic foundation of classical test theory is due to Frederick Lord, first in his 1952 monograph ^[Lord1952] and then in the canonical Lord & Novick Statistical Theories of Mental Test Scores (1968), which set the notation still in use ($X=T+E$, parallel tests, the reliability coefficient). Lee Cronbach's 1951 paper ^{[Cronbach1951]} unified a family of earlier "alpha"-style coefficients under the single formula used today and proved it to be a lower bound — a result he later revisited with unease, arguing in the 1980s that alpha had become a ritual number reported to satisfy editors rather than a substantive index of anything in particular. The Georg Rasch model ^[Rasch1960], developed for Danish reading tests, introduced the specific-objectivity property that made IRT attractive for large-scale adaptive testing.

The history of psychometrics is inseparable from the history of its misuses, and any honest treatment must record the contested questions rather than smooth them over.

Cultural bias and test fairness. At least two positions deserve separate statement. The first holds that well-constructed modern ability tests are essentially fair instruments whose group-mean differences reflect real differences in the underlying trait; on this view, "bias" should be defined strictly as statistical bias (differential item functioning, where an item behaves differently across groups of equal ability), and most modern items pass that test. The second holds that statistical item-level fairness is a weak standard that does not address construct bias: a test can be free of differential item functioning and still measure a construct shaped by the linguistic, cultural, and educational assumptions of the population on which it was normed, so that group-mean differences are artifacts of construct drift rather than differences in the trait. The 2014 Standards for Educational and Psychological Testing incorporates both views by requiring evidence of validity for each intended interpretation and use, rather than certifying tests as "fair" or "biased" in the abstract.

Heritability, IQ, and the between-groups fallacy. Behavioral-genetic studies establish that IQ scores are substantially heritable within the populations studied (twin and adoption designs converge on heritabilities around 0.5 in childhood and 0.8 in adulthood in modern Western samples). One position — associated historically with Jensen, Eysenck, and later writers of The Bell Curve — argues that high within-group heritability supports, or at least leaves open, a partially genetic explanation of between-group mean differences. The opposing position — the consensus of the American Psychological Association's 1996 task force Intelligence: Knowns and Unknowns and of the Nisbett et al. (2012) review in American Psychologist — holds that this inference is a logical error: heritability is a within-group variance ratio and is silent about the causes of between-group mean differences, exactly as high within-era heritability of height is consistent with the between-era height gap being entirely environmental (nutrition). The environmental position is further supported by the Flynn effect (raw scores rising ~3 points per decade throughout the 20th century, too fast for genetic change) and by the narrowing of observed group gaps over recent decades. A psychometrics unit cannot adjudicate these debates, but it can and must insist that the coefficient being debated — a reliability, a heritability, a factor loading — be defined precisely enough that the disputants are arguing about the same quantity.

The replication and replication-aware turn. Since 2011, psychology has seen a broad methodological reckoning, and psychometrics has not been exempt. Many published scale-validation studies have failed to replicate, the use of exploratory factor analysis with ambiguous rotation rules has been criticised as underdetermined, and there is growing agreement that confirmatory factor analysis with pre-registered factor structures, measurement invariance testing across groups, and reporting of McDonald's $\omega$ alongside Cronbach's alpha should be the new default. The methodological lessons of this unit — that reliability is population-relative, that validity is interpretation-specific, and that a single coefficient is rarely a sufficient summary of a test — are precisely the lessons the replication literature has rediscovered.

Bibliography Master

@article{spearman1904general,
  author  = {Spearman, Charles},
  title   = {``General Intelligence,'' Objectively Determined and Measured},
  journal = {American Journal of Psychology},
  volume  = {15},
  pages   = {201--293},
  year    = {1904}
}

@book{lord1952theory,
  author    = {Lord, Frederic M.},
  title     = {A Theory of Test Scores},
  series    = {Psychometric Monographs},
  number    = {7},
  publisher = {Psychometric Corporation},
  year      = {1952}
}

@article{cronbach1951alpha,
  author  = {Cronbach, Lee J.},
  title   = {Coefficient Alpha and the Internal Structure of Tests},
  journal = {Psychometrika},
  volume  = {16},
  number  = {3},
  pages   = {297--334},
  year    = {1951}
}

@article{cronbach1963generalizability,
  author  = {Cronbach, Lee J. and Rajaratnam, Nageswari and Gleser, Goldine C.},
  title   = {Theory of Generalizability: A Liberalization of Reliability Theory},
  journal = {British Journal of Statistical Psychology},
  volume  = {16},
  number  = {2},
  pages   = {137--163},
  year    = {1963}
}

@book{rasch1960models,
  author    = {Rasch, Georg},
  title     = {Probabilistic Models for Some Intelligence and Attainment Tests},
  publisher = {Danish Institute for Educational Research},
  year      = {1960},
  note      = {Reprinted by University of Chicago Press, 1980}
}

@book{lordnovick1968,
  author    = {Lord, Frederic M. and Novick, Melvin R.},
  title     = {Statistical Theories of Mental Test Scores},
  publisher = {Addison-Wesley},
  year      = {1968}
}

@book{embretsonreise2000,
  author    = {Embretson, Susan E. and Reise, Steven P.},
  title     = {Item Response Theory for Psychologists},
  publisher = {Lawrence Erlbaum Associates},
  year      = {2000}
}

@book{kaplansaccuzzo2018,
  author    = {Kaplan, Robert M. and Saccuzzo, Dennis P.},
  title     = {Psychological Testing: Principles, Applications, and Issues},
  edition   = {9},
  publisher = {Cengage Learning},
  year      = {2018}
}

@techreport{apa1996standards,
  author      = {{American Educational Research Association, American Psychological Association, National Council on Measurement in Education}},
  title       = {Standards for Educational and Psychological Testing},
  institution = {American Psychological Association},
  year        = {2014}
}

@article{nisbett2012intelligence,
  author  = {Nisbett, Richard E. and Aronson, Joshua and Blair, Clancy and Dickens, William and Flynn, James and Halpern, Diane F. and Turkheimer, Eric},
  title   = {Intelligence: New Findings and Theoretical Developments},
  journal = {American Psychologist},
  volume  = {67},
  number  = {2},
  pages   = {130--159},
  year    = {2012}
}