Silk, Alvin J.

DSpace@MIT

oon-'i'^WORKING PAPERALFRED P. SLOAN SCHOOL OF MANAGEMENTA NOTE ON TESTING FOR CONSTANT RELIABILITYIN REPEATED MEASUREMENT STUDIESAlvin J. Silk*Working Paper \Uu '[ - I S j^iy i^j^MASSACHUSETTSINSTITUTE OF TECHNOLOGY50 MEMORIAL DRIVECAMBRIDGE, MASSACHUSETTS 02139A NOTE ON TESTING FOR CONSTANT RELIABILIT"^IN REPEATED MEASUREMENT STUDIESWorking PaperAlvin J. Silk*Professor of Management ScienceSloan School of ManagementMassachusetts Institute of TechnologyABSTRACTThis paper discusses the potential usefulness of applyingtests for the equality of variances (and covariances) to data fromrepeated measurement studies prior to estimating reliability componentsand coefficients. In situations where only two rounds of repeatedmeasures are available, a test for the equality of the two (correlated)variances affords a means of checking the consistency of data with acondition necessary for a test-retest correlation to have a straight-forward interpretation as a reliability coefficient. In cases wheremore than two waves of observations have been obtained, a test ofthe hypothesis that all the variances are equal and all the covariancesare equal provides evidence as to the possible constancy of measurereliability across several waves of observations and is thereforerelevant to the selection of an appropriate method for estimatingreliability. An illustrative application of the tests is presented.^^i^saoPapers by Heise (1969) and Wiley and Wiley (1970) have discussedthe estimation of reliability coefficients from repeated measurementswhen a true change occurs between adjacent measures and the usualpsychometric error model of parallel measurements (Lord and Novick,1968, pp. 41-50) does not hold. As Wiley and Wiley (1970, pp. 113) havenoted, if the true change involves a simple additive shift whereby atrue score at time t + 1 increases (or decreases) by some fixedamount (that is identical for all observations) from their previouslevels at t , then the true score variance will be equal for bothsets of measures. Provided the error variances are stable, two roundsof measurements are sufficient in such a circumstance to estimatereliability which would, of course, by definition be the same at t andt + 1 . Winer (1962, pp. 124-30) has discussed ANOVA procedures forestimating reliability components from two or more waves of repeatedmeasurements under the assumptions of a constant differenc? between twopairs of repeated measures and stable error variance. However, Wileyand Wiley argue that the assumption of stable true score variances acrossrepeated measurements is implausible in "most cases of prictical interest"and go on to develop a method for estimating reliability coefficientsfor a situation where the process of change can be modeled in terms oflinear relationships between adjacent true scores and wnere error vari-ance remains constant. Lord and Novick (1968, p. 218) anc Coleman (1970,p. 453) had pointed out earlier that at least three waves jf repeated-1-measurements are required to identify the reliability parameters forsuch a change model.The purpose of this note is to draw attention to the potential use-fulness of some available significance tests in assessing the consis-tency of repeated measurements data with different assumptions aboutthe stability of variances. Prior to actually applying some method ofreliability estimation to a body of data from a repeated measurementstudy, consideration needs to be given to what assumptions are tenableconcerning the stability of true and error variances. For example,evaluation of the reliability of instruments used in sociologicalresearch is often based on test-retest studies (see, for example,Robinson et al., 1968) and so the question may arise as to whetheror not it is reasonable to assume such data satisfy the condition ofconstant true and error variances which is necessary for a reliabilitycoefficient to be estimated from only two sets of repeated measurements.Similarly, in the case of studies involving more than two waves ofmeasurements, one may wish to determine if the assumption of constantreliability across repeated measurements is contradicted by the natureof the data, thereby indicating the appropriateness of employing Wileyand Wiley's estimation procedure rather than that described in Winerand referred to above. To illustrate how certain available statisticaltests may be used for such diagnostic purposes, we present someempirical results from a repeated measurements study where the assump-tion of constant reliability appeared to hold.ILLUSTRATIVE APPLICATIONThe data examined here were originally reported by Palda (1966,p. 18) and consist of a measure of the "awareness" stage in the adop-tion process model found in the sociological literature on the diffusionof innovations (Rogers and Shoemaker, 1971) and routinely used in market-ing research. Awareness levels for a newly launched consumer good weremonitored in each of thirty cities at three different pcints in time,separated by two month intervals. Thus, the matrix of observations avail-able for analysis corresponds to a panel design involving three waves ofmeasurements on the same sample of size thirty (cities) . The awarenessmeasurements are proportions and were derived from telephone surveys con-ducted with samples of four hundred respondents drawn separately in eachtime period in each city. Since the sampling variance of a bionomialproportion is dependent upon the mean, over time shifts in mean awarenesslevels would lead to heterogeneity in variances if the raw awareness pro-portions were used in the analyses. A suitable variance stabilizingtransformation can be used to circumvent this condition. The angular orarcsin transformation has this property and was applied here, i.e.,-1 1/2X=sin (A),where X is the transformed score and A is the originalawareness proportion. If all the error variance is due to sampling abionomial proportion, the sampling variance in the arsin scale (degrees)is equal to 820. 7 /n. Thus, the sampling variance of a proportion sotransformed is no longer dependent upon the mean and is essentiallya constant for a given size sample, n (Snedecor and Cochran, 1967,p. 325). Table 1 presents the variance-covariance matrix and someother relevant summary statistics for the three waves of awarenessmeasurements in the arsin scale. As expected for a diffusion process,the mean awareness levels increase monotonically over time but notethat the variances exhibit a nonmonotonic pattern of fluctuations.INSERT TABLE 1 HEREConsider first the statistics for the first two periods: thecovariance, Cov (X ,X )= 34.695 and the variances, Var (X )= 38.422,Var (X„)= 40.313. The related value of the product moment correlationwas r(X ,X„)= .882. Now suppose one were interested in determiningwhether this correlation could be viewed as a conventional test-retestmeasure of reliability. For the usual error model, a necessary condi-tion for a test-retest correlation to represent a reliability coeffi-cient is that the true score and error variances be constant. If thetrue score and error components are independent, the variance of theobserved scores is simply the sum of the constant true and error vari-ances and therefore within the limits of sampling error, one shouldexpect to find that the observed variances for the test and retestscores are equal— . This suggests a test be made of the followingnull hypothesis;Var (X^) = Var (X2)Assuming the underlying distribution of the variates is liivariatenormal, the test for equality of two correlated variances due toPitman (1939) and described in Snedecor and Cochran (1967, pp. 195-197)2/may be used to assess the above hypothesis— . Rejection of the nullhypothesis here would imply that either the true score variance or theerror variance (or both) was (were) not constant for bor.h sets ofobservations and so their reliabilities could not be equal. Applyingthe aforementioned test to the above variances, Var (X ) and Var (X ),we find the value of the relevant t statistic to be .269 (df=28) whichis clearly not significant, and the hypothesis of constant observedscore variances in the first two periods appears tenable. Thus, inthis case, the test provides evidence to support acceptance ofr(X ,X„) as a measure of reliability. Testing for the equality ofobserved score variances in studies where only two rounds of repeatedmeasurements have been obtained can serve as a safeguard against test-retest correlations being misinterpreted as reliability coefficientswhen the underlying data are not a suitable basis for the assessment3/of reliability —.A related test bearing on the question of constant reliabilitym;iv be employed v^hen data .ire avnil.ible frcim mi^re tlian two waves ofrepeated measurements. By the same line of reasoning as that notedabove for the case of test-retest observations, if the score and errorvariances are constant for each wave of the repeated measurements,then it follows that all observed score variances should be equal andall the covariances between the observed scores should also be equalA consistency check for the latter conditions may be obtained for thevariance and covariances shown in Table 1 by testing the followingcomposite null hypothesis:Var (X^) = Var (X^) = Var (X^),Gov (X^,X2) = Gov (X^,X^) = Gov (X2,X^)A likelihood-ratio test for such a hypothesis under the assumptionthat the variates follow a multivariate normal distribution has beendeveloped by Box (1950, pp. 372-276) and is also described in Winer(1962, pp. 370-374) and Morrison (1976, p. 250). The value of therelevant chl square statistic for these data is 1.364 (df=4) which isnot significant (.90>p>.80) and the null hypothesis of equal variancesand equal covariances cannot be rejected. Hence the notion that allthree waves of measurements have the same reliability can be maintainedand the analysis of variance method suggested by Winer (1962,pp. 124-130) can be applied to estimate the reliability components. Inthe present context, the ANOVA model is given by:X.^ = U + C. + R + e.It 1 t Itwhere X. is the arcsin transformed value of the awareness score forItthe city i in time period t (i=l,...,30 and t=l,2,3); U is the grandmean; C. is the effect of city i; R is the effect of the t timeperiod; and e. is the random error component.Table 2 summarizes the ANOVA results. We observe that the Fstatistic for the "between time periods" effect is highly significantindicating that an additive shift occurred in the mean awareness levelin at least one of the time periods. Note that value of the error5/variance is estimated to be 5.848 . It is interesting to note thatapplying Wiley and Wiley's model of linearly related adjacent truescores to the present data, yields an estimated error variance of4.59 which is somewhat smaller than that obtained above under theassumption of a simple additive shift in true scores. Using the ANOVAcomponents from Table 2 in Winer's recommended computational procedure,we obtain an estimate of .849 for the reliability of a single awarenessmeasure.INSERT TABLE 2 HERESUMMARYThis note has discussed the possible value of applying tests forequality of variances (and covariances) to data from repeated measure-ments studies prior to estimating reliability components and coeffi-cients from them. In situations where only two rounds cf repeatedmeasures are available, a test for the equality of correleted vari-ances affords a means of checking the consistency of data vith acondition necessary for a test-retest correlation to have a straight-forward interpretation as a reliability coefficient. In cases whenmore than two waves of repeated observations have been obtained, atest of the hypothesis that all the variances are equal and all the co-variances are equal bears on the question of whether or not thereliability of the measure employed can be considered constant acrossthe several waves of observations and hence is relevant to the selec-tion of an appropriate method for estimating reliability. The normal-ity assumption underlying the tests illustrated does, of course,represent a restriction on their applicability.TABLE 1SUMMARY STATISTICS FOR THREE WAVES OF AWARENESS MEASUREMENTS*(Arcsin Transformation)1011FOOTNOTESThat is, the observed score (X) is assumed to be the sum of a truescore (t) and an error component (e) , X= T + e . With repeatedmeasurements, it is further assumed that errors are seriallyuncorrelated.If the true scores and errors were normally distributed, thenthe observed scores would also be normally distributed.It is worth noting however, that if the inequality of observedscore variances for two waves of measurement is generated bythe change model proposed by Wiley and Wiley, a test-retestcorrelation does have a reliability interpretation, albeit not astraightforward one. Assuming as Wiley and Wiley do that(i; the true score on the second test is linearily related tothat for the first test and (2) the error variances are constant,then one can easily show that the product moment correlationbetween the two sets of measures is equal to the geom.etric meanof the separate reliability coefficients for the two rounds ofmeasurements. Also, the square of such a test-retest correlationwould provide a lower bound for the two unobserved reliabilitiesbecause the values of all three of these quantities must liebetween zero and one. See Silk (1977) for a discussion of thesepoints.Recall that for the usual error model, the covariance betweenrepeated measures is equal to their (constant) true scorevariance.This may be compared to an expected value of 820.7/400=2.05 ifall the error variance was due to sampling a binomial processwith samples of four hundred respondents as were used in thesurveys for each time period and city from which thess awarenessmeasures were obtained. Thus, a substantial amount of non-sampling error appears to be present here.12REFERENCESBox, G.E.P.1950 "Problems in the Analysis of Growth and Wear Curves."Biometrics 6:362- 89.Coleman, J.S.1968 "The mathematical study of change." Pages 428-278 inH.M. Blalock and A.B. Blalock (eds.), Methodology inSocial Research . New York: McGraw-Hill.Heise, D.R.1969 "Separating reliability and stability in test-retestcorrelation." American Sociological Review 34:43-101.Lord, F.M. and Novick, M.R.1968 Statistical Theories of Mental Test Scores . Reading,Massachusetts: Addison-Wesley.Morrison, D.F.1976 Multivariate Statistical Methods . 2nd edition. New York:McGraw-Hill.Palda, K.S.1966 "The hypothesis of a hierarchy of effects: a partialevaluation." Journal of Marketing Research 3:13-24.Pitman, E.J.G.1939 "A note on normal correlation." Biometrika 31:9-12,Robinson, J. P., Rusk, J.G. , and Head, K.B.1968 Measures of Political Attitudes . Ann Arbor, Michigan:Institute of Social Research, University of Michigan.Rogers, E.M. and Shoemaker, F.F.1971 Communication of Innovations . New York: Free Press.Silk, A.J.1977 "Test-retest correlations and the reliability of copytests." Journal of Marketing Research 14:476-486.Snedecor, G.W. and Cochran, W.G.1967 Statistical Methods . 6th ed. Ames, Iowa: Iowa StateUniversity Press.Wiley, D.E. and Wiley, J. A.1970 "The estimation of measurement error in panel data."American Sociological Review 35:112-117.U' i no V , B.J.1962 Statistical Principles In Experimental Design . New York:McGraw-Hill.«# i

A note on testing for constant reliability in repeated measurement studies

http://dspace.mit.edu/bitstream/1721.1/48599/1/noteontestingfor00silk.pdf

A note on testing for constant reliability in repeated measurement studies

Abstract

Similar works

Full text

Available Versions

DSpace@MIT