On assuring valid measures for theoretical models using survey data
Introduction
Based on the articles in our major journals, marketers generally agree that specifying and testing theoretical models using unobserved variables with multiple item measures of these unobserved variables and survey data (UV–SD model tests) involve six steps: (i) defining constructs, (ii) stating relationships among these constructs, (iii) developing measures of the constructs, (iv) gathering data, (v) validating the measures, and (vi) validating the model (i.e., testing the stated relationships among the constructs). However, based on the articles reviewed,1 there appears to be considerable latitude, and confusion in some cases, regarding how these six steps should be carried out for UV–SD model tests in marketing.
For example, in response to calls for increased psychometric attention to measures in theoretical model tests, reliability and validity now receive more attention in UV–SD model tests (e.g., Churchill, 1979, Churchill and Peter, 1984, Cote and Buckley, 1987, Cote and Buckley, 1988, Heeler and Ray, 1972, Peter, 1979, Peter, 1981, Peter and Churchill, 1986). However, there were significant differences in what constitutes an adequate demonstration of measure reliability and validity in the articles reviewed. For example, in some articles Steps (v) (measure validation) and (vi) (model validation) involved separate data sets. In other articles a single data set was used to validate both the measures and the model. Further, in some articles the reliabilities of measures used in previous studies were reassessed. In other articles reliabilities were assumed to be constants that, once assessed, should be invariant in subsequent studies. Similarly, in some articles many facets of validity for each measure were examined, even for previously used measures. In other articles few facets of measure validity were examined, and validities were assumed to be constants (i.e., once judged acceptably valid a measure was acceptably valid in subsequent studies).
Thus an objective of this research is to selectively identify areas for continuous improvement in Step (v), measure validation. The research provides a selective review, albeit qualitative, of the UV–SD model testing practices of marketers in that step and the other steps as they pertain to Step (v). It also provides selective discussions of errors of omission and commission in measure validation. For example, this research discusses the implications of reliability and facets of validity as sampling statistics with unknown sampling distributions. It suggests techniques such as easily executed experiments that could be used to pretest measures, and bootstrapping for reliabilities and facets of validity. The research also suggests an estimator of average variance extracted (AVE) (Fornell and Larker, 1981) that does not rely on structural equation analysis (e.g., LISREL, EQS, AMOS, etc.). In addition, it suggests an alternative to omitting items in structural equation analysis to improve model-to-data fit that should be especially useful for older measures established before structural equation analysis became popular.
Section snippets
Measure validation
Step (v), measure validation or demonstrating the adequacy of the study measures, appeared to be the least consistent of the six steps above (see Peter and Churchill, 1986 for similar findings). Perhaps this was because there are several issues that should be addressed in validating measures. Measures should be shown to be unidimensional (having one underlying construct), consistent (fitting the model in structural equation analysis), reliable (comparatively free of measurement error), and
Unidimensionality
Assessing reliability usually assumes unidimensional measures Bollen, 1989, Gerbing and Anderson, 1988, Hunter and Gerbing, 1982. However, coefficient alpha, the customary index of reliability in marketing, underestimates the reliability of a multidimensional measure (Novick and Lewis, 1967). Thus, unidimensionality is actually required for the effective use of coefficient alpha (Heise and Bohrnstedt, 1970—see Hunter and Gerbing, 1982) (other indexes of reliability such as coefficient omega
Consistency
Many criteria for demonstrating unidimensionality have been proposed (see Hattie, 1985). Perhaps in response to calls for more work in this area (e.g., Lord, 1980), Anderson and Gerbing (1982) proposed operationalizing unidimensionality using the structural equation analysis notions of internal and external consistency (also see Kenny, 1979, Lord and Novick, 1968, McDonald, 1981) (however, see Kumar and Dillon, 1987a, Kumar and Dillon, 1987b for an alternative view).
Consistency has been defined
Procedures for attaining unidimensionality and consistency
Procedures for attaining unidimensionality using exploratory (common) factor analysis are well known. However, procedures for obtaining consistent/unidimensional measures are less well documented. Procedures using ordered similarity coefficients are suggested in Anderson and Gerbing (1982, p. 454), and Gerbing and Anderson (1988). The ordered similarity coefficients help identify inconsistent items. Alternatively, consistency/unidimensionality for constructs specified unidimensionally (i.e.,
Comments on unidimensionality and consistency
Unidimensionality in the exploratory common factor analytic sense is required for coefficient alpha, and consistency/unidimensionality is required for structural equation analysis. Further, it is well known that the reliability of a measure is necessary for its validity. Thus, there is a sequence of steps in validating a measure: establish its consistency/unidimensionality for structural equation analysis, or establish its unidimensionality using maximum likelihood exploratory common factor
Reliability
Unfortunately, the term consistency has been used in connection with reliability (see, for example, DeVellis, 1991, p. 25). In fact there has been considerable confusion over reliability and consistency (see Hattie, 1985). After discussing reliability I will discuss the distinctness of reliability from consistency as Anderson and Gerbing (1982) have defined it.
Measure reliability was usually reported in the articles reviewed. The reliability of a measure is suggested by agreement of two efforts
Validity
There was considerable variation in the demonstrations of validity among the articles reviewed. This may be because methods authors do not all agree on what constitutes an adequate demonstration of validity. Item validity is how well an item measures what it should, and a valid measure consists of valid items. Validity is important because theoretical constructs are not observable, and relationships among unobservable constructs are tested indirectly via observed variables (Jöreskog, 1993; see
Comments on measure validation
New measures frequently seemed to be underdeveloped in the articles reviewed. For example, new measure development details were not always reported. Thus it appeared that recommended procedures such as item judging, focus groups, etc. (Churchill, 1979, see Calder, 1977) were not always used to develop new measures. Several data sets should also be used to gauge the reliability and facets of the validity of measures (Campbell and Fiske, 1959; see Churchill, 1979). However, measure validation
In conclusion
Based on the articles reviewed it was difficult to escape the conclusion that reliability and validity in UV–SD model tests could be improved by simply following well-known procedures for this purpose (e.g., Churchill, 1979). For example, an examination of the equations for reliability and convergent and discriminant validity suggest that difficulties with reliability or these facets of validity could be viewed as a result of insufficient error-free variance. Thus procedures for improving the
References (81)
- et al.
A large-scale second-order structural equation model of the influence of management participation on organizational planning benefits
J Manage
(1994) - et al.
Unobservable variables in structural equation models with an application in industrial selling
J Mark Res
(1979) - et al.
Some methods for respecifying measurement models to obtain unidimensional construct measurement
J Mark Res
(1982) - et al.
The effect of sampling error on convergence, improper solutions, and goodness of fit indices for maximum likelihood confirmatory factor analysis
Psychometrika
(1984) - et al.
Structural equation modeling in practice: a review and recommended two-step approach
Psychol Bull
(1988) - et al.
On the assessment of unidimensional measurement: internal and external consistency, and overall consistency criteria
J Mark Res
(1987) Causal models in marketing
(1980)An examination of the validity of two models of attitude
Multivariate Behav Res
(1981)A prospectus for theory construction in marketing
J Mark
(1984)- et al.
The evaluation of structural equation models and hypothesis testing
A general approach to representing multifaceted personality constructs: application to self esteem
Struct Equation Model
Representing and testing organizational theories: a holistic construal
Adm Sci Q
EQS structural equations program manual
Comparative fit indexes in structural models
Psychol Bull
Robustness in regression analysis
Structural equations with latent variables
Testing structural equation models
Alternative ways of assessing model fit
Confirmatory factor-analysis structures and the theory construction process
Sociol Methods Res
Interpretational confounding of unobserved variables in structural equation models
Sociol Methods Res
Analysis of multiplicative combination rules when the causal variables are measured with error
Psychol Bull
Focus groups and the nature of qualitative marketing research
J Mark Res
Recommendations for APA test standards regarding construct trait and discriminant validity
Am Psychol
Convergent and discriminant validation by the multitrait–multimethod matrix
Psychol Bull
Personality and mood by questionnaire
The scientific use of factor analysis in behavioral and life sciences
A Paradigm for developing better measures of marketing constructs
J Mark Res
Research design effects on the reliability of rating scales: a meta analysis
J Mark Res
Applied multiple regression/correlation analyses for the behavioral sciences
Estimating trait, method and error variance: generalizing across seventy construct validation studies
J Mark Res
Measurement error and theory testing in consumer research: an illustration of the importance of construct validation
J Consum Res
Coefficient alpha and the internal structure of tests
Psychometrika
Dependability of behavioral measurements: theory of generalizability for scores and profiles
Scale development: theory and applications
Multivariate analysis: methods and applications
Marketing research in a marketing environment
Output sector munificence effects on the internal political economy of marketing channels
J Mark Res
Nonparametric estimates of standard error: the jackknife, the bootstrap, and other resampling methods
Biometrika
Two structural equation models: LISREL and PLS applied to exit-voice theory
J Mark Res
Evaluating structural equation models with unobservable variables and measurement error
J Mark Res
Cited by (334)
Why do people use Metaverse? A uses and gratification theory perspective
2024, Telematics and InformaticsInformation security policies compliance in a global setting: An employee's perspective
2023, Computers and SecurityLearning from improvisation in New Ventures
2023, Journal of Innovation and KnowledgeCoping with workplace sexual harassment: Social media as an empowered outcome
2022, Journal of Business Research