What Do We Measure? [electronic resource] : Methodological versus Institutional Validity in Student Surveys / Jeffrey Alan Johnson.
This paper examines the tension in the process of designing student surveys between the methodological requirements of good survey design and the institutional needs for survey data. Building on the commonly used argumentative approach to construct validity, I build an interpretive argument for stud...
Saved in:
| Online Access: |
Full Text (via ERIC) |
|---|---|
| Main Author: | |
| Format: | Electronic eBook |
| Language: | English |
| Published: |
[S.l.] :
Distributed by ERIC Clearinghouse,
2011.
|
| Subjects: |
| Summary: | This paper examines the tension in the process of designing student surveys between the methodological requirements of good survey design and the institutional needs for survey data. Building on the commonly used argumentative approach to construct validity, I build an interpretive argument for student opinion surveys that allows assessment of the inferences and assumptions underlying the claim that an opinion survey item is a valid measure of a construct. I then evaluate the content validity of measures of program satisfaction and academic growth in Utah Valley University's 2010 Graduated Alumni Survey and 2009 Graduating Student Survey, surveys designed to maximize conformity with existing and legacy institutional priorities and demands. Using analytical assessment and empirical tests including response change across surveys, inter-item correlation, and factor analysis I show that UVU's surveys--and by implication, all surveys developed with primarily institutional validity in mind--are subject to grave challenges to the construct validity. I conclude by suggesting that effective operationalization of institutional priorities can bring together construct and institutional validity. AIR Forum Powerpoint Presentation is appended. (Contains 9 tables and 4 footnotes.) |
|---|---|
| Item Description: | Availability: Association for Institutional Research. 1435 East Piedmont Drive Suite 211, Tallahassee, FL 32308. Tel: 850-385-4155; Fax: 850-383-5180; e-mail: air@airweb.org; Web site: http://www.airweb.org. Abstractor: As Provided. Educational level discussed: Higher Education. Educational level discussed: Postsecondary Education. |
| Physical Description: | 51 p. |
| Type of Computer File or Data Note: | Text (Reports, Evaluative) Text (Speeches/Meeting Papers) |
| Preferred Citation of Described Materials Note: | Association for Institutional Research, Paper presented at the Annual Forum of the Association for Institutional Research (51st, Toronto, Ontario, May 21-25, 2011). |