Reliability of Holistic Scoring for the 1985 MCAT Essays [microform] / Karen J. Mitchell and Judith A. Anderson.
A pilot essay was included in the 1985 Spring and Fall administrations of the Medical College Admission Test. A sample of 320 of the essays written by Fall examinees who had expressed an interest in allopathic medicine was used to calculate interrater reliability estimates. Sixteen of 20 readers who...
Saved in:
| Online Access: |
Request ERIC Document |
|---|---|
| Main Author: | |
| Other Authors: | |
| Format: | Microfilm Book |
| Language: | English |
| Published: |
[Place of publication not identified] :
Distributed by ERIC Clearinghouse,
1986.
|
| Subjects: |
| Summary: | A pilot essay was included in the 1985 Spring and Fall administrations of the Medical College Admission Test. A sample of 320 of the essays written by Fall examinees who had expressed an interest in allopathic medicine was used to calculate interrater reliability estimates. Sixteen of 20 readers who had been trained by White's suggestions for holistic scoring were included in the reliability study. Each reader rated two batches of 20 essays on the first day and again on the second. Analyses of variance were computed for (1) day; (2) batch; (3) reader; (4) subjects nested within batch; (5) interaction of day with batch; (6) interaction of day with reader; (7) interaction of reader with batch; and (8) interaction of subjects within batch with day. The interrater reliability index was estimated at .84, indicating that 71% of the variation in scores was due to level of differences between the essays themselves. The remainder was due to batch, day, day by reader, day by batch, batch by reader, and reader. When extrapolated to the operational setting, the index was .72. Methods of limiting the effects of these factors are proposed. (MGD) |
|---|---|
| Item Description: | ERIC Note: Paper presented at the Annual Meeting of the American Psychological Association (Washington, DC, August 22-26, 1986). ERIC Document Number: ED285913. |
| Physical Description: | 10 pages. |
| Reproduction Note: | Microfiche. |
| Action Note: | committed to retain |