The most widely used techniques for assessing language vitality (the Graded Intergenerational Disruption Scale, the UNESCO system, and the EL-CAT Language Endangerment Index) all derive their endangerment estimates from surveys that elicit self-reports of language proficiency and use. Unfortunately, speaker self-assessments can be distorted by a variety of factors, including confusion over what counts as proficiency in a language and a subconscious desire to understate or overstate one’s ability to use a community language that is cherished by some and criticized by others. We report here on a study that has the joint objective of (i) establishing that subjective self-assessments may often be inaccurate, and (ii) proposing a technique to reduce the effects of this problem.
Participants: 61 residents of Korea’s Jeju Island, home to the critically endangered language Jejueo. The participants ranged in age from 20 – 29; all had Korean as their dominant language, but were proficient to varying degrees in Jejueo.
Materials: Ten Jejueo sentences, constructed from items in the Swadesh list, were recorded by one male and one female speaker of Jejueo.
Method: Participants heard each Jejueo sentence twice, first in the version produced by the male speaker, then in the version produced by the female speaker. In order to ensure attention to the meaning of the Jejueo test items, participants were asked to paraphrase each sentence in written Korean right after hearing it.
Before beginning the task, participants were asked to estimate their comprehension ability in Jejueo on a scale of 1 (low) to 5 (high). At the conclusion of the task, they were asked to estimate their fluency a second time on the same scale.
Results:
Mean Self-Assessment of Proficiency Before and After Exposure to Jejueo Test Items
Before After Mean difference
3.13 2.06 1.13
As these results show, participants produced noticeably lower self-assessments after being called upon to interpret actual Jejueo sentences. This pattern was observed in 38 of 61 participants; only three participants exhibited the reverse pattern of initially underestimating their abilities.
These results suggest that traditional survey data, with its emphasis on self-assessment of language ability, may significantly overestimate proficiency levels, thereby providing an inaccurate and overly optimistic picture of language vitality. However, on a positive note, our findings suggest that this danger can be mitigated by embedding self-assessment surveys within tasks that involve exposure to the language being investigated
