The alternative and, in my view, more plausible hypothesis is that the measures are misleading due to
reference bias stemming from differences in school climate between district and charter schools.
Last year Duckworth and her colleagues found evidence of this kind
of reference bias in a study of more than 1,300 Boston eighth graders.
To illustrate the potential for
reference bias in self - reported measures of non-cognitive skills, I draw on cross-sectional data from a sample of Boston students discussed in detail in a recent working paper.
My own research has suggested the potential importance of
reference bias due to differences in school climate, leading me to caution in this series against proposals to incorporate survey - based measures of non-cognitive skills into high - stakes accountability systems.
Between reference bias and the lack of more nuanced and rigorously evaluated assessments, Duckworth and co-author David Yeager don't see grit finding its way into the world of high - stakes school accountability.
If the apparent negative effects of attending a «no excuses» charter school on conscientiousness, self - control, and grit do in fact
reflect reference bias, then what our data show is that these schools influence the standards to which students hold themselves when evaluating their own non-cognitive skills.
These new systems overcome the faking, subjectivity, and
reference bias problems that plague «first generation» measurement methods.
Nobody wants to see measurements that are flawed
by reference bias inaccuracies, create perverse incentives for cheating, or motivate extrinsically to their detriment.
This parallel evidence from research in similar settings confirms that
reference bias stemming from differences in school climate is the most likely explanation for these paradoxical findings.
Between reference bias and the lack of more nuanced and rigorously evaluated assessments, Duckworth and co-author David Yeager don't see grit finding its way into the world of high - stakes school accountability.
Key concerns include the possibility of misleading information due to
reference bias in students» self - reports and that students may simply inflate their self - ratings to improve their school's standing once stakes have been attached.
The reference bias is present not only in expression profiling, but also in SNP calling.
Of course, these data also come from self - report surveys and may themselves be subject to
reference bias.
Other recent studies of «no excuses» charter schools reinforce the plausibility of
the reference bias hypothesis.
Possibly more troublesome is
reference bias, which occurs when survey responses are influenced by differing standards of comparison.
As a result, differences in self - reports may reflect variation in normative expectations rather than true differences in skills, a phenomenon known as «
reference bias.»
It does, however, provide some preliminary evidence that the form of
reference bias that would be most problematic in the context of a school accountability system may not be an important phenomenon in the CORE districts as a whole.
[iii] To the extent that students attending schools with more demanding expectations for student behavior hold themselves to a higher standard when completing questionnaires,
reference bias could make comparisons of their responses across schools misleading.
To be sure, this analysis does not rule out the possibility that
reference bias may lead to misleading inferences about specific schools with particularly distinctive environments.
The thing I like about this measurement approach to grit is that it is, in theory, less fakeable than a self - report questionnaire and also not as susceptible to
reference bias, because you do not have to ask the applicant to rate themselves.