NAEP staff monitors the inter-rater reliability throughout scoring and tracks
the percent exact agreement between multiple scorers.
Not exact matches
School officials say the IDA did not give them the
exact amount they would receive, so they estimated it at 50
percent based on previous
agreements.
Studies of reliability indicated interrater correlations ranging from.46 to.63 (45
percent agreement based on
exact match) depending on how the scores were aggregated (e.g., within or across scoring dimensions; by individual piece or across sections of the portfolio).