Understanding Critical Difference
When exploring trends in children’s development, the broad question that we want to answer is: “Are our kindergarten-aged children doing better, worse, or about the same as in the past?” HELP has developed a method that communities and stakeholder groups can use to make informed judgements about change over time in EDI scores. The method that HELP has used to examine ‘change over time’ is described in the literature as Critical Difference.
On this page you will find information about Critical Difference, including the definition and how to calculate it.
We have also prepared a research brief on the topic.
Critical Difference: A Definition
A Critical Difference is the amount of difference in a neighbourhood’s EDI vulnerability rate that is large enough to be considered significant for a particular number of children. Statistically significant change most likely represents a real shift in a neighborhood’s EDI vulnerability rate rather than a change influenced by measurement issues.
A Critical Difference can be used in two ways:
- To calculate whether a change in the EDI vulnerability rate for one neighbourhood between two time periods is significant.
- To calculate whether the difference in EDI vulnerability rates between two neighborhoods during the same time period is significant.
When an EDI vulnerability rate is calculated for a neighbourhood, school district or some other geographic unit, there is always some uncertainty about what the true vulnerability rate actually is.
Uncertainty can be related to the sampling of children for whom EDI data are collected in a certain area. On average, EDI data are collected for about 85% of children in neighbourhoods around BC. In some neighbourhoods this sample might not be representative of the population as a whole. This is because, though the EDI is administered in most public schools, it has not universally been completed for private schools and on-reserve schools.
There is also a degree of uncertainty in EDI vulnerability rates due to the measurement process. This is common to all tools that measure complex constructs, such as health, well-being, or social status. In the case of the EDI, uncertainty can be associated with two main sources:
- Kindergarten teachers rate their perception of the developmental status of children in their class using the EDI. Uncertainty can result as teachers may rate the items differently.
- Measurement tools, such as rulers, are very precise for measuring simple qualities such as length or height. In contrast, the EDI measures complex qualities such as social competence or emotional maturity. The EDI items are proxies for these complex characteristics and cannot be precisely captured in their totality.
Uncertainty in EDI vulnerability rates is also related to population size. Uncertainty gets smaller as the number of children in an area gets larger. As seen in the power curve graph below, vulnerability rates for large populations are more precise (critical difference values are lower) than for smaller populations.
How Critical Difference Is Derived
The analytic strategy used to develop critical difference equations is micro-simulation. The ways in which uncertainty is sensitive to different sources of measurement error was tested, such as neighbourhood size and teacher effects (including leniency and consistency of scoring). For each combination of neighbourhood size and teacher-related error, the uncertainty of the vulnerability rate was calculated. The critical difference equations and associated curves show how uncertainty changes with neighbourhood size, assuming a moderate level of teacher effects.
y = critical difference value
x = number of children in area of interest (or average number of children if the count is different between time periods or areas that are being compared)
|One or More Scales||y=73.941x-0.507|
As the graph shows, vulnerability rates for large populations are more precise (critical difference values are lower) than for smaller populations. For all EDI scales, the curves end when the critical difference value reaches two percent; this is the limit placed on critical difference, regardless of group size.
How to Calculate Critical Difference
Example 1.Comparing EDI scores across time in one neighbourhood
Neighbourhood ‘A’ has a vulnerability rate on ‘one or more scales’ of 34% in Wave 2, based on scores for 52 children. In Wave 4, the vulnerability rate drops to 26%, based on scores for 63 children.
Using the critical difference equation for ‘vulnerable on one or more scales’, or by using the critical difference calculator below, the critical difference value can be determined: the critical difference is 10 percentage points in the first time period, and 9 points in the second time period. The average critical difference is 9.5 percentage points. Since the average is larger than the observed drop of 8 percentage points (34% to 26%), the vulnerability rate has not changed enough to be considered a meaningful or significant difference.
Example 2. Comparing EDI scores across neighbourhoods.
Neighbourhood ‘A’ has a vulnerability rate on ‘one or more scales’ of 34% in Wave 2, based on scores for 52 children. Neighbourhood B has a vulnerability rate of 16% in Wave 2, based on scores for 310 children. The same critical difference equations apply as in Example 1. For Neighbourhood ‘A’, the critical difference is 10 percentage points. For Neighbourhood ‘B’, it is 4 percentage points. The average critical difference is 7 percentage points. Since this is smaller than the observed difference between the neighbourhoods of 18 percentage points, Neighbourhood ‘B’s lower vulnerability rate is considered to be significantly different than Neighbourhood ‘A’s.
To make calculating critical difference easier we have created the Critical Difference Calculator, a tool that will help with interpreting change over time and differences between EDI scores in the BC context. To use the calculator, simply enter the values for each place or time and click on the Calculate button. The result will calculate automatically.
We also have an interactive map of neighbourhood critical difference for each school district. This map is part of our Neighbourhood map sets. Please click on the neighbourhood of choice and scroll down to view the critical difference map.
How Measurement Uncertainty is Managed
As HELP administers the EDI questionnaire each year, we work in constant collaboration with schools and teachers to facilitate the data gathering process and to ensure that we have high quality, reliable data on children. Annually, we invest in a standardized training package for kindergarten teachers. This package provides information on the background and use of the EDI, along with detailed instructions on EDI completion. We also have specific training content to address potential teacher bias. Our goal is to build as common an administration and interpretation of the EDI items as possible.
Frequently Asked Questions
Can I use the critical differences to compare a neighbourhood over three or more points in time, that is, to do multiple comparisons?
No. Critical difference calculations are not designed for multiple comparisons. Choose two time points representing a "beginning" and an "end". Researchers at HELP are working on a version of critical differences that can be applied to multiple time points, but the results are not yet available.
What do I do if the number of neighbourhood children in a wave is small?
Understanding change over time in small populations is more difficult because the vulnerability rate in small populations may be too unreliable to be interpreted. However, there are strategies that can increase a population’s size and thus increase the reliability. For example, it is possible to amalgamate populations across neighbourhoods or over two periods of time. HELP staff can assist you if this situation applies to your community.
In our neighbourhood, the number of children in Wave 2 is much lower than in Wave 4. Does this affect how I would use the Critical Differences Table? For instance, should I divide the group size in half for Wave 4?
No. The critical difference equations work just as well whether the group sizes are similar over time or different over time, even within the same neighbourhood. The key to the uncertainty is the number of children in the group – the more children, the less uncertainty. When a neighbourhood is measured twice in a wave, the end result is a more reliable estimate of the vulnerability rate.
Can I apply the Critical Differences Table to change over time for the whole province?
Yes. For any large region (such as the province) whose group size is beyond the point where the critical difference value reaches two, a 2% change is considered to be meaningful. Having a cutoff is necessary to avoid situations where a very small change in vulnerability over time results in a critical difference, due only to very large group sizes.