Understanding Critical Difference

When exploring trends in children’s development, the broad question that we want to answer is: “Are our kindergarten-aged children doing better, worse, or about the same as in the past?”  HELP has developed a method that communities and stakeholder groups can use to make informed judgements about change over time in EDI scores. The method that HELP has used to examine ‘change over time’ is described in the literature as Critical Difference.

On this page you will find information about Critical Difference, including the definition and how to calculate it.

We have also prepared a research brief on the topic. 

Critical Difference: A Definition

A Critical Difference is the amount of change over time in a neighbourhood’s EDI vulnerability rate that is large enough to be considered meaningful. By meaningful, we mean worthy of further discussion and exploration. We use statistical significance as a minimum requirement for identifying change as meaningful, as this reflects a high likelihood of a real shift in the vulnerability rate rather than a change due to measurement or sampling issues.

Besides using Critical Difference to assess change over time for one neighbourhood, it can also be used to assess whether two neighbourhoods have meaningfully different EDI vulnerability rates for the same time period.

Measuring Uncertainty

When an EDI vulnerability rate is calculated for a neighbourhood, school district or some other geographic unit, there is always some uncertainty about what the true vulnerability rate actually is. There are two main sources of uncertainty, relating to issues of sampling and measurement.

Sampling

On average, EDI data are collected for about 85% of children in neighbourhoods around BC. In some neighbourhoods this sample might not be representative of the population as a whole. This is because, though the EDI is administered in most public schools, it has not universally been completed for private schools and on-reserve schools.

Measurement

Uncertainty related to measurement is common for all tools that measure complex constructs, such as health, well-being, or social status. Similarly, the EDI measures complex qualities such as social competence and emotional maturity. We accept that the EDI items we use to capture these domains cannot completely represent their actual complexity, resulting in some measurement error.

In the case of the EDI, there is a second measurement-related source of uncertainty: variability in how teachers interpret EDI items when rating their students. For example, teachers may differ in how lenient or severe they are in their EDI scoring, or may differ in how consistent they are in their scoring. Teacher-related uncertainty is minimized by providing training to all teachers before they use the EDI, as well as by providing an online knowledge base to assist them while they do their scoring. However, over the hundreds of kindergarten teachers who complete the EDI for each wave of provincial data collection, some between-teacher differences are bound to persist, resulting in measurement error.

Both of these sources of uncertainty in EDI vulnerability rates are affected by the number of children in the population (e.g.,neighbourhood, school district). For statistical reasons, uncertainty inevitably gets smaller as the population size gets larger. As seen in the power curve graph below, vulnerability rates for larger populations are more precise (i.e., critical difference values are lower) than for smaller populations.

How Critical Difference Is Derived

The analytic strategy used to develop critical difference equations is micro-simulation. The ways in which uncertainty is sensitive to different sources of measurement error was tested, such as neighbourhood size and teacher effects (including leniency and consistency of scoring). For each combination of neighbourhood size and teacher-related error, the uncertainty of the vulnerability rate was calculated. The critical difference equations and associated curves show how uncertainty changes with neighbourhood size, assuming a moderate level of teacher effects.

y = critical difference value
x = number of children in area of interest (or average number of children if the count is different between time periods or areas that are being compared)

ScaleEquation
Physical y=56.532x-0.469
Social y=34.702x-0.455
Emotional y= 46.657x-0.488
Language y=52.987x-0.521
Communication y=52.539x-0.487
One or More Scales y=73.941x-0.507

Power Curves

As the graph shows, vulnerability rates for large populations are more precise (critical difference values are lower) than for smaller populations. For all EDI scales, the curves end when the critical difference value reaches two percent; this is the limit placed on critical difference, regardless of group size.

How to Calculate Critical Difference

Example 1.Comparing EDI scores across time in one neighbourhood

Neighbourhood ‘A’ has a vulnerability rate on ‘one or more scales’ of 34% in Wave 2, based on scores for 52 children. In Wave 4, the vulnerability rate drops to 26%, based on scores for 63 children.
Using the critical difference equation for ‘vulnerable on one or more scales’, or by using the critical difference calculator below, the critical difference value can be determined: the critical difference is 10 percentage points in the first time period, and 9 points in the second time period. The average critical difference is 9.5 percentage points. Since the average is larger than the observed drop of 8 percentage points (34% to 26%), the vulnerability rate has not changed enough to be considered a meaningful or significant difference.

Example 2. Comparing EDI scores across neighbourhoods.

Neighbourhood ‘A’ has a vulnerability rate on ‘one or more scales’ of 34% in Wave 2, based on scores for 52 children. Neighbourhood B has a vulnerability rate of 16% in Wave 2, based on scores for 310 children. The same critical difference equations apply as in Example 1. For Neighbourhood ‘A’, the critical difference is 10 percentage points. For Neighbourhood ‘B’, it is 4 percentage points. The average critical difference is 7 percentage points. Since this is smaller than the observed difference between the neighbourhoods of 18 percentage points, Neighbourhood ‘B’s lower vulnerability rate is considered to be significantly different than Neighbourhood ‘A’s. 

Critical Difference Calculator

To make calculating critical difference easier we have created the Critical Difference Calculator, a tool that will help with interpreting change over time and differences between EDI scores in the BC context. To use the calculator, simply enter the values for each place or time and click on the Calculate button. The result will calculate automatically.

We also have an interactive map of neighbourhood critical difference for each school district. This map is part of our Neighbourhood map sets. Please click on the neighbourhood of choice and scroll down to view the critical difference map.

EDI Scales
ONE OR MORE PHYSICAL SOCIAL EMOTIONAL LANGUAGE COMMUNICATION
EDI Data Values
Population 2
Critical Difference:-  
Change:-  
 

How Measurement Uncertainty is Managed

As HELP administers the EDI questionnaire each year, we work in constant collaboration with schools and teachers to facilitate the data gathering process and to ensure that we have high quality, reliable data on children. Annually, we invest in a standardized training package for kindergarten teachers. This package provides information on the background and use of the EDI, along with detailed instructions on EDI completion. We also have specific training content to address potential teacher bias. Our goal is to build as common an administration and interpretation of the EDI items as possible.

Frequently Asked Questions

Can I use the critical differences to compare a neighbourhood over three or more points in time, that is, to do multiple comparisons?
No. Critical difference calculations are not designed for multiple comparisons. Choose two time points representing a "beginning" and an "end". Researchers at HELP are working on a version of critical differences that can be applied to multiple time points, but the results are not yet available.

What do I do if the number of neighbourhood children in a wave is small?
Understanding change over time in small populations is more difficult because the vulnerability rate in small populations may be too unreliable to be interpreted. However, there are strategies that can increase a population’s size and thus increase the reliability. For example, it is possible to amalgamate populations across neighbourhoods or over two periods of time. HELP staff can assist you if this situation applies to your community.

In our neighbourhood, the number of children in Wave 2 is much lower than in Wave 4. Does this affect how I would use the Critical Differences Table? For instance, should I divide the group size in half for Wave 4?
No. The critical difference equations work just as well whether the group sizes are similar over time or different over time, even within the same neighbourhood. The key to the uncertainty is the number of children in the group – the more children, the less uncertainty. When a neighbourhood is measured twice in a wave, the end result is a more reliable estimate of the vulnerability rate.

Can I apply the Critical Differences Table to change over time for the whole province?
Yes. For any large region (such as the province) whose group size is beyond the point where the critical difference value reaches two, a 2% change is considered to be meaningful. Having a cutoff is necessary to avoid situations where a very small change in vulnerability over time results in a critical difference, due only to very large group sizes.

Does critical difference also apply to EDI subscale scores?
Yes, the same simulation methodology has been used to assess change over time for EDI subscales. The short-term (Wave 5 to Wave 6) and long-term (Wave 2 to Wave 6) change over time results have been included in the EDI Subscales Community Profiles (released September 2017). A Critical Difference calculator for subscales is being developed and will be available in 2018.