Understanding Critical Difference

When exploring trends in children’s development, the broad question that we want to answer is: “Are our kindergarten-aged children doing better, worse, or about the same as in the past?”  HELP has developed a method that communities and stakeholder groups can use to make informed judgments about change over time in EDI scores. The method that HELP has used to examine ‘change over time’ is described in the literature as Critical Difference.

Please find below information about Critical Difference, including the definition and how to calculate it. In addition, HELP’s Understanding Critical Difference in EDI Results Research Brief provides important information about Critical Difference.

Critical Difference: A Definition

A Critical Difference is the amount of difference in a neighbourhood’s EDI vulnerability rate that is large enough to be considered significant for a particular number of children. Statistically significant change most likely represents a real shift in a neighborhood’s EDI vulnerability rate rather than a change influenced by measurement issues.

A Critical Difference can be used in two ways.
1. A Critical Difference can be used to calculate whether a change in the EDI vulnerability rate for one neighbourhood between two time periods is meaningful/significant.
2. A Critical Difference can be used to calculate whether the difference in EDI vulnerability rates between two neighborhoods during the same time period is meaningful/significant.

Critical Difference Table 

HELP has developed a Critical Differences Table to be used for EDI vulnerability results in B.C. The table was prepared as a result of Critical Difference calculations for children included in EDI results. Its purpose is to help communities better understand change in EDI vulnerability rates. To make meaning of Critical Difference applied to EDI results, communities need to place their analysis in the context of other information about changes in the community.

The Critical Differences Table shown below applies only to vulnerability on one or more EDI scales. If you are interested in Critical Differences for one particular EDI scale, please contact HELP for assistance. These tables will be available on the HELP website shortly.

Group SizeCritical Difference
(percentage points)
45-55 11
56-69 10
70-85 9
86-108 8
109-138 7
139-200 6
201-360 5
361-535 4
536 plus 3

How to Calculate Critical Difference: Some Examples

Worksheets

HELP has created two worksheets designed to help you learn how to calculate Critical Difference:

Comparing Neighbourhood A Across Time
Comparing Neighbourhoods A and B in one Wave
 
Should you have any questions about how to calculate Critical Difference, please contact (insert). This information is included text and graphic form in HELP’s Understanding Critical Difference in EDI Results research brief.

Some Key Concepts and Considerations for Critical Difference

Critical Difference applies only to Vulnerability on One or more EDI Scales.
The Critical Differences Table is not meant to compare change over time in vulnerability on one particular scale (e.g., physical, social, emotional, language and cognitive, or communication).  It applies only to Vulnerability on One or More EDI Scales. Researchers at HELP are working on Critical Differences for one particular EDI Scale. This information will be posted to the HELP website soon. In the mean time, please contact HELP if you are interested in learning more.

Critical Difference and Small Populations
Critical Difference can be calculated for populations smaller than 45 children. However, HELP suppresses data for children where the population falls below 45 for privacy reasons. If you need to address issues of change over time and your population falls below this threshold, please contact HELP for assistance.

Uncertainty
Uncertainty is a key concept in Critical Difference. HELP’s Understanding Critical Difference in EDI Results Research Brief provides detailed and important information about uncertainty in relation to Critical Difference. Please review this brief for more information.

Critical Difference is not enough when exploring change over time.
Critical Difference is an important statistical method for understanding change over time and between communities, but to make the most meaning from the results they need to be placed in the context of social and economic changes in the community, as well as other sources of local and regional data about children, programs, and services.

Frequently Asked Questions

Is it OK to use the Critical Differences Table for vulnerability on one particular scale (e.g., physical, social, emotional, language and cognitive, or communication) instead of vulnerability on one or more scales?
No. The Critical Differences Table shown above applies only to vulnerability on one or more scales. If you are interested in Critical Differences for one particular scale, please contact HELP for assistance. These tables will be available on the HELP website shortly.

Can I use the Critical Differences Table to compare a neighbourhood over three or more points in time, that is, to do multiple comparisons?
Not yet. The table is not designed for multiple comparisons. Choose two time points representing a “beginning” and an “end” and stick with that. We at HELP are working on a version of Critical Differences that can be applied to multiple time points, but the results of that work are not yet available.

What do I do if the number of neighbourhood children in a wave is small?
Understanding change overtime in small populations is more difficult because the vulnerability rate in small populations may be too unreliable to be interpreted. However, there are strategies that can increase a population’s size and thus increase the reliability. For example, it is possible to amalgamate populations across neighbourhoods or over two periods of time. HELP staff can assist you if this situation applies to your community.

In our neighbourhood, the number of children in Wave 2 is much lower than in Wave 4. Does this affect how I would use the Critical Differences Table? For instance, should I divide the group size in half for Wave 4?
Always use the Critical Difference Table as described in the examples in this brief. It works just as well whether the group sizes are similar over time or different over time, even for the same neighbourhood. The key to the uncertainty is the number of children in the group – the more children, the less uncertainty. When a neighbourhood is measured twice in a wave, the end result is a more reliable estimate of the vulnerability rate.

Can I apply the Critical Differences Table to change over time for the whole province?
Yes. For any group size that is 536 or greater, a difference of 3 percentage points over time can be considered to be meaningful.