# Measurement uncertainty

The poverty estimates in the Poverty and Income Inequality in Scotland report are based on the Family Resources Survey (FRS). This is a sample survey, and therefore there is some degree of uncertainty around the estimates produced. For example, when it is reported that 19% of individuals are living in relative poverty after housing costs, then this should be understood not as an exact figure but as a best estimate within a range.

## Uncertainty

Two different random samples from one population give slightly different results. These results are different again from the results that would be obtained if the whole population was surveyed. The level of uncertainty around a survey estimate can be calculated and is commonly referred to as sampling error.

We calculate this by exploring how an estimate would change if we were to draw many survey samples for the same time period instead of just one. This allows us to define a range around the estimate (known as a “confidence interval”) and to state how likely it is that the real value that the survey is trying to measure lies within that range. Confidence intervals are typically set up so that we can be 95% sure that the true value lies within the range. In this case, we call it a “95% confidence interval”.

Confidence intervals provide a guide to how robust the estimates are. Tables 1 to 4 below provide confidence limits around the key poverty estimates.

For instance, Table 1 shows that the best estimate for the number of individuals in relative poverty after housing costs in 2017-20 was 19.35%, with a lower confidence limit of 17.42% and an upper confidence limit of 21.33%. This means that we can be 95% confident that the percentage of individuals in relative poverty lies between 17.42% and 21.33%. Similarly, the lower confidence limit for the number of people in relative poverty was 930,000, and the upper confidence limit was 1,140,000. So we can be 95% confident that the true number lies between those two figures.

## Change over time

When the confidence interval is wider than the magnitude of change between years, caution should be exercised when making year on year comparisons.

When calculating the difference in poverty rates between two years, the same methodology can be used to calculate a 95% confidence interval for this change. So if the range of likely values for the year on year change is either entirely greater or less than zero, that is that confidence interval does not contain zero, then the change in the latest year is 95% certain to be greater or less than zero. This is the approach used in the publication of poverty statistics to determine if a change is statistically significant.

None of the recent changes of the headline poverty estimates included in the tables below were statistically significant.

## Bootstrapping

The methodology used to calculate confidence intervals is called bootstrapping.

In the bootstrap, multiple new samples (resamples) of the dataset are created, with some samples containing multiple copies of one case with no copies of other cases. Exploring how an estimate would change if we were to draw many survey samples for the same time period instead of just one sample allows us to generate confidence intervals around the estimate.

The bootstrapping method used was improved for 2015/16. Resamples are now created by simulating stratified cluster sampling – the method used to draw the original FRS sample – and creating a unique set of grossing factors for each resample.

In the past, multiple samples were created using a simpler technique of creating simple random samples and re-using the original HBAI grossing factors.

More information on this change can be found in the DWP statistical notice (pdf). The new method widens confidence intervals for most estimates making statistically significant results less likely than before.

## Rounding

In the tables below, proportions are rounded to one decimal point in order to demonstrate the range within which the central estimates sit. In the reference tables (see Data), proportions are available with five decimal points. Please note that using decimal points when talking about poverty rates may imply accuracy that the data cannot provide. To avoid this, we refer to full percentages in the commentary for the report, such as 19%.

Population estimates are rounded to the nearest 10,000 individuals, the same as in the reference tables and in the commentary of the report.

## Describing the data

We advise to use full percentages when talking about poverty rates, and to mention the level of uncertainty around them. For example, 19% of people (+/- 2 percentage points) were in relative poverty after housing costs in 2017-20, or 24% of children (+/- 4 percentage points) were in relative poverty after housing costs in 2017-20.

## Tables

Group | Proportion | Number |
---|---|---|

People | 17.8% (15 - 20.2%) | 960,000 (800,000 - 1,080,000) |

Children | 21% (16.1 - 25.1%) | 210,000 (160,000 - 250,000) |

Working-age adults | 18.1% (14.9 - 20.8%) | 610,000 (500,000 - 700,000) |

Pensioners | 13.7% (10.7 - 16.4%) | 140,000 (110,000 - 160,000) |

Group | Proportion | Number |
---|---|---|

People | 16% (13.3 - 18.1%) | 860,000 (710,000 - 970,000) |

Children | 17.6% (13.1 - 21.6%) | 180,000 (130,000 - 210,000) |

Working-age adults | 15.5% (12.2 - 18.2%) | 520,000 (410,000 - 610,000) |

Pensioners | 15.8% (12.5 - 18.4%) | 160,000 (120,000 - 180,000) |

Group | Proportion | Number |
---|---|---|

People | 15.3% (12.9 - 17.6%) | 820,000 (690,000 - 940,000) |

Children | 18.2% (13.9 - 22.2%) | 180,000 (140,000 - 220,000) |

Working-age adults | 15.9% (12.8 - 18.5%) | 540,000 (430,000 - 620,000) |

Pensioners | 10.6% (8.1 - 13.1%) | 110,000 (80,000 - 130,000) |

Group | Proportion | Number |
---|---|---|

People | 13.3% (10.7 - 15.4%) | 710,000 (580,000 - 820,000) |

Children | 14.3% (9.9 - 17.8%) | 140,000 (100,000 - 180,000) |

Working-age adults | 13% (10 - 15.6%) | 440,000 (340,000 - 530,000) |

Pensioners | 13.1% (10.1 - 15.5%) | 130,000 (100,000 - 150,000) |

Group | Proportion | Number |
---|---|---|

Children | 6.9% (3.9 - 8.7%) | 30,000 (20,000 - 40,000) |

An after-housing-costs table for children in combined low income and material deprivation is not available. Confidence intervals are expected to be similar to the before-housing-costs measure.

Group | Proportion | Number |
---|---|---|

Pensioners | 4.6% (3.2 - 6%) | 50,000 (30,000 - 60,000) |