Measurement uncertainty
The poverty estimates in the Poverty and Income Inequality in Scotland report are based on the Family Resources Survey (FRS). This is a sample survey, and therefore there is some degree of uncertainty around the estimates produced. For example, when it is reported that 21% of individuals are living in relative poverty after housing costs, then this should be understood not as an exact figure but as a best estimate within a range.
Why estimates are uncertain
Two different random samples from one population give slightly different results. These results are different again from the results that would be obtained if the whole population was surveyed. The level of uncertainty around a survey estimate can be calculated and is commonly referred to as sampling error.
We calculate this by exploring how an estimate would change if we were to draw many survey samples for the same time period instead of just one. This allows us to define a range around the estimate (known as a “confidence interval”) and to state how likely it is that the real value that the survey is trying to measure lies within that range. Confidence intervals are typically set up so that we can be 95% sure that the true value lies within the range. In this case, we call it a “95% confidence interval”.
Confidence intervals provide a guide to how robust the estimates are. Tables 1 to 4 below provide 95% confidence limits around the key poverty estimates. For example, the proportion of individuals in relative poverty after housing costs in 2020-23 was 20.67%, with a lower confidence limit of 16.74% and an upper confidence limit of 24.63%. This means that we can be 95% confident that the percentage of individuals in relative poverty lies between 16.74% and 24.63%. Similarly, the lower confidence limit for the number of people in relative poverty was 900,000, and the upper confidence limit was 1,330,000. So we can be 95% confident that the true number lies between those two figures.
We measure uncertainty in two different ways. Bootstrapping is more accurate, but takes more time to calculate. We also calculate indicative confidence intervals which we make available in all poverty charts. These are less accurate, but they sufficiently reflect how sample size and variation affect measurement uncertainty.
The tables below contain bootstrapped confidence intervals, whereas the charts in the report contain indicative confidence intervals.
Bootstrapped intervals
The most accurate methodology used to calculate confidence intervals for poverty estimates is called bootstrapping.
In the bootstrap, multiple new samples (resamples) of the dataset are created, with some samples containing multiple copies of one case with no copies of other cases. Exploring how an estimate would change if we were to draw many survey samples for the same time period instead of just one sample allows us to generate confidence intervals around the estimate.
The bootstrapping method used was improved for 2015/16. Resamples are now created by simulating stratified cluster sampling – the method used to draw the original FRS sample – and creating a unique set of grossing factors for each resample. In the past, multiple samples were created using a simpler technique of creating simple random samples and re-using the original HBAI grossing factors.
More information on this change can be found in the DWP statistical notice (pdf). The new method widens confidence intervals for most estimates making statistically significant results less likely than before.
Indicative intervals
We also used simpler, and less accurate method for calculating 95% confidence intervals. These were used to show a measurement range in the poverty charts of the main report.
Using Scotland data for two single-year periods and four three-year periods, we calculated simplified standard errors for a range of poverty indicators and groups (all people, children, working-age adults, pensioners). Simplified standard errors were calculated using a variance estimation method that accounts for the complex sampling design. However, the simplified method treats poverty rates as linear estimates, in effect assuming that the poverty line is not subject to measurement uncertainty.
We compared the simplified standard errors to bootstrapped standard errors derived from DWP’s bootstrapped confidence intervals. Since the simplified standard errors assume that the poverty line is not subject to measurement uncertainty, they underestimate overall measurement uncertainty, resulting in too narrow confidence intervals. To widen the confidence intervals, we use a correction factor based on the average difference between simplified and bootstrapped standard errors. Note that this difference was broadly within a similar range for all groups (all people, children, working-age adults, pensioners). After applying the correction factor, we obtain indicative confidence intervals.
For three-year averages, we pooled the data to calculate standard errors. (Note that this is a different approach from calculating poverty rates for a three-year period – for that, we would take a simple mean of the three single-year estimates.) The pooling leads to a larger sample size, resulting in smaller standard errors.
As an example, for 2017-20, the correction factor ranged from 1.2 to 1.8, with a mean of 1.5.
Note that our approach made the following assumptions:
- We assume that an average correction factor can sufficiently correct for the fact that we consider the poverty line not subject to measurement uncertainty.
- The average correction factor is based on large groups (all people, all children, all working-age adults, all pensioners in Scotland), but we assume that it applies to further disaggregations equally.
- In successive years, samples are not independent (half of the primary sampling units get reused), but we treat them as if they were, and we assume that the impact on standard errors is negligible.
To mitigate the difference between (accurate) bootstrapped and (less accurate) simplified confidence intervals, we used correction factors of 1.5 for two-, three- and five-year averages, and 1.7 for single-year estimates. These correction factors are a pragmatic compromise, taking into account mean correction factors.
Applying the correction factor widened the resulting confidence intervals, reflecting the fact that the poverty line itself is subject to measurement error as well. The widened confidence intervals can be obtained for any subgroup of the population, and they are indicative of measurement uncertainty around poverty estimates.
We think that the resulting indicative confidence intervals are useful as an addition to trend charts in order to:
- visualise uncertainty for subgroups with varying sample sizes
- provide an understanding of ranges around central estimates
- provide clues as to whether trends and differences are likely to be meaningful or noise
- support a conversation about measurement uncertainty and other sources of error
Describing the data
In the tables below, proportions are rounded to two decimal points in order to demonstrate the range within which the central estimates sit. In the reference tables (see Data), proportions are available with five decimal points. Please note that using decimal points when talking about poverty rates may imply accuracy that the data cannot provide. To avoid this, we refer to full percentages in the commentary for the report, such as 21%.
We advise to use full percentages when talking about poverty rates, and to mention that the rates have some uncertainty around them. There are different ways of describing the level of uncertainty. The most important thing is to keep in mind that there is some measurement uncertainty; it is less important (and quite difficult) to try and be very precise about it.
In the charts of the main report, we include percentage ranges in the tooltips, such as 24% (19-30%). Alternatively, and to keep it more general, a rough +/- percentage point range can be given: 21% of people (+/- 4 percentage points) were in relative poverty after housing costs in 2020-23, and 24% of children (+/- 7 percentage points).
Population estimates are rounded to the nearest 10,000 individuals, the same as in the reference tables and in the commentary of the report.
Tables
All estimates and confidence intervals below were produced by DWP. Confidence intervals were obtained via bootstrapping.
Note that the latest population estimates for children and pensioners in material deprivation are both based on small sample sizes and considered not robust.
| Group | Proportion | Number |
|---|---|---|
| People | 20.67% (16.74 - 24.63%) | 1,110,000 (900,000 - 1,330,000) |
| Children | 24.19% (16.11 - 31.73%) | 240,000 (160,000 - 320,000) |
| Working-age adults | 21.32% (16.57 - 25.67%) | 720,000 (560,000 - 870,000) |
| Pensioners | 14.93% (11.1 - 18.97%) | 150,000 (110,000 - 190,000) |
| Group | Proportion | Number |
|---|---|---|
| People | 18.89% (15.08 - 22.24%) | 1,020,000 (810,000 - 1,200,000) |
| Children | 23.12% (16.54 - 30.54%) | 230,000 (170,000 - 310,000) |
| Working-age adults | 18.22% (13.5 - 22.1%) | 620,000 (450,000 - 750,000) |
| Pensioners | 16.81% (13.15 - 20.61%) | 170,000 (130,000 - 210,000) |
| Group | Proportion | Number |
|---|---|---|
| People | 17.4% (13.1 - 21.01%) | 940,000 (710,000 - 1,130,000) |
| Children | 20.96% (13.74 - 28.87%) | 210,000 (140,000 - 290,000) |
| Working-age adults | 18.4% (13.45 - 22.76%) | 620,000 (460,000 - 770,000) |
| Pensioners | 10.43% (7.19 - 13.31%) | 100,000 (70,000 - 140,000) |
| Group | Proportion | Number |
|---|---|---|
| People | 15.71% (11.72 - 19.27%) | 850,000 (630,000 - 1,040,000) |
| Children | 18.64% (10.84 - 25.92%) | 190,000 (110,000 - 260,000) |
| Working-age adults | 15.61% (10.83 - 19.49%) | 530,000 (370,000 - 660,000) |
| Pensioners | 13.07% (9.51 - 15.91%) | 130,000 (100,000 - 160,000) |
Note that numbers for children in combined low income and material deprivation, and for pensioners in material deprivation are not available in the latest period due to the small sample size.
| Group | Proportion | Number |
|---|---|---|
| Children | 9.64% (4.83 - 14.73%) | Not available |
An after-housing-costs table for children in combined low income and material deprivation is not available. Confidence intervals are expected to be similar to the before-housing-costs measure.
| Group | Proportion | Number |
|---|---|---|
| Pensioners | 6.39% (4.14 - 8.29%) | Not available |