Opinion Shamika Ravi writes: Our national surveys are based on faulty sampling

In this essay, I wish to highlight this major gap. In particular, show quantitively that the data quality related to NSS, NFHS, and PLFS needs a major sampling overhaul to reflect the true status of India’s real economy. These surveys use outdated sampling frames and hence, are not representative. In fact, the survey mechanisms are archaic and not adapted for rapid changes. As a consequence, these surveys grossly and systematically underestimate India’s progress and development and the misleading estimates from these surveys impede policy-making. Framing policies based on these estimates are unlikely to yield the desired results and we will continue to see a gap between ground realities and survey estimates.

Using projected population estimates, we find that nearly all major surveys in India that were conducted post-2011 and used the Census 2011 for the sampling frame have overestimated the proportion of the rural population significantly. This is one of the several problems with data quality, but it is a critical concern — and appropriately highlights the problem at hand.

The graph below shows the projected rural population of India over the years 2011 to 2021 and compares these with the estimates of rural population from the major surveys during this period. As is evident, every survey (except NFHS-4 of 2015-16) underestimates the proportion of the urban population or overestimates the rural population significantly compared to the population projections based on the Census 2011. A key implication is that the differences in estimates are significant and indicate major concerns in sample design across all these surveys. Moreover, it is important to appreciate that the population projections themselves fall short of the rapid pace of change on the ground. It is well documented that the proportion of the urban population based on a projection from Census 2001 grossly underestimated the speed of urbanisation in India. The proportion of the urban population that was projected to be achieved by 2016 was already achieved by 2011, according to Census 2011.

How they differ

Then there is the problem with the response rate. Not everyone who is sampled will respond to the survey questionnaire. The refusal to do so is never random, and our analysis shows that this is systematically correlated with wealth levels. The response rate falls with growth in income and wealth of households. This relationship is particularly strong for men and their response rate falls sharply with rising wealth levels. So once again, not correcting for such errors will most definitely lead to misleading estimates from the surveys, however large the sample size. If we continue to raise sample size without correcting for such non-sampling errors then the estimates from such large surveys will be “very precisely wrong”.

Given that rural and urban estimates are closely related to almost all variables of interest, such as health outcomes, wealth and asset ownership, poverty, employment and unemployment, growth etc, the resulting estimates from these surveys would over-represent the rural population and therefore systematically underestimate the improvements across the economy.

In fact, what is evident from the analysis is that the greater the improvement on the ground, the greater will be the bias in the survey estimates. The Indian economy has been incredibly dynamic in the last 30 years with significant policy reforms and subsequent major breaks in the long-term structural growth path. India is urbanising at a rapid pace, and often defying the expectations and projections based on past trends. A large part of statistical reforms should not merely focus on the availability, frequency and largeness of data, but greater emphasis should be placed on data quality.

The writer is Member, Economic Advisory Council to Prime Minister of India