Explained: What are randomised controlled trials, how do they work?

“For some things, there might be very common procedures in terms of results of randomised trials, for some other things that might be very different. There might be systematic preferences for some regions…the idea is to experiment at multiple sites and then do a better analysis and see the differences across the sites,” said Kremer on Wednesday (July 13) at the TN Srinivasan Memorial Lecture at National Council of Applied Economic Research (NCAER), Delhi.

What are randomised controlled trials?

RCTs involve dividing a population into smaller groups, in order to comparatively see the outcomes of an external stimulus. For example, if the aim of a study is to understand whether a free grains distribution scheme helped improve the nutrition levels among people living in a district, researchers will first create two groups within the population, and then put people into those groups randomly.

One group (called the control group) does not receive the grains or the external stimulus, while the other group (treatment group) does. After a designated period of time, details of how both the groups are doing would be collected. In this way, the goal is to understand what the overall impact of introducing something new could be.

At the time of their win, the Nobel citation said of the trio’s work: “The research conducted by this year’s laureates has considerably improved our ability to fight global poverty…Their new experiment-based approach has transformed development economics.”

Why are such trials criticised?

While the Nobel laureates had billed RCTs as a method to focus on more day-to-day answers to problems of poverty and deprivation, such as delivery of basic amenities, critics say the method cannot be used in all cases, while others argue it is fundamentally flawed.

Angus Deaton, the winner of the Economics Nobel in 2015, had said “randomisation does not equalise two groups”, and warned against over-reliance on RCTs to frame policies.

Like him, many believe that two or more groups carved out from a singular population living in an area may not be totally random. There may be more women in one group, or one group may have more people having some kind of distinctiveness that affects the result. As a result, the outcomes may not give an accurate view, and the very use of a scientific experiment tool in social sciences was questioned.

Further, RCTs also cannot be used to study something after it has happened, they need to be planned beforehand. And, while RCTs show results for a particular population in an area, it may not be proof that the same results will be achieved elsewhere, especially if the sample size is not big or the trial deals with a very specific kind of population sub-group. This is what Kremer was referring to with regards to India and its diversity.

Himanshu, an Assistant Professor of Economics at the Jawaharlal Nehru University, wrote in The Indian Express in 2019: “RCT has also been criticized for reducing the study of poverty to small interventions unconnected to the lived experiences of the poor. The discomfort among many established scholars is that this fashionable trend has made the historical, institutional and social structures of the persistence of poverty less relevant to understanding why the poor continue to remain poor.”

However, many agree that for a limited purpose RCTs have helped to tweak systems and improve outcomes.

For instance, they showed neither providing more textbooks nor free school meals improved learning among students. Instead, in schools in Mumbai and Vadodara, the biggest problem was that teaching was not sufficiently adapted to the students’ needs.