Prof Sanjay Kumar: ‘You have to understand what exit polls can — and cannot — do’

I never believed that the National Democratic Alliance (NDA) would touch the 400-mark. Perhaps, the figure of 340-350 could have been within the NDA’s reach, although my personal assessment was 320-330 seats. The bigger problem is that many pollsters did not give details about how they arrived at their numbers. That is what surprised me the most.

The under-300 NDA tally was a slight surprise but not altogether astounding. When we were doing our own post-poll analysis, I got a sense that the BJP was getting into a difficult situation in UP, although I did not expect it to struggle in West Bengal. So overall, I was not extremely surprised by the results.

On methodology used by pollsters

First, you have to understand what exit polls can — and cannot — do. They are designed to give you a broad direction, not give you the kind of details being put up these days by pollsters. With regard to how exit polls must be carried out, standardisation is a must. You cannot send 500 investigators to the field and tell them they are free to do what they want. Standardisation is about asking the questions more or less in the same manner to all the people from whom you are collecting the data.

Randomised sampling is also a must. You cannot tell investigators to go to anyone on the street and ask questions. A sample has to be drawn first. If people have been sent out in Maharashtra, they cannot only go to Mumbai and other “good places” to do the survey. Most important is transparency regarding the methodology — about the sampling processes, about the fieldwork exercise, about how the margin of error was arrived at, what the margin of error was.

On how a survey process works

First comes the sampling process. We do randomised sampling of parliamentary constituencies, polling stations and pick voters who need to be interviewed. Surveyors also have to be trained on questions they should ask and how to go about them. We hire people from various universities, mostly students. Nowadays, the survey instrument comes in the form of a mobile app in which questions flow one after another. It is important to maintain secrecy of the voter’s preference. That is why we tell our surveyors to pass the phone to the respondent for that question. The respondent then clicks on the symbol they voted/will vote for.

The app used for the survey is geo-enabled, meaning we can track the movement of the investigator and actually see how much time they spent per respondent and whether they asked every question or not. After the interview is complete, the data automatically gets uploaded on to our server. The surveys basically give us the voter’s preference, who they will/have cast their vote for. Collating this data gives us parties’ vote shares. This then has to be converted into seats. This is done based on what we call the “probability factor”, which takes into account vote shares and corresponding seat shares of the previous Lok Sabha elections. So, in the last

Lok Sabha elections, if a party got 12 seats from 20 per cent of the vote share, we can estimate what seats it might get with a 25 per cent
voteshare. This is done with the help of a computer programme. However, I am not sure if this is how others are doing their survey, especially those providing seat-wise predictions. To do that, one needs a sizeable sample in each constituency. I think pollsters are taking multiple shortcuts when it comes to seat projection. One obvious reason behind this is cost-cutting.

On the cost of an exit poll

The method of data collection has a huge relationship with expenses. Collecting data over the telephone is far less expensive compared to field surveys. The latter can be very expensive because when you create a randomised sample, you have to visit all kinds of places — even in the interiors. For maintaining the sanctity of the methodology, surveyors have to cover whatever the sample throws up.

In fact, one of the problems with these polls is that both people who carry out the survey and people who want to engage survey agencies want things done at a lower cost. This has resulted in compromise about how surveys are conducted. That’s why some errors are creeping in. There must be some transparency with regard to who is funding the survey, as well. People conducting exit polls or surveys should be willing to answer how their exercise was funded, they cannot evade these questions.

On judging reliability of exit polls

In recent years, there has been a race towards conducting the biggest surveys, with samples in the lakhs. I don’t think the size of a sample is as important as how representative it is. For this, randomised sampling is a must. With regard to judging a survey’s reliability, first and foremost is the voteshare estimate. If you get the voteshare estimate horribly wrong then we have every reason to say that the survey was flawed. However, even with the right vote share estimate, seatshare estimates might go terribly wrong. This is because India follows a ‘first past the post’ system. The problem is that the news channels are not satisfied with only vote share — they want the number of seats, and even specific seats. That is what draws viewers to the show.

On what can go wrong

Two kinds of problems can still emerge. While the ‘swing model’ that we use can give a broad range of seats, it might not be accurate if one party wins a large number of seats by a small margin of votes. But more than that, many respondents simply refuse to answer the voting question. The pollster has to take this number into account. The other problem is that while a randomised sample is likely to be representative of the population, it might not be so vis-a-vis the population’s actual voting behaviour. Electoral results can swing on the basis of which community turns out to vote, and in what numbers. Differential turnouts, however, make the pollsters’ sample unrepresentative. You also need to calculate the margin of error. For our polls, which had a sample size of 19,662, we calculated a margin of error of 3.08 per cent vote share.

When pollsters give massive seat ranges, I think it looks very funny. If your margin is say 280 to 360 votes, the real test of your poll is how much the actual result deviates from the mid-point of your range. But many people nowadays simply look at convenience — neeche wale se match kar liya toh humara poll achcha (if our poll matches even the lower limit, it is good).