An Expert Explains: How exit polls are conducted and read

Today, some judge the accuracy of an exit poll by looking at the survey agency that conducted it, or the television channel that commissioned it. Some others look at the sample size — a common notion is that the bigger the sample size, the more reliable the poll.

In reality, these should not be the indicators for judging the accuracy of an exit poll, which depends on many different factors.

Structured questionnaire

The science of surveys, which includes exit polls, works on the assumption that data were collected after interviewing a large number of respondents using a structured questionnaire, whether over the telephone or face to face.

This method is not new; it began back in 1957 during the second Lok Sabha elections when the Indian Institute of Public Opinion conducted a poll. But not even the best guesswork or estimate can skip the methodology required. Without a structured questionnaire, the data can neither be collected coherently nor be analysed systematically.

Sample size, representativeness

Since exit polls began in 1957, there has been enormous improvement in at least one aspect, which is sample size. Gone are the days when a national sample of 20,000-30,000 was considered large. Though the Centre for the Study of Developing Societies (CSDS) does not normally conduct exit polls, it did conduct a few — I recall the first exit poll during the 1996 Lok Sabha elections, using a sample size of 17,604. We went on to make a very accurate national projection of both vote share and seats.

The CSDS has continued its voting behaviour study (National Election Study) using post-poll surveys as a tool. Our post-poll survey of the 2019 Lok Sabha election had a sample size of little more than 25,000. Our seat projections may have been off the mark on some occasions, but the vote share estimates have been very close. (Information in public domain)

Yes, a large sample size is important, but based on my experience, I can say that more than the sample size, what matters is how representative it is, if it reflects the various types of voter profiles. But in recent years, the pressure on television channels (which in most cases are the sponsors of these exit polls) to have the largest sample has resulted in exit polls with bigger and bigger samples.

Our (Lokniti-CSDS) predictions based on the 2018 post-poll survey in Chhattisgarh got the winner wrong, and in the 2017 UP Assembly elections we predicted the winner right but were way off the mark for the final tallies for different parties. If asked if a bigger sample would have meant more accurate results, I would not have hesitated to say no. Certainly, something else went wrong with those post-poll surveys; maybe a case of some fake interviews filed by the investigators, which we could not figure out on time.

Technology — call-backs to respondents, images of interviews being conducted and phone calls from the field, WhatsApp groups and similar tools — have helped us in overcoming such shortcomings, yet there is no thumb rule for how to get the prediction correct.

Swing model and its complexities

There are other challenges. The prediction of seats is based on a swing model — the poll makes an estimate of vote shares for different parties and alliances by interviewing selected respondents, and the seat forecast is made based on the result of the previous election.

Estimating the vote share is not an easy task, given various diversities in India — of location, caste, religion, language, levels of education, economic class — all of which impact voting behaviour. Over- or under-representation of any of these diverse sections of voters can affect the accuracy of estimates.

If these were not enough, there are other difficulties. Since the swing model is applied on the previous vote shares, a change in alliances, or a split or a merger of parties between two elections poses a difficulty. For example, BJP and JD(U) in Bihar.

Measurement of swing and electoral change is easier when the contest is limited to two parties. The complexity of swings increases as more and more political players are added.

How comprehensive it is

The count method is time-consuming and labour-intensive, as one is expected to make an estimate for each seat. When agencies claim to have made seat-wise estimates, it is presented as the most comprehensive poll. This is when the sample size is as high as several lakhs.

However, some agencies have made innovations in the count method and this results in maximising gains by spending relatively less time and resources.

While an exit poll might claim to have covered all constituencies, in practice the poll is not required in some seats — for instance, why would one waste time and energy in conducting a poll in Varanasi where the Prime Minister is contesting, or in Gandhinagar where the BJP president is contesting? If one looks carefully at constituencies, state-wise, many such seats could be eliminated and one could still make the most accurate estimate.

After this elimination method combined with the count method, the survey is required in a limited number of difficult constituencies (swing constituencies). It is possible for an innovative exit poll to be far more accurate than polls conducted using traditional methodology. But while the polls using traditional methodology estimate vote share and help us analyse voting behaviour on the basis of different socioeconomic backgrounds, the count method can hardly give an estimate of vote shares, and any systematic analysis of voting behaviour could only be a dream.

Time to reflect

Many exit polls just throw a number for seats, no vote share, no methodological details. Should we even consider these as exit polls? I think the time has come when we need to distinguish between a real exit poll and estimate polls.

Vote share estimate is mandatory for any poll. If one is not estimating the vote share, the question that should be asked is: then what were you doing while conducting the polls? And what innovative method did one apply for conducting a poll that does not estimate votes, but predicts seats?

Sanjay Kumar is a Professor at the Centre for the Study of Developing Societies (CSDS).