ELECTION ANALYSIS

Exit polls and all that

ALISTAIR McMILLAN

NOW that the final results are available, we have the opportunity to look back at the accuracy and influence of the opinion and exit polls that proliferated during the campaign. The 1999 elections brought the science of psephology into the realm of publi c debate as never before. Although competition over poll forecasting by the media is by now an established feature of the Indian elections, never before have we seen so many polls and so much heat over what they said. The Election Commission's ban, the h earing in the Supreme Court, the subsequent withdrawal of the ban and the Chief Election Commissioner's call for a national debate on this question brought pollstering into public arena.

The debate focussed on two issues, although the participants were not always conscious of the fact that these are two independent questions: how reliable are these polls? and, how right is it to allow them during the electoral process? The battle lines w ere drawn in a rather simple fashion. Those who thought that these polls should be banned said that these were unreliable and vice versa. Unfortunately the intervention of the political parties in this debate resulted in a misplaced focus on the motivati ons of the pollsters rather than any informed debate on the methodology of forecasting. Now that the dust has settled down, perhaps we can look at both these questions less dispassionately.

The verdict on how the various poll forecasts have fared is not difficult to sum up: good on the national aggregates but poor on the State-wise disaggregates. And also, generally less than transparent on their methodology and less than careful in caution ing the public. Any reputable pollster makes it clear that the prediction is subject to error, and that therefore the results cannot be taken as given, even when the poll is conducted at the time and place of voting as in the case of exit polls. But what are we now to make of Narasimha Rao's prediction in The Times of India (September 23, 1999) that "the Samajwadi Party (S.P.) may end up as a marginalised force in this election and be relegated to fourth place (in Uttar Pradesh), with its seats i n single-digit figures" (it won 26 seats to finish a close second to the BJP) or that the BJP in Madhya Pradesh "is likely to end up with 15 seats" (it won 29). Like him many others could do with some caution.

The two tables here present in summary form some of the forecasts made by pollsters and carried by the media. Table 1 carries those forecasts that confined themselves to national aggregates or at the most zonal break-ups. Most of these were based on pre- poll surveys, although India Today's forecast based on post-poll survey also falls in the same category. Some of the early forecasts based on pre-poll surveys had clearly over-estimated the BJP and allies. Both the India Today-Insight poll and the two rounds of the Outlook-CMS poll overestimated the BJP and allies by at least 25 seats. It is not clear if this was a mistake on the part of the pollsters or it reflected the mood as it stood then. The predictions brought down their figu res for the BJP and allies by the time elections drew closer.

Even when results are not presented as State by State predictions, as in the polls published during the campaign by India Today, it is clear from the regional groupings that there was a similar bias, with the prediction for the Southern States und erestimating the Congress(I) seats, and for the North over-predicting the number of BJP seats by 21. The India Today polls also presented their data in a rather disingenuous manner: not making their estimate of the vote share explicit, but implici tly giving the NDA between 47 and 48 per cent of the vote (it is easy to estimate this, since they claim that the NDA received 49.3 per cent of male votes and 45.4 per cent of female, and we know that slightly more men vote than women). Their suggestion that 19.5 per cent of Muslims were voting for the BJP-led alliance is also extremely suspect. Overall, their prediction is not particularly wide off the mark, but these internal discrepancies suggest that this was more by luck than design.

Table 2 presents the second category of forecasts, which chose to reveal their hand at the State level, and compares it to the final outcome. All the polls in this category except the Times-DRS poll are exit polls. It must be noted that this kind of disaggregated forecast of seats is an unusual phenomenon in election forecasting. Pollsters all over the world would be reluctant to make this kind of disaggregated forecast for seats in a first-past-the-post system. The fact of small sample size at t he State level and the vagaries of our electoral system in non-bipolar contests makes it a psephological nightmare. The final results have once again proved that the conventional wisdom on this score is not off the mark. Perhaps the pollsters would have been better off not attempting State-wise seat forecast.

In overall terms all the exit poll forecasts except the one carried by Lokmat seem to have got the aggregate seats for the BJP alliance within a reasonable margin of error. But it is not clear if it is not the function of a verdict that leaves amp le room for the errors to cancel each other out. Everyone overestimated the Congress(I) and its allies, while underestimating the rest. The record leaves much to be desired at the State level. The DD-DRS forecast was widely off the mark in Andhra Pradesh , Bihar, Karnataka and of course U.P. The STAR-Insight poll opted for much bigger sample size, the kind one would require to make State-wise forecasts. They must be credited with getting the Andhra Pradesh forecast closer than anyone else did. But big sa mples did not help them get Bihar, Karnataka, Punjab or U.P. right. As in 1998 when no one predicted the huge change in Tamil Nadu, the pollsters have appeared to have made collective errors which must cast doubt on their methodology. No one appears to h ave predicted the Congress(I) victory in Karnataka, although the difference in votes was some five per cent, and no one saw how badly the BJP was really doing in U.P. These failures serve to remind us that making disaggregated seat forecast is indeed a v ery demanding if not an impossible task. None of the pollsters was willing to divulge exactly what methods they used for converting their vote estimates into seats, but the expectation is that some estimate is made of the vote share and distribution in e ach State, then this estimate is used to make a seats forecast for each State and finally the predicted seats for each State are totalled to get an overall picture. If the overall picture was all that was revealed then the pollsters would be justified in arguing that their model is built to cope with inaccurate forecasts for the individual States. If pollsters are going to predict results on a State by State basis in future, they should re-examine their methodology.

TABLE 3 presents the phase-wise predictions of the DD exit poll and compares it to the phase-wise outcome. As in the case of the State-wise break-up, the forecast does not do very well in terms of the phase-wise break-up. But it needs to be noted that it s figures actually underestimated the wave in favour of the BJP and its allies in the first phase of voting. It also overestimated the Congress(I) and its allies in all the phases. Clearly, accusations of manipulating the actual outcome in favour of the BJP combine are hard to substantiate.

NOW to the second question. At the core of the debate about exit and opinion polls is the question of whether they influence voting - after all, if the opinion polls have no effect on how people vote, there is little point in trying to manipulate them. T he main argument against the publication of opinion and exit polls is that they create a bandwagon effect, with voters seeing that a party is doing well and adjusting their voting decision. If this were the case we would expect to see some shift in votin g patterns in favour of the NDA, front-runner in all these polls, as the election progressed. The contrary hypothesis is that there is an underdog effect - that voters see one party or alliance doing well, and as a result vote against them.

Looking at the election results it is very difficult to distinguish any clear trend which stands out above the fluctuations determined by which States were voting. To look for a clearer picture it is possible to look at individual States in which polling was over a number of phases. Again, there is a problem that the different phases occurred in different regions within the States and so can be expected to show different patterns of voting. In the CSDS post-poll survey, we asked our sample of the elec torate whether they had either read or heard of the forecasts based on the opinion or exit polls, and if they said yes, whether it had affected their voting decision.

Of the sample of over 9,000 electors some 9 per cent said that they had read about the polls, and 21 per cent that they had heard about them. It is interesting to note that the polls respondents who did know about the polls were more likely to vote for t he NDA, a consequence of the BJP alliance having supporters with more access to the media and a higher educational profile. The pattern of responses shows little variation across the polling phases, slightly higher in the first two phases and the last ph ase, and lower in the third and fourth phases. Again, it is hard to make much of the phase-wise pattern because it is so dependent on which part of the country the polling was taking place - the first and second phases include the highly literate Souther n States, the third and fourth phases were largely in U.P., Orissa and Bihar where exposure to the media is limited, and the final phase was dominated by West Bengal, again where literacy and media exposure are high. What is apparent is that the cacophon y of polling analysis and debate during the polling did not reach out to where the voting was actually going on, and throughout the election most voters were unaware of the polls.

Of those who had heard about the opinion polls, 15 per cent said that their voting decision had been influenced by them. This figure, just over three per cent of the total in our survey, is potentially extremely politically significant - a shift in votes of this magnitude could have completely changed the outcome of the election. The proportion of those who were influenced by the polls also increased slightly in the last three phases, suggesting that the exit polls from those States voting early in the election could have influenced those voting later on. But there does not seem to be any clear systematic effect. People who said that they were influenced by the polls were more likely to have voted for the NDA, but this appears to be because the NDA vot ers are more likely to have heard about the polls. And when we compared the voting decisions of this group with how they said they had voted in the last election, the responses suggested that there were only a very small proportion who had actually chang ed their voting behaviour. This does not mean that there was actually no effect: they may have been intending to switch sides but been persuaded not to, after reading an opinion poll, but suggests that there is not any great systematic effect that could change the overall outcome.

A further way that the opinion polls published during the election could change voting is by encouraging tactical voting. The results from U.P. are a startling reminder of the fact that in a first-past-the-post electoral system how the votes are distribu ted is almost as important as the sheer numbers. In 1998 the BJP and allies won 71 per cent of the seats in the State with just 37 per cent of the vote: this year it won just 37 per cent of the seats with 31 per cent vote. The S.P. saw its vote fall by f our per cent from last year, but picked up six more seats, and the BSP picked up 10 more seats from just a two per cent rise in vote share. It seems that the anti-BJP voters shifted to whichever party put up the strongest fight in each particular constit uency, and with the Opposition less divided the BJP suffered significant losses. Did the opinion polls help make voters who wished to vote out the BJP decide whom to vote for? The evidence is not clear, and the main problem is that State-wise opinion pol ls do not help someone who wants to vote tactically very much because the decision depends on the voting patterns at the constituency level rather than at the State level. The idea that the pro-NDA polls solidified the anti-BJP vote in U.P. cannot be giv en any support.

Inaccurate opinion and exit polls can hardly help the democratic process. And when the voting process is dragged out over such a long time, a decision the Election Commission must surely regret, the uneasy sensation that the fairness of the election can be undermined arises. But the evidence from the 1999 election shows that the voter is more likely to be influenced by the regional dynamics of political power than national issues and media punditry.