होम
Political Analysis Ignoramus, Ignorabimus? On Uncertainty in Ecological Inference
Ignoramus, Ignorabimus? On Uncertainty in Ecological Inference
Martin Elff, Thomas Gschwend and Ron J. Johnstonयह पुस्तक आपको कितनी अच्छी लगी?
फ़ाइल की गुणवत्ता क्या है?
पुस्तक की गुणवत्ता का मूल्यांकन करने के लिए यह पुस्तक डाउनलोड करें
डाउनलोड की गई फ़ाइलों की गुणवत्ता क्या है?
खंड:
16
भाषा:
english
पत्रिका:
Political Analysis
DOI:
10.2307/25791917
Date:
January, 2008
फ़ाइल:
PDF, 2.32 MB
आपके टैग:
फ़ाइल 15 मिनट के भीतर आपके ईमेल पते पर भेजी जाएगी.
फ़ाइल 15 मिनट के भीतर आपकी Kindle पर डिलीवर हो जाएगी.
टिप्पणी: आप जो भी पुस्तक अपने Kindle पर भेजना चाहें इसे सत्यापित करना होगा. Amazon Kindle Support से सत्यापन ईमेल के लिए अपना मेलबॉक्स देखें.
टिप्पणी: आप जो भी पुस्तक अपने Kindle पर भेजना चाहें इसे सत्यापित करना होगा. Amazon Kindle Support से सत्यापन ईमेल के लिए अपना मेलबॉक्स देखें.
Conversion to is in progress
Conversion to is failed
0 comments
आप पुस्तक समीक्षा लिख सकते हैं और अपना अनुभव साझा कर सकते हैं. पढ़ूी हुई पुस्तकों के बारे में आपकी राय जानने में अन्य पाठकों को दिलचस्पी होगी. भले ही आपको किताब पसंद हो या न हो, अगर आप इसके बारे में ईमानदारी से और विस्तार से बताएँगे, तो लोग अपने लिए नई रुचिकर पुस्तकें खोज पाएँगे.
1


2


Society for Political Methodology Ignoramus, Ignorabimus? On Uncertainty in Ecological Inference Author(s): Martin Elff, Thomas Gschwend and Ron J. Johnston Source: Political Analysis, Vol. 16, No. 1 (Winter 2008), pp. 7092 Published by: Oxford University Press on behalf of the Society for Political Methodology Stable URL: http://www.jstor.org/stable/25791917 Accessed: 21092016 01:54 UTC JSTOR is a notforprofit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org. Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at http://about.jstor.org/terms Society for Political Methodology, Oxford University Press are collaborating with JSTOR to digitize, preserve and extend access to Political Analysis This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms Political Analysis (2008) 16:7092 doi:10.1093/pan/mpm030 Advance Access publication December 8, 2007 Ignoramus, Ignorabimus? On Uncertainty in Ecological Inference Martin Elff Faculty of Social Sciences, University of Mannheim, A5, 6, 68131 Mannheim, Germany email: eljf@sowi.unimannheim.de (corresponding author) Thomas Gschwend Center for Doctoral Studies in Social and Behavioral Sciences, University of Mannheim, D7, 27, 68131 Mannheim, Germany email: gschwend@ unimannheim.de Ron J. Johnston School of Geographical Sciences, University of Bristol, Bristol BS8 1SS, UK email: r.johnston @ bristol. ac. uk Models of ecological inference (El) have to rely on crucial assumptions about the individual level datagenerating process, which cannot be tested because of the unavailability of these data. However, these assumptions may be violated by the unknown data and this may lea; d to serious bias of estimates and predictions. The amount of bias, however, cannot be assessed without information that is unavailable in typical applications of El. We therefore construct a model that at least approximately accounts for the additional, nonsampling error that may result from possible bias incurred by an El procedure, a model that builds on the Principle of Maximum Entropy. By means of a systematic simulation experiment, we exam ine the performance of prediction intervals based on this secondstage Maximum Entropy model. The results of this simulation study suggest that these prediction intervals are at least approximately correct if all possible configurations of the unknown data are taken into account. Finally, we apply our method to a realworld example, where we actually know the true values and are able to assess the performance of our method: the prediction of districtlevel percentages of splitticket voting in the 1996 General Election of New Zealand. It turns out that in 95.5% of the New Zealand voting districts, the actual percentage of split ticket votes lies inside the 95% prediction intervals constructed by our method. 1 Introduction Many students and practitioners of social science have remained skeptical about the feasibility of sound ecological inference (El). To some, the terms "ecological inference" Authors' note: We thank three anonymous reviewers for helpful comments and suggestions on earlier versions of this paper. An appendix giving some technical background information concerning our proposed method, as well as data, R code, and C code to replicate analyses presented in this paper are available from the Political Analysis Web site. Later versions of the code will be packaged into an R library and made publicly available on CRAN (http://cran.rproject.org) and on the corresponding author's Web site. ? The Author 2007. Published by Oxford University Press on behalf of the Society for Political Methodology. All rights reserved. For Permissions, please email: journals.permissions@oxfordjournals.org 70 This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms Uncertainty in Ecological Inference 71 and "ecological fallacy" appear almost synonymous: They doubt whether it will be pos sible at all to draw any conclusions about the behavior of individuals from aggregate data. There seem to be good reasons for such skepticism: El has to rely on certain assumptions about the datagenerating process at the level of individuals that cannot directly be tested, simply because the data on which such tests could be based are unavailable to begin with. The burden of assumptions becomes especially visible in a recent survey of Bayesian approaches to El for 2 x 2 tables given by Wakefield (2004), which shows that even Markov chain Monte Carlo (MCMC) approaches are faced with this indeterminacy. Wake field considers a total of 13 different variants of prior distributions for use in MCMC analysis of 2 x 2 tables including King's original truncated normal prior, beta exponential, and Student logistic gamma compound prior distributions and compares them with respect to prediction bias for King's (1997) Louisiana party registration data. Although there is one prior distribution that performs best in this application, one may still ask whether this result can be generalized to all possible El applications to 2 x 2 tables. The question remains, as Fienberg and Robert aptly remark in their comments to Wakefield's review, "to what extent can we really distinguish between the fit of different models, hierarchical or otherwise, when only aggregate data are available?" (Fienberg and Robert 2004, 432). Nonetheless, without certain restrictive assumptions about the process generating the unknown data, it is impossible to obtain any estimates for an El problem, however pre liminary these estimates may be. It should be noted that the challenge of El is only a special case of the wider class of illposed inverse problems (King 1997). The task of El is solving a problem that is inverse insofar as only a set of summaries of the data of interest are given and ill posed insofar as the information given by these summaries is not sufficient to identify a solution. Problems of this kind abound in the technical and scientific literature and numerous approaches to their solution have been proposed (see e.g., Groetsch 1993). Therefore, we think that a general rejection of El procedures would be premature. Al though the assumptions inherent in an El procedure cannot be tested by means of statistical techniques, it is still possible to delimit the potential error that is associated with predic tions from such a procedure. Constructing bounds to this potential error is the aim of the present paper. We derive a method to construct "robust" prediction intervals, that is, intervals that contain the true values of the unknown data with (at least approximately) known probability. Further, we assess the performance of these prediction intervals by means of simulation and a realworld example, the reconstruction of splitticket votes in the 1996 General Election of New Zealand. As point of departure, we take an El procedure recently presented in this journal, the entropymaximizing approach of Johnston and Pattie (2000). When recast into a probability model, the JohnstonPattie model imposes rela tively mild and clearly structured restrictions on the unknown datagenerating process and thus suits well the exploration of the consequences of model departures. As it will turn out, both in the simulation study and in the application to ticket splitting in New Zealand, the prediction intervals that we construct have a coverage that is almost identical to their nominal level. The paper is organized as follows: In Section 2, we explain the fundamental dilemma of El, which results from the necessity of El procedures that employ certain restrictive assumptions that cannot be tested without the aid of the very data which are unavailable in a typical El application. In Section 3, we propose a secondstage correction of the error distribution of El estimates based on the Principle of Maximum Entropy and report the results of a simulation study to assess the performance of the proposed method. In Section 4, we discuss how this method can be adapted to cases where some of the data on which El is based do not come from populationlevel aggregates but from a survey sample. In This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 72 Martin Elff, Thomas Gschwend, and Ron J. Johnston Section 5, we illustrate our method with its application to splitticket voting in the General Election of New Zealand. Section 6 discusses the limits of our proposed meth whereas Section 7 summarizes our results.1 2 A Basic Dilemma of El The situation of El can be compared to reconstructing the "inner workings" of a "bl box." These inner workings may be, for example, the numbers xijk of members various ethnic groups (i = 1, ..., I) who do or do not turn out to vote (j = 1, ..., voting districts (k = 1,..., K) or the probabilities Pv(Xijk = xijk) in which these counts occur. If the total sum of the counts is n, for example, if the total population of eli voters is n, then the total number of possible configurations of counts xijk that sum to be expressed as the binomial coefficient (^^j^K_ ~\  In typical instances of El, this is a vast number: Even in a country with only 1 million (=ri) voters and three (=/) ethnic groups, whose members may or may not turn out to vote (J = 2) in one of 100 (=K) voting districts, there are n + UK  1 ^ = ( 106 + 599 ^ ^m (1) UKI J \ 599 J possible ways to arrange the voters into the black box. In the absence of any information about the marginal sums njk = Ylixijk> ?u ? or rijj. = J2k xijk>mat is? ^ onty me totalsum n ?f me counts xijk is known and if one has to make a point prediction about what configuration of counts is present inside the black box, it seems that one cannot do better than pick any of the {^^j^K_~[ 1 ^ possible configura tions at random.2 Such a random pick can be represented by a Uniform distribution on all possible configurations of counts in the (I x J x #)array that have a total sum of n. The probability of any specific configuration of being chosen then is l j ^ n ^j^K_ 7 1) Also, the probability of hitting the true configuration of counts ? =(x?jk) by accident is also l j ~^jgK_ 7 1) Thus, one could also think of this true configuration as the outcome of a random variable X=(XiJk) that has arrays T=(Xijk) as values and has a Uniform distribution on its values. Now, information about marginal sums of the cell counts, for example, the number of members of ethnic groups within each voting district, can vastly reduce the number of possible cell counts one has to consider. If we consider the case (for reasons of simplicity if not of plausibility) that the observed turnout rate in all voting districts is 50%, then one has An appendix, available online on the Political Analysis Web site, contains supplemental information: some background on maximum entropy distributions, properties of the Dirichlet and Dirichletmultinomial distribu tions relevant for the argument of our paper, details on the computation of Betabinomial prediction intervals, and some details about two nonparametric alternatives to the method proposed in our paper. The Political Analysis Web site also contains source code in R and C as well as data suitable for the replication of the analyses presented here. 2A note on notation: We abbreviate , etc., as J2i anc* S/=i Z]y=i Ylk=i as X^y,* ^tne summation limits are understood. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms Uncertainty in Ecological Inference 73 only to consider, given a combination of voting district k and turnout status y, the numb of possible triples of numbers that sum to nlJK = 106/200, that is, (jkhj 1 \ = ^ iovmo.+3 iy 107.if (2) a number that is several orders of magnitude smaller than the number of possible config urations if no information about the marginal counts was available. This suggests that predictions about the unknown cell counts that take into account the marginal tables can have a greatly improved performance as compared to predictions that do not. El methods can be considered as attempts to make such improved predictions. Yet, these methods are plagued by a serious problem, which we will expose in the following. Statistical inference usually is made with the intention to describe a population based on a sample. For example, one may try to identify a model of voting behavior that connects voters' decisions to their own and the candidates' policy positions. If a correctly specified model is found, it can be used to make outofsample predictions, which may be pre dictions about future states of affairs or contrafactual states of affairs. One can also find both types of usage of results of EL As an example of the first type of usage, Burden and Kimball (1998) try to find how much and why American voters engage in splitticket voting between Presidential and Congress elections based on aggregated Presidential and Congressional votes for American voting districts. An example of the other type of usage is predicting the effects of changing the boundaries of voting districts on the representation of racial groups in these districts and on the chances of Democratic and Republican candidates in these rearranged districts (Cirincione, Darling, and O'Rourke 2000). As with statistical inference, both usages of El need correctly specified models that describe the population. In contrast to statistical inference, El is confronted with two problems that together pose a dilemma. The first problem is that of modeling indeterminacy: If only aggregates of the variables of interest are observed, there will always be more than one model of an interrelation between these variables that even fits perfectly to the observed aggregates. The second problem is that of inferential indeterminacy: If one arrives at identifying a model describing the interrelation of the variables of interest, this model will entail certain restrictive assumptions about the population, which cannot be tested based on aggregate data alone, which usually are the only data available. Hence the dilemma: If the first problem is solved, the second one is inevitably encountered. If one tries to avoid the second problem, one cannot solve the first one. The challenge of the first problem lies in finding assumptions suitably restrictive for model identification. The criteria for suitability may vary with the application, but in most cases one will use assumptions that are plausible on the one hand and convenient on the other insofar as they lead to simple models for which estimation is feasible. However, plausibility and simplicity may conflict. Consider the case in which someone has aggregate data on turnout and the proportion of AfricanAmericans at the level of voting districts and wants to find out whether African Americans differ from other citizens with respect to turnout. In this case, a model that presupposes that in each voting district turnout and race are conditionally independent will perfectly fit to the aggregate data and thus cannot be improved based on these data alone: Let rii.k denote the number of AfricanAmericans eligible to vote in district k, n2.k the number of other citizens eligible to vote, n.ik the number of citizens eligible who actually turn out to vote, and n.2k the number of citizens eligible who do not turn out to vote in district k. Further, let p^ be the probability that an eligible citizen in district k is an This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 74 Martin Elff, Thomas Gschwend, and Ron J. Johnston AfricanAmerican (/ = 1) and turns out to vote (j = 1), is an AfricanAmerican (/ does not turn out to vote (j = 2), is not an AfricanAmerican (i = 2) and turns ou (j = 1), or is not an AfricanAmerican (/ = 2) and does not out to vote (j = I) and le the actual number of AfricanAmericans/others who turn out/do not turn out to district k. We thus have = Yljtyk and n.jk = ]C/ *(/* Then, a model that states Pij\k =Pv\kPj\k (3) can be fitted to the aggregate data using the maximum likelihood estimates PiM =Pi\kPj\k = \r\n.t Y,jnjk (4) This model implies that within each race group i in district k, the probability to turn out to vote is equal: _ Pij\k _ Pi\kP.j\k _ Pj\k Pj\ik ? ^^^7 PJ L<jPij\k 2^jPi\kPj\k 2^jPj\k that is, turnout is unrelated to race. Of course, such a model s fact that survey data indicate that turnout and race are statistic and Claggett 1984). But this independence assumption cann available aggregate data alone. On the other hand, any mode turnout to be related, that is, p?j\k = CijPi.\kp.j\k with ctj ^ 1, YlicijPij\k= Pi\k, and YlicijPij\k ~ P j\k, is empirically indistinguishable from the independence model. Thus, it seems that based on aggregate data alone, one cannot decide whether race and turnout are related or not. Ecological regression (Goodman 1953, 1959) aims to overcome this problem by using a model that is in some ways more but in other ways less restrictive. In the case of race and turnout, it requires that the conditional probabilities of turnout are different across race groups but are the same in all voting districts, that is, p^ = Pj\iPj\h Some refined eco logical regression models even relax this assumption; they allow the conditional proba bility to vary across districts according to some known properties of the districts, that is, Pij\k  Pj\ikPi \k and Pj\ik = /(Pi/o + Pc/iZi* + + P(/dZd*)? where zu, ..., zDk are the properties of the districts and pl5..., pD are the parameters to be estimated. In contrast to the conditional independence model discussed above, ecological regression models can, to some degree, empirically be tested. With maximum likelihood estimates Pj\ik =/(Pyo + Piyi^u + + %DzDk) and pL\k = m.k/ ]T\ ni.k, it is still logically possible that predicted turnout numbers per district n.jk = Y^jPj\ikni k differ from actually observed turnout numbers n.Jk. Thus, in case of a poor fit between observed and predicted rates of eligible citizens who turn out in each of the districts (indicated by k), one may conclude that the ecological regression model is wrong and some of its assumptions have to be lifted. To account for departures of observed counts in a marginal table, such as the observed turnout per voting district, from predicted turnout, various authors extend this ecological regression by a random component: In this extended ecological regression model, conditional probabilities p^ik are not fixed parameters but outcomes of a random variable. King proposes that these conditional probabilities have a truncated normal This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 75 Uncertainty in Ecological Inference distribution. Brown and Payne (1986) and Rosen et al. (2001) use the more natural assumption that the conditional probabilities have a Dirichlet distribution with mean E(Pjlik)' = Kj]ik = E,?p(^ + R,izu +  + R^) (6) and precision parameter 9. As Goodman (1953) notes, ecological regression models are suitable only if it is reasonable to assume that there is a causal relation between the properties corresponding to the marginal tables. But based on the marginal tables alone, that is, the districtlevel aggregates, it is possible neither to establish such a causal relation nor to disprove it: As just noted, in ecological regression models there may be a lack of fit between observed and predicted counts in the marginal table of turnout per district. The independence model (3) will thus almost always seem superior to any ecological regression model. But again, the fact that the marginal tables can perfectly be fitted by the independence model does not imply that, for example, turnout and race are unrelated since there are infinitely many models that pose a relation between race and turnout which may also perfectly fit to the marginal tables. Statistical relations between variables represented by the observed marginal tables are often suggested by survey samples. For example, there is evidence from election studies (e.g., Abramson and Claggett 1984) that there is a clear statistical association between race and turnout. Therefore, it is worthwhile to combine evidence from aggregate data with evidence from survey data. An El model that allows for this is the EntropyMaximizing model proposed by Johnston and Pattie (2000) in an earlier issue of this journal. In contrast to the models just discussed, Johnston and Pattie aim to directly make predictions about the unknown counts xtjk without the need of any statistical model. They show that a model of cell counts that are the most likely subject to the constraints j has i k the logl log%k They also sho procedure. Although Johnston and Pattie (2000, 337) state that their proposed procedure is "math ematical rather than statistical" and that therefore "no error terms" are attached to the predicted cell counts, this is not the case: Readers familiar with loglinear contingency table analysis (Fienberg, Holland, and Bishop 1977; King 1998) will realize that the JohnstonPattie model is actually a loglinear model for counts without threeway inter actions. However, the correct form of this model is logH,;/* = <*ij + Pi* + yjk + x, (9) where \iijk is the mean of a Poisson distribution, that is, the counts may indeed vary around this mean. The iterative scaling algorithm, which Johnston and Pattie propose, is the one usually employed to find maximum likelihood estimates for such loglinear models. That is, the "maximum likelihood solution" of Johnston and Pattie (2000, 335) is in fact the This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms = 76 Martin Elff, Thomas Gschwend, and Ron J. Johnston maximum likelihood estimates of the means of Poisson distributions that have the struc ture of equation (9). If the sum J2ijkxijk = Eijk ^ijk *s known in advance as n, which is usually the case in contingency table analysis and in El, the distribution of the cell counts can also be modeled as a multinomial distribution with cell probabilities exp(otff + pa + yjk) Pijk Ew?p(*. + P^ + yJ" 1 } Good (1963) has shown that finding the multinomial distribution that maximizes the entropy ? Y^ij,kPij^nPijk subject to the constraints j i leads k to this probabilities implies that milder restri regression m the condition More recent information Pattie (2000). bilities pjiik, similarly to e The restricti compared easi model, we pi>k 2sr,s,t = ^^iwR ^ 1st) with exP(P<*) a ni k and exP(fyO a n.jk] (12) eXP\Prf ~r for the ecological regression model (without districtlevel covariates), we have expK + Jjk) exp(oty) m = ErA,exP(a? + yw) wlth ** = E^K) ^ K "* (13) whereas for the JohnstonPattie multinomial model, we have + ptt + yjk) Ptik =exp(oey 7riT^v (14) All these are special submodels of the saturated model exp(a,7 + pa + yjk + 8fft) Pm E^exp(ars + Prt + Y,f + 5r,)' 3More on the application of information theoretic concepts to contingency table analysis and to statistics in general can be found in Kullback (1959). The argumentation of Johnston and Pattie (2000), however, follows a nonprobabilistic interpretation of entropy mentioned by Jaynes (1968). This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms Uncertainty in Ecological Inference 77 which does not pose any restrictions on the structure of the cell probabilities. T completedata loglikelihood of this model is i= ^2xijk^omjk= ij,k ij i,k j,k ij,k (16) The implications of this expansion of the completedata loglikelihood are quite ambiv alent: If, for example, the assumptions inherent in the independence model apply, that oiij = 0 and bijk = 0 for all ij, and k9 the completedata loglikelihood depends only on t observed aggregates rii.k and tij.k and the parameters of the independence model can estimated by direct maximum likelihood. Also, if the assumptions of the JohnstonPatt model apply and odds ratios are equal in all districts, that is, the threeway interactio parameters hijk are zero for all ij, and the completedata loglikelihood depends only the aggregates n,y., ntand rij.k. No such conclusion can be drawn with respect to eco ical regression: Ecological regression models are used if the aggregate table n^. is available. However, ecological regression models set parameters (3/.* and 8^ to zero bu not OLij. Therefore, the completedata loglikelihood for ecological regression is not a special case of equation (16), but still it requires assumptions that are untestable with access to the complete array of counts xijk. Thus, any of the models so far discu requires assumptions that can be tested only if the complete data xijk are available. In problems, however, they are not. This problem, which may be called inferential indeterminacy, is much more seriou than the fact that distributional assumptions of, for example, ecological regression mo of King (King 1997; King, Rosen, and Tanner 1999; Rosen et al 2001) or Brown Payne (1986) cannot be checked or that under certain circumstances an ecologica gression model like King's may be susceptible to "aggregation bias" (Openshaw Taylor 1979, 1981; Cho 1998; Steel, Beh, and Chambers 2004). For example, one m want to make predictions about the number of AfricanAmericans who will turn out to in a specific voting district. If one uses one of the models for El discussed above and assumptions inherent in this model do not hold, the predictions will be systematicall biased, irrespective of the specific estimation procedure one uses: Suppose, for examp that in a specific application of El, the complete data are generated from a multinom distribution with cell probabilities p% as in equation (15) and all structural parameters a*, PL yjh and are nonzero. Suppose further that one uses, for example, a procedure based on the JohnstonPattie model (8) for EL If n approaches infinity, the cell proportions xijkln will converge to pfjt and the proportions n^Jn, n^n, and n.jk/n in the marginal tables will converge to p% = Y<kP% P*k = HjP% d P% = respectively. Further, the parameters of equation (8) will converge to values 6t,y, and yjk that maximize the scaled expected loglikelihood (17) This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 78 Martin Elff, Thomas Gschwend, and Ron J. Johnston Now even if 6ty, pljb and yjk are equal or very close to the corresponding parameters a*, Pa, and yfk, ptjk and ptjk will in general be different because of the nonzero 8**. Conse quently, estimates of cell probabilities based on the JohnstonPattie model will have asymptotic bias pyk ? p% and thus are not consistent. However, in order to obtain an estimate of this bias, one will need an estimate of 8^, which is unavailable because only the marginal tables are known. This is what makes the dilemma posed at the beginning of this section so serious: Without making certain identifying assumptions, one will not arrive, for example, at a prediction about the number of AfricanAmericans who turn out to vote at all, apart from a random guess. The identifying assumptions, however, have crucial behavioral implications and if these assumptions are wrong, one will incur biased predictions. But these assumptions cannot be tested without access to the complete data, which are un available to begin with. Therefore, in any specific instance, one will not know how large this bias actually is. 3 Accounting for Inferential Uncertainty: A Maximum Entropy Approach Estimating cell probabilities with the help of one of the models discussed in the previous section exhausts the resources for statistical inference. The models used for estimating the cell probabilities all employ certain restrictive assumptions that are necessary for the identification of cell probability estimates. Yet, as long as one is faced with the task of El, these assumptions cannot be tested. They could be tested as a statistical hypothesis only if the complete data were available. But then, the problem no longer would be one of El. This dilemma seems to sustain a skepticism with regard to the validity of El. However, we would argue that a healthy amount of skepticism does not force us to give up the attempt at El altogether. Doing so would mean ignoring the information contained in the marginal tables; it would mean throwing out the baby with the bathwater. On the other hand, it is clear that one should not put the same confidence in predictions from an El model as one would in predictions from a welltested statistical model. In the present section, we propose a model that allows taking into account the uncertainty associated with estimates obtained from an El. Since the resources for statistical inference are already exhausted, the model we propose cannot be justified in terms of theoretical statistics. All we can do is to appeal to some general principles that have some plausibility. The principle on which our proposition is based is the Principle of Maximum Entropy, which is a generalization of the (in)famous Laplace's Principle of Indifference. Before we present our proposed model, we need to explain what the Principle of Maximum Entropy entails and in what perspective we hold it to be plausible. Laplace's Principle of Indifference, of which the Principle of Maximum Entropy is a generalization (Uffink 1995), postulates that one should assign to each elementary out come ..., {xn} of a probability experiment, in absence of any prior information, the same probability l/n. For example, if the experiment is throwing a dice, one should assign to the outcome of each number one, two, three, four, five, or six the same probability 1/6. The Principle of Maximum Entropy generalizes this principle to cases where some prior information of the probability distribution in question is available and where the probability distribution may be continuous with infinite support. It postulates that, if only some moments of the probability distribution (i.e., some nonrandom functions of the probability distribution, like, e.g., mean and variance) are given in advance, one should select the probability distribution that has maximal entropy among a set of probability distributions This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms Uncertainty in Ecological Inference 79 with the same support with the given moments. This principle leads to some common families of probability distributions such as the family of normal distributions or the family of exponential distributions. For example, the normal distribution with zero mean and variance ex2 has maximal entropy of all continuous distributions over the real line with zero mean and variance a2. The exponential distribution with parameter X has maximal entropy of all continuous distributions over the positive half real line with mean X~1 (Shannon 1948 The Maximum Entropy Principle has been used to specify "reasonable" null hypoth eses for contingency table analysis (Good 1963; Golan, Judge, and Perloff 1996), to specify noninformative priors for Bayesian inference (Jaynes 1968), and for proposing solutions to illposed inverse problems (Vardi and Lee 1993). But it has also been used for El (Johnston and Hay 1983; Johnston and Pattie 2000; Judge, Miller, and Cho 2004). Here, we use the Principle of Maximum Entropy to motivate and construct a secondstage probability model to account for inferential uncertainty. Suppose (Xijk) is an (/ x J x ?)array of counts generated by a multinomial distribution with size index n (the "population size") and cell probabilities pfjk. Suppose, further, that one has knowledge only about the marginal tables of this array and tries to make pre dictions about the cell counts. Using an El method as discussed in the previous section, one may arrive at an estimate p^k for the cell probabilities. The model by which the cell probabilities are estimated may or may not be correctly specified, that is, the "true" cell probabilities pfjk may or may not satisfy the constraints inherent in the model. Based on the El procedure of choice, one makes the prediction npijk about the cell count xijk. Then, the error of prediction of xijk by pijk can be decomposed as follows: Xijk  npijk = n  pfjk^j + n(pfjk pijk). (18) The difference (xyk/n) ? pfjk will have mean zero and variance p*(l ? pfjk) In and thus will be the smaller the larger n is, while the difference pfjk ? Pijk will not become smaller unless the model leading to p^ is correctly specified, that is, if the assumptions of the El model employed are satisfied by pfjk. In the analysis of bias at the end of the last section, we held pfjk fixed and considered the estimator as random. In contrast, we now propose to change the roles of p^k and pfjk, that is, to treat p^ as fixed, in virtue of being a function of the known marginal tables, and pfjk as the realization of a random variable Pijk. If we are completely ignorant about the data array (xijk) except for the total sum n, then any possible array of numbers (pijk) with 0 < pijk < 1 and J2ij kPijk = 1 would seem equally plausible as having generated the unknown array of counts. We can represent this, according to Laplace's Principle of Indifference, by a probability distribution with a den sity function that is Uniform for all admissible arrays (p^). Now, if each of the n indi viduals in the (I x J x /Qarray falls into the cell (ij, k) with probability Pijk, which is an element of a random array (Pijk) with a Uniform distribution, then each possible array (xijk) that may result from such a process has the same chance of occurrence. That is, there is a direct connection between the perspective that focuses on our ignorance about the counts xiJk and the perspective that focuses on our ignorance about the datagenerating process. Taking this as a baseline, we can construct a distribution of plausible arrays (pijk) that reflects our ignorance about the true cell probabilities (pfjk) that generated the unknown 4Although there is some relation between the two, "entropy" refers here to a functional of density or probability mass functions and not to entropy in the sense of statistical mechanics and thermodynamics. For the relation, see Jaynes (1957). There are several attempts at giving this principle a general axiomatic foundation, for example, Jaynes (1957), Vasicek (1980), and Cziszar (1991). This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 80 Martin Elff, Thomas Gschwend, and Ron J. Johnston data (Xijk)?an ignorance that is only reduced by the information contained in the margina tables (riij.) =should (52kxijk), = (j2j distribution xijk), and (n.jk)in the = and Such a distribution be as similar(/!/.*) to the Uniform described recovered by the EL previous paragraph as possible under the restriction that its mean is given by the estimates produced by the El method used. The Uniform distribution under consideration is a special case of a Dirichlet distribution, that is, a Dirichlet distribution with all shape parameters equal to one. Now, if we select a Dirichlet distribution that maximizes entropy under the constraint that its mean is equal to the estimates obtained from an El procedure, then we have the distribution that is, under these constraints, the most similar to the Uniform distribution in terms of the KullbackLeibler criterion for the similarity of distributions.5 Informally speaking, the maximum entropy criterion leads here to the "flattest," that is least informative distribution with the given mean. The selection of such an entropymaximizing Dirichlet distribution is possible since by its mean a Dirichlet distribution is only partially specified: If a multidimensional random variable (Pijk) has a Dirichlet distribution with parameters 0^, its components have ex pectations Kjjk : = E(Pijk) ? where 0o : = Ylr,s,t ? t Therefore, each parameter 0^ of a Dirichlet distribution can be decomposed into a mean parameter nijk and a common scale parameter 0O by 0^ = nijk%, so that an entropymaximizing Dirichlet distribution with mean niJk fixed at ptjk can be identified by maximizing Hv(9) = ^mr(0o/%)  lnr(0o) + (Onij,k  IJK)ty(%)  ^ (%pijk ~ lM%Pijk) ij,k (19) for 0O, where v/( ) is the digamma function (Abramovitz and Stegun 1964, 258).6 Since the family of Dirichlet distributions is a multivariate generalization of the family of Beta distributions, we can use this family of distributions to illustrate what finding a maximum entropy distribution entails. A Beta distribution is usually characterized by its two shape parameters, which we call here (\>i and <$>2. The mean of this distribution then is 71: = ^iA^i + ^2)5 so if we define 0O : = cj>t H fy2, the shape parameters can be reex pressed as = Qon and (J)2 = 0o(l ? n) and the variance as n(l  n)/(l + 0O). Figures 1 and 2 depict how a Beta distribution will look like for n fixed at 0.5 and 0.2, respectively, for various values of 0O below, above, and equal to 0MaxEnt> where 0MaxEnt denotes the value of 0O for which the entropy of the Beta distribution is maximal. As Fig. 1 shows, the Uniform distribution over the interval [0,1] is a special case of a Beta distribution: the Beta distribution with maximal entropy subject to the constraint that the expectation is equal to 0.5. Both figures indicate that, irrespective of the value of the expectation of the distribution, values of 0o above 0MaxEnt lead to singlepeaked densities, where the peak is the higher the larger 0O is. However, as 0O gets smaller, the density function puts more and more weight on values around zero and one. Since the variance approaches a supremum of 7i(l  71) as 0O approaches zero, the variance of a Beta density is not a good measure of uncertainty. Conversely, since the weight of the density is most evenly distributed if the 0O attains the entropymaximizing value, the entropy seems to be a much better measure of uncertainty if the expectation n is fixed. If each of the n individuals in the (I x J x ?)array falls into the cell (/, j, k) with probability Pijk, where Pijk itself is part of a random array with a Dirichlet distribution with 5For details see Appendix (Section A.l) to this paper on the Political Analysis Web site. ^or a formal proof of the validity of this formula, see Appendix (Section B.2) at the Political Analysis Web site. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 81 Uncertainty in Ecological Inference E(P) = 0.5, e > eM  " _ "MaxEnt e = 21" 0MaxE e = 24 eMay. / / / / / / E(P) = o.5, e<eM  9  9MaxEnt   e = 2"1eMi e = 24 eM. \ \ \ \ \ \ o h ,111 0.0 0.2 0.4 0.6 0.8 1.0 l1111r 0.0 0.2 0.4 0.6 0.8 1.0 Fig. 1 Density of Beta distributions with mean held fixed at parameter 0O. parameters 0^ = %iZijk, then the distribution of the possible resulting arrays is a mix ture of a multinomial distribution with a Dirichlet distribution, a compoundmultinomial or Dirichletmultinomial distribution (Mosimann 1962; Hoadley 1969). Just as the family of Dirichlet distributions is a multivariate generalization of the family of Beta distri butions, the family of Dirichletmultinomial distributions is a multivariate generalization of the family of Betabinomial distributions, which has been used to model over dispersed proportions and success counts (Skellam 1948; Crowder 1978; Prentice 1986). In fact, if an array of random variables (Pijk) has a joint Dirichlet distribution with parameter array (6^), each component of the array has a Beta distribution with parameters 0o fyijk and 0O(1  nyk) and each component of the corresponding array of counts (x^) with a Dirichletmultinomial distribution has a Betabinomial distribution with parameters Qofyjjk and 0O(1  %ijk) (Mosimann 1962), where, as before, 0o = k 0//*. Now, the counts in each cell have expectation mZyk and variance wi^Q  TC^Xn f 0o)/(l + 0o) That is, the E(P) = 0.2, 6>0MaxEnt=4.35 ? e = 21 E(P) = 0.2, G<eMaxEnt=4.35  ? _ ?MaxEnt   e = 21eMa e = 2^ eMa i i i i r 0.0 0.2 0.4 0.6 0.8 1.0 Fig. 2 Density of Beta distributions with mean held fixed at parameter 0O. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 82 Martin Elff, Thomas Gschwend, and Ron J. Johnston expectation is the same as if the counts had a multinomial distribution with cell probabil ities nijk, whereas the variance differs from the variance of such a multinomial distribution by the factor (n + 60)/(l + 60). Although proportions XiJk/n of counts Xijk with binomial distribution are asymptotically normal and although arrays of proportions X^n of arrays of counts Xijk with multinomial distribution are asymptotically multivariate normal as n approaches infinity, it will be a mistake to assume asymptotic normality in the case of counts that have a Betabinomial or Dirichletmultinomial distribution, respectively: The distribution of the proportions will converge to a Beta distribution or Dirichlet distribution, respectively, which can have a shape quite dissimilar to the normal distribution if it has maximal entropy. The above considerations make clear that one cannot assume that the asymptotic dis tribution obtained from an El procedure is normal. Therefore, one should not use the normality assumption to construct confidence or prediction intervals based on the standard errors of the estimates. Rather, we propose to use the quantile function of the Beta distri bution to construct approximate credibility intervals for the cell probabilities piJk and the quantile function of the Betabinomial distribution to construct approximate prediction intervals for the cell counts xiJk. If the counts in cell (i, j, k) have a Betabinomial distribution with parameters 0orc,y* and 0O(1  tc,^), 95% prediction intervals would be, for example, delimited by F^(0.025; 0O, ft/,*) and Fj^J(0.975; 0O, n^) + 1, where ^Bb(a;9o,7r.//fc) : =sup{jc: FBb;%,nijk(x) < ol} is the quantile function and FBb(x;%, nijk) : = X)*=o/Bb(*; 9o,rc,yj0 *s tne cumulative distribution function of the Betabinomial distribution with parameters Oo^y* and 0O(1  niJk). That is, under the assumption that the counts in cell (/, j, k) have this Betabinomial distribution, the proba bility that the counts are in these intervals will be 95%.7 Since the reasoning behind this procedure is rather heuristic than justifiable in terms of theoretical statistics or probability theory, it seems necessary to assess its performance by way of a simulation study. Therefore, we conducted a systematic simulation experiment in which we vary the size of the array of counts (x^) and its total sum n. For each considered array size / x J x AT and each "population size" n, we (1) generated 2000 arrays of random numbers (x$) with r  1, ..., 2000, (2) computed the marginal tables (njj)), (? *)> and (nS), and (3) used the JohnstonPattie procedure to generate estimates of cell probabilities (Pijk) We then (4) constructed prediction intervals for the cell counts based on the as sumption of the JohnstonPattie model that the cell counts have, jointly, a multinomial distribution and, individually, a binomial distribution with success probability pjj? and (5) prediction intervals based on the procedure proposed in the present section. We then (6) recorded whether the counts x^ are covered by the respective prediction intervals, that is, whether they are inside the intervals. The random counts are generated such that each array of counts (x^k) that sums to n has the same chance of occurrence. Thus, the generated arrays of counts are a simple random sample from all such arrays and the distribution of the generated counts represents the initial ignorance about the interior of the black box before a procedure of El is applied. The simulation results may also be generalized such that they are representative for the average performance with respect to all possible interiors of the black box. By recording the performance of prediction intervals both based on assump tions of an El procedure and based on the secondstage maximum entropy procedure, we are able, first, to demonstrate the consequences of the indeterminacy that besets El and, second, to show the degree to which our proposed secondstage maximum entropy 7For details about the computation of Betabinomial cumulative probability and quantile functions see Appendix (Section B.4) on the Political Analysis Web site. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 83 Uncertainty in Ecological Inference Table 1 Simulation study of coverage of true cell counts and true cell probabilities after 2000 replications Population size Array size 100,000 10,000,000 (a) Effective coverage by prediction 3 x 3 x 50 18.4 2.1 intervals based on the assumption of 7 x 7 x 50 28.0 2.6 a multinomial distribution 3 x 3 x 200 33.0 3.3 7 x 7 x 50 54.1 6.0 (b) Effective coverage by prediction 3 x 3 x 50 95.5 96.3 intervals based on the secondstage 7 x 7 x 50 93.1 94.1 maximum entropy method 3 x 3 x 200 95.0 96.4 7 x 7 x 200 88.5 93.8 procedure improves over a "naive" application of El and r uncertainty associated with El procedures. Table 1 presents the results of our simulation study rega of 95% prediction intervals of these two types with respe the respective arrays. Since the distribution from wh symmetric with respect to the cells in the array, any cell set of counts in the array as any other cell. For convenien (1, 1, 1). Panel (a) of Table 1 reports the coverage per intervals based on the assumption that the counts have a correctly specified cell probabilities, whereas panel (b) re of prediction intervals constructed based on the secondst A comparison of the two panels (a) and (b) in Table intervals are a large improvement in comparison to naiv vals. Panel (a) shows that the effective coverage of multi diction intervals that cover the true counts decreases with population size n is 10,000, the effective coverage may r falls short from the nominal coverage of 95%. But if the undercoverage of the naive prediction intervals is disastr are the true counts inside a 95% prediction interval. This inconsistency of the estimates of the cell probabilities: Th suppose that the standard deviation of xijk, conditional on creases with the population size n only proportional to y/n prediction intervals will decrease relative to the range of delimited by zero and n) proportional to 1 /yfn. Due to th simulation experiment the cell probabilities will almost su root mean square error of predictions based on the cell es incurred by misspecification. As n grows large, the effect of will dominate the random variation of the cell counts xijk In contrast, the maximum entropy Dirichletmultinom show an effective coverage quite close to their nominal l Table 1. That the prediction intervals based on the secondstill differ from their nominal level may have several rea be a consequence of simulation error, in which case these number of replications approaches infinity. However, thes This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 84 Martin Elff, Thomas Gschwend, and Ron J. Johnston our method is just an approximation. Our proposed method so far only takes into account the consequences of inferential indeterminacy but not the consequences of sampli variability. It takes into account that the true cell probabilities and true expected cell counts cannot be known completely even if n approaches infinity. It does not, however take into account the fact that the cell probabilities are estimated on the basis of a finite n. I may thus be possible to improve on the coverage performance of the prediction intervals a finiten correction could be applied. However, this is beyond the scope of this paper sin it is mainly concerned with the consequences of inferential indeterminacy. Another reas is that exact identity between effective coverage and nominal coverage would only be achievable if the counts were continuous and not discrete. With finite n, discrete values m just be too coarse. For example, for a population of size 100,000 and for 7 x 7 x 200 9800 cells, the actual coverage of the cell counts falls clearly short. That may be caused b the fact that in this setting, the average counts per cell are less than four which is clear below 100. Therefore, it will be almost impossible to obtain exact percentiles for su counts. Nevertheless, even if we admit that the proposed method is only an approximation it works better than prediction intervals that rely on the identifiability assumptions of t El procedure of Johnston and Pattie to hold. 4 Accounting for Sampling Variability In the previous section, we considered the uncertainty about estimates that comes from th unavailability of the complete data. The observed data, however, were assumed to b populationlevel summaries. Probabilistic methods are employed in the previous sections first, to take into account that even populationlevel counts are the outcome of som stochastic datagenerating process and, second, to take into account the uncertainty abou estimates based on these counts. If, however, at least one of the available marginal tables not a populationlevel summary but comes from a sample, for example, from a survey sample with data on ethnic identity and voting behavior, another level of uncertainty added. For example, if instead of a populationlevel crosstabulation of voting behavior and ethnic group membership only a crosstabulation xtl=(mij) from a sample is available and the true counts in the array of combinations of ethnic group membership (with categories / = 1, ..., /), voting behavior (with categories j = 1, ..., J), and voting district (with running numbers k = 1, ..., K) are xijk, then (provided that we have a simple random sample) the array of counts m,y has a multinomial distribution with cell probabilities qtj = njjJn, where = Ylkxijk> Eor given marginal tables ni=(n.jk), *l2=(ni.k), and 1*3=(n^.), the method proposed in the previous section leads to one set of parameters (0^)= 0(tti,ri2,n3) of a Dirichletmultinomial model of the counts xijk in the array. But for a given sample crosstabulation m^, there is more than one possible multinomial distribution with cell probabilities qtj and thus more than one possible marginal populationlevel table ntj. from which this sample may have been drawn. That is, there is more than one set of parameters to be considered for modeling the distribution of the unknown counts in the array (0,y*)=<Knl.n2,n3). Of course, given a sample crosstabulation, not all possible (/ x J) arrays n^. with 5^. ny. = n are equally plausible candidates. Rather, the plausibility of these candidates is adequately expressed by the values fi^3\m)=?r(^=n^\^l=Vti) of the probability mass function of the conditional distribution of 9t3given the sample table (Afly)=SDt. Now, since (Ny.) is an unobserved random variable in this perspective, it is no longer possible to simply construct prediction intervals based on a quantile function This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms Uncertainty in Ecological Inference 85 FBlJ (a; Go, n^) whose parameters are computable from fixed, observed marginal tables tt2, and tt3. Rather, an appropriate quantile function is given by F~l (a; 0O, nijk \fm) = ^ (a; 0O, nijk \ nf)f(n^ m ), (20) where the sum is over all possible tables n^=(jiij(t)) (t = 1,...) that satisfy ?V= ?. This sum has no closed form, not the least because there is no function in closed form that leads from the marginal tables t*i=(?.,*), n2=(?/.*), and 1*3=(/i,y.) to 0O and nijk (these parameters have to be computed by numerical methods). Therefore, we propose a bootstrap method to construct the limits of prediction intervals, which involves the following steps: 1. Each replication r = 1, R starts by generating random counts from an approximation of the conditional distribution of (riij) given (m,y). This is done by a doublebootstrap procedure. First, random counts (m\p) from a multinomial distribution with size index m and cell probabilities = m^/m are generated. From these random counts, a second, random set of cell probabilities p^ = m^/m is computed. These are the cell probabilities of a multinomial distribution with size index n from which a second set (n^) of counts is generated. This assures that the (random) marginal tables (n^;) are integers from a multinomial distribution with size index n and reflect the variability of the observed sample (m,y). 2. Based on the marginal tables in]?}), (rii.k), and (n.jk), firststage cell probability estimates pj^ are obtained based on the JohnstonPattie model for each replication r = 1, ..., R. 3. After setting = p^ values, 0^ are determined for each r such that the Dirichlet distribution with parameters 0^ = ^n\jk nas maximal entropy for given tt^. 4. Random numbers p~k from the Dirichlet distribution with parameters 0^ are generated for each r. (r) 5. For each r, random counts jtj^ from a multinomial distribution with probability parameters p^k are generated. For each r, the random array (jcj^) has thus a Dirichletmultinomial distribution, however, with different parameters 0^ for each r. After R replications, the predictions of the unknown cell counts xijk are given by the averages R~l Y2r xljk ?ftne random counts for all ij, and k, and the limits of the prediction interval for each cell count xtjk are given by the respective quantiles of the random counts x[jk. In the next section, we demonstrate the application of this procedure to the recon struction of percentages of splitticket votes in the 1996 General Election of New Zealand. 5 A RealWorld Example: SplitTicket Voting in New Zealand The term "splitticket voting" is a pattern of voting behavior that can emerge when voters have the opportunity to cast several votes on the same occasion. For example, American citizens have the opportunity to cast a vote both for a candidate who runs for the Presi dency and for a candidate for the House of Representatives. In parallel and mixedmember electoral systems, voters may cast two votes in general elections, one with which they can choose the candidate to represent the voting district they live in and one with which they can choose which party they want to support for the proportional tier of the electoral system. Variants of mixedmember electoral systems can be found, for example, in leg islative elections of Bolivia, Germany, New Zealand, and Venezuela; parallel voting sys tems can be found in general elections, for example, in Japan, Mexico, and South Korea. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 86 Martin Elff, Thomas Gschwend, and Ron J. Johnston Strategic voting accounts emphasize the effect of "Duverger's Law" on the amount of splitticket voting: Voters will give their most preferred party their list vote but, if they expect that the district candidate of this party to have only little chances of electoral success, restrict their choice on those candidates they deem to be potentially successful, thus not to "waste their vote." Often, available empirical data do not suffice to test such accounts. Either there are only aggregate data on list and candidate votes in individual voting districts available or the survey data, if they are available, are too sparse with regard to candidate votelist vote combinations in individual voting districts. Therefore, exam ining splitticket voting is a typical field of application for El (e.g., Burden and Kimball 1998; Gschwend, Johnston, and Pattie 2003; Benoit, Laver, and Gianetti 2004). There are, however, some rare occasions where official districtlevel data are available not only on list vote and candidate vote results but also on the numbers of straight and splitticket votes. One of these occasions is the 1996 General Election of New Zealand. This gives us the opportunity to examine the performance of the methods developed in the two preceding sections in a "reallife" setting. Three sorts of data are available on the 1996 General Election of New Zealand: (1) official data on electoral results for party lists and for party candidates and independent candidates for each of the 67 voting districts, (2) official data on total numbers of straight ticket and splitticket votes for each of the voting districts, and (3) an 8 x 8 table o combinations of list and candidate votes from a nationlevel survey sample (Levine and Roberts 1997; Johnston and Pattie 2000). That is, whereas the districtlevel aggregates tii=(n.Jk) and tt2=(/i,.*) are observed, the nationlevel aggregate tt3=(H,;/.) is not but only a sample crosstabulation m=(m#). To make an approximately valid El about the level of splitticket voting in the individual voting districts, we thus need the bootstrap method developed in the preceding section. In the context of research on splitticket voting, one is usually not interested in all counts xijk in the array made of candidate votes, list votes, and voting districts. Rather, one is interested in the proportion of splitticket and straightticket votes in the individual voting districts. This additional complication, however, can be tackled in a straightforward manner. The bootstrap method of the preceding section produces random counts x^ (r = 1, ...,/?) from which point and interval predictions of the true counts xijk are computed. Now, based on Jr) _ 2^i=jXijk * Ar) _ LiftXjjk r?n /straight,*  ^ (r) dnu /split,* ~~ ^ (r) > Z^iJ Xijk Xijk we obtain random proportions of straightticket and splitticket votes.8 The averages r Sr/straight,* an<* h Sr/spiit,* can men serve as a Pomt prediction of the proportions of straightticket and splitticket votes, whereas the simulated quantiles can serve as limits of prediction intervals. Figure 3 shows a comparison of actual against predicted percentages of ticket splitters in the voting districts of New Zealand along with 95% prediction intervals.9 As can be seen 8A further note on notation: The expression =y xijk refers to the sum of all elements xijk in unit (voting district) k for which the first index (e.g., the vote for the candidate of party 0 equals the second index (e.g., the vote for the list of party;'). The expression xijk refers to the sum of all elements in which the first and the second index differ. 9Rather than showing an ordinary scatter plot of predicted against actual percentages, this plot contains a dot plot of predicted and actual percentages against the individual voting districts (sorted by the predicted percentages) to make it easier to discern the prediction intervals of cases with similar predicted or actual percentages of ticket splitters. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms Uncertainty in Ecological Inference 87 o Predicted percentages True percentages Prediction intervals Voting district, sorted by predicted percentage of ticketsplitters Fig. 3 Application of the secondstage maximum entropy approach to splitticket voting in the 1996 General Election of New Zealand: true and predicted splitticket percentages with prediction intervals based on bootstrap percentiles, after R = 2000 replications. The coverage of the tru percentages is 95.5%. in this plot, there are three instances in which the actual percentage of ticket splitters lie outside prediction intervals. The total coverage of the actual ticketsplitting percentages b these prediction intervals thus is 95.5%, which is close to their nominal level. This con stitutes a slight overcoverage, but with only 67 voting districts, one can hardly expect to achieve a coverage exactly at nominal level. The application to the case of splitticket voting in the 1996 General Election of New Zealand is another corroboration of the method developed in the preceding sections Although, again, the match between the nominal coverage of the prediction interva and their effective coverage is not perfect, it is close enough to suggest that the methods are, if not a final solution to the problem, a substantial step in the right direction. However, the comparison of predicted and actual percentages of ticket splitting as well as the lengt of the prediction intervals shows that El cannot produce the miracle of delivering pre dictions from mere aggregates that are comparable in quality to estimates and predictions obtained from fully observed data. In the New Zealand application, the predictions o percentages of ticket splitters are in some districts roughly 10 percentage points away from the actual percentages. Also, the length of the prediction intervals amounts to, on average, 15 percentage points. This makes again clear that the basic dilemma of El should lea scholars to take greatest care when they interpret results obtained from an El procedure. 6 Discussion In Sections 3 and 4, we develop a method of accounting for the extra amount of uncertaint associated with estimates and predictions obtained from an El procedure that stems from a problem that we exposed in Section 2, the problem of inferential indeterminacy. Without This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 88 Martin Elff, Thomas Gschwend, and Ron J. Johnston the problem of inferential indeterminacy, it would be possible to model the datageneratin process of the counts xijk as a multinomial distribution. To model the consequences of inferential indeterminacy for the uncertainty about cell probabilities, we use the conjuga family of the multinomial, the family of Dirichlet distributions. To model the consequence for uncertainty about cell counts, we use the family of mixtures of multinomial Dirichlet distributions. The first to consider Dirichletmultinomial mixture distributions for El are Brown and Payne (1986). Their approach consists of an extension of the ecological regression model (12), which allows the conditional probabilities pj\ik to vary in different units k according a Dirichlet distribution with mean parameters E(pj\ik) = and precision parameter 60 According to Brown and Payne (1986), both the mean and the dispersion parameters can be estimated from the marginal tables n.jk and n^, the mean parameters based on gener alized least squares and the precision parameter based on the variance of the residuals of the regression of n.jk on (Rosen et al [2001] consider estimation of this model v MCMC and a simpler model without a Dirichlet mixing distribution estimated via no linear least squares). Since the precision parameter 00 is estimated from the residuals of this regression, this model may account for model departures with respect to those aspec in which model (12) is more restrictive than the JohnstonPattie model (8). But since th precision parameter of the model in Brown and Payne (1986) is estimated based on the observed marginal tables (n,.*) and (n.Jk), the resulting Dirichlet model of the conditiona probabilities can hardly account for those model departures that can be detected only if th full array (jc^) of counts is available. In contrast to the approach of Brown and Payn (1986), we do not try to estimate the precision parameter 0O from the observable margin tables but rather determine its value according to an a priori criterion, the Principle of Maximum Entropy. As our simulation experiments show, the values of the precisio parameter thus determined can capture much of the uncertainty caused by the possibility of undetectable model departures. Therefore, we suggest that any method of constructing prediction intervals for El should lead to intervals at least as large as those based on our proposed method. Otherwise, they cannot account for those undetectable model departures that haunt the confidence in results of El. It seems that our method of accounting for the consequences of modeling indetermi nacy itself rests on a crucial assumption that a priori, without taking into account the information contained in the marginal tables, any possible array of counts may occur wit the same probability. This, however, would be a misunderstanding. The Uniform distri bution reflects the ignorance about the true cell counts that characterizes the point of departure of El problems. It plays a role similar to a "flat" or noninformative prior in Bayesian inference. Using an informative, nonUniform distribution as a reference distr bution may increase the risk of biased predictions unless this reference distribution is sufficiently close to the true distribution of the cell counts. We use a Uniform distribution as reference specifically to avoid such possible bias. If, however, prior information available that can be summarized in a distribution of the cell probabilities, a generalization of the maximum entropy method can be used: One can then, instead of maximizing entropy select a model that minimizes the directed KullbackLeibler information divergence, also known as KullbackLeibler information criterion, relative to this prior distribution.10 10As Kullback (1959) and Good (1963) have pointed out, the Principle of Maximum Entropy is just a specia of the Principle of Minimum Discriminating Information: Choosing the distribution with maximal entropy is equivalent to minimizing the directed KullbackLeibler information divergence relative to a Uniform distribu tion. For details see Appendix (Section A) on the Political Analysis Web site. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms Uncertainty in Ecological Inference 89 As a method to construct a distribution of the cell counts, one may argue that th method developed in Section 3 is deficient insofar as it only corrects consequences of parametric assumptions of El methods that could be bypassed by directly constructing a distribution of the unknown cell counts xijk that, without any prior parametric assump tions, maximizes entropy subject to those constraints that reflect the information contained in the known marginal tables. In principle, such a construction will be straightforward; in practice, however, finding a solution will be infeasible because of its prohibitive compu tational complexity.11 The method developed in Section 3 contains some concepts of Bayesian statistical inference, that is, a probability distribution on the parameters (the cell probabilities) of a multinomial distribution, which is a member of the conjugate to the family of multinomial distributions. But it does not make use of Bayes's theorem. The combination of the JohnstonPattie model with the secondstage maximum entropy con struction may be viewed, at best, as an approximation to a posterior distribution of the cel counts. However, we are not able to show how good an approximation this is except by means of the simulation study, the results of which we report at the end of Section 3. Therefore, a direct Bayesian approach that makes use of the complete set of tools of this technique of statistical inference seems preferable to the approach proposed in this paper. As can be shown,12 a posterior constructed by the straightforward application of Bayes's theorem to a noninformative prior distribution of the cell counts would be of surprisingly simple structure. However, the computation of the posterior probabilities of the counts will be as computationally demanding as a nonparametric maximum entropy model, making a direct Bayesian approach as unfeasible as a nonparametric maximum entropy approach to modeling the distribution of the cell counts. Finally, we want to emphasize that one should not yield to the temptation of using cell counts or cell probabilities as "data" in secondstage regression models. Apart from the conceptual issues involved in fitting models to data predicted from another model and from possible bias incurred by using such secondstage regressions (Herron and Shotts 2003, 2004), conventionally estimated standard errors of a naive secondstage regression model will be highly inaccurate. Also, this problem cannot be cured by an increase in the number of spatial units considered or by the number of cases within the respective spatial units. Our simulation results reported at the end of Section 3 show how much one will be misled if one relies on asymptotic theory in these cases. Therefore, instead of treating estimates obtained from an El as data, one should consider them as parameters of the predictive distribution for unknown data and use this distribution for generating values for multiple imputation (e.g., King et al. 2001). Of course, since the amount of unknown data is quite large relative to the amount of known data, one would need a large number of imputed dat sets in order to get reliable estimates of quantities of interest and their variances. The issues connected with such a use of El results clearly deserve some further research, which is, however, beyond the scope of this paper. 7 Conclusion The point of departure of our paper is a fundamental dilemma of El and of inference in ill posed inverse problems in general: Estimates can be identified only if certain restrictive assumptions are made with respect to the structure of the datagenerating process leading nFor details see Appendix (Section A.4) on the Political Analysis Web site. 12See Appendix (Section C) on the Political Analysis Web site. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 90 Martin Elff, Thomas Gschwend, and Ron J. Johnston to the unknown data one tries to reconstruct. However, these assumptions may not be satisfied by the unknown data, leading to serious bias in the estimates relative to the tr values of the data. This may be called the dilemma of fundamental indeterminacy (Cho and Mansky forthcoming). We tackle this dilemma by proposing a method of delimitin the error one has to expect when the assumptions needed for the identification of a solut are violated by the unknown data. We focus on a special model for El, a version of the model proposed in Johnston and Pattie (2000), because this model requires only relatively mild assumptions as compared those required by ecological regression models. We examine the consequences of possib departures of the model assumptions. We find that prediction intervals based on the mo assumptions are far too narrow and show a very serious undercoverage of the unknow data. The main contribution of our paper, however, is the development of a method for t construction of prediction intervals that are at least approximately correct. The method we propose consists in combining two stages. In the first stage, poi estimates for the unknown data are constructed based on a model that contains certain assumptions inevitable for the identification of the solution. In the second stage, consider the set of all possible solutions that satisfy or do not satisfy these identificatio assumptions. We construct a probability distribution that meets the requirement that expectation is the solution from the first stage but is otherwise as neutral as possible wi respect to other possible firststage solutions. This distribution assigns the weights of possible solutions in terms of values of its density functions as equal as possible, th is, has maximal entropy, subject to the constraints on the expectation of this distributio Results from a simulation experiment show that if the population for which the El is to made is large enough, prediction intervals based on our proposed twostage method are approximately correct. We supplement this simulation experiment with a realworld application: the predic tion of districtlevel percentages of ticket splitting in the 1996 General Election of Ne Zealand based on aggregate data about candidate votes and list votes at the votingdistr level and an 8 x 8 table obtained from a survey sample. This election is a rare opportunit to check the performance of an El procedure against real data: Not only are discreteleve data available on candidate votes and list votes and a sample for candidate votelist vote combinations at the national level but also districtlevel data on the percentages of straig ticket and splitticket votes. Therefore, we are able to compare predicted with actual percentages of splitticket votes for each voting district. Within the context of this app cation, we address another issue that scholars dealing with aggregate data may face: No always are these aggregate data exact summaries of the population but rather a sample th summarizes the population. As a solution for this problem, we propose a combination o our twostage maximum entropy method with bootstrapping from the empirical distrib tion of the sample. Our application of this combination to the New Zealand data shows th it results in prediction intervals with a coverage performance roughly equal to th nominal level: In 95.5% of the New Zealand voting districts lies the actual percent of splitticket votes inside 95% prediction intervals constructed based on our proposed method. The main conclusion of our paper thus is that standard errors and confidence intervals for El problems that do not take into account the fundamental uncertainty associated with any solution to illposed inverse problems may be grossly misleading. However, the consequences of this fundamental uncertainty can be delimited, if not exactly, though approximately. Therefore, despite the problems discussed in this paper, to reject altogether the idea of El on these grounds means throwing out the baby with the bathwater. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms Uncertainty in Ecological Inference 91 References Abramovitz, Milton, and Irene A. Stegun, eds. 1964. Handbook of mathematical functions with formulas, graphs, and mathematical tables. Washington, DC: National Bureau of Standards. Abramson, Paul R., and William Claggett. 1984. Racerelated differences in selfreported and validated turnout. Journal of Politics 46:71938. Benoit, Kenneth, Michael Laver, and Daniela Gianetti. 2004. Multiparty splitticket voting estimation as an ecological inference problem. In Ecological inference: New methodological strategies, ed. Gary King, Ori Rosen, and Martin Tanner, 33350. Cambridge, UK: Cambridge University Press. Brown, Philip J., and Clive D. Payne. 1986. Aggregate data, ecological regression, and voting transitions. Journal of the American Statistical Association 81:45260. Burden, Barry C, and David C. Kimball. 1998. A new approach to the study of ticket splitting. American Political Science Review 92:533^14. Cho, Wendy K. Tarn. 1998. Iff the assumption fits ...: A comment on the King ecological inference solution. Political Analysis 7:14363. Cho, Wendy K. Tarn, and Charles F. Manski. Forthcoming. Crosslevel/ecological inference. In Oxford hand book of political methodology, ed. Janet BoxSteffensmeier, Henry Brady, and David Collier. Oxford, UK: Oxford University Press. Cirincione, C, T. A. Darling, and T. G. O'Rourke. 2000. Assessing South Carolina's congressional districting. Political Geography 19:189211. Crowder, Martin J. 1978. Betabinomial ANOVA for proportions. Applied Statistics 27:347. Cziszar, Imre. 1991. Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems. Annals of Statistics 19:203266. Fienberg, Stephen E., Paul W. Holland, and Yvonne Bishop. 1977. Discrete multivariate analysis: Theory and practice. Cambridge, MA: MIT Press. Fienberg, Stephen E., and Christian P. Robert. 2004. Comment to 'Ecological inference for 2 x 2 tables' by Jon Wakefield. Journal of the Royal Statistical Society: Series A (Statistics in Society) 167:4324. Golan, Amos, George Judge, and Jeffrey M. Perloff. 1996. A maximum entropy approach to recovering in formation from multinomial response data. Journal of the American Statistical Association 91:84153. Good, I. J. 1963. Maximum entropy for hypothesis formulation, especially for multidimensional contingency tables. Annals of Mathematical Statistics 34:91134. Goodman, Leo A. 1953. Ecological regressions and the behavior of individuals. American Sociological Review 18:6634. . 1959. Some alternatives to ecological correlation. American Journal of Sociology 64:61025. Groetsch, Charles W. 1993. Inverse problems in the mathematical sciences. Braunschweig and Wiesbaden: Vieweg. Gschwend, Thomas, Ron Johnston, and Charles Pattie. 2003. Splitticket patterns in mixedmember proportional election systems: Estimates and analyses of their spatial variation at the German federal election, 1998. British Journal of Political Science 33:10927. Herron, Michael C, and Kenneth W. Shotts. 2003. Using ecological inference point estimates as dependent variables in secondstage linear regressions. Political Analysis 11:4464. . 2004. Logical inconsistency in Elbased secondstage regressions. American Journal of Political Science 48:17283. Hoadley, Bruce. 1969. The compound multinomial distribution and Bayesian analysis of categorical data from finite populations. Journal of the American Statistical Association 64:21629. Jaynes, Edwin T. 1957. Information theory and statistical mechanics. Physical Review 106:62030. . 1968. Prior probabilities. IEEE Transactions on Systems Science and Cybernetics 4:22741. Johnston, Ron J., and A. M. Hay. 1983. Voter transition probability estimates: An entropymaximizing approach. European Journal of Political Research 11:40522. Johnston, Ron J., and Charles Pattie. 2000. Ecological inference and entropymaximizing: An alternative estimation procedure for splitticket voting. Political Analysis 8:33345. Judge, George G., Douglas J. Miller, and Wendy K. Tarn Cho. 2004. An information theoretic approach to ecological estimation and inference. In Ecological inference: New methodological strategies, ed. Gary King, Ori Rosen, and Martin Tanner, 16287. Cambridge, UK: Cambridge University Press. King, Gary. 1997. A solution to the ecological inference problem: Reconstructing individual behavior from aggregate data. Princeton: Princeton University Press. . 1998. Unifying political methodology: The likelihood theory of statistical inference. Ann Arbor, MI: Michigan University Press. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms 92 Martin Elff, Thomas Gschwend, and Ron J. Johnston King, Gary, James Honaker, Anne Joseph, and Kenneth Scheve. 2001. Analyzing incomplete political science data: An alternative algorithm for multiple imputation. American Political Science Review 95:4969. King, Gary, Ori Rosen, and Martin A. Tanner. 1999. Binomialbeta hierarchical models for ecological inference. Sociological Methods and Research 28:6190. Kullback, Solomon. 1959. Information theory and statistics. New York: Wiley. Levine, Stephen, and Nigel S. Roberts. 1997. Surveying the snark: Voting behaviour in the 1996 New Zealand general election. In From campaign to coalition: New Zealand's first general election under proportional representation, ed. Jonathan Boston, Stephen Levine, Elizabeth McLeay, and Nigel Roberts, 18397. Palmerston North, NZ: Dunmore Press. Mosimann, James E. 1962. On the compound multinomial distribution, the multivariate Pdistribution, and correlations among proportions. Biometrika 49:6582. Openshaw, S., and P. J. Taylor. 1979. A million or so correlation coefficients: Three experiments on the modifi able areal unit problem. In Statistical methods in the spatial sciences, ed. N. Wrigley, 12744. London: Pion. . 1981. The modifiable areal unit problem. In Quantitative geography: A British view, ed. N. Wrigley and R. J. Bennett, 6070. London: Routledge and Kegan Paul. Prentice, R. L. 1986. Binary regression using an extended betabinomial distribution, with discussion of corre lation induced by covariate measurement errors. Journal of the American Statistical Association 81:3217. Rosen, Ori, Wenxing Jiang, Gary King, and Martin A. Tanner. 2001. Bayesian and frequentist inference for ecological inference: The r x c case. Statistica Neerlandica 55:13456. Shannon, Claude E. 1948. A mathematical theory of communication. Bell System Technical Journal 27:379 423, 62356. Skellam, J. G. 1948. A probability distribution derived from the binomial distribution by regarding the prob ability as variable between the sets of trials. Journal of the Royal Statistical Society. Series B (Methodolog ical) 10:25761. Steel, David G., Eric J. Beh, and Ray L. Chambers. 2004. The information in aggregate data. In Ecological inference: New methodological strategies, ed. Gary King, Ori Rosen, and Martin Tanner, 5168. Cambridge, UK: Cambridge University Press. Uffink, Jos. 1995. Can the maximum entropy principle be explained as a consistency requirement? Studies in History and Philosophy of Modern Physics 26B:22361. Vardi, Y., and D. Lee. 1993. From image deblurring to optimal investments: Maximum likelihood solutions for positive linear inverse problems. Journal of the Royal Statistical Society. Series B (Methodological) 55: 569612. Vasicek, Oldrich Alfonso. 1980. A conditional law of large numbers. Annals of Probability 8:1427. Wakefield, Jon. 2004. Ecological inference for 2 x 2 tables. Journal of the Royal Statistical Society: Series A (Statistics in Society) 167:385426. This content downloaded from 150.216.68.200 on Wed, 21 Sep 2016 01:54:41 UTC All use subject to http://about.jstor.org/terms