Facts are overrated in political and moral debates. They’re fragile and impotent. They don’t do the work expected of them. This is not a rehashing of the well known inability of facts to persuade emotional beings. This is an explanation of why no one should be persuaded by most facts. This is also not a postmodern denial of truth. My point is that facts which are absolutely true can be absolutely irrelevant in overlooked ways. It’s obvious that facts can be irrelevant; everyone knows that. But some facts which seem relevant are not. For example, let’s say you ask me whether or not women face gender bias in grad school admissions. I tell you that hummingbirds can fly backwards. I just gave you an obviously irrelevant fact. But if I told you that data shows female applicants were significantly less likely than male applicants to be accepted to grad school, that would also be irrelevant. How could that fact possibly be irrelevant? It seems like exactly the kind of fact that could help answer the question. But the fact is at the wrong scale; its unit of analysis is wrong, which makes it irrelevant. This requires some explanation.
The term ‘unit of analysis’ comes from the quantitative social sciences. It refers to the specific type of thing a scientist wants to measure and compare. For example, an economist comparing the GDP of Mexico to the GDP of Japan is using ‘country’ as his unit of analysis. A psychologist conducting a controlled experiment is taking ‘treatment group’ as his unit of analysis.
Unit of analysis is not to be confused with the unit of observation, which is the unit described by one’s data. For example, a study may have a unit of observation at the individual level but may have the unit of analysis at the neighborhood level, drawing conclusions on neighborhood characteristics from data collected from individuals.
When two scientists choose two different units of analysis, they may come to apparently opposite conclusions, which are both true. In applied statistics, this is known as Simpson’s paradox. One of the best-known examples of Simpson’s paradox is a study of gender bias among graduate school admissions at University of California, Berkeley. The admission figures for the fall of 1973 showed that men applying to the university were more likely than women to be admitted. But it was found that men were not more likely than women to be admitted by any particular department. This is explained by the fact that women tended to apply to departments with low rates of admission among qualified applicants (such as in the English Department), whereas men tended to apply to departments with high rates of admission among the qualified applicants (such as in engineering and chemistry).
This table shows a simplified version of how that can work:
In this example, the statement ‘men are more likely to be accepted’ is true, taking ‘university’ as the unit of analysis. At the same time, given the same data, the statement ‘women are more likely to be accepted’ is true, taking ‘department’ as the unit of analysis. These are literally alternative facts. Both statements are true, but which is more relevant? That depends on what question the facts were meant to answer. A question about the university as an indivisible whole calls for one correct answer, while a question about the departments calls for the opposite correct answer. So the real question is, what’s the right question? Since nobody applies to the university as an indivisible whole, the right question is about the departments, and so the relevant fact is that women are admitted at a higher rate than men. The fact that women are admitted to the university at a lower rate than men is irrelevant.
Now we see that the question must specify a unit of analysis if we want a coherent answer. The same is true of questions asked in moral and political philosophy. What kind of society is most fair, or most safe, or most just, or most beneficial, or most free? No matter which question one sets out to answer, the question must specify a unit of analysis. Most fair for whom? For people. But for people at which scale? For individuals, or for families, or for neighborhoods, or for genders, or for races? Simpson’s paradox implies that two philosophers who ask the same question with different units of analysis will sometimes come to opposite conclusions.
If one were to make moral judgements, of the admissions at Berkeley for example, her conclusions would depend on the units of analysis which she chooses as her unit of moral agency, and her unit of moral harm. Suppose she takes ‘individual’ as her unit of moral harm. Then she cares about outcomes for individuals, and individual applicants only apply to single departments, so she looks for evidence of departmental treatment of applicants, and finds no bias against women. Or suppose she takes gender as her unit of moral harm. Then she has an array of contradictory units of moral agency from which to choose. Is the whole of the university an agent which produces an outcome (bias against women)? Or are the departments agents which each produce their own outcomes (no bias against women)? Or maybe she decides that a societal system of male bias is the relevant agent, which should be counteracted by a system of localized female bias in the university. Each of these a priori units of analysis leads to different conclusions about the moral character of grad school admissions at Berkeley.
Now if a single moral judgement depends on unit of analysis, then surely a political philosophy which imagines some sort of societal structure that maximizes some moral values (whatever they may be), must also depend on unit of analysis.
As we have seen, apparently conflicting conclusions can all be correct, given different a priori units of analysis. But is there such a thing as a correct unit of analysis? Philosophers have used state, economic class, gender, individual, race, neighborhood, and many other units. And there are practically infinitely many more ways in which it is possible to group people for analysis. The choice of unit is a choice between infinitely many options. I argue elsewhere that ‘individual’ is the only correct unit of moral analysis. But here, my goal is just to show that true facts are irrelevant if their units of analysis are wrong, and since there are so many possible units of analysis, a person with an opinion should have a good reason for choosing one unit over the others.
Citing true facts is child’s play. If you’re interested in moving from facts to Truth, you need a good reason to believe your facts are relevant, which means you need a good reason to believe your questions are relevant.