I admire Physicists around the world, especially the likes of Richard Feynman – Genius but Goofy. Read almost every book written on Manhattan Project and biographies of most great scientists. Recently I started reading about great mathematicians and wondered why we don’t hear or celebrate this group of scientists as much. Although some of the exploits of the great John Von Neumann around WW II era are legendary, most contributions of a lot of other mathematicians are rarely ever heard.
Andrew Wiles, a British mathematician, solved Fermat’s Last Theorem (There are no whole number solutions to the equation x^n + y^n = z^n when n is greater than 2) in 1990s. This was no trivial matter. The problem was posed by the French Mathematician, Pierre de Fermat in 1637. A number of great mathematicians tried to solve this problem over the past three centuries without much success. Dr. Wiles finally did it and yet he is not as well known among common masses as, say, Dr. Feynman. I think some of this is probably due to the abstract nature of mathematics. For example, F= ma is a simple but very elegant equation that can explain the beauty of Isaac Newton’s work. Similarly, E= mc2 for Einstein’s and Challenger disaster/o-ring explanation for Feynman’s (his Feynman diagrams may not be popular among non-scientists.) So, when I came across Abraham Wald’s work around WW II in Jordan Ellenberg’s book, “How not to be wrong”, I was ecstatic.
Abraham Wald was born in 1902 in what was then Austro-Hungarian Empire. His father was a kosher baker and yet Mr. Wald turned out to be a world class mathematician. Genius comes from all walks of life and unfortunately so does hatred. He was driven out of Europe by Nazis and Dr.Wald ended up in New York as a math professor at Columbia University around WW II era. He was also part of the Statistical Research Group (SRG), similar to Manhattan project, and helped Allies of WW II in their war efforts.
He solved math problems for the military and here is one of many interesting problems Dr. Wald solved during that era and this example has implications at so many levels in so many different fields that it is simply mind boggling!
How should Allied forces armor its fighter planes to better withstand enemy bullets? A simple question but hard to answer. A simpleton might say, “What the heck dude? It is easy. Just put the armor all over the dang plane man.” See, the problem with that is armor makes planes heavier and that makes planes consume more fuel and hard to maneuver. The simpleton’s solution solves one problem but makes the plane ineffective!
As any good scientist would do, Dr. Wald and his team asked for field data (not opinion polls but actual data) of bullet holes on the planes and they got it. The American military provided the following data.
Fuselage had 1.73 bullet holes per square foot;
Fuel System had 1.55 bullet holes per square foot;
Engines had 1.11 bullet holes per square foot; and
The rest of the plane had 1.8 bullet holes per square foot.
American military felt that the answer was obvious and staring at them. Their logic was that the military should armor the parts of the plane where they found most bullet holes (i.e., fuselage).
Dr. Wald asked a simple question: Where are the missing bullet holes? And pointed out that the armor should go where the missing bullet holes are, i.e., engines. He reasoned that Germans sprayed bullets at these planes during dogfights at random and the missing bullet holes must have been on the missing planes. Simply put most planes that got hit on engines never returned. Behavioral Psychologists call it “Survival bias”. A simple but elegant observation! By the way, Dr. Wald provided a comprehensive mathematical solution to the problem but “where are the missing bullet holes” pretty much captures the key thesis.
Dr. Ellenberg further explained the above problem in a much better way in his book. Here is his crystal clear explanation: “If you go to the recovery room at the hospital, you will see a lot more people with bullet holes in their legs than people with bullet holes in their chests. But that’s not because people don’t get shot in the chest; it’s because people who get shot in the chest don’t recover.”
You may say, “Isn’t that nice! But what is the point Mr. Faker-maestro?” Well the point is sometimes we have to worry/think about what is missing more than what is present.
Of course, I am referring to the 2016 Presidential elections of the United States of America. If you are wondering what elections have to do with randomness and uncertainty, well everything!
It is not too often that we get dumped on with a whole lot of data, opinions, polls, potential consequences and a few observations. Elections time is a party time for data hogs. Consensus call for 2016 election was for Mrs. Clinton to win the 2016 Presidential Election in a landslide but Mr. Trump won by a whisker (thanks to Electoral College). Wall Street pundits predicted a total market meltdown if Trump won the election. Markets actually closed up post his win.
Every political troll was shocked to see Donald Trump win and most “experts” started spouting opinions on how the opinion polls got it all wrong. Some newscasters even went so far as to suggest that the final outcome was a result of rage among certain groups of the US population (especially the ones with lower melanin levels and no college degrees) against intellectuals. I am not sure if college degree has anything to do with the election outcome though. Post-election reactions of most pollsters, newscasters, and some ultra-left wing citizens pretty much put the argument for college education to rest.
We evolved from autocracy (One ring rules them all) to Democracy, where 50.1% of population controls the remaining 49.9%, over the past 3,500 years or so. Democracy may not be perfect but it is the next best thing we have for now. Some may say Mrs. Clinton won the popular vote but Mr. Trump won the Electoral College (306 to 232).
I analyzed Nate Silver’s election forecasts (538.com) prior to the elections in 2016 and he had about 2 to 1 odds in favor of Hillary Clinton (most other pollsters’ odds were in the range of 85% to 99% in favor of Clinton) on the Election Day. One has to give credit where it is due and Mr. Silver cautioned his readers that the race could be close. He even wrote an article called, “Trump is just a normal poling error behind Clinton” four days before the big day. The race was too close to call because Clinton only had a 2 to 3 point advantage over Trump and 2 to 3 point polling errors are fairly common. In other words, although Clinton was ahead in polls, her lead was not statistically significant. It becomes even clearer if one looks at leads at the state level. Both Florida and Ohio were a toss-up and yet most pollsters gave these states to Clinton. Mainstream media ignored this important measure of uncertainty so blatantly. Nate’s forecast recognized that uncertainty. It is important to note that I did not root for either of the candidates. I just wanted to understand the data and it was not clear to me how the media could draw such strong conclusions form the same data sets.
Lack of understanding of uncertainty got a lot of people by surprise again (Brexit showed us a glimpse of that.) Although my base case forecast was also in favor of Clinton, the end result did not shock me. The outcome was entirely within the forecast range and not a major shock.
There is a much simpler way to explain the election result. Most democracies are rarely polarized. However, democracies with two party systems are much easier to understand. The US population can be split into 45% – 45% – 10% among Democrats, Republicans, and unaffiliated. The unaffiliated 10% pretty much determines any election. Once in a while that 10% could grow to 20% but the 45% – 45% -10% is a good and simple model. One sided elections happen when majority of the unaffiliated voters go for one party. Here is some background data to support the above point of view.
|Year||Democrat Popular Vote (%)||Republican Popular Vote (%)||Other Popular Vote (%)||Comment|
|1992||43.0%||37.4%||19.6%||Ross Perot took votes away from Bush Sr.|
|1996||49.2%||40.7%||10.1%||Ross Perot took votes away from Bob Dole|
|2000||48.4%||47.9%||3.7%||Bush won Florida|
|2004||48.3%||50.7%||1%||Bush was still riding the WMD wave|
|2008||52.9%||45.7%||1.4%||Obama beat McCain|
|2016||48.2%||46.1%||5.7%||Major candidates were polarizing figures|
Did you notice that whenever “Other” popular vote share went above 2% the election result got all messed up? Unaffiliated voters swing elections. 1992 and 2000 elections were a lot more controversial and 1992 election was especially a crazy one. Ross Perot got 18.9% of the popular vote and helped Bill Clinton win the election. Ross Perot appealed to a lot of Republican voters in both the 1992 and 1996 elections. Bill Clinton was so unpopular among democrats in 1992 that he did not even get all of his base democrat constituency. He only got 43% of overall popular vote and still won the Election.
2000 Presidential elections were a lot closer to 2016 elections in character. A lot of people were sick of Bill Clinton’s escapades in the White House and that created a lot of uncertainty among the unaffiliated voters. Al Gore (I practically invented internet fame) paid for it.
So, what happened in 2016? Almost half of the unaffiliated voters did not like both the major candidates and they voted for other candidates. Was it so hard to notice the dislike among the unaffiliated voters? Not at all! Then why did we miss the forecast? Well, Mr. Trump is a polarizing figure (same applies to Mrs. Clinton as well.) He made sure he got his 45% of votes from the Republican camp by saying some not so pleasant things and so did Mrs. Clinton but “Others” messed it up for Mrs. Clinton. It is very hard to get this minor change (“Others”) in opinion polls as the change is within the polling error range. Also fear of ridicule could have stopped a few people from openly disclosing their support for Mr. Trump. Not a surprise! Just a few thousand votes in key states swung the Electoral votes in favor of Mr. Trump and he won the overall election.
Hillary got her 45% votes from the Democrat camp and almost a third of the unaffiliated votes. Obviously I am simplifying the vote allocation here and it is entirely possible that certain moderate feminist republican voters could have voted for Hillary and vice-versa. But the point is that half of the unaffiliated voters went for other candidates. History tells us that every time the other candidates got more than 2% of the overall popular votes, the election becomes too close to call. There was nothing wrong with the opinion polls. It is just that most pollsters and newscasters simply ignored the polling error range and the margin of lead was not significant enough to warrant a 90% confidence level in favor of Mrs. Clinton. Emotions overruled simple math.
You may say that is all fine and dandy Mr. Faker but how can a person with 46.1% votes win over a person with 48.3% of popular vote. American founding fathers realized a long time ago that popular vote alone may not represent what is best for the country. They introduced Electoral College system to temper down densely populated areas overpowering countryside. Here is a simple way to explain the process from a video game perspective. Imagine a game with 50 levels (50 States) and the winner is awarded certain number of medals (Electoral College Votes) for each level depending on the difficulty of that level. This is where it gets complicated. The winner of each level is determined by the number of points (popular votes) scored by the player in that level. The person with highest number of points (popular votes) in that level wins it and gets the predetermined medals. Are you with me so far… well here is the rub. Some levels may allow you to score a lot more points than the other levels but the winner of any level only gets the predetermined number of medals allocated to that level (i.e., even if the winner gets 100% of the points in that level, the winner can’t carry over those points to other levels.) Capisce? The idea is that the winner of the game has to be good enough at each and every level.
This is where Mr. Trump played the game well. He barely managed to get enough points to win each level (he barely won most of his levels and lost big in some levels.) That did not matter as he managed to win the most number of levels and that gave him most medals (Electoral College votes.) Mrs. Clinton on the other hand won some of her levels with massive points lead and lost close battles in most levels (i.e., she did not win enough total medals.) For example, Mrs. Clinton lost Michigan by ~11,500 votes (almost 5 million votes were cast) and it did not matter. Mr. Trump got the entire 16 Electoral College votes from Michigan. Similar story in Wisconsin, Pennsylvania, and Florida. All went to Trump. On the flip side, Mr. Trump lost by a landslide in California (Mrs. Clinton won by a whopping 3.5 million vote lead and she still only got 55 Electoral College Votes.)
Here is an interesting point to highlight about the uncertainty we talked about earlier. Mr. Trump won the state of Utah with 45.9% of the popular vote vs Mrs. Clinton’s 27.8%. But here is the kicker, an independent candidate named Mr. McMullin (native son of Utah) won 21% of the state’s popular vote. The “Other” got Clinton. If you are wondering what happened in 2012, well Mr. Romney (Utah native) got 93.1% of popular vote. It is interesting to note that almost 55% of the folks from the Beehive state did not care for Mr. Trump and he still won. As you can see, majority of the unaffiliated voters (Others) in the country did not care for either of the main candidates. If not for “Others” Mrs. Clinton would have received an additional 75 electoral votes (Florida, Michigan, Pennsylvania and Wisconsin) with ease and that would have flipped the current Electoral College votes in her favor (and that would have been the consensus call of landslide victory for her.) In other words, uncertainty ruled the roost on the Election day.
One may say majority should rule and these medals for each level do not make sense. I will not argue for or against Electoral College system but only say that both players and supporters knew the rules of the game well and these rules have been in place for at least two centuries.
2016 election was a close one to call but the end result was hardly shocking. Key takeaway from all of this is that don’t ignore uncertainty. It is perfectly fine to say, “I can’t call this one with much confidence as the uncertainty around the forecast is large enough to Trump even Mrs. Clinton.”