Stay or Switch…When Intuition Goes Haywire

My friend recently raised an interesting question on intuition.  “What role did intuition play in our career choices over the years”, he asked?  I was intrigued. Although I read Kahneman’s Thinking, Fast and Slow (a good read if you are interested in behavioral psychology) a few years back, I never analyzed my own choices. Cruised through half my career as a faker and lady luck smiled on me more often than not. One important point to note is that just because an outcome turns out to be in our favor without any preparation does not mean it will work out in our favor every time. That is the beauty of randomness. However, one could improve one’s odds dramatically by systematic probabilistic thinking. I remembered Monty Hall Problem and wondered if it could offer any insights into our decision making process.

Monty Hall problem is a well-known math puzzle named after a host of a game show called Let’s Make a Deal. A lot of websites have insightful discussion on this puzzle. I recommend reading comments section of Marilyn Vos Savant’s discussion on this topic to understand how lazy, sexist and downright ugly even well-educated people get when their intuition is challenged.

The puzzle is pretty simple. There are three locked doors in a room and behind two of those locked doors are goats and behind the third door is a sports car (now you know why I am interested in this puzzle.) The game show host asks you to choose a door at random to win the car (your goal is to win the car even if you like goats very much.) Once you make a choice, the host then opens one of the two remaining doors to reveal a goat. That’s not all. He also magnanimously offers you a choice to either stay with your initial pick or switch to the unlocked door. A couple of insights: The host knows where the car is and he offers the option to change your initial pick every game.

Ok then… what’s your choice? Stay or switch?

Most people choose to stay but since it is a puzzle you know the answer has to be switch. Actually, one can win 2/3rd of the time if one chooses to switch. Humbug you say but deep down wonder why?

Most of us analyze the puzzle in the following manner. It is obvious… there is a 1 in 3 chance of winning the car.  So it really does not matter what door I choose but let us say I choose door-1 for kicks and giggles. The host then opens door-2, reveals the goat and offers the choice to switch (to door-3). Now, my intuition comes into play. The little red dude with the pointy tail and pitchfork starts whispering in my right ear, “My mighty intellectual galactic overlord, this is simple! You started out with 1 in 3 chance of winning but now you have a 50-50 chance of winning. Also my liege, the host is jealous of your killer looks. He is setting you up for failure. I humbly beg you not to switch my master.” I consider the red dude to be perceptive as he noticed and acknowledged my killer looks.

And you know life is not that simple. Poof! Another little guy deep in thought with small wings (little guys with big wings freak me out) suddenly appears floating near my left ear and says, “You doofus! Why are you staring at that door? You don’t have X-ray vision.  Did you forget Bayes theorem and all those probability theory formulae you memorized in school.  Don’t rush. Take your time and think it over. Why would the host give you a choice at all if it were to be that obvious? Don’t take this the wrong way but you are not the sharpest pencil in the drawer. You are a schlemiel (I heard this word in a movie called Independence day, circa 1996, and I finally got to use it!) What is more likely: you choosing the correct answer right away or picking the goat? Even if you don’t understand the math and believe the probability to be 50-50, switch your original pick. Here is the logic. If the switch helps, your odds of winning the car improve to 2/3rd; otherwise you are left with a 50-50 chance, no different from your original choice. Either work out the math or simply switch.”

If I am a person of simple intuition, I would go with the red dude (how can I say no– he is very perceptive. Made those nice comments about my looks) and say, “Arigatou Gozaimasu Hall-san but I will stay with my original pick.” You may ask why we do that. That is because we absolutely hate it if we find out after the switch that we had the winner in our first pick.

There was no real skill involved in making the first choice and yet we feel miserable.  This is a very big insight to keep in mind while making our choices and reacting to our decisions after the outcomes have been revealed.

You have to agree that I made a pretty compelling case for the little guy in deep thought.

In any case, here is the simple explanation for why one should switch. We always focus on our odds of winning and when the host discloses the first goat we think our options have gone down from 3 to 2. Miraculously our odds have improved in our favor we think and may even thank imaginary almighty Supreme Being for His generosity or our luck. It is arrogant of us to think that somehow our odds of picking the car went up from 1/3rd to 1/2 simply because the host opened a door that we did not choose in the first place. The odds of finding a goat behind one of the two remaining doors are 100%. The host simply disclosed the obvious and yet we use some basic math/intuition and determine that our odds have improved. Our odds are still 1/3rd if we don’t switch.

Hers is a little math and some visual entertainment for people looking for equations:

Monty Hall-1

Probability of finding the car behind any of the doors is:

P(X) = P(Y) = P(Z) = 1/3

The game begins and the host asks you to choose a door.

Let us say you chose Door X:

Then P(X) = 1/3rd in favor of the car

And P(not X) = 2/3rd (not X means you are picking a Goat). In other words, your odds of picking a goat in your first try are 2/3rd.

Then the host opens Door Z and shows you the goat and offers you a choice to switch. Should you?

Think about this part carefully. This is where the red dude told us the probability has gone from 1/3rd to 1/2 in favor of choosing the car if we just stuck to our initial pick. Did it really? Pitchfork dude is thinking that the car must be behind either Door X or Door Y and hence a 50-50 chance. Well, is it?

At the beginning of the game you knew behind one of the two remaining doors would be a goat for sure (may be even two goats if you picked the car in the first place). All that the game host did is to open that door with the goat. Remember, the host knows where the car is and he will never open that door. So, your odds did not really improve and options go from 3 to 2. Your odds of picking the car are still the same, 1/3rd. So, if you stick with your initial pick, Door X, you would lose (or a win a goat – depends on your perspective).  Now play out all three scenarios.

Scenario-1: Choose door X, host opens Door Z and you do not switch – You lose

Scenario-2: Choose door Y, host opens either X or Z and you do not switch – You Win

Scenario-3: Choose door Z, host opens Door X and you do not switch – You lose

So, you played three times and you won only one time as the math predicted. If the red dude were right you would have won half the time. But you did not. If you are not convinced, play this game a 100 times and your win percentage will be close to 1/3rd and nowhere close to 1/2.

Let us follow the little guy with small wings’ advice (I know he was rude but sometimes reality can be harsh.)

Scenario-1: Choose door X, host opens Door Z and you switch to Door Y – You Win

Scenario-2: Choose door Y, host opens either X or Z and you switch to the other Door – You lose

Scenario-3: Choose door Z, host opens Door X and you switch to Door Y – You Win

How many times did you win after switching in the above game? 2 out of 3 times! The only time you lose is when you pick the car in your first try. Holy Cow or may be Holy Goat! Play the switch scenario a 100 times and your win percentage will be close to 2/3rd. I guarantee it.

Mathematically what you are doing is very simple in the switch case.  Your chance of picking the car in your first try is only 1/3rd. That means there is a 2/3rd chance that the car is behind one of the remaining 2 doors. So, by revealing the other goat, the host is essentially giving you a choice to pick two out of 3 doors with 2/3rd odds to win a car. By switching all you are doing is to move your odds from the initial 1/3rd in favor of P(X) to 2/3rd in favor of P(not X).

Did you notice that you doubled your odds by switching (2/3rd) from staying (1/3rd).  A pessimist may say I still have a 1/3rd chance of losing. But look at it differently: you lowered your odds of losing by half by switching. Whatever rocks your boat!

The process we used to arrive at our answer is called deduction. Here is another interesting fact. Deduction is never wrong, unless of course, the experiment itself is biased.

So, why do so many people get the answer wrong? The host revealing the goat does not improve your odds one bit but provides more information about the two options you did not choose. This is very profound if you think about it. Another important observation you make after discussing this puzzle with your friends is that a lot of them argue with you even after you provide them with the above explanation. They also refuse to do the experiment themselves a 100 times. But they continue to use simple intuition to argue with you to cover up their laziness. If you happen to be in the argumentative group, simply do the experiment one weekend with your friend or spouse or child or a pet and find it for yourself.


A few insights to highlight here:

  • No matter how elegant your theory is if unbiased experimental results don’t agree with your theory, you better change your theory. Unfortunately most people try to change the experiment.
  • If you want to know the truth you have to put in the effort. Don’t just argue.

Here is a slight modification to the puzzle but provides a lot of intuition. What if there are 100 doors instead of 3. The rest of the problem stays the same – One car and 99 goats. What are the odds of you selecting a car in your first pick? 1 in 100 or 1%. That means there is a 99% chance that the car is behind one of the 99 doors that you did not pick. This point is key and very important to absorb. Please take a minute and think through. Now the host reveals 98 of the goats leaving one door unopened and offers the choice to switch. Should you stay with your initial 1% probability pick or switch to 99% probability pick? Revealing 98 goats can be thought of as the host letting you choose all the 99 doors at once. Holy Batman! You get it now. I can tell. Hope so.

Let us generalize this problem then. How do our odds change if there are “n” doors, 1 car, n-1 goats, and the host always leaves two doors unopened (the one you picked and the last door left unopened). Try to work it out yourself but here is the answer: (n-1)/n. This is very easy to derive. What is the probability of picking the car in our first try if n doors are present: 1/n. Then by switching our odds improve to 1-(1/n) = (n-1)/n.  Here is a simple table summarizing the odds with number of doors.

Table-1: Monty Hall Puzzle generalization: 1 car, n doors, n-1 goats, 2 doors left unopened

Number of Doors Odds of winning – Stay Odds of Winning – Switch
3 1/3 (33.3%) 2/3 (66.7%)
4 1/4 (25%) 3/4 (75%)
10 1/10 (10%) 9/10 (90%)
100 1/100 (1%) 99/100 (99%)
1000 1/1000 (0.1%) 999/1000 (99.9%)
175,223,510* 1/175,223,510 (not good) Not relevant

*Odds of winning a lottery are extremely low. Play if you want to have fun but know that the odds are ridiculously low to win.

Life is not that simple you say. Then let us complicate Monty Hall’s problem even further. We can imagine n doors, more than 1 car, etc., but let us start with a simple case. What if Monty Hall puzzle now has 4 doors and the host opens one door at a time and gives you the choice of switching (instead of opening 2 doors right away after you choose). How should we play this game? Should we switch every time he offers the choice or switch and stay or stay and switch? Of course we know by now that switching is always better than staying with our original pick but what are the options within switching? Let us figure it out.

Here is the game set-up: Three goats and a car are behind 4 doors. The game host allows you to switch until only two unopened doors are left.

The game is afoot!

I will provide the answers but please try to figure it out on your own. Better to spend time thinking about cars and goats than watching all the useless stuff on TV.

Scenario-1: Pick one of the four and never switch – Probability of winning the car: 1/4 (25%)

Scenario-2: Pick one followed by the host revealing a goat, we switch, and stay after that – Probability of winning the car: (1/4)*0+(3/4)*(1/2) = 3/8 (37.5%)

Scenario-3: Pick one followed by the host revealing a goat, we stay, and switch after the host reveals another goat – Probability of winning the car: (1/4)*0+(3/4)*1 = 3/4 (75%)

Scenario-4: Pick one followed by host revealing a goat, we switch, host reveals another goat, we switch again – Probability of winning the car: (1/4)*1+(3/4)*(1/2) = 5/8 (62.5%)

Amazing isn’t it. The best strategy is to stay until the last goat is revealed and switch (fairly obvious if you think through this). This strategy improves odds of winning the car from 1/4 to 3/4 (3 times).

The second best strategy is to switch every time. You might have expected this to be the best strategy but think a little and this one also becomes pretty clear. This strategy improves the odds of winning the car by 2.5 times versus staying with original pick.

The third best strategy is to switch and wait. This strategy improves the odds by 1.5 times versus staying with the original pick.


A few insights:

  • Switching helps but making the switch only after the revelation of most choices is the most prudent strategy. Now we know where the saying, “Patience is virtue” comes from.
  • Interestingly, even switching after every choice offers better odds of winning than switching first and staying put. I guess if you are not happy with your current choice, make the switch. Sure, there could be some ramifications in real world with constant switching but remember the available choices have to provide you with that massive upside. In short, it is ok to switch from time to time but the inherent assumption is that you are giving your best every time.

Anyways, you may wonder what the point of all of this is. Well, most of us do not know what we want to do in life. I am not talking about people like Madam Curie or Usain Bolt. I am talking about fakers like me. Normally we pick one out of a few choices that are available to us and hope for the best. And to make matters worse, we stick with the same choice as if it is a crime to change. Conventional wisdom is to stay put. We are brought up that way. We are told “Rolling stone gathers no moss”. Yada yada yada.

We think we make our choices to maximize our respective utility functions, whatever they may be (money, please/impress someone, sports car, goat, nice cafeteria, etc.) But subconsciously our utility functions have been tampered with all our young lives. How many times have we not heard, “one word: plastics” or “Go west, young man, go west”.

Some intellectuals (at least they think they are and usually they call themselves experts) may even display their impressive observational skills by saying, “What is one thing Bill Gates, Larry Ellison, and Mark Zuckerberg have in common?” They usually pause for a brief moment for the effect and with a smirk they say, “All three of them dropped out of school and became Billionaires. Wink-wink!” You get the point. Sounds like a solid observation but they forgot to mention one very important fact and that is Bill Gates and Zuckerberg were precocious kids. Bill Gates spent an abnormal amount of time at a young age coding on a computer when most of his contemporaries never even saw one. Zuckerberg, at the ripe old age of 12, developed a messaging program for his father’s dental office. His programming skills were legendary among his peer group and may be even among kids a lot older. These folks did not succeed because they dropped out of school. They succeeded (even this is a random low probability event) because they were better than their peers in a few aspects (I am doing ex post analysis here) and they found their true passion at an early age. They dropped out of school to pursue their passion and became successful. They did not become successful because they dropped out of school to pursue whatever.  Lady luck played a much bigger role in their success than most people give credit for.

Most of us are not that gifted.  A sobering thought but we have to be pragmatic. Another point to note is that not all kids get that Aha moment by their teens. I suspect most of us get the realization to let go of the fakery may be in our twenties or even thirties.

My point is we don’t really know what we want unless we sit and think hard. Most of us don’t even know how to think systematically without any bias. Even after all of that we may still not be able to pursue certain options due to personal commitments or our priorities may change as we get older. How can then we trust our simple intuition to make important decisions? Let us train our intuition. May be a systematic probabilistic thinking is warranted and Monty Hall puzzle gives us a chance to do that.

Of course, I am not saying we should switch every opportunity we get as we found out in Monty Hall’s puzzle. We have to figure out a way to incorporate those concepts from the puzzle in our decision making process. A lot of things about our life choices are very different compared to Monty Hall’s problem. We probably may never know all the options that are available to us ahead of time. But you don’t have to. Math can help us account for some of those variables and uncertainties. You may say, even if we know all the available choices, they are not equally weighted.  Sure but we can account for that in our math again. Also, it is highly unlikely that we will ever find a “host” who can systematically eliminate all the choices that are not good for us and offer another option to choose again. Imagine your boss playing “Let’s make a deal” with you at the year-end review. You may say I am a dreamer…

Nicola Tesla (smartest man of his generation) died penniless. If only life were that simple and all the problems can be solved with math.


Let us figure out some takeaways from all of this:

  • Don’t simply choose an option because it makes intuitive sense.
  • You may have more choices than you realize.
  • Don’t overestimate your chance of picking the winner in your first try.
  • You should dig a little deeper to better understand the odds stacked against you in making that choice.
  • Don’t take the easy choice.
  • You may meet a lot of people, who can shed some light on some of your available choices including the one you have already chosen. You should pay attention to what these people are saying. This incremental information could help improve your odds.
  • Don’t be afraid to make a change when new options become available to you with better odds.
  • Be open to the idea that your current passion may not be truly yours but drilled into you by your surroundings and upbringing. Even if it’s your own, it is possible that it may change in the future.
  • Embrace uncertainty and don’t be afraid of failure.

The last point is profound. I love this quote from Michael Jordan (retired NBA basketball player), “I’ve missed more than 9,000 shots in my career. I’ve lost almost 300 games. Twenty-six times I’ve been trusted to take the game winning shot and missed. I’ve failed over and over and over again in my life. And that is why I succeed”.

Let us wrap this up with the following:  Even with all the scientific rigor and exotic math we may still end up making wrong choices (in hindsight) in life. In the case of Monty Hall puzzle, we lose 1/3rd of the time even after switching. But if we apply scientific methodology to our analysis and introspection, we can at least understand why we did what we did instead of saying, “I am so unlucky!” or “I forgot to sacrifice a goat to almighty God Ra before the Monty Hall’s game and that’s why I won a goat instead of the car. But I still won something. God Ra acts in mysterious ways. Hail the Benevolent and the Mighty God Ra.”

 

Disclaimer: I have no knowledge of Sun God Ra granting wishes after goat sacrifices.  I also don’t know if he prefers a sheep over a goat.