A Measured Result

A recent experiment on TRACS (Laboratory Tool) investigated human performance in a "Bayesian game", called Spy TRACS, where a player must combine probabilities from multiple sources in order to make diagnoses and decisions. This task of "data fusion" (Bayesian inference)  is central to military, medical and other collaborative domains where people have to "connect the dots" (and the dots are uncertain, which is what makes it hard). Our results (Burns, 2003a; 2003b; 2004) show that most people's probability estimates are "conservative" (i.e., posterior odds too close to 50-50) and that many people are "not even in the ballpark" of the Bayesian answer (i.e., their "posterior" odds are lower than "prior" odds, when in fact they should be higher). In other words, people often fail to extract all the certainty that is available in the data. This cognitive bias is of special concern in high risk and time pressed situations, like military Command and Control, where collecting data can be expensive (and dangerous) and where taking action is time critical. 

A System Design
Based on experiments and analysis of human performance in Bayesian inference (see above), we developed a support system called "Bayesian Boxes" (Burns, 2003a; 2003b; 2004a; 2004b)  to help people improve their intuitive judgments. The system is a visual device that calculates the answer (i.e., Bayesian posterior) and at the same time illustrates the reason (i.e., Bayesian principle) that underlies this answer. Our experiments show that this dual display of "what" and "why" improves people's inferences (answers) and intuitions (reasoning). We think the display of "why" is especially important and that this is the missing ingredient in many support systems that people "refuse to use".
Here is sample problem in Bayesian inference:

Imagine that you are a military commander with limited information about a possible target, called a track. Your job is to identify the track, Friend or Foe, so you can choose a course of action. You have information from two independent sources, e.g., an intelligence source and a surveillance source. The first source reports “Foe”. This source has a success rate of 0.80 in correctly reporting Friend or Foe, so based on this source the chances are 80% that the track is a Foe. The second source reports “Foe”. This source has a success rate of 0.67 in correctly reporting Friend or Foe, so based on this source the chances are 67% that the track is a Foe. With the above information, what do you think are the chances that the track is a Foe?

When asked this question, many people say the chances are somewhere between 67% and 80%. Other people use only the most (or least) reliable source and say the chances are 80% (or 67%). Still other people multiply the two probabilities and say the chances are around 50%. What is the correct answer? To get it, you can use our colored calculator called "Bayesian Boxes" (below), where Red represents "Foe" and Blue represents "Friend". To use the calculator, move the black hash on the bottom (with your mouse) until the "Prior" probability reads 80% Red (20% Blue) - which are odds of 4:1; then move the black hash on the left until the "Likelihood" reads 67% Red (33% Blue) - which are odds of 2:1 . 

 
The "Bayesian Posterior", shown at the top, will read 89% Red (11% Blue) - which are odds of 8:1. And that is the Bayesian answer, i.e., the chances that the track is a Foe are actually 89%. Without the calculator, most people in our tests report a Posterior much less than 89%, which means that they are failing to extract as much certainty as they can from the data that they have. 

Of course, anyone can get the right answer by plugging numbers into this colored calculator. However, our preliminary experiments suggest that exposure to the calculator also improves people's conceptual understanding of Bayesian inference. Based on this finding, we are currently extending and applying the basic design of Bayesian Boxes to more complex problems typically encountered in practical applications, where there are often more than two hypotheses and where there are often more than two sources of data (and where these sources may not be "independent"). The aim of this effort is to develop "support systems" that can help people make better diagnoses and decisions under uncertainty.

 

References

Burns, K. (2003a). Improving Intuition with Bayesian Boxes: A Cognitive Aid for Combining Odds. Abstract in Proceedings of the 6th Conference on Naturalistic Decision Making.

Burns, K. (2003b). Improving Intuition with Bayesian Boxes: On Cognitive Difficulties in Combining Probabilities. Paper in Proceedings of the 6th Conference on Naturalistic Decision Making.  

Burns, K. (2004a). Bayesian Boxes: A Colored Calculator for Picturing Posteriors. Proceedings of the 3rd International Conference on the Theory and Application of Diagrams.

Burns, K. (2004b). Painting Pictures to Augment Advice. Proceedings of the 7th International Conference on Advanced Visual Interfaces.    

 


Home