Methods and Uses

In one effort, we are currently using the methods of Cognitive Task Analysis to perform a cross-comparison of several Air Force Command and Control Systems (Means & Burns, 2003; 2005.) This work is producing "Functional Decomposition Diagrams" for different missions/systems that allow us to relate decision making challenges across these missions/systems. This will be useful in two ways. First, it will help us more clearly relate the decision challenges of our laboratory tasks (card games) to those of real world tasks (warfare). Second, it will help us more clearly relate the decision making challenges of different Air Force missions/systems to one another for integrated Command and Control.

A good book on the complex and time-critical decision making challenges of Air Force operations is by Snook (2000). He tells the story of how two U.S. Air Force F-15 Eagle fighters accidentally shot down two U.S. Army Black Hawk Helicopters over Northern Iraq, killing all twenty-six peacekeepers onboard:

blinded by the squawk
eagle striking his prey hind
kills a friendly hawk

In another effort, the methods of Cognitive Task Analysis and Critical Incident Review have been use to analyze operational challenges in Air Traffic Control. These methods were used to address human performance issues arising from changes to airport layout and control systems - such as runway/tower modifications. The details of this work are documented in paper titled: Naturalistic Decision Making in the Air Traffic Control Tower: Combining Approaches to Support Changes in Procedures.

As an example of how Cognitive Engineering methods can be applied to System Engineering problems, we performed an experiment to investigate human performance in "Bayesian inference". The experiment used a game called Spy TRACS (which is a modified version of Straight TRACS, see Laboratory Tool) in which a player must combine probabilities from multiple sources in order to make diagnoses and decisions. This basic task of "data fusion" (Bayesian inference) is central to military, medical and other domains where people have to "connect the dots". The results show that most people's probability estimates are "conservative", i.e., they fail to extract all the certainty that is available in the data. This cognitive bias is of special concern in high risk and time pressed situations, like military Command and Control, where collecting data can be expensive (and dangerous) and where taking action is time critical.

Based on these experiments, we developed a support system called "Bayesian Boxes" (Burns, 2003; 2004a; 2004b; 2004c; 2005). The system is a visual device that calculates the answer (Bayesian posterior) and at the same time illustrates the reason (Bayesian principle) that underlies this answer. Further experiments have shown that this dual display of "what" and "why" improves people's inferences and intuitions. We think the display of "why" is especially important and that this is the missing ingredient in many support systems that people "refuse to use".

Here is sample problem in Bayesian inference:

Imagine that you are a military commander with limited information about a possible target, called a track. Your job is to identify the track, Friend or Foe, so you can choose a course of action. You have information from two independent sources, e.g., an intelligence source and a surveillance source. The first source reports “Foe”. This source has a success rate of 0.80 in correctly reporting Friend or Foe, so based on this source the chances are 80% that the track is a Foe. The second source reports “Foe”. This source has a success rate of 0.67 in correctly reporting Friend or Foe, so based on this source the chances are 67% that the track is a Foe. With the above information, what do you think are the chances that the track is a Foe?

When asked this question, many people say the chances are somewhere between 67% and 80%. Other people use only the most (or least) reliable source and say the chances are 80% (or 67%). Still other people multiply the two probabilities and say the chances are around 50%. What is the correct answer? To get it, you can use Bayesian Boxes (see left) where Red represents "Foe" and Blue represents "Friend". To use the calculator, move the black hash on the bottom (with your mouse) until the "Prior" probability reads 80% Red (20% Blue) - which are odds of 4:1; then move the black hash on the left until the "Likelihood" reads 67% Red (33% Blue) - which are odds of 2:1.

The "Bayesian Posterior", shown at the top, will read 89% Red (11% Blue) - which are odds of 8:1. And that is the Bayesian answer, i.e., the chances that the track is a Foe are actually 89%. Without the calculator, most people report a Posterior much less than 89%, which means that they are failing to extract as much certainty as they can from the data that they have.

Of course, human judgments in Bayesian inference are not always conservative, and non-conservative biases are also a major concern. Bayesian Boxes can be effective here as well, as discussed in a case study of "authorship attribution" (Burns, 2004c; 2005). Based on these successes, the design of Bayesian Boxes has been extended to address more complex problems. A demo version of the enhanced system appears here (right), where again you set the inputs by moving the black hashes and read the output (posterior) at the top. Other extensions to the system have also been made to address further complexities, like more than two hypotheses (handled with more than two colors) and more than two sources of data (handled with multiple-sequential panels of Bayesian Boxes).

Bonaceto, C., & Burns, K. (2003). Mapping the Mountains: A Survey of Cognitive Engineering Methods and Uses. Proceedings of the 6th Conference on Naturalistic Decision Making.

Bonaceto, C., & Burns, K. (2004). A Roadmap for Cognitive Engineering in Systems Engineering. Proceedings of the 48th Annual Meeting of the Human Factors and Ergonomics Society.

Bonaceto, C., Estes, S., Moertl, P., & Burns, K. (2005). Naturalistic Decision Making in the Air Traffic Control Tower: Combining Approaches to Support Changes in Procedures. Proceedings of the 7th International Conference on Naturalistic Decision Making.

Burns, K. (2003). Dealing with Probabilities: On Improving Inferences with Bayesian Boxes. Proceedings of the 6th Conference on Naturalistic Decision Making.

Burns, K. (2004a). Bayesian Boxes: A Colored Calculator for Picturing Posteriors. Proceedings of the 3rd International Conference on the Theory and Application of Diagrams.

Burns, K. (2004b). Painting Pictures to Augment Advice. Proceedings of the 7th International Conference on Advanced Visual Interfaces.

Burns, K. (2004c). A Bayesian Approach to Detecting a Hoax. Proceedings of the International Conference on Knowledge Engineering and Decision Support.

Burns, K. (2005). Bayesian Inference in Disputed Authorship: A Case Study of Cognitive Errors and A New System for Decision Support. Information Sciences.

Burns, K., & Means, C. D. (2005). Dynamic Diagrams for Time-Sensitive Targeting. Proceedings of the Second Annual Integrated Sensing and Decision Support Workshop.

Means, C. D., & Burns, K. (2003). Framing the Functions: Towards a Taxonomy of Command and Control. Proceedings of the 6th Conference on Naturalistic Decision Making.

Means, C. D., & Burns, K. (2005). Analyzing Decisions and Characterizing Information in C2 Systems. Proceedings of the 10th International Command and Control Research and Technology Symposium.

Snook, S. (2000). Friendly Fire: The Accidental Shootdown of U.S. Black Hawks Over Northern Iraq (Princeton University Press).