Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
What, then, is so darn unique about page five? While the format and language of the “Explanation of Estimative Language” page (hereinafter the "EEL") has undergone some changes (for the better) over the last four publicly released National Intelligence Estimates (NIEs), all of the estimates that contain such a page make the same three key points:
First, the NIE is…well…an estimate. The authors intend this to be a probabilistic judgment, not a statement of "facts". This may seem obvious but, to many casual readers, there may still be this lingering impression that the CIA, NSA and the other 14 agencies that make up the National Security Community are omniscient. Sorry, not the case and the authors of the NIEs at the National Intelligence Council (NIC) want us to know it.
Second, there is a discussion of
Estimates of Likelihood.
Specifically, this section talks about what the intelligence community commonly calls
Words of Estimative Probability (WEPs -- after the
Sherman Kent article of the same name) and what linguistics professionals usually refer to as
Verbal Uncertainty Expressions (
Thanks, Rachel!).
These are words, such as "likely", "probably", or "almost certainly", that convey a sense of probability without coming right out and saying “60%” or whatever.
Noted
MIT scholar
Michael Schrage came out quite forcefully against this type of estimative language in
a Washington Post editorial in 2005. In the same article he spoke very favorably of using percentages and
Bayesian statistical methods to get them. Despite this kind of criticism, the NIC , in the early versions of the EEL page, noted that, “Assigning precise numerical ratings to such judgments would imply more rigor than we intended”. While this language was dropped in the Iran NIE (probably due to space constraints), it likely continues to represent the NICs position.
Regardless of its desire to avoid numbers, the NIC still effectively benchmarks its WEPs in two ways.
First, it makes it clear that words such as "probably", "likely", "very likely" and "almost certainly" indicate a greater than even chance (above 50%) while words like "unlikely" and "remote" indicate a less than even chance (below 50%).
In addition, the NIC also provides a handy scale that, while it is devoid of numbers, clearly rank orders the WEPS in regular increments.
While the rank ordering is more important than the actual increments,
early versions have five increments implying roughly 20% intervals for each word.
The most recent version in the
Iran NIE has seven intervals (see the chart below) implying intervals of approximately 14%.
The EEL page also identifies the language the authors will use for improbable but potentially important events.
These words and phrases include such old standards as "possible/possibly", "may", and "might" and phrases such as "we cannot dismiss"
and "we cannot rule out".
I intend to write quite a bit about WEPs later on but one point is absolutely clear:
This move towards consistency in the use of language is an incredibly positive step forward but the “poets” in the IC have only been defeated, not routed.
Kent defined poets as the type of analysts who “… appear to believe the most a writer can achieve when working in a speculative area of human affairs is communication in only the broadest general sense. If he gets the wrong message across or no message at all-well, that is life.”
There has been, as we will see in later posts in this series, either a real hesitancy or a real lack of understanding of the value of consistent terminology on the part of many analysts in the intelligence community.
Consistent terminology, however, is something that decisionmakers have been requesting from intelligence professionals for decades.
Mercyhurst alumna, Jen Wozny, wrote a wonderful thesis on the topic (currently you can only obtain it through
inter-library loan with the Hammermill Library at Mercyhurst), exploring what over 40 decisionmakers said they wanted from intelligence.
One of the key requests, of course, was consistent terminology.
I consider it likely that the potential for broader distribution brought on by the recent Congressional requests and the public scrutiny of these latest NIEs essentially forced the Intelligence Community to adopt the more or less consistent series of terms described above.
While it may seem ludicrous to many (especially in the business or scientific communities) that this was a real debate in the intelligence community, it was and, based on the differences between what the EEL page says and what was actually done (which will make up the bulk of the remaining posts in this series), it still is.
Third and finally, the EEL page explains what the NIC means when it talks about “confidence in assessments”. This concept is difficult to explain to most people and the NIC has not been very helpful with their brief discussion of the concept.
Confidence in an assessment is a very different thing than the assessment itself. Imagine two analysts working on the same problem. One is young, inexperienced, working on what is generally considered a tough problem on a tight time schedule. He is unfamiliar with a number of key sources and cannot adequately judge the reliability of the ones he does have. When pressed to make an estimate regarding this problem, he states that he thinks that “X is likely to happen”.
The second analyst is a seasoned analyst with adequate time to think about the problem and considerable experience in the subject in question. He knows where all the sources are and knows which ones are good and which ones are to be taken with a large grain of salt. He, too, states that he thinks, “X is likely to happen.” Both analysts have given the same assessment of the same problem. The level of confidence of the first analyst is likely much lower than the level of confidence of the second analyst, however.
The important thing to note is that the analyst is expressing confidence in his probabilistic assessment. In the first case the young analyst is essentially saying “I think X is likely but for a number of reasons, not the least of which is my own inexperience, I think that this assessment could be way off. If I knew just a little bit more, I could come back to you saying that X is anything from remote to virtually certain.” In the second case, the senior analyst would say, “I think X is likely, but because I know a lot about this problem and how to do analysis, I am fairly comfortable that X is likely and even if I went out and did more research, my estimate would still probably be, “X is likely”.
How does one determine a level of analytic confidence, though? What are the appropriate elements and how are they measured? How do you know when you have crossed the line from low to moderate and the line from moderate to high (the three levels of confidence used on the EEL page)? The discussion above suggests that there are a number of legitimate factors that analysts should consider before making a statement of analytic confidence. The EEL page, strangely, does not see it that way, preferring to tie it only to the quality of the information and the nature of the problem (presumably some sort of scale running from easy to hard).
Recent research by a Mercyhurst grad student (Thanks, Josh!) suggests that a number of things legitimately influence analytic confidence including, among others, subject matter expertise (though it is likely not as important as some people think), time on target, the use of structured methods in the analysis, the degree and way in which analysts collaborate on the product, etc. I suspect that the IC is well aware of at least some of these other elements of analytic confidence (I am hard pressed to imagine, for example, senior officials in the IC stating that the subject matter expertise of their analysts doesn’t matter in their calculation of confidence yet it is not mentioned as an element in the EEL page). I find it disingenuous that they do not list these broader elements that could impact analytic confidence.
Despite these caveats and the minor weaknesses, the EEL implies a fairly comprehensive vision of what I have begun calling a theoretically complete estimate. How might such an estimate appear? Something like, “We estimate that X is likely to happen and our confidence in this assessment is high.” Translated, this might look like, “We are willing to make a rough probabilistic statement (Point 1 in the EEL) indicating that we think alternative X has about a 60-75% chance of occurring (Point 2 in the EEL). Because we have pretty good sources and this problem is not that difficult we are very comfortable that the actual range might be a bit broader but we don't think it is by much (Point 3 in the EEL).”
Ideally, decisionmakers want to know the future with certainty. Despite what the cynics in the IC might say, realistic decisionmakers understand that intelligence professionals deal with unstructured and incomplete data, some of which is deliberately deceptive, concerning difficult and even intractable problems and that certainty, as an intelligence judgment, is impossible. Under these circumstances, the structure outlined in the EEL pages of these recent NIE's seems both reasonable and useful.
Tomorrow: Part 5 -- Enough Exposition! Let’s Get Down To It…
