The Wisdom of Crowds

This is a very important book. James Surowiecki presents a wonderful spectrum of examples of how collective consciousness is superior to individual contributions to that consciousness.In the simplest example, Francis Galton, a British scientist, attended a country fair. He was curious in a weight-guessing contest to see how close the average of all guesses came in assessing the weight of an ox after it had been slaughtered and dressed. The meat was the prize for the closest estimate. He expected that the average of the 787 legible submissions would be considerably off the mark, because many people with no expertise whatsoever were participating in the hopes of winning."Many non-experts competed," Galton wrote... in the scientific journal Nature, "like clerks and others who have no expert knowledge of horses, but who bet on races, guided by newspapers, friends, and their own fancies." The analogy to a democracy, in which people of radically different abilities and interests each get one vote, had suggested itself to Galton immediately. "The average competitor was probably as well fitted for making a just estimate of the dressed weight of the ox as an average voter is of judging the merits of most political issues on which he votes," he wrote. (p. xii)The average of all guesses was 1,197 pounds and the ox weighed 1,198 pounds.Surowiecki notes that many have expressed serious skepticism about the wisdom of groups of people. Notable among these have been Charles Mackay, a Scottish journalist, who wrote about the madness of crowds in 1841; Bernard Baruch, an early 20th century speculator; Henry David Thoreau; and Friedrich Nietsche. Surowiecki acknowledges that there are situations in which crowds demonstrate execrably poor wisdom, as in the crowds who egg on people to jump when poised for suicidal leaps to their death.Countering the skeptics and the dictates of simple logic as stated by Galton, Surowiedki, with a marvelous gift of pattern recognition, expands upon his original example, considering the wisdom of crowds in addressing various types of problems. He demonstrates repeatedly, in diverse situations, how the collective wisdom of groups of people outweighs the wisdom of any of the participants in the group - even the judgments of the most educated and expert participants in these groups.A lovely example is that of the US submarine Scorpion, which disappeared in the Atlantic with no known cause.... Although the navy knew the sub's last reported location, it had no idea what had happened to the Scorpion, and only the vaguest sense of how far it might have traveled after it had last made radio contact. As a result, the area where the navy began searching for the Scorpion was a circle twenty miles wide and many thousands of feet deep. You could not imagine a more hopeless task. The only possible solution, one might have thought, was to track down three or four top experts on submarines and ocean currents, ask them where they thought the Scorpion was, and search there. But as Sherry Sontag and Christopher Drew recount in their book, Blind Man's Bluff, a naval officer named John Craven had a different plan.First, Craven concocted a series of scenarios - alternative explanations for what might have happened to the Scorpion. Then he assembled a team of men with a wide range of knowledge, including mathermaticians, submarine specialists, and salvage men. Instead of asking them to consult with each other to come up with an answer, he asked each of them to offer his bst guess about how likely each of the scenarios was. To keep things interesting, the guesses were in the form of wagers, with bottles of Chivas Regal as prizes. And so Craven's men bet on why the submarine ran into trouble, on its speed as it headed to the ocean bottom, on the steepness of its descent, and so forth.Needless to say, no one of these pieces of information could tell Craven where the Scorpion was. But Craven believed that if he put all the answers together, building a composite picture of how the Scorpion died, he'd end up with a pretty good idea of where it was. And that's exactly what he did. He took all the guesses, and used a formula called Bayes's theorem to estimate the Scorpion's final location. (Bayes's theorem is a way of calculating how new information about an event changes your preexisting expectations of how likely the event was.) when he was done, Craven had what was, roughly speaking, the group's collective estimate of where the submarine was.The location that Craven came up with ws not a spot that any individual member of the group had picked... The final estimate was a genuinely collective judgment that the group as a whole had made, as opposed to representing the individual judgment of the smartest people in it. it was also a genuinely brilliant judgment. Five months after the Scorpion disappeared, a navy ship found it. It was 220 years from where Craven's group had said it would be.(pp. xx-xxi)Cognition problemsSurowiedki examines the unusual situation of the TV show, Who wants to be a millionaire? Contestants could walk away with a million dollars if they correctly answered 15 successive multiple-choice questions of increasing difficulty. Contestants could call upon a trusted outside advisor or on the TV audience (who responded by computerized votes). We might guess that logic would suggest that the smartest person contestants could pick ought to score better than the random collection of people sitting in a TV studio on a weekday afternoon. Well, guess again. The experts answered correctly 65 percent of the time, while the audience was 91 percent on target.Surowiedki reviews many research studies of guesses similar to Galton's original situation, such as estimating beans in a jar or ranks of items by weight. Invariably, the average of group guesses is closer to the actual number than the vast majority of individual guesses. In another example, gamblers' betting odds show that the public is extremely savvy, and those who set the odds are likewise very astute at guessing outcomes of events.What is even more fascinating is that a diverse group that includes experts and non-experts in fields relevant to a problem being addressed will usually do better than a group composed only of experts in the relevant field.He then expands to consider votes by public purchases and sales of shares on the stock market following the space shuttle Challenger disaster of 1986. Within minutes following the disaster, the prices of shares of contractors that could have been involved in causing the disaster dropped: Lockheed (ground support manager); Martin Marietta (manufactured the external fuel tank); Rockwell International (builder of the shuttle and its main engines); and Morton Thiokol (built the booster rocket). By the end of the day, the price of Thiokol had dropped 12 percent, while the other prices had each rebounded from 6 percent to 3 percent drops. It took six months to identify what caused the disaster (O-rings designed by Thiokol), but the wisdom of the stock market crowd was right on target on day 1 of the disaster.Detailed investigations (including scrutiny of possible insider trading) turned up no clues to how the public immediately identified the culprit. Surowiecki believes that the wisdom of crowds explains this unusual finding. He identifies four contributing components to this wisdom: diversity of information and opinions; individual participants' independence in their contribution to the guesses; decentralization of sources of knowledge; and aggregation of the individual opinions into a collective decision.Coordination problemsThe wisdom of groups of people is challenged when they must coordinate the opinions and actions of large numbers of people. There are situations in which it is very difficult to sort out how to achieve the maximum benefits from the inputs of individual group participants, as in factories with many separate steps in production lines. Surowiecki demonstrates that the wisdom of groups of workers can often overcome these potential difficulties in successful collaborations.Cooperation problemsTrusting strangers is something we do all the time, without thought, particularly in commerce. Surowiecki discusses how such trust developed as international commerce developed, and presents various studies on how people will cooperate in market settings.The broader implications of the issues discussed in this book are far-reaching. Surowiecki makes a good case for a trust in democracy as a form of government, if the special interests of lobbying influences can be controlled.What I found of most interest was a hope in the collective wisdom of mankind to deal with the challenges of global heating.A serious deficiency in this book, however, is a total lack of consideration of intuition and collective consciousness - for which there is a major body of substantiating research. These constitute major further potential strengths in the wisdom of groups of people. A prime example is in the collective guesses that led to the location of the Scorpion.Another annoying deficiency of the book is the lack of an index.

Crowds have a bad rap. Our opinion of collective decision making and behavior has been darkened by first-hand observation of events such as the stock market bubble and crash, and by classic works such as Extraordinary Popular Delusions and the Madness of Crowds, and The Crowd: A Study of the Popular Mind. That is unfortunate, argues James Surowiecki because, under the right circumstances, groups "are remarkably intelligent, and are often smarter than the smartest people in them."If nothing else, The Wisdom of Crowds should entertain you. It may not do much more than this if you are already well read in economics, complexity theory, decision analysis, organizational theory, social psychology, prospect theory and other fields. Few readers will have delved into all the relevant areas to any great extent, so most can expect to learn something new, interesting, and quite possibly useful. Surowiecki's wide-ranging gathering of sources to support his argument is a virtue, yet it's also something of a problem. The difficulty of knowing much about all the areas on which he draws makes it easy for him to pick and choose studies and arguments selectively. While many of his points are well made, the way he supports his case sometimes seems one-sided.In evaluating and supporting the idea of the wisdom of crowds, Surowiecki looks at how collective intelligence can be applied to three kinds of problems: Cognition problems (which have definitive solutions), coordination problems, and cooperation problems (which require self-interested agents to work together). The first half of the book sets out the theory, thoroughly and entertainingly illustrated by examples. These include the smarts of the audience on game shows, how to design an excellent search engine, why short selling is a good thing, and how a group finds a lost submarine. The second half of the book applies the ideas to show various ways in which people organize toward common goals in cases such as traffic, science, juries, committees, business organizations, markets, and democracies.Among the main points that may be useful to executives, Surowiecki emphasizes that for the crowd to be wise, it must be characterized by diversity of opinion, independence of members from one another, and a specific kind of decentralization, and there needs to be a good method for aggregating opinions. He stresses that the best collective decisions result from disagreement and contest, not consensus or compromise.While corporations often rely on experts, the book does well at challenging our confidence in expertise as compared to the average of the crowd. In the course of a discussion of the role of independence, we learn that to improve your organization's decision making you should ensure that decisions are made simultaneously rather than one after the other. Finally, I have to second Surowiecki's puzzlement at the apparent lack of interest by companies in using markets (such as decision markets) for corporate strategy and market research.