Six Rules for Effective Forecasting by Paul Saffo

At the beginning of a new year, we always want to look into the future and find out what awaits us. The Internet is full of forecasts and it is very important to be able to evaluate their reliability and effectiveness. In 2007, the Harvard Business Review published an article by Paul Saffo in which he shared his views on effective forecasting. Despite the fact that quite a lot of time has passed since then, it seemed to us that this material has retained its relevance. We bring to your attention its abstract translation.

It is a common belief that expert analysts make forecasts. In fact, prediction is only possible in a world in which events are predetermined and no actions in the present can affect events in the future. This world is made up of myths and superstitions. The one we live in is completely different: little is known, nothing is predetermined, and our actions in the present often influence how events will unfold in significant and unexpected ways. Prediction is based on confidence in the future; Foresight looks at how undercurrents in the present signal possible changes in the direction of companies, societies, or the world at large. Thus, the main goal of forecasting is to identify the entire range of potential development options and opportunities, and not to list a limited set of speculative facts.

Whether a particular forecast actually turns out to be accurate is only part of the picture—even a broken clock is right twice a day. The expert's first task is to uncover uncertainty, because in a world where our actions in the present affect the future, uncertainty is also a potential opportunity.

Unlike prediction, forecast has a certain logic. This is what brings forecasting out of the dark realm of superstition. The analyst must be able to articulate and defend this logic. Moreover, in order to independently assess the quality of a forecast and give due consideration to the opportunities and risks it poses, the user must have a sufficient understanding of the process and logic behind its production. A smart consumer of a forecast is not a gullible observer, but a participant and, above all, a critic. Even after you've separated the analysts from the visionaries and prophets, you still face the challenge of distinguishing good forecasts from bad ones. To this end, Paul Saffo offers a set of common sense rules that will help you evaluate the forecasts presented to you for yourself.

Rule 1: Define the cone of uncertainty

As a decision maker, you must ultimately still rely on your intuition and judgment. In a world of uncertainty, this cannot be avoided. And effective forecasting provides context that feeds your intuition. This deepens your understanding by uncovering missed opportunities and uncovering untested assumptions about expected results. At the same time, it narrows the space within which you make decisions and apply your intuition.

Paul Saffo suggests visualizing this process as a cone of uncertainty. The forecaster's job is to define the cone in a way that allows for the best strategic decisions to be made. The shape of the cone of uncertainty is influenced by many parameters, but the most important is its width, which acts as a measure of uncertainty. Other factors—such as the relationships between elements and the ranking of possible outcomes—also need to be considered when developing a forecast, but determining the width of the cone is a critical first step.

Imagine it's 1997 and the Toyota Prius has just gone on sale in Japan. You are tasked with predicting the future of the hybrid vehicle market in the United States. External factors to consider include oil price movements and consumer attitudes toward the environment, as well as broader factors such as underlying economic trends. Within the cone will be the possible emergence of competing technologies (such as trends in the energy product market) and the level of consumer preference for small cars (such as the Mini). At the edge of the cone will be events that cannot be determined, such as possible terrorist attacks or war in the Middle East.

In general, the cone consists of three types of probabilities - areas that you know everything about and can predict with a high degree of probability; areas that you are aware of but over which your ability to influence is limited; and areas that you cannot define and that are beyond your control.

A narrower forecast cone gives you the illusion that you are in a good position and in control of the situation. A wide cone leaves you with a lot of uncertainty, but it also means there are more possible outcomes for you to take advantage of. Defining the cone broadly at the beginning of the forecasting process maximizes your ability to generate hypotheses about outcomes and possible responses. On the contrary, a cone that is too narrow leaves you open to unpleasant surprises that could have been avoided. Worse, it could cause you to miss out on the most important opportunities on your horizon.

The art of defining the boundaries of a cone lies in the ability to separate the improbable from the completely impossible. A properly defined boundary consists of regions that lie on the edge of plausibility, in which events can conceivably occur, but in which it is inconvenient to even think about. They have a low probability of occurrence (less than 10%) or sometimes their probability cannot be determined at all. Moreover, if the events had occurred, they would have had a disproportionately large impact on the development of the predicted processes. These are a kind of “black swans”. The difficulty with including such events in the forecast is that it is difficult to offer such unlikely and unusual opportunities without losing your audience. The problem—and the core of what makes forecasting difficult—is that human nature is programmed to averse uncertainty. Subconsciously, people, trying to avoid uncertainty, either completely exclude unlikely events from consideration, or try to turn them from uncertainty into certainty, which can lead to significant errors.

A striking example is the problem of the transition of information systems to the year 2000, when half of the predictions said that dramatic events would occur, and the other half were confident that the problem was insignificant and under control. In reality, the Y2K problem meant a low probability of dramatic events, thanks to the careful preparatory work of programmers around the world.

Rule 2: Look for S-curves.

Change rarely unfolds in a straight line. The development of events or trends usually follows the dynamics of an S-curve: changes start slowly and increase gradually and sometimes unnoticeably, and then make a sharp quantitative jump or even explosion, and eventually slow down or even roll back.

The basis of many S-curves that have emerged in the last 50 years is Gordon Moore's Law Curve, a brilliant hypothesis he formulated in 1965. The hypothesis states that the density of circuits on a silicon wafer doubles every 18 months. We can all feel the effects of Moore's Law in the extravagant surprises that the digital revolution is churning around us. Of course, the Moore's Law curve is still unfolding - it's still a "J" and the top of the "S" is not yet in sight. Engineers are looking for denser materials, such as nanoscale and biological ones. And each subsequent material reaches its saturation. Therefore, the broadest form of the Moore's Law curve (density regardless of material) will continue to increase for some time. The change in materials used reveals another important feature of S-curves, namely that they are fractal in nature. Very large, broadly defined curves are made up of small, well-defined and connected S-curves. For someone working on a forecast, the discovery of an incipient S-curve should lead to the suspicion that a larger, more important curve lurks in the background.

Тут следует заметить, что свою статью Пол Саффо опубликовал в 2008 г., когда действительно S-кривая Мура казалась далекой от завершения. Сегодня на горизонте появилось развитие квантовых компьютеров, которые с большой долей вероятности лягут в основу новой S-кривой — примечание автора.

The art of forecasting is to see signs of a new S-curve emerging when it first begins to emerge, long before the inflection point. The challenge is that the S-curve inevitably forces us to focus on the inflection point, that dramatic moment of takeoff when fortunes are made and revolutions begin. But at this moment it is already difficult to turn on and “catch the wave.” You need to look to the left of the inflection point in order to determine the moment of its occurrence as early as possible and prepare for it.

Oddly enough, at the beginning of the flat trajectory of the S-curve, the speed of development of events and the onset of the inflection point are often incorrectly calculated. There is a tendency to overestimate short-term prospects and underestimate long-term prospects. Our hopes lead us to conclude that the revolution will happen overnight. Then, when the cold reality does not live up to our inflated expectations, our disappointment leads us to the conclusion that the long-awaited revolution will never come at all - right before it happens.

This period is perfectly described by the concept Gartner Hype Cycle .  — примечание автора

One reason for the miscalculation is that the left side of the S-curve is much longer than most people think. It took television 20 years, plus a war break, to go from its invention in the 1930s to its takeoff in the early 1950s. Even in the center of rapid change, Silicon Valley, most ideas take 20 years to demonstrate their success. The Internet was almost 20 years old in 1988, when it began its dramatic development before the explosive growth of dot-coms in the 1990s.

Thus, once you have identified the beginning of an S-curve, it is always safer to bet that things will unfold slowly than to conclude that a sudden shift will occur very soon. “Never confuse good visibility with short distance.”

Once a tipping point occurs, people tend to underestimate the speed at which change will occur. The point is that we are all linear thinkers by nature, and phenomena driven by sudden exponential growth usually take us by surprise. Even if we notice the beginning of change, we instinctively draw a straight line diagonally through the S-curve, and although we end up in the same place, we miss both the lag at the beginning and the explosive development in the middle. Silicon Valley is littered with the corpses of companies that mistook good visibility for a short distance, and those that misjudged the magnitude of the S-curve they stumbled upon.

Also expect business opportunities to be very different from those predicted by most analysts, since even the most certain future tends to arrive in completely unexpected ways. In the early 1980s, for example, PC makers predicted that there would soon be PCs in every home, which people would use for word processing and spreadsheets and, later, to read encyclopedias on CD-ROMs. But when home PC use finally became possible, its development was driven by entertainment rather than work. And when people finally turned to encyclopedias a decade after the PC makers promised, encyclopedias were already online on the Internet. Companies that sold their encyclopedias only on CDs quickly went out of business.

Rule 3: Pay attention to what seems strange and inappropriate to you

Писатель Уильям Гибсон однажды заметил: «Будущее уже наступило. Просто оно еще неравномерно распределено». Линия переднего края формирующейся S-кривой подобна струне, свисающей из будущего, и странное событие, которое вы не можете выкинуть из головы, может быть слабым сигналом того, что отдаленная разрушающая вашу отрасль S-кривая, только начинает набирать обороты.

The entire portion of the S-curve to the left of the inflection point is paved with indicators—subtle pointers that collectively become powerful hints of what's to come. The best way to detect an emerging S-curve is to tune in to strange and unusual phenomena that people cannot categorize or even dismiss. Because of our aversion to uncertainty and our preoccupation with the present, we tend to ignore indicators that don't fit the mold. But by definition, anything truly new does not fit into existing categories.

A classic example is the first sales of characters and in-game items from the online game EverQuest on eBay in the late 1990s. Although eBay banned these sales in 2001, they were the forerunners of the explosive growth of trading in Second Life and later in the virtual worlds of computer games and the metaverse. In 2021 about 171 million subscribers participate in virtual world simulations.

More often than not, indicators of the future simply look like strange anomalies or, worse, failures. And we don’t like uncertainty and avoid failures and anomalies. But if you want to find something that will come out of nowhere in the coming years and change your business, look for interesting failures—clever ideas that seem to go nowhere.

As the Second Life example shows, indicators are grouped together. Here's another good example. Some readers will remember the flurry of news surrounding DARPA's first two big tests, in which the US Department of Defense invited inventors and researchers to design robots that could race 100-plus miles across the Mojave Desert. The first Grand Challenge, with a prize fund of $1 million, was held in March 2004. Most of the robots died within sight of the starting line, and only one made it more than seven miles. The ambitious goal of the Challenge seemed as far away as the summit of Everest. But just 19 months later, at the second Grand Challenge, five robots had already completed the course. It is noteworthy that 19 months is approximately one doubling period according to Moore's law.

On its own, it's just a curious story, but given the success of the Grand Challenge, it's another compelling example that a robotics tipping point is in the not-too-distant future.

Rule 4: Question predictions you are confident about.

One of the biggest mistakes in forecast decision making is to rely too much on seemingly reliable information because it reinforces previously drawn conclusions.

В прогнозировании множество взаимосвязанных неточных данных заслуживают гораздо большего доверия, чем один или два пункта надежной информации.  Проблема происходит из факта, что традиционный подход в научных исследованиях основан на сборе надежной информации.  И после того как исследователи прошли долгий процесс разработки красивой гипотезы, они склонны игнорировать любые доказательства, противоречащие их выводам.  Это неизбежное сопротивление противоречивой информации в немалой степени ответственно за нелинейный процесс смены парадигмы, описанный Томасом Куном в его классической книге «Структура научных революций» (The Structure of Scientific Revolutions, Thomas S. Kuhn). Once a scientific theory gains widespread acceptance, there follows a long, stable period during which it remains accepted wisdom. However, all the while, contradictory evidence quietly accumulates, which ultimately leads to a sudden shift.

Effective forecasting is the opposite: it is a process of forming confident opinions, the accuracy of which must be taken lightly. If you are making a forecast, then try to be the first to prove that it is wrong. The way to do this is to make a prediction and then try to disprove it with new data. Let's say you are forecasting the future price of oil and its impact on the economy. You initially conclude that above a certain price level, say $80 a barrel, American consumers will respond as they did during the Carter administration by wearing warm sweaters and conserving energy. Your next step is to try to figure out why this might not happen. For example - perhaps because Americans are wealthier today, and as sales statistics suggest, they may not be too interested in changing their habits based on fuel costs alone until the price of oil gets much higher. By formulating a sequence of falsified predictions, you can steadily narrow the cone of uncertainty until you feel that you have reached a comfortable option from which you can draw strategic conclusions.

Rule 5: Look back twice as often as you look forward

Маршалл Маклюэн однажды заметил: «Мы смотрим на настоящее через зеркало заднего вида. Мы пятимся, в будущее».  Это известная цитата, в которой говорится о склонности людей смотреть в будущее через комфортную призму прошлого, затрудняющую адекватное восприятие событий.  Но в процессе прогнозирования, и при правильном использовании, наше историческое зеркало заднего вида является чрезвычайно мощным инструментом. Текстуру прошлых событий можно использовать, чтобы соединить точки индикаторов, которые мы нашли  в настоящем и, таким образом, надежно отобразить траекторию будущего — при условии, что вы заглянули достаточно далеко в прошлое.

A case in point is the uncertainty created by the Internet maelstrom after the burst of the dot.com bubble, when the media market was competed for by incumbents such as Google and Yahoo, new entrants, and declining incumbents in television and print media. All of this seems to defy categorization, much less prediction, until we go back five decades to the advent of television in the early 1950s and the subsequent mass media revolution for which it was the catalyst.

The present moment has obvious parallels to that era, and examining these similarities makes today's landscape clear: we are at a moment when the old order of mass media is being replaced by a new order of personal mass communication.

The problem with history is that our love of certainty and continuity often leads us to draw the wrong conclusions. The recent past is rarely a reliable indicator of the future - if it were, one could successfully predict the next 12 months of the Dow Jones or Nasdaq by plotting a straight line connecting the performance of the last 12 months and extending the line into the future. But the Dow Jones index, like any other trend, does not behave this way. You need to look for turns, not straight lines, so you need to look far enough into the past to identify patterns. Mark Twain once said that “history never repeats itself, but it often rhymes.” Good analysts look to history to find “rhymes” rather than identical events.

So when you look back for parallels, always look back at least twice as far as you look forward. Look for similar patterns, remembering that history, especially recent history, rarely repeats itself directly. And don't be afraid to keep looking back if double spacing isn't enough.

The hardest thing about analyzing the past is recognizing when past history doesn't fit with the present. The temptation is to use history mechanically, like the drunk who looks for his keys under today's lamppost rather than under the one where he lost them. This is the biggest mistake an analyst can make, and, unfortunately, there are plenty of examples of this. Jerry Levin, for example, sold his Time Warner to AOL in the mistaken belief that he could thereby push his company into digital media, as he had successfully done with cable television and movies. As a result, he closed the deal just as AOL's decade-old model was being destroyed by new companies with models that allowed them to offer email for free.

Rule 6: Know when not to make a forecast

People have an amazing quality. They simultaneously fear change and desire it. This contradiction is built into our social vocabulary. Often when we meet, we greet a friend with the greeting: “What’s new?”

However, too much tendency to pay attention to changes is a disadvantage. The simple fact is that even in periods of dramatic and rapid change, many more elements remain that do not change than new ones appear.

Let's remember once again this seething whirlwind of the 1990s, the explosive growth of dot coms. Many new technological solutions have emerged. But at the heart of the revolution brought by the development of the Internet were deep, abiding consumer desires and, ultimately, to the chagrin of many startups, the immutable laws of economics. While focusing on innovation, many lost sight of the fact that consumers were using their new broadband channels to purchase very traditional goods such as books and engage in old human activities such as gossip, entertainment and pornography. And although those looking to the future declared that this was a time when the old rules no longer applied, the old economic imperatives came into play with a vengeance, and the dot-com bubble burst, like every other bubble before it. Anyone who has spent time studying the history of economic bubbles would have seen this happen.

Against this background, it is important to note that there are times when forecasting is relatively easy, and there are times when it is impossible to do so. The cone of uncertainty is not static; it expands and contracts as the present flows into the future. Some opportunities are realized while others are closed. Thus, there are moments of unprecedented uncertainty when the cone expands to a configuration in which the wise analyst refrains from making a forecast at all. But even at such a moment, one can take comfort in the knowledge that sooner or later everything will calm down, and with careful and careful use of intuition, a good forecast can be made.

Consider the events associated with the fall of the Berlin Wall. In January 1989, East German leader Erich Honecker declared that the wall would last for “another hundred years,” and indeed Western governments built all their plans around this assumption. The signs of internal collapse are obvious in retrospect, but at the time the world seemed locked into a bipolar superpower order that, despite the fear of nuclear war, was remarkably stable. The cone of uncertainty was thus relatively narrow, and within its confines there were a number of easily conceivable outcomes, including the horror of mutual destruction. Uncertainties arose only where the spheres of influence of the two superpowers touched and intersected. But in the fall of 1989, the Berlin Wall collapsed, and with it the certainty of the forecast based on the assumption that the world was dominated by two superpowers. The conveniently narrow cone has expanded 180 degrees, at which point a wise expert would refrain from jumping to conclusions and instead calmly look for indicators of what will emerge from the geopolitical wreckage—both the overlooked events leading to the collapse of the wall and new ones emerging from its geopolitical wreckage.

Indeed, the new order manifested itself within 12 months, and the indicator of this was the Iraqi invasion of Kuwait on August 2, 1990. Before the collapse of the USSR, such an action would have provoked a conflict similar to the Cuban Missile Crisis between the two superpowers, but without a strong Soviet Union to contain Saddam or rattle sabers in response, the result was very different. At the same time, a new geopolitical order became apparent: a cone of uncertainty narrowed, encompassing a world in which countless players, once within the ordered force field of one superpower or another, were now moving in their own directions. All the uncertainty focused on whether the only surviving superpower could remain so. Iraq II, the US invasion of Iraq from 2003–2011, answered this question: a unipolar superpower order is impossible. It turns out that we live in a world where the only remaining superpower is too strong to ignore, but too weak to change anything.

So, the dry residue. Be skeptical of obvious changes and avoid making immediate predictions—or at least don't take any predictions too seriously. The coming future will wash away many more indicators sooner than you can imagine.

Professional analysts are developing increasingly sophisticated and nuanced tools for predicting the future—futures markets, online aggregations of expert opinions, complex computer simulations, and even event horizon scanners that monitor the Internet for surprises. Therefore, it is necessary for managers to become sophisticated and active consumers of forecasts.

This doesn't mean you have to study nonlinear algebra or become a forecasting expert. After all, forecasting is nothing more than the systematic and disciplined application of common sense. It is the exercise of your own common sense that will allow you to evaluate the quality of the forecasts given to you and correctly identify the opportunities and risks they pose. But don't stop there. The best way to understand what lies ahead is to make a forecast for yourself.

A version of this article appeared in the July-August 2007 issue of Harvard Business Review.