Chapter 134 22.4 Making Big Money From the Range of Predictability

According to Farmer, there are two different kinds of complexity: intrinsic and apparent.Intrinsic complexity is the "real" complexity of a chaotic system.It creates dark unpredictability.Another kind of complexity is the other side of chaos—the apparent complexity that masks the order that can be exploited. Farmer drew a box in the air.Going up, surface complexity increases; diagonally up through the square, intrinsic complexity increases. “Physics usually works here,” Farmer says, pointing to the bottom corner where the low-end co-occurrence of the two types of complexity, the area where the simple problems lie. "And over there," said Farmer, pointing to the top corner of the box opposite this bottom corner, "it's all hard stuff. But here's where we're going to slide, and here the problems will be more interesting." -- Here the apparent complexity is high, while the real complexity remains relatively low. At this point, some elements of the complex puzzle are predictable. And those are exactly what we're looking for in the stock market. "

Forecasting firms hope to eliminate simple problems in financial markets with the help of crude computer tools that take advantage of the other side of chaos. "We're using every tool we can find," says former Chaos Society member Norman Packard, a partner at the firm.The idea is to take proven pattern-hunting strategies from various sources and turn them into data, and then "knock them out" to optimize the algorithm.Find the clearest hint of a pattern, then bring the truth to light.It's a gambler's mentality: any interest is an interest. The belief that motivated Farmer and Packard was drawn from their own experience: that the other side of chaos is stable enough to depend on.There's no reassurance like the solid cash they've earned in their Las Vegas roulette experiment.It would be foolish not to take advantage of these patterns.As the author who chronicles their high-victory venture exclaims in "Happy Pies," "Why not play roulette with a computer in your shoe?"

Beyond experience, Farmer and Packard have injected a great deal of conviction into their respected theory, which they have created through their research on chaos.Now, though, they're still testing their wildest and most controversial theories.Contrary to the skepticism of most economists, they believe that some regions of those other complex phenomena can also be accurately predicted.Packard called these regions "ranges of predictability" or "local predictability."In other words, the unpredictability is not uniformly distributed throughout the system.Most of the time, the vast majority of complex systems may not be predictable, but a small fraction of them may be predictable in the short term.Looking back, Packard believes that it was this local predictability that allowed Santa Cruz Chaos to make money predicting the approximate path of a ball on a roulette wheel.

Even if such ranges of predictability do exist, they are sure to be buried under a sea of ​​unpredictability.Signals of local predictability can be overshadowed by the swirling noise of a thousand other variables.The six stock market experts at Forecasting Inc. used a search technique that mixed old and new, high and low, to scan this vast pile of combined signals.Their software scours financial data for mathematically high-dimensional spaces as well as local regions—whatever local regions match predictable, low-dimensional patterns.They are looking for signs of order—any order—in the financial universe.

The real-time work they do can also be called "super real-time" work.Just as a bouncing ball simulated in a shoe computer will stop before the real one, so the simulated financial model of a predictive company will run faster than it actually does on Wall Street.They re-enacted a simplified part of the stock market in a computer.When they detect a wave of local order unfolding, they simulate it faster than in real life, and then place their bets on where they think the wave might end. David Barriby used a lovely metaphor to describe this process of finding the bounds of predictability in the March 1993 issue of Discover: "Looking at chaos in the market is like looking at A rough, splashing river, full of wild, rolling waves, and those unpredictable, ever-circling eddies. But suddenly, somewhere in the river, you recognize a The familiar eddies, and within five to ten seconds afterward, knew the direction of the flow in this part of the river."

Of course, you can't predict where the water will go half a mile downstream, but for five seconds -- or, on Wall Street, five hours -- you can predict how this demonstration will go. .And that's exactly what you need to get useful (or rich).Find any pattern and use it.The algorithm of the predictive company is to capture a little bit of order that is fleeting, and then use this fleeting prototype to make money.Farmer and Packard emphasize that while economists follow the code of ethics to unearth the causes of these patterns, gamblers have no such constraints.Predicting the important goals of the company is not exactly what forms the pattern.In an inductive model—the kind that forecasting companies construct—events need no abstract cause, any more than an outfielder with an imaginary baseball flight path, or a dog chasing a thrown stick. There is no need for an abstract reason.

Rather than worrying about the ambiguous relationship between cause and effect in large-scale clustered systems full of causal loops, Farmer says: "The key question to beat the stock market is: Which patterns?” Which patterns conceal order?Learning to recognize order, not cause, is the key. Before placing a bet on a model, Farmer and Packard would test it using a "backtracking" method.When using the technique of "backtracking" (a method commonly used by professional futurists), the model is built using the latest data from the workforce management model.Once the system finds some order in past data, say, the 1980s, it is fed data from past years.If the system can accurately predict the 1993 outcome based on findings from the 1980s, the pattern searcher could get its medal."The system comes out with twenty models," Farmer said. "We'll run all of them and sift through them with diagnostic statistics. Then the six of us will get together and pick the one that actually runs. .” Each round of this modeling exercise can run on company computers for several days.But once some local order is found, making predictions based on that order takes only a millionth of a second.

The final step -- which is to stuff the wads of real money into its hands to actually run the program -- requires one of the PhDs to hit the "Enter" key on a keyboard.This action throws the chosen algorithm into the world of top-flight racing with so much money that it stops the brain.Cutting off the reins of theory and running automatically, this fleshed out algorithm can only hear its creators whispering: "Place an order, fool, place an order!" "As long as we can beat the market by 5 percentage points, our investors will make money," Packard said.Here's how Packard explained the number: They were able to predict 55 percent of the market's direction, that is, 5 percentage points more than random guesses, but if they did guess correctly, they ended up with The result will be 200% higher, that is, twice the market's win rate.Those Wall Street bigwigs (currently O'Connor and affiliates) who provide financial support to the forecasting company can obtain the exclusive right to use this algorithm, and in exchange, they have to pay the company a certain amount based on the specific performance of the forecasting results obtained by the algorithm. cost. "We still have some competitors," Packard said with a smile. "I know four other companies who are thinking about the same thing," using nonlinear dynamics to capture patterns in chaos, and then use those patterns to predict. "Two of them have grown. Some of our friends are still in there."

Citibank is one of the competitors using real money transactions.Since 1990, British mathematician Andrew Colin has been working on trading algorithms.His forecasting program starts by randomly generating hundreds of hypotheses whose parameters affect monetary data, then tests those hypotheses using data from the most recent five years.The most likely influential parameters are sent to the computer neural network, which adjusts the weight of each parameter in order to better match the data, and adopts the method of weighting the best combination of parameters in order to produce better guesses.This neural network system will also constantly feed back the results obtained, and constantly polish its guesses through some kind of self-learning method.When a model fits past data, it is sent into the future. In 1992, an article in The Economist magazine wrote: "After two years of experimentation, Dr. Colin estimates that his computer's virtual trading funds can obtain a return of 25% per year... which is already the vast majority. It is several times higher than the expectations of human traders." At that time, the Bank of Milan in London had eight stock market analysis masters working on the forecasting device.They plan to have algorithms generated by computers.Still, as in forecasting companies, computer-generated algorithms are evaluated by humans “before hitting enter.”Until late 1993, they traded for real money.

One of the questions investors like to ask Farmer is how he can prove that people can actually make money in the markets with such a small advantage of information.Farmer cites a "real-life example" of people like George Soros on Wall Street making millions year after year through currency trading or whatever.Successful traders, Farmer complained, "are looked down upon by those academics who think they're just super lucky—but the evidence shows that's not the case at all."Human traders unconsciously learn how to spot patterns of local predictability in seas of random data.These traders have made millions because they first discovered patterns (though they couldn't tell them why) and then built internal patterns (though they didn't realize it) in order to make predictions. ).They don't know any more about their models or theories than they do about how they catch fly balls.That's all they did.Both models, however, are based on experience, built with the same Ptolemaic induction.And that's exactly how forecasting firms use computers to model soaring stocks -- bottom-up, starting with data.

"If we have broad-based success with what we're doing now, it proves that machines are better at forecasting than people, and that algorithms are better economists than Milton Friedman," Farmer said. .Traders are already suspicious of this thing. They feel threatened by it." The hard part is keeping the algorithm simple."The more complex the problem, the simpler the model you end up using," Farmer said. "It's not that hard to fit the data, but if you do it, you end up getting away with it. Generalization is key." After all, the predictive mechanism is actually a mechanism for producing theory, a device for generating abstractions and generalizations.The predictive mechanism chews through the seemingly random, gouged messy data of complex, living things.Given a large enough stream of data accumulated over time, the device can pick out patterns in the dots.Over time, this technology will develop a specific model internally to solve the problem of how to generate data.The machinery does not "over-tune" the model to individual data, it tends to generalize vaguely to somewhat imprecise fits.Once it gets some kind of general fit, or say, a theory, it can make predictions.In fact, prediction is the whole point of the theory."Prediction is the most useful, tangible, and in many ways, the most important consequence of building a scientific theory," declared Farmer. Although theory-making is a creative act the human brain excels at, ironically Yes, we have no rules for how to make theories.Farmer calls this mysterious "general pattern-seeking ability" "intuition."The "lucky" traders on Wall Street make use of precisely this ability. We can also see this predictive mechanism in biology.As David Reed, director of a high-tech think tank called Interval, puts it, "Dogs can't do math," but a trained dog can precalculate the path of a Frisbee and catch it with precision.Generally speaking, intelligence, or cleverness, is basically a predictive mechanism.Likewise, all adaptation and evolution are relatively milder and more sparsely distributed mechanisms for prediction and forecasting. At a private gathering of CEOs of various companies, Farmer publicly admitted: "Forecasting the market is not my long-term goal. To be honest, I am the kind of person who opens the "Wall Street Journal" and looks at the financial section. Painful people.” Nor was it surprising for an unrepentant ex-hippie.Farmer stipulated that he spend five years working on the problem of stock market forecasting, earn a fortune, and then move on to more interesting problems, such as real artificial life, artificial evolution, and artificial intelligence.And financial forecasting, like roulette, is just another puzzle. "We're interested in this problem because our dream is to produce a mechanism for prediction, one that allows us to make predictions about a lot of different things." -- weather, global climate, infectious disease, etc. Etc - "anything that generates a lot of data that we can't understand". "Ultimately," says Farmer, "we want to be able to imbue the computer with a rough form of intuition." By the end of 1993, Farmer and The Forecasting Company publicly reported that they had successfully used "computerized intuition" to make market forecasts, using real money transactions.Their agreements with investors do not allow them to talk about specific performance, as much as Farmer would have liked to.He did say, however, that in a few years they would have enough data to prove "by a scientific standard" that their trading success was more than just statistical luck: Statistically significant patterns were found in . There is indeed a range of predictability."