Seeking structure in chance and randomness...

We have a strong tendency to search for structure, and to develop powerful explanations for patterns perceived in the environment.

There are many things which happen through a complex combination of systematic, identifiable (and explicable) factors and chance. Scientific investigation has revealed a clear link between smoking and lung disease, though, not all smokers contract lung disease – a probabilistic trend. In principle there seems no reason that we can’t uncover all the particulars as to why some smokers don’t contract cancer: at present the variability is attributed to chance factors determining whether someone incurs the disease.

In this sense ‘chance’ doesn’t indicate that the situation is unknowable, only presently indeterminable.

A coin toss is referred to as a chance event, but not because it is impossible to determine the outcome by closely scrutinizing and following the various factors which produce any particular outcome, such as the composition, weight, and balance of the coin; the angle, height, and kinetic energy of the toss, and so forth. However, there is no easy way (something of an understatement!) to measure and analyze the complex interactions of all these contributing variables, yet we do understand the the statistical laws which underlie the phenomenon of “coin tossing”.

Often, though, people do not find such references to chance and randomness as satisfying. Research studies concerning how we deal with chance and random outcomes have shown that people are likely to concoct elaborate ‘theories’ to explain random outcome sequences presented to them.

Conspiracy theorists may construct complicated schemes of ‘explanation’ for what would otherwise be seen as random conjunctions of events, to provide, at least in their minds, “understanding”. Financial analysts routinely appeal to richly-textured ‘explanations’ for the frequent fluctuations of stock market prices. Whilst at times there are very definitely specific causes for various happenings on share markets, much of the variability concerns random fluctuations.

For an example of how easily one might ‘see’ what isn’t there consider the “Stock Market Game”:

A special free-edition newsletter comes into your hands from a stock market analysis firm offering insights on the market and making a speculative trading prediction for the coming fortnightly period. The newsletter does not ask you to advance any money to “take advantage” of the prediction, but asks you to take note of whether the prediction is accurate or not.

The newsletter informs you that shares in company x are a good buy because they forecast a significant rise in the price. You later notice that the stock did go up. You receive another newsletter from the same outfit a few weeks later and see that this time they are predicting a substantial fall in share price for the same company. Subsequently the prediction is borne out again. You take no action but are intrigued. The third newsletter correctly forecasts another drop for company x. Newsletter number four is ‘on the money’ with its prognostication of a good rise for company x. You’ve now worked up quite a degree of confidence and feel inclined to buy into the offer for a full year’s subscription to the analyst company’s newsletter.

But imagine this background to the experience: someone anonymous masquerading as an investment advice firm prepares a batch of 1600 newsletters to be dispatched to 1600 ‘client’ addresses lifted from the telephone directory. The newsletters are the same except that half of them predict company x’s share price will go up, and the other half predict a fall in the same share price. Once reality has crystallized the 800 hundred ‘marks’ left standing due to a successful prediction (share price rose) are now sent a second prediction: 400 are told of a spike upwards, the other 400 of a dip, and so on. At the end of the process 100 ‘lucky’ recipients have each received four successful predictions in a row and may now be ready to hand over money for a year’s subscription for this wonderful investment forecasting.

The example is simplified but illustrates how one might mistake the significance of the events encountered. Now consider how good any funds’ manager might be whom you are told, for instance by a financial magazine or TV show, has beaten their peers for the last three years. This not to say that the funds’ manager may not be a highly-competent professional – it is merely a query as to the nature and standard of evidence which will enable us to make the jump that it is really a particular element, for example, the skill of a funds’ manager, which is largely or actually responsible for a given result.

(see also: Illusory correlation, alternative explanations, cognitive illusion, expectation, Law of large numbers, Multiple endpoint fuzziness)