A new study argues that most financial research is based on statistically erroneous methods and several features of it make it particularly prone to the occurrence of false discoveries.
The academic paper titled “Finance is not excused: why finance should not flout basic principles of statistics” by David Bailey, University of California research fellow and Marcos de Prado, Global Head, Quantitative Research & Development at Abu Dhabi Investment Authority, equates finance industry back-tested analyses with the now-debunked historical medical studies funded by Big Tobacco who paid for thousands of studies in support of their bottom line during the 1950-2000 period.
According to the authors, thousands of academic articles that promote dubious investment strategies, are being published by researchers without controlling for multiple testing and are written for, funded, or promoted by investment firms with a commercial interest.
“… The sobering consequence is that a significant portion of the models, funds and strategies employed in the investment world, including many of those marketed to individual investors, may be merely statistical mirages.”
The authors suggest that three features of financial research render it prone to error. “First, the probability of finding a positive (profitable investment strategy) is very low, due to intense competition. Second, true findings are mostly short-lived, as a result of the nonstationary nature of financial systems. Third, unlike in the natural sciences, it is rarely possible to verify statistical findings through controlled experiments. Finance’s inability to conduct controlled experiments makes it virtually impossible to debunk a false claim. One would hope that, in such a field, researchers would be particularly careful when conducting statistical inference. Sadly, the opposite is true.”
The majority of financial research based on interrogating historical market data to discern “optimal” investment strategies has served up mostly “false discoveries.”
“With this ‘optimal’ design in hand, [financial researchers] tout the potential return that an investment based on this design is likely to deliver, based on its simulated performance on historical data (backtest). However, in all too many cases, such investments deliver only disappointing performance when actually fielded,” the authors write.
The paper also notes that prominent market forecasters often promote in the media their success at predicting some events, while hoping that the audience has forgotten an equal or greater number of false calls.
“Markets are not efficient by design. Instead, market efficiency is a byproduct of market competition, thus some firms must be able to extract a profit. But what firms? Among the best performing funds in history, those founded by mathematicians and natural scientists are disproportionally represented. These funds employ sophisticated and rigorous statistical techniques, using state-of-the-art computing equipment operating on numerous massive datasets. These scientists either manage their own assets and thus do not accept outside investors, or their fundraising relies on their track record, not on academic marketing of supposed economic factors, hence they have no incentive to engage in selection bias.”
“Unlike finance, medical journals today impose strict controls for multiple testing. Academic finance’s denial of its replication crisis risks its branding as a pseudoscience” the study concludes.