Missing Values and Outliers
"Some stocks have no value for this feature, and a few have values twenty standard deviations from the mean. What do you do?" A complete answer first asks why each value is missing or extreme, because the reason decides the fix. It then fits every fix on the training data only.
Why financial data goes missing
Values go missing for reasons that are rarely random.
- Not traded. A stock was suspended, or too illiquid to print a price that day.
- Not yet reported. A new listing has no twelve-month history. A small company has no analyst coverage, or has not yet published its accounts.
- Delisted. The company was taken over or went bankrupt, and its series stops.
Each reason is linked to size, liquidity or distress, and these also predict returns. Dropping every row with a missing value therefore keeps the large, liquid, surviving firms. The model is then trained on a different universe from the one it will trade. Dropping delisted firms is also survivorship bias, covered in look-ahead and survivorship bias.
The rest of this lesson is for subscribers
Unlock every lesson in Machine Learning for Quantitative Research, and every other premium course.
Subscribe to continueTest your knowledge
Keep reading Machine Learning for Quantitative Research
27 lessons in this course, and every other premium course, on one subscription.
- Every lesson in every course, with the worked examples and interactive simulators
- Graded questions on every lesson, with explanations for the wrong answers as well as the right one
- The trainers, timed assessments and brainteaser library that go with them