Backtesting (OTV)#

../_images/otv.png

Backtesting or out-of-time validation (OTV) is a generalization of the practice of evaluating a machine learning model on data it has not seen before. It is applicable in scenarios where there are clear or hypothesized temporal trends in a dataset.

OTV orders a dataset by time and then partitions the data into potentially overlapping backtests (similar to the folds in k-fold cross validation). Within each fold the data is further divided into training and validation data. Model performance can then be evaluated across the multiple backtests, all while ensuring data from the “future” is not inadvertently leaking into the model training data and skewing evaluation metrics.

Configuration#

The Configurator helper can be used to quickly and succinctly translate the most common OTV patterns into corresponding DataRobot configuration settings, handling formatting of timestamp and interval strings and toggling appropriate flags.

Usage#

import datarobotx as drx
import pandas as pd

df = pd.read_csv('https://s3.amazonaws.com/datarobot_public_datasets/DR_Demo_Google_AdWords.csv')

model = drx.AutoMLModel()
drx.Configurator(model).otv(n_backtests=5)
model.fit(df, target='SalesCount', datetime_partition_column='Date')

Additional options#

model_2 = drx.AutoTSModel()
# Explicit validation duration for each backtest
drx.Configurator(model_2).otv(n_backtests=3, validation_duration='30d')
# No holdout fold
drx.Configurator(model_2).otv(n_backtests=3, validation_duration='30d', holdout=False)
# Explicitly configured holdout
drx.Configurator(model_2).otv(n_backtests=3, validation_duration='30d', holdout=('2017-10-20', '45d'))

Advanced configuration#

DataRobot provides a significant number of options for configuring OTV which can be reviewed in detail in the DataRobot API docs. These options can be used for fine-grained control of backtests (e.g. explicitly defining each backtest fold).

The many individual parameters for controlling OTV can be directly set using the DRConfig object or passed as keyword arguments into base model constructors.