Evaluation

Evaluate a model’s performance, optionally using an external test dataset.

What is an “external test” dataset?

When we refer to an “external test” or “external data”, we are referring to a set of data that was not used to either train the model or evaluate the model relative to others (e.g. within a DataRobot project). An external holdout dataset can help in assessing how a model might perform on new data.

Evaluate in-notebook using drx

In notebook environments, users can access measures of model performance and feature importance by calling evaluate on a drx model.

import datarobotx as drx

# Evaluate on the latest data partition available
drx.evaluate(model)

# Evaluate on a specific partition of data
drx.evaluate(model, data_partition='validation')

Users can also assess the performance of a model on an external test dataset. Presently, when using an external dataset a few additional plots are available depending on the target type.

from datarobotx import evaluate

evaluate(model, evaluation_data=external_test_data_in_dataframe)

Recreating the model card

The prediction dataset identifier shown an the bottom of the model card may be used for future calls to evaluate to avoid re-uploading the data to DataRobot

Evaluate using other libraries

Where possible, drx models implement the predict and predict_proba methods in a manner that aligns with scikit-learn. This enables drx models to interoperate with scikit-learn and other libraries that expect this interface, sometimes without modification.

from sklearn.metrics import confusion_matrix

pd.DataFrame(confusion_matrix(test_df['is_bad'], model.predict(test_df)))

	0	1
0	180	0
1	9	11

Importing models with coefficients

drx allows users to bring parametric models onto the leaderboard by passing in a model formula. DataRobot will build a new Eureqa Blueprint constrained to the formula by leveraging custom expressions.

readmissions_formula = "Target = logistic(1.1*number_inpatient + 3.1*number_diagnoses)"

import_model = import_parametric_model(
   model, readmissions_formula
)

API Reference

evaluate(model[, evaluation_data, ...])	Show evaluation metrics and plots for a model.
import_parametric_model(model, model_formula)	Import a linear model onto the DataRobot leaderboard.