modelsight.curves

Submodules

• modelsight.curves._delong
• modelsight.curves.compare
• modelsight.curves.roc

Package Contents

Functions

average_roc_curves(, random_state, ax, ...)	Generate receiver-operating characteristic curves for each model in cv_preds.
roc_single_comparison(→ Dict[str, Tuple[str, str, float]])	Perform a single comparison of two areas under Receiver Operating Characteristic curves
roc_comparisons(cv_preds, target_algo)	Compares the AUC of the specified algorithm with the AUCs of all other algorithms.

modelsight.curves.average_roc_curves(cv_preds: Dict[str, modelsight._typing.CVModellingOutput], colors: List[str], model_keys_map: Dict[str, str] = {}, show_ci: bool = True, n_boot: int = 1000, bars_pos: Tuple[int, int, int, int] = (0.41, 0.01, 0.53, 0.3), random_state: modelsight._typing.SeedType = 1234, ax: matplotlib.pyplot.Axes = None, **kwargs) → Tuple[matplotlib.pyplot.Axes, matplotlib.pyplot.Axes, matplotlib.container.BarContainer, Dict[str, Dict[str, float]]]

Generate receiver-operating characteristic curves for each model in cv_preds.

Parameters:

• cv_preds (Dict[str, CVModellingOutput]) – A dictionary containing model-specific cross-validation modelling outputs.

• colors (List[str]) – A list of colors that will be used to color both curves and bars.

• model_keys_map (Dict[str, str] (default = {})) – A dictionary mapping model keys to model names.

• show_ci (bool (default = True)) – Whether bootstrapped confidence bands around curves should be shown.

• n_boot (int (default = 1000)) – Number of bootstrap iterations for generating confidence bands.

• bars_pos (Tuple[int, int, int, int]) – A tuple of four integers specifying the shape and position of the bar plot inset. (x position, y position, width, height)

• random_state (Seed (default = 1234)) – A seed for reproducibility.

• ax (plt.Axes (default = None)) – Optional Axes to plot curves onto.

• **kwargs –

  model_names_in_black: List[str]

  Names of models to show in black color, default is []

Returns:

First: the Axes containing the general plot. Second: the axes containing the bar plot inset. Third: the actual BarContainer of the bar plot inset. Fourth: A dictionary containing median (95%CI) area-under-curve over cross-validation

for each model.

Return type:

Tuple[plt.Axes, plt.Axes, matplotlib.container.BarContainer, Dict[str, Dict[str, float]]]

modelsight.curves.roc_single_comparison(cv_preds: modelsight._typing.CVModellingOutput, fst_algo: str, snd_algo: str) → Dict[str, Tuple[str, str, float]]

Perform a single comparison of two areas under Receiver Operating Characteristic curves computed on the same set of data points by the DeLong test.

Parameters:

• cv_preds (CVModellingOutput) – The output of a cross-validation process encompassing mulitple (n>=2) models.

• fst_algo (str) – The name of the first algorithm for the comparison. Must be an existing key of cv_preds.

• snd_algo (str) – The name of the second algorithm for the comparison. Must be an existing key of cv_preds.

Returns:

comparison_result – The output of the comparison. This is a dictionary where the key is of the form “<fst_algo>_<snd_algo>” and the value is a tuple of three elements, the first two are the names of the algorithms being compared and the third element is the P value for the null hypothesis that the two AUC values are equal.

Return type:

Dict[str, Tuple[str, str, float]]

modelsight.curves.roc_comparisons(cv_preds: modelsight._typing.CVModellingOutput, target_algo: str)

Compares the AUC of the specified algorithm with the AUCs of all other algorithms.

Parameters:

• cv_preds (CVModellingOutput) – The output of a cross-validation process encompassing mulitple (n>=2) models.

• target_algo (str) – The name of the target algorithm’s whose AUC will be compared with all other AUCs.

Returns:

comparisons – A dictionary containing the results of all comparisons. See output of roc_single_comparison.

Return type:

Dict[str, Tuple[str, str, float]]