modelsight.curves.compare

This file deals with the implementation of functions that allow annotating plots with statistical tests results between pairs of estimators.

Module Contents

Functions

annot_stat_vertical(text, x, y1, y2[, ww, col, ...])

Draw a vertical whisker at position x that spans through y1 to y2 with annotation specified

annot_stat_horizontal(text, x1, x2, y[, wh, col, ...])

Draw an horizontal whisker at position y that spans through x1 to x2 with annotation specified

add_annotations(comparisons, alpha, bars, direction, order)

Annotates the specified plot (ax) with the provided comparisons results either vertically or horizontally

roc_single_comparison(→ Dict[str, Tuple[str, str, float]])

Perform a single comparison of two areas under Receiver Operating Characteristic curves

roc_comparisons(cv_preds, target_algo)

Compares the AUC of the specified algorithm with the AUCs of all other algorithms.
modelsight.curves.compare.annot_stat_vertical(text: str, x: float, y1: float, y2: float, ww: float = 0.02, col: str = 'black', fontsize: int = 13, voffset: float = 0, n_elems: int = None, ax=None, **kwargs)[source]

Draw a vertical whisker at position x that spans through y1 to y2 with annotation specified by text.

Parameters:

  • text (str) – Annotation for whisker.

  • x (float) – x-position the whisker is positioned at.

  • y1 (float) – starting y position.

  • y2 (float) – ending y position.

  • ww (float, optional) – whisker width, by default 0.02

  • col (str, optional) – whisker color, by default ‘black’

  • fontsize (int, optional) – fontsize for the annotation, by default 13

  • voffset (float, optional) – vertical offset for the annotation, by default 0. Some font families and characters occupy different vertical spaces; this parameter allows compensating for such variations.

  • n_elems (int, optional) – number of discrete elements in the y-axis, by default None. This value is precomputed by the caller (add_annotations) and passed to this function as input.

  • ax (plt.Axes, optional) – a pyplot Axes to draw annotations on, by default None

  • **kwargs –

    rect_h_base: float, optional

    base height of rectangle patch for single-character annotations, by default 0.1

    fontsize_nonsignif, optional

    fontsize for multi-character annotations (here called non significant annotations to reflect the fact that single-character annotations most often use some kind of symbol to denote statistical significance, e.g. *), by default fontsize (i.e., 13)

modelsight.curves.compare.annot_stat_horizontal(text: str, x1: float, x2: float, y: float, wh: float = 0.02, col: str = 'black', fontsize: int = 13, voffset: float = 0, n_elems: int = None, ax: matplotlib.pyplot.Axes = None, **kwargs)[source]

Draw an horizontal whisker at position y that spans through x1 to x2 with annotation specified by text.

Parameters:

  • text (str) – Annotation for whisker.

  • x1 (float) – starting x position.

  • x2 (float) – ending x position.

  • y (float) – y-position the whisker is positioned at.

  • wh (float, optional) – whisker height, by default 0.02

  • col (str, optional) – whisker color, by default ‘black’

  • fontsize (int, optional) – fontsize for the annotation, by default 13

  • voffset (float, optional) – vertical offset for the annotation, by default 0. Some font families and characters occupy different vertical spaces; this parameter allows compensating for such variations.

  • n_elems (int, optional) – number of discrete elements in the y-axis, by default None. This value is precomputed by the caller (add_annotations) and passed to this function as input.

  • ax (plt.Axes, optional) – a pyplot Axes to draw annotations on, by default None

  • **kwargs –

    fontsize_nonsignif, optional

    fontsize for multi-character annotations (here called non significant annotations to reflect the fact that single-character annotations most often use some kind of symbol to denote statistical significance, e.g. *), by default fontsize (i.e., 13)

modelsight.curves.compare.add_annotations(comparisons: Dict[str, Tuple[str, str, float]], alpha: float, bars: matplotlib.container.BarContainer, direction: str, order: List[Tuple[str, str]], symbol: str = '*', symbol_fontsize: int = 22, voffset: float = 0, ext_voffset: float = 0, ext_hoffset: float = 0, P_val_rounding: int = 2, ax: matplotlib.pyplot.Axes = None, **kwargs)[source]

Annotates the specified plot (ax) with the provided comparisons results either vertically or horizontally depending on the value of direction.

Parameters:

  • comparisons (Dict[str, Tuple[str, str, float]]) – The results of models comparisons.

  • alpha (float) – The significance level used for formatting the P value of comparisons.

  • bars (matplotlib.container.BarContainer) – A list of matplotlib’s bars that is used to access the bar’s width or height when annotating horizontally and vertically, respectively.

  • direction (str) – The direction for annotation. Possible values are “horizontal” and “vertical”.

  • order (List[Tuple[str, str]]) – The order in which the comparisons should be displayed. Each entry of this list is a tuple where elements are algorithm’s names.

  • symbol (str, optional) – The symbol used in place of the P value when statistical significance is achieved accoring to the specified alpha, by default “*”.

  • symbol_fontsize (int, optional) – Fontsize for the symbol used when statistical significance is achieved, by default 22

  • voffset (float, optional) – vertical offset for the annotation, by default 0., by default 0

  • ext_voffset (float, optional) – Additional vertical offset for vertical annotations. Ignored when direction = “horizontal”, by default 0

  • ext_hoffset (float, optional) – Additional horizontal offset for horizontal annotations. Ignored when direction = “vertical”, by default 0

  • P_val_rounding (int, optional) – Number of decimal places to round P values at, by default 2

  • ax (plt.Axes, optional) – The plot to be annotated, by default None

Returns:

ax – The annotated plot.

Return type:

plt.Axes

Raises:

  • ValueError – When ax is None

  • ValueError – Whenever a comparison key doesn’t exist.

modelsight.curves.compare.roc_single_comparison(cv_preds: modelsight._typing.CVModellingOutput, fst_algo: str, snd_algo: str) → Dict[str, Tuple[str, str, float]][source]

Perform a single comparison of two areas under Receiver Operating Characteristic curves computed on the same set of data points by the DeLong test.

Parameters:

  • cv_preds (CVModellingOutput) – The output of a cross-validation process encompassing mulitple (n>=2) models.

  • fst_algo (str) – The name of the first algorithm for the comparison. Must be an existing key of cv_preds.

  • snd_algo (str) – The name of the second algorithm for the comparison. Must be an existing key of cv_preds.

Returns:

comparison_result – The output of the comparison. This is a dictionary where the key is of the form “<fst_algo>_<snd_algo>” and the value is a tuple of three elements, the first two are the names of the algorithms being compared and the third element is the P value for the null hypothesis that the two AUC values are equal.

Return type:

Dict[str, Tuple[str, str, float]]

modelsight.curves.compare.roc_comparisons(cv_preds: modelsight._typing.CVModellingOutput, target_algo: str)[source]

Compares the AUC of the specified algorithm with the AUCs of all other algorithms.

Parameters:

  • cv_preds (CVModellingOutput) – The output of a cross-validation process encompassing mulitple (n>=2) models.

  • target_algo (str) – The name of the target algorithm’s whose AUC will be compared with all other AUCs.

Returns:

comparisons – A dictionary containing the results of all comparisons. See output of roc_single_comparison.

Return type:

Dict[str, Tuple[str, str, float]]