.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "examples/xmpl_quickstart.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_examples_xmpl_quickstart.py>`
        to download the full example code. or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_examples_xmpl_quickstart.py:


=============================
Quickstart
=============================

This example shows a simple comparison of the expected calibration error of a
non-calibrated method against a calibrated method.

.. GENERATED FROM PYTHON SOURCE LINES 9-14

.. code-block:: Python

    # Author: Miquel Perello Nieto <miquel.perellonieto@bristol.ac.uk>
    # License: new BSD

    print(__doc__)








.. GENERATED FROM PYTHON SOURCE LINES 15-16

First choose a classifier

.. GENERATED FROM PYTHON SOURCE LINES 16-21

.. code-block:: Python


    from sklearn.naive_bayes import GaussianNB

    clf = GaussianNB()








.. GENERATED FROM PYTHON SOURCE LINES 22-23

And a dataset

.. GENERATED FROM PYTHON SOURCE LINES 23-36

.. code-block:: Python


    from sklearn.datasets import make_classification
    from sklearn.model_selection import train_test_split

    X, y = make_classification(
        n_samples=100000, n_features=20, n_informative=4, n_redundant=4,
        random_state=42
    )

    from sklearn.model_selection import train_test_split

    X_train, X_test, Y_train, Y_test = train_test_split(X, y)








.. GENERATED FROM PYTHON SOURCE LINES 37-38

We can see how calibrated it is after training

.. GENERATED FROM PYTHON SOURCE LINES 38-47

.. code-block:: Python


    clf.fit(X_train, Y_train)

    n_correct = sum(clf.predict(X_test) == Y_test)
    n_test = Y_test.shape[0]

    print(f"The classifier gets {n_correct} correct "
          f"predictions out of {n_test}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    The classifier gets 17938 correct predictions out of 25000




.. GENERATED FROM PYTHON SOURCE LINES 48-49

We can asses the confidence expected calibration error

.. GENERATED FROM PYTHON SOURCE LINES 49-58

.. code-block:: Python


    from pycalib.metrics import conf_ECE

    scores = clf.predict_proba(X_test)
    cece = conf_ECE(Y_test, scores, bins=15)

    print(f"The classifier gets a confidence expected "
          f"calibration error of {cece:0.2f}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    The classifier gets a confidence expected calibration error of 0.13




.. GENERATED FROM PYTHON SOURCE LINES 59-60

Let's look at its reliability diagram

.. GENERATED FROM PYTHON SOURCE LINES 60-66

.. code-block:: Python


    from pycalib.visualisations import plot_reliability_diagram

    plot_reliability_diagram(labels=Y_test, scores=scores, show_histogram=True,
                             show_bars=True, show_gaps=True)




.. image-sg:: /examples/images/sphx_glr_xmpl_quickstart_001.png
   :alt: xmpl quickstart
   :srcset: /examples/images/sphx_glr_xmpl_quickstart_001.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    <Figure size 400x400 with 2 Axes>



.. GENERATED FROM PYTHON SOURCE LINES 67-68

We can see how a calibration can improve the conf-ECE

.. GENERATED FROM PYTHON SOURCE LINES 68-72

.. code-block:: Python


    from pycalib.models import IsotonicCalibration
    cal = IsotonicCalibration()








.. GENERATED FROM PYTHON SOURCE LINES 73-75

Now we can put together a probabilistic classifier with the chosen calibration
method

.. GENERATED FROM PYTHON SOURCE LINES 75-81

.. code-block:: Python


    from pycalib.models import CalibratedModel

    cal_clf = CalibratedModel(base_estimator=clf, calibrator=cal,
                              fit_estimator=False)








.. GENERATED FROM PYTHON SOURCE LINES 82-83

Now you can train both classifier and calibrator all together.

.. GENERATED FROM PYTHON SOURCE LINES 83-96

.. code-block:: Python


    cal_clf.fit(X_train, Y_train)
    n_correct = sum(cal_clf.predict(X_test) == Y_test)

    print(f"The calibrated classifier gets {n_correct} "
          f"correct predictions out of {n_test}")

    scores_cal = cal_clf.predict_proba(X_test)
    cece = conf_ECE(Y_test, scores_cal, bins=15)

    print(f"The calibrated classifier gets a confidence "
          f"expected calibration error of {cece:0.2f}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    The calibrated classifier gets 18213 correct predictions out of 25000
    The calibrated classifier gets a confidence expected calibration error of 0.01




.. GENERATED FROM PYTHON SOURCE LINES 97-98

Now you can train both classifier and calibrator all together.

.. GENERATED FROM PYTHON SOURCE LINES 98-103

.. code-block:: Python


    from pycalib.visualisations import plot_reliability_diagram

    plot_reliability_diagram(labels=Y_test, scores=scores_cal, show_histogram=True,
                             show_bars=True, show_gaps=True)



.. image-sg:: /examples/images/sphx_glr_xmpl_quickstart_002.png
   :alt: xmpl quickstart
   :srcset: /examples/images/sphx_glr_xmpl_quickstart_002.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    <Figure size 400x400 with 2 Axes>




.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.712 seconds)


.. _sphx_glr_download_examples_xmpl_quickstart.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: binder-badge

      .. image:: images/binder_badge_logo.svg
        :target: https://mybinder.org/v2/gh/perellonieto/pycalib/gh-pages?filepath=notebooks/examples/xmpl_quickstart.ipynb
        :alt: Launch binder
        :width: 150 px

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: xmpl_quickstart.ipynb <xmpl_quickstart.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: xmpl_quickstart.py <xmpl_quickstart.py>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_