{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "# Training a Decision Tree or a Random Forest on a classification problem, and compare the latter with using adaBoost\n", "\n", "**Author: Pr Fabien MOUTARDE, Center for Robotics, MINES ParisTech, PSL Université Paris**\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Decision Trees with SciKit-Learn on a very simple dataset\n", "\n", "**We will first work on very simple classic dataset: Iris, which is a classification problem corresponding to determination of iris flower sub-species based on a few geometric characteristics of the flower.**\n", "\n", "**Please FIRST READ the [*Iris DATASET DESCRIPTION*](http://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html#sphx-glr-auto-examples-datasets-plot-iris-dataset-py).**\n", "In this classification problem, there are 3 classes, with a total of 150 examples (each one with 4 input). Please **now execute code cell below to load and view the dataset**.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(Number_of_examples, example_size) = (150, 4) \n", "\n", "Input = [5.1 3.5 1.4 0.2] , Label = 0\n", "Input = [4.9 3. 1.4 0.2] , Label = 0\n", "Input = [4.7 3.2 1.3 0.2] , Label = 0\n", "Input = [4.6 3.1 1.5 0.2] , Label = 0\n", "Input = [5. 3.6 1.4 0.2] , Label = 0\n", "Input = [5.4 3.9 1.7 0.4] , Label = 0\n", "Input = [4.6 3.4 1.4 0.3] , Label = 0\n", "Input = [5. 3.4 1.5 0.2] , Label = 0\n", "Input = [4.4 2.9 1.4 0.2] , Label = 0\n", "Input = [4.9 3.1 1.5 0.1] , Label = 0\n", "Input = [5.4 3.7 1.5 0.2] , Label = 0\n", "Input = [4.8 3.4 1.6 0.2] , Label = 0\n", "Input = [4.8 3. 1.4 0.1] , Label = 0\n", "Input = [4.3 3. 1.1 0.1] , Label = 0\n", "Input = [5.8 4. 1.2 0.2] , Label = 0\n", "Input = [5.7 4.4 1.5 0.4] , Label = 0\n", "Input = [5.4 3.9 1.3 0.4] , Label = 0\n", "Input = [5.1 3.5 1.4 0.3] , Label = 0\n", "Input = [5.7 3.8 1.7 0.3] , Label = 0\n", "Input = [5.1 3.8 1.5 0.3] , Label = 0\n", "Input = [5.4 3.4 1.7 0.2] , Label = 0\n", "Input = [5.1 3.7 1.5 0.4] , Label = 0\n", "Input = [4.6 3.6 1. 0.2] , Label = 0\n", "Input = [5.1 3.3 1.7 0.5] , Label = 0\n", "Input = [4.8 3.4 1.9 0.2] , Label = 0\n", "Input = [5. 3. 1.6 0.2] , Label = 0\n", "Input = [5. 3.4 1.6 0.4] , Label = 0\n", "Input = [5.2 3.5 1.5 0.2] , Label = 0\n", "Input = [5.2 3.4 1.4 0.2] , Label = 0\n", "Input = [4.7 3.2 1.6 0.2] , Label = 0\n", "Input = [4.8 3.1 1.6 0.2] , Label = 0\n", "Input = [5.4 3.4 1.5 0.4] , Label = 0\n", "Input = [5.2 4.1 1.5 0.1] , Label = 0\n", "Input = [5.5 4.2 1.4 0.2] , Label = 0\n", "Input = [4.9 3.1 1.5 0.2] , Label = 0\n", "Input = [5. 3.2 1.2 0.2] , Label = 0\n", "Input = [5.5 3.5 1.3 0.2] , Label = 0\n", "Input = [4.9 3.6 1.4 0.1] , Label = 0\n", "Input = [4.4 3. 1.3 0.2] , Label = 0\n", "Input = [5.1 3.4 1.5 0.2] , Label = 0\n", "Input = [5. 3.5 1.3 0.3] , Label = 0\n", "Input = [4.5 2.3 1.3 0.3] , Label = 0\n", "Input = [4.4 3.2 1.3 0.2] , Label = 0\n", "Input = [5. 3.5 1.6 0.6] , Label = 0\n", "Input = [5.1 3.8 1.9 0.4] , Label = 0\n", "Input = [4.8 3. 1.4 0.3] , Label = 0\n", "Input = [5.1 3.8 1.6 0.2] , Label = 0\n", "Input = [4.6 3.2 1.4 0.2] , Label = 0\n", "Input = [5.3 3.7 1.5 0.2] , Label = 0\n", "Input = [5. 3.3 1.4 0.2] , Label = 0\n", "Input = [7. 3.2 4.7 1.4] , Label = 1\n", "Input = [6.4 3.2 4.5 1.5] , Label = 1\n", "Input = [6.9 3.1 4.9 1.5] , Label = 1\n", "Input = [5.5 2.3 4. 1.3] , Label = 1\n", "Input = [6.5 2.8 4.6 1.5] , Label = 1\n", "Input = [5.7 2.8 4.5 1.3] , Label = 1\n", "Input = [6.3 3.3 4.7 1.6] , Label = 1\n", "Input = [4.9 2.4 3.3 1. ] , Label = 1\n", "Input = [6.6 2.9 4.6 1.3] , Label = 1\n", "Input = [5.2 2.7 3.9 1.4] , Label = 1\n", "Input = [5. 2. 3.5 1. ] , Label = 1\n", "Input = [5.9 3. 4.2 1.5] , Label = 1\n", "Input = [6. 2.2 4. 1. ] , Label = 1\n", "Input = [6.1 2.9 4.7 1.4] , Label = 1\n", "Input = [5.6 2.9 3.6 1.3] , Label = 1\n", "Input = [6.7 3.1 4.4 1.4] , Label = 1\n", "Input = [5.6 3. 4.5 1.5] , Label = 1\n", "Input = [5.8 2.7 4.1 1. ] , Label = 1\n", "Input = [6.2 2.2 4.5 1.5] , Label = 1\n", "Input = [5.6 2.5 3.9 1.1] , Label = 1\n", "Input = [5.9 3.2 4.8 1.8] , Label = 1\n", "Input = [6.1 2.8 4. 1.3] , Label = 1\n", "Input = [6.3 2.5 4.9 1.5] , Label = 1\n", "Input = [6.1 2.8 4.7 1.2] , Label = 1\n", "Input = [6.4 2.9 4.3 1.3] , Label = 1\n", "Input = [6.6 3. 4.4 1.4] , Label = 1\n", "Input = [6.8 2.8 4.8 1.4] , Label = 1\n", "Input = [6.7 3. 5. 1.7] , Label = 1\n", "Input = [6. 2.9 4.5 1.5] , Label = 1\n", "Input = [5.7 2.6 3.5 1. ] , Label = 1\n", "Input = [5.5 2.4 3.8 1.1] , Label = 1\n", "Input = [5.5 2.4 3.7 1. ] , Label = 1\n", "Input = [5.8 2.7 3.9 1.2] , Label = 1\n", "Input = [6. 2.7 5.1 1.6] , Label = 1\n", "Input = [5.4 3. 4.5 1.5] , Label = 1\n", "Input = [6. 3.4 4.5 1.6] , Label = 1\n", "Input = [6.7 3.1 4.7 1.5] , Label = 1\n", "Input = [6.3 2.3 4.4 1.3] , Label = 1\n", "Input = [5.6 3. 4.1 1.3] , Label = 1\n", "Input = [5.5 2.5 4. 1.3] , Label = 1\n", "Input = [5.5 2.6 4.4 1.2] , Label = 1\n", "Input = [6.1 3. 4.6 1.4] , Label = 1\n", "Input = [5.8 2.6 4. 1.2] , Label = 1\n", "Input = [5. 2.3 3.3 1. ] , Label = 1\n", "Input = [5.6 2.7 4.2 1.3] , Label = 1\n", "Input = [5.7 3. 4.2 1.2] , Label = 1\n", "Input = [5.7 2.9 4.2 1.3] , Label = 1\n", "Input = [6.2 2.9 4.3 1.3] , Label = 1\n", "Input = [5.1 2.5 3. 1.1] , Label = 1\n", "Input = [5.7 2.8 4.1 1.3] , Label = 1\n", "Input = [6.3 3.3 6. 2.5] , Label = 2\n", "Input = [5.8 2.7 5.1 1.9] , Label = 2\n", "Input = [7.1 3. 5.9 2.1] , Label = 2\n", "Input = [6.3 2.9 5.6 1.8] , Label = 2\n", "Input = [6.5 3. 5.8 2.2] , Label = 2\n", "Input = [7.6 3. 6.6 2.1] , Label = 2\n", "Input = [4.9 2.5 4.5 1.7] , Label = 2\n", "Input = [7.3 2.9 6.3 1.8] , Label = 2\n", "Input = [6.7 2.5 5.8 1.8] , Label = 2\n", "Input = [7.2 3.6 6.1 2.5] , Label = 2\n", "Input = [6.5 3.2 5.1 2. ] , Label = 2\n", "Input = [6.4 2.7 5.3 1.9] , Label = 2\n", "Input = [6.8 3. 5.5 2.1] , Label = 2\n", "Input = [5.7 2.5 5. 2. ] , Label = 2\n", "Input = [5.8 2.8 5.1 2.4] , Label = 2\n", "Input = [6.4 3.2 5.3 2.3] , Label = 2\n", "Input = [6.5 3. 5.5 1.8] , Label = 2\n", "Input = [7.7 3.8 6.7 2.2] , Label = 2\n", "Input = [7.7 2.6 6.9 2.3] , Label = 2\n", "Input = [6. 2.2 5. 1.5] , Label = 2\n", "Input = [6.9 3.2 5.7 2.3] , Label = 2\n", "Input = [5.6 2.8 4.9 2. ] , Label = 2\n", "Input = [7.7 2.8 6.7 2. ] , Label = 2\n", "Input = [6.3 2.7 4.9 1.8] , Label = 2\n", "Input = [6.7 3.3 5.7 2.1] , Label = 2\n", "Input = [7.2 3.2 6. 1.8] , Label = 2\n", "Input = [6.2 2.8 4.8 1.8] , Label = 2\n", "Input = [6.1 3. 4.9 1.8] , Label = 2\n", "Input = [6.4 2.8 5.6 2.1] , Label = 2\n", "Input = [7.2 3. 5.8 1.6] , Label = 2\n", "Input = [7.4 2.8 6.1 1.9] , Label = 2\n", "Input = [7.9 3.8 6.4 2. ] , Label = 2\n", "Input = [6.4 2.8 5.6 2.2] , Label = 2\n", "Input = [6.3 2.8 5.1 1.5] , Label = 2\n", "Input = [6.1 2.6 5.6 1.4] , Label = 2\n", "Input = [7.7 3. 6.1 2.3] , Label = 2\n", "Input = [6.3 3.4 5.6 2.4] , Label = 2\n", "Input = [6.4 3.1 5.5 1.8] , Label = 2\n", "Input = [6. 3. 4.8 1.8] , Label = 2\n", "Input = [6.9 3.1 5.4 2.1] , Label = 2\n", "Input = [6.7 3.1 5.6 2.4] , Label = 2\n", "Input = [6.9 3.1 5.1 2.3] , Label = 2\n", "Input = [5.8 2.7 5.1 1.9] , Label = 2\n", "Input = [6.8 3.2 5.9 2.3] , Label = 2\n", "Input = [6.7 3.3 5.7 2.5] , Label = 2\n", "Input = [6.7 3. 5.2 2.3] , Label = 2\n", "Input = [6.3 2.5 5. 1.9] , Label = 2\n", "Input = [6.5 3. 5.2 2. ] , Label = 2\n", "Input = [6.2 3.4 5.4 2.3] , Label = 2\n", "Input = [5.9 3. 5.1 1.8] , Label = 2\n" ] } ], "source": [ "import numpy as np\n", "\n", "from matplotlib import pyplot as plt\n", "from matplotlib.colors import ListedColormap\n", "\n", "from sklearn import preprocessing \n", "from sklearn.preprocessing import StandardScaler\n", "\n", "# Load Iris classification dataset\n", "from sklearn.datasets import load_iris\n", "iris = load_iris()\n", "\n", "# Print all 150 examples\n", "print(\"(Number_of_examples, example_size) = \" , iris.data.shape, \"\\n\")\n", "for i in range(0, 150) :\n", " print('Input = ', iris.data[i], ' , Label = ', iris.target[i] )\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Building, training and evaluating a simple Decision Tree classifier**\n", "\n", "The SciKit-learn class for Decision Tree classifiers is sklearn.tree.DecisionTreeClassifier.\n", "\n", "**Please FIRST READ (and understand!) the [*DecisionTreeClassifier DOCUMENTATION*](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier) to understand all parameters of the contructor.**\n", "\n", "**You can then begin by running the code block below, in which default set of parameter values has been used.** If graphical view works, look at the structure of the learnt decision tree.\n", "\n", "**Then, check the influence of MAIN parameters for Decision Tree classifier, i.e.:**\n", " - **homegeneity criterion ('gini' or 'entropy')**\n", " - **max_depth**\n", " - **min_samples_split**\n", " \n", "NB : Note that post-training *PRUNING* IS unfortunately *NOT* implemented in SciKit-Learn Decision-Trees :(" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:327: FutureWarning: The parameter 'presort' is deprecated and has no effect. It will be removed in v0.24. You can suppress this warning by not passing any value to the 'presort' parameter.\n", " FutureWarning)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "DecisionTreeClassifier(criterion='entropy', max_depth=5,\n", " min_impurity_split=1e-07, presort=False)\n", "Acuracy (on test set) = 0.9111111111111111\n", " precision recall f1-score support\n", "\n", " 0 1.00 1.00 1.00 20\n", " 1 0.85 0.85 0.85 13\n", " 2 0.83 0.83 0.83 12\n", "\n", " accuracy 0.91 45\n", " macro avg 0.89 0.89 0.89 45\n", "weighted avg 0.91 0.91 0.91 45\n", "\n", "\n", " CONFUSION MATRIX\n", "[[20 0 0]\n", " [ 0 11 2]\n", " [ 0 2 10]]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABMgklEQVR4nO2de1xVVfrwvxvtgI4WmBeUNDWt35jT1cjS8C2dVHS8YNiICN6RiwLCq6+ZeEYczUtiKoIFAQZeBhwNcWJMfjCTKCLTiE3qqAiVgTfCiQJEPOv9gzk7Dnf0HDjA+n4++6Nn77XXXvthrees86xnPY8ihEAikUgkzYNFSzdAIpFI2hNS6UokEkkzIpWuRCKRNCNS6UokEkkzIpWuRCKRNCNS6UokEkkzIpWuRCKRNCMdW7oBkqbRqVOna2VlZb1auh3mgJWV1fXS0lLblm6HRNIUFLk5onWhKIqQf7NKFEVBCKG0dDskkqYgzQsSiUTSjEilK5FIJM2IVLrtgEOHDrF//35u3LiBr68vaWlpTJ8+nbKyMjw9PXF1dWX79u0ArF27ls2bN9dZV3l5eb3PysrKYv78+UyaNImLFy8aXPv2228ZNGgQANHR0UybNo1Fixah0+ke8A0lktaDVLrtgClTpnDs2DHeffdd3nnnHQCmT5+OlZUVYWFhxMbGkp2dDYCrq2uN++/du8enn37KggULSElJqfdZw4YNIyIigqCgIFJTUw2uRUVF8dvf/hYACwsLLC0t6dGjBxYWshtK2g+yt7cTXnnlFcrLy+nVq6bjw9GjRxkxYkSt9504cQJHR0cURSE8PJzx48fzzTff4Ofnpx579uypcV9sbCxTpkxRP6ekpDBy5Eg6dOgAwKxZs9izZw+9e/cmIyPDOC8pkbQCpNJtB5SUlJCens6AAQO4cOGCwbUvv/yS48ePM2fOnFrvfeaZZ5g8eTKJiYnExsZSWlqKTqejrKxMPe7evWtwzx//+EfefvttAwV/6tQp4uPjSUtLY8+ePShKpdNB9+7d+emnn4z8xhKJ+SL9dNsBmzdvJjAwkMcff5wlS5YYmBCmT5/OG2+8QVBQEGvWrKlxb5cuXfDy8gIgNTWVv/3tb4wbN47w8PBan3Xs2DH27dvHd999x+3bt7G1tSU7O1s1a/j4+ODi4kJUVBSnTp2ipKSEiIgIE7y1RGKeSD/dVoYx/HTT0tK4desWb731Vo1reXl5JCQkEBgY+EDPaA6kn66kNSLNC+0Qa2trUlNTKSsrq3EtISGBvn37tkCrJJL2gZzptjIedKbr4+PDjh071M9CCNW+aiyysrIICgpi7ty5BrNprVZLXl4evXv3Zv369WzevJnLly+TmppKZmYm7733HkVFRXzxxRd8/fXXDT5HznQlrRFp021jXLx4kQ8++IDS0lICAwPZtm0bgwcPJjs7m4iICDIzM9FqtQwbNoy4uDjGjx/PrVu3+Pbbb7GxscHd3R0PDw8cHBzo27cvQgiGDh1Kv379CAsLIygoqME2DBs2jGXLlnHr1i2D8w899BAdOnRQF9gCAwO5d+8eCxYs4JFHHmH9+vX88MMPBAcHm0Q2Eok5IM0LbYzIyEgeffRRevbsyZkzZ9DpdPj5+eHg4MCFCxewt7dHq9XSpUsX3nzzTdzc3Lh8+TJbt27lhx9+oLy8HHt7e1auXMnp06dxdnbmwIED7N27lxkzZqjPSUlJMXAby8zMbLBt77zzDpGRkVy7do38/Hy1njfeeEMtk5CQwLRp04wvGInETJBKt41RUVGBr68v7733Hi4uLmg0Gjp06EDHjh0pLy83MCV07doVoIZ5oaKiQv23c+fO6HQ6zp07x+DBgw3KVHUbu3fvXoNt0z+nW7dulJSUAJW75SZPnqyWOX78eJ0+wxJJW0CaF9oYHh4eLF26lG7dujF16tQa14cMGYK/vz+jR49Wzw0cOJCAgACsra3RaDRkZWXh7++Pvb09AK+99hrff/+9QT1jx45l7NixtbYhJyeHLVu2UFZWxhNPPMHJkycZPXo0hw8fJicnhw4dOjBo0CDu3LlDeXm5qvwLCgro3bu30W3MEok5IRfSWhmmDu1Y3WXs8uXLrF69mvDwcFU5mgtyIU3SGpFKt5Uh4+n+glS6ktaItOlK6kSr1dbwQLgfkpOTcXBwICsrC4DExEQ8PDxYsWIFAE5OTixatIioqKgHfpZEYu5Im24bJSYmhoyMDIYMGYK7uzsbNmwgPz+f5cuXs2/fPsrKyrhy5QqjRo0iLS2NdevWERcXh06n4+bNm6xatUqtKyoqiuzsbHQ6HZs2bcLd3Z0+ffrg6elpsLhWF+PGjePatWvq588//5xdu3axZs0abty4QadOnSgvL5ebMiTtAql02ygFBQW8/PLLODk5YWFhgU6nw8bGhsTERADc3Nw4c+YMGo2GFStWqGEY3d3dAYiPj1friouLY+TIkeTl5VFYWIgQgpkzZxoo3Pj4eNLT09XPPj4+auzc6ugXynr37s21a9eIjY1FCMGcOXMYM2aMcQUhkZgZUum2UZYvX87Jkyfx9vZm0qRJjBkzhj59+rB//34AevbsiUajoXv37mg0GjU4eUVFBdVtxnZ2dmi1WvVzaGgooaGhXL9+HUdHRwDu3r1rsK24vsDk+voLCgro1asXiqKgKAqWlpZGeXeJxJyRSreNEhkZyfnz5+nZsyfPPvss69evx87Ojo4d6/+TR0REkJ+fz8aNG/noo48AGD58OL6+vpSWlhIcHExwcDDFxcVMnDhRvc/FxQUXF5da68zMzCQmJoZu3boREhLC6NGj8fLy4uGHH6ZXr14sWLAAIQQvvvii8QQgkZgp0nuhlWFK7wWtVouPjw/du3c3Sf3GRnovSFojUum2MqTL2C9IpStpjUiXMYlEImlGpNJto/j4+BilHq1Wy5YtWwDw9PRk+vTpfPLJJ1RUVLBw4ULc3Nz46quvatyXkZGBh4cHEydO5Nq1a0RFRbFo0SKeffZZzp8/X6N8SEgIc+bMISAgAAAvLy/mzZvHxo0bAZg/fz4JCQlGeSeJpCWRSreV4uvri06nIzk5mSNHjhAVFYWfnx+hoaFqmejoaLKyssjLy2Pz5s1cvHgRb29v5s6dy7lz5xr9LDc3NwDCwsLYs2cPWVlZfPHFF4wfP54PP/yQ6OjoGvcMHz6cXbt24eLiQk5ODnPmzCE8PJxnnnmGX//61zXK+/v7ExUVpXpAWFhYEBkZyZUrV4DasxRLJK0RqXRbKa+++ionTpwgJSWFMWPGIISga9euJCcn13lP9bCPehobpvGf//wnY8eO5fXXX1eD01hZWanuZtUJCwtj165d/M///A9QGev3qaeeqrVscXExLi4udO7cGQCNRsPvfvc7XnjhhcaIQyJpNUil20pxdHTkyJEjlJeXY2lpyalTpwgODlaVFlQqroqKCoqKioCaYR/1NDZM4/PPP09KSgp//etfsbW1paCggDt37qDRaGot7+npyZYtWzh06BAAe/fu5fe//32tZbt27cqePXu4e/cuN27cQFEUDh8+3KQZuUTSGpB+uq2Url27kpOTo8aiVRSFjRs3qgoWYMSIEQQHB9OrVy9sbGxqhH10cHAA6g/TqKewsJCgoCAqKioYNmwYDg4OeHl5kZiYiJ+fH9evXyc9PR0nJycAkpKSSE5OpqioSM0EcfnyZXWXWmxsLJMnT1Yjl61atYqioiJ1w0ZhYSGenp5YW1sbVW4SSUsjXcZaGc3tMtZY391jx45hY2PT6A0OGzZsYPny5Y1uR20ZjKXLmKQ1Is0Lknrp378/e/fubbDcmDFjmrSjrCkKFypnzj169GjSPRKJOSJnuq2MTp06XSsrK+vV0u0wB6ysrK6XlpbatnQ7JJKmIJVuO0BRlM5AT+BxIAFwFEKcbtlWSSTtE7mQ1j7wBp4Dfgv8XipciaTlkEq3fTAN+DVwBPjRWJW2Z1OHNG1I7hepdNs4iqJ0AV4GfgYKgavGqrusrKxXezVPKYrSLr9sJA+OVLptn5+BucAeIcSdlm6MRNLekQtpkvumPYeZlD7CkvtF+ulKmoVDhw6xf/9+bty4ga+vL2lpaUyfPp2ysjI8PT1xdXVl+/btAKxdu5bNmzfXWVddsR6qUlxczFNPPWWQzfjKlStMnToVZ2dnvvrqK6Kjo5k2bRqLFi2qN72QRGJMpHnhPmlPi0jGWDSaMmUKCxYsICUlheDgYM6fP8/06dOxsrIiLCwMqAzfCJURxaqHcbx37x5JSUkkJSXh5OTE+PHj633erl27mDBhgsG5ixcvMn/+fLp3784//vEPLCwssLS0pEePHlhYyPmHpHmQSvc+aU+LSMZaNHrllVf4+9//Tq9evWrE1D169CgjRoyo9b4TJ07whz/8AW9vb8LDw+nQoQPffPMNISEhahl7e3s1iM+FCxfo2bMnP/30k0E9L774Iq6urlRUVLB792769OmDm5sbO3fuJCMjg+HDhxvjNSWSepFf7y1E9SDjplDgubm5uLu7M3v2bH7++Wf1/OnTp5kzZw6LFi0yyXNro6SkhPT0dAYMGMCFCxcMrn355ZccP36cOXPm1HrvM888w+TJk0lMTCQ2NpbS0lJ0Op1BZLS7d++q5b/44guOHz9OUlISUVFR6vk9e/awadMmEhISiIiIUFPBd+/evYaClkhMhVxIu08aWkS6ePEiH3zwAaWlpQQGBrJt2zYGDx5MdnY2ERERjBw5EkdHR4YNG0ZcXBzjx4/n1q1bfPvtt9jY2ODu7o6HhwcODg707dsXIQRDhw6lX79+hIWFERQU1GAbV69eja+vL2fPnuXmzZs4OzsDlQHQQ0JC2L17N08//TQvvfRSQ+9a66JRUxbS1qxZg7OzM48//jhLlizB1dVVDWAzaNAg3njjDWxtbVmzZg15eXkkJCQQGBhYo57U1FTu3LnDuHHjGnymPljPd999R3Z2Ns888wwbN26kY8eOzJ07l2+++YZTp05RUlJCREREnSEqa0MupEnuF2leMBH6gOHl5eWcOXMGnU6Hn58fUVFRXLhwAXt7e7RaLWlpabz55pu4ubnh5eXFzp078fX1pby8HHt7e1auXMnixYvZsGEDa9euxdbWlhkzZqjPSUlJ4fDhw+pnFxcX7O3tASgqKqJbt27Y2dlx9uxZtYwQAgsLC+zs7CgoKGgWeVT9koiIiCAtLU39fPny5UbX8/rrrze6rFarBSpnss8//zwA+/btMyhT1+xaIjEV0rxgIqoHDNdoNHTo0IGOHTtSXl6u/rQF1JiyVc/p69D/27lzZ3Q6HefOnWPw4MEGZeoKQG5tbU1RURH5+fnY2v6yDqYoCjqdrsb55sTa2prU1FQ1PU9VEhIS6Nu3bwu0SiIxPXKmayKqBwyvzpAhQ/D392f06NHquYEDBxIQEIC1tTUajYasrCz8/f3Vmetrr73G999/b1BPfQHIZ8+ezdKlSwHYvn07O3fuZPTo0bi4uLBw4UI6duyo5j9rbp577jk1n5uPjw87duxQrwUEBNT4AnpQsrKyCAoKYu7cuQYxeU+fPs3OnTuxtLQkLCzM6M+VSKojbbr3iak3BlS3a16+fJnVq1cTHh6uzoybiwe16ZqDfRtqD4TeVPt2QzKRSBpCmhfMlP79+xssJA0aNIi4uLhmV7jGoHpCTL1928HBwcC+3aVLF9W+ffnyZbZu3coPP/xgYN8+ffo0zs7OHDhwgL1799awbzcmwWZVWsK+LWnfSKVrxmi1WoMdVffLkSNHWLhwIU5OTpSWlpKYmIiHhwcrVqwwQisbxhzs23VhDvZtSftC2nRNRExMDBkZGQwZMgR3d3c2bNhAfn4+y5cvZ9++fZSVlXHlyhVGjRpFWloa69atIy4uDp1Ox82bN1m1apVaV1RUFNnZ2eh0OjZt2oS7uzt9+vTB09PTQOnUxYQJE5gwYQLr16+nsLCQzz//nF27drFmzRpu3LhBz549TSkKs7Bv5+TksGXLFsrKynjiiSc4efKk2di3Je0LqXRNREFBAS+//DJOTk5YWFig0+mwsbEhMTERADc3N86cOYNGo2HFihWkpqYC4O7uDkB8fLxaV1xcHCNHjiQvL4/CwkKEEMycOdNA4cbHx5Oenq5+9vHxUTPvQuWsOTMzE39/f3UW2bt3b65du2Zypfvkk08SExOjftZnIZ49ezYAw4YNq3FPQECA+v+8vDzGjh1rYN/es2cP4eHhjW7DE088ocoeUF3IAF5++eVG1yORPChS6ZqI5cuXc/LkSby9vZk0aRJjxoyhT58+7N+/H4CePXuq6cY1Go0axKWioqLGLjE7OzvV5xQgNDSU0NBQrl+/jqOjIwB37941cL+qHsBFq9Vy4MAB0tPT1foLCgro1cv8w0fUZd+WSFojUumaiMjISM6fP0/Pnj159tlnWb9+PXZ2dnTsWL/IIyIiyM/PZ+PGjXz00UcADB8+HF9fX0pLSwkODiY4OJji4mImTpyo3ufi4qLGHqjO7t27yczM5Pbt2+zYsYPi4mK8vLx4+OGHW4XShcangm+IqKgoTp06xcmTJ9m3bx9btmyhQ4cODBo0qNYdcBKJsZEuY/eJKVzGjKVYjM2DuowZw7794Ycf4uPjw+HDhx/Ivq1n1qxZfPLJJyxevJiysjJef/31Or+0miITiaQh5EzXjKhqQmhLmJt9++LFizz11FMAbNu2DUVRmDNnDm+//TYdOnQwnSAkEqTSlTQD5mbf3rt3LzNnzgR+cU3r0qULFRUVUulKTI5Uus1A9W2u94tWq+Xhhx9m6dKleHp6UlhYyO9+9ztmzJiBl5cXZWVl/N//+3/5zW9+Y3Bfbm4uWq0WRVEIDQ3lV7/6lcH1xMREjhw5Qrdu3Vi/fj2xsbFkZGQYpc1gXvZtqPR+0M98/9//+3/cvn0bW1tbLC0tjfC2EkkDCCHkcR9HpegqWbJkibh375747LPPRFJSkvj444+Fr6+v2LFjhxBCCG9vbxEVFSVOnz4tcnNzxaZNm8S///1v4eXlJebMmSO+/vpr0RhWr14tbt68qX6+e/euWLJkifjf//1f8ec//1mUlpaKpUuX1rgvKChIFBYWitTUVPGnP/2pxnUfHx8hhBB/+MMfxPXr19U26/nvu9YrA2NT/V3NjbpkIg95NHTIHWlG4NVXX+XEiROkpKQwZswYhBB07dqV5OTkOu+pvjVWT2O3sv7zn/9k7NixvP766xQUFNC7d2+srKxqzR9WNcRjbVtdq/vtmgNardbsFhQlEmMgzQtGwNHRkXXr1lFeXo6lpSWnTp1i165dvP3222oZjUZDRUUFRUVFQKW9ctmyZTz66KMGdem3suqpayvr888/T0pKCp6enjg7O1NQUMCdO3dqDcRdV4hHPUK0Lr9diaQ1I5WuEejatSs5OTlMnjwZqJw5bty4UVWwACNGjCA4OJhevXphY2NTY2usfpdWfVtZ9RQWFhIUFERFRQXDhg3DwcEBLy8vEhMT8fPz4/r166Snp+Pk5ATUDPH4+eefM2DAANWuOXr0aJP77ZrCrh0QEMDt27eZMWMGY8aMMSh36dIl3n//fa5evcr69evJzc3lL3/5C//6179YuXJljcSWMTExfPHFF9y5c4dPPvmE1atXc/XqVbp168amTZtYu3YtVlZW0pdX8uC0tH2jtR6Y0J5ZF421c37++eciKyurzushISGirKys3jrqs+lSGShpJC1o175y5YrYsmWLEEIILy+vOsunp6eL2NhY9fO8efPEnTt36izv5+cnysvL1ff38PAQQgi1zVVlAmhEM/Y5ebSNQ9p0WxH9+/dn7969DZYbM2YML774Yp3X/fz86l2pj42NrXWjgaIoTymKEgzkALuqXmtuu7bejg3UmT49Pj6ewMBANcvvjz/+iJWVVZ250Dw8PLhx4wYPPfQQAwYMYNKkSfTo0aPO9gPfK4qyU1GUVxQZ/VzSSKTSbUXMnj2bxYsXm/w5rq6u+Pr6GpxTFOU0kAb8CnAChla97ujoyJEjRwzs2sHBwXTu3FktU5tdu2rIRz2NCdFoa2urLgpW98PV4+zszOHDh4mOjgbg4MGDTJkypc733rVrFy+88AI5OTl8/fXXJCYmcufOnVoXJ/+LPfA9EAVcUhRFqyhK47fFSdol0qZ7n1hZWV1XFKVdrDpZWVmVlZWVrQT+VwhRoT9fPQ5uc9q1Bw4cyNWrV/Hw8FBt12FhYXh6egJw8uRJ4uLi+PHHH/H29gYqMwlHRkYC1LBrh4SEkJOTQ2lpKUuWLMHa2hpPT08qKirqnBkLIXKBPyqKsg54EZgFHFcU5QoQC+wXQjx4QGRJ26Kl7RvyMI8D6ACMAaKBIiAZmAl0qece0dzUZde+dOmSSEhIaHQ9jbFrV6U2m66oXSYPAeOBPcB/gERgOtCptvLyaH+HDHjTzlEU5VnAFXABrlE5Q9srhGjQYdfUeeJqIzo6muLi4mYxs1Rl8+bN9O3bV3UDbEzAG0VRugJTqZwBvwgcpFK+fxNC1G4TkbR5pNJthyiK8hiVStYVeASIA2KFEOeaUk+nTp2ulZWVtQsTS3WsrKyul5aWNjq/j6IodsAMKmX+KL/I/F8maqLETJFKt52gKMrDwDQqB/3zwAHgE+C4nHU1L4qi/IbKv8NM4CaVs989QgiZGbMdIJVuG0ZRlIeAsVQO8PFAKpWK9ogQoqy+eyWmR1GUDoADleaHqcBpKhXwn4UQP7Vk2ySmo1Uq3fb0s/Y+fsYqVLoyuQJvA5eoHMh/EkIUmqaVkgdFUZROwCQq/26vAUeo/Lt9Lqp4jDQWOUbMl1apdFtiAaelaGyGAkVRBlI5YF3/eyoWiBNC5JiyfRLjoyhKDyq/MF2B/sA+Kn+hfNnYji/HiPkila6ZU71DKYryJJXuSpcURXmUSnckV2AwsJ9KZZvZbgTUxvnv33smlX/jcn75Ms1TFMUG+LUQ4kQt97WbLiCVbjPQXjuUoihPAylAOPAc8DrwGZUD8a9CiLst1U6Jafmv2egVKpXvdOAc8BdgKTBPCHG4Wvl2OUZaA21W6R46dIg7d+7w+uuv88c//pGpU6eyc+dOdu/eTVhYGGfPnqVbt268//77DUaQKi8vr3NXEsBPP/3EkiVL6Ny5s0EkrfLyct59911KSkrw8vIiMzOTw4cP06NHD3bu3FlnzIBq74oQQlEU5TkgncpdhF8CHwIHhBA/NliJpE2hKIqGyoXRBcBoQAPMFULEVCnTKsZIeno6n3zyCd988w2TJk2iU6dO9z1GGixoJrTZ2AtTpkzh2LFjvPvuu7zzzjsATJ8+HSsrK/z9/YmKilLj1rq6uta4/969e3z66acsWLCAlJSUep/VpUsXPv744xrnDx06xM8//wxU5gGzsLDA0tKSHj16NKozVeNh4CvgFGANZEuF2z4RQpQD/ws8DvwDyAS6NLUecxgjI0aMIDw8nN/85jdMnTr1QcdIq6BtvtV/eeWVVygvL68RI7a4uBgXFxeDYCxVOXHiBI6OjiiKQnh4OOPHj+ebb74xiHy1Z8+eBp+fk5PD+PHjCQwMZNeuXcyaNYs9e/bQu3dvMjIymvQuQoi/CyGGCyEchBC/FkJ82aQKJG0KIUSxEOI3QoiRQohXhBCh91NPS4+R/74LN27cwNbW9oHGSGuhzSrdkpIS0tPTGTBgABcuXDC41rVrV/bs2cPdu3drjSD1zDPPMHnyZBITE4mNjaW0tBSdTmcQ+eru3YbNp7a2tlhbW2Ntbc3PP/+sBojp3r07P/0k3TAlLYs5jBGAv//974wcORKgXYyRNhtlbPPmzQQGBvL444+zZMkSg59Hq1atoqioCI1GU6sdqkuXLnh5eQGVkan+9re/MW7cOMLDw+t8nre3N3/729/Yv38/Tz75JNnZ2UydOpWlS5ei0+lYvnw5UVFRnDp1ipKSEiIiIoz/0hJJEzCHMTJ79mwOHDjAH/7wB4B2MUba7EJaddLS0rh16xZvvfVWjWt5eXkkJCSYZSqW1rZIIDEP5BgxX9qseaE61tbWpKamGiR91JOQkEDfvn1boFUSifkgx0jz0C5mutWTIgohDAJwG4Pc3Fy0Wi2KohAaGsqvfvUrAJKTk1m3bh1btmxh2LBhhIaGkpWVBVT+lAoJCTFwzalOa/sWl5gHrWmMbN26la+++ooOHTrw4YcfsmnTJi5evEjfvn0JCgrC09OT4uJiXn755VpDera2MdImbLoXL17kgw8+oLS0lMDAQLZt28bgwYPJzs4mIiKCzMxMtFotw4YNIy4ujvHjx3Pr1i2+/fZbbGxscHd3x8PDAwcHB/r27YsQgqFDh9KvXz/CwsIICgpqsA3R0dGqAv3LX/6Cs7MzAOPGjePatV9C0164cIGoqCh8fX0pLy/H398fQM1uIJGYAnMeI35+fgAsXryYsrIyzp49yyeffEJAQAA3b94kLCwMgPnz55tMPs1JmzAvVE9wqNPp8PPzw8HBgQsXLmBvb49Wq6VLly68+eabuLm5cfnyZbZu3coPP/xAeXk59vb2rFy5ktOnT+Ps7MyBAwfYu3cvM2bMUJ9TX8LEoqIiunXrhp2dnZq7qzZee+01HB0d+fnnn9FoNA265kgkxsDcx0h+fj6WlpZYWVnh5OTEkiVLuHLlCtevXwfg6NGjjBgxonmEZWLahNKtnuBQo9HQoUMHOnbsSHl5eY1cXkCNn04VFRXqv507d0an03Hu3DmDrLj1JUy0tramqKiI/Px8bG3rDniUnJzMX/7yF55++mm++eabBl1zJBJjYM5jpKioCK1WS3BwMABTp05l27Zt9O/fn8cee4wvv/yS48ePM2fOHOMLpgVoE+aF6gkOqzNkyBD8/f0ZPXq0em7gwIEEBARgbW2NRqMhKysLf39/7O3tgcoZ6ffff29QT30JE2fPns3SpUsB2L59Ozt37mT06NH85z//ISYmhm7duhESEsLgwYNZtGgR//nPf/D09GzQNUciMQbmPEaCg4MpKyvD39+f9957j4MHD5KZmcngwYOxtrZm+vTpvPHGGwQFBbFmzRpjiaTlaOkkbfdzYOSEiNWTDl66dEm4uLiIH3/80ajPuR+oIwGiPORR3yHHiPke7cJ7oTXT2lZmJeaBHCPmS5uw6ZoCrVbLrVu3HrieI0eOsHDhQpycnCgtLeUf//gHPj4+rFixwgitlEhaDlONkU2bNrFgwYK2YUqohTZh061OTEwMGRkZDBkyBHd3dzZs2EB+fj7Lly9n3759lJWVceXKFUaNGkVaWhrr1q0jLi4OnU7HzZs3WbVqlVpXVFQU2dnZ6HQ6Nm3ahLu7O3369MHT09NgAaEuJkyYwIQJE1i/fj2FhYVERUWh0Wjo3r27KUUgkdSLOY+R6i5jPXr0MKUomp02qXQLCgp4+eWXcXJywsLCAp1Oh42NDYmJiQC4ublx5swZNBoNK1asIDU1FQB3d3cA4uPj1bri4uIYOXIkeXl5FBYWIoRg5syZBp0pPj6e9PR09bOPjw+DBg1SP2u1WjIzM/H39+df//oXn3/+OevWrePq1as89thjJpWFRFIb5jxG9C5j3333HdevX5dKtzWwfPlyTp48ibe3N5MmTWLMmDH06dOH/fv3A5WxbfWzTY1Go7pqVVRUUN0OZmdnh1arVT+HhoYSGhrK9evXcXR0BODu3bsGWyd1OsOM5lqtlgMHDpCens4TTzzBQw89xCOPPEJJSYkpXl8iaRBzHiNTp05l6tSp+Pv7t8lJSZtUupGRkZw/f56ePXvy7LPPsn79euzs7OjYsf7XjYiIID8/n40bN/LRRx8BMHz4cHx9fSktLSU4OJjg4GCKi4uZOHGiep+LiwsuLi611rl7924yMzO5ffs2O3bsoLS0FC8vLywsLHjyySeN99ISSRMw5zESFRVl4DLW1pDeC/9Fq9Xi4+NjdrbW1rYyKzEP5BgxX6TSNXNaW4eSmAdyjJgv0mVMIpFImpE2rXR9fHyMUo9Wq2XLli0ABAQEMG/ePI4dO1aj3KVLl1i0aBETJ07kq6++Ijo6mmnTprFo0aIaCwdQuer7wgsvqL6OsbGxRmuzRNIYmnuMJCcns2DBAqZMmUJhYWGN69V9djdv3syiRYt46qmn+M9//tMmxkirVrq+vr7odDqSk5M5cuQIUVFR+Pn5ERr6S46+6OhosrKyyMvLY/PmzVy8eBFvb2/mzp3LuXPnGv0sNzc3cnNzeeyxx4iMjOTgwYM1ygwePJjw8HDeeecdzp4922Bm05kzZzJp0iT1c20ZVyWSB8Hcxsi4ceP46KOPcHV1VeNKV2XChAl8+OGHvPTSSxQWFhIYGEhoaCgjRozgkUceaRNjpFUr3VdffZUTJ06QkpLCmDFjEELQtWtXkpOT67yneog7PfWFpNNTUFBA7969AepMDx0fH09gYCDDhw9vF5lNJeaNOY6RiooKjh49yqhRo2q9rtVq+eKLL9QFu5SUFN54443GvrLZ06qVrqOjI0eOHKG8vBxLS0tOnTpFcHCwQWxajUZDRUUFRUVFQM0Qd3rqC0mnx9bWVo0DWpu5AMDZ2ZnDhw8THR3dLjKbSswbcxsjQgiWLVvGsmXLsLKyqrXNWq2WefPmqZspDh06xOTJk+9fCGZGq/bT7dq1Kzk5OeofRFEUNm7cqHYegBEjRhAcHEyvXr2wsbGpEeLOwcEBqD8knZ6BAwdy9epVPDw8cHJyAiAsLAxPT08ATp48SVxcHD/++CPe3t41Mpt+/vnnDBgwQN2J89lnn5GUlEReXp5B+hKJxFiY2xiJjIwkIyODkpIS5s6di5WVFcXFxWqA8uo+u3fu3KG8vFyN8dsmaOkwZ/dzYOSwdQ2xevVqcfPmzRrnL126JBISEhpdT0hIiCgrK6u3jLe3t8FnWlnYOnmYx9Faxkh0dLQoKCho0rNa+xiRfrqNIDo6muLi4lqT4hmT2NhYCgsL8fX1Vc+1Nh9EiXkgx4j5IpWumdPaOpTEPJBjxHxplTZdKyur64qi9GrpdjQHVlZW11u6DZLWhxwj5kurnOk2BkVRhgFjgGeALsA0IcTdlm2VRCKpC0VR1gLjgM3AQ0KIT1q4SSahLSvdDwB7oAKYIIT40RTP6dSp07WysrI2OaOwsrK6XlpaWndqY0mdyH7RdBRFsQAigBeAn4QQI439DHOgLSvdQsASuAvMFULU3B5jnOe0WdtZa7OVmROyX9xXvU8BGcAdoBfQQwjx4PmAzIxWvTmiLhRF6UjlH24j8JSpFK5EIjEeQoh/A32Bd4DvgP9p2RaZhjY7020u5IxGUhuyX0jqok3OdCUSicRcaZLS7dSp0zVFUUR7ODp16nTtQYV76NAh9u/fz40bN/D19SUtLY3p06dTVlaGp6cnrq6ubN++HYC1a9eyefPmOuvS56iqj+LiYp566imDtNjl5eUsW7YMHx8fzp0712C4SYnpqa9fhISEMGfOHAICAoAH7xc//fQTc+fOrREOMS0tjTfffJNFixZx8+ZNk/QLqS9qp0lKt6ysrFdLb6FrrsMYK89Tpkzh2LFjvPvuu7zzzjsATJ8+HSsrK8LCwoiNjSU7OxuoPazjvXv3+PTTT1mwYAEpKSkNPm/Xrl1MmDDB4NyhQ4f4+eefgcpkgw2Fm5SYnvr6hb+/P1FRUWoSxwftF126dOHjjz+ucV5RFDp16oSlpSWPPPKISfqF1Be1I0ediXnllVcoLy+nV6+af5OjR4+qgT6qc+LECRwdHVEUhfDwcMaPH88333xjEFpvz549avkLFy7Qs2dPHn74YYN6cnJyGD9+PIGBgezatUuGmzQT6uoXxcXFuLi4GEQBq0pT+0VdODg48OmnnzJ27Fj+/Oc/y37RjDSr0q3+E8cUCw25ubm4u7sze/ZsdYYHcPr0aebMmcOiRYtM8tzaKCkpIT09nQEDBnDhwgWDa19++SXHjx9nzpw5td77zDPPMHnyZBITE4mNjaW0tBSdTmcQWu/u3V/2enzxxRccP36cpKQkoqKi1PO2trZYW1tjbW3Nzz//LMNNmgH19YuuXbuyZ88e7t69W6vpoKn9oi6q9wNz7BdtVV80yXuhoRXZixcv8sEHH1BaWkpgYCDbtm1j8ODBZGdnExERwciRI3F0dGTYsGHExcUxfvx4bt26xbfffouNjQ3u7u54eHjg4OBA3759EUIwdOhQ+vXrR1hYGEFBQQ22cfXq1fj6+nL27Flu3ryJs7MzUBlBPyQkhN27d/P000/z0ksvNfSujVqhrU8ma9aswdnZmccff5wlS5bg6urKrVu3eOuttxg0aBBvvPEGtra2rFmzhry8PBISEggMDKxRT2pqKnfu3GHcuHENvr8+Y+t3331HdnY2U6ZMYenSpeh0OpYvX05GRoZBuEmNRvPAMpDU5H77xapVqygqKkKj0bBlyxaj9Atvb2/+9re/sWrVKp588kmys7N59NFHSUpK4j//+Q+hoaEkJiYavV9IfVE7Ro29oI84X15ezpkzZ9DpdPj5+REVFcWFCxewt7dHq9WqRnw3Nze8vLzYuXMnvr6+lJeXY29vz8qVK1m8eDEbNmxg7dq12NraMmPGDPU5KSkpHD58WP3s4uKCvb09AEVFRXTr1g07OzvOnj2rlhFCYGFhgZ2dnRpk2dRU/aNHRESQlpamfr58+XKj63n99dcbXVar1QKVM5bnn38ewMCm9+tf/7rO2bWkeaivXwQHBze6nsb2i6qpeQC1X/zud79Tz82ZM6fZ+0V71RdGNS9Ujziv0Wjo0KEDHTt2pLy8XP0JA6hBiaue09eh/7dz587odDrOnTvH4MGDDcrUFcHe2tqaoqIi8vPzsbX9ZaeioijodLoa55sTa2trUlNT1UWSqiQkJNC3b98WaJWkpWmv/aK96gujznSrR5yvzpAhQ/D392f06NHquYEDBxIQEIC1tTUajYasrCz8/f3Vb6LXXnuN77//3qCe+iLYz549m6VLlwKwfft2du7cyejRo3FxcWHhwoV07NgRNzc3Y71yk3juuefUWYePjw87duxQrwUEBNToUA9KVlYWQUFBzJ07l7feeks9f/r0aXbu3ImlpSVhYWFGf66k8ej7gb5fCCEM/h61mRWaSm5uLlqtFkVRDDKU7N69m9TUVB555BG2bt1KXFwc77//PkePHqV79+6sXr2aq1ev0q1bNzZt2vTA7ahOe9UXRrXpPijV7VeXL19m9erVhIeHN3u6jge1W5mDvQoq/TH19kI9jbVXSZvu/aPvF+bQD+qyW7q7uxMdHc22bdsYMWIEw4YNU9cEunfvrn4hLFq0iPDw8KrvZhSb7oPSGvUFmJnLWP/+/Q2+2QcNGkRcXFyrzI9UPaOq3l7l4OBgYK/q0qWLaq+6fPkyW7du5YcffjCwV50+fRpnZ2cOHDjA3r17a9irGsrQWp2WsG+3V8yhH1S1W1b9e8+fPx8fHx/++c9/cu1aTd/+AQMGMGnSJHr06GFaId0nrVVfmJXShcqFoKo7qu6XI0eOsHDhQpycnCgtLQUqNw9Ud0MxFeZgr6oLc7BvtxfMoR/UZbd87bXXCA0NZejQoQwYMKBG27/++msSExPV5JDmiKn0RUxMDPPnz2fWrFlGaKUhRrXpxsTEkJGRwZAhQ3B3d2fDhg3k5+ezfPly9u3bR1lZGVeuXGHUqFGkpaWxbt064uLi0Ol03Lx5k1WrVql1RUVFkZ2djU6nY9OmTbi7u9OnTx88PT0NOltdTJgwgQkTJrB+/XoKCwvp0KFDs257NQd7VU5ODlu2bKGsrIwnnniCkydPmo19u71gDv2gLrvlv//9b5KTk7G2tubpp5+ukZ3a2toaT09PKioq6nUhu1/MWV+4u7vj7u6Ov78/d+/e5aGHHjLaexvVpvvee+9ha2uLk5MTFhYW/PGPf+TOnTvY2tpSUlLC9OnTOXPmDBqNhoEDB5KVlUV+fr76bZKUlERRURE+Pj78/ve/Z+TIkeTl5bFu3Tr8/f1ZtmwZL774ovq8+Ph40tPT1c8+Pj5qenOo/BbMzMzkz3/+M++99x4rVqwgICDAYAGrnndtUbuVOdirpE33/jFWvzCHflAdY40Nc9YXVlZWeHh48NNPPxEXF2c0mYCRzQvLly/nySefxNvbm88++4wxY8awYMECSkpKgMq9/xqNhu7du6PRaNSfLBUVFTV20djZ2aHVaomOjqZPnz6EhoaSlJTEX/7yF7XM3bt3DX5SVZ/JarVa5s2bR3p6Ov/4xz/U4CK5ubnGfG2T0FrtVRLj0pb7gTnrC6g0R77wwgvk5OQY9b2Nvjni/Pnz9OzZk2effZb169djZ2dHx471PyYiIoL8/Hw2btzIRx99BMDw4cPx9fWltLSU4OBggoODKS4uZuLEiep9Li4uuLi41Frn7t27yczM5Pbt2+zYsUN1jvbx8anVfmXOVF1RfhC2bt3KV199RYcOHfjwww+N1DpJS2GsfpGXl8f777+PhYUFGzZswMrKykgtrB9z1hchISHk5ORQWlrKkiVLjPfSmIHLmLE6jrEx1k8oY9itPvzwQ3x8fDh8+PAD2a30LF68mE2bNjU4uKR54f5pTf3inXfeUe22wcHBDfptt6TprbXrCzCDFOz6battlYKCAl5++WXVbqXT6bCxsSExMREANzc31W61YsUKUlNTgUofSqi0Q+mJi4tT7VaFhYUIIZg5c6bBwGrIbpWfn4+lpWWzzWYktWNO/eLs2bN88sknfPrpp5w4caLOyHfmQFvQFyZzGTOWa5ZWq2XLli1A5a6tefPmcezYsRrlLl26xKJFi5g4cSJfffUVULnzauTI2hOKJiYm4uHhwYoVKwCIjY01iTuZOdmtioqK0Gq1TdrfLzEN5tQvHnvsMbp27apGomtumltX3L59m9mzZzNr1izy8/NrXG/IfexBdcV9K11fX190Oh3JyckcOXKEqKgo/Pz8DIJrREdHk5WVRV5eHps3b+bixYt4e3szd+5czp071+hnubm5kZuby2OPPUZkZCQHD9bMMzl48GDCw8N55513OHv2LEIIkpKSeO6552qt8/PPP2fXrl106tSJGzdu1Bos2hhERkZy4MAB1W4VGxvbqNXQiIgIgoODDXaS6e1WCxcu5Pr162i1Wq5cuWIQk9XFxYXw8HD1ePLJJ9Vrixcv5ocffsDf35/bt28b9T0lTcOc+sX8+fNZtGgRiYmJjBo1yrgvivnpioMHD+Lr60twcHCtMp8wYQIffvghL730kuo+FhERQffu3bl79+4D64r7Ni+8+uqrnDhxgpSUFNauXauuqCYnJ+Pt7V3rPdWjCg0ZMgSoPwqQnoKCAnr37g1QZ2T7+Ph4QkJC+OSTT9i9ezczZsyo0z1Mb7fq3bs3165do2fPnk0TQCOZP3++weeqsW71VB1AQ4cORavVsnz5ctVupf9J5enpaXBfY1zfqhIbG9uk8hLTYU79YtiwYURERDTpnqZgbrqioKCA8ePH8+ijj9a5I1PvPubv7w+guo8Zw1/3vpWuo6Mj69ato7y8HEtLS06dOsWuXbt4++231TIajYaKigqKioqAyp9Gy5Yt49FHHzWoS7+bRk9tu6psbW05deoUQJ2bHJydnXnjjTfYunUrJSUlnDx5kuPHj/PXv/61huO43sBfUFBQa1aHlqQt2K0kxqe19gtz0xW2trYUFBSoPsG1odVqOXDgAOnp6YwePZpdu3bx/vvvk5OTwxNPPNF0IVThvs0LXbt2JScnR/2WURSFjRs3qkIDGDFiBBERESQkJAC/7M7x9/fn73//u1pu7NixBj99XnnllRrPGzhwIFevXsXDw4NJkyYBEBYWpl4/efIkPj4++Pv7M3HiRN5//33Cw8P5P//n/zB27Fg+//xzgxi2o0ePxsvLi5KSkmZTus1tu8rLy+Ptt9+uM7FhcnIyDg4OZGVlAZXO5qYys0hqp7n7RHJyMgsWLGDKlCkUFhbWWtfBgweZPn06YJw+YW66YurUqWzbto2goCBmzpzJ2bNnDRYZd+/ejY+PDwcPHuTFF18kJCRETezar1+/B5IFQJOSr1UWb15Wr14tbt68WeP8pUuXREJCQqPrCQkJEWVlZfWW8fb2Vv//33dtkkyWLFki7t27Jz777DORlJQkPv74Y+Hr6yt27Nih1h8VFSVOnz4tcnNzxaZNm8S///1v4eXlJebMmSO+/vrrRr2LXiZXrlwRW7ZsEUII4eXlVWtZ/XPqQt+eB5GBPOruF+bYJ4QQIj4+XiQnJ9c4//PPP4v169cb9IOq/xfi/sZGc3C/uiI6OloUFBQ06Vn3KxMhhPkFvKlO//792bt3b43zgwYNYtq0aY2ux8/PD0tLyzqvx8bGNsnXtTaq2q7GjBmDEEK1XdVF9ShUehoTPawxdm5Jy2KOfaKiooKjR4/WumgWFhZWw97cWrhfXeHu7t6kwE8Pqita3E+3IWbPnt0szzHGz2pzs11JWh5z6xNCCJYtW8ayZctq9dX+6quvePfddzl+/DhZWVkMGzbs/l68BWgtuqJJStfKyuq6oijmtepkIqysrK439R697Wry5MlA3bar4OBgevXqhY2NTY0oVA4ODkD9UaP0VLVdOTk5AZUzFf1q9u3bt1m5ciVXrlzhueeeo2fPnhQXF6vO75mZmcTExNCtWzdCQkKMY6+SGGBufSIyMpKMjAxKSkqYO3cuVlZWBn0iOjoaqLQ1P6jClfqiDhprh5CHedithDCt7UradFtnv2iuPiGE7BcPekhDYCvEVLarpKQks80SIKkf2SdaD00KeCOpianzQLUkMuDN/SP7haQuzH4hzdxpy3ar+7FrSyqR/UJSF3Kma2IURZkLpAKfA1uEEDtbsC0vAMnA74EngE+EEGX13yUxBYqieAFHgTQgQAixv2VbJGkupNI1IYqidAEKgFwgXgjR4uG9FEUZBcQD3wFBQogjLdykdoeiKHbAv4BC4H0hRFgDt0jaEHIhzbSMp9KEIwCHFm6LnjeAK8DTQO1h9CWmZipgBdym8u9hEjp16nRNURTRVo9OnTrVzBvfCpA2XdMyF7gLHABiWrgtejYD3wDLqfxSkDQ/7kAJsB8wWei3srKyXm35l2xrtZlL84IJURTFgv/6a7Z0W2pDURQLIYTcytbMNFe/aMseFNB6vSjkTNeEmLtCM/f2tVWk3Ns3rdqm25ZtVk21V0lZtH053E+/qI9Dhw6xf/9+bty4ga+vL2lpaUyfPp2ysjI8PT1xdXVl+/btAKxdu7bOEKGAmk6oLvLy8njllVdYtGgRFy5cMLj27bffqvnaoqOjmTZtGosWLWqz8URa9Uy3LduslCbaq6QsKmnLcgDj2jGnTJnCggULSElJITg4mPPnzzN9+nSsrKzU+LP6iGOurq5qrFs99+7dIykpiaSkJJycnBg/vv4lgs6dOyOEqBG/Oioqit/+9rdAZWQ0S0tLevTo0WYj57XNt5JIJI3ilVdeoby8vNZA/kePHq0zM/CJEydwdHREURTCw8MZP34833zzjUHoyT179qjlH3/8cVJSUvD19WXXrl3q+ZSUFEaOHEmHDh0AmDVrFnv27KF3795kZGQY+W3Ng3apdKtH6zfFzCg3Nxd3d3dmz55tkGH19OnTzJkzh0WLFpnkuU1FyuIX2pssSkpKSE9PZ8CAATV+8n/55ZccP36cOXPm1HrvM888w+TJk0lMTCQ2NpbS0lJ0Op1BxuGqWYuV/+Yk7N69Oz/99JN6/tSpU8THx5OWlsaePXvqLNeWaNXeC4pS++rsxYsX+eCDDygtLSUwMJBt27YxePBgsrOziYiIYOTIkTg6OjJs2DDi4uIYP348t27d4ttvv8XGxgZ3d3c8PDxwcHCgb9++CCEYOnQo/fr1IywsjKCgoAbbtnr1anx9fTl79iw3b97E2dkZqMyMGhISwu7du3n66ad56aWX6nq3Jq3MSlnULwcpi5qsWbMGZ2dnHn/8cZYsWYKrqyu3bt3irbfeYtCgQbzxxhvY2tqyZs0a8vLySEhIIDAwsEY9qamp3Llzh3HjxtX5rIyMDCIjIykuLmbt2rUUFxeTnZ2txsD18fFhx44dREVFcerUKUpKSoiIiECj0dT3ftJ7wVyonklUp9Ph5+dHVFQUFy5cwN7eHq1WS1paGm+++SZubm54eXmxc+dOfH19KS8vx97enpUrV7J48WI2bNjA2rVrsbW1ZcaMGepz6stMWlRURLdu3bCzs+Ps2bNqGSEEFhYW2NnZ1ZmJVMpCyqI5qPolERERQVpamvq5aj7Bhnj99dcbLDN8+HCGDx9ucO75559X/6/PYDxnzpw6Z9dthTapdKtH3j9x4gQdOnSgY8eOlJeXqz9hoDLINGBwTl+H/t/OnTuj0+k4d+4cS5YsMShTVyR/a2trioqKyM/PNwidpygKOp2O/Px8fv3rXxvxrWtHysLwPaQs6sba2pr4+HgmTpxYI6tEQkICffv2bZF2tTXapNKtHnm/OkOGDMHf35/Ro0er5wYOHEhAQADW1tZoNBqysrLw9/dXZyivvfYa33//vUE99UXynz17NkuXLgVg+/bt7Ny5k9GjR+Pi4sLChQvp2LEjbm5uxnrlOpGy+AUpi7rR/7wPDQ0FKmfeVb9wajMrNJXc3Fy0Wi2KohAaGsqvfvUroDL7bmpqKo888ghbt24lLi6O999/n6NHj9K9e3dWr17N1atX6datG5s2bXrgdrQ4LR1F/UEOTBSdv3oG3UuXLgkXFxfx448/muR5tUETo/NLWZhWDkK0bllUzzDs4eEhNm/eLGbNmiXu3LkjXnrpJbF69Wpx+PBh8fvf/17ExMSI999/X/j6+gqtVityc3PFm2++KdauXStiYmJEdHS0yMrKEjdu3BB/+MMfGtX2oKAgUVhYKFJTU8Wf/vQn9bybm5vQ6XRi69atambqqpkw9JkrPDw87lsW5nS0S++Fhujfv7/BN/ugQYOIi4tTf3K2J6QsfqE1y6J6hmG9PdvBwcHAnt2lSxfVnn358mW2bt3KDz/8YGDPPn36NM7Ozhw4cIC9e/fWsGfXlbG4qj27qt16/vz5+Pj48M9//pNr12ru/RgwYACTJk1qMxkspNJtAK1Wy61btx64nuTkZBwcHMjKyjJCq1oGKYtfMJYsjhw5wsKFC3FycqK0tNQILaudiooKfH19ee+993BxcUGj0RjNnl01Hbnenq0/GmPPfu211wgNDWXo0KEMGDCgRtu//vprEhMTuXPnToM731oDbdKmqycmJoaMjAyGDBmCu7s7GzZsID8/n+XLl7Nv3z7Kysq4cuUKo0aNIi0tjXXr1hEXF4dOp+PmzZusWrVKrSsqKors7Gx0Oh2bNm3C3d2dPn364OnpadDp6mLcuHG1fos3F1IWv2BOspgwYQITJkxg/fr1FBYW8thjj5nknc3Znv3vf/+b5ORkrK2tefrpp/nss89ISkoiLy+P0NBQrK2t8fT0pKKiol4XstZCm1a6BQUFvPzyyzg5OWFhYYFOp8PGxobExEQA3NzcOHPmDBqNhhUrVpCamgpUJusDiI+PV+uKi4tj5MiR5OXlUVhYiBCCmTNnGgys+Ph40tPT1c8+Pj7qnvKWRsriF8xNFlqtlszMTPz9/U32zk8++SQxMb9EF9Wnddf7ydaWbj0gIED9f15eHmPHjlXNK5cvX2bPnj2Eh4c3ug0DBw4kKipK/ezl5QXAU089xaRJk9Tz48ePN9hSvGXLlkY/ozXQppXu8uXLOXnyJN7e3kyaNIkxY8bQp08f9u+vzIzSs2dPNBoN3bt3R6PRqD9dKioq9AsRKnZ2dmi1WvVzaGgooaGhXL9+HUdHRwDu3r1r4CpkTgE7pCx+wdxkodVqOXDgAOnp6QYzTXOiLnu2pOm0aaUbGRnJ+fPn6dmzJ88++yzr16/Hzs6Ojh3rf+2IiAjy8/PZuHEjH330EVDp3O3r60tpaSnBwcEEBwdTXFzMxIkT1ftcXFxwcak9GUNmZiYxMTF069aNkJAQ+vXrZ7wXbQRSFr9gTrLYvXs3mZmZ3L59W90g0NrQarX4+PjQvXv3B6rnyJEjfPrpp9y6dYu4uDg6depkpBaaF21yG/CDYKwO9KAYaxvwg9AaZWEKOUDblYUx7NsffvghPj4+HD58+IHs23rWr1/PrFmzGrRvy23AbYSqPxXbO1IWv9BWZdEe7dstjVS6Ekk7Rtq3m5924adbPWTf/aLVatWV1ICAAObNm8exY8dqlLt9+zazZ89m1qxZ5Ofn17he3U81KSkJV1dXo7SxIZpbFnl5ebz99tt1Zh2oHtKwLcuioX5R3We3OWQRGRnJgQMHVPt2bGxsoxbIIiIiCA4O5q233lLP6e3bCxcu5Pr162i1Wq5cuWIQq9fFxYXw8HD1ePLJJ9Vru3fvxsfHh4MHD/Liiy8a90XNiZbeEvcgB//d5rhkyRJx79498dlnn4mkpCTx8ccfC19fX7Fjxw4hROU2wqioKHH69Gl1K2f1bZGNQb818cqVK2LLli1CCCG8vLxqlPv444/Fl19+KXJzc8XGjRtrrUvfHj36rY56uM9twOYmCyFqbp+tir69UVFRIjMz84FlQZWtr+Ymi8b0CyGEWLdunfjuu++MKgtjUnWLbkvS1DFiLkebmOm++uqrnDhxgpSUFMaMGYMQgq5du5KcnFznPdW3ReqpbxujnoKCAnr37g1Qa0oR/fXmDNOnx9xk0RBCmC6kobnJojH9QqvV8sUXX7T4gl19aLVas26fudMmbLqOjo6sW7eO8vJyLC0tOXXqFLt27eLtt99Wy2g0GioqKigqKgJqhvnTU19YPj22tracOnUKqN3/1NbWloKCAu7cuWOw3bE5MDdZNIQpQxqamywa0y/ag02zvdMmlG7Xrl3Jyclh8uTJQOVA3rhxozqQAEaMGEFwcDC9evXCxsamxrZI/Q6d+rYx6hk4cCBXr17Fw8MDJycnAMLCwvD09ARg6tSpLF26FJ1Ox7p16zh79izFxcVqvilT+qmamyxu377NypUruXLlCs899xw9e/Y0kIUpQxqamywa6hct6bOrD+34oGi1Wh5++GGWLl1KQEAAt2/fZsaMGYwZM8agXHJyMgcOHODmzZvqr4vq19etW8eWLVsYNmwYSUlJ7Nu3j9jY2AduY4vT0vaNBzkwYRi/2qjLlnXp0iWRkJBQ533R0dGioKCg3rqNZdNtLsxVFs0tByFahyzMzb6tJz4+XiQnJ9d6zdjrHuZytAmbbnPRv39/9u7dW+P8oEGDmDZtWp33ubu712tmSEpKanVh66QsfqE1yMLc7NtQabI5evQoo0aNerCXa2W0avOClZXVdUVRauaONjFVU7MYk6o+jlZWVtebcq+UxS9lW0IOYH6yqIq52beFECxbtoxly5bVSA3U1mnVSre0tLR5V6nMGCmLSqQcasfc7NuRkZFkZGRQUlLC3LlzsbKyarZ1j5amVcdekEgkdWOqOBT1UVeMisuXL5OdnV2nuSUmJoaxY8fWa26pvtjXWmMvSJuuRCIxGq3Bvt3SyJmuRNJG6dSp07WysrIWsW83B1ZWVtdbozlJKl2JpJ2iKMqjQBdgOLAFeE0IcaWF2tIbOA5sAj4FEEI073bOZkIqXYmknaIoymbgUcAR+K0Q4mwLt2cg8AVwBLAUQri3ZHtMRav2XpBIJA/EW1Qq3cOAOcy+BPC/wHRAKIpiIYQwnzxPRkIupEkk7RBFUf4HeBwoAS4B37RsiwAoAM4BPwLWwBst2hoTIc0LEkk7RFGUjsAE4LC5zSYVRVGobFuKEKK0pdtjbKTSlUgkkmZEmhckEomkGZELaRKJCWjLPrJN9Y+VsjBEmhckEhPQEltwm4umbr+VsjBEmhckEjOierJMUyir3Nxc3N3dmT17Nj///LN6vnqS0JamrcpCznQlEhNQ1+zu4sWLfPDBB5SWlhIYGMi2bdsYPHgw2dnZREREMHLkSBwdHRk2bBhxcXGMHz+eW7du8e2332JjY4O7uzseHh44ODjQt29fhBAMHTqUfv36ERYWRlBQUINtW716Nb6+vpw9e5abN2/i7OwMgK+vLyEhIezevZunn36al156qa53M8pMtz3KAuRMVyJpVqoHBtfpdPj5+eHg4MCFCxewt7dHq9XSpUsX3nzzTdzc3Lh8+TJbt27lhx9+oLy8HHt7e1auXMnp06dxdnbmwIED7N27lxkzZqjPqS/QeFFREd26dauRIFMI0yUJlbL4BbmQJpE0I9UDg584cYIOHTrQsWNHysvLqXRRraRr164ABuf0dej/7dy5MzqdjnPnzhkEUa8v0Li1tTVFRUXk5+cbRPYyZZLQ2mivspBKVyJpRqoHBq/OkCFD8Pf3N8gEPHDgQAICArC2tkaj0ZCVlYW/vz/29vYAvPbaa3z//fcG9dQXaHz27NksXboUgO3bt7Nz505Gjx5t0iShtdFuZdHSSdrkIY+2eGCiBJn6pJF6Ll26JFxcXMSPP/5okufVBmaSNLU1ykIIIRfSJBJTIN2kDMpLWVRBLqRJJK0UrVbLrVu3HrierVu3Mm/ePBYuXGiEVrUMxpJFcnIyDg4OZGVlGaFVtSNtuhJJCxATE0NGRgZDhgzB3d2dDRs2kJ+fz/Lly9m3bx9lZWVcuXKFUaNGkZaWxrp164iLi0On03Hz5k1WrVql1hUVFUV2djY6nY5Nmzbh7u5Onz598PT0ZPDgwQ22xc/PD4DFixdTVlbW7Nl5zUkW48aN49q1a6Z8Xal0JZKWoKCggJdffhknJycsLCzQ6XTY2NiQmJgIgJubG2fOnEGj0bBixQpSU1OBylxiAPHx8WpdcXFxjBw5kry8PAoLCxFCMHPmTAMlEx8fT3p6uvrZx8eHQYMGqZ/z8/OxtLRskXTo5iYLUyOVrkTSAixfvpyTJ0/i7e3NpEmTGDNmDH369GH//v0A9OzZE41GQ/fu3dFoNJSXlwOV7k/V7aN2dnZotVr1c2hoKKGhoVy/fh1HR0cA7t69a+A2pdP9Es2xqKgIrVbLBx98YKrXrRdzkkVzIJWuRNICREZGcv78eXr27Mmzzz7L+vXrsbOzo2PH+odkREQE+fn5bNy4kY8++giA4cOH4+vrS2lpKcHBwQQHB1NcXMzEiRPV+1xcXHBxcam1Tr1Zwd/fn/feew9ra2ujvWdjMCdZZGZmEhMTQ7du3QgJCaFfv37Ge9H/Ir0XJBITYIoVe61Wi4+PD927dzdqvU3FHLwXWqssQCpdicQkSDcpg/JSFlWQLmMSiUTSjEilK5G0INXDF94vWq2WLVu2ABAQEMC8efM4duxYjXLJycksWLCAKVOmUFhYWON69ZCGSUlJuLq6GqWNDdHcsrh9+zazZ89m1qxZ5Ofn17he3WfXWLKQSlciMSG+vr7odDqSk5M5cuQIUVFR+Pn5ERoaqpaJjo4mKyuLvLw8Nm/ezMWLF/H29mbu3LmcO3eu0c9yc3MjNzeXxx57jMjISA4ePFijzLhx4/joo49wdXWtdQNAbGwskZGRDB8+nKysLCZOnGi0hTVzk8XBgwfx9fUlODiYuLi4GtfHjRvH3Llz1c/GkoVUuhKJCXn11Vc5ceIEKSkpjBkzBiEEXbt2JTk5uc57qoc81FNfiEI9BQUF9O7dGwALi9qHd0VFBUePHmXUqFE1rglhupCG5iYL/fXmCmWpR7qMSSQmxNHRkXXr1lFeXo6lpSWnTp1i165dvP3222oZjUZDRUUFRUVFQM2Qh3rqC1Gox9bWllOnTgG1+58KIVi2bBnLli2rdSOEKUMampssbG1tKSgo4M6dOwZhHU2NVLoSiQnp2rUrOTk5TJ48GahUahs3blSVCsCIESMIDg6mV69e2NjY1Ah56ODgANQfolDPwIEDuXr1Kh4eHjg5OQEQFhaGp6cnUDlzzMjIoKSkhLlz52JlZUVxcTEjRowAMGlIQ3OTxdSpU1m6dCk6nY5169Zx9uxZA1mYzGe3qWHJ5CEPeTR8YKJwhnWxevVqcfPmzRrnL126JBISEuq8Lzo6WhQUFNRbt7e3t8FnzCS0Y12YsyyEENKmK5G0Bfr378/evXtrnB80aBDTpk2r8z53d/d6f1onJSXRo0cPo7SxuTB3WcjNERKJCZAbAgzKS1lUQdp0JRITYGVldV1RlF4t3Q5TYGVldb2p5aUsfkHOdCUSiaQZkTZdiUQiaUak0pVIJJJmRCpdiUQiaUak0pVIJJJmRCpdiUQiaUak0pVIJJJmRCpdiUQiaUak0pVIJJJmRCpdiUQiaUak0pVIJJJm5P8DG0k2hdZmbR4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "# Split dataset into training and test part\n", "X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3)\n", "\n", "# Learn a Decision Tree\n", "from sklearn import tree\n", "clf = tree.DecisionTreeClassifier(criterion='entropy', splitter='best', max_depth=5, \n", " min_samples_split=2, min_samples_leaf=1, \n", " min_weight_fraction_leaf=0.0, max_features=None, \n", " random_state=None, max_leaf_nodes=None, \n", " min_impurity_split=1e-07, class_weight=None, presort=False)\n", "clf = clf.fit(X_train, y_train)\n", "\n", "# Graphical view of learnt Decision Tree\n", "tree.plot_tree(clf) \n", "\n", "# Evaluate acuracy on test data\n", "print(clf)\n", "score = clf.score(X_test, y_test)\n", "print(\"Acuracy (on test set) = \", score)\n", "from sklearn.metrics import classification_report\n", "from sklearn.metrics import confusion_matrix\n", "y_true, y_pred = y_test, clf.predict(X_test)\n", "print( classification_report(y_true, y_pred) )\n", "print(\"\\n CONFUSION MATRIX\")\n", "print( confusion_matrix(y_true, y_pred) )\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Decision Trees on a MORE REALISTIC DATASET: HANDWRITTEN DIGITS\n", "\n", "**Please FIRST READ the [*Digits DATASET DESCRIPTION*](http://scikit-learn.org/stable/auto_examples/datasets/plot_digits_last_image.html#sphx-glr-auto-examples-datasets-plot-digits-last-image-py).**\n", "\n", "In this classification problem, there are 10 classes, with a total of 1797 examples (each one being a 64D vector corresponding to an 8x8 pixmap). Please **now execute code cell below to load the dataset, visualize a typical example, and train a Desicion Tree on it**. \n", "The original code uses a **voluntarily SUBOPTIMAL set of learning hyperparameters values, which reaches ~66% test acuracy. Try to play with them in order to improve acuracy.**\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number_of-examples = 1797\n", "\n", " Plot of first example\n" ] }, { "data": { "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAECCAYAAADXWsr9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAL40lEQVR4nO3dW4hd9RXH8d+vY7xGSaxWJBHtSAmIUHNBKgFpNYpWsS81RFCotCQPrRha0NiX4ptPYh+KELxU8IajBoq01gQVEVrtTIz1MrFoiJhEHSWRGAsR4+rD2SkxnTp7xv3/z5mzvh845MzMmb3WzOR39t7n7L2XI0IABtu3ZrsBAOURdCABgg4kQNCBBAg6kABBBxLoi6DbvsL2W7bftr2hcK37bE/Yfr1knSPqnWX7Odvjtt+wfXPhesfbftn2q02920vWa2oO2X7F9lOlazX1dtp+zfY226OFay2w/bjt7c3f8KKCtZY0P9Ph237b6ztZeETM6k3SkKR3JA1LOlbSq5LOK1jvYknLJL1e6ec7U9Ky5v7Jkv5V+OezpPnN/XmSXpL0g8I/468lPSzpqUq/052STqtU6wFJv2juHytpQaW6Q5I+kHR2F8vrhzX6hZLejogdEfG5pEcl/aRUsYh4QdLeUsufpN77EbG1uf+ppHFJiwrWi4g40Hw4r7kVOyrK9mJJV0m6p1SN2WL7FPVWDPdKUkR8HhGfVCp/qaR3IuLdLhbWD0FfJOm9Iz7epYJBmE22z5G0VL21bMk6Q7a3SZqQtDkiSta7S9Itkr4sWONoIekZ22O21xasMyzpI0n3N7sm99g+qWC9I62R9EhXC+uHoHuSzw3ccbm250t6QtL6iNhfslZEHIqICyQtlnSh7fNL1LF9taSJiBgrsfyvsTIilkm6UtIvbV9cqM4x6u3m3R0RSyV9Jqnoa0iSZPtYSddIGulqmf0Q9F2Szjri48WS9sxSL0XYnqdeyB+KiCdr1W02M5+XdEWhEislXWN7p3q7XJfYfrBQrf+KiD3NvxOSNqm3+1fCLkm7jtgiely94Jd2paStEfFhVwvsh6D/Q9L3bH+3eSZbI+lPs9xTZ2xbvX288Yi4s0K9020vaO6fIGmVpO0lakXEbRGxOCLOUe/v9mxEXF+i1mG2T7J98uH7ki6XVOQdlIj4QNJ7tpc0n7pU0pslah3lOnW42S71Nk1mVUR8YftXkv6q3iuN90XEG6Xq2X5E0g8lnWZ7l6TfRcS9peqpt9a7QdJrzX6zJP02Iv5cqN6Zkh6wPaTeE/ljEVHlba9KzpC0qff8qWMkPRwRTxesd5Okh5qV0A5JNxasJdsnSrpM0rpOl9u8lA9ggPXDpjuAwgg6kABBBxIg6EACBB1IoK+CXvhwxlmrRT3qzXa9vgq6pJq/zKp/OOpRbzbr9VvQARRQ5IAZ2wN9FM7ChQun/T0HDx7UcccdN6N6ixZN/2S+vXv36tRTT51Rvf37p3/OzYEDBzR//vwZ1du9e/e0vyci1BwdN22HDh2a0ffNFRHxP7+YWT8Edi5atWpV1Xp33HFH1XpbtmypWm/DhuInhH3Fvn37qtbrB2y6AwkQdCABgg4kQNCBBAg6kABBBxIg6EACBB1IoFXQa45MAtC9KYPeXGTwD+pdgvY8SdfZPq90YwC602aNXnVkEoDutQl6mpFJwKBqc1JLq5FJzYnytc/ZBdBCm6C3GpkUERslbZQG/zRVYK5ps+k+0COTgAymXKPXHpkEoHutLjzRzAkrNSsMQGEcGQckQNCBBAg6kABBBxIg6EACBB1IgKADCRB0IAEmtcxA7ckpw8PDVevNZOTUN7F3796q9VavXl213sjISNV6k2GNDiRA0IEECDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJEDQgQTajGS6z/aE7ddrNASge23W6H+UdEXhPgAUNGXQI+IFSXXPOgDQKfbRgQQ6O02V2WtA/+os6MxeA/oXm+5AAm3eXntE0t8kLbG9y/bPy7cFoEtthixeV6MRAOWw6Q4kQNCBBAg6kABBBxIg6EACBB1IgKADCRB0IIGBmL22fPnyqvVqz0I799xzq9bbsWNH1XqbN2+uWq/2/xdmrwGogqADCRB0IAGCDiRA0IEECDqQAEEHEiDoQAIEHUiAoAMJtLk45Fm2n7M9bvsN2zfXaAxAd9oc6/6FpN9ExFbbJ0sas705It4s3BuAjrSZvfZ+RGxt7n8qaVzSotKNAejOtPbRbZ8jaamkl4p0A6CI1qep2p4v6QlJ6yNi/yRfZ/Ya0KdaBd32PPVC/lBEPDnZY5i9BvSvNq+6W9K9ksYj4s7yLQHoWpt99JWSbpB0ie1tze3HhfsC0KE2s9delOQKvQAohCPjgAQIOpAAQQcSIOhAAgQdSICgAwkQdCABgg4kMBCz1xYuXFi13tjYWNV6tWeh1Vb795kRa3QgAYIOJEDQgQQIOpAAQQcSIOhAAgQdSICgAwkQdCABgg4k0OYqsMfbftn2q83stdtrNAagO22OdT8o6ZKIONBc3/1F23+JiL8X7g1AR9pcBTYkHWg+nNfcGNAAzCGt9tFtD9neJmlC0uaIYPYaMIe0CnpEHIqICyQtlnSh7fOPfozttbZHbY923COAb2har7pHxCeSnpd0xSRf2xgRKyJiRTetAehKm1fdT7e9oLl/gqRVkrYX7gtAh9q86n6mpAdsD6n3xPBYRDxVti0AXWrzqvs/JS2t0AuAQjgyDkiAoAMJEHQgAYIOJEDQgQQIOpAAQQcSIOhAAsxem4EtW7ZUrTfoav/99u3bV7VeP2CNDiRA0IEECDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJEDQgQRaB70Z4vCKbS4MCcwx01mj3yxpvFQjAMppO5JpsaSrJN1Tth0AJbRdo98l6RZJX5ZrBUApbSa1XC1pIiLGpngcs9eAPtVmjb5S0jW2d0p6VNIlth88+kHMXgP615RBj4jbImJxRJwjaY2kZyPi+uKdAegM76MDCUzrUlIR8bx6Y5MBzCGs0YEECDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJDAQs9dqz9Javnx51Xq11Z6FVvv3OTIyUrVeP2CNDiRA0IEECDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJEDQgQRaHQLbXOr5U0mHJH3BJZ2BuWU6x7r/KCI+LtYJgGLYdAcSaBv0kPSM7THba0s2BKB7bTfdV0bEHtvfkbTZ9vaIeOHIBzRPADwJAH2o1Ro9IvY0/05I2iTpwkkew+w1oE+1maZ6ku2TD9+XdLmk10s3BqA7bTbdz5C0yfbhxz8cEU8X7QpAp6YMekTskPT9Cr0AKIS314AECDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJOCI6H6hdvcL/RrDw8M1y2l0dLRqvXXr1lWtd+2111atV/vvt2LFYJ+OERE++nOs0YEECDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJEDQgQQIOpBAq6DbXmD7cdvbbY/bvqh0YwC603aAw+8lPR0RP7V9rKQTC/YEoGNTBt32KZIulvQzSYqIzyV9XrYtAF1qs+k+LOkjSffbfsX2Pc0gh6+wvdb2qO26p3YBmFKboB8jaZmkuyNiqaTPJG04+kGMZAL6V5ug75K0KyJeaj5+XL3gA5gjpgx6RHwg6T3bS5pPXSrpzaJdAehU21fdb5L0UPOK+w5JN5ZrCUDXWgU9IrZJYt8bmKM4Mg5IgKADCRB0IAGCDiRA0IEECDqQAEEHEiDoQAIDMXuttrVr11atd+utt1atNzY2VrXe6tWrq9YbdMxeA5Ii6EACBB1IgKADCRB0IAGCDiRA0IEECDqQAEEHEpgy6LaX2N52xG2/7fUVegPQkSmvGRcRb0m6QJJsD0naLWlT2bYAdGm6m+6XSnonIt4t0QyAMqYb9DWSHinRCIByWge9uab7NZJG/s/Xmb0G9Km2Axwk6UpJWyPiw8m+GBEbJW2UBv80VWCumc6m+3Visx2Yk1oF3faJki6T9GTZdgCU0HYk078lfbtwLwAK4cg4IAGCDiRA0IEECDqQAEEHEiDoQAIEHUiAoAMJEHQggVKz1z6SNJNz1k+T9HHH7fRDLepRr1a9syPi9KM/WSToM2V7NCJWDFot6lFvtuux6Q4kQNCBBPot6BsHtBb1qDer9fpqHx1AGf22RgdQAEEHEiDoQAIEHUiAoAMJ/AchD47vy2xCkAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "DecisionTreeClassifier(max_depth=5, min_impurity_split=1e-07,\n", " min_samples_split=4, presort=False)\n", "Acuracy (on test set) = 0.728587319243604\n", " precision recall f1-score support\n", "\n", " 0 0.99 0.90 0.94 91\n", " 1 0.65 0.57 0.61 89\n", " 2 0.72 0.80 0.76 74\n", " 3 0.77 0.70 0.74 91\n", " 4 0.75 0.77 0.76 96\n", " 5 0.94 0.80 0.86 90\n", " 6 0.97 0.79 0.88 97\n", " 7 0.97 0.77 0.86 92\n", " 8 0.37 0.28 0.32 93\n", " 9 0.45 0.92 0.61 86\n", "\n", " accuracy 0.73 899\n", " macro avg 0.76 0.73 0.73 899\n", "weighted avg 0.76 0.73 0.73 899\n", "\n", "\n", " CONFUSION MATRIX\n", "[[82 0 0 0 4 3 0 0 0 2]\n", " [ 0 51 4 6 6 0 0 0 18 4]\n", " [ 1 3 59 1 2 0 0 0 7 1]\n", " [ 0 2 3 64 0 1 0 2 6 13]\n", " [ 0 6 1 0 74 0 0 0 2 13]\n", " [ 0 3 1 0 5 72 0 0 3 6]\n", " [ 0 1 5 0 7 0 77 0 7 0]\n", " [ 0 4 3 0 1 0 0 71 2 11]\n", " [ 0 6 5 9 0 0 2 0 26 45]\n", " [ 0 2 1 3 0 1 0 0 0 79]]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:327: FutureWarning: The parameter 'presort' is deprecated and has no effect. It will be removed in v0.24. You can suppress this warning by not passing any value to the 'presort' parameter.\n", " FutureWarning)\n" ] } ], "source": [ "from sklearn.datasets import load_digits\n", "digits = load_digits()\n", "n_samples = len(digits.images)\n", "print(\"Number_of-examples = \", n_samples)\n", "\n", "import matplotlib.pyplot as plt\n", "print(\"\\n Plot of first example\")\n", "plt.gray() \n", "plt.matshow(digits.images[0]) \n", "plt.show() \n", "\n", "# Flatten the images, to turn data in a (samples, feature) matrix:\n", "data = digits.images.reshape((n_samples, -1))\n", "\n", "# Split dataset into training and test part\n", "X = data\n", "y = digits.target\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)\n", "\n", "# Create and train a Decision Tree Classifier\n", "clf = tree.DecisionTreeClassifier(criterion='gini', splitter='best', max_depth=5, \n", " min_samples_split=4, min_samples_leaf=1, \n", " min_weight_fraction_leaf=0.0, max_features=None, \n", " random_state=None, max_leaf_nodes=None, \n", " min_impurity_split=1e-07, class_weight=None, presort=False)\n", "clf = clf.fit(X_train, y_train)\n", "\n", "\n", "# Evaluate acuracy on test data\n", "print(clf)\n", "score = clf.score(X_test, y_test)\n", "print(\"Acuracy (on test set) = \", score)\n", "from sklearn.metrics import classification_report\n", "from sklearn.metrics import confusion_matrix\n", "y_true, y_pred = y_test, clf.predict(X_test)\n", "print( classification_report(y_true, y_pred) )\n", "print(\"\\n CONFUSION MATRIX\")\n", "print( confusion_matrix(y_true, y_pred) )\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Question: According to the confusion matrices, what digits are the most confused with each other?__\n", "\n", "__Answer:__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Finally, find somewhat optimized values for the set of 3 main hyper-parameters for DecisionTree learning, by using GRID-SEARCH WITH CROSS-VALIDATION** (see cross-validation example from the Multi-Layer Perceptron notebook used in earlier practical session). __Put the code in the cell below:__" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Question: What best value have you managed to reach for TEST accuracy of your DecisionTree after you properly gridSearched its hyper-parameters using CrossValidation?__\n", "\n", "__Answer:__\n", "\n", "\n", "In order to improve result, the most natural step is to combine SEVERAL decision trees, using the Ensemble model called Random Forest: see below" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Building, training and evaluating a Random Forest classifier\n", "\n", "The SciKit-learn class for Random Forest classifiers is sklearn.ensemble.RandomForestClassifier.\n", "\n", "**Please FIRST READ (and understand!) the [*RandomForestClassifier DOCUMENTATION*](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) to understand all parameters of the contructor.**\n", "\n", "**Then you can begin by running the code block below, in which default set of parameter values has been used.** As you will see, a RandomForest (even rather small) can easily outperform single Decision Tree. \n", "\n", "**Then, check the influence of MAIN parameters for Random Forest classifier, i.e.:**\n", " - **n_estimators (number of trees in forest)**\n", " - **max_depth**\n", " - **max_features (max number of features used in each tree)**\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n", "C:\\Users\\fabien\\anaconda3_2020-07\\envs\\envML2020\\lib\\site-packages\\sklearn\\tree\\_classes.py:310: FutureWarning: The min_impurity_split parameter is deprecated. Its default value has changed from 1e-7 to 0 in version 0.23, and it will be removed in 0.25. Use the min_impurity_decrease parameter instead.\n", " FutureWarning)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "n_estimators= 10 max_depth= None max_features= auto\n", "RandomForestClassifier(min_impurity_split=1e-07, n_estimators=10, n_jobs=1)\n", "Acuracy (on test set) = 0.9399332591768632\n", " precision recall f1-score support\n", "\n", " 0 1.00 1.00 1.00 91\n", " 1 0.89 0.96 0.92 89\n", " 2 0.90 0.99 0.94 74\n", " 3 0.94 0.89 0.92 91\n", " 4 0.97 0.97 0.97 96\n", " 5 0.95 0.98 0.96 90\n", " 6 0.99 0.98 0.98 97\n", " 7 0.96 0.93 0.95 92\n", " 8 0.86 0.88 0.87 93\n", " 9 0.95 0.83 0.88 86\n", "\n", " accuracy 0.94 899\n", " macro avg 0.94 0.94 0.94 899\n", "weighted avg 0.94 0.94 0.94 899\n", "\n", "\n", " CONFUSION MATRIX\n", "[[91 0 0 0 0 0 0 0 0 0]\n", " [ 0 85 2 0 0 0 0 0 2 0]\n", " [ 0 1 73 0 0 0 0 0 0 0]\n", " [ 0 2 3 81 0 0 0 0 3 2]\n", " [ 0 1 0 0 93 0 1 1 0 0]\n", " [ 0 1 0 0 0 88 0 0 1 0]\n", " [ 0 1 0 0 0 0 95 0 1 0]\n", " [ 0 1 1 0 3 1 0 86 0 0]\n", " [ 0 2 2 2 0 1 0 2 82 2]\n", " [ 0 2 0 3 0 3 0 1 6 71]]\n" ] } ], "source": [ "from sklearn.ensemble import RandomForestClassifier\n", "\n", "# Create and train a Random Forest classifier\n", "clf = RandomForestClassifier(n_estimators=10, criterion='gini', max_depth=None,\n", " min_samples_split=2, min_samples_leaf=1, \n", " min_weight_fraction_leaf=0.0, max_features='auto', \n", " max_leaf_nodes=None, min_impurity_split=1e-07, bootstrap=True, \n", " oob_score=False, n_jobs=1, random_state=None, \n", " verbose=0, warm_start=False, class_weight=None)\n", "clf = clf.fit(X_train, y_train)\n", "print(\"n_estimators=\", clf.n_estimators, \" max_depth=\",clf.max_depth,\n", " \"max_features=\", clf.max_features)\n", "\n", "# Evaluate acuracy on test data\n", "print(clf)\n", "score = clf.score(X_test, y_test)\n", "print(\"Acuracy (on test set) = \", score)\n", "from sklearn.metrics import classification_report\n", "from sklearn.metrics import confusion_matrix\n", "y_true, y_pred = y_test, clf.predict(X_test)\n", "print( classification_report(y_true, y_pred) )\n", "print(\"\\n CONFUSION MATRIX\")\n", "print( confusion_matrix(y_true, y_pred) )\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Finally, find somewhat optimized values the set of 3 main hyper-parameters for RandomForest, by using CROSS-VALIDATION** (see cross-validation example from the Multi-Layer Perceptron notebook used in earlier practical session). __Put the code in the cell below:__" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Question: What best value have you managed to reach for TEST accuracy of your RandomForest after you properly gridSearched its hyper-parameters using CrossValidation?__\n", "\n", "__Answer:__" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## 3. Building, training and evaluating an AdaBoost classifier\n", "\n", "The SciKit-learn class for adaBoost is sklearn.ensemble.AdaBoostClassifier.\n", "\n", "**Please FIRST READ (and understand!) the [*AdaBoostClassifier DOCUMENTATION*](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier) to understand all parameters of the contructor.**\n", "\n", "**Then begin by running the code block below, in which a default set of parameter values has been used.** \n", "\n", "**Then, check the influence of MAIN parameters for adaBoost classifier, i.e.:**\n", " - **base_estimator (ie type of Weak Classifier/Learner)** \n", " - **n_estimators (number of boosting iterations, and therefore also number of weak classifiers)**\n", " - algorithm\n", " \n", "**Finally, check which other types of classifiers can be used as Weak Classifier with the adaBoost implementation of SciKit-Learn.**\n", "NB: in principle it is possible to use MLP classifiers as weak classifiers, but not with SciKit-learn implementation of MLPClassifier (because weighting of examples is not handled by its implementation)." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Weak_learner: DecisionTreeClassifier(max_depth=6)\n", "Weights of weak classifiers: [4.1662423 5.41474306 4.7238877 4.45785944 4.16955192 4.55695778\n", " 5.33768772 5.42409322 5.7337812 4.21523304 4.44342062 4.90185404\n", " 4.98322751 5.54716878 4.83245137]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxLklEQVR4nO3de3zU5Z3o8c83d0iACAkJSbiEiyDXCBm8X4DiXUGrBWr3VLs9ai+6tdvWdrt2u+05e1y7PXX7al1LXUt7FglVCriIpWqxgGhNkGuQmxAghEsIl0Ag5PY9f/x+CUOYJDPJTGYm8337+r0y87s8v28Q5jvP8/ye5xFVxRhjTOyKC3cAxhhjwssSgTHGxDhLBMYYE+MsERhjTIyzRGCMMTHOEoExxsS4kCYCEblDRHaKyB4R+a6P47eKyGkR2eRuPwhlPMYYYy6XEKqCRSQe+CUwEygHikXkDVXd3urUtap6T6jiMMYY075Q1gimAntUda+q1gFFwKwQ3s8YY0wnhKxGAOQCB73elwPX+DjvOhHZDFQA31LV0tYniMhjwGMAqampU8aMGROCcI0xpufasGHDcVXN9HUslIlAfOxrPZ/Fx8BQVT0rIncBy4BRl12kOh+YD1BYWKglJSVBDtUYY3o2Ednf1rFQNg2VA4O93ufhfOtvoarVqnrWfb0SSBSRjBDGZIwxppVQJoJiYJSI5ItIEjAXeMP7BBHJFhFxX09146kKYUzGGGNaCVnTkKo2iMjXgVVAPPCKqpaKyBPu8ZeAB4GviEgDcB6YqzYdqjHGdCuJts9d6yMwJjLU19dTXl5ObW1tuEMxXlJSUsjLyyMxMfGS/SKyQVULfV0Tys5iY0wPVl5eTp8+fRg2bBhuC68JM1WlqqqK8vJy8vPz/b7OppgwxnRKbW0tAwYMsCQQQUSEAQMGBFxLs0RgjOk0SwKRpzP/TywRGGNMjLNEYIyJSlVVVRQUFFBQUEB2dja5ubkt7+vq6tq9tqSkhKeeeqrDe1x//fVBifW9996jX79+LfEVFBTwzjvvBKXsYLDOYmNMVBowYACbNm0C4Ic//CFpaWl861vfajne0NBAQoLvj7jCwkIKC30+QHOJ9evXByVWgJtuuokVK1a0eVxVUVXi4uJ8vm9LY2Mj8fHxXYrNagTGmB7jkUce4Zvf/CbTpk3jmWee4aOPPuL666/n6quv5vrrr2fnzp2A8w39nnucSY9/+MMf8qUvfYlbb72V4cOH8/Of/7ylvLS0tJbzb731Vh588EHGjBnDww8/TPOj9ytXrmTMmDHceOONPPXUUy3l+qOsrIyrrrqKr371q0yePJm1a9de8v7gwYN8+9vfZvz48UyYMIHFixe3xDNt2jQ+//nPM2HChC7/uVmNwBjTZd/44zfYdGRTUMssyC7ghTteCPi6Xbt28c477xAfH091dTVr1qwhISGBd955h3/4h39gyZIll12zY8cOVq9ezZkzZxg9ejRf+cpXLnsOf+PGjZSWlpKTk8MNN9zA+++/T2FhIY8//jhr1qwhPz+fefPmtRnX2rVrKSgoaHm/ZMkS4uPj2blzJ7/5zW948cUXKSsru+T9kiVL2LRpE5s3b+b48eN4PB5uvvlmAD766CO2bdsW0GOibbFEYIzpUR566KGWppLTp0/zxS9+kd27dyMi1NfX+7zm7rvvJjk5meTkZAYOHMjRo0fJy8u75JypU6e27CsoKKCsrIy0tDSGDx/e8mE8b9485s+f7/MevpqGysrKGDp0KNdee23LPu/369atY968ecTHx5OVlcUtt9xCcXExffv2ZerUqUFJAmCJwBgTBJ355h4qqampLa+fffZZpk2bxtKlSykrK+PWW2/1eU1ycnLL6/j4eBoaGvw6JxgzM3jH2/p9e+W3vq4rrI/AGNNjnT59mtzcXAAWLFgQ9PLHjBnD3r17KSsrA2hpww+Wm2++mcWLF9PY2EhlZSVr1qxh6tSpQb0HWCIwxvRg3/nOd/je977HDTfcQGNjY9DL79WrFy+++CJ33HEHN954I1lZWfTr18/nuc19BM3b66+/3mH5999/PxMnTmTSpElMnz6d559/nuzs7GD/GjbpnDGmcz755BOuuuqqcIcRdmfPniUtLQ1V5Wtf+xqjRo3i6aefDmtMvv7ftDfpnNUIjDGmC379619TUFDAuHHjOH36NI8//ni4QwqYdRYbY0wXPP3002GvAXSV1QiMMSbGWSIwxpgYZ4nAGGNinCUCY4yJcZYIjDFRqSvTUIMzcVtbs4suWLCAzMzMS5773759e7B/hYhhTw0ZY6JSR9NQd+S9994jLS2tzTUH5syZwy9+8Ys2r289/bO/00G3Nz12uFiNwBjTY2zYsIFbbrmFKVOmcPvtt3P48GEAfv7znzN27FgmTpzI3LlzKSsr46WXXuJnP/sZBQUFrF271q/yW0//3Pp9bW0tjz76KBMmTODqq69m9erVgFPDeOihh7j33nu57bbbQvb7d1ZkpSVjTHTa8A04uSm4ZV5RAFNe8Pt0VeXJJ59k+fLlZGZmsnjxYr7//e/zyiuv8Nxzz7Fv3z6Sk5M5deoU6enpPPHEE+3WIhYvXsy6deta3n/wwQfApdM/v/fee5e8/+lPfwrA1q1b2bFjB7fddhu7du1quX7Lli3079+/c38eIWSJwBjTI1y4cIFt27Yxc+ZMwGmqGTRoEAATJ07k4YcfZvbs2cyePduv8tpqGmo9/bP3+3Xr1vHkk08CzoR0Q4cObUkEM2fOjMgkAJYIjDHBEMA391BRVcaNG9fyzd3bm2++yZo1a3jjjTf48Y9/TGlpaafvEwnTRgeb9REYY3qE5ORkKisrWxJBfX09paWlNDU1cfDgQaZNm8bzzz/PqVOnOHv2LH369OHMmTNBjeHmm29m4cKFgLNS2oEDBxg9enRQ7xEKlgiMMT1CXFwcr7/+Os888wyTJk2ioKCA9evX09jYyBe+8IWWDtynn36a9PR07r33XpYuXdpmZ/HixYsveXzUn4Xsv/rVr9LY2MiECROYM2cOCxYsuGRBm0hl01AbYzrFpqGOXDYNtTHGmIBYIjDGmBhnicAY02nR1rQcCzrz/8QSgTGmU1JSUqiqqrJkEEFUlaqqKlJSUgK6zsYRGGM6JS8vj/LyciorK8MdivGSkpJCXl5eQNdYIjDGdEpiYuIlI2xN9LKmIWOMiXEhTQQicoeI7BSRPSLy3XbO84hIo4g8GMp4jDHGXC5kiUBE4oFfAncCY4F5IjK2jfP+FVgVqliMMca0LZQ1gqnAHlXdq6p1QBEwy8d5TwJLgGMhjMUYY0wbQpkIcoGDXu/L3X0tRCQXuB94qb2CROQxESkRkRJ7QsEYY4IrlIlAfOxr/cDxC8AzqtrYXkGqOl9VC1W1MDMzM1jxGWOMIbSPj5YDg73e5wEVrc4pBIpEBCADuEtEGlR1WQjjMsYY4yWUiaAYGCUi+cAhYC7wee8TVLXlIWQRWQCssCRgjDHdK2SJQFUbROTrOE8DxQOvqGqpiDzhHm+3X8AYY0z3COnIYlVdCaxstc9nAlDVR0IZizHGGN9sZLExxsQ4SwTGGBPjLBEYY0yMs0RgjDExzhKBMcbEOEsExhgT4ywRdJWqsxljTJSyRNBVW56FtwosGRhjopYlgq5oaoA98+HUFjizJ9zRGGNMp1gi6Ioj78AFd1rso38ObyzGGNNJlgi6omwhJKZDrxw4+m64ozHGmE4J6VxDPVrDOShfCkM/D00XoGIlaBOI5VZjTHSxT63OKn8DGmpg2MOQNR0uHIdTW8MdlTHGBMwSQWftfxV658HAm5xEANZPYIyJSpYIOuNCFVS8BUPnOU1BqYOhzyg4Yv0ExpjoY4mgMw68BtoAw7wWXMuaAcf+Ak314YvLGGM6wRJBZ5S9Cv3GQvqki/uyp0PDWagqCV9cxhjTCZYIAlWzHyrXOp3EIhf3D5zm/LR+AmNMlLFEEKj9Rc7PofMu3Z+S4dQQbDyBMSbKWCIIVNlCyLge0vIvP5Y9AyrXQ8P57o/LGGM6yRJBIE5tdTbvTmJvWdOdwWXH13dvXMYY0wWWCAJR9ipIPAz5nO/jA292jls/gTEmilgi8Jc2OYlg0O2Qkun7nMQ+MGCqjScwxkQVSwT+qlwP5w44cwu1J2sGnCiGutPdE5cxxnSRJQJ/lS2E+N6QN6v987KnO7WHY2u6Jy5jjOkiSwT+aKyDA793kkBiWvvnZlwH8SnWT2CMiRqWCPxx5E9Qd8IZRNaR+BTIuMHGExhjooYlAn+ULYTkATDoNv/Oz57hPGZaeyy0cRljTBBYIuhI/VkoX+48MhqX6N81LdNSrw5dXMYYEySWCDpSvgwaz/vXLNSs/xRI7Gv9BMaYqGCJoCNlCyF1qNMJ7K+4BBh4q40nMMZEBUsE7ak9BkfedsYOBLoWcdZ0OPupM1upMcZEMEsE7dn/e9DGtucWak/2DOfnEWseMsZENksE7dn/KqRPhPTxgV/bbxykDLR+AmNMxAtpIhCRO0Rkp4jsEZHv+jg+S0S2iMgmESkRkRtDGU9Azu6F4x8E1knsTcRpHjr6LqgGNzZjjAmikCUCEYkHfgncCYwF5onI2FanvQtMUtUC4EvAy6GKJ2Blrzo/h87tfBlZ0+H8YajeGZyYjDEmBEJZI5gK7FHVvapaBxQBl0zUo6pnVVu+LqcCkfHVWdV5WmjgzZA6pPPlNPcT2ChjY0wEC2UiyAUOer0vd/ddQkTuF5EdwJs4tYLLiMhjbtNRSWVlZUiCvcTJTVC9o+OZRjuSmu88emr9BMaYCNZhIhCRK0XkXRHZ5r6fKCL/6EfZ4mPfZd/4VXWpqo4BZgM/9lWQqs5X1UJVLczMbGMtgGDa/6ozinjIQ10rR8SZlvroamhqDE5sxhgTZP7UCH4NfA+oB1DVLYA/DeflwGCv93lARVsnq+oaYISIZPhRdug0NULZIhh0JyT373p5WdOh7iSc2tz1sowxJgT8SQS9VfWjVvsa/LiuGBglIvkikoSTPN7wPkFERoqIuK8nA0lAlR9lh07lGjh/qHNjB3zJducdslHGxpgI5U8iOC4iI3CbdUTkQeBwRxepagPwdWAV8Anwe1UtFZEnROQJ97TPAttEZBPOE0ZzvDqPw6PsVUhIg9x7g1Ner0HQ9yrrJzDGRKwEP875GjAfGCMih4B9gF8P16vqSmBlq30veb3+V+Bf/Y421BovwIHXIe9+SOgdvHKzZ8CnrzgL3MQnBa9cY4wJAn9qBKqqnwEygTGqeqOf10Wfireg/lTnB5G1JWs6NJ6DqtYtbMYYE37+fKAvAVDVGlU94+57PXQhhVHZQmdaiObn/4Ml61ZAbDyBMSYitdk0JCJjgHFAPxF5wOtQXyAl1IF1u7rTcOi/YeRjzjTSwZR0BfSf7HQYT/in4JZtjDFd1N4n3mjgHiAd8O45PQP8zxDGFB7lS6HpQvCbhZplzYCdP4OGGkhIDc09jDGmE9pMBKq6HFguItep6gfdGFN4lC2EtBEwYGpoys+aDp88D8fWQc7tobmHMcZ0gj9tIBtF5Gs4zUQtTUKq6nM6iKh0/rDzeOe47zujgUNh4I3OaOWjf7ZEYIyJKP50Fv8/IBu4HfgLzgjhM+1eEW32LwZt6vrcQu1JSIUB11qHsTEm4viTCEaq6rNAjar+FrgbmBDasLpZ2UK4YjL0GxPwpXWNdZyuPe3fydkz4MTHzpQTxhgTIfxJBPXuz1MiMh7oBwwLWUTdrXoXnCjpdCfxN1d9k8nzJ+PXgOis6YDC0fc6dS9jjAkFfxLBfBG5AvhHnLmCthNJo4G7quxVQGDonIAvvdBwgYVbF7L35F4OVh/s+IIB10B8b5tuwhgTUdrtLBaROKBaVU8Ca4Dh3RJVd1F1ppzOmga9L1sqoUOrPl3FqdpTABQfKmZIvw4WsYlPgoE32QR0xpiI0m6NQFWbcCaO65lOlMCZ3Z2eabRoWxH9e/UnMS6R4opi/y7KmgHVnzhPKhljTATwp2nobRH5logMFpH+zVvII+sOZQshLgkGfzbgS2vqali+czkPjX2ICVkT/E8ELdNSW/OQMSYy+DOOoHm8wNe89inR3kzU1Aj7iyD3HkhKD/jyFbtWcK7+HHPHO2v0LNq2iCZtIk46yK3pBc6UE0f/DPkhGsVsjDEB6LBGoKr5PrboTgLgfBDXHu302IGi0iIGpQ3ipiE34cnxUH2hmt1Vuzu+MC4eBt7qjCcI89ILxhgDPXU6aX/sfxUS+0Lu3QFfeqr2FCt3r2TOuDnEx8XjyfUABNA8NANq9kPNvoDvbYwxwRabiaDhPBxY4vQNxAc+keqyHcuoa6xraRYamzmWXgm9KD7kb4exLV9pjIkc7SYCcQxu75yoVLECGs50ehDZom2LyE/PZ2quM0FdQlwCkwdN9r9G0HeMs4SljScwxkSAjh4fVWBZ94TSjcpedT6IB94a8KXHao7x7t53mTt+LuI1QZ0nx8PGIxupb6xv52qXiFMrOPpn6ycwxoSdP01DH4qIJ+SRdJe6k1CxEobMdTpuA7Rk+xIatbGlWaiZJ9dDbUMtpZWl/hWUNQNqj8FpP883xpgQ8ScRTAM+EJFPRWSLiGwVkS2hDixkDiyBprpOP7q5aNsixmaOZcLAS+fd8+S4Hcb+9hNkWz+BMSYy+JMI7gRGANNxViq7h0tXLIsuZQuhz5XObKMBKq8uZ+2Btcwdd2mzEMDI/iNJT0n3v58gdaizEI71ExhjwsyfcQT7ubhc5b1Aursv+pwrh2N/cTqJO7EAzeJtiwEuaxYCEBEKcwopqSjxv8Cs6XDsPWhqCDgWY4wJlg4TgYj8HbAQGOhu/yUiT4Y6sJDYXwRo5+cWKi1iyqApjBowyudxT46Hrce2UttQ61+B2TOgvtpZo8AYY8LEn6ahvwWuUdUfqOoPgGuJ1sXryxY6axL3GRnwpXtO7KGkosRnbaCZJ8dDQ1MDm45s8q/QrGnOT1u1zBgTRv4kAgEavd43uvuiy+ntcHJTp8cOFG0rAmDOuLbXLWgZYexvh3HKQEifYP0Expiw8mfSuVeAv4rIUvf9bOA/QxZRqJzZAylZMORznbq8aFsRNw65kcH92h5fl9snl+y0bP87jMHpJ9jzK2is7dQoZ2OM6aqORhbHAX8FHgVOACeBR1X1hdCHFmR598HsQ9ArO+BLtx7dSmllKfPGz2v3PBHBk+MJMBHMcJLA8Q8DjssYY4Kh3RqBqjaJyE9V9Tog+ns0OzGADJzaQJzE8eDYBzs815PjYcWuFVRfqKZvct+OCx94M0icM54g69ZOxWeMMV3hTx/Bn0Tks9L6wfkYoaoUlRYxI38GA1MHdni+J9eDomyo2ODfDZL6QX+P9RMYY8LGn0TwTeA14IKIVIvIGRGpDnFcEaO4opi9J/d22CzUrDCnsOU6v2VNh6qPoP5MZ0I0xpgu8aeP4A5VjVPVJFXtq6p9VNWPNo+eoWhbEUnxSdx/1f1+nZ/RO4P89PzAEkH2DNAGOLa2k1EaY0zn+bN4/b91UywRp7GpkcWli7lz5J2kp6T7fZ0n1+P/I6QAGddDXLKNJzDGhIX1EbRj3YF1VJypaHcQmS+eHA/7T++nsqbSvwsSekHm9dZPYIwJi0D6COpirY+gaFsRvRN7c++Vgc2x1zITaaD9BCc3Qe3xgO5ljDFd5c+kc33cPoLEQPsIROQOEdkpIntE5Ls+jj/sTm29RUTWi8ikzvwSoVDfWM9r21/jvtH3kZqUGtC1kwdNRpDAmoeyZjg/j70X0L2MMaar/Jl0TkTkCyLyrPt+sIhM9eO6eOCXONNYjwXmicjYVqftA25R1YnAj4H5gf4CofLuvnepOl/F3HGBNQsB9Enuw5iMMYHVCAYUQkKarU9gjOl2/jQNvQhcBzRP2XkW5wO+I1OBPaq6V1XrgCJglvcJqrpeVU+6bz8E8vyKuhss2raIfsn9uGPkHZ263pPrjDBWf5eijEuEgbdYP4Exptv5kwiuUdWvAbUA7gd3kh/X5QIHvd6Xu/va8rfAW74OiMhjIlIiIiWVlX52wHZBbUMtSz9ZygNXPUByQnKnyvDkeDhWc4yD1Qc7PrlZ1nQ4s8tZN8EYY7qJP4mg3m3mUQARyQSa/LjO11NGPr8ei8g0nETwjK/jqjpfVQtVtTAzM9OPW3fNW7vf4kzdmYCfFvIW8NKV4IwnAGseMsZ0K38Swc+BpcBAEfnfwDrgX/y4rhzwnqozD6hofZKITAReBmapapUf5Ybcom2LyOydyfT86Z0uY1L2JBLiEgLrJ0ifAMkZ1jxkjOlWHU5DraoLRWQDMAPnW/5sVf3Ej7KLgVEikg8cAuZysZ8BABEZAvwB+BtV3RVo8KFw5sIZVuxawaMFj5IQ588s3b6lJKQwMWtiYIlA4pzFao68C6qdWk7TGGMC5dcnnaruAHYEUrCqNojI14FVQDzwiqqWisgT7vGXgB8AA4AX3fFqDapaGMh9gu2NnW9wvuE88yb4N7dQezw5HhZtW0STNhEn/lS+cPoJDrwGZ3ZD3yu7HIMxxnTEz0+nzlHVlap6paqOUNX/7e57yU0CqOqXVfUKVS1wt7AmAXDWJc7rm8f1g6/vclmeHA/VF6rZXbXb/4uaxxPYdBPGmG4S0kQQbU6cP8GqPauYM26O/9/g29G8dGVJRYn/F/UZCb3z4Ij1ExhjuoclAi9/+OQP1DfV+z3ldEfGZo6lV0KvAPsJxKkVHFsN6s/DWcYY0zWWCLwUbStiZP+RTB40OSjlJcQlMHnQ5MASATj9BBeq4NSWoMRhjDHtsUTgOnL2CKvLVjNv/DyCOdGqJ8fDxsMbaWhq8P+ibPex1f2LofFC0GIxxhhfLBG4Xit9jSZt6tIgMl88uR7ON5yn9Fip/xf1zoOM62D7c/B6f1h9J3zyUzi5xZqLjDFB1/kH5XuYotIiJgycwNjM1vPidY33lNSTsgOYXHXan5yBZUfegSNvw8ZvOftTBjp9CNkzIfszkDq4/XKMMaYDlgiA/af2s/7gev5luj8DpgMzsv9I0lPSKT5UzJcnf9n/CxPTIO8+ZwNn/qEj71zc9i9y9vcdDVmfcZJC1jRI6hf038EY07NZIgAWly4GYM74OUEvW0QozCkMvMO4td55MPwRZ1OF09suJoV9C2D3L52Ryf2nOklh0EwYcC3E+zM/oDEmllkiwJlb6Jrcaxh+xfCQlO/J8fCT9T+htqGWlISUrhco4sxLlD4BxjwNjXVQ9aGTFA6/Ddv/BUr/F8T3dqa2HuQ2I/Ubb9NWGGMuE/OJYMfxHWw6somf3f6zkN3Dk+OhoamBTUc2cW3etcG/QXwSDLzZ2Sb+COpOwdH3nMRw9B34+JvOeSlZTkIY9RXIvCH4cRhjolLMJ4LF2xYjCJ8b97mQ3aN5hHHxoeLQJILWktJh8GxnA6g54Exkd+QdqHgLyhbCoNth4o9hgCf08RhjIlpMPz6qqizatohbht1CTp+ckN0nt08u2WnZXe8n6KzUITDiUbhhIcw+AAX/CidKYNVU+Mt9cHJTeOIyxkSEmE4Em49uZmfVzk6tSxwIEcGT4wlfIvCWkApjvwP37XNqBMfWwltXw9qH4FQAYx2MMT1GTCeCRVsXkRCXwGfHfjbk9/LkeNh5fCfVF6pDfi+/JPaB8f8Is/bB+Gfh8CpYOQHefxiqI2JpCGNMN4nZRKCqFJUWMXP4TDJ6Z4T8fp5cD4qyoWJDyO8VkKR0p4N51j6nplC+DN68Cj54BM7uDXNwxpjuELOJ4MPyDzlw+kDQp5RoS2GOs9RCRDQP+ZI8AAqeg/v2wpV/BwcWw3+Phr8+5nQ2G2N6rJhNBIu2LSI5PpnZY2Z3y/0yemeQn54fuYmgWa8smPJ/4d5PYeTjsO+38N+joORJOH843NEZY0IgJhNBY1Mjvy/9PXdfeTd9k/t2230LcwopPhThiaBZ7xzw/ALu3Q35X4TdL8Ebw+Hjv4faY+GOzhgTRDGZCN4re4+jNUeDtgCNvzw5Hvaf3k9lTWW33rdLUofANfPh3p0wZA7sfAGW58Om7zprJhhjol5MJoKibUWkJaVx16i7uvW+LQPLIr15yJe04XDdArh7O+TNhu3POwlhyz85I5mNMVEr5hJBXWMdSz5ZwqzRs+id2Ltb7z1l0BQEiZ7mIV/6jnYGpt211RmdvO1HTkLY/CyUL4dTW6H+bLijNMYEIOammPjTp3/iZO3Jbm8WAuiT3IcxGWOis0bQWvo4uOk1Z1Tyln9yJrnzlpLl1CIu20ZAr0HOTKnGmIgQc4mgaFsRV6RcwcwRM8Nyf0+uh1V7VqGqQV0SM2yuKIBblsOFE864g7Ofuj/drfJ9Z+0E75XV4pIhLf/yBJE23NmfkBq2X8eYWBRTieBc/TmW7VjG5yd8nqQwzdPvyfHwu82/o7y6nMH9etDqYsn9nW1A4eXHGuvg3IFLE0Rzwji2FhrOXHq+d20idajzPnmg82hrirslXWG1CmOCJKYSwZu73qSmvqbbBpH54r10ZY9KBO2JT4I+I52tNVWoc2sTZz6FGu/axDrYXwTaePl1kgApmRcTQ8s28PJ9yRkQF1N/1Y0JSEz96ygqLSI7LZtbht4SthgmZU8iIS6B4kPFPHDVA2GLI2KIOKOakwf4nhJbm5xmp9qjXtuxy9+f/sR53XTB102c8r2TRd+rIOM6yLgGErtvLIkxkShmEsHp2tO8uetNHp/yOPFx8WGLIyUhhYlZE3tGh3F3kDhIyXA2xrV/rirUV/tOFN7vqz6C/YsBBQT6jXOTgrv1vdKanUxMiZlEsHznci40Xghrs1AzT46Hom1FNGkTcfaBEzwikNTP2fqOav/cutNOQjj+gbMdeA0+/bVzLOkKZ73njOsg8zoYMNVqDaZHi5lE8MBVD9A7sXf3rBDWAU+Oh19t+BV7TuzhygFXhjuc2JTUz1nLeZD79Jg2QfXOi4nh+Aew9Y+01BrSx19aa+hzpa3/bHqMmEkEaUlpPDj2wXCHAVy6dKUlggghcdDvKmcb8SVnX91pqPrrxcSwfzHsme8cs1qD6UFiJhFEkrGZY+mV0IviimIenvhwuMMxbUnqB4NuczZwaw07WtUa3nJPtlqDiV6WCMIgIS6ByYMmW4dxtJE46DfW2Ub8rbOv7hQcb6vW0B8yrr2YGAZMdVaGMybCWCIIk+Z+goamBhLsGffolZQOObc7Gzi1htOfOEmh6kPnZ8VK55jEQb/WtYZRVmswYWefQGHiyfXwwl9foPRYKZOyJ4U7HBMsEufMw5Q+DkZ+2dl3Wa1hEez5lXMseYDb13Ct1RpM2IQ0EYjIHcC/A/HAy6r6XKvjY4DfAJOB76vqv4UynkjiPcLYEkEP116toXmreNM5ZrUGEwYhSwQiEg/8EpgJlAPFIvKGqm73Ou0E8BQwO1RxRKqR/UeSnpJO8aFivjz5y+EOx3Qnn7WGk23XGhJSIb4XxCV5bYntvPfzWNIVXnM6DXGOmZgUyhrBVGCPqu4FEJEiYBbQkghU9RhwTETuDmEcEUlEnKUrrcPYgPOhnHOHs4Fba9gOxz+E06XO1BlNdc4Efk11oPWXvm66APVn2jjmdV1THc7YiFYkDnoPaWNG2OHuJH9WK+mpQpkIcoGDXu/LgWs6U5CIPAY8BjBkyJCuRxYhPDkefrL+J9Q21JKSkBLucEwkkTjncdT08cEvu6nRSQgXjkPNvouT/J1xZ4Q99Mbl61In9vOdIKw20SOEMhH4+vrg46tIx1R1PjAfoLCwsFNlRCJPjoeGpgY2HdkUESOeTYyIi4e4XpAwGFIHw8CbLz+n/uzFJHHGa42J09vg0H+7NQtXS23CTRC9c33PBJuQZrWKCBXKRFAOeM+znAdUhPB+Ucd7hLElAhNREtMgfYKztaZNcO5Qq/Ul3K18GVyo9F1mfC8f04T7mDY8ZaAzBiPYSUMVtMFtInObzOKSnYGDMS6UiaAYGCUi+cAhYC7w+RDeL+rk9sklOy3b+glMdJE4pyaROhiyfEzp3lQPtZVtTxleexRqDkBVsZM02lxvYuDFxJCcCeilH+KXvfbj2OU3cqY/z7nL2fpPicmZZ0OWCFS1QUS+DqzCeXz0FVUtFZEn3OMviUg2UAL0BZpE5BvAWFWtDlVckURE8OR4LBGYniUuEXrnOFtHLltvoo3EUb3L+YD29QRUQlrnnpqKS3T6SSr+CFv/Gbb+0Ek4OXc6SWHQbU4neQwI6TgCVV0JrGy17yWv10dwmoxilifHw4pdK6i+UE3fZJu0zMSYQNabCJUJ/wS1x+HwKmcU+KEVsO93IPHOOI7m2kL6xB7bxxF7daAIU5hTiKJsqNgQ7lCMiV0pGZD/MNywEB44BjPXw9jvQcM52PwP8FYBLBsMf30MDi5zHtXtQWyKiTBr7jAuqShhWv60MEdjjCEu3plaPPM6mPRjOH/YaT6qWAkHFjsLGMUlQubNF2sLfUdHdW3BEkGYZfTOYFj6MOsnMCZS9RoEIx51tqZ6qFzvTAlSsRI2/r2zpeZfTApZt0JC7/bLVO1cp3facGfNjCCzRBABrMPYmCgRl+g8KZV1C1z9PNTsh4q3nKSw9zew+5cQnwJpIy8d4d36A10bOnf/sc9AwXMdnxcgSwQRwJPj4bXtr1FZU0lmama4wzHG+Ct1KIx6wtkaa+HYGicp1ByA+GSQRIh3n1Ly53VH5/XODcmvYYkgAnj3E9w56s4wR2OM6ZT4lEtXtIsi9tRQBJgyaAqCWPOQMSYsLBFEgD7JfRiTMcYSgTEmLCwRRAhProfiQ8Wo9pg59YwxUcISQYTw5Hg4WnOU8urycIdijIkxlggihPfSlcYY050sEUSISdmTSIhLoPiQJQJjTPeyRBAhUhJSmJg10WoExphuZ4kggnhyPJRUlNCkTeEOxRgTQywRRBBPjofTF06z58SecIdijIkhlggiiPfSlcYY010sEUSQsZlj6ZXQy/oJjDHdyhJBBEmIS2DyoMmWCIwx3coSQYTx5HjYeHgjDU2dnKbWGGMCZIkgwnhyPZxvOE/psdJwh2KMiRGWCCKMjTA2xnQ3SwQRZmT/kaSnpNuTQ8aYbmOJIMKICIU5hVYjMMZ0G0sEEciT42Hrsa3UNtSGOxRjTAywRBCBPDkeGpoa2Hxkc7hDMcbEAEsEEahlhLE1DxljuoElggiU2yeX3D65PLv6WR5Z9gjLdizjXP25cIdljOmhLBFEIBFh6Zyl3HvlvSzfuZz7F99PxvMZzC6azW82/obKmspwh2iM6UEk2tbILSws1JKSknCH0W3qG+tZe2Aty3YsY9mOZRysPkicxHHjkBuZPXo2s8bMYvgVw8MdpjEmwonIBlUt9HnMEkH0UFU2HdnkJIWdy9hydAsAEwZOYPaY2cweM5urs69GRMIcqTEm0lgi6KH2ntzL8h3LWbZzGesOrKNJmxjcd3BLUrhpyE0kxieGO0xjTASwRBADjp87zopdK1i2YxmrPl1FbUMt6Snp3HPlPcwePZvbR95OWlJal+9T31hPTX0NZ+vOUlPn/qyvQVWZkDWB/r36B+G3McYEmyWCGFNTV8Pbe99m2Y5lrNi1gqrzVSTHJzNzxEzuGXUPfZP7tnyAt/5Ab/552T73fX1Tfbv3HpY+jMKcQqYMmuJsOVMsORgTASwRxLCGpgbeP/B+S79C2amyy85JjEskLSmN1KRU52di6iWv29yXlNryuqGpgc1HN1NSUcKGwxvYe3JvS/n56flMyZlC4aBCpuQ4CeKKXld045+CMcYSgQGczuY9J/bQpE0tH+KpSakkxScF/V4nzp/g48Mfs6FiAxsOb6CkooR9p/a1HB9+xfCWWkNhTiGTB0225GBMCIUtEYjIHcC/A/HAy6r6XKvj4h6/CzgHPKKqH7dXpiWC6NWcHJprDRsqNlyWHLyblSw5GBM8YUkEIhIP7AJmAuVAMTBPVbd7nXMX8CROIrgG+HdVvaa9ci0R9CxV56qcmsPhizUH7+arAb0GhORx2KT4pDabuXzua6/ZLCmV3om9iRMbn2kiV3uJICGE950K7FHVvW4QRcAsYLvXObOA36mTjT4UkXQRGaSqh0MYl4kgA3oPYOaImcwcMbNlX3NyKKkooby6POj3VJS6xrpLOsarzlVxoP5AS8d4TX1NwLO/9kroRXxcfNDjNabZ09c+zY+m/Sjo5YYyEeQCB73el+N86+/onFzgkkQgIo8Bj7lvz4rIzuCG2mUZwPFwBxGAaIo3amI9z3mIoniJrlghuuINSaw/dv/rpKFtHQhlIvBVn2/dDuXPOajqfGB+MIIKBREpaavKFYmiKd5oihWiK95oihWiK95oihVCO+lcOTDY630eUNGJc4wxxoRQKBNBMTBKRPJFJAmYC7zR6pw3gP8hjmuB09Y/YIwx3StkTUOq2iAiXwdW4Tw++oqqlorIE+7xl4CVOE8M7cF5fPTRUMUTYhHbbNWGaIo3mmKF6Io3mmKF6Io3mmKNvgFlxhhjgssefDbGmBhnicAYY2KcJYIuEJHBIrJaRD4RkVIR+btwx9QREYkXkY0isiLcsXTEHWD4uojscP+Mrwt3TG0RkafdvwPbRGSRiKSEOyZvIvKKiBwTkW1e+/qLyNsistv9GRHzebQR60/cvwdbRGSpiKSHMcRL+IrX69i3RERFJCMcsfnLEkHXNAB/r6pXAdcCXxORsWGOqSN/B3wS7iD89O/AH1V1DDCJCI1bRHKBp4BCVR2P83DE3PBGdZkFwB2t9n0XeFdVRwHvuu8jwQIuj/VtYLyqTsSZuuZ73R1UOxZwebyIyGCcKXYOdHdAgbJE0AWqerh5kjxVPYPzQZUb3qjaJiJ5wN3Ay+GOpSMi0he4GfhPAFWtU9VTYQ2qfQlALxFJAHoTYeNhVHUNcKLV7lnAb93XvwVmd2dMbfEVq6r+SVUb3Lcf4ow5ight/NkC/Az4Dj4GyUYaSwRBIiLDgKuBv4Y5lPa8gPMXsynMcfhjOFAJ/MZtynpZRFLDHZQvqnoI+Decb36HccbD/Cm8Ufklq3ncjvtzYJjj8deXgLfCHUR7ROQ+4JCqbg53LP6wRBAEIpIGLAG+oarV4Y7HFxG5BzimqhvCHYufEoDJwH+o6tVADZHTdHEJt219FpAP5ACpIvKF8EbVM4nI93GaZBeGO5a2iEhv4PvAD8Idi78sEXSRiCTiJIGFqvqHcMfTjhuA+0SkDCgCpovIf4U3pHaVA+Wq2lzDeh0nMUSizwD7VLVSVeuBPwDXhzkmfxwVkUEA7s9jYY6nXSLyReAe4GGN7AFQI3C+FGx2/73lAR+LSHZYo2qHJYIucBfW+U/gE1X9v+GOpz2q+j1VzVPVYTgdmX9W1Yj91qqqR4CDIjLa3TWDS6cwjyQHgGtFpLf7d2IGEdqx3cobwBfd118Elocxlna5i1w9A9ynqufCHU97VHWrqg5U1WHuv7dyYLL7dzoiWSLomhuAv8H5dr3J3e4Kd1A9yJPAQhHZAhQA/xLecHxzay2vAx8DW3H+XUXUFAMisgj4ABgtIuUi8rfAc8BMEdmN83TLc+2V0V3aiPUXQB/gbfff2UthDdJLG/FGFZtiwhhjYpzVCIwxJsZZIjDGmBhnicAYY2KcJQJjjIlxlgiMMSbGWSIwEUdEhvmayTGI5c/2nhxQRH4kIp8JUtkr3VlT00Xkq8Eo06vsb7ijVi+5VzDvYWKTPT5qIo47b9MKdybPUJS/wC3/9VCU795jGAH+Du5gNFFVn3NBuaNUC1X1eFCCNMZlNQITqRJE5Lfu/POvN38TFpEZ7iR0W9154JM72P+ciGx3y/k3EbkeuA/4iTswaYSILBCRB93zy0Tkn0XkY7esMe7+THfO/o9F5Fcist/XHPPu9Rk4g7NGuPf4iXvs2yJS7Mbyz+6+YeKstfAizoC0wSLyHyJSIs76Bs3nPYUzj9FqEVnd6l6IyDfFWQthm4h8o1XZv3bL+pOI9Gouz+vPpSgE//9MNFFV22yLqA0YhjN17w3u+1eAbwEpwEHgSnf/74BvtLO/P7CTizXfdPfnAuBBr/u1vAfKgCfd118FXnZf/wL4nvv6Dje+DB+xlwEZ7u+wzWv/bTijjQXnC9gKnGm2h+HMBnut17n93Z/xwHvARO+yfdxrCs6I5lQgDSjFmQl3GM4EbQXu+b8HvuC+rgCSvf9cbIvdzWoEJlIdVNX33df/BdwIjMaZ3G2Xu/+3OB+mbe2vBmqBl0XkAcDfOWqaJw/cgPNhinv/IgBV/SNwMsDf5zZ324jzzX8MMMo9tl9VP/Q693Mi8rF77jigo8WObgSWqmqNqp5147/JPbZPVTf5+H224Ezf8QWcZGFimCUCE6lad14pzrdpX3zuV2chk6k4s8POBv7o570vuD8bcabDbvMeARDg/6hqgbuNVNX/dI/VtJwkko9T+5mhzmpcb+LUeDoquy0XvF57/z53A7/EqU1sEGdBHROjLBGYSDVELq5RPA9YB+wAhonISHf/3wB/aWu/OOtE9FPVlThNRQXu8TM4E5gFYh3wOQARuQ3oaH3f1vdYBXzJjQkRyRURXwvB9MVJDKdFJAu4s50ym60BZruzn6YC9wNr2wpMROKAwaq6GmehonScJiUTo+xbgIlUnwBfFJFfAbtxFqipFZFHgdfcb7DFwEuqesHXfpw+guXiLCQvwNNu2UXAr90O2Af9jOefgUUiMgcn+RzG+WD2SVWrROR99zHYt1T12yJyFfCB83AQZ4Ev4HxL975us4hsxGnn3wu873V4PvCWiBxW1Wle13zsPgn1kbvrZVXd6D655Es88F8i0g/nz+VnGtnLgJoQs8dHjfGD+xRSo6o2uDWV/1DVgjCHZUxQWI3AGP8MAX7vNqvUAf8zzPEYEzRWIzDGmBhnncXGGBPjLBEYY0yMs0RgjDExzhKBMcbEOEsExhgT4/4/w2TcQGqHG1kAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "n_estimators= 15\n", "Acuracy (on test set) = 0.9210233592880979\n", " precision recall f1-score support\n", "\n", " 0 0.99 0.98 0.98 91\n", " 1 0.86 0.96 0.90 89\n", " 2 0.96 0.97 0.97 74\n", " 3 0.97 0.81 0.89 91\n", " 4 0.97 0.93 0.95 96\n", " 5 0.93 0.94 0.94 90\n", " 6 0.93 0.93 0.93 97\n", " 7 0.93 0.97 0.95 92\n", " 8 0.85 0.85 0.85 93\n", " 9 0.84 0.88 0.86 86\n", "\n", " accuracy 0.92 899\n", " macro avg 0.92 0.92 0.92 899\n", "weighted avg 0.92 0.92 0.92 899\n", "\n", "\n", " CONFUSION MATRIX\n", "[[89 0 0 0 0 0 2 0 0 0]\n", " [ 0 85 0 0 0 0 0 0 0 4]\n", " [ 1 0 72 0 0 0 0 0 1 0]\n", " [ 0 1 0 74 0 3 1 2 6 4]\n", " [ 0 3 0 0 89 0 3 1 0 0]\n", " [ 0 1 0 0 1 85 0 0 2 1]\n", " [ 0 4 1 0 1 0 90 0 1 0]\n", " [ 0 0 0 0 1 0 0 89 0 2]\n", " [ 0 3 2 1 0 2 1 2 79 3]\n", " [ 0 2 0 1 0 1 0 2 4 76]]\n" ] } ], "source": [ "from sklearn.ensemble import AdaBoostClassifier\n", "\n", "# Create and train an adaBoost classifier using SMALL Decision Trees as weak classifiers\n", "weak_learner = tree.DecisionTreeClassifier(max_depth=6)\n", "clf = AdaBoostClassifier(weak_learner, n_estimators=15, learning_rate=1.0, algorithm='SAMME', \n", " random_state=None)\n", "clf = clf.fit(X_train, y_train)\n", "print(\"Weak_learner:\", clf.base_estimator)\n", "print(\"Weights of weak classifiers: \", clf.estimator_weights_)\n", " \n", "# Plot training curves (error = f(iterations))\n", "n_iter = clf.n_estimators\n", "from sklearn.metrics import zero_one_loss\n", "ada_train_err = np.zeros((clf.n_estimators,))\n", "for i, y_pred in enumerate(clf.staged_predict(X_train)):\n", " ada_train_err[i] = zero_one_loss(y_pred, y_train)\n", "ada_test_err = np.zeros((clf.n_estimators,))\n", "for i, y_pred in enumerate(clf.staged_predict(X_test)):\n", " ada_test_err[i] = zero_one_loss(y_pred, y_test)\n", "fig = plt.figure()\n", "ax = fig.add_subplot(111)\n", "ax.plot(np.arange(n_iter) + 1, ada_train_err,\n", " label='Training Error',\n", " color='green')\n", "ax.plot(np.arange(n_iter) + 1, ada_test_err,\n", " label='Test Error',\n", " color='orange')\n", "ax.set_ylim((0.0, 0.5))\n", "ax.set_xlabel('boosting iterations')\n", "ax.set_ylabel('error rate')\n", "leg = ax.legend(loc='upper right', fancybox=True)\n", "plt.show()\n", "\n", "# Evaluate acuracy on test data\n", "print(\"n_estimators=\", clf.n_estimators)\n", "score = clf.score(X_test, y_test)\n", "print(\"Acuracy (on test set) = \", score)\n", "from sklearn.metrics import classification_report\n", "from sklearn.metrics import confusion_matrix\n", "y_true, y_pred = y_test, clf.predict(X_test)\n", "print( classification_report(y_true, y_pred) )\n", "print(\"\\n CONFUSION MATRIX\")\n", "print( confusion_matrix(y_true, y_pred) )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Question:__ Looking at the training curves, you can see that **training error goes down to zero rather quickly, but that test_error still continues, after training error is zero, to diminish with increasing iterations**. __Is it normal, and why?__ (check the course!)\n", "\n", "__Answer:__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Now, for the case of _DecisionTree_ weak classifiers, find somewhat optimized values of (max_depth, n_estimators) by using CROSS-VALIDATION.** __Put the code below:__" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Question: What best value have you managed to reach for TEST accuracy of your AdaboostClassifier after you properly gridSearched its hyper-parameters using CrossValidation?__\n", "\n", "__Answer:__" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.8" } }, "nbformat": 4, "nbformat_minor": 1 }