{"id":5927,"date":"2021-11-22T11:42:49","date_gmt":"2021-11-22T10:42:49","guid":{"rendered":"https:\/\/joapen.com\/blog\/?p=5927"},"modified":"2021-11-22T11:42:53","modified_gmt":"2021-11-22T10:42:53","slug":"intermediate-machine-learning-by-kaggle","status":"publish","type":"post","link":"http:\/\/joapen.com\/blog\/2021\/11\/22\/intermediate-machine-learning-by-kaggle\/","title":{"rendered":"Intermediate Machine Learning, by Kaggle"},"content":{"rendered":"\n

Some notes of this course offered by Kaggle, for my poor memory.<\/p>\n\n\n\n

Cross validation<\/h2>\n\n\n\n

Cross-validation gives a more accurate measure of model quality, which is especially important if you are making a lot of modeling decisions.\u00a0<\/p>\n\n\n\n

Use pipelines for doing cross-validation, you will save a lot of time.<\/p>\n\n\n\n

\"tut5_crossval\"\/<\/figure>\n\n\n\n

XGBoost = Gradient boosting<\/h2>\n\n\n\n

We refer to the random forest method as an “ensemble method”. By definition,\u00a0ensemble methods<\/strong>\u00a0combine the predictions of several models (e.g., several trees, in the case of random forests).<\/p>\n\n\n\n

Gradient boosting is an ensemble method too that goes through cycles to iteratively add models into an ensemble.<\/p>\n\n\n\n

  1. Naive model is build as initial prediction that will be used as basis for following predictions.<\/li>
  2. Make predictions, we use the first prediction or the iterative predictions to generate predictions for each observation in the dataset. These predictions are added in the ensemble.<\/li>
  3. Calculate loss, we track the data returned by a loss function (like mean square error (MAE)).<\/li>
  4. Train new model, we use the loss function to fit a new model that will be added to the ensemble. Specifically, we determine model parameters so that adding this new model to the ensemble will reduce the loss. (Side note: The “gradient” in “gradient boosting” refers to the fact that we’ll use\u00a0gradient descent<\/a>\u00a0on the loss function to determine the parameters in this new model.<\/em>).<\/li>
  5. Add new model to ensemble, and repeat again and again.<\/li><\/ol>\n\n\n\n
    \"tut6_boosting\"\/
    Gradient Boosting process<\/figcaption><\/figure>\n\n\n\n

    Data leakage<\/h2>\n\n\n\n

    Data leakage happens when your training data contains information about the target, but similar data will not be available when the model is used for prediction. This leads you to obtain high performance on the training set (and possibly even the validation data), but low performance in production (and a lot of frustration).<\/p>\n\n\n\n

    Types of data leakage:<\/p>\n\n\n\n