DCG Insights

What Is XGBoost?

XGBoost stands for Extreme Gradient Boosting. It is a supervised machine learning algorithm well-known for modeling complex datasets with high performance, flexibility, and scale. At a high level, XGBoost creates an ensemble of classification and regression trees through gradient boosting (a calculus-based optimization method) and then uses the ensemble to make a prediction.

Let's break this down.

What Is a Supervised Machine Learning Algorithm? 

Supervised machine learning algorithms learn how to make predictions based on data with the benefit of already knowing the answer in advance. An algorithm's learning (or training) process tests how accurate its predictions are against the known outcome and can therefore adjust to reach an optimal state. Because the training process is directed to a known target, it is classified as “supervised.”

The alternative would be an unsupervised algorithm. When an outcome is not already known during training, the algorithm explores data to find patterns. An example of an unsupervised algorithm would be discovering groups within a dataset, called “clustering.”   

In general, practitioners use supervised algorithms to do two things.

The first is to observe characteristics and predict the class of a new observation. For example, will this loan application default 90 days after origination, yes or no? The algorithm can learn from previous applications and determine the characteristics, like FICO or debt-to-income, that may help determine future default status.  

The second is to observe characteristics and estimate a continuous numeric value. In the above example, it may be better to know more than just yes or no. It may be more helpful to know the probability of default instead (a continuous value between 0 and 1). You may recognize the similarity to logistic regression. 

XGBoost performs both classification and regression. For classification problems such as “will the loan default or not,” XGBoost can use the estimated probability of default and a threshold to determine a final classification of a new applicant. While this article focuses on XGBoost classification models, know also that XGBoost can use regression to solve for other prediction problems, like forecasting prices.

About Decision Trees

 A decision tree is a simple modeling technique that is easy to interpret and visually represent. Decision trees will sequentially “branch” or “split” data into smaller and smaller groups to create homogeneous sub-groups. Decision trees can be used for classification (predicting categorical outcomes) or regression problems (predicting continuous outcomes). 

Classification and Regression Tree (CART) is the decision tree algorithm that XGBoost uses for classification and regression. The CART algorithm scans an initial dataset, evaluates each feature and split value, and determines the best way to divide the data. It does this by finding the feature and split value that create the best result (i.e., the most homogeneity in the resulting subgroup). It is known as a "greedy" approach because it considers all possible features.

XGBoost also allows other approaches, such as a histogram-based or approximate algorithm, to find the split feature and value. The flexibility in split finding is one of the reasons that XGBoost is highly efficient with big datasets. A greedy approach may be cumbersome and time-consuming for high-dimensional datasets. However, by having the option to pivot to a histogram-based or approximate algorithm for split finding instead, XGBoost can decrease training time while preserving predictive performance.

Once the algorithm determines the best feature and feature value, it splits the data into two groups: datapoints with that feature's value above the split value, and datapoints with that feature's value below the split value. From there, the algorithm recursively splits the data by the best feature and split value until either a stopping condition is met, or further splitting offers no more improvement in the performance metric.  

XGBoost’s next step is to create an ensemble of these decision tree models through a process known as gradient boosting.  

About Gradient Boosting 

Gradient boosting is an ensemble method, which means it takes an extensive collection of "weak" models and uses them to generate a prediction based on the predictions of the individual weak models. As the name implies, a weak model contains fewer features and provides a less reliable prediction. For example, a weak model for predicting snow today could be, Is the temperature below freezing? A cold day is a prerequisite for snow, but snow does not fall on every cold day.

A gradient boosting algorithm iteratively adds new trees as it learns from the prior tree's mistakes. Continuing the previous example, consider the predictive power of adding another weak model, Is it cloudy? This corrects the earlier model's mistake of predicting snow on cold but clear days. With the addition of more weak models, the overall prediction gets stronger.

XGBoost modifies traditional gradient boosting for flexibility and performance. It does this by adding a regularization term to the objective function (i.e., a kind of penalty for models that try to get too fancy), which helps it to balance making accurate predictions with keeping the model simple.

Mathematically, the gradient of the loss function guides the model towards the optimal results, which in turn minimizes the overall error of the model. In layperson's terms, the loss function is like a judge that observes each attempt of the model to grow and picks the attempt that results in the best improvement in predictive power.

Practical Limitations and Considerations

XGBoost's effectiveness is entirely based on data and training parameters. The model learns from the data, so the broader the view it can learn from, the better it will perform when facing new information.

During development, management must consider both the model's objective and what data are available. The balance between the two influences data preparation and model training. For example, when modeling a sequence such as a time series, model developers should prepare the training data using sequential sampling. Otherwise, if the model is to solve a problem where the order of observation is not essential, then stratified or randomized data splitting may be more appropriate.

After determining the objective and available data, model developers must “tune” the model, i.e., adjust its settings (parameters) so it will perform better on new data.

Modelers can tune many parameters in XGBoost, and the model's performance is sensitive to the tuning process. Developers and management should review various performance metrics to assess the quality of the model as it is tuned. Because XGBoost is so flexible, model developers can use many different metrics, but the selected metrics ultimately depend on the model's purpose and objective function (the function the algorithm solves to minimize loss and improve predictive performance). For classification models, some metrics that may help evaluate a model are learning curves, confusion matrices, or ROC and PR curves. 

Extra Credit: Algorithm Details for the Math-Inclined

How does gradient boosting work to solve problems in XGBoost?