What Is Xgboost and How actually it works internally?

What Is Xgboost ?

It is an efficient implementation of gradient boosting (GB). I don't think it has any new mathematical breakthrough. It is just a practically well designed version of GB for optimal use of multi CPU and caching hardware.

It is a penalized gradient boosting method with a variety of baselearners. 

XGBoost, short for (extreme) gradient boosting, is a fast, portable, and distributed implementation of the gradient boosting (trees) algorithm.

What is gradient boosting? => It is an ensemble (i.e. meta) machine learning algorithm that builds a strong model based on many weaker ones sequentially. To do so, it uses gradient descent. When this technique is used with decision trees, it is called gradient boosting trees.

Why is it fast? => The library is written in C++ and offers many useful wrappers in higher-level languages (Python and R for instance).

What does portable and distributed mean in this context? Portable means that it can be easily used in different settings/architectures. Distributed means that it can run on multiple cores (single machine) and/or on multiple servers (in a cluster).

If you want to understand some of the theory behind boosting trees, I refer to this excellent introduction from the official website.

Now, for the second part of your question about the best way to optimize its (hyper)parameters.

As with many real-world applications, the answer is invariably “it depends”?.

In fact, it depends on the size of the dataset at hand, the metric to optimize, the used features, the computational power available, and many more things.

That being said, here is a systematic strategy that you can adapt (that I often use myself):

Start with “good candidates” for the problem you are trying to solve. By good candidates, I mean standard values for hyper parameters that experts have been using and are known to yield good results. This is enough in a lot of cases. For a table of “good candidates”, I refer to this table from this great article.

If the performance is still not acceptable by your standards, try random search and/or grid search. This type of approach is highly parallelizable (sometimes dubbed embarrassingly parallel problem) and you are only limited by the number of nodes you can set in your computing cluster.

Finally, you can get clever by trying one the many Bayesian optimization techniques. If you are a Python user, I recommend trying hyperopt and/or skopt. Other similar libraries exist of course.

Alternatively, you can decide that you aren’t good at this and you look for a “HpoaaS” solution (short for hyperparameters optimization as a service). Sigopt and/or Datarobot are interesting ones to try.