Single-Task and Multi-Task XGBoost on Tabular Regression Tasks

GitHub: Repo

Context

This project was completed as the course project for ECSE 507 (Optimization & Optimal Control). This paper looked to explore the benefits of a single task vs multi task strategies with XGBoost with a focus towards the optimization strategies.

Abstract

Tabular data remains a difficult domain for the application of machine learning techniques, due to a lack of structure in the data. Although deep learning methods have been applied in recent years, gradient boosting models like XGBoost have remained competitive in many tabular scenarios. This paper investigates the performance of single-task and multi-task XGBoost models on three mid-sized tabular datasets. Our experiments yield mixed results, the multi-task models improve performance on some tasks within some datasets but no model dominates across all tasks and datasets. Additionally, we investigate the impact of standardizing features and targets but find that it did not meaningfully impact performance. These results suggest that multi-task XGBoost may offer improvements in some circumstances but it should be investigated on a case-by-case basis rather than being applied universally.