Soil Organic Content Prediction
LightGBM regression pipeline for soil organic content; training on the raw target beat the log target.
Problem
The COMPFEST 18 Data Science Academy selection task: predict the organic content of soil samples from spectral, chemical and geographic data, scored by RMSE on a Kaggle leaderboard. A research notebook answering ten analysis questions about the training data came with it.
Data
The training set has 11,210 samples and 52 columns: 15 spectral principal components per band, soil chemistry such as cation exchange capacity, and land cover and region labels. The target is strongly right-skewed (skewness 2.03), and the second spectral band is missing for about 85% of samples.
Approach
I added soil-science features such as the fine-to-coarse particle ratio, total base cations and base saturation. Models went up in steps: Ridge reached a cross-validated RMSE of 16.81 (R² 0.475), Random Forest 13.41 (R² 0.666) and LightGBM 11.44 (R² 0.757). Because the official metric is raw RMSE, the final LightGBM trains on the raw target instead of the log target, averaged over three seeds.
Validation
Samples from the same source lab look alike, so I compared random KFold with GroupKFold on source_id. RMSE rose from about 11.4 to 18.30 (R² 0.378), which showed the random split was too optimistic. SHAP confirmed that source_id and cation exchange capacity drive the predictions most.
Findings
Organic content differs strongly by region: the South zone averages 57.3 against 29.1 in the Southeast, about 1.97 times higher. The notebook keeps every number reproducible from the training data.
