Predicting carbon and water vapor fluxes using machine learning and novel feature ranking algorithms

Highlights: • Accurate NEE and ET prediction models could be constructed using feature-refined SVM trained by small datasets. • Prediction model performance greatly depends on ecosystem type, climates, prediction targets, and training algorithms. • Prediction algorithm and target have much greater impacts on modeling performance than time-series autocorrelation effects. • Gap length has great impacts on prediction accuracy. • The gap-filling performance of REddyProc and ONEFlux is limiting compared to machine learning models. Gap-filling eddy covariance flux data using quantitative approaches has increased over the past decade. Numerous methods have been proposed previously, including look-up table approaches, parametric methods, process-based models, and machine learning. Particularly, the REddyProc package from the Max Planck Institute for Biogeochemistry and ONEFlux package from AmeriFlux have been widely used in many studies. However, there is no consensus regarding the optimal model and feature selection method that could be used for predicting different flux targets (Net Ecosystem Exchange, NEE; or Evapotranspiration –ET), due to the limited systematic comparative research based on the identical site-data. Here, we compared NEE and ET gap-filling/prediction performance of the least-square-based linear model, artificial neural network, random forest (RF), and support vector machine (SVM) using data obtained from four major row-crop and forage agroecosystems located in the subtropical or the climate-transition zones in the US. Additionally, we tested the impacts of different training-testing data partitioning settings, including a 10-fold time-series sequential (10FTS), a 10-fold cross validation (CV) routine with single data point (10FCV), daily (10FCVD), weekly (10FCVW) and monthly (10FCVM) gap length, and a 7/14-day flanking window (FW) approach; and implemented a novel Sliced Inverse Regression-based Recursive Feature Elimination algorithm (SIRRFE). We benchmarked the model performance against REddyProc and ONEFlux-produced results. Our results indicated that accurate NEE and ET prediction models could be systematically constructed using SVM/RF and only a few top informative features. The gap-filling performance of ONEFlux is generally satisfactory (R² = 0.39–0.71), but results from REddyProc could be very limited or even unreliable in many cases (R² = 0.01–0.67). Overall, SIRRFE-refined SVM models yielded excellent results for predicting NEE (R² = 0.46–0.92) and ET (R² = 0.74–0.91). Finally, the performance of various models was greatly affected by the types of ecosystem, predicting targets, and training algorithms; but was insensitive towards training-testing partitioning. Our research provided more insights into constructing novel gap-filling models and understanding the underlying drivers affecting boundary layer carbon/water fluxes on an ecosystem level.