Next Generation Error Metrics for Model Validation

Sean Ziegeler
High Performance Technologies, Inc

Abstract:
Part of the process of model validation includes the use of error metrics that directly compare model forecast to ground truth. Such metrics include simpler mean differences, root-mean-square (RMS) differences, and cross-correlations, to more composite quantities like skill scores, to even more elaborate manual feature tracking studies. Spatial displacement can be computed automatically using deformable registration algorithms. The result is a vector field indicating the movement of data points between the forecast and ground truth data sets. This displacement field can be used as an alternative indication of error. Theoretical advantages include a spatially-based metric value and the ability to separate between spatial and amplitude error. The effectiveness of several registration algorithm configurations were compared by introducing a synthetic warping to a data set, then determining which configuration recovered the synthetic displacement best. Artificial biases were also introduced to gauge performance in such conditions. The synthetic trials were performed on NCOM model output from 2009 in 20 cut-out regions using two time steps per month, for a total of 480 trials. Results of the trials are provided along with some background on the concepts of displacement and deformable registration.