Next Generation Error Metrics for Model Validation
Sean Ziegeler
High Performance Technologies, Inc
Abstract:
Part of the process of model validation includes the use of error
metrics that directly compare model forecast to ground truth. Such
metrics include simpler mean differences, root-mean-square (RMS)
differences, and cross-correlations, to more composite quantities like
skill scores, to even more elaborate manual feature tracking studies.
Spatial displacement can be computed automatically using deformable
registration algorithms. The result is a vector field indicating the
movement of data points between the forecast and ground truth data
sets. This displacement field can be used as an alternative indication
of error. Theoretical advantages include a spatially-based metric value
and the ability to separate between spatial and amplitude error. The
effectiveness of several registration algorithm configurations were
compared by introducing a synthetic warping to a data set, then
determining which configuration recovered the synthetic displacement
best. Artificial biases were also introduced to gauge performance in
such conditions. The synthetic trials were performed on NCOM model
output from 2009 in 20 cut-out regions using two time steps per month,
for a total of 480 trials. Results of the trials are provided along
with some background on the concepts of displacement and deformable
registration.