Fuzzy forecast verification – giving credit to forecasts that are "close enough"

Beth Ebert
CAWCR

Abstract:

High resolution forecasts from nowcasts and numerical models can look quite realistic and provide the forecaster with very useful guidance. However, when spatial forecasts are verified using traditional metrics such as probability of detection, false alarm ratio, and equitable threat score, they often score quite poorly because of the difficulty of predicting an exact match to the observations at high resolution. Recent years have seen the development of "fuzzy" verification approaches that reward closeness by relaxing the requirement for exact matches between forecasts and observations. Some of these fuzzy methods compute standard verification metrics for deterministic forecasts using a broader definition of what constitutes a "hit". Other fuzzy methods treat the forecasts and/or observations as probability distributions and use verification metrics suitable for probability forecasts. Implicit in each fuzzy verification method is a particular decision model concerning what constitutes a good forecast.

The key to the fuzzy approach is the use of a spatial window or neighborhood surrounding the forecast and/or observed points. The treatment of the points within the window may include averaging (upscaling), thresholding, or generation of a PDF, depending on the fuzzy method used. The size of this neighborhood can be varied to provide verification results at multiple scales, thus allowing the user to determine at which scales the forecast has useful skill. Other windows could be included to represent closeness in time, closeness in intensity, and/or closeness in some other important aspect.

This talk will describe a framework for fuzzy verification that incorporates several fuzzy verification methods. It will be demonstrated on high resolution WRF model precipitation forecasts and radar observations from the central U.S. The fuzzy verification results will be interpreted to show the additional information that can be gleaned from this approach.