TRF - JTRF Volume 44, Number 3 (Multi-Measure Quality Analysis of Weather Forecasts Used for Winter Highway Maintenance Purposes)

State Departments of Transportation spend a considerable portion of their winter maintenance budgets getting accurate weather forecasts that are essential for proactive operations and are interested in getting the best value for their money. With that intention, the Kansas Department of Transportation obtained weather forecasts from three commercial weather data providers for four selected locations within the state of Kansas. This paper presents a quality analysis of those forecasts to identify the most accurate and reliable provider. In doing so, researchers used the prediction of frost events as the performance measure. Several different measures that were capable of quantifying the selected attributes (reliability, accuracy, and skill) were used to evaluate different forecasts for each location and for combined data.

Introduction

Surface transportation dominated by the highway mode plays an invaluable role in moving people and goods from one place to another. Highway transportation affects our day-to-day lives at least several times a day and the whole economy is dependent on its efficient functioning. However, safety and efficiency, primary operational goals of the highway system, are often affected by various environmental conditions such as precipitation (whether it is in the form of snow, rain, or sleet), high winds, reduced levels of visibility, and many others. This intersection of weather with the safety and efficiency of the surface transportation system is therefore an important aspect for many transportation agencies throughout the United States.

According to the estimates by National Highway Traffic Safety Administration, each year 7,000 fatalities and 800,000 injuries occur as a result of weather-related adverse road conditions (Office of The Federal Coordinator For Meteorological Services And Supporting Research 2002). The estimated cost of deaths, injuries, and property damage as a result of these weather-related crashes amounts to almost $42 billion. Additionally, there are many hours of vehicle delay on highways because of adverse weather conditions.

While there are many aspects to the safety of road users and traffic operations under adverse weather conditions, from the viewpoint of the highway agency the most important one becomes highway maintenance operations. Predominantly, it includes road surface treatment for snow and ice control in the winter. Until recently, snow and ice control had been generally reactive in nature, where agencies would typically perform mechanical removal of snow accompanied by deicing chemicals or traction enhancement with abrasives as snow accumulates (Boselly 2001). With increasing expectations of the traveling public and companies relying more on just-in-time delivery, the reactive strategy is becoming more and more unacceptable. Instead, anti-icing practice has become more and more popular among many highway agencies as a proactive strategy.

Anti-icing is the snow and ice control practice of preventing the formation or development of bonded snow and ice by timely applications of a chemical freezing point depressant (Federal Highway Administration 1996). It makes the maintenance manager capable of maintaining the roads in the best possible condition during a winter storm. While this approach has benefits in terms of improving safety and efficiency of the highway system during adverse weather, it requires use of considerable judgment in the decision-making process, application of the available information in an effective manner, and taking prompt action. Accordingly, the whole process in proactive winter maintenance operations is dependent upon the accuracy of the forecasts available to the decision maker.

To fulfill the need for obtaining better weather forecasts in a timely manner, transportation agencies use Road Weather Information Systems (RWIS). In 1990, more than 42 states operated such RWIS and the number could be larger by now (Kelley 1990). A RWIS collects information from pavement temperature sensors and ice detectors, meteorological sensors in the atmosphere, and weather forecasts from commercial vendors. Mainly, these forecasts obtained from commercial suppliers are the basis for the decision making related to anti-icing because the others provide real time or actual data rather than forecasts. Highway transportation agencies spend a considerable portion of their maintenance budgets in obtaining these forecasts so that the highway system can be kept in usable condition under adverse weather conditions. With increasingly tight budgets and higher expectations from users, it is essential for the agencies to get the best and most accurate weather forecast in a timely manner.

To accomplish this, the Kansas Department of Transportation (KDOT) decided to obtain weather forecasts from three vendors for winter 2002-03. To avoid endorsing any commercial product, the providers are labeled here as providers 1, 2 and 3. This study conducted analyses based on the data provided by the three vendors with the intention of measuring the quality and value of each forecast. Even though there are some other important parameters, prediction of frost events was used as the performance measure in this study because that data element was the most complete and accurate as compared to others in the dataset. Accordingly, this paper describes a methodology that could be used in evaluating quality of weather forecasts for highway winter maintenance purposes and shows the results obtained by applying the methodology for Kansas data. The same methodology could be applied to data from any other state to evaluate weather forecasts.

Methodology

Attributes Considered

In a well-known work done by Stanski et al. (1989) regarding verification methods in meteorology, it is emphasized that no single verification measure provides complete information about the quality of a product. Because they all provide information on one or more attributes, it is necessary for a verification system to compute several measures chosen to describe the attributes that are more critical to the selected measure. Accordingly, Multi-Measure Quality Analysis of forecasting a road-weather event is utilized in this study to evaluate the forecasts provided by the vendors. Based on past studies, the total quality of a weather forecast consists of six attributes (Stanski et al. 1989). These are: (1) reliability, (2) accuracy, (3) skill, (4) resolution, (5) sharpness, and (6) uncertainty. Of these six attributes, road maintenance is more closely associated with the first three attributes (Thornes and Stephenson 2001). Therefore, multiple measures, each of which measures one of the three attributes, reliability, accuracy, and skill are used to evaluate the quality of road weather forecasts provided by the vendors.

Brief descriptions of these attributes are as follows (Stanski et al. 1989, and Thornes 1996).

Selected Parameter

The accuracy of the forecasts were analyzed in this study by considering frost events which are very strongly related with one of the most important parameters in terms of winter maintenance operations, pavement temperature. In this analysis, for simplicity and also for the purpose of having a sufficiently large sample size for analysis purposes, a frost event is defined as a day during which the lowest surface temperature falls below 0° C or 32° F, irrespective of surface moisture. Because the consideration of days where the temperature was far above freezing was not considered critical for winter maintenance purposes, only the days during which the lowest surface temperature is no greater than 5° C or 41° F were considered for analysis. The cut-off value of 41° F was selected to be consistent with previous work done by other researchers (Thornes and Stephenson 2001) and allow for errors in the forecast. From the larger dataset, days satisfying this condition, i.e. days with surface temperature no greater than 41° F, were selected and analyzed as described in the following sections.

Contingency Table Development

In this case, it is necessary to evaluate a road weather forecast with regard to how well it forecasts a frost event. This could be done by developing a simple contingency table, where there are two possibilities (frost or no frost) for both forecast and actual conditions.

The total number of events (n) considered in the analysis would be the summation of all the events, i.e. n = a + b + c + d. Note that this is equal to the size of the sample mentioned earlier where temperature conditions were satisfied, i.e. surface temperature no greater than 41° F. Additionally, the second category (b) could be considered as an over-forecasting of events where more frost events were forecast than actually happened and the third category (c) corresponds to an under-forecasting of events where fewer frost events were forecast than actually happened.

Multi-Measures

Based on the frequency values in the developed contingency table with respect to any specific event, the following multi-measures that quantify each of the selected attributes could be used in the evaluation and/or comparison process.

When B = 1 it indicates a perfect situation where number of misses in each category, i.e. events forecast but did not happen and events not forecast but did happen, are equal. However, a perfect bias score does not mean the forecast is accurate. Thus, such reliability only reflects that the forecast has equal probability of over-forecast vs. under-forecast as defined earlier. B>1 indicates an over-forecasting whereas B<1 indicates under-forecasting.

Skill measures: Attempts to assess how much better the forecasts are than those which could be generated by climatology, persistence and chance.

Higher values of PSS and ORSS (where max value =1) both indicate better conditions as compared to lesser values. For example, PSS takes the maximum possible value of 1.0 when both miss rate and false alarm rate are zero, which happens as a result of having both b and c values equal to zero. If b and c are zero, ORSS = 1.0. On the other hand when there are high miss and/or false alarm rates, and b and c are large, PSS and ORSS have smaller values indicating that the predictions are less accurate.

The Data

Currently, KDOT maintains 41 RWIS stations spread throughout the state as shown in Figure 1. From these stations, four locations were selected for obtaining trial forecast data from the three vendors. Those are: Edson (NW); Great Bend (Central); Bull Creek (Kansas City Area); and Chanute (SE). The contract period extended from Dec. 11, 2002 to Mar. 31, 2003 which covered the actual winter period in the state. Each vendor displayed their forecasts and some selected real-time data for the four selected stations on the KDOT website. The data posted on the websites, both forecast and actual, were archived and then made available for this research. The word "actual" is used throughout this paper to represent the data gathered from the KDOT weather stations.

The data were presented in four text data files, one for each vendor and one for actual data. Data parameters and typical format of data are represented in Table 1. The only structural difference between the actual and forecast datasets was that the value for precipitation probability which contained no value for the actual data. The preliminary investigation of the available data indicated there was a very poor correlation between the actual data and any of the vendor-provided data. After time-series plots were made, it was discovered that the time stamp was Greenwich Mean Time (GMT) for the actual data and Central Standard Time (CST) for each of the three vendors. This discrepancy was corrected before performing any further analysis.

Even though the vendors provided several updates daily, for the sake of comparison, the forecasts made immediately prior to 3:00 p.m. each day were used. The software for the KDOT website uploaded the actual data from four of the 41 sites at approximately 10-minute intervals. For the purpose of comparison, the actual data element closest to 3:00 p.m. was used for the analysis.

Actual Data Concerns

This study used the above-described multi-measures to evaluate and compare the quality of the weather forecasts provided by the three vendors for the particular winter season. However, the whole process was based on the comparisons between actual and forecast datasets. Accordingly, it was dependent on the fact that actual values were truly the real conditions prevailing at the locations. To verify this critical assumption a sample of RWIS stations were assessed for their accuracy. This was done by validating the pavement temperature sensors at these locations, where several techniques were taken into consideration. The first was to compare air temperature to the pavement temperatures, looking for a pattern that could be used. This was quickly discarded for several reasons. The air temperature sensors are located a distance from the pavement sensors at a higher elevation, and are shielded from solar radiation. Both these combine to add considerable difference to the measurements, with the possibility of capturing only the grossest errors.

The second method combined the placement of electronic thermistors in the shoulder of the roadway and recording the pavement temperature on five-minute intervals. An infrared gun was also used to make a series of temperature measurements next to the pavement sensors and to the thermistors to validate pavement temperature sensors. A Campbell Scientific CR10 data logger was placed in the median of the roadway in close proximity to the road sensors.¹ A series of infrared readings were taken under two conditions: (1) when weather conditions were mild and there had been no snow and (2) temperatures had been below freezing for several days and there had been significant snowfall.

Through data analysis it was found that under mild conditions the temperatures measured by the methods discussed in the previous paragraph compared favorably, until temperatures exceeded about 40° F. It was also noted that the infrared readings showed poor correlation with the pavement sensors above this temperature. This was expected, because heat radiating from the pavement surface would influence the infrared readings. Temperatures in this range were outside the critical range considered in this study. Under very cold and snowy conditions, however, there was an excellent correlation of the readings indicating the accuracy of the observations representing the actual conditions. Based on both conditions, because only the low temperatures are important in making winter weather decisions and only these values are treated in the analysis, it was decided that the data does not violate the assumption of the study, i.e. observations on pavement temperature (actual values) indicated the true conditions.

Empirical Results

Data analysis was conducted first on each station separately and then on the combined data of all four stations for each of the vendors.

Separate Frost Forecast Analysis for Four Locations

Summary of the contingency tables developed for Edson, Great Bend, Bull Creek and Chanute are given in Tables 2 through 5, respectively. Such separate analyses were thought to be useful since the forecasting in one location may or may not be more difficult than another. Even though there were 111 days from Dec. 11, 2002 through Mar. 31, 2003, the time period of the forecasts, it should be noted that in no case the total base value (n) equaled 111 indicating missing data in databases or days with temperatures above the cut-off value.

Results of the analysis based on the multi-measures that quantify the three selected attributes are summarized in Table 6. Some of the findings based on each station are as follows.

Edson. Based on the value of bias B, providers 1 and 3 over-forecast the frost events at Edson, whereas the other provider under-forecast such events. However, provider 1 was better in terms of reliability as the value is much closer to 1.0. When considering percent correct and miss rate, provider 1 was more accurate than the other two, even though it had a higher false alarm rate than provider 2. In terms of the two skill scores, providers 1 and 2 were better than the third.

Great Bend. Provider 2 was the best in terms of reliability as indicated by bias, closely followed by provider 1, even though all of them under-forecast frost events. Provider 2 was also the most accurate as indicated by percent correct and miss rate and had the highest skill values as well. Provider 3 had the lowest false alarm rate. Therefore, when forecasting frost events at Great Bend, provider 2 has very clearly out performed the other two.

Bull Creek. All three providers slightly over-forecast frost events at Bull Creek station. As far as accuracy is considered, provider 1 performed better than the other two, which was illustrated by all three measures percent correct, miss rate, and false alarm rate being more accurate for provider 1. Both skill scores were also better for provider 1.

Chanute. From among all the stations, the worst forecasting of frost events was for Chanute station. All three providers over-forecast such events by a considerable margin. Even for the best provider, percent correct was only 78.7, whereas for the other three stations the values were more than 88 for all providers. For all providers the miss rate was relatively low, but that was only because they had a very high false alarm rate due to over-forecasting. The skill scores were lower for all three providers than for the other three stations. Even with non-impressive numbers, provider 1 did a better job than the other two.

Combined Frost Forecast Analysis Considering All Four Locations

By considering the fact that the average or overall quality and value of the forecasts are more important to the agency than the accuracy at each specific location, data from all four locations were combined and the same quality analysis was performed. This, in practical terms provides a more meaningful evaluation since only one vendor is contracted to provide services for all locations within the state. In other words, overall quality is more important than site-specific quality. This approach also allowed for a larger sample size and bigger values in the contingency table making the analysis statistically more reliable.

The contingency table developed for the overall quality analysis is in Table 7 and the overall frost forecast statistics are presented in Table 8. Based on this overall analysis, provider 2 was better in terms of reliability even though all three vendors slightly over-forecast frost events. When considering percent correct and miss rate, provider 1 was more accurate than the other two. However, provider 1 has a higher false alarm rate arising from the over-forecasting and provider 2 had the lowest false alarm rate. In terms of skill scores, however, provider 1 was better, closely followed by the second provider.

Conclusions

This study tested a methodology for evaluating road weather forecasts used for winter highway maintenance purposes. Similar methodology could be adopted by any other state in evaluating their own weather forecasts.

This study utilized a multi-measure quality analysis approach to evaluate the weather forecasts provided by commercial vendors for winter highway maintenance purposes, where frost forecast prediction was used as the parameter for performance measurement. When considering individual analysis for each station, provider 1 did better in the majority of cases followed by provider 2. However, all three providers were not capable of forecasting frost events at Chanute. It is worth further investigation to see whether this is because of the actual nature of weather in the area (which could be more unpredictable than others) or because of some malfunctioning of the sensors at the RWIS location.

When considering the overall analysis, both providers 1 and 2 provided a better service in forecasting frost events. From the six multi-measures considered, four of the measures indicated better values for provider 1, whereas provider 2 did better on the other two measures. However, determination of the best provider cannot simply be done by looking at those facts. Eventually, the agency has to determine the cost-benefit effects of the consequences of having an over-forecast (false alarm) vs. missing an event that would actually happen. Not only do repeated false alarms cause lost credibility, they are costly to the agency as well, particularly in the context of anti-icing. If chemical agents are applied in response to a false alarm, the value of the chemicals and wages of the maintenance personnel are wasted. Lost wages become even more critical if the event was forecasted during a weekend because overtime pay would be required. On the other hand, a missed event risks the safety of road users. It is necessary to consider all these factors and many others in the final decision-making process with respect to which vendor is to be selected.² Based on the data analyzed in this study and on limited events and with only one parameter (frost events) evaluated, provider 1 performed better on the multi-measures analyzed. However, it should be noted that analyzing data from one trial year may not be very accurate and a multi-year study might be required to clearly differentiate one forecast provider from another. Another idea for further research would be to derive an overall index which yields a single composite score, taking into account the weighting for each of the three score types (reliability, skill, and accuracy). This approach, however, might be challenging since there is no objective way of determining associated weighting factors.

Endnotes

1. If the pavement temperature experiment is to be repeated at some date for a longer period of time, consideration could be given to using embedded, self-logging devices such as the Dallas Semiconductor Thermochron that can remain active in the pavement without replacement for up to 10 years.

2. The relative prices of the forecast providers are a relevant selection factor, but the authors did not have access to this information.

References

Boselly, S. E. Benefit/Cost Study of RWIS and Anti-Icing Technologies. Final Report, Prepared for National Cooperative Highway Research Program, National Research Council, Washington D.C., 2001.

Federal Highway Administration. Manual of Practice for Effective Anti-Icing Program: A Guide for Highway Winter Maintenance Personnel. FHWA-RD-95-202, Federal Highway Administration, U.S. Department of Transportation, Washington, D.C., 1996.

Kelley, J. R. "Solutions to Improve Ice and Snow Control Management on Road, Bridge, and Runway Surfaces." Transportation Research Record 1276, Transportation Research Board, National Research Council, Washington, D.C., (1990): 48-51.

Office of The Federal Coordinator For Meteorological Services And Supporting Research. Weather Information For Surface Transportation: National Needs Assessment Report. FCM-R18-2002, Washington, D.C., 2002.

Stanski, H. R., L. J Wilson, and W.R. Borrows. Survey of Common Verification Methods in Meteorology. WMO/TD-No 358, World Meteorological Organization, Switzerland, Geneva, 1989.

Thornes, J. E. and D. B. Stephenson. "How to Judge the Quality and Value of Weather Forecast Products." Meteorological Applications 8, (2001): 307-314.

Thornes, J. E. "The Quality and Accuracy of a Sample of Public and Commercial Weather Forecasts in the UK." Meteorological Applications 3, (1996): 63-74.

Acknowledgements

The study described in this paper was conducted under a research project sponsored by the K-TRAN program of the Kansas Department of Transportation. The cooperation and excellent support provided by project manager Peter Carttar and other members of KDOT are greatly appreciated. Opinions, findings and conclusions presented in this paper are those of the authors and may not necessarily represent the view of the sponsoring agency.

Sunanda Dissanayake is an assistant professor attached to the Department of Civil Engineering at Kansas State University. She received her B.Sc. (Eng), M.Eng., and Ph.D. degrees from the University of Moratuwa in Sri Lanka, Asian Institute of Technology in Thailand, and University of South Florida, respectively. Her research interests include various aspects of traffic engineering, highway safety, and access management.

Dean Landman is an adjunct professor attached to the Department of Civil Engineering at Kansas State University. He received his B.S. and M.S. degrees from Kansas State University and has more than 40 years of experience working in the areas of transportation planning and traffic engineering.

Wei He was a graduate research assistant working towards his Ph.D. in Civil Engineering at the time this research was conducted. He obtained his B.E. and M.S. degrees from Northern Jiaotong University in China.

Mary Knapp is a State Climatologist attached to the Information & Educational Technology Center at the Department of Communications at Kansas State University. She has a number of years of experience working in the areas of weather forecasts and climatology.