The validation technique could help scientists make more precise forecasts | MIT News

Should you grab your umbrella before leaving the door? Verify the weather forecast in advance will only be useful if that forecast is necessary.

Spatial prediction problems, such as the weather forecast or the estimation of air pollution, involve predicting the value of a variable in a new location based on values ​​known in other locations. Scientists generally use proven validation methods to determine how much to trust these predictions.

But MIT researchers have shown that these popular validation methods can fail a lot for spatial prediction tasks. This could lead to someone to believe that a prognosis is necessary or that a new prediction method is effective, when in reality that is not the case.

The researchers developed a technique to evaluate prediction validation methods and used it to demonstrate that two classical methods can be substantially incorrect in spatial problems. Then they determined why these methods can fail and create a new method designed to handle the types of data used for spatial predictions.

In experiments with real and simulated data, its new method provided more precise validations than the two most common techniques. The researchers evaluated each method using realistic spatial problems, including the prediction of wind speed at Chicago O-Hare airport and predicting air temperature in five Us metro locations.

Its validation method could be applied to a variety of problems, from helping climatic scientists to predict the surface temperatures of the sea to help epidemiologists estimate the effects of air pollution on certain diseases.

“Hopefully, this will lead to more reliable evaluations when people present new predictive methods and better understanding how well the methods are working,” says Tamara Broderick, an associated teacher in the Department of Electrical and Informatics Engineering of the MIT (EECS) , member of the Information and Decision Systems Laboratory and the Data, Systems and Society Institute, and an Affiliate of the Computer and artificial intelligence Laboratory (CSAIL).

Broderick joins in the paper by the main author and the postdoc of Mit David R. Burt and the student graduated from the CEE Yunyi Shen. The investigation will be presented at the International Conference on artificial intelligence and Statistics.

Validation evaluation

Broderick's group has recently collaborated with atmospheric oceanographers and scientists to develop automatic learning prediction models that can be used for problems with a strong spatial component.

Through this work, they noticed that traditional validation methods can be inaccurate in spatial environments. These methods maintain a small amount of training data, called validation data, and use to evaluate the precision of the predictor.

To find the root of the problem, they performed an exhaustive analysis and determined that traditional methods make inappropriate assumptions for spatial data. The evaluation methods are based on assumptions on how the validation data is related and the data that one wants to predict, called test data.

Traditional methods assume that the validation data and test data are independent and distributed identically, which implies that the value of any data point does not depend on the other data points. But in a spatial application, this often is not the case.

For example, a scientist may be using EPA air pollution sensor validation data to prove the precision of a method that predicts air pollution in conservation areas. However, EPA sensors are not independent: they were located based on the location of other sensors.

In addition, perhaps the validation data comes from EPA sensors near cities, while conservation sites are found in rural areas. Because these data are of different locations, they are likely to have different statistical properties, so they are not distributed identically.

“Our experiments showed that you get some really incorrect answers in the spatial case when these assumptions made by the validation method decompose,” says Broderick.

The researchers had to find a new assumption.

Specifically spatial

Thinking specifically about a spatial context, where the data is collected from different locations, designed a method that assumes the validation data and the test data vary without problems in space.

For example, it is unlikely that air pollution levels change drastically between two neighboring houses.

“This regularity assumption is appropriate for many spatial processes, and allows us to create a way to evaluate space predictors in the spatial domain. As far as we know, no one has performed a systematic theoretical evaluation of what went wrong when finding a better approach, “says Broderick.

To use your evaluation technique, you would enter your predictor, the locations that you want to predict and your validation data, then automatically do the rest. In the end, it estimates how precise the predictor's prognosis for the location in question will be. However, effectively evaluating its validation technique proved to be a challenge.

“We are not evaluating a method, but we are evaluating an evaluation. So, we had to step back, think carefully and be creative about the appropriate experiments we could use, ”explains Broderick.

First, they designed several tests using simulated data, which had unrealistic aspects, but allowed them to carefully control the key parameters. Then, they created more realistic and semi-suggested data by modifying real data. Finally, they used real data for several experiments.

Use of three types of realistic problems, such as predicting the price of an apartment in England depending on its location and forecasting the wind speed, allowed them to perform a comprehensive evaluation. In most experiments, his technique was more accurate than any traditional method with which they compared it.

In the future, researchers plan to apply these techniques to improve the quantification of uncertainty in space environments. They also want to find other areas where the regularity assumption could improve the performance of the predictors, as with the temporary series data.

This investigation is financed, in part, by the National Science Foundation and the Naval Research Office.