Watch videos with subtitles in your language, upload your videos, create your own subtitles! Click here to learn more on "how to Dotsub"

BITC: Ecological Niche Model Evaluation, part 1

0 (0 Likes / 0 Dislikes)
Hello, my name is Town Peterson, and I am here today to talk with you about model evaluation. Essentially, in this world of ecological niche modeling and species distribution modeling, we produce a lot of spatial predictions and we need to have some idea ofwhether those predictions are useful and robust, and so this module of the Biodiversity Informatics Training Curriculum is designed to give you some basic concepts, working towards a methodology for testing and evaluating model predictions. Essentially what we are doing is asking two questions: one is whether the models we are producting give us predictions that are better than random, essentially is there a signal that's distinguishable from the noise. and the other is whether the models are giving us a prediction that is good enough for purposes that we are conducting the study for. So let's start out by talking about some introductory concepts, and specifically we will start talking about presence data versus absence data. So presence data is one that records the occurrence of the species at a particular place at a particular time. and one thing about the presence data is that, under most circumstances, these data will be correct and informative. The relatively unusual circumstances where they are not will be things like misidentifications and georeferencing errors. But, most of the time, a presence datum is correct. Now, in absence, we have a very different situation. Absence data can be very heterogeneous in terms of why the species is not there. It could be, for example, that a group of researchers visited the place, the species was present, but the species wasn't encountered ... some species are hard to detect. So that's an absence datum that is positively misleading. Other reasons for absence might be that the place was visited, but the species wasn't there because it had never gotten there. Why are there no elephants living in South America? Probably not because they couldn't survive there. Probably rather because they've never been there. So, the absence data are very complex because we don't know which absence data points are absent because the species is not encountering appropriate environments. So, I don't want to go into a lot of detail with this, because we will treat this in other modules of this curriculum, but the basic outcome is that we place a lot more confidence in presence data than in absence data. So, for that reason, we want to build this difference in weighting into our methodologies. Okay, a second major introductory concept that we want to discuss is the idea that we have calibration data versus evaluation data. Calibration data are essentially those data that we feed into our modeling algorithm, and that the model is based on. So clearly the model is not independent of the calibration data; quite clearly, these are not the data that we want to use to test whether our model has some predictive ability. Evaluation data are independent, but that independence is a strange thing. They're still detecting presences of the species in question, so, in that sense, they're not independent, but ideally we would have some source of occurrence data that doesn't come from the same source as our calibration data. We can talk about this in other modules as well, but what I am after is that we would like to have some degree of independent confirmation of model predictions, so that our model evaluations are not circular. Now, a third important concept is that of overfitting, and I will give you a very simple example of overfitting ... Let's imagine that we had a data set that looks like this ... And we want to develop a model, so this might be a prediction, and this might be truth. This just a very simple idea, but we might produce a model that does this. And you can see that the prediction explains every bit of the truth. Alright? Each one of these occurrence points is predicted exactly by our model. But, we may have a problem here, because if we gave it more data ... these are our independent evaluation data ... maybe our evaluation data would look like this. And so you can see that our model didn't do a really good job of predicting these independent data points. And maybe what we need is a simpler model that looks like that, And that may actually be a better solution ... a simpler solution that gives us a more general answer to our prediction challenge. So, this is just a very simple example, but essentially what I'm after is that overfitting can sometimes give you a very precise answer to the calibration data, but a very imprecise, or very error-laden, view of independent data. So, we want to get essentially the simplest model that has the most general predictive ability. And that's essentially the concept of overfitting. And then the final introductory concept that i would like to mention is the difference between performance and significance. Significance is simply the idea of whether there is a signal that's distinguishable from random, essentially whether your model is performing better than random expectations. Performance is actually how well the model explains an independent data set. I'll give you another example ... imagine that we are flipping coins ... if we were flipping coins, we're expecting a 50% probability of success. Well, maybe I can predict heads versus tails in a coin flip with a 51% accuracy. Now, if I do a lot of coin flips, that's going to be statistically significant, but do I want to base strong inferences on a 1% improvement in predictive accuracy? Probably not. So those are some introductory concepts, and in the next section of this module, we'll talk about the practicalities of evaluating models.

Video Details

Duration: 8 minutes and 10 seconds
Country: United States
Language: English
Producer: A. Townsend Peterson
Director: A. Townsend Peterson
Views: 164
Posted by: townpeterson on Dec 14, 2012

A seminar on how to evaluate ecological niche model performance.

Caption and Translate

    Sign In/Register for Dotsub to translate this video.