Be careful with state-of-the-art models

The three stages of model building and the consequences of playing stupid popularity contests.

To build a model, you go through 3 stages.

First, your model sucks. You are better off making a random guess than using it.

After some care, you get it working and can show it has some predictive power. Using the model at this point is better than tossing a coin, but it doesn't generalize well to data outside your small dataset.

The final stage is when the…