Be careful with state-of-the-art models
The three stages of model building and the consequences of playing stupid popularity contests.
To build a model, you go through 3 stages.
First, your model sucks. You are better off making a random guess than using it.
After some care, you get it working and can show it has some predictive power. Using the model at this point is better than tossing a coin, but it doesn't generalize well to data outside your small dataset.
The final stage is when the model generalizes to samples it hasn't seen before. You can deploy this model and start using it in production applications. This is the dream, the final destination.
Moving from the first to the second stage is not hard. The more experienced you are, the faster you can beat a random baseline.
Moving from the second to the third state is much more complex. This is where people spend most of their effort.
I've read many papers that show "state-of-the-art" models.
Unfortunately, most of these models don't generalize to data outside the author's heavily curated evaluation dataset.
They claim “state-of-the-art,” but their models are in the second stage at best.
The system incentivizes academics to participate in a stupid popularity contest: the more citations they get, the more prestige they collect.
You can get more citations by publishing more papers and making bigger claims. Who cares if the models work?
If you are in the business of delivering value, be careful with state-of-the-art models.
In most situations, sticking to tried-and-true techniques will be better.