[Corpora-List] "Cargo cult" NLP?

John F Sowa sowa at bestweb.net
Thu Apr 10 00:01:39 CEST 2014


On 4/9/2014 12:16 AM, Jason Eisner wrote:
> But this hardly shows that A will do better than B in other settings.
> The advantage might not carry over to other pairs of implementations,
> other training sets, other test distributions, or other evaluation metrics.

For more examples, Google the phrase "damn lies and benchmarks".

Vendors who design new hardware and software have been advertising their performance on "benchmarks" for over half a century. But customers have learned that benchmarks don't answer the question whether a particular system will solve their problem.

NLP is primarily a branch of engineering. The challenge for an engineer is not to get a better score on some "metric", but to solve a client's problem within the constraints of budgets, deadlines, and available resources.

John



More information about the Corpora mailing list