Older versions of the ROUGEeval summarization evaluation system were easier to fool

• We show some limitations of the ROUGE evaluation method for automatic summarization. We present a method for automatic summarization based on a Markov model of the source text. By a simple greedy word selection strategy, summaries with high ROUGE-scores are generated. These summaries would however not be considered good by human readers. The method can be adapted to trick different settings of the ROUGEeval package.

Subject headings

NATURVETENSKAP  -- Data- och informationsvetenskap (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences (hsv//eng)

Keyword

automatic summarization

automatic evaluation

Markov models

Information technology

Informationsteknik

Publication and Content Type

ref (subject category)

art (subject category)