Skip to Main content Skip to Navigation
New interface
Journal articles

A Survey on evaluation of summarization methods

Abstract : The increasing volume of textual information on any topic requires its compression to allow humans to digest it. This implies detecting the most important information and condensing it. These challenges have led to new developments in the area of Natural Language Processing (NLP) and Information Retrieval (IR) such as narrative summarization and evaluation methodologies for narrative extraction. Despite some progress over recent years with several solutions for information extraction and text summarization, the problems of generating consistent narrative summaries and evaluating them are still unresolved. With regard to evaluation, manual assessment is expensive, subjective and not applicable in real time or to large collections. Moreover, it does not provide re-usable benchmarks. Nevertheless, commonly used metrics for summary evaluation still imply substantial human effort since they require a comparison of candidate summaries with a set of reference summaries. The contributions of this paper are three-fold. First, we provide a comprehensive overview of existing metrics for summary evaluation. We discuss several limitations of existing frameworks for summary evaluation. Second, we introduce an automatic framework for the evaluation of metrics that does not require any human annotation. Finally, we evaluate the existing assessment metrics on a Wikipedia data set and a collection of scientific articles using this framework. Our findings show that the majority of existing metrics based on vocabulary overlap are not suitable for assessment based on comparison with a full text and we discuss this outcome.
Complete list of metadata
Contributor : Accord Elsevier CCSD Connect in order to contact the contributor
Submitted on : Monday, October 25, 2021 - 12:34:30 PM
Last modification on : Tuesday, August 16, 2022 - 4:40:22 PM
Long-term archiving on: : Wednesday, January 26, 2022 - 7:51:14 PM


Files produced by the author(s)


Distributed under a Creative Commons Attribution - NonCommercial 4.0 International License



Liana Ermakova, Jean-Valère Cossu, Josiane Mothe. A Survey on evaluation of summarization methods. Information Processing and Management, 2019, 56 (5), pp.1794-1814. ⟨10.1016/j.ipm.2019.04.001⟩. ⟨hal-02130700⟩



Record views


Files downloads