Proceedings of the 59th Annual Meeting of the Association for Computational Linguisticsand the 11th International Joint Conference on Natural Language Processing, pages 404–414

August 1–6, 2021. ©2021 Association for Computational Linguistics

404

A Training-free and Reference-free Summarization Evaluation Metric viaCentrality-weighted Relevance and Self-referenced Redundancy

Wang Chen1∗ Piji Li2 Irwin King1

1Department of Computer Science and Engineering,The Chinese University of Hong Kong, Shatin, N.T., Hong Kong

2Tencent AI Lab1{wchen, king}@cse.cuhk.edu.hk

2pijili@tencent.com

Abstract

In recent years, reference-based and super-vised summarization evaluation metrics havebeen widely explored. However, collectinghuman-annotated references and ratings arecostly and time-consuming. To avoid theselimitations, we propose a training-free andreference-free summarization evaluationmetric. Our metric consists of a centrality-weighted relevance score and a self-referencedredundancy score. The relevance score iscomputed between the pseudo reference builtfrom the source document and the givensummary, where the pseudo reference contentis weighted by the sentence centrality toprovide importance guidance. Besides anF1-based relevance score, we also design anFβ-based variant that pays more attention tothe recall score. As for the redundancy scoreof the summary, we compute a self-maskedsimilarity score with the summary itself toevaluate the redundant information in thesummary. Finally, we combine the relevanceand redundancy scores to produce the finalevaluation score of the given summary. Ex-tensive experiments show that our methodscan significantly outperform existing methodson both multi-document and single-documentsummarization evaluation. The source codeis released at https://github.com/Chen-Wang-CUHK/Training-Free-and-Ref-Free-Summ-Evaluation.

1 Introduction

Text summarization systems have been developedrapidly due to the appearance of sequence-to-sequence frameworks (Sutskever et al., 2014; Bah-danau et al., 2015; See et al., 2017; Chan et al.,2020), transformer architectures (Vaswani et al.,2017) and large-scale pre-training models (Devlinet al., 2019; Liu et al., 2019). How to accurately

∗This work was mainly done when Wang Chen was anintern at Tencent AI Lab.

evaluate the summaries generated from these sys-tems also attracts more and more attention in thisresearch area. One of the most accurate evaluationmethods is human evaluation. However, humanevaluation is expensive, time-consuming, and non-reproducible. Thus, it is necessary to develop au-tomatic evaluation metrics for text summarizationsystems. Existing automatic summarization evalu-ation metrics can be roughly categorized into twogroups: reference-based metrics and reference-freemetrics. In this work, we focus on reference-freemetrics.

Reference-free summarization evaluation met-rics have been developed in parallel in multi-document summarization and single-documentsummarization. The SOTA reference-free methodfor multi-document summarization evaluation, SU-PERT (Gao et al., 2020), predicts a relevance scorefor each (document, summary) pair to estimate theinformativeness of the summary and then averagesall the scores from multiple documents as the fi-nal evaluation score. For each pair, SUPERT em-ploys the top-ranked sentences which are ranked bythe position or centrality as a pseudo reference ofthe document and then applies BERTScore (Zhanget al., 2020) to produce a relevance score betweenthe pseudo reference and the given summary. TheSOTA single-document summarization reference-free evaluation metric, LS Score (Wu et al., 2020),combines a learned linguistic scorer for the sum-mary and a cosine similarity scorer for the (docu-ment, summary) pair to produce the final score.

Although SUPERT and LS Score achieve theSOTA performance on their own areas respectively,they still have several drawbacks. For example,SUPERT only considers the relevance score be-tween the document and the summary while ignor-ing the other aspects such as how much redundantinformation is contained in the summary. Besides,SUPERT assumes that all pseudo reference sen-

405

tences are equally-important. However, in the realworld, the key information of a document is un-evenly distributed over sentences. Therefore, suchan assumption may introduce extra noise for theevaluation. Note that although SUPERT may em-ploy sentence centrality to select document sen-tences as a pseudo reference, they ignore the sen-tence centrality after the selection and still treatthe selected sentences equally-important. As forLS Score, although it does not require a referenceduring the evaluation of a summary, it requiresa large-scale training dataset with reference sum-maries to train the linguistic scorer. Besides theintrinsic drawbacks in these SOTA methods, to ourbest knowledge, there is no reference-free evalua-tion metric showing that it can achieve the SOTAperformance on both multi-document and single-document summarization.

To solve the above limitations, based on SU-PERT, we propose a novel training-free andreference-free metric for both multiple and singledocument summarization evaluation. Our metric iscomposed of a centrality-weighted relevance scoreand a self-referenced redundancy score.

For the relevance score which is employed toestimate the informativeness of the summary, weincorporate the following new features. First, un-like previous work which only utilizes the token-level representations, motivated by Clark et al.(2019), we engage a hybrid way that contains bothtoken-level representations and sentence-level rep-resentations to encode the document and the sum-mary. The purpose of the hybrid representationis to enable our method to consider richer map-ping styles (i.e., token-to-token, sentence-to-token,and sentence-to-sentence) and help to produce amore comprehensive evaluation score. Second,we utilize the sentence centrality computed fromsentence-level representations of the source doc-ument to produce the importance weights of thepseudo reference sentences and tokens. Based onthe weights, we compute a weighted relevancescore that is more precise by considering the rela-tive importance. Third, besides the F1 version ofour relevance score, we also propose an adaptiveFβ version where recall is considered β times asimportant as precision. β is computed based on thelength ratio between the pseudo reference and thegiven summary. The motivation is to punish theshort summary that can easily get high precisionwhile covering very limited important information

in the pseudo reference (i.e., low recall).To measure the redundancy of a summary, we

design a simple but effective self-referenced simi-larity score. If a summary contains much redundantinformation, there must exist plenty of semanticallysimilar tokens or sentences. Based on this assump-tion, we use the summary itself as the referenceand input a (summary, summary) pair into a self-masked BERTScore to produce a redundancy scorethat evaluates the averaged degree of semantic sim-ilarity of each token or sentence with other tokensor sentences.

After obtaining the centrality-weighted rele-vance score and the self-referenced redundancyscore, we combine them to predict the final evalua-tion score. Depending on either F1 or Fβ is appliedin our relevance score, we propose two variants ofour method: the F1-based version and the Fβ-basedversion. Extensive experiments are conducted onboth multi-document and single-document summa-rization datasets. The results show that our F1-based method already outperforms all the SOTAbaselines on all datasets. Moreover, our Fβ-basedmethod can further improve the performance onmulti-document summarization datasets.

Our contributions are summarized as follows:(1) A novel training-free and reference-free summa-rization evaluation metric which considers both rel-evance and redundancy; (2) A centrality-weightedrelevance score that effectively utilizes the sentencecentrality of the documents to provide importanceguidance for the pseudo reference tokens and sen-tences. Besides the F1 version, we also developan Fβ based relevance score which pays more at-tention to recall; (3) A self-referenced redundancyscore that utilizes a self-masked BERTScore todetect the duplicated information of the given sum-mary; (4) To the best of our knowledge, we arethe first evaluation metric that can achieve SOTAperformance on both multiple and single documentsummarization under the reference-free setting.

2 Preliminary

Notations. We denote vectors as bold lowercasecharacters and matrices as bold uppercase charac-ters. The characters that are not bold are used todenote scalars. Calligraphy uppercase charactersare utilized to represent sets.

Problem Definition. We formally define thereference-free summarization evaluation problemas follows. Give a set of documents D =

406

Summary 𝑥

Document 𝑑#

BERT

𝐰%&

…

𝐰'&

𝐬%&

…

𝐬)&

𝐰%*

…

𝐰+*

𝐬%*

…

𝐬,*

Centrality-based Sentence Selection

Centrality-weighted BERTScore (𝐹% or 𝐹.)

Self-masked BERTScore (𝐹%)

Sentence Weights

Merge Final Score

Averaged Relevance

Score

Redundancy Score

𝐾

PseudoReference 𝑟

Figure 1: Overall framework of our method. w and s are the token-level and sentence-level representations. nand N (m and M ) are the token number and the sentence number of the summary (pseudo reference). For multi-document summary (i.e., K > 1), we compute relevance scores between the summary x and each document dk,and then average them as the final relevance score.

{d1, d2, ..., dK} and a generated summary x, thegoal is to predict a score to represent the overallquality of the summary. K = 1 and K > 1 indi-cate single-document and multi-document summa-rization respectively.

3 Our Methodology

The overall framework is illustrated in Figure 1.Our final evaluation score of a summary consists ofan averaged centrality-weighted relevance scoreand a self-referenced redundancy score. Bothscores are calculated on a semantic-level instead ofutilizing n-gram overlapping. The averaged rele-vance score is computed from the relevance scorebetween the summary and each document in thedocument set. The redundancy score is calculatedbased on the summary itself.

3.1 Centrality-weighted Relevance ScoreOur relevance score aims to estimate the informa-tiveness of the given summary. We first encodeeach document in the document set and the sum-mary into hidden representations. Then, for eachdocument, we select essential sentences by central-ity to build a pseudo reference. Next, we computea centrality-weighted relevance score between thesummary and each pseudo reference. Finally, weaverage all the relevance scores as the final rel-evance score of the summary. We use the k-thdocument dk and a summary x as an example toshow the workflow.

Encoding. Following SUPERT (Gao et al., 2020),we first split the document dk and the summary xinto sentences. Then, the pre-trained SBERT1 isemployed to encode the tokens of each sentenceinto token-level contextual hidden representations.We also apply max-pooling on all the tokens of asentence to obtain the sentence-level hidden repre-sentation. Following previous work, when utilizingthe token-level representations to compute the rel-evance and redundancy scores, we will filter outthe non-informative tokens such as stop-words toimprove the efficiency.

Building Pseudo Reference. We do not chooseall the document sentences of dk to evaluate therelevance of the summary. Because the whole doc-ument usually contains plenty of unimportant sen-tences which may introduce extra noise for the rel-evance evaluation. Thus, we select important docu-ment sentences to build a pseudo reference r for theevaluation. The sentence selection is based on thecentrality of each sentence, which is computed bythe unsupervised algorithm, PacSum (Zheng andLapata, 2019), using the sentence-level represen-tation. After obtaining the centrality scores of allsentences of the document, we choose the top-M2

sentences as the pseudo reference. Besides, wenormalize the centrality scores to [0, 1] and denotethe normalized centrality scores of the selected sen-

1bert-large-nli-stsb-mean-tokens2In experiments, we follow the default configuration of

SUPERT and set M as 12 for all the datasets.

407

tences as as = [as1, as2, ..., a

sM ] where asi ∈ [0, 1]

and the superscript s means sentence-level. Wedenote the pseudo reference building process asPacSumTopM.

Computing Relevance Score with One PseudoReference. Instead of only using token-level rep-resentations, we also leverage the sentence-levelrepresentations to provide multi-level information.The hybrid representations of the summary x andthe pseudo reference r are denoted as follows:

X = [wx1 , ...,w

xn, s

x1 , ..., s

xN ], (1)

Rk = [wr1, ...,w

rm, s

r1, ..., s

rM ], (2)

where n and N (m and M ) are the token numberand sentence number of the summary (pseudo ref-erence). w and s represent the token and sentencehidden representations respectively.

Besides the hybrid representations, we also in-troduce a centrality weighting scheme to weightthe tokens and sentences of the pseudo reference,which is different from previous work that eithertreats them equally or uses the surface statisticslike IDF as the weights. Based on the centralityscores of the selected pseudo reference sentencesi.e., as = [as1, a

s2, ..., a

sM ], we assign the weights

of the pseudo reference tokens as follows:

aw = [aw1 , aw2 , ..., a

wm], (3)

awj = asi:wj∈si , (4)

where ai:wj∈si indicates the token wj inherits thecentrality score from its sentence si. Since wehave already removed the non-informative tokensin the token-level representations of each sentence,the remaining tokens capture the key informationof the sentence and consequently it is reasonableto perform such a weight inheritance. Next, wecombine token weights aw and sentence weights as

to get the final normalized centrality-based weightsof the hybrid representations:

a = [aw1 , ..., awm, a

s1, ..., a

sM ], (5)

awj = awj /sum([aw; as]), (6)

asi = asi/sum([aw; as]), (7)

where “[·; ·]” represents concatenation.Based on the hybrid representations (i.e., X

and Rk) and the centrality-based weights of thepseudo reference tokens and sentences (i.e., a),we compute the relevance score between the sum-mary and the pseudo reference by a weighted

BERTScore (Zhang et al., 2020). For brevity, wedenote the j-th element of X as xj , the i-th elementof Rk as ri, and the i-th element of a as ai:

Recall =

∑i ai maxj Sim(ri,xj)∑

i ai, (8)

Precision =

∑j maxi Sim(ri,xj)

|X|, (9)

F1 =2 ∗Recall ∗ PrecisionRecall + Precision

, (10)

where “Sim” denotes the cosine similarity and |X|equals to n + N . Recall, Precision, and F1 arein the range of [-1, 1].

Besides the F1 version, we also propose an adap-tive Fβ version of relevance score as follows:

Fβ =(1 + β2) ∗Recall ∗ PrecisionRecall + β2 ∗ Precision

, (11)

β2 =

1, if ( |Rk|

|X| )1/γ ≤ 1

2, if ( |Rk||X| )

1/γ ≥ 2

( |Rk||X| )

1/γ , otherwise

, (12)

where |Rk| = m+M , |X| = n+N , and γ is a pos-itive integer hyper-parameter. In our experiments,γ is set as 2 after fine-tuning on the validationdataset and is fixed for all the testing datasets. Thephysical meaning of β is that the Recall score isconsidered β times as important as the Precisionscore. In summarization evaluation, the coverageof the key information is always the most importantquality indicator of the summary. Thus, we set thelower bound of β as 1. On the other hand, the met-ric should not only evaluate the key informationcoverage, containing less unimportant content inthe summary should also be considered. Therefore,we set the upper bound of β as

√2. As shown in

Eq.12, within the range of [1,√

2], β adaptivelychanges according to the ratio between |Rk| and|X|. The intuition comes from that a longer pseudoreference implies more key information needs tobe covered by the summary. Besides, a shortersummary can easily get high precision but coversvery limited important information in the pseudoreference. Thus, we give Recall a higher weightto punish such short summaries when the pseudoreference is long.

Final Averaged Relevance Score. After com-puting the centrality-weighted relevance score be-tween the summary and the pseudo reference ofeach source document, we employ the average as

408

the final relevance score of the summary:

scorerel = mean([F 1∗ , ..., F

k∗ , ..., F

K∗ ]), (13)

where * is 1 for the F1 variant and β for the Fβvariant. The superscript k indicates the F∗ scoreis computed with the k-th document. Note thatscorerel ∈ [−1, 1] and higher is better.

3.2 Self-referenced Redundancy ScoreIn this section, we introduce our self-referencedredundancy score. We engage the summary itselfas the reference to evaluate the degree of the se-mantic similarity between each summary token orsentence with the other tokens or sentences. Theaveraged semantic similarity degree is used as theredundancy score. The computation is based on aself-masked BERTScore as follows:

scorered =

∑i maxj:i 6=j Sim(xj ,xi)

|X|, (14)

where “j : i 6= j” means we do not consider thesimilarity between xi and itself, i.e, self-masked.Because of the symmetric property, the F1, preci-sion, and recall scores are equal with each other.This is also the reason that we use precision inEq.14 as the final redundancy score. Note thatscorered ∈ [−1, 1] and lower is better.

3.3 Final Evaluation ScoreAfter obtaining the relevance score and the redun-dancy score, we apply a linear combination to pro-duce the final evaluation score of the summarybased on the document set:

score =scorerel − λ ∗ scorered

1 + λ, (15)

where 0 < λ ≤ 1 is a hyper-parameter toscale the redundancy score and score ∈ [−1, 1].Higher score means better summary quality. Inour experiments, after fine-tuning on the vali-dation set, λ is set as 0.6 and is fixed for allthe testing datasets. We denote the variants ofour final method as Ours(Fβ)-PacSumTopM andOurs(F1)-PacSumTopM depending on whetherthe adaptive Fβ is employed.

4 Experiment Setup

4.1 DatasetsFor comprehensively investigating our summariza-tion evaluation methods, we test our methods onboth multi-document and single-document sum-marization datasets. We leverage TAC3 datasets

3https://tac.nist.gov/

Dataset |Topic| Document Summary|Set| Ave.S Ave.T |Systems| Ave.S Ave.T

Valid. TAC-2010 46 10 23.2 651.8 43 4.3 118.9

Test.

TAC-2011 44 10 20.1 560.5 50 4.3 120.9TAC-2009 44 10 24.9 705.8 55 4.1 117.6TAC-2008 48 10 23.3 660.0 58 4.2 119.6CNNDM 499 1 36.0 921.1 4 3.5 73.2

Table 1: Statistics of datasets. “Valid.” and “Test.” in-dicate the dataset is used for validation and testing, re-spectively. “|Topic|” is the number of topics. Undereach topic, a set of documents is given and summariesare from different systems associating with human-annotated quality scores. “|Set|” is the number of doc-uments in the document set. “Ave.S” and “Ave.T” rep-resent the averaged sentence number and token numberper document or summary. Note that the token numberis counted after the tokenization. “|Systems|” denotesthe number of summarization systems in the dataset.

for multi-document summarization evaluation test-ing. We choose TAC-2010 as the validation datasetand TAC-2008/TAC-2009/TAC-2011 as the testingdatasets. Following previous work, we only uti-lize the initial summaries in TAC datasets, i.e., thesummaries for the document set A. For the single-document summarization evaluation, we employCNNDM4 (Chaganty et al., 2018) as the testingdataset. The statistics of these datasets are shownin Table 1. Note that the hyper-parameters of ourmethods are fine-tuned on TAC-2010 and then fixedfor all the testing datasets.

For TAC datasets, we compute correlation coef-ficients between predicted scores of an evaluationmethod and the annotated Pyramid scores of sum-maries to measure the effectiveness of the method.Following Gao et al. (2020), a correlation is com-puted for each topic. Then, the averaged correlationfrom all the topics is engaged as the final correla-tion of the method with human ratings.

For CNNDM dataset, correlations are calculatedwith the human scores in three dimensions includ-ing Overall, Grammar, and Redundancy. Follow-ing Wu et al. (2020), the correlation is computedbetween predicted scores of the 499 × 4 = 1996(document, summary) pairs with corresponding hu-man ratings.

4.2 Baselines

In this section, we briefly introduce our baselines.We choose TF-IDF, JS (Louis and Nenkova,

2013), and REPEAR (Rioux et al., 2014) as tra-ditional reference-free baselines. All these tradi-tional baselines do not build pseudo references and

4https://bit.ly/price-of-debiasing

409

Method TAC-2011 TAC-2009 TAC-2008r ρ τ r ρ τ r ρ τ

TF-IDF 0.313 0.294 0.209 0.372 0.382 0.279 0.375 0.341 0.243JS 0.377 0.333 0.240 0.376 0.381 0.279 0.385 0.338 0.242REAPER 0.377 0.334 0.237 0.358 0.357 0.256 0.287 0.261 0.187Ours(F1)-All 0.495 0.451 0.329 0.478 0.476 0.353 0.466 0.426 0.310Ours(Fβ)-All 0.498 0.455 0.332 0.480 0.471 0.348 0.462 0.423 0.307ROUGE-1-PacSumTopM 0.436 0.377 0.274 0.418 0.406 0.301 0.397 0.348 0.252ROUGE-2-PacSumTopM 0.429 0.388 0.287 0.380 0.419 0.314 0.410 0.355 0.259ROUGE-L-PacSumTopM 0.436 0.370 0.272 0.427 0.415 0.306 0.385 0.336 0.245MoverScore-PacSumTopM 0.521 0.475 0.351 0.483 0.485 0.362 0.479 0.440 0.323S+WMS-PacSumTopM 0.291 0.292 0.211 0.350 0.358 0.264 0.364 0.358 0.260C-ELMO-PacSumTopM 0.386 0.302 0.217 0.317 0.235 0.167 0.210 0.162 0.114C-SBERT-PacSumTopM 0.332 0.293 0.207 0.314 0.277 0.197 0.183 0.196 0.143SUPERT-PacSumTopM 0.511 0.481 0.357 0.486 0.494 0.368 0.493 0.457 0.334SUPERT-IDF-PacSumTopM 0.507 0.476 0.353 0.485 0.492 0.367 0.489 0.450 0.328Ours(F1)-PacSumTopM 0.531 0.493 0.365 0.502 0.506 0.381 0.495 0.461 0.337Ours(Fβ)-PacSumTopM 0.541 0.505 0.374 0.507 0.508 0.380 0.500 0.465 0.339

Table 2: Main results on multi-document summarization datasets. Pearson’s r, Spearman’s ρ, and Kendall’s τ withhuman scores are reported. The best results are bold and the second-best results are underlined.

Method Overall Grammar Redundancyr ρ τ r ρ τ r ρ τ

TF-IDF 0.264 0.249 0.187 0.186 0.170 0.127 0.281 0.253 0.187JS 0.265 0.232 0.174 0.210 0.180 0.136 0.317 0.278 0.208REAPER 0.036 0.032 0.024 0.004 -0.006 -0.005 -0.020 -0.031 -0.024LS Score (Wu et al., 2020) − 0.334 − − 0.266 − − 0.288 −Ours(F1)-All 0.390 0.370 0.281 0.306 0.306 0.232 0.413 0.381 0.287Ours(Fβ)-All 0.361 0.337 0.255 0.273 0.270 0.204 0.395 0.356 0.268ROUGE-1-PacSumTopM 0.224 0.215 0.159 0.126 0.114 0.084 0.289 0.254 0.186ROUGE-2-PacSumTopM 0.347 0.335 0.253 0.254 0.240 0.181 0.398 0.369 0.274ROUGE-L-PacSumTopM 0.235 0.224 0.166 0.135 0.122 0.090 0.300 0.264 0.193MoverScore-PacSumTopM 0.373 0.341 0.259 0.264 0.240 0.181 0.411 0.359 0.267S+WMS-PacSumTopM 0.324 0.353 0.267 0.240 0.256 0.193 0.360 0.385 0.286C-ELMO-PacSumTopM 0.355 0.297 0.223 0.232 0.201 0.151 0.425 0.354 0.262C-SBERT-PacSumTopM 0.405 0.378 0.286 0.295 0.299 0.225 0.415 0.373 0.279SUPERT-PacSumTopM 0.384 0.374 0.284 0.318 0.317 0.240 0.381 0.369 0.277SUPERT-IDF-PacSumTopM 0.382 0.373 0.283 0.316 0.314 0.238 0.377 0.365 0.274Ours(F1)-PacSumTopM 0.416 0.404 0.308 0.341 0.341 0.259 0.428 0.408 0.308Ours(Fβ)-PacSumTopM 0.400 0.381 0.290 0.314 0.311 0.235 0.427 0.395 0.298

Table 3: Main results on single-document summarization dataset (CNNDM). Pearson’s r, Spearman’s ρ, andKendall’s τ with human scores are reported. The best results are bold and the second-best results are underlined.

directly utilize the full content of the documents.For fairness, we also show the performance of ourmethods without building pseudo reference. Wedenote them as Ours(F1)-All and Ours(Fβ)-Allsince they use the whole document as a reference.

We also extend several popular reference-based methods as baselines. We adapt ROUGE-1/2/L (Lin, 2004), MoverScore (Zhao et al., 2019),and S+WMS (Clark et al., 2019) into the reference-free scenario via building the pseudo reference withthe PacSumTopM method. We add the suffix “-PacSumTopM” to these baseline names to indi-cate the pseudo reference building process.

Besides, the SOTA reference-free summary eval-uation metrics are also selected as our strong base-lines, including C-ELMO/C-SBERT (Sun andNenkova, 2019), SUPERT/SUPERT-IDF (Gaoet al., 2020), and LS Score (Wu et al., 2020). C-ELMO (C-SBERT) encodes the document and the

summary using the pre-trained ELMO (SBERT)and then computes their cosine similarity. SUPERT-IDF is an extension of SUPERT, which utilizes theinverse document frequency (IDF) as the impor-tance weight of each token. For fair comparisons,we also apply the same pseudo reference build-ing process i.e., PacSumTopM, to C-ELMO/C-SBERT/SUPERT/SUPERT-IDF and add the suffix“-PacSumTopM” to the their names.

5 Results and Analysis

5.1 Main Results

The main experimental results on multi-documentsummarization datasets are shown in Table 2.We find that our F1 version (i.e., Ours(F1)-PacSumTopM) already consistently outperformsall the baselines, which indicates the effectivenessof our centrality-weighted relevance score and ourself-referenced redundancy score. The results also

410

1 2 3 5 7 9 allDifferent |Set|

−0.02

−0.01

0.00

0.01

0.02O

urs(F β

)'s ρ

- O

urs(F 1

)'s ρ

0.004

0.011

0.007 0.0

10.0

08 0.009 0.0

12

4 8 12 16 20 28 36 44 allDifferent |Systems|

−0.02

−0.01

0.00

0.01

0.02

Our

s(F β

)'s ρ

- O

urs(F 1

)'s ρ

-0.02

4-0

.007

0.007

0.014

0.018

0.018

0.014

0.014

0.012

Figure 2: The gap of Spearman’s ρ between Ours(Fβ)and Ours(F1) on TAC-2011 for different |Set| and|Systems|. Positive gaps mean our Fβ can improve theperformance while negative gaps indicate our Fβ de-grades the performance. When changing one of them,the other is fixed. “all” means the full size is applied,i.e., 10 for |Set| and 50 for |Systems|.

demonstrate that our Fβ version can further im-prove the performance of multi-document sum-marization evaluation. By comparing Ours(Fβ)-PacSumTopM and Ours(Fβ)-All, we see that thepseudo reference building process can significantlyimprove the performance. This is also the reasonwhy we apply the same pseudo reference buildingprocess into SOTA baselines for fair comparisons.In the remaining part of this paper, we omit thesuffix “-PacSumTopM” for simplicity when wemention a method.

We also test our methods on the single-documentsummarization dataset without further fine-tuningthe hyper-parameters. The main results are dis-played in Table 3. We note that our F1 version stilloutperforms all the baselines, which manifests thehigh generalization ability of our F1-based method.One interesting finding is that the performance sig-nificantly drops after incorporating the Fβ score.

To study the reason for the performance degrada-tion on CNNDM after incorporating Fβ , we com-pare CNNDM and TAC datasets first. From Table 1,we note the main differences between them are thesize of the document set for each topic (i.e., |Set|)and the number of the summarization systems (i.e.,|Systems|). CNNDM has much smaller |Set| and|Systems|. We use the TAC-2011 dataset as an ex-ample to investigate whether our Fβ is unsuitablefor smaller |Set| and |Systems|. We change |Set|and |Systems| respectively and report the gap ofSpearman’s ρ between Ours(Fβ) and Ours(F1) inFigure 2. From the results, we observe that our Fβ

Method TAC CNNDM2011 2009 2008 Overall Grammar Redundancy

Ours(F1) 0.493 0.506 0.461 0.404 0.341 0.408Ours(Fβ) 0.505 0.508 0.465 0.381 0.311 0.395MoverScore 0.475 0.485 0.440 0.341 0.240 0.359+CentralityW. 0.472 0.467 0.431 0.350 0.257 0.364+Redundancy 0.237 0.202 0.221 0.448 0.326 0.546+Both 0.261 0.220 0.241 0.455 0.341 0.545

Table 4: Spearman’s ρ of incorporating the central-ity weighting and redundancy score into MoverScorebased framework. “+Both” means these two featuresare simultaneously applied.

can consistently improve the performance for dif-ferent |Set|. For the single-document summariza-tion setting, i.e., |Set|=1, it still obtains a positivegap. Nevertheless, when the |Systems| is smallsuch as 4, applying our Fβ leads to a dramatic per-formance dropping. From Table 1, we also seethat CNNDM and TAC-2011 have different sum-mary lengths (73.2 for CNNDM and 120.9 for TAC-2011). However, when we limit the |Systems| ofTAC-2011 to smaller numbers, the average lengthof generated summaries is still around 120, whichindicates the performance degeneration is indeedfrom the change of system numbers. Therefore, wesuggest using Ours(Fβ) when |Systems| is largelike 12 and employing Ours(F1) when |Systems|is small like 4.

5.2 Ablation Study

For better understanding the contributions of ourproposed components, we conduct ablation stud-ies on the best-performed method on each dataset,i.e., Ours(Fβ) for the multi-document summariza-tion datasets and Ours(F1) for the single-documentsummarization dataset. We display results of therank-based Spearman’s ρ in Figure 3.

As shown in the figure, after removing one ofthe three components (i.e., the centrality weight-ing, the hybrid representation, and the redundancyscore), the performance of our methods becomeworse in most cases. This finding demonstratesthe effectiveness of our proposed components. Be-sides, we also note that removing the redundancyscore significantly degrades the performance onthe redundancy evaluation on CNNDM, which in-dicates our redundancy score effectively capturesthe redundancy degree of the summaries.

5.3 Apply Centrality Weighting andRedundancy Score into MoverScore

Besides basing on BERTScore, we also studywhether our key features i.e., the centrality weight-ing and redundancy score, can work well in a

411

TAC2011 TAC2009 TAC20080.30

0.35

0.40

0.45

0.50

0.55

0.60Sp

earm

an's ρ 0.5

05

0.508

0.4650.5

01

0.51

0.46

0.502

0.509

0.4680.4

98

0.496

0.466

Ablation Study on TAC Datasets

Ours(Fβ)

-CentralityW.-HybridR.-Redundancy

Overall Grammar Redundancy0.20

0.25

0.30

0.35

0.40

0.45

0.50

Spea

rman

's ρ 0.4

04

0.341

0.408

0.399

0.332

0.406

0.4

0.337

0.406

0.382

0.33

0.374

Ablation Study on CNNDM Dataset

Ours(F1)-CentralityW.-HybridR.-Redundancy

Figure 3: Ablation studies for Ours(Fβ) on TACdatasets and Ours(F1) on CNNDM. “-CentralityW.”means that we remove the centrality weighting whencomputing relevance scores. “-HybridR.” representswe only utilize the token-level representations whencalculating relevance and redundancy scores. “-Redundancy” indicates we omit the redundancy score.

MoverScore based framework (i.e., the relevanceand redundancy scores are computed using Mover-Score). Note that our Fβ is not applicable to Mover-Score since it is not an F -measure. The results arelisted in Table 4. We find that these two featuressignificantly improve the performance of the orig-inal MoverScore on single-document summariza-tion evaluation while degrading the performancedramatically on multi-document summarizationevaluation. On CNNDM, the enhanced Mover-Score even outperforms Ours(F1) on the “Overall”and “Redundancy” aspects, which indicates Mover-Score is a promising basis for our proposed new fea-tures. We leave solving the performance droppingof the enhanced MoverScore on multi-documentsetting as future work.

5.4 Robustness Analysis

We investigate the robustness of our method onthe following factors and report the experimentalresults on the validation dataset (i.e., TAC-2010) inFigure 4: (1) the hyper-parameter λ for scaling theredundancy score; (2) the hyper-parameter γ in Fβ ;(3) the number of selected sentences for pseudo ref-erence i.e., M ; (4) different pre-trained contextualencoding models including BERT-base5, BERT-large6, RoBERTa-base7, and RoBERTa-large8.

5bert-base-nli-stsb-mean-tokens6bert-large-nli-stsb-mean-tokens7roberta-base-nli-stsb-mean-tokens8roberta-large-nli-stsb-mean-tokens

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Different λ

0.550

0.575

0.600

0.625

Cor

rela

tions

rρ

1 2 3 4 5 6Different γ

0.550

0.575

0.600

0.625

Cor

rela

tions

rρ

3 6 9 12 15 18 21Different M

0.550

0.575

0.600

0.625

Cor

rela

tions

rρ

BERT-baseBERT-large

RoBERTa-baseRoBERTa-large

0.550

0.575

0.600

0.625

Cor

rela

tions

rρ

Figure 4: The performance of Ours(Fβ) on TAC-2010under different λ, γ, M , and encoding models. Whenwe change one of them, the others are fixed. The Pear-son’s r and Spearman’s ρ are reported.

Since both Spearman’s ρ and Kendall’s τare rank-based correlation coefficients, we omitKendall’s τ for simplicity. From this figure, weobserve that the performance of our method is rel-atively stable for different λ and γ. We also findthat a small M leads to lower correlations becausemuch important information may be abandonedwhen building the pseudo references. But a largeM will also degenerate the correlations since morenoises are introduced. Thus, a moderateM is better.As for encoding models, we note that large encod-ing models obtain better performance than baseencoding models. However, large models needmore computation resources and time to encodethe input text. Note that for our final method, weonly fine-tune λ and γ on the TAC-2010 and setthem as 0.6 and 2. As for M and encoding mod-els, following the configuration of SUPERT (Gaoet al., 2020), we directly set M as 12 and employthe BERT-large as the encoding model. All thesefactors are fixed for all testing datasets.

5.5 Performance on Bad/Good Summaries

In this section, we evaluate the ability of ourmethod to distinguish bad and good summaries.The bad and good summaries are selected by hu-man ratings. We use TAC-2011 as an example andchoose SUPERT as a strong baseline. The corre-sponding distributions of the reversed rank for badand good summaries are illustrated in Figure 5. Asmaller (larger) reversed rank represents the sum-mary is assigned with a lower (higher) score. Fromthe figure, we find that compared with SUPERT,Our(Fβ) has a better ability to assign bad sum-

412

SUPERT Ours(Fβ)

0

10

20

30

40

Rev

erse

d R

ank

Reversed Rank of Bad Summaries

SUPERT Ours(Fβ)0

10

20

30

40

Rev

erse

d R

ank

Reversed Rank of Good Summaries

Figure 5: Distributions of the reversed rank from SU-PERT and Ours(Fβ) for bad and good summaries onTAC-2011. The bar in the middle indicates the median.

maries lower scores and good summaries higherscores, which demonstrates the effectiveness ofour method again. Moreover, we also note thatboth SUPERT and Ours(Fβ) are good at giving badsummaries lower scores while having difficulty inassigning good summaries higher scores. We leavesolving this problem as another future work underthe reference-free setting.

6 Related Work

Reference-based Evaluation Metrics mainlymeasure the relevance between the human-annotated references and the system-generatedtext, which are widely adopted in text summa-rization (Lin, 2004; Zhao et al., 2019), machinetranslation (Papineni et al., 2002; Zhang et al.,2020), and dialogue systems (Papineni et al., 2002;Gao et al., 2021; Xiang et al., 2021). For exam-ple, ROUGE (Lin, 2004) evaluates the token se-quence overlapping. BERTScore (Zhang et al.,2020), S+WMS (Clark et al., 2019), and Mover-Score (Zhao et al., 2019) measure the semanticsimilarity between the references and the summaryvia a greedy or optimized minimum Earth Mover’sDistance.

Reference-free Evaluation Metrics have beendeveloped to avoid the dependency on human-annotated references, which obtain more and moreattention in recent years (Bohm et al., 2019; Gaoet al., 2020; Wu et al., 2020; Chan et al., 2021).Some of them need to train a scorer (Peyrard andGurevych, 2018; Xenouleas et al., 2019; Scialomet al., 2019; Bohm et al., 2019). For example,LS Score (Wu et al., 2020) designs a metric which

combines a linguistic quality scorer trained fromthe built positive and negative summaries, and arelevance scorer based on cosine similarity. Theothers do not require training (Louis and Nenkova,2013; Rioux et al., 2014; Peyrard, 2019; Sun andNenkova, 2019). For instance, SUPERT (Gao et al.,2020) builds the pseudo references from the sourcedocument first and then engages BERTScore tocompute the relevance score between the pseudoreference and the summary.

7 Conclusion

In this paper, we propose a novel training-freeand reference-free summarization evaluation met-ric consisting of a relevance score and a redun-dancy score. Experiments on multi-document andsingle-document summarization settings show theeffectiveness of our methods. One promising fu-ture direction is to solve the performance droppingissue after applying our key features into Mover-Score and the other is to tackle the problem thatcurrent metrics struggle to assign higher scores forgood summaries.

Acknowledgements

The work described in this paper was partiallysupported by the Research Grants Council of theHong Kong Special Administrative Region, China(CUHK 2410021, Research Impact Fund (RIF),R5034-18).

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural machine translation by jointlylearning to align and translate. In 3rd Inter-national Conference on Learning Representations,ICLR 2015, San Diego, CA, USA, May 7-9, 2015,Conference Track Proceedings.

Florian Bohm, Yang Gao, Christian M. Meyer, OriShapira, Ido Dagan, and Iryna Gurevych. 2019. Bet-ter rewards yield better summaries: Learning to sum-marise without references. In Proceedings of the2019 Conference on Empirical Methods in Natu-ral Language Processing and the 9th InternationalJoint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 3110–3120, Hong Kong,China. Association for Computational Linguistics.

Arun Chaganty, Stephen Mussmann, and Percy Liang.2018. The price of debiasing automatic metrics innatural language evalaution. In Proceedings of the56th Annual Meeting of the Association for Com-putational Linguistics (Volume 1: Long Papers),

413

pages 643–653, Melbourne, Australia. Associationfor Computational Linguistics.

Hou Pong Chan, Wang Chen, and Irwin King. 2020.A unified dual-view model for review summariza-tion and sentiment classification with inconsistencyloss. In Proceedings of the 43rd International ACMSIGIR conference on research and development inInformation Retrieval, SIGIR 2020, Virtual Event,China, July 25-30, 2020, pages 1191–1200. ACM.

Zhangming Chan, Lemao Liu, Juntao Li, HaisongZhang, Dongyan Zhao, Shuming Shi, and Rui Yan.2021. Enhancing the open-domain dialogue evalua-tion in latent space. In Proceedings of the 59th An-nual Meeting of the Association for ComputationalLinguistics: Findings.

Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith.2019. Sentence mover’s similarity: Automatic eval-uation for multi-sentence texts. In Proceedings ofthe 57th Conference of the Association for Compu-tational Linguistics, ACL 2019, Florence, Italy, July28- August 2, 2019, Volume 1: Long Papers, pages2748–2760. Association for Computational Linguis-tics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 4171–4186, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Jun Gao, Wei Bi, Ruifeng Xu, and Shuming Shi. 2021.Ream#: An enhancement approach to reference-based evaluation metrics for open-domain dialoggeneration. In Proceedings of the 59th Annual Meet-ing of the Association for Computational Linguistics:Findings.

Yang Gao, Wei Zhao, and Steffen Eger. 2020. SU-PERT: towards new frontiers in unsupervised evalu-ation metrics for multi-document summarization. InProceedings of the 58th Annual Meeting of the As-sociation for Computational Linguistics, ACL 2020,Online, July 5-10, 2020, pages 1347–1354. Associa-tion for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for auto-matic evaluation of summaries. In Text Summariza-tion Branches Out, pages 74–81, Barcelona, Spain.Association for Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,Luke Zettlemoyer, and Veselin Stoyanov. 2019.Roberta: A robustly optimized BERT pretraining ap-proach. CoRR, abs/1907.11692.

Annie Louis and Ani Nenkova. 2013. Automaticallyassessing machine summary content without a goldstandard. Comput. Linguistics, 39(2):267–300.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In Proceedings ofthe 40th Annual Meeting of the Association for Com-putational Linguistics, pages 311–318, Philadelphia,Pennsylvania, USA. Association for ComputationalLinguistics.

Maxime Peyrard. 2019. A simple theoretical model ofimportance for summarization. In Proceedings ofthe 57th Annual Meeting of the Association for Com-putational Linguistics, pages 1059–1073, Florence,Italy. Association for Computational Linguistics.

Maxime Peyrard and Iryna Gurevych. 2018. Objec-tive function learning to match human judgementsfor optimization-based summarization. In Proceed-ings of the 2018 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018,Volume 2 (Short Papers), pages 654–660. Associa-tion for Computational Linguistics.

Cody Rioux, Sadid A. Hasan, and Yllias Chali. 2014.Fear the REAPER: A system for automatic multi-document summarization with reinforcement learn-ing. In Proceedings of the 2014 Conference onEmpirical Methods in Natural Language Processing(EMNLP), pages 681–690, Doha, Qatar. Associationfor Computational Linguistics.

Thomas Scialom, Sylvain Lamprier, Benjamin Pi-wowarski, and Jacopo Staiano. 2019. Answersunite! unsupervised metrics for reinforced summa-rization models. In Proceedings of the 2019 Con-ference on Empirical Methods in Natural LanguageProcessing and the 9th International Joint Confer-ence on Natural Language Processing (EMNLP-IJCNLP), pages 3246–3256, Hong Kong, China. As-sociation for Computational Linguistics.

Abigail See, Peter J. Liu, and Christopher D. Manning.2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th An-nual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Computa-tional Linguistics.

Simeng Sun and Ani Nenkova. 2019. The feasibilityof embedding based automatic evaluation for sin-gle document summarization. In Proceedings ofthe 2019 Conference on Empirical Methods in Nat-ural Language Processing and the 9th InternationalJoint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 1216–1221, Hong Kong,China. Association for Computational Linguistics.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.Sequence to sequence learning with neural networks.In Advances in Neural Information Processing Sys-tems 27: Annual Conference on Neural Informa-tion Processing Systems 2014, December 8-13 2014,Montreal, Quebec, Canada, pages 3104–3112.

414

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Advances in Neural Information Pro-cessing Systems 30: Annual Conference on NeuralInformation Processing Systems 2017, 4-9 Decem-ber 2017, Long Beach, CA, USA, pages 5998–6008.

Hanlu Wu, Tengfei Ma, Lingfei Wu, TariroManyumwa, and Shouling Ji. 2020. Unsuper-vised reference-free summary quality evaluation viacontrastive learning. CoRR, abs/2010.01781.

Stratos Xenouleas, Prodromos Malakasiotis, MariannaApidianaki, and Ion Androutsopoulos. 2019. SUM-QE: a BERT-based summary quality estimationmodel. In Proceedings of the 2019 Conference onEmpirical Methods in Natural Language Processingand the 9th International Joint Conference on Natu-ral Language Processing (EMNLP-IJCNLP), pages6005–6011, Hong Kong, China. Association forComputational Linguistics.

Jiannan Xiang, Yahui Liu, Deng Cai, Huayang Li, DefuLian, and Lemao Liu. 2021. Assessing dialogue sys-tems with distribution distances. In Proceedings ofthe 59th Annual Meeting of the Association for Com-putational Linguistics: Findings.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q.Weinberger, and Yoav Artzi. 2020. Bertscore: Eval-uating text generation with BERT. In 8th Inter-national Conference on Learning Representations,ICLR 2020, Addis Ababa, Ethiopia, April 26-30,2020. OpenReview.net.

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris-tian M. Meyer, and Steffen Eger. 2019. MoverScore:Text generation evaluating with contextualized em-beddings and earth mover distance. In Proceedingsof the 2019 Conference on Empirical Methods inNatural Language Processing and the 9th Interna-tional Joint Conference on Natural Language Pro-cessing (EMNLP-IJCNLP), pages 563–578, HongKong, China. Association for Computational Lin-guistics.

Hao Zheng and Mirella Lapata. 2019. Sentence cen-trality revisited for unsupervised summarization. InProceedings of the 57th Conference of the Asso-ciation for Computational Linguistics, ACL 2019,Florence, Italy, July 28- August 2, 2019, Volume1: Long Papers, pages 6236–6247. Association forComputational Linguistics.