|
ProjectsDUC 2005DESCRIPTION:The focus of DUC 2005 was the evaluation of automatic summarization system.CELCT’S ROLE:Celct, in collaboration with Columbia University, tested a new manual method for summarisation evaluation called “Pyramid Method”, which was developed in an attempt to address a key problem in summarisation: namely, the fact that different humans choose different content when writing summaries.The pyramid method addresses the problem by using multiple human summaries to create a gold-standard and by exploiting the frequency of information in the human summaries in order to assign importance to different semantic units (SCUs). The SCUs are then organized in a ‘pyramid’, which has at the top the most important information, i.e. that information which is present in all manual summaries. The SCUs at the top are assigned the highest ‘weight’. Through an annotation interfaces, the automatic sommarie themselves are then divided into SCUs which are matched to the ones in the pyramid. In this way, they get scored, proportionally to the weight of the pyramid SCUs. LINK:http://duc.nist.gov/ |
|
