Masters Thesis Defense “ResQu: A Framework for Automatic Evaluation of Knowledge-Driven Automatic Summarization” By Nishita Jaykumar

Committee:  Drs. Amit Sheth, Advisor, TK Prasad, Thomas Rindflesch, and Delroy Cameron

ABSTRACT:

Automatic generation of summaries that capture the salient aspects of a search result set (i.e., automatic summarization) has become an important task in biomedical research. Automatic summarization offers an avenue for overcoming the information overload prevalent in large online digital libraries. However, across many of the knowledge-driven approaches for automatic summarization it is not always clear which features have the most impact or influence on the quality of a summary. Instead, there has been considerable focus on utilizing schema knowledge to facilitate browsing and exploration of generated summaries a posteriori. Such informative features should not be ignored, since they could be utilized to help optimize the models that generate these semantic summaries in the first place.

In this research, we adopt a leave-one-out approach to assess the impact of various features on the quality of automatically generated summaries that contain structured background knowledge. We first create the gold standard summaries, using information-theoretic methods, by extraction and validation; the semantic summaries are then transformed into an equivalent textual format. Finally, various similarity metrics, such as cosine similarity, Euclidean distance, and Jensen-Shannon divergence are computed under different feature combinations to assess summary quality against a textual gold standard. We report on the relative importance of the various features used to automatically generate the semantic summaries in a biomedical application. Our evaluation suggests that the proposed approach is an effective automatic evaluation method for assessing feature importance in automatically generated semantic summaries.