Structured Literature Image Finder

Amr Ahmed1,6, Andrew Arnold1, Luıs Pedro Coelho2,3,4, Joshua Kangas2,3,4,Abdul-Saboor Sheikh3, Eric Xing1,5,6, William Cohen1, Robert F. Murphy1,2,3,4,5,7∗

Abstract

The Structured Literature Image Finder tacklestwo related problems posed by the vastness of thebiomedical literature: how to make it more accessi-ble to scientists in the field and how to take advan-tage of the primary data often locked inside papers.Towards this goal, the slif project developed an in-novative combination of text and image processingmethods.

Images from papers are classified according totheir type (fluorescence microscopy image, gel, . . . )and their caption is parsed for biologically relevantentities such as protein names. This enables tar-geted queries for primary data (a feature that auser study revealed to be highly valued by scien-tists). Finally, using a novel extension to latenttopic models, we model papers at multiple levelsand provide the ability to find figures similar to aquery and refine these findings with interactive rel-evance feedback.

Slif is most advanced in processing fluorescentmicroscopy images which are further categorised ac-cording to the depicted subcellular localization pat-tern.

The results of slif are made available to thecommunity through a user friendly web interface(http://slif.cbi.cmu.edu).

1 Introduction

Biomedical research worldwide results in a veryhigh volume of information in the form of publica-tions. Biologists are faced with the daunting task ofquerying and searching these publications to keep

1Machine Learning Department, 5Department of Bi-ological Sciences, and 7Department of Biomedical Engi-neering, Carnegie Mellon University; 2Joint Carnegie Mel-lon University–University of Pittsburgh Ph.D. Program inComputational Biology; 3Center for Bioimage Informatics,Carnegie Mellon University; 4Lane Center for Computa-tional Biology, Carnegie Mellon University; 6Language Tech-nologies Institute, Carnegie Mellon University. ∗to whomcorrespondence should be addressed.

Figure 1: Screenshot of the slif search engineshowing the results of a search.

up with recent developments and to answer specificquestions.

In the biomedical literature, data is most of-ten presented in the form of images. A fluores-cent micrograph image (fmi) or a gel is sometimesthe key to a whole paper. Compared to figuresin other scientific disciplines, biomedical figures areoften a stand alone source of information that sum-marizes the findings of the research under consid-eration. A random sampling of such figures inthe publicly available PubMed Central databasereveals that in some, if not most of the cases, abiomedical figure can provide as much informationas a normal abstract. The information-rich, highly-evolving knowledge source of the biomedical liter-ature calls for automated systems that would helpbiologists find information quickly and satisfacto-rily. These systems should provide biologists witha structured way of browsing the otherwise unstruc-tured knowledge in a way that would inspire themto ask questions that they never thought of before,or reach a piece of information that they would havenever considered pertinent to start with.

Relevant to this goal, we developed the first sys-tem for automated information extraction from im-ages in biological journal articles (the “SubcellularLocation Image Finder,” or slif, first described in2001 [9]). Since then, we have made major enhance-ments and additions to the slif system [3, 8, 7],

1

and now report not only additional enhancementsbut the broadening of its reach beyond fluorescentmicroscopy images. Reflecting this, we have nowrechristened slif as the “Structured Literature Im-age Finder.”

Slif reached the final stage in the Elsevier GrandChallenge (4 out of 70), a contest sponsored by El-sevier to “improve the way scientific information iscommunicated and used.”

2 Overview

Slif provides both a pipeline for extracting struc-tured information from papers (illustrated in Fig-ure 2) and a web-accessible searchable database ofthe processed information (depicted in Figure 1).

The pipeline begins by finding all figure-captionspairs. Each caption is then processed to identifybiological entities (e.g., names of proteins and celllines) and these are linked to external databases.Pointers from the caption to the image are identi-fied, and the caption is broken into “scopes” so thatterms can be linked to specific parts of the figure.

The image processing module begins by split-ting each figure into its constituent panels, andthen identifying the type of image contained in eachpanel. The patterns in fmis are described using aset of biologically relevant image features [9], andthe subcellular location depicted in each image isrecognized.

The last step in the pipeline is to discover latenttopics that are present in the collection of papers.These topics serve as the basis for visualization andsemantic representation. Each topic consists of atriplet of distributions over words, image features,and proteins (possibly extended to include gene on-tology terms and subcellular locations). Each figurein turn is represented as a distribution over thesetopics, and this distribution reflects the themes ad-dressed in the figure. This representation servesas the basis for various tasks like image-based re-trieval, text-based retrieval and multimodal-basedretrieval. Moreover, these discovered topics providean overview of the information content of the col-lection, and structurally guide its exploration.

All results of processing are stored in a database,which is accessible via a web interface or SOAPqueries. The results of queries always include linksback to the panel, figure, caption and the full paper.Users can query the database for various informa-tion appearing in captions or images, including spe-cific words, protein names, panel types, patterns infigures, or any combination of the above. Using the

latent topic representation, we built an innovativeinterface that allows browsing through figures bytheir inferred topics and jumping to related figuresfrom any currently viewed figure.

3 Caption Processing

In order to identify the protein depicted in an im-age, we look for protein names in the caption.The structure of captions can be complex (espe-cially for multipanel figures). We therefore imple-mented a system for processing captions with threegoals: identifying the “image pointers” (e.g., “(A)”or“(red)”) in the caption that refer to specific panellabels or panel colors in the figure [3], dividing thecaption into fragments that refer to an individualpanel, color, or the entire figure, and recognizingprotein and cell types.

Errors in optical character recognition can leadto low accuracy in matching image pointers to panellabels. Using regularities in the arrangement of thelabels (e.g., if the letters A through D are found asimage pointers and the panel labels are recognizedas A,B,G and D, then the G should be corrected toa C) corrects some of the errors [7]. Using a testset from PNAS, the precision of the final matchingprocess was found to be 83% and the recall to be74% [5].

Recognition of named entities (such as proteinand cell types) in free text is a difficult task thatmay be even more difficult in condensed text suchas captions. We have implemented two schemes forrecognizing protein names. The first (which is alsoused for cell type recognition) uses prefix and suffixfeatures along with immediate context to identifycandidate protein names. This approach has a lowprecision but an excellent recall (which is useful toenable database searches on abbreviations or syn-onyms that might not be present in structured pro-tein databases). The second approach [6] uses a dic-tionary of names extracted from protein databasesin combination with soft match learning methodsto obtain a recall and precision above 70%. Theoccurrences of the names found in the captions arestored as being associated either with a panel or afigure, depending on the scope in which the proteinname was found. The system also assigns subcellu-lar locations to proteins using lookup of GO termsin the Uniprot database, making it possible to findimages depicting particular subcellular patterns.

Finally, the task of simply segmenting a paperand extracting the caption, even without namedentity recognition or panel scoping, has proven very

2

Figure 2: Slif Pipeline. This figure shows the general pipeline through papers are processed.

useful to our users, allowing easy search of free textwhich can be limited to the captions, and thereforethe figures, of a paper.

4 Image Processing

Since, in most cases, figures are composed of mul-tiple panels, the first step in our image process-ing pipeline is to divide the figures into panels.We employ a figure-splitting algorithm that recur-sively finds constant-intensity boundary regions inbetween panels, a method which we have previ-ously shown can effectively split figures with com-plex panel layouts [9].

Slif was originally designed to process only fmipanels, and subsequent systems created by othershave included classifiers to distinguish other figuretypes [11, 4]. We have now expanded the classifi-cation to other panel types: (1) fmi, (2) gel, (3)graph or illustration, (4) photograph, (5) X-ray, or(6) light microscopy. Using active learning [10], weselected ca. 700 panels to label.

Given its importance to the working scientists,we focused on the gel class. Currently, the systemproceeds through 3 classification levels: the firstlevel, classifies the image into fmi or non-fmi us-ing image based features (as previously reported);the second level, uses textual features to identifygels with high-precision (91%, and moderate recall:66%); finally, if neither classifier has fired, a generalpurpose support vector machine classifier, operat-ing on image-based features does the final classifi-cation (accuracy: 61%).

Perhaps the most important task that slif sup-ports is the classification of fmi panels based on thedepicted subcellular localization. To provide train-ing data for pattern classifiers, we hand-labeled aset of images into four different subcellular locationclasses: (1) nuclear, (2) cytoplasmic, (3) punctate,and (4) other, again selected through active learn-

ing.We computed previously described features to

represent the image patterns. If the scale is in-ferred from the image, then we normalize this fea-ture value to square microns. Otherwise, we as-sume a default scale of 1µm/pixel. On the 3 mainclasses (Nuclear, Cytoplasmic, and Punctate), weobtained 75% accuracy (as before, reported accu-racies are estimated using 10 fold cross-validationand the classifier used was libsvm based). On thefour classes, we obtained 61% accuracy.

5 Topic Discovery

The goal of the topic discovery phase is to en-able the user to structurally browse the otherwiseunstructured collection. This problem is reminis-cent of the actively evolving field of multimediainformation management and retrieval. However,structurally-annotated biomedical figures pose a setof new challenges due to the the hierarchical struc-ture of the domain (panels contained within fig-ures) which results in scoped and global annotationschemes, and the presence of various image annota-tions (free form text, protein mentions,etc.) in thecaption with different frequency profiles.

Our model, the structured correspondence topicmodel [1], addresses the aforementioned challengesby extending the correspondence latent Dirichletallocation model that was successfully employedfor modeling annotated images [2]. The input tothe topic modeling system is the panel-segmented,structurally and multimodally annotated biomedi-cal figures. The goal of our approach is to discover aset of latent themes in the collection. These themesare called topics and serve as the basis for visual-ization and semantic representation. Each biomed-ical figure, panel, and protein entity is then rep-resented as a distribution over these latent topics.This unified representation enables comparing fig-

3

ures with radically different number of panels andserves as the basis for various tasks like image-basedretrieval, text-based image retrieval, multimodal-based image retrieval and image annotation. Wecompared our model to various baselines with fa-vorable results [1].

Furthermore, the latent topic representation fa-cilitates the implementation of features such asfinding similar objects to an example that the userhas found as interesting (this can be done at anylevel: panel, figure, or paper).

6 Discussion

We have presented slif, a system which analyzesimages in the biomedical literature. It processesboth text and image, combining them through la-tent topic discovery. This enables users to browsethrough a collection of papers by looking for re-lated topics or images that are similar to an imageof interest.

Although it is crucial that individual componentsachieve good results (and we have shown good re-sults in our sub-tasks), good component perfor-mance is not sufficient for a working system. Slifis a production system that has been shown to yieldusable results in real collections of papers.

The project is on-going and many avenues forimprovement are being exploited. Among those arebetter semantic understanding of fmi data, moreadvanced image processing of gels, exploitation ofthe full-text, as well as a continuing improvementof all the components in the pipeline.

6.1 Acknowledgements

Development of slif is suported by National Insti-tutes of Health grant R01 GM078622.

References

[1] Amr Ahmed, Eric P. Xing, William W. Cohen,and Robert F. Murphy. Structured CorrespondenceTopic Models for Mining Captioned Figures in Bio-logical Literature. In KDD ’09: Proceedings of theFiftenth ACM SIGKDD International Conferenceon Knowledge Discovery and Data Mining, 2009.

[2] David M. Blei and Michael I. Jordan. Modelingannotated data. In SIGIR ’03: Proceedings ofthe 26th annual international ACM SIGIR confer-ence on Research and development in informaion re-trieval, pages 127–134, New York, NY, USA, 2003.

ACM Press.

[3] William W. Cohen, Richard Wang, and Robert F.Murphy. Understanding captions in biomedical pub-lications. In KDD ’03: Proceedings of the ninthACM SIGKDD international conference on Knowl-edge discovery and data mining, pages 499–504, NewYork, NY, USA, 2003. ACM.

[4] Jan-Mark Geusebroek, Minh Anh Hoang, Jan vanGernert, and Marcel Worring. Genre-based searchthrough biomedical images. volume 1, pages 271–274, 2002.

[5] Zhenzhen Kou, William W. Cohen, and Robert F.Murphy. Extracting information from text and im-ages for location proteomics. In Mohammed JaveedZaki, Jason Tsong-Li Wang, and Hannu Toivonen,editors, BIOKDD, pages 2–9, 2003.

[6] Zhenzhen Kou, William W. Cohen, and Robert F.Murphy. High-recall protein entity recognition usinga dictionary. In Bioinformatics, vol. 21, pages i266–273, 2005.

[7] Zhenzhen Kou, William W. Cohen, and Robert F.Murphy. A stacked graphical model for associatingsub-images with sub-captions. In Russ B. Altman,A. Keith Dunker, Lawrence Hunter, Tiffany Mur-ray, and Teri E. Klein, editors, Pacific Symposiumon Biocomputing, pages 257–268. World Scientific,2007.

[8] Robert F. Murphy, Zhenzhen Kou, Juchang Hua,Matthew Joffe, and William W. Cohen. Extract-ing and structuring subcellular location informationfrom on-line journal articles: The subcellular loca-tion image finder. In IASTED International Confer-ence on Knowledge Sharing and Collaborative En-gineering, pages 109–114, 2004.

[9] Robert F. Murphy, Meel Velliste, Jie Yao, and Gre-gory Porreca. Searching online journals for fluo-rescence microscope images depicting protein sub-cellular location patterns. In BIBE ’01: Proceed-ings of the 2nd IEEE International Symposium onBioinformatics and Bioengineering, pages 119–128,Washington, DC, USA, 2001. IEEE Computer So-ciety.

[10] Nicholas Roy and Andrew Mccallum. Toward op-timal active learning through sampling estimationof error reduction. In In Proc. 18th InternationalConf. on Machine Learning, pages 441–448. Mor-gan Kaufmann, 2001.

[11] Hagit Shatkay, Nawei Chen, and DorotheaBlostein. Integrating image data into biomedicaltext categorization. In Bioinformatics, vol. 22,pages 446–453, 2006.

4