Upload
ngocong
View
217
Download
3
Embed Size (px)
Citation preview
HAVING A BLAST: ANALYZING GENE SEQUENCE DATA WITH BLASTQUEST –
WHERE DO WE GO FROM HERE?
William G. Farmerie1, Joachim Hammer2, Li Liu1, and Markus Schneider2
University of FloridaGainesville, FL 32611, U.S.A.
AbstractIn this paper, we pursue two main goals. First, we describe a new, user-driven tool, called BlastQuest, for managing BLAST query results. BlastQuest provides interactive, Web-enabled query, analysis, and visualization facilities beyond what is possible by current BLAST interfaces. Specifically, the BLAST results, which are in XML format, are extracted, structured, and stored persistently in a relational database to support a series of built-in analysis operations that can be used to select, filter, and order data from multiple BLAST results efficiently and without referring to the original result files. In addition, users have the option to interact with the BLAST data through a mask-oriented, non-SQL query interface.
Despite BlastQuest’s recognized benefits for biologists, its functionality is limited in several important ways. The second goal of this paper is to analyze these shortcomings and describe a new concept based on two main pillars. (1) A Genomics Algebra, which provides an extensible set of high-level genomic data types (GDTs) together with a comprehensive collection of appropriate genomic functions, and (2) a Unifying Database, which allows us to integrate and manage the semi-structured contents of publicly available genomic repositories and to transfer these data into GDT values.
1. Introduction
Biologists are nowadays confronted with two main problems, namely the exponentially growing volume
of biological data of high variety, heterogeneity, and semi-structured nature, and the increasing
complexity of biological applications and methods afflicted with an inherent lack of biological
knowledge. As a result, many and very important challenges in biology and genomics are now challenges
in computing and here especially in advanced information management and algorithmic design.
The currently most widely used and accepted tool for conducting similarity searches on gene
sequences is BLAST (Basic Local Alignment Search Tool) [1]. BLAST comprises a set of similarity
search programs that employ heuristic algorithms and techniques to detect relationships between gene
sequences and rank the computed ‘hits’ statistically. An essential problem for the biologist is currently the
processing and evaluation of BLAST query results, since a BLAST search yields its result exclusively in
a textual format (e.g., ASCII, HTML, XML). This format has the benefit of being application-neutral but
at the same time impedes its direct analysis. In this paper, we describe a new powerful tool, called
BlastQuest, for managing BLAST results stemming from multiple individual queries. This tool provides
1 Affiliation: ICBR Molecular Services Division: DNA Sequencing Core, Interdisciplinary Center for Biotechnology Research.2 Affiliation: Deptartment of Computer & Information Science & Engineering.
the biologist with interactive and Web-enabled query, analysis, and visualization facilities beyond what is
possible by current BLAST interfaces. In particular, BLAST results from multiple queries are imported,
structured, and stored in a relational database to support a series of built-in analysis operations that can be
used to select, filter, group, and order these data efficiently and without referring to the original BLAST
result files. In addition, users have the option to interact with the data through a user-tailored, screen-
mask oriented, non-SQL query interface based at a deeper, hidden level on a well-defined subset of SQL.
Section 2 elaborates on the current, main challenges in genomics and emphasizes the need for tools
capable of processing BLAST results. Section 3 describes the main features and data analysis options of
the BlastQuest user interface. In Section 4, we view our BlastQuest system from the system architecture
perspective. Section 5 considers desired improvements to BlastQuest. Specifically, we illustrate why new,
sophisticated concepts, tools, and non-standard database technology, which altogether should lead us far
beyond BLAST technology, are indispensable in order to advance biological and genomic research and
progress. We conclude the section with a description of a new, data model, language, and architecture for
integrating, processing and querying genomic information. Section 6 summarizes the paper.
2. The Challenge of Genomics and Its Effect on Computer Science
Genomics is a biological discipline focused on understanding living organisms at the level of the whole
genome. It goes beyond a gene-by-gene approach and instead takes a global view of the complete genetic
system. Genomic scientists examine the full catalogue of genes, the process that control them, gene inter-
relationships and inter-dependencies, and how the organism responds to changes in environment through
the expression of genetic information. In order to illustrate the challenges faced by scientists in this field,
we first review the most important concepts underlying gene sequencing.
2.1. Gene Sequencing
DNA is an information storage macromolecule to encode all of the heritable information passed from
generation to generation of living organisms. In biological systems, genetic information flows from DNA
(genes) to proteins, which are the molecules responsible for mediating or catalysing biological processes.
In other words, inherited information is selectively converted into active biomolecules in response to
changing environmental conditions or demands. The molecular information pathway from gene to protein
goes through an intermediate class of molecules known as messenger RNA (mRNA). The synthesis of
mRNA is known as transcription, and the conversion of mRNA into protein is a process known as
translation. Both transcription and translation are important regulatory steps used to control which
genetic information is expressed, and when and where protein molecules will be made by the cell. The
constellation of mRNA molecules in a cell at any moment represents the expressed genome. The
2
expressed genome is also referred to as the transcriptome. Identifying all the genes present in the
transcriptome effectively infers the proteins being utilized by the cell (also known as the proteome) and
essentially defines the current biochemical process of the cell. While characterizing the global cellular
proteome would be most direct and informative, this is not possible using currently available technology.
Instead genomics scientists use high throughput DNA sequencing to characterize the genome and the
transcriptome. Genome sequencing involves determining the nucleotide sequence of extensive
chromosomal regions or in some cases a complete nucleotide sequence of the whole genome.
Characterization of the transcriptome on the other hand involves full or partial sequence characterization
of mRNA molecules. Partial sequences of mRNA molecules are known as Expressed Sequence Tags
(EST) sequences. While the process of DNA sequencing is routine, nucleotide sequences do not directly
reveal their biological meaning or function. The possible biological function of a gene sequence must be
determined either through direct empirical experimentation, or more often through inferencing of gene
function using nucleotide sequence homology searches of gene databases such as GenBank [5].
2.2. Gene Homology Searches
Gene homology searches most often use the BLAST algorithm [1]. The BLAST search engine takes a
query nucleotide sequence and searches it against the database for entries matching the query. The
BLAST algorithm calculates statistical scores (bit scores and e-values) making real sequence homology
matches easier to distinguish from matches that might happen by chance. Other information included in
the BLAST result includes a short text string summarizing the biological properties of the database
match, and several unique identification numbers, the GI Number (unique ID for Genbank records) and
Accession Number, linking the matched sequence back to the GenBank database and to additional
information stored in the full database record. Each nucleotide query sequence submitted to the BLAST
search engine returns as few as zero (no matching homologous sequence) to hundreds of matching
database records. Results of BLAST searches are usually interpreted by reviewing the text output.
However, large-scale genomics projects often generate tens of thousands of nucleotide sequences
and the prospect of manually manipulating, summarizing, and interpreting the thousands of BLAST
output files is impractical at best. Scientists facing this informatics challenge may become discouraged or
might overlook important information because they simply cannot find it. Clearly, methods or tools are
needed to help manage the process of identifying and evaluating unknown nucleotide sequences and the
sometimes-overwhelming information obtained in large-scale nucleotide sequence homology searches.
3
2.3. BlastQuest as an Answer to Tool Requirements from a Biologists Perspective
The Biotechnology Program at the University of Florida through the Interdisciplinary Center for
Biotechnology Research (ICBR) operates a Genomics Core facility with the personnel and equipment to
carry out large-scale DNA sequencing projects. The ICBR Genomics Core provides DNA sequencing
services to campus researchers as well as biological scientists and their collaborators located throughout
the world. A typical DNA sequencing project involves collections of several hundred to tens of thousands
of DNA sequences. As outlined in Section 2.2, nucleotide sequence homology searches are frequently the
first step toward identifying the biological function of unknown nucleotide sequences. Most university-
based investigators lack the computational expertise and infrastructure to initiate and manage BLAST
homology searches on the hundreds or thousands of nucleotide sequences generated by their projects.
Biological scientists want to gain insight from their data without first having to overcome the
management of their data.
With this in mind, there has been a clear need to build a centralized system to manage BLAST
results. The BlastQuest project was initiated to help with the challenge of managing BLAST results and
make this information available in a web-based interface accessible to client researchers located anywhere
with internet access. It began with several modest goals, foremost the delivery of a web-based tool for
viewing, searching, filtering, and summarizing large numbers of BLAST results files. Our solution began
with asking our user community for ideas about the types of analysis they would like to perform. The
result of these interviews drove our initial list of functional requirements for the BlastQuest system:
A BLAST results viewing tool accessible to research groups at remote locations . Users should
have access to their BLAST results from anywhere on the Web including the ability to share
results with colleagues in other locations.
Selective browsing of BLAST homology search results. As a first step, biologists want a broad
overview of the possible biological functions of the many genes sequences represented in their
DNA sequence data. The ability to reduce and summarize BLAST data to only the most
significant results is initially very informative.
Search capability on a variety of criteria, such as text terms on biological properties or gene
functions. As biological scientists identify their most interesting gene sequences they need a
way to focus and retrieve only those search results related to the precise topic of interest.
Selective data filtering on various BLAST statistical criteria such as e-value or bit score .
These statistical parameters help discriminate between real sequence homology matches and
matches that might happen by chance. There are no hard limits to the significance of these
statistical parameters. The user will choose parameters giving either a more relaxed or
restricted view as needed.
4
Selective data grouping on criteria such as GI number, or a defined number of top-scoring
results. For example, viewing the three statistically best-scoring results for each query sequence
is a convenient way to summarize and browse BLAST results for many query sequences.
Grouping query sequences by GI number collects all of the query sequences having sequence
homology matches with the same sequences from the database. Two or more query sequences
sharing the same database homology match imply the query sequences are related to each other
and suggest additional analysis of the relationship is warranted.
Privacy constrained sharing of results among the scientists. DNA sequence data is often
proprietary and may constitute intellectual property. Such data should not be made public until
properly protected.
A convenient interface for getting queries into and BLAST results out of the system . The
interface must be attractive and logically implemented so users will be able to find and use the
tools the system provides.
To the best of our knowledge, the functionalities of WebBLAST 2.0 [3] and the Ontario Center for
Genomic Computing OCGC BLAST [2] match some of our requirements but fall short in several
important aspects. For example, there is no provision in WebBLAST for applying global filtering and
grouping operations, or a mechanism for searching all BLAST results on user-supplied text terms. The
OCGC BLAST results manager appears closest to BlastQuest in functionality, allowing selected viewing
and data filtering on up to five criteria. However, OCGC BLAST is not available to genomics scientists
outside of the Province of Ontario, Canada. The BlastQuest Project is designed to meet our immediate
specific requirements, but most important, provide a platform we might freely modify to test our notions
of Genomics Algebra (Section 5.2), an advanced query language for biological information.
3. BlastQuest User Interface
BlastQuest simplifies large-scale analysis in gene sequencing projects by providing scientists with a
means to filter, summarize, sort, group, and search BLAST output data. BlastQuest extracts gene data
from XML files, which are returned as the result of homology searches from BLAST engines, and stores
them in an underlying relational database. This allows the user to benefit from well-known relational
concepts like transactions, controlled sharing, and query optimization. Finally, BlastQuest also allows
users to perform homology searches of their proprietary sequence data against public domain data, such
as NCBI databases, etc.
The most frequently used user operations are hard-wired in the user interface and accessible via
command buttons. Their execution rests on SQL that is hidden from the user. To enable data analysis that
is not directly supported by the built-in user interface operations, BlastQuest offers a more flexible, mask-
5
oriented, and especially non-SQL query interface since biologists object to SQL due to its complexity and
low-level abstraction. This interface essentially allows the user to construct complex boolean expressions
as selection conditions which include logical operators and substring search predicates. The underlying
query execution is based on parameterized SQL queries, which are instantiated and automatically
translated into executable SQL code by the DBMS.
Another interesting feature of BlastQuest is that it can be linked to the so-called SMART (Simple
Modular Architecture Research Tool [6]). The integration of BlastQuest output into SMART is in direct
response to the desire by scientists for new tools and interfaces capable of accessing and integrating
external resources into one system. In Section 5, we describe our plans to develop a Genomics Algebra
query software that operates on a unifying database whose contents can include data from existing
genomics repositories.
Finally, BlastQuest enables to manage BLAST data on a per-project or per-user basis using the
security features of the underlying database while at the same time allow controlled sharing of this data
in order to support collaboration. A startup page facilitates the extraction of gene data from original,
external BLAST files into a MySQL database. Due to the large volume of data, a simple page-by-page
viewing is not helpful to the user but selection mechanisms are needed to find the data of interest. The
overall strategy is to apply a sequence of consecutive operations on the data to gradually approach the
data of interest. In the following we describe the main user interface features for doing this.
The first feature is to let BlastQuest create a summary page for selected sequence segments.
Users require this high level summarization of their sequences because the volume of BLAST output data
for large-scale sequencing project is well beyond simple page-by-page viewing. This summary page gives
an abbreviated overview of each query sequence with possible function. For each query DNA sequence,
only the sequence database match with the best statistical score calculated by BLAST is displayed with a
summary of important biological information, usually text terms describing a gene or protein name, and
sometimes including possible biological functions. The summary page also contains, for each matching
sequence, the GenBank sequence ID, gene definition, and expect value.
The second feature is user-controlled selection. Unfortunately, the statistically calculated ranking
of matching sequences provided by BLAST does not necessarily correspond to the biological knowledge
and experience of the user. The user may apply their biological knowledge or insight to tag a different
result as better for expressing the possible function of the query sequence. By manually selecting a
specific query result, the user can get additional information such as the percentage of identity, or
alignment of the query sequence and the matching sequence. Even a detailed display of sequence
alignments is available, which is identical to the free-text formatted BLAST result to which most BLAST
users are accustomed.
6
The third feature is related to built-in selection facilities, which can be activated by a mouse-click
and operate on all query sequences and their query results. Examples are the displays of hits with expect
values less than a particular threshold by selecting from a pull-down menu (e.g., shown later in Figure 2),
or restricting the display to the best n database matches for each query sequence. All filtering facilities
together give researchers the ability to adjust their analysis process to the particular research focus,
project status, and prior knowledge of query sequences, to reduce the original BLAST result to a
manageable size, and especially to remove results of low quality.
The fourth feature comprises ordering and grouping functions. These help the user to discover
relationships among genes or expression patterns. For examples, there may be more than one sequence or
contig that are derived from different regions of the same mRNA or gene, grouping on GI number will
cluster these related sequences, identifying them for further analysis of their relationship. One of the new
features we are implementing is grouping sequences on UniGene ID. This is an additional step to identify
EST sequences that come from gene orthologs or gene paralogs3. Another example is that biologists
sometimes want to know which sequences have their functions well resolved by blast search, and which
have not. By ordering query sequences by the expect values of top scoring BLAST hits, users identify
sequences with high-quality hits, sequences with only low-quality hits, or even sequences having no hit.
This step rapidly classifies sequences for different types of additional analysis. For example, if the user
asks for grouping on GI number or query sequence, related sequences and their BLAST results are
grouped together rather than appear randomly or out of context. This is also a proven method to identify
EST sequences that come from different regions of the same mRNA, gene orthologs, or gene paralogs4.
The fifth feature enables user-defined, mask-oriented, non-SQL queries. This feature refers to the
problem that the built-in functionality of BlastQuest is sometimes insufficient for specific analysis tasks.
For example, if a user wants to find out which sequences are homologous to genes with reverse
transcriptase function, which is not hypothetical but is proved by empirical data, BlastQuest does not have
built-in selection facilities for this specific query. In fact, we recognize that it is impossible to exhaust
such ad hoc queries. On the other hand, we cannot expect our users to learn SQL in order to query their
results. To solve this problem, BlastQuest provides a Web page which allows the user to interactively and
textually construct complex boolean filter expressions which may include logical operators like “AND”
and “OR” as well as substring search predicates like “Contains” or “Not Contains.” A search field (like
“Hit Definition” in Figure 1) to which the Boolean expression is compared can be selected by a drop-
down menu. Figure 1 shows two textual representations of the same Boolean expression under
3 Gene orthologs are genes that are derived by divergent evolution, such as the -hemoglobin gene from human and from mouse. Gene paralogs are genes that are duplications, such as -hemoglobin and -hemoglobin.4 Gene orthologs are genes that are derived by divergent evolution, such as the -hemoglobin gene from human and from mouse. Gene paralogs are genes that are duplications, such as -hemoglobin and -hemoglobin.
7
construction. The second representation expresses the condition in a way nearer to natural language. The
first representation is a test mode translating the ‘natural language’ condition into SQL. In a later version
the SQL test mode will disappear. The construction of the Boolean expression and hence of the query is
completed by clicking the “Commit” button. BlastQuest assembles the SQL query, sends it to the MySQL
driver, receives the results and displays them. In the example in Figure 1, the user is specifying a query
which will produce matches that contain the word ‘reverse’, but not ‘hypothetical’.
Figure 1: User-defined query construction tool.
The sixth feature to be mentioned is interoperability between BlastQuest and other biological
information systems. Creating links to other systems in order to make use of their specific functionality
becomes more and more important for the biologist. In the context of BlastQuest, after having examined
the query sequences and their probable identities, we wish to derive the protein sequences encoded by the
nucleotide sequence. Rather than translate the nucleotide sequence directly, BlastQuest takes the ‘best’
match, which represents a homologous gene closely related to the unknown query sequence, and retrieves
the corresponding protein sequence as translated by BLAST. After grouping search results by query
sequence (e.g., the best five statistical matches) the user is presented with the screen shown in the top half
of Figure 2. Next, the user checks the ‘amino conversion’ box at the right top of the screen, and the check
box adjacent to the query sequence they wish to translate into an amino acid sequence. When the user
clicks the ‘Details’ button, the ‘Sequence Analysis’ screen shown in the bottom half of Figure 2 appears.
The user may submit the derived protein sequence to the SMART protein analysis Web site by simply
clicking on the amino acid sequence. Results of the SMART analysis will appear in the browser window.
8
Figure 2: Filtering and grouping BLAST results on a project basis.
The seventh and final major feature is the capability to perform BLAST searches against the users’
own sequence database. This functionality gives the user the ability to query their own sequence data with
a specific nucleotide or protein sequence. If a user obtains an interesting sequence from other resources,
internal blast search helps to find out whether s/he owns similar sequences. If similar sequences are
found, the corresponding clone is identified and retrieved from the users clone bank where it may be used
for further experiments. In the example shown in Figure 3, the user pasted the query sequence into the top
text area. The interface also allows input of a sequence file location for uploading. From drop-down
menus, the user chooses from among different BLAST programs and different local target databases that
s/he owns or has a “guest” privilege for. BlastQuest also provides choices for choosing a homology
matrix via a drop-down menu. However, BlastQuest does not provide users to specify other parameters,
such as expect value threshold, word size, gap-open penalty, and gap-extension penalty. The rationale
behind this is that we allow retrieval of all hits with expect values less than 10. Users may filter, order,
group or perform other manipulations later by using the built-in functionalities of the BlastQuest system.
After the user clicks the “BLAST” button, the query sequence is submitted with selected parameters. For
individual blast query, the result will be displayed in HTML format. If the user has “owner” privilege,
s/he can choose to either parse and store this blast output persistently into the MySQL database or delete
9
it when the session ends. For batch queries, BLAST results will be parsed and automatically stored in the
MySQL database automatically for later examination and analysis.
Figure 3: Internal BLAST search user databases.
All operations described here can be combined to analyze data generated in a larger project. For
example, one may use BlastQuest to retrieve hits with expect value lower than 0.05, followed by grouping
on gene ID, and only display the top five matching hits per GI number (as illustrated in Figure 2).
4. Architectural Overview of the BlastQuest System
Figure 4 depicts a conceptual overview of the 3-tiered BlastQuest system architecture. Tier 1 contains the
database backend, which is implemented using the MySQL5 RDBMS. Since BlastQuest is mainly a
proof-of-concept prototype rather than a production-strength system, our choice for a DBMS was
governed by availability of source code and platform compatibility rather than performance and richness
in features. The database backend stores and manages BLAST and PHRAP (Phragment Assembly
Program) [4] results, which are represented as XML and ACE6 (ArChivE) documents and whose structure
has been mapped into the relations Query, Assembly, Hit, and Query_Hit shown later in Figure 5.
5 See http://www.mysql.com/.6 See http://bozeman.mbt.washington.edu/phrap.docs/phrap.html for an example and documentation on the format.
10
Figure 4: Conceptual overview of the BlastQuest system architecture.
For each query sequence submitted to the BLAST server (shown in the upper right-hand corner of Fig. 4),
the relation Hit stores detailed hit information, such as hit definition, expect value, bit score, pairwise
alignments and so forth. For queries, which do not produce a match in the homology search, the fields are
marked as NULL. From a biological point of view, sequences with no homologous sequence match often
lead to new genes and are analyzed in a different manner (outside of BlastQuest). In addition, the
homology search criteria for each BLAST search, such as the BLAST program name, database name,
matrix, and date, are stored in Query_Hit table. These parameters are important to users because for the
same query sequence, BLAST generates different results based on different criteria. For example,
BLASTN results and BLASTX results may indicate different functions for the same query sequence. In
addition, the same BLAST search on different days may generate different hits since BlastQuest’s
BLAST server is regularly updated with the latest version of the NCBI data files. The MySQL database
also stores information about how related gene segments are assembled into single consensus DNA
sequences by PHRAP, which is external to BlastQuest and invoked before the DNA sequence results are
submitted to BLAST. PHRAP outputs its results in an ACE file, which is mapped into the relation
Assembly. If the user considers the results of the BLAST search interesting, s/he may want to extract
the physical clones from which the specific query sequences are generated or assembled. This is possible
by joining the Assembly and Query tables via the “qid” foreign key to retrieve all segments and
corresponding clone names that are clustered into a specific query sequence.
11
Figure 5: Relational Schema of the BlastQuest MySQL database.
The database also maintains information about users and their corresponding gene sequencing
projects, which are stored in the three remaining relations, User, Project, and User_Proj. The
relation User_Proj represents the many-many relationship between scientists and the projects to which
they belong. Since all sequence data is organized by project (using the PID foreign key in relation
Query), BlastQuest provides control over which user has access to which data.
Tier 2 contains the multi-threaded BlastQuest application program, which is divided into five
modules: The BLAST Server, which is used to conduct BLAST searches against NCBI as well as internal
data owned by the users; the client interface module, which handles communication with the Web clients
in tier 1; the two loader modules for extracting and loading data from the XML and ACE input files into
the database; and the SQL constructor for assembling the queries to be sent to the database. The BLAST
Server is downloadable freeware from NCBI. The client interface module is implemented as a series of
Java Server pages (JSPs) that execute inside a Tomcat server. The remaining three modules are
implemented as Java classes. We briefly highlight the functionality of each module.
BlastQuest maintains a local version of NCBI’s “NR” database, which is, updated monthly with
new releases and can be searched with a local copy of the BLAST server (labeled “Blastable Data” in
Figure 4). In addition to public domain data, this local BLAST database also contains blastable data from
each user’s proprietary query sequences. The conversion of query sequence data into blastable data is
done using the “formatdb” program provided with NCBI’s BLAST search engine.
The XML loader parses each BLAST result file into a Document Object Model (DOM)
representation using the Xerces Java Parser 1.4.4. The XML loader then extracts the relevant data items
needed to populate the Hit and Query_Hit tables. Specifically, the loader module contains several
12
Underlined attributes denote the unique identifiers (primary keys) for each relation. Attributes with a superscript are foreign keys (superscript denotes the referenced relation).
classes whose data structures correspond to the tables in the database schema. When the loader collects
data from an XML file, it populates the appropriate class objects with the extracted values. At the end, the
objects are passed to the SQL constructor, which creates the SQL commands to insert the values into the
relational database. The ACE loader works in a similar fashion. However, since there was no standard
ACE parser available, we created our own. Our event-based parser detects the presence of certain
keywords in the ACE input file and extracts the information associated with that keyword. Other, more
efficient loading options are possible, for example, by using the bulk loading utilities of the DBMS.
However, by making our loader modules part of the Web-based middleware, users can load BLAST
results into their BlastQuest accounts from anywhere on the Web.
The SQL constructor is the gateway between the database and the middleware. It connects to the
MySQL relational database engine via the JDBC driver, which is part of the Java developer’s kit, and is
supported as a part of MySQL’s call level interface. In addition to creating the SQL load commands, the
SQL constructor translates commands from the user interface into SQL queries, which can be executed by
MySQL. Analogously, it processes the resulting record sets and creates the Java objects that are used by
the client interface to generate the Web pages.
Another important function of the SQL constructor is the management of connections to the
relational database engine. Each time a query is to be processed, a database connection is required.
Establishing a JDBC connection with a relational database is relatively time consuming and may take as
much as two seconds7. Since it is anticipated that queries from multiple clients will need to be
simultaneously supported, there is a need to manage the limited number of database connections
carefully. Because of the overhead required to establish a new database connection, the SQL constructor
maintains a “pool” of open database connections. Each time a client needs to process a query, it requests
a connection from the pool, eliminating in most cases the need to re-establish and wait for a database
connection. It is important to note that BlastQuest is only supporting read-queries, avoiding the need to
support updates and the overhead of transaction management inside the MySQL database.
Tier-3 is a (thin) client interface, which is implemented as dynamic Web pages displayed inside a
Web browser. Client-side processing is limited to validation of user input, submitting requests to the
BlastQuest application and displaying HTML results.
5. Planned Improvements
The BlastQuest system described above has been successfully employed and testet by scientists in the
gene-sequencing lab at ICBR and by several of their clients for over six months; the feedback from the
7 While two seconds does not appear to be a long time, consider that most commercial database engines can process queries against tens of thousands of tuples in less than one second!
13
clients has been positive. However, we also received important feedback regarding the limitations of the
current system. For example, there is a desire for additional, more sophisticated analysis functionality and
the ability to integrate data from external repositories.
5.1. User Requirements
As a starting point for the development of a more sophisticated management system for genomics data,
we have identified all of the biological needs that are currently not supported in BlastQuest 8. In the
interest of space, we provide the readers with an overview of the most important ones:
1. The ability to query, search and analyze data from external genomics repositories (in addition
to those accessible through BLAST). An extension of this is the ability to integrate related
results from multiple repositories in a meaningful manner, for example, to fill in missing values
or correct inconsistencies that exist across different repositories.
2. A representation of the genomics data that is semantically richer than the current textual
representation provided by BLAST and most other repositories. For example, BLAST query
results are more or less collections of textual strings and numerical values and are not expressed
in biological terms such as genes, proteins, and nucleotide sequences. As a result, BLAST and
BlastQuest operations are limited to basic string manipulation (e.g., shortest common substring)
rather than high-level, gene-specific operations such as transcribe, translate, etc.
3. Integration of new specialty evaluation functions. The possibility to evaluate data from BLAST
results as well as self-generated data with publicly available methods is insufficient. Thus, it
must be possible to create, use, and integrate user-defined functions that are capable of
operating on both kinds of data into the analysis interface of the tool. However, this requires an
extensible database management system, query language, and user interface, which is currently
not part of BlastQuest.
4. The ability to create and store new knowledge. A biologist generates new biological data from
their own research or experimental work, for example, by analyzing BLAST results. Hence,
scientists have expressed a strong desire to store and manage this newly created knowledge
together with the source data. For example, there is a need to annotate data in BLAST results
and to store the annotations persistently so that they can be re-used (e.g., by linking a record in
a new BLAST result to an existing annotation in the repository).
8 In fact, based on a survey of the related literature, we have found that most of the existing integration and management systems for genomics data such as K2/KLEISLI (http://db.cis.upenn.edu/K2/), Tambis (http://imgproj.cs.man.ac.uk/tambis/index.html), SRS (http://srs.ebi.ac.uk), etc. only support some of the functionality described in this list.
14
5. Support for controlled collaboration among multiple scientists. It is of great value for scientists
to share some of the their findings in a controlled manner with colleagues. For example, it
should be possible among the users of a genomics repository system to grant write access to
some of the annotations but read-only or no access to others.
6. The ability to connect DNA sequence identities inferred from BLAST results with gene-
associated biological functions described through the efforts of the Gene Ontology (GO)
Consortium [7]. This type of cross-referencing is the best way to describe the functionality of a
newly, discovered gene. This functionality will help biologists to annotate and catalog the
genes by universally accepted GO IDs and hence help them to discover new genes.
5.2. The New Approach
Based on previous list of needs, which illustrates the complexity of the information-related challenges
that confront biologists and computer scientists, we decided to redesign our current system from the
ground up. For example, to provide users with a semantically rich representation of the genomics data as
well as support for specialty functions (needs 2 and 3 above), requires the design of a new data type
system and operations, which must be integrated with the underlying database management system for
efficient query processing and persistence. Another example, access to multiple genomics repositories
(need 1), requires the ability to extract, translate, and reconcile heterogeneous data from multiple sources
and store the integrated result using a global schema, which has been constructed either from the local
schemas of the sources or based on general knowledge of the domain.
In response to our requirements analysis, we are developing a new genomics integration and
management system that is based on two fundamental pillars: (1) A Genomics Algebra software system to
provide an extensible set of high-level genomic data types (GDTs) (e.g., gene, protein, nucleotide)
together with a comprehensive collection of appropriate genomic functions (e.g., translate, transcribe,
decode). (2) A Unifying Database to manage the semi-structured contents of publicly available genomic
repositories and to transfer these data into GDT values. These values then serve as arguments of
Genomics Algebra operations, which can be embedded into a DBMS query language.
5.3. Genomics Ontology
A precondition for a successful construction of our Genomics Algebra is the design of an ontology for
molecular biology and bioinformatics. By ontology, we are referring to “a specification of a
conceptualisation.” That is, in general, an ontology is a description of the concepts and relationships that
define an application domain. Applied to bioinformatics, an ontology is a “controlled vocabulary for the
description of the molecular functions, biological processes and cellular components of gene products”.
15
An obstacle to its unique definition is that the multitude of heterogeneous and autonomous genomic
repositories has induced terminological differences (synonyms, aliases, formulae), syntactic differences
(file structure, separators, spelling) and semantic differences (intra- and interdisciplinary homonyms). The
consequence is that data integration is impeded by different meanings of identically named categories,
overlapping meanings of different categories, and conflicting meanings of different categories. Naming
conventions of data objects, object identifier codes, and record labels differ between databases and do not
follow a unified scheme. Even the meaning of important high-level concepts (e.g., the notion of gene or
protein function) that are fundamental to molecular biology is ambiguous.
If the user queries a database with such an ambiguous term, until now (s)he has full responsibility
to verify the semantic congruence between what (s)he asked for and what the database returned. An
ontology helps here to establish a standardized, formally and coherently defined nomenclature in
molecular biology. Each technical term has to be associated with a unique semantics that should be
accepted by the biological community. If this is not possible, because different meanings or
interpretations are attached to the same term but in different biological contexts, then the only solution is
to coin a new, appropriate, and unique term for each context. Uniqueness of a term is an essential
requirement to be able to map concepts into the Genomics Algebra.
Consequently, one of our current research efforts and challenges is to develop a comprehensive
ontology, which defines the terminology, data objects and operations including their semantics that
underlie genome sequencing. Besides posing such a genomic ontology, a main challenge is to find or
even devise an appropriate formalism for its unique specification.
5.4. Genomics Algebra
The Genomics Algebra is the derived, formal, and executable instantiation of the resulting genomic
ontology. Entity types and functions in the ontology are represented directly using the appropriate data
types and operations supported by our Genomics Algebra. This algebra has to satisfy two main tasks.
First, it has to serve as an interface between biologists, who use this interface, and computer scientists,
who implement it. An essential feature of the algebra is that it incorporates high-level biological
terminology and concepts. Hence, it is not based on the low-level concepts provided by database
technology. Second, as a result, this high-level, domain-specific algebra will greatly facilitate the
interactions of biologists with genomic information stored in our Unifying Database (see Section 5.5) and
incorporating at least the knowledge of the genome repositories. To our knowledge, no such algebra
currently exists in the field of bioinformatics.
Our Genomics Algebra is a domain-specific algebra incorporating a type system for biological data.
Its sorts, operators, domains, and functions will be derived from the genomic ontology developed in the
16
first step. The sorts are called genomic data types (GDTs) and the operators genomic operations. To
illustrate the concept, we assume the following, very simplified signature:
sortsgene, primaryTranscript, mRNA, protein
opstranscribe: gene primaryTranscriptsplice: primaryTranscript mRNAtranslate: mRNA protein
This “mini algebra” contains four sorts or genomic data types for genes, primary transcript, messenger
RNA, and protein as well as three operators transcribe, which for a given gene returns its primary
transcript, splice, which for a given primary transcript identifies its messenger RNA, and translate, which
for a given messenger RNA determines the corresponding protein. We can assume that these sorts and
operators have been derived from our genomic ontology. Hence, the high-level nomenclature of our
genomic ontology is directly reflected in our algebra. The algebra now allows us to (at least) syntactically
combine different operations by (function) composition. For instance, given a gene g, we can
syntactically construct the term translate(splice(transcribe(g))), which yields the protein determined by g.
The Genomics Algebra develops its full expressiveness and usability if it is designed and integrated
as a collection of abstract data types (ADTs) into the type system and query language of a database
system (Section 5.6). ADTs encapsulate their implementation and thus hide it from the user or another
software component like the DBMS. From a modeling perspective, the DBMS data model and the
application-specific algebra or type system are separated. This enables the software developer to focus on
the application-specific aspects embedded in the algebra. Consequently, this procedure supports
modularity and conceptual clarity and permits the reusability of an algebra for different DBMS data
models. It requires extensibility mechanisms at the type system level in particular and at all levels of the
architecture of a DBMS in general, starting from user interface extensions down to new, external
representation and index structures. From an implementation point of view, ADTs support modularity,
information hiding, and the exchange of implementations without changing the interface.
5.5. Unifying Database
The Unifying Database is the second pillar of our integrating approach. By Unifying Database, we are
referring to a data warehouse, which integrates data from multiple genomic repositories. We have chosen
the data warehousing approach to take advantage of the many benefits it provides, including superior
query processing performance in multi-source environments, the ability to maintain and annotate
extracted source data after it has been cleansed, reconciled and corrected, and the option to preserve
17
historical data from those repositories that do not archive their contents. Equally important, the Unifying
Database is also the persistent storage manager for the Genomics Algebra.
The component most visible to the user is the integrated schema. We distinguish between the
portions of the schema that house the restructured and integrated external data (i.e., the entities that store
the genomic data brought in from the sources) and which is publicly available to every user, and those
that contain the user data (i.e. the entities that store user-created data including annotations), which may
be private. The schema containing the external data is read-only to facilitate maintenance of the
warehouse; user-owned entities are updateable by their owners. Separating between user and public space
provides privacy but does not exclude sharing of data between users, which can be controlled via the
standard database access control mechanism. Since all information is integrated in one database using the
same formats and representation, cross-referencing, linking, and querying can be done using the
declarative database language provided by the underlying database management system (DBMS), which
has been extended by powerful operations specific to the characteristics of the genomic data. However,
users do not interact directly with the database language; instead, they use the commands and operations
provided by the Genomics Algebra, which may be embedded in a graphical user interface.
Conceptually, the Unifying Database may be implemented using any DBMS as long as it is
extensible. By extensible we are referring to the capability to extend the type system and query language
of the database with user-defined data types. For example, all of the object-relational and most object-
based database management systems are extensible. We believe our integration of the Genomics Algebra
with the Unifying Database represents a dramatic improvement over current technologies (e.g., a query-
driven integration system connected to BLAST sources) and will cause a fundamental change in the way
biologists will conduct sequence analysis.
5.6. Interaction Between Genomics Algebra and Unifying Database
A conceptual overview of the high-level architecture that integrates the Genomics Algebra with the
Unifying Database is shown in Figure 6. The Unifying Database is managed by the DBMS and contains
the genomic data, which comes either from the external sources or is user generated. The link between the
Genomics Algebra and the Unifying Database is established through the DBMS-specific adapter.
Extracting and integrating data from the external sources is the job of the extract-transform-load (ETL)
tool shown on the right-hand side of Figure 6. User-friendly access to the functionality of the Genomics
Algebra is provided by the GUI component depicted in the top center.
The adapter provides a DBMS-specific coupling mechanism between the ADTs together with their
operations in the Genomics Algebra and the DBMS managing the Unifying Database. The ADTs are
plugged into the adapter by using the user-defined data type (UDT) mechanism of the DBMS. UDTs
18
provide the ability to efficiently define and use new data types in a database context without having to re-
architect the DBMS. The adapter is registered with the database management system at which point the
UDTs become add-ons to the type system of the underlying database.
Figure 6: Integration of the Genomics Algebra with the Unifying Database through a DBMS-specific adapter.
The component responsible for loading the Unifying Database and making sure its contents are up-
to-date is referred to as ETL (Extract-Transform-Load). In our system architecture, ETL comprises four
separate activities:
1. Monitoring the data sources and detecting changes to their contents. This is done by the source
monitors.
2. Extracting relevant new or changed data from the sources and restructuring the data into the
corresponding types provided by the Genomics Algebra. This is done by the sources wrappers.
3. Merging related data items and removing inconsistencies before the data is loaded into the
Unifying Database. This is done by the warehouse integrator.
4. Loading the cleaned and integrated data into the unifying database. This is done by the loader.
As we can see from Figure 6, the ETL component interfaces with a DBMS-specific adapter instead of the
DBMS directly. This adapter, which implements the interface between database engine and Genomics
Algebra, is the only component that has knowledge about the types and operations of the Genomics
Algebra as well as how they are implemented and stored in the DBMS.
6. Conclusion
In this paper we have described BlastQuest, a Web-based and interactive tool for importing and
persistently storing genomic data from multiple BLAST queries in a relational database, applying DBMS
19
functionality for processing and querying these data, and visualizing them appropriately. Limitations of
the underlying concept, which will inevitably be reached even through some meaningful improvements,
require new concepts and advanced tools.
The Genomic Algebra is a promising approach in this direction. We believe our new approach will
cause a fundamental change in the way biologists analyze genomic data. No longer will biologists be
forced to interact with hundreds of independent data repositories each with their own interface. Instead,
biologists will work with a unified database through a single user interface specifically designed for
biologists. Our high-level Genomics Algebra will allow biologists to pose questions using biological
terms, not SQL statements. Managing user data will also become much simpler for biologists, since
his/her data can also be stored in the Unifying Database and no longer will s/he have to prepare a custom
database for each data collection. Biologists should, and indeed want to invest their time being biologists,
not computer scientists.
From a computer science perspective, our project leverages and extends the benefits and
possibilities of current database technology. In particular, we demonstrate the elegance and expressive
power of modeling and integrating non-standard and extremely complex data by the concept of abstract
data types into databases and query languages. In addition, our approach is independent of a specific
underlying DBMS data model. That is, the Genomics Algebra can be embedded in a relational, object-
relational, or object-oriented DBMS as long as it is equipped with the appropriate extensibility
mechanisms. In addition, we believe we will gain valuable knowledge about the design and
implementation of new, sophisticated data structures and efficient algorithms in the non-standard
application field of biology and bioinformatics.
References
20