Embed Size (px)
Citation preview
MD2 – Getting Users Involved in the
Development of Data Warehouse Applications
Gilmar M. FreitasProdabel
Av. Presidente Carlos Luz, 127531230-901 Belo Horizonte MG
Alberto H. F. LaenderDepartment of Computer Science
Federal University of Minas Gerais31270-901 Belo Horizonte MG
Maria Luiza CamposDepartment of Computer Science
Federal University of Rio de Janeiro21941-590 Rio de Janeiro RJ
Abstract
User participation has become paramount for anysuccessful data warehouse initiative. In this paper, wepresent MD2, a tool based on the dimensional datamodeling approach, which facilitates the userparticipation in the development of a data warehouseapplication. MD2 assists users on identifying theiranalytical needs in order to help data warehousedesigners to better specify business requirements andlatter translate them into appropriate design elements.The tool is supported by a data repository that helpsgather metadata used to build the applicationdimensional schema and to specify reports that can begenerated from the data warehouse and visualizedthrough a Web interface.
1 Introduction
Data warehousing has become a key technologyfor many organizations. From a general point ofview, a data warehouse can be seen as anintegrated data repository that is generated andthen used across an organization for supportingthe processes of strategic planning and decisionmaking [4,7,8,10,18]. However, it is a generalunderstanding that this is a complex technologythat involves a number of difficulties for itseffective implementation. Among the maindifficulties pointed out by several authors[4,7,10,19] is the existence in the organizationsof a large spectrum of users with distinctcharacteristics and information needs, whichmakes user participation paramount for anysuccessful data warehouse initiative.
Dimensional modeling has been the predominantapproach to designing data warehouse appli-cations [2,3,10,11,15,18]. The main dimensionalmodeling primitives are facts and dimensions.
Facts represent measurements of the business orrecord events. Dimensions are attributes used toconstraint, group or browse facts. On a relationalimplementation these primitives are representedby interrelated tables whose structure is usuallydescribed by a star schema [6,10,12,15]. Thus,the correct understanding and identification offacts, dimensions and their interrelationships arekey to precisely design a data warehouse thatconforms to the users’ data requirements andanalytical needs. To achieve this, it is veryimportant an effective user participation whendeveloping a data warehouse application.
In this paper, we present MD2, a tool based onthe dimensional data modeling approach, whichfacilitates the user participation in several stepsinvolved in the development of a data warehouseapplication. MD2 assists users in defining theiranalytical needs and modeling the application.The tool is supported by a data repository thathelps gather metadata used to build theapplication dimensional schema and to specifyreports that can be generated from the datawarehouse and visualized through a Webinterface. In addition to describing the mainfacilities provided by our tool, we also discuss,based on the Business Dimensional Lifecycle[10], a case study that illustrates how the tool isused to develop a data warehouse application.
The remainder of the paper is organized asfollows. Section 2 discusses related work.Section 3 presents an overview of MD2. Section4 discusses the interaction of MD2 with theBusiness Dimensional Lifecycle methodology.The case study is described in Section 5. Finally,Section 6 presents our conclusions.
2 Related Work
The concepts and techniques related to datawarehouses have been widely discussed in theliterature [4,7,10,11]. In what follows, we focusour attention on the dimensional model and onmethodologies that adopt this approach (or someof its variations) to design, develop, and deploydata warehouse applications.
The dimensional model has been extensivelyaddressed in the literature. A method for derivingdimensional schemas from traditional entity-relationship schemas is presented in [15]. Thedimensional fact model, proposed by Golfarelliet al. [5], provides a graphical notation forrepresenting data warehouses based on thedimensional model. The mapping of ER schemasinto this model is discussed in [6]. Kripperndorfand Song [12] address the problem of mapping astar schema into an ER schema, and Song et al.[18] analyze many-to-many relationshipsbetween fact and dimension tables. An extensionof the ER model for capturing the dimensionalparadigm is presented in [17].
Several methodologies for data warehousedesign have also been proposed. The design ofdata warehouses according to the dimensionalfact model is described in [7]. Object-oriented-based methodologies are presented byGiovinazzo [4] and by Trujillo et al. [19]. In [8],the authors propose an approach to datawarehouse design that resembles the traditionaldatabase design process. Like in [1] and [2], theyadvocate that at least a conceptual or logicalmodeling activity should precede the actualimplementation of a data warehouse.
In [10] and [11], Kimball et al. provide a detaileddiscussion of the dimensional model conceptsand use. In [10], a complete framework forapplying the dimensional modeling approach,called Business Dimensional Lifecycle, isdescribed. From the authors’ perspective,requirements definition constitutes a fundamentalstep as it has a major impact on the wholeproposed lifecycle. They suggest interviews andfacilitated sessions as the basic techniques forthis step. The difficulty of identifyingdimensions is also addressed by Kimball in [9],
and considered to be mostly a data warehousedesigner’s decision and responsibility.
To face the problem of decision supportrequirements being indeterminate and unstable,Lambert [13] presents suggestions on how to getusers more involved, but yet with no directsupport for this interaction. Firestone [3]proposes a dimensional object modelingapproach, which strongly relies on informationsystem use cases tightly coupled with strategicgoals and objectives. According to the author,“the use case provides a view of the system fromthe view point of its users and their businesspurposes, … and it follows that the whole systemarchitecture will be determined by the use casesand ultimately by the users who specified them.”
Although many of these works consider and evenemphasize the user involvement on early stagesof requirements analyzes, little effort has beenput on developing tools to directly support thistask. Many commercial solutions for softwarecomponents of data warehouse environmentsaddress the definition and specification ofdimensions, fact variables and hierarchies.Modeling tools like Computer Associates'sAllFusion ERwin Data Modeler, Oracle's CASE2000 and MicroStrategy's Architect havecomprehensive features to represent and describestar and snowflake schemas, but they do notinclude facilities for the participation of end-users in early stages of development. The samehappens with ETL systems and OLAP tools, likeSAS Administrator, Microsoft DTS andMicrosoft OLAP Services, which contemplatelater stages of data warehouse development.Thus, MD2 may be seen as an attempt towardssupporting the user involvement in thedevelopment of data warehouse applications.
3 Overview of MD2
In this section, we present an overview of theMD2 tool. MD2 has been designed to workalong with other tools that comprise a typicaldata warehouse architecture. Figure 1 depicts theMD2 environment, illustrating how its maincomponents and the actors involved are relatedto each other. This environment is brieflydiscussed next.
As we can see from Figure 1, we distinguishthree actors in this scenario. The informationanalyst is responsible for the development of thedata warehouse application. He is the one whoinputs the MD2 data repository with apreliminary definition of the dimensions,hierarchies, facts, and attributes identified for theapplication. We notice that sometimes this rolemight be played by the data warehouse designertoo. The expert user is an application specialistthat is responsible for structuring the contents ofthe data repository according to a hierarchy ofapplication related themes. In doing so, he alsohelps to identify new elements (dimensions,hierarchies, facts, and attributes) that are relevantto the application. He is also responsible for lateranalyzing the data warehouse contents (usingsome specific data access tool) in order to createand generate application reports that will bestored in the Web server to be made availablethrough a Web interface. This interface is alsogenerated by the expert user based on thethematic structures defined in the MD2 datarepository. When performing these tasks, boththe information analyst and the expert user maybe assisted by a number of specific reportsgenerated by MD2. Finally, the application user(or end-user) might be any person that, throughthe Internet or a company Intranet, accesses theapplication reports using a Web browser.Application users play an important role in this
environment because they help the expert user toidentify the hierarchy of themes according towhich the data repository is structured.
A more detailed discussion of how the MD2 toolis used to assist the development of datawarehouse applications is postponed to Section4. In what follows, we describe the MD2 datarepository and present an overview of the MD2user interface.
3.1 The Data Repository
The MD2 data repository is used tohierarchically represent the application data interms of thematic structures. Each thematicstructure corresponds to a domain (main subject)that is hierarchically composed of structuredsubjects, optional composed facts, and items.These components are called structural elements.In addition to structural elements, independentelements are used to compose or characterizeother elements. The independent elementsrepresented in the data repository are: subjects,facts, attributes, dimensions, dimensionhierarchies, reports, and report types. Except forsubjects, reports and report types, theseindependent elements correspond to the usualdimensional modeling primitives [10,11]. Detailsof how such thematic structures and thecorresponding elements are created andmanipulated in the data repository are given next.
MD2Tool
Expert User
Information Analyst
WebBrowser
Application User
Web Server
Data AcessTool
Expert User
Application Reports
Web Interface
Data Metadata
Generation
Generation
ReadingChecking
Data Repository
Data
Definitions of ThematicStructures, Attributes,
Facts, Dimensions,Hierarchies, and Reports
MD2 Reporfts
Definitions of Attributes,
Facts, Dimensions and
Hierarchies
MD2 Reports
Information Analyst
DW/DM
Figure 1: The MD2 environment.
MD2
File
Exit
Edit
Copy
Cut
Paste
Delete
Property
Insert
Structure
Domain
Structed Subject
Composed Fact
Item
Subject
Attribute
Fact
Dimension
Dimension Hierarchy
Report Type
Report
Structure
Update
Expand
Shrink
Show Description
Show Report
Tools
Checking
Web Interface...
Production Version
Options
Tabs
Structure
Domain
Subject
Attribute
Fact
Dimension
Dimension Hierarchy
Report Type
Report
MD2 Reports
Lists of
Domains
Subjects
Facts
Items
Attributes
Dimensions
Conformed Dimensions
Conformed Facts
Associated Facts andDimensions
Checking Report
Help
About
Figure 2: MD2 function menu diagram.
3.2 The User Interface
MD2 provides a Windows-like graphicalinterface for user interaction. This interface hasbeen implemented using standard Visual Basicclasses [14] and includes function bars, pull-down menus, and specific function buttons.Figure 2 presents a diagram describing the tool’sfunction menu. The upper level functions (File,Edit, Insert, Structure, Tools, Tabs, Reports, andHelp) correspond to general functions that areincluded in the main screen bar. Clicking on anyof these functions pops-up a pull-down menu forselecting a specific function.
The thematic structures and the elementsdescribed in the data repository (domains,subjects, facts, attributes, dimensions, dimensionhierarchies, reports, and report types) are createdand further manipulated by selecting specifictabs. For instance, by selecting the (Structure) tab (see Figure 3), the user can createor modify a thematic structure by inserting,updating, and deleting its structural elements(domains, structured subjects, composed facts,and items). In addition, this function also allowsthe user to visualize some of the independentelements that might be associated with thethematic structure (reports associated with anitem, dimensions associated with a report,dimension hierarchies associated with adimension, and attributes associated with adimension hierarchy).
Analogously, by selecting any other tab the userhas access to the corresponding element screen.For example, Figure 4 shows the fact screen thatis selected by clicking on the (Fact) tab (seeFigure 3). Notice that when a fact in the list boxis selected, the fact properties (name, description,type, default aggregation rule, etc.) are presentedin the text boxes below. The buttons (Insert), (Update Properties),and (Delete) allow the user to,respectively, insert a new fact, update theproperties of an existing fact, and delete anexisting fact.
Figure 3: Elements of a thematic structure.
Figure 4: Fact screen
The menus (Tools) and (MD2 Reports) provide additional facilities forsupporting the work of the information analystand of the expert users when modeling theapplication. These facilities include functionsfor checking the consistency of thematicstructures and elements represented in the datarepository, as well as for producing analyticreports to support some of the steps in the datawarehouse developing process and generating aWeb interface to publish the application reportsthat will be extracted from the data warehouse.Figure 5 presents the Web interface generatedfor the structure shown in Figure 3. Theinterface includes a navigation area and apresentation area. The navigation area displaysthe thematic structure allowing the user tonavigate through its constituent elements.When an item is selected, the associated reportis displayed in the presentation area. Thecommand bar includes buttons for expandingand shrinking the navigation structure,activating the floating menu, and selecting thenavigation mode (by item or dimension).
Figure 5: The Web interface
4 MD2 and the BusinessDimensional Lifecycle
Developing a data warehouse application isa complex and time-consuming processthat involves several steps. The MD2 toolaims at supporting some of these steps. Toillustrate this, in this section we brieflydiscuss how it can be used in the context of
the framework provided by the BusinessDimensional Lifecycle. Figure 6 describes thebasic steps of this methodology as proposed in[10]. The shadow boxes correspond to themethodology steps in which MD2 can beapplied.
In the Business Requirement Definition step,the tool can be used by the expert user toexpress, in terms of the dimensional model, hisdata analysis needs, facilitating a betterunderstanding of the facts, dimensions,hierarchies, and attributes involved in theapplication. It also allows other thematicstructures to be queried, revealing newelements that might be useful for data analysis.
The Dimensional Modeling step can beconsiderably facilitated by the contents of theMD2 data repository input in the previous step.From the thematic structures described in thedata repository, the information analyst canidentify possible data marts. As a firstassumption, we assume that the last level ofeach subject corresponds to a data mart. It isalso possible to identify associated facts anddimensions, descendents of a structuredsubject, and dimensions used to aggregate afact. Table 1 lists the MD2 reports that canhelp execute the activities of this step.
In the Data Staging Design and Developmentstep, some MD2 reports (e.g., List of Facts,List of Attributes, and List of Dimensions) canbe used to help the information analyst createthe data staging area metadata, especially thedefinition of attribute sources and formulas,
algorithms and transformations rules forattributes and facts.
MD2 can also assist some activities during theEnd-User Application Specification andDevelopment steps. Particularly, the Webinterface generated based on the thematicstructures described in the data repository canbe seen as an end-user application. Informationin the List of Reports, especially on dimensionsand related hierarchy attributes, report types,constraints and parameters, can help thespecification of new application reports. Inaddition, other MD2 reports listing elementssuch subjects, dimensions, attributes and factscan help populate metadata for data accesstools.
Finally, in the Maintenance and Growth step,the expert user can input in the data repositorynew elements that represent new requirementsidentified for the application.
5 Case Study
In this section, we briefly describe a case studyin which the MD2 tool was used to design anddevelop a small data warehouse application.This application is a data mart prototypedeveloped for BHTRANS, the Transport andTraffic Authority of Belo Horizonte, the thirdlargest city in Brazil. The use of MD2 in thisapplication followed the steps of the BusinessDimensional Lifecycle, as discussed in theprevious section.
Project Plannig
BusinessRequeriment
Definition
TechnicalArchitecture
Design
End-UserApplication
Specification
DimensionalModeling
ProductSelection &Instalation
PhysicalDesign
Data StagingDesign &
Development
End-UserApplication
Development
DeploymentMaintenance &
Growth
Project Management
Figure 6: Steps of the Business Dimensional Lifecycle.
MD2 Reports
Activities in the Dimensional Modeling step of theBusiness Dimensional Lifecycle
Dim
ensi
ons
Fac
ts
Rep
orts
Con
form
edD
imen
sion
s
Ass
ocia
ted
Fac
tsan
d D
imen
sion
s
Sub
ject
s an
dD
omai
ns
Att
ribu
tes
Building the matrix of data marts and dimensions X
Defining the fact table granularity and selection of thedimensions according to the defined granularity
X X
Selecting facts X X
Drawing the fact table diagram X
Detailing the fact tables X X
Detailing the dimension tables X X
Mapping data from source to target X X X
Table 1: MD2 reports and activities in the Dimensional Modeling step.
This data mart corresponds to the RequestRecord, a subject related to the domain CitizenEnquiry. The main data source for thisapplication is the system that records therequests, suggestions and complains from thecity’s transport system users. These requests,suggestions and complains, generally known asRequest Records (RRs), are recorded fromdistinct sources (e.g., phone calls, letters,electronic forms, etc.) and stored in an Oracledatabase. An RR is classified according to itssubject and type of occurrence. In addition, everyRR is associated with an identified user. An RRcan be sent to distinct administrative sectors andwhile it has not yet received an answer it isconsidered as pendent. At any time, it is possibleto follow the status of an RR. When an RR isclosed, its final status is recorded in the databaseand an answer is sent to the user who made therequest, suggestion or complain from which ithas been issued.
Several users from different administrativesectors of BHTRANS participated in thedevelopment of this application. Afteraccomplishing the Project Planning step (seeFigure 6), MD2 was used next in the BusinessRequirement Definition step for expressing, interms of the dimensional model, the application
data analysis needs. The main dimensionalmodel elements were identified by analyzing thedatabase entity-relationship schema, and theattributes, facts, dimensions and hierarchiesidentified were input in the MD2 data repositoryby the information analyst. Figure 7 illustrates aform used to insert the
. This fact has AVG (average) as its defaultaggregate rule. In addition, since it is onlyconsidered for RRs that have been closed, aconstraint is imposed on it.This fact has also been classified as (Derived) because it is the result of a formula.
Figure 7: Insert Fact Form.
Figura 8: An application thematic structure.
After the input of the elements in the datarepository, the application thematic structure wasdefined by an expert user. Figure 8 presents partof this structure that corresponds to the domainCitizen Enquiry and to the first level subject
, which has as its descendants thesubjects and
. Figure 8 also illustrates how thecomposed facts (nodes F) are defined in terms ofitems (nodes I), which reports (nodes R) arerelated to each item, and the dimensions (nodesDI), hierarchies (nodes H) and attributes (nodesAt) that compose each item. For instance, thesubject includes three of
composed facts: , and
.
Then, the Dimensional Modeling step wascarried out. In addition to information gatheredfrom interviews in the previous step, severalMD2 reports were used to help build thedimensional schema. Particularly, the reportAssociated Facts and Dimensions was used tovalidate the schema consistency and the reportConformed Dimensions was used to check thedimensions related to the two fact tables. Figure9 depicts the application dimensional schema.The use of MD2 reports in the Data StagingDesign and Developing step was not significantin this application.
In the End-User Application Specification andDevelopment steps, several MD2 reports wereused to help input metadata for the OracleDiscoverer [16] data access tool. For this task,the List of Reports was especially important toidentify which dimensions and respectivehierarchies should be extracted from the datawarehouse to generate the application reports.The reports generated by Oracle Discovererwere then stored in the Web server for lateraccess by end-users. To make possible the accessto these reports, the Web interface was generatedby an expert user. Figure 10 illustrates anapplication report generated through OracleDiscoverer and displayed by the Web interface.
<<Fact>>R R Analys is
<<D im ens ion>>A ns wer T ype
<<D im ens ion>>Solution
<<D im ens ion>> R egional A dm in is tration
<<D im ens ion>>D is tric t
<<D im ens ion>>S treet
<<D im ens ion>>Bus Line
<<D im ens ion>>R R
<<D im ens ion>>S ourc e
<<D im ens ion>>T im e <<D im ens ion>>
R R T ype<<D im ens ion>>
R R S ubjec t<<D im ens ion>>
U ser T ype
<<Fact>>Pendency A nalys is
<<D im ens ion>>Adm in is trative S ector
<<D im ens ion>>Pendency T ype
C losing
O penning
OpenningClosing
Oco
rrenc
y Loc
ality
Use
r
User
O corrency Locality
O co rrency Loc ality
C losing
O pen ning
Q uantityR esponse T im e
Q uantityR espons e T im e
Figura 9: Dimensional schema of the data mart Request Record.
Figura 10: HTML report generated through Oracle Discoverer and displayed by the Web interface.
6 Conclusions
Any successful data warehouse initiative isfirst and foremost dependent on its businessusers. They must be engaged in the projectfrom the very beginning as well as theirexcitement and motivation maintained alongthe whole development process. Most of all,they play a fundamental role during therequirements analysis and must be activelyinvolved on the elucidation of relevant factsand perspectives.
We have presented in this paper MD2, a toolbased on the dimensional modeling approachthat assists the users’ participation in thedevelopment of data warehouse applications.The tool is supported by a data repository andprovides a user friendly graphical interface thathelps to gather metadata for describing thedimensional schema as well as to specifyreports that can be generated from the datawarehouse and visualized through a Webinterface. In addition, the tool includes anumber of reports that help users perform theiractivities throughout the process of developinga data warehouse application.
To illustrate the use of MD2, we havediscussed a case study within the framework ofthe Business Dimensional Lifecycle [10]. Thiscase study has been based on a data martprototype developed for BHTRANS, the
Transport and Traffic Authority of BeloHorizonte, and emphasizes the userparticipation in some of the key steps of thismethodology. However, it should be pointedout that the use of MD2 is not centered on theBusiness Dimensional Lifecycle steps and,therefore, the tool can be used with any otherdata warehouse design methodology based onthe dimensional modeling approach, such as[8] and [15]. Particularly, MD2 might be usedto capture the dimensional modeling elementsderived from an entity-relationship schemaaccording to the method proposed in [15].
As the MD2 tool is conceived to complementand interoperate with other tools usually foundon data warehouse environments, theadherence to metadata standards [4] currentlybeing discussed by OMG and other groups is afundamental requirement. We believe the MD2data repository metamodel can be easilyadapted in the future to any unified standardthat might result from current metadatastandard initiatives.
References
[1] Agrawal, R., Guppta, A., and Sarawagi,S. Modelling MultidimensionalDatabases. Proceedings of the ThirteenthInternational Conference on DataEngineering, Birmingham, UK, 1997, pp.232-243.
[2] Cabibbo, L., and Torlone, R. A LogicalApproach to MultidimensionalDatabases. Proc. of the 6th Int’lConference on Extended DatabaseTechnology, Valencia, Spain, 1998, pp.187-197.
[3] Firestone, J.M. Dimensional ObjectModeling. Executive InformationSystems, Inc., White Paper n. 7, April1998 (available at http://www.dkms.com/DOM.htm).
[4] Giovinazzo, W. A. Object-Oriented DataWarehouse Design. Prentice Hall, NewJersey, NJ, 2000.
[5] Golfarelli, M., Maio, D., and Rizzi, S.The dimensional fact model: a conceptualmodel for data warehouses. InternationalJournal of Cooperative InformationSystems 7, 2-3 (1998), 215-247.
[6] Golfarelli, M., Maio, D., and Rizzi, S.Conceptual Design of Data Warehousesfrom E/R Schemas. Proc. of the 31st
Hawaii Int’l Conference on SystemSciences, Vol. VII, Kona, Hawaii, 1998,pp. 334-343.
[7] Golfarelli, M., and Rizzi, S. Designingdata warehouses: key steps and crucialissues. Journal of Computer Science andInformation Management 2, 3 (1999).
[8] H semann, B., Lechtenb rger, J., andVossen, G. Conceptual Data WarehouseDesign. Proc. of the Int’l Workshop onDesign and Management of DataWarehouses, Stockholm, Sweden, 2000,pp. 6.1-6.11.
[9] Kimball, R. Mystery Dimensions.Intelligent Enterprise Magazine 3, 5(March 2000).
[10] Kimball, R., Reeves, L., Ross, M., andThomthwaite, W. The Data WarehouseLifecycle Toolkit: Tools and Techniquesfor Designing, Developing andDeploying Data Warehouses. John Wiley& Sons, New York, 1998.
[11] Kimball, R. A Dimensional Modeling
Manifesto. DBMS 10, 9 (August 1997).
[12] Krippendorf, M., and Song, I.-Y. TheTranslation of Star Schema into EntityRelationship Diagrams. Proc. of theEighth Int’l Workshop on Database andExpert Systems Applications, DEXA'97,Toulouse, France, 1997, pp. 390-395.
[13] Lambert, B. Break Old Habits To DefineData Warehousing Requirements. DataManagement Review (December 1995).
[14] Microsoft. Visual Basic Developer’sGuide. 1999.
[15] Moody, L.D., and Kortink, M.A.R. FromEnterprise Models to DimensionalModels: A Methodology for DataWarehouses and Data Mart Design. Proc.of the Int’l Workshop on Design andManagement of Data Warehouses,Stockholm, Sweden, 2000, pp. 5.1-5.12.
[16] Oracle. Oracle Discoverer 3.1 Release3.1: Administration Guide. 1998.
[17] Sapia, C., Blaschka, M., H fling, G., andDinter, B. Extending the E/R Model forthe Multidimensional Paradigm. Proc. ofthe Int’l Workshop on Data Warehousingand Data Mining, Singapore, 1998, pp.105-116.
[18] Song, I.-Y., Rowen, W., Medsker, C.,and Ewen, E. An Analysis of Many-to-Many Relationships Between Fact andDimension Tables in DimensionModeling. Proc. of the Int’l Workshop onDesign and Management of DataWarehouses, Interlaken, Switzerland,2001, pp. 6.1-6.13.
[19] Trujillo, J., Palomar, M., G mez, J., andSong, I.-Y. Designing Data Warehouseswith OO Conceptual Models. IEEEComputer 34, 12 (2001), 66-75.
[20] Tryfona, N., Busborg, F., andChristiansen, J. starER: A ConceptualModel for Data Warehousing Design.Proc. of the ACM Second Int’l Workshopon Data Warehousing and OLAP, KansasCity, MI, 1999, pp. 3-8.
