Data warehouse in_reliance

Corporate ProfileType : Private Conglomerate

Founded : 1966 as Reliance Commercial Corporation

Headquarters : Navi Mumbai, Maharashtra, India

Key People : Anil Ambani, Chairman

Industry : Energy, Telecommunications, Media

Products : Electricity, Wireless Services, Movies

Revenue : Crosses Rs. 200,000 Crore

Net Profit : Rs. 16,236 Crore

Total Assets :

Employees :

Website : http://www.relianceadagroup.com/

RELIANCE COMMUNICATION Ltd | Prepared by Er. S.Syed Nizamudeen,

up to 23rd March, 10

Subsidiaries


Reliance Entertainment

Reliance Capital

Reliance Communication

Reliance Infrastructure

Reliance Power

Reliance Health

Reliance Natural Resources Limited

Definition of DWA subject-oriented, integrated, time-variant, non updatable collection of data used in support of management decision-making processes

•Subject-oriented: e.g. customers, patients, students, products•Integrated: Consistent naming conventions, formats, encoding structures; from multiple data sources•Time-variant: Can study trends and changes•Non-updatable: Read-only, periodically refreshed

--Jeffrey A. Hoffer, Mary B. Prescott, Fred R. McFadden

Modern Database ManagementPrentice Hall, 7th Edition


AbbrevationsCDR Call Detail RecordSMP Shared-memory MultiprocessorNUMA Non Uniform Memory AccessRISC Reduced Instruction Set ComputingIP Internet ProtocolTB Tera Byte(1 TB = 1024 Giga Bytes)DTH Direct to HomeIPTV Internet Protocol TelevisionZFS Zettabyte File SystemSATA Serial Advanced Technology AttachmentETL Extract, Transform and LoadRAID Redundant Array of Independent DisksBI Business IntelligenceH / W HardwareI / O Input / OutputDW Data Warehouse


DW-ing ScopeBroadly the project may involve any or all of the following •Sizing and database design •Define the data warehousing Model (generally a star schema) •Populating the data warehouse using ETL (or ELT) •Representing the data to the users using BI tools •Creating of aggregation content and optimization It is very important to clearly define what concepts form a part of this project.


Designing DW•Problem Definition Do I need a Data Warehouse?What Specific Problems will it Solve?What are my available Resources(time, money, personnel)?What criteria will I use to measure success?Should I outsource all, some or none of the development & operation?Am I upgrading and existing system, converting from a legacy system, or developing from scratch?•Requirement Analysis•Designing and PrototypingRapid PrototypingStructured Development•Development and Documentation•Test and Review•Deployment and Training•Operation, Enhancement and Help Desk


Problem DefinitionDo I Need a Data Warehouse?

This is an obvious question, but should be taken seriously. For some businesses the answer is an easy yes, but for others, there may be better solutions. It is wise to seek professional advice when answering this question. Having an outside professional examine the question may cost some money up front, but they are far less costly early on in the development.


Problem DefinitionWhy do I Need a Data Warehouse?

•Many inconsistencies are identified and resolved before loading of information in data warehousing. This makes the reporting and analyzing process simpler.

•Data warehousing enhances the value of operational business applications and customer relationship management systems.

•Data warehousing also leads to proper functioning of support system applications like trend reports, exception reports and the actual performance analyzing reports.

Precisely, a data warehouse system proves to be helpful in providing collective information to all its users. It is mainly created to support different analysis, queries that need extensive searching on a larger scale.


Problem DefinitionWhat specific Problems will it solve? Write down the four, five, ten, whatever, number of problems that having a data warehouse will solve. The problems should be clearly stated, be very specific, and have testable criteria for success. Make sure you publicize these problems and get user and management feedback.


Problem DefinitionWhat are my available resources (time, money, and personnel)?

Knowing what your actual resources are at the beginning is critical for defining the development path. If your budget is low, consider down-scaling the effort. If time is short, consider using off-the-shelf products extensively. If your personnel resources are thin, consider outsourcing. Being realistic about your actual resources will help you prevent overruns and project disappointments. Promising a full-blown data warehouse when you only have resources for a database, will always doom a project. Also, don't be afraid to tell upper management that the resources are too small for solving the problem. Believe me, they would rather know up front than get a surprise during deployment.


Problem DefinitionWhat criteria will I use to measure success?

This is an often overlooked step in the problem definition. For every problem stated, you must define a means for determining the success of the solution. If you can't think of success criteria, then the problem is not defined specifically enough. Stay away from problem statements such as "The data warehouse must hold all our accounting data." Restate the problem in quantifiable terms, like "The data warehouse must handle the current 20GB of accounting data including all metadata and replicated data with an expected 20% growth per year."


Problem DefinitionShould I outsource all, some, or none of the development and operation?

If you have in-house personnel that are under-utilized or have time to be assigned to the development process, then keeping most of the development in-house makes sense. If not, then you can either hire additional staff or outsource some or most of the development. I recommend that you do not outsource all of the development. You must have some in-house expertise available or at least strong upper management support. The outsourcing contractor can supply the needed expertise and personnel at the various development phases. And when a particular phase is finished, you are not left with a staff member looking for something to do.


Problem DefinitionAm I upgrading an existing system, converting from a legacy system, or developing from scratch?

Developing a data warehouse from scratch, as strange as it sounds, is by far the easiest. If you are in this situation, count your blessings. If not, upgrading existing databases or converting from one or more legacy systems will be your lot. Fortunately, you will have a long list of "things that don't work right" to begin with. Make sure that you fully understand what systems will still be in place after the migration and how they will be integrated into your intranet. If your budget is low, then consider using middleware and "web-like" products to layer on top of the existing system. Remember that computer hardware, memory, and disk space are cheap. It's the software and operations that are expensive. Powerful hardware can make even today's bloated software work faster. With a higher budget, consider migrating legacy databases to a single powerful commercial database.


Requirement Analysis1) Clearly state the problem(s) you wish to solve. 2) Identify all data sources and formats. 3) Identify the users of the completed system. 4) Formulate a specific budget - time, money, personnel. 5) Ask identified users to specifically state what they expect the system to do. 6) Ask management to specifically state their success criteria 7) Separate their requirements from their "desirements." Only design to

requirements. The enhancement phase is where you address the "desirements." 8) Group and "bubble-up" requirements. 9) Generate a prioritized requirements table listing the requirement, where it came

from, the success criteria, and priority. Keep this table high-level. A table with a dozen requirements will be much easier to manage than one with hundreds.

10)Produce a detailed development schedule including hardware, software, personnel, documentation, and reviews. Include outsourcing requirements and long lead-time items.

11)Get a sign-off of the requirements, resource allocation, and schedule from top management before you go any further.


Data ModelingInformation and Data modeling, along with the definition of the metadata, is the single most important activity in the design of a data warehouse. If this step is done correctly, success is almost ensured. If not, then areas such as flexibility, scalability, and usability will suffer. Information modeling is best left to the experts. I recommend that you hire a professional information modeling firm or consultant. The consultant should be brought in at the tail end of the requirements analysis and be kept though the design and prototyping and then be called in as necessary during the development and documentation phases. Don't skimp on this phase or you will regret it.


Designing PrototypingRapid Prototyping There are 5 keys to a successful rapid prototyping methodology: 1)Assemble a small, very bright team of database programmers, hardware technicians, designers, quality assurance technicians, documentation and decision support specialists, and a single manager. 2)Define and involve a small "focus group" consisting of users (both novice and experienced) and managers (both line and upper). These are the people who will provide the feedback necessary to drive the prototyping cycle. Listen to them. 3)Generate a user's manual and user interface first. You will be amazed at what you will find out by producing a user's manual first! 4)Use tools specifically designed for rapid prototyping. Stay away from C, C++, COBOL, SQL, etc. Instead use the visual development tools included with the database. 5)Remember a prototype is NOT the final application. Prototypes are meant to be copied into production models. Once the prototypes are successful, then begin the development processing using development tools, such as C, C++, Java, SQL, etc.


Designing PrototypingStructured Development

When a project has more than 10 people involved or when multiple companies are performing the development, a more structured development management approach is required. Note that rapid prototyping can be a subset of the structured development approach. This approach applies a more disciplined approach to the data warehouse development. Documentation requirements are larger, quality control is critical, and the number of reviews increases. While some parts may seem like overkill at the time, they can save a project from problems, especially late in the development cycle.


Development & DocumentationOnce the requirements analysis is well underway, the prototypes are working, and the focus groups are becoming happy, it's time to begin the development. Coordinating hardware and software purchases and upgrades, server and hardware installation, software and database development, documentation guides and manuals, reviews, and testing can become a full-time job. The key to keeping a handle on all of this to maintain a good written schedule that everyone can view and to have periodic "all-hands" reviews. Remember that working with vendors can be a frustrating experience. Hardware incompatibilities, data format incompatibilities, software bugs, late deliveries, etc. are more the norm than the exception. Outsourcing can help, but you must be continually involved to ensure success.


Test & ReviewTesting and Reviews take place throughout the development cycle, including prototyping, development, deployment, operations, and enhancements. It never ends. It's wise to place a single individual in charge of testing and reviews. This is not a popular job, but it is critical for developing a system that works and meets each of the requirements. Be sure to empower this person (usually a quality assurance engineer) with the appropriate authority. Also, provide them with an appropriately sized staff. Testing is time consuming, tedious work and preparing for reviews and analyzing results can take much longer than you might think. Fortunately this person can save you from being surprised at budget review time and usually catches most problems before they become too big. If you outsource this task, make sure that you make it clear to the others on the team what the outsourcer's role is and what level of authority they have.


Deployment & TrainingOK, the development is complete, quality assurance is satisfied, the documentation is ready, and all the "off-the-shelf" products have arrived. Now it's time to put everything together. This can be a highly disruptive time. Make sure that you have full management support and that they understand the nature and effect of the installation and deployment disruption. Scheduling training sessions concurrently with the installation can be an effective use of time. Don't skimp on the training. Make sure you have training in the budget from the beginning and don't dip into it. The best way to ensure success is to effectively train the users so that they will actually use the system and possibly sing its praises. Also remember that training is ongoing. New employees or employees being moved or promoted will need to be trained. Each time enhancements are added, new training sessions must be scheduled.


OperationData warehouses usually contain two or more servers. Tasks such as backups, bug fixes, software updates, hardware maintenance and upgrades, media services, account maintenance, security patches, and other similar tasks must be performed regularly. Operation and maintenance of such services requires an operations staff. It is not enough to "let the users take care of it." If you are providing these services in-house then you will need on-site support from either an outsourcing agency or in-house staff. The current trend is to outsource most of these services. More and more companies are outsourcing the entire data warehouse and access it via the Internet or private network. Outsourcing can result in a substantial savings. Just make sure that your provider can supply the services you require and is available when you need them. Also, be sure to discuss security requirements with them before you hire them.


EnhancementThere is always one thing you can count on: "Requirements Creep." The more successful the data warehouse, the faster requirements creep will occur. As your users become more sophisticated they will want more and more capabilities. If you can respond quickly and efficiently, your users will again sing your praises (and upper management will definitely take notice :-). Make sure that you have designed in the ability to add features from the very beginning. Remember to design in scalability and flexibility at all phases of development.


Help DeskYou might think that good manuals and good training would be sufficient to effectively use your data warehouse. Not so. A knowledgeable, available, responsive help desk is critical to the overall success of the project. Users will always find new uses for a well-designed system and problems will inevitably occur. Without a help desk, a data warehouse can become dated and under-utilized. In my experience, deployment of an excellent help desk (with telephone, fax, online, and e-mail capabilities) is the single most important function that ensures the continued success of a data warehouse.


Choosing H / W for DWPicking your data warehousing hardware is actually quite easy. Typical needs include:

•Scalable, multi-processor server platform•Sufficient memory for parallel processing•Highly scalable disk storage system•Sufficient disk cache for > 50% hit ratio•Striping for better I/O performance•Data redundancy for data protection•Support for both fast reads and writes


Continued…Combined with our proposed software architecture requirements for:

•Single server

•Single instance

•Single database

•Oracle 8i or 9i

•RISC-based UNIX

•Parallel load programs


Continued…The recommended minimum data warehousing hardware platform is:•SMP or NUMA server

•8 / 6 64-bit RISC CPUs

•True 64-bit UNIX OS

•4 / 8 GB RAM

•Mid- to large-sized disk array

•2 / 4 GB disk cache

•RAID 0+1 via hardware

•18GB or 36GB disk drives


Implementing DW1. Recognize that the job is probably harder than you expect.

Experts frequently report that 30-to-50 percent of the information in a typical database is missing or incorrect. That situation may not be noticeable -- or may even be acceptable -- in an operational system that focuses on swiftly and accurately processing current transactions. But that percentage of error is totally unacceptable in a data warehousing system designed to sort through millions of historical records to identify trends or select potential customers for a new product. And, even when the data is correct, it may not be usable in a data warehouse environment.


Implementing DW2. Understand the data in your existing systems.

The first step in any data warehousing project is to perform a detailed analysis of the status of all databases that will either contribute -- or potentially contribute -- to the data warehouse. An important part of understanding the existing data is determining interrelationships between various systems. Interrelationships must be maintained as the data is moved into the warehouse. In addition, the data warehouse implementation often involves making changes to database schema. You must have a clear understanding of data relationships among heterogeneous systems to determine -- in advance -- how any change may impact the system. Otherwise, it's possible for changes to create inconsistencies that ripple across the entire enterprise, creating enormous headaches.


Implementing DW3. Be sure to recognize equivalent entities.

One of the most important aspects of preparing for a data warehousing project is identifying equivalent entities and heterogeneous systems. The problem arises because the same essential piece of information may appear under different field names in different parts of the organization. For example, two different divisions may be servicing the same customer but have the name entered in a slightly different manner (such as AIG and American International Group). You can use a data transformation product that's capable of fuzzy matching to identify and correct this and similar problems.


Implementing DW4. Use metadata to support data quality.

Metadata is crucial to a successful data warehousing implementation. By definition, metadata is data about data, such as the tags that indicate the subject of a web document. There are many types of metadata that can be associated with a database to characterize and index data, facilitate or restrict access to data, determine the source and currency of data, etc. One major challenge is trying to synchronize the metadata between different vendor products, different functions, and different metadata stores. A major data center might easily have a 500-page schema layout or COBOL copybooks with 200,000 or more lines. But the chances are that many undocumented changes were made to minimize the work involved in dealing with historical data.


Implementing DW5. Select the right data transformation tools.

Data transformation tools extract data from the operational sources, clean it, and load it into the data warehouse while capturing the history of that process. This transformation process may include the creation and population of new fields from the operational data, summarizing data to an appropriate level for analysis, performing error checking operations to validate the integrity of the data, etc. Look for a tool that makes it possible to map data from source to target with a simple point-and-click interface. It's also useful to track and manage the relationships of interrelated data entities that are affected when changing database structures. Finally, try to find a tool that can capture and store metadata during the conversion process.


Implementing DW6. Take advantage of external resources.

External sources of information, such as data from a customer's transaction processing systems, or market research data provided by a third party, can greatly increase the value of internal information. Rather than simply comparing sales against the same month last year, using external data might make it possible, for example, to compare sales growth against the increase in the overall market. Or suppose that you want to estimate the income of each of your customers. You may be able to obtain a database containing the average income for every ZIP code in the country. This won't provide accurate answers for individual customers, but it will let you obtain a good estimate of an aggregate grouping, such as everyone from 30-to-45 years old in Kansas, that might be used in a targeted marketing campaign.


Implementing DW7. Use new information distribution methods.

The biggest technological improvements in recent years in the data warehousing field have come in the area of information delivery. In the past, highly skilled analysts were needed to prepare information in the format requested by users. Today, information can be delivered in several different ways directly to people who need it. Users can subscribe to regular reports and have them delivered via e-mail. If the report contains data that meets the subscription criteria, a customer-filtered view of the data can be delivered to the user by means of an e-mail attachment. The report data stays securely and economically on the server, and only the pages requested by the authorized user are sent on demand. Another option is to let the user log in, search for data, and open the reports that provide the needed information. Only the pages requested by the user are sent over the network to minimize traffic.


Implementing DW8. Focus on high-payback marketing applications.

Most of the really hot applications in data warehousing right now involve marketing because of the potential for an immediate payback in increased revenues. For example, catalog manufacturers are using data warehousing to match personal characteristics to purchases of specific items, and then they use this empirical data to produce special demographic editions that generate higher sales because they're targeted more closely to customers' needs.


Implementing DW9. Emphasize early wins to build support throughout the organization.

The wide range of off-the-shelf solutions has made it possible to drastically reduce cost and lead-time requirements for data warehousing applications. Off-the-shelf solutions won't usually complete project objectives, but they often can be used to provide point solutions in a short time that serve as a training and demonstration platform and, most importantly, build momentum for full-scale implementation. Even for the largest-scale applications, you should perform technology surveys to maximize the use of pre-built technology.


Implementing DW10. Don't underestimate hardware requirements.

The hardware requirements for data warehousing database servers are high. The primary reason is the large number of CPU cycles required to slice and dice data over and over again to meet the varying needs of users throughout the organization. Database size also plays in server performance requirements, ranging up to terabytes and higher. For those reasons, be sure to select a scalable platform regardless of how much headroom you have provided in your server specification. The typical data warehouse implementation starts out at the departmental level and grows over time to an enterprise-wide solution. Purchasing servers that can be expanded with additional processors is one possible approach. A more ambitious idea is to combine loosely-coupled systems that let companies spread the database over multiple servers, although the database still appears as a single entity to users.


Implementing DW11. Consider outsourcing your data warehouse development and maintenance.A large percentage of medium and large companies use outsourcing to avoid the difficulty of locating and the high cost of retaining skilled IT staff members. Most data warehousing applications fit the main criteria for a good outsourcing project -- a large project that has been defined to the point that it doesn't require day-to-day interaction between business and development teams. There have been many cases where an outsourcing team is able to make dramatic improvements in a new or existing data warehouse. Typically these improvements don't necessarily stem from an increased level of skill on the outsourcing team, but flow from the nature of outsourcing. The outsourcing team brings fresh ideas and perspective to their assignment, and is often able to suggest methods and solutions that they've developed and proven successful in previous assignments. The outsourcing team also doesn't have to deal with manpower shortages and conflicting priorities faced by the previous internal team.


Process of DW


Architecture of DW


Reliance DWAbout

Greenplum was co-founded in 2003 by Scott Yara, former VP for Sandpiper Networks, and Luke Lonergan, former CEO and founder at Didera

Products

•Greenplum Database •Greenplum's Single Node Edition •Greenplum Chorus


http://en.wikipedia.org/wiki/File:Greenplumlogotype.jpg


Reliance Communications World’s largest network roll-out covering almost a

billion people in India

World’s largest IP enabled optic fiber cable network - 230,000 kms

Pioneered wireless affordability in India

Submarine cable assetsOf 110,000 Rkms

Terrestrial fiber cable metro & intercity

Handset prices down from>$ 50 to <$ 20

Driven call rates down from >10 cents to <1 cent

per minute

Coverage increasing from 10,000 to 23,000 towns

Covering almost 100% of rail & roadway routes

Coverage + Affordability = Profitable growth

Launching nationwide GSM services


Robust Infrastructure to Support Customers

• More than 1500 Million CDRs per day

• More than 1.5 million eRecharge per day

• More than 1,800 TB data storage

• IT support in more than 6,000 towns for over 60,000 IT users

• Data ware house growth : more than 160 GB a day

• More than 7 lakh emails a day across group

IT Operations

New Products & Services

• DTH services to provide DVD quality picture & better viewing

experience

• IPTV to provide interactive TV & Video services, movies & music

on demand, interactive program guide and gaming

DW Systems - ReliancePreviously Used System

•S1004 model of Sun Data Warehouse Appliance

•S1004 integrates Greenplum Database with 4 Sunfire X4500 Systems as well as a Sunfire X4200 System

•All systems run Solaris Operating System

•Which includes ZFS, Suns’s Revolutionary File System

•The System Provide 20 CPU Cores

•192 SATA Drives and 40 Tera Bytes of usable database capacity


DW Systems - RelianceUpgrade

•Reliance simply added two X4500 systems to the existing S1004, to construct what is effectively an S1006 appliance.

•Today, the system supports 60 terabytes of usable capacity and 200 SATA drives with 28 CPU cores.

•Reliance’s two primary data sets represent over 28 terabytes of actual data, growing at a pace of 1 terabyte per week.


Sun Fire X4500 Server

Best Implementation of 2010PUNJAB NATIONAL BANK •5000+ branches, there was a need of creating a central data repository •gather data from various disparate systems and MIS reports/statements to facilitate decision making and exploit historical data •TCS is the Implementation Partner

Before Deployment•Reports/statements were collected manually or were obtained from the branches •Capturing the leads, and monitoring them was not possible.

After Deployment•5000 man days are being saved per year •With the implementation of the Dashboard, monitoring of branches along with focused follow-ups are being maintained. •Timely reports to statutory authorities, follow-ups of the alerts, and improvement in monitoring of transactions in accounts has also been facilitated.


DW Vendor Comparison


Thank You.


Education

Data warehouse in_reliance