BI - cdn.ttgtmedia.comcdn.ttgtmedia.com/BeyeNETWORK/downloads/BI+Trends...box remains very real. Organiza - tions with storage space to spare are carving out small slices of their

Data SanDboxeS Help analyStS Dig Deep intoCorporate info Page 3 x

MiSConCeptionS HolDing baCk USe of Data integration toolS Page 7

figHt tHe fearS tHreatening tHe riSe of Data ViSUalization Page 10

BITrends + Strategies

august 2012, volume 1, issue 8insight and advice on Bi, analytics and data warehousing

In their own little data warehouse playgrounds, analytics pros toy around with data sets—and unearth valuable insights.

BI Trends + sTraTegIes • augusT 2012 2

sand has a lot of negative connota-tions. You can get stuck in it. Im- polite people might tell you to go pound it. You can perish in the parched desert. And of course, there’s the Wicked Witch of the West and her deadly hourglass timer—feel free to join me in shud-dering at the thought of it. On the other hand, there’s the beach—and the sandbox, that quintessential source of childhood fun.

For analytics professionals, the concept of digging around in a sand-box remains very real. Organiza-tions with storage space to spare are carving out small slices of their data warehouses and analytical databas-es where data scientists, business analysts and other users can store, manipulate and analyze specific data sets. In the lead story in this issue of BI Trends + Strategies, Search-BusinessAnalytics.com’s Nicole Laskowski reports on the benefits of these data sandboxes and the issues that companies should consider

before setting them up.Next, consultant Rick Sherman,

writing for SearchDataManagement.com, assesses the continuing reli-ance on manually coded data inte-gration scripts at many companies. Sherman’s take: Misconceptions about data integration software are leading many IT managers to stick with inefficient SQL coding process-es instead of taking advantage of automated tools.

And BeyeNETWORK expert Lee Feinberg addresses the organiza-tional resistance that can sometimes be triggered by efforts to use data visualization technology to distrib-ute business intelligence data more widely in companies. Feinberg’s advice for BI managers facing such resistance is to be flexible—but stick to your guns. Craig StedmanExecutive Editor, SearchBusinessAnalytics.com and SearchDataManagement.com

playing in SanD Still in Style for analytiCS proS

HoMe

eDitor’S letter

Data SanDboxeS Help analyStS Dig Deep into Corporate info

MiSConCeptionS HolDing baCk USe of Data integration toolS

figHt tHe fearS tHreatening tHe riSe of Data ViSUalization

Editor’s LEttEr


HoMe

eDitor’S letter




data sandboxEs HELp anaLysts dig dEEp into CorporatE info

ou need to think big when you think about eBay Inc.’s auction and shopping website; for example, picture 100

million site users, 300 million active items, 50,000 product categories and an average of $2,100 worth of goods sold every second. The same applies if you think of eBay as a data management and business analytics company: It generates 50 terabytes of data a day and supports efforts to analyze that data by 7,500 business users and analysts.

“We can become swamped if people are asking for different views of the data—different reports or dashboards,” said Chris Rogaski, eBay’s senior director of analytics application technology, during a pre-sentation at the Gartner Business Intelligence Summit in Los Angeles

in April. “We needed to get ahead of that … so that our business analysts and product managers can make data-informed decisions.”

The San Jose, Calif., company has taken steps to help it stay in front of the user demand. Its data analytics platform is composed of a Teradata-based enterprise data warehouse (EDW) that stores struc-tured transaction data; a separate “deep storage” Teradata database, called Singularity by eBay, that holds semi-structured data such as analy-ses of the behavior of site users; and a Hadoop system for unstructured data, including the raw user behavior data, other forms of machine-gener-ated info and text. The three pillars provide about 90 petabytes of stor-age space, Rogaski said.

In addition, eBay is liberally hand-ing out virtual data marts inside the

Data SanDboxeS Help analyStS Dig Deep into Corporate infoGiving analytics professionals control of small amounts of space in data warehouses lets them manipulate and experiment with data sets—in a controlled and managed environment. By Nicole Laskowski

Y



HoMe

eDitor’S letter




EDW to employees who want to explore, manipulate and even add to specific data sets on their own. The data marts are part of the company’s Analytics as a Service, or A3S, pro-gram for users involved in analyzing data. Using a tool created by eBay’s IT department, business users and data analysts can apply for, and are usually granted, 100 GB of space—giving them what are known in busi-ness intelligence (BI) circles as data sandboxes to play around in.

Sandboxes are walled-off areas that keep experimentation with data separate from the production database environment. At eBay, users have access to the data in the EDW and can copy information that they want to analyze into their data marts. And with the help of a sec-ond eBay-developed tool, they can upload additional data to work with. “If people have a new data source we don’t know about, we can’t be in the way of that data becoming a part of their analysis,” Rogaski said.

Family Feud Frustrates analytics eFFortsThe long-standing feud between the IT department and the business in many organizations is well docu-mented. It can be chalked up partly to differing priorities: While business users have pressing business prob-lems to resolve, IT teams are tasked with governing the use of data and maintaining data quality standards.

For analytics professionals looking to dig deep into the most current data that’s available, the divide can be a source of frustration.

Often, “analysts need data that’s not yet in the data warehouse,” said Wayne Eckerson, a BI consultant

and research director for TechTar-get Inc.’s business applications and architecture media group. “It’s not there because it hasn’t been sourced or it’s not yet loaded.”

In other cases, he said, data ana-lysts may view the BI and analytics tools deployed by their companies as inflexible compared with Excel—leading them to go their own way by surreptitiously setting up Excel-based spreadmarts outside of IT’s purview. But stretching Excel across the enterprise for data analysis uses is hardly ideal, Eckerson added: “Everyone knows analysts deliver valuable information, but organiza-tions cannot run on spreadsheets.”

That’s where data sandboxes come into play, according to Eck-erson. He said sandboxes can help bring spreadmarts and other so-

“ organizationS Cannot rUn on SpreaDSHeetS.” —Wayne eckerson BI consultant and research director, TechTarget Inc.



HoMe

eDitor’S letter




called data shadow systems out of the dark corners of an organization by ensuring that analytics users have access to the data they need and can exert some level of control over the information.

For BI and IT managers, a well-managed data sandbox offers a safe place for users to experiment with

corporate data inside a company’s data management infrastructure. It’s an environment “that is not stor-ing the primary copy of the data but is storing [information] in a format suitable for analysis,” said Gordon Linoff, founder and principal of con-sultancy Data Miners Inc. in New York and co-author of Data Mining

HaDoop SySteMS offer a HoMe for SanDboxeSwith petabytes oF data storage space available across three high-powered analytics platforms, ebay has ample wiggle room before it needs to start worrying about the virtual data marts that it sets up for data analysts and other users affecting the per-formance of its enterprise data warehouse (EdW) system. but for many other compa-nies, performance issues could be a valid concern with data sandboxes, and a reason to put them outside of an EdW.

one alternative location is a standalone data mart. a Hadoop system is another. “Most people don’t use the term sandbox when implementing Hadoop,” said Wayne Eckerson, research director for techtarget inc.’s business applications and architec-ture media group. “but in many ways, companies want to do data mining and explora-tion there.”

the open source distributed computing technology is free-standing but can be con-nected to data warehouses to exchange data, and Hadoop clusters should be able to provide space for data scientists and other skilled analytics users who may be taking up valuable computing resources in an EdW system, Eckerson said. He cautioned, though, that people using Hadoop sandboxes will have to be adept at manipulating the Mapreduce programming framework and familiar with related technologies such as Hive and pig.

yet another possible sandbox host, said gordon Linoff, founder and principal of new york-based consultancy data Miners inc., is “a separate system running sas or spss—analytics tools that are not database-oriented and are designed more for stat-isticians.” n


HoMe

eDitor’S letter





Techniques: For Marketing, Sales and Customer Relationship Management.

Data sandboxes can be construct-ed in data warehouses and analyti-cal databases or outside of them as standalone data marts. In eBay’s case, hosting sandboxes as virtual data marts inside the EDW keeps data movement down and reduces the need for users to make copies of data and store them in other sys-tems, Rogaski said.

best when analyzed by this dateHe acknowledged that a “minimal” amount of data duplication occurs as users populate their sandboxes. “But it happens, and that’s a cost of the way we’re doing business,” he said. To decrease the instances of duplication, eBay uses an expi-ration date system, with analysts typically setting an end date for their use of a data set. Once the limit is reached, Rogaski’s team confers with analysts before expunging their data from the system—a process that eBay refers to as “garbage collection.”

Because sandboxes by their very nature involve playing with data, Linoff believes that having the right skills is an important part of a suc-cessful deployment. Data miners and other users may need to manipulate data and analyze what they’re look-ing at on the fly. “This is about learn-ing new things,” he said. “And you need the skill set to make use of it.”

That may be a good rule of thumb for many businesses but not for all. Rogaski said one of its goals is to make its BI and analytics data acces-sible to “a wide swath of people.” Even a business user “who really just wants to be told what they need to know” can apply for a virtual data mart, he added.

Managing usage was one of the big challenges that Eckerson cited for organizations looking to set up data sandboxes. For example, he said that before users distribute any reports containing unique views of the data they’re working with to other people, the manipulated infor-mation should be checked by the corporate BI team to make sure the metrics are correct and no errors have crept into the data.

“You can give users access [to data], but you also have to give them some guidelines,” Eckerson said. “They don’t like restrictions, but if they’re going to use corporate resources, they have to agree to certain things.” n

Data SanDboxeS Can be ConStrUCteD in Data WareHoUSeS anD analytiCal DatabaSeS or oUtSiDe of tHeM aS StanD-alone Data MartS.


MisConCEptions HoLding baCk UsE of data intEgration tooLs

here’s no question that data integration tech-nology is a good thing. So why aren’t business-es using it as much as

they should be? Data integration software has

evolved significantly from the days when it primarily consisted of extract, transform and load (ETL) tools. The technologies available now can automate the process of integrating data from source sys-tems around the world in real time if that’s what companies want. Data integration products can also increase IT productivity and make it easier to incorporate new data sources into data warehouses and business intelligence (BI) systems for users to analyze.

But despite tremendous gains in the capabilities and performance

of data integration tools, as well as expanded offerings in the market-place, much of the data integration projects in corporate enterprises are still being done through manual cod-ing methods that are inefficient and often not documented. As a result, most companies haven’t gained the productivity and code-reuse benefits that automated data integration pro-cesses offer. Instead, they’re deluged with an ever-expanding backlog of data integration work, including the need to continually update and fix older, manually coded integration programs.

Even large companies that use automated tools to integrate and load data into their enterprise data warehouses are still relying on homegrown SQL scripts to load data marts, online analytical processing cubes and other data structures used

MiSConCeptionS HolDing baCk USe of Data integration toolSMany organizations continue to rely on manual integration processes that are often inefficient. Automated tools have matured greatly, but IT managers don’t always see that. By Rick Sherman

THoMe

eDitor’S letter





HoMe

eDitor’S letter





in BI applications. And as you might expect, small and medium-sized businesses aren’t widely using inte-gration tools.

I think the biggest reason more organizations aren’t taking advan-tage of data integration technology is they don’t fully understand what it can do. Let’s clear up some miscon-ceptions about that.

stuck in the integration pastMany IT managers don’t realize how far data integration software has come in recent years. And it did have a long way to go. The first generation of ETL tools were simple code gen-erators that were expensive and had limited functionality. Lots of com-panies that evaluated them found it more effective to develop their own custom integration code.

Second-generation ETL products offered more functionality, but they were primarily batch-oriented and didn’t perform well. Based on those two sets of tools, many people in IT were left with the feeling that ETL software wasn’t worth the effort to learn and wouldn’t be able to meet their performance needs.

But what IT professionals should realize is that the current generation of data integration offerings consists of full-fledged suites that include ETL, enterprise application integra-tion, real-time integration and data virtualization functionality as well as data cleansing and data profiling

tools. The suites can support data integration processes in traditional batch mode or in real or near real time through the use of Web ser-vices. Built-in best practices can help improve both the software’s performance and user productivity. Meanwhile, vendors specializing in technologies such as data virtualiza-tion and complex event processing have emerged to offer more targeted alternatives to the suites. At this point, there’s no good reason to be stuck in the past about the capabili-ties of automated integration tools.

sQl not the answer to all integration QuestionsAnother common misperception is that manual SQL coding is sufficient to perform all data integration tasks. Although there is no shortage of people who can crank out SQL code, the reality is that data integration is often a much more complex under-taking than merely writing a sequen-tial string of SQL statements. Manu-ally written integration scripts can be laborious to create and usually do not scale or age well.

Over the years, developers with extensive experience working for software vendors have designed sophisticated workflows and data transformation routines to handle the myriad types of data integration that most enterprises need. IT and data management pros doing man-ual coding typically don’t have the


HoMe

eDitor’S letter





same level of experience. In essence, instead of leveraging reusable inte-gration workflows and transforms, these SQL coders are starting from a blank slate on each project.

Another issue is that IT depart-ments generally do not invest in effective data integration training. This is a problem, even when IT groups do choose to use automated tools. Although they may engage in tool training, they neglect to learn data integration best practices or make an effort to fully understand how integration processes work. Without that kind of understanding, companies can’t maximize the value of their data integration tools. Some end up just going back to relying on manual coding.

Because of these misconceptions,

data integration in many organiza-tions continues to be laborious and time-consuming—much more so than it needs to be. To make matters even worse, enterprises can’t truly leverage the data at their disposal, and they often are forced to invest in upgrading and expanding their IT infrastructures to support ineffective or inefficient data integration pro-cesses.

Fortunately, there are many capa-ble tools available today and plenty of IT pros and consultants who are well-versed in sound integration techniques and practices. But com-panies have to recognize that there’s a problem, and viable solutions to that problem, before they can take advantage of what’s out there to help break the data integration backlog. n

Mapping oUt a better Way forWarD on Data integrationthere are some simple steps that companies can take to get out of the data integration quandary many find themselves in. for example, they should do the following:

n stop manual coding and start using data integration tools.

n invest in proper training on data integration architecture, design and best practices.

n get experienced people on board to design data integration processes and mentor the it staffers who will be doing the automated development work. n


HoMe

eDitor’S letter




figHt tHE fEars tHrEatEning tHE risE of data VisUaLization

ou Finally have the soft-ware and hardware required to enable the use of data visualization technology through-

out your company. You also have a vision of how reports and end-user dashboards with built-in visualiza-tions will generate new levels of insight that can help improve busi-ness performance.

But as your work begins to attract attention, you may be confronted with a range of new behaviors that can threaten to slow progress. Stay alert: “Fears” about making business intelligence (BI) data more broadly available can suddenly emerge after long being suppressed by existing organizational controls over the dis-tribution of information.

The fact is that most organizations are still unable to effectively analyze

and put into action the data in their BI systems and data warehouses. Yet they continue to operate under cen-tralized control models that closely regulate how data is handled, used and shared. Faced with the prospect

of BI data being unleashed so more business users have access to it, pro-ponents of tight controls cite risks such as data security and system performance problems as well as a

figHt tHe fearS tHreatening tHe riSe of Data ViSUalizationIt’s time to change the way your company thinks about disseminating BI data to a broader set of business users, particularly in visual formats. But be prepared for resistance. By Lee Feinberg

YMoSt organizationSare Still Unable to effeCtiVely analyze anD pUt into aCtion tHe Data in tHeir bi SySteMS anD Data WareHoUSeS.


HoMe

eDitor’S letter





lack of analysis skills among the tar-geted new users. But as companies continue to place considerably more value on their data, the real threat is if the control level increases.

As a BI manager, you must take a leadership role to help your organi-zation understand and get past these new fears. But you cannot face them down alone. You must work closely with your peers—yes, even the ones who prefer the control model. Set aside your beliefs that the fears are being created or cultivated by any one person or group. Focus on building a path forward by working to remove existing barriers and pro-

moting new ideas.As part of that process, you’ll likely

find yourself needing to do the fol-lowing:

n be understanding. Your company’s processes, operations and beliefs have developed over many years, and now you’re working to create a foundational shift, a breakthrough beyond making a single business team successful through data visu-alization. Be sensitive and observe reactions; explore what’s behind opinions and fill in where there is a lack of information. You really need to collaborate at the personal level to

source: decIsIonVIz

Figure 1: BI benefits are proportional to the number

of business users who can view, analyze and make decisions based on BI data.


HoMe

eDitor’S letter





uncover and understand motivations.n be courageous. Turn risks and fears into opportunities. Don’t debate whether the risks are real or perceived. Foster open conversations that get to the root of the actual risks. You are likely to find that some fears just fade away while you build concrete agreements on how best to handle others. But don’t back down: Maintain an “abundance mind-set” that the more people who are con-nected with BI data, the better.

Together with others in the organi-zation, you need to consider who can view and analyze the data and use it to make decisions, and then isolate the cases in which justified data access limits are needed (see figure 1, page 11).

n be flexible. Even with executive sponsorship, organizations do not like to change how they do things, or at least they don’t change easily. Since you are forging new ground and disturbing the status quo, you may feel it’s necessary to create a detailed plan to prove how the data visualization strategy will unfold in a

successful way. Instead, be agile, because you will encounter both good and bad reactions that you didn’t consider in advance. Demon-strating your own team’s ability to

adapt helps create a natural atmo-sphere of cooperation. Develop a high-level roadmap and take your first step with the end goal in mind. And remember: There is no perfect first step. But the sooner you start the change process, the sooner you can show some positive results.

n be a risk taker. Evangelize the role of data visualization in a much broader context than its contribu-tion to the output of reports and dashboards. Emphasize that data has zero value sitting in a database unused. Help your company see that fully integrating data visualiza-tion capabilities into a broadened BI strategy can create opportunities to improve the corporate bottom line. Your message should be clear: It’s time to consider new possibilities. n

DeMonStratingyoUr oWn teaM’S ability to aDapt HelpS Create a natUral atMoSpHereof Cooperation.

You can find more articles by Lee Feinberg in his Beye NETWORK Expert Channel.

http://www.b-eye-network.com/channels/5628/

http://www.b-eye-network.com/channels/5628/


HoMe

eDitor’S letter




aboUt tHE aUtHors

BI Trends + Strategies is a joint e-publication of BeyeNETWORK, SearchBusinessAnalytics.com

and SearchDataManagement.com.

Jason sparapani Managing Editor, E-Publications

Jean schauer Editor in Chief, BeyeNETWORK

Craig stedman Executive Editor,

SearchBusinessAnalytics.com and SearchDataManagement.com

Linda koury Director of Online Design

Mike bolduc Publisher

[email protected]

Ed Laplante Director of Sales

[email protected]

techtarget 275 Grove Street, Newton, MA 02466

www.techtarget.com

© 2012 TechTarget Inc. No part of this publication may be transmitted or reproduced in any form or by any means without written permission from the publisher. TechTarget reprints are available through

The YGS Group.

About TechTarget: TechTarget publishes media for information technology professionals. More than 100 focused websites enable quick access to a deep store of news, advice and analysis about the technologies, prod-ucts and processes crucial to your job. Our live and virtual events give you direct access to independent expert com-mentary and advice. At IT Knowledge Exchange, our social community, you can get advice and share solutions with

peers and experts.

Nicole Laskowski is the news editor for SearchBusinessAnalytics.com. She covers business intelligence, analytics and data visualization technologies and trends. Email her at [email protected].

Rick Sherman is the founder of Athena IT Solutions, a consultancy in Stow, Mass., that focuses on BI, data integra-tion and data warehousing. He is also an adjunct faculty member at Northeastern University’s Graduate School of Engineer-ing, and he offers independent analysis on his blog, The Data Doghouse. Email him at [email protected].

Lee Feinberg is the founder of manage-ment consultancy DecisionViz, which helps companies develop and implement data visualization strategies and process-es. Feinberg is also a frequent speaker at conferences in the U.S. and other coun-tries. Email him at [email protected].

http://www.b-eye-network.com/

http://searchbusinessanalytics.techtarget.com/

http://searchdatamanagement.techtarget.com/

http://reprints.ygsgroup.com/m/techtarget

http://searchbusinessanalytics.techtarget.com/

mailto:[email protected]



Documents

BI - cdn.ttgtmedia.comcdn.ttgtmedia.com/BeyeNETWORK/downloads/BI+Trends...box remains very real. Organiza - tions with storage space to spare are carving out small slices of their