Capitalizing Big Data Value in an Enterprise

Embed Size (px)

Citation preview

  • 7/30/2019 Capitalizing Big Data Value in an Enterprise

    1/5

    IntroductionWe have had lot of talk around SOA, Mobility and Cloud Computing in recent time. And with social

    networking adoption by consumers and enterprises, we are seeing impact of enterprise social

    collaboration to this mix. In reality though, all these technologies feed each other to create a range ofnew capabilities that could potentially benefit enterprises in some way or other. So if we look diligently,

    all of these enterprise applications, consumers, and other IT assets rely upon and create huge dataBig

    Data!!!

    In the sections below, we will see what is Big Data and how do we use the output of Big Data analysis in

    an enterprise to generate events and in turn trigger business processes to achieve high productivity,

    increase in utilization, reduction in cost, business process improvement and higher customer

    satisfaction.

    What is Big Data?I am sure all of you would have heard about Big Data which is nothing but data; however its huge,

    generated by applications which are accessible and actionable with the help of social collaboration or

    smart communication systems. Cloud computing makes the management of Big Data easy in terms of

    deployment and capacity. There are two distinct factors that can be considered: First, commoditization

    of the ability to process large data sets. Second, the use of cloud computing platforms.

    With the advent of Big Data and other related technologies, we have seen a paradigm shift in terms of

    having more focus on the business concepts, applications and goals, and using the technology as a tool

    to achieve those goals. In fact, there was a report in recent time stating that leading companies are

    using big data analytics to gain competitive advantage. It predicted a 60% margin increase for retail

    companies who are able to harvest the power of Big Data.

    Lets look at an example where the comparison of unstructured data and relying on structured data lead

    to improvement in customer satisfaction. The consumers are facing issues with their GPS units and they

    post the same in some forums or on their social media profiles. If the manufacturer has some means of

    gathering and analyzing that information and loop back to take necessary steps to take care of issue

    proactively then that can surely improve customer satisfaction.

    Big Data AnalyticsAs seen earlier, the "Big Data" which is generated on large scale usually has one or more of the following

    characteristics:

    Very large data volumes measured in terabytes or petabytes Variety of structured, unstructured and semi-structured data High velocity, rapidly changing data

  • 7/30/2019 Capitalizing Big Data Value in an Enterprise

    2/5

    Also, we know that using traditional database management and BI tools is difficult and uneconomical as

    datasets grow very large. At the same time, if we analyze big data appropriately and get value out of it

    then it would help analysts, data scientists and business users to make better decisions using

    information that was previously inaccessible to them.

    The ability to store, aggregate, and combine huge amounts of data, and perform insightful analytics onthe results, has finally become more accessible and cost-effective the technical and economic barriers

    are falling fast. Companies that use Big Data to gain business insight and take action will outperform

    their peers.

    Major pain point that enterprises are facing is with traditional database vendor to get the

    technology enabled for processing huge data sets for any business purpose. Now, with the

    introduction of Apache Hadoop and other related technologies, which provides a divide and

    conquer approach called MapReduce to the huge data, we now have the Big Data problem

    solved using an open source solution. Hadoop, when used with commodity hardware, is able to

    process terabytes or even petabytes of data in a matter of seconds or minutes which used to

    take hours or days with traditional database technology and models.

    As stated earlier, the whole Big Data setup becomes interesting when coupled with cloud

    computing wherein the cloud providers provide access to hundreds of servers that may be

    provisioned on demand to support the distribution of processing needed for Big Data analytics.

    With the following advantages of the available technologies, we can visualize the true power of

    those technologies.

    Making big data more accessible in a timely manner Using data and experimentation to expose variability and improve performance Segmenting populations to customize actions Replacing and supporting human decision-making with automated algorithms Innovating new business models, products, and servicesInstead of just generating report after report, Big Data systems can automatically trigger

    changes to core business processing based on the analysis of the data sets. At the same time,

    there is an ability to check all operational data directly, instead of dealing with small,

    specialized databases set up specifically to support BI.

    Unlike conventional BI, Big Data, the value is discovered through a refining modeling process:make a hypothesis, create statistical, visual, or semantic models, validate, and then make a new

    hypothesis. It either takes a person interpreting visualizations or making interactive knowledge-

    based queries, or by developing machine learning adaptive algorithms that can discover

    meaning.

    http://hadoop.apache.org/http://hadoop.apache.org/http://hadoop.apache.org/
  • 7/30/2019 Capitalizing Big Data Value in an Enterprise

    3/5

    Integrating Big Data Analysis with event driven SOA

    Once you've collected all that data, what to do with it? For Example from CRM perspective, Big

    Data will not be an exercise in merely collecting massive amounts of this data; rather it will be

    about making the right information accessible and action-oriented for both the company and

    the customer for core CRM. But at the same time, we should be cognizant of the fact that there

    are down side of using Big Data, while analytics are great, they is no replacement for a human

    touch. You'll still need to interact with current and potential customers, whether in person,

    over the phone, or via some other one-on-one connection. Relying too much on Big Data

    analytics has got a risk of losing the personal approach to selling.

    From Big Data analysis perspective, we need to keep few key points in mind before we could

    use Big Data analysis for a given use case,

    - As the input data has unstructured data realms and due to the fact that the data setsare huge, we wont be able to move the raw data directly to a data warehouse.

    However, after MapReduce processing we may integrate the reduction result into the

    data warehouse environment so that we can leverage conventional BI reporting,

    analytics and use the output in an enterprise.

  • 7/30/2019 Capitalizing Big Data Value in an Enterprise

    4/5

    - Also, for many of the use cases, we need to capture constantly changing andunpredictable data. And to analyze that data, we would need a new and separate

    architecture.

    - In some use cases, the analysis cannot be done without combining the structured data,so to get a better result, Big Data analytics requires context and understanding.

    - Another important aspect to keep in mind is data governance. If we apply strict dataprecision rules, we might lose the value hidden in the Big Data.

    - To get better results and reduce the risk of redundant investment for handling big dataanalysis, better option is to incorporate the Big Data results into existing Data

    Warehousing platform.

    - This integration between Big Data sources and existing data warehousing platform isvery critical component which should extend across all of the data types and domains,

    and bridge the gap between traditional and new data acquisition and processing

    framework.

    - The analytics platform should have an ability to access structured information alongwith the output of Big Data analysis to get better value.

    Lets look at various use cases wherein we can use the actionable output of Big Data analysis.

    One of the use cases where the output of Big Data analysis can be used is Complex Event

    Processing to trigger business process events. Mostly to achieve the complex event processing,

    its difficult to analyze the large amount of data however with HDFS and MapReduce, its

    possible to incorporate all the detailed data points and use as part of complex processing. In

    conjunction with HDFS and MapReduce, NoSQL can also be used to capture and store low

    latency and large volume of data from various sources in a flexible data structure. Both of these

    can be integrated with advance analytics engine to produce automatic alerts and can trigger the

    business processes to take appropriate actions.

    The above diagram shows that an output from an application of Big Data Analysis on the large

    amount of real-time and static data allows enterprises to trigger business process as

    appropriate. And even though its not shown in this diagram however the processing

    transactions and results are stored in database which could be looped back to Big Data analysis.

    Most of the enterprises have different types of business processes defined like processes which

    are pre-defined using services which are centrally orchestrated as part of typical SOA platform

    and other processes which are triggered through events that occur across, or outside of,

    specific business process. These complex events with a pattern of activities both random and

    scheduled should trigger a set of services being used in these business processes to achievespecific business goal.

    Event driven SOA has an ability to generate high-level business events which triggers the

    business processes from numerous low-level system events. As stated earlier, these events are

    generated by filtering static and dynamic data.

  • 7/30/2019 Capitalizing Big Data Value in an Enterprise

    5/5

    Another example wherein an enriched event like an escalation in the rate of customer shopping

    cart abandonment in last few days, then by analyzing the data, analytical engine could send a

    notification to the marking department to initiate research into what competitors are doing and

    what are the prices being offered by the competition.

    Another example where cancellation of a big order in manufacturing could result in surplus andloss of revenue. However if captured by analytical engine as part of real-time data analysis,

    could trigger marketing department to initiate a special sales campaign that would resell the

    excess capacity and reducing the loss of revenue.

    In nut-shell, business processes triggered by events generated from big data analytics should

    directly support revenue growth with cost containment, responsiveness to business conditions,

    improve customer satisfaction or ability to pursuenew market opportunities. These resultingbusiness processes could also measure operational progress toward achieving goals, control

    operational costs by communicating just what is needed to just who needs to know, or report

    performance status of key processes to key decision makers.

    References1. http://eyeonibm.com/2012/01/03/big-data-driving-soa-driving-more-big-data/2. http://www.infosysblogs.com/infytalk/2012/02/data_-_structured_or_unstructu.html#more3. http://www.ebizq.net/web_resources/whitepapers/BPM-SOA_wp.pdf(page 12)4. http://www.talend.com/products/enterprise-di.php5. http://www.pentaho.com/big-data/6. http://en.wikipedia.org/wiki/Event-driven_SOA7. http://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-

    2012.00111.png

    8. http://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdf

    http://eyeonibm.com/2012/01/03/big-data-driving-soa-driving-more-big-data/http://eyeonibm.com/2012/01/03/big-data-driving-soa-driving-more-big-data/http://www.infosysblogs.com/infytalk/2012/02/data_-_structured_or_unstructu.html#morehttp://www.infosysblogs.com/infytalk/2012/02/data_-_structured_or_unstructu.html#morehttp://www.ebizq.net/web_resources/whitepapers/BPM-SOA_wp.pdfhttp://www.ebizq.net/web_resources/whitepapers/BPM-SOA_wp.pdfhttp://www.talend.com/products/enterprise-di.phphttp://www.talend.com/products/enterprise-di.phphttp://www.pentaho.com/big-data/http://www.pentaho.com/big-data/http://en.wikipedia.org/wiki/Event-driven_SOAhttp://en.wikipedia.org/wiki/Event-driven_SOAhttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdfhttp://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdfhttp://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdfhttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://en.wikipedia.org/wiki/Event-driven_SOAhttp://www.pentaho.com/big-data/http://www.talend.com/products/enterprise-di.phphttp://www.ebizq.net/web_resources/whitepapers/BPM-SOA_wp.pdfhttp://www.infosysblogs.com/infytalk/2012/02/data_-_structured_or_unstructu.html#morehttp://eyeonibm.com/2012/01/03/big-data-driving-soa-driving-more-big-data/