Upload
mukundmulkar
View
213
Download
0
Embed Size (px)
Citation preview
7/30/2019 Capitalizing Big Data Value in an Enterprise
1/5
IntroductionWe have had lot of talk around SOA, Mobility and Cloud Computing in recent time. And with social
networking adoption by consumers and enterprises, we are seeing impact of enterprise social
collaboration to this mix. In reality though, all these technologies feed each other to create a range ofnew capabilities that could potentially benefit enterprises in some way or other. So if we look diligently,
all of these enterprise applications, consumers, and other IT assets rely upon and create huge dataBig
Data!!!
In the sections below, we will see what is Big Data and how do we use the output of Big Data analysis in
an enterprise to generate events and in turn trigger business processes to achieve high productivity,
increase in utilization, reduction in cost, business process improvement and higher customer
satisfaction.
What is Big Data?I am sure all of you would have heard about Big Data which is nothing but data; however its huge,
generated by applications which are accessible and actionable with the help of social collaboration or
smart communication systems. Cloud computing makes the management of Big Data easy in terms of
deployment and capacity. There are two distinct factors that can be considered: First, commoditization
of the ability to process large data sets. Second, the use of cloud computing platforms.
With the advent of Big Data and other related technologies, we have seen a paradigm shift in terms of
having more focus on the business concepts, applications and goals, and using the technology as a tool
to achieve those goals. In fact, there was a report in recent time stating that leading companies are
using big data analytics to gain competitive advantage. It predicted a 60% margin increase for retail
companies who are able to harvest the power of Big Data.
Lets look at an example where the comparison of unstructured data and relying on structured data lead
to improvement in customer satisfaction. The consumers are facing issues with their GPS units and they
post the same in some forums or on their social media profiles. If the manufacturer has some means of
gathering and analyzing that information and loop back to take necessary steps to take care of issue
proactively then that can surely improve customer satisfaction.
Big Data AnalyticsAs seen earlier, the "Big Data" which is generated on large scale usually has one or more of the following
characteristics:
Very large data volumes measured in terabytes or petabytes Variety of structured, unstructured and semi-structured data High velocity, rapidly changing data
7/30/2019 Capitalizing Big Data Value in an Enterprise
2/5
Also, we know that using traditional database management and BI tools is difficult and uneconomical as
datasets grow very large. At the same time, if we analyze big data appropriately and get value out of it
then it would help analysts, data scientists and business users to make better decisions using
information that was previously inaccessible to them.
The ability to store, aggregate, and combine huge amounts of data, and perform insightful analytics onthe results, has finally become more accessible and cost-effective the technical and economic barriers
are falling fast. Companies that use Big Data to gain business insight and take action will outperform
their peers.
Major pain point that enterprises are facing is with traditional database vendor to get the
technology enabled for processing huge data sets for any business purpose. Now, with the
introduction of Apache Hadoop and other related technologies, which provides a divide and
conquer approach called MapReduce to the huge data, we now have the Big Data problem
solved using an open source solution. Hadoop, when used with commodity hardware, is able to
process terabytes or even petabytes of data in a matter of seconds or minutes which used to
take hours or days with traditional database technology and models.
As stated earlier, the whole Big Data setup becomes interesting when coupled with cloud
computing wherein the cloud providers provide access to hundreds of servers that may be
provisioned on demand to support the distribution of processing needed for Big Data analytics.
With the following advantages of the available technologies, we can visualize the true power of
those technologies.
Making big data more accessible in a timely manner Using data and experimentation to expose variability and improve performance Segmenting populations to customize actions Replacing and supporting human decision-making with automated algorithms Innovating new business models, products, and servicesInstead of just generating report after report, Big Data systems can automatically trigger
changes to core business processing based on the analysis of the data sets. At the same time,
there is an ability to check all operational data directly, instead of dealing with small,
specialized databases set up specifically to support BI.
Unlike conventional BI, Big Data, the value is discovered through a refining modeling process:make a hypothesis, create statistical, visual, or semantic models, validate, and then make a new
hypothesis. It either takes a person interpreting visualizations or making interactive knowledge-
based queries, or by developing machine learning adaptive algorithms that can discover
meaning.
http://hadoop.apache.org/http://hadoop.apache.org/http://hadoop.apache.org/7/30/2019 Capitalizing Big Data Value in an Enterprise
3/5
Integrating Big Data Analysis with event driven SOA
Once you've collected all that data, what to do with it? For Example from CRM perspective, Big
Data will not be an exercise in merely collecting massive amounts of this data; rather it will be
about making the right information accessible and action-oriented for both the company and
the customer for core CRM. But at the same time, we should be cognizant of the fact that there
are down side of using Big Data, while analytics are great, they is no replacement for a human
touch. You'll still need to interact with current and potential customers, whether in person,
over the phone, or via some other one-on-one connection. Relying too much on Big Data
analytics has got a risk of losing the personal approach to selling.
From Big Data analysis perspective, we need to keep few key points in mind before we could
use Big Data analysis for a given use case,
- As the input data has unstructured data realms and due to the fact that the data setsare huge, we wont be able to move the raw data directly to a data warehouse.
However, after MapReduce processing we may integrate the reduction result into the
data warehouse environment so that we can leverage conventional BI reporting,
analytics and use the output in an enterprise.
7/30/2019 Capitalizing Big Data Value in an Enterprise
4/5
- Also, for many of the use cases, we need to capture constantly changing andunpredictable data. And to analyze that data, we would need a new and separate
architecture.
- In some use cases, the analysis cannot be done without combining the structured data,so to get a better result, Big Data analytics requires context and understanding.
- Another important aspect to keep in mind is data governance. If we apply strict dataprecision rules, we might lose the value hidden in the Big Data.
- To get better results and reduce the risk of redundant investment for handling big dataanalysis, better option is to incorporate the Big Data results into existing Data
Warehousing platform.
- This integration between Big Data sources and existing data warehousing platform isvery critical component which should extend across all of the data types and domains,
and bridge the gap between traditional and new data acquisition and processing
framework.
- The analytics platform should have an ability to access structured information alongwith the output of Big Data analysis to get better value.
Lets look at various use cases wherein we can use the actionable output of Big Data analysis.
One of the use cases where the output of Big Data analysis can be used is Complex Event
Processing to trigger business process events. Mostly to achieve the complex event processing,
its difficult to analyze the large amount of data however with HDFS and MapReduce, its
possible to incorporate all the detailed data points and use as part of complex processing. In
conjunction with HDFS and MapReduce, NoSQL can also be used to capture and store low
latency and large volume of data from various sources in a flexible data structure. Both of these
can be integrated with advance analytics engine to produce automatic alerts and can trigger the
business processes to take appropriate actions.
The above diagram shows that an output from an application of Big Data Analysis on the large
amount of real-time and static data allows enterprises to trigger business process as
appropriate. And even though its not shown in this diagram however the processing
transactions and results are stored in database which could be looped back to Big Data analysis.
Most of the enterprises have different types of business processes defined like processes which
are pre-defined using services which are centrally orchestrated as part of typical SOA platform
and other processes which are triggered through events that occur across, or outside of,
specific business process. These complex events with a pattern of activities both random and
scheduled should trigger a set of services being used in these business processes to achievespecific business goal.
Event driven SOA has an ability to generate high-level business events which triggers the
business processes from numerous low-level system events. As stated earlier, these events are
generated by filtering static and dynamic data.
7/30/2019 Capitalizing Big Data Value in an Enterprise
5/5
Another example wherein an enriched event like an escalation in the rate of customer shopping
cart abandonment in last few days, then by analyzing the data, analytical engine could send a
notification to the marking department to initiate research into what competitors are doing and
what are the prices being offered by the competition.
Another example where cancellation of a big order in manufacturing could result in surplus andloss of revenue. However if captured by analytical engine as part of real-time data analysis,
could trigger marketing department to initiate a special sales campaign that would resell the
excess capacity and reducing the loss of revenue.
In nut-shell, business processes triggered by events generated from big data analytics should
directly support revenue growth with cost containment, responsiveness to business conditions,
improve customer satisfaction or ability to pursuenew market opportunities. These resultingbusiness processes could also measure operational progress toward achieving goals, control
operational costs by communicating just what is needed to just who needs to know, or report
performance status of key processes to key decision makers.
References1. http://eyeonibm.com/2012/01/03/big-data-driving-soa-driving-more-big-data/2. http://www.infosysblogs.com/infytalk/2012/02/data_-_structured_or_unstructu.html#more3. http://www.ebizq.net/web_resources/whitepapers/BPM-SOA_wp.pdf(page 12)4. http://www.talend.com/products/enterprise-di.php5. http://www.pentaho.com/big-data/6. http://en.wikipedia.org/wiki/Event-driven_SOA7. http://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-
2012.00111.png
8. http://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdf
http://eyeonibm.com/2012/01/03/big-data-driving-soa-driving-more-big-data/http://eyeonibm.com/2012/01/03/big-data-driving-soa-driving-more-big-data/http://www.infosysblogs.com/infytalk/2012/02/data_-_structured_or_unstructu.html#morehttp://www.infosysblogs.com/infytalk/2012/02/data_-_structured_or_unstructu.html#morehttp://www.ebizq.net/web_resources/whitepapers/BPM-SOA_wp.pdfhttp://www.ebizq.net/web_resources/whitepapers/BPM-SOA_wp.pdfhttp://www.talend.com/products/enterprise-di.phphttp://www.talend.com/products/enterprise-di.phphttp://www.pentaho.com/big-data/http://www.pentaho.com/big-data/http://en.wikipedia.org/wiki/Event-driven_SOAhttp://en.wikipedia.org/wiki/Event-driven_SOAhttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdfhttp://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdfhttp://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdfhttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://blogs-images.forbes.com/davefeinleib/files/2012/07/Big-Data-Landscape-Jul-4-2012.00111.pnghttp://en.wikipedia.org/wiki/Event-driven_SOAhttp://www.pentaho.com/big-data/http://www.talend.com/products/enterprise-di.phphttp://www.ebizq.net/web_resources/whitepapers/BPM-SOA_wp.pdfhttp://www.infosysblogs.com/infytalk/2012/02/data_-_structured_or_unstructu.html#morehttp://eyeonibm.com/2012/01/03/big-data-driving-soa-driving-more-big-data/