BlueMix Hands-On Workshop - Meetupfiles.meetup.com/14755122/IBM BlueMix Workshop - Lab E - MapReduce Service _Big Data...deploy a BlueMix Java application that reads the data. The

BlueMix HandsLab E - Using the BlueMix MapReduce Service to build an IBMBig SQL application

Version : 3.00Last modification date : 05/Owner : IBM

BlueMix Hands-On WorkshopUsing the BlueMix MapReduce Service to build an IBM

.00/11/2014

IBM Ecosystem Development

Using the BlueMix MapReduce Service to build an IBM

Copyright IBM Corporation 2013-2014. All rights reserved. 3

Table of Contents

Lab E: Using the BlueMix MapReduce Service to build an IBM Big SQL application .................................... 5

1. Creating a BlueMix MapReduce Service instance............................................................................ 5

2. Uploading the social media data to the BlueMix MapReduce Service instance............................... 5

3. Importing the data into BigSheets ..................................................................................................... 7

4. Exporting the workbook data............................................................................................................. 8

5. Launch Eclipse to bring up the sample Big SQL application ........................................................... 9

6. Define the BlueMix Server ..............................................................................................................10

7. Push the BigInsights sample application from Eclipse ...................................................................11

8. Running the application...................................................................................................................13

9. Validating the data import using the BigSQL console.....................................................................16

Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix


Lab E: Using the BlueMix MapReduce Service to build an IBMBig SQL application

Lab Objectives: This lab will guide you to understand and use the sample application that is provided for the

BlueMix MapReduce Service. You will learn how to work with the BlueMix MapReduce cloud offering.

As a Hadoop developer, you can get started quickly by using the BlueMix MapReduce Service (that is powered by

IBM® InfoSphere® BigInsights™) to create big data applications.

There are many advantages of developing this type of application on BlueMix.

• You can avoid the overhead of acquiring and provisioning your own hardware cluster.

• You can create an application by combining multiple BlueMix services

• You can scale yourhardware environment as your application needs increase.

You will use InfoSphere BigInsights BigSheets component that comes with BlueMix Map reduce Service to learn how

to explore social media data. This data is a look at the global coverage of the “IBM Watson” brand. Then, you will

learn how to query the data by using IBM Big SQL, which is the InfoSphere BigInsights SQL interface. You will then

deploy a BlueMix Java application that reads the data. The result of this exercise is an application that you

create for your users to dynamically run queries on that data.

Lab Duration : 30 minutes

1. Creating a BlueMix MapReduce Service instance

In this section you'll create the instance of the BlueMix Map Reduce Service that will be used throughoutteh rest of this lab.

Step 1 In your browser go to the BlueMix URL https://ace.ng.bluemex.net and login if necessary

Step 2 Make sure you're in the Dashboard tab (if not click on the Dashboard link at the top of thepage to take you there)

Step 3 Scroll down to the Services section and click on Add a service

Step 4 Scroll down to the Big Data category and click on the icon for the MapReduce service

Step 5 Click on CREATE AND ADD TO APP

Step 6 For Add to: select [Do Not Associate] in the pulldown. Leave the default name.

Step 7 Click Create

Step 8 Your new service should now appear in the Services section

2. Uploading the social media data to the BlueMix MapReduce Serviceinstance

In this section you'll upload social media data about Watson to the instance of the BlueMix Map ReduceService that you just created

Step 1 In the Services section of your BlueMix Dashboard click on the hexagon for the service youjust created to see more information about it



Figure 1 MapReduce service icon

Step 2 You see the URL and the login information that you need to open the services underlyingInfoSphere BigInsights Console.

Step 3 Write down the Username assigned to this instance (note: it is case sensitive). You'll need itlater on in the lab.

Step 4 Click on Launch the Console to bring up the console in another tab

Figure 2 Launching the console

Step 5 From the console click the Files tab

Step 6 In the DFS navigator, expand the user directory and select the directory with the same username that was in the service details

Step 7 Click the Upload icon.



Figure 3 Upload icon

Step 8 Click Browse and select the file \BlueMixLabs\LabE\blogs-data.txt where \BlueMixLabs isthe root folder of the files provided to you by the instructor. Click OK.

3. Importing the data into BigSheets

Now that you have the sample data uploaded to HDFS, you can import the data into BigSheets andcreate a workbook that contains that data.

Step 1 From the console click the BigSheets tab and click New workbook.

Figure 4 Create BigSheets Workbook

Step 2 Name the workbook WatsonBlogData. In the distributed file system browser, select the fileblogs-data.txt that you uploaded in the previous step.

Step 3 In the preview pane on the right , select a new reader to map the data into a spreadsheetformat. (Currently the reader is Line Reader.) Click the edit icon next to 'Line Reader' andselect JSON Array from the drop-down list.

Figure 5 Changing the reader

Step 4 Click the green check mark to change the reader. Click Fit Columns in the preview pane to



make the tabular data appear more compact.

Figure 6 Fit Columns to width

Step 5 Click the green check at the bottom to save the workbook.

Step 6 From the BigSheets workbook page, click Build new workbook to create a workbook with thesame data.

Step 7 Because you do not need all of the columns to complete your design, drop some of thecolumns. Click the arrow in any column heading, and select Organize columns

Step 8 Click the x next to the following columns to remove them:

▪ Crawled

▪ Inserted

▪ IsAdult

▪ PostSize

Click the check at the bottom to confirm removal.

Step 9 Click Save. Then, click Save and Exit,.

Step 10 Change the name of the workbook to WatsonBlogsDataV2 and click Save.

Step 11 Click Run to run the workbook. Wait for the run to complete.

4. Exporting the workbook data

In this section you’ll export the data from your updated workbook so that it can be loaded into BigSQLlater and then accessed using standard SQL queries

Step 1 In the menu bar of the WatsonBlogDataV2 workbook, click Export data.

Figure 7 Export Data menu item

Step 2 In the drop-down window, select the TSV type in the Format Type field

Step 3 Specify Export to File.

Step 4 Click Browse to select a destination directory in the HDFS file system. Select your path/user/<username and then type the name for the new file, WatsonBlogs Click OK .



Figure 8 Naming the exported file

Step 5 Make sure that the Include Headers check box is cleared, because you want to export thedata only. Click OK. A message dialog shows that the workbook is successfully exported , ClickOK.

Figure 9 Don't include headers

5. Launch Eclipse to bring up the sample Big SQL application

In this section you’ll start Eclipse and load the workspace that contains the sample Big SQL applicationyou'll use to interact with your BlueMix MapReduce service

Step 1 Open Windows Explorer and launch the program \BlueMixLabs\eclipse\eclipse.exe where\BlueMixLabs is the root folder of the files provided to you by the instructor.

Step 2 Select the workspace \BlueMixLabs\workspaces\LabE when prompted and click OK



Step 3 The sample BigInsightsSample should appear in the Project Explorer

Figure 10 BigInsightsSample project in Eclipse

Step 4 Feel free to explore the code . BigSQLJdbcClient.java is the interface to BigSQL.BigInsightsServlet.java handles all the URL requests

6. Define the BlueMix Server

In this section you'll define the BlueMix server so you can automatically push your Eclipse code out toBlueMix when needed

Step 1 Click on the Servers tab at the bottom of the Eclipse UI as shown in Figure 24.

Figure 11 Eclipse Servers tab

Step 2 Click the link labeled ...Click this link to create a new server.

Step 3 Expand the IBM category and select IBM BlueMix. Click Next.



Figure 12 Select IBM BlueMix as server type

Step 4 In the next dialog enter your BlueMix credentials and click on Validate Account

Figure 13 Validate your BlueMix account

Step 5 Verify that the dialog reports that your account is valid and then click Finish.

Step 6 Your server definition should now appear in the Servers tab

Figure 14 Your server definition

7. Push the BigInsights sample application from Eclipse

In this section push the sample appication to BlueMix, bind it to your existing MapReduce service andstart it from within Eclipse .

Step 1 In Eclipse go to the Servers tab and select your server, right click and select Add andRemove from the context menu



Figure 15 Add project to server

Step 2 Select the BigInsightsSample application and click Add .

Figure 16 Add project to BlueMix

Step 3 Click Finish. Click Next in the resulting dialog

Step 4 Append a unique string to the Subdomain (eg your first initial + last name ) since this has to beunique systemwide. See Figure 17 for an example. Click Next

Figure 17 Specify unique Subdomain for app

Step 5 Select your previously create MapReduce service. Click Next



Figure 18 Add existing service

Step 6 Click Finish to push and start your application.

Step 7 Messages should start appearing on the console. Wait for the message that says The serverdefaultServer is ready to run a smarter planet.

Figure 19 Console messages

8. Running the application

Now that the application has been deployed and started, you can run it from inside Eclipse

Step 1 From the Eclipse menu select Window>Web Browser and then select one of your installedWeb browsers (eg Firefox)



Figure 20 Select an external browser

Step 2 In the Project Explorer select your project, right click and then select Run As->Run onServer from the context menu.

Figure 21 Launch application

Step 3 Click on Finish in the resulting dialog. .The browser you selected in Step 1 should launch withthe application

Step 4 The application will load the file you exported in BigSheets into BigSQL. You have to providethe name of the file and the name of the new table in BigSQL that will host it's contents. Type in



WatsonData.tsv as the TSV File and my_demotable as the BigSQL table name

Figure 22 Exported Data File and BigSQL Table Name

Step 5 Clock Load. Verify that the returned message indicates a successful load. Note: if it failscontact your instructor

Figure 23 Successful Load

Step 6 Click Fetch data and verify that the first 5 rows of data are returned from BigSQL



Figure 24 Successful fetch

9. Validating the data import using the BigSQL console

In this section you’ll the data import performed by the application by looking directly at the BigSQLconsole

Step 1 In your browser go to the BlueMix URL https://ace.ng.bluemex.net and login if necessary

Step 2 Make sure you're in the Dashboard tab (if not click on the Dashboard link at the top of thepage to take you there).

Step 3 In the Services section of your BlueMix Dashboard click on the hexagon for the Map Reduceservice you've been using in this lab

Step 4 You see the URL and the login information that you need to open the services underlyingInfoSphere BigInsights Console.

Step 5 Click on Launch the Console to bring up the console in another tab

Step 6 In the console on the right side of the screen under Quick Links click on Run Big SQLQueries

Figure 25 Launch Big SQL console



Step 7 You'll need your BigSQL database name to proceed.Go to tab in your browser where you haveyour BlueMix environment running.

Step 8 Go to the Dashboard area and click on the square icon for you BigInsightsSample application

Step 9 Under App Runtime click on the round icon for the Liberty Runtime.

Step 10 Scroll down to Environment Variables and note the BigSQL database name as shown inFigure 26.

Figure 26 Big SQL database name

Step 11 Go back to the Big SQL console and enter the following command (replacing [db name] withthe actual database name you noted in the previous step.

Step 12 use [db_name]

Step 13 Click Run and verify that you get a successful status message.

Figure 27 Status message after use command

Step 14 Replace the use command with the following command and click Run again

select * from my_demotable

Step 15 Verify that the data you loaded via the applications is returned



Figure 28 Query results

Congratulations! You’ve successfully completed Lab E where you explored the BlueMix MapReduceservice that is powered by IBM BigInsights

Documents

BlueMix Hands-On Workshop - Meetupfiles.meetup.com/14755122/IBM BlueMix Workshop - Lab E - MapReduce Service _Big Data...deploy a BlueMix Java application that reads the data. The