Solr & R to Deploy Custom Search Interface: Presented by Patrick Beaucamp, Bpm-Conseil

O C T O B E R 1 3 -‐ 1 6 , 2 0 1 6 • A U S T I N , T X

Solr & R to deploy Custom Search Interfaces

Patrick Beaucamp

Chairman – Bpm-‐Conseil -‐ France patrick.beaucamp@bpm-‐conseil.com

PresentaHon Agenda

Solr & R IntegraHon inside AklaBox

AklaBox PresentaHon

AklaBox & Solr + R & GoJS & OSM

Demo Pla;orm : AklaBox

Going further : Vanilla Air, Spark & R & Solr

Cer@fied on Cloudera & HortonWorks

Run on Hadoop : Solr/Cloud, Hdfs ...

Ready for OpenStack

Aklabox PresentaHon

Aklabox PresentaHon

User Interface

Aklabox PresentaHon

Upload your documents

Share your documents

Collaborate on documents

Search on documents

Synchronize your

documents

Publish your documents

Document Viewer

Aklabox PresentaHon

WorkFlow

Synchro

Mobile

Aklabox PresentaHon

Standard Search Interface


•  Why do I get this list when I search inside the document repository ?

•  What does value when I run a search : weight of every words ? •  If a word is 100 @mes in a document, is the document more valuable for my search ?

•  May be the document I’m looking for has not the exact word spelling ?

•  How do I take into account mul@ language support ?


•  We need to review our module and rethink how we can help user to deploy their own search policy

•  R was a natural choice to create a new search algorithm •  We use R for our Data Mining development •  R contains packages to inspect documents •  R has virtually no limit to analyze and classify documents •  We read a lot about R & Search engine …


•  When do we analyze documents with R : •  Before Solr Indexa@on •  AZer Solr Indexa@on

•  Choice : •  Before Solr Indexa@on •  We add Metadata on every document, like top words, class of document ….

•  We create classes for documents, and rela@on between classes


Keywords are added inside Solr Index




R Packages : •  tm, textmining func@ons (stemming, words frequency, words manipula@on,

etc...) •  TF IDF funcHon (Term Frequency)

•  Matrix, for complex ma@rx manipula@on

•  cluster -‐ fanny & kmeans func-ons, to calculate classes on various group

•  libsvm -‐ fonc@uns svm, predict e& tune, for automa@c words classifica@on

•  Sampling – to create & manipulate different data sets


+ •  R algorithm runs when the document is uploaded

•  We keep only a few number of words per documents (parameter) •  We create classes for documents •  We can managed other concerns, such as interna@onalisa@on

•  R Package can be switch (other algorithm, new deployment) •  easy & flexible to deploy and maintain

•  No impact on Solr

-‐ •  Solr index is a gold mine … and we don’t run analysis on it



Mind Map with Words associa@on


Map Visualiza@on

OSM Visualiza@on

DemonstraHon

DemonstraHon

•  Other Business Cases

•  Document Management : Pre-‐classifica@on of documents (pharmaceu@cal industry)

•  Search engine : Analysis of WebSite during crawling process

•  Open Door to New development

•  Phone@cs search (to solve the word spelling problem)

Vanilla Air, Spark, Spark Sql for Solr

New Technologies are emerging … well : it’s already there !!!


•  Vanilla Air – Can Process R Packages – Can scale with growing number of documents

www.vanillasmartdata.com


Easy Switch in Architecture -‐> scalability

Vanilla Air, Spark, Spark & R & Solr

Spark 1.5 Version 1.5 (sept 2015) support for YARN cluster mode in R


We have now Spark & Solr Tools : SolrRDD Tools for reading data from Solr as a Spark RDD and indexing objects from Spark into Solr using SolrJ

hlps://github.com/LucidWorks/spark-‐solr


Admin Side – Runing complex R program on Solr index, using Vanilla Air

Lucky One !

Technology

Solr & R to Deploy Custom Search Interface: Presented by Patrick Beaucamp, Bpm-Conseil