[HCM Scrum Breakfast - January 2018] ElasticSearch In Action

Scrum breakfast Jan, 2018

Elasticsearch in actions

www.axon.vnfb.com/AxonActiveVietNam

Who we are?

Lương Văn Thắng


Who we are?

Vũ Hiển Khang


Who we are?

Đỗ Việt Trung


Theme: Learning mindset


1. OVERVIEW2. Searching3. SCALE4. SECURITY5. MIGRATION6. LIMITATION7. DEMO

Agenda


OVERVIEW


What is Elasticsearch?

❖ Search engine based on Lucene

❖ Real-time distributed, full-text search engine

❖ RESTful API

❖ Schema-free

❖ First public release in Feb 2010

Wikipedia


WHO are using ELASTICSEARCH?


WhY Elasticsearch?

● NoSQL DB for indexing JSON contents

● Schema-free

● Distributed

● High performance

● REST semantics

● Graph capabilities

● Great documentation

● Open source!


WhY Elasticsearch?


WORK WITH Big data


Searching – poor performance


Schema changes frequently


Scaling database


Searching – faster and smarter


Our solution


Searching In Action


Get started


❖ Node – a started instance of Elasticsearch

❖ Cluster - collection of connected nodes of Elasticsearch

Cluster and Nodes


❖ A collection of types

❖ Similar to database

Indexes


❖ Collection of documents

❖ Has schema (implicit or explicit)

❖ Similar to table

Types


❖ Self-contained data

❖ Have id

❖ Have schema

❖ Similar to record

Documents


❖ Documents are structured in fields

❖ Special fields: _id, _uid, _index, _type, _all, _source, …

❖ Similar to column

Fields


❖ text, keyword

❖ long, integer, short, double, float, byte

❖ date

❖ boolean

❖ binary

❖ geo

❖ object

❖ nested

❖ ip

❖ arrays

❖ …

Data types


Mappings and settings

Mappings – schema of documents+ dynamic mappings+ explicit mappings

Settings – configured for each index+ good enough when using default settings+ index analysis – expert users


Download and setup

https://www.elastic.co/guide/en/elasticsearch/reference/5.4/setup.html


Setup Elasticsearch


{"from": 0,"size": 20,"min_score": 5,"query": {"bool": {

"should": [],"must": [],"must_not": [],"filter": {}

}},"sort": [],"aggs": {}

}

Query structure



"should": [],"must": [],"must_not":

[],"filter": {}

}},"sort": [],"aggs": {}

}

Query structure

// do pagination




}},"sort": [],"aggs": {}

}

Query structure

// limit hitlist return




}},"sort": [],"aggs": {}

}

Query structure

// statistic



"should": [{..}, {..},..],"must": [{..}, {..},..],"must_not": [{..}, {..},..],"filter": {}

}},"sort": [],"aggs": {}

}

Query structure

// and

// or

// not


{

"bool": {

"must": [

{“strong”},

{“is man”}

]

}

}

Exercise 1


{"bool": {"must": [{“strong”},{“is man”}]

}}

Exercise 1


{"bool": {

"must": [{“has red on suite”}],“should“:[{“strong”},{“is man”}]

}}

Exercise 2


{"bool": {"must": [{“has red on suite”}],"should":[{“strong”},{“is man”}]

}}

Exercise 2


Exercise 3{

"bool": {"must": [{“strong”},{“has

cape”}],“must_not“:[{“be seen the face”} ]

}}


{"bool": {"must":

[{“strong”},{“has cape”}],

“must_not“:[{“be seen the face”}]

}}

Exercise 3


❖ Mail-search server

❖ Dataset of >470,000 emails

❖ Legal copy from a dissolved

company - US

Workflow of building a query


❖ Has ‘bomb’ in content

❖ Sent by [email protected]

❖ Interested in receiver

[email protected]


mailto:[email protected]

mailto:[email protected]




Synonym


Synonym token filter


Synonym token filter



SCALE


What – Why?


How to Scale


How to Scale




SECURITY


WHY

Elasticsearch has no concept of a user.

Essentially, anyone that can send arbitrary requests to your

cluster is a “super user”.


WHAT

❖ Authentication & authorization

❖ Access control

❖ Encryption

❖ Auditing



Authentication & Authorization

Role-based access control

Encryption

Auditing

License PaidPaid/UnpaidPaid/Unpaid Paid/Unpaid


Prevent unauthorized access

Keep Data Integrity

what happen on system


Installing….


User Authentication


Identify who send the request


built-in users

elastic kibana logstash_system


ADD More UserPUT localhost:9200/_xpack/security/user/trungdo

{

"full_name" : "Trung Do",

"email" : "[email protected]",

"password" : "axonvn",

"roles" : [ "admin", "superior", "kibana_user" ],

"metadata" : {

"workingYear" : 5

},

"enabled": true

}


How authentication works

Native(Basic Auth)

LDAP

file(Basic Auth)

ActiveDirectory

PKI Custom

Realms


Realms chain

Native file LDAP1 LDAP2


xpack.security.authc:

realms:

file: //id of realm

type: file //type of realm

order: 0 //order in chain

native:

type: native

order: 1

ldap1:

type: ldap

order: 2

enabled: false

url: 'url_to_ldap1'

…

ldap2:

type: ldap

order: 3

enabled: false

url: 'url_to_ldap2'

Configure realms

chainelasticsearch.yml


Authorization


❖ Identify permission to execute request

❖ Support by Role Based Access Control (RBAC)


Secured Resource

RBAC

Role

Privilege

Permissions

User


Secured Resource

Object need to be restricted accessity

Indices Document Field

User Cluster


Privilege

❖ One or group of actions user can execute

❖ Two types: Cluster and Indice

all read create index delete

See full privileges: https://www.elastic.co/guide/en/x-pack/5.4/security-privileges.html

all monitor manage_watcher manage_security

Indice privilege

Cluster privilege

https://www.elastic.co/guide/en/x-pack/5.4/security-privileges.html


Permission

Set of privileges on a secured resource

read on kibana indices

monitor on cluster

write on kibana indices

kibana system permission


Built-in Role

superuser kibana_user kibana_system logstash_system

monitoring_user reporting_user watcher_admin watcher_user machine_learning_user

See more at: https://www.elastic.co/guide/en/x-pack/5.4/built-in-roles.html

https://www.elastic.co/guide/en/x-pack/5.4/built-in-roles.html


ADD ROLEPOST localhost:9200/_xpack/security/role/scrum_admin

{

"run_as": [ ... ],

"cluster": [ ... ],

"indices": [ ... ]

}

//submit request on behalf of other user

//privilege on cluster

//permission entry on indices


ADD ROLEPOST localhost:9200/_xpack/security/role/scrum_admin

{

"run_as": [ "khangvu" ],

"cluster": [ "monitor" ],

"indices": [

{

"names": [ "scrum*" ],

"privileges": [ "read" ],

"field_security" : {

"grant" : [ "category", "@timestamp", "message" ]

},

"query": "{\"match\": {\"event.type\": \"technical\"}}"

}

]

}

//scrum_admin can submit request for khangvu user

//can monitor cluster

//name of targeted-indices: all indexes started with scrum

//privilege on targeted-indices

//allowed field to access

//list of targeted-document


Auditing

❖ Audit activities/events that occur in the system

❖ Output to logfile or index

❖ Enable this feature byxpack.security.audit.enabled to true in elasticsearch.yml.

See more at https://www.elastic.co/guide/en/x-pack/5.4/auditing.html

https://www.elastic.co/guide/en/x-pack/5.4/auditing.html



Encrypting communication

❖ Encrypt traffic to, from and within an ES cluster using

SSL/TLS certificate.

❖ Able to increase strength of encryption by Java

Cryptography Extension (JCE) plugin

❖ Separate port of node-to-node and transport client

See more at https://www.elastic.co/guide/en/x-pack/5.4/encrypting-communications.html

https://www.elastic.co/guide/en/x-pack/5.4/encrypting-communications.html


How to Scale



migration with logstash


missions

❖ Data migration tool

❖ Lightweight

❖ Scalabled

❖ Managabled

❖ Monitoring UI


Logstash is data collector pipeline


How ?❖ Event processing pipeline

❖ Has three stages:

➢ Input

➢ Filter

➢ Output

❖ Has bunch of plugins for you to play with


usecase - Collect rdbms to es server

Database Logstash filter ElasticSearch

Store Read

Kibana





Database Logstash ElasticSearch

StoreRead



ElasticSearch

Read

Kibana


Input stage❖ Plugins:

➢ jdbc

➢ file

➢ kafka

➢ redis

➢ beats

➢ …

See more at https://www.elastic.co/guide/en/logstash/current/input-plugins.html

https://www.elastic.co/guide/en/logstash/current/input-plugins.html


Filter stage❖ Plugins:

➢ grok

➢ mutage

➢ geoip

➢ ...

See more at https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

https://www.elastic.co/guide/en/logstash/current/filter-plugins.html


output stage❖ Plugins:

➢ elasticsearch

➢ file

➢ redis

➢ kafka

➢ ...

See more at https://www.elastic.co/guide/en/logstash/current/output-plugins.html

https://www.elastic.co/guide/en/logstash/current/output-plugins.html


scaling

Server-01

Logstash filter ElasticSearch

Read

Kibana

shipper-1

Garther Store

shipper-2

shipper-3

Server-02

Server-03

your business, Your imagination



LIMITATION


❖ ES has no transaction management, can’t rollback data.❖ Problem with index when change data property

{

"firstname": "Trung" ,

"lastname": "Do",

"email": "[email protected]" ,

"phone": "0909999999" ,

"street": "39B Truong Son" ,

"city": "Ho Chi Minh"

}

{

"firstname": "Trung" ,

"lastname": "Do",

"contact": {

"email": "[email protected]" ,

"phone": "0909999999" ,

"street": "39B Truong Son" ,

"city": "Ho Chi Minh"

}

}



Technology

[HCM Scrum Breakfast - January 2018] ElasticSearch In Action