GETTING STARTED WITH ELASTICSEARCH ON

WINDOWS AND .NET WITH NEST

A short introduction

Oslo/NNUG Meetup

Tomas Jansson

29/01/2014

THIS IS ME

Tomas JanssonManager & Group Lead .NETBEKK Oslo

@TomasJanssontomas.jansson@bekk.nogithub.com/mastojblog.tomasjansson.com

TL;DR;

https://github.com/mastoj/NestDemo

AUDIENCE

N00b

N00b Expert

Expert

BACKGROUND

This is the data and we need this new application

THE MASTERPLAN

WHAT I WANT TO SHOW YOU IS...

Elasticsearch is awesome

Indexing using NEST

Querying using NEST

... not about advanced elasticsearch hosting

INSTALLATION

Great news, install as a service added in 0.90.5

Powershell to the rescue

NEST

Abstraction over

Elasticsearch

There is an low level abstraction as well called RawElasticClient

Abstraction over

Elasticsearch

NEST

Fluent & Strongly

typed

Functional C#

FUNC DEMO

C:\Dev\git> scriptcs

scriptcs (ctrl-c or blank to exit)

> Func<int, int, int> add = (x, y) => x + y;

> add(1, 3)

4

Func executable

SIMPLE EXPRESSION DEMO

> using System.Linq.Expressions;

> Expression<Func<int, int, int>> addExpr = (x, y) => x + y;

> addExpr(1, 3)

(1,1): error CS1955: Non-invocable member 'addExpr' cannot be used like a method.

> var binExpr = addExpr.Body as BinaryExpression;

> Console.WriteLine(binExpr);

(x + y)

> var add2 = addExpr.Compile();

> add2(3, 1);

4

Expression ”function description”

MORE COMPLEX EXPRESSION DEMO

> public class SomeClass { public string MyString { get; set; } }

> Expression<Func<SomeClass, object>> propExpr = y => y.MyString + y.MyString;

> var compExpr = propExpr.Compile();

> var obj = new SomeClass { MyString = "Hello world" };

> compExpr(obj)

Hello worldHello world

> var body = propExpr.Body as BinaryExpression;

> Console.WriteLine(body);

(y.MyString + y.MyString)

> var left = body.Left as MemberExpression;

> Console.WriteLine(left.Member.Name);

MyString

MORE COMPLEX EXPRESSION DEMO

> public class SomeClass { public string MyString { get; set; } }

> Expression<Func<SomeClass, object>> propExpr = y => y.MyString + y.MyString;

> var compExpr = propExpr.Compile();

> var obj = new SomeClass { MyString = "Hello world" };

> compExpr(obj)

Hello worldHello world

> var body = propExpr.Body as BinaryExpression;

> Console.WriteLine(body);

(y.MyString + y.MyString)

> var left = body.Left as MemberExpression;

> Console.WriteLine(left.Member.Name);

MyString

Enables us to translate from one domain to another in an ”easy” manner

Show me the code!

ELASTICSEARCH CONNECTION

public class ElasticClientWrapper : ElasticClient{ private static string _connectionString = Settings.ElasticSearchServer;

private static ConnectionSettings _settings = new ConnectionSettings(new Uri(_connectionString)) //http://demoserver:9200 .SetDefaultIndex(Settings.Alias) //"customer_product_mapping" .UsePrettyResponses();

public ElasticClientWrapper() : base(_settings) { }}

//usagevar client = new ElasticClientWrapper();

MAPPING

public class Product{ public double UnitPrice { get; set; } public int TotalQuantity { get; set; } [ElasticProperty(Index = FieldIndexOption.not_analyzed)] public string ProductName { get; set; } [ElasticProperty(Index = FieldIndexOption.not_analyzed)] public string CategoryName { get; set; }}

public class Customer{ public string CustomerID { get; set; } public string CompanyName { get; set; } public string Address { get; set; } public string City { get; set; } public string Country { get; set; } [ElasticProperty(Type = FieldType.nested)] public Product[] Products { get; set; }}

MAPPING & INDEXING

_client = new ElasticClientWrapper();_client.CreateIndex("indexName", s => s.AddMapping<Customer>(m => m.MapFromAttributes()));

var customers = _customerRepo.GetCustomers();_client.IndexMany(customers, "indexName");

Mapping created from attributes

Indexing will use the mapping for the specified index

There is async versions of the methods

ALIAS

_client = new ElasticClientWrapper();_client.Alias("indexName", "aliasName");

Index_01

Alias

SWAPPING

_client = new ElasticClientWrapper();_client.Swap("aliasName", new [] { "Index_01" }, new [] { "Index_02" } );

Index_01 Index_02

Alias Alias

1. Create new index

2. Swap

3. Delete old index

MY QUERY OBJECT (WILL BE USED IN THE EXAMPLES)

public class SearchModel{

private int? _numberToTake;public string Query { get; set; }public Dictionary<string, IEnumerable<string>> Filter { get; set; }

public int? NumberToTake{

get { return _numberToTake.HasValue ? _numberToTake.Value : 25; }

set { _numberToTake = value; }}

}

QUERYING

{ "query": { "query_string": { "query": "tomas" } }}

_client.Search<Customer>(sd => sd.QueryString(Input.Query));

Elasticsearch

NEST

FUZZY

{ "query": { "fuzzy": { "_all": { "min_similarity": 0.6, "prefix_length": 1, "value": "tomas" } } }}

_client.Search<Customer>(sd => sd.Query(q => q

.Fuzzy(fd => fd

.OnField("_all")

.MinSimilarity(0.6).PrefixLength(1)

.Value(Input.Query))));

Elasticsearch

NEST

Will enable us to search for both «Thomas» and «Tomas» when writing «Tomas»

FUZZY IMPROVED (USING BOOL QUERY) - ELASTICSEARCH

{ "query": { "bool": { "should": [{ "match": { "_all": { "query": "tomas" } } }, { "fuzzy": { "_all": { "boost": 0.1, "min_similarity": 0.6, "prefix_length": 1, "value": "tomas" } } }] } }}

FUZZY IMPROVED (USING BOOL QUERY) - NEST

_client.Search<Customer>(sd => sd.Query(q => q

.Bool(b => b.Should(new Func<QueryDescriptor<Customer>,

BaseQuery>[]{

_ => _.Match(m => m.OnField("_all")

.QueryString(Input.Query)),_ => _.Fuzzy(fd => fd

.OnField("_all")

.MinSimilarity(0.6)

.PrefixLength(1)

.Value(Input.Query)

.Boost(0.1))}))));

HIGHLIGHT RESULT - ELASTICSEARCH

{ "query": { // see previous example }, "highlight": { "pre_tags": [ "<span class='highlight'>" ], "post_tags": [ "</span>" ], "fields": { "companyName": { "fragment_size": 100, "number_of_fragments": 1 } } }}

HIGHLIGHT RESULT - NEST

_client.Search<Customer>(sd => sd.Query( /* See previous example */ ).Highlight(h => h

.PreTags("<span class='highlight'>")

.PostTags("</span>")

.OnFields(new Action<HighlightFieldDescriptor<Customer>>[]{

_ => _.OnField(c => c.CompanyName).NumberOfFragments(1).FragmentSize(100)

})));

FACETS - ELASTICSEARCH

{ "query": { /* See previous example */ }, "highlight": { /* See previous example */ }, "facets": { "products.productName": { "nested": "products", "terms": { "field": "products.productName", "size": 1000 } }, "products.categoryName": { "nested": "products", "terms": { "field": "products.categoryName", "size": 1000 } }, "country": { "terms": { "field": "country", "size": 1000 } } }}

FACETS - NEST

_client.Search<Customer>(sd => sd.Query( /* See previous example */ ).Highlight( /* See previous example */ ).FacetTerm(f => f

.Nested(c => c.Products)

.OnField(c => c.Products[0].ProductName)

.Size(1000)).FacetTerm(f => f

.Nested(c => c.Products)

.OnField(c => c.Products[0].CategoryName)

.Size(1000)).FacetTerm(f => f

.OnField(c => c.Country)

.Size(1000)));

http://go-gaga-over-testing.blogspot.no/2011/09/solution-to-warning-in-quality-center.html

FILTERS - ELASTICSEARCH

{ "query": { "filtered": { "query": { /* See previous example */ }, "filter": { "bool": { "must": [ { "terms": { "country": ["usa"] } }, { "nested": { "query": { "terms": { "products.categoryName": ["Condiments", "Seafood"] } }, "path": "products" } }, { "nested": { "query": { "terms": { "products.productName": ["Chai"] } }, "path": "products" } } ] } } } }, "facets": { /* See previous example */}, "highlight": { /* See previous example */ }}

FILTERS – NEST – PART 1, THE CUSTOMERS FILTER

private static BaseFilter AddCustomerFilter(IEnumerable<string> items, Expression<Func<Customer, object>> propExpr)

{return Filter<Customer>.Terms(propExpr, items.ToArray());

}

FILTERS – NEST – PART 1, THE PRODUCTS FILTER

private static BaseFilter AddProductsFilter(IEnumerable<string> items,Expression<Func<Customer, object>> propExpr)

{return Filter<Customer>.Nested(sel => sel

.Path(c => c.Products) .Query(q => q.Terms(propExpr, items.ToArray())));}

FILTERS – NEST – PART 1, THE MAGIC DICTIONARY

public Dictionary<string, Func<IEnumerable<string>, BaseFilter>> FilterDesc =

new Dictionary<string, Func<IEnumerable<string>, BaseFilter>>(){

{"products.productName", ps => AddProductsFilter(ps, c => c

.Products[0].ProductName)},{"products.categoryName", cs => AddProductsFilter(cs, c =>

c

.Products[0].CategoryName)},{"country", cs => AddCustomerFilter(cs, c => c.Country)}

};

FILTERS – NEST – PART 1, ALL THE HELPERS

private static BaseFilter AddCustomerFilter(IEnumerable<string> items, Expression<Func<Customer, object>> propExpr)

{return Filter<Customer>.Terms(propExpr, items.ToArray());

}

private static BaseFilter AddProductsFilter(IEnumerable<string> items,Expression<Func<Customer, object>> propExpr)

{return Filter<Customer>.Nested(sel => sel

.Path(c => c.Products) .Query(q => q.Terms(propExpr, items.ToArray())));}

public Dictionary<string, Func<IEnumerable<string>, BaseFilter>> FilterDesc =

new Dictionary<string, Func<IEnumerable<string>, BaseFilter>>(){

{"products.productName", ps => AddProductsFilter(ps, c => c

.Products[0].ProductName)},{"products.categoryName", cs => AddProductsFilter(cs, c =>

c

.Products[0].CategoryName)},{"country", cs => AddCustomerFilter(cs, c => c.Country)}

};

FILTERS – NEST – PART 2, THE QUERY_client.Search<Customer>(sd => sd

.Query(q => q.Filtered(fq =>

{fq.Query(qs =>{

if (!string.IsNullOrEmpty(Input.Query))

{qs.Bool( /* See previous

example */ ));}else{

qs.MatchAll();}return qs;

});if (Input.Filter.Count > 0){

var filters = Input.Filter.Select(_ =>

FilterDesc[_.Key](_.Value)).ToArray();fq.Filter(fs => fs.Bool(bf =>

bf.Must(filters)));}

})).Highlight( /* See previous example */ ).FacetTerm( /* See previous example */ ).FacetTerm( /* See previous example */ ).FacetTerm( /* See previous example */ );

Easy installation

Awesome search engine

Strongly typed client

Fluent

Abstraction over Elasticsearch

Elasticsearch

NEST

SUMMARY

Demo code: https://github.com/mastoj/NestDemo

Nest documentation: http://nest.azurewebsites.net/

Nest source code: https://github.com/Mpdreamz/NEST

Slideshare: http://www.slideshare.net/mastoj/getting-started-with-elasticsearch-and-net

Sense (great tool to query elastic search in the browser): https://github.com/bleskes/sense

RESOURCES

Questions?

Thank you!

@TomasJansson