Take Cloud Hybrid Search to the Next Level

Jeff Fried

BA Insight

@jefffried

we love hybrid search - it's amazing how fast usage is growing

Jeff Teper @jeffteper

GO

LD

BR

ON

ZE

/

PR

IZE

SS

ILV

ER

Focused on Search and

SharePoint since 2004

Longtime

Search Nerd

• CTO, BA Insight

• Senior PM, Microsoft

• VP, FAST

• SVP, LingoMotors

About Jeff Fried

Passionate About

• Search

• SharePoint

• Search-driven

applications

• Information Strategy

Blog:

BAinsight.com/blog

Technet Column

“A View from the

Crawlspace”

jeff.fried@bainsight.com

About BA Insight

– Connectivity

– Applications -

– Classification -

– Analytics

KCTCS (background)

Search is not stationary

–

–

–

–

–

Why Hybrid SharePoint?

The

Evolution

of

SharePoint:

HYBRID Management ExtensibilityExperiences

| Server

Experiences Management Extensibility

| Server | Server

HYBRID

Team

Sites

Portals

Enterprise

Content Mngt

BI

Approaches to Hybrid SharePoint

Split Workload

different tools in

different places

Split User

task uses content or

sites across ‘the divide’

Exchange, SharePoint, Skype

OneDrive, Yammer, PowerBI, Delve

Extranet, Mysites, Team Sites, Project Sites

Portals, Intranet, Services/Applications

Links Search

Search Provides a Unified View

–

–

–

–

3 approaches for hybrid search

14

“Classic” Hybrid Search is Federated

not a single result set OOB

SharePoint 2013/2016 Search Architecture

Web Service (CEWS)

–

–

–

–

–

–

Case Study B: Crawling O365

Cloud Hybrid Search

Benefits of Cloud Hybrid Search

2) Makes finding content easy, wherever the content lives

1) Simpler, easier, and less costly to run search

SharePoint Server

(On-premises or Hosted)Office 365

SharePoint Online Content

Onedrive for Business ContentSharePoint Content

Cloud Hybrid Search

Case Study C: Split Users with SharePoint

Setting up Cloud Hybrid Search

•

•

1.

2.

3.

4.

https://support.office.com https://technet.microsoft.com

https://social.technet.microsoft.com/Forums/en-us/home?forum=CloudSSA

Microsoft doc and forums

24

New Sites to bookmark

The Cloud SSA

Use search verticals with Cloud Hybrid Search

SharePoint Online

Custom result source using Local SharePoint results plus a filter which excludes results from on-premises

TIP: Can be used during validation of hybrid search in the production tenant.

Result source query:

{searchTerms} NOT(IsExternalContent:1)

Start with “Everything”?

This is the default result source using Local SharePoint results but it has been renamed to «Everything» in the Search Navigation configuration.

SharePoint Online Everything

Result Sources are your friend

The Support Search vertical only searches sites that are relevant to the Support team.

It uses Local SharePoint results plus a filter on which sites to include in the search results

Result source query:

{searchTerms} (

Path:»http://sp2010» OR

Path:»file://fileshare» OR

Path:»http://demohybrid.../../supportforum»)

SharePoint Online Support Search

SharePoint 2016 Hybrid

Cloud Hybrid

Search User Profiles Following

Extranet

Compliance

(DLP/e-

Discovery)

Config

Experience

Built on Search

Differences between Cloud Hybrid Search

in SP2013 versus SP2016?

PRO

CON

Cloud SSA Pro/Con

External Content

(on-premises and/or

in the cloud)

SharePoint Server

(On-premises or Hosted)Office 365

SharePoint Online Content

Onedrive for Business Content

Co

nnect

ors

SharePoint Content

Adding External Content

Cloud Hybrid Search

Connectors to MANY Enterprise Systems

•

•

•

•

ERP and Portal Systems•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

External Content in O365 UX

Unified view across all content

- on-premises and on-line

- inside and outside SharePoint

DLP Sensitive Data Search works with hybrid

Search for sensitive data across on-premises and SharePoint Online

All Built-in sensitive types

Identification and export

Extends to data in OneDrive

Sensitive Information type detection through KQL searches

Get instant statistics

Preview & export results

Current Caveats:

1) don’t see thumbnails, just file icons

2) Have to query for it to show up

–

–

–

–

Case Study C: Cloud SSA, external content

Large global company

in materials science

Scaling

Item Limits and Pricing

1M items of external content in index for every 1TB storage in O365

1TB included by default

+ 0.5 GB per licensed O365 user

No limit on number of items from O365 in the index

2000 users x 0.5 GB = 1TB

+ 1TB default = 2 TB total

-> 2M external items indexed

+ Can also buy the “Office 365 Extra File Storage” Add-on

$0.20/GB/Month = $200/TB/Month = $200/M items/Month

50,000 users x 0.5 GB = 25TB

+ 1TB default = 26 TB total

-> 26M external items indexed

External Content

(on-premises and/or

in the cloud)

Custom

Processing

CEWS

Bottlenecks:

1) Source systems

2) Content Processing

3) Indexer

….

External Content

(on-premises and/or

in the cloud)

Bottlenecks:

1) Uplink

2) Source systems

….

43

Performance

500K items crawled on an Azure D3

50 DPS 100 DPS

1 hour

SUPPORTED

– Custom IFilter

– BCS connectors

– Partner connectors

Customizations with Cloud Hybrid Search

SUPPORTED

– Tenant level schema mapping

– Query rules

– Result sources

Cloud SSA SCS/O365

NOT SUPPORTED

• Content that requires custom security trimming

NOT SUPPORTED

• Site collection level schema mapping

• Custom security trimming

• Custom entity extraction

• Content enrichment web service

Issues with Cloud Hybrid Search (1)Cloud Hybrid Search "annoyances"

Performance Characteristicsslower query latency for on-prem queries against Cloud SSA

SharePoint Online Limitationsno synonyms

no site-level schema

no full trust code access

Hybrid Administration Weaknessesclunky metadata mapping

can't remove on-premises search results from Cloud SSA

trickier to test & debug crawls

can't reset index from Cloud SSA

Should I run index reset?

NO!

Issues with Cloud Hybrid Search OOB

48

Content Enrichmentno CEWS

no Entity Extraction

Securityno Custom Security Trimming

Can't crawl across Multiple Domains

Can't Crawl SP in Classic Auth Mode

Data Sovereigntyexport-restricted content

can't be put in O365 index

Limitations of Cloud SSA

External Content

(on-premises and/or

in the cloud)

SharePoint Server

(On-premises or Hosted)

SPO Content

OneDrive Content

Co

nnect

ors SharePoint Content

Connector

Framework

Office 365

AutoClassifier

(app version)

CEWS

Custom

Processing

Case study D:Content Enrichment

Content

CloudSSA

Connector Framework

IndexingConnectors

Smart Pipeline

AutoClassifierCustom Stage A

CustomStage C

Custom Stage B

Online

On-Prem

Cloud Hybrid Search under the coversSecurity = identity sync + ACL mapping

Cloud SSACloud SSA

ParseCrawl

SCS

ACL Map Process

Blob

storequeue

•

•

Directory Synchronization

SID S-1-5-21-1212121212-1212121212-1212

jaden@corp.hybridsearch.com

msOnline-

OnPremiseSecurity

Identifier

S-1-5-21-1212121212-1212121212-1212

PUID PUID-XXXX-XXXXXXXXXX

Mapping of Access Control Lists

Allow: S-1-5-21-1212121212-1212121212-1212 Allow: PUID-XXXX-XXXXXXXXXX

• User SIDs are mapped to PUIDs

• Group SIDs are mapped to Object IDs

• «Everyone» and «Authenticated users» are mapped to

«Everyone except external users»

Case Study E: Crawling Cross-Domain

A global single index solution

Cloud SSA

Cloud SSA

Cloud SSA

Cloud SSA

Cloud SSA

BUT export-restricted content

can’t be in the global index

Connect & Crawl

Federate

“Classic” Hybrid Search is Federated

not a single result set OOB

BA Insight Federator

Case study F:Data Sovereignty & Federation

Issues with Cloud Hybrid Search OOB

Content Enrichmentno CEWS

no Entity Extraction

Securityno Custom Security Trimming

Can't crawl across Multiple Domains

Can't Crawl SP in Classic Auth Mode

Data Sovereigntyexport-restricted content

can't be put in O365 index

Limitations of Cloud SSA BA Insight Solution

Connector Framework

AutoClassifier

Connector Framework

can 'map down' to AD groups

can 'map across' cross-domain

can crawl and map security

Federator

Key Considerations for Hybrid: Workloads, Environment, Data, Customizations

Availability of features Online versus

On-Premises on particular workloads

Significant investments in

customization of On-Premises

workloads

Concerns over global network

performance with remote sites

Regulatory

considerations

Manageability concerns

Thank you!

Toronto Enterprise Collaboration User Group

Change Management, Governance, SharePoint, Office 365, Yammer,

PowerBI, etchttp://www.meetup.com/TSPBUG/

Toronto SharePoint Users Group

http://tspug.com/

THANK YOU & See you next year!

Join us for SharePint after the event @ 5:30pm

6982 Financial Dr. and don’t forget to submit feedback after each

session for your chance to win great prizes at the end of the day!

https://www.surveymonkey.com/r/spstoronto2016

Contact:Jeff.Fried@BAinsight.comwww.BAinsight.com

Questions

References

http://technet.microsoft.com/en-us/library/dn197172(v=office.15).aspx

http://sp2013searchtool.codeplex.com/

https://github.com/OfficeDev/PnP-Tools/tree/master/Scripts/SharePoint.Hybrid.Search.Configuration

References - Blogs

http://blogs.msdn.com/b/spses/archive/2015/09/15/cloud-hybrid-search-service-application.aspx

http://blogs.msdn.com/b/spses/archive/2013/10/22/office-365-configure-hybrid-search-with-directory-synchronization.aspx

http://blogs.msdn.com/b/spses/archive/2014/01/05/office-365-configure-hybrid-search-with-directory-synchronization-password-sync-part2.aspx

http://blogs.msdn.com/b/spses/archive/2014/01/07/identity-federation-amp-single-sign-on-deployment-for-hybrid-search-in-office-365-sharepoint-online-part3.aspx

http://blogs.msdn.com/b/spses/archive/2015/03/19/configuring-microsoft-web-application-proxy-server-for-inbound-hybrid-topology-with-office-365-and-microsoft-sharepoint-server-2013-part7.aspx

https://www.youtube.com/watch?v=JWEZx9SHDb0&list=PLvmwu6WYeFdjNbiy7SISJAZd1HjzIJoz5

https://azure.microsoft.com/en-us/documentation/articles/active-directory-aadconnect/

https://azure.microsoft.com/en-us/documentation/articles/active-directory-aadconnect/

http://blogs.msdn.com/b/spses/archive/2015/09/15/cloud-hybrid-search-service-application.aspx

References – Installing with SP2016