Upload
waleed-badawy
View
2.892
Download
4
Embed Size (px)
DESCRIPTION
Microsoft Fast Search server for Sharepoint 2010 server
Citation preview
FAST Search for Sharepoint
Brad FreelsTechnology [email protected]
Microsoft’s Search Vision & Strategy
People & Expertise
Connect to all your Content
Search is Everywhere
Big Bet
Enterprise
Desktop Enterprise Internet Devices
Employee ProductivityMonetization
High Value SearchInternet
Consumer Portals / Partner Portals Marketing / B2B / …
Best of Microsoft - Best of SharePoint - Best of High End Search
Interactive Visual Search - Personalization - Social Networks
eDiscoveryResearch Portals
360o customer viewsCompetitive Intelligence
…
Introducing FAST Search for SharePoint
OOB User Experience
Tailoring General Productivity
Search Platform and Architecture
Search Driven Applications
Deployment and Administration
Summary and Resources
Microsoft Enterprise Search The 2010 Wave
General productivity searchLight customization
Customized productivity search and search driven
applications
• UI Framework• Social search features and integration• SharePoint platform integration • End user and site administrator enablement
Common across the product line
• Connector Framework (BDC)• APIs and developer Experience• Admin & deployment capabilities• Operations advantages (SCOM, scripting)
SharePoint vs. FAST Search for SharePoint
User InterfaceCentral AdministrationCrawler and Connector
Similarities
SharePoint Front-end
Central Administration
UI
Connectors
• SharePoint• File Traverser
• Web • BDC• Exchange• Notes
• Query and Result Processing
• Content Processing Pipeline
• Customizability and Scalability
Differences
Query and Result
ProcessingAdvancedContent
ProcessingSearch
Core
SharePoint and FAST Search Server
Content SSA
Query SSA
Content Sources
Search Results
Introducing FAST Search for SharePoint
OOB User Experience
Tailoring General Productivity
Search Platform and Architecture
Search Driven Applications
Deployment and Administration
Summary and Resources
Explore using navigatorsthat help you understand the scope of information available,with exact counts of results
Improve relevance with usebased on how people tag and rate content in SharePoint and on click-throughs of search results
Visualize your resultsso you can quickly recognize what you’re looking for without opening separate applications
Get better answers, fasterusing a visual, interactive search experienceQuery
completion
Document thumbnail
s
Scrolling previews
Read in Office Web
Apps
Relatedsearches & people
Federated results
Sorting on any
property
Find peoplewith address book search that matches names phonetically & lets you browse organizations
Use MySite profile pagesthat recognize and track your colleagues and make it easier to share your work with others
Discover expertisebased on terms matched frommined Outlook inbox data and SharePoint MySite profiles
Connect with people and expertiseand streamline how you find and collaborate with othersFilter by title, expertise &
other attributes
Expertise matching
Phonetic name
lookup
Org browsin
gFind recent
content
Real-timepresence
Deep Refinement and SortingEnables precise control of results
Enables conversational experience across all of the results
You will never miss any content Enabling better findability and explorationDiscover non-obvious relationships across the entire result set
Exact counts shows relative weightProvides analytic view of your resultsIndicates priority and importanceThe right lever to slice and dice your content
Sort on any fieldEmpower the user to use the relevance model that best fits their needsRearrange the result set to meet specific criteriaAlphabetical, numeric, and date
Out of the Box
Sorting OptionsExact Counts
Demo
Introducing FAST Search for SharePoint
OOB User Experience
Tailoring General Productivity
Search Platform and Architecture
Search Driven Applications
Deployment and Administration
Summary and Resources
Customize search to meet your business needs
Deliver results that are contextually relevant
Search in the language of your business
Tune relevancy to improve accuracy
Create structure from unstructured content
Configure the UI to extend your application Example:
Similarity Search
Custom Query Suggestions
Visual Best BetsIdentify static content that is always relevant
Built on SharePoint KeywordsMatches keywords and synonyms that are contextually relevant to users. Include banners, videos, external websites.
Easy and quick to setupPoint and click setup for site admins. Set and forget with content expiration dates . Web Parts allow for easy page customization
Visual Notification Set Vertical Orientation
Example:Visual Best Bets
Audience-specific search experiencesUse User Context to meet the needs of diverse groups
User context
Inform-atio
n contextApplicati
on context
Social
context
Renee LoEngineeringContoso Consulting”What should I know about implementing ERP?”
Alan BrewerSales ManagerContoso Consulting”What should I know about selling ERP consulting?”
Username & Group Memberships
LocationLanguages
Business UnitDepartment
TeamTime of Day
Preferred SitesSharePoint Audiences
Interests & Current ProjectsContext of Current Task
Quickly build a contextual experienceUser based tools for creating results that are relevant to your users
Pick the right ingredients Match the proper terms and contexts to boost relevancy for targeted users to ensure your users are always finding the right content
One-way synonymsKeywords map to other termsTwo-way synonymsKeywords become equivalent to other termsBest BetsHighlights key resources that are always relevant to a keywordVisual Best BetsExtend Best Bets with pictures, video, Silverlight controlsDocument Promotion / DemotionTailor specific document relevancy
Create new user contextsSite administrators create contexts based on user profiles to deliver relevant results to the right audiences
Create new keywordsSite Administrators have powerful and simple tools to configure the search experience for groups of users
Search in the language of your businessIdentify what is important to improve the search experience
Use language that has specific meaning to your business
Users can quickly refine content using familiar termsBuild confidence that you found the correct answers the first time
Leverage corporate knowledge to make content findable
Corporate taxonomiesBusiness terminologyProduct namesAcronyms
Define custom rules to identify unique terms
Handle complex terms such as part numbers or formsSearching for ”XXX 123 abc“ finds “XXX-123-abc“ and “GG^XXX-123-abc_HH“
Profit
Taxonomy
risk
best practices
Strategy Development
customer relations
revenue
brand management
complianceSOX
supply chain
Disaster Recovery
merger
acquisition
target markets
cloud computing
mobile workforce
quality
cost savings
market share
Productivity
Social Media
IP Telephony
communications
Competition
part numbers
Global presence
direct mail
storagearchiveaudit
XML
Introducing FAST Search for SharePoint
OOB User Experience
Tailoring General Productivity
Search Platform and Architecture
Search Driven Applications
Deployment and Administration
Summary and Resources
FAST Search Extends SharePoint Superset of capabilities; common base functionality and platform
End Users
Content
Content search
People search
Common query and federation
APIs
Common connectors and
content APIs
Best High-End Content Search based on FAST ESP
Unique Social Search from SharePoint Server
2010
Common IT Pro and developer experience
FAST Search for SharePoint
SharePoint Server 2010
Advanced Content Processing
+ Profiles
FAST Search for SharePointHigh Level Architecture
Federa
tion Op
enSea
rch
Content Processo
rCrawlerIndexer
Query Processo
r
Search Center
Content
User Profiles
…
Form
at
Con
ver
sio
n
Lan
gu
ag
eD
ete
cti
on
En
tity
Extr
acti
on
Lem
ma
tizati
on
Map
per
…
Content Processing Pipeline
FAST Server(s)
SharePoint Server(s)
Other Server(s)
People
Searc
h
Index Partition
FAST Search for SharePointHigh Level Architecture
Federa
tion Op
enSea
rch
Content Processo
rCrawlerIndexer
Query Processo
r
Search Center
Content
User Profiles
…
Form
at
Con
ver
sio
n
Lan
gu
ag
eD
ete
cti
on
En
tity
Extr
acti
on
Lem
ma
tizati
on
Map
per
…
Content Processing Pipeline
FAST Server(s)
SharePoint Server(s)
Other Server(s)
People
Searc
h
Index Partition
FAST Query SSA
FAST ConnectorSSA
Extensible Content ProcessingEnables search that has a deep understanding of your information
Transform contentusing a processing pipeline that normalizes and cleanses all of your information
Use globallywith linguistics processing for 45 languages and recognition of content in 80+ languages
Add or extend stagesthat apply sentiment analysis, translation, or other business-specific processing you need
…
Format Converter
LanguageDetector
Lemmatizer
Word Breaker
Entity Extractor
Date/TimeNormalizer
Vectorizer
Web Analyzer
Properties Mapper
How does the pipeline work?A systematic approach to interpreting your contentSequential stages perform specific tasks while ingesting content
Breaks down content to the smallest addressable chunks to build meaningUnderstands file encoding, data formats, and written languages Supports 400+ file formats, 80+ languages
Process your content to make it searchableNormalizes content so that a consistent relevancy model can be appliedIdentifies structured and unstructured metadata in your contentMaps document metadata to SharePoint Crawled Properties
Map Crawled Properties
Maps all of the metadata that was discovered by the various pipeline stages
Web Link Analysis Analyzes documents for hyperlinks extracting anchor text which reinforces the authority ranking of a document.
Document Vector Creates a unique representation of a document that reflects important terms and frequency of occurrence. Used to find similar documents.
Date and Time Normalization
Converts dates and times to a standard representation, to handle locale specific representations. For example, knows that 14-Mar-10 is equivalent March 14, 2010.
Entity Extraction Finds terms in the content and maps them to predefined categories. Out of the box support for People, Companies and Locations, but can be extended to any category.
Lemmatization Finds the root of a word for a given language. For English it maps run, runs, running and ran back to a single lemma. Understands language specific grammar and context.
Tokenization Apply the language specific rules for identifying words, concepts, idioms and phrases. Also applies custom word breakers found in part numbers or telephone numbers.
Language Encoding and
Detection
Identifies the native written language and locale specific encoding so that the proper dictionaries can be used by the tokenization and lemmatization stages
Format Conversion Extracts plain text from multiple file formats, encodings, and applications
Extending Pipeline capabilitiesStraightforward way to add custom text analysis functionality
Configure Optional Processing StepsXML Properties mapperOffensive Content FilterField CollapsingVerbatim (wholeword) extractor
Use a dictionary for custom extraction
Pipeline ExtensibilityCalls external applications for custom item processing
Add Custom ProcessingPipeline Extensibility is a specially defined stage that takes a set of crawled properties, as flat text as input and maps output to another crawled property
Sandboxed executionExecutable arguments and temporary files are automatically handled with timeouts.Runs just before the Crawled Property Mapper, providing accessibility within SharePoint
Add needed metadatawith pre-built extractors that automatically tag people, locations, and company names
Extend easilyto recognize business-specific terms and concepts—tailoring search for your information
Surface in navigationrecognized properties, making search results more relevant and discoverable
Powerful Entity ExtractionEnables search-driven navigation that is relevant to your business
CONCEPT
PRODUCT
COMPANY
Tune relevancy to improve accuracyChanging content and users need require a flexible solution
Start with great relevance OOBTuned for great general productivity experienceAutomatically improves relevancy with social click-throughs and link text analysis
Create new relevance modelsMultipleRank ProfilesBlend static and dynamic ranking parameters to instantly improve search resultsCreate with simple PowerShell commandsExpose as new sorting options
Standard Sorting Options
Custom Rank Profiles
Tunable RelevanceEnables unique, business-specific search results for diverse roles
Tailor search resultsby tuning relevance ranking based on content freshness, source and 7 other attributes
Make search your ownby configuring relevance to recognize synonyms, acronyms, and other business terms
Highlight featured content using configurable best bets that visually highlight editorialized search results for targeted users
Search 2010 “Stack”The platform for Search Customization
SearchWeb Parts
SharePoint Search Index
OpenSearch /Custom Source
SharePoint Indexer
Federation OM
Web Service,
RSS
FASTSearch Index
FAST Indexer
What’s New in 2010?Primary Search Web Parts now UnsealedFederation now a key Public OM layer
All Web Parts built on federationQuery alteration, custom Runtimes, blending results from multiple sources
Web Service / RSS Enhancements
FAST Search / SharePoint Search:Shared Web Parts, RSS, Web ServiceShared Federation OMIndex and Crawling SeparateFAST unique Content Processing Pipeline
SearchWeb Parts
Federation / Query
OM
Web Service,
RSS
Content Processing
Robust query languageUse FAST Query Language (FQL) for precise query development
FQL provides a robust and expressive query language
Wildcard support - *, ?Numeric Data types (Integer, Float, Decimal, Datetime)
OperatorsDirect field access (e.g., title:othello, author:shakespeare)Numeric (COUNT, RANGE, <, <=, >, >=)Boolean (AND, OR, ANY, NOT)Rank (RANK, XRANK)Proximity (NEAR, ONEAR)Sorting (SORT, SORTFORMULA)
String (operator support for strings) Boundary (starts-with, ends-with, equals)Filter
Introducing FAST Search for SharePoint
OOB User Experience
Tailoring General Productivity
Search Platform and Architecture
Search Driven Applications
Deployment and Administration
Summary and Resources
Search Driven ApplicationsMeet all the search application needs you have across your business
“How do I support the
unique search needs of teams and work
that impact our business?”
To do so, you need a search platform that has• A deep understanding
of your information• Flexible relevance to
meet diverse needs• A customizable UX to
increase user efficiency
Sales: 360o Customer Insight
Services: Knowledge Browser
Marketing: Competitive Intelligence
Research & Development:Innovation Portal
Support:Call Center Advisor
Operations:Systems/Logistics Portal
Legal, HR, IT, Finance, ……
real estate risk
News and external opinion to monitor and send to clients
Top information from Woodgrove…new market
view report to send to clients
Drilldown to single view with all clues about a customer: portfolio, holdings, communications, annual and quarterly
customer plans, etc…
Set of Customers to explore, with rollup
Experts to help, with availability
and rating
Immediate actions on
selected items
View of information across different pivots, with
drilldown
How would you create this?Content Crawling: bring in data from lots of places
OOB connectors to SharePoint (reports, account documents), exchange public folders, shared files; BDC with customization in SPD (no code) for customer portfolio/holdings
Content processing: creating metadataNames of holdings, offerings, key concepts, companies, peopleSynonyms for key concepts (real estate ~ REIT)
OOB web parts configured for style Federation, People Search, Search actions
Custom web parts for visual navigationRoll-up configured via results collapsingCustom relevance profileSharePoint workflows for act-on-selected-items
Introducing FAST Search for SharePoint
OOB User Experience
Tailoring General Productivity
Search Platform and Architecture
Search Driven Applications
Deployment and Administration
Summary and Resources
Secure, unified access to informationIndex or federate with content, applications, and services
Access content securelywith a full-featured content crawler and both index- and query-time trimming of results
Federate queriesusing OpenSearch to quickly access existing search indexes and online information services
Extend your reachusing Business Connectivity Services to access your content sources and business applications
OpenSearch Federation
Indexing Connectors
EnterpriseContent
Business Application
s
Information Services
User Experience
Search Index
Websites and Intranet
File Shares
IBM Lotus Notes
EMC Documentum
SharePoint Server
Exchange Public Folders
AD & LDAP Profiles
Etc …
FAST Search for SharePointSummary of architectural elements
Custom Front-End
OpenSearch or Other Sources
SharePoint Front-end
People Search
Qu
ery
Obje
ct
Mod
el
Query and Result
Processing
Security AccessModule
SearchCore
Indexing
Federation Object Model
Query Web Service
AdvancedContent
Processing
Linguistics
Web LinkAnalysis
Connectors
• Web Crawler• JDBC
Connectors
• SharePoint• File
Traverser• Web • BDC• Exchange• Notes• Documentu
mMicrosoft System Center Operations Manager
Monitoring Services
Administration and Schema Object Model
Site Collection Level Admin UI
• Keyword Management• User Context
Management• Site
Promotion/Demotion
PowerShell
• Schema configuration• Admin configuration• Deployment
configuration
Central Administration UI
• Property mapping• Property extraction• Spell-checking
FAST Server(s)
SharePoint Server(s)
Other Server(s)
Content
FAST Search HW – Best Practices
CPU: 2 x 2GHz+ (Quad/six core)Memory: 24-48 GBDisk:
2 x 300 GB, SAS, 10K RPM (RAID 1)
CPU: 2 x 2GHz+ (Quad/six core)Memory: 24-48 GBDisk alternatives:
1.0 TB: 8 x 300 GB, SAS, 10K RPM (RAID10)1.8 TB: 8 x 300 GB, SAS, 10K RPM (RAID 5)3.6 TB: 16 x 300 GB, SAS, 10K RPM (RAID 5+0)New: 7.2 TB: 16 x 600 GB, SAS, 10K RPM (RAID 5+0)SAN: Configured for “database performance”
Storage Server
Admin / Processing Server
FAST Search – Main ComponentsSharePoint Crawler
Capacity: ~30 mill items per crawler node, SQL server needs to be scaled for high IO
Web AnalyzerCPU/disk footprint can vary by a factor of 10 depending on the content: - number of links - length of links - internal cross link ratioAverage capacity: ~30 mill items per web analyzer nodeCan be deployed with the Indexer in normal scenarios
Indexer/search nodeTwo supported models: - Normal mode:
~15 mill items per node~25 QPS
- High Density Mode:~ 40 mill. items per node~ 7 QPS
SharePoint Crawler
Crawl DBSP CrawlPeople Crawl
Indexer/Search
FAST-FSTIDX-11Index/Search
Web Analyzer
FAST-WA-1Web Analyzer
Columns give you more indexingNeed more Doc Processors and Content Distributor roles
Rows give you more query and redundancy
More Query roles
Rows and Columns
FAST Search – Pilot/Dev Deployment
LimitationsQuery rate: < 2 QPSContent: < 1 M items
Note:This configuration should work as an absolute minimum production environment running on VMs with a SAN backend but with very limited capacity and performance. Not tested.
SP2010 Farm
All roles
FAST Search for SP 2010 Farm
All roles
FAST Search – Extra Small FarmLimitations
Query rate: < 5 QPSContent: < 8 M items
SP2010 Farm FAST Search for SharePoint 2010 Farm
Web Front EndQuery
SP CrawlPeople CrawlSQL Server
Web Front EndQuery
SP CrawlPeople CrawlSQL Server
AdminIndex (Search)
Content DistributorIndexing Dispatcher
Web Analyzer4 Docprocs+
(Index) SearchContent DistributorIndexing Dispatcher
Web Analyzer4 Docprocs+
SP2010 Farm FAST Search for SharePoint 2010 Farm
SQL 2008 Cluster
Web Front EndQuery
Web Front EndQuery
Search Admin DBCrawl DB
SharePoint
AdminContent DistributorIndexing Dispatcher
12 Docprocs+Web Analyzer
Index (Search)Content DistributorIndexing Dispatcher
12 Docprocs+Web Analyzer
QR Server
(Index) SearchQR Server
SP CrawlPeople Crawl
SP CrawlPeople Crawl
**
*
Note:Servers marked with * are only
needed for high availability
FAST Search – Small Deployment
LimitationsQuery rate: < 25 QPSContent: < 15 M items
FAST Search – Medium DeploymentFAST Search for SharePoint 2010 Farm
AdminContent Distributor
Web Analyzer12 Docprocs+
Index (Search)Content Distributor
Web Analyzer12 Docprocs+
Index (Search)Web Analyzer
Indexing Dispatcher12 Docprocs+
(Index) SearchQR Server
(Index) SearchQR Server
Index (Search)Web Analyzer
Indexing Dispatcher12 Docprocs+
(Index) SearchQR Server
SP2010 Farm
SQL 2008 Cluster
WFEQuery SSA
WFEQuery SSA
SP CrawlPeople Crawl
SP CrawlPeople Crawl
Search Admin DBCrawl DB
SharePoint DB
LimitationsQuery rate: < 25 QPSContent: < 45 M items
FAST Search – Large DeploymentSP2010 Farm
FAST Search for SharePoint 2010 Farm
SQL 2008 Cluster
Web Front EndQuery
Web Front EndQuery SP Crawl
People CrawlSP Crawl
People Crawl
Search Admin DBCrawl DB
SharePoint
AdminConfigServer
Content DistributorWeb Analyzer12 Docprocs+
Index (Search)Content Distributor
Web Analyzer12 Docprocs+
Index (Search)Indexing Dispatcher
Web Analyzer12 Docprocs+
(Index) SearchQR Server
(Index) SearchQR Server
Index (Search)Indexing Dispatcher
Web Analyzer12 Docprocs+
(Index) SearchQR Server
Index (Search)Web Analyzer12 Docprocs+
Index (Search)Web Analyzer12 Docprocs+
(Index) SearchQR Server
(Index) SearchQR Server
Index (Search)Web Analyzer12 Docprocs+
(Index) SearchQR Server
LimitationsQuery rate: < 25 QPSContent: < 100 M items
Introducing FAST Search for SharePoint
OOB User Experience
Tailoring General Productivity
Search Platform and Architecture
Search Driven Applications
Deployment and Administration
Summary and Resources
Tools – QR Server
Neil Richard’s Blog Enabling the QR Server Blog Post - http://tinyurl.com/3b9ren4
To access1. Go to your FAST
Search for SharePoint Server
2. Open the browser
3. Access the URL http://localhost:13280
http://fastforsharepoint.codeplex.com/Connect to web app running FAST SSA (SP box)Use it to test FQL
FAST Search for Sharepoint Query Tool
Useful ResourcesFAST University TrainingMSDN & TechNetBlogs
Leonardo De Souza’s Blog http://searchunleashed.wordpresss.com
Thomas Svensen’s Bloghttp://blogs.msdn.com/b/thomsven/
Comperio Search Nuggets http://nuggets.comperiosearch.com/
Books