Context Attentive Document Ranking and Query...

Context Attentive Document Ranking and Query Suggestion

Wasi Uddin AhmadUniversity of California,

Los Angeles

Kai-Wei ChangUniversity of California,

Los Angeles

Hongning WangUniversity of Virginia

https://github.com/wasiahmad/context_attentive_irCodes will be released soon!

Search Logs Provide Rich Context to Understand Users’ Search Tasks

ContextAttentiveRankingandSuggestion 2CS@UCLA

5/29/2012 5/29/2012 14:06:04 coney island cincinnati5/29/2012 14:11:49 sas5/29/2012 14:12:01 sas shoes

5/30/2012 5/30/2012 12:12:04 exit #72 and 275 lodging5/30/2012 12:25:19 6pm.com5/30/2012 12:49:21 coupon for 6pm

5/31/20125/31/2012 19:40:38 motel 6 locations5/31/2012 19:45:04 hotels near coney island

5/29/2012 5/29/2012 14:06:04 coney island cincinnati5/29/2012 14:11:49 sas5/29/2012 14:12:01 sas shoes

5/30/2012 5/30/2012 12:12:04 exit #72 and 275 lodging5/30/2012 12:25:19 6pm.com5/30/2012 12:49:21 coupon for 6pm

5/31/20125/31/2012 19:40:38 motel 6 locations5/31/2012 19:45:04 hotels near coney island

Task: an atomic information need that may result in one or more queries.

Clicks Further Enrich Context to Understand Users’ Search Tasks

CS@UCLA ContextAttentiveRankingandSuggestion 3

𝑄$low cost decoration

Clicked document

Skipped document

decoration options for home, patio, outdoor etc. low cost bedroom decorations.

decorate bedroom on a limited budget

furniture for decoration

cheap furniture for bedroom. buying used furniture is an alternative route.

used and cheap futon, dressers, cupboards.

Guides to reformulate query

Previous clicks help to rank documents

Summary of clicked documents

Our Proposal

• Attend on previous queries/clicks to perform retrieval tasks• Learning to utilize context in multiple retrieval activities

q2 q3 q6q5q0

Rank candidate documents

D2D3 D5

Predict next query

A Context Attentive Solution• A two-level hierarchical

structure

• Task context embedding• Session-query and

session-click encoders• Context attentive

representations

• Multi-task Learning• Document Ranking• Query Suggestion

SD1 SD

SQ1 SQ

Q1 Q2 Q3

Ranker

Rec Q3

SD1 SD

SQ1 SQ

Q1 Q2 Q3

Ranker

Recomm

Query Encoder

Query Session Encoder

Document Session Encoder

Inner AttentionInner Attention

it masters nysoftware engineer ny it position ny healthcare2018

Inner Attention

Clicked Documents

Documents

Inner AttentionDocument

Encoder

master’s degree hospital jobs

Recommender

Ranker

Encoders?

This one should give us better visibility

Ranker

Recomm

Query Encoder

Session-Query Encoder

Session-Click Encoder

Encoder

Inner Attention

Clicked Documents

Documents

Ranker

Recomm

enderQuery Encoder

Encoder

Inner Attention

Clicked Documents

Ranker Recommender

………..

Maxout neuron

Projection + Softmax

Session-levelClick Attention

Session-levelQuery Attention

Ranker

Recomm

Query Encoder

Encoder

Inner Attention

Clicked Documents

Context-AttentiveRepresentation

lower-level

Context Attentive Ranking and Suggestion

structure

representations

SD1 SD

SQ1 SQ

Q1 Q2 Q3

Ranker

Rec Q3

SD1 SD

SQ1 SQ

Q1 Q2 Q3

Ranker

Recomm

Query Encoder

Inner Attention

Clicked Documents

Documents

Encoder

Recommender

Ranker

Encoders?

Ranker

Recomm

Query Encoder

Encoder

Inner Attention

Clicked Documents

Documents

Ranker

Recomm

enderQuery Encoder

Encoder

Inner Attention

Clicked Documents

Ranker Recommender

………..

Maxout neuron

Ranker

Recomm

Query Encoder

Encoder

Inner Attention

Clicked Documents

upper-level

structure

representations

SD1 SD

SQ1 SQ

Q1 Q2 Q3

Ranker

Rec Q3

SD1 SD

SQ1 SQ

Q1 Q2 Q3

Ranker

Recomm

Query Encoder

Inner Attention

Clicked Documents

Documents

Encoder

Recommender

Ranker

Encoders?

Ranker

Recomm

Query Encoder

Encoder

Inner Attention

Clicked Documents

Documents

Ranker

Recomm

enderQuery Encoder

Encoder

Inner Attention

Clicked Documents

Ranker Recommender

………..

Maxout neuron

Ranker

Recomm

Query Encoder

Encoder

Inner Attention

Clicked Documents

structure

representations

SD1 SD

SQ1 SQ

Q1 Q2 Q3

Ranker

Rec Q3

SD1 SD

SQ1 SQ

Q1 Q2 Q3

Ranker

Recomm

Query Encoder

Inner Attention

Clicked Documents

Documents

Encoder

Recommender

Ranker

Encoders?

Ranker

Recomm

Query Encoder

Encoder

Inner Attention

Clicked Documents

Documents

Ranker

Recomm

enderQuery Encoder

Encoder

Inner Attention

Clicked Documents

Ranker Recommender

………..

Maxout neuron

Ranker

Recomm

Query Encoder

Encoder

Inner Attention

Clicked Documents

Multi-task Learning Objective

• Optimized via Regularized Multi-task Learning

ContextAttentiveRankingandSuggestion 9WasiUddinAhmad@UCLA

Negative log-likelihoodRegularization

𝑄$ low cost decoration

decorate bedroom on a limited budget

Task 1: document ranking

furniture for decoration

Task 2: next query prediction𝑄#

Experiments

Data Source

• AOL search log – 13 weeks search log of ~650k users• Background set – 5 weeks• Training set – 6 weeks• Validation and Test set – 2 weeks

• Aggregation of candidate documents for ranking• Candidates are sampled from the top 1000 documents retrieved by

BM25 from a pool of 1M documents

• Task Segmentation• Averaging the query term vectors → Query embedding• Consecutive queries* with cosine similarity > 0.5

* within 30 mins interval

Data Statistics

• Only the document titles are utilized in the experiments

Evaluation Metrics and Baselines

• Document Ranking• Metrics – MAP, MRR, NDCG@k (where k=1,3,10)• Baselines – BM25, QL, FixInt, DSSM, DUET, MNSRF etc.

• Query Suggestion• Metrics – MRR, F1, BLEU-k (where k=1,2,3,4)• Baselines – HRED-qs, Seq2seq+Attn, MNSRF etc.

MRR – assesses discrimination ability, rank a list of candidate queries that might follow a given query

Evaluation Metrics and Baselines

• Document Ranking• Metrics – MAP, MRR, NDCG@k (where k=1,3,10)• Baselines – BM25, QL, FixInt, DSSM, DUET, MNSRF etc.

• Query Suggestion• Metrics – MRR, F1, BLEU-k (where k=1,2,3,4)• Baselines – HRED-qs, Seq2seq+Attn, MNSRF etc.

F1, BLEU-k – assesses generation ability, measures overlapping between the generated query term sequence and ground-truth sequence.

Evaluation on Document Ranking

− Vocabulary gap limits the performance− Do not utilize any search context information

− Do not utilize any search context information

+ Jointly learns ranking and suggestion− Utilizes query history but no click history

+ Jointly learns ranking and suggestion+ Utilizes both query and click history

Evaluation on Query Suggestion

Do not utilize any context

Do not utilize click history

+ Utilizes both in-task query and click history

Ablation Study

• Modeling search context is beneficial• Joint learning of related retrieval tasks results in improvements

∗ indicates that the attention in the query recommender

Ablation Study

Performance drops

Ablation Study

Performance dropsPerformance

increases

Effect of Context Modeling

Hypothesis – longer tasks are intrinsically more difficult.

Short tasks (with 2 queries)Medium tasks (with 3–4 queries)Long tasks (with 5+ queries)

Effect of Context Modeling

• Context information helps more on short/medium tasks• Longer tasks are intrinsically more difficult.

Sample Complexity

In terms of #parameters, CARS > MNSRF > M-Match Tensor

10/14/19 NeuralTaskModeling 28

Case Analysis

How in-task previous queries and clicks impact predicting the next query and click for it?

Conclusion & Future Works

• A task-based approach of learning search context• Exploiting users’ on-task search query and click sequence• Jointly optimized on two companion retrieval tasks

• Future works• Modeling across-task relatedness, e.g., users’ long-term search

interest• Apply to any scenario where a user sequentially interacts with a

system

Thank You!Q&A

In-task Context: a richer way to understand users’ search intent

https://github.com/wasiahmad/context_attentive_irCodes will be released soon!

Context Attentive Document Ranking and Query...

Documents

74LVC574A Octal D-type flip-flop with 5 V tolerant inputs/outputs; … · 2017-07-20 · D3 Q2 D2 Q1 D1 Q0 D0 D FF1 Q CP CP D FF2 Q CP D FF3 Q CP D FF4 Q CP D FF5 Q CP D FF6 Q CP

Q H WD [ D $ UD F H D H IUR P . D OLP D Q WD Q % R UQ H R

D W D +D Q G OLQ J ,P S R U W &OH D Q LQ J D Q G 9LV X D ... · 'D W D +D Q G OLQ J ,P S R U W &OH D Q LQ J D Q G 9LV X D OLV D W LR Q / H FW X UH 3 UR JUD P P LQ J Z LW K 'D W D

24 dias letivos D S T Q Q S S D S T Q Q S S D S T Q Q S S

IR Models based on Predicate Logic · IR and Databases The Logic View Retrieval IDB: given query q, nd objects o with o !q IIR: given query q, nd documents d with high values of P(d

A Model of Product Design & Information Disclosure Investments · MODEL DEFINITION 1/2 d 1 q 1 (d 1,q 1) S 1 S 2 S 3 S 4 (d 2,q 2) d 2A d 2B q 2A q 2B (d 3,q 3) q 3A q 3B d 3 d 4A

Implement D Flip-flop by T Flip-flop 0 1 0 1 0 1 1 0 DT Q Q 0101 0101 T = D Q’ + D’ Q D D’ T

c Ken Q Q (D o 81:) (D O Q (D (D o Q c c 03 o Q Q 3 m O 3 ...€¦ · Q O c Q c Q (D (D Q o o Q O O 3 o Q (.n Q Q o (D Q Q O n Q O c (D o Q Q . Created Date: 3/22/2017 3:29:48 PM

Z& ODj Q D )>Z Q UU)D> - HospiVision

Jeopardy 3-d shapes2-d figuresHeading4 Heading5 Q $100 Q $200 Q $300 Q $400 Q $500 Q $100 Q $200 Q $300 Q $400 Q $500 Final Jeopardy variables

Q g d,Q q gggh

Q@=#'=#',D*'?#'%, ' !#$%&'()&**$ +,#'$(-.deddingtontennis.com/windmill.pdf · Q@=#'=#',D*'?"#'%, ' Created Date: 20180730195230Z

Y L S LV Y -LV P S L7 8 - KRUWANDEEq d luq lp v q d x lx a p l d q lq x p d z q" y -lv p s l7 8 k g q x ( q d x lld q llx % x d aq x q p q d liy x a q a k iol q d a 7 x y l e d q lq

Q ? D )> Q>D

)>Z Q> )D> )D>

Q ) Q ) D 8 Z ) = D U Q i Z E Q ) D U

,Q G H [ WR Q H Z Q D PH V D Q G F R P E LQ D WLR Q V D S

R F D O 0 D Q D J H P H Q W R I $ Q G H D Q : H WOD Q G ... · / r f d o 0 d q d j h p h q w r i $ q g h d q : h wod q g v lq 6 d md p d 1 d wlr q d o 3 d un % r oly ld $ x wk r

D S T Q Q S S D S T Q Q S S D S T Q Q S S D S T Q Q S S D ... · Calendário Acadêmico 2016 D S T Q Q S S D S T Q Q S S D S T Q Q S S D S T Q Q S S D S T Q Q S S D S JAN 01 02 03

Í > ¬¿¬ Èù^q D «q ^q C · Í > ¬¿¬ Èù^q D «q ^q C y R å D Í > ¬¿¬ Èù^q «qx D Ô t B^ z D Ô p w Ô wq8pz Í >¢ ¬ ×Ï·ï» ^ ÔtSMo 5^ `h{y «qpxz Ú « z