Data Mining & Machine Learning

CS57300Purdue University

April 10, 2018

1

Predicting Sequences

2

But first, a detour to Noise Contrastive Estimation

3

Bruno Ribeiro

} Machine learning methods are much better at classifying examples than generating new ones

◦ In classification tasks, we use the exact derivatives to find a solution that maximizes the likelihood

◦ In generative tasks, we can only compute an estimate of the derivative

} Because we have better techniques to classify data than to generate data…

◦ Can we make generative tasks look more like classification tasks?

4

Learn a Classifier to Distinguishing Noise from Data

• This is key idea behind noise contrastive estimation (NCE): make generative tasks look like classification tasks

• Pioneered by Hastie, Tibshirani, Friedman in The Elements of Statistical Learning in 2008, Section 14.2.4 “Unsupervised Learning as Supervised Learning”

• The idea is quite simple, consider the task of learning to distinguish noise from the data, i.e., search for a good classifier f(x ;W)

5

RandomNoise

Examples

TrainingData

Examples

True example generator:

Classifier .

Label y1=1 (= true data) Label y=0 (= false data)x1=

x2=

: noise generator

Generation Task Using Classifier

• Now use the learned classifier (which distinguishes noise from the data) to generate new data… but how?

• Naïve approach: Generate examples from the random noise… whatever gets classified as “real data” will be our generated examples

• What is the problem with this naïve approach?

• In very high dimensions (e.g., images), true random noise will not generate any interesting examples

6

RandomNoise

Examples

x1= x2=

x1 x2 x3…

Go to iPython notebook

7

Back to sequences

8

} In this lecture we will focus on word sequences (a.k.a. text)

} The techniques we see are applicable to any type of sequence

} A sequence is a succession of elements from a set (likely finite)

} We will write a sequence of n elements as x1, …, xn

} The temporal ordering is key to learning the sequence

9

Sequences

} Embed words w.r.t. their sentences◦ “Bring me a constant woman to her husband” ◦ “Forgetting, like a good man your late

censure” of his wife

} Under the Markov assumption ◦ P[bring, me, a, constant, woman, to, her, husband] =

P[bring]P[me | bring] P[a | me] P[constant| a] … P[husband | her]

◦ Go to ipython notebook

Word Sequences “Rank-3” word embedding

www.tensorflow.org

© 2017 Bruno Ribeiro

} Embed words w.r.t. their sentences◦ “Bring me a constant woman to her husband” ◦ “Forgetting, like a good man your late

censure” of his wife

} Initial idea by (Chen et al. 2012) was named Latent Markov Embedding, rediscovered by (Mikolov et al. 2013) named word2vec

} Main difference: ◦ Application: (Mikolov et al. 2013) paid attention to the composition of latent

vectors in sentences◦ Otherwise, techniques equivalent

Word Sequences & Embeddings

husbandwife

woman

man

“Rank-3” word embedding

Chen, S., Moore, J. L., Turnbull, D., & Joachims, T. (2012). Playlist prediction via metric embedding. In ACM SIGKDD.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In NIPS

www.tensorflow.org

© 2017 Bruno Ribeiro

How the math works◦ “Bring me a constant woman to her husband” ◦ “Forgetting, like a good man your late

censure” of his wife

} Conditional bag-of-words assumption (SKIPGRAM):All words are independent given the target word◦ P[bring, constant, husband | woman] =

P[bring | woman] P[constant | woman] P[husband | woman]

◦ P[forgetting, like, good, late, censure, wife | man] = P[forgetting | man] P[like | man] P[good | man] P[late | man] P[censure | man] P[wife | man]

Word2vec Embeddings

husbandwife

woman

man

“Rank-3” word embedding

www.tensorflow.org

© 2017 Bruno Ribeiro

Word2vec Type Embeddings III

+u1

v1σ1

u2

v2σ2

1stco

ordi

nate

of c

ondi

tiona

l wor

d

Embedding of “woman”as conditional word

2nd

coor

dina

te o

f con

ditio

nal w

ord

1st coordinate of output words

woman

2nd coord. of output words

husband

woman

husband

X =exp

The machine learning challenge is not summing over all words © 2017 Bruno Ribeiro

Matrix Factorization as a Neural Network (v2)

• But neural networks can have isomorphic network representations

• Matrix factorization:

• Neural network representation:

14

… … … …

X ⇡ U⌃V T =KX

k=1

�kukvTk

<latexit sha1_base64="cIPMw6+Acjie9pp8Tt1pQFH35g4=">AAAHoHic3VVNbxMxEN0USkr4auHIxW0VKRQabSIQSChSJS6VemgrmrYiu428XiexYq9Xtrdtau3P4MfxOzhyYfaDkDRtbyCBpdU+z8zz2G9GdhBzpo3rfqss3bu//KC68rD26PGTp89W154fa5koQrtEcqlOA6wpZxHtGmY4PY0VxSLg9CQYf8r8J+dUaSajIzOJqS/wMGIDRrABU3/16ynycBwreYmsly9nFQ3TbvprFmI1HipKo9T7zIYCp9O4gCc0Rcfp2RHqIE8nom/HnVZ6tocWueDPyf3xXJpk1pCvdw6Ws6P+6qbbdPOBFkGrBJtOOQ76a8vfvVCSRNDIEI617rXc2PgWK8MIp2nNSzSNMRnjIe2F5yzWERZU+/Yyz52iOvhDNJAKvsig3DpLslhoPREBRApsRvq6LzPe5OslZvDBtyyKE0MjUiQaJBwZibJ6oJApSgyfAMBEMdguIiOsMDFQtbks2dpGSq7T2lwGgcdUSSl8S3BEKJ93W8PGV6Upg5wFCquJ1SMcU90UTBNQJz8vp2amZDb/rb9z1wMOS0FQRC+IFAJH4Zb19gbQgkGghqm1noadx0azK1rzsiQ9TTCnHbfZ8m2tXq8jGg4pnCZU+KJnRizyUaP9xn2FtrdRw/0N2gVoA/gIPASdC4JMmQPGeSdrkzfZNEd+ySdMQZ1Ro9WODXBn4qHPinAAfpnk9ujp+QvOdOoXu7qDubCv9g37SmvpgpBQeAE6/ikhp65bFb3jzP+ctjd1KWj717v0f9J0RlLrYTUUDG4HT8ZUYSNVdpFeMDPiTDCjbemfr0RBw5d308B/nRaYETU4K14geZhdwJJ7hQ1C4ZloXX8UFsFxu9mCGh++3dzZLx+MFeels+E0nJbz3tlxdp0Dp+sQ50dlvbJVeV3dqO5W96uHRehSpeS8cOZG9ctPtZaCjQ==</latexit><latexit sha1_base64="cIPMw6+Acjie9pp8Tt1pQFH35g4=">AAAHoHic3VVNbxMxEN0USkr4auHIxW0VKRQabSIQSChSJS6VemgrmrYiu428XiexYq9Xtrdtau3P4MfxOzhyYfaDkDRtbyCBpdU+z8zz2G9GdhBzpo3rfqss3bu//KC68rD26PGTp89W154fa5koQrtEcqlOA6wpZxHtGmY4PY0VxSLg9CQYf8r8J+dUaSajIzOJqS/wMGIDRrABU3/16ynycBwreYmsly9nFQ3TbvprFmI1HipKo9T7zIYCp9O4gCc0Rcfp2RHqIE8nom/HnVZ6tocWueDPyf3xXJpk1pCvdw6Ws6P+6qbbdPOBFkGrBJtOOQ76a8vfvVCSRNDIEI617rXc2PgWK8MIp2nNSzSNMRnjIe2F5yzWERZU+/Yyz52iOvhDNJAKvsig3DpLslhoPREBRApsRvq6LzPe5OslZvDBtyyKE0MjUiQaJBwZibJ6oJApSgyfAMBEMdguIiOsMDFQtbks2dpGSq7T2lwGgcdUSSl8S3BEKJ93W8PGV6Upg5wFCquJ1SMcU90UTBNQJz8vp2amZDb/rb9z1wMOS0FQRC+IFAJH4Zb19gbQgkGghqm1noadx0azK1rzsiQ9TTCnHbfZ8m2tXq8jGg4pnCZU+KJnRizyUaP9xn2FtrdRw/0N2gVoA/gIPASdC4JMmQPGeSdrkzfZNEd+ySdMQZ1Ro9WODXBn4qHPinAAfpnk9ujp+QvOdOoXu7qDubCv9g37SmvpgpBQeAE6/ikhp65bFb3jzP+ctjd1KWj717v0f9J0RlLrYTUUDG4HT8ZUYSNVdpFeMDPiTDCjbemfr0RBw5d308B/nRaYETU4K14geZhdwJJ7hQ1C4ZloXX8UFsFxu9mCGh++3dzZLx+MFeels+E0nJbz3tlxdp0Dp+sQ50dlvbJVeV3dqO5W96uHRehSpeS8cOZG9ctPtZaCjQ==</latexit><latexit sha1_base64="cIPMw6+Acjie9pp8Tt1pQFH35g4=">AAAHoHic3VVNbxMxEN0USkr4auHIxW0VKRQabSIQSChSJS6VemgrmrYiu428XiexYq9Xtrdtau3P4MfxOzhyYfaDkDRtbyCBpdU+z8zz2G9GdhBzpo3rfqss3bu//KC68rD26PGTp89W154fa5koQrtEcqlOA6wpZxHtGmY4PY0VxSLg9CQYf8r8J+dUaSajIzOJqS/wMGIDRrABU3/16ynycBwreYmsly9nFQ3TbvprFmI1HipKo9T7zIYCp9O4gCc0Rcfp2RHqIE8nom/HnVZ6tocWueDPyf3xXJpk1pCvdw6Ws6P+6qbbdPOBFkGrBJtOOQ76a8vfvVCSRNDIEI617rXc2PgWK8MIp2nNSzSNMRnjIe2F5yzWERZU+/Yyz52iOvhDNJAKvsig3DpLslhoPREBRApsRvq6LzPe5OslZvDBtyyKE0MjUiQaJBwZibJ6oJApSgyfAMBEMdguIiOsMDFQtbks2dpGSq7T2lwGgcdUSSl8S3BEKJ93W8PGV6Upg5wFCquJ1SMcU90UTBNQJz8vp2amZDb/rb9z1wMOS0FQRC+IFAJH4Zb19gbQgkGghqm1noadx0azK1rzsiQ9TTCnHbfZ8m2tXq8jGg4pnCZU+KJnRizyUaP9xn2FtrdRw/0N2gVoA/gIPASdC4JMmQPGeSdrkzfZNEd+ySdMQZ1Ro9WODXBn4qHPinAAfpnk9ujp+QvOdOoXu7qDubCv9g37SmvpgpBQeAE6/ikhp65bFb3jzP+ctjd1KWj717v0f9J0RlLrYTUUDG4HT8ZUYSNVdpFeMDPiTDCjbemfr0RBw5d308B/nRaYETU4K14geZhdwJJ7hQ1C4ZloXX8UFsFxu9mCGh++3dzZLx+MFeels+E0nJbz3tlxdp0Dp+sQ50dlvbJVeV3dqO5W96uHRehSpeS8cOZG9ctPtZaCjQ==</latexit><latexit sha1_base64="cIPMw6+Acjie9pp8Tt1pQFH35g4=">AAAHoHic3VVNbxMxEN0USkr4auHIxW0VKRQabSIQSChSJS6VemgrmrYiu428XiexYq9Xtrdtau3P4MfxOzhyYfaDkDRtbyCBpdU+z8zz2G9GdhBzpo3rfqss3bu//KC68rD26PGTp89W154fa5koQrtEcqlOA6wpZxHtGmY4PY0VxSLg9CQYf8r8J+dUaSajIzOJqS/wMGIDRrABU3/16ynycBwreYmsly9nFQ3TbvprFmI1HipKo9T7zIYCp9O4gCc0Rcfp2RHqIE8nom/HnVZ6tocWueDPyf3xXJpk1pCvdw6Ws6P+6qbbdPOBFkGrBJtOOQ76a8vfvVCSRNDIEI617rXc2PgWK8MIp2nNSzSNMRnjIe2F5yzWERZU+/Yyz52iOvhDNJAKvsig3DpLslhoPREBRApsRvq6LzPe5OslZvDBtyyKE0MjUiQaJBwZibJ6oJApSgyfAMBEMdguIiOsMDFQtbks2dpGSq7T2lwGgcdUSSl8S3BEKJ93W8PGV6Upg5wFCquJ1SMcU90UTBNQJz8vp2amZDb/rb9z1wMOS0FQRC+IFAJH4Zb19gbQgkGghqm1noadx0azK1rzsiQ9TTCnHbfZ8m2tXq8jGg4pnCZU+KJnRizyUaP9xn2FtrdRw/0N2gVoA/gIPASdC4JMmQPGeSdrkzfZNEd+ySdMQZ1Ro9WODXBn4qHPinAAfpnk9ujp+QvOdOoXu7qDubCv9g37SmvpgpBQeAE6/ikhp65bFb3jzP+ctjd1KWj717v0f9J0RlLrYTUUDG4HT8ZUYSNVdpFeMDPiTDCjbemfr0RBw5d308B/nRaYETU4K14geZhdwJJ7hQ1C4ZloXX8UFsFxu9mCGh++3dzZLx+MFeels+E0nJbz3tlxdp0Dp+sQ50dlvbJVeV3dqO5W96uHRehSpeS8cOZG9ctPtZaCjQ==</latexit>

U<latexit sha1_base64="20r+kfvoJZRFZ60S1mDkDrHzzOQ=">AAAHF3ic3VXNbhMxEHYLDSX8tIUjly1VpIDaaBOBQEKVKnFB4kCRSFspu1Re7ySxYq9Xtrdtau17wBFehBviypHn4AWYza5C07S9gQSWVvt5Zj7P+BvLjlLBjfX9HwuL164v1W4s36zfun3n7srq2r09ozLNoMuUUPogogYET6BruRVwkGqgMhKwH41eFv79I9CGq+SdHacQSjpIeJ8zatH03gWTJZyGOO/mh6sbfsufDG8etCuwQaqxe7i29DOIFcskJJYJakyv7ac2dFRbzgTk9SAzkFI2ogPoxUc8NQmVYEJ3Mkmbew30x15fafwS602sZ0mOSmPGMsJISe3QnPcVxot8vcz2n4eOJ2lmIWFlon4mPKu8QgYv5hqYFWMElGmO5XpsSDVlFsWayVKsbZUSJq/PZJB0BFopGTpGEwZi1u0sH51WpgIKHmmqx84MaQqmJblhqM5kvwKsi6keDTRAkrvJb/2pvx4JXAqDEjhmSkqaxI9d8LqPnY8iPcidCwxWnlrDT6EeFEl6hlEB236rHbp6o9HwIB4A7ibW9LhnhzwJvWZn03/kbW15Tf836JSgg+AF8jw8MCjIlNnnQmxHIoPNYjpBYcVnXGOfvWa7k1rknonHE1WGIwirJJdHT/dfcqbTsKzqCuZcXZ0L6srr+ZyQ2HiJOv4pIaeuSxW9Ys//nLYXnVLU9q+f0v9J0zOSuoDqgeR4OwQqBU2t0sVFesztUHDJrXGVf7YTJY2eXE1D/3laZIdgadG8SIm4uICVCEobhuIz0T7/KMyDvU6rjT1++2Rj5031YCyTB+QhaZI2eUZ2yCuyS7qEEU0+kE/kc+1j7Uvta+1bGbq4UHHuk5lR+/4LKtZP9A==</latexit><latexit sha1_base64="20r+kfvoJZRFZ60S1mDkDrHzzOQ=">AAAHF3ic3VXNbhMxEHYLDSX8tIUjly1VpIDaaBOBQEKVKnFB4kCRSFspu1Re7ySxYq9Xtrdtau17wBFehBviypHn4AWYza5C07S9gQSWVvt5Zj7P+BvLjlLBjfX9HwuL164v1W4s36zfun3n7srq2r09ozLNoMuUUPogogYET6BruRVwkGqgMhKwH41eFv79I9CGq+SdHacQSjpIeJ8zatH03gWTJZyGOO/mh6sbfsufDG8etCuwQaqxe7i29DOIFcskJJYJakyv7ac2dFRbzgTk9SAzkFI2ogPoxUc8NQmVYEJ3Mkmbew30x15fafwS602sZ0mOSmPGMsJISe3QnPcVxot8vcz2n4eOJ2lmIWFlon4mPKu8QgYv5hqYFWMElGmO5XpsSDVlFsWayVKsbZUSJq/PZJB0BFopGTpGEwZi1u0sH51WpgIKHmmqx84MaQqmJblhqM5kvwKsi6keDTRAkrvJb/2pvx4JXAqDEjhmSkqaxI9d8LqPnY8iPcidCwxWnlrDT6EeFEl6hlEB236rHbp6o9HwIB4A7ibW9LhnhzwJvWZn03/kbW15Tf836JSgg+AF8jw8MCjIlNnnQmxHIoPNYjpBYcVnXGOfvWa7k1rknonHE1WGIwirJJdHT/dfcqbTsKzqCuZcXZ0L6srr+ZyQ2HiJOv4pIaeuSxW9Ys//nLYXnVLU9q+f0v9J0zOSuoDqgeR4OwQqBU2t0sVFesztUHDJrXGVf7YTJY2eXE1D/3laZIdgadG8SIm4uICVCEobhuIz0T7/KMyDvU6rjT1++2Rj5031YCyTB+QhaZI2eUZ2yCuyS7qEEU0+kE/kc+1j7Uvta+1bGbq4UHHuk5lR+/4LKtZP9A==</latexit><latexit sha1_base64="20r+kfvoJZRFZ60S1mDkDrHzzOQ=">AAAHF3ic3VXNbhMxEHYLDSX8tIUjly1VpIDaaBOBQEKVKnFB4kCRSFspu1Re7ySxYq9Xtrdtau17wBFehBviypHn4AWYza5C07S9gQSWVvt5Zj7P+BvLjlLBjfX9HwuL164v1W4s36zfun3n7srq2r09ozLNoMuUUPogogYET6BruRVwkGqgMhKwH41eFv79I9CGq+SdHacQSjpIeJ8zatH03gWTJZyGOO/mh6sbfsufDG8etCuwQaqxe7i29DOIFcskJJYJakyv7ac2dFRbzgTk9SAzkFI2ogPoxUc8NQmVYEJ3Mkmbew30x15fafwS602sZ0mOSmPGMsJISe3QnPcVxot8vcz2n4eOJ2lmIWFlon4mPKu8QgYv5hqYFWMElGmO5XpsSDVlFsWayVKsbZUSJq/PZJB0BFopGTpGEwZi1u0sH51WpgIKHmmqx84MaQqmJblhqM5kvwKsi6keDTRAkrvJb/2pvx4JXAqDEjhmSkqaxI9d8LqPnY8iPcidCwxWnlrDT6EeFEl6hlEB236rHbp6o9HwIB4A7ibW9LhnhzwJvWZn03/kbW15Tf836JSgg+AF8jw8MCjIlNnnQmxHIoPNYjpBYcVnXGOfvWa7k1rknonHE1WGIwirJJdHT/dfcqbTsKzqCuZcXZ0L6srr+ZyQ2HiJOv4pIaeuSxW9Ys//nLYXnVLU9q+f0v9J0zOSuoDqgeR4OwQqBU2t0sVFesztUHDJrXGVf7YTJY2eXE1D/3laZIdgadG8SIm4uICVCEobhuIz0T7/KMyDvU6rjT1++2Rj5031YCyTB+QhaZI2eUZ2yCuyS7qEEU0+kE/kc+1j7Uvta+1bGbq4UHHuk5lR+/4LKtZP9A==</latexit><latexit sha1_base64="20r+kfvoJZRFZ60S1mDkDrHzzOQ=">AAAHF3ic3VXNbhMxEHYLDSX8tIUjly1VpIDaaBOBQEKVKnFB4kCRSFspu1Re7ySxYq9Xtrdtau17wBFehBviypHn4AWYza5C07S9gQSWVvt5Zj7P+BvLjlLBjfX9HwuL164v1W4s36zfun3n7srq2r09ozLNoMuUUPogogYET6BruRVwkGqgMhKwH41eFv79I9CGq+SdHacQSjpIeJ8zatH03gWTJZyGOO/mh6sbfsufDG8etCuwQaqxe7i29DOIFcskJJYJakyv7ac2dFRbzgTk9SAzkFI2ogPoxUc8NQmVYEJ3Mkmbew30x15fafwS602sZ0mOSmPGMsJISe3QnPcVxot8vcz2n4eOJ2lmIWFlon4mPKu8QgYv5hqYFWMElGmO5XpsSDVlFsWayVKsbZUSJq/PZJB0BFopGTpGEwZi1u0sH51WpgIKHmmqx84MaQqmJblhqM5kvwKsi6keDTRAkrvJb/2pvx4JXAqDEjhmSkqaxI9d8LqPnY8iPcidCwxWnlrDT6EeFEl6hlEB236rHbp6o9HwIB4A7ibW9LhnhzwJvWZn03/kbW15Tf836JSgg+AF8jw8MCjIlNnnQmxHIoPNYjpBYcVnXGOfvWa7k1rknonHE1WGIwirJJdHT/dfcqbTsKzqCuZcXZ0L6srr+ZyQ2HiJOv4pIaeuSxW9Ys//nLYXnVLU9q+f0v9J0zOSuoDqgeR4OwQqBU2t0sVFesztUHDJrXGVf7YTJY2eXE1D/3laZIdgadG8SIm4uICVCEobhuIz0T7/KMyDvU6rjT1++2Rj5031YCyTB+QhaZI2eUZ2yCuyS7qEEU0+kE/kc+1j7Uvta+1bGbq4UHHuk5lR+/4LKtZP9A==</latexit>

⌃<latexit sha1_base64="Sm8SXrapGkH1G+uhJbid190WSxc=">AAAHJHic3VXdahQxFE7VrnX9a/VKvEktC6u0y8yiKEih4I3ghRXtD+wMJZM5Oxs2mQxJpu02DD6NXuqLeCdeeONL+AJmdoZ1t9v2TkEDw3w553w5J98JSZRxpo3nfV+4dPnKYuPq0rXm9Rs3b91eXrmzq2WuKOxQyaXaj4gGzlLYMcxw2M8UEBFx2IuGL0r/3iEozWT6zowyCAVJUtZnlBhnOli+Z4PxIjYmapgogLQI3rJEkOJgec3reOOB54FfgzVUj+2DlcWfQSxpLiA1lBOte76XmdASZRjlUDSDXENG6JAk0IsPWaZTIkCH9nhcQYFbzh/jvlTuSw0eW6dJlgitRyJykYKYgT7tK41n+Xq56T8LLUuz3EBKq0T9nGMjcakJjpkCavjIAUIVc+ViOiCKUOOUm8lSrm2k5LpozmQQZAhKShFaSlIKfNZtDRue1KYSchYpokZWD0gGuiOYpk6d8X45mKlW2PFv9Ym3GnG3lAtK4YhKIUgaP7LBq747BlGkksLaQLvKM6PZCTSDMklPU8Jh0+v4oW22Wi0McQJuN7EiRz0zYGmI29117yHe2MBt7zfoVqDrwHPHw+70OEEmzD7jfDPiOayX0zEKaz5lyvUZt/1uZhx3Kl5BXIU7ENZJzo+e7L/iTKZhVdUFzLm6umfUVTSLOSFd44XT8U8JOXGdq+gFe/7ntD3rlDpt//op/Z80nZLUBkQlgrnbIZAZKGKkKi/SI2YGnAlmtK39s52oaOT4Yprzn6ZFZgCGlM2LJI/LC1jyoLK5UPdM+KcfhXmw2+34rsdvHq9tva4fjCV0Hz1AbeSjp2gLvUTbaAdR9B59QJ/Q58bHxpfG18a3KvTSQs25i2ZG48cv7EhVAw==</latexit><latexit sha1_base64="Sm8SXrapGkH1G+uhJbid190WSxc=">AAAHJHic3VXdahQxFE7VrnX9a/VKvEktC6u0y8yiKEih4I3ghRXtD+wMJZM5Oxs2mQxJpu02DD6NXuqLeCdeeONL+AJmdoZ1t9v2TkEDw3w553w5J98JSZRxpo3nfV+4dPnKYuPq0rXm9Rs3b91eXrmzq2WuKOxQyaXaj4gGzlLYMcxw2M8UEBFx2IuGL0r/3iEozWT6zowyCAVJUtZnlBhnOli+Z4PxIjYmapgogLQI3rJEkOJgec3reOOB54FfgzVUj+2DlcWfQSxpLiA1lBOte76XmdASZRjlUDSDXENG6JAk0IsPWaZTIkCH9nhcQYFbzh/jvlTuSw0eW6dJlgitRyJykYKYgT7tK41n+Xq56T8LLUuz3EBKq0T9nGMjcakJjpkCavjIAUIVc+ViOiCKUOOUm8lSrm2k5LpozmQQZAhKShFaSlIKfNZtDRue1KYSchYpokZWD0gGuiOYpk6d8X45mKlW2PFv9Ym3GnG3lAtK4YhKIUgaP7LBq747BlGkksLaQLvKM6PZCTSDMklPU8Jh0+v4oW22Wi0McQJuN7EiRz0zYGmI29117yHe2MBt7zfoVqDrwHPHw+70OEEmzD7jfDPiOayX0zEKaz5lyvUZt/1uZhx3Kl5BXIU7ENZJzo+e7L/iTKZhVdUFzLm6umfUVTSLOSFd44XT8U8JOXGdq+gFe/7ntD3rlDpt//op/Z80nZLUBkQlgrnbIZAZKGKkKi/SI2YGnAlmtK39s52oaOT4Yprzn6ZFZgCGlM2LJI/LC1jyoLK5UPdM+KcfhXmw2+34rsdvHq9tva4fjCV0Hz1AbeSjp2gLvUTbaAdR9B59QJ/Q58bHxpfG18a3KvTSQs25i2ZG48cv7EhVAw==</latexit><latexit sha1_base64="Sm8SXrapGkH1G+uhJbid190WSxc=">AAAHJHic3VXdahQxFE7VrnX9a/VKvEktC6u0y8yiKEih4I3ghRXtD+wMJZM5Oxs2mQxJpu02DD6NXuqLeCdeeONL+AJmdoZ1t9v2TkEDw3w553w5J98JSZRxpo3nfV+4dPnKYuPq0rXm9Rs3b91eXrmzq2WuKOxQyaXaj4gGzlLYMcxw2M8UEBFx2IuGL0r/3iEozWT6zowyCAVJUtZnlBhnOli+Z4PxIjYmapgogLQI3rJEkOJgec3reOOB54FfgzVUj+2DlcWfQSxpLiA1lBOte76XmdASZRjlUDSDXENG6JAk0IsPWaZTIkCH9nhcQYFbzh/jvlTuSw0eW6dJlgitRyJykYKYgT7tK41n+Xq56T8LLUuz3EBKq0T9nGMjcakJjpkCavjIAUIVc+ViOiCKUOOUm8lSrm2k5LpozmQQZAhKShFaSlIKfNZtDRue1KYSchYpokZWD0gGuiOYpk6d8X45mKlW2PFv9Ym3GnG3lAtK4YhKIUgaP7LBq747BlGkksLaQLvKM6PZCTSDMklPU8Jh0+v4oW22Wi0McQJuN7EiRz0zYGmI29117yHe2MBt7zfoVqDrwHPHw+70OEEmzD7jfDPiOayX0zEKaz5lyvUZt/1uZhx3Kl5BXIU7ENZJzo+e7L/iTKZhVdUFzLm6umfUVTSLOSFd44XT8U8JOXGdq+gFe/7ntD3rlDpt//op/Z80nZLUBkQlgrnbIZAZKGKkKi/SI2YGnAlmtK39s52oaOT4Yprzn6ZFZgCGlM2LJI/LC1jyoLK5UPdM+KcfhXmw2+34rsdvHq9tva4fjCV0Hz1AbeSjp2gLvUTbaAdR9B59QJ/Q58bHxpfG18a3KvTSQs25i2ZG48cv7EhVAw==</latexit><latexit sha1_base64="Sm8SXrapGkH1G+uhJbid190WSxc=">AAAHJHic3VXdahQxFE7VrnX9a/VKvEktC6u0y8yiKEih4I3ghRXtD+wMJZM5Oxs2mQxJpu02DD6NXuqLeCdeeONL+AJmdoZ1t9v2TkEDw3w553w5J98JSZRxpo3nfV+4dPnKYuPq0rXm9Rs3b91eXrmzq2WuKOxQyaXaj4gGzlLYMcxw2M8UEBFx2IuGL0r/3iEozWT6zowyCAVJUtZnlBhnOli+Z4PxIjYmapgogLQI3rJEkOJgec3reOOB54FfgzVUj+2DlcWfQSxpLiA1lBOte76XmdASZRjlUDSDXENG6JAk0IsPWaZTIkCH9nhcQYFbzh/jvlTuSw0eW6dJlgitRyJykYKYgT7tK41n+Xq56T8LLUuz3EBKq0T9nGMjcakJjpkCavjIAUIVc+ViOiCKUOOUm8lSrm2k5LpozmQQZAhKShFaSlIKfNZtDRue1KYSchYpokZWD0gGuiOYpk6d8X45mKlW2PFv9Ym3GnG3lAtK4YhKIUgaP7LBq747BlGkksLaQLvKM6PZCTSDMklPU8Jh0+v4oW22Wi0McQJuN7EiRz0zYGmI29117yHe2MBt7zfoVqDrwHPHw+70OEEmzD7jfDPiOayX0zEKaz5lyvUZt/1uZhx3Kl5BXIU7ENZJzo+e7L/iTKZhVdUFzLm6umfUVTSLOSFd44XT8U8JOXGdq+gFe/7ntD3rlDpt//op/Z80nZLUBkQlgrnbIZAZKGKkKi/SI2YGnAlmtK39s52oaOT4Yprzn6ZFZgCGlM2LJI/LC1jyoLK5UPdM+KcfhXmw2+34rsdvHq9tva4fjCV0Hz1AbeSjp2gLvUTbaAdR9B59QJ/Q58bHxpfG18a3KvTSQs25i2ZG48cv7EhVAw==</latexit>

Element-wise product

V<latexit sha1_base64="RjX2VIY0zv0WBBOsgMdNfNkNeHM=">AAAHG3ic3VXNbhMxEHYLDSX8pXDk4lJFCqiNdiMQSKhSJS5IHCgS/ZGyq8rrnSRW7PXK9qZNrX0TOMKLcENcOfAcvADeH4WmaXsDCSyt9vPMfJ7xN5YdpZxp43k/lpavXV9p3Fi92bx1+87de621+/taZorCHpVcqsOIaOAsgT3DDIfDVAEREYeDaPyq8B9MQGkmk/dmmkIoyDBhA0aJcaajVssG5SI24hnkeD8/am14Xa8ceBH4NdhA9dg9Wlv5GcSSZgISQznRuu97qQktUYZRDnkzyDSkhI7JEPrxhKU6IQJ0aE/KxDluO3+MB1K5LzG4tJ4lWSK0norIRQpiRvq8rzBe5OtnZvAitCxJMwMJrRINMo6NxIUUOGYKqOFTBwhVzJWL6YgoQo0TbC5LsbaRkuu8OZdBkDEoKUVoKUko8Hm3NWx8WpsKyFmkiJpaPSIp6K5gmjp1yv1yMDYmajxUAEluy9/6M2894m4pF5TAMZVCkCR+YoM3A9f9KFLD3NpAu8pTo9kpNIMiSV9TwmHb6/qhbbbbbQzxENxuYkWO+2bEkhB3epveY7y1hTveb9CrQM+Bl46H3aFxgsyYA8b5dnFGNotpicKaT5lyfcYdv5caxz0TryCuwh0I6ySXR8/2X3Fm07Cq6grmQl29C+rKm/mCkK7xwun4p4ScuS5V9Io9/3PaXnRKnbZ//ZT+T5qekdQGRA0Fc7dDIFNQxEhVXKTHzIw4E8xoW/vnO1HRyMnVNOc/T4vMCAwpmhdJHhcXsORBZXOh7pnwzz8Ki2C/1/Vdj9893dh5Wz8Yq+gheoQ6yEfP0Q56jXbRHqJogj6gT+hz42PjS+Nr41sVurxUcx6gudH4/gtSIlDH</latexit><latexit sha1_base64="RjX2VIY0zv0WBBOsgMdNfNkNeHM=">AAAHG3ic3VXNbhMxEHYLDSX8pXDk4lJFCqiNdiMQSKhSJS5IHCgS/ZGyq8rrnSRW7PXK9qZNrX0TOMKLcENcOfAcvADeH4WmaXsDCSyt9vPMfJ7xN5YdpZxp43k/lpavXV9p3Fi92bx1+87de621+/taZorCHpVcqsOIaOAsgT3DDIfDVAEREYeDaPyq8B9MQGkmk/dmmkIoyDBhA0aJcaajVssG5SI24hnkeD8/am14Xa8ceBH4NdhA9dg9Wlv5GcSSZgISQznRuu97qQktUYZRDnkzyDSkhI7JEPrxhKU6IQJ0aE/KxDluO3+MB1K5LzG4tJ4lWSK0norIRQpiRvq8rzBe5OtnZvAitCxJMwMJrRINMo6NxIUUOGYKqOFTBwhVzJWL6YgoQo0TbC5LsbaRkuu8OZdBkDEoKUVoKUko8Hm3NWx8WpsKyFmkiJpaPSIp6K5gmjp1yv1yMDYmajxUAEluy9/6M2894m4pF5TAMZVCkCR+YoM3A9f9KFLD3NpAu8pTo9kpNIMiSV9TwmHb6/qhbbbbbQzxENxuYkWO+2bEkhB3epveY7y1hTveb9CrQM+Bl46H3aFxgsyYA8b5dnFGNotpicKaT5lyfcYdv5caxz0TryCuwh0I6ySXR8/2X3Fm07Cq6grmQl29C+rKm/mCkK7xwun4p4ScuS5V9Io9/3PaXnRKnbZ//ZT+T5qekdQGRA0Fc7dDIFNQxEhVXKTHzIw4E8xoW/vnO1HRyMnVNOc/T4vMCAwpmhdJHhcXsORBZXOh7pnwzz8Ki2C/1/Vdj9893dh5Wz8Yq+gheoQ6yEfP0Q56jXbRHqJogj6gT+hz42PjS+Nr41sVurxUcx6gudH4/gtSIlDH</latexit><latexit sha1_base64="RjX2VIY0zv0WBBOsgMdNfNkNeHM=">AAAHG3ic3VXNbhMxEHYLDSX8pXDk4lJFCqiNdiMQSKhSJS5IHCgS/ZGyq8rrnSRW7PXK9qZNrX0TOMKLcENcOfAcvADeH4WmaXsDCSyt9vPMfJ7xN5YdpZxp43k/lpavXV9p3Fi92bx1+87de621+/taZorCHpVcqsOIaOAsgT3DDIfDVAEREYeDaPyq8B9MQGkmk/dmmkIoyDBhA0aJcaajVssG5SI24hnkeD8/am14Xa8ceBH4NdhA9dg9Wlv5GcSSZgISQznRuu97qQktUYZRDnkzyDSkhI7JEPrxhKU6IQJ0aE/KxDluO3+MB1K5LzG4tJ4lWSK0norIRQpiRvq8rzBe5OtnZvAitCxJMwMJrRINMo6NxIUUOGYKqOFTBwhVzJWL6YgoQo0TbC5LsbaRkuu8OZdBkDEoKUVoKUko8Hm3NWx8WpsKyFmkiJpaPSIp6K5gmjp1yv1yMDYmajxUAEluy9/6M2894m4pF5TAMZVCkCR+YoM3A9f9KFLD3NpAu8pTo9kpNIMiSV9TwmHb6/qhbbbbbQzxENxuYkWO+2bEkhB3epveY7y1hTveb9CrQM+Bl46H3aFxgsyYA8b5dnFGNotpicKaT5lyfcYdv5caxz0TryCuwh0I6ySXR8/2X3Fm07Cq6grmQl29C+rKm/mCkK7xwun4p4ScuS5V9Io9/3PaXnRKnbZ//ZT+T5qekdQGRA0Fc7dDIFNQxEhVXKTHzIw4E8xoW/vnO1HRyMnVNOc/T4vMCAwpmhdJHhcXsORBZXOh7pnwzz8Ki2C/1/Vdj9893dh5Wz8Yq+gheoQ6yEfP0Q56jXbRHqJogj6gT+hz42PjS+Nr41sVurxUcx6gudH4/gtSIlDH</latexit><latexit sha1_base64="RjX2VIY0zv0WBBOsgMdNfNkNeHM=">AAAHG3ic3VXNbhMxEHYLDSX8pXDk4lJFCqiNdiMQSKhSJS5IHCgS/ZGyq8rrnSRW7PXK9qZNrX0TOMKLcENcOfAcvADeH4WmaXsDCSyt9vPMfJ7xN5YdpZxp43k/lpavXV9p3Fi92bx1+87de621+/taZorCHpVcqsOIaOAsgT3DDIfDVAEREYeDaPyq8B9MQGkmk/dmmkIoyDBhA0aJcaajVssG5SI24hnkeD8/am14Xa8ceBH4NdhA9dg9Wlv5GcSSZgISQznRuu97qQktUYZRDnkzyDSkhI7JEPrxhKU6IQJ0aE/KxDluO3+MB1K5LzG4tJ4lWSK0norIRQpiRvq8rzBe5OtnZvAitCxJMwMJrRINMo6NxIUUOGYKqOFTBwhVzJWL6YgoQo0TbC5LsbaRkuu8OZdBkDEoKUVoKUko8Hm3NWx8WpsKyFmkiJpaPSIp6K5gmjp1yv1yMDYmajxUAEluy9/6M2894m4pF5TAMZVCkCR+YoM3A9f9KFLD3NpAu8pTo9kpNIMiSV9TwmHb6/qhbbbbbQzxENxuYkWO+2bEkhB3epveY7y1hTveb9CrQM+Bl46H3aFxgsyYA8b5dnFGNotpicKaT5lyfcYdv5caxz0TryCuwh0I6ySXR8/2X3Fm07Cq6grmQl29C+rKm/mCkK7xwun4p4ScuS5V9Io9/3PaXnRKnbZ//ZT+T5qekdQGRA0Fc7dDIFNQxEhVXKTHzIw4E8xoW/vnO1HRyMnVNOc/T4vMCAwpmhdJHhcXsORBZXOh7pnwzz8Ki2C/1/Vdj9893dh5Wz8Yq+gheoQ6yEfP0Q56jXbRHqJogj6gT+hz42PjS+Nr41sVurxUcx6gudH4/gtSIlDH</latexit>

One-hot encoding of user One-hot encoding of product1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0

Linear activationLinear activation

X̃1,7 =4X

k=1

�ku1,kv7,k<latexit sha1_base64="6JpLcbwsZUrX8PqDW+OaF+hhNAE=">AAAHcXic3VVLbxMxEHYLbUp4tXBCXNxWkQK00W5UVCQUqRIXJA4UiT6k7BJ5vU5irb27sr1tU2v/IX+Av0GPcGD2oZD0dQMJLEWZnW8+z/ibkR2kgmvjON8WFu/cXVpurNxr3n/w8NHj1bUnhzrJFGUHNBGJOg6IZoLH7MBwI9hxqhiRgWBHQfSuwI9OmNI8iT+bScp8SUYxH3JKDLgGq0PPcBEye5wPrLu1m+Me9nQmBzbqufmXHWy9MocNiYpGirE4B5yPJMkH0RRULMyzcoMonzoDkbH8BLy74B2sbjodp1z4quHWxiaq1/5gbenCCxOaSRYbKojWfddJjW+JMpwKlje9TLOU0IiMWD884amOiWTat2dl9hy3AA/xMFHwiw0uvbMkS6TWExlApCRmrC9jhfM6rJ+Z4Rvf8jjNDItplWiYCWwSXAiMQ64YNWICBqGKQ7mYjoki1EAb5rIUe5skETpvzmWQJGIqSaRvKYkpE/OwNTw6r12FKXigiJpYPSYp0x3JNQV1yvMKZmYaZ8u/9dfOeiBgKwiK2SlNpCRx+NJ6H4YwU0GgRrm1nobKU6P5OWt6RZK+pkSwntNxfdtstVqYhSMGpwkVOe2bMY993O5uOS/w9jZuO7+NbmV0wXgLPAyjCIJMmUMuRK8YlK3is7T8mk+5gj7jtttNDXBn4mHaqnAw/DrJzdHT81ec6adfVXUL80pd3Wvqypv5FSGh8RJ0/FNCTqEbFb3lzP+cttdNKWj716f0f9J0RlLrETWSHG4HL0mZIiZRxUV6ys1YcMmNtjU+34mKRs5upwF+mRaYMTOkaF6QiLC4gBPhVT4IhWfCvfwoXDUOux0XevxpZ3PvY/1grKDnaAO1kYt20R56j/bRAaLoK/qOfqCfyxeNZw3c2KhCFxdqzlM0txqvfgG5j3D0</latexit><latexit sha1_base64="6JpLcbwsZUrX8PqDW+OaF+hhNAE=">AAAHcXic3VVLbxMxEHYLbUp4tXBCXNxWkQK00W5UVCQUqRIXJA4UiT6k7BJ5vU5irb27sr1tU2v/IX+Av0GPcGD2oZD0dQMJLEWZnW8+z/ibkR2kgmvjON8WFu/cXVpurNxr3n/w8NHj1bUnhzrJFGUHNBGJOg6IZoLH7MBwI9hxqhiRgWBHQfSuwI9OmNI8iT+bScp8SUYxH3JKDLgGq0PPcBEye5wPrLu1m+Me9nQmBzbqufmXHWy9MocNiYpGirE4B5yPJMkH0RRULMyzcoMonzoDkbH8BLy74B2sbjodp1z4quHWxiaq1/5gbenCCxOaSRYbKojWfddJjW+JMpwKlje9TLOU0IiMWD884amOiWTat2dl9hy3AA/xMFHwiw0uvbMkS6TWExlApCRmrC9jhfM6rJ+Z4Rvf8jjNDItplWiYCWwSXAiMQ64YNWICBqGKQ7mYjoki1EAb5rIUe5skETpvzmWQJGIqSaRvKYkpE/OwNTw6r12FKXigiJpYPSYp0x3JNQV1yvMKZmYaZ8u/9dfOeiBgKwiK2SlNpCRx+NJ6H4YwU0GgRrm1nobKU6P5OWt6RZK+pkSwntNxfdtstVqYhSMGpwkVOe2bMY993O5uOS/w9jZuO7+NbmV0wXgLPAyjCIJMmUMuRK8YlK3is7T8mk+5gj7jtttNDXBn4mHaqnAw/DrJzdHT81ec6adfVXUL80pd3Wvqypv5FSGh8RJ0/FNCTqEbFb3lzP+cttdNKWj716f0f9J0RlLrETWSHG4HL0mZIiZRxUV6ys1YcMmNtjU+34mKRs5upwF+mRaYMTOkaF6QiLC4gBPhVT4IhWfCvfwoXDUOux0XevxpZ3PvY/1grKDnaAO1kYt20R56j/bRAaLoK/qOfqCfyxeNZw3c2KhCFxdqzlM0txqvfgG5j3D0</latexit><latexit sha1_base64="6JpLcbwsZUrX8PqDW+OaF+hhNAE=">AAAHcXic3VVLbxMxEHYLbUp4tXBCXNxWkQK00W5UVCQUqRIXJA4UiT6k7BJ5vU5irb27sr1tU2v/IX+Av0GPcGD2oZD0dQMJLEWZnW8+z/ibkR2kgmvjON8WFu/cXVpurNxr3n/w8NHj1bUnhzrJFGUHNBGJOg6IZoLH7MBwI9hxqhiRgWBHQfSuwI9OmNI8iT+bScp8SUYxH3JKDLgGq0PPcBEye5wPrLu1m+Me9nQmBzbqufmXHWy9MocNiYpGirE4B5yPJMkH0RRULMyzcoMonzoDkbH8BLy74B2sbjodp1z4quHWxiaq1/5gbenCCxOaSRYbKojWfddJjW+JMpwKlje9TLOU0IiMWD884amOiWTat2dl9hy3AA/xMFHwiw0uvbMkS6TWExlApCRmrC9jhfM6rJ+Z4Rvf8jjNDItplWiYCWwSXAiMQ64YNWICBqGKQ7mYjoki1EAb5rIUe5skETpvzmWQJGIqSaRvKYkpE/OwNTw6r12FKXigiJpYPSYp0x3JNQV1yvMKZmYaZ8u/9dfOeiBgKwiK2SlNpCRx+NJ6H4YwU0GgRrm1nobKU6P5OWt6RZK+pkSwntNxfdtstVqYhSMGpwkVOe2bMY993O5uOS/w9jZuO7+NbmV0wXgLPAyjCIJMmUMuRK8YlK3is7T8mk+5gj7jtttNDXBn4mHaqnAw/DrJzdHT81ec6adfVXUL80pd3Wvqypv5FSGh8RJ0/FNCTqEbFb3lzP+cttdNKWj716f0f9J0RlLrETWSHG4HL0mZIiZRxUV6ys1YcMmNtjU+34mKRs5upwF+mRaYMTOkaF6QiLC4gBPhVT4IhWfCvfwoXDUOux0XevxpZ3PvY/1grKDnaAO1kYt20R56j/bRAaLoK/qOfqCfyxeNZw3c2KhCFxdqzlM0txqvfgG5j3D0</latexit><latexit sha1_base64="6JpLcbwsZUrX8PqDW+OaF+hhNAE=">AAAHcXic3VVLbxMxEHYLbUp4tXBCXNxWkQK00W5UVCQUqRIXJA4UiT6k7BJ5vU5irb27sr1tU2v/IX+Av0GPcGD2oZD0dQMJLEWZnW8+z/ibkR2kgmvjON8WFu/cXVpurNxr3n/w8NHj1bUnhzrJFGUHNBGJOg6IZoLH7MBwI9hxqhiRgWBHQfSuwI9OmNI8iT+bScp8SUYxH3JKDLgGq0PPcBEye5wPrLu1m+Me9nQmBzbqufmXHWy9MocNiYpGirE4B5yPJMkH0RRULMyzcoMonzoDkbH8BLy74B2sbjodp1z4quHWxiaq1/5gbenCCxOaSRYbKojWfddJjW+JMpwKlje9TLOU0IiMWD884amOiWTat2dl9hy3AA/xMFHwiw0uvbMkS6TWExlApCRmrC9jhfM6rJ+Z4Rvf8jjNDItplWiYCWwSXAiMQ64YNWICBqGKQ7mYjoki1EAb5rIUe5skETpvzmWQJGIqSaRvKYkpE/OwNTw6r12FKXigiJpYPSYp0x3JNQV1yvMKZmYaZ8u/9dfOeiBgKwiK2SlNpCRx+NJ6H4YwU0GgRrm1nobKU6P5OWt6RZK+pkSwntNxfdtstVqYhSMGpwkVOe2bMY993O5uOS/w9jZuO7+NbmV0wXgLPAyjCIJMmUMuRK8YlK3is7T8mk+5gj7jtttNDXBn4mHaqnAw/DrJzdHT81ec6adfVXUL80pd3Wvqypv5FSGh8RJ0/FNCTqEbFb3lzP+cttdNKWj716f0f9J0RlLrETWSHG4HL0mZIiZRxUV6ys1YcMmNtjU+34mKRs5upwF+mRaYMTOkaF6QiLC4gBPhVT4IhWfCvfwoXDUOux0XevxpZ3PvY/1grKDnaAO1kYt20R56j/bRAaLoK/qOfqCfyxeNZw3c2KhCFxdqzlM0txqvfgG5j3D0</latexit>

u1 v1

Word2vec Embedding as Neural Network

• Neural network representation of word2vec:

15

… … … …U<latexit sha1_base64="20r+kfvoJZRFZ60S1mDkDrHzzOQ=">AAAHF3ic3VXNbhMxEHYLDSX8tIUjly1VpIDaaBOBQEKVKnFB4kCRSFspu1Re7ySxYq9Xtrdtau17wBFehBviypHn4AWYza5C07S9gQSWVvt5Zj7P+BvLjlLBjfX9HwuL164v1W4s36zfun3n7srq2r09ozLNoMuUUPogogYET6BruRVwkGqgMhKwH41eFv79I9CGq+SdHacQSjpIeJ8zatH03gWTJZyGOO/mh6sbfsufDG8etCuwQaqxe7i29DOIFcskJJYJakyv7ac2dFRbzgTk9SAzkFI2ogPoxUc8NQmVYEJ3Mkmbew30x15fafwS602sZ0mOSmPGMsJISe3QnPcVxot8vcz2n4eOJ2lmIWFlon4mPKu8QgYv5hqYFWMElGmO5XpsSDVlFsWayVKsbZUSJq/PZJB0BFopGTpGEwZi1u0sH51WpgIKHmmqx84MaQqmJblhqM5kvwKsi6keDTRAkrvJb/2pvx4JXAqDEjhmSkqaxI9d8LqPnY8iPcidCwxWnlrDT6EeFEl6hlEB236rHbp6o9HwIB4A7ibW9LhnhzwJvWZn03/kbW15Tf836JSgg+AF8jw8MCjIlNnnQmxHIoPNYjpBYcVnXGOfvWa7k1rknonHE1WGIwirJJdHT/dfcqbTsKzqCuZcXZ0L6srr+ZyQ2HiJOv4pIaeuSxW9Ys//nLYXnVLU9q+f0v9J0zOSuoDqgeR4OwQqBU2t0sVFesztUHDJrXGVf7YTJY2eXE1D/3laZIdgadG8SIm4uICVCEobhuIz0T7/KMyDvU6rjT1++2Rj5031YCyTB+QhaZI2eUZ2yCuyS7qEEU0+kE/kc+1j7Uvta+1bGbq4UHHuk5lR+/4LKtZP9A==</latexit><latexit sha1_base64="20r+kfvoJZRFZ60S1mDkDrHzzOQ=">AAAHF3ic3VXNbhMxEHYLDSX8tIUjly1VpIDaaBOBQEKVKnFB4kCRSFspu1Re7ySxYq9Xtrdtau17wBFehBviypHn4AWYza5C07S9gQSWVvt5Zj7P+BvLjlLBjfX9HwuL164v1W4s36zfun3n7srq2r09ozLNoMuUUPogogYET6BruRVwkGqgMhKwH41eFv79I9CGq+SdHacQSjpIeJ8zatH03gWTJZyGOO/mh6sbfsufDG8etCuwQaqxe7i29DOIFcskJJYJakyv7ac2dFRbzgTk9SAzkFI2ogPoxUc8NQmVYEJ3Mkmbew30x15fafwS602sZ0mOSmPGMsJISe3QnPcVxot8vcz2n4eOJ2lmIWFlon4mPKu8QgYv5hqYFWMElGmO5XpsSDVlFsWayVKsbZUSJq/PZJB0BFopGTpGEwZi1u0sH51WpgIKHmmqx84MaQqmJblhqM5kvwKsi6keDTRAkrvJb/2pvx4JXAqDEjhmSkqaxI9d8LqPnY8iPcidCwxWnlrDT6EeFEl6hlEB236rHbp6o9HwIB4A7ibW9LhnhzwJvWZn03/kbW15Tf836JSgg+AF8jw8MCjIlNnnQmxHIoPNYjpBYcVnXGOfvWa7k1rknonHE1WGIwirJJdHT/dfcqbTsKzqCuZcXZ0L6srr+ZyQ2HiJOv4pIaeuSxW9Ys//nLYXnVLU9q+f0v9J0zOSuoDqgeR4OwQqBU2t0sVFesztUHDJrXGVf7YTJY2eXE1D/3laZIdgadG8SIm4uICVCEobhuIz0T7/KMyDvU6rjT1++2Rj5031YCyTB+QhaZI2eUZ2yCuyS7qEEU0+kE/kc+1j7Uvta+1bGbq4UHHuk5lR+/4LKtZP9A==</latexit><latexit sha1_base64="20r+kfvoJZRFZ60S1mDkDrHzzOQ=">AAAHF3ic3VXNbhMxEHYLDSX8tIUjly1VpIDaaBOBQEKVKnFB4kCRSFspu1Re7ySxYq9Xtrdtau17wBFehBviypHn4AWYza5C07S9gQSWVvt5Zj7P+BvLjlLBjfX9HwuL164v1W4s36zfun3n7srq2r09ozLNoMuUUPogogYET6BruRVwkGqgMhKwH41eFv79I9CGq+SdHacQSjpIeJ8zatH03gWTJZyGOO/mh6sbfsufDG8etCuwQaqxe7i29DOIFcskJJYJakyv7ac2dFRbzgTk9SAzkFI2ogPoxUc8NQmVYEJ3Mkmbew30x15fafwS602sZ0mOSmPGMsJISe3QnPcVxot8vcz2n4eOJ2lmIWFlon4mPKu8QgYv5hqYFWMElGmO5XpsSDVlFsWayVKsbZUSJq/PZJB0BFopGTpGEwZi1u0sH51WpgIKHmmqx84MaQqmJblhqM5kvwKsi6keDTRAkrvJb/2pvx4JXAqDEjhmSkqaxI9d8LqPnY8iPcidCwxWnlrDT6EeFEl6hlEB236rHbp6o9HwIB4A7ibW9LhnhzwJvWZn03/kbW15Tf836JSgg+AF8jw8MCjIlNnnQmxHIoPNYjpBYcVnXGOfvWa7k1rknonHE1WGIwirJJdHT/dfcqbTsKzqCuZcXZ0L6srr+ZyQ2HiJOv4pIaeuSxW9Ys//nLYXnVLU9q+f0v9J0zOSuoDqgeR4OwQqBU2t0sVFesztUHDJrXGVf7YTJY2eXE1D/3laZIdgadG8SIm4uICVCEobhuIz0T7/KMyDvU6rjT1++2Rj5031YCyTB+QhaZI2eUZ2yCuyS7qEEU0+kE/kc+1j7Uvta+1bGbq4UHHuk5lR+/4LKtZP9A==</latexit><latexit sha1_base64="20r+kfvoJZRFZ60S1mDkDrHzzOQ=">AAAHF3ic3VXNbhMxEHYLDSX8tIUjly1VpIDaaBOBQEKVKnFB4kCRSFspu1Re7ySxYq9Xtrdtau17wBFehBviypHn4AWYza5C07S9gQSWVvt5Zj7P+BvLjlLBjfX9HwuL164v1W4s36zfun3n7srq2r09ozLNoMuUUPogogYET6BruRVwkGqgMhKwH41eFv79I9CGq+SdHacQSjpIeJ8zatH03gWTJZyGOO/mh6sbfsufDG8etCuwQaqxe7i29DOIFcskJJYJakyv7ac2dFRbzgTk9SAzkFI2ogPoxUc8NQmVYEJ3Mkmbew30x15fafwS602sZ0mOSmPGMsJISe3QnPcVxot8vcz2n4eOJ2lmIWFlon4mPKu8QgYv5hqYFWMElGmO5XpsSDVlFsWayVKsbZUSJq/PZJB0BFopGTpGEwZi1u0sH51WpgIKHmmqx84MaQqmJblhqM5kvwKsi6keDTRAkrvJb/2pvx4JXAqDEjhmSkqaxI9d8LqPnY8iPcidCwxWnlrDT6EeFEl6hlEB236rHbp6o9HwIB4A7ibW9LhnhzwJvWZn03/kbW15Tf836JSgg+AF8jw8MCjIlNnnQmxHIoPNYjpBYcVnXGOfvWa7k1rknonHE1WGIwirJJdHT/dfcqbTsKzqCuZcXZ0L6srr+ZyQ2HiJOv4pIaeuSxW9Ys//nLYXnVLU9q+f0v9J0zOSuoDqgeR4OwQqBU2t0sVFesztUHDJrXGVf7YTJY2eXE1D/3laZIdgadG8SIm4uICVCEobhuIz0T7/KMyDvU6rjT1++2Rj5031YCyTB+QhaZI2eUZ2yCuyS7qEEU0+kE/kc+1j7Uvta+1bGbq4UHHuk5lR+/4LKtZP9A==</latexit>

V<latexit sha1_base64="RjX2VIY0zv0WBBOsgMdNfNkNeHM=">AAAHG3ic3VXNbhMxEHYLDSX8pXDk4lJFCqiNdiMQSKhSJS5IHCgS/ZGyq8rrnSRW7PXK9qZNrX0TOMKLcENcOfAcvADeH4WmaXsDCSyt9vPMfJ7xN5YdpZxp43k/lpavXV9p3Fi92bx1+87de621+/taZorCHpVcqsOIaOAsgT3DDIfDVAEREYeDaPyq8B9MQGkmk/dmmkIoyDBhA0aJcaajVssG5SI24hnkeD8/am14Xa8ceBH4NdhA9dg9Wlv5GcSSZgISQznRuu97qQktUYZRDnkzyDSkhI7JEPrxhKU6IQJ0aE/KxDluO3+MB1K5LzG4tJ4lWSK0norIRQpiRvq8rzBe5OtnZvAitCxJMwMJrRINMo6NxIUUOGYKqOFTBwhVzJWL6YgoQo0TbC5LsbaRkuu8OZdBkDEoKUVoKUko8Hm3NWx8WpsKyFmkiJpaPSIp6K5gmjp1yv1yMDYmajxUAEluy9/6M2894m4pF5TAMZVCkCR+YoM3A9f9KFLD3NpAu8pTo9kpNIMiSV9TwmHb6/qhbbbbbQzxENxuYkWO+2bEkhB3epveY7y1hTveb9CrQM+Bl46H3aFxgsyYA8b5dnFGNotpicKaT5lyfcYdv5caxz0TryCuwh0I6ySXR8/2X3Fm07Cq6grmQl29C+rKm/mCkK7xwun4p4ScuS5V9Io9/3PaXnRKnbZ//ZT+T5qekdQGRA0Fc7dDIFNQxEhVXKTHzIw4E8xoW/vnO1HRyMnVNOc/T4vMCAwpmhdJHhcXsORBZXOh7pnwzz8Ki2C/1/Vdj9893dh5Wz8Yq+gheoQ6yEfP0Q56jXbRHqJogj6gT+hz42PjS+Nr41sVurxUcx6gudH4/gtSIlDH</latexit><latexit sha1_base64="RjX2VIY0zv0WBBOsgMdNfNkNeHM=">AAAHG3ic3VXNbhMxEHYLDSX8pXDk4lJFCqiNdiMQSKhSJS5IHCgS/ZGyq8rrnSRW7PXK9qZNrX0TOMKLcENcOfAcvADeH4WmaXsDCSyt9vPMfJ7xN5YdpZxp43k/lpavXV9p3Fi92bx1+87de621+/taZorCHpVcqsOIaOAsgT3DDIfDVAEREYeDaPyq8B9MQGkmk/dmmkIoyDBhA0aJcaajVssG5SI24hnkeD8/am14Xa8ceBH4NdhA9dg9Wlv5GcSSZgISQznRuu97qQktUYZRDnkzyDSkhI7JEPrxhKU6IQJ0aE/KxDluO3+MB1K5LzG4tJ4lWSK0norIRQpiRvq8rzBe5OtnZvAitCxJMwMJrRINMo6NxIUUOGYKqOFTBwhVzJWL6YgoQo0TbC5LsbaRkuu8OZdBkDEoKUVoKUko8Hm3NWx8WpsKyFmkiJpaPSIp6K5gmjp1yv1yMDYmajxUAEluy9/6M2894m4pF5TAMZVCkCR+YoM3A9f9KFLD3NpAu8pTo9kpNIMiSV9TwmHb6/qhbbbbbQzxENxuYkWO+2bEkhB3epveY7y1hTveb9CrQM+Bl46H3aFxgsyYA8b5dnFGNotpicKaT5lyfcYdv5caxz0TryCuwh0I6ySXR8/2X3Fm07Cq6grmQl29C+rKm/mCkK7xwun4p4ScuS5V9Io9/3PaXnRKnbZ//ZT+T5qekdQGRA0Fc7dDIFNQxEhVXKTHzIw4E8xoW/vnO1HRyMnVNOc/T4vMCAwpmhdJHhcXsORBZXOh7pnwzz8Ki2C/1/Vdj9893dh5Wz8Yq+gheoQ6yEfP0Q56jXbRHqJogj6gT+hz42PjS+Nr41sVurxUcx6gudH4/gtSIlDH</latexit><latexit sha1_base64="RjX2VIY0zv0WBBOsgMdNfNkNeHM=">AAAHG3ic3VXNbhMxEHYLDSX8pXDk4lJFCqiNdiMQSKhSJS5IHCgS/ZGyq8rrnSRW7PXK9qZNrX0TOMKLcENcOfAcvADeH4WmaXsDCSyt9vPMfJ7xN5YdpZxp43k/lpavXV9p3Fi92bx1+87de621+/taZorCHpVcqsOIaOAsgT3DDIfDVAEREYeDaPyq8B9MQGkmk/dmmkIoyDBhA0aJcaajVssG5SI24hnkeD8/am14Xa8ceBH4NdhA9dg9Wlv5GcSSZgISQznRuu97qQktUYZRDnkzyDSkhI7JEPrxhKU6IQJ0aE/KxDluO3+MB1K5LzG4tJ4lWSK0norIRQpiRvq8rzBe5OtnZvAitCxJMwMJrRINMo6NxIUUOGYKqOFTBwhVzJWL6YgoQo0TbC5LsbaRkuu8OZdBkDEoKUVoKUko8Hm3NWx8WpsKyFmkiJpaPSIp6K5gmjp1yv1yMDYmajxUAEluy9/6M2894m4pF5TAMZVCkCR+YoM3A9f9KFLD3NpAu8pTo9kpNIMiSV9TwmHb6/qhbbbbbQzxENxuYkWO+2bEkhB3epveY7y1hTveb9CrQM+Bl46H3aFxgsyYA8b5dnFGNotpicKaT5lyfcYdv5caxz0TryCuwh0I6ySXR8/2X3Fm07Cq6grmQl29C+rKm/mCkK7xwun4p4ScuS5V9Io9/3PaXnRKnbZ//ZT+T5qekdQGRA0Fc7dDIFNQxEhVXKTHzIw4E8xoW/vnO1HRyMnVNOc/T4vMCAwpmhdJHhcXsORBZXOh7pnwzz8Ki2C/1/Vdj9893dh5Wz8Yq+gheoQ6yEfP0Q56jXbRHqJogj6gT+hz42PjS+Nr41sVurxUcx6gudH4/gtSIlDH</latexit><latexit sha1_base64="RjX2VIY0zv0WBBOsgMdNfNkNeHM=">AAAHG3ic3VXNbhMxEHYLDSX8pXDk4lJFCqiNdiMQSKhSJS5IHCgS/ZGyq8rrnSRW7PXK9qZNrX0TOMKLcENcOfAcvADeH4WmaXsDCSyt9vPMfJ7xN5YdpZxp43k/lpavXV9p3Fi92bx1+87de621+/taZorCHpVcqsOIaOAsgT3DDIfDVAEREYeDaPyq8B9MQGkmk/dmmkIoyDBhA0aJcaajVssG5SI24hnkeD8/am14Xa8ceBH4NdhA9dg9Wlv5GcSSZgISQznRuu97qQktUYZRDnkzyDSkhI7JEPrxhKU6IQJ0aE/KxDluO3+MB1K5LzG4tJ4lWSK0norIRQpiRvq8rzBe5OtnZvAitCxJMwMJrRINMo6NxIUUOGYKqOFTBwhVzJWL6YgoQo0TbC5LsbaRkuu8OZdBkDEoKUVoKUko8Hm3NWx8WpsKyFmkiJpaPSIp6K5gmjp1yv1yMDYmajxUAEluy9/6M2894m4pF5TAMZVCkCR+YoM3A9f9KFLD3NpAu8pTo9kpNIMiSV9TwmHb6/qhbbbbbQzxENxuYkWO+2bEkhB3epveY7y1hTveb9CrQM+Bl46H3aFxgsyYA8b5dnFGNotpicKaT5lyfcYdv5caxz0TryCuwh0I6ySXR8/2X3Fm07Cq6grmQl29C+rKm/mCkK7xwun4p4ScuS5V9Io9/3PaXnRKnbZ//ZT+T5qekdQGRA0Fc7dDIFNQxEhVXKTHzIw4E8xoW/vnO1HRyMnVNOc/T4vMCAwpmhdJHhcXsORBZXOh7pnwzz8Ki2C/1/Vdj9893dh5Wz8Yq+gheoQ6yEfP0Q56jXbRHqJogj6gT+hz42PjS+Nr41sVurxUcx6gudH4/gtSIlDH</latexit>

One-hot encoding of output word One-hot encoding of input word1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0

Linear activationLinear activationu1 v1