30
1 11 ,.!. . ."11 .. . "1 1 . 1" ." .1 . 'J#G0G=S:>P4)N+AGY? 2US:; &LB%5 AI#K 7(@V

[DL Hacks 実装]Attention is All You Need

Embed Size (px)

Citation preview

1 11,. . . . 11 . . . 1 1 . 1 . . 1 .

�������������������������� ���� ����������������������������������������

J G G S P N GY USLAI K

V

:

• 22 : b 2 N• .2 n A

– / . /: 2– n gfd gfd t Nm ci k

– A N o ab Y– ln gfd

rN N s G u N– . .2 ci k m p 2 b N e h

a w A

A 4B3A23:7 B 4

- 45 3 A53AB 51 5 6 AA5 A

& 2 7.5 B A

5 4 SR M M A5 6 I

DIG IA D

• g vsg M7 gnRe e Nsq Qr N V Q ( – T o s M g tv V Q d

• 7 TQ M o lM D T o d q R VPo• q ld o M g l e N w q o• SiM. 29 ) g1892q Q M g - 0 ) . q Q UUo

– q Q g kVe Q• 6 A 7 GG DI G C IL G O g Qo• DK C IA D C 8 D I 8 D 1 GDAD DK8 8 O vsc t

v 8 9

• M II DIA D a q Q M Tn tv g8 9 q

• II DIA D q Qo dcMu g e q c o– ). c8 9 g

B E

• s z Sf pn i e ba x dN Ug– ) D . EB 2– ID . D–– P ..a kro b P g kro w i f

T M

• DD D d t u Sf w i f– x dN L ml hg Mf

• ml– C BD D B BI DG B C B B - BI i BI

DG B f • ml

– ( A C DD D - B . DEB E B– A B B B CDB D CE B D

• c b– DBE DEB DD D D )

– yv C DD D i ml

2D A ,F H HIF

• rgl w[ n D F D F dbe R– D F mz s x h p D F y1 EIH >4D H D A . > hP– 6 > u i ]. D Fcfa N3– 6 2IAH ,HH H D 5 I A

D H D A F DF A M H D DF F 5 I AD H D A F DF A M H D

- D F EIH– 6 6 Rto

( - D Fcfa N3– 6 2IAH ,HH H D 5 I A D H D A F

DF A M H D 2IAH ,HH H D EIH 6 5 I AD H D A F DF A M H D DF F 5 I AD H D A F DF A M H D

) D=H

1

2

3

4

5

6

( 3 1 3 3 . 1

• ( 3 ] [D– 1 go N A ) 13

dt – - 3A1– 1 1 1

• 3 ] [D– ( 3 R aM E( 3 e e

RN A 13 fE A ) 133 R sk

– A ) 13 3 Em cS3 3 uy S L i nl

1 N rh H1

2

3

4

5

6

.

1

2

3

4

5

6

.

1

2

3

4

5

6

��

) 2

• ko 2= 2 l S a ! ) 2 q DA 2l 2 t S gP, 21 = 1 ) 2– 2= ko !D 2 ko ! n D 2 ko v!" s p

• 11Pa 2 x a Du cdem n• !" f i D#$% f P D 3 ko

f P D Pa yr a ]D !" s p

– ko (ko i D 2 3 ) 2 a– ko (ko i D [ -ko (2 ko (12. i D

&'()*+,(./0

12) ko go 2

��

* ) ,HH H

) 1 H / ,HH H– ru = ,HH H _ehT Mtih - H 3 H ,HH H _fN

dad cT mN N Q PV( #0# lo p

.K H= E # H # H H H=) n D H SMqs q H

.K H= H H= D H # A # K ) # K 2# E# H

* - H 3 H ,HH H

��

-- - . - -- -

��

2 - 2– d e ������������������� ����

– S AM c fEM

– A M A g A Mha 3 .EM M• D 3 . 3 2 i• - 3 −∞ EM

��

2 .32

. .32 3 2 .32 .2 3 3– 2 3 3 2 .32 3 M e

M .2-E �����– 2 3 2 3 2 .32 3 3 2 .32 .2-

2 .32 E D 2 .32 ka A dcf gM i

��

-- - . - -- -

��

, - 3

• ) . ( . NP 1 1 -3D

– k N( 3 D EE P. F– EED. E iP d C Pa e E cf C

��

44 4 D

• 3 4 I > 44 E 3 4 I > 44 E 3 4 I> 44 E U OM LS ..

– WT - > 44 > 4• 3 4 . 3 4 I > 44 E 3 4 I > 44 L

��<->����, ��<->�� �������voc �����

���<->��������������

��

5 5 ( 25

• ))[ ))R ERA ce m nsiE– n S 5 5 ( 25 b– v A P 3 5 g [ P– 5 5 3 3225 . AE ]aA a S v– R dl o S tC siE N

��

-

• RNN.CNN, '(<��-�)�0(!/1. 7169%*-�����2. ���+���3. 534892,"$/����-�&:�&;

• ���+�*'(<������# !

��

=R O OI

=R O OI 2 T O 1 T OI– 7= )AGO GB. ) N MM PO SGOTGO G RS (, TPLGOS– 7= )AGO HRB. ( 7 SGOTGO GS O S M T TPLGOS OTP (– rzbtcy mad k ilh[e u VP cSPUR G T RIGTva4 R W RG O G UMG– - 8 5250 : 3: S– 1 SG NP GM. )SG TR O OI STG 1 I NP GM. SG TR O OI STG– 1 SG NP GM. STG S PURS 1 I NP GM. ( STG S ( S

( T N GR– 0 N !" = 0.9, !( = 0.98 ϵ = 10,-

) GIUM R T PO– 2RP PUT. RGS U M bw nf]bopa RP PUT R TG/ g s– 0TTGOT PO RP PUT

– 6 DGM NPPT OI. ./0 = 0.1

��

/6

- 48 6 1=

– 6 6 0.1 E TRU 2 OM A 2 B– 6 = UTRUL A 2 B– O S

��

G

– , B: D NMO D = = BH D = BD B SR5 BB 0 GD = 4D B +D : = .G : = 4D B 6

• + & ! = 0.6– ,DB BG– 1G G = G = '

��

2

– !" 6 6– . 2– D

��

2

– !" 6 6– . 2– D

��

. 3– %���� transfomer������ ��������������$# "�" �&�!��– RecurrentNeuralNetworkGrammar�������

��

– Selfattention������� ���� encoder-encoderattention�������

& on NDN . I

• on l e I B Xh A. . IicE

– af on–– . m– P–

��

- 03/17 i s s I D E BE i ha c

•– e N gf E : B Md d g h cN p m

P P v cN II DIAEDgfh b r v P P dS

– i d N i o h r gf d N iet326 dgXN 26i gfn egr

• .– v r hN cv re upg idN_i i d

p h i ilv rw– i E E hT cN: v r– . E i g gid– P yP i e i)ai

• II D I AD B DAD (& & & : I B

��

) 0D e e I

• A– 3X f S C I yNd A a tow m

– xsvxi f I C f d– ( nEl S fdc f IG C u ruv Pe Id C % S

��

-- D bIl l > Ab

• . Ab c n eT– b c x P b E m– b b f e b N m b

• c b n l bNi N N l Nb N b l T

• . Ab c 1 TeT– 0 Nc e– 0 X 1 n go ld swp tv urv

N c

��