Lecture 6 Value Func.on Approxima.on

Tabular presentation, so far

s0 10

1 12

c0 8

1 7

r0 14

1 10

<latexit sha1_base64="OIVYOcZ7MwPQ/7LBO2ZNkmrR/KI=">AAAB+3icbVBNS8NAEJ34WetX1aOXxSJ4KokoehGKXjy2YD+gDWWz3bRLdzdhdyOUkL/gVe/exKs/xqu/xE2bg219MPB4b4aZeUHMmTau++2srW9sbm2Xdsq7e/sHh5Wj47aOEkVoi0Q8Ut0Aa8qZpC3DDKfdWFEsAk47weQh9zvPVGkWySczjakv8EiykBFscqmJ7tCgUnVr7gxolXgFqUKBxqDy0x9GJBFUGsKx1j3PjY2fYmUY4TQr9xNNY0wmeER7lkosqPbT2a0ZOrfKEIWRsiUNmql/J1IstJ6KwHYKbMZ62cvF/7xeYsJbP2UyTgyVZL4oTDgyEcofR0OmKDF8agkmitlbERljhYmx8SxsCURWtqF4yxGskvZlzbuuuc2rav2+iKcEp3AGF+DBDdThERrQAgJjeIFXeHMy5935cD7nrWtOMXMCC3C+fgEN/5Qa</latexit>

Q =

We need to store |S||A| number of cells

What if there are too many states ?note: cells are independent and isolatedcannot generalize to similar states

What if there are infinite number of states (continuous)?What if the action is continuous?

Feature vectors

sunny

rainy

cloudy

0.2/2

0.1/-1

0.4/-1

0.3/1

0.3/2

0.7/1

0.2/2

0.5/1

0.3/-1

MDP with state ID State feature vector

[temperature, lightness,humidity, rainfall]

feature vectors also relates the statese.g. sunny is close to cloudy than rainy

allows generalization over states

Value function approximation

function approximation

function f

as a supervised learning problem

data approximation h

Value function approximation

value function approximation

Q value

V value st

at

st

h(s)

h(s,a)

V(s)

Q(s, a)

commonly, we parameterize the approximation function with parameter

<latexit sha1_base64="fT8DpBI47XxMrJmuWA5G114UFPM=">AAAB/XicbVA9SwNBEN3zM8avqKXNYhCswp0oWgZtLCOYD0iOsLeZS9bs3h27c0I4gn/BVns7sfW32PpL3CRXmMQHA4/3ZpiZFyRSGHTdb2dldW19Y7OwVdze2d3bLx0cNkycag51HstYtwJmQIoI6ihQQivRwFQgoRkMbyd+8wm0EXH0gKMEfMX6kQgFZ2ilRgcHgKxbKrsVdwq6TLyclEmOWrf00+nFPFUQIZfMmLbnJuhnTKPgEsbFTmogYXzI+tC2NGIKjJ9Nrx3TU6v0aBhrWxHSqfp3ImPKmJEKbKdiODCL3kT8z2unGF77mYiSFCHis0VhKinGdPI67QkNHOXIEsa1sLdSPmCacbQBzW0J1LhoQ/EWI1gmjfOKd1lx7y/K1Zs8ngI5JifkjHjkilTJHamROuHkkbyQV/LmPDvvzofzOWtdcfKZIzIH5+sX84WV0g==</latexit>

✓<latexit sha1_base64="aCJFijjWq3uInfVhByp5cu1kNAs=">AAACG3icbVC7SgNBFJ2Nrxhfq5Y2Q4IQIYRdUbQM2lgmYB6QLMvsZDYZMvtg5q4QlvT+hL9gq72d2FrY+iVOki3y8MCFM+fcy517vFhwBZb1Y+Q2Nre2d/K7hb39g8Mj8/ikpaJEUtakkYhkxyOKCR6yJnAQrBNLRgJPsLY3up/67ScmFY/CRxjHzAnIIOQ+pwS05JrFltuDIQOCy8qFiwpuLL4rRGuuWbKq1gx4ndgZKaEMddf87fUjmgQsBCqIUl3bisFJiQROBZsUeoliMaEjMmBdTUMSMOWks1sm+FwrfexHUlcIeKYuTqQkUGoceLozIDBUq95U/M/rJuDfOikP4wRYSOeL/ERgiPA0GNznklEQY00IlVz/FdMhkYSCjm9pixdMCjoUezWCddK6rNrXVatxVardZfHk0RkqojKy0Q2qoQdUR01E0TN6RW/o3XgxPoxP42vemjOymVO0BOP7D1S3oAw=</latexit>

V✓(st), Q✓(st, at)

What are the approximation models?

exactly the supervised learning models

– linear models– linear models with kernels– nearest neighbors– decision trees– neural networks–…

RL usually requires more complex models than SL

Approximation objective

learning a model that approximate the true value func.

<latexit sha1_base64="pUGBVfdZ9Ab/s6NQn+EnMVPg5lk=">AAACQ3icbVDLSgMxFM34tr6qLt0Eq+CqzAiiy6IbERcKVoW2lkx6pw3mMSR3lFL6T/6EvyC4Uty6E7eCmbELXxcCJ+fck3tz4lQKh2H4GIyNT0xOTc/MlubmFxaXyssr585klkOdG2nsZcwcSKGhjgIlXKYWmIolXMTXB7l+cQPWCaPPsJ9CS7GuFongDD3VLh8dA9KNpsqumqnYoB3QBoFiD6jDooXZPu34PayIs/xOTVJI4GhipDS3Qnf9A97cLlfCalgU/QuiEaiQUZ20yy/NjuGZAo1cMucaUZhia8AsCi5hWGpmDlLGr1kXGh5qpsC1BsWfh3TTMx2/gvVHIy3Y744BU871Vew7FcOe+63l5H9aI8NkrzUQOs0QNP8alGSSoqF5gD4MCxxlngrjVvhdKe8xyzj6mH9MidWw5EOJfkfwF5xvV6Odani6Xantj+KZIWtknWyRiOySGjkkJ6ROOLkjD+SJPAf3wWvwFrx/tY4FI88q+VHBxydbSbF6</latexit>

Let µ⇡ denote the stationary distribution of states following ⇡

why mean square? V and Q are expectations, mean square leads to unbiased approximation

<latexit sha1_base64="6/xHh+TqtNIxvVCsboCUpCylFT8=">AAACs3icfVFdaxNBFJ1dv2r8SvXRl4tBSUDDbrHYB4WqIOKLCZq0kEmWu7OTdOjM7DIzawlr/pj/xFd/iTNpCjYtXhg499x7z72cySsprEuS31F84+at23d27rbu3X/w8FF79/HYlrVhfMRKWZrjHC2XQvORE07y48pwVLnkR/npx1A/+sGNFaX+7pYVnypcaDEXDJ2nsvavF/Cle9aDd0CFdtk3oKqe0Up0bQ/oB7HojmeNT1chfwXj7Aw8CoXebA+o70Z3YhQUYIHS1v/EAvMeQoY/L8SHF+IvMcgP1/IeX7cAYWtf1u4k/WQdcBWkG9Ahmxhk7T+0KFmtuHZMorWTNKnctEHjBJN81aK15RWyU1zwiYcaFbfTZu3xCp57poB5afzTDtbsvxMNKmuXKved4Ua7XQvkdbVJ7eYH00boqnZcs/NF81qCKyF8GBTCcObk0gNkRvhbgZ2gQeb8t17akqtVy5uSbltwFYz3+ul+Pxm+7hy+3dizQ56SZ6RLUvKGHJLPZEBGhEUQfYq+RoN4P57EeVyct8bRZuYJuRSx+gtm8Mql</latexit>

J(w) =

Z

Sµ⇡(s)

⇣V ⇡(s)� Vw(s)

⌘2ds

J(w) =

Z

Sµ⇡(s)

Z

A⇡(a|s)

⇣Q⇡(s, a)�Qw(s, a)

⌘2da ds

<latexit sha1_base64="HKYlrW2R4vMpWEGPNbQQtFGEfbg=">AAACb3icbZFdS8MwFIbT+j2/pl54IcjBoWygoxVFbwSZCOLVBm4K6yxplm3BpC1JqoyyHyp45T/wH5huFZzzQODNe87DCW+CmDOlHefdsufmFxaXllcKq2vrG5vFre2WihJJaJNEPJJPAVaUs5A2NdOcPsWSYhFw+hi83GT9x1cqFYvCBz2MaUfgfsh6jGBtLL/4egT35bcKXMGtnypPMeHFbARejfXLrec0u5RVBU6g5b+BUVmj8nwKnleYIo/xNNv4YY9xRjfGtNE57xdLTtUZF8wKNxcllFfdL3563YgkgoaacKxU23Vi3Umx1IxwOip4iaIxJi+4T9tGhlhQ1UnH+Yzg0Dhd6EXSnFDD2P1NpFgoNRSBmRRYD9TfXmb+12snunfZSVkYJ5qGZLKol3DQEWRhQ5dJSjQfGoGJZOatQAZYYqLNl0xtCcSoYEJx/0YwK1qnVfe86jTOSte1PJ5ltIcOUBm56AJdoztUR01E0IdlW6vWmvVl79r7NkxGbStndtBU2ZVvFE+08Q==</latexit>

J(w) = Es⇠⇡

⇣V ⇡(s)� Vw(s)

⌘2

J(w) = Es,a⇠⇡

⇣Q⇡(s, a)�Qw(s, a)

⌘2

Solve the parameters

online environment: stochastic gradient on single sample

<latexit sha1_base64="c+75lnLlCk7WJeAtR9TGqXpBhgU=">AAACTHicbVBNSwMxFMzWr1q/qh69BIvQipbdoiiCUBRBPLVotdBtl2ya1mCSXZKspSz9Yf4J754ET3r3JoLZ2kNtHQjMm3mP9zJ+yKjStv1ipWZm5+YX0ouZpeWV1bXs+satCiKJSQ0HLJB1HynCqCA1TTUj9VASxH1G7vyH88S/eyRS0UDc6H5Imhx1Be1QjLSRvOx1r7ULT6GLZNflVMCrfK8wXsMLL1Z7yFWUuyEdQPeMdvPVVpwUeWMU4D6sej045IlZaJW8bM4u2kPAaeKMSA6MUPGyb247wBEnQmOGlGo4dqibMZKaYkYGGTdSJET4AXVJw1CBOFHNePj5AdwxSht2Amme0HCojk/EiCvV577p5Ejfq0kvEf/zGpHuHDdjKsJIE4F/F3UiBnUAkyRhm0qCNesbgrCk5laI75FEWJu8/2zx+SBjQnEmI5gmt6Wic1i0qwe58skonjTYAtsgDxxwBMrgElRADWDwBF7BO/iwnq1P68v6/m1NWaOZTfAHqfkfquuv6A==</latexit>

w⇤ = argmin J(w) = argminEs,a⇠⇡

⇣Q⇡(s, a)�Qw(s, a)

⌘2

for one state-action data sample

<latexit sha1_base64="Aw1B8sTznC6TpOtA5Ic9giBexpk=">AAACLnicbZDLSgMxFIYz9VbrrerSTbAIrWiZKRZFEIpuXLZgL9BpSyZN29AkMyQZSxn6JL6Er+BW94ILEVz5GKaXhW09EPj4/3M4J78XMKq0bX9YsZXVtfWN+GZia3tndy+5f1BRfigxKWOf+bLmIUUYFaSsqWakFkiCuMdI1evfjf3qI5GK+uJBDwPS4KgraIdipI3USuYHzVN4A10kuy6nAkLo3tJuutSM3ICO0uoMZeA5LLUGcMJjM9PMtZIpO2tPCi6DM4MUmFWxlfx22z4OOREaM6RU3bED3YiQ1BQzMkq4oSIBwn3UJXWDAnGiGtHkeyN4YpQ27PjSPKHhRP07ESGu1JB7ppMj3VOL3lj8z6uHunPViKgIQk0Eni7qhAxqH46zgm0qCdZsaABhSc2tEPeQRFibROe2eHyUMKE4ixEsQyWXdfJZu3SRKlzP4omDI3AM0sABl6AA7kERlAEGT+AFvII369l6tz6tr2lrzJrNHIK5sn5+AU0GpWA=</latexit>

w⇤ = argmin⇣Q⇡(s, a)�Qw(s, a)

⌘2

how to solve ? assume differentiable<latexit sha1_base64="6+83Y4EkqO0Rt4r+NUrGV+plbVQ=">AAACOHicbVDLSgMxFM3UV62vqks3wSK0oGWm+NqIBTfiqgVbhU4d7qRpDWYyQ5KxlKGf40/4C2514U7ciFu/wHRawdeBwOGcc7k3x484U9q2n63M1PTM7Fx2PrewuLS8kl9da6owloQ2SMhDeemDopwJ2tBMc3oZSQqBz+mFf3My8i9uqVQsFOd6ENF2AD3BuoyANpKXP3YjkJoB9/r4rNgv4SOMd3AFF3H9yo1YUW1Daafu9VNSwq4An4PJfklevmCX7RT4L3EmpIAmqHn5V7cTkjigQhMOSrUcO9LtZHQE4XSYc2NFIyA30KMtQwUEVLWT9KNDvGWUDu6G0jyhcap+n0ggUGoQ+CYZgL5Wv72R+J/XinX3sJ0wEcWaCjJe1I051iEetYY7TFKi+cAQIJKZWzG5BglEm25/bPGDYc6U4vyu4C9pVsrOXtmu7xaq+5N6smgDbaIictABqqJTVEMNRNAdekCP6Mm6t16sN+t9HM1Yk5l19APWxye2HKh/</latexit>

@wJ(w) = �2(Q⇡(s, a)�Qw(s, a))rwQw(s, a)

update w towards negative derivative<latexit sha1_base64="GGjcqyV/58qGSoVsTWfpnArJdWo=">AAACM3icbZDLSgMxFIYzXmu9jbp0EyxCBS0zUlQKQkEXLluwF+jU4Uya2tBMZkgyllL6Lr6Er+BWt+KuuPUdTC+CVn8IfPznHM7JH8ScKe04b9bC4tLyympqLb2+sbm1be/sVlWUSEIrJOKRrAegKGeCVjTTnNZjSSEMOK0F3atxvfZApWKRuNX9mDZDuBeszQhoY/l2wbumXAPu4UvsAY87gLO4fOfFLKuO4eik7PcmcIQ9AQEHv4e/Ld/OODlnIvwX3Blk0Ewl3x55rYgkIRWacFCq4Tqxbg5AakY4Haa9RNEYSBfuacOggJCq5mDyxyE+NE4LtyNpntB44v6cGECoVD8MTGcIuqPma2Pzv1oj0e2L5oCJONFUkOmidsKxjvA4MNxikhLN+waASGZuxaQDEog2sf7aEoTDtAnFnY/gL1RPc+5ZLl/OZ4qFWTwptI8OUBa56BwV0Q0qoQoi6BE9oxf0aj1Z79bI+pi2LlizmT30S9bnF6Twp6U=</latexit>

�w = ↵(Q⇡(s, a)�Qw(s, a))rwQw(s, a)

Recall the Q update rules

Recall the errors:MC update:TD update:

target

Q(st, at)+ = ↵(R�Q(st, at))

Q(st, at)+ = ↵(rt+1 + �Q(st+1, at+1)�Q(st, at))

model

replace Q table updates by parameter updates

<latexit sha1_base64="GGjcqyV/58qGSoVsTWfpnArJdWo=">AAACM3icbZDLSgMxFIYzXmu9jbp0EyxCBS0zUlQKQkEXLluwF+jU4Uya2tBMZkgyllL6Lr6Er+BWt+KuuPUdTC+CVn8IfPznHM7JH8ScKe04b9bC4tLyympqLb2+sbm1be/sVlWUSEIrJOKRrAegKGeCVjTTnNZjSSEMOK0F3atxvfZApWKRuNX9mDZDuBeszQhoY/l2wbumXAPu4UvsAY87gLO4fOfFLKuO4eik7PcmcIQ9AQEHv4e/Ld/OODlnIvwX3Blk0Ewl3x55rYgkIRWacFCq4Tqxbg5AakY4Haa9RNEYSBfuacOggJCq5mDyxyE+NE4LtyNpntB44v6cGECoVD8MTGcIuqPma2Pzv1oj0e2L5oCJONFUkOmidsKxjvA4MNxikhLN+waASGZuxaQDEog2sf7aEoTDtAnFnY/gL1RPc+5ZLl/OZ4qFWTwptI8OUBa56BwV0Q0qoQoi6BE9oxf0aj1Z79bI+pi2LlizmT30S9bnF6Twp6U=</latexit>

�w = ↵(Q⇡(s, a)�Qw(s, a))rwQw(s, a)

Update with value function approximation

MC update:

TD update:

<latexit sha1_base64="88AoaJYJLmKz65imDgsrpg67sTU=">AAACNHicbVDLSgNBEJz1bXxFPXoZDEICGnZFVBRB0INHI0YD2bD0TiZmyOzsMtNrCCEf40/4C171KHhSvPoNTh4HoxY0FFXddHeFiRQGXffVmZicmp6ZnZvPLCwuLa9kV9duTJxqxssslrGuhGC4FIqXUaDklURziELJb8PWWd+/vefaiFhdYyfhtQjulGgIBmilIHvsn3OJQNv0hPogkybQPL2iO7QUtPMmwG0IsFCgvoJQQtAek4Nszi26A9C/xBuRHBnhMsi++/WYpRFXyCQYU/XcBGtd0CiY5L2MnxqeAGvBHa9aqiDiptYdPNmjW1ap00asbSmkA/XnRBciYzpRaDsjwKb57fXF/7xqio3DWleoJEWu2HBRI5UUY9pPjNaF5gxlxxJgWthbKWuCBoY217EtYdTL2FC83xH8JTe7RW+/uFfay50ejeKZIxtkk+SJRw7IKbkgl6RMGHkgT+SZvDiPzpvz4XwOWyec0cw6GYPz9Q3id6hY</latexit>

�w = ↵(R�Qw(st, at))rwQw(st, at)

<latexit sha1_base64="KqQ8zE0akUD1+/eu7EUtt0krqDk=">AAACWHicbZBNixNBEIYr48fuxo+N69FLYRASVsOMLCqCsKAHj7tgdhcyYajpdJJmu3uG7hpDGPLz/BF69OpV73aSOZhdC5p+ed4qqnjzUivPcfy9Fd25e+/+3v5B+8HDR48PO0+OLnxROSGHotCFu8rJS62sHLJiLa9KJ8nkWl7m1x/X/uVX6bwq7BdelnJsaGbVVAnigLJOln6SmgkX+AFT0uWcsIcuq/k4WeExpjMyhvA8W/T8Fr6k7d/HVw3mgLjfx9RSrilb7OCs040H8abwtkga0YWmzrLOz3RSiMpIy0KT96MkLnlck2MltFy108rLksQ1zeQoSEtG+nG9CWKFLwKZ4LRw4VnGDf13oibj/dLkodMQz/1Nbw3/540qnr4b18qWFUsrtoumlUYucJ0qTpSTgvUyCBJOhVtRzMmR4JD9zpbcrNohlORmBLfFxetB8mZwcn7SPX3fxLMPz+A59CCBt3AKn+EMhiDgG/yC3/Cn9SOCaC862LZGrWbmKexUdPQXKsKxuw==</latexit>

�w = ↵(rt+1 + �Qw(st+1, at+1)�Qw(st, at))rwQw(st, at)

MC RL with function approximation

w = 0 for i=0, 1, ..., m

generate trajectory <s0, a0, r1, s1, ..., sT> by for t=0, 1, ..., T-1

R = sum of rewards from t to T

end for update policy

end for

⇡✏

<latexit sha1_base64="zxqRHJzRAkVaeM3JonGc1+QnQjM=">AAACJ3icbVDLSsQwFE19O75GXboJDoKiDq0oulFENy4VHBWmY0kzqV5Mk5LcCkPp37jxV9wIKqJL/8TMOAtfBwKHc84l9544k2DR99+9gcGh4ZHRsfHKxOTU9Ex1du7M6txw0eBaanMRMyskKNFAQCkuMiNYGktxHt8cdv3zW2EsaHWKnUy0UnalIAHO0ElRdS9ESIWlYWZ0O1RaQgpoowJ2cTUoL4vT9aCkYWIYL1wGlm0Ea5RFsFIWWQRlVK35db8H+pcEfVIjfRxH1aewrXmeCoVcMmubgZ9hq2AGgUtRVsLciozxG3Ylmo4q5pZrFb07S7rklDZNtHFPIe2p3ycKllrbSWOXTBle299eV/zPa+aY7LQKUFmOQvGvj5JcUtS0WxptgxEcZccRxg24XSm/Zq4UdNVWXAnB75P/krONerBV9082a/sH/TrGyAJZJMskINtknxyRY9IgnNyRB/JMXrx779F79d6+ogNef2ae/ID38QkGLKYA</latexit>

⇥YT�1

i=t+1

⇡(si, ai)

pi

<latexit sha1_base64="hSMy76CPD4DNGstP8xTA3D628JI=">AAACBXicbVDLSgNBEJz1GeNr1aMeBoOQgIRdUfQiBL14TMA8ILssvZPZZMjsg5lZNYRcvPgrXjwo4tV/8ObfOEn2oIkFDUVVN91dfsKZVJb1bSwsLi2vrObW8usbm1vb5s5uQ8apILROYh6Llg+SchbRumKK01YiKIQ+p02/fz32m3dUSBZHt2qQUDeEbsQCRkBpyTMPnIQVZQlfYgdE1wnhwQNc8+6L8hhKnlmwytYEeJ7YGSmgDFXP/HI6MUlDGinCQcq2bSXKHYJQjHA6yjuppAmQPnRpW9MIQird4eSLET7SSgcHsdAVKTxRf08MIZRyEPq6MwTVk7PeWPzPa6cquHCHLEpSRSMyXRSkHKsYjyPBHSYoUXygCRDB9K2Y9EAAUTq4vA7Bnn15njROyvZZ2aqdFipXWRw5tI8OURHZ6BxV0A2qojoi6BE9o1f0ZjwZL8a78TFtXTCymT30B8bnD7qDls0=</latexit>

⇡(s) = argmaxa

Qw(s, a)

<latexit sha1_base64="N7J5BNts9wZSSXkhgeJ0r8qCh3c=">AAACHXicbVDLSgNBEJz1GeNr1aOXwSBE1LArQb0IQS8eo5gYSMLSO5mYIbOzy0yvIYT8iBd/xYsHRTx4Ef/GSczBV0FDUdVNd1eYSGHQ8z6cqemZ2bn5zEJ2cWl5ZdVdW6+aONWMV1gsY10LwXApFK+gQMlrieYQhZJfh92zkX99y7URsbrCfsKbEdwo0RYM0EqBW+zRE9qju7QBMulA/nL/IujlTYB7EODODm0oCCUEPfpdDtycV/DGoH+JPyE5MkE5cN8arZilEVfIJBhT970EmwPQKJjkw2wjNTwB1oUbXrdUQcRNczD+bki3rdKi7VjbUkjH6veJAUTG9KPQdkaAHfPbG4n/efUU28fNgVBJilyxr0XtVFKM6Sgq2hKaM5R9S4BpYW+lrAMaGNpAszYE//fLf0n1oOAfFooXxVzpdBJHhmySLZInPjkiJXJOyqRCGLkjD+SJPDv3zqPz4rx+tU45k5kN8gPO+yf/Ep9w</latexit>

w = w + ↵(R�Qw(st, at))rwQw(st, at)

Q-learning with function approximation

w = 0, initial state for i=0, 1, ...

s’, r = do action from policy a’ =

s = s’, a = a’ end for

⇡✏

⇡(s0)

<latexit sha1_base64="hSMy76CPD4DNGstP8xTA3D628JI=">AAACBXicbVDLSgNBEJz1GeNr1aMeBoOQgIRdUfQiBL14TMA8ILssvZPZZMjsg5lZNYRcvPgrXjwo4tV/8ObfOEn2oIkFDUVVN91dfsKZVJb1bSwsLi2vrObW8usbm1vb5s5uQ8apILROYh6Llg+SchbRumKK01YiKIQ+p02/fz32m3dUSBZHt2qQUDeEbsQCRkBpyTMPnIQVZQlfYgdE1wnhwQNc8+6L8hhKnlmwytYEeJ7YGSmgDFXP/HI6MUlDGinCQcq2bSXKHYJQjHA6yjuppAmQPnRpW9MIQird4eSLET7SSgcHsdAVKTxRf08MIZRyEPq6MwTVk7PeWPzPa6cquHCHLEpSRSMyXRSkHKsYjyPBHSYoUXygCRDB9K2Y9EAAUTq4vA7Bnn15njROyvZZ2aqdFipXWRw5tI8OURHZ6BxV0A2qojoi6BE9o1f0ZjwZL8a78TFtXTCymT30B8bnD7qDls0=</latexit>

⇡(s) = argmaxa

Qw(s, a)

<latexit sha1_base64="1znwYJ/HCCxsdvtiXCADtZxvk0o=">AAACK3icbZDLSgMxFIYzXmu9VV26CRaxxVpmRLQboejGZQv2Am0ZzqRpG0wyQ5KxlNL3ceOruNCFF9z6HqYXxNsPgS//OYfk/EHEmTau++rMzS8sLi0nVpKra+sbm6mt7aoOY0VohYQ8VPUANOVM0ophhtN6pCiIgNNacHM5rtduqdIslNdmENGWgK5kHUbAWMtPXfTxOe7jQ9wEHvUgo8bYBSEAl/1+Rh/k4CCLj6aXHGSzuCkh4OD3vyw/lXbz7kT4L3gzSKOZSn7qsdkOSSyoNISD1g3PjUxrCMowwuko2Yw1jYDcQJc2LEoQVLeGk11HeN86bdwJlT3S4In7fWIIQuuBCGynANPTv2tj879aIzadQmvIZBQbKsn0oU7MsQnxODjcZooSwwcWgChm/4pJDxQQY+NN2hC83yv/hepx3jvNn5RP0sXCLI4E2kV7KIM8dIaK6AqVUAURdIce0DN6ce6dJ+fNeZ+2zjmzmR30Q87HJw+SorY=</latexit>

w = w + ↵(r + �Qw(s0, a0)�Qw(s, a))rwQw(s, a)

Approximation model

<latexit sha1_base64="DA2MOBe5wGOecAM/dhBmlDbmbz4=">AAACBHicbVDLSsNAFJ34rPUVddnNYBEqSElE0Y1QdOOyBfuAJobJdNIOnUzCzMRSQhdu/BU3LhRx60e482+cpllo64ELh3Pu5d57/JhRqSzr21haXlldWy9sFDe3tnd2zb39lowSgUkTRywSHR9JwignTUUVI51YEBT6jLT94c3Ubz8QIWnE79Q4Jm6I+pwGFCOlJc8sNbxRRZ6gY3gFR/eOimLoxAOaSZ5ZtqpWBrhI7JyUQY66Z345vQgnIeEKMyRl17Zi5aZIKIoZmRSdRJIY4SHqk66mHIVEumn2xAQeaaUHg0jo4gpm6u+JFIVSjkNfd4ZIDeS8NxX/87qJCi7dlPI4UYTj2aIgYVBFcJoI7FFBsGJjTRAWVN8K8QAJhJXOrahDsOdfXiSt06p9XrUaZ+XadR5HAZTAIagAG1yAGrgFddAEGDyCZ/AK3own48V4Nz5mrUtGPnMA/sD4/AHIaZZE</latexit>

Qw(s, a) = w>�(s, a)

Linear model

encode state-actions into one vector<latexit sha1_base64="GMY4QeEZ2oTxhuWLgRmB9s41oq8=">AAAB8HicbVBNSwMxEJ2tX7V+rXr0EixCBSm7ouix6MVjBfsh7VKyabYNTbJLkhXK0l/hxYMiXv053vw3pu0etPXBwOO9GWbmhQln2njet1NYWV1b3yhulra2d3b33P2Dpo5TRWiDxDxW7RBrypmkDcMMp+1EUSxCTlvh6Hbqt56o0iyWD2ac0EDggWQRI9hY6bGbDFlFn+HTnlv2qt4MaJn4OSlDjnrP/er2Y5IKKg3hWOuO7yUmyLAyjHA6KXVTTRNMRnhAO5ZKLKgOstnBE3RilT6KYmVLGjRTf09kWGg9FqHtFNgM9aI3Ff/zOqmJroOMySQ1VJL5oijlyMRo+j3qM0WJ4WNLMFHM3orIECtMjM2oZEPwF19eJs3zqn9Z9e4vyrWbPI4iHMExVMCHK6jBHdShAQQEPMMrvDnKeXHenY95a8HJZw7hD5zPH9clj8Y=</latexit>

�(s, a)

<latexit sha1_base64="PN5W+Bg68kVWfFImNan5PAbxQeU=">AAACBHicbZDLSgMxFIYz9VbrbdRlN8EiVJAyI4puhKIbly3YC7RlOJOmbWgmMyQZSxm6cOOruHGhiFsfwp1vYzrtQlt/CHz85xxOzu9HnCntON9WZmV1bX0ju5nb2t7Z3bP3D+oqjCWhNRLyUDZ9UJQzQWuaaU6bkaQQ+Jw2/OHttN54oFKxUNzrcUQ7AfQF6zEC2lienW8L8Dl4I1z1RkV1CifX7WjAUvLsglNyUuFlcOdQQHNVPPur3Q1JHFChCQelWq4T6U4CUjPC6STXjhWNgAyhT1sGBQRUdZL0iAk+Nk4X90JpntA4dX9PJBAoNQ580xmAHqjF2tT8r9aKde+qkzARxZoKMlvUiznWIZ4mgrtMUqL52AAQycxfMRmABKJNbjkTgrt48jLUz0ruRcmpnhfKN/M4siiPjlARuegSldEdqqAaIugRPaNX9GY9WS/Wu/Uxa81Y85lD9EfW5w95BJaw</latexit>

rwQw(s, a) = �(s, a)

the encoding is crucial

<latexit sha1_base64="FBHqC0KJIvsq5F7BVxZyCQb5sx0=">AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PAi8cEzAOSJcxOepMxs7PLzKwQlnyBFw+KePWTvPk3TpI9aGJBQ1HVTXdXkAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4bua3n1BpHssHM0nQj+hQ8pAzaqzU0P1yxa26c5BV4uWkAjnq/fJXbxCzNEJpmKBadz03MX5GleFM4LTUSzUmlI3pELuWShqh9rP5oVNyZpUBCWNlSxoyV39PZDTSehIFtjOiZqSXvZn4n9dNTXjrZ1wmqUHJFovCVBATk9nXZMAVMiMmllCmuL2VsBFVlBmbTcmG4C2/vEpaF1Xvquo2Liu16zyOIpzAKZyDBzdQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/gfP4A3FuM7w==</latexit>s

<latexit sha1_base64="KlM+5J+nG5hvTBDiOu2bABLcbII=">AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PAi8cEzAOSJcxOepMxs7PLzKwQlnyBFw+KePWTvPk3TpI9aGJBQ1HVTXdXkAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4bua3n1BpHssHM0nQj+hQ8pAzaqzUoP1yxa26c5BV4uWkAjnq/fJXbxCzNEJpmKBadz03MX5GleFM4LTUSzUmlI3pELuWShqh9rP5oVNyZpUBCWNlSxoyV39PZDTSehIFtjOiZqSXvZn4n9dNTXjrZ1wmqUHJFovCVBATk9nXZMAVMiMmllCmuL2VsBFVlBmbTcmG4C2/vEpaF1Xvquo2Liu16zyOIpzAKZyDBzdQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/gfP4AwROM3Q==</latexit>a

<latexit sha1_base64="FBHqC0KJIvsq5F7BVxZyCQb5sx0=">AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PAi8cEzAOSJcxOepMxs7PLzKwQlnyBFw+KePWTvPk3TpI9aGJBQ1HVTXdXkAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4bua3n1BpHssHM0nQj+hQ8pAzaqzU0P1yxa26c5BV4uWkAjnq/fJXbxCzNEJpmKBadz03MX5GleFM4LTUSzUmlI3pELuWShqh9rP5oVNyZpUBCWNlSxoyV39PZDTSehIFtjOiZqSXvZn4n9dNTXjrZ1wmqUHJFovCVBATk9nXZMAVMiMmllCmuL2VsBFVlBmbTcmG4C2/vEpaF1Xvquo2Liu16zyOIpzAKZyDBzdQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/gfP4A3FuM7w==</latexit>s

<latexit sha1_base64="KlM+5J+nG5hvTBDiOu2bABLcbII=">AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PAi8cEzAOSJcxOepMxs7PLzKwQlnyBFw+KePWTvPk3TpI9aGJBQ1HVTXdXkAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4bua3n1BpHssHM0nQj+hQ8pAzaqzUoP1yxa26c5BV4uWkAjnq/fJXbxCzNEJpmKBadz03MX5GleFM4LTUSzUmlI3pELuWShqh9rP5oVNyZpUBCWNlSxoyV39PZDTSehIFtjOiZqSXvZn4n9dNTXjrZ1wmqUHJFovCVBATk9nXZMAVMiMmllCmuL2VsBFVlBmbTcmG4C2/vEpaF1Xvquo2Liu16zyOIpzAKZyDBzdQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/gfP4AwROM3Q==</latexit>a010

0

0

<latexit sha1_base64="FBHqC0KJIvsq5F7BVxZyCQb5sx0=">AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PAi8cEzAOSJcxOepMxs7PLzKwQlnyBFw+KePWTvPk3TpI9aGJBQ1HVTXdXkAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4bua3n1BpHssHM0nQj+hQ8pAzaqzU0P1yxa26c5BV4uWkAjnq/fJXbxCzNEJpmKBadz03MX5GleFM4LTUSzUmlI3pELuWShqh9rP5oVNyZpUBCWNlSxoyV39PZDTSehIFtjOiZqSXvZn4n9dNTXjrZ1wmqUHJFovCVBATk9nXZMAVMiMmllCmuL2VsBFVlBmbTcmG4C2/vEpaF1Xvquo2Liu16zyOIpzAKZyDBzdQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/gfP4A3FuM7w==</latexit>s

0

1

0

<latexit sha1_base64="KlM+5J+nG5hvTBDiOu2bABLcbII=">AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PAi8cEzAOSJcxOepMxs7PLzKwQlnyBFw+KePWTvPk3TpI9aGJBQ1HVTXdXkAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4bua3n1BpHssHM0nQj+hQ8pAzaqzUoP1yxa26c5BV4uWkAjnq/fJXbxCzNEJpmKBadz03MX5GleFM4LTUSzUmlI3pELuWShqh9rP5oVNyZpUBCWNlSxoyV39PZDTSehIFtjOiZqSXvZn4n9dNTXjrZ1wmqUHJFovCVBATk9nXZMAVMiMmllCmuL2VsBFVlBmbTcmG4C2/vEpaF1Xvquo2Liu16zyOIpzAKZyDBzdQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/gfP4AwROM3Q==</latexit>a<latexit sha1_base64="GMY4QeEZ2oTxhuWLgRmB9s41oq8=">AAAB8HicbVBNSwMxEJ2tX7V+rXr0EixCBSm7ouix6MVjBfsh7VKyabYNTbJLkhXK0l/hxYMiXv053vw3pu0etPXBwOO9GWbmhQln2njet1NYWV1b3yhulra2d3b33P2Dpo5TRWiDxDxW7RBrypmkDcMMp+1EUSxCTlvh6Hbqt56o0iyWD2ac0EDggWQRI9hY6bGbDFlFn+HTnlv2qt4MaJn4OSlDjnrP/er2Y5IKKg3hWOuO7yUmyLAyjHA6KXVTTRNMRnhAO5ZKLKgOstnBE3RilT6KYmVLGjRTf09kWGg9FqHtFNgM9aI3Ff/zOqmJroOMySQ1VJL5oijlyMRo+j3qM0WJ4WNLMFHM3orIECtMjM2oZEPwF19eJs3zqn9Z9e4vyrWbPI4iHMExVMCHK6jBHdShAQQEPMMrvDnKeXHenY95a8HJZw7hD5zPH9clj8Y=</latexit>

�(s, a)

Approximation model

Linear model

each action has a vector<latexit sha1_base64="JXArQ9P9iKysUmfJy0Biy10sh+k=">AAACBnicbZDLSgMxFIYzXmu9jboUIViEuikzouhGKLpx2YK9QGccMmmmDc1kQpJRytCVG1/FjQtF3PoM7nwb03YW2vpD4OM/53By/lAwqrTjfFsLi0vLK6uFteL6xubWtr2z21RJKjFp4IQlsh0iRRjlpKGpZqQtJEFxyEgrHFyP6617IhVN+K0eCuLHqMdpRDHSxgrsg3qQPQRoVFbH8BIauvN0IqAn+tRYgV1yKs5EcB7cHEogVy2wv7xugtOYcI0ZUqrjOkL7GZKaYkZGRS9VRCA8QD3SMchRTJSfTc4YwSPjdGGUSPO4hhP390SGYqWGcWg6Y6T7arY2Nv+rdVIdXfgZ5SLVhOPpoihlUCdwnAnsUkmwZkMDCEtq/gpxH0mEtUmuaEJwZ0+eh+ZJxT2rOPXTUvUqj6MA9sEhKAMXnIMquAE10AAYPIJn8ArerCfrxXq3PqatC1Y+swf+yPr8AURml7Y=</latexit>

Qwa(s) = w>a �(s)

<latexit sha1_base64="rANlHypPWTIJUUVbL+TgIR6XO/M=">AAACCHicbVC7SgNBFJ2Nrxhfq5YWDgYhNmFXFG2EoI1lAuYB2WW5O5kkQ2Znl5lZJSwpbfwVGwtFbP0EO//GyaPQxAPDPZxzL3fuCRPOlHacbyu3tLyyupZfL2xsbm3v2Lt7DRWnktA6iXksWyEoypmgdc00p61EUohCTpvh4GbsN++pVCwWd3qYUD+CnmBdRkAbKbAPPQEhhyB7CGCEa9NaUidXXtJnpgZ20Sk7E+BF4s5IEc1QDewvrxOTNKJCEw5KtV0n0X4GUjPC6ajgpYomQAbQo21DBURU+dnkkBE+NkoHd2NpntB4ov6eyCBSahiFpjMC3Vfz3lj8z2ununvpZ0wkqaaCTBd1U451jMep4A6TlGg+NASIZOavmPRBAtEmu4IJwZ0/eZE0TsvuedmpnRUr17M48ugAHaESctEFqqBbVEV1RNAjekav6M16sl6sd+tj2pqzZjP76A+szx/PXJku</latexit>

rwaQwa(s) = �(s)

question: can we modify Q(s,a) or the policy easily?

Approximation model

Linear model with kernels

given a set of “training data” (si, ai)

<latexit sha1_base64="fSgvn80Mazd8vETXdMymF2qqCzk=">AAACIHicbZDLSgMxFIYzXmu9jbp0EyxCC6XMiFI3haIbwU0L9gJtHTJp2oYmmSHJWMrQR3Hjq7hxoYju9GlMp11o64HAx/+fw8n5/ZBRpR3ny1pZXVvf2Extpbd3dvf27YPDugoiiUkNByyQTR8pwqggNU01I81QEsR9Rhr+8HrqNx6IVDQQd3ockg5HfUF7FCNtJM8uVr1RVuVRDpZgW0Xci2nJndxzOPIovM22wwFN7DycoUfzyKO5nGdnnIKTFFwGdw4ZMK+KZ3+2uwGOOBEaM6RUy3VC3YmR1BQzMkm3I0VChIeoT1oGBeJEdeLkwAk8NUoX9gJpntAwUX9PxIgrNea+6eRID9SiNxX/81qR7l12YirCSBOBZ4t6EYM6gNO0YJdKgjUbG0BYUvNXiAdIIqxNpmkTgrt48jLUzwruRcGpnmfKV/M4UuAYnIAscEERlMENqIAawOARPINX8GY9WS/Wu/Uxa12x5jNH4E9Z3z/PxqBH</latexit>

Qw(s, a) =mX

i=1

wiK(�(s, a),�(si, ai))

and a kernel function K

<latexit sha1_base64="qgun0BfUi/29PGGHrtpXDKvA0QU=">AAACZXicbVHLSgMxFM2Mr1pf9YEbFwaL0EIpM0XRjSB1I7hpwarQGYY7adoGk8yQZJRS/Ul3bt34G6bTLrT2QuDcc87lJidxypk2nvfpuEvLK6trhfXixubW9k5pd+9BJ5kitEMSnqinGDTlTNKOYYbTp1RREDGnj/HzzUR/fKFKs0Tem1FKQwEDyfqMgLFUVHoLJMQcolfcjl4rugZVfIWDJht08V0lSIcs52p4CiO/BpFftf0isWHFxkQMeC8xeqFHWI+oVvMVYVQqe3UvL/wf+DNQRrNqRaWPoJeQTFBpCAetu76XmnAMyjDC6XsxyDRNgTzDgHYtlCCoDsd5Su/41DI93E+UPdLgnP09MQah9UjE1inADPW8NiEXad3M9C/DMZNpZqgk00X9jGOT4EnkuMcUJYaPLACimL0rJkNQQIz9mKINwZ9/8n/w0Kj753WvfVa+bs7iKKAjdIIqyEcX6BrdohbqIIK+nIKz6+w53+6We+AeTq2uM5vZR3/KPf4BlsqwBw==</latexit>

rwQw(s, a) =hK(�(s, a),�(s1, a1)),K(�(s, a),�(s2, a2)), . . .K(�(s, a),�(sm, am))

i

kernel trick brings nonlinearity into linear model<latexit sha1_base64="dLwDwA/6Gm69U7kLyTHHLNW8R1E=">AAACNHicbVDLattAFB2lzaNOmrjJspuhpmBDY6SQkGwCodkUsnGgfoDlmNHoyh48ejBzlVrI+qhs8iHZhEIXDSHbfkPHj4Xr9MDA4ZxzuXOPl0ih0bZ/Wmtv3q5vbG69K23vvN/dK3/Yb+k4VRyaPJax6nhMgxQRNFGghE6igIWehLY3upz67VtQWsTRd8wS6IVsEIlAcIZG6pevrqrjL9nERRhj/kP4OCxq9JzCOKm6gWI8p4fuZHyYuRNa5HQ5Rl3ux0hdLQYhuzkqaK1frth1ewb6mjgLUiELNPrlB9ePeRpChFwyrbuOnWAvZwoFl1CU3FRDwviIDaBraMRC0L18dnRBPxvFp0GszIuQztTliZyFWmehZ5Ihw6Fe9abi/7xuisFZLxdRkiJEfL4oSCXFmE4bpL5QwFFmhjCuhPkr5UNmukLTc8mU4Kye/Jq0jurOSd2+Pq5cfF3UsUU+kk+kShxySi7IN9IgTcLJHXkkv8mTdW/9sp6tl3l0zVrMHJB/YP35C/pJqwU=</latexit>

K(x, y|width) = exp(�kx� ykwidth · �2

)example:

Approximation model

Decision-tree model

decision-tree model can do regression

decision-tree model is not differentiable

use regression to learn Q values

decision-tree model is more interpretable

model-update is a re-train

Approximation model

follow the BP rule to pass the gradient

Neural network model

<latexit sha1_base64="GMY4QeEZ2oTxhuWLgRmB9s41oq8=">AAAB8HicbVBNSwMxEJ2tX7V+rXr0EixCBSm7ouix6MVjBfsh7VKyabYNTbJLkhXK0l/hxYMiXv053vw3pu0etPXBwOO9GWbmhQln2njet1NYWV1b3yhulra2d3b33P2Dpo5TRWiDxDxW7RBrypmkDcMMp+1EUSxCTlvh6Hbqt56o0iyWD2ac0EDggWQRI9hY6bGbDFlFn+HTnlv2qt4MaJn4OSlDjnrP/er2Y5IKKg3hWOuO7yUmyLAyjHA6KXVTTRNMRnhAO5ZKLKgOstnBE3RilT6KYmVLGjRTf09kWGg9FqHtFNgM9aI3Ff/zOqmJroOMySQ1VJL5oijlyMRo+j3qM0WJ4WNLMFHM3orIECtMjM2oZEPwF19eJs3zqn9Z9e4vyrWbPI4iHMExVMCHK6jBHdShAQQEPMMrvDnKeXHenY95a8HJZw7hD5zPH9clj8Y=</latexit>

�(s, a)<latexit sha1_base64="jzJqBlvX45fhcj78tibXgHxkSX4=">AAAB73icbVBNSwMxEJ3Ur1q/qh69BItQQcquKHosevHYgv2AdinZNNuGZrNrklXK0j/hxYMiXv073vw3pu0etPXBwOO9GWbm+bHg2jjON8qtrK6tb+Q3C1vbO7t7xf2Dpo4SRVmDRiJSbZ9oJrhkDcONYO1YMRL6grX80e3Ubz0ypXkk7804Zl5IBpIHnBJjpXa991TWZ+S0Vyw5FWcGvEzcjJQgQ61X/Or2I5qETBoqiNYd14mNlxJlOBVsUugmmsWEjsiAdSyVJGTaS2f3TvCJVfo4iJQtafBM/T2RklDrcejbzpCYoV70puJ/XicxwbWXchknhkk6XxQkApsIT5/Hfa4YNWJsCaGK21sxHRJFqLERFWwI7uLLy6R5XnEvK079olS9yeLIwxEcQxlcuIIq3EENGkBBwDO8wht6QC/oHX3MW3MomzmEP0CfP/ujj0Y=</latexit>

Qw(s, a)

<latexit sha1_base64="iGEq+Ie4Mw75QMQYdtR9hnEpNcY=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4Kokoeix68diC/YA2lM120q7dbMLuRimhv8CLB0W8+pO8+W/ctjlo64OBx3szzMwLEsG1cd1vZ2V1bX1js7BV3N7Z3dsvHRw2dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6nfqtR1Sax/LejBP0IzqQPOSMGivVn3qlsltxZyDLxMtJGXLUeqWvbj9maYTSMEG17nhuYvyMKsOZwEmxm2pMKBvRAXYslTRC7WezQyfk1Cp9EsbKljRkpv6eyGik9TgKbGdEzVAvelPxP6+TmvDaz7hMUoOSzReFqSAmJtOvSZ8rZEaMLaFMcXsrYUOqKDM2m6INwVt8eZk0zyveZcWtX5SrN3kcBTiGEzgDD66gCndQgwYwQHiGV3hzHpwX5935mLeuOPnMEfyB8/kD5geM/w==</latexit>wparameters

Neural network model is differentiable<latexit sha1_base64="1znwYJ/HCCxsdvtiXCADtZxvk0o=">AAACK3icbZDLSgMxFIYzXmu9VV26CRaxxVpmRLQboejGZQv2Am0ZzqRpG0wyQ5KxlNL3ceOruNCFF9z6HqYXxNsPgS//OYfk/EHEmTau++rMzS8sLi0nVpKra+sbm6mt7aoOY0VohYQ8VPUANOVM0ophhtN6pCiIgNNacHM5rtduqdIslNdmENGWgK5kHUbAWMtPXfTxOe7jQ9wEHvUgo8bYBSEAl/1+Rh/k4CCLj6aXHGSzuCkh4OD3vyw/lXbz7kT4L3gzSKOZSn7qsdkOSSyoNISD1g3PjUxrCMowwuko2Yw1jYDcQJc2LEoQVLeGk11HeN86bdwJlT3S4In7fWIIQuuBCGynANPTv2tj879aIzadQmvIZBQbKsn0oU7MsQnxODjcZooSwwcWgChm/4pJDxQQY+NN2hC83yv/hepx3jvNn5RP0sXCLI4E2kV7KIM8dIaK6AqVUAURdIce0DN6ce6dJ+fNeZ+2zjmzmR30Q87HJw+SorY=</latexit>

w = w + ↵(r + �Qw(s0, a0)�Qw(s, a))rwQw(s, a)

Batch updatelearning on single sample introduces large variance, particularly for high-capacity models

Batch mode is straightforward:collect trajectory and history data

solve batch least square objective

J(w) = ED[�V ⇡ � V̂ (s)

�2]

linear function: closed formneural networks: batch update/repeated stochastic update

LSMC, LSTD, LSTD(λ)

D = {(s1, V ⇡1 ), (s2, V

⇡2 ), . . . , (sm, V ⇡

m)}

Batch methods

Batch mode policy iteration: LSPI with linear model

Q0 = 0, initial state for i=0, 1, ...

collect data

end for

<latexit sha1_base64="hSMy76CPD4DNGstP8xTA3D628JI=">AAACBXicbVDLSgNBEJz1GeNr1aMeBoOQgIRdUfQiBL14TMA8ILssvZPZZMjsg5lZNYRcvPgrXjwo4tV/8ObfOEn2oIkFDUVVN91dfsKZVJb1bSwsLi2vrObW8usbm1vb5s5uQ8apILROYh6Llg+SchbRumKK01YiKIQ+p02/fz32m3dUSBZHt2qQUDeEbsQCRkBpyTMPnIQVZQlfYgdE1wnhwQNc8+6L8hhKnlmwytYEeJ7YGSmgDFXP/HI6MUlDGinCQcq2bSXKHYJQjHA6yjuppAmQPnRpW9MIQird4eSLET7SSgcHsdAVKTxRf08MIZRyEPq6MwTVk7PeWPzPa6cquHCHLEpSRSMyXRSkHKsYjyPBHSYoUXygCRDB9K2Y9EAAUTq4vA7Bnn15njROyvZZ2aqdFipXWRw5tI8OURHZ6BxV0A2qojoi6BE9o1f0ZjwZL8a78TFtXTCymT30B8bnD7qDls0=</latexit>

⇡(s) = argmaxa

Qw(s, a)

<latexit sha1_base64="3G4UBNlS/l6Q2XLdsjzpvSv6qi4=">AAACOXicbVDLSgMxFM34tr6qLt0Ei9CilhlRdCMIunBZwarQlOFOmrahSWZIMpYy9Lfc+BfuBDcuFHHrD5jWWfg6EDj3nHu5uSdKBDfW9x+9icmp6ZnZufnCwuLS8kpxde3KxKmmrE5jEeubCAwTXLG65Vawm0QzkJFg11HvdORf3zJteKwu7SBhTQkdxducgnVSWKz18TEmoDtEchX2MTGpDLOy2YEK4QqfDXFZb2PSASkBX4R955CEl02lgnfzGiquIEmXj3lYLPlVfwz8lwQ5KaEctbD4QFoxTSVTlgowphH4iW1moC2ngg0LJDUsAdqDDms4qkAy08zGlw/xllNauB1r95TFY/X7RAbSmIGMXKcE2zW/vZH4n9dIbfuomXGVpJYp+rWonQpsYzyKEbe4ZtSKgSNANXd/xbQLGqh1YRdcCMHvk/+Sq71qcFD1L/ZLJ0d5HHNoA22iMgrQITpB56iG6oiiO/SEXtCrd+89e2/e+1frhJfPrKMf8D4+AfRxqK4=</latexit>

w = argminw

X

(s,a)2D

(r + �Qw(s,⇡(s))�Qw(s, a)))�(s, a)

<latexit sha1_base64="wI4mqrU3HrGloH+0EWaqeYIXTJ0=">AAACDHicbZDLSgMxFIYz9VbrrerSTbAILZQyUxS7EQq6cFnBXqAzDJlM2oZmkiHJCGXoA7jxVdy4UMStD+DOtzFtZ6GtPwQ+/nMOJ+cPYkaVtu1vK7e2vrG5ld8u7Ozu7R8UD486SiQSkzYWTMhegBRhlJO2ppqRXiwJigJGusH4elbvPhCpqOD3ehITL0JDTgcUI20sv1i6gVfQTcvKd6rIdypVQ3VD9UrVZaHQCrpT02XX7LngKjgZlECmll/8ckOBk4hwjRlSqu/YsfZSJDXFjEwLbqJIjPAYDUnfIEcRUV46P2YKz4wTwoGQ5nEN5+7viRRFSk2iwHRGSI/Ucm1m/lfrJ3rQ8FLK40QTjheLBgmDWsBZMjCkkmDNJgYQltT8FeIRkghrk1/BhOAsn7wKnXrNuajZd+elZiOLIw9OwCkoAwdcgia4BS3QBhg8gmfwCt6sJ+vFerc+Fq05K5s5Bn9kff4A1rqYUg==</latexit>

D = {(s1, a1), (s2, a2), . . .}

More objectives

MSE: mean square error<latexit sha1_base64="e0kiJKUuT95mOTdCGIO/MfOSaSw=">AAACLnicbZDLSgMxFIYz9VbrrerSTbAI7cIyU+ploxRFEFcV7AU67ZBJ0zY0mRmSjKUMfRJfwldwq3vBhQiufAwzbRe29UDgy/+fw0l+N2BUKtP8MBJLyyura8n11Mbm1vZOenevKv1QYFLBPvNF3UWSMOqRiqKKkXogCOIuIzW3fx37tUciJPW9BzUMSJOjrkc7FCOlJSd9cpcd5OAFvHEiaUvK7YCOoH1Fu9lqK4ovWZmDx7DqDKCm2Mi1CtBJZ8y8OS64CNYUMmBaZSf9bbd9HHLiKcyQlA3LDFQzQkJRzMgoZYeSBAj3UZc0NHqIE9mMxt8bwSOttGHHF/p4Co7VvxMR4lIOuas7OVI9Oe/F4n9eI1Sd82ZEvSBUxMOTRZ2QQeXDOCvYpoJgxYYaEBZUvxXiHhIIK53ozBaXj1I6FGs+gkWoFvLWab54X8yULqfxJMEBOARZYIEzUAK3oAwqAIMn8AJewZvxbLwbn8bXpDVhTGf2wUwZP79PxKX/</latexit>

J(w) = Es⇠⇡

⇣V ⇡(s)� Vw(s)

⌘2

MSBE: mean square Bellman error<latexit sha1_base64="S/yae9xq0ttJlRM4fGOs+0tlhxs=">AAACLXicbZDLSgMxFIYz9VbrbdSlm2AR2oVlphR1oxRFEFcVeoNOHTJp2oYmM0OSsZShL+JL+Apude9CEBdufA3Ti2Bbfwh8/OcczsnvhYxKZVnvRmJpeWV1Lbme2tjc2t4xd/eqMogEJhUcsEDUPSQJoz6pKKoYqYeCIO4xUvN6V6N67YEISQO/rAYhaXLU8WmbYqS05ZqF20w/C8/htRtLR1LuhHQInUvayZRh1e3DjMzC418a+dn7PHTNtJWzxoKLYE8hDaYqueaX0wpwxImvMENSNmwrVM0YCUUxI8OUE0kSItxDHdLQ6CNOZDMe/24Ij7TTgu1A6OcrOHb/TsSISzngnu7kSHXlfG1k/ldrRKp91oypH0aK+HiyqB0xqAI4igq2qCBYsYEGhAXVt0LcRQJhpQOd2eLxYUqHYs9HsAjVfM4+yRXuCunixTSeJDgAhyADbHAKiuAGlEAFYPAInsELeDWejDfjw/ictCaM6cw+mJHx/QM+bqTU</latexit>

J(w) = Es⇠⇡

⇣TVw(s)� Vw(s)

⌘2

<latexit sha1_base64="WKryE8Bgv0IHdLbPu72GhFJXM34=">AAACPXicbVBLSwMxGMz6rPVV9eglWKQtlrIrRb0oBRU8VukLusuSTdM2NNldkqxSlv4h/4R/wWs9Ct7Eq1fTB2JbBwKTmW/4kvFCRqUyzaGxtLyyurae2Ehubm3v7Kb29msyiAQmVRywQDQ8JAmjPqkqqhhphIIg7jFS93rXI7/+SISkgV9R/ZA4HHV82qYYKS25qZsKrLlPMCtz8BLeurHM2JJyWM7KvB1SLecGsPnwe8vLTA6eQLuDOEejZFYLjptKmwVzDLhIrClJgynKburdbgU44sRXmCEpm5YZKidGQlHMyCBpR5KECPdQhzQ19REn0onHvx3AY620YDsQ+vgKjtW/iRhxKfvc05Mcqa6c90bif14zUu0LJ6Z+GCni48midsSgCuCoOtiigmDF+pogLKh+K8RdJBBWuuCZLR4fJHUp1nwFi6R2WrDOCsX7Yrp0Na0nAQ7BEcgCC5yDErgDZVAFGDyDVzAEb8aL8WF8Gl+T0SVjmjkAMzC+fwDdg6oN</latexit>

TVw(s) = Es0⇠P (s,⇡(s))[R(s,⇡(s), s0) + �Vw(s0)]

TD

GTD (gradient TD)

[Baird, L. C. Residual algorithms: Reinforcement learning with function approximation. In ICML’95][Baird, L. C. Reinforcement Learning Through Gradient Descent. PhD thesis, Carnegie-Mellon University, 1999][Hamid Reza Maei. Gradient Temporal-Difference Learning Algorithms. PhD thesis, University of Alberta, 2011. https://era.library.ualberta.ca/items/fd55edcb-ce47-4f84-84e2-be281d27b16a/view/373459a7-72d1-4de2-bcd5-5f51e2f745e9/Hamid_Maei_PhDThesis.pdf]

An example representation should be considered

[Richard S. Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, Eric Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. ICML 2009: 993-1000]

MSBE: mean square Bellman error

<latexit sha1_base64="WGuNonGDiESJ19L5E1U+YkysnD8=">AAACRnicbVDdShtBGP02VhvTqtFeevNhEBOUsCuh9kYRSsGbQhSjQnYNs5NJMmRmd5iZrYQlT9WX6Ct4a++9E287+aEY9cDAmXO+n5kTK8GN9f17r7D0YXnlY3G19Onz2vpGeXPryqSZpqxFU5Hqm5gYJnjCWpZbwW6UZkTGgl3Hw+8T//oX04anyaUdKRZJ0k94j1NindQp/7zEu1ANOFZNDY/xRyc3e6HhEptVcxAq7uTaGNsX/28HZq+G+xj2iZQE725Dm6rJgKrTo0654tf9KfAtCeakAnM0O+WHsJvSTLLEUkGMaQe+slFOtOVUsHEpzAxThA5Jn7UdTYhkJsqn3x7jrlO62Eu1O4nFqfqyIyfSmJGMXaUkdmBeexPxPa+d2d63KOeJyixL6GxRLxNoU5xkiF2uGbVi5Aihmru3Ih0QTah1SS9sieW45EIJXkfwllwd1oOv9cZ5o3J6Mo+nCNuwA1UI4AhO4Qya0AIKv+EeHuCv98d79J6851lpwZv3fIEFFOAfJ5utRQ==</latexit>

Tw�(s) = Es0⇠P (s,⇡(s))[R(s,⇡(s), s0) + �w>�(s0)]

<latexit sha1_base64="LXE+OvTpUxv8YbpbiKzK9NZNITo=">AAACNnicbVDNSgMxGMz6W+tf1aOXYBG2B8uuiHpRiiKIpwqtLXTbkk3TNpjshiRrKcs+jS/hK3jVkxcPgnj1EUzrHmzrQGCYmY/vy/iCUaUd582am19YXFrOrGRX19Y3NnNb23cqjCQmVRyyUNZ9pAijAalqqhmpC0kQ9xmp+feXI7/2QKSiYVDRQ0GaHPUC2qUYaSO1c2c39qAAz+BVO1aeotwTNIHeBe3ZFTiAnuhTWxXgARxkW54ORSqMAoXWIWzn8k7RGQPOEjcleZCi3M59eJ0QR5wEGjOkVMN1hG7GSGqKGUmyXqSIQPge9UjD0ABxoprx+JsJ3DdKB3ZDaV6g4Vj9OxEjrtSQ+ybJke6raW8k/uc1It09bcY0EJEmAf5d1I0Y1CEcdQY7VBKs2dAQhCU1t0LcRxJhbZqd2OLzJGtKcacrmCV3h0X3uHh0e5Qvnaf1ZMAu2AM2cMEJKIFrUAZVgMEjeAYv4NV6st6tT+vrNzpnpTM7YALW9w8ywaj1</latexit>

J(w) = Es⇠⇡

⇣Tw�(s)� w>�(s)

⌘2

can be out of the representation space of w

GTD2MSPBE: mean square projected Bellman error<latexit sha1_base64="9LJllAqmMA0wpBI91vFxrkzDIaw=">AAACOnicbVDLSgMxFM34tr6qLt1cLEK7sMyUom4UHwjiqkKrQqeWTJq2oclMSDKWMvR7/Al/wa3i1p249QNM6yyseiBwOOdc7s0JJGfauO6rMzU9Mzs3v7CYWVpeWV3Lrm9c6yhWhNZIxCN1G2BNOQtpzTDD6a1UFIuA05ugdzbyb+6p0iwKq2YgaUPgTsjajGBjpWb25DLfL8AhnDcT7WsmfMmG4J+yTt6vMKhCH3zZZXldgF3oZ+58E8lUGIUKdyVoZnNu0R0D/hIvJTmUotLMvvmtiMSChoZwrHXdc6VpJFgZRjgdZvxYU4lJD3do3dIQC6obyfirQ9ixSgvakbIvNDBWf04kWGg9EIFNCmy6+rc3Ev/z6rFpHzQSFsrY0JB8L2rHHEwEo96gxRQlhg8swUQxeyuQLlaYGNvuxJZADDO2FO93BX/Jdano7RXLV+Xc8VFazwLaQtsojzy0j47RBaqgGiLoAT2hZ/TiPDpvzrvz8R2dctKZTTQB5/ML/wSqUg==</latexit>

J(w) = Es⇠⇡

⇣⇧Tw�(s)� w>�(s)

⌘2

<latexit sha1_base64="0VPLA7A/faO9kpSQz/jAgyfJYA0=">AAACAnicbVA9SwNBEJ3zM8avqKXNYhBiE+4kqJUEbCwj5AuSI+xt9pIlu7fH7p4SjnT+BVvt7cTWP2LrL3GTXGESHww83pthZl4Qc6aN6347a+sbm1vbuZ387t7+wWHh6LipZaIIbRDJpWoHWFPOItowzHDajhXFIuC0FYzupn7rkSrNZFQ345j6Ag8iFjKCjZU6dfSEuvGQlfRFr1B0y+4MaJV4GSlChlqv8NPtS5IIGhnCsdYdz42Nn2JlGOF0ku8mmsaYjPCAdiyNsKDaT2cnT9C5VfoolMpWZNBM/TuRYqH1WAS2U2Az1MveVPzP6yQmvPFTFsWJoRGZLwoTjoxE0/9RnylKDB9bgoli9lZEhlhhYmxKC1sCMcnbULzlCFZJ87LsXZUrD5Vi9TaLJwencAYl8OAaqnAPNWgAAQkv8ApvzrPz7nw4n/PWNSebOYEFOF+/QpCW/w==</latexit>

Tw�(s)

<latexit sha1_base64="aI1sjE/8oZ0Z7YtAqTMpGxPgZO8=">AAACCHicbVDLSsNAFL2pr1ofjbp0M1iEuimJFHUlBTcuK/QFTSiT6aQdOpOEmYlSSn/AX3Cre3fi1r9w65c4bbOwrQcuHM65l3M5QcKZ0o7zbeU2Nre2d/K7hb39g8OifXTcUnEqCW2SmMeyE2BFOYtoUzPNaSeRFIuA03Ywupv57UcqFYujhh4n1Bd4ELGQEayN1LOLXp2hBnpCXjJkZXXRs0tOxZkDrRM3IyXIUO/ZP14/JqmgkSYcK9V1nUT7Eyw1I5xOC16qaILJCA9o19AIC6r8yfzxKTo3Sh+FsTQTaTRX/15MsFBqLAKzKbAeqlVvJv7ndVMd3vgTFiWpphFZBIUpRzpGsxZQn0lKNB8bgolk5ldEhlhiok1XSymBmBZMKe5qBeukdVlxryrVh2qpdpvVk4dTOIMyuHANNbiHOjSBQAov8Apv1rP1bn1Yn4vVnJXdnMASrK9fR5aYjQ==</latexit>

⇧Tw�(s)<latexit sha1_base64="GrY3BQeh6DRNcTQVStlLvYacAEI=">AAACAHicbVA9TwJBEJ3DL8Qv1NJmIzHBhtwZolaGxMYSEwENXMjesgcbdvcuu3sacqHxL9hqb2ds/Se2/hIXuELAl0zy8t5MZuYFMWfauO63k1tZXVvfyG8WtrZ3dveK+wdNHSWK0AaJeKTuA6wpZ5I2DDOc3seKYhFw2gqG1xO/9UiVZpG8M6OY+gL3JQsZwcZKD0+oEw9YWZ92iyW34k6BlomXkRJkqHeLP51eRBJBpSEca9323Nj4KVaGEU7HhU6iaYzJEPdp21KJBdV+Oj14jE6s0kNhpGxJg6bq34kUC61HIrCdApuBXvQm4n9eOzHhpZ8yGSeGSjJbFCYcmQhNvkc9pigxfGQJJorZWxEZYIWJsRnNbQnEuGBD8RYjWCbNs4p3XqneVku1qyyePBzBMZTBgwuowQ3UoQEEBLzAK7w5z8678+F8zlpzTjZzCHNwvn4BPP+Wdw==</latexit>

w�(s)