View
570
Download
8
Category
Preview:
Citation preview
• S. Boucheron, G. Lugosi and P. Massart:Concentration Inequalities: A Nonasymptotic Theory of Independence.Oxford Univ. Pr., 2013.
• / /
• “theory of independence”• (cf: Talagrand (1996))
3
1. Introduction ( )
2. – 9. & • Chernoff bound / Hoeffding / Bernstein
• (Efron-Stein / Poincaré)
• (Han / Pinsker / Ent. / Birge)
• Sobolev
•
•
•
10. – 15. advanced (?)• 11. – 13. sup
4
•
• (concentration inequality)•
• / / / / / / / etc…
• Twitter bio
• Talagrand (1995) •
Chernoff
• Q. (smoothness condition)
6
Hoeffding• 𝑌: [𝑎, 𝑏]
𝑉𝑎𝑟 𝑌 ≤𝑏−𝑎 2
4
• “exponential change” ( lem2.2)
𝜓𝑌−𝐸𝑌 𝜆 ≤𝜆2 𝑏−𝑎 2
8
• Hoeffding• 𝑋1, … , 𝑋𝑛 : [𝑎𝑖 , 𝑏𝑖]
• 𝑍 = 𝑖 𝑋𝑖
𝜓𝑍−𝐸𝑍 𝜆 =
𝑖
𝜓𝑋𝑖−𝐸𝑋𝑖(𝜆) ≤
𝜆2𝑣
2
• where 𝑣 ≔ 𝑖𝑏𝑖−𝑎𝑖
2
4= cumulant 𝑍 sub-Gaussian
9
(BDC)• smoothness condition
• (bdd. difference condition)
• 𝑥𝑖
• Hamming 𝑑𝑐 𝑥, 𝑦 = 𝑖 𝑐𝑖1 𝑥𝑖≠𝑦𝑖
1-Lipschitz
• : BDC
10
• 𝑓: BDC
• 𝑍 = 𝑓(𝑋1, … , 𝑋𝑛)
• 𝑍• Δ𝑖 ≔ 𝐸 𝑍 𝑋1, … , 𝑋𝑖 − 𝐸[𝑍|𝑋1, …𝑋𝑖−1 ]
• 𝑍 − 𝐸𝑍 = 𝑖 Δ𝑖
• BDC ⇔ Δ𝑖 𝑐𝑖
• Hoeffding ineq.
𝜓𝑍−𝐸𝑍 𝜆 ≤𝜆2
2⋅
1
4 𝑐𝑖
2
• bounded distance inequality / McDiarmid
11
• (isoperimetry)•
• 𝑛- (Lebesgue 𝜆)• 𝐴 ⊂ ℝ𝑛 : ( )
• 𝐴𝑡 ≔ {𝑥 ∈ ℝ𝑛 ; 𝑑 𝑥, 𝐴 < 𝑡} 𝐴 𝑡-blowup ( )
• 𝐴 𝑛- 𝐵
𝐴
𝑡∀𝑡 > 0, 𝜆 𝐴𝑡 ≥ 𝜆(𝐵𝑡)
15
• 𝑆𝑛−1 (Lévy )• 𝑆𝑛−1
(= )
• 𝜇 𝐴 ≥1
2
•
𝜇 𝐴𝑡𝑐 ≤ 𝜇 𝐵𝑡
𝑐 = exp −𝑛 − 1 𝑡2
2
• 𝜇 𝐴 ≥1
2𝐴𝑡
𝑡• 𝑛 − 1 (= )
≤𝐴 𝐵
16
Lipschitz (2)• 𝐴 𝑑 𝑡
• 𝐴
• 𝑥 ∈ 𝐴𝑡 𝑓 𝑥 < 𝑀𝑓 𝑋 + 𝑡• 𝑑 𝑥, 𝑦 < 𝑡 𝑦 ∈ 𝐴
𝑓 1-Lipshitz𝑓 𝑥 − 𝑀𝑓 𝑋 ≤ 𝑓 𝑥 − 𝑓 𝑦 ≤ 𝑑 𝑥, 𝑦 < 𝑡
18
Gauss• Gauss (Gauss 𝛾 )
• Borell (1975), Tsirelson, Ibragimov & Sudakov (1976)
• ( Sec10.4)
• Gauss 𝐻 extremal set
• ( ) 𝛼(𝑡) explicit
• 𝑃 𝐴 ≥1
2
20 (GP)
Efron-Stein• 𝑋 = (𝑋1, … , 𝑋𝑛)
• 𝑋(𝑖) = (𝑋1, … , 𝑋𝑖−1, 𝑋𝑖+1, … , 𝑋𝑛)
• Efron-Stein (Sec. 3.1)
• [Efron & Stein 1981] 𝑓
• [Steele 1986] 𝑓
• ( : r.v. + Jensen)
30
Sobolev• ≤
Sobolev
• Gaussian log-Sobolev (Chap. 5)
• : Gauss Sobolev
• log-Sobolev (Chap. 6)
• Gaussian Sobolev• Gaussian vector
•
32
Sobolev (1)Herbst
• Sobolev
• log-Sobolev: ≤ *
• 𝑓: ℝ𝑛 → ℝ 1-Lipshitz• ∇𝑓(𝑋) ≤ 1
• 𝑔 𝑥 = exp𝜆𝑓 𝑥
2(𝜆 > 0)
33
≤ 1
(2)( )
•• 𝑋~𝑃 𝑇 𝑌 = 𝑇(𝑋) 𝑄
𝑇
• 𝑥 y = 𝑇(𝑥) 𝑐(𝑥, 𝑇 𝑥 )• 𝑐 𝑥, 𝑦 = 𝑑(𝑥, 𝑦) ( )
• ≒ 𝑇
• 𝑇• : 1 2
• well-defined
• [Villani08, Chap. 4]
39
• /
• P. Massart: Concentration Inequalities and Model Selection. Springer, 2003.
• M. Ledoux: The Concentration of Measure Phenomenon. AMS, 2001.
• :
(pdf)
• M. Ledoux• Concentration of measure and logarithmic Sobolev inequalities
http://www.math.duke.edu/~rtd/CPSS2007/Berlin.pdf
• Isoperimetry and Gaussian analysishttp://www.math.univ-toulouse.fr/~ledoux/Flour.pdf
• G. Lugosi• Concentration-of-measure inequalities (@MLSS03/05)
http://www.econ.upf.edu/~lugosi/anu.pdf
• S. Boucheron• Concentration inequalities with machine learning applications ( )
www.proba.jussieu.fr/pageperso/boucheron/SLIDES/tuebingen.pdf
44
Recommended