Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
Matlab8:K-NearestNeighborClassifiers
Cheng-HsinHsuNa#onalTsingHuaUniversity
DepartmentofComputerScience
SlidesarebasedonthematerialsfromProf.RogerJang
CS3330ScienAficCompuAng 1
04/12/16 2
Concept of KNNC • Steps:
1. Find the first k nearest neighbors of a given point
2. Determine the class of the given point by a majority vote among these k neighbors
Feature1
Feature2 :class-Apoint
:class-Bpoint:pointwithunknownclass
3-nearestneighborsThepointisclassBvia3NNC.
04/12/16 3
Flowchart for KNNC
Featureextrac*on
Datamodeling
Evalua*on
GeneralflowchartofPR: KNNC:
Fromrawdatatofeatures
DistanceComputa@onForKNNC
Clustering(op@onal)
04/12/16 4
Decision Boundary for 1NNC Delaunay triangulation: maximize the minimum angle of all the angles of each triangle Voronoi diagram: piecewise linear boundary
Characteristics of KNNC
• Strengths of KNNC – Very Intuitive – No data modeling required
• Drawbacks of KNNC – Massive computation required when dataset is
big – No straightforward way to choose the value of K – Rescaling the dataset along each dimension may
be tricky
04/12/16 5
04/12/16 6
Preprocessing/Variants for KNNC
• Preprocessing: – Data rescaling to have zero mean and unit
variance along each feature – Value of K obtained via trials and errors
• Variants: – Weighted votes – Nearest prototype classification – Edited nearest neighbor classification – k+k-nearest neighbor
Given (x1, x2, . . . , xN
), x
0n
=x
n
� µ
x
�
x
8n = 1, 2, . . . , N
Demos by Cleve
• Delaunay triangles and Voronoi diagram – http://mirlab.org/jang/books/dcpr/example.rar
04/12/16 7
Natural Examples of Voronoi Diagram
04/12/16 8
Natural Examples of Voronoi Diagram (cont.)
04/12/16 9
1NNC Decision Boundaries
• 1NNC Decision boundaries
04/12/16 10
1NNC Distance/Posterior as Surfaces and Contours
04/12/16 11
Using Prototypes in KNNC
• No. of prototypes for each class is 4.
04/12/16 12
10 20 30 40 50 60
5
10
15
20
25
sepal length
sepa
l wid
th
iris
10 20 30 40 50 60
5
10
15
20
25
Input_1
Inpu
t_2
Dataset and decision boundary
Decision Boundaries of Different Classifiers
• Quadratic classifier
• 1NNC classifier
• Naive Bayes classifier
04/12/16 13
10 20 30 40 50 60
5
10
15
20
25
sepal lengthse
pal w
idth
iris
Matlab#7Homework(M7)1. (1%)Giventwosetsof2D
points:Redonesat{(0,1),(2,3),(4,5)};andblueonesat{(2,0),(5,2),(7,3)}.PleaseuseEuclideandistanceinthisexercise.a) DrawtheKNNCdecision
boundaryforK=1.Markthecolorsforeachpoint.
b) MulAplethe6points’xvaluesby10.Replotthefigure,andexplainhowthechangeaffectsthedecisionboundary.Howwouldthisaffectrealworldproblem?
c) Whatistheoutputof3-NNCofthepoint(1,2)?Ifyoucanaddpointsinthe2Dpoints,inordertofliptheclassificaAonof(1,2)?Showyourwork,e.g.,whereyouaddthepointsandthevaluesofthem.
d) WhatisthecomplexityofKNNCalgorithm?Showyourwork.
CS3330ScienAficCompuAng 14
QuesAons?
CS3330ScienAficCompuAng 15