Seminar I on Agricultural Process Engineering 農産 …...Seminar I on Agricultural Process...

Seminar I on Agricultural Process Engineering農産加工学演習　I

No.6

農学研究科　地域環境科学専攻農学研究科　地域環境科学専攻

近藤　直・清水　浩近藤　直・清水　浩

Naoshi Kondo, Hiroshi ShimizuDivision of Environmental Science & Technology,

Graduate School of Agriculture, Kyoto University

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Applications as robotic eyes

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Strawberry harvesting robot

Color conversionPreprocessingPreprocessing

BinarizationBinarization

Processing ofProcessing ofbinary, gray levelbinary, gray levelimagesimages

Feature extractionFeature extraction

RecognitionRecognition

3 3 D understandingD understanding

Software of machine vision as robotic eyes

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Stereo imaging for Depth measurement for 3 D reconstruction

Active stereo visionPassive stereo vision

Stereo geometryScene pointP

Z

Xf f

pL pR

b

xL xR

CL CR

bf

: Baseline: Focal length: disparity: P’s x coordinate on left camera: P’s x coordinate on right camera: Lateral offset: Distance from camera

XZ

d

Identical parallel cameras

xL

xR

d = ‐ =Z

f

Zf=

XxL

Zf=

xR ＋

，

d is proportional to f and binversely proportional to Z

Image plane

xLxR

X b

b

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Area-based stereo matching(use of larger image regions (or areas) that containenough information to yield unambiguous matches)Feature-based stereo matching(Feature extraction by color or edge detection and dealwith only points that can be matched unambiguoisly)

Correspondence problem

Occlusion problem, Ambiguous matching

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

1) Kise M., et al. : A Stereovision-based Crop Row Detection Method for Tractor-automatedGuidance, Biosystems Engineering, 90(5) 357-367 (2005).

Stereo cameraStereo camera mounted tractor 1)

Area-based stereo vision

Disparity image(darker pixel is farther)

Image size: 320 X 240Mask size m: 25 X 25d’= 0～32 (setting value)

E(d’) = Σ Σ |IL(x+i, y+j) - IR(x+i+d’, y+j)|

L R

i=-m/2

i=m/2 j=m/2

j=-m/2

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Feature-based stereo vision

Dx

Dy

b

P1

P2

Px d

Y

X

Dy＝d b/(P2-P1) Dx＝Px Dy/d Dz＝Py Dy/d

Q1: Describe how we can getQ1: Describe how we can get Dy Dy＝＝d bd b/(/(PP22-P-P11))

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Left Right

Strawberry fruit images from stereo vision

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Left Right

Matching criteria: Horizontal level and strawberry sizeshould be almost equal

Matching on binary images

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Cluster of Strawberry fruits

1

2

3

4

56

56

4

3

2

Left Right

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

1

2

3

4

56

5

6

4

3

2

Left Right

No 1: out of view in right imageNo.3: occluded on right imageNo.4: different immature parts from different angle

Un-Matched fruits (No.1, 3, 4)

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Depth measurement by use of differential object size

Dy=P1 L/(P2-P1)Dx=P2 Dy/d

Dy=L√Na1/(√Na2‐√Na1)

Dy= Nl1L/(Nl2‐Nl1)

P1

DyL

Dx

P2

d

DyL

Na1

Nl1

Na2

Nl2

Nai: area of object on image Nli : length of object on image

Q2:Describe how we can getQ2:Describe how we can get Dy= Nl1L/(Nl2‐Nl1)

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Na1: pixel numberrecognizing fruit

at position P1 at position P2

Actual images from camera attached to manipulator end

Dy=L√Na1/(√Na2‐√Na1)

Na2: pixel numberrecognizing fruit

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

3 D image from active range finder

Operating principleOperating principle(Time of flight)(Time of flight)

TransmitterReceiver

Rotating mirror

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

3 dimensional shape recognition

Phyllotaxis

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Machine vision forfruit grading system

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Extracted features from images

SizesColorsShapeDefectsMaturity Dullness (Gloss)……

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

An apple fruit with discoloration

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Color component images

R　　　　　　　　　　　　G　　　　　　　　　　　B

G‐BR‐BR‐G

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

FFT

Internal defects detection by X-ray CT

Peach with

Split-pits

Moth suckeddefect

Rotten core

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Experimental result（sprit-pit of peach）

(a) Appearance (b) Cut sample (Sprit-pit)

(d) Top view(c) Side view

Transparent ImageOutput Voltage: 50keV

Y=250

Intersection at Y=250

Gra

y Le

vel

CT Image

Split-pit

Peach with Split-pits

X ray image

X ray GeneratorMonochromeTV camera

Scintillator

Rotten core onion

Hollow potatoes

X-ray images of orange fruits

Empty portion

Top view Side view

Empty portion

Rind-puffingfruit

Normalfruit

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Ultra violet image

Visible camera

UV camera

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Review

Describe what was the phenomenon Describe what was the phenomenon ““fluorescencefluorescence””..

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Rotten part

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Fluorescence image

Fluorescence reaction

Rotten part

Color images fluorescence images

(365nm exciting light)

Rotten part

UV LEDs

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Infrared

Light

Rotten portion

Transmitted image

Texture on bioproducts

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Cooccurrence matrix and textural features

Haralick, R.M., K. Shanmugam and Its'hak Dinstein. Textural features for image classification,IEEE Transactions on systems, man, and cybernetics, Vol.SMC-3, No.6, 610-621.1973.

1

2 3

0 00 00 2

22 2

11 1

3Gray level image

0 1 2 30123

4 2 1 02 4 0 01 0 6 10 0 1 2

d=1, θ =0

Distance and direction

dθ

i j

jj

ASM:ΣΣ{p(i,j)} 2i=0 j=0

n n

ΣΣ p(i,j)/{1+(i-j)2}IDM:i=0 j=0

n n

ΣΣ (i-j)2p(i,j)CON:i=0 j=0

n n

Measurement Box Power source, PC,

Spectroscopy,PLC, amp board

Antenna of DGPS

Chisel(sensorprobe

housing)

※ Date: 2004. 11. 21　Depth: 150mm, Speed: About 30cm/ secCooperation with SHIBUYA MACHINERY CO.,LTD., TUAT

SAS1000 – Outline (Real time soil sensor)

Ready to sell!

Measurement

Nitrogen MC SOM EC pH Compaction + Image data

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Illumination fiber

EC Electrode

Laser Displacementsensor Color camera

CondensedfiberGround surfaceGround surface

Soil flattener

Arrangement of equipments in chisel

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Under-ground-Images by the soil sensor

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

a b c

d e f

g h i

(max{a,b,c,d,e,f,g,h,i}-min{a,b,c,d,e,f,g,h,i})×k

(a+b+c+d+e+f+g+h+I)/9

Textural analysis

USAD-ARS Imaging Research TeamUSAD-ARS Imaging Research Team

RDA 2004 Symposium October 13, 2004

Common ApertureCamera

Carcass

IntelligentBird Washer

Method & System for Intelligent Washing ofFecal & Ingesta Contaminants

RDA 2004 Symposium October 13, 2004

Multispectral imaging

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Machine vision for seedling production

Cutting sticking robot Transplanting robot

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

A fruit grading robot with machine vision

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

Assignment

Describe how you may apply the imagingDescribe how you may apply the imagingtechnologies to your own research project.technologies to your own research project.

KYOTO UNIVERSITY京都大学

AgriculturalAgriculturalProcessProcessEngineeringEngineeringLaboratoryLaboratory

For improving my lecture

Tell me anything (teaching techniques) I canTell me anything (teaching techniques) I canimprove for your understanding in myimprove for your understanding in mylecture. (For example, small voice, too fastlecture. (For example, small voice, too fastPPT page changing, hard to see letters onPPT page changing, hard to see letters onblackboard, more explanation, use moreblackboard, more explanation, use moreblackboardblackboard……..)..)