Embed Size (px)
Citation preview
Le me• Day job python developer.
• Hobby technologies
o Machine Learning
o Hardware Hacking
o Image processing
o Graphics & Video design
Introduction • Open Computer Vision (OpenCV) developed &
donated to FOSS by Intel.
• Open Source Image Processing library in C/C++
• Optimized for real time image / video processing applications
• OS/hardware independent with ports to Java, Python with wrappers to numerous other languages and platforms.
Why OpenCV?• No need to reinvent the wheel to deal with image
processing solutions.
• Has both high level and low level components.
• Lets you treat images as a 2D Matrix.• Image processing comes down to simple
mathematical operations on this Matrix.
• Also provides a framework for machine learning, 3D scene information, depth profile, object recognition, motion tracking etc..
Suiting up• Sun JDK 6
o Very important you DO NOT use Sun JDK 7. unless you want to deal with intermittent weird run time issues that make no sense. In that case go right ahead.
• Android SDK + NDK o Keep it updated once in a while.o Remember:You need “ia32-lib” if you’re running a 64 bit ubuntu/debian o NDK is required only if native code is erquired to be compiled.
• Android SDK tools rev14+o fun fact: using older versions can cause heartburn.
• SDK Platform API 8 is minimum requiredo But usually a device running 11 would have the processing power you need to play with image
processing application without lag.
• Eclipse with ADT + CDT o Duh.
OpenCV4Android SDK• Obtain CV-2.4.2-android-sdk.zip from sourceforge.• SDK
o Contains the openCV libraries and Java classes for Android.o /java, /native and /etc
• APKo Opencv manager apk that must be installed to make the openCV libaries on the
android device available.o Android market installation can also be done.
• Sampleso Contains a decent amount of openCV samples for android.o These samples are written as documentation.
o See the openCV wiki for setting up Eclipse with OpenCV.
Overview of some openCV classes
in /sdk/java
• Contains the Async loader that you’ll need to connect your application to the openCV libraries installed on your device.
• Utility function to convert between openCV data types and android ones.
• Some other loader classes for your viewing pleasure.
org.opencv.android
• Contains the core building blocks of openCV
• Matrix, point, rect and Bitmap data types defined here.
org.opencv.core
• Allows you to draw some GUI components on the screen.
• Also has the openCV videcapture class as an alternative to the android camera interface.
org.opencv.highgui
• The main image processing functions exist here, like blurring edge detection, colour conversion.
• This file also contains documentation and usage of each function.
org.opencv.imgproc
• Contains classes to assist in developing almost any machine learning algorithm in images.
• Trees, statistical models, Bayes, Markov models, K nearest all provided here with in-code commented documentation
org.openCV.ml
Programming methodology
The application structure
• Best way (as suggested in samples) is as follows.
• MainActivityo Extends the android activity class & is the Launcher Activity.o Initializes the openCV libraries.o Handles the android activity lifecycle functions.
• opencvViewo Extends opencvViewBaseo Running openCV code goes here.
• opencvViewBaseo Extends surfaceView as a holder for the video streamo Abstract class to simplify usage of OpenCVo Sets up and handles camera / preview & drawing
MainActivity.java• onCreate()
o This is called when the activity is first created.o OpenCVLoader.initAsync(…) attempts to start the openCV library
and on completion calls mOpenCVCallBack.
• mOpenCVCallBack()o Inside there is a function called onManagerConnected() which
checks to see if no errors were thereo If openCV loaded sucessfully, then an instance of Sample1View can
be created called say mView
• Also this class has the onPause(), onResume() functions which start / stop camera streaming as per the android activity lifecycle.
opencvView.java• This class contains code relevant for your application, this
directly extends the absract opencvViewBase class.
• processFrame(byte[] data)o Input comes in as byte[] data as raw camera feed
o Using the mat.put() this picture can be stored in a matrix
o Any openCV functions from imgproc etc.. can be performed on this mat object
o Display must be in bitmap format, so the mat object is converted to bmp
o This bmp object is returned to run() which displays it on the screen.
• Camera mCamera; o Camera object to open/close/ retrieve video information
• SurfaceHolder mHolder; o Holder to the surfaceview to enable a live streaming of video
• Mat mRgba;o Matrix Object that openCV uses for image manipulation
• Bitmap mBitmap;o Bitmap object that can be displayed on the surface view
• mRgba.put(0, 0, data);o Returns the raw data feed in an openCV matrix format.
• Imgproc.cvtColor(mGraySubmat, mRgba, Imgproc.COLOR_GRAY2RGBA, 4);o How to convert an RGB matrix to a greyscale
• Utils.matToBitmap(mRgba, bmp);o Converting a matrix to a bitmap
Real Time Applications
Real time Face detection
• fdActivity example is an openCV sample that shows how to implement face detection algorithm on a real time video stream.
• It is also an example on how to efficiently implement an algorithm in C and call it via the JNI interface.
• Also implements an FPS meter class to judge the usability of the algorithm used.
• A Haar Classifier from the ml library with the openCV trained xml file is used on the image stream to return if the image contains a face or not.
• It is the standard openCV way of face detetcion.
Possible Applications on android devices
• Obtain features from any image and send it over network for matching against an online database for object recognition.
• Use the front camera to monitor head movement and provide a hands free interface for some games.
• Auto-tagging of friends in photos taken and associating them with facebook/android contacts.
