Upload
jonathon-hare
View
178
Download
0
Tags:
Embed Size (px)
Citation preview
ImageTerrierwww.imageterrier.org
OpenIMAJIntelligent Multimedia Analysis In Java
www.openimaj.org
OpenIMAJ and ImageTerrierJava Libraries and Tools for Scalable Multimedia
Analysis and Indexing of Images
Jonathon Hare, Sina Samangooei & David Dupplaw The University of Southampton
What is OpenIMAJ?
Scalable Clustering
Scalable KNN
OpenIMAJIntelligent Multimedia Analysis In Java
Image
Analysis
LibrariesCommand-line Tools
Hadoop Map-Reduce Implementations
HardwareMachine Learning
Video
Web Audio
Analysis
AnalysisProcessing
ProcessingProcessing
Analysis Rendering
Kinect Audio & Video
CaptureExotic
Hardware
LibrariesCommand-line Tools
Hadoop Map-Reduce Implementations
Efficient, Compressed
Inverted Index
Query Engine
Term Metadata
IntegrationBased on
Terrier Constructable from Images
Distance Metrics
scaleRe-Ranking affine
orientation
Compressed
Extensible
OpenIMAJIntelligent Multimedia Analysis In Java
Scalable
e.g SIFT BoVW (x, y, scale, etc)
What is ImageTerrier?Efficient inverted index construction for image/video features (and potentially audio features too)
Augmented inverted index, capable of storing almost any kind of meta information per term
Easily extensible
Provided implementations contain things like (x,y,scale) for SIFT BoWV
Indexing is scalable across machines using Hadoop
Complete OpenIMAJ integration
Included tools can build an index straight from a set of images using common techniques like SIFT BoVW or read features extracted from other tools.
This can even be done on Hadoop!
Easy to define searching and re-ranking strategies
Various distances, weightings and constraints (such as Affine & Homographic) are provided
Tested with corpuses in excess of 10M images
A Little history...IAMMediaEngine
c. 2005
MapSnapc. 2006
JCVMLTKc. 2009
OpenIMAJ & ImageTerrier
May 2011
MoVEc. 2010
Pure-Java SIFT & Matching
Extensible Java (& JNI) Content-
Based Retrieval System
Refactoring of older code, addition of face detection
and machine learning components
Initial attempts at scalable BoWV indexing using terrier,
shape analysis, etcMany new features added.
Improvements to ImageTerrier, face pipeline, demos, video capture and
hardware, ... PhotoCopainc. 2005
Image classifiers
SMS-NLPc. 2010
Machine learning, clustering
Massive code cleanup and documentation effort. First
Open Source Release
It’s pure Java... (with the exception of thin interfaces for hardware)
Isn’t it slow?
Why another set of libraries?Scalability + Portability. OpenIMAJ projects work on:
Android mobile devices
Hadoop clusters
...without recompilation!
Easy to use and extend
Implement many important Multimedia algorithms
While allowing easy adaption and extensions.
Software engineering principles adopted
clear separations of concerns
maintainable, understandable
We simplify data representations:Images are 2D-arrays of pixels;
Videos are iterable streams of Images
Audio is an array of samples
Both OpenIMAJ and ImageTerrier have been designed from the ground-up to be easily extensible.
We’ve made it easy to modify and understand algorithms
and techniques by separating different components.
Dominant(orienta+on(of(an(orienta+on(histograms(
Replace'with'a'Null'orienta1on'extractor.'
Regular(SIFT(feature(provider(
Replaced(with(Irregular(Binning(
Rapid Application Development
“Make it easy to use”
This philosophy lets researchers build complex applications quickly
Example: Southampton Goggles
synchronised: 6 cameras, GPS device, compass
feature extraction and index construction
mobile integration
…all programmed over 2 days!
Find out more at: http://bit.ly/SOTONGOG
Student projects
Our students use OpenIMAJ (…for 2 years now!)
OpenIMAJ lets them:
Get something basic working quickly
Understand the algorithms they are using
Implement new algorithms in a structured way
Contribute to a living, open source project
To support our students we wrote a comprehensive OpenIMAJ tutorial.
Research applications
Through combination of OpenIMAJ and ImageTerrier various multimedia research applications can be addressed
- We have done so for over 6 years!
1)Guess The Building - part of the LiveMemories project
Estimate geolocation and semantic information for an un-annotated image of a landmark
2)Stock Photo Finder - part of the Living Knowledge project
Find the source of a web image in a stock photo database
Any Questions?Try the OpenIMAJ tutorial for yourself:
http://openimaj.org/tutorial.pdf