Upload
mark-pavlidis
View
1.576
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Core Image talk at Toronto Area Cocoa and WebObjects Developers Group
Citation preview
Core Image
Mark PavlidisCo-founderFlixel Photos Inc.
Toronto Cocoa and WebObjects Developers Group2013/01/08
Motivation Why use Core Image?
• Image processing and analysis
• Fast, efficient filters
• Auto enhancement
• Feature detection
• Still and “real-time” video
• Simple Objective-C API
Definitions• Image Filter
• a single transform or effect
• built-in or loaded from an Image Unit plugin (OS X only)
• Framework
<CoreImage/CoreImage.h>
• Key Classes
CIImage, CIFilter, CIContext
Overview
• Pixel Accurate
• Non-destructive
• Filters can be chained together
MonochromeFilter
Apply an image filter to the source pixel data
Core Image RuntimeHow it works
Core Image RuntimeHow it works
• Operates on image data types from:• Core Graphics• Core Video• Image I/O
Core Image RuntimeHow it works
• Built-in or plugin filters apply effects• Written in the Core Image Kernel Language• JIT complier assembles instruction pipeline
Core Image RuntimeHow it works
• Executed on the GPU or CPU• Low-level graphics processing encapsulated by CI• Performance: GLSL capabilities of the GPU or
processing power of the CPU
Core Image Inputs• Photo Library and Files
imageWithContentsOfURL:
• Live Video CaptureimageWithCVPixelBuffer:
imageWithCVImageBuffer:
• In MemoryimageWithCGImage:
•GL TextureimageWithTexture:size:flipped:colorSpace:
Core Image Outputs • CGImageRef
• Output to UIImage, ImageIO, ALAssetLibrarycreateCGImage:fromRect:
• CAEAGLLayerdrawImage:inRect:fromRect:
• CVPixelBufferRefrender:toCVPixelBuffer:
• Bitmaprender:toBitmap:rowBytes:bounds:format:colorSpace:
Simple Filter
// Create a CIImage CIImage *ciImage = [CIImage imageWithContentsOfURL:myURL];
// Create a CIFilterCIFilter *filter = [CIFilter
filterWithName:@”CIColorMonochrome”];[filter setValue:ciImage forKey:kCIInputImageKey];[filter setValue:@(0.5) forKey:@”inputIntensity];
// Render the filter output image into a UIImageUIImage *uiImage = [UIImage
imageWithCIImage:filter.outputImage];
Filter Chains// Filter 1CIImage *output = [CIFilter
filterWithName:@”CIColorMonochrome”] keysAndValues: kCIInputImageKey, ciImage, @”inputIntensity, @(0.5), nil].outputImage;
// Filter 2output = [CIFilter
filterWithName:@”CIVignette”] keysAndValues:kCIInputImageKey, output,@”inputIntensity, @(0.5), @”inputRadius, @(1.5), nil].outputImage;
•CI defers pixel processing until render is requested•CI optimizes the render graph (e.g., sepia then scale)
Math!• An image filter is kernel
•Output is the convolution of the source & kernel
• Commutative, associative, distributive, associative with scalar multiplication, multiplicative identity
1 1 1
-1 2 1
-1 -1 1
2 2 2
2 1 3
2 2 1
6
SourceKernel
Output
Demo - QuartzComposer
Demo - Simple filter
Tips and Best Practices• CIImage and CIFilter are autoreleased
• Use autorelease pools
• CIImage, CIContext are immutable and thread-safe, CIFilter is not
• CIContext is expensive
• Stores a lot of state information so reuse them
• Avoid CA animations when rendering on GPU
•Use smaller images when possible
• Use Core Animation to upscale view, texture, framebuffer
•Disable colour management
• If real-time performance and/or reduced quality is unnoticeable
Face Detection• Identifies rectangles that contain human faces
• Feature detection
• eyes & mouth position, tracking ID and frame count (video)
CIContext *context = [CIContext contextWithOptions:nil];
NSDictionary *opts = @{CIDetectorAccuracy, ! ! ! ! ! ! ! CIDetectorAccuracyHigh};
CIDetector *detector = [CIDetector ! ! ! ! ! ! detectorOfType:CIDetectorTypeFace! ! ! ! ! ! ! ! context:context ! ! ! ! ! ! ! ! options:opts];
NSArray *features = [detector featuresInImage:myImage ! ! ! ! ! ! options:opts];
CIDetector
Demo - Redacted Faces
OS X and iOS Differences• Some CIImage create methods differ
•Has more built-in filters
• Permits custom filter plugins
• filter.outputImage is iOS only, use [filter valueForKey:kCIOutputImage]
• Remember to [filter setDefaults]
• All input parameter keys have strings defined
Portability
Alternatives•GPUImage (iOS)
✓Hand-tuned OpenGL ES shaders that are much faster than CI
✓Custom filters
- Lacks advanced features of Core Image
• Aviary
✓ Everyone is using it (Twitter, Flickr)
- Everyone is using it
• Only still images
•Hardcore - Write your own GL shaders
• Don’t be silly
Summary
• Fast and efficient image processing framework
• Simple Objective-C API
• Encapsulates the low-level instruction details
•Many built-in filters
• Advanced processing features:
• Face Detection
• Auto Enhancement
• Real-time video filters
Core Image
More Information• Core Image Programming Guide
•WWDC 2012 Videos • Session 510 - Getting Started with Core Image
• Session 511 - Core Image Techniques
• Sample Code
• https://github.com/mhpavl/FilterDemo
• https://github.com/mhpavl/RedactedFaces
• Core Image Funhouse (TBD)
[self release];
Mark Pavlidis
Twitter: @mhp
ADN: @mhp