Soft Real-Time Systems - NUS Computingcs3249/lecture/soft rts.pdf · CS3249 Soft Real-Time Systems 1 CS3249 CS3249 Soft Real-Time Systems 1 Soft Real-Time Systems Leow Wee Kheng CS3249

CS3249 Soft Real-Time Systems 1CS3249

Soft Real-Time Systems 11CS3249 Soft Real-Time Systems

Soft Real-Time Systems

Leow Wee KhengCS3249 User Interface Development

CS3249 Soft Real-Time Systems 2CS3249 2Soft Real-Time Systems

Real-Time Systems

Real-time systems must react to inputs “quickly”.

CS3249 Soft Real-Time Systems 3CS3249 Soft Real-Time Systems 3CS3249Soft Real-Time Systems 3Soft Real-Time Systems 3CS3249

aircraft autopilot


GPS navigation system


car engine control unit


If your car breaks take 10 sec to respond...


Real-time systems are

closer than you think!


video player


video surveillance


virtual reality


interactive system with kinect


my fancy computer system


Types of Real-Time Systems

Hard real-time systems Timing is critical. Failure to meet deadline can result in disaster. Take CG2271 Real-Time Operating Systems.

Soft real-time systems Timing is less critical. Failure to meet deadline results in degraded performance.


Activation Modes

Time-triggered Task is activated by timer. Periodic: If activation is at fixed interval. Aperiodic: If activation is not at fixed interval.

Event-triggered Task is activated by event, e.g., mouse click. Usually aperiodic.


Processor Load

For simplicity, let's consider periodic tasks e.g., displaying video frames at regular interval.

Ti : period or interval task i

Ci : computation time of task i

Processor utilisation

Total processor utilisation

U=∑i=1

n

U i

i

ii TCU


Processor Load

Overload Not enough time to service all demands. Total processor utilisation U > 1.

Overrun A task misses its deadline. May or may not cause overload.

Remedy Detect overload or overrun and manage the problem.


Decoding time of a sequence from Star Wars.

Worst case is rare butmuch longer than average.


load

average

time

No overload butinefficient use of CPU resources


load

average

time

Efficient use of CPU resourceswith transient overload


Methods for Handling Overload

Adaptive quality of service Change to low-quality service when load is high.

Multithreading Run tasks in multiple threads.

Task skipping Skip (less important) tasks.

Others


Adaptive Quality of Service

Change quality of service when needed. Provide high-quality service when CPU load is low. Switch to low-quality service when CPU load is high.

during rotation (high CPU load) after rotation (low CPU load)


Case Study: Video Capture and Processing

image widget for image widget forcaptured video frame processed frame

consoleExample 1: Single Thread


Console

inputimage widget

outputimage widget

Webcam ImageProcessor

OpenCVvideo capture

library

OpenCVimage processing

library

setimage

framecaptured

frameprocessed

queryframe

processimage

processframetimer

event

frame ratecomputed

frame ratecomputed


// Webcam.h

class Webcam: public QObject{ Q_OBJECT

public: Webcam(int width, int height, ImageProc *p); ~Webcam(); bool setDevice(int device); void setFrameRate(int fps); void start(); void stop();

signals: void frameCaptured(QImage frame); protected: void timerEvent(QTimerEvent *event);


private: ImageProc *imageProc; int width, height, frameRate; CvCapture *capture; IplImage *frame; int timerId;};


// Webcam.cpp

bool Webcam::setDevice(int dev){ if (capture) { cvReleaseCapture(&capture); // reset capture = NULL; } capture = cvCaptureFromCAM(dev); if (!capture) return false; cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, width); cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, height); return true;}


void Webcam::setFrameRate(int fps){ frameRate = fps;}

void Webcam::start(){ timerId = startTimer(1000 / frameRate);}

void Webcam::stop(){ killTimer(timerId); timerId = 0;}


void Webcam::timerEvent(QTimerEvent *event){ if (event->timerId() == timerId) { frame = cvQueryFrame(capture); QImage image = IplImage2QImage(frame); emit frameCaptured(image);

// Direct function call to imageProc. imageProc->processFrame(frame); } else QObject::timerEvent(event);}


// ImageProc.h

class ImageProc: public QObject{ Q_OBJECT public: void setProcessType(int type); void processFrame(IplImage *frame); signals: void frameProcessed(QImage image); private: int procType; IplImage *process(IplImage *frame);};


// ImageProc.cpp

void ImageProc::setProcessType(int type){ if (type >= 1 && type


IplImage *ImageProc::process(IplImage *source){ IplImage *target = cvCreateImage( cvSize(source->width, source->height), source->depth, source->nChannels); switch(procType) { case 1: // do nothing return cvCloneImage(source); case 2: // demo fast algo: fast smoothing cvSmooth(source, target, CV_GAUSSIAN, 5); return target;

case 3: // demo slow algo: filter with large kernel ... return target; }}


// Console.cpp

Console::Console(Webcam *w, ImageProc *p){ webcam = w; connect(webcam, SIGNAL(frameCaptured(QImage)), this, SLOT(getCameraImage(QImage))); imageProc = p; connect(imageProc, SIGNAL(frameProcessed(QImage)), this, SLOT(getProcessedImage(QImage))); inputPanel = new ImageWidget(WIDTH, HEIGHT); connect(inputPanel, SIGNAL(frameRateComputed(float)), this, SLOT(showInputFps(float))); outputPanel = new ImageWidget(WIDTH, HEIGHT); connect(outputPanel, SIGNAL(frameRateComputed(float)), this, SLOT(showOutputFps(float))); ...}


Test Results

Process Type 1, 2: fast Input frame rate = output frame rate = 25 fps (desired rate). All CPUs run at about 25% load or less.

Type 1, 2 Type 3


Test Results

Process Type 3: slow Input frame rate = output frame rate = 10 fps < desired rate. Video displays lack behind user actions. Sometimes, 1 CPU is fully loaded. Why input frame rate = output frame rate?


Case Study: Video Capture and Processing

Example 2: Multiple Threads

Console, image widgets Main thread: thread 1.

Webcam Secondary thread: thread 2.

Image processor Secondary thread: thread 3.

Buffer Shared buffer between webcam and image processor. Maintains mutex lock.


Console

inputimage widget

outputimage widget

Webcam ImageProcessor

OpenCVvideo capture

library

OpenCVimage processing

library

setimage

framecaptured

frameprocessed

queryframe

processimage

timerevent

frame ratecomputed

frame ratecomputed

Bufferadd del loop ortimer

event

status


// Webcam.h

class Webcam: public QThread{ Q_OBJECT public: Webcam(int width, int height, Buffer *b); ~Webcam(); bool setDevice(int device); void setFrameRate(int fps); void start(); void stop();

signals: void frameCaptured(QImage frame); protected: void timerEvent(QTimerEvent *event);


private: Buffer *buffer; int width, height, frameRate; CvCapture *capture; IplImage *frame; volatile int timerId;};


// Webcam.cpp

void Webcam::start(){ timerId = startTimer(1000 / frameRate); buffer->clear();}

void Webcam::stop(){ killTimer(timerId); timerId = 0;}


void Webcam::timerEvent(QTimerEvent *event){ if (event->timerId() == timerId) { frame = cvQueryFrame(capture); QImage image = IplImage2QImage(frame); emit frameCaptured(image);

// Doesn't call imageProc directly. buffer->addToFront(frame); } else QObject::timerEvent(event);}


Image Processing Types

Type Process Speed Mechanism

1 copy fast Explicit loop

2 copy fast Timer event

3 fastsmoothing fast Explicit loop

4 fast smoothing fast Timer event

5 slowfiltering slowExplicit loop

Don't skip frames

6 slowfiltering slowExplicit loopSkip frames


// ImageProc.h

class ImageProc: public QThread{ Q_OBJECT public: ImageProc(Buffer *b); void setProcessType(int type); void start(); void stop(); protected: void run(); void timerEvent(QTimerEvent *event); signals: void frameProcessed(QImage image);


private: Buffer *buffer; int frameRate; volatile int procType; volatile bool stopped; volatile int timerId; void processFrame(); IplImage *process(IplImage *frame);};


// ImageProc.cpp

void ImageProc::start(){ if (procType == 2 || procType == 4) // Use timer event. timerId = startTimer(1000 / frameRate); else { stopped = false; // Don't use timer event. QThread::start(); // Will call run(). }}

void ImageProc::stop(){ stopped = true; killTimer(timerId); timerId = 0;}


void ImageProc::run(){ while (!stopped) // Run explicit loop. if (!(buffer->isEmpty())) processFrame();}

void ImageProc::timerEvent(QTimerEvent *event){ if (event->timerId() == timerId) processFrame(); else QObject::timerEvent(event);}


void ImageProc::processFrame(){ IplImage *frame = NULL; if (procType != 6) frame = buffer->delAtBack(); // Don't skip frames. else frame = buffer->skipToFront(); // Skip frames.

if (frame) { IplImage *pframe = process(frame); QImage image = IplImage2QImage(pframe); emit frameProcessed(image); cvReleaseImage(&frame); cvReleaseImage(&pframe); }}


// Buffer.h

class Buffer: public QObject{ Q_OBJECT public: Buffer() {front = back = 0;} // Init to empty buffer. bool isEmpty(); void addToFront(IplImage *frame); IplImage *delAtBack(); IplImage *skipToFront(); void clear();

signals: void status(int value);


protected: enum {size = 100}; IplImage *data[size]; int front; // Points to front of buffer. int back; // Points to back of buffer. QMutex mutex;};


// Buffer.cpp

bool Buffer::isEmpty(){ return (back == front);}


void Buffer::addToFront(IplImage *frame){ mutex.lock(); if (front != back - 1 and !(front == size - 1 and back == 0)) // Not full { ++front; // Increment front. if (front == size) front = 0; // Wrap around.

IplImage *clone = cvCloneImage(frame); data[front] = clone; emit status(0); // Buffer not full. } else emit status(1); // Buffer full. mutex.unlock();}


IplImage *Buffer::delAtBack(){ IplImage *frame = NULL; mutex.lock(); if (back != front) // Not empty. { ++back; // Increment back. if (back == size) back = 0; // Wrap around.

frame = data[back]; // Get data. emit status(0); // Buffer not empty. } else emit status(-1); // Buffer empty mutex.unlock();

return frame;}


Similarly for IplImage *Buffer::skipToFront()

void Buffer::clear()


Test Results

For process type 1 – 4: fast Input frame rate = output frame rate = 25 fps (desired rate).

Type 1, 3 Type 2, 4


Test Results

Process Type 1, 3: Most of the time, 1 CPU is at (almost) full load.

Process Type 2, 4: All CPUs are at 25% load or less.

What causes the difference?


Test Results

Process Type 6: slow Output frame rate < 1/2 input frame rate = 25 fps. By skipping frames, image processor is in sync with webcam. Buffer is not full.


Multithreading vs. Efficiency

Multithreading lets multiple threads run concurrently.

But, multithreading ⇒ efficient processing. Two factors:

Processing time of tasks: long or short. Dependency between tasks.

Example: ImageProc depends on Webcam thru' Buffer.


Multithreading can't speed up slow tasks.


Speeding Up Slow Tasks

How to speed up a slow task? Use efficient algorithm.

e.g., O(n log n) is more efficient than O(n2). Reduce overhead in program.

e.g., pass pointers instead of whole objects. Parallel processing.


Parallel processing: Split task into multiple independent sub-tasks. Run sub-tasks concurrently, e.g., in multiple threads.

1 2

3 4

processingtime isreduced toabout 1/4


Splitting slow task into independent sub-tasks.


Summary

Soft real-time system Overload degrades system performance.

Overload handling methods adaptive quality of service multithreading task skipping

Explicit loop that does nothing wastes CPU resource.

Event activation saves CPU resource, if done properly.

In multithreading, secondary threads communicate with main thread using signals and slots.


Further Reading

Soft real-time systems: [Buttazzo2005].


Exercise

Write the functions Buffer::skipToFront() and Buffer::clear().


Reference

G. Buttazzo, G. Lipari, L. Abeni, M. Caccamo. Soft Real-Time Systems: Predictability vs. Efficiency, Springer, 2005.

Documents

Soft Real-Time Systems - NUS Computingcs3249/lecture/soft rts.pdf · CS3249 Soft Real-Time Systems 1 CS3249 CS3249 Soft Real-Time Systems 1 Soft Real-Time Systems Leow Wee Kheng CS3249