MP design and implementation

CS414: Multimedia SystemInstructor: Klara Nahrstedt

April 13, 2012

Covered Aspects of Multimedia in CS414 MP

Image/VideoCapture

MediaServerStorage

Transmission

CompressionProcessing

Audio/VideoPresentationPlaybackAudio/Video

Perception/ Playback

Audio InformationRepresentation

Transmission

AudioCapture

A/V Playback

Image/Video InformationRepresentation

MP1: AV Capture and Playback

Image/VideoCapture

Compression/ Processing

Audio InformationRepresentation

AudioCapture

Audio/Video Perception/ Playback

MP1: Video Capture and Store in a File

Src

Sink Src Sink Src

Video Source Video Filter Media Encoder

Sink Src

Video Muxer

Sink

Video Sink

final Element videosrc = ElementFactory.make("v4l2src", "source");

final Element videofilter = ElementFactory.make("capsfilter", "flt");videofilter.setCaps(Caps.fromString("video/x-raw-yuv, width=640, height=480, framerate=10/1")); final Element formatConverter = ElementFactory.make("ffmpegcolorspace", "formatConverter"); final Element encoder = ElementFactory.make("ffenc_mpeg4", "encoder");final Element muxer = ElementFactory.make("fmux_mpeg", "muxer");final FileSink filesink = (FileSink) ElementFactory.make("filesink", "filesink");filesink.setLocation("capture01.avi");

Snippet

pipe.addMany(videosrc,formatConverter, videofilter, encoder, filesink);Element.linkMany(videosrc, formatConverter, videofilter, encoder, filesink);pipe.setState(State.PLAYING);Gst.main();pipe.setState(State.NULL);

START

Video Capture and Store in a Queue

Video Source Video Filter

final Element videosrc = ElementFactory.make("v4l2src", "source");

final Element videofilter = ElementFactory.make("capsfilter", "flt");videofilter.setCaps(Caps.fromString(Caps.fromString(String.format("video/x-raw-yuv, width=%s, height=%s" + ", bpp=24, depth=16, framerate=%s/1",WIDTH+"", HEIGTH+"", CAPTURERATE+""))); videorate = ElementFactory.make("videorate", "videorate");ratefilter = ElementFactory.make("capsfilter", “rateFilter");ratefilter.setCaps(Caps.fromString(String.format("video/x-raw-yuv, framerate=%s/1”,FRAMERATE+""))); final Element encoder = ElementFactory.make(” theoraenc", ”H264");final Element muxer = ElementFactory.make(”oggmux", "muxer");final AppSink appsink = (FileSink) ElementFactory.make(”appsink", ”appsink");

Snippet

Src

Sink Src

Media Encoder Video Muxer

Sink Src Sink Src Sink

App SinkVideo Rate Rate Filter

Sink Src Sink Src

Gstreamer-java AppSink

Sink

App Sink Element

A callback function is used to get the frames from the AppSink Element

appsink.set("emit-signals", true);appsink.setSync(false);appsink.connec(new AppSink.NEW_BUFFER(){

public void newBuffer(AppSink as){Buffer buffer = as.getLastBuffer();ByteBuffer buf = buffer.getByteBuffer();byte[] b = new byte[buf.remaining()];buf.get(b);int timestamp =

(int)buffer.getTimestamp().toMillis();ApplicationPacket packet = new

ApplicationPacket(b, timestamp)Queue.enQueue(packet);

}});

QueuePacket

ArrayBlockingQueue

MP1: Video Playback from a File

Pipeline pipe = new Pipeline(”Decode pipeline");Element videosrc = ElementFactory.make("filesrc", ”filesrc");src.set("location", "Avatar.avi");DecodeBin2 decodeBin = new DecodeBin2("Decode Bin");Element decodeQueue = ElementFactory.make("queue", "Decode Queue");final VideoComponent videoComponent = new VideoComponent();

pipe.addMany(videosrc, decodeQueue, decodeBin);Element.linkMany(videosrc, decodeQueue, decodeBin);

decodeBin.connect(new DecodeBin2.NEW_DECODE_PAD()){public void newDecodePad (DecodeBin2 elem, Pad pad,

boolean last){Caps cap = pad.getCaps();Structure struct = caps.getStructure(0);if(struct.getName().startsWith(“video/”){

pad.link(videoComponent.getElement.getStaticPad(“Sink”));} } }

Src

Sink Src

Media Source Media Decoder

Sink Src

Media DeMuxer

Sink

Media Sink

Sink Src

DecodeBin

Video Playback from a Queue

Src

Sink Src Sink Src

App Source Video Filter Media Encoder

Sink Src

Video Muxer

Sink

Video Sink

final AppSrc appsrc = (AppSrc) ElementFactory.make("appsrc", ”appsrc");appsrc.setCaps(Caps.fromString(Caps.fromString(String.format("video/x-raw-yuv, width=%s, height=%s" + ", bpp=24, depth=16, framerate=%s/1",WIDTH+"", HEIGTH+"", CAPTURERATE+"")));

final Element decodeBin = ElementFactory.make("decodebin2","decode");final VideoComponent videoComponent = new VideoComponent();

pipe.addMany(appsrc, decodeBin);Element.linkMany(appsrc, decodeBin);

decodeBin.connect(new DecodeBin2.NEW_DECODE_PAD()){public void newDecodePad (DecodeBin2 elem, Pad pad, boolean last){

Caps cap = pad.getCaps();Structure struct = caps.getStructure(0);if(struct.getName().startsWith(“video/”){

pad.link(videoComponent.getElement.getStaticPad(“Sink”));}

}}

Snippet

Sink Src

Decode Bin

Gstreamer-java AppSrc

Source

App Source Element

A callback function is used to put the frames into the AppSrc Element

appsrc.set("emit-signals", true);appsrc.setSync(false);appsrc.connect(new AppSrc.NEED_DATA() { public void needData(AppSrc as, int size) {

ApplicationPacket packet = Queue.dequeue();

while (packet == null) {System.out.println("Null

data. trying again");packet = Queue.dequeue();

}byte[] data = packet.getData();Buffer buffer = new

Buffer(data.length);buffer.getByteBuffer().put(data);as.pushBuffer(buffer);

}{);

Queue ArrayBlockingQueuePacket

MP2 Design Space

Image/VideoCapture

MediaServerStorage

Transmission

CompressionProcessing

Audio/VideoPresentationPlaybackAudio/Video

Perception/ Playback

Audio InformationRepresentation

Transmission

AudioCapture

A/V Playback

Image/Video InformationRepresentation

MP2: Audio Video Streaming

Server Client

• Server– Capture Audio and Video at a fixed rate– Video: 30 fps, Audio: 8000Hz

• Client– Requests for Video and/or Audio– Works in Two modes: Active Mode and

Passive Mode

Control Channel

Data Channel

• Active Mode: Media type: Audio, VideoVideo Rate: 15 to 25 fpsAudio Rate: 8000HzVideo Resolution: 640X480

• Passive Mode: Media type: VideoVideo Rate: 10 fpsVideo Resolution: 320X240

Video Data Flow Architecture: Server

RTP

Pack

et

Rate Control

Network

Video Source Video Filter Media Encoder Video Muxer App Sink

Payload

Media Type Timestamp RTP headerID

UDP Packet Payload

Segment

UDP Communication: Server

Payload

Media Type Timestamp

RTP Packet

ID

UDP Packet Payloads

PID SEQ SIZE PAYLOAD SIZE

UDP Packet Payload

System.arraycopy(pid_byte, 0, senddata, 0, pid_byte.length);System.arraycopy(seq_byte, 0, senddata, 4, seq_byte.length);System.arraycopy(size_byte, 0, senddata, 8, size_byte.length);

While(true){int remaining = (databyte.length - from);if(remaining < 0 ) break;else if(remaining > UDP_PAYLOAD_SIZE){

byte[] payload_size_byte = toBytes(UDP_PAYLOAD_SIZE); System.arraycopy(payload_size_byte, 0, senddata, 12, payload_size_byte.length); System.arraycopy(databyte, from, senddata, 16, UDP_PAYLOAD_SIZE);

}else{ byte[] payload_size_byte = toBytes(remaining); System.arraycopy(payload_size_byte, 0, senddata, 12, payload_size_byte.length); System.arraycopy(databyte, from, senddata, 16, remaining); }}

sendData(7777, ”IP_Address", senddata); seq++;from += UDP_PACKET_SIZE;

DatagramSocketDatagramPacket

UDP Packet Header

Video Data Flow Architecture: Client

Network

RTP

Pack

et

App SourceDecode Bin2Video Sink

Synchronization

Payload

Media Type Timestamp RTP headerID

UDP Packet Payload

Merge

UDP Communication: Client

Payload

Media Type Timestamp

RTP Packet

ID

PID SEQ SIZE PAYLOAD SIZE

UDP Packet Payload

System.arraycopy(recv_data, 0, pid_byte, 0, 4);System.arraycopy(recv_data, 4, seq_byte, 0, 4); System.arraycopy(recv_data, 8, size_byte, 0, 4);System.arraycopy(recv_data, 12, payload_size_byte, 0, 4);

Hash

Table

index w

ith

PID

SIZE

if(!received_frame.containsKey(pid)){ ApplicationPacket app_frame = new ApplicationPacket(); app_frame.frame = new byte[size]; app_frame.array_size = size; System.arraycopy(recv_data, 16, app_frame.frame, seq*UDP_PACKET_SIZE, payload_size); received_frame.put(pid, app_frame); app_frame.array_size -= payload_size;}else{ ApplicationFrame app_frame = (ApplicationFrame) received_frame.get(pid); System.arraycopy(recv_data, 16, app_frame.frame, seq*UDP_PACKET_SIZE, payload_size); app_frame.array_size -= payload_size; received_frame.put(pid, app_frame);}if(app_frame.array_size == 0)finish_frame(pid);

received_frame

Video Data Flow Architecture

RTP

Pack

et

Rate Control

Network

Video Source Video Filter Media Encoder Video Muxer App Sink

RTP

Pack

et

App SourceDecode Bin2Video Sink

Synchronization

Payload

Media Type Timestamp RTP headerID

UDP Packet Payload

Segment

Merge

Rate Control: QoS Enforcement

• You can build leaky bucket• You can implement token bucket• You can use gstreamer rate control• You can implement your own

method

Sleep (100) => 10 fps

You can simply read using a loopWhile(true){ SendPacket(); Thread.sleep(100);}

Resource Admission: Client

• Client– Available Application Bandwidth ABN

– Application Frame Size, MN = ?

– Application Frame Rate, RN = ?

– Audio Bandwidth = 8000 * 16– Request Bandwidth

• Active Mode: BN = (MN * RN) + Audio Bandwidth

• Passive Mode: BN = (MN * RN)

resource.txt

Optimistic Allocation

8000Hz Audio Signal

How to define RN for active mode?

Show your computation on the Screen

fps: 10

Resource Admission: Client

• Client– Available Application Bandwidth ABN

– Application Frame Size, MN = ?

– Application Frame Rate, RN = ?

– Request Bandwidth• Active Mode: BN = (MN * RN) + Audio

Bandwidth

resource.txt

Optimistic Allocation

Show your computation on the Screen

ABN = (MN*RN) + Audio Bandwidthif RN > 25, RN = 25if RN < 15, REJECTelse RN

Resource Admission: Client

• Server– Available Application Bandwidth ABN

– Used Bandwidth for Server 1 B1

– Available Bandwidth ABN = ABN –B1

– Admission is successful for B2 if B2 <= ABN

resource.txt

Server 1 60 Kbps

Server 2 35 KBps

Resource Table

Using RtpBin @ ServerVideo Source Video Filter

Tee Video Scale Scale Filter

Video Rate Rate Filter Video Encoder RTP Payload

RTP Bin

UDP Sink

Queue

UDP SinkUDP Source

RTCP

RTCP

RtpBin combines • RtpSession• RtpSsrcDemux• RtpPtDesmux• RtpJitterBuffer

Using RtpBin @ Client

Video SinkVideo DecoderRTP DePayload

RTP Bin

UDP Source

UDP SourceUDP Sink

RTCP

RTCP

rtpBin.connect(new Element.PAD_ADDED() {public void padAdded(Element element, Pad pad) {

if (pad.getCaps().toString().contains("video")) {

//LINK TO VIDEO SINK

} else if (pad.getCaps().toString().contains("audio")) {

// LINK TO AUDIO SINK }

}});

Internals of RtpBin

http://gstreamer.freedesktop.org/documentation/rtp.html

MP3: Multi-view Surveillance System

Window for server 1

Window for server 2

• Server– Capture Audio and Video at a fixed rate– Video: 30 fps, Audio: 8000Hz

• Client– Requests for Video and/or Audio– Works in Two modes: Active Mode and

Passive Mode

• Active Mode: Media type: Audio, VideoVideo Rate: 15 to 25 fpsAudio Rate: 8000HzVideo Resolution: 640X480

• Passive Mode: Media type: VideoVideo Rate: 10 fpsVideo Resolution: 320X240

Additional Features for MP3?

• It is highly recommended that you build an additional feature in MP3 to SURPRISE US!

• Bonus Point: 20• Bonus point will be awarded based on the

amount of surprise!