Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
Investigating Wavelet Investigating Wavelet Based Video Based Video
Conferencing SystemConferencing SystemTeam Members:Team Members:
oo AhtshamAhtsham Ali Ali oo Adnan Ahmed (in Newzealand for grad studies)Adnan Ahmed (in Newzealand for grad studies)oo Adil Nazir (starting MS at LUMS now)Adil Nazir (starting MS at LUMS now)oo WaseemWaseem KhanKhanoo WaseemWaseem KhanKhanoo Farah ParvaizFarah Parvaizoo Muhammad Hussnain (pursuing MS from Sweden)Muhammad Hussnain (pursuing MS from Sweden)oo Ahmed Majeed, Farooq Ali (are in USA for grad studies)Ahmed Majeed, Farooq Ali (are in USA for grad studies)
Supervisors:Supervisors:oo Dr. Shahid MasudDr. Shahid Masudoo Dr. Nadeem Dr. Nadeem A.KhanA.Khan
Lahore University of Management SciencesLahore University of Management Sciences
1
Investigating Wavelet Based Video Investigating Wavelet Based Video
ConferencingConferencing
� We are presenting a wavelet based video conferencing system.
� Openphone
� Peer to peer video conferencing system� Peer to peer video conferencing system
� Dirac
� Wavelet based video codec
2
OutlineOutline� Introduction
� Video Conferencing system
� Openphone
� Wavelet based video codec� Wavelet based video codec
� Dirac
� Embedding Process
� Improvements in Openphone
� Improvements in Dirac
� Scalability
� Summary
3
Video Conferencing Video Conferencing ––
System OverviewSystem Overview
4
Open Phone in Open Phone in Video Conferencing Video Conferencing
SystemSystem
� What is Openphone?
PWLIB� PWLIB
� SIP
� RTP
5
OutlineOutline� Introduction
� Video Conferencing system
� Openphone
� Wavelet based video codec� Wavelet based video codec
� Dirac
� Embedding Process
� Improvements in Openphone
� Improvements in Dirac
� Scalability
� Summary
6
Dirac: Wavelet Based Video Codec Dirac: Wavelet Based Video Codec
�� What is Dirac?What is Dirac?
�� Open Source technologyOpen Source technology
�� High Performance, simple and Modular Design Video High Performance, simple and Modular Design Video High Performance, simple and Modular Design Video High Performance, simple and Modular Design Video CodecCodec
�� Support Multiple Picture FormatsSupport Multiple Picture Formats
�� Good Subjective quality due to Psycho Visual property Good Subjective quality due to Psycho Visual property of waveletof wavelet
7
Dirac Encoder ArchitectureDirac Encoder Architecture
8
OutlineOutline� Introduction
� Video Conferencing system
� Openphone
� Wavelet based video codec� Wavelet based video codec
� Dirac
� Embedding Process
� Improvements in Openphone
� Improvements in Dirac
� Scalability
� Summary
9
Dirac Integration In OpenphoneDirac Integration In Openphone
�� Dirac version 0.9.1 has successfully embedded in Dirac version 0.9.1 has successfully embedded in Openphone.Openphone.
�� MethodsMethods
�� Source Code IntegrationSource Code Integration
�� Dirac Library PlugDirac Library Plug--inin
�� Support in OpenphoneSupport in Openphone
�� Registration of Dirac in OpenphoneRegistration of Dirac in Openphone
�� Dirac Bridging ClassesDirac Bridging Classes
10
OutlineOutline� Introduction
� Video Conferencing system
� Openphone
� Wavelet based video codec� Wavelet based video codec
� Dirac
� Embedding Process
� Improvements in Openphone
� Improvements in Dirac
� Scalability
� Summary
11
Buffer I/OBuffer I/O�� SenderSender SideSide
�� TakeTake videovideo fromfrom thethe cameracamera�� EncodesEncodes itit usingusing thethe DiracDirac encoderencoder andand directsdirects thethe outputoutput
bitbit--streamstream toto aa bufferbuffer..�� TheThe outputoutput bitbit--streamstream bufferbuffer isis thenthen handledhandled byby
OpenphoneOpenphone�� TransmitsTransmits thethe bitbit--streamstream overover networknetwork inin orderorder toto sendsend itit�� TransmitsTransmits thethe bitbit--streamstream overover networknetwork inin orderorder toto sendsend itit
toto thethe receiverreceiver sideside..�� ReceiverReceiver SideSide
�� OpenphoneOpenphone receivesreceives thethe bitbit--stream,stream,�� DecodesDecodes itit usingusing DiracDirac decoderdecoder andand�� DisplaysDisplays thethe videovideo onon thethe locallocal displaydisplay windowwindow..
12
Parameter passing from OpenphoneParameter passing from Openphone
�� WeWe havehave modifiedmodified DiracDirac encoderencoder // decoderdecoder toto getget thetheparameterparameter listlist fromfrom OpenphoneOpenphone insteadinstead ofof acquiringacquiring ititthroughthrough consoleconsole inputinput..
�� WidthWidth
�� HeightHeight�� HeightHeight
�� FrameFrame raterate
�� BitBit--raterate
�� VideoVideo formatformat
�� QualityQuality etcetc..
�� whereaswhereas DiracDirac encoder/decoderencoder/decoder usesuses thisthis listlist totoencode/decodeencode/decode asas appropriateappropriate..
13
BitBit--stream Fragmentationstream Fragmentation
� Divide the bit-stream into network supported size packets.
14
Size Up gradation (CIF)Size Up gradation (CIF)
�� We have added support for CIF size (352x288) as well We have added support for CIF size (352x288) as well as for QCIF (176x144) size.as for QCIF (176x144) size.
�� Practical issues involved were increase in encoding Practical issues involved were increase in encoding time and size of bittime and size of bit--stream.stream.time and size of bittime and size of bit--stream.stream.
�� Frame rate reduced to 2,3 fpsFrame rate reduced to 2,3 fps
15
Frame Rate OptimizationFrame Rate Optimization�� Dirac usually generates a bitDirac usually generates a bit--stream of 5 to 75 stream of 5 to 75
kilobyteskilobytes
�� modified the Openphone architecture to transmit the modified the Openphone architecture to transmit the whole bitwhole bit--stream of one frame before grabbing the stream of one frame before grabbing the next frame.next frame.next frame.next frame.
�� Increased the frame rate to 12 fps.Increased the frame rate to 12 fps.
16
INTER ModeINTER Mode�� Motion Estimation SupportMotion Estimation Support
�� BitBit--stream size reductionstream size reduction
�� Maintaining the starting and ending sequence of the Maintaining the starting and ending sequence of the frameframe
�� Maintaining the initialization and reMaintaining the initialization and re--initialization of initialization of Dirac encoder.Dirac encoder.
�� Also, some buffers are introduced for maintaining the Also, some buffers are introduced for maintaining the bitbit--stream before transmission.stream before transmission.
17
GUI SupportGUI Support�� The user can configure video conferencing options as The user can configure video conferencing options as
desireddesired
�� Dial / Listen Dial / Listen
�� Video Size Video Size
�� CIF or QCIFCIF or QCIF
�� QualityQuality
�� Encoding mode (INTER / INTRA)Encoding mode (INTER / INTRA)
18
GUI Main Window GUI Main Window
19
Crashing ProblemCrashing Problem
�� Video conferencing lasted only a couple of hoursVideo conferencing lasted only a couple of hours
�� Memory leakages and bugs have been removed from Memory leakages and bugs have been removed from the code by extensive debugging.the code by extensive debugging.the code by extensive debugging.the code by extensive debugging.
�� Tested for 50 hours.Tested for 50 hours.
20
Discrete Wavelet TransformDiscrete Wavelet Transform
�� Old DWT filterOld DWT filter
�� DD9_7DD9_7
�� 10 frames per second10 frames per second
�� Experimentations have been done over all the available Experimentations have been done over all the available �� Experimentations have been done over all the available Experimentations have been done over all the available wavelet filters wavelet filters
�� Replaced with HAAR0Replaced with HAAR0
�� Performance gain Performance gain
�� 25 %25 %
�� Frame rate increased to 15 frames per secondFrame rate increased to 15 frames per second
21
Dirac 1.0.2Dirac 1.0.2
�� Latest release of Dirac video codec version 1.0.2 has Latest release of Dirac video codec version 1.0.2 has been successfully implemented in Openphone.been successfully implemented in Openphone.
�� Frame rate is around 15 frames per secondFrame rate is around 15 frames per second
22
Client / Server ConfigurationsClient / Server Configurations
�� Quality and Size Quality and Size
�� To be set from both sidesTo be set from both sides
�� Now only Dialer has the option and whatever he Now only Dialer has the option and whatever he selects becomes the communication size for video selects becomes the communication size for video conferencing.conferencing.
23
Experiments and Results (Cont.)Experiments and Results (Cont.)� Comparison of Dirac and other video codec
Comparison of Openphone Codecs
45
50
Average PSNR (dB)
20
25
30
35
40
45
50 100 150 200 250 300 350 400 450 500
Bitrate
Average PSNR (dB)
H263-Intra
H261-Intra
Dirac-Intra
� Bit rate Vs PSNR of Openphone Codec
24
Experiments and Results (Cont.)Experiments and Results (Cont.)
� Subjective Quality Difference in H.261,H.263 and Dirac at 300 Kbps
H.261 H.263 Dirac
25
OutlineOutline� Introduction
� Video Conferencing system
� Openphone
� Wavelet based video codec� Wavelet based video codec
� Dirac
� Embedding Process
� Improvements in Openphone
� Improvements in Dirac
� Scalability
� Summary
26
Improvements in DiracImprovements in Dirac�� Improvement in Motion EstimationImprovement in Motion Estimation
�� 3D Recursive Search Algorithm3D Recursive Search Algorithm
�� Reduced Computational ComplexityReduced Computational Complexity
�� PerformancePerformance
�� ScalabilityScalability
�� Three Different Bit StreamsThree Different Bit Streams
�� Base Layer, Enhancement Layer1, Enhancement Base Layer, Enhancement Layer1, Enhancement Layer2Layer2
�� Three TypesThree Types
�� SNR, Spatial, TemporalSNR, Spatial, Temporal27
Improvements in Motion EstimationImprovements in Motion Estimation
Existing Motion Estimation Algorithm: Three StagesExisting Motion Estimation Algorithm: Three Stages
Motion vectors are found for each block of 1 pixel accuracy
using Hierarchical Motion using Hierarchical Motion Estimation
Vectors are refined to sub pixel accuracy
Mode Decision
28
Continued…Continued…� Process of down conversion in hierarchical motion
estimation
29
Continued…Continued…� In lowest level, two candidate vector lists are generated , one is
centered at zero motion vector and the other spatially predicted motion vector.
� In all other levels, there candidate vector lists are generated, two above and one guide vector list ,which is the best motion vector of the block at the immediate lower level.of the block at the immediate lower level.
� Sum of absolute difference is used as a cost function.
30
Continued…Continued…�� Complexity of Existing Complexity of Existing
AlgorithmAlgorithm
� Using Hierarchical Motion Estimation
Huge amount of SAD
MB in previous frame
BA
X
MB in current frame
Current MB
C
� Huge amount of SAD calculations, taking 80% of total encoding time
�� 3DRS Algorithm3DRS Algorithm
�� Spatial PredictorSpatial Predictor
�� Temporal PredictorTemporal Predictor
E
Current MB
31
Continued…Continued…�� Proposed ME Scheme Based on 3DRSProposed ME Scheme Based on 3DRS
32
Experiments and ResultsExperiments and ResultsThe Comparison of Motion Estimation (ME) Results for CIF with GOP Size = 36, B Frames = 18
SequenceAlgorithm File Size
(KB)
PSNR-Y
(dB)
Avg(SAD)
/Frame
%SAD
Reduction of
3DRS
%ME Time
Reduction of
3DRS
%Total Time
Reduction of 3DRS
Foreman
Original 1316 37.9834 122871
62% 61% 49%Foreman 62% 61% 49%3DRS 1281 37.7774 47285
Container
Original 825 35.8540 142651
59% 59% 46%3DRS 769 35.5432 58371
News
Original 791 39.8091 76566
63% 60% 43%3DRS 781 39.7951 27965
Highway
Original 7231 39.2548 107357
57% 55% 42%3DRS 6732 39.0925 46962
33
Experiments and ResultsExperiments and ResultsThe Comparison of Motion Estimation (ME) Results for CIF with GOP Size = 54, B Frames = 36
SequenceAlgorithm File Size
(KB)
PSNR-Y
(dB)
Avg(SAD)
/Frame
%SAD
Reduction of
3DRS
%ME Time
Reduction of
3DRS
%Total Time
Reduction of 3DRS
Foreman
Original 1129 37.3678 126498
62% 62% 49%Foreman 62% 62% 49%3DRS 1075 37.1229 48095
Container
Original 697 37.7331 79323
61% 59% 43%3DRS 635 37.7676 30872
News
Original 681 39.4961 7847964% 60% 44%3DRS 656 39.4300 28453
Highway
Original 6050 38.7711 11004656% 57% 43%3DRS 4692 38.5662 48299
34
ConclusionConclusion
� Implemented modified adaptive 3DRS algorithm by using temporal and spatial block motion vectors.
� Reduces the average number of SAD calculations per block by 50% to 65%.block by 50% to 65%.
� Reduces the motion estimation time from 50% to 60%.
� Reduces the total encoding time from 30% to 50%.� Average PSNR and bit stream size does not show
significant variations.
35
OutlineOutline� Introduction
� Video Conferencing system
� Openphone
� Wavelet based video codec� Wavelet based video codec
� Dirac
� Embedding Process
� Improvements in Openphone
� Improvements in Dirac
� Scalability
� Summary
36
ScalabilityScalability�� Three different bitThree different bit--streamsstreams
�� Base Layer (7Base Layer (7--16)16)
�� Enhancement Layer 1 (4Enhancement Layer 1 (4--6)6)
�� Enhancement Layer 2 (1Enhancement Layer 2 (1--3)3)
�� Three typesThree types
�� SNRSNR
�� SpatialSpatial
�� Temporal ( In progress)Temporal ( In progress)
37
Dirac Bit Stream SyntaxDirac Bit Stream Syntax
38
Dirac Bit Stream SyntaxDirac Bit Stream Syntax
39
Dirac Architecture For ScalabilityDirac Architecture For Scalability
40
Dirac Bit stream SplitterDirac Bit stream Splitter
41
Bit Stream Layers After SplittingBit Stream Layers After Splitting
� Bit Stream Layers After Splitting for Inter Frame
42
Dirac Bit stream JoinerDirac Bit stream Joiner
43
Bit Stream After joiningBit Stream After joining
� Bit Stream After Joining for Inter Frame
44
Scalability In OpenphoneScalability In Openphone�� Three different bitThree different bit--streamsstreams
�� Base Layer (7Base Layer (7--16)16)
�� Enhancement Layer 1 (4Enhancement Layer 1 (4--6)6)
�� Enhancement Layer 2 (1Enhancement Layer 2 (1--3 )3 )
�� Three typesThree types
�� SpatialSpatial
�� SNRSNR
�� Temporal (Not completed)Temporal (Not completed)
45
Scalability in OpenphoneScalability in Openphone•• User receive video depending on his capabilities.User receive video depending on his capabilities.
46
OutlineOutline� Introduction
� Video Conferencing system
� Openphone
� Wavelet based video codec� Wavelet based video codec
� Dirac
� Embedding Process
� Improvements in Openphone
� Improvements in Dirac
� Scalability
� Summary
47
Tasks CompletedTasks Completed� Understanding the code of different modules of Dirac
video codec� Documenting the algorithmic details� Improvement in encoding time of Dirac
� 3DRS implementation, Comp complexity reduction , wavelet transform improvementwavelet transform improvement
� Implementing SNR and Spatial scalability in Dirac� Integrating Dirac in Open phone� Implementing Multi-streaming in open phone� Providing Graphical User Interface support� Support up till CIF format� Improvement Dirac to real time
� Extended frame rate from 3 to 1548
Ongoing and Future WorkOngoing and Future Work
� Addressing other modules regarding performance perspective
� Implementing Wavelet coefficient parent child relationship for scalable video performance improvementimprovement
� Implementing Temporal scalability
� Support extended for HDTV.
� Multi-user support for scalability in open phone
� Association of audio stream
� Improvement of GUI according to new developments
� Chat, File sharing49