1. Kurento: a media server architecture and API Live video technologyLuis Lopez lulop@kurento.org

2. Who am I? Associate Prof. Computer Science Multimedia protocols and systems15 years of experience in multimedia applications and protocols research Director of the FUN-LAB groupOpen Source Software enthusiast and contributorCoordinating the Kurento.org project: a media server architecture and API http://www.kurento.org2 3. Whats Kurento? Kurento Esperanto translation of streamEsperanto philosophy Openness Simplicity InteroperabilityKurento Open source development framework for creating rich real-time multimedia applications.: a media server architecture and API http://www.kurento.org3 4. Kurentos history 2004March 2013 Different research efforts from private and public organizations in the area of multimedia Kurento receives 4M by European Commission through different research initiatives2012December 2013 The Kurento stack goes fully open source LGPL v2.1 Kurento first release available: a media server architecture and API http://www.kurento.org4 5. Why WebRTC? WebRTC is a movement by Tsahi Levent-LeviAre RTCs new? Skype, H.323, Facetime, WebExWhy WebRTC is raising such expectations Millions of WWW developersWWW dev. communityRTC/IMS dev. communityWebRTC WebRTC = Multimedia RTC is just another feature of your WWW application : a media server architecture and API http://www.kurento.org5 6. Why Kurento? WWW RTC developer experience Before WebRTC Developing the client sideAfter WebRTCBeginNext natural stepEnd Begin Unified APIs Standards FOSS MultiplatformDeveloping the infrastructure side: a media server architecture and API http://www.kurento.orgEnd Unified APIs Standards FOSS MultiplatformBeginEnd Unified APIs Standards FOSS Multiplatform6 7. Kurento media framework components Low level media capabilitiesSignaling and control capabilities SIP, HTTP, RESFul, etc. Java Spring + JBoss + Mobicents SIP ServletTransport, encode/decode, transcode, CV, mix, etcC/C++ C++ Media Server + Gstreamer pluginsKurento Media ServerKurento Application ServerKurento Android SDKKurento WWW SDKRTC library for Android Java and C Multicodec support: a media server architecture and API http://www.kurento.orgHTML5 multimedia development Video tag, WebRTC Javascript7 8. Kurento Application Server: extending the WWW development model Signaling request: I want this mediaHTTP request: I want this resourceProcess WWW request - DDBB access - Transactions - Security tools - Etc.HTTP response: The resourceIntuition behind traditional WWW Applications (Servlets, ASP, PHP, Rails, etc.): a media server architecture and API http://www.kurento.orgProcess media request- Media API -DDBB access Transactions Security tools Etc. Signaling response: The media is hereIntuition behind Kurento development APIs: Multimedia RTC is just another feature of your application 8 9. Media API: media elements and media pipelines Media Element Provides a specific media functionalitySend/receive media Process media Transform mediaThe Media API provides the capability of creating media pipelines by joining media elements of the toolboxThe Media API provides a toolbox of media elements ready to be used. New media elements can be addedMedia ElementPlayerEndPointSinkSRC: a media server architecture and API http://www.kurento.orgWebRtcEndPointFilter SinkSRCSRCthe desired media functionality.Building blockSink Media pipeline Chain of media elements implementing9 10. Application architectureOther systems and databasesSignaling and WWW trafficMediaMediaSignaling and WWW trafficHTTP ServletSIP ServletWeb service sDD.BB.JMSJava EE compatible container Specific application logic Media API Decode VideoAugmented RealityVideo Playing and RecordingComputer VisionEncode VideoKurento Media Server Infrastructure: a media server architecture and API http://www.kurento.org10 11. Possible use cases: just integrate with Java EE and GStreamer Verticals E-Health Kurento + HAPI (http://hl7api.sourceforge.net/) P2D video conferences as Electronic Health Records Smart cities Kurento + NGSI + OpenCV + Google Maps City crowds movement tracking Traffic density tracking Telco infrastructures Kurento + Mobicents IMS application server B2B & B2C WWW RTC Kurento + CRM APIs Enriched video conferencing with customer personal data Kurento + ESB Billing, video event processing, physical security, etc.: a media server architecture and API http://www.kurento.org11 12. Application example: requirementsKurento Media Server Infrastructure: a media server architecture and API http://www.kurento.org12 13. Application example: code @WebRtcMediaService(name = "MyWebRtcService", path = "/pathToService") public class MyWebRtcService implements WebRtcMediaHandler { public void onMediaRequest(WebRtcMediaRequest request) { //I can authenticate using any of the Java EE available mechanisms MediaPipelineFactory mpf = request.getMediaPipelineFactory(); MediaPipeline mp = mpf.createMediaPipeline(); //I could decide the type of processing connecting to a DDBB JackVaderFilter filter = mp.newFilter().withType(JackVaderFilter.class).build(); RecorderEndpoint recorder = mp.newRecorderEndpoint().withUri("file:///myFile.webm"); filter.connect(recorder); HttpEndpoint httpEndpoint = mp.newHttpEndpoint().build(); filter.connect(filter); //I could connect only audio or video separately request.startMedia(filter, filter); }To fileMedia SinkRecorderEndpointMedia SinkSinkVideo tag or CDNSRCMedia SinkHttpEndpoint : a media server architecture and API JackVaderFilter http://www.kurento.orgTo NetworkMedia SourceFrom network WebRtcEndpoint13 14. Application example: result: a media server architecture and API http://www.kurento.org14 15. Media element toolbox TransportRepositoryGroup communications WebRtcEndpoint RtpEndpoint HttpEndpoint PlayerEndPoint RecorderEndPoint MainMixer GridMixer RoundRobinMixerFilters FaceRecognitionFilter (events) JackVaderFilter QR/Barcode detector PlateRecognitionFilter (events) ColorTrackingFilter (events)And growing Available as part of: a media server architecture and API http://www.kurento.org15 16. Media elements: WebRtcEndpoint Full implementation of the RTCWeb protocol stack SRTP ICE DTLS Allow sending and receiving WebRTC flows at the media server infrastructureMedia SourceWebRTC trafficMedia SinkWebRTC trafficWebRtcEndpoint: a media server architecture and API http://www.kurento.org16 17. Media elements: HttpEndpoint Media downloading compatible with the HTML5 video tag WebM (Chrome, Firefox) MP4 (Chrome, Firefox, IE, Safari) Media uploading compatible with HTML file input tag Multipart supportMedia SinkHTTP WebM/MP4 HttpEndpoint: a media server architecture and API http://www.kurento.org17 18. Media elements: RtpEndpoint Full-duplex RTP multimedia exchange H.264 H.263 VP8 Many different audio codecs supportedMedia SourceRTP trafficMedia SinkRTP trafficRtpEndpoint: a media server architecture and API http://www.kurento.org18 19. Media elements: UriEndpoints PayerEndpoint Play media from file or URL Support for most popular formats RecorderEndpoint Record media to file or URL WebM MP4 Media SinkRecorderEndpointMedia SourcePlayerEndpoint : a media server architecture and API http://www.kurento.org19 20. Media elements: Mixers (in progress) Make possible group communications ForwardingMixer One-to-many replication of flows A source can be assigned to any of the sinks Multiple sources supported MainMixer Mixes media A source can be assigned to a combination of sinks Multiple sources supportedMedia Source Media Sink Media Source Media SinkMixerMedia Sink: a media server architecture and API http://www.kurento.orgMedia Source 20 21. Filters Seamless integration into OpenCV Face recognition Augmented reality Subtitle adding Color manipulation QR detection People counter Plate recognition Etc.Filter SRC Sink: a media server architecture and API http://www.kurento.org21 22. Filter with events Filters can provide events to the application Events are generated at the media server Events can be propagated to the client app Code example: MediaPipeline mp = mpf.create(); PlayerEndPoint playerEndPoint = mp.newPlayerEndPoint( "https://ci.kurento.com/video/barcodes.webm").build(); ZBarFilter filter = mp.newZBarFilter().build(); playerEndPoint.connect(filter);EventsSRCMedia SourceSinkfilter.addCodeFoundDataListener(new MediaEventListener() { @Override public void onEvent(CodeFoundEvent event) { session.publishEvent(new ContentEvent(event.getType(), event.getValue())); ...PlayerEndpoint ZBarFilter : a media server architecture and API http://www.kurento.org22 23. The magic of pipelines: Transparent media adaptation Agnostic media adaptor Acts every time a source is connected to a sink Adapts media formats as required by the involved media elements 100% transparent for the application developer An agnostic media adaptor is hidden behind every connection making compatible element formats transparentlyVP8H.264: a media server architecture and API http://www.kurento.orgSRCMedia Element SinkSRCSinkMedia Element23 24. Complex examples: Heterogeneous group communications Media SourceMedia SinkMedia SinkRecorderEndpointWebRtcEndpoint Media SourceMixerMedia SinkWebRtcEndpoint Media SourceSRCMedia SinkSinkFilterRtpEndpoint : a media server architecture and API http://www.kurento.org24 25. Complex examples: WebRTC to HTTP Media SourceMedia SinkMedia SinkRecorderEndpointMixerWebRtcEndpoint Media SinkHttpEndpointMixerMedia SinkHttpEndpointMedia SinkHttpEndpoint : a media server architecture and API http://www.kurento.org 26. Complex examples: WebRTC Playing Media SourceMedia SinkMedia SinkRecorderEndpointMixerWebRtcEndpoint Media SinkFilterHttpEndpointSinkSRCMedia SourcePlayerEndpoint : a media server architecture and API http://www.kurento.org26 27. Kurentos Roadmap Q1 2014 Installer and documentation Group communications Super scalable media repositoryQ4 2014 Kurento cloud infrastructure Hardware accelerationQ2 20142015 WebRTC support on Android SDK Kurento iOS SDK Javascript Media API Elastric scalable Kurento PaaS platform SDN Kurento QoS cloud connectivity manager IMS/Telco infrastructure integration 3D/Kinect multimedia support: a media server architecture and API http://www.kurento.org27 28. Upcoming opportunities for developers FI-WARE is the platform of the FI-PPP initiative European Commission FP7 Kurento is providing multimedia capabilities to FI-WARE FI-WARE is giving 800K in prices to developers You can opt to them developing on top of Kurento http://www.fi-ware.eu/challenges/ FI-PPP is providing 100M in funding to SMEs and entrepreneurs http://www.fi-ppp.eu/how-to-participate/: a media server architecture and API http://www.kurento.org28 29. Collaborations welcome http://www.github.com/kurentoThank you very much for your attention Complains, suggestions and comments can be sent to: Luis Lpez lulop@kurento.org: a media server architecture and API http://www.kurento.org29