Upload
luis-lopez
View
2.641
Download
9
Embed Size (px)
DESCRIPTION
This presentation contains a tutorial devoted to showing how Java developers can create rich multimedia applications with Kurento. Java developers will find natural Kurento development model, which is based on standard Java EE technologies and is inspired on the WWW Servlet model. If you have ever developed a Web application, you may be familiar with this scheme. At the browser, HTML and JavaScript code is in charge of user interaction and generates HTTP requests to the server. This code is usually programmed with the help of APIs such as jQuery, DOM, XHR or others. Upon reception, HTTP requests are processed by some kind of server side technology (e. g.. PHP, Java, Ruby, etc.) using service APIs providing features such as DB access, communications, transactions, XML parsing, and others. As a result, an HTTP response is issued and sent back to the client. Following this scheme, both server and client side APIs are just capabilities simplifying developer work and providing abstractions for programming faster and more efficiently. Kurento technologies adapt to the Web development model so that, from a programmer perspective, Kurento can be seen just as an additional set of APIs. Developers does not need to learn novel programming schemes and can reuse all their knowledge and previous background on WWW application development. When you need multimedia, just use Kurento APIs. For the rest, use your preferred APIs or reuse previous code. Kurento APIs have been designed for simplicity and Web developers will find them familiar and intuitive. Most of the low level details related to codecs, formats, protocols, profiles and containers are abstracted by the framework. Programmers just concentrate on specifying the sequence of processing steps that they want to execute on the media flows.
Citation preview
www.kurento.org
Real-time Multimedia Stream ProcessingDeveloping rich multimedia applications with KurentoDeveloping rich multimedia applications
with Kurento: a tutorial for Java developers
[email protected]://www.kurento.org
www.kurento.org
Multimedia infrastructures for the Future Internet
Media ishere Media got
there
Media gotthere
AnalyzeTransform
Store
Transport
EnrichAugment
Adapt
SensorsContext
EventsMedia is
here
Traditional infrastructures
Future Internet infrastructures
2
www.kurento.org
Enrich, augment, adapt, analyze, transform, store: what’s the problem?
Complexity
3
www.kurento.org
Future Internet
Multimedia Infrastructure
Simple Development
APIs
Kurento: the equation
4
www.kurento.org
• Interoperable media exchange (multiplatform/multiprotocol)• WebRTC, RTP, HTTP (video tag), etc.
• Process media (Computer vision, augmented reality, media indexing, etc.)• Media and metadata recording and recovery• Transform and adapt media (H.264, H.263, VP8, Ogg, and others)• Media routing and mixing• Etc.
Multimedia infrastructure
• REST API• JavaScript API• Java API
APIs
• LGPL 2.1
Is distributed through a flexible FOSS license
What’s Kurento
5
www.kurento.org
Kurento Media Server (KMS): the nucleus of Kurento
• KMS is a middleware for media streams– Receives the stream– Process the stream– Issues the stream
SendReceiveAnalyze
AugmentEnrich
TransformTranscode
RecordProcess
Replicate
MediaSource
MediaSink
KMS
6
www.kurento.org
The Media API: The API for accessing KMS capabilities
SendReceiveAnalyze
AugmentEnrich
TransformTranscode
RecordProcess
Replicate
MediaSource
MediaSink
KMS
JavaMedia API
JavaScriptMedia API
RESTAPI
Applications define the processingof streams getting through KMS
7
www.kurento.org
Media API: Media Elements and Media Pipelines
Media Element
• Provides a specific media functionality
› Send/receive media
› Process media
› Transform media
• Exchange media through
› Sources
› Sinks
Media pipeline
• Chain of media elements implementing the desired media logic.
• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox
Media Element
Sink
SRC
8
www.kurento.org
Media API: trivial example (Java)MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build();
HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();
playerEndpoint.connect(httpEndpoint);
httpEndpoint.getUrl(); //URL where the media is made available
Media Pipeline (inside KMS)
HttpGetEndpoint
Media fromfile or URI
HTTP mediastreaming Si
nk
SRC
PlayerEndpoint
9
www.kurento.org
Is that enough? Think about the WWW development model
Process WWW request- DDBB access- Authentication- XML processing- Etc.
HTTP request: I want this resource
HTTP response:The resource
• Intuition behind the WWW model – Client asks what it wants
• GET /this_resource HTTP/1.1
– Server side APIs execute the associated processing
You develop this using …• Servlet• JSP• ASP• PHP• Ruby on Rails• CGI• Etc.
10
www.kurento.org
We need an equivalent model: the role of the Signaling Plane
• We need “something” with the capability of negotiating what, when and how media is exchanged
• A protocol for “saying”– I want “this media” …
• Identification of the media to exchange– File in hard-drive, IP camera, user, etc.
– in “this way” …• Identification of the processing of media
– Augmented, analyzed, etc.
– with “this format” …• Quality
– Codec, screen-size, frame-rate, etc.
– at “this moment”• Stream control
– Play, stop, start, pause, etc.
11
www.kurento.org
Don’t get it? think about WWW development again …
Process WWW request- DDBB access- Authentication- XML processing- Etc.
HTTP request: I want this resource
HTTP response:The resource
Process media request- Media API- DDBB access- Authentication- XML processing- Etc.
Signaling request: I want this media
Signaling response:The media is here
This is how you create WWW applications
(Servlets, ASP, PHP, Rails, etc.)
This is how you createKurento enabled
applications:Multimedia RTC is just
another feature of your application
You, as a programmer,
create this
12
www.kurento.org
Abstracting the signaling plane: The Content Handler
KMS
Sink
SRC
Sink
SRC
Sink
SRC
Sink
Media API
REST API (Open API protocol) The Content HandlerEquivalent to a Servlet/ASP/PHP scriptExposes APIs for specifying:- when receiving “this request”…- … execute “this logic”Developer can use the media API
Code building the media pipeline and executing the application logic the
developer wants
13
www.kurento.org
Kurento Application Server: The container of Handlers
• Is an extension of Java EE technologies.• Compatible with all Java EE Servlet
containers• Hold Handlers
– Your specific application code• Receives signaling requests:
– I want “this media” in this way …• Dispatches request to the appropriate
handler– @annotations based mapping
• Generate an answer showing “where and how to find the media”– URL where media is located– Negotiated SDP
Java EE compatible container
HTTP Servlet
SIP Servlet
Web service
s
Kurento REST API
Specific handler implementations
Media API DD.BB.
Kurento Application Server (KAS)
Other java APIs.
14
www.kurento.org
Kurento Architecture: putting it all together
Kurento Media Server (KMS)
ReceiveVideo
AugmentedReality
SendVideo
Computer Vision
Video Playingand Recording
Java EE compatible container
HTTP Servlet
SIP Servlet
Web service
s
Kurento REST API
Specific handler implementations
Signaling and WWW traffic
Med
ia
Med
ia
Signaling and WWW traffic
Media API DD.BB.
Kurento Application Server (KAS)
Other java APIs.
15
www.kurento.org
Application execution flowClientCode
ApplicationServer (KAS)
MediaServer (KMS)
I want this media in this way …(JSON-RPC request)
Commands requestingthe creation of a pipeline
What you want is here …(JSON-RPC answer)
Media negotiation
phase
Media exchange
phase
1
2
Specific applicationlogic at the server-side
(Content Handler)
Mediapipelinecreation
Media exchange between client and server
16
www.kurento.org
Content Handler: trivial example//Specify the type of service provided by this handler: Player, Recorder, WebRTC, RTP, etc.@HttpPlayerService(path = "/player”) //Mapping of handler specified in pathpublic class MyPlayerHandler extends HttpPlayerHandler {
@Overridepublic void onContentRequest(HttpPlayerSession contentSession) {
//Thie client wants the media this handler provides//Create the pipeline for providing the media
}
@Overridepublic void onContentStarted(HttpPlayerSession contentSession) {
//Media started flowing, you can execute additional actions}
@OverridePublic void onSessionTerminated(HttpPlayerSession contentSenssion){
//Media exchange termianted, you can collect your resources}
17
www.kurento.org
Let’s develop with Kurento
• What you need– A Kurento instance
• You can install your own Kurento instance• You can launch a Kurento instance at the FI-LAB (FI-WARE project)
– http://lab.fi-ware.org
• Getting help– Kurento web site
• http://www.kurento.org
– Kurento mailing list• https://groups.google.com/forum/#!forum/kurento
– Twitter• @kurentoms
18
www.kurento.org
Kurento Hello World: Playing a file with an HTML5 client
Media Pipeline
HttpGetEndpoint
Media fromfile or URI
HTTP mediastreaming Si
nk
SRC
PlayerEndpoint
Media API
REST API (Open API protocol)
Handler codeI want “this media”
Media is “at this URL”
19
www.kurento.org
Playing a file with an HTML5 client: Handler code
@HttpPlayerService(path = "/player”)public class MyPlayerHandler extends HttpPlayerHandler {
@Overridepublic void onContentRequest(HttpPlayerSession contentSession) throws Exception {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint("http://media.w3.org/2010/05/sintel/trailer.webm").build();contentSession.setAttribute("player", playerEndpoint);HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();playerEndpoint.connect(httpEndpoint);contentSession.start(httpEndpoint);
}
@Overridepublic void onContentStarted(HttpPlayerSession contentSession) {
PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAttribute("player");playerEndpoint.play();
}}
Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerHandler.java
20
www.kurento.org
Playing a file with an HTML5 client: client code<!DOCTYPE html>
<html><head>
<script src="./js/kws-content-api.js"></script><script>
var conn;function start() {
var options = {remoteVideoTag : "remoteVideo"
};conn = new kwsContentApi.KwsContentPlayer("./player", options);
}function terminate() {
conn.terminate();}
</script></head><body>
<button onclick="start();">Start</button><button onclick="terminate();">Terminate</button><br /><video id="remoteVideo" autoplay></video>
</body></html>
Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/webapp/player.html
21
www.kurento.org
Playing a file with an HTML5 client: See the example working
https://www.youtube.com/watch?v=n5BQlhYgGSo
22
www.kurento.org
Media Pipeline
Adding Computer Vision
HttpGetEndpointMediafrom
file or URI
HTTP mediastreaming Si
nk
SRC
PlayerEndpoint
SRC
Sink
JackVaderFilter
Media API
REST API (Open API protocol)
Handler codeI want “this media”
Media is“at this URL”
23
www.kurento.org
Adding Computer Vision: Handler code@HttpPlayerService(path = "/playerWithFilter”)public class MyPlayerHandler extends HttpPlayerHandler {
@Overridepublic void onContentRequest(HttpPlayerSession contentSession) throws Exception {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint("http://media.w3.org/2010/05/sintel/trailer.webm").build();contentSession.setAttribute("player", playerEndpoint);JackVaderFilter filter = mp.newJackVaderFilter().build();HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();filter.connect(httpEndpoint);playerEndpoint.connect(filter);contentSession.start(httpEndpoint);
}
@Overridepublic void onContentStarted(HttpPlayerSession contentSession) {
PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAttribute("player");playerEndpoint.play();
}}
Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java
24
www.kurento.org
Adding Computer Vision: Client code<!DOCTYPE html><html>
<head><script src="./js/kws-content-api.js"></script><script>
var conn;function start() {
var options = {remoteVideoTag : "remoteVideo"
};conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", options);
}function terminate() {
conn.terminate();}
</script></head><body>
<button onclick="start();">Start</button><button onclick="terminate();">Terminate</button><br /><video id="remoteVideo" autoplay></video>
</body></html>
Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/webapp/playerFilter.html
25
www.kurento.org
Adding Computer Vision: See the example working
https://www.youtube.com/watch?v=yJAQs23eoXw
26
www.kurento.org
WebRTC loopback
Media Pipeline
WebRTCStreaming
Media API
REST API (Open API protocol)
Handler codeI want “this media (SDP)”
Media is“at here (SDP)”
Sink
SRC
27
www.kurento.org
WebRTC loopback: Handler code
28
Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcLoopback.java
@WebRtcContentService(path = "/webRtcLoopback")public class MyWebRtcLoopback extends WebRtcContentHandler {
@Overridepublic void onContentRequest(WebRtcContentSession contentSession) throws Exception {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build();webRtcEndpoint.connect(webRtcEndpoint);contentSession.start(webRtcEndpoint);
}}
28
www.kurento.org
WebRTC loopback: client code
29
<!DOCTYPE html><html>
<head><script src="./js/kws-content-api.js"></script><scriptsrc="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script><script>
var conn;function start() {
var options = {localVideoTag : "localVideo",remoteVideoTag : "remoteVideo"
};conn = new kwsContentApi.KwsWebRtcContent("./webRtcLoopback", options);}
function terminate() {conn.terminate();
}</script>
</head><body>
<button id="start" onclick="start();">Start</button><button id="terminate" onclick="terminate();">Terminate</button><br /><video id="localVideo" autoplay></video><video id="remoteVideo" autoplay></video>
</body></html>
Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/webapp/webrtcLoopback.html
29
www.kurento.org
WebRTC loopback: see the example working
30
https://www.youtube.com/watch?v=HaVqO06uuNA
30
www.kurento.org
WebRTC recording
Media Pipeline
WebRTCStreaming
Media API
REST API (Open API protocol)
Handler codeI want “this media (SDP)”
Media is“at here (SDP)”
Sink
SRC
Sink
Mediato
file or URI
31
www.kurento.org
WebRTC recorder: Handler code
32
Source https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java
Source of handler playing the recorded video:https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerRecording.java
@WebRtcContentService(path = "/webRtcRecorder")public class MyWebRtcRecorder extends WebRtcContentHandler {
@Overridepublic void onContentRequest(WebRtcContentSession contentSession) throws Exception {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build();RecorderEndpoint recorderEndpoint = mp.newRecorderEndpoint("file:///tmp/recording").build();contentSession.setAttribute("recorder", recorderEndpoint);webRtcEndpoint.connect(webRtcEndpoint);webRtcEndpoint.connect(recorderEndpoint);contentSession.start(webRtcEndpoint);
}
@Overridepublic void onContentStarted(WebRtcContentSession contentSession) {
RecorderEndpoint recorderEndpoint = (RecorderEndpoint) contentSession.getAttribute("recorder");recorderEndpoint.record();
}}
32
www.kurento.org
WebRTC recording: client code
33
<!DOCTYPE html><html>
<head><script src="./js/kws-content-api.js"></script><scriptsrc="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script><script>
var conn;function start() {
var options = {localVideoTag : "localVideo",remoteVideoTag : "remoteVideo"
};conn = new kwsContentApi.KwsWebRtcContent("./webRtcRecorder", options);}
function terminate() {conn.terminate();
}</script>
</head><body>
<button id="start" onclick="start();">Start</button><button id="terminate" onclick="terminate();">Terminate</button><a href="./playerRecording">Play recorded content</a><br /><video id="localVideo" autoplay></video><video id="remoteVideo" autoplay></video>
</body></html>
Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java
33
www.kurento.org
WebRTC recording: see the example working
34
https://www.youtube.com/watch?v=ogN81PGkMuE
34
www.kurento.org
WebRTC one-to-many (Handler omitted for simplicity)
Media Pipeline
Sink
SRC
SRC
Sink
SRC
Sink
SRC
Sink
35
www.kurento.org
WebRTC one-to-many: Handler code
36
Source https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcOneToMany.java
@WebRtcContentService(path = "/webRtcOneToMany")public class MyWebRtcOneToMany extends WebRtcContentHandler {
private WebRtcEndpoint firstWebRtcEndpoint;private String sessionId;
@Overridepublic synchronized void onContentRequest(WebRtcContentSession contentSession) throws Exception {
if (firstWebRtcEndpoint == null) {MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);firstWebRtcEndpoint = mp.newWebRtcEndpoint().build();sessionId = contentSession.getSessionId();contentSession.releaseOnTerminate(firstWebRtcEndpoint);firstWebRtcEndpoint.connect(firstWebRtcEndpoint); //LoopbackcontentSession.start(firstWebRtcEndpoint);
} else {MediaPipeline mp = firstWebRtcEndpoint.getMediaPipeline();WebRtcEndpoint newWebRtcEndpoint = mp.newWebRtcEndpoint().build();contentSession.releaseOnTerminate(newWebRtcEndpoint);newWebRtcEndpoint.connect(firstWebRtcEndpoint);firstWebRtcEndpoint.connect(newWebRtcEndpoint); //Latest client gives media to the master endpointcontentSession.start(newWebRtcEndpoint);
}}
}
36
www.kurento.org
WebRTC one-to-many: client code
37
<!DOCTYPE html><html>
<head><script src="./js/kws-content-api.js"></script><scriptsrc="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script><script>
var conn;function start() {
var options = {localVideoTag : "localVideo",remoteVideoTag : "remoteVideo"
};conn = new kwsContentApi.KwsWebRtcContent("./webRtcOneToMany", options);}
function terminate() {conn.terminate();
}</script>
</head><body>
<button id="start" onclick="start();">Start</button><button id="terminate" onclick="terminate();">Terminate</button><br /><video id="localVideo" autoplay></video><video id="remoteVideo" autoplay></video>
</body></html>
Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java
37
www.kurento.org
WebRTC one-to-many: see the example working
38
https://www.youtube.com/watch?v=TBkrl3fmHWI
38
www.kurento.org
WebRTC game (Handler omitted for simplicity)
Media Pipeline
Sink
SRC
Sink
SRC
Sink
SRC
39
www.kurento.org
WebRTC game: Handler code
40
Source https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcWithFilters.java
@WebRtcContentService(path = "/webRtcWithFilters")public class MyWebRtcWithFilters extends WebRtcContentHandler {
@Overridepublic void onContentRequest(WebRtcContentSession contentSession)throws Exception {
final MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);final WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build();final PointerDetectorFilter pointerDetectorFilter = mp.newPointerDetectorFilter().build();final FaceOverlayFilter faceOverlayFilter = mp.newFaceOverlayFilter().build();PointerDetectorWindowMediaParam start = new PointerDetectorWindowMediaParam("start", 100, 100, 280, 380);start.setImage("http://ci.kurento.com/imgs/start.png");pointerDetectorFilter.addWindow(start);pointerDetectorFilter.addWindowInListener(new MediaEventListener<WindowInEvent>() {public void onEvent(WindowInEvent event) {
faceOverlayFilter.setOverlayedImage("http://ci.kurento.com/imgs/mario-wings.png",-0.35F, -1.2F, 1.6F, 1.6F);}
});
webRtcEndpoint.connect(pointerDetectorFilter);pointerDetectorFilter.connect(faceOverlayFilter);faceOverlayFilter.connect(webRtcEndpoint);contentSession.start(webRtcEndpoint);
}
}
40
www.kurento.org
WebRTC game: client code
41
<!DOCTYPE html><html>
<head><script src="./js/kws-content-api.js"></script><scriptsrc="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script><script>
var conn;function start() {
var options = {remoteVideoTag : "remoteVideo"
};conn = new kwsContentApi.KwsWebRtcContent("./webRtcWithFilters", options);}
function terminate() {conn.terminate();
}</script>
</head><body>
<button id="start" onclick="start();">Start</button><button id="terminate" onclick="terminate();">Terminate</button><br /><video id="remoteVideo" autoplay></video>
</body></html>
Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java
41
www.kurento.org
WebRTC game: see the example working
42
https://www.youtube.com/watch?v=5eJRnwKxgbY
42
www.kurento.org
Collaborations welcomehttp://www.github.com/kurento
Thank you very much for your attention
Suggestions, comments and complains:[email protected]
43