Visual Coding Framework

Visual Coding Framework, a feature of Intel® Integrative Native Developer Experi-ence (Intel® INDE) enables developers to create applications with audio and video on Android* and/or Windows*. VCF generates source code with a simple drag-and-drop interface. The source code can then be integrated into an existing or new application using C/C++ bindings for rapid deployment. VCF also enables rapid prototyping to explore media capabilities and performance.

The VCF framework has three components:

• VCF Designer, the front-end user interface (UI)

• VCF runtime API, a C/C++ programming API/SDK

• VCF runtimes for Android and Windows

VCF is a framework for visual creation, prototyping, and analyzing of visual com-puting workloads. The current release of VCF is focused on creation of end-to-end media/graphics workloads. The VCF Designer user interface enables rapid visual workload creation via flow graphs. The workloads created can be instantiated in an application and executed using the VCF runtime via the VCF API.

Abstract

This white paper explains how to use Visual Coding Framework (VCF) to rapidly prototype, create, and deploy media applications. VCF enables quick and powerful ways to explore real-time performance and platform features without building an application. The difficult task of adding media to applications is no longer a road-block to creating modern, media-rich applications.

The following chapters explain how to use VCF to add media such as audio and vid-eo to new or existing applications on Android and Windows. Additional chapters explain the VCF architecture, workflow and feature set. For further details on how to use the framework, please refer to VCF manual, user guides, and sample code.

Harshdeep Brahmasury Jain Intel Corporation

Petter Larson Intel Corporation

Ankur Jain Intel Corporation

Introducing Visual Coding Framework, a Beta feature of

Intel® INDE

white paper

How to Protoytpe, Create, and Deploy Media Applications using Visual Coding FrameworkVisual Coding Framework -Beta, a feature of Intel® INDE

The VCF Designer component (colored darker blue in Figure 1) provides a natural way for designing and prototyping the desired workload by combining functional blocks or nodes such as video decode/encode, audio decode/encode, file I/O, cam-era, or renderer. The components colored green are part of the VCF SDK.

VCF Nodes

A node here represents the fundamental, functional, and execution block used to build graph workloads. VCF currently supports these functional nodes:

• Audio Decode: Decoding compressed bitstreams into raw audio (PCM) samples

• Audio Encode: Encoding from audio (PCM) samples to compressed bitstreams

• Camera: Video capture from camera, supporting both raw and compressed data

• File Sink: Output raw or compressed bitstreams with multiplexing options

• File Source: Input from raw or compressed bitstreams, including demuxing options

• Null Sink: Null output, primarily used for benchmarking

• Renderer: Render audio and/or video

• Video Decode: Decode compressed bitstreams into raw video frames

• Video Encode: Encode bitstreams from raw video frames

• Video Process: Supports processing capabilities such as denoising, deinter lacing, image stabilization, and more

Table of Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . .1

Visual Coding Framework . . . . . . . . . . . . .1

Architecture . . . . . . . . . . . . . . . . . . . . . . .2

VCF Nodes . . . . . . . . . . . . . . . . . . . . . . . . .2

Application Development

Workflow . . . . . . . . . . . . . . . . . . . . . . . . .3

Build and Run Workloads with Visual

Coding Framework . . . . . . . . . . . . . . . . . . .4

Android Device Link . . . . . . . . . . . . . . . .6

Benchmarking and Analytics . . . . . . . .8

Integrating graphs into Applications 8

Conflusion . . . . . . . . . . . . . . . . . . . . . . . . . 10

References . . . . . . . . . . . . . . . . . . . . . . . . . 10

Architecture

Figure 1 provides a high-level overview of VCF architecture.

How to Prototype, Create, and Deploy Media Applications using Visual Coding Framework 2

HARDWARE AND SOFTWARE REQUIREMENTS

OPERATING SYSTEM VERSIONS PLATFORMS

Mircosoft Windows Windows 7, 8.1, and 10 3rd generation and higher Intel® Core™ processors

Google Android KitKat*, Lolipop* Intel® Atom™ processors with Intel® Processor Graphics and Intel® Quick Sync Video

Application Development Workflow

Figure 2 shows a complete VCF workflow from an idea to deployment of an application. Blocks outlined in purple indicate how VCF is utilized. The VCF installation package is supported on 64-bit Windows operating systems only. Table 1 includes mini-mum hardware and software requirements supported by VCF run-time.

How to Prototype, Create, and Deploy Media Applications using Visual Coding Framework

3

3

Figure 3 . VCF Designer

Build and Run Workloads with Visual Coding Framework

Launch VCF from the desktop start menu, which brings up VCF Designer. VCF Designer includes the options shown in Figure 3. In VCF Designer, select Designer Mode (on the left side of VCF Designer next to Presets) to select VCF nodes.

Figure 4 . Designer Modes in VCF

How to Prototype, Create, and Deploy Media Applications using Visual Coding Framework 4

VCF also includes a Presets mode which includes a set of preconfigured graphs representing common usage of available functional nodes. You can execute the preset graph or make changes to any of the graph and/or functional node configuration parameters and save changes as new graph. For more information about a preset, click the “i” button next to each available preset.

If you try to execute a preset which depends on a media (e.g., video files and more) content package file. Click here to down-load the media Content Package File. If not installed, you will be prompted and suggested to install the package.

Now let’s build a simple transcode graph (File Source->Video Decode->Video Encode ->File Sink). Select Video File Transcode from Presets. You should now have a workload similar to Figure 5. Run the workload. You can also examine or make changes to default parameters by selecting each node. When you are satisfied with the workload design, save it as a GraphML file, which completely describes the workload.

Depending on the target system, you can select one of the following modes to build a graph workload as shown in Figure 4.

• Compatibility: Default mode ensuring cross-OS and cross-platform parity with a limited set of functional node capabili- ties and parameters.

• Windows: Exposes Windows-specific capabilities.

• Android: Exposes Android-specific capabilities.

Figure 5 . Video Transcode Workload

How to Prototype, Create, and Deploy Media Applications using Visual Coding Framework 5

Figure 6 . Android* Device Link

Android* Device Link

In addition to local workload execution, VCF also supports:

• Running workloads directly on devices

• Performing real-time workload analysis using Android device link capability

This feature is supported for Android KitKat* (4.4) and Android Lollipop* (5.x) on Intel® Atom™ processors with Intel® Processor Graphics and Intel® Quick Sync Video. Enabling this feature requires:

• Android Debug Bridge (adb) to be installed and added to the machine environment variables.

• Developer Options must be enabled on the Android device.

• The ADB USB Driver must be installed on the host to enable and connect to the device via USB.

After the above items are installed, with the target device connected to the host machine via USB, verify that all the items are installed correctly. Open a Command window in Administrator mode from the Start menu and run command line “adb devic-es”. If all the items are correctly installed, you should see your devices under “list of devices attached”.

How to Prototype, Create, and Deploy Media Applications using Visual Coding Framework 6

Restart VCF from the Start menu, select and click on the Device Link option from the toolbar in VCF Designer (Figure 6). Select the device on which you want to run the workload from the dialog box. Device connection status can be:

• Connected: A device that is connected to the host

• Online: A device that is ready and can be connected to the host

•Offline:A device that is not ready to work with VCF

Connect to the device on which you want to run the workload and run the graph. This will run the graph on the selected device and perform benchmarking, status checks, and progress updates of your workload from the device in real time on VCF user interface. Figures 6 and 7 show a step-by-step procedure for Android device linking. To switch back to local Windows runtime, select the Device Link option in the toolbar and select Windows Runtime from the dialog box.

Figure 7 . Workload on Android* Device

How to Prototype, Create, and Deploy Media Applications using Visual Coding Framework 7

To modify the benchmarking metrics according to your environment, select the Graph Properties tab (on the right side of VCF Designer). Expand the Benchmarking/Analysis item to revise settings related to benchmarking and analysis (Figure 8).

Integrating Graphs into Application

Graphs can be integrated into application code using the VCFAPI, which supports both C and C++ language bindings for both Windows and Android operating systems.

The GraphML workload is interpreted, instantiated, executed, and controlled dynamically using the VCF runtime API and as shown in the code snippet below:

1 // ... App start...2 VCFAPI myVCF; // Create VCF API instance3 myVCF.Init(); // Initialize VCF runtime4 5 VCFGraph* myGraph;6 myVCF.LoadFile(“workload.gml”, &myGraph); //Load and validate a VCF graph7 myGraph->Run(); // Execute the graph89 // ... Dynamic graph control, callbacks, etc... 10 11 myVCF->WaitForSingle(myGraph); // Wait for graph execution to complete12 myGraph->Release(); // Release all allocated resources with graph workload13 // ... App close...

Benchmarking and Analytics

You can capture and compare benchmarking metrics like frame rate, execution time, or latency between different workloads. Frame rate, also known as frame frequency, is the frequency (rate) at which an imaging device displays consecutive images called frames. Frame rate is expressed in frames per second (FPS). Latency is the amount of time between when the instant a frame is captured and the instant that frame is displayed. Execution time represents the time the workload took to complete the process.

Figure 8 . Benchmarking Workload

How to Prototype, Create, and Deploy Media Applications using Visual Coding Framework 8

Figure 9 . VCF API Overview

• VCFAPI: This is the top level the API that harbors all the run-time level functions such as run-time initialization, loading of graph workloads, and run-time level event subscriptions.

• VCFGraph: This API controls individual graph workloads including operations such as Run, Stop, and Pause. The API also exposes the ability to subscribe to graph-level events. Instances of the VCFGraph API are created via the VCFAPI:Load or VCFAPI:LoadFile operation.

• VCFNode: This API represents the basic functional execution block unit. The API controls the behavior of individual nodes. Users may subscribe for node events or invoke commands on the node. Nodes may expose a set of generic events and/or commands and also a set of node-specific commands. Instances of the VCFNode API are created via the VCFGraph: GetNode operation. (Please refer to VCF SDK manual for more details on node level operations.)

• Callbacks: For all VCF APIs, this provides callback interfaces associated with the subscribed event.

Figure 10 provides an overview of the all the blocks in VCF API.

How to Prototype, Create, and Deploy Media Applications using Visual Coding Framework 9

Conclusion

By utilizing VCF, developers can save many months of development time and invest fewer resources, since they do not have to spend time and resources mastering low-level APIs.

VCF is currently in the Beta phase, which means that Intel may make changes if issues are discovered. This also means that we are eager to get your feedback and requests. Please send your comments, questions, and issues via the VCF User Forum at https://software.intel.com/forums/vcf.

Learn More

VCF Webpage

VCF User Manual

VCF SDK Manual

VCF Samples Guide

VCF Release Notes

VCF GitHub samples

How to Prototype, Create, and Deploy Media Applications using Visual Coding Framework

1 Footnotes and Legal Text. Note the double-tab arrangement that allows for hanging footnote numbers. Agna consed deliquisl ut vullum velisis nim doloborper sisiscil enit dit ute mod te tismolesto estrud magna feugiam dolut loreet, vel dit vel ex exeros ero odolore magnit lore modolut nit irilit nulputem eu feugiam consed min ut eriuscip enissenis nullamc onsecte consequis ad dolenissi.

2 Agna consed deliquisl ut vullum velisis nim doloborper sisiscil enit dit ute mod te tismolesto estrud magna feugiam dolut loreet, vel dit vel ex exeros ero odolore magnit lore modolut nit irilit nulputem eu feugiam consed min ut eriuscip enissenis nullamc onsecte consequis ad dolenissi.

3 Vullum velisis nim doloborper sisiscil enit dit ute mod te estrud magna dolut loreet, vel dit vel ex exeros ero odolore magnit lore modolut nit irilit nulputem eu feugiam consed min ut eriuscip enissenis nullamc onsecte consequis ad dolenissi.

4 Tismolesto estrud magna feugiam dolut loreet, vel dit vel ex exeros ero odolore magnit lore modolut nit irilit nulputem eu feugiam consed min ut eriuscip enissenis nullamc onsecte consequis ad dolenissi. 10 Vel dit vel ex exeros ero odolore magnit lore modolut nit irilit nulputem eu feugiam consed min ut eriuscip enissenis nullamc onsecte consequis ad dolenissi. Please include disclaimers and copyright information appropriate for your document content. Get approval from Trademarks and Brands Legal at http://legal.intel.com/Trademarks/index.htm

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked “reserved” or “undefined.” Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or by visiting Intel’s Web site at www.intel.com.

Copyright © 2014 Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Printed in USA XXXX/XXX/XXX/XX/XX Please Recycle XXXXXX-001US

10