Embed Size (px)
Citation preview
POI360: Panoramic Mobile Video Telephony over LTECellular Networks
Xiufeng XieUniversity of Wisconsin-Madison
Xinyu ZhangUniversity of California San Diego
ABSTRACTPanoramic or 360◦ video streaming has been supported by awide range of content providers and mobile devices. Yet ex-isting work primarily focused on streaming on-demand 360◦videos stored on servers. In this paper, we examine a morechallenging problem: Can we stream real-time interactive360◦ videos across existing LTE cellular networks, so as totrigger new applications such as ubiquitous 360◦ video chatand panoramic outdoor experience sharing? To explore thefeasibility and challenges underlying this vision, we designPOI360, a portable interactive 360◦ video telephony systemthat jointly investigates both panoramic video compressionand responsive video stream rate control. For the challengethat the legacy spatial compression algorithms for 360◦ videosu�er from severe quality �uctuations as the user changesher region-of-interest (ROI), we design an adaptive compres-sion scheme, which dynamically adjusts the compressionstrategy to stabilize the video quality within ROI under vari-ous user input and network condition. In addition, to meetthe responsiveness requirement of panoramic video tele-phony, we leverage the diagnostic statistics on commodityphones to promptly detect cellular link congestion, hencesigni�cantly boosting the rate control responsiveness. Exten-sive �eld tests for our real-time POI360 prototype validateits e�ectiveness in enabling panoramic video telephony overthe highly dynamic cellular networks.
CCS CONCEPTS• Networks → Cross-layer protocols;
KEYWORDS360 degree video; LTE; cellular network
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for pro�t or commercial advantage and that copies bearthis notice and the full citation on the �rst page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior speci�c permission and/or a fee. Requestpermissions from [email protected] ’17, Incheon, Republic of Korea© 2017 ACM. 978-1-4503-5422-6/17/12. . . $15.00DOI: 10.1145/3143361.3143381
360° live video streamControl
Virtual 360° cockpit
Out of Wi-Fi coverage
Cellular BS
Figure 1: Example application scenario for portable in-teractive 360◦ video telephony over cellular networks.
1 INTRODUCTIONPanoramic videos, also known as 360◦ videos, is gainingtraction in the consumer electronics and mobile applicationecosystem. Many video content providers [9, 45] recentlystarted to stream 360◦ videos, which are usually capturedby omnidirectional cameras, and can be viewed by virtualreality head-mounted-displays (VR HMDs), such as OculusRift, Google Daydream, HTC VIVE, and Samsung GearVR.Besides fetching 360◦ videos stored on a server, the emergingmobile 360◦ stereoscopic cameras [22, 25, 29, 34], combinedwith the mobile HMDs, can potentially enable a new waveof interactive 360◦ video services that capture & stream 360◦videos in real-time. Some examples include elevating current2D video chat into 360◦ video chat, panoramic live broad-casting for outdoor events, and �ying a drone remotely asif sitting inside a virtual cockpit (Fig. 1). However, existingworks mainly investigate on-demand 360◦ video streamingfrom contend providers, while the area of interactive 360◦video telephony remains mostly unexplored.
In this paper, we present POI360, the �rst POrtable Interac-tive 360◦ video telephony system running over existing LTEcellular network. Two major challenges emerge in realizingPOI360: (i) Portability. To perform ubiquitous live streamingeven out of the Wi-Fi coverage, POI360 relies on the mobilebroadband. However, a 360◦ video contains views in everydirection and must entail high resolution for the near-eye dis-play of HMD, which translates into a bitrate of tens of Mbpsand drastically overloads the LTE uplink (median bandwidtharound 2.2 Mbps [13]). (ii) Interactivity. POI360 confronts anintrinsic con�ict between the strict latency requirement ofinteractive video and the highly dynamic cellular networkcondition. The situation is even more challenging when the
CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea Xiufeng Xie and Xinyu Zhang
heavy 360◦ video tra�c pushes the cellular link to its limit,as a �uctuating link bandwidth can easily fall below the hightra�c load. POI360’s rate control must detect this and reducethe tra�c rate promptly to prevent video freezes.
To meet these challenges, POI360 design achieves bothe�cient compression to curtail the overwhelming bitrateof 360◦ video stream, and responsive rate control to facili-tate smooth real-time video telephony. (i) Although existingworks on 360◦ video compression [11, 24, 30, 33] can reducethe tra�c load by prioritizing the encoding quality of theregion-of-interest (ROI) in a 360◦ video frame, the highlydynamic cellular network latency may easily fail such ROI-based compression, as the sender’s ROI knowledge may lagbehind the remote user’s actual ROI, causing mismatchesbetween the sender-allocated encoding quality and user-perceived display quality, and thus severe quality �uctua-tions (§3). POI360 answers this challenge with an adaptivespatial compression algorithm – It dynamically tunes the en-coding quality distribution across the 360◦ panorama basedon the ROI update responsiveness, thus guarantees a stableuser-perceived quality even during user ROI change. (ii) Toboost the responsiveness of rate control, since cellular uplinkis the typical network bottleneck for interactive video (§3),POI360 monitors the cellular device’s uplink �rmware bu�eroccupancy to instantaneously detect any uplink congestion.Upon congestion, POI360 degrades its uploading bitrate tothe windowed-sum of cellular uplink transport block size(TBS), which is the guaranteed bandwidth.
We have implemented a POI360 prototype over commer-cial LTE networks. Targeting VR HMD using smartphone asscreen, POI360 runs in mobile browsers, and our implementa-tion integrates WebRTC [18] which serves as the back-end ofmost interactive video services [40], and WebGL, which con-verts the 360◦ video stream to the HMD-compatible stereo-scopic view in the browser. We implement POI360’s cellular-link-informed rate control in the WebRTC module of theChromium browser. The required cellular physical-layerstatistics, including the �rmware bu�er level and TBS, areread from the diagnostic interface on commodity phones.Therefore, POI360 requires no hardware modi�cation and isreadily deployable. Our run-time experiments demonstratethat POI360’s adaptive spatial compression algorithm sig-ni�cantly outperforms existing benchmark algorithms, andPOI360’s rate control is much more responsive than conven-tional solutions that rely on end-to-end delay/loss metrics.We have also conducted �eld tests under various locationsand mobility levels, which con�rm POI360’s e�ectiveness inenabling real-time interactive 360◦ video services over LTE.
The contributions of POI360 are summarized as follows:(i) To the best of our knowledge, POI360 is the �rst portable
interactive 360◦ video telephony system customized for ex-isting LTE cellular network infrastructure.
tCompression
tViewer
Frame interval
Frame latency
Streaming
Raw frames
ROI
Region-Of-Interest (ROI)
Sender
High qualityLow quality
Figure 2: ROI-based compression for 360◦ video.
(ii) POI360 features a novel adaptive spatial compressionscheme for 360◦ video, which dynamically tunes the compres-sion strategy to �t the varying cellular network conditions.(iii) POI360’s awareness of cellular link statistics enables
prompt adaptation to facilitate smooth 360◦ video telephony.
2 BACKGROUND FOR 360◦ VIDEO360◦ videos are typically shot using an array of cameras [32]whose �eld-of-view (FoV) together covers 360◦. The videoframes captured from each camera at the same moment arestitched and then projected onto a sphere, which is furthermapped to a planar format as one 360◦ video frame. Thisis known as equirectangular projection and the world mapis a classical example. To further compress the video data,alternative conversion schemes instead project the sphereinto a cube [8] or pyramid [10]. Nonetheless, the bitrate ofa 360◦ video (e.g., tens of Mbps for 4K video [11]) typicallyexceeds the bandwidth of wireless links [15].
To alleviate the bandwidth crunch, the region-of-interest(ROI) streaming has been proposed [6, 24, 33, 35, 39]. Humaneyes have limited FoV, and the vision resolution is high atthe central visual �eld (fovea) but drops almost quadraticallywith distance from the center. Accordingly, the resolutionof regions near the visual periphery or even out of FoV canbe compressed more aggressively. As shown in Fig. 2, byestimating the viewer’s foveation using head-orientationsensors inside a VR HMD, the video sever can selectivelycompress the video quality—the ROI will be delivered withhigh resolution while the remaining regions will have lowerquality. Such ROI streaming mechanisms can bring the band-width cost to a level that can be supported even in cellularnetworks [30]. Besides orientation sensors, alternative ap-proaches can further narrow down the ROI, e.g., using aneye-tracking camera to track the viewer’s focus [35], using se-mantic analysis of the video frame content or crowd-sourcedviewing statistics [11, 28, 30]. Although this work focuses onthe head-orientation-based ROI compression, our solutioncan be potentially applied to such systems.
POI360 CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea
Highquality
Lowquality
Lowquality
Sender
Viewer
Frame
Highquality
Lowquality
Lowquality
Viewer
Sender
FrameViewer
changes ROI
New ROI
Figure 3: Problem for ROI-based compression: viewerROI change causes user-perceived quality �uctuation.
Conservative compression
Aggressive compression
Video quality
Spatial position in frame
Smooth quality dropSharp quality drop
ROI change
ROI center
ROI change
Figure 4: Trade-o� between compression ratio anduser-perceived quality stability during ROI update.3 CHALLENGES & SOLUTION SPACEIn this section, we investigate the challenges for realizingportable interactive panoramic video telephony over cellularnetworks, and explore the corresponding solution spaces.
3.1 ROI Quality Fluctuation.Challenge I. When running over highly dynamic cellularnetworks with much longer and unstabler latency than thewireline access networks [46], ROI-based spatial compres-sion causes frequent user-perceived video quality �uctua-tions for the ROI region as the user view changes. As shownin Fig. 3, due to the network latency of ROI feedback, thepanoramic video sender and viewer may have inconsistentknowledge of the viewer’s current ROI, thus there can bea temporary mismatch between the sender-allocated qual-ity and user-perceived quality for the ROI region wheneverthe user view changes, causing quality �uctuations and usersickness. In §6, we further verify this phenomenon throughreal-time experiments (Fig. 12b). We note that motion-basedROI prediction [2, 21] cannot fully solve this problem as itsprediction horizon is still shorter than the end-to-end latencyof mobile interactive video sessions (§8).
Solution space I. To mitigate the quality �uctuation, weneed to address the trade-o� between the compression ag-gressiveness and the stability of ROI quality. An aggressivecompression mode can focus the quality on a small ROIwhile minimizing the non-ROI bitrate. But it is sensitiveto the ROI changes as the user’s FoV can easily enter thenon-ROI region. So it applies only to networks with small
0
1
2
3
4
5
6
0 5 10 15 20 25Sum
UL T
BS
/s (
Mbps)
Firmware Buffer Status (KByte)
Figure 5: The relation be-tween �rmware bu�er oc-cupancy and TBS.
0
0.2
0.4
0.6
0.8
1
0 10 20 30 40 50
CD
F
Firmware Buffer Level (KByte)
Figure 6: Uplink �rmwarebu�er level under We-bRTC rate control.
feedback latency. On the other hand, a conservative compres-sion mode that distributes the quality more evenly acrossspatial regions is suitable for high-latency networks in orderto keep stable video quality in ROI region even under laggyROI feedback, but has to sacri�ce the quality itself to �t theentire frame within limited network bandwidth. To makethe situation worse, cellular networks latency is also highlydynamic due to the varying wireless environment, causingunstable responsiveness of the end-to-end ROI update.
To resolve these issues, instead of sticking to a �xed com-pression mode, we can dynamically switch across di�erentmodes based on the responsiveness of ROI update, as shownin Fig. 4. In this way, we can aggressively reduce the tra�cload when current network condition allows the sender toresponsively update its ROI knowledge, and conservativelycompress the stream to maintain a stable video quality whenthe end-to-end ROI update becomes sluggish.
3.2 Video Rate Control ResponsivenessChallenge II. The rate control algorithms must responsivelyadapt the panoramic video encoding bitrate based on net-work conditions, to avoid congestion and video freeze. How-ever, existing video bitrate adaptation algorithms for interac-tive videos commonly adopt end-to-end network conditionestimation (§2).Such strategies fail to meet the responsive-ness requirement of interactive panoramic video, especiallyover highly dynamic cellular networks (experimental evi-dences will be shown in §6).
Solution space II. For panoramic video telephony, cel-lular uplink is typically the network bottleneck, but needsto sustain the same tra�c load as the downlink or wirelinesegments. Therefore, the sender can bypass the sluggish end-to-end metrics, and directly execute congestion detectionand bandwidth estimation for the cellular uplink alone.
3.3 Uplink Bandwidth Underutilization.Challenge III. Due to the use of proportional fair schedul-ing in LTE networks [44], the user device’s uplink �rmwarebu�er has an unique feature – its service rate depends on itsown bu�er occupancy. Fig. 5 showcases this e�ect, wherewe measure the bu�er size and uplink throughput Rphy (indi-cated by the TBS) on an LTE smartphone. With small bu�er
CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea Xiufeng Xie and Xinyu Zhang
FirmwareBufferAwareRateControl
FirmwareBuffer
Buffer level
360° Cam
Encoded stream
RTP packets
Cellular uplink
ROI
Congestion indicator
Sender
AdaptiveSpatialCompression
Compressed frames
WebRTC
Viewer
Figure 7: POI360 system architecture.size, the Rphy increases almost linearly with the bu�er size,and saturates when reaching the uplink bandwidth. Unawareof this e�ect, generic video rate control frames, like WebRTC,simply set the RTP sending rate Rr tp to match the video bit-rate Rv . However, a temporary uplink bandwidth surge candrain the bu�er quickly, and the RTP sender may leave thebu�er empty as it needs to follow the Rv cap, even if thereexists pending tra�c at the video application layer. As anevidence, Fig. 6 plots the CDF of uplink bu�er level whenstreaming a 4K panoramic video through an LTE phone(more details about the setup are in §5). We see that the up-link bu�er becomes empty for 40% of the time, even thoughthe tra�c always exceeds the available bandwidth.
Solution space III.The knowledge of the uplink �rmwarebu�er level opens a new opportunity to address this chal-lenge. In particular, the sender can push the �rmware bu�erlevel to a “sweet” region which is far from congestion butstill high enough to harness the bandwidth provisioned bythe basestation’s uplink scheduling. As the video encodingbitrate Rv should not be changed too fast to avoid quality�uctuations that may degrade user experience, we keep up-dating the RTP sending rate Rr tp based on the instantaneousbu�er level to keep it in the “sweet” region.
4 POI360 SYSTEM DESIGNIn this section, we present our POI360 system design to ex-ploit the solution space discussed in §3. As outlined in Fig. 7,the POI360 sender �rst applies adaptive spatial compressionto perform ROI-based spatial compression to the input 360◦video stream while adapting the compression mode basedon current network condition, the ROI-compressed streamis then encoded by the legacy WebRTC framework[18] (bydefault the VP8 encoder) which performs temporal compres-sion to the frames. The sender then uses �rmware bu�eraware rate control to e�ciently deliver the encoded stream tothe remote viewer wearing a VR HMD. Packet losses and re-bu�ering during streaming are handled by WebRTC’s builtin
ROI center
Raw 360° frame
Compressed frame
Compress(Sender)
Unfold(Client)
Tile size(quality)
Spatial position
Compression mode
x
y
Figure 8: POI360 videocompression model.
PacketPacer
Video bitrate
RTP bitrate
PHY throughput
CellularFirmwareBuffer
Applicationlayer
Transportlayer
Physicallayer
PanoramicVideoEncoding
Figure 9: POI360 rate con-trol model.
mechanisms[14]. In what follows, we introduce POI360’ssystem model (§4.1), and then describe its two major designmodules (§4.2 and §4.3) in more detail.
4.1 System Model360◦ frame compression. As illustrated in Fig. 8, POI360compresses 360◦ video stream at the video frame level. Eachraw 360◦ video frame is �rst spatially segmented into smalltiles, which are then compressed separately based on their rel-ative position w.r.t. the user’s current ROI center. Finally, alltiles of a video frame are stitched together and sent throughthe cellular uplink. Based on Fig. 8, we de�ne several com-pression parameters to be used across the paper. Let i denotea tile’s position in the x-axis while j denote its position inthe y-axis in a 360◦ video frame. The center of the user’scurrent ROI is a tile whose position is denoted as r = (i∗, j∗).We de�ne compression level li j as the ratio of tile size beforeand after compression, and compression matrix L is the ma-trix of compression level li j for all tiles. We further de�necompression mode F as the mapping that determines thecompression level li j of each tile based on the distance be-tween its position (i, j) and the current ROI center r = (i∗, j∗),i.e., li j = F (i − i∗, j − j∗). The basic principle behind POI360’scompression mechanism is to use higher compression levelfor the tiles further away from the ROI center r , and the cen-ter itself should always have the lowest compression levellmin . In our current POI360 prototype, we implement thecompression by scaling down the width and length of eachtile based on its distance to the ROI center, i.e., the compres-sion ratio for the tile width is Ci−i∗ and similarly C j−j∗ forthe tile height, where C > 1 is a constant value controllingthe compression aggressiveness for this compression mode.As a result, the compression mode is essentially de�ned as:
li j = F (i − i∗, j − j∗) = C(i−i
∗)+(j−j∗) (1)Under a certain compression mode, when the ROI center
shifts with the user’s head position, the compression matrixwill be updated following Eq. (1), which is essentially a cyclicshift based on the shift of ROI center.
POI360 CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea
360◦ video stream rate control. As shown in Fig. 9,POI360 uses a cross-layer rate control model at the videosender, which contains two important bu�ers: the videobu�er in the application layer, and the �rmware bu�er inthe LTE PHY layer. Below are the crucial model parameters:360◦ video encoding bitrate (Rv ) determines the bitrate of
the compressed video source, i.e., tra�c injection rate (videoquality) into the video bu�er.RTP sending rate (Rr tp ) in the transport layer parameter
controlling the outgoing tra�c of the video bu�er and theinput tra�c into the cellular �rmware bu�er.
PHY-layer throughput (Rphy ) determines the outgoing traf-�c of the cellular �rmware bu�er. Rphy is controlled by theBS’s uplink scheduling policy. However, as will be clari�edsoon, POI360 can indirectly a�ect Rphy by controlling Rr tp .
From the diagnostic interface on commodity smartphones,we can extract the LTE physical-layer statistics including theuplink �rmware bu�er level (B) and uplink transport blocksize or TBS (Ptbs ), for every LTE uplink subframe [23].
4.2 Adaptive Spatial CompressionPOI360’s adaptive compression module aim to tame the360◦ video quality �uctuations caused by ROI changes. Asdiscussed in §4.1, for ROI-based compression, the senderneeds to keep updating the compression matrix L(t) basedon rs (t) = (i
∗s , j∗s ), its current knowledge of the client-side
ROI. Due to the end-to-end feedback latency df , the sender’sknowledge of the viewer’s ROI position rs (t) lags behind theactual value rc (t), i.e., rs (t) = rc (t −df ). As a result, when theviewer shifts her ROI to rs (t) at time t , the sender will keepusing the previous ROI rc (t −df ) to compress the 360◦ videostream until t +df . In this duration, the user view has shiftedto the new region, while the sender has not updated thecompression level for this region yet. So the viewer observesa quality drop in ROI since time t , and then experiences aquality surge at time t + df + dv (where dv denotes the one-way video frame delay), as it takes some time for the senderto be aware of an ROI change and deliver the video framescompressed with updated ROI to the viewer. Such quality�uctuations severely degrade the user experience[19].
To address this issue, POI360’s adaptive spatial compres-sion allows the sender to gauge the responsiveness of theend-to-end ROI update, and then adopt the spatial compres-sion mode F that best �ts current responsiveness level. Un-der swift ROI update, the sender can aggressively compressthe non-ROI regions. On the other hand, under sluggish ROIupdate, the sender should create a smooth quality transitionfrom ROI to the non-ROI regions, as the compressed contentmay be temporarily displayed in the new ROI. The key chal-lenge here is to precisely estimate the end-to-end ROI updateresponsiveness. As shown in Fig. 10, this responsiveness is af-fected by 3 factors: (i) The ROI feedback delay df determines
tROI Stream compressed with new ROI
ROI mismatch time
New ROI quality converged
Sender
Viewer tROI change
Update ROI knowledge
Figure 10: ROI mismatch due to network latency.how long it takes for the sender to be aware of an ROI change.(ii) The one-way video frame delay dv re�ects how long ittakes for the 360◦ video stream compressed with the updatedROI to arrive at the client. (iii) The ROI stability shows howoften the user changes the ROI and by how much.
POI360 employs a single metric, the ROI mismatch time(M) to capture all these factors. As shown in Fig. 10, M isde�ned as the time interval during which the 360◦ videosender and client have inconsistent knowledge of the user’sROI. When the network causes higher ROI feedback latency,M will be higher. Similarly, when the network causes highervideo frame delay, it takes longer for the frames with updatedROI to reach the client, leading to higher M as well. Finally,when the user switches the ROI consecutively, inconsistencybecomes more severe, again leading to higher M .
POI360 measures M by monitoring the ROI video qualityat the client side: Ideally, the ROI region should always enjoythe highest quality. In other words, the ROI compression levell(i∗c , j
∗c ) should equal lmin . However, when the user quickly
switches to a new ROI, her head may temporarily point to alow-quality region. Following Eq. (2), POI360 measures M atthe time scale of every 360◦ video frame.
M =
{max(t − t0,dv ), if l(i∗c , j
∗c ) , lmin
dv , otherwise(2)
where the client starts counting the time on detecting theROI change at time t0, and computes M = max(t − t0,dv ) forthe frame coming at time t > t0 until the quality in the newROI converges to the highest level. For the frames in whichcurrent ROI already has the lowest compression level, e.g.,when the user keeps the ROI, we have M = dv , as the updatelatency for potential ROI change must be higher than currentframe delay dv . The client then uses a sliding time windowto average across the frame-level measurements of M , andperiodically feeds back the average M to the sender, with thefeedback interval the same as the video frame interval.
After receiving M , to achieve a balance between tra�cload reduction and the stability of user-perceived ROI qual-ity, the sender then switches between K pre-de�ned com-pression modes F1, ...,FK listed in the order of decreasingcompression aggressiveness (smoother quality drop for re-gions further away from ROI). Our implementation usesK = 8 and empirically selects the compression mode Fimwith im = max(8, dM/200mse), and the constant C in Eq. (1)is selected from [1.1, 1.2, ..., 1.8] for the 8 compression modes,representing 8 levels of compression aggressiveness.
CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea Xiufeng Xie and Xinyu Zhang
4.3 Firmware Bu�er Aware CongestionControl
POI360’s Firmware Bu�er Aware Congestion Control (FBCC)aims to address the challenge of irresponsive rate adaptation,following the solution principle in §3. In what follow, wedescribe the details and rationales behind our design choices.
4.3.1 Panoramic video encoding bitrate control. The coreprinciple behind FBCC’s video bitrate control is to let thesender promptly detect the overuse of cellular uplink band-width based on its local cellular �rmware bu�er occupancy,instead of waiting for the remote peer’s end-to-end con-gestion measurement feedback. Upon detecting bandwidthoveruse, the panoramic video sender immediately reduces itsencoding bitrate to �t the instantaneous uplink bandwidth,which is again computed based on the PHY-layer statistics.
How to detect cellular uplink congestion? Conven-tional congestion control algorithms, like the GCC used inWebRTC, treat the network as a blackbox and rely on theend-to-end network delay/loss metrics to detect congestion.As a result, if a congestion happens, it takes at least one RTTfor the sender of conventional rate control to detect it. Ine�ect, due to the large bu�ers at cellular basestations [42],those metrics become extremely irresponsive, taking up toseconds to react to a congestion.
In contrast, FBCC is customized for interactive panoramicvideo services running over cellular networks, exploitingthe fact that the last-hop cellular path typically bottlenecksthe network path. FBCC is essentially a network-assistedrate control algorithm, which leverages the uplink �rmwarebu�er occupancy to enable quick detection of potential con-gestion. Speci�cally, let J ∈ {0, 1} be the congestion indicatorwith J = 1 showing a congestion. B(t) denotes the �rmwarebu�er occupancy at time t , and ∆t denotes the report inter-val of �rmware bu�er occupancy from the phone’s chipset,FBCC estimates a bandwidth overuse and potential conges-tion based on the following conditions:
J =
1, if B(t − (n − 1)∆t) > B(t − n∆t)
for n = 1, ...,K and B(t) > Γ(t)
0, otherwise(3)
Essentially, the FBCC estimates a congestion on detectingboth of the following events: (i) A continuous increases of�rmware bu�er occupancy B(t) for K consecutive reportsfrom the chipset. To guarantee the responsiveness, POI360uses a small K = 10. (ii) Current �rmware bu�er occupancyB(t) exceeds a threshold Γ(t), which is set to the long-termaverage bu�er level and keeps being updated online.
How to throttle the rate on detecting an uplink con-gestion? After detecting the uplink bandwidth overuse, therate control algorithm should reduce the video encoding bi-trate to avoid potential congestion. While classical congestion-control solutions throttle the rate following empirical curves,
POI360 can precisely cut the encoding bitrate to �t the in-stantaneous cellular uplink bandwidth. In particular, the sumof the TBS (Pwtbs ) for allW LTE subframes (with 1ms length)in a time window divided by the length of the time windowW is the current PHY-layer LTE uplink throughput:
Rphy =
∑Ww=1 P
wtbs
W(4)
When an uplink bandwidth overuse is detected (J = 1), thethroughput Rphy on this saturated uplink channel is exactlythe available uplink bandwidth Rbw :
Rbw
{= Rphy , if J = 1> Rphy , otherwise
(5)
As a result, after detecting a uplink bandwidth overuse, FBCCdirectly sets the encoding bitrate toTu based on Eq. (4). Notethat POI360 resorts to Tu only under congestion, because Turepresents the current rate that the client can exploit, butcannot indicate the potential rate it can explore.
Handling congestion elsewhere. In certain rare cases,congestion can happen elsewhere along the end-to-end path,e.g., due to the unpredictable heavy cross tra�c. POI360detects and handles such cases by degrading to the legacyWebRTC [18] end-to-end congestion control (discussed in§7), Google Congestion Control (GCC). More speci�cally,let Rдcc denote the bitrate selected by the legacy GCC al-gorithm, and t∗ denote the moment when a cellular uplinkcongestion is detected, POI360 combines both the cellularuplink congestion control and legacy end-to-end congestioncontrol for encoding bitrate adaptation as follows:
Rv (t) =
{∑Ww=1 P
wtbs
W , if t∗ ≤ t ≤ t∗ + 2RTTRдcc (t), otherwise
(6)
As shown in the above equation, (i) When the cellular uplinkbecomes the bottleneck (J = 1 detected at t = t∗), POI360immediately reduces the encoding bitrate Rv to the cellu-lar uplink bandwidth Rbw computed by Eq. (5). In this way,POI360 is more responsive than GCC as its congestion con-trol does not wait for the reduced Rдcc feedback from theremote peer. Furthermore, our solution is also more accurateas it directly converges to the cellular uplink bandwidth, in-stead of sharply reducing the rate and probing the bandwidthagain. (ii) When J = 0, the congestion happens elsewhere,or there is no congestion at all. By setting the encodingbitrate Rv to Rдcc , the selected bitrate from the WebRTCclient, POI360 reuses the legacy WebRTC congestion controllogic to handle network congestion elsewhere. Similarly, ifthere is no congestion, uplink congestion control will not betriggered, so we still have Rv = Rдcc .
Note that after a bandwidth overuse occurs at the cellularuplink (t∗ < t ≤ t∗ +RTT ), legacy GCC at the client side willalso detect a bandwidth overuse and then noti�es the senderto throttle the rate by sharply reducing the Rдcc to feedback,
POI360 CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea
which arrives the sender one RTT after our algorithm reducesthe rate to Rbw . In other words, the di�erent responsivenessof sender-based and client-based algorithms may causes twoconsecutive rate reductions on a single bandwidth overuseevent. To avoid this, as in Eq. (6), our design keeps forcing thevideo encoding bitrate to the PHY-layer uplink bandwidthRbw within a duration of 2 RTTs (in cases the RTT itselfchanges) after an uplink bandwidth overuse is detected.
We also emphasize that FBCC reduces the 360◦ video en-coding bitrate, rather than the transport-layer bitrate, whenbandwidth overuse occurs. As shown in Fig. 9, if we throttlethe transport-layer bitrate on an uplink bandwidth overuse,it simply changes the queuing location from the �rmwarepacket bu�er to the application-layer packet bu�er, but thetotal queue length remains unchanged. The only ways toclear the queues are to throttle the source tra�c rate andincrease the physical-layer service rate. The encoding ratecontrol essentially throttles the source tra�c rate. In whatfollows, we introduce how FBCC boosts the physical-layerservice rate by properly adapting the transport-layer bitrate.
4.3.2 Cellular-Link-Informed RTP Rate Control. As dis-cussed in §3.3, conventional rate control may unnecessarilyleave the cellular �rmware bu�er empty due to the poorinteraction between the video bitrate control and transportrate control. FBCC addresses this issue by pushing the local�rmware bu�er level to the “sweet spot” that maximizesbandwidth utilization without causing congestion. Since itcannot directly change the �rmware bu�er level, FBCC in-directly controls the bu�er level by controlling the bu�er’sinput – the RTP sending bitrate. The algorithm works witha time granularity of Dp , which is the feedback interval ofthe bu�er level from the chipset (40ms in our test device).
At the beginning of each time epoch with duration Dp ,the RTP rate controller monitors the occupancy of �rmwarebu�er B(t). When it drops below the target bu�er level B∗phycorresponding to the optimal cellular uplink rate, it updatesits current RTP sending rate Rr tp (t) following:
Rr tp (t) = Rr tp (t − Dp ) +(B∗phy − B(t)
)/Dp (7)
The physical meaning of Eq. (7) is to increase the RTPbitrate so that the bu�er level can reach the target level whenthe next time epoch starts. The optimal �rmware bu�er levelB∗phy depends on the consistent uplink scheduling policy ofthe LTE BS, which can be learnt from previous transmissions.In the rare cases when the cellular uplink is not the networkbottleneck, the video encoding bitrate Rv (t) is bounded bythe network bottleneck bandwidth as in Eq. (6), and we mayhave Rv (t) < Rr tp (t), i.e., the uplink �rmware bu�er can stilldrain out, but that prevents the congestion at the networkbottleneck. We also note that this design does not a�ect thefairness to other cellular users, because the LTE BS controlsthe fairness in uplink scheduling.
5 IMPLEMENTATIONBrowser-based adaptive spatial compression. We imple-ment POI360’s spatial compression algorithm using HTML5.Speci�cally, every raw 360◦ video frame is divided to smalltiles (12 × 8 tiles in our implementation) by writing eachtile to a separate HTML5 canvas. The size of each tile iscompressed based on current compression mode F . Thecompressed tiles are then stitched together in one HTML5canvas, in which the sender also embeds its current compres-sion mode and its knowledge of the viewer’s ROI. Finally,the browser’s WebRTC module captures and encodes theoutgoing video stream from this canvas. After receiving thecompressed video stream, the client �rst infers the sender’scompression matrix using the information embedded insideeach received video frame and then unfolds this compressedframe accordingly. Then the client uses WebGL to render theunfolded video frame as stereoscopic view for left and righteyes, which can be viewed with stereoscope VR headsets likeGoogle cardboard. Since the 360◦ video stream is capturedfrom real world, rather than generated based on a real-time3D engine, and the ROI-compressed video frame dimensionbecome comparable to that of conventional video, the com-putational overhead and latency of 360◦ video encoding andrendering for POI360 are comparable to WebRTC-based con-ventional video telephony like Google Hangout.
Besides rendering the stream, the client also keeps up-dating current ROI based on the user input like the motionsensor reading on the phone, and periodically feeds back cur-rent user ROI to the sender via WebRTC data channel. Thefeedback interval is set to the video frame interval 1. Mean-while, the client also keeps monitoring the responsiveness(M discussed in §4.2) of the end-to-end ROI update, whichis also fed back via the WebRTC data channel. Based on M ,the sender then adapts its compression mode following §4.2.
FirmwareBu�erAwareCongestionControl. POI360’scellular-link-informed rate control obtains the physical-layerstatistics from the diagnostic interface, which is available onmost commodity phones. So it requires no hardware modi�-cations and is readily deployable. Following the implemen-tation of MobileInsight[23], the POI360 prototype uses ourcustomized real-time log decoder to read the phone’s diag-nostic interface and obtains the LTE uplink TBS and theuplink �rmware bu�er level for every 40ms. These cellularphysical layer statics are then written to shared memorywhich is also accessed by POI360’s FBCC algorithm. We in-tegrate FBCC into the WebRTC module in the open-sourceChromium browser by modifying the GCC implementationin WebRTC source code.1Since both the WebRTC data channel and video stream run over UDP,current LTE BS gives the same priority to the ROI feedback and video data.In future 5G networks with more �exible �ow control, the BS may schedulethe ROI feedback at higher priority to further improve performance.
CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea Xiufeng Xie and Xinyu Zhang
360◦ video telephony performancemeasurement. (i)End-to-end video frame delay measurement. The core idea isto embed the sending timestamp of a video frame insidethe frame itself, the receiver then extracts the sending timefrom the received frame and subtracts it from local time tocompute the delay. The sender and receiver are synchronizedusing NTP. Each decimal bit of the sending timestamp (withmillisecond resolution) is encoded using a colored squareblock, with the number from 0 to 9 mapping to 10 colors withuniform separation in the RGB code space. These blocks arethen appended to the side of the outgoing video frame.Atthe client side, the RGB values of pixels within each coloredblock region of the received frame are averaged and thenmapped back to the decimal number.(ii) 360◦ video quality measurement. Video quality mea-
surement requires comparison between the original and thereceived video frame. However, for 360◦ video, the usersonly care about the quality within ROI, as other regions arenot rendered. Therefore our measurement system matchessender-side and client-side frames in both time and space do-main to facilitate 360◦ video quality measurement. In particu-lar, the client only dumps the ROI of its received frame, alongwith the corresponding ROI position (i∗c , j∗c ). The sender, onthe other hand, may have stale ROI knowledge due to thenetwork latency, so it has to dump the entire frame. Finally,in the ROI comparison, we crop the ROI region from the orig-inal frame based on the client’s corresponding ROI position.
6 POI360 EVALUATIONIn this section, we �rst use micro-benchmark experiments(§6.1) to validate POI360’s design components, includingthe adaptive spatial compression (§6.1.1) and the FBCC ratecontrol (§6.1.2), and demonstrate the individual performancegain contributed by each of them, Then we conduct extensivesystem level tests for our real-time POI360 prototype undervarious network conditions (§6.2). Our experiments run ona commercial LTE network using LG Nexus 5 D820 smart-phones. To obtain representative human-controlled ROI, weinvite 5 users to participate in the experiments. Since thebehavior of an user’s ROI is highly correlated with the videocontent, we use di�erent 360◦ video for each user to avoidover�tting. To guarantee repeatable 360◦ video tra�c loadin experiments, we use virtual webcam[41] to live streamthe same 360◦ video with 4K resolution when repeating theexperiments for a user. In our experiments, each 360◦ videotelephony session runs for 5 minutes, and each user repeatsthe test session for 10 times. Due to the limited LTE band-width, other network applications running on the same de-vice can degrade POI360 performance. To isolate such e�ects,we turn o� other data consuming applications on the phone.
6.1 Microbenchmarks
MOS Video Quality PSNR Range (dB)Excellent > 37
Good 31 ∼ 37
Fair 25 ∼ 31
Poor 20 ∼ 25
Bad < 20
Table 1: Mapping PSNR to Mean Opinion Score (MOS).
0
10
20
30
40
POI360 Conduit Pyramid
PS
NR
(d
B)
(a) PSNR over wireline.
0
10
20
30
40
POI360 Conduit Pyramid
PS
NR
(d
B)
(b) PSNR over cellular.
0
0.2
0.4
0.6
0.8
1
Bad Poor Fair Good EXCP
DF
POI360ConduitPyramid
(c) MOS over wireline.
0
0.2
0.4
0.6
0.8
1
Bad Poor Fair Good EXC
PD
F
POI360ConduitPyramid
(d) MOS over cellular.
Figure 11: Comparing video quality under POI360compression and benchmark algorithms.
6.1.1 Adaptive Spatial Compression. We �rst verify theadaptive spatial compression. To highlight the e�ectivenessof POI360’s design customized for cellular networks, thesame set of experiments are also repeated over a wireline net-work on a university campus. Speci�cally, we run a POI360session with two laptops, and the Internet access of bothlaptops are provided by either wireline connection or LTE.To isolate the impact of network protocols, we use GCC,WebRTC’s default rate control (§2), as the transport layer.
We compare POI360 with two benchmark panoramic videocompression schemes including Conduit[1] and Pyramidencoding[7]. Conduit crops the ROI region from the panoramavideo frame and only streams the cropped parts. To avoiddisplaying blank frame, we still send the non-ROI regionsfor Conduit but with the lowest possible quality. Pyramidencoding centers the panorama video frame at the ROI, andthen allocates the center region the highest quality and ag-gressively compresses the contents in the 4 corners based ontheir distance to the center. Based on our system mode (§4.1),Conduit can be considered as a very aggressive compressionmode with a sharp quality distribution curve, whereas Pyra-mid encoding adopts a conservative compression schemewith smooth quality distribution, so it can better handle theROI shifting at the cost of higher tra�c load. However, bothalgorithms are incapable of dynamically adapting the com-pression modes based on the network conditions. Besides,motion-prediction based compression[30] does not work inour application scenario (§8), thus is not compared here.
POI360 CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea
User-perceived video quality in ROI. We �rst evaluatethe user-perceived ROI quality based on the aforementionedsetup. First, Fig. 11a and 11b plot the average PSNR withinthe ROI region over both cellular and wireline networks,with error bars showing the standard deviation of PSNRvalue. We can observe that POI360’s compression algorithmachieves the highest PSNR within the ROI region. In wirelinenetworks, all the algorithms perform reasonably well. How-ever, Conduit and Pyramid su�er from signi�cant qualityloss when running over cellular networks, with 11 to 13 dBlower PSNR than POI360. Such disparate performance comesfrom POI360’s capability to adapt the compression modesbased on the network latency that a�ects the end-to-end ROIupdate responsiveness. POI360 can e�ectively switch to theproper spatial compression mode that best �ts the currentnetwork condition. In contrast, when a viewer changes ROIin Conduit, the remote peer cannot update its knowledge ofthe viewer’s ROI in time. So the viewer can only see com-pressed regions with low quality. Pyramid shows slightlyhigher PSNR than Conduit, as it has a smoother spatial qual-ity distribution, and hence less quality drop during the ROIshift.
To understand the viewer’s subjective experience, in Fig. 11cand 11d, we further plot the PDF of the Mean Opinion Score(MOS) computed based on the frame-level PSNR, followingthe well-known relation between the MOS and PSNR[36]summarized in Table 1. We see that POI360 achieves the bestMOS over both cellular and wireline networks. In particular,over cellular networks, 52% of its frames have good qualitywhile 4% have excellent quality. Conduit does not provideany frame with good or excellent quality, and Pyramid onlyhas 7% frames with good quality. Even in wireline networks,POI360 has more than half of the frames with excellent qual-ity and the rest of the frames with good quality, substantiallyoutperforming both Conduit and Pyramid.
We emphasize that POI360’s performance gain mainlycomes from its principle of adaptive compression. The com-pression itself is taking e�ect just like other schemes. Forexample, given a 360◦ video with 12.65Mbps raw bitrate, thereceived stream bitrate at the POI360 client is only around3Mbps, reduced by 76% (more in §6.1.2 and Fig. 16a).
Video quality stability in ROI. Besides video qualityitself, the short-term stability of video quality is also vitalfor user experience[19]. We characterize this stability by thestandard deviation of the ROI compression level in a 2 sec-ond sliding window. The CDF plot in Fig. 12a shows thatall the algorithms have small quality variation in wirelinenetworks with relatively stable network condition and smallRTTs. In contrast, in cellular networks (Fig. 12b), Conduitand Pyramid show poor stability, with 5× and 14× higherstd. than POI360. This is because the large and unstable RTTof cellular networks makes the end-to-end ROI update laggy,
0
0.2
0.4
0.6
0.8
1
0 2 4 6 8 10
CD
F
Short-term comp. level variation (std.)
POI360ConduitPyramid
(a) ROI region compressionlevel variation over wireline.
0
0.2
0.4
0.6
0.8
1
0 2 4 6 8 10
CD
F
Short-term comp. level variation (std.)
POI360ConduitPyramid
(b) ROI region compressionlevel variation over cellular.
Figure 12: Short-termROI compression level variation(std. value of ROI compression level in a 2s window).causing the �uctuating compression level in the actual ROI.The problem is especially severe for Conduit, as it only has 2compression levels, thus ROI shifting triggers unacceptablevideo quality oscillation between the high/low levels. Usingmore compression levels, Pyramid smooths out the transi-tion, thus achieving better stability than Conduit. However,its rigid spatial compression mode still cannot �t all condi-tions in the highly dynamic cellular networks. In contrast,POI360 achieves the best stability as it can always adapt tothe compression mode best �tting current network condi-tion.
360◦ video frame delay.We now analyze the frame-leveldelay performance collected from the same set of experi-ments. Fig. 13a plots the CDF of frame delay over the wirelinenetwork. We can observe that POI360’s compression algo-rithm achieves the lowest video delay. Furthermore, in cellu-lar networks (Fig. 13b), POI360 achieves a median delay of460ms, 15% less than Conduit which aggressive compression,because its adaptive design can switch to more aggresivecompression modes than Conduit under bad network condi-tion, or prioritize the video quality when the network allows.In other words, POI360’s better ROI quality and stabilitycome without sacri�cing the crucial delay performance. Thisdelay is also comparable to the conventional video telephonydelay over cellular networks measured in existing works[46].We emphasize that end-to-end frame delay is not frame in-terval, as shown in Fig. 2, e.g., a video stream can be delayedby 460ms while keeping a 36FPS frame rate. Pyramid encod-ing has higher video delay than Conduit, as it compressesthe non-ROI regions less aggressively. In contrast, POI360strikes a balance between video quality and delay: When theend-to-end ROI update is responsive, it can aggressively cropand stream only the high-quality ROI like Conduit; Whenthe ROI update becomes sluggish, it can distribute the videoquality more evenly across the frame like Pyramid encoding.
Video frame freezing ratio. Based on the video framedelay, we then compute the frame freezing ratio, the mostcrucial user experience metric. We de�ne freezing ratio (FR )as the percentage of video frames that experience higher than600ms delay. Fig. 14a shows that all algorithms work prop-erly in wireline networks, with FR below 2%, and POI360has the lowest FR of 0.6%. However, in cellular networks
CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea Xiufeng Xie and Xinyu Zhang
0
0.2
0.4
0.6
0.8
1
0 200 400 600 800 1000
CD
F
Video Frame Delay (ms)
POI360ConduitPyramid
(a) Frame delay (wireline).
0
0.2
0.4
0.6
0.8
1
0 200 400 600 800 1000
CD
F
Video Frame Delay (ms)
POI360ConduitPyramid
(b) Frame delay (cellular).
Figure 13: 360◦ video frame delay under POI360 com-pression and benchmark algorithms.
0
4
8
12
16
20
POI360 Conduit Pyramid
Vid
eo
Fre
eze
Ra
tio
(%
)
(a) Freeze ratio (wireline).
0
4
8
12
16
20
POI360 Conduit Pyramid
Vid
eo
Fre
eze
Ra
tio
(%
)
(b) Freeze ratio (cellular).
Figure 14: 360◦ video freeze ratio under POI360 com-pression and benchmark algorithms.(Fig. 14b), both Conduit and Pyramid encoding fail with anunacceptable FR of 8% to 17%. In contrast, POI360’s adap-tive compression tames the FR below 3% under the networkdynamics.
In sum, POI360’s adaptive compression substantially out-performs the benchmark algorithms in terms of video quality,stability and delay, especially in cellular networks with longRTT and highly dynamic network conditions. However, thereis still a performance gap between cellular and wireline , es-pecially in terms of freezing ratio. In what follows, we showhow POI360’s cross-layer rate control helps bridge this gap.
6.1.2 Firmware Bu�er Aware Congestion Control. Recallthat POI360’s FBCC module aims to improve the cellularuplink bandwidth utilization by pushing the �rmware bu�eroccupancy to a proper level. To verify its e�ectiveness, werepeat the panoramic video telephony sessions with eachlasting for 200 seconds for both POI360 over FBCC andPOI360 over GCC—the default WebRTC rate control usedin the foregoing experiments. Both con�gurations share thesame application-layer dynamic panoramic video compres-sion module in §6.1.1. The scatter plot in Fig. 15 showsthe �rmware bu�er level and the per-second uplink TBS(throughput) collected across the panoramic video telephonysessions. We split this �gure into 3 regions: (i) the low usageregion with uplink throughput below 2 Mbps; (ii) the highusage region where the uplink throughput exceeds 2Mbpsand keeps increasing with the bu�er level; (iii) the overuseregion where the uplink throughput no longer increases withthe �rmware bu�er level as the uplink is saturated. It is clearthat FBCC maintains the bu�er level around the “sweet spot”in the high usage region that is far from congestion but highenough to secure su�cient bandwidth utilization. In con-trast, GCC’s bu�er occupancy stays at the low usage region
0 1 2 3 4 5
0 5 10 15 20UL T
BS
/s (
Mbps)
Firmware Buffer Level (KByte)
GCCFBCC
Low usage High usage Overuse (saturation)
Figure 15: FBCC is allocated higher uplink TBS thanGCC as it maintains a proper �rmware bu�er level.
0
1
2
3
4
5
FBCC GCC 0
2
4
6
8
10
Th
rou
gh
pu
t (M
bp
s)
Vid
eo
Fre
eze
Ra
tio (%
)
Mean ThroughputVideo Freeze Ratio
(a) Video freeze ratio.
0
0.2
0.4
0.6
0.8
1
Bad Poor Fair Good EXC
PD
F
FBCC
GCC
(b) Video quality (MOS).
Figure 16: ComparingPOI360 performancewith FBCCor GCC (WebRTC’s default rate control).for a substantial fraction of the samples, indicating that itseverely underutilizes the network bandwidth.
We proceed to evaluate the impact of the transport layerrate control on the performance of panoramic video tele-phony. Following the same experimental setup discussed atthe beginning of this microbenchmark, we compare POI360running over FBCC against POI360 running over legacy GCC.From Fig. 16a, we can observe that the throughput of bothcon�gurations are almost identical at around 3Mbps, as theyshare the same compression algorithm which compressesthe panoramic video stream bitrate to such a level below thecurrent cellular uplink bandwidth.
However, the throughput variation di�ers a lot. Speci�-cally, GCC’s throughput has a 1.2Mbps standard deviation,57% higher than FBCC. This unstable throughput is mainlycaused by its ine�cient adaptation logic, which exploresthe network bandwidth by gradually increasing the sendingrate, and reacts to the bandwidth overuse by sharply throt-tling the tra�c bitrate. In contrast, FBCC converges to thenetwork bandwidth quickly as it can directly estimate thecellular uplink bandwidth based on the �rmware bu�er leveland TBS following Eq. (5) when the bu�er status indicates acongestion, and then responsively throttles the rate to theestimated bandwidth. As a result, the extra cross-layer in-formation helps FBCC achieve a low FR of around 1.6% overcellular networks, which is signi�cantly lower than the FR ofGCC (4.7%), and is even close to the wireline case (Fig. 14b).The stability of throughput also translates into high videoquality. Fig. 16b shows the MOS quality computed basedon the PSNR collected from the same experiment. We seethat FBCC has much higher percentage of video frames withgood (69%) and excellent (23%) quality than GCC. In contrast,
POI360 CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea
0
10
20
30
40
50
Idle CH Busy CH 0
2
4
6
8
10
PS
NR
(d
B)
Vid
eo
Fre
eze
Ra
tio (%
)
Mean PSNRVideo Freeze Ratio
(a) Freeze ratio & PSNR withdi�erent background load.
0
0.2
0.4
0.6
0.8
1
Bad Poor Fair Good EXC
PD
F
Light background loadHeavy background load
(b) Video quality (MOS) withdi�erent background load.
0
10
20
30
40
50
Weak Moderate Strong 0
2
4
6
8
10
PS
NR
(d
B)
Vid
eo
Fre
eze
Ra
tio (%
)
Mean PSNRVideo Freeze Ratio
(c) Freeze ratio & PSNR withdi�erent signal strength.
0
0.2
0.4
0.6
0.8
1
Bad Poor Fair Good EXC
PD
FWeak signal(-115dBm)
Moderate signal(-82dBm)
Strong signal(-73dBm)
(d) Video quality (MOS) withdi�erent signal strength.
0
10
20
30
40
50
15mph 30mph 50mph 0
2
4
6
8
10
PS
NR
(d
B)
Vid
eo
Fre
eze
Ra
tio (%
)
Mean PSNRVideo freeze ratio
(e) Freeze ratio & PSNR at dif-ferent driving speed.
0
0.2
0.4
0.6
0.8
1
Bad Poor Fair Good EXC
PD
F
Slow driving(15mph)
Urban driving(30mph)
Highway driving(50mph)
(f) Video quality (MOS) at dif-ferent driving speed.
Figure 17: System-level POI360 evaluation.GCC has more than 40% of the frames with only fair quality,as its throughput frequently vibrates to low levels.
6.2 System Level EvaluationWe now integrate POI360’s design components and conducta system level evaluation under various network conditionsthat lead to di�erent level of network bandwidth and ROIfeedback latency.
Di�erent background tra�c load. Since POI360 runson cellular networks with the uplink being the typical bot-tleneck, we verify its resilience under di�erent uplink tra�cload within the same cell. We conduct the experiments intwo campus environments: (i) light tra�c load environmentduring early morning when most users are o� campus; (ii)heavy tra�c load environment in the noon just after class,when most roam around campus with their cell phones. Allexperiments are performed at the same static location to ruleout the in�uence of signal strength and mobility.
Fig. 17a plots the video freezing ratio FR and PSNR. We ob-serve that the overall FR is only around 1% under light tra�cload. Even during the busiest hour, the freezing rate is onlyaround 4%, and PSNR drops by 2 dB. Under light tra�c load,the MOS metric shows a relatively larger fraction of frameswith excellent quality. But even with heavy competing traf-�c, majority of the frames are in either excellent or goodcondition, and none is poor or bad condition. Therefore, ourresults demonstrate that POI360 is robust to the background
tra�c load. Such robustness owes to its awareness of cellularlink statistics like the uplink �rmware bu�er level, and itsability to e�ectively maintain an adequate uplink rate bymanipulating the bu�er level.
Di�erent LTE channel quality. Since POI360 aims torealize the ubiquitous capturing and streaming of 360◦ video,it is important to understand the impact of the location-dependent cellular link quality on its performance. To createdi�erent channel qualities, we place our prototype at di�er-ent locations and evaluate the performance under di�erentlevels of received signal strength (RSS): (i) weak signal in-side a concrete parking garage, where the RSS is −115dBas reported by the phone’s �rmware. (ii) moderate signal atan outdoor parking lot partially shadowed by a tall build-ing (−82dB RSS). (iii) Strong signal in an open parking lotwithout tall buildings nearby (−73dB RSS). The experimentsare conducted during weekends when the cellular channelis mostly idle.
Fig. 17c shows the resulting PSNR and video freeze per-formance. We �rst see that the FR is not a�ected much bythe RSS. Even under weak channel, the FR is below 3%. How-ever, this comes at the cost of lower video quality (Fig. 17d).Under a weak channel, none of the frames show excellentquality, whereas the strong channel has 31% of the frameswith excellent quality. The di�erent trends for video freezeand video quality can be explained by the stability of RSS.Although the channel can be weak which forces POI360 touse low video quality, as long as the RSS does not �uctuate,POI360’s rate control can always converge to the networkbandwidth and stably stream the video without freezing.
Di�erent mobility level. One potential application ofPOI360 is to enable 3D experience sharing through dronesor inside moving vehicles. Di�erent mobility levels in turna�ects the LTE network condition. To evaluate the impact,we place our experimental platform inside a moving vehiclewith 3 di�erent speed con�gurations: (i) 15 mph, a typicalslow driving speed within the residential area; (ii) 30 mph,driving along a road in an urban area; (iii) 50 mph, drivingalong a highway. We repeat each test 10 times along the sameroute. During the experiments, the phone is under seamlessLTE coverage.
Fig. 17e shows the performance in terms of FR . We seethat slow driving has almost no impact on FR compared withthe static experiments. But FR increases to 7% under citydriving speed and 9% on the highway. Fig. 17f further plotsthe MOS video quality computed based on the PSNR, wecan observe that although the video quality decreases underhigher mobility level, even under the speed of 50 mph, allframes still have either excellent (20%) or good (80%) quality,which mainly owes to the less building blockage and thushigher RSS (typically −60dB) around the highway test route.
CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea Xiufeng Xie and Xinyu Zhang
7 RELATEDWORKThe past decade of research in 360◦ video delivery focusedon the encoding aspect [4], i.e., mapping the original sphereview into various planar formats [8, 10, 11, 26, 31]. The con-cept of splitting an 360◦ frame into tiles and only deliveringa fraction of the tiles based on the viewer’s ROI has beeninvestigated [33] to ensure high ROI resolution under certainbandwidth and latency constraint. To date, 360◦ video stream-ing mainly deliver pre-captured videos on demand, whichcan tolerate second-level bu�ering latency [20]. Throughsimulation, Qian et al. [30] veri�ed the feasibility of deliver-ing on-demand 360◦ videos through cellular networks.
Driven by portable 360◦ cameras, real-time immersiveexperience sharing is emerging as a new class of applications.Through a 360◦ camera mounted on a drone [22, 29], headset[25], or handheld device [34], real-time 360◦ scenes can beteleported to remote users, who can experience �rst-person360◦ view with a VR HMD. Microsoft’s Holoportation [27]represents the state-of-the-art in streaming 360◦ videos fortelepresence. However, current Holoportation requires 30-50Mbps bandwidth, and only works within Wi-Fi range.
Ultimately, 360◦ videos call for a more agile adaptation tothe network conditions than conventional video as its heavytra�c is more sensitive to the bandwidth variations. Pro-prietary video chat systems commonly build their own ratecontrol atop of UDP [46], and measurement studies revealsevere performance limitations, such as sluggish response tobandwidth change [5] and frequent freezing [46] especiallyover cellular networks. Recently, the IETF chartered the RM-CAT working group [17] to standardize congestion controlfor real-time media. Google Congestion Control (GCC) [12]has been a leading proposal in RMCAT, and acts as the me-dia transportation framework in mainstream browsers likeChrome and Firefox. Alternative proposals [47, 48] regulatesending rate based on explicit congestion noti�cation (ECN),but require ECN support on intermediate routers. In recentLinux kernel, Byte Queue Limits (BQL) can avoid starvationand unnecessary queuing for the driver queue [37]. But ithas no access to the cellular �rmware bu�er level or the linkquality. In [44], Xu et al. proposed a machine learning modelto predict cellular network performance based on historicalthroughput/latency, which, however, still needs at least oneRTT to detect a network change as an end-to-end solution.
Parallel to the interactive media research, on-demandvideo streaming protocols are converging towards the HTTPbased dynamic adaptive streaming (DASH) [38]. The keyissue is to accurately estimate the available bandwidth (bymonitoring playback bu�er level [16] or historical through-put [19], etc.), and then choose a video rate below it. Recently,piStream [43] proposed to adapt the video bitrate accordingto LTE downlink resource utilization. Such on-demand videostreaming systems leverage the large client bu�er of several
seconds to absorb bandwidth estimation inaccuracy. In con-trast, interactive video has orders of magnitude tighter delayconstraints , thus calls for a responsive rate control design.
8 DISCUSSION & FUTUREWORKROI prediction. (i) For rendering local 360◦ video, motion-based ROI prediction [2, 21] helps smooth the display quality.However, such prediction only works reliably at a short timescale. In typical use, the average head angular velocity is60 deд/sec while acceleration can go up to 500 deд/sec2 asreported by Oculus [21], i.e., the head position after 120 msis unpredictable, which is below the typical video latencyover LTE. (ii) For on-demand 360◦ video multicast, recentwork [3] suggests using the motion patterns learnt from pre-vious users to predict the ROI of current user, as users per-form similar motion patterns given the same video content.However, 360◦ video telephony is di�erent from on-demandstreaming, as the content is generated in real time. Thereforethere are no existing contents to learn from.
Improving the ROI update responsiveness. AlthoughPOI360 smooths out the ROI quality transition by adaptivecompression, it still takes the cellular RTT before the qualityof new ROI converges. The key to reduce this convergencetime is shortening the end-to-end network path. In current4G LTE network, tra�c still goes to the Internet even bothends of the video telephony connect to the same BS. In futureworks, mobile edge computing can be used to enable therelaying at the edge BS, thus signi�cantly shortens the pathand accelerate the quality convergence of POI360.
9 CONCLUSIONAlthough on-demand 360◦ video streaming for recorded con-tents has been studied extensively, real-time interactive 360◦video telephony remains mostly unexplored. In this paper, weinvestigate the feasibility of realizing interactive 360◦ videotelephony services over LTE cellular networks, and presentour readily-deployable solution POI360, which employs adap-tive compression to balance the tra�c load reduction anduser-perceived video quality. POI360 also features a novelcellular-link-informed rate control algorithm to boost theresponsiveness to potential uplink congestion. Finally, a real-time prototype of POI360 is implemented, which validatesour design in various network conditions. We believe thiswork can inspire new applications that bring the interactivevideo services to the 360◦ world.
ACKNOWLEDGMENTSWe appreciate the anonymous reviewers for their insightfulcomments. This research was supported in part by a GoogleFaculty Research Award, and by the NSF under Grant CNS-1506657, CNS-1518728, CNS-1343363 and CNS-1350039.
POI360 CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea
REFERENCES[1] Aakash Patel and Gregory Rose. 2015. Conduit: E�cient Video Com-
pression for Live VR Streaming. (2015). https://avp.github.io/conduit/[2] Ronald Tadao Azuma. 1995. Predictive Tracking for Augmented Real-
ity. In Ph.D. dissertation, University of North Carolina at Chapel Hill.[3] Yanan Bao, Tianxiao Zhang, Amit Pande, Huasen Wu, and Xin Liu.
2017. Motion-Prediction-Based Multicast for 360-Degree Video Trans-missions. In IEEE SECON.
[4] J. Chakareski. 2013. Adaptive Multiview Video Streaming: Challengesand Opportunities. IEEE Communications Magazine 51, 5 (2013).
[5] Luca De Cicco, Saverio Mascolo, and Vittorio Palmisano. 2011. SkypeVideo Congestion Control: An Experimental Investigation. ComputerNetworks 55, 3 (2011).
[6] Eduardo Cuervo and David Chu. 2016. Poster: Mobile Virtual Realityfor Head-mounted Displays With Interactive Streaming Video andLikelihood-based Foveation. In Proceedings of ACM MobiSys.
[7] Evgeny Kuzyakov and David Pio. 2016. Next-generation video encod-ing techniques for 360 video and VR. (2016). https://code.facebook.com/posts/1126354007399553/next-generation-video-encoding
[8] Facebook. 2016. Cubemap Transform Open Source Code. (2016).https://github.com/facebook/transform
[9] Facebook. 2016. Facebook 360 Video. (2016). https://facebook360.fb.com
[10] Facebook. 2016. Next-generation video encoding techniques for360 video and VR. (2016). https://code.facebook.com/posts/1126354007399553/next-generation-video-encodin
[11] V. R. Gaddam, M. Riegler, R. Eg, C. Griwodz, and P. Halvorsen. 2016.Tiling in Interactive Panoramic Video: Approaches and Evaluation.IEEE Transactions on Multimedia 18, 9 (2016).
[12] Google. 2015. A Google Congestion Control Algorithm for Real-TimeCommunication. https://tools.ietf.org/html/draft-alvestrand-rmcat-congestion-03. (2015).
[13] Yihua Guo, Feng Qian, Qi Alfred Chen, Zhuoqing Morley Mao, andSubhabrata Sen. 2016. Understanding On-device Bu�erbloat for Cellu-lar Upload. In Proceedings of the 2016 ACM on Internet MeasurementConference.
[14] Stefan Holmer, Mikhal Shemer, and Marco Paniconi. 2013. HandlingPacket Loss in WebRTC. In Image Processing (ICIP), 2013 20th IEEEInternational Conference on. IEEE.
[15] Junxian Huang, Feng Qian, Yihua Guo, Yuanyuan Zhou, Qiang Xu,Z. Morley Mao, Subhabrata Sen, and Oliver Spatscheck. 2013. AnIn-depth Study of LTE: E�ect of Network Protocol and ApplicationBehavior on Performance. In Proc. of ACM SIGCOMM.
[16] Te-Yuan Huang, Ramesh Johari, Nick McKeown, Matthew Trunnell,and Mark Watson. 2014. A Bu�er-based Approach to Rate Adaptation:Evidence from a Large Video Streaming Service. In Proc. of ACMSIGCOMM.
[17] IETF. 2015. RTP Media Congestion Avoidance Techniques (RMCAT).https://datatracker.ietf.org/wg/rmcat/documents/. (2015).
[18] IETF/W3C. 2015. WebRTC. (2015). https://webrtc.org/[19] Junchen Jiang, Vyas Sekar, and Hui Zhang. 2012. Improving Fairness,
E�ciency, and Stability in HTTP-based Adaptive Video Streamingwith FESTIVE. In Proc. of ACM CoNEXT.
[20] Ngo Quang Minh Khiem, Guntur Ravindra, and Wei Tsang Ooi.2011. Towards Understanding User Tolerance to Network Latency inZoomable Video Streaming. In Proceedings of the ACM InternationalConference on Multimedia.
[21] Steven M LaValle, Anna Yershova, Max Katsev, and Michael Antonov.2014. Head Tracking for The Oculus Rift. In Robotics and Automation(ICRA), 2014 IEEE International Conference on.
[22] Nick Lavars. 2016. Exo360 Drone Shoots VR Content on the Fly. (2016).http://newatlas.com/exodrone360-vr-content/43854/
[23] Yuanjie Li, Chunyi Peng, Zengwen Yuan, Jiayao Li, Haotian Deng, andTao Wang. 2016. MobileInsight: Extracting and Analyzing CellularNetwork Information on Smartphones. In ACM MobiCom.
[24] M. Makar, A. Mavlankar, P. Agrawal, and B. Girod. 2010. Real-TimeVideo Streaming With Interactive Region-of-Interest. In IEEE Interna-tional Conference on Image Processing (ICIP).
[25] Mariella Moon. 2016. 360�y Puts 4K Video Cams onHelmets. (2016). https://techcrunch.com/2016/10/06/oculus-shows-o�-facebook-messenger-video-calls-in-virtual-reality/
[26] Khiem Quang Minh Ngo, Ravindra Guntur, and Wei Tsang Ooi. 2011.Adaptive Encoding of Zoomable Video Streams Based on User AccessPattern. In Proceedings of the Annual ACM Conference on MultimediaSystems (MMSys).
[27] Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, WayneChang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L. David-son, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, CharlesLoop, Qin Cai, Philip A. Chou, Sarah Mennicken, Julien Valentin,Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli,Yuliya Lutchyn, Cem Keskin, and Shahram Izadi. 2016. Holoporta-tion: Virtual 3D Teleportation in Real-time. In Proceedings of ACMSymposium on User Interface Software and Technology (UIST).
[28] Derek Pang, Sherif Halawa, Ngai-Man Cheung, and Bernd Girod. 2011.Mobile Interactive Region-of-interest Video Streaming with Crowd-driven Prefetching. In Proceedings of International ACM Workshop onInteractive Multimedia on Mobile and Portable Devices.
[29] Ben Popper. 2016. Drones and Virtual Reality Headsets are a DeliciousCombination. (2016). http://www.theverge.com/2014/11/24/7274997/parrot-bebop-drone-virtual-reality
[30] Feng Qian, Lusheng Ji, Bo Han, and Vijay Gopalakrishnan. 2016. Op-timizing 360 Video Delivery over Cellular Networks. In Proceedings ofthe Workshop on All Things Cellular.
[31] Ngo Quang Minh Khiem, Guntur Ravindra, Axel Carlier, andWei Tsang Ooi. 2010. Supporting Zoomable Video Streams with Dy-namic Region-of-interest Cropping. In Proceedings of the ACM SIGMMConference on Multimedia Systems (MMSys).
[32] Richoh. 2016. Theta S 360 Mobile Camera. (2016). https://theta360.com/en/about/theta/s.html
[33] Patrice Rondao Alface, Jean-François Macq, and Nico Verz-ijp. 2012. Interactive Omnidirectional Video Delivery: A Bandwidth-E�ective Approach. Bell Labs Technical Journal 16, 4 (2012).
[34] Adam Ryder. 2016. VR Action Camera Field Test: RicohTheta S and 360Fly 4K. (2016). http://www.popphoto.com/hands-on-with-two-360deg-action-cameras
[35] Jihoon Ryoo, Kiwon Yun, Dimitris Samaras, Samir R. Das, and GregoryZelinsky. 2016. Design and Evaluation of a Foveated Video StreamingService for Commodity Client Devices. In Proceedings of the Interna-tional Conference on Multimedia Systems (MMSys).
[36] Sayandeep Sen, Syed Gilani, Shreesha Srinath, Stephen Schmitt, andSuman Banerjee. 2010. Design and Implementation of an "Approx-imate" Communication System for Wireless Media Applications. InSIGCOMM.
[37] Dan Siemon. 2013. Queueing in the Linux Network Stack. LinuxJournal (2013).
[38] Iraj Sodagar. 2011. The MPEG-DASH Standard for Multimedia Stream-ing Over the Internet. In IEEE Multimedia.
[39] Xinding Sun, J. Foote, D. Kimber, and B. S. Manjunath. 2005. Region ofInterest Extraction and Virtual Camera Control Based on PanoramicVideo Capturing. IEEE Transactions on Multimedia 7, 5 (2005).
[40] John Tan. 2015. WebRTC Mobile: Facebook and Slack Are In. (2015).https://www.sinch.com/opinion/webrtc-facebook-google-�refox/
[41] Umlaeute. 2017. v4l2loopback - A Kernel Module to Create V4L2Loopback Devices. (2017). https://github.com/umlaeute/v4l2loopback
CoNEXT ’17, December 12–15, 2017, Incheon, Republic of Korea Xiufeng Xie and Xinyu Zhang
[42] Keith Winstein, Anirudh Sivaraman, and Hari Balakrishnan. 2013.Stochastic Forecasts Achieve High Throughput and Low Delay overCellular Networks. In USENIX NSDI.
[43] Xiufeng Xie, Xinyu Zhang, Swarun Kumar, and Li Erran Li. 2015.piStream: Physical Layer Informed Adaptive Video Streaming OverLTE. In ACM MobiCom.
[44] Qiang Xu, Sanjeev Mehrotra, Zhuoqing Mao, and Jin Li. 2013. PRO-TEUS: Network Performance Forecast for Real-time, Interactive MobileApplications. In Proc. of ACM MobiSys.
[45] YouTube. 2016. Uploading 360-degree Videos. (2016). https://support.google.com/youtube/answer/6178631?hl=en
[46] Chenguang Yu, Yang Xu, Bo Liu, and Yong Liu. 2014. “Can you SEEme now?”: A Measurement Study of Mobile Video Calls. In Proc. ofIEEE INFOCOM.
[47] Jing Zhu, R. Vannithamby, C. Rodbro, Mingyu Chen, and S. Vang An-dersen. 2012. Improving QoE for Skype video call in Mobile BroadbandNetwork. In Proc. of IEEE Global Communications Conference (GLOBE-COM).
[48] Xiaoqing Zhu and Rong Pan. 2013. NADA: A Uni�ed CongestionControl Scheme for Low-Latency Interactive Video. In InternationalPacket Video Workshop (PV).
