Embed Size (px)
Citation preview
0
White Paper
Probabilistic Approach to Provisioning of Resources for Delivery of Interactive TV
Synopsis
This paper discusses how to provision network and computer resources needed to provide the services
delivered by streaming multimedia platform. Using this information, network operators can now estimate
the cost-benefits of implementing centralized interactive services with Interactive TV streaming
processor. A detailed proprietary spreadsheet model was developed on the basis of this paper. The
foundation for this model is discussed.
By: Amos Kohn
Table of Contents
Introduction ..................................................................................................................................... 1
System Architecture .................................................................................................................... 1
Analysis Approach ...................................................................................................................... 2
User Behavior ................................................................................................................................. 3
Peak Usage .................................................................................................................................. 3
Application Usage ....................................................................................................................... 4
Impact of User Interactivity ........................................................................................................ 5
Data Packaging ............................................................................................................................... 5
MPEG Characteristics ................................................................................................................. 5
Frame Rate .................................................................................................................................. 6
Statistical Multiplexing ............................................................................................................... 6
Display Area for Streaming Media ............................................................................................. 7
Bit Rate Generation......................................................................................................................... 7
Forward Path Data Transport ...................................................................................................... 9
Return Path Data Transport .......................................................................................................... 10
Overview ................................................................................................................................... 10
Motorola .................................................................................................................................... 10
Processing Requirements .............................................................................................................. 12
Validation ...................................................................................................................................... 14
Financial Model ............................................................................................................................ 14
Key Issues ................................................................................................................................. 14
Views ........................................................................................................................................ 14
ROI Calculations ....................................................................................................................... 14
Typical Results.............................................................................................................................. 15
Conclusions ................................................................................................................................... 16
Need for Model ......................................................................................................................... 16
How Often Is the Peak Exceeded? ............................................................................................ 17
Overall Model Uncertainty ....................................................................................................... 17
Appendix A ................................................................................................................................... 18
Appendix B ................................................................................................................................... 20
1
Introduction
Digital delivery of TV content offers a fundamental change in the way the industry should view a
“channel.” Instead of one channel equals one service, digital transmission offers the concept of
one channel equals N services. Since an interactive TV channel is a limited resource for most
network operators, they must deploy a delivery method that can provide the most services to their
subscribers at the least cost.
A comparison for effective utilization of a TV channel is the revenue obtained from Video on
Demand (VOD). Each movie consumes 3.75 Mbps of bandwidth and generates approximately
$4 over a two-hour period. This represents $2 per channel per hour in revenue. Thus, making a
channel available for interactive content needs to generate more than $2 per hour in direct plus
indirect benefits to make business sense.
The model developed on the basis of this paper estimates the specific computer and network
resources needed to deploy Interactive TV streaming processor as well as the expected cost and
revenue.
System Architecture
As illustrated in Figure 1, the interactive TV platform is comprised of Set-top Box proxy
Headend processor servers, an Interactive TV Operation Manager, and a small software module
that is downloaded into the set-top box. The main function of the software downloaded to the set-
top box is to collect keystrokes generated by the user and forward this data to the headend. The
headend processes the signals and returns an MPEG-2 stream to the set-top box where it is
decoded and displayed on the subscriber’s TV.
Internet
IP BackboneEdge QAM
Interactive TV
Operation Server
Set-top Proxy
MPEG and Media
Processor
MPEG/IP to
QAM Modulator
MSO
Back Office
Return Path
Equipment
MPEG/UDP/IP
SPTS’s
Data
Router
Headend Primary HUB
RF
Co
mb
ine
r
Settop Box
MP
EG
& A
ud
io
Ele
me
nta
ry Stre
am
HFC Network
RF
RF
Settop Box
OOB & Return
Path Equipment
Local
Interactive
App Server
UD
P/IP
Pa
ckets
OoB Data
HF
C D
istribu
tion
Ne
two
rk
Firewall
Application
Server
OoB Data
Figure 1: End-to-End System Architecture
More services
per channel
offer higher
revenue
potential per
channel.
Uniform
content
programming
for display
on any set-
top box.
Interactive TV streaming processor supports any WEB content, browser plug-ins, and
executables. It employs a software approach to processing real-time digital MPEG video and
audio. By supporting variable bit rate encoding, session switching, and load balancing, each set-
top box extender can support about 24 simultaneous sessions for the situation we considered.
Any content or application type is encoded by the Interactive TV streaming processor system to
MPEG video and digital audio streams. These streams are converted from elementary streams
into multiple unicast Single Program Transport Streams (SPTS) and then encapsulated into IP
packets for transport over an IP network to regional headends or distribution HUBs.
Interactive TV streaming processor also supports multiplexed program transport streams and
QAM modulation where MPEG streams are directly taped into the cable distribution plant. The
software downloaded into the set-top boxes satisfies three important requirements: a) it can reside
in any set-top box, b) it requires a small memory footprint, and c) it sends user keystrokes back to
the headend.
Analysis Approach
This paper is divided into the following areas:
User behavior: The average user is characterized in terms of how many use each
application and how many minutes per day they use that application. This information is
used to estimate peak simultaneous usage which establishes the design condition.
Data packaging: The approach THE DELIVERY SYSTEM uses to package MPEG
images onto the network is discussed. Frame rate, compression, and multiplexing are key
issues. Understanding these issues helps optimize bandwidth utilization.
Bit rate generation: The bit rate for each application is determined by understanding the
behavior and characteristics of MPEG and by measuring key parameters during test runs
of the application. The resulting data provides the basis for a model of the traffic
generated by the headend and transported through the network.
Forward path data transport: To familiarize the reader with the key configuration issues,
the path from the headend to the set-top boxes is reviewed.
Return path data transport: The user interacts with the headend through the “return
path” network. Keystrokes from the subscriber’s remote control are captured by the set-
top box and then transported through the return path back to the headend. Although this
traffic is relatively small, it requires analysis to achieve a good user experience.
Processors: Set-top Box proxy Headend processor server processing requirements vary
significantly from one application to the other. This section discusses how test data is
used to estimate the number of Set-top Box proxy Headend processor server needed to
produce the MPEG image streams.
Financial results: With a model of the required computer and network resources, the
model can compute the capital cost of a deployment. ROI, cash-flow, and other financial
parameters are also computed.
A detailed analysis of the return path for Motorola is presented in Appendix A.
The model developed is presented in Appendix B.
Determine
what services
the typical
user is likely
to choose.
Understand
how the
data flows
through the
network.
Compute the
resources
needed,
associated cost,
and then
perform
financial
analysis.
The focus in the remainder of this paper is to discuss each of these bulleted topics in detail.
User Behavior
Peak Usage
A delivery system must be designed to handle reasonably likely demands from subscribers. The
system should satisfy the demands most of the time, but not all the time, because to do so is
unnecessarily expensive. For example, if 500 households are connected to one node, the node
needs to provide acceptable quality of service to the subscribers who are likely to use the service
at the busiest time. It is not necessary to design the system to service 500 simultaneous users
because that is very unlikely to occur. However, a peak of 5 (i.e., 1%) simultaneous users might
be a reasonable number for provisioning purposes. If more users try to connect, the system
should either degrade gracefully or deny further logons.
To gain further insight into the concept of peak simultaneous users, consider the following
assumptions: 500 users randomly log onto the system throughout the day and each user is active 5
minutes per day. In this case, there would be an average of 1.7 (=500*5/[24*60]) people logged
on simultaneously. What is the peak utilization in this case? A Poisson model, which is the
workhorse of queuing theory, estimates the number of simultaneous users at the 95-percentile
(two standard deviations) to be about 5 people which is 1% of the 500 subscribers.
From the previous paragraph we can infer the meaning of peak usage as the value that is
exceeded no more than 5% of the time. From the ratio of 5/1.7, we can also infer that the peak is
approximately 3 times the average value. These are useful numbers to keep in mind when
performing a provisioning analysis and they are fundamental to our model.
We reviewed data from one of Interactive TV Operation Manager’s customer sites where
Interactive TV streaming processor is installed on a free trial basis. We found the peak usage to
be 0.89% of the subscribers who had signed up for the Interactive TV streaming processor
service. The average log-on time was 17 minutes. If each subscriber logged on once every three
days (i.e., 5 minutes per day on the average), the measured data are close to the Poisson model
discussed in the previous two paragraphs.
Figure 2 provides further insight into the usage pattern at a typical site. This figure shows the
number of simultaneous sessions over five minutes intervals. The average number of
simultaneous sessions is 13 and the maximum is 29; thus, the peak-to-average is 2.2.
Figure 2: Five Minutes Averages of Number of Sessions for a Typical 24 hour Period
Peak and
average
utilizations are
related by a
Poisson
model.
Peak
simultaneous
usage
establishes the
key design
condition.
Figure 3 shows the number of simultaneous sessions for two hour averages over the period of a
month. The average is 11 and the maximum is 24; thus, the peak-to-average is 2.2.
Figure 3: Two Hour Averages of Number of Sessions for a Month Period
We believe that the peak-to-average represents the behavior of the user population while the
average logon time is a reflection of the user’s satisfaction with the products. There is good
reason to believe that increasing customer satisfaction (i.e., increasing logon time) will result in
more revenue. Thus, the network operators do not mind adding capacity to satisfy increasing
average demand, but they prefer to minimize buying spare capacity to satisfy peak demand.
The above discussion provides justification that the peak-to-average is a practical concept
because it has a relatively stable value. Furthermore, since the average value from one day to the
next is quite repeatable, it validates the use of peak simultaneous usage as a percent of the number
of subscribers because it is a fairly constant value. Another observation is that the Poisson model
presented above appears to be conservative relative to the observed data.
Application Usage
The model asks the system operator to estimate the total number of subscribers, the take rate for
the services from each content provider, and the number of minutes per day that the average
subscriber might use these applications. With this information, the model computes the average
number of simultaneously active users. By using the peak-to-average approach presented in the
previous section, the peak number of simultaneous users is computed.
To determine the resources consumed during peak conditions, two alternative approaches are
available:
If the processor and bandwidth requirements are known, these values can be specified as
inputs.
If the plug-ins used to render the applications are known, the model can use this
information to compute the processor and bandwidth requirements.
Since many content providers offer a bundle of applications, it is necessary to either specify a
bundle average resource requirement as estimated by the content provider, or to ask the network
operator to specify their estimate of usage pattern for individual portions of the bundle.
Interactive TV streaming processor employs variable bit rate algorithms to produce MPEG video
and AC3 audio streams. The output bit rate is directly affected by the degree to which the user
interacts with the application. For instance, applications that have higher screen update rates will
produce higher bit rates when encoded, and the applications that use persistent audio will
generate higher bit rates then the ones that use intermittent audio.
Since users will run a number of applications during their sessions, at any given point in time the
accumulated output bit rate will be defined by the application mix across all active sessions. By
estimating the application mix and the number of simultaneous sessions, we estimate the output
bit rate.
To represent the impact of the wide range of resources required by the different applications, we
group the applications into categories based on the plug-ins that are used. These categories allow
us to establish the relationship between the media the applications are built with and the expected
output bit rate. Using this approach, we estimate the bit rate for any given application mix. This
approach reduces the overall uncertainty in the results compared to an approach that only uses
average bit rates and CPU values.
The model computes the average resource requirements per Interactive TV Operation Manager
subscriber based on take rate, multiplied by the length of time the average user spends with each
application, multiplied by the resources associated with the selected applications. The results are
representative estimates of both processor and bandwidth requirements.
Impact of User Interactivity
An active user produces more keystrokes than slow users. This has an impact on the traffic on
the return path. This is modeled by estimating the average rate of keystrokes associated with
each application.
All applications are interactive in the sense that they can be started, stopped, and terminated by
user interactions. Many applications (particularly games) have the added feature of users sending
keystrokes back to the Headend processor server to move the applications forward. Thus, slow
users consume fewer resources than fast users. We used data obtained from what we believed
was a representative tester.
Data Packaging
Interactive TV streaming processor packs the MPEG streams tightly by using a combination of
compression, multiplexing, low frame rate, and small display area. We found that the average
session generates approximately 523 Kbps of video plus audio. This is 14% of the 3.75 Mbps
bandwidth required by a VOD stream through a dedicated channel. This shows that Interactive
TV streaming processor is lean in its use of bandwidth. The following explains how this is
accomplished.
MPEG Characteristics
Interactive TV streaming processor produces a compressed MPEG-2 stream consisting of two
types of frames:
I: Intraframe frames. These are encoded without reference to another frame.
P: Predictive frames are encoded using the previous frame as reference. P frames are
much smaller than I frames.
To achieve high compression, only the differences between successive frames are encoded
instead of the frames themselves. One image is constructed as a “prediction” based on the
previous image. To increase the compression, more P frames are used. To do that, the GOP
Understanding
MPEG
compression
helps you
select good
input
parameters.
(“Group of Pictures”), which represents the distance between I frames, is used. For example, if
GOP is 6, the video stream looks like: I P P P P P I P P P P P I P P . . .
Frame Rate
The frame rate and GOP are determined by the application developers. For the applications we
tested, 8 fps and 13 fps were typical. A GOP of 20 was used for all these applications. This low
frame rate is used because most of the screen images have no reference point in terms of
rendering speed (e.g., the images in game applications have no reference in terms of “live”
speed). The industry standard for movie delivery is 29.97 fps.
Statistical Multiplexing
The traditional approach to providing VOD is to allocate a fixed bandwidth (e.g., 3.75 Mbps)
channel to each stream to achieve high quality. However, an interactive application that uses
video transport as a delivery method does not need all the allocated bandwidth most of the time
because there are typically few changes in the images. For optimum utilization of the bandwidth,
compression is used to send several streams through the same bandwidth and “time-slice”
(statistically multiplex) several simultaneous streams.
Figure 4 shows conceptually how data from three sessions (green, gray, and orange) are
statistically multiplexed into a channel bandwidth that is less than the combined peak bit rate of
the incoming streams.
Figure 4: Squeezing of Offered Bit Rate into Limited Bandwidth
If several streams require high bandwidth at the same time, the multiplexer determines how to
deal with that need. It can delay frames by slicing (buffering) I frames into a few smaller frames,
it can drop frames, or it can instruct the encoder module on the Set-top Box proxy Headend
processor server to increase the compression rate or reduce the frame rate (future
implementation). Any of these actions is performed automatically based on pre-established
priorities.
By these mechanisms, Interactive TV streaming processor can operate at a high utilization of the
allocated bandwidth. It can run up to 90% of the allocated bandwidth, on the average, before
requesting additional bandwidth. As a result, our estimates of the needed QAM resources are
based on the average bit rate, plus a small amount (e.g., 10%) for headroom and 1 Mbps for
overhead.
If all the streams were movies, the multiplexing would need to be conservative in terms of
allocating bandwidth to ensure delivery of live motion. However, for data, games, and other
Low frame
rate is used to
regulate
output
intensity.
More streams
through fixed
bandwidth
results in
more services
at same cost.
Channel Bandwidth
Channel Bandwidth
Portion of frame delayed until next frame
Multiplexing
is used to
package the
MPEG
frames into a
small
bandwidth.
streams where live action is not involved, Interactive TV Operation Manager has adopted a less
resource-intensive approach. Most of the streams produced by Interactive TV streaming
processor consist of sequences of user interface images that do not have a reference with respect
to speed of display. Thus, the user experience will be good as long as the interactive
responsiveness of the system is comfortable to the user.
Display Area for Streaming Media
Streaming media applications delivered by Interactive TV streaming processor displayed on one
quarter of the available TV screen generate a video bit rate of approximately 600 Kbps. This
compares to 3.75 Mbps generated by regular VOD. This factor of 6 difference is primarily due to
Interactive TV Operation Manager’s frame rate being about half and the screen display area being
one quarter.
Bit Rate Generation
We measured the bit rates for many applications by packet capture. Inspection of the bit rates
generated by these services revealed the following.
Audio
Audio streams generated on Interactive TV streaming processor are encoded with 128 Kbps of
payload based on the Dolby standard. By selecting the mono option, the audio payload can be
reduced by 50%. When the audio stream is encoded, 5 Kbps of MPEG headers are added. When
the audio becomes a TCP/IP stream, an additional 15 Kbps of headers are added. The overall
result is a 148 Kbps audio stream when measured on an Ethernet LAN. Some applications have
intermittent audio wherein the audio stream temporarily stops when no sound is needed.
Video
Based on a combination of packet captures on the Ethernet network and understanding the
various plug-ins used to generate the applications that we investigated, we developed the results
in the Table 1.
Table 1: Overview of How Bit Rates Depend on Plug-ins for NTSC Deployments
Category Plug-ins Update Rate Interactive TV
Operation Manager
Frame Rate
Typical Bit Rate
Thick
plug-ins
for games
Media Player,
Quick Time,
Real Player,
executables
15 fps or 24 fps.
Steady 128 Kbps audio.
Limited to 13 fps.
Usually steady audio.
Video typically updates
¼ of the screen.
Video: 400 – 500 Kbps
Audio: 148 Kbps
Total avg: 550 Kbps
Thick
plug-ins
Same as
above.
Same as above. Same as above. Same as above.
Flash Macromedia’s
Flash Player
Between 0 to 30 fps
depending on application
developer.
Steady 128 Kbps audio.
Limited to 13 fps.
Usually steady video
and audio.
Video: 400 – 500 Kbps
Audio: 148 Kbps
Total avg: 570 Kbps
Thin Flash Macromedia’s
Flash Player
Between 0 to 15 fps
depending on application
developer.
Intermittent 128 Kbps
Limited to 8 fps.
Intermittent video and
audio.
Video: 0 – 250 Kbps
Audio: 0 - 148 Kbps
Total avg: 300 Kbps
To estimate
bandwidth
needs, it is
necessary to
determine bit
rate for each
plug-in.
audio.
Thin plug-
ins
HTML,
DHTML, Java,
JavaScript, …
Between 0 to 8 fps
depending on the
application.
Intermittent 128 Kbps
audio depending on app.
Limited to 8 fps.
Intermittent video and
audio.
Video: 0 – 250 Kbps
Audio: 0 to 148 Kbps
Total avg: 260 Kbps
Mixed
plug-ins
For example,
Flash and Java
Intermittent video and
audio.
Video: 0 – 500 Kbps;.
Audio: 0 to 148 Kbps
Total avg: 240 Kbps
Figure 5 shows the cumulative bytes flowing from an Set-top Box proxy Headend processor
server to a MUX for a typical instance of menu browsing. The plot to the right is an expansion of
10 seconds of data. The following features are of interest:
There are seven small (2 seconds) gaps caused by the tester taking short breaks. This
behavior justifies our approach of first determining the maximum bit rate that can be
associated with the service and then multiplying by the speed of user interactivity to
determine the effective bit rate.
The vertical sections of the “staircase” behavior are due to I frames that appeared every
2.5 seconds when the tester was working within one menu.
There is one I frame after every 19 P frames; i.e., GOP=20. Each I frame is
approximately 40 Kbytes while the P frames typically range between 500 bytes to 2,000
bytes.
The average throughput was 190 Kbps while the bit rate generated by the application
reached as high as 230 Kbps. This implies that the user’s degree of interactivity was 83%
(i.e., “thinking time” was 17%). Furthermore, 14% of that thinking time was made up of
the 7*2=14 seconds mentioned in the first bullet item listed above.
Figure 5: Example of Bit Rate Time Sequence for Typical Menu Browsing
Based on detailed analysis of packet captures from more than 20 applications for the four
categories of plug-ins we are dealing with, we developed the following model to estimate the total
bit rate per second:
(P + I / GOP) * fps + audio
Packet capture
provides a
view of the I
and P frames.
where P and I are parameters with the values shown in Table 2 and fps is the frame rate. GOP is
usually defaulted to 20 by Interactive TV streaming processor for NTSC deployments. Parameter
P is the impact of the P frames and I is the impact of the I frames. The audio values are computed
as average over a typical session. This model provides “high” estimates that include the margin
needed to account for the variability in the bit rates. A margin is needed to obtain a conservative
estimate of the QAM resources needed to deliver these streams.
Table 2: Bit Rate Model for Different Plug-ins
Category Frame
Rate
Parameter
P
Parameter
I
Video Bit
Rate; Kbps
Audio;
Kbps
Total;
Kbps
Thick plug-ins 13 16 350 436 113 549
Flash 13 16 350 436 130 566
Thin Flash 8 10 360 221 75 296
Thin plug-ins 8 10 330 209 50 259
Mixed plug-ins 8 10 330 209 24 233
Forward Path Data Transport
Figure 6 is an overview of the forward path of the delivery network. Engineering experience is
needed to determine a particular configuration. Interactive TV Operation Manager provides
recommendations based on discussions with their clients. Here are three common configurations.
Basic system transport: This is the easiest configuration to provision. All Set-top Box
proxy Headend processor servers, MUXs, and QAMs are installed at a central headend. The
services are generated, delivered, modulated, and taped onto the network at that location.
This results in the most efficient use of bandwidth because no long haul transport and third-
party QAM overhead are involved.
Figure 6: Overview of Forward Path Delivery of Interactive TV Operation Manager
Services
RF
QAM MUX Set-top proxy
Generation
Transport
Distribution
Content
IP IP IP
Set-top Proxy located at Headend.
MUXs may be at headend or may be distributed. MUX and QAM may be at same or different locations.
Content may be at headend and/or remote locations.
500 to 1,500 homes per Node.
If location of QAM is distributed, it is called “Edge QAM” and the location is a HUB. There may be many QAMs per HUB.
Transport is across IP network until images reach a QAM which converts them to RF signals for distribution through the cable network. Switches are used on IP network if distributed architecture is
involved. STXs encode the content into MPEG-2 images that are “packaged” by MUXs that send data to QAMs for modulation onto cable.
Transport stream over GigE: The MPEG images produced by the Set-top Box proxy
Headend processor server are delivered across a Gigabit Ethernet (GigE) infrastructure to an
“Edge QAM” located at remote distributed HUBs. Interactive TV streaming processor
supports many third-party QAM types and vendors.
Session transport over GigE infrastructure: This is the most general case in which any
number of streams is delivered across GigE networks to any number of distributed QAMs.
Each stream headed to a particular HUB can be transported in either of two ways:
o SPTS streams that consume a fixed bandwidth (approximately 1.0 Mbps) large
enough to accommodate the variable bit rate of a single MPEG session.
o MPTS streams that consume a partial or a full QAM rate to accommodate the
variable bit rate of multiple sessions. Interactive TV streaming processor can also
use the VOD model where it acquires a QAM rate of 7.5 Mbps (2*3.75) and use this
fix bandwidth for multiplexing multi sessions in variable bit rate. If more bandwidth
is required to accommodate additional Interactive TV streaming processor sessions,
increments of 3.75 Mbps can be added to the initial 7.5 Mbps bandwidth by sending a
bandwidth request inquiry to the session resource manager.
Return Path Data Transport Overview
Interactive TV streaming processor uses the return path network to transport interactive TV
keystrokes from the subscriber’s set-top box back to the network operator’s headend. The
subscriber sends keystrokes to the set-top box from a remote control or wireless keyboard. This
communication goes from the set-top box to the headend through the return path. Depending on
the return path technology and the network topology, the network operators only need to allocate
a small portion of the available return-path bandwidth capacity to interactive TV service. It is
important to estimate the return path traffic associated with typical use of the application to
adequately provision the requisite amount of return-path bandwidth.
Interactive TV streaming processor is deployed over several industry standard return path
systems: ALOHA (for Motorola), DAVIC (for Scientific Atlanta and DVB), and DOCSIS (both
US and Europe). Each system requires a different implementation analysis, with Motorola being
the most limiting in terms of available bandwidth.
Motorola
Figure 7 shows an overview of the return path for the Motorola system. The traffic on the return
path is generated as follows: A subscriber initiates traffic by pressing keys on a remote control
device. This creates payload packets that are carried by 65.5 bytes ATM cell transport from the
set-top box to a demodulator card (DM 1000). Up to seven payload packets can be carried by one
cell. These packets are repackaged into 76 bytes UDP/IP packets for transfer to the network
controller (NC 1500). The NC 1500 produces an acknowledgement for each of these packets.
The ACK goes through the out-of-band modulator (OM 1000) back to the set-top box.
Figure 7: Motorola Up-Stream System
The key performance parameters associated with Motorola are summarized in Table 3. Appendix
A contains a detailed discussion. The key throughput limitation is imposed by the ALOHA
network which is used to transport data from the set-tops to the DM cards. The maximum
theoretical throughput for each of these cards is 18% of their 256 Kbps capacity. The
recommended design limit is 13.5%. The effect is that each DM card can support a maximum of
33 keystrokes per second. If each user generates one keystroke per second, this implies that 33
users on the same node will reach the design limit of one DM card. To handle more simultaneous
users, the network operator can do one of two things: a) add a frequency to the existing cable
network by adding more DM cards to the existing infrastructure, or b) break the network into
more nodes.
Table 3: Key Performance Parameters for Motorola
Performance Parameter Comments
Return path data packets from set-
tops to DM cards.
524 bits in each packet. Low-end remotes allow max of 3
packets per second per user.
Capacity of return path cable per
DM card.
Max theoretical throughput of 256*18% = 46 Kbps. Max
throughput for design calculations: 13.5% = 34.56 Kbps
Return path Demodulator
RPD 2000
Each DM card supports 66 packets of 524 bits per second.
6 DM cards per RPD was standard. 8 cards are used in
newer versions.
Network Controller
NC 1500
Each NC 1500 supports up to 32 RPDs and a total of
50,000 users.
Out-of-Band Modulator
OM 1000
Each OM can support ~20,000 homes with data from
headend to set-tops at max speed of 2.048 Mbps.
Motorola has a
low capacity
return path that
requires attention
to details to
assure acceptable
solution.
Return Path Traffic
The bit rate through the return path depends on the speed by which the users press the buttons on
the remote controls and how fast the return path is able to carry the traffic. Unfortunately, it is
difficult to characterize the average user and it is just as difficult to measure this traffic and
compute meaningful aggregate values. Thus, we chose to take a conservative approach of
computing values that we believe are on the high end of what will be seen on the average. Table
4 summarizes our recommended values for a Motorola environment. These values are inputs to
the model; i.e., they can easily be changed.
Table 4: Suggested Keystroke Values for Motorola Environment
Category Keystrokes per second Resulting Bit Rate
Thick plug-ins; games 2 2096 bps
Thick plug-ins 0.25 262 bps
Flash 0.5 524 bps
Thin Flash 1 1048 bps
Thin plug-ins 0.5 524 bps
Mixed plug-ins 0.25 262 bps
Most applications that are rendered with thick plug-ins do not involve much interaction by the
subscribers; however, high-end games are noticeable exceptions. High-end games are typically
also associated with high user interactivity (i.e., as many keystrokes as the system allows). The
bit rates associated with these applications can be input as special cases.
Another issue is the effect of “stuffing” (i.e., more than one packet in one ATM cell) which
occurs when a subscriber presses the remote control buttons in rapid sequence. The effect is that
the network traffic does not increase beyond its capacity (e.g., 1.5 keystrokes/sec) but the speed
by which the application can move forward may increase by the degree of stuffing.
Processing Requirements
The model contains measured values of average Set-top Box proxy Headend processor server
processor consumption for more than 30 applications. It is not necessary to measure every new
application that comes available in the future. However, when a significant group of applications
are added, it is advisable to either ask the content provider or a tester to provide reasonably
accurate measurements of the resource requirements.
The test machine was a single motherboard, 2.4 GHz P4 processor. By testing many applications,
and by understanding the various plug-ins used to generate the applications that we investigated,
we developed the results in the Table 5. This table shows our extrapolation from the test
computer to the typical deployment Set-top Box proxy Headend processor server (3.2 GHz with
hyper-threading). In summary, we found the following:
Applications produced by high-end plug-ins (Media Player, Quick Time, Real Player and
Flash) consume the largest amount of processing resources. A substantial amount of
processing is needed to decode the original streaming format and then encode this into
MPEG format.
Most games on Interactive TV streaming processor use Flash as the plug-in. Flash
consumes a medium amount of processing resources; however, the developers have many
options to choose among to produce their applications. Furthermore, the developers can
To estimate
processor
needs, it is
necessary to
determine
processor
consumption
per
application.
stop both the video and audio streams while waiting for user inputs. Thus, Flash
applications have a wide statistical variation.
Applications developed with thin plug-ins consume the least amount of processing
resources due to the relatively simple format of the source.
The simple average CPU consumption for a single motherboard Set-top Box proxy
Headend processor server for the applications for which we had data was 12%.
Table 5: Processor Consumption For Different Plug-ins
Categories
Set-top Box proxy
HE processor
server
Processor
Consumption1
Estimated Average
Thick plug-ins; games 8% - 60% 30%
Thick plug-ins 8% - 60% 23%
Flash 13% 20% 17%
Thin Flash 5% - 20% 8%
Thin plug-ins 1% - 17% 7%
Mixed plug-ins 1% - 20% 4%
Note 1: Estimated for single motherboard 3.2 GHz processor with hyper-threading.
Figure 8 shows the Set-top Box proxy Headend processor server utilization for a single
motherboard processor as a function of time for an application written in HTML. The averaging
interval was 1 second. For the data in Figure 8, the average value is 4%.
Figure 8: Example of Set-top Box proxy Headend processor server Utilization as Function
of Time for a Typical Game
Looking at a plot like Figure 8, the session startup and session breakdown consume about 30% of
the CPU for about 5 seconds. By knowing the average length of time of the user sessions, as
discussed in the User Behavior section, we compute the contribution of these bursts in CPU
consumption.
To estimate the processor requirements for several simultaneous applications, ideally a
probabilistic aggregation of individual probabilities would be performed. In practice, that
is more effort than is warranted due to the high uncertainty in the data. Thus, we add the
average values and then assume a processor efficiency of 95%, and we allocate 35%
CPU Utilization as Function of Time
0
5
10
15
20
0 50 100 150 200 250 300
Time in seconds
% C
PU
uti
lizati
on
Information
about
variability in
processor
consumption
is needed.
“headroom” for variabilities and uncertainties to provide sufficient processor resources to
accommodate randomly changing resource needs.
Validation
We compared our results with tests where we added users until the users felt that their
applications slowed down due to cpu congestion. We chose 12 different applications; mostly
games rendered by thin-Flash plug-ins. The average number of simultaneous sessions for both
the tests and the model calculations were 12 for a double-motherboard Set-top Box proxy
Headend processor server. The standard deviation for the difference between the two tests was 2
sessions; i.e., 18%. This is consistent with the belief that the overall uncertainty in the data is
about 20%. Thus, we consider this to be a good validation of our approach.
Financial Model Key Issues
From the network operator’s point of view, the main business considerations associated with
deploying Interactive TV streaming processor are:
Financial: Is there a compelling return on the investment for this deployment? The
model developed on the basis of this paper provides calculations to help answer these
questions.
Network: What, if any, upgrades of the existing network are needed? Our model
provides information needed to make these decisions. Experienced engineers from both
Interactive TV Operation Manager and the network operator need to conduct joint
discussions to make final decisions.
Views
To make it easy for system operators to begin the analysis, the following modular approach was
developed:
Customer view: A summary view that focuses on specifying information about the
subscribers and the applications offered to these subscribers. The calculations are
initially based on an aggregate model. As the engineering team refines the system
specifications, the accuracy of the results will improve.
Financial view: Information related to content subscription fees, advertising, software
license, and capital equipment costs.
Engineering view: Detailed information about bandwidth and processor requirements.
The inputs for this view are usually prepared by Interactive TV Operation Manager
engineering staff in cooperation with their customers.
ROI Calculations
From a business perspective, it is important to estimate the return on the investment. A payback
period of less than 18 months is a common expectation in the cable TV industry. The cumulative
cash flow plot below shows how much, and when, cash has to be provided.
System
operators focus
on financial
returns and
impact on their
network.
Different
views for
different
people.
Figure 9 shows an example of revenue, expenses, and cash flow. A provisioning case typically
begins at year 0 with a significant capital outlay representing the initial purchase of equipment.
Revenue starts at year 1 followed by increasing slope due to an expected growth in number of
subscribers. We also assign a growth factor to the expenses.
Figure 9: Example of Financial Projections
Projections never fully match reality. Thus, it is necessary to perform sensitivity analysis to
identify the key parameters that may affect the financial projections. The uncertainties in take
rate and length of time per application dominate the uncertainties in the financial results.
Revenue and expenses are approximately a linear function of the number of subscribers. The
primary non-linear effect is the potential cost of changing the cable plant (e.g., providing fiber
close to the subscriber homes). Although those costs cannot typically be justified only by the
benefits of interactive TV, a bundle of services might provide the necessary justification.
Typical Results
Figure 10 shows typical results produced by the model. These results are of interest to network
operators who are considering a Interactive TV streaming processor deployment. The focus is on
summarizing the key financial, subscriber, and resource results that are needed to make a decision
to proceed on a serious evaluation of Interactive TV streaming processor.
Figure 11 shows the key results related to the network traffic on the return path. This information
helps the network operators decide if the return traffic can be accommodated by the current
network, or if capacity expansion is required.
ROI considers
length of
payback, cash
flow, and
return on
investment.
Figure 10: Screenshots of Typical Summary Results
Figure 11: Screenshots of Typical Return Path Results
Conclusions
Need for Model
A provisioning model is a prerequisite for cost-effective roll-out of interactive TV. Our model
estimates the resources (Set-top Box proxy Headend processor servers, MUXs, QAMs, etc.)
needed to deliver the specified services and it computes the key financial numbers associated with
such deployments.
With a model, network operators can compare interactive TV products with other services in
terms of cost, revenue, and impact on their networks. For example, they can compute the cost of
installing Interactive TV streaming processor as well as the expected revenue per channel. They
A model
provides
insight,
illuminates
key issues,
and helps
estimate the
cost benefit of
deployment.
can also determine which services are consuming resources and compare that with their
individual revenue contributions.
How Often Is the Peak Exceeded?
There will occasionally (5% of the time) be situations where the peak simultaneous usage is
beyond the provisioned capacity. In those cases, the network will slow the users down so they all
can share the available resources. If too many people try to log on, Interactive TV streaming
processor will deny service.
If exceeding capacity 5% of the time is too frequent, the system operator can increase the peak-
to-average input parameter and then redo the calculations prior to concluding the provisioning
analysis.
Overall Model Uncertainty
The dominant uncertainty is associated with the user behavior. It is not known with certainty
what services the users will select, the length of time they will interact with these services, or the
intensity by which they will interact with these services. However, by analyzing user behavior
over time, these uncertainties can be reduced.
A major objective of our model is to reduce the uncertainties in the deployment decisions. In our
experience, a straightforward, aggregated approach, with no details in terms of application mix
and probabilities is associated with an accuracy in the range of -50% to +200%; i.e., if you
estimate number of Set-top Box proxy Headend processor servers to be 100, the reality I expected
to be somewhere between 50 and 200. However, with the model discussed in this paper, we
believe the uncertainties are reduced to the range of -20% to +30% for a given specification of
user behavior. Uncertainty in user behavior will obviously increase the overall uncertainty.
Uncertainties
are part of
any business
decision.
How
often are
design
conditions
exceeded?
Appendix A
Detailed Analysis of Motorola Return Path
This Appendix explains the upstream capacity limitations inherent in Motorola implementations.
Log-on Traffic
When a user activates a service, there is a rapid exchange of a few packets between the set-top
box and the Set-top Box proxy Headend processor server at the headend. Five packets are
transmitted from the set-top within a second or two. When many users log in at the same time,
several packets may stack together into a single cell that is transported through the upstream path.
RF Traffic between Set-Top Box and RPD
ATM adaptation layer 5 (AAL-5) is the method for delivering packets over the upstream network.
The ATM cell size in the return path, including the Aloha protocol headers, is 65.5 bytes (524
bits) instead of an ordinary 53 bytes ATM cell. If there are no packet collisions in the return path,
every key press transmitted from the set-top box will be contained in one cell of 524 bits. An
additional 524-bit packet is created when the button on the remote control is released. However,
when several users on the same RF network press keys at the same time, up to seven keystrokes
can be aggregated in one ATM packet. Likewise, if a user presses the remote control buttons in
rapid sequence, up to seven messages can be contained in one packet which would represent more
than 524 bits of information. For design purposes, we assume that 35% of the packets carry five
keystrokes. For the remote control device we tested, the time between the first and second packet
was about 300 ms.
The capacity of the network between set-top boxes and RPDs is 256 Kbps. The Aloha protocol
for transporting the packets provides a maximum throughput of 18% of the 256 Kbps available
capacity. In practice, 13.5% is the recommended maximum for design calculations. Loads
beyond this create high rates of collisions which decrease the effective throughput. So, for
planning purposes, each demodulator (DM) card can only accept 66 ATM cells per second. Since
one remote control button press followed by a release (this combination defines one keystroke)
produces two packets, the result is that a maximum of 33 keystrokes per second (worst case; i.e.,
no combined keystrokes) can be supported by each DM card. The DM cards convert the 65.5
byte packets into 76 bytes for transmission from the DM card to the network controller (NC),
which repacks them into 66 bytes UDP/IP packets for transmission across the application network
in the headend.
Throttling of Remote Control
Our testing, using a handheld remote control, showed that Motorola was not able to accept
keystrokes faster than once every 700 ms which throttled the number of packets to 3 per second
per set-top box. This is a limitation in the handheld remote control devices for Motorola set-top
boxes. Other input devices (e.g., keyboard and wired devices) are faster. The performance
limitations in the input devices are believed to be configurable.
Traffic through DM Cards
Based on the discussion in the previous paragraph, the maximum traffic generated by one user on
the Aloha network is 3 packets/sec * 524 bits/packet = 1.5 Kbps. Most users will be much slower
because they need time to read the screen. Thus, the expected traffic is more likely to be one-
third to one-half of the maximum (i.e., they may each generate 1 to 1.5 packets per second or 0.5
to 0.7 kbps). If we consider a situation where the peak number of simultaneous users is 2% of
500 households, this results in 15% to 22% of the available capacity for one DM card. In the
worst case, each DM card can serve a maximum of 22 simultaneous Interactive TV streaming
processor interactive TV users.
Traffic through Network Controller
On the path between the RPD and the NC, the network is running at 10 Mbps. If all active users
are generating the maximum number of keystrokes (3 packets/sec), that part of the network can
support 5,482 users:
10 Mbps / (3packets/sec * 76 bytes/packet * 8 bits/byte) = 5,482 users
This is such a high number that it will almost certainly not be a constraint.
The network controller (NC 1500) sends an acknowledgement back to the set-top box for each
packet it receives from the RPD. This ACK is carried in a 188 byte MPEG packet together with
other data and then modulated by the out-of-band modulator (OM 1000) and sent back to the set-
top box through the downstream path.. We estimate the incremental contribution for this packet
due to the ACK to be 94 bytes (=188/2). In addition to this traffic, the OM broadcasts a small
amount of network traffic associated with control, synchronization, and electronic program guide
data to all the households connected to the OM.
Traffic through Out-of-Band Modulator
Since each OM usually supports about 20,000 homes, the ACKs can add a noticeable amount of
traffic on the forward path on the out-of-band network. Each OM 1000 has a bandwidth of 2.048
Mbps. If we assume that 200 (i.e., 1% of the homes) simultaneous users are supported by one
OM, and if we assume they all are using the system at maximum speed, the bit rate generated by
the acknowledgements is 451 Kbps:
3 packets/sec * 94 bytes/ACK * 8 bits/byte * 200 simultaneous users = 451 Kbps
This represents 22% of capacity. Since most users do not interact at maximum speed, the traffic
will most likely be one-half or one-third of this value.
Appendix B
Model of Resource Utilization
This section presents the key formulas we use in the model resulting from this study. The focus
is on the essential issues. Various special cases and small effects are ignored in this paper to
avoid distracting the reader. Separate documentation exists for the spreadsheet implementation of
the model.
Inputs
P = peak to average value
N = # of Interactive TV Operation Manager subscribers (modified to be number of set-top boxes
when there are more than one set-top per home),
n = number of applications,
a1, a2, a3, …, an = applications (services) available for selection,
t1, t2, t3, …, tn = length of time (e.g., minutes) for each application per day per subscriber divided
by length of a day (e.g., 24*60),
r1, r2, r3, …, rn = take rate; i.e., percent of total Interactive TV Operation Manager subscriber that
subscribe to these applications,
Set-top Box proxy Headend processor server (ai) = Set-top Box proxy Headend processor server
requirement for rendering and encoding application ai on reference Set-top Box proxy Headend
processor server,
Forward.path.bit.rate(ai) = bit rate generated by Set-top Box proxy Headend processor server for
application ai for the forward path.
Forward Path Bit Rate
We have developed the following relationship between MPEG parameters and the resulting bit
rate as measured on an Ethernet wire between Set-top Box proxy Headend processor server and
MUX. All traffic is TCP/IP.
Typical size of I-frames: 45,000 to 60,000 bytes
Typical size of P-frames: 500 to 2,000 bytes
# of P-frames per I-frame = GOP -1 = 19
# of frames per second: usually 8 or 13
Audio bit rate (for NTSC): 148 Kbps but sometimes zero for apps with intermittents
audio, and
Estimated bit rate: Isize/Ifps + Psize/Pfps + audio
where:
Ifps = (# of I frames per second) / GOP
The actual traffic will be a little higher than this if the user switches between services because
such switching forces generation of more I-frames.
The frame sizes were determined by packet capture on a switch between the Set-top Box proxy
Headend processor server and MUX. The IP headers of 58 bytes should only be included when
the numbers are used to compute traffic on Ethernet. The headers are removed by the MUX. The
reduction in packet size by removing the headers is 7.5%.
Return Path Bit Rate
Return.path.bit.rate (ai) = bit rate associated with application ai for the return path produced by
the user clicking his remote control.
The upstream bit rate on the return path depends on the following factors:
The maximum speed that the system is willing to accept inputs from the user. As
discussed in this paper, 1.5 keystrokes per second (equivalent to 1,572 bps) is a common
upper limit. One keystroke is defined as one button press followed by one button release.
When the return path network is congested due to many simultaneous users or when the
user rapidly presses the buttons on their remote controls, the result is that up to 7
keystrokes can be packed in one ATM cell from the set-top to the RTD. We have
developed a flexible model that can be adjusted to relevant tests when available. On the
average, the effect is that the return traffic is reduced by about half compared to the case
where there is no packing.
Average Consumption per Active User
Average SET-TOP BOX EXTENDER SERVER processor consumption per Interactive TV
Operation Manager subscriber:
i=n
Average consumption of STB proxy Headend processor server = Σ [ri * ti * HE STB Proxy (ai)]
i=1
Average bit rate produced by Set-top Box proxy Headend processor server computers for
transport through the forward path per Interactive TV Operation Manager subscriber:
i=n
Average.forward.path.bit.rate = Σ [ri * ti * Forward.path.bit.rate(ai)]
i=1
Average bit rate produced by user interactions for transport through the return path between home
and RPD per Interactive TV Operation Manager subscriber:
i=n
Average.return.path.bit.rate = Σ [ri * ti * Return.path.bit.rate (ai)]
i=1
As discussed in the body of this paper, the return path traffic differs slightly, depending on what
part of the path we are dealing with, as a result of changes in packet size as the packets move
from the homes to the demodulator to the network controller.
Aggregate Results
Total Set-top Box proxy Headend processor server requirements = P * N * Average.STB PROXY
SERVER.consumption * (1 + headroom) / (2 * efficiency)
This value is used to compute the number of Set-top Box proxy Headend processor servers
needed. The reason for dividing by 2 is to account for the testing being performed on a single
motherboard Set-top Box proxy Headend processor server while deployed Set-top Box proxy
Headend processor server have dual motherboards. Furthermore, the number of Set-top Box
proxy Headend processor server is scaled by knowing the clock speed of the deployment Set-top
Box proxy Headend processor server relative to that of the test Set-top Box proxy Headend
processor server. The estimates of number of Set-top Box proxy Headend processor servers are
rounded up to nearest integer.
Total bit rate produced by Set-top Box proxy Headend processor server computers for forward
path: Total.forward.path.bit.rate = P * N * Average.forward.path.bit.rate
This is the total aggregated bit rate produced by all simultaneous sessions in the forward
direction. In a distributed system implementation, this bit rate represents the total amount of
traffic introduced into the backbone.
To estimate the number of QAMs needed to support the generated forward bit rate, we first need
to select the QAM being used which is determined by the following table:
Bandwidth Constellation
64 256
NTSC (US) 6 MHz 27 Mbps 38.8 Mbps
PAL (Europe) 8 MHz 38.4 Mbps 51 Mbps
We subtract 1 Mbps for overhead and another 10% to account for bursting beyond average. We
then compute as follows:
Number of QAMs = Total.forward.path.bit.rate/effective.QAM.capacity
effective.QAM.capacity = (QAM.constellation – 1Mbps)*0.9
where QAM constellation is as shown in the table above.
The total upstream return path bit rate is as follows:
Total.return.path.bit.rate = P * N * Average.return.path.bit.rate
There are a few other components that need to be computed to represent a complete system.
Examples are: number of LANs, number of firewalls, etc. However, these issues are of minor
importance in the high-level discussion in this paper.
Cost
The total system price is determined by multiplying the computed number of Set-top Box proxy
Headend processor servers, MUXs, RPDs, and other devices by their respective prices and
summing the results.
Glossary
Aloha: A network transport protocol.
Application: A computer program producing display
on cable TV.
Bit rate: Bits per second of network traffic.
DAVIC: Digital Audio Visual Council (located in
Switzerland). DAVIC is creating the industry
standard for end-to-end interoperability of broadcast
and interactive digital audio-visual information, and
of multimedia communication.
Digital subscriber: Household that subscribes to
cable service and which has a connection that can use
digital signals.
DOCSIS: Data over Cable Interface Specification.
An industry standard that defines how cable modems
communicate over a cable television line.
Ethernet: Networking of computers in a LAN using
copper cabling. Ethernet can handle 10Mbps, 100
Mbps, or 1Gbps and can be used to connect almost
any kind of computer.
Frame Rate: Number of updates of TV screen per
second.
GigE: Gigabit Ethernet network.
GOP: Group of Pictures; distance between I
frames.
Headend: Electronic control center -- generally
located at the antenna site of a cable TV system --
usually including antennas, preamplifiers, frequency
converters, demodulators, modulators and other
related equipment which amplify, filter and convert
incoming broadcast TV signals to cable system
channels.
HFC: Hybrid Fiber Coax cable.
HUB: Location in the cable network where the signal
is changed from digital to RF.
I frames: Intraframes in video streams; i.e., reference
frames. They usually contain a large number of bytes.
IR: Infrared. Used to send signals from a handheld
remote control device to a set top box.
IRR: Internal rate of return on investment.
iTV: Interactive TV.
Keystroke: One press followed by one release of a
button on a remote control. The press and the release
create one data packet each.
LAN: Local area network.
MPEG: Moving Picture Experts Group. Standard for
digital video and audio compression.
MPTS: Multiple Program Transport Stream.
MUX: Multiplexer.
NC: Network controller.
Node: Location in the cable network where coax
cable splits into branches.
OM: Out-of-band modulator.
OOB: Out-of-band.
Payback period: Time to generate revenue equal to
the cost of capital equipment.
Peak simultaneous usage: Maximum number of
homes that simultaneously use a service during a
period of 5 minutes. To avoid once-in-a-lifetime
situations, the peak is that case that occurs no more
than 5% of the time.
Poisson model: Statistical model to represent
processes of random arrivals.
QAM: Quadrature amplitude modulation. Modulates
an MPEG bit stream onto a cable for transmission to a
cable TV.
RF: Radio frequency.
ROI: Return on investment.
Router: Network device for routing traffic based on
IP address.
RPD: Return path demodulator.
Service: A computer program producing display on
cable TV.
Session: The activity resulting from one user being
connected to an Set-top Box proxy Headend
processor server. Number of sessions is equal to
number of users logged on.
SPTS: Single Program Transport Stream.
Stream: Data flowing across a network associated
with one user being connected.
STB PROXY SERVER: Processor that run in a
remote server behalf of a Set-top Box.
Subscriber: Household that is paying for service of
interest.
Switch: Network device for routing traffic based on
IP address.
Take rate: Percent of subscribers who are paying for
specific services; in this case Interactive TV
Operation Manager services.
TCP: Network protocol to transport data across a
network. TCP is “reliable” in the sense that lost
packets are resent if they are not acknowledged by the
receiver.
VOD: Video on demand.
UDP: Network protocol to transport data across a
network. UDP does not require acknowledgement. WAN: Wide Area Network.
About the author
Amos Kohn was Vice President of Solutions Engineering at ICTV Inc. until 2006. He has more then 15
years of multi-national executive management experience in convergence technology development,
marketing & business strategy and solutions engineering at Telecomm and new media emerging
organizations. Prior to joining ICTV Inc., Amos Kohn held a senior position at Liberate Technologies and
Golden Channels.
.
About Interactive TV Streaming Processor
The Interactive TV streaming processor platform enables network operators to deliver interactive content
and applications, including animation, high-fidelity audio and full-motion video, to any digital set-top
box. Using headend-based processors and the existing digital plant, Interactive TV streaming processor
removes set-top box and middleware constraints on the delivery of interactive multimedia content and
allows interactive TV that cannot be matched by satellite solutions.
With Interactive TV streaming processor application Servers, digital subscribers can interact with
multimedia applications developed with standard PC, Web, and Java tools using existing set-top boxes
and remote controls. Interactive TV streaming processor supports the latest versions of HTML,
JavaScript, Java, Windows Media Player, Flash, RealPlayer and QuickTime, enabling network operators
to provide subscribers with an array of content and applications.
