Gluster Software Defined Storage for

Media and Entertainment

Workflows

© 2018, Coughlin Associates

Coughlin AssociatesCoughlin Associates

ã 2018 Coughlin Associates,

1

Gluster Software Defined Storage for Media and Entertainment Workflows Tom Coughlin, Coughlin Associates The size and amount of content in media and entertainment workflows will continue to grow. Higher resolutions, frame rates and dynamic range as well as the growth in multi-camera projects will swell the data that media workflows generate and which must be stored for current and future use. The demand for more storage must be matched against storage budgets that are flat or only increasing by a small amount annually. Infrastructure experts in the industry are looking for new ways to handle data that are secure, provide needed performance and are cost effective. One approach that many industries faced with this dilemma have taken to meet these needs are using software defined storage. Let’s look at whether and where software defined storage can be effectively used in modern media and entertainment workflows. Media and Entertainment Workflows A general feature of digital media content is that as the content resolution increases, the data rate and resulting capacity demand increases as well. For instance, going from raw uncompressed content with 24 fps, 8-bit color HDTV to 60 fps, 10-bit color 4K UHDTV can require 12 X more storage capacity. Raw uncompressed 8K UHDTV at 120 fps, 12-bit color can be 120 X more. In addition to working with higher resolution 4K and even 8K content, media and entertainment professionals are also using higher frame rates and greater dynamic range. Video professionals also tend to capture more content with today’s digital recording technologies than they ever did with film and older recording. An annual survey conducted by Coughlin Associates looked at many factors that impact video content growth. One of these was hours of video shot for an hour of completed work. Figure 1 shows the results of this survey on hours shot. In 2017 46% of the survey participants said that they captured 6 hours or more of original content for an hour of completed work. 7.0% said they captured more than 51 hours of video for an hour of final content in 2017. In 2017 3.0% captured over 100 hours of content for a completed hour. We expect that multi-camera sports events and 360-degree video will increase the number of hours shot and thus the required storage capacity per event even more. With the increases in storage capacity expected for the media and entertainment industry, traditional approaches for buying storage appliances and the resulting complexity for managing these independent storage systems just won’t scale.

ã 2018 Coughlin Associates,

2

Figure 1. Content Shot for an Hour of Completed Work

So far we have discussed the generation of digital content, but there are many steps between content generation and its eventual distribution and monetization. These steps consist of many operations and the complete set of operations from content capture and creation to distribution and archiving is referred to as a workflow. Figure 2 represents a digital entertainment content workflow showing various aspects of digital storage used in the creation, distribution, and archiving of digital entertainment content1. As the amount and size of content grows, the storage requirements for various operations in the workflow will be different. This workflow includes content ingest, editing and other post production operations, final conformation and assembly of the finished product and then transcoding for delivery over various distribution channels and finally archiving and backup. Different operations in this workflow have different digital storage performance requirements. For instance, content capture and post production often require very low latency access to data, while content distribution libraries and content archives might live with longer latencies. These differences can impact the types of storage technologies and storage systems architectures that work best with the applications. 1Tom Coughlin, Digital Storage for Media and Entertainment Report, www.tomcoughlin.com, 2017.

ã 2018 Coughlin Associates,

3

Figure 2. Digital Entertainment Content Workflow.

Storage Technology Developments There are exciting developments in basic digital storage technologies. Flash memory is increasing in size and gradually decreasing in price. Samsung recently announced a 128 TB solid state drive (SSDs) available in the near future and multiple TB SSDs are becoming primary storage in many enterprise applications. HDDs up to 14 TB are available for cooler storage applications and with recent announcements by HDD companies 40 TB HDDs should be available by 2025. In addition, the growth of NAND flash and other solid-state storage devices have led to new storage interfaces and networking protocols as well as new uses for existing interfaces. Flash memory and other solid-state storage technologies are now being used as DIMM cards in computer memory busses. The NVMe protocol is based upon the PCIe bus in computer systems and will empower the next generations of SSDs for client, consumer and enterprise applications. NVMe provides a high-performance storage interface for storage devices and NVMe over Fabric (NVMe-oF) will power upcoming storage networking using primary Ethernet hardware. The development of higher capacity and lower cost per TB storage devices as well as new networking technologies that take advantage of the performance advantage of flash memory and emerging non-volatile memory technologies will be a critical element in storage for future media and entertainment workflows. A server-based storage architecture can cost effectively take advantage of the

ã 2018 Coughlin Associates,

4

appropriate storage technology in order to get the best trade-off between cost and performance. Scale-Out Software-Defined Storage for Media It is possible to make very cost-effective on-premise or hybrid cloud storage using commodity hardware and software. Software-defined storage is a term used to describe storage infrastructure built on industry-standard infrastructure. This is in direct contrast to traditional storage that tightly couples the storage controller software and the hardware on which it is deployed. Software-defined storage appeals to organizations that want to reduce their overall storage costs and increase flexibility in the way they scale storage to meet evolving demands. In addition, software-defined storage has benefits to users. such as virtualization for data storage. Virtualization allows making many virtual machines out of physical ones. This lets organizations abstract and pool storage capacity as well as compute and networking resources across on-premise and cloud environments to scale independently beyond the capability of individual hardware components. Open, software-defined storage combines community-powered open source software and volume economics by running this software on standards-based commodity hardware. This results in flexible, scalable, loosely coupled storage environments that can be useful for unstructured data such as that commonly found in professional media and entertainment. The emergence of software-defined file storage has made it attractive for studios and broadcasters to find ways to use this technology to take the place of traditional NAS storage arrays. Software defined storage using industry standard server hardware is well-suited for the high-throughput and archival workloads associated with these needs, providing excellent price-performance efficiency versus traditional storage appliances. Red Hat Gluster Storage - Software Defined Red HatÒ Gluster Storage is an open, software-defined scale-out storage platform to easily manage unstructured data for physical, virtual, and cloud environments that allows cost effective storage and management of petabyte size content libraries. It can be used for storing various kinds of unstructured data including rich media content like videos, images and audio files, backup-images and nearline archives, virtual machine images and big data. Red Hat Gluster Storage eliminates storage silos by enabling global access to data through multiple file and object protocols. It is also designed to work seamlessly with industry standard x86 commodity servers. Red Hat Gluster Storage can easily be deployed on-premise, in public cloud infrastructures, and in hybrid cloud environments. It provides a continuous

ã 2018 Coughlin Associates,

5

storage fabric across physical, virtual, and cloud resources so customers can transform their big, semi-structured, and unstructured data from a burden to an asset. Red Hat Gluster Storage is optimized for storage-intensive enterprise workloads such as archiving and backup, rich media content delivery, enterprise drop-box, cloud and business applications, virtual and cloud infrastructure storage, as well as emerging workloads such as co-resident applications, containers, and big data analytics workloads. Legacy storage appliances cannot deliver the scale and flexibility required by modern workloads such as containers. Developers don’t want to wait on storage administrators to provision storage for their applications. Dynamic provisioning with container-native storage helps developers to choose what type of storage they need, and provision it right away. Red Hat Gluster Storage offers these capabilities, which will become increasingly important for media and entertainment applications. In fact, Red Hat Gluster Storage is deeply integrated with Red Hat OpenShift Container Platform for a seamless customer experience - one support call for the entire stack as opposed to many weeks of finger pointing between multiple vendors (container host, app dev platform, storage etc.) Red Hat Gluster Storage lets customers deploy cost-effective and highly available storage without compromising scale or performance. It is designed for unlimited scalability; storage can be added or removed while data continues to be available. Bit-rot detection helps ensure the integrity of that data. Additionally, its elastic, scale-out architecture lets users seamlessly grow storage to meet the dynamic needs of virtual environments. Red Hat Gluster Storage is a mature, proven distributed solution with proven success across a number of use cases such as rich media and active archive, owing to its elasticity and ease of use. Media and Entertainment Gluster Customer Experiences Proximus Group is a telecommunication & ICT company operating in the Belgian and international markets, providing services to residential, enterprise and public customers. They are the leading provider of telephony, internet, television and network-based ICT services in Belgium through their Proximus and Scarlet brands. The consolidated PVRs of 1.5 million customers from 280 channels generate a continuous load of 8-10Gbits/sec with an average of 500 simultaneous FTP transfers. 9000 TV emissions are recorded daily in each site. They serve national broadcasters in Belgium with over 50,000 videos in their VOD network. The company uses Red Hat Gluster Storage with one system in production and one in non-production, starting with their implementation in late 2016. Proximus records and copies this data in two geographically separate

ã 2018 Coughlin Associates,

6

sites. The Gluster storage resides on HPE ProLiant 3600 servers with HPE D6020 storage enclosures filled with 8 TB HDDs. Their VOD system ingests about 500 video objects per day at about 2 GB size for each object. They also delete about 300 video objects per day. Proximus says that the most valuable feature of the Gluster storage systems is its high streaming throughput and scale out capability, including the stretch cluster between their two facilities. Gluster Storage solution is also used to store copies of customer’s personal video recordings (nPVR). Gluster saves 2 extras copy of customers recordings for a period of 10 days, allowing Proximus to recover them in case of data loss. The master copy is stored for a longer period on other storage. Figure 3 show the volume of VOD streamed to customers on a single day and Figure 4 shows the VOD data streamed over the course of a week.

Figure 3. VOD Data Flow in One Day at Proximus.

Figure 4. VOD Data Flow in One Day at Proximus.

ã 2018 Coughlin Associates,

7

Proximus currently uses their Gluster storage for two resolutions. Their SD channels operate at 2.5 Mbits/sec and their HD channels operate at 6.2 Mbits/sec. Their Gluster storage initial capacity was 1.2 PB and they are increasing this to 2.4 PB to meet their growing storage needs. Conclusions Software defined storage running on industry standard server hardware provides a cost-effective storage environment for many media and entertainment industry applications, from content delivery to archiving. This is especially important for M&E because the number and size of content created is increasing dramatically, while budgets are not. Red Hat Gluster Storage is replacing traditional NAS storage systems in media and entertainment applications. Customers find that Gluster storage provides good performance and scale out capabilities allowing them to meet the growing needs of their media workflows.

ã 2018 Coughlin Associates,

8

About the Author:

Tom Coughlin, President, Coughlin Associates is a widely respected digital storage analyst as well as business and technology consultant. He has over 37 years in the data storage industry with multiple engineering and management positions at high profile companies. Dr. Coughlin has many publications and six patents to his credit. Tom is also the author of Digital Storage in Consumer Electronics: The Essential Guide, which is in its second edition with Springer. Coughlin Associates provides market and technology analysis as well as Data Storage Technical and Business Consulting

services. Tom publishes the Digital Storage Technology Newsletter, the Media and Entertainment Storage Report, the Emerging Non-Volatile Memory Report and other industry reports. Tom is also a regular contributor on digital storage for Forbes.com and other blogs. Tom is active with SMPTE, SNIA, the IEEE (he is President-elect of IEEE_USA and active in the Consumer Electronics Society where he is chairman of the Future Directions Committee) and other professional organizations. Tom is the founder and organizer of the Annual Storage Visions Conference (www.storagevisions.com), a partner to the International Consumer Electronics Show, as well as the Creative Storage Conference (www.creativestorage.org). He has been the general chairman of the annual Flash Memory Summit, the world’s largest independent storage event. He is a Fellow of the IEEE and a member of the Consultants Network of Silicon Valley (CNSV). For more information on Tom Coughlin and his publications go to www.tomcoughlin.com.

Coughlin Associates408-202-5098www.tomcoughlin.com