HDR and its influence on visual

components

Sebastian Björk

Computer Graphic Arts, bachelor's level

2018

Luleå University of Technology

Department of Arts, Communication and Education

Foreword This thesis marks the last assignment in my studies at LTU in Skellefteå. I would like to thank my wife and kid, who’s been putting up with the endless late nights and long weekends in front of the computer during these three years.

Sammanfattning Syftet med detta examensarbete var att analysera och jämföra HDR mot SDR i relation till de visuella komponenter som finns i en bild. Frågan som ställdes var hur stor inverkan HDR har på bildens visuella komponenter. HDR-simulerade bilder från samma miljö analyserades baserat på de visuella komponenter för att se hur de ändrades. Defintionen av vad som räknas som en visuell komponent kommer från Bruce Blocks bok The Visual Story. Resultatet av analysen var att HDR förändrar de nuvarande komponenterna och underkomponenter. En till slutsats var att metoden har sina svagheter och skulle behöva studeras mer ingående, svårigheterna är att ett specifikt hårdvarustöd krävs för att kunna visa HDR korrekt.

Abstract The purpose of this thesis was to analyse and compare HDR to SDR in relation to the visual components which exists in a picture. The questions asked how big of an impact HDR would have on the visual components in a picture. HDR simulated images of the same scene was analysed based on the visual component to compare in what ways they would be altered. The definition of a visual component is based on Bruce Blocks book The Visual Story. The result of the analysis exposed that HDR alters the current visual components and its sub components. Another conclusion was that the method used has it flaws and would need more research, the complication being the hardware needed to properly view HDR content.

Table of contents 1 Prelude ......................................................................................................... 1 1.1 Introduction ........................................................................................................................................ 1 1.2 Background ......................................................................................................................................... 1 1.3 Purpose ................................................................................................................................................. 1 1.4 Limitations ........................................................................................................................................... 2

2 Theory ........................................................................................................... 3 2.1 What is HDR? ...................................................................................................................................... 3

2.1.1 The Dynamic Range of HDR ........................................................................................................... 4 2.1.2 Colors of HDR ........................................................................................................................................ 6

2.2 The visual components ................................................................................................................... 7 2.2.1 Space ......................................................................................................................................................... 7 2.2.2 Line............................................................................................................................................................. 7 2.2.3 Shape ......................................................................................................................................................... 7 2.2.4 Tone ........................................................................................................................................................... 8 2.2.5 Color .......................................................................................................................................................... 8 2.2.6 Movement ............................................................................................................................................... 8 2.2.7 Rhythm ..................................................................................................................................................... 9

3 Method ....................................................................................................... 10 3.1 Analysis .............................................................................................................................................. 10

3.1.1 Football court, simulated image ................................................................................................ 11 3.1.2 Parrot, simulated image ................................................................................................................ 12 3.1.3 Balls, simulated image .................................................................................................................... 13

3.2 Method critique .............................................................................................................................. 13

4 Result .......................................................................................................... 14

5 Discussion ................................................................................................... 14

6 Conclusion ................................................................................................... 15

7 References .................................................................................................. 16

8 Glossary ...................................................................................................... 17

1

1 Prelude

1.1 Introduction High dynamic range (HDR) is a relatively new technology that still is figuring out how it’s supposed to work and look. There’s currently a conflict in which standard is going to be used, in some ways it reminds of the HDDVD and Blu-ray “war” in late 00’s and Betamax and VHS in the 80’s. (Holmes, 2017) It’s probably since it has possibility to make a big impact. Media has looked the same for a long time, it’s mostly been increases in spatial and temporal resolution. The end picture however has been constrained to the fact that all content is viewed in either sRGB or BT.709. For a long time it’s been possible to print a picture from a movie on a piece of paper, and in most ways, it will look the same as on the screen. With HDR that might change, because the hardware will finally have the prerequisite to go beyond the paper with the increased dynamic range and color. The promise of HDR is more realistic pictures than before. (Schulte & Barsotti, 2016)

1.2 Background Understanding what an image is made of, should be of interest for an artist. The visual components has been the base of explaining to me and fellow students at the University how a picture is built. As my first encounter with HDR had such a huge impact, I was curious to dig deeper in how it can affect the image and its visual components.

1.3 Purpose The purpose and idea behind this thesis is to explore the impact of HDR on the visual components, and try to grasp if/in what ways it changes them.

- How does it change the building blocks and components of the currently existing ones?

2

1.4 Limitations This report will analyse HDR against the visual components as defined in Bruce Block’s The Visual Story. The report will also use simulated pictures of the difference of HDR versus SDR, it’s because the hardware limitation, you can’t view HDR content properly if you don’t watch them on an HDR screen. This is currently a common way HDR is compared to SDR. The report will not cover the technical challenges in how HDR is delivered, mastered and the different standards surrounding it.

3

2 Theory

2.1 What is HDR? A common misconception is to think that HDR is same technology that has existed in the realms of cameras and capturing hardware. Its two completely different things, it’s a tech where the goal is to capture as much dynamic range as possible. It’s just the first part of the new infrastructure. You’re still constrained to displaying it on a display with sRGB or BT.709, that’s where the new HDR tech comes in. HDR is a new way to show pictures (moving or still) with increased fidelity and detail, which comes in the form of increased contrast, dynamic range and colors. It will increase those areas several times compared to SDR with the help of new standards for both the displays, post processing, mastering and distribution. The diagram below (Figure.1) is a simplified way of viewing it, it’s specifically for Dolby Vision which is one of the HDR standards, but the base concept applies for the others as well. (Schulte & Barsotti, 2016)

Figure 1. Dolby Vision infrastructure, Image credit: Dolby

4

2.1.1 The Dynamic Range of HDR The limitations and standard set by the Cathode Ray Tube (CRT) television still lives today, two significant way it still does is; -“The color gamut (as defined by the rare phosphors used) -“The brightness was limited to ~100 candela/m2 (often referred to as “nits”) not only to control large area flicker, but also to prevent the electron beam from spreading and reducing spatial resolution.” (Brooks, 2014) Dolby created an experimental display to find out how much dynamic range was favoured by the viewer in the aspects of “black level”, “diffuse white level” and “highlight level”. According to the report of the test, most of the viewers (90%) would be satisified with a dynamic range from 0.001 nit to 10,000nits which equals to 22 F-stops. (Brooks, 2014) An object which in the real world only emits 50 cd/m2 is represented by the same 8 bit value of 255 as a bright object such as the sun which emits more then 105 cd/m2. Which means that for our eyes it’s the same brightness. But with HDR they will be represented with a float value which equals to their actual brightness. However the sun is such a bright object that some adaption still has to occur even in a HDR pipeline. (Boitard et al., 2018) Figure 2 and 3 below gives a approximate overview of some real world light levels.

Figure 2. Real world light levels, Image credit: SMPTE (Holmes, 2017)

Figure 3. Light levels, Dolby Vision (Dolby, 2016)

5

Figure 4 below demostrates a scene where the dynamic range of HDR really benefits the picture, the big difference in dark bright values. Not all scenes displays such a huge gain however. The figure 5 below shows an increase in shadow detail but the difference is not as big as figure 4.

Figure 4. Histogram representation of HDR compared to SDR, (Boitard et al., 2018)

Figure 5. Histogram representation of HDR compared to SDR, (Boitard et al., 2018)

6

2.1.2 Colors of HDR Color is another aspect where HDR is set to increase the percevaible amount by a large amount compared to sRGB and BT.709. The current color space gamut BT.709 is covering 35,9% of the “full visible gamut when represented in the CIE xyY color space” BT.2020 increases that amount to 75.8%, that demands a higher bit depth than the current 8 bit SDR standard otherwise you will end up with distortions, 10 bit HDR standard removes those distortions. (Boitard et al., 2018) The current limitation of 100 nits also affects the colors, since “the brighter the color, the closer it becomes to white, so bright colors become less saturated.” The brightest saturated blue currently only displays at 7 nits on current displays, which doesn’t give justice to “bright blue skyes” as an example. (Dolby, 2016)

Figure 6. BT.2020 compared to BT.709, Image credit: (Boitard et al., 2018)

7

2.2 The visual components There are seven basic visual components which exists in all pictures. These are the building blocks that together creates the visual structure of any picture. (Block, 2008) To understand the concept of the visual components, one needs to understand the principle of contrast and affinity, which applies to all the visual components. ” The greater the contrast in a visual component, the more the visual intensity or dynamic increases. The greater the affinity in a visual component, the more the visual intensity or dynamic decreases.” (Block, 2008)

2.2.1 Space The first one is space, which can be categorized in to the space behind, in front of the camera and the physical size and shape of the screen. The first part describes space through the difference in perspective, camera settings such as field of view and movement of the camera itself. But also, the component called tonal separation which describes that typically lighter objects seem closer and darker farther away. The same principle also applies to warm and cold colors, with warm seem closer and cold farther away.

2.2.2 Line Lines are everywhere even though you can’t see them, they appear in contrast between tones and colors. “Line differs from the other visual components, because lines appear only due to tonal or color contrast. Depending on this contrast, a line can be revealed or obscured” (Block, 2008) Lines can appear from many places, one is edges which is the “apparent line around the borders of any two-dimensional object”, contours appear from the borders we distinguish from three dimensional objects. They can also appear in other ways. Linear motif is a sub component which is achieved through looking at a picture in high contrast simplifying it to only lines.

2.2.3 Shape Shapes are closely related to lines and are described in the same chapter. There are three basic shapes, in a two-dimensional representation they are the circle, equilateral triangle and square. In a three-dimensional representation they are sphere, cube and pyramid.

8

2.2.4 Tone Tone is defined as the brightness of the image. “There are three ways to control the tone, or brightness, of objects in a shot: reflective control (art direction), incident control (lighting), and exposure (camera and lens adjustments).” (Block, 2008) A sub component of tone is coincidence and noncoincidence of it, which is described as the “relationship between the tonal organization of the shot and the subject of the shot.” Coincidence of tone is when subject is exposed by the tonal range and noncoincidence is the other way around.

2.2.5 Color The two main ways of combining and establishing color is through the additive and subtractive system. The base components of color are hue, brightness and saturation. They are described as this; - “Hue is the position of a color on the color wheel: red, orange, yellow, green, cyan, blue, violet (or purple), and magenta.” - “Brightness (sometimes called value) is the addition of white or black to the hue. Brightness is the position of a color in relation to the gray scale.” -“Saturation (sometimes called chroma or intensity) and its opposite, desaturation. Saturation refers to the purity of a hue.” (Block, 2008)

2.2.6 Movement Movement as a visual component exists in three ways, the first being movement of objects, second the camera and third being people’s eyes as they watch the image. There’s four ways in which movement can be created in the image. The first being objects moving in the real world such as vehicles, animals and humans. Second one is the way which movies and video are represented and that is apparent movement. It’s the effect that makes real world objects looking like they move, but, it’s just a series of pictures played back in a rate of 24 and upwards frames per second. Third way is the induced movement, which occurs when a moving object makes a stationary close by object appear to move by “transferring” its movement to it and become stationary itself. The fourth and final way is relative movement, which also is dependent on other objects, but this movement is exposed by the other objects, otherwise it would appear to be stationary.

9

2.2.7 Rhythm Rhythm is an experience-based component. It’s heard, seen and felt. Most common its sound based it also applies to the visuals. The same subcomponents which rhythm is based also applies, they are alternation, repetition and tempo.

10

3 Method

3.1 Analysis HDR increases color and dynamic range (contrast) and allowing for more detail in shadows and highlights. (SMPTE, 2015) Bruce says that more visual intensity is gained by increasing contrast in the visual components, he also states that “every visual component can be described and used in terms of contrast and affinity”. That increase in contrast is achieved in different ways for each visual component and subcomponent, but they “must be related back to contrast and affinity”. He also states that it’s impossible to remove the visual components from a picture, except color if you shoot in black and white. (Block, 2008) This is a good representation of a common difficult scene to capture the amount of dynamic range that exists, a common issue when shooting indoors and some windows are visible. It usually makes for a good surface divider but that changes with HDR. Color, shapes and lines appear and changes the mood and visual structure of the image. These simulated pictures below demonstrate the effect of the increase in dynamic range and color with HDR. Which, depending on the image, impacts the visual components/structure and subcomponents. When commenting about contrast in the image analysis below, that’s referring to the principle of contrast and affinity per visual component.

Figure 7. Effects of HDR, Image credit: Sony

11

3.1.1 Football court, simulated image Space: The subcomponent tonal separation gets an increase. Decrease in surface divider based on the increased detail in dark and bright areas Lines: Increase cause of the new edges, contours and linear motif which adds to additional contrast Shapes: New shapes that weren’t visible before which also adds to an increased contrast Tone: Increase in brightness and darkness, which in turn alters the coincidence and noncoincidence of tone, also contrast increase Color: Change in brightness, saturation and hue, also a contrast increase Movement: Increased relative movement produced by the new shapes and lines Rhythm: Changes cause of difference on the other visual components Image visual structure: Changes since the picture has an increased visual intensity

Figure 8. SDR compared to HDR, Image credit: Sony

12

3.1.2 Parrot, simulated image Space: The subcomponent tonal separation gets an increase Lines: Increase edges and contours which adds to additional contrast Shapes: More profound shapes which in turn adds increased contrast Tone: Increase in brightness and darkness (contrast), which in turn alters the coincidence and noncoincidence of tone, also contrast increase Color: Change in brightness, saturation and hue, also a contrast increase Movement: Becomes more apparent cause of the increased clarity in shapes and color Rhythm: Changes cause of difference on the other visual components Image visual structure: Changes since the picture has an increased visual intensity

Figure 9. HDR left, SDR right, Image credit: Dolby

13

3.1.3 Balls, simulated image Space: An increase in the subcomponent tonal separation Lines: Increase edges and contours which adds to additional contrast Shapes: More defined shapes add to increase in contrast Tone: Increase in brightness and darkness (contrast), which adjusts the coincidence and noncoincidence of tone, plus a contrast increase Color: Change in brightness, saturation and hue, also a contrast increase Movement: Increases cause of the increased difference in shapes, tone and color Rhythm: Changes cause of difference on the other visual components Image visual structure: Changes since the picture has an increased visual intensity

3.2 Method critique This method only analyses against Bruce Blocks defined visual components. There might be other ways to define a picture, its visual components and intensity. Also, the simulated images can never correctly represent HDR cause of the constraints the technology inherently has. Another way to properly compare HDR against SDR pictures wasn’t discovered during research.

Figure 10. HDR left, SDR right, Image credit: Dolby

14

4 Result The increase in space tonal separation in figure 8 might seem wrong, but that’s only because it’s viewed on a non-HDR screen. The overblown look on the left on the same figure is an effect of the 100 nits limit that also effects color. (Dolby, 2016) The increased values in brightness output would display all the areas the sun hits and the blue sky brighter on an HDR screen cause of the dynamic range output difference. The image analysis shows an overall increase in contrast and visual intensity on all simulated images according to the principle of contrast and affinity. HDR alters the current visual components and subcomponents without adding anything that isn’t already there. The only visual component you can remove is color. You can remove HDR and the result will be a decrease in most of the components intensity, but the images will still have the seven base ones left.

5 Discussion My first encounter with HDR was with a Panasonic representative at a photo convention. He demonstrated a few video clips he shot himself with a camera that can capture 10-bit HLG. All the shots were shown on an OLED TV that has capability of 1000 nits. One of the clips was an interior scene a lot like the figure 7 scene, the effect was the same. The amount of dynamic range captured was impressive, it looked more like a real-life capture than a photo capture. As an instinct I took a picture with my phone, and of course the impressive picture as displayed on the TV in front of me, looked nothing like it on the SDR display of my phone. This is a theme of this report, the challenge of analysing something that dependant on a certain hardware to display properly. Even if you show a histogram with correct value differences, it’s still up to the reader’s imagination and simulated pictures to describe the difference, which I found to be a real challenge. The question that this thesis set out to answer is how HDR changes the visual components and its building blocks. The result of the analysis shows that it has a big difference on most of them. HDR would qualify as a subcomponent or overall visual intensity modifier for most of them. The amount it alters them depends on image, even though they are all simulated, which will also be true for real HDR pictures as the dynamic range theory describes. It’s a powerful tool for the people that’s responsible for creating the visual structure in movies and video. Just as lighting, cameras and post processing are a tool for the creators. As an artist myself, it’s a huge subject to wrap you’re head around. The technical aspects as much as the artistic ones. While I haven’t encountered it yet in my limited time in production, I will for sure do it in the future.

15

The report and its result are up for interpretations since the analysis is based on simulated pictures, not real HDR. More research would be needed to find another method of doing the analysis.

6 Conclusion I would like to see if there is a better way to compare HDR to SDR pictures. During my research I found many examples of pictures of two screens showing the same content with and without HDR. Without knowing all the technical information about the screens, content and camera, it’s difficult to make use of them. And I found no other way to do comparisons than to use the simulated images. That’s why they were used, which is not ideal. Something that also would be interesting, is to explore and research about the emotional impact HDR can have on a viewer. With 10,000 nits soon accessible for consumers, the sun or flashes of light can be represented in a better way than the current white pixels displayed at 100 nits. Most of the reports and research surrounding HDR is currently aimed on solving technical and infrastructural challenges. Standards are being defined in how to capture, master, process and display HDR properly. It’s clear that it’s a huge leap in how content is displayed since it alters the image in so many ways.

16

7 References Block, B., 2008. The Visual Story. 2nd ed. Oxford: Focal Press. Boitard, R., Nasiopoulos, P. & Pourazad, M.T., 2018. Compression Efficiency of High Dynamic Range and Wide Color Gamut Pixel's Representation. Ieee Transactions On Broadcasting, 64(1), pp.1-10. [Accessed May 2018]. Brooks, D.G., 2014. The Art of Better Pixels. Paper. USA: Dolby Laboratiories, Inc. Dolby, 2016. https://www.dolby.com. [White Paper] Dolby Laboratories, Inc Available at: https://www.dolby.com/us/en/technologies/dolby-vision/dolby-vision-white-paper.pdf [Accessed May 2018]. Holmes, S., 2017. HDR, Who's, How., 9 June 2017. SMPTE. ITU-R, 2015. Parameter values for the HDTV standards. Recommendation. International Telecommunication Union. Schulte, T. & Barsotti, J., 2016. HDR Demystified. White Paper. Australia: SpectraCal, Inc. SMPTE, 2015. Study Group Report High-Dynamic-Range (HDR) Imaging Ecosystem. Report. Society of Motion Picture and Television Engineers ®, Inc.

17

8 Glossary High dynamic range (HDR)

A new technology standard to capture and display scenes with increased dynamic range, contrast and color

Standard dynamic range (SDR) Current display standard as defined by the ITU-R Hybrid log gamma (HLG)

“The Hybrid Log Gamma name refers to the fact that the OETF is a hybrid curve that applies a standard gamma curve for darker pixels in the legacy SDR range and a logarithmic curve for higher brightness highlight” (Schulte & Barsotti, 2016)

Rec. 709 (BT.709) The current standard of high-definition televisions as defined by

(ITU-R, 2015) Candela (cd)

“Unit of luminious intensity in the international system of units (SI)” often refered to as “nits”

Spatial resolution Number of lines (pixels), i.e. resolution Temporal resolution Frame rate