A Novel Blind Watermarking Scheme for Depth-Image-Based Rendering 3D Images

Yu-Hsun Lin2 and Ja-Ling Wu1,2 National Taiwan University

1Department of Computer Science and Information Engineering 2Graduate Institute of Networking and Multimedia {lymanblue,wjl}@cmlab.csie.ntu.edu.tw

ABSTRACT The content protection for image-based 3D data is getting more importance with the advance of low cost 3D display devices. The depth-image-based rendering (DIBR) 3D image is one of the image-based 3D data which consists of the center image and the depth image generated by the content provider. The left-eye image and the right-eye image are rendered from the center image and the depth image at the content consumer side. The blind watermarking for DIBR 3D image is rarely studied in the literature. In this paper, a novel blind multiple watermarking scheme is proposed to deal with the content protection problem of DIBR 3D images. Besides the usual requirement of mutual orthogonality among reference patterns for multiple watermark embedding, we found that proper embedding order plays an even more important role in watermarking the DIBR 3D images. Experimental results show that the proposed scheme is robust against the JPEG compression and noise adding attacks. More interestingly, it is found that the proposed watermarking can also tolerate large range variations of the depth image during rendering. Categories and Subject Descriptors I.4.0 [Image Processing and Computer Vision]: General General Terms Algorithms, Performance, Design, Experimentation, Security.

Keywords Blind watermarking, multiple watermarking, Depth-Image-Based Rendering (DIBR), 3D-image

1. INTRODUCTION With the emerging of low cost 3D display devices, different kinds of 3D applications and the amount of 3D content are booming up, recently. The digital watermarking system for 3D content protection will play an important role for developing and promoting the 3D entertainment industry. Human visual system

(HVS) perceives the depth illusion of a 3D image from two different viewpoint images, one from the left-eye and another from the right-eye. Generally, there are two major image-based representations for 3D images. The first one is the so-called stereo image, where the left-eye image and right-eye image are recorded directly. The second one is the depth-image-based rendering (DIBR) 3D image which records the center image and the depth image. The studies in ATTEST project [4] showed the advantage of DIBR 3D image representation for 3D TV broadcasting is that it can be compressed to a much lower bit rate than a normal color representation does. This is the reason why DIBR 3D image is chosen to be the target that we are going to protect.

There are some watermarking schemes proposed for stereo images [5, 6] and DIBR 3D image representations [10]. Due to the specific data dependency of 3D images, all the previous works belong to the informed and the semi-informed watermarking categories. Therefore, in this paper, a blind watermarking scheme for DIBR 3D images is proposed and investigated. Figure 1 specifies that the watermark embedded in the center image has to be kept on the left- and right-eye images after rendering.

If the left-eye and right-eye images are illegally re-distributed, the depth image may not be available for watermark detection. Without the aid of depth image, the rendering operation can be viewed as a kind of local geometric modification, which is an effective attack for many blind

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 3DVP’10, October 29, 2010, Firenze, Italy. Copyright 2010 ACM 978-1-4503-0159-6/10/10...$10.00.

Figure 1. Blind watermark scheme for DIBR 3D images, where the message embedded in the center image should be kept on the left- and right-eye images after rendering.

Figure 2. (a) Center image. (b) Depth image. (c) The mapping

between intensity and depth value.

watermarking schemes. There are lots of robust blind watermarking schemes for 2D images [1, 3, 11] that can resist local geometric modification attacks. However, they cannot be applied to 3D images directly. Although the rendering operation is one kind of local geometric modifications; fortunately, we have the full knowledge about the rendering operation in the watermark embedding phase. That means we may reduce its negative effect by utilizing our full understanding of the rendering operation.

This paper is organized as follows. Section 2 reviews the basic DIBR operations. Section 3 presents the proposed watermarking scheme for DIBR 3D images. Experimental results are demonstrated in Section 4 and Section 5 concludes this work.

2. THE DIBR 3D IMAGE SYSTEMS The depth image indicates the depth values of the objects in a 3D scene. DIBR warps the center image according to the depth values to synthesize left-eye and right-eye images.

2.1. Rendering Operation The rendering operation for synthesizing the left-eye and/or the right-eye images is performed by warping the center image pixel-wisely. Figure 2 shows an example of center image, depth image and the corresponding mapping between intensity and depth value. The depth value ranges from Znear to Zfar, where Znear is the nearest clipping plane and Zfar is the farthest clipping plane in the 3D scene. The depth value maps the intensity value linearly into the range [Zfar, Znear] (which will be [0, 255] in an 8-bit depth image). The rendering operation warps the pixel location according to the depth value by the following formula [7]:

⎟⎠⎞

⎜⎝⎛ ×+=

Zft

xx xCL 2

⎟⎠⎞

⎜⎝⎛ ×−=

Zft

xx xCR 2

(1)

where xC is the x-coordinate of a pixel in the center image, xL/xR is the corresponding x-coordinate of the pixel in the left- eye/right-eye image, f is the focal length of camera, tx is the baseline distance or camera distance, and Z represents the depth value of the current pixel in the center image.

Figure 3. (a) The left-eye image with holes. (b) The image with

holes being filled by linear interpolation.

According to the visibility property of a 3D scene, the near object will occlude the object with a distant; therefore, in the rendering process, the pixels with the farthest depth value should be warped first.

2.2. The Hole-Filling Problems There is a well-known problem for DIBR 3D images: the left-eye or right-eye image may have holes due to sharp changes in the depth image. The existence of holes represents the situation that the 3D objects which are occluded in the center image but are visible in the left-eye or right-eye image. The Hole-Filling algorithm can simply be a linear interpolation done in the receiver side, of course, there are lots of studies [4, 7] dealt with the Hole-Filling problem by providing better synthesis quality according to the depth image. Figure 3 shows an example of the left-eye image with holes and the image with holes being filled by linear interpolation. In order to provide the flexibility for Hole-Filling algorithm selection in a DIBR 3D image system, this study takes only the number of holes into account for designing watermarking scheme. Without loss of generality and for simplicity, this paper uses linear interpolation as the Hole-Filling algorithm in the following discussions. Of course, better quality of the reconstructed images could be expected if a more complicated Hole-Filling algorithm were used.

3. THE WATERMARKING SCHEME Let IC, IL and IR denote the center, the left-eye and the right-eye images, respectively. A multiple watermarking scheme is proposed to protect IC, IL and IR by using different reference patterns wC, wL and wR correspondingly. The reference patterns wC, wL and wR are chosen to be pair-wise orthogonal. Since the communication model of DIBR 3D image system transmits only IC and depth image to the content consumer side, the watermarks for protecting IL and IR must be embedded in and generated from IC. As prescribed, on the basis of the knowledge of rendering, we can embed the watermark in IC to protect IL and IR. Let blocki,L and blocki,R denote the i-th block of IL and IR, respectively.

By utilizing the knowledge of DIBR, we can synthesize Render-1(blocki,L) and Render-1(blocki,R), where Render-1(blocki,L) and Render-1(blocki,R) respectively represent the corresponding pixels in IC for IL and IR. Without loss of generality, Figure 4 shows the embedding block diagram for protecting IL, in the content provider side, where M stands for the message to be embedded which consists of N bits, say b1,b2,…,and bN. The value of bi is +1 or -1. The left-eye image IL is divided into N blocks (each of them is of size m m pixels), and we embed bi

Figure 4. The watermark embedding process.

Figure 5. The proposed watermark detection process.

into Render-1(blocki,L) (following the raster-scan order) by modulating the reference pattern wL. In the embedding process, we skip Render-1(blocki,L) when the number of holes in the block exceeds a given threshold Th. By embedding the watermark into the pixels of Render-1(blocki,L) and Render-1(blocki,R) in IC, we can eliminate the effect of rendering operation performed in content consumer side. Since IL and IR are synthesized by the same corresponding pixels in IC. In order to detect the source of suspect image, the proposed scheme embeds the watermark into all the blocki,C, Render-1(blocki,L) and Render-1(blocki,R) in IC, by modulating different reference patterns (say wC, wL and wR) respectively.

3.1 The Embedding and the Detection Procedures We use a linear approximation of the improved spread spectrum (ISS) technique [9] to insert a reference pattern wP into pixels of blocki,P, where blocki,P can be blocki,C, Render-1(blocki,L), or Render-1(blocki,R). Let’s denote the inner product between s and wp as Pws, .

The proposed watermark embedding scheme can be represented as:

( ) Pwsbsx 'λα −+= , (2)

where

PP

P

wwws

s,

,'= . (3)

s’ in (3) is the interference of the original data to the embedded watermark when detection process is conducted. The variable α

adjusts the signal strength of the inserted bit b, the variable λ indicates the reduction ratio of the interference of s’ and the variable x represents the watermarked signal.

For a suspected image, the watermark detection process is shown in Figure 5, where the correlation operation is performed by computing the normalized inner product between y and wP. That is

( ) '1,

,sb

wwwy

PP

P λαγ −+== (4)

Table 1. The bit error rate for each image

Image IC IL

(2.65% skipped blocks) IR

(2.72% skipped blocks)

BER(IC, wC) = 0% BER(IL, wC) = 42.35% BER(IR, wC) = 37.64%

BER(IC, wL) = 41.76% BER(IL, wL) = 5.44% BER(IR, wL) = 56.75% BER(IP, wP)

BER(IC, wP) = 41.49% BER(IL, wR) = 51.47% BER(IR, wR) = 4.14%

Figure 6. The gray circles illustrate the pixels of blocki,C. The white circles denote the pixels in IC which corresponding to

Render-1(blocki,L).

The estimated bit b’ is determined by checking the sign of γ

sign(γ ). The bit error rate of extracting message M from a given image I, by using the reference pattern wP, is denoted as BER(I, wP) in the rest of this paper. BER(I, wP) can be defined as:

BER(I, wP)=#(bi’ in M’ ≠ bi in M)/N . (5)

3.2 The Effect of Watermark Embedding Order A major design principle for multiple watermarking system is the pair-wise orthogonality among the reference patterns to avoid mutual interference. By contrast, the multiple watermarking scheme for DIBR 3D image faces a different challenge due to the existence of rendering operations. Figure 6 demonstrates the interference between IC and IL. For bit bi, the embedding pixel positions in blocki,C, Render-1(blocki,L) and Render-1(blocki,R) are not the same.Therefore, the traditional mutual orthogonality among reference patterns is not the only thing we have to consider in designing an effective watermarking scheme for protecting DIBR 3D images.

Notice that the reference patterns wC, wL and wR will introduce noises to each other during watermark detection. In this case, the embedding order of wC, wL and wR becomes critical. Since the former embedded reference pattern will suffer the noise generated by the latter embedded one, and the latter embedding process can remove the effect of noise generated from the former embedding by applying the prescribed ISS techniques. We determine the embedding order based on the qualities of IC, IL and IR. The image with better quality should be protected robustly by embedding with the corresponding reference pattern. We use the number of holes as the measure of image quality. Since IC contains no holes, wC is the last one to be embedded. Without loss of generality, we assume IL has more holes than IR, so we embed IL with wL first, and then IR with wR, and IC with wC last.

Figure 7. Left: the original image. Right: the multiple

watermarked image (PSNR = 39.40 dB).

4. EXPERIMENTAL RESULTS The center image and the depth image of the test data “Interview” are obtained from HHI (Heinrich-Hertz-Institute), Germany [4]. The sizes of IC and the depth image are both 720x576 pixels, and the depth image is an 8-bit gray-level image. The parameters for watermark embedding are α =1 and λ =1 which entirely remove

the interference from the original cover. Since the recommended maximum disparity between the left-eye and the right-eye images is about 3% ~5% of the image width for providing comfortable viewing [4, 7], the baseline distance tx is set to 36 pixels. The focal length f is set to 1. In order to examine the watermark sensitivity to depth image variation, we set Zfar= tx/2 and Znear=1 which correspond to the most varying scenario for the rendering operations in Eq. (1). The block size is chosen to be 8x8 pixels, and the length is set to 20 for reference patterns wC, wL and wR. We embed the reference patterns in the middle band of the DCT domain, as suggested by [2]. We skip those blocks with the number of holes exceeds 10% of their size. Figure 7 shows the original center image and the multiple watermarked center image (where the PSNR is 39.40 dB).

4.1. The Effectiveness of Multiple Watermarking Table 1 illustrates the effectiveness of the proposed multiple watermarking scheme, from which, it is clear that the bit error rate is unacceptable for embedding only one image. For example, the bit error rate is quite high if we embed only one reference pattern wC in IC. The proposed multiple watermarking scheme will decrease the bit error rate significantly, as indicated in BER(IL, wL) and BER(IR, wR). The multiple watermarking scheme can also be used to indicate the origin of the suspected image in which the bit error rate increases drastically for mismatched IP and wP. also demonstrates the correctly extracted watermark from IC, IL and IR for “Interview”.

(a) (b)

Figure 8. The bit error rates for: (a) JPEG compression and (b) Gaussian noise adding

(a) (b) (c)

Figure 9. The bit error rates for (a) JPEG compressed depth image and (b) Gaussian noised depth image. (c) The snapshot of the Gaussian noised depth image with variance = 20000.

4.2. The JPEG Lossy Compression and Gaussian Noise Addition Figure 8(a) illustrates the bit error rates for applying different quality JPEG compression attacks to the proposed scheme. The bit error rate reaches to almost 50%, for JPEG quality factor equals to 50, where the compression ratio is up to 32. The experimental results for adding different zero mean Gaussian noises, with different variances, are shown in Figure 8(b). The bit error rate is about 40% for adding a zero mean Gaussian noise with variance 200. Notice that in this setting, the induced distortion is noticeable.

4.3. The Depth Image Variation Similar to the center image, the depth image will be JPEG compressed and suffer from the channel noise.

The receiver side may render the left-eye and the right-eye images based on the modified depth image. This experiment illustrates that the effect of depth image variation to the proposed watermarking scheme. Figure 9(a) demonstrates that there is a negligible increase in bit error rate for applying JPEG compression attack to the depth image (with quality factor 10), where the compression ratio is up to 136. Figure 9(b) illustrates that the proposed watermarking scheme can tolerate high depth image variation, in which the variance of the added zero mean

Gaussian noise even reaches to 20000. Figure 9(c) shows the snapshots of the original depth image and the noise added depth image. Clearly, from Figure 9, the proposed multiple watermarking scheme is robust to the variation of depth image.

5. CONCLUSION In this paper, a blind watermarking scheme incorporating with the rendering operation is proposed to deal with the content protection of DIBR 3D images. The blind watermarking scheme is preferable because it does not need the original data and the depth image during watermark detection. The proposed multiple watermarking scheme is effective to reduce the resultant bit error rate in DIBR 3D image watermark detection. This study also shows different requirements and advantages of the proposed multiple watermarking scheme for protecting DIBR 3D images, as compared with traditional ones. The embedding order becomes very important in DIBR 3D image watermarking, since the interference from latter embedded reference patterns are unavoidable. The proposed scheme is proved to be robust to JPEG compression and noise adding attacks. Moreover, it also tolerates high variation of depth image in rendering. The digital watermarking for image-based 3D data is still in its infancy stage, this work is just our first step toward exploiting this interesting and important topic and, of course, there are lots of different issues worthy of further investigations.

6. REFERENCES [1] M. Barni, “Effectiveness of exhaustive search and template matching against watermark desynchronization,” IEEE Signal Processing Letters, vol. 12, no. 2, pp. 158-161, 2005. [2] H. Chiou-Ting, and W. Ja-Ling, “Hidden digital watermarks in images,” IEEE Transactions on Image Processing, vol. 8, no. 1, pp. 58-68, 1999. [3] J. L. Dugelay, S. Roche, C. Rey et al., “Still-image watermarking robust to local geometric distortions,” IEEE Transactions on Image Processing, vol. 15, no. 9, pp. 2831-2842, 2006. [4] C. Fehn, “Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV,” Proc. SPIE, San Jose, CA, USA, pp. 93-104, 2004. [5] D. C. Hwang, K. H. Bae, and E.-S. Kim, “Stereo image watermarking scheme based on discrete wavelet transform and adaptive disparity estimation,” Proc. SPIE, San Diego, CA, USA, pp. 196-205, 2004. [6] D.-C. Hwang, K.-H. Bae, M.-H. Lee et al., “Real-time stereo image watermarking using discrete cosine transform and adaptive

disparity maps,” Proc. SPIE, Orlando, FL, USA, pp. 233-242, 2003. [7] Z. Liang, and W. J. Tam, “Stereoscopic image generation based on depth images for 3D TV,” IEEE Transactions on Broadcasting, vol. 51, no. 2, pp. 191-199, 2005. [8] C. Y. Lin, M. Wu, J. A. Bloom et al., “Rotation, scale, and translation resilient watermarking for images,” IEEE Transactions on Image Processing, vol. 10, no. 5, pp. 767-782, 2001. [9] H. S. Malvar, and D. A. F. Florencio, “Improved spread spectrum: a new modulation technique for robust watermarking,” IEEE Transactions on Signal Processing, vol. 51, no. 4, pp. 898-905, 2003. [10] Z. Ning, D. Guiguang, and W. Jianmin, “A Novel Digital Watermarking Method for New Viewpoint Video Based on Depth Map,” the 8-th International Conference on Intelligent Systems Design and Applications, pp. 3-7, 2008. [11] X. Shijun, K. Hyoung Joong, and H. Jiwu, “Invariant Image Watermarking Based on Statistical Features in the Low-Frequency Domain,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 18, no. 6, pp. 777-790, 2008.