ECE 4110: Random Signals in Communications andSignal Processing

ECE department, Cornell University, Fall 2011Instructor: Salman Avestimehr

Homework 4 SolutionsBy: Sina Lashgari

1. Let Yi = X + Zi for i = 1, 2, . . . , n be n observations of a signal X ∼ N(0, P ).The additive noise components Z1, Z2, . . . , Zn are zero-mean jointly Gaussian randomvariables that are independent of X. Furthermore, assume that the noise componentsZ1, . . . , Zn are uncorrelated, each with variance N . Find the best MSE estimate of Xgiven Y1, Y2, . . . , Yn and its MSE. Hint: It might be convenient to assume a form ofthe estimator and use the orthogonality principle to claim optimality.

Problem 1 Solution

According to the hint we try to come up with a guess and prove the optimality ofthat guess in terms of mean square error. First, note that since X and ~Y are jointlyGaussian, we have E[X|~Y ] = L[X|~Y ]. Therefore, MSE is a linear combination of Yi’s.Thus, it is sufficient to search through the class of linear functions for MMSE. In otherwords, it is sufficient to find a linear combination of Yi’s whose error is orthogonal toany of Y1 through Yn. We consider X̂ = c

∑ni=1 Yi, where c is an unknown constant.

For E[(X − c∑n

i=1 Yi)Yj] to be equal to zero for any j ∈ {1, 2, · · · .n} we should have

c = PnP+N

. Therefore, X̂ = PnP+N

∑ni=1 Yi.

The error is

E[(X − X̂)2] = P − nP 2

nP +N.As n→∞ or N → 0 this error goes to zero.

2. Suppose (X, Y1, Y2) is a zero-mean Gaussian random vector, with covariance matrix

K =

4 2 12 4 21 2 1

(1)

(a) Find the conditional pdf, fX|Y1,Y2(x|y1, y2).(b) Calculate E[X|Y1, Y2].

Problem 2 Solution

(a) Since we are dealing with jointly Gaussian random variables, given Y1 and Y2, Xis a Gaussian random variable with mean E[X|Y1, Y2] and variance equal to the

MMSE error. K~Y is singular and therefore, ~Y does not have a density. We have

E[(Y1 − 2Y2)2] = var(Y1)− 4cov(Y1, Y2) + 4var(Y2) = 0 (2)

which shows Y1 = 2Y2. Thus, we can just consider fX|Y1 .

For Gaussian random varibles E[X|Y1] = L[X|Y1]. Therefore, We need

E[(X − aY1)Y1] = 0⇒ cov(X, Y1)− avar(Y1) = 0⇒ a = 0.5 (3)

Since we are dealing with zero mean Gaussian random variables; having correlationzero implies (X−0.5Y1) and Y1 are independent. We have X = X−0.5Y1 +0.5Y1,thus given Y1 = y1, we have X ∼ N(0.5y1, σ

2), where σ2 = var(X − 0.5Y1) = 3.

⇒ fX|Y1,Y2(x|y1, y2) =1√

2π × 3exp{−1

6(x− 0.5y1)

2} if y1 = 2y2 (4)

(b)E[X|Y1, Y2] = E[X|Y1] = 0.5Y1 (5)

3. Suppose that g(~Y ) is the linear least-square error (LLSE) estimator for X given ~Y :

g(~Y ) = L[X|~Y ] = KX~YK−1~Y

(~Y − E[~Y ]) + E[X].

Determine the mean square error

E[(X − g(~Y ))2]

in terms of the means, covariances, and cross-covariances of X and ~Y .

Problem 3 Solution

E[(X − g(~Y ))2] = E[(X − E[X]−KX~YK−1~Y

(~Y − E[~Y ]))2]

= E[(X − E[X])2] + E[(KX~YK−1~Y

(~Y − E[~Y ]))2]

− 2E[(X − E[X])(KX~YK−1~Y

(~Y − E[~Y ]))]

= var(X) +KX~YK−1~YE[(~Y − E[~Y ])(~Y − E[~Y ])T ](K−1~Y

)TKTX~Y

− 2KX~YK−1~YE[(X − E[X])(~Y − E[~Y ])]

= var(X) +KX~YK−1~YK~YK

−1~YKT

X~Y− 2KX~YK

−1~YKT

X~Y

= var(X)−KX~YK−1~YKT

X~Y

(6)

2

4. Let ~X be Gaussian random vector with mean [1 4 6]T and covariance matrix 3 1 01 2 10 1 1

.(a) Compute E[X1|X2] and E[(X1 − E[X1|X2])

2].

(b) Compute E[X1|X3] and E[(X1 − E[X1|X3])2].

(c) Compute E[X1|X2, X3] and E[(X1 − E[X1|X2, X3])2].

(d) Note that X1 and X3 are uncorrelated, and hence independent. Yet E[X1|X2, X3]is a function of both X2 and X3. Why is that?

Problem 4 Solution

(a) MMSE estimator in this case is an affine function.

E[X1|X2] = E[X1] + cov(X1, X2)(var(X2))−1(X2 − E[X2]) =

X2

2− 1

E[(X1 − E[X1|X2])2] = var(X1)−

(cov(X1, X2))2

var(X2)=

5

2

(7)

(b) X1 and X2 are Gaussian and uncorrelated, therefore independent.

E[X1|X3] = E[X1] = 1

E[(X1 − E[X1|X3])2] = 3

(8)

(c) Let ~Y T = [X2X3].

E[X1|~Y ] = E[X1] +KX1,~YK−1~Y

(~Y − E[~Y ])

=[

1 0] [ 2 1

1 1

]−1([X2X3]

T − [46]T ) + 1

= X2 −X3 + 3

E[(X1 − E[X1|Y ])2] = var(X1) + +KX1,~YK−1~Y

KTX1,~Y

= 2

(9)

(d) Independence of X1 and X3, does not imply their independence when conditionedon X2, because X2 might contain some common information between X1 and X3.In general, we have the following statement: X1, X3independent ; (X1, X3)|X2

independent.

5. Let Xn be a sequence of i.i.d. equiprobable Bernoulli random variables and let

Yn = 2nX1X2 . . . Xn.

3

(a) Does this sequence converge almost surely, and if so, to what limit?

(b) Does this sequence converge in mean square, and if so, to what limit?

Problem 5 Solution

(a)

Yn = 2nX1X2 . . . Xn =

{2n X1 = X2 = . . . = Xn = 10 Otherwise

(10)

since P (Yn 6= 0) = 12n→ 0, Yn

a.s.→ 0.

(b)

E[Yn] = 2n × P [X1 = X2 = . . . = Xn = 1]

+ 0× (1− P [X1 = X2 = . . . = Xn = 1])

= 2n(1

2n) = 1

(11)

Furthermore,

E[Y 2n ] = (2n)2 × (

1

2n) = 2n →∞ (12)

Thus Yn does not converge to zero in m.s. sense.

6. Suppose Xnm.s.→ X and Yn

m.s.→ Y . Show that

(a) Xn +Ynm.s.→ X+Y . Hint: You may find the following inequality useful, (a+ b)2 ≤

2a2 + 2b2.

(b) E[(Xn + Yn)2]→ E[(X + Y )2].

(c) E[XnYn]→ E[XY ].

Problem 6 Solution

(a)

E[(Xn + Yn − (X + Y ))2] = E[((Xn −X) + (Yn − Y ))2]

≤ E[2(Xn −X)2 + 2(Yn − Y )2]

= 2E[(Xn −X)2] + 2E[(Yn − Y )2]→ 0 as n→∞(13)

where we have used the hint and the fact that expectation is a monotonic function.

4

(b)

E[(Xn + Yn)2] = E[((X + Y ) + (Xn + Yn − (X + Y )))2]

= E[(X + Y )2] + E[(Xn + Yn − (X + Y ))2]

+ 2E[(X + Y )(Xn + Yn − (X + Y ))]

(14)

From part (a) we know that the second term goes to zero as n goes to∞. We usethe Cauchy-Schwarz inequality to show that the last term also converges to zero.

E[|(X+Y )(Xn +Yn− (X+Y ))|] ≤ E[(X+Y )2]0.5E[(Xn +Yn− (X+Y ))2]0.5 → 0(15)

Therefore, we getE[(Xn + Yn)2]→ E[(X + Y )2] (16)

(c)

E[XnYn] = E[(X + (Xn −X))(Y + (Yn − Y ))]

= E[XY ] + E[X(Yn − Y )]

+ E[Y (Xn −X)] + E[(Xn −X)(Yn − Y )]

(17)

We already know that Yn → Y in m.s. sense, using Cauchy-Schwarz inequalitywe have

E[|X(Yn − Y )|] ≤ E[X2]0.5E[(Yn − Y )2]0.5 → 0 (18)

similarly, third and fourth terms also go to zero and therefore

E[XnYn]→ E[XY ] (19)

You can also use the fact that XnYn = (Xn+Yn)2−X2n−Y 2

n

2. From part (b), we know

that E[(Xn+Yn)2]→ E[(X+Y )2], and we can similarly show that E[X2n]→ E[X2]

and E[Y 2n ]→ E[Y 2]. Using these facts, we have

E[XnYn] = E[(Xn + Yn)2 −X2

n − Y 2n

2]→ E[

(X + Y )2 −X2 − Y 2

2] = E[XY ]

(20)

7. Let X1, X2, . . . be a sequence of variables with mean µ and covariance

COV (Xi, Xj) = σ2ρ|i−j|,

where |ρ| < 1. Let

Sn =

∑ni=1Xi

n.

Show that Snm.s.→ µ.

Problem 7 Solution

5

E[(Sn − µ)2] =1

n2E[

n∑i=1

n∑j=1

(Xi − µ)(Xj − µ)]

=1

n2

n∑i=1

n∑j=1

E[(Xi − µ)(Xj − µ)]

=1

n2

n∑i=1

n∑j=1

COV (XiXj)

=1

n2[nσ2 + 2(n− 1)σ2ρ+ 2(n− 2)σ2ρ2 + . . .+ 2σ2ρn−1]

≤ 1

n2[2nσ2 + 2nσ2ρ+ 2nσ2ρ2 + . . .+ 2nσ2ρn−1]

=2σ2

n[1 + ρ+ ρ2 + . . .+ ρn−1]

=2σ2

n

1− ρn

1− ρ→ 0

(21)

where to derive the last line, we have used the fact that |ρ| ≤ 1.

6