CHANNEL CODING

Introduction

The mutual information I(A,B), with input A and output B, measures the amount of information that the channel is able to convey over the source. If the channel is noiseless, then this is equal to the information content of source A, I(A,B)=H(A). But if the channel is noisy, there is an uncertainty of equivocation that reduces the mutual information to I(A,B)=H(A) – H(A|B). The presence of noise manifests itself as random errors in the transmission of the symbols.

The random errors may be tolerable ( eg. Occasional bit errors in the transmission of an image may not yield any noticeable degradation in picture quality) or fatal (e.g., with Huffman codes, any bit errors in the encoded variable-length output will create a cascade of errors in the decoded sequence due the loss of coding synchronization ) .

When the errors introduced by the information channel are unacceptable, then channel coding is needed to reduce the error rate and improve the reliability of the transmission. Since the digital communication system uses a binary channel and the source coder will encode the source to a binary code, then the intervening channel coder will code binary msgs to binary codes.

Our concern is the binary block codes, that is, both the channel coder inputs and outputs will be binary and fixed length.

Definition of channel coding

The channel encoder separates or segments the incoming bit stream into equal length blocks of L binary digits and maps each L-bit message block into an N-bit code word where N > L and the extra N –L check bits provide the required error protection.

There are M=2L messages and 2L code words of length N bits

The channel decoder has the task of detecting that there has been a bit error and (if possible) correcting the bit error

The channel decoder can resolve bit errors by two scheme of error-control .

ARQ (Automatic-Repeat-Request ) If the channel decoder performs error detection then errors can be detected and a feedback channel from the channel decoder to the channel encoder can be used to control the

retransmission of the code word until the code word is received without detectable errors.

FEC (Forward Error Correction) If the channel decoderperforms error correction then errors are not only detected but the bits in error can be identified and corrected (by bit inversion)

Code Rate

The extra N –L bits requires a higher bit-rate or bandwidth of a channel

Assume that the source coder generates messages at an average bit rate of ns bit per second

If the channel encoder maps each L-bit message into N-bit code word, then the channel bit rate will be

nc = ( N/L )ns

It infers that more bits transmitted through the channel, so the channel must transmit bits faster than the source encoder

The general expression for the code rate isR=H(M)/n

Where H(M) is the average number of bits transmitted with each source message and n is the length of code wordFor the case of channel coding, we have M= 2L and each message is transmitted as the code word of length N, then H(M)=log2M=L and n = N

R = L/N = ns/nc = Tc/Ts

The code rate measure the relative amount of information conveyed in each code word

Higher code rate implies fewer redundant check bits relative to the code word length

The message rate and the channel rates are fixed design parameters, the code word length is then selected:

N = Lnc/ns

Example A system has bps and the channel bit rate is bps. This requires a channel coder with If L = 3, we have to choose N = 4

If L = 5, we have to choose N = 6, and the channel coder has code rate of 5/6 yields loss of synchronization, we have to compensate with 2 “dummy’ bits every 3 code word

Decoding RulesThe form of decoding rule will govern how the channel codes are used and the level of error protection provided

Let ai = ai1ai2….aiN be the i-th N-bit code word transmitted through the channel

Let b = b1b2….bN be the corresponding N-bit word produced at the output of the channel

Let BM represent the set of M valid N-bit code words and its complement BC

M is the remaining N-bit which are not code words

The BN = BM BCM is the set of all possible N-bit binary

sequences and b = BN whereas ai

The channel decoder has to apply the decoding rule on the received word to decide which code word was transmitted.

If the decision rule is D(.) then ai = D(b)

Let PN (b | ai) be the probability of receiving b given ai was transmitted

For discrete memoryless channel, the probability is

Let PN(ai) be the a priori probability for the message corresponding to the code word ai

By the Bayes’ Theorem we get

If the decoder decodes b into the code word ai then the

probability that this is correct is PN(ai | b) and the probability that it is wrong is 1 – PN(ai| b)

If we want to minimize the error, we have to maximize the

PN(ai | b) (It is the minimum-error decoding rule)

Definition of minimum-error decoding rule:We choose DME(b)=a* (a* ) such that

PN (a* | b) >= PN(ai | b) i

By using Bayes’ Theorem and noting that PN(b) is independent of the code word, the condition becomes

PN (b | a*) PN (a*) >= PN(b | ai) PN (ai) irequiring both knowledge of the channel probabilities and channel input probabilities.This decoding rule guarantees minimum error in decoding

Another rule is called maximum-likelihood decoding rule, by

maximizing PN(b | ai) , the likelihood that b is received given ai was transmitted

The rule is simpler as the channel input probabilities are not required

Definition of Maximum-Likelihood Decoding:

We choose DML(b)=a* where a* is such that

PN (b | a*) >= PN(b | ai) i

requiring only knowledge of the channel probabilities. This decoding rule does not guarantee minimum error in decoding

The maximum-likelihood decoding rule is the same as the minimum-error decoding rule when the channel input probabilities are equal

Assume we have decoding rule D(.) and Bi is the set of all possible received words b such that D(b) = ai

The probability of decoding error given code word ai is transmitted is

The overall probability of decoding error is

Hamming Distance

The important parameter in error coding is the number of bit errors or differences between two different code words ai, and aj

This measure is provided by Hamming distance on the space of binary words of length N

The properties of Hamming distance are instrumental in establishing the operation and performance of channel codes for both error detection and error correction

Definition of Hamming Distance:

Consider the two N-length binary words a = a1a2….aN and b = b1b2….bN

The Hamming distance between a and b, d(a,b) is defined as the number of bit positions in which a and b differ

The Hamming distance is a metric on the space of all binary words of length N since for arbitrary words a, b, c, the Hamming distance obeys the following conditions:d(a,b)>=0 with equality when a= bd(a,b)= d(b,a)d(a,b)+ d(b,c)>=d(a,c) (triangle inequality)

Hamming Distance for BSCs

Consider the maximum likelihood decoding rule where the code

word, ai , which maximizes the conditional probability PN(b | ai) has to be found

Consider a BSC with bit error probability q

Let the Hamming distance d(b,ai) = D represent the number of bit errors between the transmitted code word and received word

This gives the following expression for the conditional probability

If q< 0.5, then eq. above is maximized when ai is chosen such that d(b,ai) is minimized

Hamming Distance Decoding Rule

The binary word, b, of length N is received upon transmission of one of the M possible N-bit binary code words, ai BM, through a BSC.

Assuming the maximum likelihood decoding rule we choose the most likely code word as follows:

If b = ai , for a particular i then the code word ai was sent

If b ai , for any i, we find the codeword a* B which is closest to b in the Hamming sense:

If there is only one candidate a* then the t-bit error, where t=d(a*,b), is corrected and a* was sent

If there is more than one candidate a* then the t-bit error, where t=d(a*,b) , can only be detected

Error Detection/Correction Using Hamming Distance

Errors are detected when the received word is equidistance from more than one code word

Errors are corrected when the received word is closest to only one code word

Definition of minimum distance of a code:

The minimum distance of a block code Kn, where Kn identifies the set of length n code words, is given by

d(Kn) = mind(a,b) |a,b Kn and a b

that is, the smallest Hamming distance over all pairs of distinct code words

A block code Kn detects up to t errors if and only if its minimum distance is greater than t

A block code Kn corrects up to t errors if and only if its minimum distance is greater than 2t

Upper Bounds on M and the Hamming Bound

Definition of Upper Bound on M:

For a block code KN of length N and minimum distance d(KN) the maximum number of code words , and hence the msgs, M, to guarantee that such a code is exist is given by :

M = B(N,d(KN))

The following are elementary results for B(N,d(KN)) :

B(N,1) = 2N B(N,2) = 2N-1 B(N,2t+1) = B(N+1,2t+2) B(N,N) = 2

Where B(N,2t+1) = b(N+1,2t+2) indicates that if we know bound for odd d(KN) equivalent 2t+1 then we can obtain the bound for the next even d(KN)+1 = 2t+2, and vice versa

Theorem: Hamming Bound

If the block code of length N is a t-bit error correcting code, then the number of code words, M, must satisfy the following inequality :

which is an upper bound (the Hamming bound) on the number of code words

We have already known that for a t-bit error correcting code d(KN)>2t

A t-bit error correcting code will have a minimum distance of d(KN)=2t+1 or more

Consider the problem of finding the upper bound for a code with d(KN)=2t+1

Since d(KN)>2t this is a t-bit error correcting code and subject to the Hamming bound

Maximal Codes and Gilbert Bound

The code K3= {000, 011} has d(K3)=2 and contains M= 2 code words of length N= 3 implying L= 1 and a code rate R= 1/3

The code can be made more efficient by augmenting it to the code K3= {000, 011, 101, 110} which is also d(K3) = 2but contains M= 4 code words of length N= 3 implying L= 2 and a code rate R= 2/3

The second one is more efficient than the first code without sacrificing the minimum distance

Definition of Maximal Codes:

A code KN of length N and minimum distance d(KN) with M code words is said to be maximal if it is not part of, or cannot be augmented to, another code KN of length N, minimum distance d(KN) but with M+ 1 code words. It can be shown that a code is maximum if and only if for all words b of length N there is a code word ai, such that d(b,ai) <d(KN)

Example

The code K3= {000, 011} with d(K3)=2 is not maximal since for word b= 101, d(a1,b)=d(000,101)=2 , and d(a1,b)=d(011,101)=2

and thus there is no code word such that d(b,ai) < 2

If code KN is a maximal code then it satisfies the Gilbert bound

Theorem: Gilbert Bound

If a block code KN of length N is a maximal code with minimum distance d(KN) then the number of code words, M, must satisfy the following :

which provides a lower bound (the Gilbert bound) on the number of code words

Error Probabilities

There are two important measures for a channel code : The code rate. Codes with higher code rate are more

desirable

The probability of error. Codes with lower probability of error are more desirable. For error correcting codes the block error probability is important. For error detecting codes the probability of undetected error is important

There is a tradeoff between the code rate and probability of error

To achieve a lower probability of error it may be necessary to sacrifice code rate

The bit error probability, Peb(Kn), is the probability of bit error between the message and the decoded message

For a BSC system without a channel coder : Peb(Kn)=q

The block error probability, Pe(Kn), is the probability of a decoding error by the channel decoder

The probability that the decoder picks the wrong code word when applying the Hamming distance decoding rule

For L = 1, the bit error and block error probabilities are the same

For L > 1 the calculation of the block error probability is straightforward if we know the minimum distance of the code

Example :

Consider code K6 as shown below :

Say message 000 is transmitted as code word 000000 Two bit errors occur in the last two bits, so the received word is 000011

The channel decoder will select the nearest code: 100011

The block error has occured

If this always occurs then Pe(Kn)=1

The code word 100011 is decoded as message 100 which has only the first bit wrong and the remaining two bits correct when compared with the original message

Then, the bit error is Pb(Kn)=1/3