Coding Theory and Applications (I): A Quick...

Coding Theory and Applications (I):

A Quick Introduction to Coding Theory

Pre-Conference Tutorial, AMC 2005

San LingDivision of Mathematical Sciences

School of Physical and Mathematical Sciences

Nanyang Technological University, Singapore

0-0

1 Introduction

Information media – not absolutely reliable

because of noise or interference.

Coding theory – detects/corrects errors.

source coding and channel coding.

Source coding – changing message source to

a suitable code to be transmitted through the

channel. E.g., ASCII code.

0-1

A simple communication model:

Message Source Receiver

↓ ↑Source Encoder −→ Channel −→ Source Decoder

0-2

Example 1.1

apple → 00, banana → 01,

cherry → 10, grape → 11.

Suppose “apple” (00) transmitted.

Assume 01 received.

apple banana

↓ ↑00 −→Channel−→ 01

↑noise

0-3

Receiver may not realize message was cor-

rupted.

Channel coding – encode message again after

source coding by introducing some form of re-

dundancy so that errors can be detected or

even corrected.

0-4

Message Source Receiver

↓ ↑Source Encoder Source Decoder

↓ ↑Channel Encoder −→ Channel −→ Channel Decoder

0-5

Example 1.2

In Example 1.1, introduce redundancy of 1

bit:

00 → 000, 01 → 011,

10 → 101, 11 → 110.

Suppose “apple” (000) is transmitted over a

noisy channel, and only one error.

Received word is 100, 010 or 001 – none among

encoded messages.

0-6

Detects errors at the cost of reducing trans-

mission speed (transmit 3 bits for a message

of 2 bits).

Does not correct errors.

E.g., if 100 is received, then do not know whether

100 comes from 000, 110, or 101.

With more redundancy, can correct errors. E.g.,

00 → 00000, 01 → 01111,

10 → 10110, 11 → 11001.

0-7

Suppose “apple” is transmitted over a noisy

channel, and only one error.

Received word must be one of: 10000, 01000,

00100, 00010, 00001.

Assume 10000 received.

Can be sure that 10000 comes from 00000 be-

cause there are at least two errors between

10000 and each of the others: 01111, 10110

and 11001.

0-8

apple apple

↓ ↑00 00000

↓ ↑00000 −→Channel −→ 10000

↑noise

0-9

Goal of channel coding – to construct encoder

and decoder for

1. fast encoding of messages,

2. easy transmission of encoded messages,

3. fast decoding of received messages,

4. maximum transfer of information per unit

time, and

5. maximal detection or correction capabil-

ity.

0-10

Mathematically, the primary goal – 4 & 5 (but

not compatible).

Therefore, need trade-off.

0-11

2 Error Detection, Correction and

Decoding

2.1 Some definitions

Definition 2.1

A = {a1, a2, . . . , aq} – set of size q, – code

alphabet; its elements – code symbols.

(i) q-ary word of length n over A: sequence

w = w1w2 · · ·wn or vector (w1, . . . , wn) in

An.

0-12

(ii) q-ary block code of length n over A: non-

empty set C of An.

(iii) Element of C: codeword in C.

(iv) Number of codewords in C, denoted by |C| –

size of C.

(v) Code of length n and size M : (n,M)-code.

0-13

Example 2.2

Code over F2 = {0, 1} – binary code.

(i) C1 = {00, 01, 10, 11}: (2,4)-code.

(ii) C2 = {000, 011, 101, 110}: (3,4)-code.

Code over F3 = {0, 1, 2} – ternary code.

0-14

2.2 Hamming Distance

Definition 2.3

x, y: words of length n over A.

(Hamming) distance from x to y, de-

noted by d(x,y) – number of places at which

x and y differ.

0-15

Example 2.4

Let A = {0, 1} and let x = 01010, y =

01101, z = 11101, then

d(x,y) = 3

d(y, z) = 1

d(z,x) = 4

0-16

2.3 Nearest Neighbour/Minimum DistanceDecoding

Suppose codewords from C are being sent over

a communication channel.

If x is received, nearest neighbour de-

coding rule (or minimum distance de-

coding rule) will decode x to cx such that

d(x, cx) = minc∈C

d(x, c). (1)

0-17

For received word x, if two or more codewords

cx satisfy (1), then

• complete decoding rule arbitrarily selects

one to be most likely word sent,

• incomplete decoding rule requests for re-

transmission.

0-18

Remark 2.5

Assume that it is much more likely that a

symbol is sent correctly than if an error occurs.

Example 2.6

Suppose codewords from binary code

C = {0000, 0011, 1000, 1100, 0001, 1001}are being sent. Assuming x = 0111 is re-

ceived, then

0-19

d(0111, 0000) = 3

d(0111, 0011) = 1

d(0111, 1000) = 4

d(0111, 1100) = 3

d(0111, 0001) = 2

d(0111, 1001) = 3

By using nearest neighbour decoding, we de-

code x to 0011.

0-20

Example 2.7

C = {000, 011}: binary code. Incomplete

decoding table for C is as follows, where “–”

means that a retransmission is sought:

0-21

received x d(x, 000) d(x, 011) decode to

000 0 2 000

100 1 3 000

010 1 1 –

001 1 1 –

110 2 2 –

101 2 2 –

011 2 0 011

111 3 1 011

0-22

2.4 Distance of a Code

Definition 2.8

|C| ≥ 2. (Minimum) distance of C,

denoted by d(C) –

d(C) = min{d(x,y) : x,y ∈ C, x 6= y}.Definition 2.9

Code of length n, size M , and distance d:

(n,M, d)-code.

n,M, d: parameters of the code.

0-23

Example 2.10

C = {00000, 00111, 11111}: binary code,

then d(C) = 2 since

d(00000, 00111) = 3,

d(00000, 11111) = 5,

d(00111, 11111) = 2.

C: binary (5,3,2)-code.

0-24

Definition 2.11

u: positive integer.

C: u-error-detecting if, whenever a code-

word incurs at least one but at most u errors,

the resulting word is not a codeword.

C: exactly u-error-detecting if it is u-

error-detecting but not (u+1)-error-detecting.

0-25

Example 2.12

Binary C = {00000, 00111, 11111}: 1-error-

detecting since:

00000 → 00111 needs to change 3 bits,

00000 → 11111 needs to change 5 bits,

00111 → 11111 needs to change 2 bits.

C exactly 1-error-detecting: changing the first

two positions of 00111 will result in another

codeword 11111 (so C not 2-error-detecting).

0-26

Theorem 2.13

C: u-error-detecting if and only if d(C) ≥u + 1, i.e.,

a code with distance d is an exactly (d−1)-

error-detecting code.

0-27

Definition 2.14

v: positive integer.

C: v-error-correcting if minimum dis-

tance decoding is able to correct v or fewer er-

rors, assuming that incomplete decoding rule

is used.

C: exactly v-error-correcting if it is v-

error-correcting but not (v+1)-error-correcting.

0-28

Example 2.15

Binary C = {000, 111}.Using minimum distance decoding,

• if 000 is sent and one error occurs, then re-

ceived word (100, 010 or 001) will be decoded

to 000;

• if 111 is sent and one error occurs, then re-

ceived word (110, 101 or 011) will be decoded

to 111.

Hence, C is 1-error-correcting.

0-29

If ≥ 2 errors occur, the decoding rule may

produce the wrong codeword.

E.g., if 000 is sent and 011 is received, then

011 will be decoded to 111 using minimum

distance decoding.

Hence, C is exactly 1-error-correcting.

0-30

Theorem 2.16

C: v-error-correcting if and only if d(C) ≥2v + 1, i.e.,

code with distance d is an exactly b(d −1)/2c-error-correcting code.

0-31

3 Linear Codes

3.1 Basic Notions

Fq: finite field of order q

Definition 3.1

Linear code C of length n over Fq: sub-

space of Fnq .

0-32

Example 3.2

Examples of linear codes:

(i) C = {(λ, λ, . . . , λ) : λ ∈ Fq}. (Repeti-

tion code)

(ii) (q = 2) C = {000, 001, 010, 011}.(iii) (q = 3) C = {0000, 1100, 2200, 0001,

0002, 1101, 1102, 2201, 2202}.

0-33

Definition 3.3

C: linear code in Fnq .

(i) Dual code of C:

C⊥ = {v ∈ Fnq : v · c = 0 for all c ∈ C}.

(ii) Dimension of linear code C: dim(C).

0-34

Theorem 3.4

C: linear code of length n over Fq, then:

(i) |C| = qdim(C);

(ii) C⊥ is a linear code and

dim(C) + dim(C⊥) = n;

(iii) (C⊥)⊥ = C.

0-35

Remark 3.5

q-ary [n, k]-code or simply [n, k]-code.

Also (n, qk)-linear code.

If distance d of C is known, then [n, k, d]-

linear code.

0-36

Definition 3.6

C: linear code.

(i) C is self-orthogonal if C ⊆ C⊥.

(ii) C is self-dual if C = C⊥.

Example 3.7

(q = 2) C = {0000, 1010, 0101, 1111} is

self-dual.

0-37

3.2 Hamming Weight

Definition 3.8

x ∈ Fnq . (Hamming) weight of x, de-

noted by wt(x): number of nonzero coordi-

nates in x, i.e.,

wt(x) = d(x,0).

Lemma 3.9

x,y ∈ Fnq . Then

d(x,y) = wt(x− y).

0-38

Definition 3.10

C: code (not necessarily linear).

Minimum (Hamming) weight wt(C) of

C: smallest of the weights of the nonzero code-

words of C.

0-39

Theorem 3.11

C: linear code over Fq.

Then d(C) = wt(C).

Example 3.12

Binary linear C = {0000, 1000, 0100, 1100}.wt(1000) = 1,

wt(0100) = 1,

wt(1100) = 2.

Hence, d(C) = 1.

0-40

3.3 Bases for Linear Codes

Algorithm A

Input: nonempty subset S of Fnq .

Output: A basis for C = < S > and

C⊥.

Description: Form the matrix A whose

rows are the words in S.

Use elementary row operations to place

A in RREF.

0-41

Let G be the k×n matrix consisting

of all the nonzero rows of the RREF:

A →

G

O

.

Rows of G give a basis for C.

G contains k leading columns.

Permute the columns of G to form

G′ = (Ik|X) .

0-42

Form a matrix H ′ as follows:

H ′ =(−XT |In−k

).

Apply the inverse of the permutation

applied to the columns of G to the

columns of H ′ to form H .

Rows of H form a basis for C⊥.

0-43

Example 3.13

Let q = 3. Find a basis for C⊥ if the RREF

of A is

G =

1 2 3 4 5 6 7 8 9 10

1 0 2 0 0 2 0 1 0 2

0 0 0 1 0 1 0 0 0 1

0 0 0 0 1 0 0 2 0 0

0 0 0 0 0 0 1 0 0 1

0 0 0 0 0 0 0 0 1 2

.

Leading columns of G: 1, 4, 5, 7 and 9. Per-

0-44

mute columns of G into the order 1, 4, 5, 7, 9,

2, 3, 6, 8, 10 to form matrix G′ = (I5|X), i.e.,

G′ =

1 4 5 7 9 2 3 6 8 10

1 0 0 0 0 0 2 2 1 2

0 1 0 0 0 0 0 1 0 1

0 0 1 0 0 0 0 0 2 0

0 0 0 1 0 0 0 0 0 1

0 0 0 0 1 0 0 0 0 2

.

Form H ′ and finally rearrange columns of

0-45

H ′ using the inverse permutation to get H :

H ′ =

1 4 5 7 9 2 3 6 8 10

0 0 0 0 0 1 0 0 0 0

1 0 0 0 0 0 1 0 0 0

1 2 0 0 0 0 0 1 0 0

2 0 1 0 0 0 0 0 1 0

1 2 0 2 1 0 0 0 0 1

,

0-46

H =

1 2 3 4 5 6 7 8 9 10

0 1 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0

1 0 0 2 0 1 0 0 0 0

2 0 0 0 1 0 0 1 0 0

1 0 0 2 0 0 2 0 1 1

.

By Algorithm A, the rows of H form a basis

for C⊥.

0-47

3.4 Generator and Parity-Check Matrices

Definition 3.14

(i) Generator matrix for linear code C: ma-

trix G whose rows form a basis for C.

(ii) Parity-check matrix H for linear code C:

generator matrix for dual code C⊥.

0-48

Definition 3.15

(i) A generator matrix of the form (Ik|X): stan-

dard form.

(ii) A parity-check matrix in the form (Y |In−k):

standard form.

0-49

Lemma 3.16

C: [n, k]-linear code over Fq, with gen-

erator matrix G.

v ∈ Fnq belongs to C⊥

mv orthogonal to every row of G;

i.e.,v ∈ C⊥ ⇔ vGT = 0.

0-50

In particular, given (n − k) × n matrix

H, then

H: parity-check matrix for C

mrows of H linearly independent

and HGT = O.

0-51

Theorem 3.17

C: linear code

H: parity-check matrix for C. Then

(i)

C has distance ≥ d

many d− 1 columns of H

are linearly independent;

0-52

(ii)

C has distance ≤ d

mH has d columns

that are linearly dependent.

0-53

Corollary 3.18

C: linear code with parity-check matrix

H.

C has distance d

many d− 1 cols of H : lin indep

and

H has d cols : lin dependent.

0-54

Example 3.19

C: binary linear code with parity-check ma-

trix

H =

10100

11010

01001

.

There are no zero columns and

no two columns of H sum to 0T ,

so any two columns of H are linearly indepen-

dent.

0-55

However, columns 1, 3, and 4 sum to 0T , and

hence are linearly dependent.

Therefore, the distance of C is d = 3.

0-56

Theorem 3.20

If G = (Ik|X) is the standard form gen-

erator matrix of an [n, k]-code C,

then a parity-check matrix for C is H =

(−XT |In−k).

Remark 3.21

Not every linear code has a generator ma-

trix in standard form.

0-57

3.5 Equivalence of Linear Codes

Definition 3.22

Two (n,M)-codes over Fq are equivalent

if one can be obtained from the other by a

combination of following operations:

(i) permutation of the n digits of the codewords;

and

(ii) multiplication of the symbols appearing in a

fixed position by a nonzero scalar.

0-58

Example 3.23

q = 3 and n = 3.

C = {000, 011, 022}.Permuting first and second positions,

followed by multiplying the third position by

2,

obtain equivalent code

C ′ = {000, 102, 201}.

0-59

Theorem 3.24

Any linear code C is equivalent to a lin-

ear code C ′ with a generator matrix in stan-

dard form.

0-60

4 Some Interesting Codes

Aq(n, d) – the largest possible size for all codes,

defined over any alphabet of size q, of length

n and of minimum distance d.

0-61

4.1 Hamming Bound and Hamming Codes

Theorem 4.1 (Hamming or sphere-packing

bound)

q > 1: integer

n, d: integers s.t. 1 ≤ d ≤ n

Aq(n, d) ≤ qn

∑b(d−1)/2ci=0

(ni

)(q − 1)i

.

Any code meeting the Hamming bound is

called a perfect code.

0-62

Definition 4.2

r ≥ 2. Binary linear code, n = 2r − 1,

parity-check matrix H : columns – all nonzero

vectors of Fr2 – binary Hamming code of

length 2r − 1. Ham(r, 2).

Remark 4.3

(i) Order of columns of H not fixed.

(ii) Rows of H linearly independent since H con-

tains all r columns of weight 1 words.

Hence, H is indeed a parity-check matrix.

0-63

Example 4.4

Ham(3, 2): length 7 with parity-check ma-

trix

H =

0 0 0 1 1 1 1

0 1 1 0 0 1 1

1 0 1 0 1 0 1

.

0-64

Proposition 4.5 (Properties of binary

Hamming codes)

(i) All binary Hamming codes of a given length

are equivalent.

(ii) Dimension of Ham(r, 2): k = 2r − 1− r.

(iii) Distance of Ham(r, 2): d = 3, hence Ham(r, 2)

is exactly 1-error-correcting.

(iv) Binary Hamming codes are perfect codes.

0-65

4.2 Golay Codes

Binary Golay Codes

Definition 4.6

G: 12× 24 matrix G = (I12|A),

where I12: 12× 12 identity matrix

A: 12× 12 matrix

0-66

A =

0 1 1 1 1 1 1 1 1 1 1 11 1 1 0 1 1 1 0 0 0 1 01 1 0 1 1 1 0 0 0 1 0 11 0 1 1 1 0 0 0 1 0 1 11 1 1 1 0 0 0 1 0 1 1 01 1 1 0 0 0 1 0 1 1 0 11 1 0 0 0 1 0 1 1 0 1 11 0 0 0 1 0 1 1 0 1 1 11 0 0 1 0 1 1 0 1 1 1 01 0 1 0 1 1 0 1 1 1 0 01 1 0 1 1 0 1 1 1 0 0 01 0 1 1 0 1 1 1 0 0 0 1

.

Binary linear code with generator matrix G –

extended binary Golay code G24.

0-67

Remark 4.7

(i) Used in Voyager 1 and 2 spacecraft – launched

towards Jupiter and Saturn in 1977 – encod-

ing and decoding of the general science and

engineering (GSE) data for the missions.

(ii) Any code equivalent to the linear code with

generator matrix G – extended binary Golay

code.

0-68

Proposition 4.8 (Properties of extended

binary Golay code)

(i) Length of G24: 24, dimension: 12.

(ii) Parity-check matrix for G24:

H = (A|I12).

(iii) G24 is self-dual, i.e., G24⊥ = G24.

(iv) Another parity-check matrix for G24:

H ′ = (I12|A)(= G).

0-69

(v) Another generator matrix for G24:

G′ = (A|I12)(= H).

(vi) Weight of every codeword in G24: multiple

of 4.

(vii) No codeword of wt 4, so d = 8.

(viii) G24: exactly 3-error-correcting code.

0-70

Definition 4.9

Binary Golay code G23 – code obtained from

G24 by deleting last coordinate of every code-

word.

0-71

Proposition 4.10 (Properties of binary

Golay code)

(i) Length of G23: 23, dimension: 12.

(ii) Parity-check matrix for G23: 11×23 matrix

H = (AT |I11).

(iii) Distance of G23 is d = 7.

(iv) G23: perfect exactly 3-error-correcting code.

0-72

Ternary Golay Codes

Definition 4.11

Extended ternary Golay code G12:

ternary linear code with generator matrix G =

(I6|B), where

B =

0 1 1 1 1 11 0 1 2 2 11 1 0 1 2 21 2 1 0 1 21 2 2 1 0 11 1 2 2 1 0

.

0-73

Remark 4.12

Any linear code equivalent to above code –

extended ternary Golay code.

G12: self-dual ternary [12, 6, 6]-code

0-74

Definition 4.13

Ternary Golay code G11: code obtained

by puncturing G12 in last coordinate.

G11 satisfies Hamming bound

hence perfect ternary [11, 6, 5]-code.

0-75

4.3 Singleton Bound and MDS Codes

Theorem 4.14 (Singleton bound)

q > 1, n and d: integers s.t. 1 ≤ d ≤ n.

Then

Aq(n, d) ≤ qn−d+1.

In particular, when q: prime power,

parameters [n, k, d] of any linear code over

Fq satisfy

k + d ≤ n + 1.

0-76

Definition 4.15

Linear code with parameters [n, k, d] such

that k + d = n + 1 – maximum distance

separable (MDS) code.

0-77

Definition 4.16

MDS code C over Fq is trivial if and only

if C satisfies one of:

(i) C = Fnq ;

(ii) C equivalent to code generated by 1 = (1, . . . , 1);

or

(iii) C equivalent to dual of code generated by 1.

Otherwise, C is nontrivial.

0-78

Remark 4.17

q = 2: all MDS codes are trivial ones.

Interesting family of MDS codes: Reed-Solomon

codes.

0-79

4.4 Reed-Solomon Codes

Definition 4.18

k: positive integer,

Pk(q): set of all polynomials of degree < k

with coefficients in Fq, i.e.,

Pk(q) = {f0 + f1x + · · · + fk−1xk−1 :

f0, f1, . . . , fk−1 ∈ Fq}.

0-80

Definition 4.19

Fq = {0, α1, . . . , αq−1}.k ≤ q − 1: positive integer,

Reed-Solomon code

RSk(q)

||{(f (α1), f (α2), . . . , f (αq−1)) :

f (x) ∈ Pk(q)}.

0-81

Remark 4.20

(i) Order of α1, . . . , αq−1 is not important.

(ii) Used in applications like: CD players, mobile

communications, digital TV and high-speed

modems, etc.

0-82

Theorem 4.21

RSk(q) is linear [q−1, k, q−k]-code over

Fq, so MDS.

0-83

References

[1] R. Hill, A First Course in Coding Theory,

Clarendon Press, Oxford, 1986

[2] W.C. Huffman & V. Pless, Fundamentals

of Error-Correcting Codes, Cambridge

University Press, 2003

[3] S. Ling & C. Xing, Coding Theory:

A First Course, Cambridge University

Press, 2004

0-84

[4] F.J. MacWilliams & N.J.A. Sloane,

The Theory of Error-Correcting Codes,

North-Holland, 1977

[5] V. Pless, Introduction to the Theory of

Error-Correcting Codes, Third Edition,

John Wiley & Sons, 1998

0-85