Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Coding Theory and Applications (I):
A Quick Introduction to Coding Theory
Pre-Conference Tutorial, AMC 2005
San LingDivision of Mathematical Sciences
School of Physical and Mathematical Sciences
Nanyang Technological University, Singapore
0-0
1 Introduction
Information media – not absolutely reliable
because of noise or interference.
Coding theory – detects/corrects errors.
source coding and channel coding.
Source coding – changing message source to
a suitable code to be transmitted through the
channel. E.g., ASCII code.
0-1
A simple communication model:
Message Source Receiver
↓ ↑Source Encoder −→ Channel −→ Source Decoder
0-2
Example 1.1
apple → 00, banana → 01,
cherry → 10, grape → 11.
Suppose “apple” (00) transmitted.
Assume 01 received.
apple banana
↓ ↑00 −→Channel−→ 01
↑noise
0-3
Receiver may not realize message was cor-
rupted.
Channel coding – encode message again after
source coding by introducing some form of re-
dundancy so that errors can be detected or
even corrected.
0-4
Message Source Receiver
↓ ↑Source Encoder Source Decoder
↓ ↑Channel Encoder −→ Channel −→ Channel Decoder
0-5
Example 1.2
In Example 1.1, introduce redundancy of 1
bit:
00 → 000, 01 → 011,
10 → 101, 11 → 110.
Suppose “apple” (000) is transmitted over a
noisy channel, and only one error.
Received word is 100, 010 or 001 – none among
encoded messages.
0-6
Detects errors at the cost of reducing trans-
mission speed (transmit 3 bits for a message
of 2 bits).
Does not correct errors.
E.g., if 100 is received, then do not know whether
100 comes from 000, 110, or 101.
With more redundancy, can correct errors. E.g.,
00 → 00000, 01 → 01111,
10 → 10110, 11 → 11001.
0-7
Suppose “apple” is transmitted over a noisy
channel, and only one error.
Received word must be one of: 10000, 01000,
00100, 00010, 00001.
Assume 10000 received.
Can be sure that 10000 comes from 00000 be-
cause there are at least two errors between
10000 and each of the others: 01111, 10110
and 11001.
0-8
apple apple
↓ ↑00 00000
↓ ↑00000 −→Channel −→ 10000
↑noise
0-9
Goal of channel coding – to construct encoder
and decoder for
1. fast encoding of messages,
2. easy transmission of encoded messages,
3. fast decoding of received messages,
4. maximum transfer of information per unit
time, and
5. maximal detection or correction capabil-
ity.
0-10
Mathematically, the primary goal – 4 & 5 (but
not compatible).
Therefore, need trade-off.
0-11
2 Error Detection, Correction and
Decoding
2.1 Some definitions
Definition 2.1
A = {a1, a2, . . . , aq} – set of size q, – code
alphabet; its elements – code symbols.
(i) q-ary word of length n over A: sequence
w = w1w2 · · ·wn or vector (w1, . . . , wn) in
An.
0-12
(ii) q-ary block code of length n over A: non-
empty set C of An.
(iii) Element of C: codeword in C.
(iv) Number of codewords in C, denoted by |C| –
size of C.
(v) Code of length n and size M : (n,M)-code.
0-13
Example 2.2
Code over F2 = {0, 1} – binary code.
(i) C1 = {00, 01, 10, 11}: (2,4)-code.
(ii) C2 = {000, 011, 101, 110}: (3,4)-code.
Code over F3 = {0, 1, 2} – ternary code.
0-14
2.2 Hamming Distance
Definition 2.3
x, y: words of length n over A.
(Hamming) distance from x to y, de-
noted by d(x,y) – number of places at which
x and y differ.
0-15
Example 2.4
Let A = {0, 1} and let x = 01010, y =
01101, z = 11101, then
d(x,y) = 3
d(y, z) = 1
d(z,x) = 4
0-16
2.3 Nearest Neighbour/Minimum DistanceDecoding
Suppose codewords from C are being sent over
a communication channel.
If x is received, nearest neighbour de-
coding rule (or minimum distance de-
coding rule) will decode x to cx such that
d(x, cx) = minc∈C
d(x, c). (1)
0-17
For received word x, if two or more codewords
cx satisfy (1), then
• complete decoding rule arbitrarily selects
one to be most likely word sent,
• incomplete decoding rule requests for re-
transmission.
0-18
Remark 2.5
Assume that it is much more likely that a
symbol is sent correctly than if an error occurs.
Example 2.6
Suppose codewords from binary code
C = {0000, 0011, 1000, 1100, 0001, 1001}are being sent. Assuming x = 0111 is re-
ceived, then
0-19
d(0111, 0000) = 3
d(0111, 0011) = 1
d(0111, 1000) = 4
d(0111, 1100) = 3
d(0111, 0001) = 2
d(0111, 1001) = 3
By using nearest neighbour decoding, we de-
code x to 0011.
0-20
Example 2.7
C = {000, 011}: binary code. Incomplete
decoding table for C is as follows, where “–”
means that a retransmission is sought:
0-21
received x d(x, 000) d(x, 011) decode to
000 0 2 000
100 1 3 000
010 1 1 –
001 1 1 –
110 2 2 –
101 2 2 –
011 2 0 011
111 3 1 011
0-22
2.4 Distance of a Code
Definition 2.8
|C| ≥ 2. (Minimum) distance of C,
denoted by d(C) –
d(C) = min{d(x,y) : x,y ∈ C, x 6= y}.Definition 2.9
Code of length n, size M , and distance d:
(n,M, d)-code.
n,M, d: parameters of the code.
0-23
Example 2.10
C = {00000, 00111, 11111}: binary code,
then d(C) = 2 since
d(00000, 00111) = 3,
d(00000, 11111) = 5,
d(00111, 11111) = 2.
C: binary (5,3,2)-code.
0-24
Definition 2.11
u: positive integer.
C: u-error-detecting if, whenever a code-
word incurs at least one but at most u errors,
the resulting word is not a codeword.
C: exactly u-error-detecting if it is u-
error-detecting but not (u+1)-error-detecting.
0-25
Example 2.12
Binary C = {00000, 00111, 11111}: 1-error-
detecting since:
00000 → 00111 needs to change 3 bits,
00000 → 11111 needs to change 5 bits,
00111 → 11111 needs to change 2 bits.
C exactly 1-error-detecting: changing the first
two positions of 00111 will result in another
codeword 11111 (so C not 2-error-detecting).
0-26
Theorem 2.13
C: u-error-detecting if and only if d(C) ≥u + 1, i.e.,
a code with distance d is an exactly (d−1)-
error-detecting code.
0-27
Definition 2.14
v: positive integer.
C: v-error-correcting if minimum dis-
tance decoding is able to correct v or fewer er-
rors, assuming that incomplete decoding rule
is used.
C: exactly v-error-correcting if it is v-
error-correcting but not (v+1)-error-correcting.
0-28
Example 2.15
Binary C = {000, 111}.Using minimum distance decoding,
• if 000 is sent and one error occurs, then re-
ceived word (100, 010 or 001) will be decoded
to 000;
• if 111 is sent and one error occurs, then re-
ceived word (110, 101 or 011) will be decoded
to 111.
Hence, C is 1-error-correcting.
0-29
If ≥ 2 errors occur, the decoding rule may
produce the wrong codeword.
E.g., if 000 is sent and 011 is received, then
011 will be decoded to 111 using minimum
distance decoding.
Hence, C is exactly 1-error-correcting.
0-30
Theorem 2.16
C: v-error-correcting if and only if d(C) ≥2v + 1, i.e.,
code with distance d is an exactly b(d −1)/2c-error-correcting code.
0-31
3 Linear Codes
3.1 Basic Notions
Fq: finite field of order q
Definition 3.1
Linear code C of length n over Fq: sub-
space of Fnq .
0-32
Example 3.2
Examples of linear codes:
(i) C = {(λ, λ, . . . , λ) : λ ∈ Fq}. (Repeti-
tion code)
(ii) (q = 2) C = {000, 001, 010, 011}.(iii) (q = 3) C = {0000, 1100, 2200, 0001,
0002, 1101, 1102, 2201, 2202}.
0-33
Definition 3.3
C: linear code in Fnq .
(i) Dual code of C:
C⊥ = {v ∈ Fnq : v · c = 0 for all c ∈ C}.
(ii) Dimension of linear code C: dim(C).
0-34
Theorem 3.4
C: linear code of length n over Fq, then:
(i) |C| = qdim(C);
(ii) C⊥ is a linear code and
dim(C) + dim(C⊥) = n;
(iii) (C⊥)⊥ = C.
0-35
Remark 3.5
q-ary [n, k]-code or simply [n, k]-code.
Also (n, qk)-linear code.
If distance d of C is known, then [n, k, d]-
linear code.
0-36
Definition 3.6
C: linear code.
(i) C is self-orthogonal if C ⊆ C⊥.
(ii) C is self-dual if C = C⊥.
Example 3.7
(q = 2) C = {0000, 1010, 0101, 1111} is
self-dual.
0-37
3.2 Hamming Weight
Definition 3.8
x ∈ Fnq . (Hamming) weight of x, de-
noted by wt(x): number of nonzero coordi-
nates in x, i.e.,
wt(x) = d(x,0).
Lemma 3.9
x,y ∈ Fnq . Then
d(x,y) = wt(x− y).
0-38
Definition 3.10
C: code (not necessarily linear).
Minimum (Hamming) weight wt(C) of
C: smallest of the weights of the nonzero code-
words of C.
0-39
Theorem 3.11
C: linear code over Fq.
Then d(C) = wt(C).
Example 3.12
Binary linear C = {0000, 1000, 0100, 1100}.wt(1000) = 1,
wt(0100) = 1,
wt(1100) = 2.
Hence, d(C) = 1.
0-40
3.3 Bases for Linear Codes
Algorithm A
Input: nonempty subset S of Fnq .
Output: A basis for C = < S > and
C⊥.
Description: Form the matrix A whose
rows are the words in S.
Use elementary row operations to place
A in RREF.
0-41
Let G be the k×n matrix consisting
of all the nonzero rows of the RREF:
A →
G
O
.
Rows of G give a basis for C.
G contains k leading columns.
Permute the columns of G to form
G′ = (Ik|X) .
0-42
Form a matrix H ′ as follows:
H ′ =(−XT |In−k
).
Apply the inverse of the permutation
applied to the columns of G to the
columns of H ′ to form H .
Rows of H form a basis for C⊥.
0-43
Example 3.13
Let q = 3. Find a basis for C⊥ if the RREF
of A is
G =
1 2 3 4 5 6 7 8 9 10
1 0 2 0 0 2 0 1 0 2
0 0 0 1 0 1 0 0 0 1
0 0 0 0 1 0 0 2 0 0
0 0 0 0 0 0 1 0 0 1
0 0 0 0 0 0 0 0 1 2
.
Leading columns of G: 1, 4, 5, 7 and 9. Per-
0-44
mute columns of G into the order 1, 4, 5, 7, 9,
2, 3, 6, 8, 10 to form matrix G′ = (I5|X), i.e.,
G′ =
1 4 5 7 9 2 3 6 8 10
1 0 0 0 0 0 2 2 1 2
0 1 0 0 0 0 0 1 0 1
0 0 1 0 0 0 0 0 2 0
0 0 0 1 0 0 0 0 0 1
0 0 0 0 1 0 0 0 0 2
.
Form H ′ and finally rearrange columns of
0-45
H ′ using the inverse permutation to get H :
H ′ =
1 4 5 7 9 2 3 6 8 10
0 0 0 0 0 1 0 0 0 0
1 0 0 0 0 0 1 0 0 0
1 2 0 0 0 0 0 1 0 0
2 0 1 0 0 0 0 0 1 0
1 2 0 2 1 0 0 0 0 1
,
0-46
H =
1 2 3 4 5 6 7 8 9 10
0 1 0 0 0 0 0 0 0 0
1 0 1 0 0 0 0 0 0 0
1 0 0 2 0 1 0 0 0 0
2 0 0 0 1 0 0 1 0 0
1 0 0 2 0 0 2 0 1 1
.
By Algorithm A, the rows of H form a basis
for C⊥.
0-47
3.4 Generator and Parity-Check Matrices
Definition 3.14
(i) Generator matrix for linear code C: ma-
trix G whose rows form a basis for C.
(ii) Parity-check matrix H for linear code C:
generator matrix for dual code C⊥.
0-48
Definition 3.15
(i) A generator matrix of the form (Ik|X): stan-
dard form.
(ii) A parity-check matrix in the form (Y |In−k):
standard form.
0-49
Lemma 3.16
C: [n, k]-linear code over Fq, with gen-
erator matrix G.
v ∈ Fnq belongs to C⊥
mv orthogonal to every row of G;
i.e.,v ∈ C⊥ ⇔ vGT = 0.
0-50
In particular, given (n − k) × n matrix
H, then
H: parity-check matrix for C
mrows of H linearly independent
and HGT = O.
0-51
Theorem 3.17
C: linear code
H: parity-check matrix for C. Then
(i)
C has distance ≥ d
many d− 1 columns of H
are linearly independent;
0-52
(ii)
C has distance ≤ d
mH has d columns
that are linearly dependent.
0-53
Corollary 3.18
C: linear code with parity-check matrix
H.
C has distance d
many d− 1 cols of H : lin indep
and
H has d cols : lin dependent.
0-54
Example 3.19
C: binary linear code with parity-check ma-
trix
H =
10100
11010
01001
.
There are no zero columns and
no two columns of H sum to 0T ,
so any two columns of H are linearly indepen-
dent.
0-55
However, columns 1, 3, and 4 sum to 0T , and
hence are linearly dependent.
Therefore, the distance of C is d = 3.
0-56
Theorem 3.20
If G = (Ik|X) is the standard form gen-
erator matrix of an [n, k]-code C,
then a parity-check matrix for C is H =
(−XT |In−k).
Remark 3.21
Not every linear code has a generator ma-
trix in standard form.
0-57
3.5 Equivalence of Linear Codes
Definition 3.22
Two (n,M)-codes over Fq are equivalent
if one can be obtained from the other by a
combination of following operations:
(i) permutation of the n digits of the codewords;
and
(ii) multiplication of the symbols appearing in a
fixed position by a nonzero scalar.
0-58
Example 3.23
q = 3 and n = 3.
C = {000, 011, 022}.Permuting first and second positions,
followed by multiplying the third position by
2,
obtain equivalent code
C ′ = {000, 102, 201}.
0-59
Theorem 3.24
Any linear code C is equivalent to a lin-
ear code C ′ with a generator matrix in stan-
dard form.
0-60
4 Some Interesting Codes
Aq(n, d) – the largest possible size for all codes,
defined over any alphabet of size q, of length
n and of minimum distance d.
0-61
4.1 Hamming Bound and Hamming Codes
Theorem 4.1 (Hamming or sphere-packing
bound)
q > 1: integer
n, d: integers s.t. 1 ≤ d ≤ n
Aq(n, d) ≤ qn
∑b(d−1)/2ci=0
(ni
)(q − 1)i
.
Any code meeting the Hamming bound is
called a perfect code.
0-62
Definition 4.2
r ≥ 2. Binary linear code, n = 2r − 1,
parity-check matrix H : columns – all nonzero
vectors of Fr2 – binary Hamming code of
length 2r − 1. Ham(r, 2).
Remark 4.3
(i) Order of columns of H not fixed.
(ii) Rows of H linearly independent since H con-
tains all r columns of weight 1 words.
Hence, H is indeed a parity-check matrix.
0-63
Example 4.4
Ham(3, 2): length 7 with parity-check ma-
trix
H =
0 0 0 1 1 1 1
0 1 1 0 0 1 1
1 0 1 0 1 0 1
.
0-64
Proposition 4.5 (Properties of binary
Hamming codes)
(i) All binary Hamming codes of a given length
are equivalent.
(ii) Dimension of Ham(r, 2): k = 2r − 1− r.
(iii) Distance of Ham(r, 2): d = 3, hence Ham(r, 2)
is exactly 1-error-correcting.
(iv) Binary Hamming codes are perfect codes.
0-65
4.2 Golay Codes
Binary Golay Codes
Definition 4.6
G: 12× 24 matrix G = (I12|A),
where I12: 12× 12 identity matrix
A: 12× 12 matrix
0-66
A =
0 1 1 1 1 1 1 1 1 1 1 11 1 1 0 1 1 1 0 0 0 1 01 1 0 1 1 1 0 0 0 1 0 11 0 1 1 1 0 0 0 1 0 1 11 1 1 1 0 0 0 1 0 1 1 01 1 1 0 0 0 1 0 1 1 0 11 1 0 0 0 1 0 1 1 0 1 11 0 0 0 1 0 1 1 0 1 1 11 0 0 1 0 1 1 0 1 1 1 01 0 1 0 1 1 0 1 1 1 0 01 1 0 1 1 0 1 1 1 0 0 01 0 1 1 0 1 1 1 0 0 0 1
.
Binary linear code with generator matrix G –
extended binary Golay code G24.
0-67
Remark 4.7
(i) Used in Voyager 1 and 2 spacecraft – launched
towards Jupiter and Saturn in 1977 – encod-
ing and decoding of the general science and
engineering (GSE) data for the missions.
(ii) Any code equivalent to the linear code with
generator matrix G – extended binary Golay
code.
0-68
Proposition 4.8 (Properties of extended
binary Golay code)
(i) Length of G24: 24, dimension: 12.
(ii) Parity-check matrix for G24:
H = (A|I12).
(iii) G24 is self-dual, i.e., G24⊥ = G24.
(iv) Another parity-check matrix for G24:
H ′ = (I12|A)(= G).
0-69
(v) Another generator matrix for G24:
G′ = (A|I12)(= H).
(vi) Weight of every codeword in G24: multiple
of 4.
(vii) No codeword of wt 4, so d = 8.
(viii) G24: exactly 3-error-correcting code.
0-70
Definition 4.9
Binary Golay code G23 – code obtained from
G24 by deleting last coordinate of every code-
word.
0-71
Proposition 4.10 (Properties of binary
Golay code)
(i) Length of G23: 23, dimension: 12.
(ii) Parity-check matrix for G23: 11×23 matrix
H = (AT |I11).
(iii) Distance of G23 is d = 7.
(iv) G23: perfect exactly 3-error-correcting code.
0-72
Ternary Golay Codes
Definition 4.11
Extended ternary Golay code G12:
ternary linear code with generator matrix G =
(I6|B), where
B =
0 1 1 1 1 11 0 1 2 2 11 1 0 1 2 21 2 1 0 1 21 2 2 1 0 11 1 2 2 1 0
.
0-73
Remark 4.12
Any linear code equivalent to above code –
extended ternary Golay code.
G12: self-dual ternary [12, 6, 6]-code
0-74
Definition 4.13
Ternary Golay code G11: code obtained
by puncturing G12 in last coordinate.
G11 satisfies Hamming bound
hence perfect ternary [11, 6, 5]-code.
0-75
4.3 Singleton Bound and MDS Codes
Theorem 4.14 (Singleton bound)
q > 1, n and d: integers s.t. 1 ≤ d ≤ n.
Then
Aq(n, d) ≤ qn−d+1.
In particular, when q: prime power,
parameters [n, k, d] of any linear code over
Fq satisfy
k + d ≤ n + 1.
0-76
Definition 4.15
Linear code with parameters [n, k, d] such
that k + d = n + 1 – maximum distance
separable (MDS) code.
0-77
Definition 4.16
MDS code C over Fq is trivial if and only
if C satisfies one of:
(i) C = Fnq ;
(ii) C equivalent to code generated by 1 = (1, . . . , 1);
or
(iii) C equivalent to dual of code generated by 1.
Otherwise, C is nontrivial.
0-78
Remark 4.17
q = 2: all MDS codes are trivial ones.
Interesting family of MDS codes: Reed-Solomon
codes.
0-79
4.4 Reed-Solomon Codes
Definition 4.18
k: positive integer,
Pk(q): set of all polynomials of degree < k
with coefficients in Fq, i.e.,
Pk(q) = {f0 + f1x + · · · + fk−1xk−1 :
f0, f1, . . . , fk−1 ∈ Fq}.
0-80
Definition 4.19
Fq = {0, α1, . . . , αq−1}.k ≤ q − 1: positive integer,
Reed-Solomon code
RSk(q)
||{(f (α1), f (α2), . . . , f (αq−1)) :
f (x) ∈ Pk(q)}.
0-81
Remark 4.20
(i) Order of α1, . . . , αq−1 is not important.
(ii) Used in applications like: CD players, mobile
communications, digital TV and high-speed
modems, etc.
0-82
Theorem 4.21
RSk(q) is linear [q−1, k, q−k]-code over
Fq, so MDS.
0-83
References
[1] R. Hill, A First Course in Coding Theory,
Clarendon Press, Oxford, 1986
[2] W.C. Huffman & V. Pless, Fundamentals
of Error-Correcting Codes, Cambridge
University Press, 2003
[3] S. Ling & C. Xing, Coding Theory:
A First Course, Cambridge University
Press, 2004
0-84
[4] F.J. MacWilliams & N.J.A. Sloane,
The Theory of Error-Correcting Codes,
North-Holland, 1977
[5] V. Pless, Introduction to the Theory of
Error-Correcting Codes, Third Edition,
John Wiley & Sons, 1998
0-85