Study Guide Part a Topic1

F A C U L T Y O F A R T S A N D E D U C A T I O N

Applications of Mathematics

STUDY GUIDE

PART A

This Study Guide has been produced for EMM400 Applications of Mathematics, which is one of the units offered by the Faculty of Arts and Education in Deakin University’s flexible learning program. Published by Deakin University, Geelong, Victoria 3217, Australia First published 2009 © Deakin University 2009 Produced by the Faculty of Arts and Education Printed in Penang, Malaysia These materials were printed from ready-to-print copy submitted by Susie Groves. Unless otherwise indicated, the following notice will apply:

COMMONWEALTH OF AUSTRALIA Copyright Regulations 1969

WARNING This material has been reproduced and communicated to you by or on behalf of Deakin University

pursuant to Part VB of the Copyright Act 1968 (the Act). The material in this communication may be subject to copyright under the Act.

Any further reproduction or communication of this material by you may be the subject of copyright protection under the Act.

Do not remove this notice.

Deakin University CRICOS Provider Code: 00113B (VIC)

Contents

Part A

Topic 1. Introduction 1

References 2

Topic 2. Introducing error-correcting codes 3

The Mariner 6 space mission 3

Error-correcting codes 6

An example of a single-error-correcting code 7

Repetition codes 9

A decoding algorithm for a repetition code of length 5 9

Are repetition codes worth bothering with? 10

Modular arithmetic, groups and fields 12

Matrix calculations over the field GF(2) 14

Single parity-check codes 15

The main problem 16

Shannon’s Theorem 19

Answers to activities 20

References 21

Topic 3. Linear codes 22

An example of a linear code 22

Finding the words corresponding to a given syndrome 25

Cosets 28

Decoding a received word 29

Linear codes in general 30

Hamming codes 31

Decoding a received word revisited 32

Answers to selected activities 36

References 36

T O P I C 1

Introduction P R E P A R E D F O R T H I S U N I T B Y S U S I E G R O V E S

This unit, EMM400 Applications of Mathematics, draws on topics you have studied in earlier units, to demonstrate their use in a range of contemporary contexts.

Much of this unit will be concerned with two quite different aspects of codes and coding – secret codes and the problems associated with encryption, and what are called error-correcting codes and the problems associated with ensuring high quality storage and transmission of digital data.

The history of secret codes and encryption is long and fascinating – too long for us to go into great detail about the historical aspects in the ordinary course of your lectures. For this reason, you will have a significant group project that will allow you to look at and “teach” the rest of your class about the history of codes and how some of these aspects can be incorporated into your teaching – perhaps to a mathematics club at school, if not in your regular classes. This section will also deal with the rather amazing new developments in the area known as public-key-cryptography. The need for powerful (and public) encryption has grown out of our ever increasing reliance on digital information – you will no doubt have seen yourselves how when you sometimes go to or leave a web page you are told that the page is (or is not) encrypted for security purposes. Security issues arise not only in the public domain but also in the secrecy that surrounds military operations. Not only has this resulted in mathematical research being classified as secret, but also in conflict between the police and governments trying to restrict the use of encryption which they regard as a tool for terrorists, and civil libertarians who argue that encryption is necessary to protect privacy. So we can see that mathematics is not always a neutral and “pure” science – even when the techniques used may come from the “purest” mathematics of all – number theory!

Another product of the digital age is error-correcting codes. This area of mathematics began just 60 years ago in 1948. A mere thirty years later, MacWilliams and Sloane (1977) listed almost 1500 articles in their bibliography and the area has continued to grow at a rapid rate in the thirty years since then! For example, a search in Google Scholar for articles published in 2008 with the phrase “error-correcting code” in the title gives 735 results! Typically these articles use very sophisticated mathematics, but in this unit we will look at some of the earliest work in the area to get a small idea of what it is all about.

2 E M M 4 0 0 T O P I C 1

The remaining area we will look at will be the use of mathematical modeling in biology and ecology. This will draw on and extend the recent work you have done with matrices.

This unit is intended as what is often called a “capstone” unit in your mathematics discipline major. It aims to draw together work from many of the other units in order to allow you to: find out about some modern applications of various mathematical topics you have studied; understand more about the nature of mathematics and its applications; and become increasingly aware of mathematics as an ongoing cultural activity.

I hope you will enjoy this unit and see it as an introduction to a life-long interest in the applications of mathematics in the contemporary world.

References

MacWilliams, F.J. & Sloane, N.J.A. (1977). The theory of error-correcting codes. Amsterdam: North-Holland.

T O P I C 2

Introducing error-correcting codes P R E P A R E D F O R T H I S U N I T B Y S U S I E G R O V E S

Have you ever wondered how satellites send photographs back to earth? Or why you can’t just invent a credit card number when you try to buy something on the internet? Or how you can be sure when you deposit money into an ATM that the correct amount will go to your account?

The topic of error-correcting codes looks at an important application of a number of different strands of pure mathematics to data transmission and storage – something that is becoming more and more important every day.

We will begin by looking at:

• what are error-correcting codes and why we need them;

• the Mariner 6 space mission; and

• some examples of error-correcting codes.

The Mariner 6 space mission

In 1965, the USA’s Mariner 4 was the first spaceship to photograph Mars. At that time, each picture took 8 hours to transmit. Later Mariner missions, such as Mariner 6, produced improved pictures, using what are called error-correcting codes.

FIGURE 2.1 Mariner 6 spaceship sending pictures from Mars

4 E M M 4 0 0 T O P I C 2

Before going on to look at error-correcting codes in more detail, we will first explain how the Mariner 6 spaceship sent back pictures to Earth from Mars in 1969.

In order to transmit pictures, a fine grid was placed on the picture and each “square” or what we would now refer to as a pixel, was given a degree of blackness ranging from 0 to 63 – see Figure 2.2.

Each of these numbers was written as a sequence of six 0’s and 1’s, for example by writing it in the binary system (that is, in base 2), as shown below.

0

000000

1 000001

2 000010

3 000011

4 000100

5 000101

6 000110

7 000111

8 001000

9 001001

…

43 101011

…

63 111111

So we have “degree of blackness = 43” → 101011.

In the case of Mariner 6, each picture was broken up into 700 x 832 squares, so if each square was coded using 6 binary digits, each picture would consist of a sequence of 6 x 700 x 832 = 3 494 400 binary digits.

However, while the degree of blackness of each square consisted of a six binary digits, the message that was sent actually used many more digits for each degree of blackness – in fact 32 binary digits were sent for each square, so each picture consisted of a sequence of 32 x 700 x 832 = 18 636 800 binary digits.

So we have:

One picture → long sequence of 0’s and 1’s → Jet Propulsion Laboratory of the California Institute of Technology.

FIGURE 2.2 Converting Mariner 6 pictures to binary digits

I N T R O D U C I N G E R R O R - C O R R E C T I N G C O D E S 5

Figure 2.3 shows the process of transmitting a message through a communication channel in the form of a binary code.

Message to be transmitted → Coder →

Binary code to be transmitted

↓

Decoder ← Binary code

received ← Channel

transmitting code

↓

Message received

However, it is not quite as simple as this. Regardless of whether the communication channel is a satellite link, or wireless, or a telephone line, typically the communication channel adds noise, which causes errors – as we have all experienced for example when we are speaking on a “bad” telephone line. Figure 2.4 illustrates how this might happen.

In this example, the message 01101 was sent, but the message received was unclear. It does not seem possible to decipher the middle digit – the received message looks like 01?01.

What should a person do if they receive a message like this? The answer depends on the situation. For example, it may be possible to ask for the message to be sent again – for example if it was a telephone message you could ask the person to repeat it, or if someone is swiping your credit card they will try again if the numbers are not clear (payment not approved) rather than “guess” the missing digit. But in other cases, like the Mariner space mission, you cannot ask for the picture to be sent again and it may be worth taking the risk and decoding the message as best we can. (Of course, these messages are not really being deciphered by people but by computers.)

In general, the effect of noise in a communication channel is to produce errors so that the message received is different from the message sent. So, in our Mariner 6 example, we could get the situation shown in Figure 2.5 where 43 was transmitted, but the message received was 11.

FIGURE 2.3 Transmitting data through a communication channel

FIGURE 2.4 The effect of noise in a communication channel

6 E M M 4 0 0 T O P I C 2

If we have many errors like this they will affect the quality of the picture that is received and we cannot really ask for the message to be repeated.

This is the reason we need error-correcting codes.

Error-correcting codes

Error-correcting codes combat error by introducing redundancy – by including more symbols than is necessary for the message.

In ordinary language, there is usually a lot of redundancy, usually supplied through our knowledge of both the language and the context in which it is being used – this is how we can usually recognize typographical errors and correct them as we read.

For example, if I sent the word misprint, this could easily arrive as mispront or perhaps as misprink. In the context of this topic, you might easily recognize not only that there was a typographical error, but also could guess that the most likely word to have been sent was misprint.

So, as mentioned earlier, Mariner 6 was using 6 binary digits to encode each of the tiny squares (or pixels) in each picture of Mars. But instead of just sending 6 digits for each square, it sent 32 digits, with 26 (= 32 – 6) being redundant digits. But the really amazing part is that every string with less than 8 errors could be interpreted correctly!

So we have: Each string of six 0’s and 1’s → string of thirty-two 0’s and 1’s → strings with < 8 errors decoded correctly.

How can this happen?

We will illustrate this shortly using a smaller example. But before we do, we will look at one way that we might write letter or numbers as string of binary digits.

The process of encoding a message usually begins with the conversion of “ordinary” text into a string of numbers by means of a “digital alphabet’ like the one shown below. In the code shown in Figure 2.6, each letter (and a few punctuation marks) are represented by a sequence of 0’s and 1’s of length 5 – we can think of these sequences as being the numbers between 0 and 32 written in the binary (base 2) system.

FIGURE 2.5 Errors introduced by noise


= 00000 A = 00001 B = 00010 C = 00011 D = 00100

E = 00101 F = 00110 G = 00111 H = 01000 I = 01001

J = 01010 K = 01011 L = 01100 M = 01101 N = 01110

O = 01111 P = 10000 Q = 10001 R = 10010 S = 10011

T = 10100 U = 10101 V = 10110 W = 10111 X = 11000

Y = 110001 Z = 11010 , = 11011 . = 11100 ? = 11101

: = 11110 ; = 11111

So, for example, if I were to write my name in this code I would get:

SUSIE – 10011 10101 10011 01001 00101

Write your name using the binary code above.

An example of a single-error-correcting code

In the case of Mariner 6, a powerful Reed-Muller code was used for error correction. As stated earlier, this transformed 6 binary digit messages into 32 binary digit strings called codewords.

We will use a simple example to illustrate how we can not only detect errors but also correct them.

In this example, we will assume that every message we want to transmit consists of just 3 binary digits. There are 8 such possible messages, so we could think of them as representing the integers from 0 to 7.

In this example, we will add an extra 5 redundant digits to each message as shown below to give codewords of length 8. (Right now we won’t try to explain how we choose the extra digits.)

0 = 000 000 00000

1 = 001 001 10110

2 = 010 010 10101

3 = 011 → 011 00011

4 = 100 100 10011

5 = 101 101 00101

6 = 110 110 00110

7 = 111 111 10000

Consider the codeword 00110110 corresponding to the integer 1. If we compare it with the codeword 00000000 corresponding to the integer 0, it is easy to see that these two codewords differ in exactly four places (the third, fourth sixth and seventh places). Similarly, if we compare 00110110 with the codeword 01010101 , we see that again the two codewords differ in exactly four places – this time the second, third, seventh and eighth places.

FIGURE 2.6 A binary code using the numbers 0 to 32 in base 2

ACTIVITY 2.1

FIGURE 2.7 A single-error-correcting code of length 8

8 E M M 4 0 0 T O P I C 2

Check that every codeword differs from every other codeword in exactly four places.

Notice that because we only have 8 messages, we have only 8 codewords out of a possible 28 = 256 strings of eight binary digits. This is what is going to allow us to not only detect when an error has occurred but also to correct a single error, as we will show now.

Imagine that we receive the following message: 00111110

It is easy to check that this is not one of our codewords, so an error must have occurred – actually we cannot be sure that only one error has occurred, instead we know that at least one error must have occurred.

While we cannot be certain what the original message was, we can use what is called the principal of maximum likelihood to decode the received message. We do this by comparing the received message with the 8 codewords and look to see which codeword is closest to the received message.

When we do this, we see that the codeword closest to 00111 110 is 00110 110. It differs in just one place – the fifth place (underlined here).

Since every codeword differs from every other one in exactly four places, our received message 00111110 must differ from all the others in at least three places.

So, as long as we can assume that errors rarely occur, the most likely codeword to have been transmitted is 00110110 . In this case, as long as there was only one error (and this is the most likely case) we are able to correct it.

This is true for all cases when just a single error occurs – hence the name single-error-correcting code.

In this example, we have:

8 digit codewords with

3 information digits and

5 redundant digits.

So we can say this code has information rate = 8

3

. In general, if we have

n digits }

– – – … – – – … – – }

}

k message digits

r check (redundant)

digits

the information rate, R = n

k.

For Mariner 6, the information rate was R =32

6.

ACTIVITY 2.2

INFORMATION RATE OF A CODE


We will look at some more examples now.

Repetition codes

One simple way of introducing redundancy is to repeat everything. So if we have a message we could decide to code it by repeating each digit n times.

For example, if n = 5, we get a repetition code of length 5.

In my example earlier, my name would be coded as shown below.

S → 10011 → 11111 00000 00000 11111 11111

U → 10101 → 11111 00000 11111 00000 11111

S → 10011 → 11111 00000 00000 11111 11111

I → 01001 → 00000 11111 00000 00000 11111

E → 00101 → 00000 00000 11111 00000 11111

Notice that if we send S = 10011 as 11111 00000 00000 11111

11111 , we can still receive this as any sequence of 0’s and 1’s of length 25.

We need a rule (or algorithm) for decoding a received message.

Moreover, because it will be a computer decoding messages, not a person who might be able to make educated guesses based on the context, etc, we need an exact rule.

For example what should we do if we receive the following message:

11011 00110 11000 10000 10111 ?

A decoding algorithm for a repetition code of length 5 The following is an example of a decoding algorithm for any one 5-digit block:

1. Count the number of 1’s.

2. If number of 1’s ≥ 3 , write 11111 .

3. If number of 1’s ≤ 2 , write 00000 .

Notice that this code can correct 2 errors, but it has a very low information rate

of 5

1.

If we chose n = 4 (that is, repeated each digit 4 times), what should we do if we received 0011 ? Maybe we could ask for the message to be repeated, but usually this is not an option!

1 0 E M M 4 0 0 T O P I C 2

Are repetition codes worth bothering with? Even with repetition codes, errors still can occur. So are they worth bothering with?

How much do they improve our chances of coding the message correctly?

To answer this question, we will assume that we have what is called a binary symmetric channel – that is, the probability of receiving an incorrect symbol is the same whether the symbol 0 or the symbol 1 was sent. This is illustrated in Figure 2.8.

Binary symmetric channel

probability of incorrect symbol being received = p

For example, if p = 100

1 , that is, the probability of a single digit being

received incorrectly is 100

1 = 0.01, then the probability of a single digit being

received correctly is 100

99 = 0.99.

We will assume that errors occur randomly – that is, independently of one another. [Notice that this may be an invalid assumption – for example, lightning may cause errors to occur close together.]

To simplify our calculations we will use a repetition code of length 3.

Will we improve our 100

99 = 0.99 probability of correctly receiving a “word”

consisting of a single digit?

Possible message

Message sent

Possible messages received

Decoded message

0 → 000 →

000 001

010 100 → 0

1 → 111 →

101 011

110 111 → 1

Assume we send the message 000 .

Consider the probabilities for each of the possible messages received:

FIGURE 2.8 A binary symmetric channel transmitting information

message received }{ 0

1

0

1

p

pmessage sent

1 – p

1 – p

I N T R O D U C I N G E R R O R - C O R R E C T I N G C O D E S 1 1

Pr (000 ) = 100

99 x 100

99 x 100

99 = 0.970299

Pr (001 ) = 100

99 x 100

99 x 100

1 = 0.009801

Pr (010 ) = 100

99 x 100

1 x 100

99 = 0.009801

Pr (100 ) = 100

1 x 100

99 x 100

99 = 0.009801

So the probability of decoding our message as 0 is given by:

Pr (0) = Pr (000 ) + Pr (001 ) + Pr (010 ) + Pr (100 )

= 0.970299 + 3 x 0.009801

= 0.999702 .

So now, on average, instead of decoding our sent message of 0 incorrectly as 1 once in every 100 times, we will get fewer than 3 in every 10 000 (or approximately 1 in 3000) incorrect! This is a big improvement.

For a repetition code of length n, we have

n digits }

– – – … – – }

}

1 message digit

n – 1 check digits

So the information rate, R = n

1. This is very small!

Repetition codes are good at error correcting, but have very low information rates!

You receive the following message, which was sent using the alphabet from Activity 2.1 and a repetition code of length 5.

00000 10010 11011 11000 01111

11110 01010 01000 01011 00001

00111 10000 01100 11100 00000

01000 11111 00111 10111 11101

01111 00010 01000 10111 10000

a) Convert the message to the 5 digit binary code.

b) Use the alphabet from Activity 2.1 to convert it to ordinary letters.

ACTIVITY 2.3

1 2 E M M 4 0 0 T O P I C 2

Using the same code as in Activity 2.3, you receive the following message:

00000 11011 01111 00100 11111

01000 11111 00100 00001 11101

11101 00100 00000 11110 01111

11111 10000 01000 00000 00100

10111 00010 10000 00111 01100

01100 10001 01010 00011 11001

01010 10111 01111 11111 00000

11111 00010 11011 01000 00000

a) Convert the message to the 5 digit binary code.

b) Use the alphabet from Figure 2.6 to convert it to ordinary letters.

c) What message do you think was really intended?

A repetition code of length 3 is being used to transmit a message. If the probability of an error occurring in a single digit is 0.01, and we assume that errors occur independently of one another, calculate the probability that if 000 is transmitted 111 will be received?

How could you check your answer using the calculations on the previous page?

Modular arithmetic, groups and fields

Before proceeding with our next example of an error-correcting code (well actually the next example is only an error-detecting code, sorry!), we need to revise and extend some things you have previously learned in this course.

You will recall from EMM302 Linear and Abstract Algebra, that we can define addition and multiplication modulo a natural number, n, by looking at the remainders after dividing by n.

So, for example, if n = 2, we get the following addition and multiplication tables modulo 2 (or more briefly mod 2),

+ 0 1 x 0 1

0 0 1 0 0 0

1 1 0 1 0 1

Notice that really is just the same as looking at what happens when we do arithmetic with even and odd numbers (even numbers correspond to 0, while odd numbers correspond to 1).

ACTIVITY 2.4

ACTIVITY 2.5


Draw up the addition and multiplication tables for the following:

a) The set of integers {0, 1, 2, 3, 4} using modulo 5 arithmetic.

b) The set of integers {0, 1, 2, 3, 4, 5} using modulo 6 arithmetic.

In each case the set of integers forms a commutative group under addition.

In one of the above cases we also get a commutative group when we exclude 0 from the set.

Which case gives the group under multiplication?

Which group properties are not satisfied in the other case?

In fact, it turns out that for every n, the set {0, 1, 2, … , n – 1} forms a cyclic group under addition, while the set {1, 2, … , n – 1} forms a group exactly when n is a prime number.

If n is not a prime, then every factor of n (and every multiple of every factor of n) turns out not to have an inverse. For example, in your multiplication table for modulo 6, you would have found that none of the rows or columns for 2, 3, or 4 contained the multiplicative identity 1.

It is the existence of inverses (except of course for 0) that essentially allow us to divide as well as multiply in arithmetic.

So it seems that when we work in multiplication modulo a prime p , things are pretty similar to “ordinary” arithmetic, while this is not true when we try to use modulo n arithmetic when n is not a prime.

The set {0, 1, 2, … , p – 1} under arithmetic modulo p, for every prime p, is a special case of what is called a Galois field.

Below we will give a very long definition of a field, but as you will see shortly, it is quite easy to remember what a field is using your knowledge of groups.

A field is a set F , containing a least two elements, on which an addition, + ,

and a multiplication, . , are defined in such a way that, for every x, y and z in F:

F1 x + y is in F – i.e. F is closed under +

F2 x + y = y + x is in F – i.e. + is commutative

F3 (x + y) + z = x + (y + z) – i.e. + is associative

F4 There is an element 0 in F such that x + 0 = 0 + x = x – i.e. 0 is the additive identity

F5 For each x in F , there is an element – x in F such that x + (– x) = (– x) + x = 0

– i.e. – x is the additive inverse of x

F6 x . y is in F

– i.e. F is closed under .

ACTIVITY 2.6

DEFINITION OF A FIELD

1 4 E M M 4 0 0 T O P I C 2

F7 x . y = y . x is in F

– i.e. . is commutative

F8 (x . y) . z = x . (y . z)

– i.e. . is associative

F9 There is an element 1 in F such that x . 1 = 1 . x = x – i.e. 1 is the multiplicative identity

F10 For each x ≠ 0 in F , there is an element x – 1 in F such that

x . x – 1 = x – 1 . x = 1 – i.e. x – 1 is the multiplicative inverse of x

F11 x . (y + z) = x . y + x . z and (x + y) . z = x . z + x . z – i.e. . is distributive over + .

Notice that F forms an abelian (i.e. commutative) group over addition (from F1 – F5) and that the non-zero elements of F form an abelian group under multiplication. The distributive property F11 connects multiplication and addition.

Basically, fields are just structures that behave in pretty much the same way as “ordinary arithmetic” – that is, we can add, subtract, multiply and divide “as usual”.

Some familiar fields under “ordinary addition and multiplication” are: the set of real numbers, R , and the set of all rational numbers, Q . Both of these fields are infinite.

However neither the set of natural numbers, N , nor the set of integers , Z

forms a field under addition and multiplication – why not?

It turns out that for every prime p and every natural number n, there exists a finite field of order pn and that (up to isomorphism) no other finite fields exist.

These finite fields are called Galois fields in honour of the mathematician Galois and are usually written as GF(pn).

When n = 1, we get the Galois fields, GF(p), referred to earlier consisting of the elements {0, 1, 2, … , p – 1} under arithmetic modulo p.

When n > 1, things are a bit more complicated, but luckily for us here we are only interested in the very simple case of GF(2) – i.e. mod 2 arithmetic.

Matrix calculations over the field GF(2)

In EMM302 Linear and Abstract Algebra, we only looked at vector spaces with real numbers as our scalars. However, we could have used any field for our scalars and it is usual to talk about a vector space over a field. We won’t have to worry too much about this here, but in order to look at some of our codes, we need to be able to do matrix calculations in modulo 2 arithmetic – which of course turns out to be much easier than ordinary arithmetic!


Let A =

− 111

103

701

, B =

101

432 , C =

−316

012 .

a) Which combination of the matrices A, B and C can be added or multiplied together?

b) For each possible combination, carry out the calculations.

c) Now let A′ = A (mod 2), B′ = B (mod 2), and C′ = C (mod 2),

so, for example, A′ =

111

101

101

.

Find B′ and C′. d) Repeat each of the calculations from part b) using A′, B′ and C′

instead of A, B and C.

e) In part b), one of your calculation was B + C. Find (B + C)′ = (B + C) (mod 2) and check that (B + C)′ = B′ + C′.

f) Repeat part e) for each of the calculations from part b).

We can see from Activity 2.7 that, once we get used to it, arithmetic in mod 2 is much simpler than ordinary arithmetic! This is especially true for matrix addition and multiplication so we are lucky as we will be using it a lot from now on.

Single parity-check codes

Single parity-check codes are the other extreme from repetition codes. While repetition codes have a single message digit, single parity-check codes have only one check digit.

The check digit is obtained by taking the sum of the information digits mod 2.

So, for example, using the code in Figure 2.6, we get

A

000001 1

}

←

5

message digits

1check digit

B

000010 1

C

000011 0

D

000100 1

…

In general, the codewords look like c1 c2 c3 c4 c5 c6 where

ACTIVITY 2.7

1 6 E M M 4 0 0 T O P I C 2

c6 = c1 + c2 + c3 + c4 + c5 (mod 2).

Find the codewords representing each of the following letters in the single-parity-check code above: J , L , Q , S , G , X.

Write the message NO ERRORS in the single-parity-check code above.

The following message is received in the single-parity-check code:

000011 000000 001111 011110 010110 001001 000000 100100 001010 100111 101001 011000 101000

a) Detect the places where the errors occur.

b) Decode the remaining letter.

c) Use guesswork to complete the message.

For our single-parity-check code, we have:

n digits

} -- … - - }

←

k = n – 1 message

digits

1check digit

So the information rate is n

k =

n

n 1− , which is very high!

However, a single-parity-check code can only detect an odd number of errors and cannot correct them!

The main problem

Error-correcting codes apply to many areas – for example, information transmission via satellites, computer storage of data, electronic transmission of data via the internet, to name just a few.

We have considered two extreme cases in this topic – repetition codes which

have very low information rates, but which can correct up to 2

1−n errors, and

single-parity-check codes which have a very high information rate but cannot correct any errors and can only detect an odd number of errors.

The mathematics comes in when we try to find codes that have high information rates and good error-correcting capabilities.

ACTIVITY 2.8

ACTIVITY 2.9

ACTIVITY 2.10


Recall our first example. What made it good?

codewords of length 8 }

– – – – – – – – }

}

3 message

digits 5 check digits Information rate =

8

3

Altogether there were 8 codewords, since we had 3 message digits (– – –) with each digit being 0 or 1, giving 2 x 2 x 2 = 8 possibilities.

How many possible words are there of length 8? Since we have 8 digits (– – – – – – – –) with each digit again being 0 or 1, we have 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 28 = 256 possibilities.

So there are lots of possible choices for our 8 codewords out of the 256 possible words of length 8.

The actual codewords have been chosen to be as far apart from each other as possible – each codeword differs from every other one in at least 4 places, which is what allows single error correction.

We can think of a code with codewords of length n as vertices of a unit n-dimensional cube.

You are probably thinking “Help! Whatever can that mean?”

Let us try to think in terms of dimension 3 for a start. If n = 3, we are looking at an “ordinary” 3-dimensional cube. A “unit” cube just means its sides are 1 unit in length. Figure 2.9 shows a 3-dimensional unit cube with the vertices labelled with its x,y,z co-ordinates.

The 8 vertices of the cube represent the 8 possible binary words of length 3.

Vertices that share an edge (for example, 100 and 110) differ in exactly one place – you should check this!

Vertices that are on opposite “corners” of the same face (for example, 100 and 111) need you to “move along” two edges to get from one to the other and differ in exactly two places.

FIGURE 2.9 A 3-dimensional unit cube

1 8 E M M 4 0 0 T O P I C 2

Vertices that are not on the same face are “opposite” one another (for example, 100 and 011) need you to “move along” three edges to get from one to the other and differ in exactly three places.

If we want to create an error-correcting code, then we need all of our codewords to differ from one another in all three places so that if there is a single error in our received word we can still correctly choose the most likely codeword that was sent as the message.

This means that we must choose as our codewords opposite vertices of the cube (for example, 000 and 111 marked with ● in Figure 2.9). So we can choose at most two codewords “as far apart as possible” if we want them to differ in all 3 places.

If we choose any more codewords then we can only make them differ in 2

places (for example, the four codewords 000, 101, 011 and 110 marked with ▪ in Figure 2.9).

One possible set of codewords of length 3 that differ in all 3 places is {000, 111}. Find all the other such sets.

What about a four-dimensional cube? It is difficult to draw this in two dimensions, but Figure 2.10 is an attempt at this. The drawing started with a three-dimensional cube which then had a new cube drawn on each face – it looks better in colour!

This time we have 2 x 2 x 2 x 2 = 24 = 16 “vertices” on the “cube”, so there are 16 possible choices for our codewords.

The vertices marked with a ○ indicate 8 codewords that could be chosen in

such a way that each one differs from all the others in exactly 2 places – of course this would only give us an error-detecting code and not an error-correcting one!

ACTIVITY 2.10

FIGURE 2.10 A 4-dimensional unit cube


Shannon’s Theorem

The mathematical area of algebraic coding theory is usually regarded as having started with Shannon’s remarkable theorem that showed that under certain conditions we can make the probability of error for a decoded message as small as we like without lowering the information rate (Shannon, 1948).

While we cannot prove this here (or even state it precisely) we will discuss it briefly to give you an idea of what it means.

The capacity C (p) of a binary symmetric channel where the probability of a single random error occurring is p is given by

C (p) = 1 + plog2 p+ (1 – p)log2 (1 – p).

Figure 2.11 shows the graph of C (p).

1.2

1

0.8

0.6

0.4

0.2

-0.2

0.5 1

As we will see shortly, the capacity of a channel acts as an upper bound for the information rate with an arbitrarily small probability of error. Notice that the graph of the capacity of the channel is symmetric about p = 0.5, at which point it has value 0. This indicates the fact that if there is a 50% probability of an error occurring it will never be possible to decode a message with any certainty. Also, notice that a probability of 1 of an error occurring is just as “good” as a probability of 0 – in the case we would just change every 0 into 1 and vice versa. Similarly, errors occurring with probability p is equivalent to errors occurring with probability 1 – p.

For a binary symmetric channel with probability p of a single random error occurring, let R be a number such that R < C (p). Then for any ε > 0 there exists, for sufficiently large n a binary code of length n with information rate

n

k ≥ R and probability of error after decoding less than ε.

In other words, this is a truly amazing theorem says that, depending on the probability of error, p, there is a fixed information rate that can always be achieved while making the probability of overall error as small as we like! Of course, we will need to make the code length sufficiently large.

This theorem has a complicated proof, using, among other things, probability theory. But it only tells us that such codes exist – it gives us no idea how to find them. So the question still remains “How can we find good codes?” – that

DEFINITION Capacity of a binary symmetric channel

Figure 2.11 Graph of the capacity of a binary symmetric channel

SHANNON’S THEOREM

C (p)

p

←

→

2 0 E M M 4 0 0 T O P I C 2

is, codes with a high information rate and good error-correcting capability – or even “Can we find good codes?”

The first single-error-correcting codes were discovered by Hamming around 1950. They are relatively simple, as we will see in the next topic. Double-error-correcting codes are much more complicated and were first found by Bose, Changhuri and Hocquenghan around 1960 – they are usually referred to as BCH codes. These were generalized almost immediately to t-error-correcting codes for all t.

The mathematical area of algebraic coding theory continues to be an active area of research, which uses many areas of “pure” mathematics, such as linear algebra (vectors, matrices) abstract algebra (groups, fields, rings), number theory, geometry, and probability theory.

In the next few topics we will find out more about this important new branch of mathematics.

Answers to activities

You need to check each codeword against each of the other ones.

For example, the codeword 01010101 corresponding to the message “2 = 010 ” differs from the codeword 00000000 in 4 places (the second, fourth, sixth and eighth places), from the codeword 00110110 in 4 places (the second, third, seventh and eighth places), etc.

a)

Answer to

Ac t i v i t y 2 .2

Answer to

Ac t i v i t y 2 .6 + 0 1 2 3 4 x 0 1 2 3 4

0 0 1 2 3 4 0 0 0 0 0 0

1 1 2 3 4 0 1 0 1 2 3 4

2 2 3 4 0 1 2 0 2 4 1 3

3 3 4 0 1 2 3 0 3 1 4 2

4 4 0 1 2 3 4 0 4 3 2 1


b)

The set of integers {1, 2, 3, 4} under modulo 5 multiplication forms a group.

The set of integers {1, 2, 3, 4, 5} under modulo 6 multiplication do not form a group since:

(i) the set is not closed as 0 appears in several rows and columns; and

(ii) the elements 2, 3 and 4 do not have an inverse.

J

01010 0

L

01100 0

Q

10011 1

S

10011 1

G

00111 1

X

11000 0

References

Hill, R. (1986). A first course in coding theory. Oxford: Clarendon Press.

Shannon, C. E. (1948). A mathematical theory of communication. Bell Syst. Tech. J., 27, 379–423; 623–656.

van Lint, J.H. (1983). Algebraic coding theory. In M. Zweng, T. Green, J. Kilpatrick, H. Pollack, & M. Suydam (Eds). Proceedings of the Fourth International Congress on Mathematical Education (pp. 299 – 303). Boston, MA: Birkhäuser.

+ 0 1 2 3 4 5 x 0 1 2 3 4 5

0 0 1 2 3 4 5 0 0 0 0 0 0 0

1 1 2 3 4 5 0 1 0 1 2 3 4 5

2 2 3 4 5 0 1 2 0 2 4 0 2 4

3 3 4 5 0 1 2 3 0 3 0 3 0 3

4 4 5 0 1 2 3 4 0 4 2 0 4 2

5 5 0 1 2 3 4 5 0 5 4 3 2 1

Answer to

Ac t i v i t y 2 .8

T O P I C 3

Linear codes P R E P A R E D F O R T H I S U N I T B Y S U S I E G R O V E S

The theory of error-correcting codes began with the work of Richard Hamming in 1947. Hamming was a mathematician who had access to the Bell Telephone Laboratories’ computer to carry out calculations. At that time, programs would take a very long time to run and he was frustrated by the fact that he would come in to work at the weekends only to find that the computer had detected an error and terminated his program. He wondered why if the computer could detect an error it could not also correct it!

In this topic we will look at a class of codes called linear codes that include our previous examples from Topic 2, as well as codes that are now named after Hamming – that is, Hamming Codes.

An example of a linear code

In this example, we will consider words of length 6 that have 3 message digits and 3 check digits as shown in Figure 3.1.

A codeword of length 6 }

c1 c2 c3 c4 c5 c6 }

}

3 message

digits

3 check digits

Recall that in our single-parity-check we defined our check digit as the sum, mod 2, of the message digits.

Similarly, for this code, for any given message c1 c2 c3 we must define how to obtain the check digits. In this case, for reasons that we won’t try to explain here, we will use the following linear equations to define the check digits.

Given the message c1 c2 c3 we set

c4 = c1 + c2 (mod 2)

c5 = c1 + c3 (mod 2)

c6 = c2 + c3 (mod 2)

to get the codeword C= [c1 c2 c3 c4 c5 c6].

Figure 3.1 A codeword of length 6

CALCULATING THE CHECK DIGITS

L I N E A R C O D E S 2 3

That is,

6

5

4

c

c

c

=

110

101

011

3

2

1

c

c

c

.

Don’t forget that we are working in arithmetic modulo 2. So all of our matrices are over the field of scalars consisting of 0 and 1 under modulo 2 arithmetic – that is, the Galois field GF(2).

So, for example, 010 will be transmitted as [010101].

a) Write down the codewords corresponding to the following messages:

(i) 111 (ii) 101

b) How many three digit message are there?

c) List all the codewords for this code.

d) Work in groups of four to find the minimum distance between any two codewords.

The full codeword C = [c1 c2 c3 c4 c5 c6 ] satisfies the parity-check equations

c1 + c2 + c4 = 0 (mod 2)

c1 + c3 + c5 = 0 (mod 2)

c2 + c3 + c6 = 0 (mod 2).

The reason these equations are called parity-check equations is that they check the parity or evenness of the sum of the digits in the codewords – in order to get 0 on the right-hand side of each of these equations, we need and even number of 1’s on the left-hand side of each equation.

We can also write the parity-check equations as

100110

010101

001011

6

5

4

3

2

1

c

c

c

c

c

c

=

0

0

0

,

or H CT = 0 ,

where H is the parity-check matrix above.

[The reason we use CT instead of writing out the whole vertical transpose of the vector C in the equation above is to save space – it also helps us to write further equations in a more concise, and therefore more easily remembered, form.]

What happens when we try to transmit information using our codewords?

ACTIVITY 3.1

PARITY-CHECK EQUATIONS

PARITY-CHECK MATRIX

2 4 E M M 4 0 0 T O P I C 3

If we start with the codeword

C = [c1 c2 c3 c4 c5 c6]

the channel adds noise (or error)

E = [e1 e2 e3 e4 e5 e6]

resulting in the received word

R = [r1 r2 r3 r4 r5 r6]

where r i = ci + ei (mod 2) .

a) If C = [100110] , E = [000101], find R.

b) If R = [001000], E = [000011], find C.

c) If R = [010000], C = [111000], find E.

Unfortunately, the examples in Activity 3.2 are just exercises. In actual fact, we only know the received word R. So the problem is how to find the codeword C if we receive the word R.

We do this by finding the error E first.

Of course, we won’t be able to find the error E exactly. Instead, when we detect an error, we will assume that as few as possible error digits are 1 – that is, given a choice for E, we will always choose the one corresponding to the fewest errors occurring in transmission. This method depends on another assumption – that the probability of error is small.

We will now show how to find E.

For received word R = [r1 r2 r3 r4 r5 r6], we define the syndrome s = [s1 s2 s3] of R by

s1 = r1 + r 2 + r 4 (mod 2)

s2 = r 1 + r 3 + r 5 (mod 2)

s3 = r2 + r 3 + r 6 (mod 2).

That is,

3

2

1

s

s

s

=

100110

010101

001011

6

5

4

3

2

1

r

r

r

r

r

r

or more simply

sT = H RT

.

Since the syndrome digits are defined by the same parity-check equations as the codewords, the syndrome digits will reveal the pattern of parity-check failures, as we will see shortly.

By the way, the reason that we call s the syndrome of R is that it exhibits the “characteristic symptoms” of the error without identifying the cause, just as we call diseases where we know the symptoms but not the real cause “syndromes” – for example, SIDS, which stands for Sudden Infant Death Syndrome.

ACTIVITY 3.2

CALCULATING THE SYNDROME


Work in groups of four to find the syndrome of all 8 codewords.

Your results from Activity 3.3 should have shown that every codeword has syndrome 0 = [000].

This should not be surprising, since we defined our codewords as satisfying

H CT = 0 .

We will now show a very important result, namely that the syndrome of the received word is the same as the syndrome of the error.

Recall that our received word R is the sum of our codeword C and the error E. That is, R = C + E.

So C = R – E and

0 = H CT = H (R – E)T

= H (R T – E T)

= H R T – H E T

So sT = H RT = H E T.

So the received word R has the same syndrome as the error E.

This is very useful because it means that if R is the received word, the set of possible errors is just the set of vectors with the same syndrome as R.

From the above, if R and E have the same syndrome, we can reverse the argument to show that they differ by a codeword – that is, if H RT

= H E T , then R – E = C , where C is a codeword.

So the number of words with the same syndrome is equal to the number of codewords.

In this case, we have 23 = 8 possible syndromes and each corresponds to

exactly

88

64

8

2 6

== words.

So, for example, the 8 codewords correspond exactly to the 8 words with syndrome [000].

Finding the words corresponding to a given syndrome Since the syndrome of a received word R is the same as the syndrome of the error E, one of the things we need to be able to do in order to decode a received word is to find all the words that have the same syndrome as R.

As an example, we will try to find all words with syndrome [001] when we use our linear code defined above. So we have

r1 + r 2 + r 4 = 0 (mod 2)

r1 + r 3 + r 5 = 0 (mod 2)

r2 + r 3 + r 6 = 1 (mod 2).

ACTIVITY 3.3

THE SYNDROME OF THE RECEIVED WORD

2 6 E M M 4 0 0 T O P I C 3

We could solve these equations “barehanded” or we could use our knowledge of systems of equations and vector spaces – that is find solutions to

100110

010101

001011

6

5

4

3

2

1

r

r

r

r

r

r

=

1

0

0

,

or H RT =

1

0

0

= sT.

It is clear that the rank of the matrix H is 3. [Why?]

Recall Theorem 3.5 from EMM302 Linear and Abstract Algebra, which is repeated below.

Let A be an m × n matrix and suppose b is an m × 1 vector.

The system Ax = b is consistent (that is it has solutions) if and only if [A b] has the same rank as A.

If the system is consistent, then the solution consists of precisely r variables expressed in terms of the remaining n − r variables, which have arbitrary values.

It is also clear that the rank of the augmented matrix [H sT] is 3. [Why?]

So the equations are consistent.

Since H is a 3 x 6 matrix, with n = 6 rows and rank r = 3, this theorem tells us that there are 3 = 6 – 3 independent variables (that is, variables that we can choose arbitrarily).

This makes sense. If we can make the first three variables independent, we have 2

3 = 8 possible choices of 0’s and 1’s for the first three variables and these

correspond exactly to our codewords!

Similarly, we can find all the words corresponding to our syndrome [001] by listing all 8 possible choices of 0’s and 1’s for the first three variables and finding the values for the remaining three variables.

For example, if we start with r1 = 0, r2 = 0, r3 = 0, we see from the three equations above that we must have r4 = 0, r5 = 0, r6 = 1.

So one of the 8 words corresponding to syndrome [001] is [000001].

We can obtain all 8 words in the same way. For example, if r1 = 0, r2 = 0, r3 = 1, we get r4 = 0, r5 = 1, r6 = 0, giving the word [001010].

Find the remaining 6 words corresponding to the syndrome [001].

ACTIVITY 3.4


a) List all the words with syndrome [010].

b) List all the words with syndrome [111].

Figure 3.2 lists all the words corresponding to each syndrome for the linear code described in this topic.

The set of words {[r1 r2 r3 r4 r5 r6] such that r i = 0 or 1, for i = 1, 2, …, 6} forms a vector space of dimension 6 over the Galois Field GF(2).

Since the codewords are the solutions of H CT = 0, the codewords form a

subspace of the vector space consisting of all words. The dimension of this subspace is 6 – 3 = 3.

In particular, this means that the set of codewords forms a group under addition (mod 2) and also that the sum of any two codewords is also a codeword.

Work in groups of four to check that the sum of any two codewords is also a codeword.

Work in groups of four.

Choose a syndrome not equal to [000] – make sure each person in your group chooses a different syndrome.

a) Choose one codeword and add it to each of the words in the row corresponding to your chosen syndrome. What do you find?

b) Using the same syndrome, choose a different codeword and repeat part a).

c) Using the same syndrome, check that each word corresponding to your syndrome in Slepian’s Standard Array can be obtained as the sum of the first word in the row (the coset leader) and the codeword in the same column.

d) Compare your answers to parts a), b) and c) with the other members of your group.

ACTIVITY 3.5

Figure 3.2 Slepian’s Standard Array

ACTIVITY 3.6

ACTIVITY 3.7

2 8 E M M 4 0 0 T O P I C 3

Cosets

Last semester in EMM302 Linear and Abstract Algebra, we didn’t have time to prove Lagrange’s Theorem regarding the order of a subgroup of a finite group, and we don’t really have time here either.

However, an important idea used in the proof of Lagrange’s Theorem is that of a coset.

If H is a subgroup of a group G and g is an element of G, then the left coset of H by g is the set gH of all products gh where h is in H.

Right cosets are defined in a similar way.

The important things about cosets are that:

• every element of G belongs to some coset of H;

• if H is finite, every coset of H has the same number or elements as H; and

• any two cosets of H are either disjoint or co-incide.

This last property ensures that the whenever we have a finite group G of order n with a subgroup H of order m, then if we have r left (or right) cosets of H, then n = m x r.

This is essentially the proof of Lagrange’s Theorem showing that the order of any subgroup of a finite group must divide the order of the whole group.

The codewords form a subgroup of the group of all words under addition. In Slepian’s Standard Array, the codewords are listed as the first row, starting with the codeword [000000].

Each further row of the array consists of a coset of the codewords. In each coset, the words are arranged in each row in such a way that the first word in each row has the fewest 1’s. The first word in each row of Slepian’s Standard Array is called the coset leader.

In the first row, corresponding to the syndrome [000], the first word in the row has no 1’s; the first word in each of the next six rows has exactly one 1; while the first word in the last row is one of the three words in that row with exactly two 1’s.

The weight of a word is the number of 1’s in the word.

A Slepian Standard Array can be constructed for a linear code in the following way:

1. In the first row, list all the codewords starting with 0.

2. Choose any word, W, of minimum weight which is not a codeword (that is, which is not listed in the first row) and list it as the first element in the next row.

3. Starting with W, list all the coset elements W + C, where C is a codeword, in the same order as the list of codewords in the first row.

4. Repeat steps 2 and 3 using a new word X, where X is not in either of the first two rows.

LEFT AND RIGHT COSETS

WEIGHT OF A WORD


5. Repeat step 4 using new words not in the previous rows until all the words have been listed.

Check that Slepian’s Standard Array for our linear code shown in Figure 3.2 can be constructed using steps 1 to 5 above.

We will now look at how we can use all of this information to help us to decode a received word that may have some errors.

Decoding a received word We can decode a received word R = [r1 r2 r3 r4 r5 r6] by carrying out the following steps:

1. Calculate the syndrome s = [s1 s2 s3] of R. This gives the syndrome of E.

2. Use Slepian’s Standard Array to find the word with syndrome s with the least number of 1’s. Choose this word for E.

3. Let C = R – E.

Use steps 1 to 3 above to find C if R = [101110].

Notice that in fact by using this array we don’t even have to calculate the syndrome. Since R and E have the same syndrome, it is enough to look for R in the table. Then E will be the first word in the row containing R.

Find C if (i) R = [111111], (ii) R = [111011], (iii) R = [110011].

Work in pairs and carry out the following:

1. Choose a codeword C to send as your message.

2. Choose an error E.

3. Calculate R = C + E and “send” it to your partner.

4. Decode your partner’s word.

5. Repeat this a total of three times: once choose E of weight 1 (that is with a single 1 in the word); once with E = 0; and once with E of weight 2 (that is with two 1’s in the word).

6. In which cases were you able to identify the codeword?

In Part d) of Activity 3.1 you should have found that the minimum distance between any two codewords in this linear code is 3. This means that this code will always correct any single error – which is what you should also have confirmed in Activity 3.11.

ACTIVITY 3.8

ACTIVITY 3.9

ACTIVITY 3.10

ACTIVITY 3.11

3 0 E M M 4 0 0 T O P I C 3

Linear codes in general

In the last section, we only looked at a simple example of linear codes. However, much of what we saw there is characteristic of linear codes in general. So we will use this section to define some of the terms we used and extend our ideas to linear codes in general.

A code is a linear code or group code if its codewords are the set of vectors C satisfying a system of equations H CT

= 0, where H is a parity-check matrix.

In the last section, we looked at a linear code with a 3 x 6 parity-check matrix, H .

What about our previous two examples from Topic 2, the single-parity-check code and the repetition code?

In our single-parity-check code, the digits c1, c2, c3, c4, c5, c6 of the codeword [c1 c2 c3 c4 c5 c6 ] satisfied the parity-check equation

c6 = c1 + c2 + c3 + c4 + c5 (mod 2),

which, since we are doing our calculations mod 2, is the same as

c1 + c2 + c3 + c4 + c5 + c6 = 0 (mod 2).

We could write this as H CT = 0, where H = [111111].

In general, if we have a single-parity-check code of length n, then the parity-check matrix corresponding to that code is the 1xn matrix H = [111… 1].

Our length 5 repetition code could have been defined using the following parity-check equations:

c1 + c2 = 0 (mod 2)

c1 + c3 = 0 (mod 2)

c1 + c4 = 0 (mod 2)

c1 + c5 = 0 (mod 2).

We could write this as H CT = 0, where H =

10001

01001

00101

00011

.

In general, if we have a length n repetition code, we get an (n – 1) x n parity-check matrix H .

Not all codes are linear codes, but the fact that the codewords in a linear code are the solutions of a set of homogeneous equations and form a subspace of the vector space over the field of scalars means that there is a basis for the codewords which simplifies everything a great deal. So, in our previous example, since the dimension of the set of codewords was 3, any basis had just 3 elements and every codeword could be found as a linear combination of our

LINEAR CODES

SINGLE-PARITY-CHECK CODE REVISITED

REPETITION CODE REVISITED


three basis codewords. So, instead of needing to do everything with all 8 (= 23)

codewords, we only need to consider the three basis codewords. This isn’t much of a saving for such a small code, but it is a big saving in computation when we have a larger code. This is one reason why linear codes are used such a lot.

A code in which each codeword is a sequence of a fixed number, n, of symbols is called a block code of block length n. In the case of a linear code, the block length is the number of columns in H.

While we haven’t stated this explicitly before, but all the codes we are considering here are block codes – that is, the codewords all have fixed length. Of course there are other codes – for example Morse Code – where the length of the codewords varies.

The syndrome, s, of a received word R is given by sT = H RT.

A coset consists of all words having a given syndrome.

The weight of a word is the number of 1’s in the word.

Within a coset, a word of least weight is chosen as the coset leader.

To decode a received word R:

1. calculate its syndrome s;

2. find the coset leader E; and

3. calculate C = R – E.

This is alright in theory and worked very nicely for the cases when n is small – we can calculate coset leaders and actually find parity-check matrices H . But in general, just how does one find these matrices and find the coset leaders?

We won’t be able to answer this question here, but will just begin to look at the problem by describing one of the first error-correcting codes.

Hamming codes

As stated earlier, Hamming codes were developed by Richard Hamming around 1950. These codes are linear codes that correct a single error.

In order to describe these codes we will first go back to look at the example of a linear code at the start of this topic.

In this case we had H =

100110

010101

001011

and

sT = H RT = H ET.

BLOCK LENGTH

SYNDROMES

COSETS

WEIGHT

COSET LEADER

DECODING R

3 2 E M M 4 0 0 T O P I C 3

Now if the error, E = [e1 e2 e3 e4 e5 e6], we can write the equation above as

3

2

1

s

s

s

= e1

0

1

1

+ e2

1

0

1

+ e3

1

1

0

+ e4

0

0

1

+ e5

0

1

0

+ e6

1

0

0

.

That is, the syndrome is the sum of those columns of H in which the channel errors occur.

So

• if any column of H is 0, an error in that position cannot be detected; and

• if any two columns of H are identical, we cannot tell the difference between a single error in those positions.

So a linear code can only correct all patterns of single errors if the columns of H are distinct and non-zero.

Conversely, if all the columns of H are distinct and non-zero, then a single error in different positions will always result in different syndromes.

A linear binary code is capable of correcting all patterns of not more than one channel error if and only if all columns of its parity-check matrix H are distinct and non-zero.

To decode a received word R, compute its syndrome s.

If s is zero, assume there is no error.

If s is non-zero and equal to a column of H, assume a single error occurred in that position.

If s is non-zero and not equal to any column of H, this decoding procedure fails.

Decoding failures and errors can only occur is two or more channel errors occur.

For example, let H =

101000111

011001110

101011100

011111000

.

If we receive the word R = [101000101 ], then we can find s = [1100 ].

So sT is the fifth column of H and we assume that E = [000010000 ].

So C = R – E = [101010101 ].

However, if R = [101000101 ], we get s = [1101 ], and in this case sT is none of the columns of H , so we must have ≥ 2 errors and the decoding procedure fails.

For the parity-check matrix H above, try to decode each of the following received words:

THEOREM 3.1

Decoding a received word revisited

ACTIVITY 3.12


(i) R = [101001101 ] (ii) R = [111000101 ] (iii) R = [101000111 ]

Recall our earlier diagram of an n-digit codeword.

n digits }

– – – … – – – … – – }

}

k message digits

r check (redundant)

digits What is the largest possible block length for a single-error-correcting code having at most r check digits?

Since we want a single-error-correcting code, we can ask instead what is the maximum number of distinct, non-zero columns of a binary matrix with at most r rows?

Since the entries are all 0’s or 1’s, the answer to this question is 2r – 1.

The columns of the parity-check matrix, H , of a Hamming Code consist of the

2r – 1 non-zero binary r-tuples arranged in any order.

Recall Theorem 3.8 from EMM302 Linear and Abstract Algebra, which is repeated below.

If A is an m × n matrix with rank r, then the dimension of the null space of A is n − r.

Since H contains all possible non-zero columns, it contains each of the columns of an r x r identity matrix, and so it has rank r.

So H is an r x n matrix of rank r, and the dimension of the subspace satisfying H CT

= 0 is n − r = k.

So the number of message digits = k = n − r = 2r – 1 – r.

For each positive integer r, there exists a Hamming Code with r check digits,

block length n= 2r – 1 and k = n − r = 2

r – 1 – r.

This code can correct a single error in any digit.

Furthermore, since every non-zero r-tuple occurs as a column, decoding failures never occur.

So the single-error decoding procedure is complete.

However, the code cannot detect ≥ 2 errors.

Sometimes we add another parity-check digit to allow us to detect (but not correct) 2 errors.

While the columns of the parity-check matrix H can be arranged in any order, some arrangements are vastly superior to others for computational reasons that we won’t go into here.

HAMMING CODES

3 4 E M M 4 0 0 T O P I C 3

The information rate for a Hamming Code is given by

R = n

k =

12

12r

r

−−− r

= 1 – 12r −

r.

As r → ∞, R→1.

That is, by constructing Hamming Codes of large block length, we can get very high information rates.

So Hamming Codes are a great improvement on single-parity-check codes, but they still cannot correct ≥ 2 errors.

As mentioned at the end of Topic 2, Hamming found the first single-error-correcting codes around 1950. Double-error-correcting codes (BCH codes), which are much more complicated, were first found by Bose, Changhuri and Hocquenghan around 1960. These were generalized almost immediately to t-error-correcting codes for all t.

a) Which of the following parity-check matrices defines a single-error- correcting code? Give reasons for your answer.

(i) H H H H =

0111011100

0001000101

1100110011

1010101010

(ii) H H H H =

110110000

000110110

000011011

011011000

b) Does either of these matrices correspond to a Hamming Code?

Explain why or why not.

Use matrix (ii) from Activity 3.13 to decode each of the following received words.

a) R = [111101000 ] b) R = [110101011 ]

c) R = [100010001 ] d) R = [010010010 ].

For each of the two codes defined by the set of codewords given below, find the minimum distance between pairs of codewords.

a) 00011, 00101, 01001, 10001, 00110,

01010, 10010, 01100, 10100, 11000.

b) 000000, 001110, 010101, 011011,

100011, 101101, 110110, 111000.

ACTIVITY 3.13

ACTIVITY 3.14

ACTIVITY 3.15


Let h be the minimum distance for a code.

a) How many errors can it detect if

(i) h is odd (ii) h is even?

b) How many errors can it correct if

(i) h is odd (ii) h is even?

a) From the diagram above, name a pair of points differing in

(i) all four co-ordinates (such pairs of points form the ends of a diagonal of the hypercube);

(ii) three co-ordinates; and

(iii) two co-ordinates.

b) Describe geometrically the line joining your selected pairs of points in cases (ii) and (iii) above.

Consider the Hamming Codes defined by the three parity-check matrices below.

(i) H H H H =

1010101

1100110

1111000

(ii) H H H H =

1001011

0101101

0011110

(iii) H H H H =

1010101

0110011

0001111

a) For each of these three codes, decode the following received words: R = [1110000 ] , R = [1111000 ] .

Show that two of these three matrices define identical codes. Hint: Show that the rows of either are linear combinations of the other.

ACTIVITY 3.16

ACTIVITY 3.17

ACTIVITY 3.18

3 6 E M M 4 0 0 T O P I C 3

Answers to selected activities

a) (i) 111000 (ii) 101101

b) 8

c) 000000 001011 010101 011110 100110 101101 110011 111000

d) The minimum distance between any two codewords is 3.

a) R = C + E = [100110 ] + [000101 ] = [100011 ]

b) C = R – E = [001000 ] – [000011 ] = [001011 ]

c) E = R – C = [010000 ] – [111000 ] = [101000 ]

1. We use

3

2

1

s

s

s

=

100110

010101

001011

6

5

4

3

2

1

r

r

r

r

r

r

to get the syndrome of

R = [101110 ]. This gives s = 011

2. The first word in the row corresponding to 011 in Slepian’s Standard Array is 001000 . So let E = [001000 ].

3. C = R – E = [101110 ] – [001000 ] = [100110 ].

(i) C = [101011101 ] (ii) C = [111000001 ] (iii) decoding failure

References

Hamming, R.W. (1950). Error correcting and error detecting codes. Bell Syst. Tech. J. 29, 147–160.

Hill, R. (1986). A first course in coding theory. Oxford: Clarendon Press.

van Lint, J.H. (1983). Algebraic coding theory. In M. Zweng, T. Green, J. Kilpatrick, H. Pollack, & M. Suydam (Eds). Proceedings of the Fourth International Congress on Mathematical Education (pp. 299 – 303). Boston, MA: Birkhäuser.

Answer to

Ac t i v i t y 3 .1

Answer to

Ac t i v i t y 3 .2

Answer to

Ac t i v i t y 3 .9

Answer to

Ac t i v i t y 3 .12

Documents

Study Guide Part a Topic1