','

~956/57, No. 7 FLYING-SPOT SCANNER FOR 35 mm FILM 201

picture tube will be slightly distorted over 3 to 4lines, which is not, however, perceptible. As may beseen from fig. 9, distortion of this nature generally

I occurs at two positions in the picture. If the .film isnot stationary during slip, the distortion is spreadover the whole picture, but at the same time itbecomes less pronounced and thus remains im-percept~le in the television picture.

Summary. A Hying-spot scanner for televising 35 mm filmhas been developed in the Philips Research Laboratories,Eindhoven. In such scanners the film moves with constantspeed and, this complicates the problem of interlaced"

'scanning. The problem is solved by using two, opticalsystems. The two objective lenses have to be closely identicaland should be as free as possible from distortion andvignetting.Since there is always some shrinkage in a film and sincethe

drive is effected in the normal way via a sprocket, thefllm doesnot in fact travel at a strictly constant speed. A detailedexplanation is given of why this is not troublesome.

ENTROPY IN SCIENCE AND TECHNOLOGY

'IV. ENTROPY AND INFORMATION

by J. D. FAST and F. L. H . .M. STUMPERS. 536.75: 621.391

In Volume 16 of this Review one of the authors (J.D.F) devoted a series of three articles tothe. application of the concept of entropy to several widely varying problems in -science and.technology, Sóme of the subjects discussed were: affinity in chemical reactions, paramagnetism,the attainment of very low temperatures, heat engines, refrigerating machines and he~t pumps,radiation, specific heat of solids, vacancies and diffusion in solids, dlloys, the elasticity ofrubber, and the sol!Ltion of macro-molecules. This article, which concludes the series, is some-what different from the previous ones in that its purport is rat/~er negative than positive. Ininformation theory it is possible to arrive at a concept that can beformally defined and manipu-lated a~alogously to the concept of entropy in thermodynamics, This is illustrtued by variousapplications. Hmoener, the use of the term entropy for this new concept might ~vrongly suggestthat all thermodyruimiccl implications in the term were equally valid hete .. Some caution istherefore necessary.

-Introduction

The concept of information plays l!n essentialpart in two recently developed branches of tech-nology, viz. communication theory and cyhernetics.The domain of communication theory includes thetransmission of messages by means of acoustical,mechanical, electrical or optical signals. Its mathe-matical foundation has becn largely laid down. byShannon 1). Even more ambitious in scope is thescience. of control or cybernetics, so termed byWiener 2) who has done much fundamental workon this subject. The subtitle of Wiener's book:"Control and communication in the" animal arid'the machine" states succinctly the scope of, thisnew science. More recently 3) the Sánit} author hasparticularly applied himself to the' sociologicalaspects of his. theory. At present a synthesisembracing both communication theory and oyberne-

1) C: E. Shannon, A mathematical theory of communication,Bell System tech. J. 27, 379-423 and 623-656, 1948.

2) N. Wiener, Cybernetics, Hermann at Cie., Paris 1947,John Wiléy, New York 1948. "

3) N. Wiener, The human use of human beings: Cyberneticsand society, Houghtcn Mifflin, Boston '1950.

tics, is developing. In various symposia workers inwidely diverging branches of science, such ascommunication engineers, linguists, psychologistsand neurologists have considered to what extenttheir problems, which are all concerned with thetransmission and the use of information, are of acommon nature.In the literature that has sprung up on this sub-

ject since about 1945, the term "entropy" is fre-quently encountered. Mention is made of theentropy of a source of information, the entropy of alanguage, the entropy of a TV picture, etc. Un:fortunately, however, the word entropy is oftenused here in a sense that bears but little relationto the well-established thermodynamical-statistical'entropy, discussed 'in the precèding three articles ofthis series 4). We shall return to this after havingfirst said something more about coinmunication,cybernetics and information.4) J. D. Fast, Entropy in science and technology: I. The

concept of entropy, Philips tech. Rev. 16, 258-269, 1954/55;11. Examples and applications, ibid. pp. 298-308, Ill.Examples and applications (cont.), ibid. pp. 312-332.

202 PHILIPS ITECHNICAL REVIEW '.VOLUME 18'

CommunicationIt is self-evident that communication plays a

predominant role in modern civilization. No society .however, primitive, is conceivable without com-'munication _:__primarily in the for~ of a languageof gestures and spoken words. In modern societythis is augmented by the written and printed word,photography, telegraphy, telephony, teleprinter,radio, the cinema a~d television. The word com-municatio~ in~~ns, quite generally, any transmissionof information from one place to another. Theentities and the media that, participate in thisconstitute together a communication system. In itssimplest form such a system (fig. 1) consists of asource of information plus a transmitter at the oneend, and a receiver plus a recipient at the other end,with a communication channel linking the two.

r-I ~ .IY]-+----------~~----------~ ~

SEC R., T 88116. '

Fig. 1. Schematic representation of a communication system.S source of information (selector), E transmitter + codingdevice, C communication channel, R receiver + decodingdevice, T recipient.

. ,

If the communication consists of .somethingspoken by a person A to a person B, the brain andthe organs ofspeech of A constitue the source andthe transmitter, whilst the ear and the brain of Bare the receiver and the recipient. The communica-tion channel in this case is the air.

The source of information preselects a message,i.e. it choses a specific sequence of the words, lettersor figures in which the message is embodied. From.this the receiver tries to reconstruct the originalmessage, after which it is passed along to the rec-ipient. In the case of the spoken word the signalconsists of sound waves in the air, in the case'of line'telephony of variations in the electric current in awire. The transmitter in the latter case is a micro-phone, which converts' the varying sound pressureof the voice into a varying electric current; the

, telephone is part ofthe receiver. The sound waves in~he air and the varying electric current in the wirecorrespond to the message to be transmitted accord-ing to a certain "cod e" .The application of carrier-wave telephony 5)

makes it possible to transmit' several, telephoneconversations over a single line, i.e, over one con,-duotor pair. The low-frequency electric vibrationsderived from corresponding acoustic vibrations

5) See e.g, H. N. Hansen and H. Feiner, Coaxial cable as atransmission medium for carrier telephony, (Philips tech.Rev. '14; 141-150, 1952/53.

from speech, are here superimposed on the high-frequency vibrations of the carrier wave. Whencarriers of different frequencies are separatelymodulated in this way with audio-frequency \speech, several conversations can be transmitted atthe same time. The substantially increased efficien-cy of information-transmission thus brought aboutgives rise to the question: what is the maximumamount of information per 'unit time that can betransmitted over a single telephone line? Before .wecan apswer this question we need á measure toexpress "amount, of .information". The fact thatthere is a finite limit to the capacity of a communi-cation channel to carry information follows fromthe inevitable occurrence of thermal agitation noise(see i) in every communications system. This biitrs:the details of the transmitted symbols, thus limitingthen: intelligibility. Only after we have introduceda measure for information can we deal with theabove-mentioned question.

, Cybernetics

Cybernetics is primarily concerned with self-regulating systems and processes. A very simpleexample of a self-regulating system is a thermostat,with which a space is a~tomatically kept at a nearlyconstant temperature. This form of control canbe effected, for example, with a special type ofthermometer in which the mercury closes a contactas soon as the temperature exceeds a predeterminedvalue. The electrical circuit is arranged so that a'relay is tripped which reduces the supply of heat.When subsequently the temperature drops belowthe desired value, the thermometer contact isbroken and the relay automatically causes anincreased supply of heat. The' essence of the matteris that here we are nor concerned with a one waysequence of cause and effect, as with a room witha normal thermometer, where the position of themercury-in the thermometer depends simplyon thetemperature of the room; instead, we have here atwo-way sequence of cause and effect, whereby the. temperature' of the room is also influenced by theposition of the mercury.We may put it that the above automatic control is

due to the fact that the information regarding thete~perature is fed back. from the thermometer tothe source of heat. This feedback is the commoncharacteristic of the systems with which cyberneticsis concerned. The information regarding a deviationfrom the desired condition is fed back to the controlmeehanism, which solely on the strength of thisinformation can fulfill its controlling action.

! ,

1956/57, No.. 7 IV. ENTROPY AND INFORMATION 203

A more elaborate example of automatic- controlby information-feedback is the automatic aimingof an anti-aircraft gun. A radar installation suppliesdata regarding the position of an aircraft to anelectronic device, which, via servomotors, movesthe gun into the appropriate dir~ction. In order to'function properly and to give the right i~structionsto the servomotors the electronic device must notonly know the position of the aircraft, but also thatof the gun. Of the latter it,is automatically informedby electric signals transmitted by a device connectedto the gun. These signals thus. again provide the'necessary feedback of information.

Control instruments of the latter kind are further character-ized by the fact that they predict the probable future positionof the aircraft (or other variable) from the data so far received.The mathematical theory of these prediction problems hasbeen examined by Wiener and Kolmogoroff. .

The ambitious scope of the, cyberneticists will bebetter appreciated when one :reflects·on the manyautomatic processes that are continuously occurringin 'living creatures e.g. in the human body: itsbreathing, its heartbeat, its reaction to stimuli, etc.Many of these processes are involuntary, beingcontrolled by reflexes, which are formally compar-able to the above-mentioned mechanisms employingthe feedback principle. An example is the flexor-reflex, which occurs for instance if one inadvertentlytouches a hot object with the hand: Along thesensory nerves a signal in the form of series ofelectromechanical impulses of short duration ispassed to the spinal cord. Here the informationis passed.on to certain motor neurones, along whichcommands are sent to the flexor muscles of the armin question, resulting in the withdrawal ofthe handfrom the hot object. Regulation by means of hor-mones (organic substances secreted by certain glandsin the body) may likewise be considered as a feed-back mechanism. A well-known hormone is insulin,which is produced in the pancreas and plays animportant part in regulating the metabolism ofcarbohydrates. We are probably justified in sayingthat all vegetative functions in the body, e.g. thecontrol of the composition, the pressure and the pHof the blood, the osmotic pressure, the body tempe-rature, and the composition of the gas in the lungs,are based on a form of feedback, either hormon al ornervous in characte;r. This control has the task. .ofproviding the tissues with a constant environment,whi~h is indispensable to the proper functioning ofthe organism. The name "homeostasis" has beengiven by Cannon 6) to this system of regulation.

6) W. B. Cannon, The wisdom of the body, Paul, Trench,Trubner & Co., London 1939.

With any automatic control by feedback alone wehave to put up with minor deviations from the de-sired value, since it is precisely these 'deviations 'thatar~ employed to, execute the necessary corrections.We are thus always concerned with oscillationsaround the desired value and it is our aim to reducethese to a 'minimum. Some cyberneticists believethat it, will eventually become possible to mitigatethe recurring economic crises of our society by theapplication of the methods of feedback engineering.They proceed from the premise that these economicfluctuations are based on a feedback of informationin the sense that economic activity is determined bythe rate of investment, but that the latter, in,connection with the profits to be expected, is in itsturn dependent upon economic activity 7). They holdthat by taking certain timely action it should inprinciple be possible to put a curb on eC0I:10micfluctuations,A field of great importance is .opened up by the

appl~cation of the principles of cybernetics ~o thedevelopment of "automatic factories", that is thegradual but increasing change to almost wholly.automatic plants and machines.With this '~automation" the human ~lement is

entirely or partially replaced .by robots that act in acontrolling and regulating capacity upon themanufacturing process. Like the thermometer ill thethermostat, the robot performs both the measuringarid the regulating functions. It may be given thepower of human muscles, or more than human powerby amplification, and may. even be given, in theform of an electronic computer, some of the elemen-tary faculties of the human mind. Modern .eleetroniccomputer~ are capable of completely, taking overcertain secondary functions of the human brain andwithin these restricted fields of action often improveon the brain with regard to speed and accuracy.

The concept of information

In daily life we speak of receiving iitf~rmationwhen we learn something th,at was yet unknown to .us: Information removes uncertainty, or rather,may remove uncertainty. For the communications'engineer only the latter is of importan~e. He isinterested only in how much information can beconta~ed in a message, and how much difficulty is,involved in .transmitting it. Its significance to the

. receiver or its intrinsic value do not concern him. . .7) A. Tustin, The mechanism of economic systems: An- approach to the problem of economic stabilisation from thepoint of view of control-system engineering, Heinemann,London 1953. . ,

204 PHILIPS TECHNICAL REVIE;W VOLUME 18

For' the communications engineer all messages'of equallength (i.e. symbol content) are equivalent.F~r the person who wishes to transmit a message the

'choice of the symbols is of considerable importance.If the message to 'be transmitted consists, e.g. ofa series of 1000 symbols, chosen from an alphabet of32 symbols (26 letters + some punctuation symbols)then the message will he one out of a total of 321000

possible messages. If the message were to 'contalnonly one symbol, there would be 32 possibilities; ifthe message ~\Tere2 symbols long, then each of the32 symbols might be followed by any of 32 symbols,so that there would be 322 possihilities, etc. Innormal practice, as regards conversation, telephonyand telegraphy, one is restricted to those series ofsymbols that constitute ah intelligible message in.one of the known languages or in an accepted code.For a message of 1000 symbols one then still hasthe choice of a multitude of possibilities, butconsiderably less than 321000• •

Let us, for the time being, consider the symbols asindependent of each other and occurring with equalprobability; then the general expression for a num-ber q of possible messages of length N symbols,seleéted from m different symbols is q = mN

• Letanother message have á length of M symbols, thenthe number of possible selections is q' = mM. If thetwo messages are transmitted in direct succession,and are considered as a single message, then achoice has been made out of Q = mN X mMpossible messages. On the other hand it is onlylogical to claim that the total service rendered bythe telegraph office is the sum of the two separateservices. These services consist of the transmissionof information. We must, therefore, find a,definitionof the concept "quantity of information" I as afunction of the number of possible messages, in sucha form that:

, .To express the.additive character of the function,the quantity of information in a message of a givenlength is defined as the logarithm of the number.of possible messages of that length. For the caseconsidered here we can put:

First quantity of information log q = N log mSecond quantity of information: log q' = M log m

Total quantity of information : log Q = (N+M), log m.

Any separate symbol from the series of m equallyprobable and mutually independent symbols thusrepresents a quantity of infor:rn:ation

"

i = log m = - log p, (IV, 1)

where p = l/m is the a priori probability of thesymbols. The choice of the base of the logarithms isquite arbitrary since this merely determines themagnitude of the unit. If a chóice of two equallyprobable . possibilities is transmitted, then the'quantity of information supplied is, according to(IV, 1), i = log 2. This is thè simplest situationconceivable, and it is therefore reasonable to choosethis quantity 'of information as the unit: log 2-assumes the value unity if 2 is chosen as the base ofthe logarithms. This has ïr- fact been generallyadopted' in communication theory. This unit ofinformation is called a "bit" (short for "binarydigit"). Formula (IV, 1) appropriately allots ahigher numerical value t.o information according asthe freedom of choice is greater, i.e. according as theresult is more uncertain (less ,probable).

It will be clear from the foregoing that the use of alogarithmic function for the definition (IV,' 1) isanalogous to the logarithmic definition (I, 12) ofstatistical-thermodynamical entropy: both informa-tion content and entropy are additive, while theprobabilities of independent events are multiplica-tive in both cases (seè I, pp. 265-266).

The "entropy" of a series of events

A single symbol from the language contammg32 = 25 equally probable and mutually independentsymbols, supplies, according to formula (IV, 1), 5bits of information. The symbols of a language are,in fact, not equally probable; moreover they cannotalways occur independently, In order to estimate ofthe actual information content per symbol, weshall first of all get rid of the restrietion that thesymbols are equally probable. Later we shall alsodrop the restrietion that -t.hey are independent.

Let us begin by considering an imaginarylanguage in which the symbols occur with unequalprobability and a:reindependent of each other. Forany given language the probabilities are immediatelyapparent from the frequencies with which thesymbols occur in a number of random texts dn thatlanguage. The frequency of letters in the Englishlanguage is given below in Table 18).

It is found that letter frequencies are onlyslightly influenced by the subject and the author ofthetext, ' ,

The quantity of information conveyed by a singleletter we shall define by analogy with formula

8) Taken from Fletcher Pratt, Secr~t and Urgent, Blue RibbonBooks, New York 1942, '

1956/57, ~o: 7 IV. ENTROPY AND INFORMATION 205

..'

Table I. Probabilities of occurrence of the various letters average amount of inforI?-ation per event is given byin the English language.

Letter Probahility Letter Probability

.e 0.131 m 0.025t 0.105 u 0.025 ,a 0.086 g 0.0200 0.080 Y , 0.020n 0.071 p 0.020r 0.068 w 0.015i 0.063 b 0.014s 0.061 v 0.0092h 0.053 k 0.0042d 0.038 x 0.0017I 0.034 j 0.0013f 0.029

, q 0.0012c 0.028 z 0.00077

(IV, I) as -log2P, p representing the relativèfrequency with which this letter occurs. The quesrionnow arises how much information will be conveyedon the average by a symbol' from an English text.There is a chance pe that it will be an e, in which casea quantity of information -log2Pe will be received.There is further ~ chance Pt that it will be a, t, inwhich case a quantity of information -log2]1t will bereceived, etc. The average amount of informationper symbol received is therefore '

H= -log2Pp = - :£ pp log, pp bits per symbol.. ,v (IV, 2)

With the exception of a constant we have met aformula 'of the same form in (1,16), where it servedas a definition of entropy in statistical thermodyna-mies. Following Shannon, the above-definedquantity H has also been designated "entropy". Inthe opinion of the present authors, this is a regrett-able practice, not to be recommended. Instead,some such term as information-content per symbolis ;to be preferred, in view of the fact that therelationship between this information-entropy andthe thermodynamical-statistical entropy is only of aformal mathematical nature, 'due to the commonstatistical background. The thermodynamical con-cept of entropy, however, also h_asphysical implica-tions: it is directly connected with the concept oftempérature and with the quantity of heat that canbe exchanged in the course of a process between asystem and its surroundings (see I).

The definition .of the (average) quantity ofinformation per symbol, or entropy, is generally,'valid for series of events that can each occur with a.ce~tain probability. If only two events are possible

, with probabilities p and (I - p) - then the

'.H = - P log2P- (I-p) log2·(I-p). (IV,3)

In fig. 2, H is plotted as a function of p. We seethat H reaches a maximum (= log2 2 or I bit) whenthe two possibilities are equally probable, Le, whenp = 1- P = 0.5, in other words, when the a prioriuncertainty with regarp..'to the result is greatest.In the extreme cases, p = 1 (certainty) and p = 0(impossibility), formula (IV, 3) gives H = 0, sincethen there exists no uncertainty whatsoever.Returning from formula (IV, 3) to the genera)formula (IV, 2), we observe that in this case too

H

t0,8 /

V ...........-,V \/ 1\I . \I \Ij \

0,2

oo 0,2 0,6

-p0,8 1,0

88117

Fig. 2. The functionH = -p log2P- (1 - p) log2 (1 - p).

the quantity of information per event (i.e. H) is amaximum if all probabilities are equally great,which means that there is a maximum freedom ofchoice, that is to say, there is the greatest uncertaintywith regard to the outcome.

In connectionwiththe aboveconsiderations,J.F. Schouten9)has pointed out that the activity or "manipulation" leadingto the informationshouldbe distinguishedfrom the informa-tion.itself, the latter beingmerely passive.Schouten expressesthe quantity of manipulation in' bics (binary choice); it is .equal to the maximumquantity of information'(in bits) that 'one may expect from a process. This distinction betweenthe"active" unit bic and the "passive" unit bit has not been,generallyrecoguized.

As an illustration we' may mention the well-known 'radio parlour-game "Twenty Questions" inwhich the team has to find the name of an object

9) J. F. Schouten,Proceedingsofthe Symposium onlnf~rma-tion Theory, London1950, page 195.

206 PHILlPS TECHNICAL REVIEW VOLUME 18

by putting questions that can be answered only byyes or no. The best policy for the team is to putquestions that have an equal chance of beinganswered by yes or, by no. A simple example toillustrate this principle is the following. If an objectis to be located on a chessboard (64 = 26 squares),then the quickest procedure is to divide the hoardeach time into halves (lower half, upper half, lefthalf, and right half, etc.).' In this way only 6 ques-tions are necessary. Somewhat more complicatedisthe following' problem. We have a pair of scales(without weights) and twelve coins of ideu'ticalappearance. One of the coins.is counterfeit. Thequestion is how 'many weighings are at leastrequired to decide which coin is counterfeit' andwhether this coin is too light or too heavy. We must,therefore, locate one coin .out ,of twelve and further-more make a choice of two possibilities. This re-.quires log212+ log22 = 4.585 bits of information.One weighing may have 3 results: equilibrium, thescale tips to the left, the scale tips to the right. Wecan, therefore, expect log23 = 1.585 bits of informa-tion at the most from each weighing. For this reasonit may be expected that 3 weighings will be sufficient. provided that they are carried out properly. If wearrange our experiments in such a way that the 3possible results of every weighing are a~out equallyprobable, this can indeed be realized, though notwithout some trouble.

The total amount of information required (lOg224 bits) isless than the maximum amount of information that can bederived from three weighings (3 X log23= log227 bits).Consequently to solvethe problem in three weighings a numberof slightly differentstrategies may be possible. An example of agood strategy is the following.Divide the twelvecoins into three groups of 4 and weightwo

groups against each other. If the scale dips, then we knowthat one of these gronps of 4 may contain a coin that is too'lightor the other group of 4 may contain a coin that is too heavy.For the second weighingwe place on each scale pan two of thecoins that are possibly too light together with one çf the coinsthat Ul;epossibly too heavy. If there is equilibrium then weknow that one of the 2 remaining coins is counterfeit and tooheavy. A single weighing will then show which of them iscounterfeit. If there is no equilibrinm then we weigh the twopossibly-too-light coins that were on the scale-pan that went. up, against each other. If ,vith this third weighing thescaledips, then the coinon the scale-pan that goes up is counterfeitand too light. If there is equilibrium however then the coin"that was possibly too heavy on the scale-pan 'that went downin the second weighingis the eulprit.There remains the possibility that in the first weighing the

scales were balanced.We than take three of the four remainingcoins and one of the 8 good coins and place 2 of these fourcoins on each scale.If there is equilibrium then the remainingtwelfth coin is counterfeit and a singleweighing will establishwhether it is too light or too heavy. If there Is-no equilibrium,then weweigh the two coins on the pan not containing the good

"

coin against each other. The result of this third weighing isthen decisive. It is instructiv~ to evaluate the informationobtained from these experiments by means of formula (IV, 2)'for each weighing.A similarpuzzleconcerns 13coins of identic-al appearance, one of which may be counterfeit. Here the pro-blem is to find the counterfeit coin (if there at uIl) and todecide whether it is too light or too heavy in three weighings.A 14.th,good standard coin is provided. In this problem boththe required and the maximum obtainable quantity of in-formation are log227 bits, and there exists only one goodstrategy.

The ."entropy" of a language

If we evaluate the average quantity of informa-tion per letter of the English language from theprobabilities given in Table I, then we obtain4.16 bits/letter. If the 26 letters' had' had equalprobabilities, then H would have been log226 =4·.70 bits/letter. The quantity of information perletter has therefore been 'Teduced 10), even for an in-dcp~ndent choice. Moreover, the probability of aletter is also somewhat dependent on the lettersearlier transmitted or written down. We shall 'illustrate this with 2 examples: 1) The probabilityof the letter u is not particularly great, but theprobability that a u follows a q approaches unity.2) If we have received, as part of a message, thefollowing .symbols: Arrived Saturday mor .... "then it is highly probable that the next letters willbe "ning", although "ose", "e", "eover" or "occo"are not inconceivable. Many other examples ofcorrelation: could be given. They all show that thenumber of different meaningful messages which arepossible in a message of a given length is reducedby the presence of correlation.

An artifice to compute roughly the quantity of .information per symbol is the following: let P", informula (IV, 2) represent the probability of theoccurrence of super-symbols each of length Asymbols. According as we make A larger, the inter-.c'orrelations between the super-symbols will becomesmaller. The super-symbols, if sufficiently 'long, cantherefore be chosen nearly independently of eachother. In a very long message of N super-symbolsthere will be on the average PIN specimens .of thefirst super-symbol" P2N specimens of the second.super-symbol, etc. For the information per super-symbol we then obtain, according to (IV, 2):

- ~ Pv(A) 10gp",(A).v

10) Wh~n determining the -probability per symbol we shouldactually also take the spacings and punctuations intoaccount. This has not been done here, in order to make use,of existing tables.

19~6/57, No. 7 , IV. ENTROPY ·AND INFORMATION 207

Since each super-symbol comprises A symbols, theaverage ,inf~rmation per, 'symbol is represented by:

H = -lim Al ~-pp(A) 10gpp(A) bits per symbol,A+a::> v ... (IV,4)

, .An increasingly closer approximation to H can be

, reached by choosing successively 1, 2, 3, ...... for A,i.e. by determining the' frequencies with whichsingle symbols, symbol, pairs, symbol triples, etc.occur in the language. Another estimate of thequantity of inform~tion per symbol can be arrivedat by estahlishing experimentally how many timesa person has to guess before he has found the nextletter in a given text, and then the next letter, etc.There have not yet been enough experiments of thiskind made to establish accurately the actualinformation .content per symbol of a 'language, butit has been found that for the modern Europeanlanguages the information content is around 1.5bits per symbol.

The part played by the statistical structure ina language has been aptly demonstrated byShannon 1).He :6.rstconsiders a sequence of symbolsconstructed from a 27-letter alphabet'{êf letters anda space) each symbol being chosen quite randomly(i.e. with equal probability) and independently.This can be regarded as a zero-order approximationto any language using the roman alphabet. Thefirst-order approximation to (say) English is thenobtained by choosing successive letters independent-ly but with each letter and the word spacingoccurring with the natural frequency that they havein English (seeTable I). Thus e is chosen with proba-hility 0.13 (roughly its relative frequency in English)and w with probability 0.015. For the second orderapproximation, pair-struc~ure 11) is introduced:after a letter is chosen, the next one is chosen inaccordance with the natural frequencies of the vari-ous pairs (e.g. th, ed, are frequent pairs in English).In the third-order approximation, triple-structureis introduced: each letter is then chosen with proba-bilities which depend on the preceding two letters.

11r Pair and triple frequencies are also given in the hook byFletcher Pratt quoted in footnote 8). Word frequencies aretabulated in Relative frequency of English speech sounds,hy G. Dewey, Harvard University Press 1923. ,Shannon 1) also mentions a simple method for constructingthese examples without the use of such tables: To constructa second order letter approximation, for example,' open abook at random and select a letter at random on the page.This letter is recorded. The hook is then opened to anotherpage and one reads until this letter is encountered. Thesucceeding letter.is then recorded. Turning to another pagethis second letter is searched for and the succeedingletter is-recorded; ond so on. This process was actually used hyShannon for constructing the examples (c) and (d) and forthe word approximations (e) and (I).

Proceeding in this way·we can construct better andbetter approximations to, English:a) Zero-order approximation (symbols independent

and all equally probable).xfoml rxkhrjffjuj zlpwèjwkcyjjJjevvkcqsgxyd qpaamkbzaacibzlhjqd ,

b) First-order approximation (symbols independentbut with frequencies as in English).ocro hli rgun:' nmielwis eu ll, nbnesebya th eeialherihttpa oobttva nah brl :

c) Second-order approximation (pair-structure as inEnglish) ,on ie antsoutinys are t inctore st be s deamyachin d ilonasiee tucoowe at teasonare fusotizin andy tobe seace ctisbe '

d) Third-order approximation (triple-structure asin English).

,in no ist lat whey eratiet froure birs grocid,pondenome of demonstures of the reptagin isregoactiona of ere

In the last example (3rd-order approximation)some words and parts of words are beginning to berecognizable, Further approximations become moreand more time-consuming and are therefore notattempted. ,An analogous synthesis, a~d one that sooner

resembles the language, may be achieved by con-sidering whole words instead ofletters:e) First-order word approximation (words chosen

independently but with their natural frequencies).representing and speedily is an good apt orcome can different natural here he the a incame the to of to expert gray some to furnishes,the line message had 'be these.'

.f) Sec~nd-order word approximation (the transi-tion probabilities of the words, i.e. their naturalpair frequency is taken into account).the. head and in frontal attack on an english,writer that the character of this point is there-fore another method for the 'letters that thetime of toh» ever told the problem for an un-expected., '

Again the resemblance to normal English in-creases noticeably at each stage.All these examples clearly demonstrate that the

more we take' the structure of the language intoaccount the more the choice is restricted. Theinformation content per symbol is consequentlysmaller than it would be with a free, independentchoice. I'his difference 'between the maximuminformation content per symbol and the actualinformation content' is called the redundancy R:

(IV, 5)

208 . PHILlPS .TECHNICAL REVIEW VOLUME.18

The relative redundanèy is also often considered; thisis the ratio' of the redundancy to the maximuminformation content:

" Hmàx-EIr=----

EImax... . '. . (IV,6)

For languages such as E~gÜsh, French, German,Dutch the relative redundancy is 60-70%. (At thebeginning of this section, P: 206, Hmax is evaluatedat 4.70 for the ~oman alphabet, whilst on the nextpage, EI is given as 1.5 for the modem Europeanlanguages.) For the t.elegraphic transmission ofinformation in one of the above languages it wouldin principle be possible to save 70% of the time byconverting the messages into such' a code as toreduce the redundancy almost to zero. This obvious-ly requires the coding en bloc of long series ofsymbols which have hardly any mutual éorrelationleft. The use of such a complicated code would in itsturn cause a certain delay. This might be acceptableif it were not for the fact that a finite redundancyhas some advantages of its own. A finite redundancymakes it possible to reconstruct a message mutilatedby interference or noise at the receiving end. If wereceive a telegram: "Adrive mext suntay" then weassume immediately that the meaning was "arrivenext sunday". In a code with maximum informationcontent per symbol, however, any alteration willcompletely alter the significance of the messageand it is by definition no longer possible to recoverthe original meaning from the context. Sincedistortion during transmission is inevitable, ir-respective of the system of communication, (noisewith electric transmission and ambient noise withthe spoken word), a language should always have a'certain redundancy. The fact that the redundanciesof the known languages, in spite of (or because of?)their very long history, vary only slightly gives riseto the supposition t.hat this "general" redundancyis not far removed from the optimum value ~2).A slight reduction of the redundancy, however, ispossible without harm being done. This is demon-strated by certain of the simplifications adopted inthe American version of the English language.

The "entropy'" of a television signal

The general observations on the "entropy" of aseries of events are applicable to every form. ofcommunication. We shall deal here briefly with theirbearing on television. Suppose that the separate

12) Exhaustive studies on this and other structural phenomenaof the language have been made by B. Mandelbrot. See e.g.Structure formelle des textes et communication, Word 10,1-17, 1945.

eleme~ts of an ordinary black-and-white television-picture can assume a number n of different bright-ness levels .. Let us again indicate the probabilitiesof 'the occurrence of these levels by Pp, then the"entropy" of every tele~ision picture could be. expressed by

nEI = - ~ Pp logPp bits per picture element,

v_I -

~f the brightness levels' of the separate pictureelements were completely independent of each other.(Here, too, it would be better not to speak of entropybut of the quantity of information per' pictureelement.) The brightness levels of the pictureelements, how~ver, show,a definite correlatión withregard to both place and time. Most picture elementsin an ordinary picture vary only slightly in bright-ness from adjacent elements. We ,~ould furthermore. not get an impression of continuous motion if themajority of the picture elements did not varygradually in brightness with time. Like a language,the television picture shows a considerable 'redun-dancy, and similarly, a picture that only possessedthe statistically correct brightness distributionwould never approximate to a normal picture. Theefficiency of the transmission of information fromthe television transmitter to the receiver is decidedlyimpaired by this considerable redundancy. This ismanifested by the necessity of a very large bandwidthfor the present television systems. Recently someinteresting suggestions were made as to how, withthe aid of a suitable system of coding, the redun-dancy and hence the bandwidth of the signal to betransmitted might be reduced 13). Informationtheory defines the limits of what is attainable, butthe future must decide how closely our technologycan approach them.

Capacity of a communication channel with noise

In the foregoing we have mentione-d in passing thepossible interference with and mutilation of theinformation content of a message. during trans-mission. As a rule the signals' comprising the messagein some form of code reach their destination via atransmission path (telephone' line, radio link) .inwhich interference occurs. The main source of.interference, which is universally present, is random. noise. This is caused by the thermal agitation inresisto~s and valves of transmitter, receiver andamplifying stations, and by many other randomfluctuations on the transmission path: Owing to theinterfering influence of the noise, we .cannot becertain whether the signal received is the one that

13) E. C. Cherry' and G. G. Gouriet, Proc. Instn. Electr. Engrs100, 9-18, 1953.

-~~--~~--,-~--~---~~.~~-""~--~.. . "

.1956/57, No. 7 IV. ENTÎWPY AND INFORMATION , 20~

was actually transmitted. The information received" is consequently less than it would be if no noise werepresent. Let us consider a case in which there are atotal of 'n symbols that can all be transmitted withequal a priori probability. The reception of eachsymbol would then supply, in the absence of noise,log2n bits of information. If, however, the presenceof noisè has the effect that reception of a givensymbol only means that w~ have a choice of threeèqually probable "adjacent" symbols, then welack log23 bits. of information, and we have ~husreceived only log2n - log23 bits ..A similar influence of the noise on a communi-

cation system, can be imagined as follows: theinformation contained in a message is coded at thetransmitter end in the form of short, 'similarelectric pulses, .which are transmitted in rapidsuccession (fig. 3). Each letter then corresponds,

Fig. 3. Transmission of information by series of sinusoidalpulses. The information is contained in the amplitude of thepulses. At the receiving end (b) the pulses are received in amutilated condition, so that an unequivocal recognition of theoriginal symbol at the trarisrnitting end (a) is not possible.

say, to a specific pulse height. The effect ofnoise willnow be that the pulses arrive at the receiving endin a mutilated condirion, so that there will be somedegree of uncertainty regarding the original heightof each pulse. According as the influence of the noiseis greater compared with the minimum difference inlevel between two pulses representing. differentsymbols, so the uncertainty regarding each received. signal will he greater and the information ohtainedcorrespondingly less. '. '

It is possible to _formulate' these thoughts in asomewhat stricter mathematical form. In the follow-ing, some general and well-known formulae are deriv-ed concerning the maximum quatity of information

per symbol that can be obtained at the receiving endof a communication system with noise, and concern:ing 'th~ 'minimu~ power that is required for thetransmission of information. Remarkably enough,with this' latter problem, the thermodynamicalentropy re-enters the picture,

Let the probability t~at .a given. symbol istransmittéd be Pv- In the case of a correct trans-

. 'mission of mutually independent symbols, theaverage information content' per symbol at thereceiving end would amount to

H = - '~Pv Iog p,• (IV, 2)

This and all subsequent summations have to bemade over the whole range of available symbols.

Let Pf.lV be the chance that a transmitted symbolf1, is detected at the receiving end as the symbol,"p f.l1'may he. cálled a transition probability: In a noise-free channel

P iff1,= 'V.

Pu» = (0 iff1, ~ 'V.(IV,7)

If.a random, long series of symbols is transmitted,then the nett chance that each received symbol willbe 11 is

(IV, 8)

and the chance that any received symbol 'V isactually the result of the tran~mission of a symbolA is

(IV, 9)

The reception of symbol 'V leaves, according to (IV,9), an additional chance for each symbol that it wasactually transmitted as such. The uncertainty as towhether the received symbol is the' same as thetransmitted symbol is therefore

. hv = - ~ PAv log PJ.v.).

(IV, 10)

The' average uncertainty after reception of any symbolis then:- PJ.PJ.vh = ~ Ql,hv = -~~PJ.PJ.v log. .. (IV,l1)

Il • ), ~ Pf.lPf.lV '.. Il.

The average quantity of information reteived persymbol is t?erefore .

H-h--.....:..~ [pvlogp;-~PJ.PJ.vlog PJ.pJ,v ]. (IV, 12)'~ ;. ~Pf.lPf.lv

. .. . Il "

The noise properties of the communicationchannel are entirely determined by the transition.probabilities Pu»: For a, noise-free channel, to'which (IV, 7) applies, it can be readily-verified that

"

'."

210 PHILlPS T1j:CHNICALREVIEW VOLUME 18

ïï = 0.' If the transition prohahilities are given, it ispossible to choose the relative probabilities pp of the,code symbols to be transmitted ill such a way thatthe symbols most liable to interference (with the',highest hp-values) are least used. This means aspecial choice of coding. With an optimum choiceof ..coding, ïï is small and the quantity ofinformation transmitted-per .symbol is maximum.The quantity of information th~ transmitted iscalled the capacity' of the. communication channel.Shannon has demonstrated that the capacity ofa communication channel' can in' principle befully utilized 'by dividing each message into longseries of symbols (letters) and representing eachseries by a unique set. of code symbols (series of ,pulses). Of all sets of- symbols that it w~uld bepossible' to use, only those are chosen for the code

. that differ- so much that they are least liable toconfusion. In practice -this would require a sort' ofdictionary of all kinds of extensive messages, .ready'for use and transl~ted into' sets of code symbols.The scope of the code-books and the delay in lookingfor the "translation" obviously impose a limit to thepossibility of realizing -this ideal in practice.

VIe shall now apply the foregoing to the earlier-described simple 'model of a communication system,in which, coding is effected in the form of voltagepulses whose height determines the symbol. In anoise free communication channel an infinitenumber of symbols is in principle available, 'sinceboth at the transmitting and at' the receiving end aclear distinction can be made between arbitrarilysmall differences in amplitude. In this way anymessage could be expressed in a single pulse bymeasuring its height to millions of decimal places,the first two decimals. representing the first letter;the next twó decimals the second letter, etc. In. reality, noise is always present, with the result thatthe original height of a transmitted pulse can bedetermined at the receiving end only' with i givendegree of uncertainty, so that only pulses 'thatalready possessed a finite difference in height atthe transmitting end can be distinguished. ., Thermal noise has, at constant temperature and inlinear nétworks, 'a stationary character, -i.e. theprobability distribution of the fluctuating noisevoltage V does not depend on time: This probabilitydistrihution is Gaussian: the probability that. thevoltage at' a given moment lies betweén V andV+dV is given by

~ (V) dV ' 1/~e-ar-:' d;. (IV, ~3)

.The root mean 'square of; the noise voltage, or the, '

power-developed in a resistance of In, then amountsto: '/

+co

N .: V2 ' l' /~ fV2 'e-aV'dV ~!__. '(IV;14), I 'TC 2a

-co

Two transmitted pulses of different 'height canonly be distinguished at the receiving end if thedifference in level, expressed as a measureof energy,is at least of the same order as the amount' given ,'by (14). A further calculation shows that if'fhe 'voltage levels differ hyan amount i21V , Y I/a,there is probability of about' t (0.48 to be exact)that noise will cause pulse di~tortion to the extent of,half the distance to an adjácent level. There is thusa chance of O.4S that a .transmitted. pulse will befaultily received (too high or too low}, The levels.should therefore be placed somewhat- further apart.If intervals of i6N are used, the chance of faultyreception becomes 0,.22,which is considered accept-able in practice. For a communication system of the ; 1type described here the levels will, t~erefote, heplaced at voltage intervals of i6N_. 9nly a finitenumber of pulse heights is then, of course, permissi-ble, viz. iEllV' + 1= lts+ 1, E repreaenting the 'mean signal power, and hence s the signal-to-noiseratio (in terms of energy).

, The above number is arrived at as follows, Let the maximumpulse voltage be Vmax- If the number of equidistant pulseheights is n + 1, than the various pulse amplitudes are given.by gVmnxJn, where g may have any integral value between 0and n, Each pulse represents a power !g2Vmnx2Jn2, and if allpulse heights ?ccur with equal probability, the mean signalpower amounts to

E = !Vmnx212 + 22 + ...-+ n2

RI Vmnx2.

n2 n + 1 6

Hence Vmnx RI If 6E. Since the distance between the levelswas fixed at If 6N, the number of permissible levels becomes

'1/6E VE ,_.,': 6N + 1 = N + 1 = I' ,8 + 1. .'

. ,In practice not only are the pulse heights restriet- '

,ed but also the pulse frequency, i.e. the number ofpulses that can be distinguished per second, This is, ' ., attributable to the fact that in every communicationchannel only a limited frequency range is availablefor the trans~ission. Let the available bandwidth beJo; in othér words, assume that only frequencies,between a given' frequency Je and the frequencyJc + J() are transmitted. It can be proved : on'general grounds that· two, signals each of dura-tion r seconds can no longer be distinguished if'they have the same amplitude at more than: 2for ' •• iÓ.

instants.,',

.,, ,

. -... ", .~\ "

1956/57, No; 7 IV. ENTROPY ÀND INFORMATION

.'

A peri~dic signal of~eriod • ca~ he repres~~ted by the follow-' ° cable must have' a" reflection-free terI~lÏn~tion. lÏiing Fourier-series: ' 'that case, 'as mentioned in the- first articie of this. ~ .,. ,

a co • , t ,co' t series (Vol. 16, p. 262), a noise power. N === kTafo inv(t) = -20 + ~ ak cos 2nk- + ~ bk sin 2nk ---:.1 • • 1 • a bandwidth jo is produced by the cable at 'a cable

If only the frequencies betweenD and.fo = 1/. occur. only teinperáture Ta; k represents Boltsmann's constant,21 + 1 ='2fo. + 1 ',terms occur in these expansions, and If,the signal power that can be extracted from thev(t)cis thus completely determined by 'the,21 + 1 available cable amounts to S, then the ':rhàxim~m quantityofcoefficients (ao• aH ...... al. blO ••. : .• bi). More coefficients do not information that can b'~transmitted in the time. is,define the shape of the signal any better. The 1may be neglect-ed if 2.foi~ 1 which is always the case. For a _non-periodic accordin~ to (IV, 16):, 'signalof duration. the series is replaced by a Fourier-integral, S' .but the conclusion that v(t) is completely defined by 210data 1,= for log2 (1+_'_).per 'second remains valid. These conclusions. drawn for the , , -, kTafofré~ency range between 0 andfo. are equnlly applicable 'to thearbitrary frequency range between fe and fe' + fo. .

Pulse series lasting 7: seconds are .thus completelydefined by the amplitude at 210. instants. This meansthat the maximum quantity' of information that canbe contained within a bandwidth of 10 cis in a pulse .'series of durati~n. seco~ds is attained ifin that time2fo. pulses are transmitted.". .Since, as we have shown above, à choice can he

made of ys + 1 discrete heights for the trans-mission of each pulse, the total number of distinct(i.e, different) pulse series of length • amounts to(is + 1)2!,T; In such a series of length r, therefore,the maximum quantity of infor~ation is

I ° 2fodog2 (ys +1) bits. (IV, IS)

This largest possible quantity ,of information iswhat we earlier termed the channel' capacity; thistime, however, it is expressed' in terms of thebandwidth and the signal-to-noise ratio 'of thecommunication channel in question.

The derivation given abóve is broadly that givenby Tuller 14)., Shannon 1) arrives at the directlycomparable result: '

(IV,16)

For strong signals this expression is virtuallyidentical with (IV, 15). The relationship betweenefficiency of information-transmisslon and band-width of thé communication channel; mentionedbriefly' above in. connection with the 'television 'picture, will now be clear. Thegreater .the redund- 'ance of the message to be transmitted the lèss, efficiently will the channel 'capacity be 'utilized, ~nd_a bandwidth greaterthan follows {rpm (IV, 16) willthen be required for transmitting a given quantityof informätion, . .

Information and energy

If a signal is transmitted ,over a cable. and wew!sh ~o receive the maximum .signal ~nérgy, the

14) VI: G. Tuller, Pr~c: I~st. Radio. Engrs. 37. 4~8-478; 1949.

211

.'

(IV, p)

The quantity of information available per unit timeand per unit signal power is then:' :'

~,' .&. log2 (1+__S_)S... S kTafo .

Considered as a function 'of the variable Slfo, this'function always has a negative differential coefficierrtConsequently it reaches its maximum in the limitingcase where the signal power is zero. By- expandingformula (IV, 18) in terms of SlkTafo, the' maximumavailable quantity of information per unit time ándper unit power is found'to be- "

(IV,18)) ,

,',. ,~,

, ,, "';j

"

(!_) , _1 log2 e.S. max ,kTa

Thus, to gain' I, bit of information per second werequire at the very least (i.e. for very small Silo) anenergy of kTa/log2e, i.e. kTaln2: T~isconsequence ofShannon's calculations has been' pointed .out byFélker 15): According as the bandwidth is smaller andthe signal power greater, the energy required perbit increases.

In this" connection, comtnunication theory prov-ides a.means of solving a classical paradox relating. to the second law of thermodynamics, viz. that. concerning "Maxwell's 'demon". Maxwell hadpointed out that a being capable of acting on a ,molecular scale' wocld be able to cause changes,in state that are in direct conflict, with the second'law. Let us imagine, e.g. a gas-filled vessel whose,walls are perfectly insulating, A -pàrtîtion, also aperfect insulator, with a small hole-in- it-dividesthevessel into two' compartments. The "demon" canclose tMs :hole with a .valve, which' he operates 'insuch a way that only rapid. moiecules are passed

• from right to, left and slow ones from left to right. •In this manner he, divides the gas into' a hot and acold part, thus reducing the statistical-thermo-dynamical entropy of the system without doing anywork. It" has often -heen pointed out that thisreasoning is unsatisfactory, for several reas~ns.·,16) J.'H. Felker, Pröc, Instn. Radio Engrs;'40. 728:729.1953~ •

(IV, 19) " '

... ,- ~.,....'

212 PHILIPS TECHNICAL REVIËW VOLUME 18

Brillouin in particular has drawn attention to thefact that the demon would require informationregarding the speed of the molecules for his manipu-lations. The demon' would he unable to "see"anything in a space of constant temperature andwould have to use the additional energy of a "lamp".This information would cost at least as much energyas corresponds to the gain in entropy -=- and, inpractice, far more, even with the most favourablemeasuring technique conceivable. '

It is interesting from a historical point of vie~,that as early as 1929Szilard 16), on thermodynamicalconsiderations, came to the conclusion that 1bit ofinformation {though he expressed it otherwise] in asystem of constant temperature would have tocost an energy of at least kTln 2 in order to avoidconflict. with the second làw. .Szilard's imaginaryexperiment' can be briefly described as follows. A

r single molecule is contained in a vessel, kept at aconstant temperature by a heat reservoir. In thevessel is a piston with an opening that can be closedwith a sliding valve. The piston is initially in themiddle of. the vessel. At the beginning of theexperiment the observer doses the opening andlooks to see whether the molecule is at the right orat the left. Then he allows the piston to be graduallymoved in the opposite direction. The molecule hitsthe piston several times, thus expending work uponit. This is the energy which corresponds to the iso-thermal expansion of a perfect gas from the volumetV to the volume V, during which an amount ofheat kT In 2 per molecule is abstracted from thereservoir (This is deducible from I, equation 15).After some time the piston reaches its extremeposition. The 'valve is now opened and the pistonis returned to its initial position, after whichthe whole process can be repeated.At first sight the foregoing conflicts with the.

second law, since energy is being continuouslyextracted from a constant temperature sourceapparently without doing any work. In order toavoid such a' discrepancy, however, Szilard assumesthat the measurement itself, which involves deter-mining the place of the molecule _:_ i.e. supplyingone bit of information (right or left, i.e. two equallyprobable' possibilities). -:- requires an energy of atleast kT In 2. The application of ideas such as entro-

. py, 'temperature, isothermal expansion etc. tothis gas, which consists of one molecule, as well asthe idea of action on a molecular scale in thisimaginary experiment deserves to be subjected tocriticism. Szilard himself partly answers this

10) L. Ssilard, Z. Phys. 53, 840-846, 1929.

criticism by extending the argument to large num-bers of molecules; others such as Rayniond, havesince described 'analogous imaginary experimentswith gases consisting of more than one molecule.However, the merit· of Szilard's achievementremains: he derived an interesting result whichsatisfactorily agrees with present ideas in informa-lion theory. .

In this àrticle we have seen that th~re is a connec-tion between the "entropy" of informati~n theory,and the statis'tical-thermodynarnical entropy. 'Yefound this connection when determining the amountof energy needed for the transfer of informationthrough, ..a channel ~ith noise. Since the intensity ofthe noise depends on the temperature, the energymust also depend on temperature, and it is thereforehardly surprising that the thermodynamical notionof entropy should come into these considerations.Discussion of Szilard's 'experiment, however, showshow little the term "entropy" is justified in inform-ation theory. The use of this term has sometimesbeen responsible for the following explanation of theresult obtained by Szilard: 1 bit of "informationentropy" equals a "thermodynamical entropy" ofkIn 2. However, the introduetion of Boltzmann'sconstant k in pure information theory is really quiteirrational (as opposed to its natural appearance inthermodynamical considerations of communicationchannels, etc.). For example, what could be thesignificanee of Boltzmann's constant w'hen dealingwith the "entropy" of a language, where there is noquestion of either noise temperature or energy?What indeed could be the thermodynamical"system" ofwhich the language forms a part? In theabsence of such a system, Boltzmann's constantclearly has no place.So long as we confine ours~lves to the statistical

aspect of the notion of entropy there is' nodanger of confusing ,thermodynamics and informa-tion theory. The mathematical analogy, however,does not imply that the experimental laws whichare valid for thermodynamical entropy (second law),should have a corresponding physical significance ininformation theory.

Summary. In the relatively new branch of science known asinformation theory, which is mainly concerned with communi-cation and .cybernetics, the notion of information is given aquantitative meaning on the basis of statistical considerations.A quantity termed "information content per symbol" is thusderived, which is formally closely analogous to statistical-thermodynamical entropy (see the previous articles I, II andIII of this series). The concept can be easily applied to any givenseries of events each of which may occur with a certain degreeof probability, as for example the occurrence of letters in alanguage. This has led some to speak of the "entropy" of alanguage, or the "entropy" of a television picture.

1956/57, No. 7",'

IV. ENTROPY AND INFORMATION 213

An important application of information theory lies in theinvestigation of the efficiency of the transmission of informa-tion, e.g. by electric signals. By methods which show a certainformal agreement with those applied in. statistical thermo-dynamics, it is possible to derive highly general results con-cerning the way in which the noise and' bandwidth of thecommunication channel affect thc capacity of the channel.Finally, the minimum quantity of energy needed for thc trans-mission of a given quantity of information per unit time is

discussed. Owing to the effect of thermal noise, it is found todepend on the temperature of the communication channel.This 'dependence on temperature establishes a connection withthermodynamical entropy, but the connection is based ontrivial grounds, and constitutes no justification whatsoever'forthe common use of the' term "entropy" in the theory ofinformàtion, This abuse may lead us wrongly to expectanalogies between thermodynamics and the information theorywhich go beyond their common statistical background.

ABSTRACTS OF RECENT SCIENTIFIC PUBLICATIONS BY THE STAFF OFN.V. PHILIPS' GLOEILAMPE~FABRIEKEN

Reprints of these papers not marked with an asterisk .. can be obtained free of chargeupon application to the Philips Research 'laboratory, Eindhoven, Netherlands.

2330: A. M. Kruithof: Quelques idées sur les t.rans- 'formations du verre (Verres et Réfractaires9, 311-319, 1955).

Measurements made with different glasses showthat ány 'glass can have various densities at roomtemperature.' The difference in density is the resultof different thermal histories of the glass samples.

, On the basis of a theory of the fre~zing-in, ofstructure states, t-he existence of different densitiesat room temperature can be explained with theuse of the contraction curves. Following the defini-tion of Tool, these structure states can he character-ized by a "fictive temperature". Ideas on changes instructure states can explain the mutual differencesbetween expansion curves as well as differencesbetween these curves and the contraction curves.The conceptions of "transformation point" and"transformation range" are discussed on the basisof this theory. Other physical properties e.g.viscosity and electrical conductivity (direct current)show similar phenomena. The curve representing thelog of the specific resistance as a function of theinverse of the absolute temperature consists of threestraight lines, Both points of intersection maycorrespond to transformations in the glass. Thermaldifferential analysis was used as a check on thishypothesis. That the lower point does correspond toa transformation is not, however, completelyestablished.

2331: H. G. van Bueren: Theory of the formationoflattice defects during plastic strain (ActàMet. 3, 519-524, 1955).'

A simple theory is presented by which a relationbetween the plastic strain and the concentration ofvacancies, interstitials and dislocations is obtained.Dislocations are formed' by sources under the actionof an applied stress, the other defects by the move-

"

ment of jogs, in the dislocations. The action of adislocation source under a varying stress is studied,for the case of static as well as of dynamic generation. 'After the first few percent of strain, both methods ofgeneration yield the same resulting defect concentra-tions. By suitable elimination, the influence qf workhardening could he left out of the theory, and thedefect concentrations could be expressed as functionsof the amount of plastic strain. The theoreticaldeductions should be valid between plastic strainsof 0.05 up to 1. They are compared with the observedcritical shear stress, the elementary structure andthe resistivity-strain relation in slightly deformedcopper.

2332: G. Brouwer: Electrical analog of the eddy-current-limited domain-boundary motion inferromagnetics (J. appl. Phys. 26, 1297-13011955).

-The observed losses in a ferromagnetic core are

always greater than those calculated on the basisof a homogeneous permeability. In a magnetizationprocess by domain-boundary displacements thepermeability certainly is not homogeneous, but'attains extreme values in' the' domain walls. As aresult the loss factor due t? eddy-currents is in~creased. This is, the well-known eddy-currentanomaly. The effect increases with the domain size.'An electric circuit analogue was used to determinethe eddy current distribution and domain-boundarymotion in a number of idealized cases.

2333*: A. van Weel: Unnouveausystème de mesuredes angles de phase (Aéröélectronique IerCongrès International, Dunot, Paris 1955,pp. 565-572.

. 1£ a four-pole is inserted in the feedback loop ofan oscillator, the frequency of the latter is affected

214 PHILIPS TECHNICAL REVIEW VOLmm 18

by the phase change suffered by th~ waveform onpassing through the four-pole (See also Philips tech.Rev. 15, 307-3;t6, 1953/54.) This principle offersconsiderable advantages for phase angle measure-ment. By' transforming phase varîations intofrequency variations, phase meters with the follow:ing properties can he designed: range 0-360° or more,precision 1°; small variations of phase measurableto an accuracy better than 0.01°; direct indicationon a 'calibrated linear scale. For certain applicationsthe unknown phase angle originates in a four-polewhich cannot be inserted in the oscillator loop.To measure the phase relation between two A.C.voltages a new principle has been devised whichpermits the unknown phase angle to be introduced.lito', a separate oscillator of special configuration.To the above-mentioned properties may then beadded: insensitivity to not-too-large frequencyvariations in the given voltages. Possible applica-tions include altimeters and rangefinders.

2334: H. B. G. Casimir: On the theory' of super-conduotivity (Reprinted from: Niels Bohrand the development of physics, PergamonPress, Londo~ 1955).

Survey of certain aspects of superconductivityand of proposed theories of this phenomenon.

2335: J. de Jonge and B. H. Biho: The preparatienof ortho and para hydroxybenzyl alkylethers (Rec. trav. Chim. 'Pays Bas 74,1448-1452, 1955).

Alkyl ethers of ortho and para hydroxybenzyl-alcohol can he obtained with. a good yield by heatingthe hydroxybenzyl alcohol with the appropriatealkyl alcohol at 150°C.

"

2336: G. Thirup: Wide-band three-phase RC-generators fo'r complex measurements oftwo-poles and four-poles (J. Brit. Instn,Rad. Engrs. IS, 597-605, 1955).

From a 3-phase R-C oscillator t'Y0 voltages arederived, the complex ratio of which can be varied.The complex ratio .is independent of the frequency.The two voltages are used in a composition circuit

. for measuring the parameters of two-poles and four-poles. Two equipments are described covering thefrequency ranges 20 c/s-22 kc/s and 22 kc/s-Iû Mc/s.The possible error is ± 0.5 db and ± 2° in thefrequency range 100 els - 3Mc/s. Outside this rangethe phase error may Increase about 3 times while theamplitude error remains nearly the same.

2337: P. Jongenburger: The magneto-resistance ofmetals deformed atlowtemperatures (Suppl.Bull. Inst. Int. du Froid, AnIl;exe 1955~.

Carefully annealed pure ~opper wires were plastic-ally deformed at 20 OK. Their magneto-resistancewas measured at the same temperature before andafter the deformation. When the results were plottedin a Kohler diagram, a large influénce of thedeformation was observed. This effect was shown tobe due to' dislocations only; it was independent of .the simultaneous presence of.point-defects.

2338: F. A. Kröger:' The physical chemistry ofcrystal phosphors (Proc. I.R.E. 43, 1941-1944,.1955, No. 12).

After an historical introduction, a survey is givenof the present views regarding the constitution andthe preparatien of inorganic crystals phosphors.Partienlar attention is paid te:' the incorporationof atoms with a valency deviating from that..of the,atoms of the base material, and to the stabilizationof atoms in a particular valency.

2339: E. W. Gorter: Some properties of ferrites in 'connection with thêir chemistry (Proc.I.R.E. 43, 1945-1973, 1955, No. 12).

Mter an elementary introduetion on the origin ofthe magnetism of oxides, it is shown how the mole-cular-field hypothesis can amount for the magneticproperties of ferromagnetics and antiferromagnetics,and for those of non-compensated antiferro-magnetics, with which latter materials we areconcerned here, A brief description of the spinellattice is given, and also an account of the crystalchemistry of the spinels, which is necessary to under-stand the experimental saturation magnetizationsdiscussed. A short suryey of methods of preparatiónis given. The second part discusses the anisotropies,and' some of the magnetization prpc,esses whichinfluence permeability, and the factors whichinfluence high-frequency permeability and losses.Among these are the ferromagnetic resonancephenomenon and the dimensional resonance andrelax,ation phenomena. The ,way in which thesefaétors are influenced by chemical composition andpreparation technique' is indicated. Finally, a shorthistory of the development of ferrites is given .

2340: C.O. Jonkers: Die Philips Oaskältemaschine(Allgemeine Wärmetechnik 6,203-206,1955).

Abbreviated version (in German) of articlepublished in Philips tech. Rev. 16, 69~78, and105-115, 1954/55.

""

.' ," - .~i

1956/57, No. 7 ABSTRACTS OF RECENT SCIENTIFIC PUBLICATIONS 2152341: J. S. van Wieringen: Paramagnetic- reson-

ance of divalent manganese incorporated invarious lattices (Disc. Faraday Soc. No, 19,1955)., '

Paràmagnetic resonance was observed in powdersamples containing Mn diluted in various diamag-netic compounds, The measurements were rr{ade at

, , 3.2 and 1.25 cm, at room temperature. Three effectswere found: narrowing of hyperfine splitting bycovalent bonding and by exchange, and a g-factorsomewhat larger than the free-electron value inZnSe-MnSe and CdTe-MnTe mixtures. The firsteffect suggests 10-20%, 30%, 35% and 40%covalent bo.nding in Mn-o.xygen compounds, 'MnS,MnSe and MnTe respectively. '

2342: A. A. Kruithof and J. L. Ouweltjes: Colourrendering .hy de luxe fluorescent lamps(Proc. Int. Comm, Ill. Zürich, 1955). '

Description of a method of improving the colourrendering o.f fluorescent lamps by the addition of aspecial phosphor. The principles governing thecolour rendering are' discussed and the' applicationof these principles demonstrated with regard to.some newly developed "de luxe" lamps:

2343: J. J. Balder and G. J. Fortuin: The influenceof time of ohservation on the visibility ofsta!io.nary objects (Proc. Int. Comm, Illn.Zürich, 1955).

Interim repo.rt of new measurements, with newapparatus, of-the threshold values of visual acuity.Apart from the variables concerned in an earlierinvestigation byFortuin (sec these abstracts R,170and~R 174), the effect of object ohservation timewas also examined.

2344: H, A. Klasens, ·P. Zalm and G. Diemer:Characteristics of electroluminescent cells[Proc. Int. Comm. Illn. Zürich, 1955).

A consideration of the chemical nature of anelectroluminescent phosphor. is followed by a dis-cussion of the electrical and optical characteristicsofan electre-luminescent cell, making use of a simpleequivalent circuit. On the basis of these conaidera-'tions, and direct ohservations, the properties of theindividual phosphor crystals arederived. A relationis found expr~ssing the brightness B in terms o.f thefrequency co and the applied voltage V. A theory isproposed for the mechanism of electroluminescencewhich explains this relation, Finally, some commentsare made on the practical application of electro-luminescent _lamps. .

2345: H. Zijl: Computed coefficients of utiliaation(Proc. Int. Comm. Hln, Zürich, 1955).

j

. A method is described by which utilization tab,les<?anbe computed which take account o.f the natureof the illumination of the working surface and thelight distribution over all the surfaces enclosing theilluminated space: It is possible to. present the.datain the form of simple graphs and tables.

2346: J. B. de Boer and J. F. T. van HeemskerckVeeckens:' Observatio.ns on discomfort glar-ein street-lighting; influence of the colour ofthe :light (Proc. Int. Comm, Illn. Zurich,19~5).

Experimental investigation into. the effect o.f thecolour of the light on glare in street lighting. Of the'various lamps used, the observers showed ~ definitepreference for the "warmer" -coloured light sources,The acceptable luminance (to. ensure a certain degreeof comfort) was found to be approximately pro.-portional to. the square root of the road luminance.Differences in the results from those of otherinvestigators can partly be attributed to. differencesin the measuring techniques. .

2347: H. Bremmer: Diffraction problems of micro-waveoptics (I.R.E.Trans. 4, 1955).

Survey of current methods applied in microwavediffraction theory.

2348: A. A. Kruitho.f: Chromatic adaption ~th,near white backgrounds (Die Farbe 4,147-158, 1955).

Repetition and extension of work carried outearlier in collaboration with P. J. Bouma (Philipstech. Rev. 9, 257-266, 1947/48).

2349: J. L. Meijering, G. W. Rathenau, M. G. vander Steeg and P. B. Braun, A miscibility gapin the face-centred cubic phase of the co.pper-nickel-chromium system (J. Inst. Metals84, 118-12~, 1955/56).

The copper-nickel-chromium system has beenstudied by metallographic and Xeray-diffractionmethods, particular attention being paid' to. theisothermal section at 930°C. In addition to. thebody-centred cubic phase rich in chromium, two.face-centred cubic phases have. been found to. exist,although ,the binary copper-nickel alloys showcomplete miscibility.

2350: C. Meyer: Gasentladungslampen und ihrephysikalischen Prohleme (Physikalische-Blätter 12, 19-28 and 62-73, 1956). (Gasdischarge lamps and their physical problems;in German) . ' . '

Following; a short historical review, questionsrelating to. the efficiency of Iuminous sources,

_ ....._ ~"

216 PHlLIPS TECHNICAL REVIEW VOLUME 18

fluorescence, colour rendering and the conditions of 2354: G. D.' Rieck: Fragmentation in tungstenoperatien (striking and stahilisation] are discussed, crystals (~cta, Met. 4, 47-51, 1~56).The lamps discussed include sodium vapour lamps, 'Back-reflexion-Laue diagrams were made of Whigh pressure and super-high P!es,s~re mercury crystals in recrystallized lamp filament wires, using \lamps, high pressure xenon lamps and ..electronic a sp~cially collimated X-ray mic;o~beam. :Differ-fl'ash tubes. The article ends with a t~bie showing ences in orientation of 1-2' could be' observed.the spectral distribution of the ,light from a number .,;rl'opetimes two crystals with orientation differenceof gas-discharge lamps as compared with daylight .::of".i 0_20, are irradiated at the same time. The ' ,d .. d 1 li h - s, ..OJ:>£I-lj S O'i:.s fJ,o.,'an mean escent amp g t. ,crystals oftenshowed Laue diagrams with spots split

in the direction of the wire. axis, with angular ._differences of 2'-30'. ,After cold bending, fragmen-tation was found, which was more clearly seen afterre-straightening, for then the deformation asterismdiminished. The fragmentation occurred mainly intwo directions. One corresponds with the normalbreaking-up-of the lattice in the direction ofbending.The other was approximately perpendicular to thelatter and gave smalllines parallel to the wire axis,arising from particles with orientation differences of1'-10' and dimensions of about 10 {Jo. Bearing inmind the split spots occurring in unbent wires andthe fact that during drawing contaminations in thewire are stretched out along its length, we, mayassume a predisposition of the crystals to break alongplanes parallel to the axis. .

2355: W. K. Westmijze: The fundamentals ofmagnetic recording (T. Ned. Radiogenoot-schap 21, 1-15, 1956).

A short description is given of the magneticrecording method in general, and some details aretreated more elaborately. For the understanding of 'the h.f. biasing method use is made of a simplifiedhysteresis curve. In this way an explanation can hegiven of some ~f the peculiarities met in recordingand erasing. The recorded signal is attenuated bythe demagnetizing field, and during reproductionnot all the flux in the tape is reproduced. This isdiscussed for short, long and intermediate ,wave-lengths. Finally, the factors are surveyed which.influence distortion, frequency response, noise andprint effect.

2356: G. P. Bakos: Mechanical aspects ofmagnetic-recorder design (T.: Ned. Radio-'genootschap 21, 17-37, 1956).

The quality of magnetic recording is deferminedto a great extent by the electro-mechanical design .of the tape-drive mechanism. The task of thismechanism is to make the tape run with a constantspeed and in intimate contact with the magneticheads. It depends on the constructional details howfar this task is fulfilled. This article deals with thevarious factors which have to be taken into accountwhen designing tape drive .mechanisms,

2351: O.W. Memelink: The distribution of impuri-ty in à semi-infinite solidified melt (Proc.Phys. Soc. B 69, 119-120, 1956).

The transient distribution of impurity segregatingupon solidification in a semi-infinite melt has beencalculated. The expression obtained, which is of,closed form, contains the following parameters:the constant velocity of the moving solid-liquidinterface, the ratio of impurity solubility in the solidto that in the liquid state and the diffusion constantof the impurity in the liquid state.

2352: J. E. Rombouts and J. Links: The chemicalnature of the antibacterial substance presentin Aucuba japonica Thunbg (Experientia 12,78-80,1956).

An inactive precursor of the antibàcterial sub-stance present in the aucuba plant has been isolatedand identified from the seeds. The precursor wasidentified with aucubin, a glycoside of a furanderivative. From the leaves an enzyme could beextracted which produced from the aucubin (pre-sumably by hydrolysis) 'the antibacterial substanceaucubigenin. The activity spectrum of this sub-stance is described. It is asserted that the anti-bacterial substance present ID crude juices, from.leaves is identical with the hydrolysis product ofaucubin.

,2353:\

H. J. G. Meyer: Interaction of excitons withlattice vibrations in polar crystals 1 (Physica22, 109-120, 1956).

A theory describing' slow excitons which interact, with ,longitudinal 'optical lattice. vibrations isdeveloped within' the framework of the effectivemass approximation. In many respects' the theoryis very similar to --;: and makes explicit use of - theordinary theory of polarons. Interesting differenceswith polaron theory are introduced by the fact thatan exciton has an internal degree of freedom and isas a whole electrically neutral. The effect of latticevibrations on optical exciton transitions is investigat-ed with the aid of the above theory.