The Combinatorial Basis of Entropy (“MaxProb”)
22nd Canberra International Physics Summer School ANU, Canberra
11 December 2008
by Robert K. Niven
Marie Curie Incoming International Fellow, 2007-2008 Niels Bohr Institute, University of Copenhagen, Denmark
School of Aerospace, Civil and Mechanical Engineering
The University of New South Wales at ADFA Canberra, ACT, Australia
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 2
Lectures
1. The Combinatorial Basis of Entropy (“MaxProb”)
2. Jaynes’ MaxEnt, Riemannian Metrics and the Principle of Least Action
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 3
Contents • Historical overview
- combinatorics - probability theory
• Combinatorial basis of entropy / MaxProb principle
generalised combinatorial definitions of entropy and cross-entropy
explanation of MaxEnt / MinXEnt
• Applications 1. Multinomial systems (asymptotic vs non-asymptotic) 2. (In)distinguishable particles or categories 3. “Neither independent nor identically distributed” sampling
• Future applications …
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 4
(Advertisement)
Courses at UNSW@ADFA, Canberra:
• Short course in “Maximum Entropy Analysis”, 14-15 May 2009
(fee paying $1270).
• Masters course: ZACM8327 Maximum Entropy Analysis, semester 2,
2009 (fee paying or UNSW@ADFA enrolled student)
- 3 hours of lectures + tutorials per week
- based on similar course at Niels Bohr Institute
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 5
Historical Overview
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 6
Combinatorics Knowledge is very old! (Edwards, 2002)
(a) Number-patterns
Pythagoras (500BC)
Egypt (300BC)
Theon of Smyrna, Nicomachus (100AD)
Higher dimensions: Tartaglia (1523, publ. 1556)
figurate numbers fk
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 7
(b) Binomial coefficients
= coefficients of (a + b)N
Al-Karaji (1007); Al-Samawal (1180); Al-Kashi (1429)
Chia Hsien (1100); Yang Hui (1261); Chu Shih-chieh (1303)
Cardano (1570), etc
- applied to solution of equations; finding roots; etc
Chu Shih-chieh (1303)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 8
binomial coefficients
N
k= f
k , where =dimension
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 9
(c) Combinations + Permutations
Ancient; e.g.
- Susruta (600BC), Jains (300BC): combinations of 6 tastes
- Pingala (200BC): combinations of syllables
No. of permutations of N things = N !
- Hebrew Book of Creation (700); Bhaskara (1150)
No. of groups of N things, taken k at a time: - Mahavira (850); Bhaskara (1150); ben Gerson (1321)
Without replacement With replacement
Combinations
CkN
=N
k=
N!
k !(N k)!
wC
kN
=N + k 1
k=
(N + k 1)!
k !(N 1)!
Permutations PkN
= N(k) =N!
(N k)!
wPkN
= Nk
with
N
kk=0
N= 2
N
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 10
Pascal (1654) - equivalence of figurate numbers AND binomial coefficients AND
numbers of combinations without replacement
Multinomial weight - Bhaskara (1150); Mersenne (1636)
= no. of permutations of N objects, containing ni of each category
i = 1,...,s , is:
W =N !
n1!n
2! ... n
s!= N !
1
ni!
i=1
s
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 11
Probability Theory (a) Classical period (e.g. Cardano (1560s), Pascal, Fermat, Huygens, the Bernoullis,
Montmort, de Moivre, Laplace)
Probability =
No. of outcomes of interest
Total no. of outcomes
(b) “Frequentist” school (e.g. Venn, Pearson, Neyman, Fisher, von Mises, Feller)
- probability = measurable frequency, for an infinite number of repetitions of a “random experiment”
- attempt to define probabilities as certainties
- narrow applicability
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 12
(c) “Bayesian” or “Plausibility” school (Bayes, Laplace, Jeffreys, Polya, Cox, Jaynes, 1957; 2003)
- probability = “plausibility” = assignment based on what you know
- need not be a measurable frequency - manipulate using sum + product rules (Jaynes, 2003)
- “subjective” = “information-dependent”
- different observers, with different information, can assign different probabilities to the same event
- more useful; encompasses all frequentist situations
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 13
Probability Distributions “Measures of Central Tendency”
Continuous parameter x Discrete parameter x p(x) = probability density function (pdf) p(x) = pi = probability mass function
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 14
Combinatorial (or Probabilistic) Definition of Entropy
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 15
Definitions Entity = a discrete particle, object or agent, or an item in a sequence, which is
separate but not necessarily independent of other entities
Category = possible assignment of an entity
Probabilistic System = a set of entities K assigned to a
set of categories C by a discrete random variable : K C
e.g. physics: particles energy levels
gambling: die throws die sides
communications: signal bits letters of alphabet
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 16
Configuration = distinguishable permutation of entities amongst categories e.g.: physics: microstate; information theory: sequence
Realization = aggregated arrangement of entities amongst categories = set of configurations e.g.: physics: macrostate; information theory: type
Commonly define realizations by the no. of entities in each category {n
i}
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 17
MaxProb Principle MaxProb Principle (Boltzmann, 1877; Planck, 1901; Vincze, 1974; Grendar & Grendar, 2001)
- “A system can be represented by its most probable realization”
principle for probabilistic inference
- does not depend on asymptotic limits
- does not give certainty
BB GB BG GG
Superset of 2nd Law “A system tends towards its most probable realization”
- not just thermodynamics!
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 18
MaxProb Principle MaxProb Principle (Boltzmann, 1877; Planck, 1901; Vincze, 1974; Grendar & Grendar, 2001)
- “A system can be represented by its most probable realization”
principle for probabilistic inference
- does not depend on asymptotic limits
- does not give certainty
BB GB BG GG
Superset of 2nd Law “A system tends towards its most probable realization”
- not just thermodynamics!
Ergodicity
Inference
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 19
Multinomial Systems (Boltzmann, 1877)
N distinguishable balls (entities)
s distinguishable boxes (categories)
qi = source (“prior”) probability of ball falling in ith box
= normalised degeneracy gi / gii=1
s
Probability of a given realization {n
i} is given by the multinomial
distribution:
Pmult = N ! qi
ni
ni !i=1
s
qi =1/s
Pmult
=W
mult
sN
; Wmult
= N !1
ni!
i=1
s
If categories equiprobable, can use multinomial weight
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 20
MaxProb: want to maximise:
Easier to maximise:
lnPmult = lnN !+ ni lnqii=1
s
lnni !
i=1
s
Asymptotic limit for N (rigorously by Sanov (1957) theorem; crudely by Stirling’s approx. lnm! m lnm m ):
DKL = limN
lnPmult
N= pi ln
pi
qii=1
s
where pi =
ni
N.
If qi = 1/s = constant:
hSh= lim
N
lnWmult
N= pi lnpi
i=1
s
Shannon entropy
Kullback - Leibler cross - entropy
= directed divergence
= negative of relative entropy
Pmult = N ! qi
ni
ni !i=1
s
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 21
Summary
• Kullback-Leibler and Shannon functions are asymptotic forms of the
multinomial distribution P
mult
• If minimise D
KL (MinXEnt) or maximise
hSh (MaxEnt) of a multinomial
system, subject to constraints
obtain asymptotic MaxProb realization
Boltzmann principle:
Define entropy and cross-entropy by: h=
lnW
N,
D =
lnP
N
(compare S
total= SN = k lnW )
hence always consistent with MaxProb
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 22
Jaynes’ MaxEnt • Jaynes (1957)
- minimise D or maximise h , subject to constraints
“most uncertain” distribution = distribution which contains the least
information
• BUT how do we define uncertainty?
- Jaynes only considers D
KL or
hSh axiomatic basis of Shannon
(1948)
• However, a system: - need not be multinomial ! - need not be asymptotic !
Kullback-Leibler or Shannon functions will not give the MaxProb distribution
If know P P
mult or N , must include this information !
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 23
Application to Multinomial Systems (Asymptotic + Non-Asymptotic)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 24
Why the Multinomial? (Niven, AIP Conf. Proc. 954 (2007) 133; Blower, pers. comm.)
Pmult = N ! qi
ni
ni !i=1
s
with pi =
ni
N
1. Frequentist approach
P
mult, {qi } = measurable frequencies
2. Bayesian approach
P
mult, {qi } = Bayesian probabilities
If ignorant about choice of model P , then all models equiprobable
must choose multinomial (“central model theorem”)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 25
Asymptotic Analysis Jaynes’ (1957) Algorithm
Minimise
DKL = pi lnpi
qii=1
s
subject to
pii=1
s= 1 and
pii=1
sfri = fr , r = 1,...,R
Form Lagrangian, differentiate w.r.t. pi
pi*= qi e 0
'
r frir=1
R
=1
Zqi e r frir=1
R
Z = e 0
'
= qi e r frir=1
R
i=1
s
Boltzmann
distribution
with 0
'=
0+1
Jaynes’ (1957, 1963, 2003) analysis many more (generic) relations
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 26
Isolated Thermodynamic System:
Isolated from rest of universe
e.g. microcanonical ensemble
Natural:
pii=1
s= 1
Mean energy:
pii=1
si = U
pi
*= qi e 0
'
1 i =1
Zqi e 1 i
D *
h*=
0
'+
1U
compare S* = k lnZ +U
T
or F = kT lnZ = TS * + U
Hence 0
'= lnZ = and
1=
1
kT
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 27
Non-Asymptotic Analysis (Niven, Phys. Lett. A, 342(4) (2005) 286; Physica A, 365(1) (2006) 142)
Use raw multinomial:
Minimise
D(N)
=lnP
N=
1
NlnN !+ ni lnqi
i=1
s
lnni !
i=1
s
subject to
nii=1
s= N and
n
ii=1
sfri= F
r, r = 1,...,R
Form Lagrangian, differentiate w.r.t. n
i
pi#=
ni#
N=
1
N
1 lnN !
N+ lnqi 0
(N)r(N)fri
r =1
R
1
where ( )= digamma function
Non - asymptotic
distribution
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 28
Example 1:
Multinomial
n1,n2,n3
s=3
q =
1
2,3
8, 1
8
subject to
n
i= N
i=1
s
only
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 29
Example 2: Multinomial
n1,n2,n3
s=3
q =
1
14, 4
14, 9
14
subject to
n
i= N
i=1
s
n
i i= E
i=1
s
T
with
= 1,2,4[ ]
U =
ET
N=
5
3
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 30
Example 2 (cont’d):
For constant
U =
ET
N=
5
3,
obtain
0(N)
=(N)
1(N)
=1
kT(N)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 31
Thermodynamic Double System:
System 1: N
1 particles; energy levels
i
System 2: N
2 particles; energy levels
j
Probs. of realizations {ni }, {nj } are:
Maximise P1P
2 subject to
n
i= N
1i=1
s,
njj=1
m= N
2 and
most probable distrib.:
pi#=
1
N1
1
lnN1!
N1
+ lnqi 0a
(N1)1 i 1
pj#=
1
N2
1 lnN2 !
N2
+ lnqj 0b
(N2)1 j 1
“Zeroth law” upheld
( 1 in common)
P1 = N1! qi
ni
ni !i=1
s
, P2 = N2 ! qj
nj
nj !j=1
m
ET = ni ii=1
s+ nj jj=1
m
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 32
Summary
- combinatorial approach straightforward analysis using system (not
ensemble) parameters
steps towards non-asymptotic thermodynamics
(without thermodynamic limit !)
Application to Information Theory
(Niven, Phys. Lett. A, 342(4) (2005) 286; Physica A, 365(1) (2006) 142)
Adopt Boltzmann principle as definition of information:
I =h
ln2=
log2 W
N or
I =
D
ln2=
log2 P
N (in bits)
non-asymptotic coding ?
non-asymptotic network theory ?
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 33
Application: (In)distinguishability
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 34
Statistics Consider role of distinguishability:
Disting. balls Indisting. balls
Disting.
boxes
Maxwell-Boltzmann
(Lynden-Bell)*
Bose-Einstein
(Fermi-Dirac)*
Indisting. boxes
? ?
* maximum of 1 ball per box
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 35
Statistics Consider role of distinguishability:
Disting. balls Indisting. balls
Disting.
boxes
Maxwell-Boltzmann
(Lynden-Bell)*
Bose-Einstein
(Fermi-Dirac)*
Indisting. boxes
D:I statistic I:I statistic
* maximum of 1 ball per box
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 36
(In)distinguishability Allocate students to PhD supervisors:
1. Disting. students disting. supervisors - consider personal interactions
2. Indisting. students disting. supervisors
- e.g. Dean
3. Disting. students indisting. supervisors
- e.g. student club
4. Indisting. students indisting. supervisors
- e.g. Government department Choice of statistic - and hence entropy - depends on purpose
Tseng & Caticha (2002): “Entropy is not a property of a system … [it] is a property of our description of a system.”
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 37
(a) Maxwell-Boltzmann
WMB = N ! gi
ni
ni !i=1
s
(b) Bose-Einstein
WBE =(gi + ni 1)!
ni !(gi 1)!i=1
s
(c) Fermi-Dirac
WFD =gi !
ni !(gi ni )!i=1
s
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 38
Entropy functions: Name Asymptotic Entropy Non-asymptotic Entropy
(Niven, 2005, 2006) MB
hMB= pi ln
pi
gii=1
s
hMB(N)
=1
Nln[(piN)!]
i=1
s
+1
Npi ln[N!]+ pi lngi
BE
hBE = ( i + pi )ln( i + pi )
i=1
s
i ln i pi lnpi
hBE(N)
=1
Nln ( iN + piN 1)!{
i=1
s
ln ( iN 1)! ln (piN)! }
FD
hFD = ( i pi )ln( i pi )
i=1
s
+ i ln i pi lnpi
hFD(N)
=1
Ni=1
s
ln ( iN piN)!{
+ ln ( iN)! ln (piN)! }
where i = gi / N = relative degeneracy
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 39
MaxProb: maximise h subject to
pii=1
s
= 1,
pi frii=1
s
= fr , r = 1,...,R
Name Asymptotic distribution Non-asymptotic distribution (Niven, 2005, 2006)
MB
pMB,i*
= gi e 0' r frir=1
R
pMB,i#
=1
N
1 ln[N!]
N+ lngi 0 ' r fri
r=1
R
1
BE
pBE,i*
=i
e 0+ r frir=1
R
1
pBE,i#
=
1
N
1( iN + pBE,i
# N) 0 r frir=1
R
1
FD
pFD,i*
=i
e 0+ r frir=1
R
+1
pFD,i#
=
1
N
1( iN pFD,i
# N +1) 0 r frir=1
R
1
where = digamma f’n; -1 =inverse digamma f’n
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 40
(d) D:I Statistic
(Niven, CTNEXT07)
- disting. entities
- indisting. categories, each with g indisting. subcategories
Can show
WD:I =N
n1,n2,...,nk ,0,...,0(g)
=N !
ni !
i=1
k
rj !
j=1
N
ni
g=1
min(g,ni )
i=1
k
where rj = no. of occurrences of j in
{n
i}
ni
g = Stirling no. of 2nd kind
Curious behaviour!
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 41
(e) I:I Statistic
(Niven, NEXT 07)
- indisting. entities
- indisting. categories, each with g indisting. subcategories
Can show
WI:I(g) =N
n1,n2,...,nk ,0,...,0(g)
= P ( j)
=1
min(g, j)rj
j=1
n1
where rj = no. of occurrences of j in
{n
i}
P ( j) = partition number
a + b + ...( )m
= Wronski aleph = “combinatorial polynomial”
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 42
Normal Polynomials Wronski (1811) alephs
(a + b)2 = a2+ 2ab + b
2
(a + b)3 = a3+ 3a
2b + 3ab
2+ b
3
(a + b)m =m
ta
tb
m t
t=0
m
(a + b)2 = a2+ ab + b
2
(a + b)3 = a3+ a
2b + ab
2+ b
3
(a + b)m = atb
m t
t=0
m
Hence
a
=1
m
= a1
t1
t1,t
2,...,t
a2
t2 ...a
t with
t = m
=1
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 43
Example: Non-Degenerate MB and BE statistics
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 44
Example: Non-Degenerate D:I statistic
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 45
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 46
Example: Non-Degenerate I:I statistic
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 47
Summary (non-degenerate, no moment constraints)
Disting. balls Indisting. balls
Disting. boxes
MB statistic
MaxProb; MeanProb
Highly symmetric
Strongly asymptotic
uniform distrib.
BE statistic
MeanProb only
Highly symmetric
Strongly asymptotic uniform
distrib.
Indisting. boxes
D:I statistic
MaxProb; MeanProb
Highly asymmetric
Slowly asymptotic, s N
Non-asymptotic, s N ?
I:I statistic
MeanProb only
Highly asymmetric
Non-asymptotic, s N
Monotonic asymptote for s N
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 48
Application:
Pólya Distribution
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 49
Pólya Distribution (Grendar & Niven, cond-mat/0612697)
- urn: M disting. balls, with mi of each
category,
mi= M
i=1
s
- sample: N balls, with ni of each category
- scheme: draw of ball of category i, return to urn + add c balls of same category to urn
“neither independent nor identically distributed” (ninid) sampling
Prob. of each realization {ni} is:
PPólya =M c + N 1
N
1 mi c + ni 1
nii=1
s
Multivariate
Polya
distribution
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 50
Pólya cross-entropy: put qi = mi/M, and =N/M: Name Asymptotic cross-entropy Non-asymptotic cross-entropy
(without Stirling approx.) Pólya (c>0)
DPólyax
=1
Npi ln
(N +1) ( Nc)
( Nc+ N)i=1
s
+ ln(qiN
c+ piN)
(piN +1) (qiN
c)
Pólya (c<0)
DPólyaSt pi
c( c +1)ln( c +1)
i=1
s
+1c
(qi + pi c)ln(qi + pi c)
1c
qi lnqi pi lnpi
DPólyax
=1
Npi ln
(N +1) ( Nc
N +1)
( N
FD
+1)i=1
s
+ ln(
qiN
c+1)
(piN +1) (qiN
cpiN +1)
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 51
Pólya MaxProb
Name Asymptotic distribution Non-asymptotic distribution Pólya (c>0)
pPólya,i#
=1
N
1F(N, c) + (
qiN
c+ pPólya,i
# N) 0 r frir=1
R
1
Pólya (c<0)
pPólya,i*
=qi
e 0+ r frir=1R
c
pPólya,i#
=1
N
1K(N, c) (
qiN
cpPólya,i
# N +1) 0 r frir=1
R
1
Compare Acharya-Swamy (1994) ansatz for “anyons”
pi* 1
e 0+
1xi
, with [ 1,1]
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 52
Future Applications: Graphs and Networks
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 53
Graph Entropy (Körner & Longo, 1973; Körner & Orlitsky, 1998)
- vertices = categories (alphabet)
- lines (edges) connect disting. categories
H(G,P) = limsupN
1
Nlog2 (G
P
N ) +1( )
where P = prob. distrib on vertex set
(GP
N ) = chromatic no. of graph G
P
N , for N-sequence
consider “heterogeneous” distinguishability of categories
BUT is asymptotic (does not consider entities)
Strong connection to networks + coding theory
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 54
Conclusions
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 55
Conclusions • MaxProb principle: choose realization of highest probability
principle of probabilistic inference
explanation for MaxEnt, MinXEnt
generalised definitions of D and h
• Non-asymptotic theory
- finite N thermodynamics (microcanonical) - other applications!
• Other statistics:
- MB, BE, FD - indisting. categories - Polya sampling (“ninid”)
• Strong connections to graphs, networks + coding
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 56
Acknowledgments:
Thanks to:
• The University of New South Wales, Australia
• The European Commission, for Marie Curie Incoming
International Fellowship at University of Copenhagen
• Dr Bjarne Andresen + Dr Flemming Topsøe
• COSNET, ANU and (Prof. R. Dewar)2 for opportunity to present
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 57
References Acharya, R., Narayana Swamy, P. (1994) J. Phys. A: Math. Gen., 27: 7247-7263. Boltzmann, L. (1872) Sitzungsberichte Akad. Wiss., Vienna, II, 66: 275-370; English transl.: Brush,
S.G. (1966) Kinetic Theory: Vol. 2 Irreversible Processes, Permagon Press, Oxford, 88-175. Boltzmann, L. (1877), Wien. Ber., 76: 373-435, English transl., Le Roux, J. (2002), 1-63,
http://www.essi.fr/~leroux/. Clausius, R. (1865) Poggendorfs Annalen 125: 335; English transl.: R.B. Lindsay, in J. Kestin (ed.)
(1976) The Second Law of Thermodynamics, Dowden, Hutchinson & Ross, PA, (1976) 162. Clausius, R. (1876) Die Mechanische Wärmetheorie (The Mechanical Theory of Heat), F. Vieweg,
Braunschwieg; English transl.: W.R. Browne (1879), Macmillan & Co., London. Edwards, A.W.F. (2002) Pascal’s Arithmetical Triangle: The Story of a Mathematical Idea, 2nd ed.,
John Hopkins U.P., Baltimore. Grendar, M., Grendar, M. (2001) What is the question that MaxEnt answers? A probabilistic
interpretation, in A. Mohammad-Djafari (ed.) Bayesian Inference and Maximum Entropy Methods in Science and Engineering, AIP (Melville), 83.
Grendar, M., Niven, R.K. (in submission), http://arxiv.org/abs/cond-mat/0612697. Jaynes, E.T. (1957), Physical Review, 106: 620-630. Jaynes, E.T. (Bretthorst, G.L., ed.) (2003) Probability Theory: The Logic of Science, Cambridge
U.P., Cambridge. Körner, J., Longo, G. (1973) IEEE Trans. Information Theory IT-19(6): 778. Körner, J., Orlitsky, A., (1998) IEEE Trans. Information Theory 44(6) 2207. Kullback, S., Leibler, R.A. (1951), Annals Math. Stat., 22: 79-86.
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 58
Lilly, S. (2002), A Practical Guide to Runes, Caxton Editions, London. Niven, R.K. (2005), Physics Letters A, 342(4): 286-293. Niven, R.K. (2006), Physica A, 365(1): 142-149. Niven, R.K. (in submission) CTNEXT07, 1-5 July 2007, Catania, Sicily, Italy, http://arxiv.org/
abs/0709.3124. Niven, R.K. (2005-07) Combinatorial information theory: I. Philosophical basis of cross-entropy
and entropy, cond-mat/0512017. Niven, R.K., Suyari, H. (in submission) Combinatorial basis and finite forms of the Tsallis entropy
function. Pascal, B. (1654), Traité du Triangle Arithmétique, Paris. Paxson, D.L. (2005) Taking Up the Runes, Red Wheel/Weiser, York Beach, ME, USA. Pennick, N. (2003) The Complete Illustrated Guide to Runes, HarperCollins, London. Planck, M. (1901) Annalen der Physik 4: 553. Sanov, I.N. (1957) Mat. Sb. 42, 11-44; English transl. Selected Transl. Math. Stat. Prob. 1 (1961),
213-224. Shannon, C.E. (1948), Bell System Technical Journal, 27: 379-423; 623-659. Suyari, H. (2006), Physica A 368(1): 63. Vincze, I, (1974) Progress in Statistics, 2: 869-895. Historical references prior to 1800AD are given in Edwards (2002).
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 59
Appendix 1: Runic alphabet: (Refs: Lilly, 2002; Pennick, 2003; Paxson, 2005; Wikipedia)
fuTarkgw hnijIpzs tbemlNod ...
f.u.th.a.r.k.g.w h.n.i.j.eo.p.z.s t.b.e.m.l.ng.o.d
- used across Germanic + central Europe, Britain + Scandinavia, 5th-10th cent.; in Sweden to 17th cent.
- derived from Etruscan alphabet (not Greek or Roman) - each rune has symbolic meaning Anglo-Saxon h (“Haegl”) = old German h (“Hagalaz”) = hail, hailstones - symbolic of destructive force of Nature, but melts and gives
new life - evokes need to accept what is inevitable; to “go with the flow”;
i.e. rune of transformation
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 60