Information modification and particle collisions in distributed computationJoseph T. Lizier,1, 2, a) Mikhail Prokopenko,1 and Albert Y. Zomaya21)CSIRO Information and Communications Technology Centre, PO Box 76, Epping, NSW 1710,Australia2)School of Information Technologies, The University of Sydney, NSW 2006,Australia

(Dated: 14 September 2010)

Distributed computation can be described in terms of the fundamental operations of information storage,transfer and modification. To describe the dynamics of information in computation, we need to quantifythese operations on a local scale in space and time. In this paper we extend previous work regarding thelocal quantification of information storage and transfer, to explore how information modification can bequantified at each spatiotemporal point in a system. We introduce the separable information, a measure whichlocally identifies information modification events where separate inspection of the sources to a computation ismisleading about its outcome. We apply this measure to cellular automata, where it is shown to be the firstdirect quantitative measure to provide evidence for the long-held conjecture that collisions between emergentparticles therein are the dominant information modification events.

PACS numbers: 89.75.Fb, 89.75.Kd, 89.70.Cf, 05.65.+bKeywords: information modification, information processing, distributed computation, intrinsic computation,cellular automata, complex systems, self-organization, information theory, particles, particle collisions

The nature of distributed computation has longbeen a topic of interest in complex systems sci-ence, physics, artificial life, bio- and neuroinfor-matics. In all of these relevant fields, distributedcomputation is often discussed in terms of mem-ory, communication, and processing. Distributedcomputation is any intrinsic or designed pro-cess conducted by multiple inter-connected enti-ties (e.g. cells, neurons, software agents, trans-port hubs, network nodes, sensors, etc.) thatinvolves these operations on information: infor-mation storage, transfer and modification. Cel-lular automata (CAs) are a well-studied exam-ple, where these operations are described in termsof emergent coherent structures known as parti-cles. General interpretation holds that stationaryparticles implement information storage, mov-ing particles implement information transfer, andparticle collisions implement information modifi-cation. We seek a complete framework to quan-tify the qualitative understanding of these oper-ations. Specifically, we seek to quantify each ofthese operations or information dynamics of dis-tributed computation on a local scale in space andtime. We have previously addressed informationstorage and transfer, quantitatively confirmingtheir embodiment in stationary and moving par-ticles in CAs. In this paper, we explore how in-formation modification can be quantified. We in-troduce a new measure for this purpose, the sepa-rable information, and demonstrate how it iden-tifies particle collisions in CAs as the dominant

a)jlizier@it.usyd.edu.au

information modification events therein. This re-sult is important not only in providing the firstdirect quantitative support for this long-held con-jecture about distributed computation, but alsoin being the first technique which can separatelyfilter particle collisions.

I. INTRODUCTION

Information modification is a crucial part of dis-tributed computation, whether that computation isspecifically designed or simply intrinsic1. It has beenviewed as a particularly important operation for biolog-ical neural networks and models thereof2–5, where it hasbeen suggested as a potential biological driver3. It is alsoa key operation in collision-based computing6,7, includ-ing in soliton dynamics and collisions8.

Our perspective is that it is important as one of threefundamental operations of intrinsic distributed compu-tation in complex systems: information storage, trans-fer and modification. We seek a complete frameworkto quantify these fundamental operations of distributedcomputation. In particular, we seek to quantify these in-formation dynamics on a local scale in space and timein a system. In previous work we have described howinformation storage9 and transfer10 can be quantifiedinformation-theoretically in this fashion. Here, we seek toestablish how information modification can be quantifiedalongside these related measures within this framework.

Information modification is often colloquially de-scribed as the processing of information into a new form.More specifically, it has been interpreted to mean inter-actions between transmitted and/or stored informationwhich result in a modification of one or the other11. We

2

accept this interpretation in our perspective of distri-bution computation, as it specifically juxtaposes mod-ification against storage and transfer, viewing it as adynamic combination or synthesis of information fromdifferent sources. Modification therefore involves a non-trivial processing of information rather than a trivial re-trieval, movement or translation of one source of infor-mation alone.

The term has however remained elusive to appropri-ate quantitative definition, despite several attempts2,4,5.A major issue is that these attempts typically focus onmeasuring trivial processing as the movement or interpre-tation of information rather than specifically the modi-fication or non-trivial processing of information synthe-sized from several sources. Furthermore, some2,5 havebeen too specific to allow portability across system types(e.g. by focusing on the capability of a system to solvea known problem, or measuring properties related to theparticular type of system being examined). Also, theyare not amenable to measuring the dynamics of infor-mation modification at local space-time points within adistributed system. This is a crucial requirement of sucha measure, since only a local perspective can describewhen, where and how modification occurs.

Despite the current lack of a quantitative measure,the qualitative notion of information modification in dis-tributed computation is well-understood in cellular au-tomata (CAs), a popular model of distributed computa-tion. As such, to derive quantitative insights here we willfocus on CAs. CAs are discrete dynamical lattice systemsinvolving an array of cells which synchronously updatetheir states as a homogeneous deterministic function (orrule) of the states of their local neighbors12. Here wewill use Elementary CAs (ECAs), which consist of a one-dimensional array of cells with binary states, with eachupdated as a function of the previous states of themselvesand one neighbor either side (i.e. neighborhood size 3 orrange r = 1). CAs are widely used to study complex com-putation, since certain rules (e.g. ECA rules 110 and 54,defined using the Wolfram 12 numbering scheme) exhibitemergent coherent structures which are not discerniblefrom their microscopic update functions but which pro-vide the basis for understanding the macroscopic com-putations being carried out13. These coherent structuresagainst a background domain region are known as parti-cles. Regular or periodic particles are known as gliders,while stationary gliders are known as blinkers. Particlesare also often referred to as domain walls, since they arediscontinuities in the background domain pattern.

There are several long-held conjectures regarding therole of these emergent structures in the intrinsic dis-tributed computation in CAs; i.e. how the cells pro-cess information in order to determine their collectivefuture state. Blinkers are generally held to be the dom-inant information storage elements, since local patternmaintenance is an information storage process. In con-trast, particles are held to be the dominant informationtransfer agents, since they communicate coherent infor-

mation about the dynamics in one area of the system toanother11,13.

Most importantly for our work here, in the context ofCAs interactions between transmitted and/or stored in-formation are generally interpreted to mean collisions ofparticles (including blinkers), with the resulting dynam-ics involving something other than the incoming particlescontinuing unperturbed. The resulting dynamics couldinvolve zero or more particles (with an annihilation leav-ing only a background domain), perhaps including someof the incoming particles14. Given our earlier interpreta-tion of modification as involving the synthesis of severalsources resulting in changes to their information, it isplausible to associate a collision event with the modifi-cation of transmitted and/or stored information, and tosee it as a non-trivial information processing event. In-deed, as an information processing event the importantrole of particle collisions in determining future dynam-ics is widely acknowledged for CAs11,15–18, e.g. in thedensity classification task with rule φpar

19,20. In short:particles provide a short-hand description of the dynam-ics in a CA; collisions change that description, modifyingthe information in the dynamics19. The interpretation isparalleled in studies of collision-based computing6–8.

Identifying particle collisions in CAs is a simple enoughtask: there are several measures which can quantitativelyidentify particles10,21–24, and their collision points canthen be identified be eye. Yet such an approach doesnot suffice as the identification of information modifica-tion. First, it provides no direct quantitative basis forunderstanding information modification itself. Further,it is not capable of identifying information modificationevents if and where they are manifested beyond particlecollisions.

The intersection of the aforementioned interpretationsand approaches presents an excellent opportunity toquantify information modification on a local scale inspace and time. The foundation measures in our frame-work of information storage and transfer have been estab-lished, and been demonstrated to provide the first quanti-tative evidence for the conjectures that blinkers are dom-inant information storage entities and (moving) parti-cles are dominant information transfer agents in CAs9,10.This, coupled with the well-understood qualitative no-tion of information modification as particle collisions inCAs, means that CAs are an ideal experimental tool here.Furthermore, our foundation measures are information-theoretic (meaning they can be generically applied to thedynamics of other systems) and quantify the dynamics ona local scale in space and time (meaning they can mea-sure the information properties of specific spatiotemporalstructures). In building on these foundation measures,our quantification of information modification will alsohave these desirable properties.

We begin by introducing the foundation measures inour framework for distributed computation in Section II,and describing what they reveal about information stor-age and transfer on a local scale in CAs. We then derive

3

the separable information in Section III as a tool to de-tect information modification events, where separate in-spection of information sources is misinformative aboutthe next state of a destination. We hypothesize that thismeasure should identify particle collisions in CAs as thedominant information modification events, and demon-strate this to be the case in Section IV. The separableinformation is thus shown to be the first measure to pro-vide direct quantitative evidence that particle collisionsin CAs are the dominant information modification eventstherein. We also use the measure to reveal a number ofother interesting instances of information modification inCA dynamics beyond particle collisions, and we classifythe types of modification events using their local infor-mation dynamics properties. Finally, we identify appro-priate parameter settings for the measure.

II. FOUNDATION MEASURES OF INFORMATIONDYNAMICS

We seek a framework to quantify the local informationdynamics of distributed computation. The two founda-tion components of this framework, information storageand transfer, have previously been addressed9,10. There,we have shown how these fundamental operations ofdistributed computation can be quantified at each spa-tiotemporal point in a complex system. Built on infor-mation theory25, these measures provide a common, non-linear, application-independent language in which to an-alyze and design complex computation.

A. Information storage

Information storage refers to the amount of informa-tion in the past of a variable that is relevant to predictingits future. The active information storage measures thestored information that is currently in use in computingthe next state of a variable9. Specifically, the local activeinformation storage for variable X is the local (or un-averaged) mutual information between its semi-infinite

past x(k)n = {xn−k+1, . . . , xn−1, xn} (as k → ∞) and its

next state xn+1 at time step n+ 1:

aX(n+ 1) = limk→∞

log2

p(xn+1 | x(k)n )

p(xn+1). (1)

Finite-k estimates are represented as aX(n, k). The ac-tive information storage is the average over time: AX =〈aX(n)〉n; and both average and local measures are givenin bits. The local active information storage aX(n + 1)is the stored information that is currently in use by vari-able X in computing its next state xn+1 at time n + 1.This contrasts with the excess entropy26, which mea-sures the average total information stored by the vari-able (which will be in use either at the next state or ata later time); the active information storage, measuring

only that which is currently in use, represents a portionof or lower bound to the excess entropy9. We focus on theactive information here since it yields an immediate con-trast in the relative contributions of storage and transferto each computation of the next state of a variable (seeAppendix A), and it is this computation we need to focuson in examining information modification.

We can generalize the measure for Xi in a lattice sys-tem with spatially-ordered variables (such as a CA) as:

a(i, n+ 1) = limk→∞

log2

p(xi,n+1 | x(k)i,n)

p(xi,n+1), (2)

and use a(i, n + 1, k) to denote finite-k estimates there.We note that a(i, n, k) gives a spatiotemporal profileacross the lattice system (i, n). We also note that inhomogeneous lattice systems, the average value A =〈a(i, n, k)〉i,n is taken across both space and time (i.e.

over the local values at each spatiotemporal point).The average value AX will always be positive, but is

limited by the average entropy HX (i.e. the amount ofinformation in the next state xn+1). The local valuesaX(n + 1) are not bound in this manner however, withlarger values indicating that the particular past of a vari-able provides strong positive information about its nextstate. Furthermore, we can have aX(n + 1) < 0, wherethe past history of the variable is actually misinforma-tive about its next state. An observer is misinformedwhere the probability of observing the given next state

in the context of the past history, p(xn+1 | x(k)n ), is lowerthan the probability p(xn+1) of observing that next statewithout considering the past.

The dynamics of the local active information storagehave been studied in CAs9. Here, we discuss the re-sults of that study9 with respect to sample profiles ofa(i, n, k = 16) for ECA rules 54, 110 and 18 in Fig. 1(b),Fig. 3(b) and Fig. 2(b). Rule 54 and 110, consideredcomplex, contain traveling and stationary gliders. Rule18 contains domain walls against a seemingly chaoticbackground domain. The profiles here were generatedusing CA runs of 10 000 cells with periodic boundaryconditions, initialized from random states, with 600 timesteps retained (after 30 initial time steps). The profilesare generated by calculating a(i, n+ 1) (from Eq. (2)) atevery spatiotemporal site (i, n + 1). For each CA, this

is done by evaluating p(xi,n+1 | x(k)i,n) and p(xi,n+1) for

the actual xi,n+1 and x(k)i,n at the site (i, n+ 1), with the

probability distribution functions (PDFs) defined usingcounts across all spatiotemporal points (i, n + 1) in thishomogeneous system. Note that even though the systemis deterministic, the PDFs are non-trivial since they donot take the whole neighborhood into account. Throughthe large number of cells we attempt reduce dependencyof the results on initial conditions.

Considering rule 54 first, the positive values ofa(i, n, k = 16) are concentrated in the domain areas,as well as in the blinkers (referred to as α and β

4

previously17), due to strong temporal periodicity in bothareas. We additionally showed9 that while the blinkersand domain areas were currently using similar amountsof information storage, the blinkers had larger total in-formation storage (i.e. their local excess entropy waslarger). These results confirmed the conjecture thatblinkers, followed by the regular domain regions, are thedominant information storage entities in CAs. Similarresults are shown for rule 110, and in analogy positivevalues of a(i, n, k = 16) for rule 18 were found where itsdomain walls were stationary.

Furthermore, there is quite interesting informationstorage structure in the seemingly irregular backgrounddomain of rule 18. In contrast to rule 54, the back-ground domain for rule 18 contains both positive andnegative values of a(i, n, k = 16). Considering these com-ponents together, a pattern of spatial and temporal pe-riod 2 emerges, corresponding to the period-2 ε-machinegenerated to recognize the background domain for rule18 by Hanson and Crutchfield 21 . Every second site is a“0”, and contains a small positive a(i, n, k = 16); thisinformation storage indicates that we have reached thisprimary temporal phase of the period and is sufficientto predict the next state here. The alternate site is ei-ther a “0” or a “1”, and contains either a small nega-tive a(i, n, k = 16) at the “0” sites or a larger positivea(i, n, k = 16) at the “1” sites. The stored informationfrom the past of the cell (which indicates that it is cur-rently in the alternate temporal phase) is strongly in useor active in computing the “1” sites, since the “1” sitesonly occur in the alternate phase. In contrast however,this stored information (that the cell is in the alternatetemporal phase) is misinformative with regard to the “0”sites, since they occur more frequently with the primaryphase. Indeed, encountering a “0” at the alternate sitescreates ambiguity in the future (since it makes determi-nation of the phase more difficult) so in this sense it canbe seen as detracting from the overall storage.

Perhaps most importantly for this work, the only neg-ative values of a(i, n, k = 16) for rule 54 are concen-trated in the traveling gliders (labeled as γ+ and γ−

previously17), while the strongest negative values for rule18 are where the domain walls are moving. In both cases,when a traveling particle is encountered at a given cell,the past history of that cell (being part of the backgroundregular domain) becomes misinformative about the nextstate of the cell. This is because the domain sequence wasmore likely to continue than be interrupted. In fact, thedynamics at these points are due to information transfer,which we explore in the next section.

B. Information transfer

Information transfer is formulated by the transfer en-tropy27 (TE) as the information provided by a sourceabout a destination’s next state that was not containedin the past of the destination. Specifically, the local trans-

fer entropy10 from a source Y to a destination X is thelocal mutual information between the previous state ofthe source yn and the next state of the destination xn+1,conditioned on the semi-infinite past of the destination

x(k)n (as k →∞):

tY→X(n+ 1) = limk→∞

log2

p(xn+1 | x(k)n , yn)

p(xn+1 | x(k)n ). (3)

Again, the transfer entropy is the (time) averageTY→X = 〈tY→X(n)〉, while finite-k estimates are rep-resented as tY→X(n + 1, k). The TE can be measuredfor any two time series, but only represents informationtransfer when measured on a causal link28 and in thelimit k → ∞.10 Importantly, the TE properly measuresa directed, dynamic flow of information, unlike previousinferences with the mutual information which measurecorrelations only. Also, in measuring the contribution ofthe source in the context of the past of the destination, itspecifically juxtaposes the information transfer from thatsource against the storage of the destination. That is tosay, conditioning on the past of the destination removesany stored information from being considered as transfer.(See further discussion in Appendix A.)

For lattice systems, we can represent the local transferentropy from Xi−j to Xi (i.e. across j cells to the right)at time n+ 1 as:

t(i, j, n+ 1) = limk→∞

log2

p(xi,n+1 | x(k)i,n , xi−j,n)

p(xi,n+1 | x(k)i,n). (4)

We use t(i, j, n + 1, k) to denote finite-k estimates here.We note that t(i, j, n+1, k) is defined for every spatiotem-poral information destination (i, n), forming a spatiotem-poral profile for every information channel or directionj. As above, the TE is only interpretable as informa-tion transfer where j corresponds to causal informationsources, i.e. for CAs sources within the cell range |j| ≤ r.

In a similar fashion to AX , the average TE TY→X isalways positive but limited by the average entropy HX .And again, the local transfer entropy t(i, j, n, k) may beeither positive or negative. Negative values occur where(given the destination’s history) the source is actuallymisinformative about the next state of the destination.Specifically, an observer is misinformed by the source

where the probability p(xi,n+1 | x(k)i,n , xi−j,n) of observingthe actual next state given the past history and source

value, is lower than the probability p(xi,n+1 | x(k)i,n) ofobserving the actual next state given the past historyalone.

As shown in Appendix A, we refer to the above formu-lation of the TE as the apparent transfer entropy10, incontrast with other formulations that condition on addi-tional information sources. We also demonstrate in Ap-pendix A that the total information in a destination (thelocal entropy) can be decomposed as a sum of the activeinformation storage and (iteratively-conditioned) trans-fer entropy terms from each causal source.

5

The dynamics of the local transfer entropy have beenstudied in CAs previously10. Here, we summarize therelevant results of that study with respect to sample pro-files of t(i, j, n, k = 16) for rule 54 (with the j = 1 profilein Fig. 1(c), and j = −1 profile in Fig. 1(d)) and rule 18(with the j = 1 profile in Fig. 2(c), and j = −1 profilein Fig. 2(d)). The results for rule 110 (in Fig. 3(c) andFig. 3(d)) mirror those for rule 54.

The primary result of the application is that the lo-cal apparent TE highlights particles (both gliders anddomain walls) as strong positive information transferagainst background domains. Crucially, the particlesare measured as information transfer in their directionof macroscopic motion, as expected. In rule 54 for exam-ple, the right-moving γ+ gliders are shown in Fig. 1(c)to carry strong positive information transfer in the j = 1(or 1 step to the right per time step) channel. Similarly,in rule 18 we see that each profile t(i, j = 1, n, k = 16)and t(i, j = −1, n, k = 16) measures strong positive in-formation transfer when the domain wall is moving inthe direction of the information channel they are mea-suring. These results provided quantitative evidence forthe long-held conjecture that particles were the dominantinformation transfer agents in CAs.

We have also demonstrated10 that there is zero in-formation transfer in an infinite periodic domain: theirstates are completely predicable from their pasts, mean-ing that they only involve information storage. In con-trast, there is a small but non-zero information transferin finite domains acting as a background to gliders. Thisambient transfer certainly exists for rule 54, though thevalues are too small to appear in Fig. 1(c) and Fig. 1(d).The ambient transfer is not a finite-k effect, but exists be-cause when a periodic domain is occasionally punctuatedby gliders, the next state of a cell is not completely pre-dictable from its periodic history. There is scope for theneighboring sources to add information about the nextstate of that destination, effectively indicating whethera glider is incoming or is absent. Such ambient trans-fers are stronger in the wake of a glider, indicating theabsence of (relatively more common) following gliders.

The dynamics in the background domain of rule 18 areslightly different. Certainly the primary phase (where ev-ery second site is a “0”) exhibits a dominant informationstorage operation similar to periodic domain sites. Theinformation transfer to these sites approaches zero, butbecause of the possibility of a domain wall encounter herewe again have some ambient transfer. In contrast, wherethe past history indicates a site in the alternate phase,a “0” or a “1” are roughly equally likely, and there ismuch scope or capacity for the sources to add informa-tion here. The apparent TE also approaches zero herehowever, because these sites are determined by an XORoperation between both transfer sources29. As describedin Appendix A, the apparent TE cannot measure transferdue to interactions, such as that in XOR operations.

Also interesting are the results when the apparent TEis measured in the orthogonal direction to particles, e.g.

see the t(i, j = 1, n, k = 16) profile in Fig. 1(c) at theγ− gliders. At some points here, a positive informationtransfer is measured. This occurs for a similar reason tothe ambient transfer in the domain, simply: the sourcedoes add positive information about the next state of thedestination. At other points, the apparent TE measuresnegative values. In general this is because the source, aspart of the domain, suggests that this same domain foundin the past of the destination is likely to continue; how-ever since the next state of the destination forms part ofthe particle, this suggestion proves to be misinformative.

III. SEPARABLE INFORMATION AS A DETECTORFOR INFORMATION MODIFICATION

We begin our investigation of the quantitative natureof information modification by reviewing our qualitativeunderstanding of what it means for a particle in a CAto be modified. For the simple case of a regular glider,a modification is simply an alteration to the predictableperiodic pattern of the glider’s dynamics. At such points,an observer would be surprised or misinformed aboutthe next state of the perturbed glider, having not takenaccount of the entity about to perturb it.

This interpretation is a clear reminder of our earliercomments that local active information storage was mis-informative at moving gliders (Section II A), and localapparent transfer entropy was misinformative at gliderstraveling in the orthogonal direction to the measurement(Section II B). For these unperturbed gliders, one ex-pects the local apparent transfer entropy measured inthe direction of motion to be more informative aboutits continuation than any misinformation conveyed fromother sources (see Fig. 4(a)). However, where the glideris modified by a collision with another glider, we canno longer expect the local apparent transfer entropy inits macroscopic direction of motion to remain informa-tive about its evolution (see Fig. 4(b)). Assuming thatthe incident glider is also perturbed, the local apparenttransfer entropy in its macroscopic direction of motionwill also not be informative about the dynamics at thiscollision point. We expect the same argument to be truefor domain walls and their collisions.

As such, we make the hypothesis that at the spa-tiotemporal location of a local information modificationevent or collision, separate inspection of each informationsource will misinform an observer overall about the nextstate of the modified information destination.

To be specific, the information sources referred to hereare the past history of the destination (via the local activeinformation storage from Section II A) and each othercausal information contributor: these are examined inthe context of the past history of the destination, viatheir local apparent transfer entropies from Section II B.

We have shown in Appendix A how these sources pro-vide the total information for the computation of the nextstate of the destination. Crucially, this total information

6

(a)Raw CA (b)a(i, n, k = 16) (c)t(i, j = 1, n, k = 16)

(d)t(i, j = −1, n, k = 16) (e)s(i, n, k = 16) (f)Close up of raw CA for collision “A”

FIG. 1. (color online) Local information dynamics in rule 54 (35 time steps displayed for 35 cells, time increases down the pagefor all CA plots). (a) Raw states, with “0” in white and “1” in black. All other profiles are discretised into 16 levels, with bluefor positive values and red for negative. (b) Local active information storage, max. 1.07 bits, min. -12.27 bits. Local apparenttransfer entropy: (c) one cell to the right, max. 7.93 bits, min. -4.04 bits, (d) one cell to the left, max. 7.93 bits, min. -4.21bits. (e) Local separable information, max. 8.40 bits, min. -5.27 bits. (f) Close up of raw states of rule 54 in collision type “A”.“x” and “+” mark some positions in the γ+ and γ− gliders respectively. Note their point of coincidence in collision type “A”,with “∗” marking what initially appears to be the collision point and “o” marking the subsequent information modification asdetected using s(i, n, k = 16) < 0.

is only obtained from a unified inspection of all causalsources which accounts for all source interactions (seeEq. (A22))30. In contrast, we quantify the total informa-tion gained from separate observation of the informationstorage and transfer contributors as the local separableinformation sX(n):

sX(n) = limk→∞

sX(n, k), (5)

sX(n, k) = aX(n, k) +∑

Y ∈VX\X

tY→X(n, k). (6)

Note that the transfer entropy terms are summed over allsources Y in the set of causal information sources VX toX (except for X itself). We use sX(n, k) for finite-k es-timates, though in practice recommend that as large a kas possible is used. This is because the measure relies on

the correctness of aX(n, k) and the tY→X(n, k), whichhave this requirement as previously shown9,10. Indeed,the true separation of elements of information storageand transfer (facilitated by k →∞ providing the contextof the past) is critical to our consideration of informationmodification from the perspective of distributed compu-tation here31.

For CAs, where the causal information contributors arehomogeneously within the neighborhood r, we write thelocal separable information in lattice notation as:

s(i, n) = limk→∞

s(i, n, k), (7)

s(i, n, k) = a(i, n, k) +

+r∑j=−r,j 6=0

t(i, j, n, k). (8)

7

(a)Raw CA (b)a(i, n, k = 16) (c)t(i, j = 1, n, k = 16)

(d)t(i, j = −1, n, k = 16) (e)s(i, n, k = 16)

FIG. 2. (color online) Local information dynamics in rule 18 (67 time steps displayed for 67 cells). (a) Raw states, with “0”in white and “1” in black. All other profiles are discretised into 16 levels, with blue for positive values and red for negative.(b) Local active information storage, max. 1.98 bits, min. -9.92 bits. Local apparent transfer entropy: (c) one cell to the right,max. 11.90 bits, min. -7.44 bits, (d) one cell to the left, max. 11.90 bits, min. -7.30 bits. (e) Local separable information, max.1.98 bits, min. -14.37 bits.

We show s(i, n, k) diagrammatically in Fig. 5.

As inferred in our hypothesis, we expect the local sepa-rable information to be positive or highly separable whereseparate observations of the information contributors areinformative overall regarding the next state of the des-tination. We define this situation with sX(n, k) > 0 tobe trivial information processing, because an observer ispositively informed about the outcome even without ac-counting for any interactions between the sources. Assuch, information storage and transfer are not interact-ing in any significant manner. For example, a periodicprocess executes information storage alone, trivially up-dating its state using its past history as a sole informationsource.

More importantly, we expect the local separable infor-mation to be negative or non-separable at spatiotemporalpoints where an information modification event or colli-sion takes place. Here, separate observations are mislead-ing overall because the outcome is largely determined by

the interaction of the information sources. We define thissituation with sX(n, k) < 0 to be an information modi-fication event or equivalently non-trivial informationprocessing, understanding this as the interaction be-tween information storage and transfer. For example,a particle collision involves non-trivial information pro-cessing because an observer needs to collectively examinemultiple sources and their interaction in order to be pos-itively informed about the next state of the destination.Importantly though, this argument has distilled the keydynamical features of information during a modificationevent, and the technique may identify additional modifi-cation events beyond particle collisions.

Interestingly also, this formulation of informationmodification echoes descriptions of complex systems asconsisting of (a large number of) elements interactingin a non-trivial fashion32, and of emergence as where“the whole is greater than the sum of its parts”33. Here,we quantify the sum of the parts in sX(n), whereas

8

(a)Raw CA (b)a(i, n, k = 16) (c)t(i, j = 1, n, k = 16)

(d)t(i, j = −1, n, k = 16) (e)s(i, n, k = 16)

FIG. 3. (color online) Local information dynamics in rule 110 (55 time steps displayed for 55 cells). (a) Raw states, with “0”in white and “1” in black. All other profiles are discretised into 16 levels, with blue for positive values and red for negative.(b) Local active information storage, max. 1.22 bits, min. -9.21 bits. Local apparent transfer entropy: (c) one cell to the right,max. 9.99 bits, min. -5.56 bits, (d) one cell to the left, max. 10.43 bits, min. -6.01 bits. (e) Local separable information, max.5.47 bits, min. -5.20 bits.

“the whole” refers to examining all information sourcestogether. The whole is greater where all informationsources must be examined together in order to receivepositive information on the next state of the examinedentity. We emphasize that there is no quantity repre-senting “the whole” as such, simply the indication thatthe sources must be examined together. We also empha-size that sX(n) is not the total information an observerneeds to predict the state of the destination: this is mea-sured by the single-site entropy hX(n) (see Appendix A).Instead, sX(n) is the total obtained by inspecting thesources separately, ignoring any interaction or redundan-cies (and so could be larger or smaller than hX(n)).

The average separable information can certainly be de-

fined also:

SX = 〈sX(n)〉n , (9)

SX(k) = 〈sX(n, k)〉n , (10)

SX = limk→∞

SX(k), (11)

though the quantity is only truly understood it terms ofwhat its local values imply. Similarly, for lattice systemswe have:

S(i) = 〈s(i, n)〉n , (12)

S(i, k) = 〈s(i, n, k)〉n , (13)

S(i) = limk→∞

S(i, k). (14)

For homogeneous agents we have S(k) = 〈s(i, n, k)〉i,n,and define S in the limit k →∞.

9

(a)Unperturbed glider (b)Glider collision

FIG. 4. Our expectations for the local information dynamics of storage and transfer for unperturbed gliders (coherentstructures) and glider collisions. (a) For unperturbed gliders in channel j = 1, we expect the transfer t(i, j = 1, n, k) in thedirection of glider motion to be positively informative, and indeed more informative than the misinformation conveyed througha and t(i, j = −1, n, k). (b) For a collision perturbing a glider moving in the channel j = 1, we can no longer expect thetransfer t(i, j = 1, n, k) in the direction of glider motion to be positively informative at the collision point. We cannot expectthe transfer t(i, j = −1, n, k) in the direction of the incident glider to be positively informative at the collision point either.

FIG. 5. Separable information for 1D CAs s(i, n+1, k): infor-mation gained about the next state of the destination xi,n+1

from separately examining each causal information source in

the context of the destination’s past x(k)i,n . For CAs these

causal sources are within the cell range r.

IV. LOCAL INFORMATION MODIFICATION INCELLULAR AUTOMATA

We applied the local separable information with k = 16to study information modification in our same sampleECA runs of rules 54 (Fig. 1(e)), 18 (Fig. 2(e)) and 110

(Fig. 3(e))34.

The key results from this application are discussedhere:

• Negative values of local separable information pro-vide the first direct quantitative identification ofhard collisions between particles as dominant infor-mation modification events in Section IV A. This isthe case for both regular gliders and domain walls,and we observe a short time delay between theapparent collisions and the identified informationmodification points.

• The separable information identifies informationmodification events occurring separately from par-ticle collisions, for example in soft collisions be-tween gliders and background domains (SectionIV B), storage modifications in non-periodic back-ground domains (Section IV C), as well as through-out the chaotic dynamics of rule 22 (Section IV D).After identification of these events using s(i, n, k) <0, we use the underlying local information dynam-ics values to perform a secondary classification ofthe event type (summarized in Table I).

• Appropriately large values of past history k arerequired to provide the perspective of distributedcomputation and identify modification points.

10

A. Hard particle collisions as dominant modificationevents

The simple gliders in ECA rule 54 give rise to rela-tively simple collisions which we focus on in our discus-sion of s(i, n, k = 16) here (see Fig. 1(e)). Notice thatthe positive values of s(i, n, k = 16) are concentrated inthe domain regions and at the stationary gliders (α andβ). As expected, these regions are undertaking trivialcomputations only. More importantly, the negative val-ues of s(i, n, k = 16) are also shown in Fig. 1(e), withtheir positions circled there. The dominant negative val-ues are clearly concentrated around the areas of parti-cle collisions, including collisions between the travelinggliders only (marked by “A”) and between the travelinggliders and the stationary gliders (marked by “B”, “C”and “D”). This clearly confirms our hypothesis thatglider collisions have negative separable informa-tion and upholds our argument that this is a statisticfor inferring information modification events. We termthese events hard collisions because they are collisionsbetween explicit emergent structures.

As a detailed example, collision “A” involves the γ+

and γ− particles interacting to produce a β particle(γ+ + γ− → β17). The only information modificationpoint highlighted is one time step below that at whichthe gliders initially appear to collide – these points aremarked “o” and “∗” respectively in the close-up of rawstates in Fig. 1(f). The periodic pattern from the past ofthe destination breaks at “∗”, however the neighboringsources are still able to support separate prediction of thestate, i.e.: a(i, n, k = 16) = −1.09 bits, t(i, j = 1, n, k =16) = 2.02 bits and t(i, j = −1, n, k = 16) = 2.02 bits,giving s(i, n, k = 16) = 2.95 bits. This is no longer thecase however at “o” where our measure has identified themodification point; there we have a(i, n, k = 16) = −3.00bits, t(i, j = 1, n, k = 16) = 0.91 bits and t(i, j =−1, n, k = 16) = 0.90 bits, with s(i, n, k = 16) = −1.19bits suggesting an information modification event or non-trivial information processing.

A delay is also observed before the identified informa-tion modification points of collision types “B” (γ+ +β →γ−, or vice-versa in γ-types), “C” (γ−+α→ γ−+α+2γ+,or vice-versa) and “D” (2γ+ + α + 2γ− → α). Possiblythese delays represent a time-lag of information process-ing. Not surprisingly, the results for these other collisiontypes imply that the information modification points areassociated with the creation of new behavior : in “B” and“C” these occur along the newly created γ gliders, andfor “C” and “D” in the new α blinkers.

We observe similar results for ECA rule 110 inFig. 3(e). This rule contains more complex glider struc-tures and collisions, with information modification pointseven more delayed from the initiation of the collision. Wealso observe a collision (not shown) where an incidentglider is absorbed by a blinker without any modificationto the absorbing blinker (because the information stor-age for the absorbing blinker is sufficient to predict the

dynamics at this interaction).Furthermore, as displayed in Fig. 2(e), the separa-

ble information quite clearly identifies the hardcollision between the domain walls as dominantinformation modification events for rule 18. The ini-tial information modification event is clearly where onewould initially identify the collision point, yet it is fol-lowed by two secondary information modification pointsseparated by two time steps. At the raw states of thesethree collision points in Fig. 2(a), the outer domains haveeffectively coalesced (inferred by spatial scanning). Thesource in say the left outer domain (in the context ofthe past in the inner domain) indicates that the domainwalls should intrude into the inner domain; the domainwall is not observed though because the outer domainshave coalesced, and this is misinformative. The same ap-plies for the source in the right domain, so the separableinformation is negative at these points due to the transfersources. Indeed an observer following the computationalperspective and scanning the temporal pattern cannotbe certain that the outer domains have taken hold atthis particular cell until observing a “1” at the alternatephase (see discussion of the two phases of the domain inSection II A). As such, these information modificationevents continue to be observed until a “1” confirms theouter domains have joined35. This in some ways paral-lels the observation of delays in information processingobserved earlier.

Importantly, this result provides evidence that colli-sions of irregular particles are information modificationevents, as expected. It is also worth noting that thesecollisions always result in the destruction of the domainwalls (and the inner domain), indicating that our methodcaptures destruction-type modification events as well ascreation. (This is also true for the γ+ + γ− + β → �event in rule 54, not shown).

Interestingly also, note that the glider collisions in rules54 and 110 always occurred with t(i, j, n, k) > 0 for atleast one j = ±1. In contrast, the domain wall collisionsin rule 18 have a different basis for modification becausewe have t(i, j, n, k) < 0 for both j while the a(i, n, k)remains positive.

While particle collisions are the dominant informationmodification events, they are not the only such eventsidentified by the separable information. In the next sec-tions we discuss the manifestation of information modifi-cation in gliders, non-periodic background domains, andtheir proliferation in chaotic dynamics. Their identifica-tion here is important, because it would not be possiblehad we used an approach focussing directly on particlecollisions.

B. Soft collisions between gliders and the domain

Interestingly, weak information modification pointscontinue to be identified at every second point along allthe γ+ and γ− particles in rule 54 after the initial colli-

11

sions. These are generally too weak to appear in Fig. 1(e)but can be seen for a similar glider in rule 110 in Fig. 3(e).This was unexpected from our earlier hypothesis. How-ever, these events can be understood as non-trivial com-putations of the continuation of the glider in the absenceof a collision. From another perspective, they are softcollisions of the glider with the periodic struc-tures and ambient transfer in the domain. Recallthat ambient transfer refers to the small but non-zeroinformation transfer in periodic domains indicating theabsence of gliders (see Section II B). The term soft col-lisions indicates the qualitative contrast with hard col-lisions between particles, and that the gliders continueunperturbed. These soft collision events are more sig-nificant closer to the hard collisions, since the ambienttransfer is stronger in the wake of the gliders that causedthese collisions (see Section II B).

Also, note that these soft collision events occur witha(i, n, k) < 0 (since the domain is misinformative at glid-ers) and at least one t(i, j, n, k) > 0 (since the glidercontains strong transfer). Importantly also, some softcollision events (e.g. in rule 110) occur with a larger mag-nitude s(i, n, k) < 0 than for some hard collision events.These facts together mean that hard and soft collisionsare not differentiable using s(i, n, k) alone, or by examin-ing the underlying values of a(i, n, k) and t(i, j, n, k) thatproduce these values s(i, n, k) < 0. From the perspectiveof these measurements, they are both simply occurrencesof information modification. In future work, we will in-vestigate methods to formally quantify the difference be-tween these collision types.

In contrast to the gliders, the domain walls in rule 18appear to give rise to only positive values of s(i, n, k =16). This indicates that the domains walls contain onlytrivial information processing, in contrast with regulargliders which required a small amount of non-trivial in-formation processing in order to compute their continu-ation. This is perhaps akin to the observation by Shaliziet al. 24 that the domain walls in rule 146 are largely de-termined by the dynamics on either side, i.e. they arenot the result of any interaction per se but of dominancefrom a single source at each time step.

C. Storage modifications in non-periodic domains

Also, as displayed in Fig. 2(e), the background domainof rule 18 takes values of s(i, n, k = 16) as either positiveor negative with a(i, n, k = 16), since t(i, j = 1, n, k =16) and t(i, j = −1, n, k = 16) vanish at these points.As described in Section II A, a(i, n, k = 16) is positivefor the “0” values at every second site, whereas for thealternate sites it is positive where these are “1” but nega-tive where they are “0”. This indicates that the “0” sitesfor every second point and the “1”’s which only occur inthe alternate phase are trivial computations dominatedby information storage. In contrast then, some minorinformation processing is required to compute the “0”

sites in the alternate phase. The occurrence of a “0” inthis phase is an information modification event becauseit makes the task of temporally determining the phasemore ambiguous in the future. With a(i, n, k = 16) < 0as the determining factor here, it could be viewed as astorage modification.

D. Proliferation of information modification in chaoticdynamics

We have also applied s(i, n, k = 16) to ECA rule 22(not shown, see additional results36 or raw CA and plotsof a(i, n, k = 16) and t(i, j = −1, n, k = 16) here37).Rule 22 is considered complex by some authors38–40, butchaotic by others41. Certainly it does not contain anycoherent particle-like structures24 (confirmed by local in-formation storage and transfer measures10,36).

The s(i, n, k = 16) profile for rule 22 contains manypoints of both positive and negative local separable in-formation. Indeed the presence of negative values impliesthe occurrence of information modification, yet theredoes not appear to be any structure to these profiles (inalignment with10,24,36).

In rule 22, we observe information modification eventswith both a(i, n, k) < 0 and t(i, j, n, k) < 0 for all j, inaddition to the other sign combinations for these eventspreviously observed in rules 110, 54 and 18 (see sum-mary in Table I). Indeed these events, along with thosewith a(i, n, k) > 0 and t(i, j, n, k) < 0 for all j, are themost prevalent and strongest modification events for rule22. We cannot conclude that these types of modifica-tions events are prohibited between coherent structuresthough: at this stage we have no theoretical basis for sucha conclusion, and indeed such modifications between co-herent structures may exist in other examples. Furtherinvestigation is required on this topic. Nonetheless, theresult itself is important since it demonstrates that infor-mation modification events can occur with all sign combi-nations of a(i, n, k), t(i, j = 1, n, k) and t(i, j = −1, n, k)(except for all of them being positive, since this wouldleave s(i, n, k) > 0).

E. Modification only understood in context of past history

We have previously observed9,10 and inferred in Sec-tions II A and II B that for appropriate measurement ofinformation storage and transfer, k should be selected tobe as large as possible for accuracy, at least larger thanthe scale of the period of the regular background domainfor CA filtering purposes. Indeed, using sufficiently largevalues of k provides the perspective of distributed com-putation, by establishing the context of the past state ofthe destination and thus properly separating informationstorage and transfer. Since the definition of s(i, n, k) isdependent on this perspective of distributed computa-tion, and accurate measurement of information storage

12

and transfer, the preceding text has assumed the sameapplies here. As such, k = 16 was used since it was foundto be sufficient for storage9 and transfer10. In testingthis assumption, we note that for rule 54 k < 4 could notdistinguish any collision points clearly from the domainsand particles, and even k < 8 could not distinguish allof them (results not shown). Correct quantification ofseparable information requires satisfactory estimates ofinformation storage and transfer, and accurate distinc-tion between the two.

V. CONCLUSION

We have introduced the local separable information inSection III as a tool to describe information modificationon a local scale in space and time in complex systems.Importantly, the measure describes the manner in whichinformation storage and transfer interact to produce non-trivial computation or information processing where “thewhole is greater than the sum of the parts”. Informationmodification events are defined to occur where the sepa-rable information is negative, indicating that separate orindependent inspection of the causal information sourcesis misleading because of non-trivial interaction betweenthese sources. In Section IV the local separable informa-tion was demonstrated to provide the first direct quan-titative evidence that particle collisions in CAs are thedominant information modification events therein. Themeasure is capable of identifying events involving bothcreation and destruction, and interestingly the locationof an information modification event often appears de-layed perhaps due to a time-lag in information process-ing. Also, the measure identified a number of other in-formation modification events. A secondary classificationof these event types was made using their local informa-tion dynamics properties, as summarized in Table I. Fur-thermore, in order to separate information storage andtransfer and properly identify information modification,the measure required appropriately long values of pasthistory k in establishing the context of the destination’spast and taking the perspective of distributed computa-tion.

This presentation of the separable information to iden-tify information modification completes our frameworkof the fundamental operations of distributed computa-tion. Together, the measures of the frameworkhave provided the first direct quantitative evi-dence for all of the conjectures about the roleof emergent structures in distributed computa-tion in CAs: that blinkers implement information stor-age, particles are information transfer agents, and par-ticle collisions are information modification events. Theframework is unique in relating these three operationsof computation, and in providing such evidence for ourqualitative understanding of their embodiment.

With the framework in place, in future work we willapply these measures to study various systems, and ex-

plore what they can tell us about the nature of complexcomputation. For example, we are investigating what theinformation dynamics can tell us about coherent struc-ture in computation37. We also intend to explore howthis work relates to other approaches which measure thehow information content in a destination variable is de-pendent on the number of interacting source variablesgenerating its next state42,43.

ACKNOWLEDGMENTS

A brief exposition of the local separable informationwas previously presented44. The authors thank MelanieMitchell for helpful comments and suggestions regardingan early version of this manuscript. The CA diagramswere generated using enhancements to the software ofWojtowicz 45 .

Appendix A: Total information composition

In this appendix, we show how the information stor-age and transfer components from Sections II A and II Btogether form the total information composition of thedestination, or in other words, compute the destination’snext state.

First, we note that TE can be formulated to condi-tion on the states of other causal information contrib-utors Z.10,27 This mitigates against their influence be-ing attributed to the source Y , as well as allowing theobservation of interaction-based transfer between Y andZ. For example, if X is the result of an exclusive-OR(XOR) operation between Y and Z (with independent,maximum entropy Y and Z), then the original TE for-mulation will measure zero information transfer. For-mally here we have: xn+1 = yn ⊕ zn, and e.g. tY→X =

i(xn+1; yn|x(k)n ) = 0. We refer to the original formula-tion as the apparent transfer entropy10, since it gives theapparent transfer without considering multi-source inter-actions. In contrast, we label a formulation conditioningon additional sources Z as conditional transfer entropy,with local values:

tY→X|Z(n+ 1) = limk→∞

log2

p(xn+1 | x(k)n , yn, zn)

p(xn+1 | x(k)n , zn). (A1)

Conditional transfer entropy measurements can detectthe interaction-based transfer due to Y and Z in theXOR operation for example (see measurements of thebackground domain of rule 18 in previous work10). Wenote that the apparent TE remains useful in its own rightthough, because it can only be large where the source hasa coherent effect on the destination (without requiringinteraction with other sources).

Additionally, we label the special case where we condi-tion on all causal information contributors to the destina-tion X (the set VX) except the source Y , as the complete

13

TABLE I. Classification of information modification event types (i.e. with s(i, n) < 0) in cellular automata using informationstorage and transfer properties as a secondary tool (after s(i, n) < 0).

transfer entropy10. Referring to the value of VX exclud-ing Y and X itself at time n as vyx,n, the local values ofthe complete TE are:

tcompY→X(n+ 1) = limk→∞

log2

p(xn+1 | x(k)n , yn,vyx,n)

p(xn+1 | x(k)n ,vyx,n), (A2)

vyx,n = {zn | ∀Z ∈ VX , Z 6= Y,X} . (A3)

Since the complete TE takes all interactions of the sourceinto account, it cannot become negative at all in de-terministic systems, and the only occasions where itis negative in stochastic systems are due to statisticalfluctuations10.

Furthermore, we can measure the collective transferentropy from the set of all causal information contribu-tors VX to X, with local values defined as:

tcollX (n+ 1) = limk→∞

log2

p(xn+1 | x(k)n ,vx,n)

p(xn+1 | x(k)n ), (A4)

vx,n = {zn | ∀Z ∈ VX , Z 6= X} . (A5)

Importantly, the collective TE is a conditional mutualinformation (as per the apparent TE), written locally as:

tcollX (n+ 1) = limk→∞

i(vx,n;xn+1 | x(k)n

). (A6)

Finally, we represent finite-k estimates as tY→X|Z(n +

1, k) , tcompY→X(n+ 1, k) and tcollX (n+ 1, k).We can also represent these in lattice notation for CAs

where the set of causal information contributors to Xi isthe neighborhood Vi,r of Xi within the range r:

Vi,r = {Xi−q | ∀q : −r ≤ q ≤ +r} . (A7)

For the complete TE we have:

tcomp(i, j, n+ 1) = limk→∞

tcomp(i, j, n+ 1, k),

(A8)

tcomp(i, j, n+ 1, k) = log2

p(xi,n+1 | x(k)i,n , xi−j,n,v

ji,r,n

)p(xi,n+1 | x(k)i,n ,v

ji,r,n

) ,

(A9)

vji,r,n = {xi−q,n | ∀q : −r ≤ q ≤ +r, q 6= j, q 6= 0} .(A10)

We then consider how the information required to pre-dict the next state of a variable can be written in termsof information storage and transfer. This total informa-tion in a variable is of course its entropy, written in localnotation as hX(n + 1) = − log2 p(xn+1) = h (xn+1). Wehave previously shown9 that that the the local entropycan be decomposed into the amount predictable from thevariable’s past (the local active information storage) andthe remaining uncertainty after examining this memory(the local temporal entropy rate hµX):

hX(n+ 1) = aX(n+ 1, k) + hµX(n+ 1, k), (A11)

hµX(n+ 1) = limk→∞

hµX(n+ 1, k), (A12)

hµX(n+ 1, k) = − log2 p(xn+1 | x(k)n ), (A13)

= h(xn+1 | x(k)n

). (A14)

Note that the correctness of Eq. (A11) is independent ofthe value of k; increasing k simply increases the propor-tion of information in the next state that is attributed tothe variable’s past (i.e. the active information storage).

First we decompose the local temporal entropy rate as

14

a sum of collective transfer entropy plus intrinsic uncer-tainty uX :46

hµX(n+ 1, k) = tcollX (n+ 1, k) + uX(n+ 1, k), (A15)

uX(n+ 1, k) = h(xn+1 | x(k)n ,vx,n

), (A16)

using Eq. (A14) and Eq. (A6). The collective TE canthen be decomposed itself by incrementally taking ac-count of the contribution of each causal informationsource. Let us consider an arbitrary ordering (using in-dex g) of the causal sources in {VX \X}: Z1, Z2, ..., ZG.We can then write an arbitrarily ordered subset of g − 1sources as:

V<gX = {Zc | ∀c : 1 ≤ c < g} , (A17)

v<gx,n = {zc,n | ∀c : 1 ≤ c < g} , (A18)

and then make the decomposition:

tcollX (n+ 1, k) =∑g

i(zg,n;xn+1 | x(k)n ,v<gx,n). (A19)

We have a sum of incrementally conditioned mutual in-formation terms: each term is the information addedby the given source Zg that was not contained either inthe past of the destination or in the previously inspectedsources V<g

X . Each term is a transfer entropy itself, andif we expand this sum:

tcollX (n+ 1, k) =i(z1,n;xn+1 | x(k)n )+

i(z2,n;xn+1 | x(k)n , z1,n)+

i(z3,n;xn+1 | x(k)n , z1,n, z2,n) + . . .+

i(zG,n;xn+1 | x(k)n ,v<Gx,n ), (A20)

we see that the first term is the apparent TE from sourceZ1, the last term is the complete TE from source ZG(since all other causal sources are conditioned on), andthe intermediate terms are conditional transfer entropies(see Eq. (A1)). The collective transfer entropy captures(while accounting for redundancies) all transfers fromthe sources to the destination, incorporating both single-source and interaction-based transfers. Importantly, it isnot a simple sum of the apparent TE from each source,nor the sum of the complete TE from each source.

Finally, we can combine Eq. (A11), Eq. (A15) andEq. (A19) together47:

hX(n+ 1) =aX(n+ 1, k) + tcollX (n+ 1, k) + uX(n+ 1, k),

(A21)

hX(n+ 1) =aX(n+ 1, k) +∑g

i(zg,n;xn+1 | x(k)n ,v<gx,n)

+ uX(n+ 1, k). (A22)

This explicitly demonstrates that the information re-quired to predict the next state of a destination is thesum of:

• the information gained from the past of the desti-nation (i.e. the active information storage); plus

• the collective information transfer from all causalsources (as a sum of incrementally conditionedtransfer entropies); plus

• any remaining intrinsic uncertainty in the destina-tion.

Of course, these equations are valid for the averages aswell as the local values.

Eq. (A22) leads to a simple relationship in ECAs,where we have no intrinsic uncertainty, and the TE termsare the apparent TE from one neighbor plus the completeTE from the other:

h(i, n+ 1) =a(i, n+ 1, k) + t(i, j = −1, n+ 1, k)+

tcomp(i, j = 1, n+ 1, k), (A23)

or vice-versa in j = 1 and −1.

1D. P. Feldman, C. S. McTague, and J. P. Crutchfield, “The orga-nization of intrinsic computation: Complexity-entropy diagramsand the diversity of natural information processing,” Chaos, 18,043106 (2008).

2O. Kinouchi and M. Copelli, “Optimal dynamical range ofexcitable networks at criticality,” Nature Physics, 2, 348–351(2006).

3J. J. Atick, “Could information theory provide an ecological the-ory of sensory processing?” Network: Computation in NeuralSystems, 3, 213 (1992).

4M. A. Sanchez-Montanes and F. J. Corbacho, “Towards a newinformation processing measure for neural computation,” in Pro-ceedings of the International Conference on Artificial NeuralNetworks (ICANN 2002), Madrid, Spain, Lecture Notes in Com-puter Science, Vol. 2415, edited by J. Dorronsoro (Springer-Verlag, Berlin/Heidelberg, 2002) pp. 637–642.

5T. Yamada and K. Aihara, “Spatio-temporal complex dynam-ics and computation in chaotic neural networks,” in Proceedingsof the IEEE Symposium on Emerging Technologies and FactoryAutomation (ETFA ’94), Tokyo (IEEE, 1994) pp. 239–244.

6M. H. Jakubowski, K. Steiglitz, and R. Squier, “Informationtransfer between solitary waves in the saturable schrodinger equa-tion,” Physical Review E, 56, 7267 (1997).

7A. Adamatzky, ed., Collision-Based Computing (Springer-Verlag, Berlin, 2002).

8D. E. Edmundson and R. H. Enns, “Fully 3-dimensional collisionsof bistable light bullets,” Optics Letters, 18, 1609–1611 (1993).

9J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, “Local mea-sures of information storage in complex distributed computa-tion,” (2010), in preparation.

10J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, “Local infor-mation transfer as a spatiotemporal filter for complex systems,”Physical Review E, 77, 026110 (2008).

11C. G. Langton, “Computation at the edge of chaos: phase transi-tions and emergent computation,” Physica D, 42, 12–37 (1990).

12S. Wolfram, A New Kind of Science (Wolfram Media, Cham-paign, IL, USA, 2002).

13M. Mitchell, “Computation in cellular automata: A selectedreview,” in Non-Standard Computation, edited by T. Gramss,S. Bornholdt, M. Gross, M. Mitchell, and T. Pellizzari (VCHVerlagsgesellschaft, Weinheim, 1998) pp. 95–140.

14The number of particles resulting from a collision has been stud-ied elsewhere17.

15N. Boccara, J. Nasser, and M. Roger, “Particlelike structuresand their interactions in spatiotemporal patterns generated byone-dimensional deterministic cellular-automaton rules,” Physi-cal Review A, 44, 866–875 (1991).

15

16B. Martin, “A group interpretation of particles generated byone dimensional cellular automaton, Wolfram’s 54 rule,” Inter-national Journal of Modern Physics C, 11, 101–123 (2000).

17W. Hordijk, C. R. Shalizi, and J. P. Crutchfield, “Upper boundon the products of particle interactions in cellular automata,”Physica D, 154, 240–258 (2001).

18G. J. Martinez, A. Adamatzky, and H. V. McIntosh, “Phe-nomenology of glider collisions in cellular automaton rule 54 andassociated logical gates,” Chaos, Solitons and Fractals, 28, 100–111 (2006).

19M. Mitchell, J. P. Crutchfield, and P. T. Hraber, “Evolvingcellular automata to perform computations: Mechanisms andimpediments,” Physica D, 75, 361–391 (1994).

20M. Mitchell, J. P. Crutchfield, and R. Das, “Evolving cellularautomata with genetic algorithms: A review of recent work,”in Proceedings of the First International Conference on Evolu-tionary Computation and Its Applications, Moscow, edited byE. D. Goodman, W. Punch, and V. Uskov (Russian Academy ofSciences, Russia, 1996).

21J. E. Hanson and J. P. Crutchfield, “The attractor-basin portaitof a cellular automaton,” Journal of Statistical Physics, 66, 1415–1462 (1992).

22A. Wuensche, “Classifying cellular automata automatically:Finding gliders, filtering, and relating space-time patterns, at-tractor basins, and the Z parameter,” Complexity, 4, 47–66(1999).

23T. Helvik, K. Lindgren, and M. G. Nordahl, “Local informationin one-dimensional cellular automata,” in Proceedings of the In-ternational Conference on Cellular Automata for Research andIndustry, Amsterdam, Lecture Notes in Computer Science, Vol.3305, edited by P. M. A. Sloot, B. Chopard, A. G. Hoekstra(Springer, Berlin/Heidelberg, 2004) pp. 121–130.

24C. R. Shalizi, R. Haslinger, J.-B. Rouquier, K. L. Klinkner, andC. Moore, “Automatic filters for the detection of coherent struc-ture in spatiotemporal systems,” Physical Review E, 73, 036104(2006).

25D. J. MacKay, Information Theory, Inference, and Learning Al-gorithms (Cambridge University Press, Cambridge, 2003).

26J. P. Crutchfield and D. P. Feldman, “Regularities unseen, ran-domness observed: Levels of entropy convergence,” Chaos, 13,25–54 (2003).

27T. Schreiber, “Measuring information transfer,” Physical ReviewLetters, 85, 461–464 (2000).

28J. T. Lizier and M. Prokopenko, “Differentiating informationtransfer and causal effect,” European Physical Journal B, 73,605–615 (2010).

29P. Grassberger, “Some more exact enumeration results for 1d cel-lular automata,” Journal of Physics A: Mathematical and Gen-eral, 20, 4039–4046 (1987).

30Interactions are accounted for in that equation because of the in-cremental conditioning on previous sources (i.e. in conditionaland complete transfer entropies). Consideration of apparenttransfer entropies only in Eq. (6) means that interactions arenot being accounted for.

31The separable information has parallels to the sum of “first orderterms” (the apparent contribution of each source without con-sidering interactions) in the total information of a destinationdecomposition by Ludtke et al. 42 . The local separable informa-tion however is distinguished in considering the contributions inthe context of the destination’s past, and evaluating these on alocal scale – both features are critical for an understanding ofdistributed computation.

32M. Prokopenko, F. Boschietti, and A. J. Ryan, “An information-theoretic primer on complexity, self-organization, and emer-gence,” Complexity, 15, 11–28 (2009).

33D. G. Green, “Emergent behavior in biological systems,” Com-plexity International, 1 (1994), paper ID: green01.

34We report similar results36 for other CA rules with gliders, e.g.the density classification rule φpar from Mitchell et al. 19,20 .

35Only then does the active information storage go negative, be-

cause from a temporal perspective the inner domain no longercontinues. The separable information is positive here though, asthe transfer sources provide positive information about their do-mains intruding (having coalesced).

36J. T. Lizier, The local information dynamics of distributed com-putation in complex systems, Ph.D. thesis, The University of Syd-ney (2010).

37J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, “Coherent infor-mation structure in complex computation,” (2010), submittedto Theory in Biosciences, Guided Self-organization 2009 specialissue.

38P. Grassberger, “Long-range effects in an elementary cellular au-tomaton,” Journal of Statistical Physics, 45, 27–39 (1986).

39P. Grassberger, “Toward a quantitative theory of self-generatedcomplexity,” International Journal of Theoretical Physics, 25,907–938 (1986).

40R. Badii and A. Politi, “Thermodynamics and complexity of cel-lular automata,” Physical Review Letters, 78, 444 (1997).

41H. Gutowitz and C. Domain, “The topological skeleton of cellularautomaton dynamics,” Physica D, 103, 155–168 (1997).

42N. Ludtke, S. Panzeri, M. Brown, D. S. Broomhead, J. Knowles,M. A. Montemurro, and D. B. Kell, “Information-theoretic sen-sitivity analysis: a general method for credit assignment in com-plex networks,” Journal of The Royal Society Interface, 5, 223–235 (2008).

43T. Kahle, E. Olbrich, J. Jost, and N. Ay, “Complexity mea-sures from interaction structures,” Physical Review E, 79, 026201(2009).

44J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, “Detectingnon-trivial computation in complex dynamics,” in Proceedingsof the 9th European Conference on Artificial Life (ECAL 2007),Lisbon, Portugal, Lecture Notes in Artificial Intelligence, Vol.4648, edited by F. Almeida e Costa, L. M. Rocha, E. Costa,I. Harvey, and A. Coutinho (Springer, Berlin / Heidelberg, 2007)pp. 895–904.

45M. Wojtowicz, http://psoup.math.wisc.edu/mcell/mjcell/mjcell.html“Java cellebration v.1.50,” (2002), Online software.

46Note that any dependence of uX on k is due to statistical fluctu-ations rather than a causal effect. (Of course, this is for k largeenough to include all direct causal contributors in the past of thedestination).

47Similarly motivated are other decompositions of the total infor-mation for a destination42,48. The approach of Ludtke et al. 42

however decomposes the total information in terms of a sum ofmutual information terms of increasing size, rather than incre-mentally conditioned mutual information terms. The approachof Bettencourt et al. 48 does use incrementally conditioned mu-tual information terms, though again this is merely as a meansto infer the contribution to the total information from terms ofincreasing size or order k. Our presentation is distinct from bothof these in considering the destination’s history as a single source,regardless of whether these states are causal contributors or not.Establishing the context of the past is critical for an understand-ing of distributed computation. Also, our approach is distinct inconsidering the decomposition on a local scale.

48L. M. A. Bettencourt, V. Gintautas, and M. I. Ham, “Identifica-tion of functional information subgraphs in complex networks,”Physical Review Letters, 100, 238701 (2008).