Approximating the Smallest Grammar:Kolmogorov Complexity in Natural Models

Moses CharikarPrinceton University

Eric LehmanMIT

Ding LiuPrinceton University

Rina PanigrahyCisco Systems

Manoj PrabhakaranPrinceton University

April RasalaMIT

Amit SahaiPrinceton University

abhi shelatMIT

February 20, 2002

Abstract

We consider the problem of finding the smallest context-free grammar that generates exactly one given string oflength n. The size of this grammar is of theoretical in-terest as an efficiently computable variant of Kolmogorovcomplexity. The problem is of practical importance in ar-eas such as data compression and pattern extraction.

The smallest grammar is known to be hard to approxi-mate to within a constant factor, and an o(log n/log log n)approximation would require progress on a long-standingalgebraic problem [10]. Previously, the best proved ap-proximation ratio was O(n1/2) for the BISECTION algo-rithm [8]. Our main result is an exponential improvementof this ratio; we give an O(log(n/g)) approximation al-gorithm, where g is the size of the smallest grammar.

We then consider other computable variants of Kolo-mogorov complexity. In particular we give an O(log2 n)approximation for the smallest non-deterministic finiteautomaton with advice that produces a given string. Wealso apply our techniques to advice-grammars andedit-grammars, two other natural models of string com-plexity.

e-mail: {moses,dingliu,mp,sahai}@cs.princeton.edue-mail: {e lehman,arasala,abhi}@mit.edue-mail: rinap@cisco.com

1 IntroductionThis paper addresses the smallest grammar problem;namely, what is the smallest context-free grammar thatgenerates exactly one given string ? For example, thesmallest context-free grammar generating the string:

= a rose is a rose is a rose

is as follows:

S BBAA a roseB A is

The size of a grammar is defined to be the total numberof symbols on the right sides of all rules. The small-est grammar is known to be hard to approximate withina small constant factor. Further, even for a very re-stricted class of strings, approximating the smallest gram-mar within a factor of o

(log n

log log n

)would require progress

on a well-studied algebraic problem [10]. Previously, thebest proven approximation ratio was O(n1/2) for the BI-SECTION algorithm [8]. Our main result is an exponentialimprovement of this ratio.

Several rich lines of research connect this elegant prob-lem to fields such as Kolmogorov complexity, data com-pression, pattern identification, and approximation algo-rithms for hierarchical problems.

1

The size of the smallest context-free grammar generat-ing a given string is a natural, but more tractable variantof Kolmogorov complexity. (The Kolmogorov complex-ity of a string is the description length of the smallest Tur-ing machine that outputs that string.) The Turing machinemodel for representing strings is too powerful to be ex-ploited effectively; in general, the Kolmogorov complex-ity of a string is uncomputable. However, weakening themodel from Turing machines to context-free grammars re-duces the complexity of the problem from the realm of un-decidability to mere intractability. How well the smallestgrammar can be approximated remains an open question.In this work we make significant progress toward an an-swer: our main result is that the grammar complexityof a string is O(log(n/g))-approximable in polynomialtime, where g is the size of the smallest grammar.

This perspective on the smallest grammar problem sug-gests a general direction of inquiry: can the complex-ity of a string be determined or approximated with re-spect to other natural models? Along this line, we con-sider the problem of optimally representing an input stringby a non-deterministic finite automaton guided by an ad-vice string. We show that the optimal representation ofa string is O(log2 n)-approximable in this model. Thenwe consider context-free grammars with multiple produc-tions per nonterminal, again guided by an advice string.This modified grammar model turns out, unexpectedly, tobe essentially equivalent to the original.

The smallest grammar problem is also important in thearea of data compression. Instead of storing a long string,one can store a small grammar that generates it, providedsuch a grammar can be found. This line of thought has ledto a flurry of research [8, 12, 16, 7, 1, 9] in the data com-pression community. Kieffer and Yang show that a goodsolution to the grammar problem leads to a good uni-versal compression algorithm for finite Markov sources[7]. Empirical results indicate that grammar-based datacompression is competitive with other techniques in prac-tice [8, 12]. Surprisingly, however, the most prominentalgorithms for the smallest grammar problem have beenshown to exploit the grammar model poorly [10]. In par-ticular, the SEQUITUR [12], BISECTION [8], LZ78 [16],and SEQUENTIAL [14] algorithms all have approximationratios that are (n1/3). In this paper, we achieve a loga-rithmic ratio.

The smallest grammar problem is also relevant to iden-

tifying important patterns in a string, since such patternsnaturally correspond to nonterminals in a compact gram-mar. In fact, an original motivation for the problem was toidentify regularities in DNA sequences [11]. Since then,smallest grammar algorithms have been used to highlightpatterns in musical scores and uncover properties of lan-guage from example texts [4]. All this is possible becausea string represented by a context-free grammar remainsrelatively comprehensible. This comprehensibility is animportant attraction of grammar-based compression rela-tive to otherwise competitive compression schemes. Forexample, the best pattern matching algorithm that oper-ates on a string compressed as a grammar is asymptoti-cally faster than the equivalent for LZ77 [6].

Finally, work on the smallest grammar problem extendsthe study of approximation algorithms to hierarchical ob-jects, such as grammars, as opposed to flat objects, suchas graphs, CNF-formulas, etc. This is a significant shift,since many real-world problems have a hierarchical na-ture, but standard approximation techniques such as lin-ear and semi-definite programming are not easily appliedto this new domain.

The remainder of this paper is organized as follows.Section 2 defines terms and notation. Section 3 con-tains our main result, an O(log(n/g)) approximation al-gorithm for the smallest grammar problem. Section 4follows up the Kolmogorov complexity connection andincludes the results on the complexity measure in othermodels. We conclude with some open problems and newdirections.

2 DefinitionsA grammar G is a 4-tuple (, , S, ), where is a setof terminals, is a set of nonterminals, S is the startsymbol, and is a set of rules of the form T ().A symbol may be either a terminal or nonterminal. Thesize of a grammar is defined to be the total number ofsymbols on the right sides of all rules. Note that a gram-mar of size g can be readily encoded using O(g log g) bits,and so our measure is close to this natural alternative.1

1More sophisticated ways to encode a grammar [7, 14] relate theencoding size in bits to the above definition of the size of a grammar ina closer manner, and give theoretical justification to this definition.

2

Since G generates exactly one finite string, there is ex-actly one rule in defining each nonterminal in . Fur-thermore, G is acyclic; that is, there exists an orderingof the nonterminals in such that each nonterminal pre-cedes all nonterminals in its definition. The expansion ofa string of symbols is obtained by replacing each non-terminal in by its definition until only terminals remain.The expansion of is denoted , and the length of is denoted [].

Throughout, symbols (terminals and nonterminals) areuppercase, and strings of symbols are lowercase Greek.We use for the input string, n for its length, g for thesize of a grammar that generates , and g for the size ofthe smallest such grammar.|G| denotes the size of grammar minus number of non-

terminals. Without loss of generalily we do not allowunary rules of the form A B; then |G| is at least halfthe size of G.

3 An O(log(n/g)) ApproximationAlgorithm

We now give an O(log(n/g))-approximation algorithmfor the smallest grammar problem. The description is di-vided into three sections. First, we introduce a variant ofthe well-known LZ77 compression scheme. This servestwo purposes: it gives a lower bound on the size of thesmallest grammar for a string and is the starting point forour construction of a small grammar. Second, we intro-duce balanced binary grammars, a class of well-behavedgrammars that our procedure employs. We also introducethree basic operations on balanced binary grammars. Fi-nally, we present the main algorithm, which translates astring compressed using our LZ77 variant into a gram-mar at most O(log(n/g)) times larger than the smallest.

3.1 LZ77Our algorithm for approximating the smallest grammaremploys a variant of the LZ77 compression scheme [15].In this variant, a string is represented by a list of pairs(p1, l1), . . . , (ps, ls). Each pair (pi, li) represents a string,and the concatenation of these strings is . In particular,if pi = 0, then the pair represents the string li, which is

a single terminal. If pi 6= 0, then the pair represents aportion of the prefix of that is represented by the pre-ceding i 1 pairs; namely, the li terminals beginning atposition pi in . Note that, unlike in the standard LZ77,we require that the substring referred to by (pi, li) be fullycontained within the prefix of generated by the previouspairs. The size of the LZ77 representation of a string is s,the number of pairs employed. The smallest LZ77 repre-sentation can be easily found using a greedy algorithm inpolynomial time. More efficient algorithms are possible;for example, [5] gives an O(n log n) time algorithm forour variant of LZ77.

As observed in [12], a grammar G for a string can beconverted in a simple way to an LZ77 representation. Itis not hard to see that this transformation gives an LZ77list of at most |G|+ 1 pairs. Thus we have:

Lemma 1 The size of the smallest grammar for a stringis lower bounded by the size of the smallest LZ77 repre-sentation for that string.

Our O(log(n/g))-approximation algorithm essen-tially inverts this process, mapping an LZ77 sequence toa grammar. This direction is much more involved, be-cause the strings represented by pairs can arbitrarily over-lap each other whereas the rules in a grammar have lessflexibility. Hence, one of our main technical contributionsis an efficient procedure to perform this conversion.

3.2 Balanced Binary GrammarsOur approximation algorithm works exclusively with arestricted class of well-behaved grammars referred to asbalanced binary grammars. A binary rule is a grammarrule with exactly two symbols on the right side. A binarygrammar is a grammar in which every rule is binary. Twostrings of symbols, and , are -balanced if

1 []

[] 1

for some constant between 0 and 12 . Intuitively, -balanced means about the same length. Note that in-verting the fraction [][] gives an equivalent condition. Thecondition above is also equivalent to saying that the lengthof the expansion of each string ([] and []) is between an

3

and a 1 fraction of the length of the combined ex-pansion ([]). An -balanced rule is a binary rule inwhich the two symbols on the right are -balanced. An-balanced grammar is a binary grammar in which ev-ery rule is -balanced. For brevity, we usually shorten-balanced to simply balanced.

If G is a binary grammar, |G| = size of G2 = number ofnon-terminals in G. So in the rest of this section we shallconsider the number of non-terminals of a binary gram-mar instead of its size.

The remainder of this section defines three basic opera-tions on balanced binary grammars. Each operation addsa small number of rules to an existing balanced grammarto produce a new balanced grammar that has a nontermi-nal with specified properties. These operations are sum-marized as follows.

AddPair: Produces a balanced grammar containinga nonterminal with expansion XY from a bal-anced grammar containing symbols X and Y . Thenumber of rules added to the original grammar isO(1 +

log [X][Y ] ).AddSequence: Produces a balanced grammar con-

taining a nonterminal with expansion X1 . . . Xtfrom a balanced grammar containing symbolsX1 . . . Xt. The number of rules added isO(t(1 + log [X1...Xt]t

)).

AddSubstring: Produces a balanced grammar con-taining a nonterminal with expansion from a bal-anced grammar containing a nonterminal with asa substring of its expansion. Adds O(log ||) newrules.

For these operations to work correctly, we require that be selected from the limited range 0 < 1 12

2,

which is about 0.293. These three operations are detailedbelow.

3.2.1 The AddPair Operation

We are given a balanced grammar with symbols X andY and want to create a balanced grammar containing anonterminal with expansion XY . Suppose that [X ] [Y ]; the other case is symmetric.

First, we decompose Y into a string of symbols as fol-lows. Initially, this string consists of the symbol Y itself.Thereafter, while the first symbol in the string is not inbalance with X , we replace it by its definition. A routinecalculation, which we omit, shows that balance is eventu-ally achieved. At this point, we have a string of symbolsY1 . . . Yt with expansion Y such that Y1 is in balancewith X . Furthermore, note that Y1 . . . Yi is in balance withYi+1 for all 1 i < t by construction.

Now we create new rules. Initially, we create a newrule Z1 XY1 and declare this to be the active rule.The remainder of the operation is proceeds in steps. Atthe start of the i-th step, the active rule has the form Zi AiBi, and the following three invariants hold:

1. Zi = XY1 . . . Yi2. Bi is a substring of Y1 . . . Yi.3. All rules in the grammar are balanced, including the

active rule.

The relationships between strings implied by the first twoinvariants are indicated in the following diagram:

Zi X Y1 Y2 . . .

Ai

. . . Yi1 Yi Bi

Yi+1 . . . Yt

After t steps, the active rule defines a nonterminal Zt withexpansion XY1 . . . Yt = XY as desired.

The invariants stated above imply some inequalitiesthat are needed later to show that the grammar remainsin balance. Since Y1 . . . Yi is in balance with Yi+1, wehave:

1 [Yi+1]

[Y1 . . . Yi] 1

Since Bi is a substring of Y1 . . . Yi by invariant 2, wecan conclude:

1 [Yi+1]

[Bi](1)

On the other hand, since Zi is a superstring ofY1 . . . Yi by invariant 1, we can conclude:

4

[Yi+1]

[Zi] 1

(2)

Below we describe how during a step a new active ruleis created. For clarity, we supplement the text with dia-grams. In these diagrams, a rule Zi AiBi is indicatedwith a wedge:

Preexisting rules are indicated with broken lines, and newrules with dark lines.

At the start of the i-th step, the active rule is Zi AiBi. Our goal is to create a new active rule that definesZi+1 while maintaining the three invariants. There arethree cases to consider.Case 1: If Zi and Yi+1 are in balance, then we create anew rule:

Zi+1 ZiYi+1

This becomes the active rule. It is easy to check that thethree invariants are maintained.

If case 1 is bypassed, then Zi and Yi+1 are not in bal-ance; that is, the following assertion does not hold:

1 [Yi+1]

[Zi] 1

Since the right inequality is (2), the left inequality mustbe violated. Thus, hereafter we can assume:

1 >[Yi+1]

[Zi](3)

Case 2: Otherwise, if Ai is in balance with BiYi+1, thenwe create two new rules:

Zi+1 AiTiTi BiYi+1

The first of these becomes the active rule. It is easy tocheck that the first two invariants are maintained. But thethird, which asserts that all new rules are balanced, re-quires some work. The rule Zi+1 AiTi is balancedby the case assumption. What remains is to show that therule Ti BiYi+1 is balanced; that is, we must show:

1 [Yi+1]

[Bi] 1

The left inequality is (1). For the right inequality, beginwith (3):

[Yi+1] [Ai]

[BiYi+1](4)

Combining inequalities (3) and (4), one can use algebraicmanipulation to establish the following bounds, whichhold hereafter:

[Ai]

[Bi]

1 2 (5)[Yi+1]

[Bi]

1 2 (6)

5

Then [Bi] > 1, as say [Ai] 1 > 12 ; i.e., Bi is not aterminal. So let Bi be defined by the rule Bi UV .

Case 3: If we bypass cases 1 and 2, we create three newrules:

Zi+1 PiQiPi AiUQi V Yi+1

The first of these becomes the active rule. We must checkthat all of the new rules are in balance. We begin withPi AiU . In one direction, we have:

[Ai]

[U ] [Ai]

[Bi]

1

The first inequality follows as [Bi] = [UV ] > [U ]. Thesecond inequality uses the fact that Ai and Bi are in bal-ance. In the other direction, we have:

[Ai]

[U ] [Ai]

[Bi]

11 2

1

The first inequality uses the fact that Bi UV is bal-anced, and the second follows from (5). The last inequal-ity holds for all < 0.293.

Next, we show that Qi V Yi+1 is balanced. In onedirection, we have:

[Yi+1]

[V ] [Yi+1]

[Bi]

11 2

1

The first inequality uses the fact that Bi UV is bal-anced, the second uses (6), and the third holds for all < 0.293. In the other direction, we have:

[Yi+1]

[V ] [Yi+1]

[Bi]

1

The first inequality follows as [Bi] > [V ], and the secondis (1).

Finally, we must check that Zi+1 PiQi is in balance.In one direction, we have:

[Pi]

[Qi]=

[AiU ]

[V Yi+1]

[Ai] + (1 )[Bi][Bi] + [Yi+1]

=

[Ai][Bi]

+ (1 ) + [Yi+1][Bi]

12 + (1 ) + 1

1

The equality follows from the definitions of Pi and Qi.The first inequality uses the fact that the rule Bi UVis balanced. The subsequent equality follows by dividingthe top and bottom by [Bi]. In the next step, we use (5) onthe top, and (1) on the bottom. The final inequality holdsfor all 13 . In the other direction, we have:

[Pi]

[Qi]=

[AiU ]

[V Yi+1]

[Ai] + [Bi](1 )[Bi] + [Yi+1]

=

[Ai][Bi]

+

(1 ) + [Yi+1][Bi]

1 +

(1 ) + 12

1

6

As before, the first inequality uses the definitions of Piand Qi. Then we use the fact that Bi UV is balanced.We obtain the second equality by dividing the top and bot-tom by [Bi]. The subsequent inequality uses the fact thatAi and Bi are in balance on the top and (6) on the bottom.The final inequality holds for all 13 .

All that remains is to upper bound the number of rulescreated during the ADDPAIR operation. At most threerules are added in each of the t steps. Therefore, it suf-fices to upper bound t. Recall that t is determined whenY is decomposed into a string of symbols: each time weexpanded the first symbol in the string, the length of theexpansion of the first symbol decreases by a factor of atleast 1 . When the first symbol is in balance withX , the process stops. Therefore, the number of steps isO(log([Y ]/[X ])). Since the string initially contains onesymbol, t is O(1 + log([Y ]/[X ])). Therefore, the num-ber of new rules is O (1 + log([Y ]/[X ])). The case of[X ] > [Y ] is symmetric, and the common bound for thetwo cases can be written as O

(1 +

log [X][Y ] ).3.2.2 The AddSequence Operation

The ADDSEQUENCE operation is a generalization of AD-DPAIR. We are given a balanced grammar with nonter-minals X1 . . . Xt and want to create a balanced grammarcontaining a nonterminal with expansion X1 . . .Xt.

But we solve this problem in a much simpler manner,using the ADDPAIR operation repeatedly. Consider plac-ing the Xi at the leaves of a balanced binary tree. (To sim-plify the analysis, assume that t is a power of two.) Wecreate a nonterminal for each internal node by combiningthe nonterminals at the child nodes using ADDPAIR. Re-call that the number of rules that ADDPAIR creates whencombining nonterminals X and Y is:

O

(1 +

log [X ][Y ])

= O (1 + log [X ] + log [Y ])

Let c denote the hidden constant on the right, and let sequal [X1 . . .Xt]. Creating all the nonterminals on thebottom level of the tree generates at most

ct/2 + ct

i=1

log [Xi] ct/2 + ct log st

rules. (The inequality follows from the concavity of log.)Similarly, the number of rules created on the second levelof the tree is at most ct/4+c(t/2) log st/2 , because we pairt/2 nonterminals, but the sum of their expansion lengthsis still s. In general, on the i-th level, we create at most

ct/2i+1 + c(t/2i) logs

t/2i= c(t/2i) log

s

t+ ct(i +

1

2)/2i

new rules. Summing i from 0 to log t, we find that thetotal number of rules created is at most

log ti=0

c(t/2i) logs

t+ ct(i +

1

2)/2i = O

(t(1 + log

s

t

))where s = [X1 . . . Xt].

3.2.3 The AddSubstring Operation

We are given a balanced grammar containing a nontermi-nal with as a substring of its expansion. We want tocreate a balanced grammar containing a nonterminal withexpansion exactly .

Let T be the nonterminal with the shortest expansionsuch that its expansion contains as a substring. LetT XY be its definition. Then we can write = ps,where the prefix p lies in X and the suffix s lies inY . (Confusingly, p is actually a suffix of X, and sis a prefix of Y .) We generate a nonterminal that ex-pands to the prefix p, another that expands to the suffixs, and then merge the two with ADDPAIR. The last stepgenerates only O(log ||) new rules. So all that remainsis to generate a nonterminal that expands to the prefix, p;the suffix is handled symmetrically. This task is dividedinto two phases.

In the first phase, we find a sequence of nonterminalsX1 . . . Xt with expansion equal to p. To do this, we be-gin with an empty sequence and employ a recursive pro-cedure. At each step, we have a remaining suffix (initiallyp) of some current nonterminal (initially X). Duringeach step, we consider the definition of the current non-terminal, say X AB. There are two cases:

1. If the remaining suffix wholly contains B, then weprepend B to the nonterminal sequence. The remain-ing suffix becomes the portion of the old suffix that

7

overlaps A, and the current nonterminal becomesA.

2. Otherwise, we keep the same remaining suffix, butthe current nonterminal becomes B.

A nonterminal is only added to the sequence in case 1.But in that case, the length of the desired suffix is scaleddown by at least a factor 1 . Therefore the length ofthe resulting nonterminal sequence is t = O(log ||).

This construction implies the following inequality,which we will need later:

[X1 . . .Xi]

[Xi+1] 1

(7)

This inequality holds because X1 . . . Xi is a suffix ofthe expansion of a nonterminal in balance with Xi+1.Consequently, X1 . . . Xi is not too long to be in balancewith Xi+1.

In the second phase, we merge the nonterminals in thesequence X1 . . . Xt to obtain the nonterminal with expan-sion p. The process goes from left to right. Initially, weset R1 = X1. Thereafter, at the start of the i-th step, wehave a nonterminal Ri with expansion X1 . . . Xi andseek to merge in nonterminal Xi+1. There are two cases,distinguished by whether or not the following inequalityholds:

1 [Ri]

[Xi+1]

If so, then Ri and Xi+1 are in balance. (In-equality (7) supplies the needed upper bound on[Ri]/[Xi+1].) Therefore, we add the rule Ri+1 RiXi+1.

If not, then Ri is too small to be in balance withXi+1. We use ADDPAIR to merge the two, whichgenerates O(1+log ([Xi+1]/[Ri])) new rules. Since[Xi+1] [Ri+1], the number of new rules is O(1 +log ([Ri+1]/[Ri])).

Summing the number of rules created in this processgives:

ti=1

O

(1 + log

[Ri+1]

[Ri]

)= O(t + log[Rt])

= O(log ||)The second bound follows from the fact, observed previ-ously, that t = O(log ||) and from the fact that Rt =p. Generating a nonterminal for the suffix s requiresO(log ||) rules as well. Therefore, the total number ofnew rules is O(log ||) as claimed.

3.3 The AlgorithmWe now combine all the tools of the preceding two Sec-tions to obtain an O(log(n/g))-approximation algorithmfor the smallest grammar problem.

First, we apply the LZ77 variant describedin Section 3.1. This gives a sequence of pairs(p1, l1) . . . (ps, ls). By Lemma 1, the length of thissequence is a lower bound on the size of the smallestgrammar for ; that is, s g. Now we employ the toolsof Section 3.2 to translate this sequence to a grammar.We work through the sequence from left to right, buildingup a balanced binary grammar as we go. Throughout, wemaintain an active list of grammar symbols.

Initially, the active list is l1, which must be a singlecharacter since p1 = 0 necessarily. In general, at the be-ginning of the i-th step, the expansion of the active list isthe string represented by (p1, l1) . . . (pi, li). Our goal forthe step is to augment the grammar and alter the activelist so that the expansion of the symbols in the active listis the string represented by (p1, l1) . . . (pi+1, li+1).

If pi+1 = 0, we can accomplish this goal by simplyappending li+1 to the active list. If pi+1 6= 0, then it spec-ifies a substring i of the expansion of the active list. Ifi is contained in the expansion of a single symbol in theactive list, then we use ADDSUBSTRING to create a non-terminal with expansion i using O(log |i|) rules. Thisnonterminal is then appended to the active list.

On the other hand, if i is not contained in the expan-sion of a single symbol in the active list, then it is theconcatenation of a suffix of X, all of A1 . . . Ati, and aprefix of Y , where XA1 . . . AtiY are consecutive sym-bols in the active list. We then perform the following op-erations:

8

1. Construct a nonterminal M with expansionA1 . . . Ati using ADDSEQUENCE. The number ofnew rules produced is O(ti(1 + log(|i|/ti))).

2. Replace A1 . . . Ati in the active list by the singlesymbol M .

3. Construct a nonterminal X with expansion equalto the prefix of i in X using ADDSUBSTRING.Similarly, construct a nonterminal Y with expan-sion equal to the suffix of i in Y using ADDSUB-STRING. This produces O(log |i|) new rules in to-tal.

4. Create a nonterminal N with expansion X MY using ADDSEQUENCE on X , M , and Y . This cre-ates O(log |i|) new rules. Append N to the end ofthe active list.

Thus, in total, we add O(ti+ti log(|i|/ti)+log |i|) newrules during each step. The total number of rules createdis:

O

(s

i=1

ti + ti log(|i|/ti) + log |i|)

= O

(s

i=1

ti +

si=1

ti log(|i|/ti) +s

i=1

log |i|)

The first sum is upper bounded by the total number ofsymbols inserted into the active list. This is at most twoper step (M and N ), which implies si=1 ti 2s.To upper bound the second sum, we use the concavityinequality:

si=1

ai log bi (

si=1

ai

)log

(si=1 aibisi=1 ai

)

and set ai = ti, bi = |i|/ti to give:

si=1

ti log|i|ti

(

si=1

ti

)log

(si=1 |i|si=1 ti

)

= O(s log

(ns

))

The latter inequality uses the fact thats

i=1 |i| n andthat

si=1 ti 2s. Note that the function x log(n/x) is

increasing for x up to n/e, and so this inequality holdsonly if 2s n/e. This condition is violated only wheninput string (length n) turns out to be only a small factor(2e) longer than the LZ77 sequence (length s). If we de-tect this special case, then we can output the trivial gram-mar S and achieve a constant approximation ratio.

By concavity again, the third sum is upper bounded bys log

|i|s s log ns and thus total grammar size is at

most:

O(s log

n

s

)= O

(g log

n

g

)

where we use the inequality s g and, again, the ob-servation that x log(n/x) is increasing for x < n/e. Thepreceeding analysis establishes the following theorem.

Theorem 1 The optimum grammar for a string can beapproximated efficiently within a factor of O(log (n/g)),where n is the length of the string and g is the size of thesmallest grammar.

Remark: Universal source codes are of great interestin information theory. Grammar-based codes have beenshown to be universal provided there is a good grammartransform and assuming a finite alphabet [7, 14]. [7] de-fines a grammar-transform to be asymptotically compactif for every string , it produces a grammar G such that|G |/|| = o(1), as || goes to infinity. Though not themotivation for our work, we note that our algorithm isasymptotically compact. To see this, note that for a fi-nite alphabet, the number of pairs in the LZ77 represen-tation of is g = O( nlog n ), where n = || (e.g. Lemma12.10.1, [3]). So g log(n/g) = O(n log log nlog n ) and hence|G |/n = O( log log nlog n ) = o(1). It then follows from Theo-rem 7 in [7] that there is a source code based on our gram-mar transform which is universal with a maximal point-wise redundancy of O( (log log n)

2

log n ).2

2In fact by Theorem 8 in [7], by modifying the transform to givean irreducible grammar a better redundancy of O( log log n

log n) can be

achieved. One way to modify the algorithm is to post-process the gram-mar in a straight-forward way by applying the Reduction Rules in [7](Section VI). This can be done in polynomial time.

9

4 Kolmogorov complexity in relatedmodels

The complexity of a string is a natural question of theo-retical importance. Kolmogorov complexity is an elegantway to define this notion. It is the description length of thesmallest Universal Turing machine program which printsthe string. But it is well-known that this is an uncom-putable measure of complexity. Thus we are led to theidea of defining a computable measure of the complexityof strings. A natural approach to defining such a measureis to restrict the computation model under which the stringis generated.

We take the most natural candidate restrictions: in-stead of Turing machines we restrict ourselves to non-deterministic finite automata and context-free grammars.We make precise the notion of generating strings undersuch a model later. Then we ask the question of whetherthe string complexity under these models can be effi-ciently estimated. Although the problems are known tobe NP-hard, we show that efficient algorithms with goodapproximation ratios exist.

The grammar model, which forms the focus of thiswork, turns out to be closely related to both the modelsmentioned above. The problem cast in the NFA modelreduces to a two-level grammar model, for which wedevelop an O(log2 n) approximation algorithm. In thegrammar model above we used a CFG which generates alanguage containing a single string to define the complex-ity of the string. Another natural possibility, which wecall the advice-grammar model, encodes an input stringwith a CFG (which can potentially generate an infinitenumber of strings) and an advice string. The advice stringspecifies how the production rules of the CFG should beapplied such that only the specified string is produced. Weshow that the complexities in these two models are equiv-alent within a constant factor. Hence our main result ongrammar approximation implies that we can approximatethe CFG-based complexity of a string (either definition)within a logarithmic factor.

4.1 NFA-complexity of a stringA Non-deterministic Finite Automaton (NFA) is specifiedby a set of states and a transition function between the

states. To associate an NFA with a string we consider it asa Mealy machine, which outputs a string on each transi-tion. An NFA is said to generate the string that it outputsduring an entire sequence of transitions from the start stateto a final state. Thus an NFA can generate many strings,and we have to provide an advice to specify which tran-sition to take at each point when multiple transitions arepossible. To measure the complexity of the input string,we count both the NFA cost, which is the total length ofthe strings appearing on the transition edges, and the ad-vice cost which is simply the number of advices required.Having fixed this measure of complexity, it is easy to seethat without loss of generality we can have just one state(apart from a final state) and all the transitions are fromthis state to itself.3 This is identical to a two-level restric-tion of our grammar model- with one rule (for the startsymbol) corresponding to the advice string, and one rulefor each transition.

In a two-level grammar, only the definition of the startsymbol may contain non-terminals; the definitions of allother non-terminals contain only terminals. The prob-lem is to find the smallest two-level grammar for a string.This problem resembles the optimal dictionary problemin which the goal is to find the best possible dictionaryof codewords used to compress the input string. Storer[13] shows that this problem is NP-hard. However, sev-eral practical algorithms have been developed to addressthis problem. In this section, we outline an algorithm witha O(log2 n) approximation ratio.

Theorem 2 The 2-level grammar compression problemis O(log2 n)-approximable.

Proof: The main idea is to recast the two-level gram-mar compression problem as a disjoint set cover problem.Given a string , define the ground set to consist of then positions in the input string such that each positionof contributes a distinct element. For every collectionof non-overlapping instances of a substring of , definea covering set consisting of all the positions contained inthose instances. For example, if the string abc appears

3Note that this is possible only because we charge unit cost for eachadvice, which is unjustified if we consider the bit representation of theadvice. But this affects the complexity measure by at most a factor oflogarithm of the number of different transitions of the NFA.

10

k times without overlap, then we generate 2k 1 cov-ering sets for this substring. The cost of a covering setwith string and some t non-overlapping instances is thelength of plus t. In addition, for each character in , wedefine a covering set with cost one that contains only thatcharacter. It is not hard to see that this reduces the small-est two-level grammar problem to the problem of coveringthe ground set with a minimum cost collection of disjointcovering sets.

If we relax the condition that the cover consist of dis-joint sets, then the cost of the optimum cover can only de-crease. Furthermore, we can obtain an overlapping coverthat has cost at most 1+ lnn times the optimum using thewell-known greedy algorithm for set-cover [2].

This overlapping cover is converted to a two-levelgrammar as follows. First we modify the cover so thatno instance of a string in any covering set is redundant forachieving the cover, by simply removing the redundant in-stances systematically. This ensures that a disjoint cover-ing of can be obtained from the instances in the coveringsets by taking at most one substring of each instance. Nowconsider a covering set (in the modified cover) consistingof instances of a substring . We define a non-terminalfor , non-terminals for the halves of , quarters of andso forth until is partitioned into single characters. Thetotal length of all these definitions generated by is atmost log n times the length of . Now any substring of is expressible using a sequence of at most log n of thesenon-terminals. In particular, for each instance of , wecan express the portion of that instance not covered bypreviously-selected sets. Thus for each instance we addat most log n symbols to the rule of the start-symbol (andto the size of the grammar). Since we incur an approx-imation factor of log n in approximating the cover, theresulting grammar is at most O(log2 n) larger than theoptimum.

All that remains is to describe how the greedy set coveralgorithm is applied. Potentially, there could be exponen-tially many covering sets. To overcome this, at each stepwe consider every distinct substring of . There areat most n2 substrings. For each i from 1 to ||/||, weuse dynamic programming to find the collection of i non-overlapping instances of that cover the greatest num-ber of previously uncovered characters in the ground set.Thus, for each and i we narrow the search to one can-didate covering set. From among the candidate covering

sets, we select the one which maximizes the number ofnew elements covered divided by (||+ i).

4.2 Advice-GrammarInstead of restricting the grammar to produce exactly onestring one could think of a general CFG and a programor advice-string for expanding the CFG such that ex-actly one string is produced. In a general CFG a sin-gle non-terminal may have many production rules andcyclic dependencies are also allowed. With at most aconstant factor increase in the size of the CFG, we re-strict the non-terminals to be of one of the two kinds:concatenation non-terminals, with exactly one binary pro-duction, or choice non-terminals with rules of the formA X0|X1| . . . |XtA , where the Xi are any symbols.We call such a CFG a canonical CFG.

Advice-grammar CA consists of a CFG C and anadvice-string of integers, A. To expand the advice-grammar, one starts off expanding the CFG depth-first,but on encountering a choice non-terminal the next inte-ger from the advice-string is read and used to choose theproduction to be used. We define the complexity of theadvice-grammar as |CA| = |C| + |A|, where |C| is thenumber of non-terminals in C and |A| is the length of theadvice-string A.

Remarkably, as we show now, this seemingly morepowerful model is not more than a constant factor moreefficient than the grammar we have been considering.

Define the partial expansion of a concatenation non-terminal as an expansion till terminals or choice non-terminals. Note that there cannot be any production loopwithout a choice non-terminal involved, and so the par-tial expansion gives rise to a finite tree, called the partialexpansion tree (PET for short), with terminals and choicenon-terminals as the leaves (the latter will be referred to asthe choice leaves of the PET). Two non-terminals are saidto be equivalent if they have the same partial expansion.A canonical CFG G is a transform of a canonical CFG Gif they have the same terminals and choice non-terminalsand for every concatenation non-terminal in G there is anequivalent non-terminal in G , and the choice rules in G are derived from that in G by replacing any concatenationnon-terminal by its equivalent in G .

11

Lemma 2 Suppose C is a canonical CFG with |C| = g.Then we can construct a transform of C, C such that|C| = O(g) and every concatenation non-terminal A inC satisfies the following: If more than one choice non-terminal appears in the

partial expansion of A, then A XY where bothX and Y have choice non-terminals in their partialexpansions.

If A has only one choice non-terminal in its partialexpansion, then the corresponding choice leaf ap-pears in the PET of A at a depth of at most 2.

Then, in As PET, the number of nodes in the spanningsub-tree containing the root and all the choice leaves in itis at most a constant times the number of choice leaves.

In the appendix we prove the above Lemma and use itto prove the following:

Theorem 3 Suppose CA is an advice-grammar, with|C| = g and |A| = a. Then there exists a grammar Gwith |G| = O(g + a), producing the same string as CA.

4.3 Edit GrammarsThe fact that both the natural variants of CFGs are equiva-lent suggests the robustness of grammar based string com-plexity. We further explore the robustness of the model,by allowing edit-rules into the grammar. We show thatthis extended model can affect the complexity of the stringby at most a logarithmic factor. Further our algorithmgives a logarithmic approximation under this model too.

An edit-rule is a rule of the form A X [editop],where X is a symbol (which might in turn be a non-terminal defined by an edit-rule), and editop is a singleedit operation; we restrict ourselves to the following threekinds of edit operations: insert, delete and replace a char-acter. The edit operation specifies the position in Xwhere the edit is to be performed, and in case of insert andreplace, the character to be introduced.4 The semantics of

4It is sometimes reasonable to define other edit operations. For in-stance one could allow a prefix/suffix operation. In this case the result-ing edit-grammar becomes as powerful as LZ77 representation, within aconstant factor. Or, one could allow insert/replace to introduce arbitrarysymbols instead of terminals, which doesnt change the model signifi-cantly.

the edit operation is the natural one: A is obtained byperforming editop on X. An edit rule incurs unit cost.An edit-grammar is called binary or balanced dependingon the non-edit (concatenation) rules in it.

We show that edit-grammars are well-approximated byusual grammars. For this we introduce a representationof the string called edit-LZ77. In the edit-LZ77 represen-tation of a string, instead of pairs as in LZ77, we havetriplets (pi, li, [editlist]). editlist consists of a list ofprimitive operations (insert/delete/replace) to be appliedto the sub-string generated by the (pi, li) part of the triplet,one after the other. Each primitive operation will pointto a location in the sub-string (which might have alreadybeen modified by the preceding operations in the list) andprescribe an insert, delete or replace operation. The edit-cost of a triplet is the number of edit operations in thetriplets list.

Along the lines of Lemma 1 we get the followinglemma.

Lemma 3 If G is a binary edit-grammar with g concate-nation rules and k edit rules, then there is an edit-LZ77representationL of the string generated by G, with at mostg + 1 triplets and a total edit cost of k.

We observe that edit-LZ77 is approximated within aconstant factor by the usual LZ77.

Lemma 4 If L is an edit-LZ77 list for a string , with ttriplets and a total edit cost of k, then there is an LZ77 listL for with at most t + 2k pairs.

Proof: We can replace a triplet (pi, li, [editlist]) whereeditlist has ki operations, by at most 1 + 2ki pairs: atmost 1 + ki pairs to refer to the sub-strings into whichthe ki edit operations split the sub-string generated by thetriplet, and at most ki pairs to represent the inserts andreplaces.

From the above two lemmas and the procedure fromSection 3.3 we conclude the following.

Theorem 4 The optimum edit-grammar for a string canbe approximated efficiently within a factor of O(log n) bya grammar, where n is the length of the string.

12

5 Future WorkThe smallest grammar problem has a theoretically inter-esting connection to Kolmogorov complexity, practicalrelevance in the areas of data compression and patternextraction, and a hierarchical structure that is evident inother real-world problems.

One line of research leading from this problem is thestudy of string complexity with respect to other natu-ral models. For example, the grammar model could beextended to allow a non-terminal to take a parameter.One could then write a rule such as T (P ) PP , andwrite the string xxyzyz as T (x)T (yz). Presumably asmodel power increases, approximability decays to un-computability.

The O(log ng )-approximation algorithm presented hereruns in near-linear time and hence can be used in a po-tentially practical grammar-based compression algorithm.Empirical study is required to determine its effectivenesscompared to other compressors. On the theoretical front,it would be interesting to explore new probabilistic mod-els of sources for which it performs well.

Also, while an o(

log nlog log n

)approximation algorithm

for the smallest grammar problem would require progresson the well-studied addition chain problem, it is onlyknown that the optimal grammar cannot be approximatedto within a small constant unless P = NP. Thus, nailingdown the actual hardness of the smallest grammar prob-lem remains an intriguing open problem.

Finally, many other important real-world problemsshare the hierarchical structure of the smallest grammarproblem. For example, suppose one must design a digitalcircuit that computes the logical AND of various, spec-ified subsets of the input signals. How many two-inputAND gates suffice? This amounts to a variant of thesmallest grammar problem where non-terminals representsets rather than strings. We know of neither an approxi-mation algorithm nor hardness of approximation result forthis natural question.

AcknowledgmentsWe sincerely thank Yevgeniy Dodis, Martin Farach-Colton, Michael Mitzenmacher, Madhu Sudan and theanonymous reviewers.

References[1] A. Apostolico and S. Lonardi. Some theory and

practice of greedy off-line textual substitution. InJ. A. Storer and M. Cohn, editors, Data Compres-sion Conference, pages 119128, Snowbird, Utah,1998.

[2] V. Chvatal. A greedy heuristic for the set-coveringproblem. Mathematics of Operations Research,4(3):233235, 1979.

[3] T. M. Cover and J. A. Thomas. Elements of Infor-mation Theory. John Wiley & Sons, New York, NY,USA, 1991.

[4] C. de Marcken. Unsupervised Language Acquisi-tion. PhD thesis, Massachusetts Institute of Tech-nology, 1996.

[5] M. Farach-Colton. Personal communication.

[6] T. Kida, Y. Shibata, M. Takeda, A. Shinohara, andS. Arikawa. A unifying framework for compressedpattern matching. In SPIRE/CRIWG, pages 8996,1999.

[7] J. C. Kieffer and E. hui Yang. Grammar-basedcodes: A new class of universal lossless sourcecodes. IEEE Transactions on Information Theory,46(3):737754, 2000.

[8] J. C. Kieffer, E. hui Yang, G. J. Nelson, and P. Cos-man. Universal lossless compression via multilevelpattern matching. IEEE Transactions on Informa-tion Theory, 46(4):12271245, 2000.

[9] N. J. Larsson and A. Moffat. Offline dictionary-based compression. In Data Compression Confer-ence, pages 296305, 1999.

[10] E. Lehman and A. Shelat. Approximations algo-rithms for grammar-based compression. In Thir-teenth Annual Symposium on Discrete Algorithms(SODA02), 2002.

[11] C. Nevill-Manning. Inferring Sequential Structure.PhD thesis, University of Waikato, 1996.

13

[12] C. G. Nevill-Manning and I. H. Witten. Compres-sion and explanation using hierarchical grammars.The Computer Journal, 40(2/3):103116, 1997.

[13] J. A. Storer. Data Compression: Methods and Com-plexity Issues. PhD thesis, Princeton University,1979.

[14] E. h. Yang and J. C. Kieffer. Efficient universal loss-less data compression algorithms based on a greedysequential grammar transformpart one: Withoutcontext models. IEEE Transactions on InformationTheory, 46(3):755777, 2000.

[15] J. Ziv and A. Lempel. A universal algorithm for se-quential data compression. IEEE Transactions onInformation Theory, IT-23(3):337343, 1977.

[16] J. Ziv and A. Lempel. Compression of individualsequences via variable-rate coding. IEEE Transac-tions on Information Theory, IT-24:530536, 1978.

A.1 Proof of Theorem 3First we prove Lemma 2.

PROOF of LEMMA 2. Non-terminals whose partial ex-pansions contain only terminals are copied as such into C .We describe how to introduce equivalent non-terminals inC for each of the remaining concatenation non-terminalin C by adding at most a constant number of (concate-nation) rules. We process the remaining concatenationnon-terminals in C (which contain at least one choice non-terminal in their partial expansion) in increasing order ofthe size of (number of symbols in) their partial expan-sions. For all non-terminals A that we process we producean equivalent non-terminal A in C ensuring the follow-ing (in addition to the condition in the lemma): In thePET of A, the depth of both the leftmost choice leaf andthe rightmost choice leaf is at most 3.

We proceed inductively. Consider processing A XY . If neither X nor Y had to be processed before, wejust set A = A and add the rule A XY (this is thebase case). If X (resp Y ) is a concatenation non-terminalin C we would have already processed it and added anequivalent non-terminal X (resp Y ) to C. Consider sim-ply making a concatenation rule X Y . This may havethe (at most) two choice leaves of interest- the leftmost

!!!

!!!

!!!

!!!

"""

"""

"""

"""

##

##

##

$$

$$

$$

X

A

Y

Figure 1: Making a new rule in the proof of Lemma 2. Theshaded sub-trees are the choice leaves of interest.

and the rightmost ones- at a depth of at most four in theresulting PET. Figure 1 shows a general scenario, with amaximum of eight possible sub-trees, two of which arethe leaves of interest. We can arrange the (at most) eightsub-trees into a tree such that the leaves are at depth threeor higher, and they fall on either side of the root. Thiscan be done by making at most 7 new rules, the one atthe root being A. If there is only one leaf of interest,it can be pushed up to depth 2. When we finish pro-cessing the concatenation non-terminals in C, we com-plete the construction of C by adding rules for each ofthe choice non-terminals from C (replacing any concate-nation non-terminal appearing in the RHS by equivalentnon-terminals).

PROOF of THEOREM 3. By Lemma 2 we assumew.l.o.g that C satisfies the condition in the lemma, with|C| = O(g). A symbol is called choice-free if its partialexpansion has only terminals. First we add all rules forchoice-free non-terminals in C to G. There are O(g) suchrules. Figure A.1 gives a recursive algorithm to add otherrules to G; we call this algorithm on the start symbol of C.

It is easy to see that in the resulting grammar G, thenon-terminal returned by the top-level call to DE-ADVICEexpands to the string produced by CA. We claim that thenumber of rules added to G by this call is O(a). Notethat new rules are produced at nodes in C which have atleast one choice leaf below it in the PET of the last choicenon-terminal occurring in the recursion; that is, the nodesat which new rules are produced are exactly the ones inthe spanning sub-tree of a PET which spans the root ofthe PET and the choice leaves. At each choice leaf in

14

Procedure DE-ADVICE(X){ i is a global variable, initialized to 0}if the symbol X is choice-free then

return X; { No new rule is produced}else if X is a choice non-terminal then

Let X X0|X1| . . . |XtXb = Aii = i + 1

return DE-ADVICE(Xb) { No new rule is produced}else

Let X Y ZM := DE-ADVICE(Y )P := DE-ADVICE(Z)Produce new rule N MP {a new non-terminal is createdhere}return N

each occurrence of a PET in the recursion, an entry of theadvice string is consumed, and hence the total number ofchoice points is a. So by Lemma 2 the number of rulesadded by DE-ADVICE is O(a). So the total number ofrules in G is O(g + a).

15