Part 3: Maximally recoverable local

reconstruction codes

Lingfei Jin

2021 IEEE East Asian School of Information Theory

04/08/2021

Outline

1 Backgrounds

2 State-of-art results

3 Constructions via rational function fields

2

Local repairing via global repairing

• However, an LRC achieving the Singleton-type bound doesn’t

necessarily repair local and global errors simultaneously.

• A much stronger requirement on global fault-tolerance, called

Maximal Recoverability was introduced in [1], motivated by

applications to storage on solid-state devices.

• The terminology maximally recoverable codes was coined in

[2]

[1] M. Blaum, J. Lee Hafner and S. Hetzler, Partial-MDS codes and their application to RAID type ofarchitectures, IEEE TIT, 2013.

[2] P. Gopalan, C. Huang, B. Jenkins and S. Yekhanin, Explicit maximally recoverable codes with locality,IEEE TIT, 2014.

3

Local repairing via global repairing

• However, an LRC achieving the Singleton-type bound doesn’t

necessarily repair local and global errors simultaneously.

• A much stronger requirement on global fault-tolerance, called

Maximal Recoverability was introduced in [1], motivated by

applications to storage on solid-state devices.

• The terminology maximally recoverable codes was coined in

[2]

[1] M. Blaum, J. Lee Hafner and S. Hetzler, Partial-MDS codes and their application to RAID type ofarchitectures, IEEE TIT, 2013.

[2] P. Gopalan, C. Huang, B. Jenkins and S. Yekhanin, Explicit maximally recoverable codes with locality,IEEE TIT, 2014.

3

Local repairing via global repairing

• However, an LRC achieving the Singleton-type bound doesn’t

necessarily repair local and global errors simultaneously.

• A much stronger requirement on global fault-tolerance, called

Maximal Recoverability was introduced in [1], motivated by

applications to storage on solid-state devices.

• The terminology maximally recoverable codes was coined in

[2]

[1] M. Blaum, J. Lee Hafner and S. Hetzler, Partial-MDS codes and their application to RAID type ofarchitectures, IEEE TIT, 2013.

[2] P. Gopalan, C. Huang, B. Jenkins and S. Yekhanin, Explicit maximally recoverable codes with locality,IEEE TIT, 2014.

3

Maximally recoverable local reconstruction codes

An maximally recoverable (n, r , h, a)`-local reconstruction code (or

MR (n, r , h, a)`-LRC) with n = gr and k = n − ga − h has a

generator matrix of the form

G = (B1|B2| · · · |Bg) ∈ Fk×n`

satisfying

• each Bi has size k × r ;• the row span of each Bi is an [r , r − a, a + 1]`-MDS code for

1 ≤ i ≤ g (note that Bi is not a generator matrix of this MDS

code in general);• after puncturing a columns from each Bi , the remaining

matrix of G generates an [n − ga, k, h + 1]`-MDS code.

4

Maximally recoverable local reconstruction codes

An maximally recoverable (n, r , h, a)`-local reconstruction code (or

MR (n, r , h, a)`-LRC) with n = gr and k = n − ga − h has a

generator matrix of the form

G = (B1|B2| · · · |Bg) ∈ Fk×n`

satisfying

• each Bi has size k × r ;

• the row span of each Bi is an [r , r − a, a + 1]`-MDS code for

1 ≤ i ≤ g (note that Bi is not a generator matrix of this MDS

code in general);• after puncturing a columns from each Bi , the remaining

matrix of G generates an [n − ga, k, h + 1]`-MDS code.

4

Maximally recoverable local reconstruction codes

An maximally recoverable (n, r , h, a)`-local reconstruction code (or

MR (n, r , h, a)`-LRC) with n = gr and k = n − ga − h has a

generator matrix of the form

G = (B1|B2| · · · |Bg) ∈ Fk×n`

satisfying

• each Bi has size k × r ;• the row span of each Bi is an [r , r − a, a + 1]`-MDS code for

1 ≤ i ≤ g (note that Bi is not a generator matrix of this MDS

code in general);

• after puncturing a columns from each Bi , the remaining

matrix of G generates an [n − ga, k, h + 1]`-MDS code.

4

Maximally recoverable local reconstruction codes

An maximally recoverable (n, r , h, a)`-local reconstruction code (or

MR (n, r , h, a)`-LRC) with n = gr and k = n − ga − h has a

generator matrix of the form

G = (B1|B2| · · · |Bg) ∈ Fk×n`

satisfying

• each Bi has size k × r ;• the row span of each Bi is an [r , r − a, a + 1]`-MDS code for

1 ≤ i ≤ g (note that Bi is not a generator matrix of this MDS

code in general);• after puncturing a columns from each Bi , the remaining

matrix of G generates an [n − ga, k, h + 1]`-MDS code.4

Repairing erasure errors of MR LRC

An MR (n, r , h, a)`-LRC can correct

• h erasure errors at arbitrary positions, together with

• a erasure errors in each of g groups

• An MR LRC repair local and global errors simultaneously.

↓ ↓ ↓ ↓ ↓

A1 A2 X B1 X B3 X X C3

This is an MR (9, 3, 1, 1)-LRC.

• First repair C1 from A1,A2,B1,B3,C3.

• Then repair A3 from A1 and A2, B2 from B1 and B3, C2

from C1 and C3, respectively.

5

Repairing erasure errors of MR LRC

An MR (n, r , h, a)`-LRC can correct

• h erasure errors at arbitrary positions, together with

• a erasure errors in each of g groups

• An MR LRC repair local and global errors simultaneously.

↓ ↓ ↓ ↓ ↓

A1 A2 X B1 X B3 X X C3

This is an MR (9, 3, 1, 1)-LRC.

• First repair C1 from A1,A2,B1,B3,C3.

• Then repair A3 from A1 and A2, B2 from B1 and B3, C2

from C1 and C3, respectively.

5

Repairing erasure errors of MR LRC

An MR (n, r , h, a)`-LRC can correct

• h erasure errors at arbitrary positions, together with

• a erasure errors in each of g groups

• An MR LRC repair local and global errors simultaneously.

↓ ↓ ↓ ↓ ↓

A1 A2 X B1 X B3 X X C3

This is an MR (9, 3, 1, 1)-LRC.

• First repair C1 from A1,A2,B1,B3,C3.

• Then repair A3 from A1 and A2, B2 from B1 and B3, C2

from C1 and C3, respectively.5

A characterization

• A matrix

G = (B1|B2| · · · |Bg) ∈ Fk×n` (1)

is a generator matrix of an MR (n, r , h, a)`-LRC if and only if

(i) for every Bi , every k × (r − a) submatrix has rank r − a;

(ii)every k × k submatrix of G with at most r − a columns

per block Bi is invertible,i.e.,

• for any submatrices Ai ∈ Fk×ki` of Bi with ki ≤ r − a and∑g

i=1 ki = k, the matrix (A1|A2| · · · |Ag) ∈ Fk×k` is invertible.

6

A characterization

• A matrix

G = (B1|B2| · · · |Bg) ∈ Fk×n` (1)

is a generator matrix of an MR (n, r , h, a)`-LRC if and only if

(i) for every Bi , every k × (r − a) submatrix has rank r − a;

(ii)every k × k submatrix of G with at most r − a columns

per block Bi is invertible,i.e.,

• for any submatrices Ai ∈ Fk×ki` of Bi with ki ≤ r − a and∑g

i=1 ki = k, the matrix (A1|A2| · · · |Ag) ∈ Fk×k` is invertible.

6

An equivalent definition

An `-ary [n, k]-linear code with a parity-check matrix of the form

H =

A1 O · · · OO A2 · · · O...

.... . .

...O O · · · Ag

D1 D2 · · · Dg

∈ F(n−k)×n

` (2)

is called an MR (n, r , h, a)`-LRC if Skip parity-matrix of RFF Skip Theorem 2

• each Ai has size a × r and each Di has size h × r ;• each Ai generates an [r , a, r − a + 1]`-MDS code• every ag + h columns consisting of any a columns in each

group and other arbitrary h columns are F`-linearly

independent. 7

An equivalent definition

To see the equivalence

• each Ai is actually a parity-check matrix of the code

generated by Bi

• We can construct MR LRCs from either of the definitions.

However, the main results of this talk come from the

constructions based on parity-check matrices of the required

form.

8

An equivalent definition

To see the equivalence

• each Ai is actually a parity-check matrix of the code

generated by Bi

• We can construct MR LRCs from either of the definitions.

However, the main results of this talk come from the

constructions based on parity-check matrices of the required

form.

8

Outline

1 Backgrounds

2 State-of-art results

3 Constructions via rational function fields

9

Goal

• Since encoding a linear code and decoding it from erasures

involve performing numerous finite field arithmetic operations,

it is highly desirable to have codes over small fields (preferably

of characteristic 2).

• Obtaining MR LRCs over finite fields of minimal size has

therefore emerged as a central problem in the area of codes

for distributed storage.

10

Goal

• Since encoding a linear code and decoding it from erasures

involve performing numerous finite field arithmetic operations,

it is highly desirable to have codes over small fields (preferably

of characteristic 2).

• Obtaining MR LRCs over finite fields of minimal size has

therefore emerged as a central problem in the area of codes

for distributed storage.

10

Known results

Field size is denoted by `

Blaum-Hafner-Hetzler, 2013 Gopi-Guruswami-Yekhanin, 2017

↓ ↓

for h ≤ 1, ` = O(r) for h = 2, ` = O(n)

for h = 3, ` = O(n3)

[1] M. Blaum, J. Lee Hafner and S. Hetzler, Partial-MDS codes and their application to RAID type ofarchitectures, IEEE TIT, 2013.

[2] S. Gopi, V. Guruswami and S. Yekhanin, On maximally recoverable local reconstruction codes, SODA2019.

11

Known results

For other h:

Gabrys-Yaakobi-Blaum-Siegel, 2017

` = O(r · n(a+1)h−1

)` = max

O(n

r ), O(r)h+ah

• The bound at the left outperforms the bound at the right when

r = Ω(n), while the bound at the right is better when r n. In both

the bounds, the field size grows exponentially with h and a.

Skip consequence of Theorem 2a Skip consequence of Theorem 2b Skip consequence of Theorem 3a

[1] R. Gabrys, E. Yaakobi, M. Blaum and P. Siegel, Construction of partial MDS codes over small finitefields ISIT, 2017.

12

Known results

Neri, Horlemann-Trautmann, 2018

` = O(

rn(r−a)

r

)` = O

((n−1k−1))

= 2O(n)

↑ ↑

via rank-metric codes via probabilistic arguments

• When r = Ω(n), and a is close to r or h is large, this bound is better

than the bounds in the previous slide.

Skip consequence of Theorem 1

[1] A. Neri, A.-L. Horlemann-Trautmann, Random Construction of Partial MDS Codes, arXiv, 2018

13

Known results

A lower bound: an (n, r , h, a)` MR LRC must obey

Gopi, V. Guruswami and S. Yekhanin, 2019

↓

` = Ωa,h(n · rmina,h−2

)↑

via combinatorial arguments

• The lower bound is still quite far from the upper the bounds.

[1] S. Gopi, V. Guruswami and S. Yekhanin, On maximally recoverable local reconstruction codes, SODA2019.

14

Outline

1 Backgrounds

2 State-of-art results

3 Constructions via rational function fields

15

Moore matrix

Let ` be a power of q. For elements α1, . . . , αh ∈ F`, the Moore

matrix is defined by

M =

α1 α2 · · · αh

αq1 αq

2 · · · αqh

...... . . . ...

αqh−1

1 αqh−1

2 · · · αqh−1

h

∈ Fh×h

` .

Then α1, α2, . . . , αh are Fq-linearly independent ⇔ det(M ) 6= 0

16

Constructions via generator matrix

• Choose g ≤⌈

qm

m

⌉polynomials p1(x), p2(x), . . . , pg(x) ∈ Fq [x]

of degree m such that gcd(pi(x), pj(x)) = 1

• We can form an Fq-vector space

Vi := f (x)

pi(x) : f (x) ∈ Fq [x], deg(f (x)) ≤ m − 1

of dimension m.

• Find r functions fi1(x), . . . , fir(x) ∈ Vi such that any r − a

polynomials out of fi1(x), . . . , fir(x) are Fq-linearly

independent.

17

Constructions via generator matrix

• Choose g ≤⌈

qm

m

⌉polynomials p1(x), p2(x), . . . , pg(x) ∈ Fq [x]

of degree m such that gcd(pi(x), pj(x)) = 1

• We can form an Fq-vector space

Vi := f (x)

pi(x) : f (x) ∈ Fq [x], deg(f (x)) ≤ m − 1

of dimension m.

• Find r functions fi1(x), . . . , fir(x) ∈ Vi such that any r − a

polynomials out of fi1(x), . . . , fir(x) are Fq-linearly

independent.

17

Constructions via generator matrix

• Choose g ≤⌈

qm

m

⌉polynomials p1(x), p2(x), . . . , pg(x) ∈ Fq [x]

of degree m such that gcd(pi(x), pj(x)) = 1

• We can form an Fq-vector space

Vi := f (x)

pi(x) : f (x) ∈ Fq [x], deg(f (x)) ≤ m − 1

of dimension m.

• Find r functions fi1(x), . . . , fir(x) ∈ Vi such that any r − a

polynomials out of fi1(x), . . . , fir(x) are Fq-linearly

independent.

17

Constructions via generator matrix

p1(x) p2(x) ... pg(x)

↓ ↓ ... ↓

V1 V2 ... Vg

↓ ↓ ... ↓

f11, . . . , f1r f21, . . . , f2r ... fg1, . . . , fgr

Assume that m < r and there is a q-ary

[r , r −m,≥ r − a + 1]-linear code. Then we can require that any

r − a polynomials in each block are Fq-linearly independent.

18

Constructions via generator matrix

Choose an irreducible polynomial Q(x) ∈ Fq [x] of degree

minkm, gm = min

km, nmr

= min

(n − an

r − h)m, nmr

.

Put

` := qmin(n− anr −h)m, nm

r = qminkm, nmr

For a polynomial f (x) ∈ Fq [x], denote by f (Q) the residue class of

f (x) modulo Q(x), i.e.,

f (Q) ∈ Fq [x]/Q ' Fqdeg(Q) = F`.

19

Constructions via generator matrix

Choose an irreducible polynomial Q(x) ∈ Fq [x] of degree

minkm, gm = min

km, nmr

= min

(n − an

r − h)m, nmr

.

Put

` := qmin(n− anr −h)m, nm

r = qminkm, nmr

For a polynomial f (x) ∈ Fq [x], denote by f (Q) the residue class of

f (x) modulo Q(x), i.e.,

f (Q) ∈ Fq [x]/Q ' Fqdeg(Q) = F`.

19

Constructions via generator matrix

Choose an irreducible polynomial Q(x) ∈ Fq [x] of degree

minkm, gm = min

km, nmr

= min

(n − an

r − h)m, nmr

.

Put

` := qmin(n− anr −h)m, nm

r = qminkm, nmr

For a polynomial f (x) ∈ Fq [x], denote by f (Q) the residue class of

f (x) modulo Q(x), i.e.,

f (Q) ∈ Fq [x]/Q ' Fqdeg(Q) = F`.

19

Constructions via generator matrix

Define the k × r matrix Bi as follows.

Bi =

fi1(Q) fi2(Q) · · · fir(Q)

f qi1(Q) f q

i2(Q) · · · f qir(Q)

......

......

f qk−1

i1 (Q) f qk−1

i2 (Q) · · · f qk−1

ir (Q)

∈ Fk×r

` . (3)

20

Constructions via generator matrix

Theorem 1The `-ary code C with the generator matrix

G = (B1|B2| · · · |Bg) ∈ Fk×n` is an MR (n, r , h, a)`-LRC.

• Idea of proof: By the previous characterization, we have to

show that (i)each Bi generate an MDS code; (ii) every k × k

submatrix A of G with at most r − a columns per block Bi is

invertible.

• This is true due to a fact about the Moore matrix.

21

Constructions via generator matrix

Theorem 1The `-ary code C with the generator matrix

G = (B1|B2| · · · |Bg) ∈ Fk×n` is an MR (n, r , h, a)`-LRC.

• Idea of proof: By the previous characterization, we have to

show that (i)each Bi generate an MDS code; (ii) every k × k

submatrix A of G with at most r − a columns per block Bi is

invertible.

• This is true due to a fact about the Moore matrix.

21

Constructions via generator matrix

Theorem 1The `-ary code C with the generator matrix

G = (B1|B2| · · · |Bg) ∈ Fk×n` is an MR (n, r , h, a)`-LRC.

• Idea of proof: By the previous characterization, we have to

show that (i)each Bi generate an MDS code; (ii) every k × k

submatrix A of G with at most r − a columns per block Bi is

invertible.

• This is true due to a fact about the Moore matrix.

21

Some consequences of Theorem 1

Theorem 1

` ≤

2minrk,n ≤ 2n if r ≥ log n2minkdlog ne, n

r dlog ne if r ≤ log n` ≤ 2r n

r br−a

2 c

• The probabilistic bound gives the same size as the bound at the left.

• The bound at the right clearly outperforms the bound obtained viarank-metric codes when k < n/r . Skip prob bound

22

Constructions via parity-check matrix

• For any Choose g ≤⌈

qm

m

⌉polynomials

p1(x), p2(x), . . . , pg(x) ∈ Fq [x] of degree m such that

gcd(pi(x), pj(x)) = 1

• we can form an Fq-vector space

Vi := f (x)

pi(x) : f (x) ∈ Fq [x], deg(f (x)) ≤ m − 1

of dimension m.

• Find r functions fi1(x), . . . , fir(x) ∈ Vi such that any h + a

polynomials out of fi1(x), . . . , fir(x) are Fq-linearly

independent.

23

Constructions via parity-check matrix

• For any Choose g ≤⌈

qm

m

⌉polynomials

p1(x), p2(x), . . . , pg(x) ∈ Fq [x] of degree m such that

gcd(pi(x), pj(x)) = 1

• we can form an Fq-vector space

Vi := f (x)

pi(x) : f (x) ∈ Fq [x], deg(f (x)) ≤ m − 1

of dimension m.

• Find r functions fi1(x), . . . , fir(x) ∈ Vi such that any h + a

polynomials out of fi1(x), . . . , fir(x) are Fq-linearly

independent.

23

Constructions via parity-check matrix

• For any Choose g ≤⌈

qm

m

⌉polynomials

p1(x), p2(x), . . . , pg(x) ∈ Fq [x] of degree m such that

gcd(pi(x), pj(x)) = 1

• we can form an Fq-vector space

Vi := f (x)

pi(x) : f (x) ∈ Fq [x], deg(f (x)) ≤ m − 1

of dimension m.

• Find r functions fi1(x), . . . , fir(x) ∈ Vi such that any h + a

polynomials out of fi1(x), . . . , fir(x) are Fq-linearly

independent.

23

Constructions via parity-check matrix

p1(x) p2(x) ... pg(x)

↓ ↓ ... ↓

V1 V2 ... Vg

↓ ↓ ... ↓

f11, . . . , f1r f21, . . . , f2r ... fg1, . . . , fgr

Assume that m < r and there is a q-ary

[r , r −m,≥ h + a + 1]-linear code. Then we can require that any

h + a polynomials in each block are Fq-linearly independent.Skip construction via FF

24

Constructions via parity-check matrix

Choose an irreducible polynomial Q(x) ∈ Fq [x] of degree

minhm, gm = min

hm, nmr

= min

(n − an

r − h)m, nmr

.

Put

` := qmin(n− anr −h)m, nm

r = qminhm, nmr

For an polynomial f (x) ∈ Fq [x], denote by f (Q) the residue class

of f (x) modulo Q(x), i.e.,

f (Q) ∈ Fq [x]/Q ' Fqdeg(Q) = F`.

25

Constructions via parity-check matrix

Choose an irreducible polynomial Q(x) ∈ Fq [x] of degree

minhm, gm = min

hm, nmr

= min

(n − an

r − h)m, nmr

.

Put

` := qmin(n− anr −h)m, nm

r = qminhm, nmr

For an polynomial f (x) ∈ Fq [x], denote by f (Q) the residue class

of f (x) modulo Q(x), i.e.,

f (Q) ∈ Fq [x]/Q ' Fqdeg(Q) = F`.

25

Constructions via parity-check matrix

Choose an irreducible polynomial Q(x) ∈ Fq [x] of degree

minhm, gm = min

hm, nmr

= min

(n − an

r − h)m, nmr

.

Put

` := qmin(n− anr −h)m, nm

r = qminhm, nmr

For an polynomial f (x) ∈ Fq [x], denote by f (Q) the residue class

of f (x) modulo Q(x), i.e.,

f (Q) ∈ Fq [x]/Q ' Fqdeg(Q) = F`.

25

Constructions via parity-check matrix

• Define the h × r matrix Di as follows:

Di =

fi1(Q) fi2(Q) · · · fir(Q)

f qi1(Q) f q

i2(Q) · · · f qir(Q)

......

......

f qh−1

i1 (Q) f qh−1

i2 (Q) · · · f qh−1

ir (Q)

∈ Fh×r

` .

• Let Ai ∈ Fa×rq be a generator matrix of an [r , a]q-MDS code.

Then we obtain a parity-check matrix H of the form (2).Skip equivalent definition Skip parity of FF

26

Constructions via parity-check matrix

• Define the h × r matrix Di as follows:

Di =

fi1(Q) fi2(Q) · · · fir(Q)

f qi1(Q) f q

i2(Q) · · · f qir(Q)

......

......

f qh−1

i1 (Q) f qh−1

i2 (Q) · · · f qh−1

ir (Q)

∈ Fh×r

` .

• Let Ai ∈ Fa×rq be a generator matrix of an [r , a]q-MDS code.

Then we obtain a parity-check matrix H of the form (2).Skip equivalent definition Skip parity of FF

26

Constructions via parity-check matrix

Theorem 2Let r , g, a, h,m be positive integers with a ≤ r . Suppose that

q ≥ r is a prime power satisfying qm ≥ mnr and there is a q-ary

[r , r − a, a + 1]-linear code. If (i) m ≥ r ; or (ii) m < r and there

exists a q-ary [r , r −m,≥ h + a + 1]-linear code, then the code

with H defined in (2) is an MR (n, r , h, a)-LRC with n = rg over a

field of size ` = qminhm, nmr .

• Idea of proof: By the equivalence Definition, we can show

certain linear independence via Moore Matrix.

Skip equivalent definition

27

Some consequences of Theorem 2

For a = 1:

Theorem 2

` ≤(

max

O( nr ), (2r)b

h+12 c)minh, n

r ` ≤

(max

O( n

r ), 2r)minh, nr

• The bound at the left outperforms the second bound byGabrys-Yaakobi-Blaum-Siegel due to the quadratically better exponent forr . Skip general bound

• The bound at the right outperforms the bound at the left forr

log r <⌊ h+1

2

⌋.

28

Some consequences of Theorem 2

For a > 1:

Theorem 2

` ≤(max

O( n

r ), (2r)h+a)minh, nr ` ≤

(max

O( n

r ), (2r)r)minh, nr

• The bound at the left outperforms the bound the second bound byGabrys-Yaakobi-Blaum-Siegel for h > n

r . Skip general bound

• The bound at the right outperforms the bound at the left for r < h + a,and hence it beats the bound by Gabrys-Yaakobi-Blaum-Siegel fornh < r < h + a.

29

Future research

The known results are far from satisfaction. In particular:

• There is a big gap between lower bounds and upper bounds.

How to bridge this gap?

• Is it possible to improve the lower bounds (or upper bounds)

significantly?

• Can we find some new tools for constructions of MR LRCs

(although there are various constructions, none of them can

outperform the rest for all parameter regime).

30

Future research

The known results are far from satisfaction. In particular:

• There is a big gap between lower bounds and upper bounds.

How to bridge this gap?

• Is it possible to improve the lower bounds (or upper bounds)

significantly?

• Can we find some new tools for constructions of MR LRCs

(although there are various constructions, none of them can

outperform the rest for all parameter regime).

30

Future research

The known results are far from satisfaction. In particular:

• There is a big gap between lower bounds and upper bounds.

How to bridge this gap?

• Is it possible to improve the lower bounds (or upper bounds)

significantly?

• Can we find some new tools for constructions of MR LRCs

(although there are various constructions, none of them can

outperform the rest for all parameter regime).

30

Future research

The known results are far from satisfaction. In particular:

• There is a big gap between lower bounds and upper bounds.

How to bridge this gap?

• Is it possible to improve the lower bounds (or upper bounds)

significantly?

• Can we find some new tools for constructions of MR LRCs

(although there are various constructions, none of them can

outperform the rest for all parameter regime).

30