Sampling and Soundness: Can We Have Both?

Carla Gomes, Bart Selman, Ashish Sabharwal

Cornell University

Jörg Hoffmann

DERI Innsbruck

…and I am: Frank van Harmelen

Nov 11, 2007 ISWC’07 2

Talk Roadmap

A Sampling Method with a Correctness Guarantee Can we apply this to the Semantic Web? Discussion

Nov 11, 2007 ISWC’07 3

How Might One Count?

Problem characteristics:

Space naturally divided into rows, columns, sections, …

Many seats empty

Uneven distribution of people (e.g. more near door, aisles, front, etc.)

How many people are present in the hall?

Nov 11, 2007 ISWC’07 4

#1: Brute-Force Counting

Idea: Go through every seat If occupied, increment counter

Advantage: Simplicity, accuracy

Drawback: Scalability

Nov 11, 2007 ISWC’07 5

#2: Branch-and-Bound (DPLL-style)

Idea: Split space into sections

e.g. front/back, left/right/ctr, … Use smart detection of

full/empty sections Add up all partial counts

Advantage: Relatively faster, exact

Drawback: Still “accounts for” every single

person present: need extremely fine granularity

Scalability

Framework used in DPLL-basedsystematic exact counterse.g. Relsat [Bayardo-et-al ’00], Cachet [Sang et al. ’04]

Nov 11, 2007 ISWC’07 6

#3: Naïve Sampling Estimate

Idea: Randomly select a region Count within this region Scale up appropriately

Advantage: Quite fast

Drawback: Robustness: can easily under-

or over-estimate Scalability in sparse spaces:

e.g. 1060 solutions out of 10300 means need region much larger than 10240 to “hit” any solutions

Nov 11, 2007 ISWC’07 7

Idea: Identify a “balanced” row split or

column split (roughly equal number of people on each side)

Use local search for estimate

Pick one side at random Count on that side recursively Multiply result by 2

This provably yields the true count on average! Even when an unbalanced row/column is picked accidentally

for the split, e.g. even when samples are biased or insufficiently many Surprisingly good in practice, using a local search as the sampler

Sampling with a Guarantee

Nov 11, 2007 ISWC’07 8

Algorithm SampleCount

Input: Boolean formula F

1. Set numFixed = 0, slack = some constant (e.g. 2, 4, 7, …)

2. Repeat until F becomes feasible for exact counting

a. Obtain s solution samples for F

b. Identify the most balanced variable and variable-pair [“x is balanced” : s/2 samples have x = 0, s/2 have x = 1 “(x,y) is balanced” : s/2 samples have x = y, s/2 have x = –y]

c. If x is more balanced than (x,y), randomly set x to 0 or 1Else randomly replace x with y or –y ; simplify F

d. Increment numFixed

Output: model count 2numFixed – slack exactCount(simplified F) with confidence (1 – 2– slack )

Note: showing one trial

[Gomes-Hoffmann-Sabharwal-Selman IJCAI’07]

Nov 11, 2007 ISWC’07 9

Correctness Guarantee

Key properties: Holds irrespective of the quality of the local search estimates

No free lunch! Bad estimates high variance of trial outcome min(trials) is high-confidence but not tight

Confidence grows exponentially with slack and t

Ideas used in the proof: Expected model count = true count (for each trial) Use Markov’s inequality Pr[X>kE[X]] < 1/k to bound error

probability (X is outcome of one trial)

Theorem: SampleCount with t trials gives a correct lower bound with probability (1 – 2– slack t )

e.g. slack =2, t =4 99% correctness confidence

Nov 11, 2007 ISWC’07 10

Circuit Synthesis, Random CNFs

1.4 x 1014

7 min

1.4 x 1014

2 hrs

1.6 x 1013

4 min1.4 x 1014wff-3-3.5

1.8 x 1021

3 hrs

4.0 x 1017

12 hrs

1.6 x 1020

4 min1.8 x 1021wff-3-1.5

---

---

2.1 x 1029

True Count

1.0 x 1014

12 hrs

1.8 x 1012

12 hrs

8.0 x 1015

2 minwff-4-5.0

---

12 hrs

---

12 hrs

5.9 x 101339

32 min3bitadd_32

2.1 x 1029

2 sec

2.1 x 1029

66 sec

2.4 x 1028

29 sec2bitmax_6

Cachet

(exact)

Relsat

(exact)SampleCount (99% conf.)

Instance

Nov 11, 2007 ISWC’07 11

Talk Roadmap

A Sampling Method with a Correctness Guarantee Can we apply this to the Semantic Web? Discussion

Nov 11, 2007 ISWC’07 12

Talk Roadmap

A Sampling Method with a Correctness Guarantee Can we apply this to the Semantic Web?

[Highly speculative] Discussion

Nov 11, 2007 ISWC’07 13

Counting in the Semantic Web…

… should certainly be possible with this method Example: given RDF database D, count how many

triples comply with query q Throw a constraint cutting the set of all triples in half If feasible, count n triples exactly; return n*2#constraints-slack

Else, iterate “Merely” technical challenges:

What are “constraints” cutting the set of all triples in half? How to “throw” a constraint? When to stop throwing constraints? How to efficiently count the remaining triples?

Nov 11, 2007 ISWC’07 14

What about Deduction?

Does follow from ? Exploit connection “implication UNSAT upper bounds”?

A similar theorem does NOT hold for upper bounds Nutshell: Markov’s inequality Pr[X>kE[X]] < 1/k does not have

a symmetric “Pr[X<kE[X]]” counterpart An adaptation is possible but has many problems → does not

look too promising Heuristic alternative:

Add constraints into to obtain ’; check whether ’ implies If “No”, stop; if “yes”, goto next trial After t successful trials, output “it’s enough, I believe it” No provable confidence but may work well in practice

Nov 11, 2007 ISWC’07 15

What about Deduction?

Does follow from ? Much more distant adaptation:

“Constraint” = something that removes half of !! Throw some and check whether ’

Confidence problematic: Can we draw any conclusions if ’ NOT ? May be that 1, 2 in with 1 2 , but a constraint

separated 1 from 2

May be that all relevant are thrown out Are there interesting cases where we can bound the

probability of these events??

Nov 11, 2007 ISWC’07 16

Talk Roadmap

A Sampling Method with a Correctness Guarantee Can we apply this to the Semantic Web?

[Highly speculative] Discussion

Nov 11, 2007 ISWC’07 17

Discussion

In prop CNF, one can efficiently obtain high-confidence lower bounds on nr of models, by sampling

Application to Semantic Web: Adaptation to counting tasks should be possible Adaptation for , via upper bounds, is problematic

Promising: heuristic method sacrificing confidence guarantee Alternative adaptation weakens instead of strengthening it

“Sampling the knowledge base” Confidence guarantees??

Your feedback and thoughts are highly appreciated!!

Nov 11, 2007 ISWC’07 18

What about Deduction?

Does follow from ? Straightforward adaptation:

There is a variant of this algorithm that computes high-confidence upper bounds instead

Throw “large” constraints, check if ’ is SAT If SAT, no implication; if UNSAT in each of t iterations,

confidence on upper bound on #models Many problems:

Is the ’ actually easier to check?? “Large” constraints are tough even in propositional CNF context! (“Large” = involves half of the prop vars; needed for confidence) Upper bound on #models is not confidence in UNSAT!