HyperLogLog

in Hive How to countsheep efficiently?

Phillip Capper: Whitecliffs Sheep

@bzamecnik

Agenda

● the problem – count distinct elements● exact counting● fast approximate counting – using HLL in Hive● comparing performance and accuracy● appendix – a bit of theory of probabilistic counting

○ how it works?

The problem: count distinct elements

● eg. the number of unique visitors● each visitor can make a lot of clicks● typically grouped in various ways● "set cardinality estimation" problem

Small data solutions

● sort the data O(N*log(N)) and skip duplicates O(N)○ O(N) space

● put data into a hash or tree set and iterate○ hash set: O(N^2) worst case build, O(N) iteration○ tree set: O(N*log(N)) build, O(N) iteration○ both O(N) space

● but: we have big data

Example:

~100M unique values in 5B rows each day

32 bytes per value -> 3 GB unique, 150 GB total

�

Problems with counting big data

● data is partitioned○ across many machines○ in time

● we can't sum cardinality of each partition○ since the subsets are generally not disjoint○ we would overestimate

count(part1) + count(part1) >= count(part1 ∪ part2)

● we need to merge estimators and then estimate cardinality

count(estimator(part1) ∪ estimator(part2))

SELECT COUNT(DISTINCT user_id)

FROM events;

single reducer!

Exact counting in Hive

Exact counting in Hive – subquery

SELECT COUNT(*) FROM (

SELECT 1 FROM events

GROUP BY user_id

) unique_guids;

Or more concisely:

SELECT COUNT(*) FROM (

SELECT DISTINCT user_id

FROM events

) unique_guids;

� many reducers

� two phases

cannot combine more aggregations

Exact counting in Hive

● hive.optimize.distinct.rewrite○ allows to rewrite COUNT(DISTINCT) to subquery○ since Hive 1.2.0

Probabilistic counting

● fast results, but approximate● practical example of using HLL in Hive● more theory in the appendix

● klout/brickhouse○ single option○ no JAR, some tests○ based on HLL++ from stream-lib (quite fast)

● jdmaturen/hive-hll○ no options (they are in API, but not implemented!)○ no JAR, no tests○ compatible with java-hll, pg-hll, js-hll

● t3rmin4t0r/hive-hll-udf○ no options, no JAR, no tests

Implementations of HLL as Hive UDFs

● User-Defined Functions● function registered from a class (loaded from JAR)● JAR needs to be on HDFS (otherwise it fails)● you can choose the UDF name at will● work both in HiveServer2/Beeline and Hive CLI

ADD JAR hdfs:///path/to/the/library.jar;

CREATE TEMPORARY FUNCTION foo_func

AS 'com.example.foo.FooUDF';

● Usage:

SELECT foo_func(...) FROM ...;

UDFs in Hive

● to_hll(value)○ aggregate values to HLL○ UDAF (aggregation function)○ + hash each value○ optionally can be configured (eg. for precision)

● union_hlls(hll)○ union multiple HLLs○ UDAF

● hll_approx_count(hll)○ estimate cardinality from a HLL○ UDF

HLL can be stored as binary or string type.

General UDFs API for HLL

● Estimate of total unique visitors:

SELECT hll_approx_count(to_hll(user_id))

FROM events;

● Estimate of total events + unique visitors at once:

SELECT

count(*) AS total_events

hll_approx_count(to_hll(user_id))

AS unique_visitors

FROM events;

Example usage

Example usage

● Compute each daily estimator once:

CREATE TABLE daily_user_hll AS

SELECT date, to_hll(user_id) AS users_hll

FROM events

GROUP BY date;

● Then quickly aggregate and estimate:

SELECT hll_approx_count(union_hlls(users_hll))

AS user_count

FROM daily_user_hll

WHERE date BETWEEN '2015-01-01' AND '2015-01-31';

https://github.com/klout/brickhouse - Hive UDFhttps://github.com/addthis/stream-lib - HLL++

$ git clone https://github.com/klout/brickhouse

disable maven-javadoc-plugin in pom.xml (since it fails)

$ mvn package

$ wget http://central.maven.org/maven2/com/clearspring/analytics/stream/2.3.0/stream-

2.3.0.jar

$ scp target/brickhouse-0.7.1-SNAPSHOT.jar \

stream-2.3.0.jar cluster-host:

cluster-host$ hdfs dfs -copyFromLocal *.jar \

/user/me/hive-libs

Brickhouse – installation

Brickhouse – usage

ADD JAR /user/zamecnik/lib/brickhouse-0.7.1-15f5e8e.jar;

ADD JAR /user/zamecnik/lib/stream-2.3.0.jar;

CREATE TEMPORARY FUNCTION to_hll AS 'brickhouse.udf.hll.

HyperLogLogUDAF';

CREATE TEMPORARY FUNCTION union_hlls AS 'brickhouse.udf.

hll.UnionHyperLogLogUDAF';

CREATE TEMPORARY FUNCTION hll_approx_count AS 'brickhouse.

udf.hll.EstimateCardinalityUDF';

to_hll(value, [bit_precision])

● bit_precision: 4 to 16 (default 6)

Hive-hll usage

ADD JAR /user/zamecnik/lib/hive-hll-0.1-2807db.jar;

CREATE TEMPORARY FUNCTION hll_hash as 'com.kresilas.hll.

HashUDF';

CREATE TEMPORARY FUNCTION to_hll AS 'com.kresilas.hll.

AddAggUDAF';

CREATE TEMPORARY FUNCTION union_hlls AS 'com.kresilas.hll.

UnionAggUDAF';

CREATE TEMPORARY FUNCTION hll_approx_count AS 'com.

kresilas.hll.CardinalityUDF';

We have to explicitly hash the value:

SELECT

hll_approx_count(to_hll(hll_hash(user_id)))

FROM events;

Options for creating HLL:

to_hll(x, [log2m, regwidth, expthresh, sparseon])

hardcoded to:

[log2m=11, regwidth=5, expthresh=-1, sparseon=true]

Hive-hll usage

Nice things

● HLLs are additive○ can be computed once

○ various partitions can be merged and estimated for cardinality later

● we can count multiple unique columns at once○ no need to subquery○ we can do wild grouping (by country, browser, …)

● HLLs take only little space

Rolling window

-- keep reasonable number of task for month of dataSET mapreduce.input.fileinputformat.split.maxsize=5368709120;-- keep low number of output files (HLLs are quite small)SET hive.merge.mapredfiles=true;-- maximum precisionSET hivevar:hll_precision=16;

-- HLL for each dayCREATE TABLE guids_parquet_hll ASSELECT '${year}' AS year, '${month}' AS month, day, to_hll(guid, ${hll_precision}) AS guid_hllFROM parquet.dump_${year}_${month}GROUP BY day;

-- for each day estimate number of guids 7-days back

CREATE TABLE zamecnik.guids_parquet_rolling_30_day_countASSELECT `date`, hll_approx_count(guids_union) AS guid_countFROM ( SELECT concat(`year`, '-', `month`, '-', `day`) as `date`, union_hlls(guid_hll) OVER w AS guids_union FROM guids_parquet_hll WINDOW w AS ( ORDER BY `year`, `month`, `day` ROWS 6 PRECEDING )) rolling_guids;

Rolling window

● when JARs are not on HDFS the query fails (why?)● computing on many days of raw clickstream fails in

Beeline (works in Hive CLI), parquet is ok

● HIVE-9073 WINDOW + custom UDAF → NPE○ fixed in Hive 1.2.0

● DISTRO-631

Pitfalls

Approximation error

● Typically < 1-2 %● Can be controlled by the parameters● Example: 1 year of guids

Appendix – more interesting things

● trade-off: some approximation error for far better performance and memory consumption

● sketch - streaming & probabilistic algorithm● KMV - k minimal values● linear counter● loglog counter

Probabilistic counting

LogLog counter

● run length of initial zeros● multiple estimators (registers)● stochastic averaging

○ single hash function○ multiple buckets

● hash → (register index, run length)

Linear counter

m = 20 # size of the registerregister = bitarray(m) # register, m bits

def add(value): h = mmh3.hash(value) % m # select bit index register[h] = 1 # = max(1, register[h])

def cardinality(): u_n = register.count(0) # number of zeros v_n = u_n / m # relative number of zeros n_hat = -m * math.log(v_n) # estimate of the set cardinality return n_hat

● structure like loglog counter● harmonic mean to combine registers● correction for small and large cardinalities● values needs to be hashed well – murmur3

HyperLogLog (HLL)

HLL union

● just take max of each register value● no loss – same result as HLL of union of streams● parallelizable● union preserves error bound, intersection/diff do not

Further reading

● very nice explanation of HLL● Probabilistic Data Structures For Web Analytics And

Data Mining● Sketch of the Day: HyperLogLog — Cornerstone of a

Big Data Infrastructure● HyperLogLog in Pure SQL● Use Subqueries to Count Distinct 50X Faster● It is possible to combine HLL of different sizes

Papers

● HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm

● https://github.com/addthis/stream-lib#cardinality

Other problems & structures

● set membership – bloom filter● top-k elements – count-min-sketch, stream-summary