Hue/Beeswax – A Hive UI

2010.7.6 bc Wong<bcwalrus [at] cloudera.com>

A Demo Is Worth 210 Slides

Beeswax Server as the Hive Connector

● A proxy that supports concurrent queries

● Needed to work around HiveServer● 617 lines of code● Apache License v2.0

Beeswax Server Talks Thrift

QueryHandle query(Query);QueryState get_state(QueryHandle);Results fetch(QueryHandle, bool start_over);ResultsMetadata get_results_metadata(QueryHandle);

QueryExplanation explain(Query);

string get_log(LogContextId);

A Handle for Each Query

QueryHandle query(Query);QueryState get_state(QueryHandle);Results fetch(QueryHandle, bool start_over);ResultsMetadata get_results_metadata(QueryHandle);

QueryExplanation explain(Query);

string get_log(LogContextId);

{string query;list<string> configuration;string user;list<string> groups;

}

{string id;LogContextId;

}

Asynchronous Query Processing

QueryHandle query(Query);QueryState get_state(QueryHandle);Results fetch(QueryHandle, bool start_over);ResultsMetadata get_results_metadata(QueryHandle);

QueryExplanation explain(Query);

string get_log(LogContextId);

MAIN THREADhandle = new QueryHandle(...);query = new RunningQueryState(handle, ...);query.compile();<thread pool execution>return handle; WORKER THREAD

Hive.closeCurrent();Hive.get(conf);driver.execute();notify();

Hue Polls on the QueryState

QueryHandle query(Query);QueryState get_state(QueryHandle);Results fetch(QueryHandle, bool start_over);ResultsMetadata get_results_metadata(QueryHandle);

QueryExplanation explain(Query);

string get_log(LogContextId);

CREATED,INITIALIZED,COMPILED,RUNNING,FINISHED,EXCEPTION

Results Delimiter Is a Big Problem

QueryHandle query(Query);QueryState get_state(QueryHandle);Results fetch(QueryHandle, bool start_over);ResultsMetadata get_results_metadata(QueryHandle);

QueryExplanation explain(Query);

string get_log(LogContextId);

{bool ready;list<string> columns;list<string> data; // list of tab delimited fieldsint start_row;bool has_more;

}

driver.getResults();

Metadata on Results for Direct Access

QueryHandle query(Query);QueryState get_state(QueryHandle);Results fetch(QueryHandle, bool start_over);ResultsMetadata get_results_metadata(QueryHandle);

QueryExplanation explain(Query);

string get_log(LogContextId);

// Only for “ selects”{

hive_metastore.Schema;string results_dir; // temporary results dirstring table_name; // for select *string delim;

}

Logs Collected Per Query

QueryHandle query(Query);QueryState get_state(QueryHandle);Results fetch(QueryHandle, bool start_over);ResultsMetadata get_results_metadata(QueryHandle);

QueryExplanation explain(Query);

string get_log(LogContextId);

Query_a <-- LogContextId_a <-- -- Thread_1Query_b <-- LogContextId_b <-- -- Thread_2

dynamic thread to ctx

map

LogDivertAppender