View
144
Download
10
Category
Tags:
Preview:
Citation preview
GO’ING TO THE NEXT LEVEL, LESSONS FROM GO AT COUCHBASE AND INTRODUCING THE GO SDK
Marty Schoch and Matt Ingenthron
Lessons from Goat Couchbase
Lessons from Go at Couchbase
©2015 Couchbase Inc. 4
Couchbase Labs
2011 2 engineers 0 rules
©2015 Couchbase Inc. 5
What if we used Go?
Easy concurrency Easy HTTP Server Easy HTTP Client Easy JSON Rich Standard Library
©2015 Couchbase Inc. 6
Timeline
January 2012go-couchbase
2012
2014
September 2012sync getaway
September 2012cbfs
October 2012
tuq -> tuqtng -> n1ql -> sql for documents
November 2012cbgb
January 2013cbugg
February 2013
Godu, vbmap, and other utils
August 2013
Secondary indexing April 2014
bleve
August 2014goxdcr
October 2014cbft
December 2014gocb
©2015 Couchbase Inc. 7
Why do developers like Go?
Excellent tooling CPU/memory profilers Race detector Integrated Unit Tests
Channels for synchronization as well as communication
Satisfy interface without importing its package
©2015 Couchbase Inc. 8
What don’t developers like about Go?
No good debugger gdb works in some cases not others Others still immature
error as interface is powerful construct … but too often its fmt.Errorf(“…”) … which leads to string matching later Define constants so that errors can be differentiated and
handled accordingly
Garbage Collection
©2015 Couchbase Inc. 10
Don’t Make Garbage
©2015 Couchbase Inc. 11
Sounds Simple
Be mindful of allocations Be mindful of operations that copy data
string <-> []byte conversions Many standard library functions operating on []byte But, once you avoid making copies you need clear notion
of ownership Re-use already allocated objects when possible
Include Reset() methods in your API sync.Pool offers solution when objects are
reusable and have clear start/end lifecycle
©2015 Couchbase Inc. 12
Do I really have a GC problem?
Review CPU Profiles
©2015 Couchbase Inc. 13
Do I really have a GC problem?
Graph GC Pause Time (exposed by expvar)
©2015 Couchbase Inc. 14
Do I really have a GC problem?
Visualize Garbage Collector Details Using gcvis by Dave Cheney
©2015 Couchbase Inc. 15
Is this variable on the stack or the heap?
Have the compiler tell you. Build with: go build -gcflags=-m
./main.go:54: map[string]interface {} literal escapes to heap./main.go:112: (*matchPhraseQuery).Validate q does not escape
©2015 Couchbase Inc. 16
Solutions for Advanced Users
Slab allocation See github.com/couchbase/go-slab
Off-heap allocation Lots of pointers, even if largely static, may cause
unreasonable GC Large maps, tree structures with child pointers, etc GC has to check pointers in map, even though you never
plan on freeing them Several new projects in this area…
©2015 Couchbase Inc. 17
Other Memory Surprises?
©2015 Couchbase Inc. 18
Sync Gateway – Many Concurrent Users
Observation: often 20GB+ memory usage Action: Review heap profile
(pprof) top25Total: 3972.5 MB 3875.0 97.5% 97.5% 3875.0 97.5% compress/flate.(*compressor).initDeflate 19.0 0.5% 98.0% 19.5 0.5% github.com/couchbaselabs/sync_gateway/db.(*Database).MultiChangesFeed 12.5 0.3% 98.3% 12.5 0.3% net/http.newBufioWriterSize 10.5 0.3% 98.6% 10.5 0.3% bufio.NewReaderSize 10.0 0.3% 98.9% 10.0 0.3% compress/flate.newHuffmanBitWriter 7.0 0.2% 99.0% 7.0 0.2% newdefer 4.5 0.1% 99.1% 4.5 0.1% github.com/couchbaselabs/sync_gateway/channels.readString 3.0 0.1% 99.2% 3.0 0.1% net/textproto.(*Reader).ReadMIMEHeader 2.0 0.1% 99.3% 2.0 0.1% github.com/couchbaselabs/sync_gateway/auth.func·001 2.0 0.1% 99.3% 2.0 0.1% reflect.mapassign 1.5 0.0% 99.4% 1.5 0.0% github.com/couchbaselabs/sync_gateway/channels.TimedSet.Copy 1.5 0.0% 99.4% 1.5 0.0% net/http.Header.clone 1.5 0.0% 99.4% 1.5 0.0% net/textproto.MIMEHeader.Set 1.5 0.0% 99.5% 1.5 0.0% net/url.parseQuery 1.5 0.0% 99.5% 1.5 0.0% runtime.malg 1.0 0.0% 99.5% 1.0 0.0% compress/flate.(*huffmanEncoder).generate 1.0 0.0% 99.6% 1.0 0.0% concatstring 1.0 0.0% 99.6% 1.0 0.0% encoding/json.(*decodeState).objectInterface 1.0 0.0% 99.6% 1.0 0.0% github.com/couchbaselabs/sync_gateway/channels.(*ChangeLog).TruncateTo 1.0 0.0% 99.6% 5.5 0.1% github.com/couchbaselabs/sync_gateway/channels.decodeChangeLog 1.0 0.0% 99.7% 1.0 0.0% github.com/gorilla/context.Set 1.0 0.0% 99.7% 1.0 0.0% github.com/gorilla/mux.(*routeRegexpGroup).setMatch 1.0 0.0% 99.7% 4.5 0.1% net/http.ReadRequest 0.5 0.0% 99.7% 3885.5 97.8% compress/flate.NewWriter 0.5 0.0% 99.7% 0.5 0.0% compress/gzip.NewWriterLevel
©2015 Couchbase Inc. 19
Gzipped Responses
Sync Gateway clients are mobile, gzipped response makes a lot of sense.
Initial implementation, invoked compress/flate NewWriter on each response
Each flate.Writer consumed 1.4MB Rewritten to use sync.Pool
Acquire flate.Writer instances from pool Call Reset() and return to pool when done
©2015 Couchbase Inc. 20
Slices are Great… But…
Read 1MB of bytes off the wire into a buffer Parse the bytes, find the data you’re interested
in Return []byte with the key (say 16 bytes) The ENTIRE 1MB buffer is held in memory while
the slice is referenced. If these slices are long-lived and small relative to
their parent array, may be cheaper to copy them.
Go Routines / Channels
©2015 Couchbase Inc. 22
Too many channels?
Channels are synchronized
Synchronization has a cost
Use wisely, not haphazardly
©2015 Couchbase Inc. 22
Repeatable Builds
©2015 Couchbase Inc. 24
New/Prototype Projects
Bleeding Edge ‘go get’ everything Low friction Things break, and we fix them
But no one pretends this works for production!
24
©2015 Couchbase Inc. 25
Sync Gateway Builds
First Go project to need a solution to the problem Decided on a custom solution
Dependencies are vendored using git submodules Use wrapper scripts around native tools, go.sh, build.sh,
run.sh Scripts setup custom $GOPATH containing only
sync_gateway and its dependencies Works well for sync_gateway team
Separate $GOPATH can make for more complex integrations with IDE’s and tools like Go oracle
©2015 Couchbase Inc. 26
Projects integrating with Couchbase Server
Couchbase Server already has a well established process for repeatable builds using ‘repo’ All external dependencies are cloned into a separate
github organization “couchbasedeps” In our repo manifests, we get sources from couchbase
deps, but install them into $GOPATH using their canonical package name
<project name="protobuf" remote="couchbasedeps" revision="655cdfa588ea190e901bc5590e65d5621688847c" path="godeps/src/github.com/golang/protobuf"/>
©2015 Couchbase Inc. 27
Set $GOPATH and all Go tools are happy!
©2015 Couchbase Inc. 27
Using cgo
©2015 Couchbase Inc. 29
cgo on Windows Depends on gcc
No option to use MSVC We recently went to great pains to remove dependency
on GCC and now we had to add it back Considered dynamically loading DLL
High maintenance for wrappers when APIs change Current solution is to install gcc on Windows
build boxes
©2015 Couchbase Inc. 30
Reliable Core Dumps with crash inside cgo
With gdb attached, we see what we expect However, if no debugger is attached, resulting
core file does not have correct stack trace Using gccgo does work, but not comfortable shipping this
to customers
Thoughts…
©2015 Couchbase Inc. 32
Go at Couchbase, at a Crossroads
©2015 Couchbase Inc. 32
©2015 Couchbase Inc. 33
Real Problems, Real Solutions
Solve the problems Contribute back to the community
Solve the problems Don’t contribute them back
Don’t solve them Move away from Go in the future
The Couchbase Go SDKgocb
©2015 Couchbase Inc. 35
Couchbase SDKs
What does it mean to be a Couchbase SDK?
Cluster
Bucket
CRUDView
QueryN1QL Query
FunctionalManage connections to the bucket within the cluster for different services.Provide a core layer where IO can be managed and optimized.Provide a way to manage buckets.
APIinsertDesignDocument()flush()listDesignDocuments()
FunctionalHold on to cluster information such as topology.
APIReference Cluster ManagementopenBucket()info()disconnect()
FunctionalGive the application developer a concurrent API for basic (k-v) or document management
APIget()insert()upsert()remove()
FunctionalAllow for querying, execution of other directives such as defining indexes and checking on index state.
APIclient.NewN1QLQuery( “SELECT * FROM default LIMIT 5” ) .Consistency(gocouchbase.RequestPlus);
FunctionalAllow for view querying, building of queries and reasonable error handling from the cluster.
APIabucket.NewViewQuery().Limit().Stale()
©2015 Couchbase Inc. 36
Being True to Go Philosophy
Easy to read, write, understand with high-level concurrency (goroutines), type safety. Low level access if needed.
gocbcore
gocb
See more in a gist from Brett Lawson.
var user interface{}
bucket.Get("user_1", &user)fmt.Printf("Got user_1: %v\n", user)
bucket.Get("user_2", &user)fmt.Printf("Got user_2: %v\n", user)
bucket.Get("user_3", &user)fmt.Printf("Got user_3: %v\n", user)
signal := make(chan int)go func() {
var user interface{}bucket.Get("user_1",
&user)fmt.Printf("Got user_1:
%v\n", user)signal <- 1
}()go func() {
var user interface{}bucket.Get("user_2",
&user)fmt.Printf("Got user_2:
%v\n", user)signal <- 1
}()go func() {
var user interface{}bucket.Get("user_3",
&user)fmt.Printf("Got user_3:
%v\n", user)signal <- 1
}()for i := 0; i < 3; i++ {
<-signal}
// Note that this does not perform transcoding (ie: JSON marshalling)agent := bucket.IoRouter()signal := make(chan int)
_, err = agent.Get([]byte("user_1"), func(value []byte, flags uint32, cas uint64, err error) {
fmt.Printf("Got user_1: %v\n", string(value))
signal <- 1})
_, err = agent.Get([]byte("user_2"), func(value []byte, flags uint32, cas uint64, err error) {
fmt.Printf("Got user_2: %v\n", string(value))
signal <- 1})
_, err = agent.Get([]byte("user_3"), func(value []byte, flags uint32, cas uint64, err error) {
fmt.Printf("Got user_3: %v\n", string(value))
signal <- 1})
for i := 0; i < 3; i++ {<-signal
}
gocb is iterating toward interface stability
gocbcore will be a volatile interface for some time
©2015 Couchbase Inc. 37
JSON and Other Data Types
cbgo uses Go’s support for JSON internally
type BeerFull struct {Type string `json:"type"`Id string
`json:"id,omitempty"`BreweryId string
`json:"brewery_id"`Name string `json:"name"`Description string
`json:"description"`Style string `json:"style"`Category string
`json:"category"`Abv float64 `json:"abv"`Ibu float64 `json:"ibu"`Srm float64 `json:"srm"`Upc float64 `json:"upc"`
}
var beer BeerFull_, _, err := bucket.Get(id, &beer)if err != nil {
fmt.Fprintf(w, "Get Error: %v\n", err)return
}
// reference things like thisbeer.Name
Given this struct
You can unmarshal with a Get()
©2015 Couchbase Inc. 38
gocbcore – Where all of the Performance Tricks Live
High level concurrency in Go is great, but… Better still (for go) is avoiding the use of the
primitives
gocbcore is (mostly) lockless Has a concept of a “pipeline” and only two locks
within it Coarse grained “lock” as an RWMutex used only in the
case of death for cleanup Fine grain mutex on the operation list (memdOpMap) when
adding and removing items to the “pipeline”== minimal blocking of your goroutines
DemoTour of a Web Application
©2015 Couchbase Inc. 40
Current Status and Roadmap
Status Current version: 0.2.0 Release early and often to get user feedback, iterate
Roadmap Continue to iterate on 0.x releases adding features… N1QL Support Durability Requirements Complete test coverage && Fully Review Couchbase QE
tests Release 1.0!
Q&A
Thanks!Marty Schoch – @mschoch
Matt Ingenthron – @ingenthr
Get Started with Couchbase Server 4.0: www.couchbase.com/beta
Get Trained on Couchbase: training.couchbase.com
Recommended