digitalocean.com

Go Generate Pipelines

Pipelines?

Pipelines• https://blog.golang.org/pipelines

• concurrency pattern

• “stages connected by channels"

Pipelines• produce a channel of stuff

• creates at least one goroutine

• source -> [middle] -> [middle] -> sink

Example

• https://play.golang.org/p/4Awrr2PgFo

Pattern• directional channels

• goroutine

• close what you create

• for/range to completion (previous stage closes)

Concurrency• increase the concurrency by stage

• easy to add

• https://play.golang.org/p/_qQVZ0Rh73

Generators vs Pipeline• generators create values when you ask for them

• pipeline produce values before you ask for them

• https://play.golang.org/p/R_XHMu1jBz

Real Program Concerns• leaking goroutines

• cancelation

• errors

Leaking Goroutines• every goroutine must terminate!

• pipelines create many goroutines

• https://play.golang.org/p/jUZIv6H4l2

Cancelation Strategies• exit program

• close source stage let downstream drain naturally

• close all stages

Exit program

• easy

• really really easy

Close source & drain downstream

• source closing trickles down to consumers

• all goroutines can exit

• https://play.golang.org/p/pRpNspMG2z

Close all stages• `out <-` must be interruptible everywhere

• every channel producing function provided “done”

• https://play.golang.org/p/Rd-hEVic0O

Errors• can happen anywhere

• second output channel

• https://play.golang.org/p/aUv5MrD7Bj

Errors Take 2• https://godoc.org/golang.org/x/net/context

• need a “pipeline context”

• signal done and place to write errors

• WIP idea

• https://play.golang.org/p/-sn3QGpc7X

Pipelines• `out <-` always interruptible by `done`

• close what you create

• error channel/pipeline context

• often repetitive…

Go Generate

Go Generate• since go1.4

• “comments” in your go project

• //go:generate [command] [args…]

Just a command• generate say hi (for fun)

• generate rm -fr / (for evil)

• generate printenv (for understanding)

Not a shell…

• go:generate echo $EDITOR > /tmp/nope

• go:generate cat /tmp/file | grep something

But shellish…• $GOFILE

• $GOPACKAGE

• $GOARCH

• $GOOS

• go:generate something -out=$GOARCH_something.go

Meant to write files• stderr for communicating to the user

• silent in proper operation

• side effect (file) then committed to repository

• can also generate _test.go file

Pipeliner

My common pipelines• batch(<-chan Event) (<-chan []Event)

• flatMap(<-chan gzipfile) (<-chan Event)

• from([]Event) (<-chan Event)

Pipeliner• side-project

• saves me time

• may annoy you (bad error messages)

• github.com/supershabam/pipeliner

digitalocean.com

close(talk)

digitalocean.com

questions?