View
218
Download
0
Category
Preview:
Citation preview
8/13/2019 Concurrency in the Go Programming Language
1/18
An overview of Go
by Dan Callahan
Let’s start at the beginning: What is Go?
Go is a “fast, statically typed, compiled language that feels like a dynamically typed,
interpreted language! "t is a systems language competiti#e $ith C, C%%, and &a#a, but
designed to deal gracefully $ith concurrency 'o $it:
“Go makes much more sense for the class of problems that C%% $as originally intended to
sol#e! ()ruce *ckel
Go $as released by Google in late +-, and .ersion / should land in the first fe$ months
of +/+
Why should you care?
0ou should care about Go for fi#e reasons:
/ "t has an impeccable pedigree
+ "t embodies an e1tremely pragmatic design philosophy
2 "t implements many inno#ati#e ideas
3 "t can already be used for real $ork
4 "t has an absolutely fantastic mascot
Go’s Pedigree
Go is being de#eloped by )ell Labs #eterans augmented by a handful of other brilliant
engineers 'he core team includes:
• Rob Pike 56'789, lan - from )ell Labs;
• Ken Thompson 5C, 6ni1, lan - from )ell Labs;
• Russ o! 5lan - from )ell Labs,
8/13/2019 Concurrency in the Go Programming Language
2/18
)y taking these lessons into account, Go hopes to offer the de#eloper8time efficiency of a
language like ython and the run8time efficiency of C
7or e1ample:
• *#en though Go is a systems language, it uses garbage collection• Go has been designed to compile extremely @uickly
• Go’s synta1 is clean and minimal
• Go offers first8class functions
• 7unctions in Go are closures
• 7unctions in Go can return multiple #alues
• 7unction definitions can ha#e named return parameters
• Go offers sane error handling, in contrast to many languagesA model of thro$ing and
catching e1ceptions
Let’s look at a fe$ of these more closely
Quick Compile Times
Go programs are structured in a manner that drastically limits the number of passes re@uired
to produce a binary By / G> dual8core laptop compiles pkg/math 5E+ files, 4 FLC; inunder / second:
$ time make --quietmake --quiet 0.89s user 0.08s system 97% cpu 0.991 total
Clean Syntax
)elo$ is a simple “hello, $orld! in Go
package main
import "fmt"
func main!
fmt.#rintln"ello 世界"!&
'he anatomy of this program should be self8e#ident: "t declares that it is a member of the
package main, imports any other packages it needs, and then defines a function called main,
$hich is e1ecuted $hen the compiled program is in#oked
"t’s $orth pointing out the use of 6nicode )y definition, Go source files are encoded in
6'789, $hich makes it easy to use 6nicode literals >a#e you e#er $anted to name a #ariable? Well, no$ you can
+
8/13/2019 Concurrency in the Go Programming Language
3/18
First Class Functions
"n Go, functions can be created or passed around like any other structure "n the code belo$,
"’#e defined t$o unary functions: 'ou(le and square *ach accepts a single int for input
and returns a single int
"’#e also defined apply, $hich accepts a t$o arguments: a function, and an integer "t then
calls the function and returns an integer
package main
import "fmt"
func 'ou(le) int! int return ) * )&
func square) int! int return ) + )&
func applyf funcint! int ) int! int return f)!&
func main! fmt.#rintlnapply'ou(le ,!! fmt.#rintlnapplysquare ,!!&
)ecause Go is strongly and statically typed, function declarations must e1plicitly name thetypes of their inputs and return #alues 'hus, func applyf funcint! int ) int! int
creates a ne$ function named apply $hich takes t$o inputs, a funcint! int called f and
an int called ) "t then returns an int
0ou can read funcint! int the same $ay: "t’s a function that accepts an integer and
returns an integer
With that information, the compiler is able to #erify that e#erything in the program is of the
e1pected type "t’s $orth noting that Go does not do implicit or automatic type coercion
Closures
H closure is Iust a function that can reference #ariables that $ould typically be considered
outside of its scope 'o illustrate, the code belo$ calculates the first fe$ 7ibonacci numbers
using a closure
package main
func fi(! func! int a ( 0 1 return func! int
a ( ( a*( return ( &
2
8/13/2019 Concurrency in the Go Programming Language
4/18
&
func main! f fi(! printlnf! f! f! f! f!!&
'hat might look a bit $eird 'o re#ie$, note that named function definitions ha#e three parts:
/ 'he func key$ord
+ 'he name of the function, its inputs, and their types, like square) int!
2 'he function’s return type, like int
=o func fi(! func! int abo#e is creating a function named fi( that doesn’t ha#e any
inputs, and $hich returns a func! int H func! int is, in turn, a function that doesn’t
ha#e any inputs, and $hich returns an integer
'hus, $hen $e in#oke fi(! $e get another function When $e in#oke that , $e get an
integer
)ecause functions in Go are closures, the function returned by fi(! has access to the #alues
of a and (, e#en though they $ere defined outside of the inner function itself
Go’s #nnova"ions
'he follo$ing items aren’t necessarily uni@ue to Go, but they are distinguishing features
• Go ships $ith its o$n pretty printer, gofmt• Go ships $ith an old8to8ne$ Go compiler, gofi1
• Go sports a brilliantly light$eight type system
• Go has per#asi#e, primiti#e support for concurrency
• Go makes it easy to defer e1ecution of a function until the end of a block
'he type system and concurrency features are $orthy of their o$n dedicated articles, so let’s
look at the other items
Gofmt, the Go Pretty-Printer
Go is some$here bet$een C and ython in the rigidity of its synta1 Whitespace is
insignificant and semicolons can be omitted, but the opening brace of a construct must be on
the same line as that construct
=till, Go maintains high degree of readability across authors thanks to gofmt, a command line
tool that pretty8prints Go source code in a single, consistent style
'ake the follo$ing #ersion of the 7ibonacci program from abo#e "t still compiles and runs
perfectly, but it has semicolons e#ery$here, horribly inconsistent indentation, and e#en
combines t$o statements on one line $ith a ( ( a*( return (
package main
3
8/13/2019 Concurrency in the Go Programming Language
5/18
func fi(! func! int a ( 0 1 return func! int a ( ( a*( return ( & &
func main! f fi(!printlnf! f! f! f! f!!&
ne pass through gofmt, and out pops:
package main
func fi(! func! int a ( 0 1 return func! int a ( ( a*( return ( &&
func main! f fi(! printlnf! f! f! f! f!!&
Gofix, the Old-to-Ne Go Compiler
Hs Go is still under rapid de#elopment, it’s not uncommon for components of the standard
library to change in back$ards8incompatible $ays
8/13/2019 Concurrency in the Go Programming Language
6/18
package main
import "url"
func main! url url.#arse"http//e)ample.com/search4qfoo"!
printlnurl.5uery!.6et"q"!!&
Jote that import "http" changed into import "url" and http.#arse23 $as changed
into url.#arse ne of the more subtle details is that my #ariable, url, $as automatically
renamed to url to a#oid clashing $ith the ne$ import statement >o$ cool is that?
"f you’re coming from ython, you can think of gofi) as being similar to to
!eferred "xecution
'ake the follo$ing program:
package main
func goo'(ye! println"6oo'(ye:"!&
func main! goo'(ye! println";our"! println"
8/13/2019 Concurrency in the Go Programming Language
7/18
println"
8/13/2019 Concurrency in the Go Programming Language
8/18
>eroku has also been using Go to build Dooer , a consistent, highly8a#ailable data store 'o
@uote Feith
8/13/2019 Concurrency in the Go Programming Language
9/18
8/13/2019 Concurrency in the Go Programming Language
10/18
13etBs go: D 0000Coffee is rea'y: D 000
8/13/2019 Concurrency in the Go Programming Language
11/18
i there, " lo#e cakeK!
in your bro$ser, since http.an'le;unc"/" han'ler! ensures that all re@uests get passed off to the han'ler function, $hich simply displays that message Digging in to the
source for the http package $ill re#eal that it takes each incoming re@uest, spins up a
goroutine to handle it, and then goes back to listening
'hus, a great use of goroutines $ithout needing to kno$ ho$ to communicate $ith them
$hile they run =till, there are situations $here you need to communicate $ith the goroutines,
as per the coffee N tea bre$ing code abo#e Luckily, Go pro#ides channels to do Iust that
//
8/13/2019 Concurrency in the Go Programming Language
12/18
Wha" is a channel?
H channel is a type8safe pipe for communicating bet$een different parts of your program
Let’s start $ith an impro#ed #ersion of the bre$ing program as an e1ample:
package main
import "time"
func ?s2ea'y>hat string secon's int@ ch chanJ- (ool! time.Aleepsecon's + 1e9! // nanosecon's println>hat "is rea'y:"! ch J- true&
func main! ch makechan (ool! println"3etBs go:"! go ?s2ea'y"Coffee" ch! go ?s2ea'y"orl'" K se' -e Bs/hello/goo'(ye/gB K >c -c 1@
/+
8/13/2019 Concurrency in the Go Programming Language
13/18
'he shell commands abo#e $ork like they do since echo Iust takes a string, slaps a Ln
character on the end, and spits it back out Bean$hile >c -c reads in data and counts the
number of characters 'hus, it counts /+ in the first instance 54 % 4 % a space % a ne$line;
=imilarly, the se' in#ocation takes its input and replaces any occurances of “hello! $ith
“goodbye!, lengthening the original string by +
What if $e $anted to simulate that sort of a pipeline in Go? 7irst, let’s define functions to
replace echo, se', and >c *ach one $ill take a channel for input, and a channel for output:
package main
import "rege)p" "strconF"!
func echoin chan string out chan string!
out J- J-in * "Ln"!&
func se'in chan string out chan string! re rege)p.MustCompile"hello"! out J- re.2eplaceIllAtringJ-in "goo'(ye"!&
func >cin chan string out chan string! out J- strconF.?toalenJ-in!!&
"n the abo#e functions, echo simply reads from its input channel, adds a ne$line to the end of
it $ith J-in * "Ln", and then sends the result to the out channel
'he se' function $orks similarly, first compiling a regular e1pression that matches hello,
and then calling that rege1’s 2eplaceIllAtring method 'he method is told to grab a string
from the in channel and use “goodbye! as the replacement for matches 'he result gets sent
to the out channel
Lastly, >c simply reads from the input channel 5J-in;, then gets the length of the input $ith
len, and lastly con#erts the integer length into a string $ith strconF.?toa before sending
the result on the out channel
'he main function is simple, but a bit #erbose
func main! ch1 makechan string! ch makechan string! ch makechan string! ch@ makechan string!
go echoch1 ch! go se'ch ch! go >cch ch@!
ch1 J- "hello >orl'" printlnJ-ch@!
/2
8/13/2019 Concurrency in the Go Programming Language
14/18
&
"t simply makes four channels, and hooks them up so that the output of echo is the input of
se', and the output of se' is the input of >c
*ach of those subfunctions get started in their o$n goroutines, and then sit and $ait for input'he string “hello $orld! gets sent on ch1, $hich flo$s through echo, then se', then >c and
out on ch@ 'he result, “/3!, gets printed:
$ /pipes11@
*1plicitly specifying the input and output channels is needlessly #erbose We ha#e to gi#e
each function an input channel, but if it made and returned its o$n output channel, then $e
could tri#ially chain together as many functions as $e $ant
Let’s do that:
package main
import "rege)p" "strconF"!
func echoin chan string! chan string out makechan string! go func! out J- J-in * "Ln"!
&! return out&
func se'in chan string! chan string out makechan string! re rege)p.MustCompile"hello"! go func! out J- re.2eplaceIllAtringJ-in "goo'(ye"! &! return out&
func >cin chan string! chan string out makechan string! go func! out J- strconF.?toalenJ-in!! &! return out&
'his might look comple1, but it’s really not too different from the pre#ious code 'he
signatures changed to Iust accepting a single chan string, each function starts $ith creating
an output channel, and each function ends by returning that channel "n the middle, the “real
$ork! gets $rapped in an anonymous function $hich gets in#oked in a goroutine 'his
means that $hen you call something like echo, it immediately returns its ne$ly8made outputchannel, and as a side effect, spins off a goroutine to process any data that gets sent on the in
/3
8/13/2019 Concurrency in the Go Programming Language
15/18
channel "t’s basically saying “kay, "’m ready and $aiting on in When " get something, "’ll
gi#e it back to you on this channel that "’m returning!
Jote that because functions in Go are closures, $e didn’t ha#e to e1plicitly pass in a
reference to the input channel, in, $hen $e defined an anonymous $orker function cse'echoin!!! inJ-"hello >orl'" printlnJ- out!&
We make an input channel, then $e get an output channel by composing our filters: out
>cse'echoin!!! Hgain, because the output of echoin! is a channel, $e can use it as
the input to se', and so on Hnd it $orks e1actly like you $ould e1pect:
$ ./pipes1@
7unctions that start goroutines and return channels are a some$hat uni@ue and interesting Go
idiom
)ow do channels and gorou"ines work "oge"her?Channels and goroutines $ork together to implement a much more sane model of
concurrency than traditional threads and shared memory 'he central idea is that, for
unbuffered channels, the act of reading and $riting across a channel forces t$o goroutines to,
at that moment, be synchronied ther$ise, things are free to e1ecute concurrently or in
parallel
'his paradigm is often e1pressed as:
Do not communicate by sharing memoryO instead, share memory by communicating (
*ffecti#e Go
7or a some$hat silly e1ample of channels and goroutines $orking together, check out this
prime number sie#e:
package main
func 6eneratech chanJ- int! for i i** ch J- i &&
func ;ilterin J-chan int out chanJ- int prime int!
/4
8/13/2019 Concurrency in the Go Programming Language
16/18
for i J-in if i%prime : 0 out J- i & &
&
func main! ch makechan int! go 6eneratech! for i 0 i J 10 i** prime J-ch printprime "Ln"! ch1 makechan int! go ;ilterch ch1 prime! ch ch1 &&
6enerate simply starts at + and sends e#er increasing numbers out on a gi#en channel 7irst
+, then 2, then 3, and so on
;ilter sits in bet$een t$o channels and silently drops any number that’s di#isible by its
prime input parameter ther$ise, it Iust passes the number along H ;ilterin out !,
for instance, $ould not let any multiples of + through it 'hus, $hen hooked up to the output
of 6enerate, first 2, then 4, then E, $ould pop out the other end
Can you see $here this is going? We $ant to build a pipeline that successi#ely filters each
prime, so that $ith each number popping out the end, $e sa#e it, and then add on a filter for
its o$n multiples:
G6enerateH -N
G6enerateH -N G;ilter!H -N
G6enerateH -N G;ilter!H -N G;ilter!H -N ,
and so on
>ence the loop in the main function, $hich successi#ely reads from the rightmost end of the
pipeline, prints that #alue, and then adds on a filter for it
$ ./primes,711117199
/
8/13/2019 Concurrency in the Go Programming Language
17/18
)ecause all of the $ork is happening in goroutines, as soon as any t$o adIacent goroutines
are ready, a ne$ #alue can begin tra#ersing through the pipeline 'he adIacent goroutines
synchronie upon e1changing data
hannels can also be mul"iple!ed
Hny number of goroutines can $rite to a single channel, and any number of goroutines can
read from a single channel Go e#en has a special select statement $hich, gi#en a list of
channels, reads from and e1ecutes a block for a random, $aiting channel 'his allo$s
channels to be easily de8multiple1ed
7or another e1ample, let’s use the random selection amongst $aiting channels to implement a
ridiculous random number generator:
package main
func main! ch1 makechan int! ch makechan int!
go func! for J-ch1 & &!
for select case ch1 J- 0 print0! case ch1 J- 1 print1! case ch J- print! & &&
*1ecuting this prints out a random series of /’s and ’s Why? Well, first $e set up t$o
channels for integers, and then kick off a goroutine that does nothing but drain one of those
channels
'hen $e loop infinitely, selecting one of the ready cases =ince the goroutine is al$ays
$aiting to consume any #alue sent on ch1, either of the t$o case statements that send on ch1
could run )y definition, select chooses one those at random, creating a stream of random
data
)ecause nothing is e#er $aiting to read from ch, the case that sends on ch is ne#er
e1ecuted
'hese features make it easy to implement things like @ueues, signaling, etc
/E
8/13/2019 Concurrency in the Go Programming Language
18/18
*roader Applica"ions
*#en if you ne#er plan on using Go, Iust learning about goroutines and channels can pro#ide
a rich mental model for designing concurrent systems "’m particularly fond of this @uote
describing the benefits of good notation:
)y relie#ing the brain of all unnecessary $ork, a good notation sets it free to concentrate on
more ad#anced problems, and, in effect, increases the mental po$er of the race (Hlfred
Jorth Whitehead, Hn "ntroduction to Bathematics, /-//
Hnd if you’d like to go straight to the notation’s source, goroutines and channels take their
inspiration from C H oare’s “Communicating =e@uential rocesses!, a formal language
for describing patterns of interaction in concurrent systems
Py"hon and oncurrency
Puite a fe$ proIects e1ist to make concurrent programming easier in ython "n particular:
• )oth yC= and ython8C= implement C= in ython, and are some$hat acti#e
'he ython8C= proIect is currently planning on merging into yC=
• 'he ))C’s Famaelia is a frame$ork for doing *rlang8style concurrency in ython
• =tackless ython is a fork of Cython $ith added support for “tasklets! and channels
+"her Resources
6sing C= is 'ony >oare’s book on C=, a#ailable as a free D7
'he Go $ebsite at golangorg has absolutely fantastic resources, including great docs, a
bro$ser8based “ playground! N pastebin, an interacti#e tour of the language, a great blog, and
much more
redi"s
Recommended