Ruby meets GoDec. 12, 2015

Masaki Matsushita

About Me

● Masaki Matsushita● CRuby Committer

○ 138 Commits■ Mainly for performance improvement■ Marshal.load, Hash#flatten, etc.

● Software Engineer at NTT Communications○ Contribution to OpenStack○ Slide at OpenStack Summit Tokyo

http://goo.gl/OXTYor● Twitter: @_mmasaki Github: mmasaki

Today’s Topic

● Go 1.5 Feature: buildmode “c-shared”○ Cgo Basics

● Using Go Function from Ruby○ FFI and Fiddle without ruby.h

● Writing Extension Library with Go (and C)○ Define Functions Equivalent to C Macros○ Avoid Copy of Strings○ Propagate Reference from Ruby to Go○ Creating Gem including Go code

Buildmode “c-shared”

● Go 1.5 relased in August 2015● Buildmode “c-shared” was introduced

○ go build -buildmode c-shared○ Build C shared library with cgo

● cgo enables:○ Refer to C functions, types and variables○ Export Go functions for use by C

Cgo Example: Say hello with puts() in Cpackage main

/*

#include <stdlib.h>

#include <stdio.h>

*/

import "C"

import "unsafe"

func main() {

cstr := C.CString("Hello, world!")

defer C.free(unsafe.Pointer(cstr))

C.puts(cstr)

}

Include C header file

Convert Go string into C String

Cgo Example: define and use C functionpackage main

/*

char *hello(void) {

return "Hello, world!";

}

*/

import "C"

import "fmt"

func main() {

cstr := C.hello()

fmt.Println(C.GoString(cstr))

}

Define C Function

Convert into Go String

Call C Function from Go

Try c-shared: add.go

package main

import "C"

//export add

func add(a C.int, b C.int) C.int {

return a + b

}

func main() {}

● go build -buildmode c-shared -o add.so add.go

Export Go Function for use by C

Load c-shared Libraries

● ruby-ffi○ https://github.com/ffi/ffi○ gem install ffi

● fiddle○ Standard ext library

● useful to call Go functions simply(without ruby.h)

Call Go Function from Ruby: ruby-ffi

require "ffi"

module Int

extend FFI::Library

ffi_lib "int.so"

attach_function :add, [:int, :int], :int

end

p Int.add(15, 27) #=> 42

Load c-shared library

Add Go Function to Module

Call Go Function from Ruby: fiddle

require "fiddle/import"

module Int

extend Fiddle::Importer

dlload "int.so"

extern "int add(int, int)"

end

p Int.add(15, 27) #=> 42

Go String and C String: str.sopackage main

import "C"

import "fmt"

//export hello

func hello(cstr *C.char) {

str := C.GoString(cstr)

fmt.Println("Hello, " + str)

}

func main() {}

Receive C String

Convert to Go String

Returning String: ruby-ffi

require "ffi"

module Hello

extend FFI::Library

ffi_lib "str.so"

attach_function :hello, [:string], :void

end

Hello.hello("world") #=> "Hello, world"

Ruby String can be passed

Returning String: fiddle

require "fiddle/import"

module Hello

extend Fiddle::Importer

dlload "str.so"

extern "void hello(char *str)"

end

Hello.hello("world") #=> "Hello, world"

Cgo Functions to Convert String

● C.CString(goString string) *C.char

○ copy Go String to C String

○ Users are responsible to free C String

● C.GoString(cString *C.char) string

● C.GoStringN(cString *C.char, length C.int) string

○ copy C String to Go String

Writing Extension Library with Go

● Naruse-san’s Amazing Talk:“Writing extension libraries in Go”at OedoRubyKaigi 05https://speakerdeck.com/naruse/writing-extension-libraries-in-go

● gohttp: https://github.com/nurse/gohttp○ Implementation of extension library in Go

C Extension Library Basicsstatic VALUE

rb_magic_number(VALUE self)

{

return INT2NUM(42);

}

void

Init_test(void)

{

rb_cFoo = rb_define_class("Foo");

rb_define_method(rb_cFoo, "magic_number", rb_magic_number, 0);

}

C Extension Library Basicsstatic VALUE

rb_magic_number(VALUE self)

{

return INT2NUM(42);

}

void

Init_test(void)

{

rb_cFoo = rb_define_class("Foo");

rb_define_method(rb_cFoo, "magic_number", rb_magic_number, 0);

}

Method Implementation

C Extension Library Basicsstatic VALUE

rb_magic_number(VALUE self)

{

return INT2NUM(42);

}

void

Init_test(void)

{

rb_cFoo = rb_define_class("Foo");

rb_define_method(rb_cFoo, "magic_number", rb_magic_number, 0);

}

Using C Macro

C Extension Library Basicsstatic VALUE

rb_magic_number(VALUE self)

{

return INT2NUM(42);

}

void

Init_test(void)

{

rb_cFoo = rb_define_class("Foo");

rb_define_method(rb_cFoo, "magic_number", rb_magic_number, 0);

}

Function Pointer

Minimal Go Ext Example?//export rb_magic_num

func rb_magic_num(self C.VALUE) C.VALUE {

return INT2NUM(42)

}

//export Init_foo

func Init_foo() {

rb_cFoo = rb_define_class("Foo", C.rb_cObject)

rb_define_method(rb_cFoo, "magic_num", C.rb_magic_num, 0)

}

Writing Extension Library with Go

● Wrapper Function equivalent to C Macro○ C macros can’t be used by Cgo

● Convert Go String into Ruby without Copy● Propagate Ruby Reference to Go● Create gem including Go code

○ Modify Rakefile and extconf.rb

C Macros for Ruby Extention Libraries

● Useful C macros are defined in ruby.h○ INT2NUM: C int to Ruby Numeric○ NIL_P: true if obj is nil○ RSTRING_PTR: pointer to buffer of String○ RSTRING_LEN: lengh of String

● These macros can’t be used from Cgo…● Define Go functions equivalent to C macros

○ Use equivalent C function○ Wrap C macros with C function

Use Equivalent C Function

func LONG2NUM(n C.long) C.VALUE {

return C.rb_long2num_inline(n)

}

func NUM2LONG(n C.VALUE) C.long {

return C.rb_num2long(n)

}

Wrap C macros with C functionpackage main

/*

long rstring_len(VALUE str) {

return RSTRING_LEN(str);

}

*/

import "C"

func RSTRING_LEN(str C.VALUE) C.long {

return C.rstring_len(str)

}

Convert Go String into Ruby without Copy

● Go String -> C String -> Ruby String

● C.CString(goString string) *C.char

○ copy Go String to C String

○ Users are responsible to free C String

● VALUE rb_str_new(const char *ptr, long len)

○ copy C String to Ruby String

Basic Usage of C.CString()

// go/doc/progs/cgo4.go

func Print(s string) {

cs := C.CString(s)

defer C.free(unsafe.Pointer(cs))

C.fputs(cs, (*C.FILE)(C.stdout))

}

● Call C func and discard C str soon

Basic Usage of C.CString()

str := "Hello, world!"

// Copy #1

cstr := C.CString(str) // will be discarded soon

// Copy #2

rbstr := C.rb_str_new(cstr, C.long(len(str)))

● Need to copy twice!

Avoid Copy of Strings

● Get *C.char from Go String without Copy

func GOSTRING_PTR(str string) *C.char {

bytes := *(*[]byte)(unsafe.Pointer(&str))

return (*C.char)(unsafe.Pointer(&bytes[0]))

}

// example of use

cstr := GOSTRING_PTR(str) C.rb_utf8_str_new

(cstr, C.long(len(str)))

Avoid Copy of Strings

● Technique to Get []byte from Go w/o Copy http://qiita.com/mattn/items/176459728ff4f854b165

func GOSTRING_PTR(str string) *C.char {

bytes := *(*[]byte)(unsafe.Pointer(&str))

return (*C.char)(unsafe.Pointer(&bytes[0]))

}

Avoid Copy of Strings

● Get *C.char from []bytefunc GOSTRING_PTR(str string) *C.char {

bytes := *(*[]byte)(unsafe.Pointer(&str))

return (*C.char)(unsafe.Pointer(&bytes[0]))

}Cast to char

Example Usage of GOSTRING_PTR()

func RbString(str string) C.VALUE {

if len(str) == 0 { return C.rb_utf8_str_new(nil, C.long(0)) }

return C.rb_utf8_str_new(GOSTRING_PTR(str), GOSTRING_LEN(str))

}

func rb_define_class(name string, parent C.VALUE) C.VALUE {

return C.rb_define_class(GOSTRING_PTR(name), parent)

}

func rb_define_method(klass C.VALUE, name string, fun unsafe.Pointer, args int) {

cname := GOSTRING_PTR(name)

C.rb_define_method(klass, cname, (*[0]byte)(fun), C.int(args))

}

Propagate Ruby Reference to Go

● Go’s GC doesn’t know refs from Ruby● Go obj referenced from Ruby can be collected● We have to propagate Ruby Refs to Go● Use Map to keep reference to Go Objects

Propagate Ruby Reference to Govar objects = make(map[interface{}]int)

//export goobj_retain

func goobj_retain(obj unsafe.Pointer) {

objects[obj]++ // increment reference count

}

//export goobj_free

func goobj_free(obj unsafe.Pointer) {

objects[obj]-- // decrement reference count

if objects[obj] <= 0 { delete(objects, obj) }

}

Propagate Ruby Reference to Gostatic const rb_data_type_t go_type = {

"GoStruct",

{NULL, goobj_free, NULL},

0, 0, RUBY_TYPED_FREE_IMMEDIATELY|RUBY_TYPED_WB_PROTECTED

};

VALUE

NewGoStruct(VALUE klass, void *p)

{

goobj_retain(p);

return TypedData_Wrap_Struct((klass), &go_type, p);

}

Increment Reference Count

Decrement Reference Count

Create Gem including Go code

● Directory Structure● Rakefile● extconf.rb

Directory Structure

● Use “bundle gem --ext foo”

├── ext

│ └── foo

│ ├── extconf.rb // configured to use go build

│ ├── foo.c // helper functions for use by Go

│ └── foo.h // export helper functions

│ ├── foo.go // created by hand

│ └── wrapper.go // created by hand

└── lib

Rakefile

require 'bundler'

Bundler::GemHelper.install_tasks

require 'rake/extensiontask'

task :default => [:compile]

spec = eval File.read('foo.gemspec')

Rake::ExtensionTask.new('foo', spec) do |ext|

ext.lib_dir = File.join(*['lib', 'foo', ENV['FAT_DIR']].compact)

ext.ext_dir = 'ext/foo'

ext.source_pattern = "*.{c,cpp,go}"

end

● Add .go into source_pattern

extconf.rb

require 'mkmf'

find_executable('go')

$objs = []

def $objs.empty?; false ;end

create_makefile("memberlist/memberlist")

case `#{CONFIG['CC']} --version`

when /Free Software Foundation/

ldflags = '-Wl,--unresolved-symbols=ignore-all'

when /clang/

ldflags = '-undefined dynamic_lookup'

end

● Some techniques to build successful

extconf.rb

File.open('Makefile', 'a') do |f|

f.write <<-EOS.gsub(/^ {8}/, "\t")

$(DLLIB): Makefile $(srcdir)/memberlist.go $(srcdir)/wrapper.go

CGO_CFLAGS='$(INCFLAGS)' CGO_LDFLAGS='#{ldflags}' \

go build -p 4 -buildmode=c-shared -o $(DLLIB)

EOS

end

● Modify Makefile to use go build

Ruby meets Go

● Buildmode “c-shared” and Cgo Basics● Using Go Function from Ruby

○ FFI and Fiddle without ruby.h● Writing Extension Library with Go

○ Define Functions Equivalent to C Macros○ Avoid Copy of Strings○ Propagate Reference from Ruby to Go○ Creating Gem including Go code

● Let’s Hack Go for Ruby together!