Xtreams - Michael Lucas-Smith and Martin Kobetic

Xtreams

'Don't cross the streams!'

Michael Lucas-SmithMartin Kobetic

Cincom Smalltalk EngineeringSmalltalk Solutions 2011

Status

Trunk: StableDocumentation: http://code.google.com/p/xtreamsLicense: MITProject started: November 30th 2007

PlatformsVisualWorksSqueak and Pharo (Nicolas Cellier)Gemstone (Dale Henrichs)Slate (Brian Rice)

... thanks to all who have provided feedbackand patches

Why?

simplification and consistencyIncomplete vs atEndpositioning and 'surprising' buffersread and write are different

compositionno deep class hierarchy any more

capabilitiestransformationssubstreams

scalabilitychunks vs elementsrecycling

Structure

coreDefines the API and Core Classes.Support package is for portability.

terminalsUnique to each platform, wraps the source and destination objects in to streams.

transformscollecting, rejecting, selecting...character encodingbinary transformationsobject marshaling

Structure

substreams / parsinglimiting, element or subcollection endingslicing and stitchingparsing expression grammarsmultiplexing (experimental)

xtrascompressioncryptographychunking

applicationsSSH2 (Xtreams-SSH2)IRC (Xtreams-IRC)EXIF Metadata (Xtreams-Photo)

reading / writing

collections'example' reading.OrderedCollection new writing.Transcript writing.

(RingBuffer new: 4) writing.SharedQueue new writing.

reading / writing

externals'example.txt' asFilename reading.OS.Stdin reading.(External.CIntegerType unsignedChar

gcMalloc: 50) writing.(OS.SocketAccessor

newTCPclientToHost: 'localhost'port: 1234) reading.

transcendentals[0] reading. /dev/zero[:x | ] writing. /dev/nullKernel.ObjectMemory reading.Random new reading.

API

commonIncomplete.stream terminal.stream close.stream isReadable; isWritable; isPositionable.

readinginput get.input read: 5.input read: 5 into: collection.input read: 5 into: collection at: 2.input rest.

do:, do:separatedBy:, select:, reject:, collect:, fold:,detect:(ifNone:), inject:into:, groupedBy:

API

writingoutput put: 'example'.output write: $x.output write: input.output write: 'example'.output write: 4 from: input.output write: 4 from: 'example'.output write: 3 from: 'example' at: 4.output conclusion.

output insert: 'example'.output insert: 4 from: 'example'.output insert: 3 from: 'example' at: 4.

output print: #( 1 2 3 ).output cr; bell; q; qq; space; space: 2; tab; tab: 3

API

seekingpositionable := stream positioning.

transform a non-positionable stream in to a positionable stream

stream ++ 200.seek forward from where we are

stream -- 200.seek backward from where we are

stream += 200.seek forward from the start of the stream

stream -= 0. (skip to end)seek backward from the end of the stream

API

seekingstream explore: [stream read: 2].

seek within the block, but return to where we were once done

stream position. stream position: 4.change the position of the stream

stream available.elements left to consume on the stream(not to be mistaken with data left in socket buffer)

stream length.elements in the stream

QA

Questions?

Transforms

Collection Stylecollecting:, selecting:, injecting:into:, doing:, ...

Specialized Transformsencoding:, encodingBase64, encodingHex,compressing, en/decrypting:key:iv:, hashing:interpreting:, marshaling

General Transformstransforming: [ :in :out | out put: in get ]

Collection Transforms

random := Random new reading.random := random collecting: [ :f | (f * 256) floor ].random contentsSpecies: ByteArray.random read: 10.

current := 0.integers := [current := current + 1] reading.integers read: 10.even := integers selecting: [ :i | i even ].even read: 10.

Character Encoding

input := 'xtreams.cha' asFilename reading.input := input encoding: #utf8.input read: 50.input close.

(#[13 10 10 13] reading encoding: #ascii) rest.(ByteArray new writing encoding: #ascii)

cr; conclusion

Composition / Stacking

ones := [ 1 ] reading.twoAndUp := ones

injecting: 1into: [ :previous :one | previous + one ].

sieve := OrderedCollection new.primes := twoAndUp rejecting: [ :i |

(sieve anySatisfy: [ :p | i \\ p = 0 ])ifTrue: [ true ]ifFalse: [ sieve add: i. false ] ].

primes read: 10.

Morse Code

Message... -- .- .-.. .-.. - .- .-.. -.-

DecodingTree - << * >> . T E M N A I O G K D W R U S Q Z Y C X B J P L F V H

($ ($T ($M ($O) ($G ($Q) ($Z))) ($N ($K ($Y) ($C)) ($D ($X) ($B))) ($E ($A ($W ($J) ($P)) ($R () ($L))) ($I ($U () ($F)) ($S ($V) ($H)))

Morse Code Decoding

($ ($T ($M ($O) ($G ($Q) ($Z))) ($N ($K ($Y) ($C)) ($D ($X) ($B))) ($E ($A ($W ($J) ($P)) ($R () ($L))) ($I ($U () ($F)) ($S ($V) ($H)))

('... -- .- .-.. .-.. - .- .-.. -.- ' readingtransforming: [ :in :out || node beep |

node := MorseTree.[ beep := in get.

beep = Character space ] whileFalse: [

node := beep = $.ifTrue: [ node at: 3 ]ifFalse: [ node at: 2 ] ].

out put: node first ]) rest

Morse Code Encoding

(String new writingtransforming: [ :in :out |

out write: (Morse at: in get);space ]

) write: 'SMALLTALK';conclusion

QA

Questions?

Substreams

* sentences in text* parts of messages in a protocol* files in an archive

size(Object comment reading limiting: 10) rest.

bounding criteria(Object comment reading ending: $.) rest.(Object comment reading ending: [:e | '.?!' includes:(Object comment reading ending: ' is the') rest.

streams of substreamsslicing streams into substreamsstitching streams from substreams

Substreams - limiting:

output := String new writing.Number withAllSubclasses do: [ :class |

[ (output limiting: 40) write: class comment.] on: Incomplete do: [ :ex | output -- 3; write: '...' ].output cr ].

output conclusion.

Substreams - ending:

output := String new writing.Number withAllSubclasses do: [ :class |

[ (output ending: $. inclusive: true)write: class comment

] on: Incomplete do: [].output cr ].

output conclusion

Slicing

readinginput := 'xtreams.cha' asFilename reading.input := input encoding: #utf8.[ lines := (input ending: Character cr) slicing.

(lines ++ 10000; get) rest] ensure: [ input close ].

writingoutput := String new writing.blurbs := (output limiting: 40) slicing.Number withAllSubclasses do: [ :class |

[ blurbs get write: class comment.] on: Incomplete do: [ :ex | output -- 3; write: '...' output cr ].

output conclusion.

Slices of Slices

readinginput := 'aaa#bb#c##!1#22#33#444' reading.messages := (input ending: $!) slicing.parts := (messages get ending: $#) slicing.parts collect: [ :p | p rest ].

writingoutput := String new writing.messages := (output closing: [ output put: $! ])

slicing.#((aa bb cc dd ee) (xxx yy z)) do: [ :m |

message := messages get.parts := (message closing: [ message put: $# ])

slicing.m do: [ :p | parts get write: p ] ].

output conclusion

Stitching Reads

((1 to: 5) reading, (6 to: 10) reading) rest.((1 to: 10) reading limiting: 3) slicing stitching rest.

files := '..' asFilename reading.[ | fn | fn := files get.

fn isDirectory ifTrue: [ files := fn reading, files ].fn

] reading collect: [ :f | f asString ].

directories := ElasticBuffer new: 10 class: Array.directories put: '..' asFilename.[ directories get reading doing: [ :filename |

filename isDirectory ifTrue: [directories put: filename]]

] reading stitching collect: [ :f | f asString ].

Stitching Writes

output := ByteArray new writing.buffer := RingBuffer on: (ByteArray new: 5).[ (buffer writing limiting: buffer cacheSize)

closeBlock: [output put: buffer readSize;

write: buffer ];yourself

] reading stitchingwrite: (1 to: 12);close.

output conclusion

QA

Questions?

Back to Binary

Stream --> ObjectsInterpreting streamsObject marshalingParsing Expression GrammarsErlang 'Bit Syntax'

Objects --> StreamInterpreting streamsObject marshaling

Interpreting

stream interpreting: #long.stream interpreting: #signedLonglong_be.

(input interpreting: #float) read: 10.(output interpreting: #double) put: Double pi.

Marshaling

goalsCross-Smalltalk (not achieved yet)Space EfficiencySpeed Efficiency

to replace...BOSSOpentalk-STSTParcels (sort of)SIXX

Marshaling

ObjectMarshalerSeparate from the Read/Write streams.Pluggable marshaling strategy, for different protocolsUses Pragmas to allow extensions with a computed hash to detect different versions.

AnalysisRead without instantiating objects to diagnose a stream.

Marshaling

output := ByteArray new writing marshaling.output put: 100 asValue.

output conclusion = #[83 84 83 84 20 4 21 199 91 7 32 29 82 111 111 116 46 83 109 97 108 108 116 97 108 107 46 85 73 46 86 97 108 117 101 72 111 108 100 101 114 0 1 28 22 100]

Marshaling

(Xtreams.ObjectAnalyseStream on:output conclusion reading) rest =

'0+10 header: #[83 84 83 84 20 4 105 117 236 8]10+36 record

10+1 class id: 3211+31 class description: UI.ValueHolder42+1 object id: 143+3 object: UI.ValueHolder

43+1 dependents43+1 class id: 28 Core.UndefinedObject

43+1 class id: 2844+0 nil

44+2 value44+2 class id: 22 Core.SmallInteger

44+1 class id: 2245+1 byte integer: 100'

Marshaling

output conclusion reading marshaling get.a ValueHolder on: 100

Parsing

Parsing Expression GrammarsErlang 'bit syntax'Reusable GrammarsGrammar composition

Parsing

Parsing Expression Grammars

grammar := 'Sentence <- Whitespace? (Word / Punctuation)* Word <- [a-zA-Z'']+Punctuation <- [,;:()]'.

sentenceParser := PEG.Parser parserPEGparse: 'Grammar'stream: grammaractor: PEG.ParserParser new

Parsing

ApplicationsTransformed Namespaced class names of Xtreams during import/export via Monticello using the Smalltalk PEG grammar with an Actor that contained a single method.

Implemented an IRC client where data read from the client stream was parsed with a grammar and interpreted using a single method on the IRC client itself. The whole IRC implementation is one class.

Parsing

ApplicationsTransformed Smalltalk, Javascript, XML, CSS3 grammars in to a Javascript lexer to perform syntax color highlighting in javascript in the web browser for WebVelocity 1.1

This presentation. The slides of this presentation are built by parsing Smalltalk methods comprised of wiki syntax in comments and smalltalk code. The wiki grammar is used with an actor to generate Text objects that display what you see right now.

Parsing

Parsing Expression GrammarsPEG: character streams onlyRead one character: .Read one of many characters: [a-zA-Z]Read a sequence of characters: ''TITLE''

Moving beyond... Grammar CompositionCompile grammar into bytecode on lightweight classRead one character: {character/utf8}Read one byte: {integer/unsigned/little}:8Read one of many characters in utf16:

[a-zA-Z]/utf16Read a sequence of characters in utf8:

''TITLE''/utf8

QA

Questions?

Xtras

Various non-core transformations

Compressioncalls ZLib

Cryptographycalls BCrypt (CNG) on Windowscalls OpenSSL's libcrypto everywhere elsehashing (MD5, SHA1, SHA256, SHA512, ...)hashing with key (HMAC with MD5, SHA1, ...)encryption (RC4, AES, DES, ...)

HTTP Chunking

Xtras - Examples

hashing(ObjectMemory imageFilename reading

hashing: 'MD5') -= 0; close; digest.

encryptionkey := random read: 16.((String new writing

encodingBase64encrypting: 'RC4' key: key)compressingencoding: #utf8

) write: Object comment;conclusion.

Xtras - SSH2

doestransportauthentication: public-key, passwordchannel/session managementdata transferchannel requests: exec, env, exit-statusscp

does notterminal session/emulation: pty-req, shellport forwarding (tunnels)scp atime/mtime

Xtras - SSH2

keys := SSH2KeysTest sampleKeys.config := SSH2Configuration new keys: keys.server := ('-thishost-' asIPv4: 2222) listen accepting.[ (SSH2ServerConnection on: server get)

configuration: config;when: SSH2Announcementdo: [ :m | Transcript cr; print: m ];accept;waitForDisconnect;close

] ensure: [ server close. keys release ]

Xtras - SSH2

home := '$(HOME)' asLogicalFileSpecification asFilenamuser := home tail.keys := SSH2Keys fromUser: home.config := SSH2Configuration new keys: keys.client := ('localhost' asIPv4: 22) connect.client := SSH2ClientConnection on: client.client configuration: config.client when: SSH2Announcement

do: [ :m | Transcript cr; print: m ].[ service := client connect: user.

session := service session.[ session exec: 'ls -l'] ensure: [ session close ]

] ensure: [ client close. keys release ]

QA

Questions?