Software for a Concurrent World

Joe Armstrong

Dallas, Texas • Raleigh, North Carolina

Introduction xiii

Part I — Why Erlang?

1. Introducing Concurrency 3 1.1 Modeling Concurrency 3 1.2 Benefits of Concurrency 6 1.3 Concurrent Programs and Parallel Computers 8 1.4 Sequential vs. Concurrent Programming Languages 9

2. A Whirlwind Tour of Erlang 11 2.1 The Shell 11 2.2 Processes, Modules, and Compilation 13 2.3 Hello, Concurrency 15

Part II — Sequential Programming

3. Basic Concepts 25 3.1 Starting and Stopping the Erlang Shell 25 3.2 Simple Integer Arithmetic 27 3.3 Variables 28 3.4 Floating-Point Numbers 32 3.5 Atoms 33 3.6 Tuples 34 3.7 Lists 37 3.8 Strings 39 3.9 Pattern Matching Again 41

4. Modules and Functions 43 4.1 Modules Are Where We Store Code 43 4.2 Back to Shopping 50

vi • Contents

4.3 Funs: The Basic Unit of Abstraction 52 4.4 Simple List Processing 57 4.5 List Comprehensions 59 4.6 BIFs 63 4.7 Guards 64 4.8 case and if Expressions 68 4.9 Building Lists in Natural Order 70 4.10 Accumulators 71

5. Records and Maps 75 5.1 When to Use Maps or Records 75 5.2 Naming Tuple Items with Records 76 5.3 Maps: Associative Key-Value Stores 79

6. Error Handling in Sequential Programs 87 6.1 Handling Errors in Sequential Code 88 6.2 Trapping an Exception with try...catch 89 6.3 Trapping an Exception with catch 92 6.4 Programming Style with Exceptions 93 6.5 Stack Traces 95 6.6 Fail Fast and Noisily, Fail Politely 96

7. Binaries and the Bit Syntax 99 7.1 Binaries 99 7.2 The Bit Syntax 101 7.3 Bitstrings: Processing Bit-Level Data 110

8. The Rest of Sequential Erlang 113 8.1 apply 115 8.2 Arithmetic Expressions 116 8.3 Arity 116 8.4 Attributes 117 8.5 Block Expressions 120 8.6 Booleans 120 8.7 Boolean Expressions 121 8.8 Character Set 122 8.9 Comments 122 8.10 Dynamic Code Loading 122 8.11 Erlang Preprocessor 126 8.12 Escape Sequences 126 8.13 Expressions and Expression Sequences 127

Contents • vii

8.14 Function References 128 8.15 Include Files 128 8.16 List Operations ++ and - - 129 8.17 Macros 129 8.18 Match Operator in Patterns 131 8.19 Numbers 132 8.20 Operator Precedence 133 8.21 The Process Dictionary 134 8.22 References 135 8.23 Short-Circuit Boolean Expressions 135 8.24 Term Comparisons 136 8.25 Tuple Modules 137 8.26 Underscore Variables 137

9. Types 141 9.1 Specifying Data and Function Types 141 9.2 Erlang Type Notation 143 9.3 A Session with the Dialyzer 148 9.4 Type Inference and Success Typing 152 9.5 Limitations of the Type System 155

10. Compiling and Running Your Program 159 10.1 Modifying the Development Environment 159 10.2 Different Ways to Run Your Program 161 10.3 Automating Compilation with Makefiles 166 10.4 When Things Go Wrong 169 10.5 Getting Help 172 10.6 Tweaking the Environment 173

Part III — Concurrent and Distributed Programs

11. Real-World Concurrency 177

12. Concurrent Programming 181 12.1 The Concurrency Primitives 182 12.2 Introducing Client-Server 184 12.3 Processes Are Cheap 189 12.4 Receive with a Timeout 191 12.5 Selective Receive 193 12.6 Registered Processes 194

12.7 A Word About Tail Recursion 12.8 Spawning with MFAs or Funs

196 197

Errors in Concurrent Programs 199 13.1 Error Handling Philosophy 199 13.2 Error Handling Semantics 202 13.3 Creating Links 203 13.4 Groups of Processes That All Die Together 204 13.5 Setting Up a Firewall 205 13.6 Monitors 205 13.7 Error Handling Primitives 206 13.8 Programming for Fault Tolerance 207

Distributed Programming 211 14.1 Two Models for Distribution 212 14.2 Writing a Distributed Program 213 14.3 Building the Name Server 213 14.4 Libraries and BIFS for Distributed Programming 219 14.5 The Cookie Protection System 222 14.6 Socket-Based Distribution 224

Part IV — Programming Libraries and Frameworks

Interfacing Techniques 231 15.1 How Erlang Communicates with External Programs 232 15.2 Interfacing an External C Program with a Port 234 15.3 Calling a Shell Script from Erlang 240 15.4 Advanced Interfacing Techniques 240

Programming with Files 243 16.1 Modules for Manipulating Files 243 16.2 Ways to Read a FUe 244 16.3 Ways to Write a File 251 16.4 Directory and File Operations 255 16.5 Bits and Pieces 258 16.6 A Find Utility 258

Programming with Sockets 263 17.1 Using TCP 263 17.2 Active and Passive Sockets 272 17.3 Error Handling with Sockets 275

Contents • ix

17.4 UDP 276 17.5 Broadcasting to Multiple Machines 280 17.6 A SHOUTcast Server 281

18. Browsing with Websockets and Erlang 287 18.1 Creating a Digital Clock 288 18.2 Basic Interaction 291 18.3 An Erlang Shell in the Browser 292 18.4 Creating a Chat Widget 293 18.5 IRC Lite 295 18.6 Graphics in the Browser 299 18.7 The Browser Server Protocol 301

19. Storing Data with ETS and DETS 305 19.1 Types of Table 306 19.2 ETS Table Efficiency Considerations 308 19.3 Creating an ETS Table 309 19.4 Example Programs with ETS 310 19.5 Storing Tuples on Disk 315 19.6 What Haven't We Talked About? 318

20. Mnesia: The Erlang Database 321 20.1 Creating the Initial Database 321 20.2 Database Queries 322 20.3 Adding and Removing Data in the Database 326 20.4 Mnesia Transactions 328 20.5 Storing Complex Data in Tables 332 20.6 Table Types and Location 333 20.7 The Table Viewer 336 20.8 Digging Deeper 337

21. Profiling, Debugging, and Tracing 339 21.1 Tools for Profiling Erlang Code 340 21.2 Testing Code Coverage 341 21.3 Generating Cross-References 342 21.4 Compiler Diagnostics 343 21.5 Runtime Diagnostics 346 21.6 Debugging Techniques 347 21.7 The Erlang Debugger 350 21.8 Tracing Messages and Process Execution 352 21.9 Frameworks for Testing Erlang Code 355

x • Contents

22. Introducing OTP 359 22.1 The Road to the Generic Server 360 22.2 Getting Started with gen_server 368 22.3 The gen_server Callback Structure 372 22.4 Filling in the gen_server Template 376 22.5 Digging Deeper 377

23. Making a System with OTP 381 23.1 Generic Event Handling 382 23.2 The Error Logger 384 23.3 Alarm Management 392 23.4 The Application Servers 394 23.5 The Supervision Tree 396 23.6 Starting the System 400 23.7 The Application 403 23.8 File System Organization 405 23.9 The Application Monitor 406 23.10 How Did We Make That Prime? 407 23.11 Digging Deeper 409

Part V — Building Applications

24. Programming Idioms 413 24.1 Maintaining the Erlang View of the World 413 24.2 A Multipurpose Server 416 24.3 Stateful Modules 418 24.4 Adapter Patterns 419 24.5 Intentional Programming 422

25. Third-Party Programs 425 25.1 Making a Shareable Archive and Managing Your Code with

Rebar 425 25.2 Integrating External Programs with Our Code 428 25.3 Making a Local Copy of the Dependencies 430 25.4 Building Embedded Web Servers with Cowboy 431

26. Programming Multicore CPUs 439 26.1 Good News for Erlang Programmers 440 26.2 How to Make Programs Run Efficiently on a Multicore

CPU 441 26.3 Parallelizing Sequential Code 445

Contents • xi

26.4 Small Messages, Big Computations 447 26.5 Parallelizing Computations with mapreduce 451

27. Sherlock's Last Case 457 27.1 Finding Similarities in Data 458 27.2 A Session with Sherlock 458 27.3 The Importance of Partitioning the Data 463 27.4 Adding Keywords to the Postings 464 27.5 Overview of the Implementation 467 27.6 Exercises 469 27.7 Wrapping Up 470

Al. OTP Templates 471 Al. 1 The Generic Server Template 471 A 1.2 The Supervisor Template 474 Al.3 The Application Template 475

A2. A Socket Application 477 A2.1 An Example 477 A2.2 How lib_chan Works 479 A2.3 The lib_chan Code 483

A3. A Simple Execution Environment 493 A3.1 How Erlang Starts 494 A3.2 Running Some Test Programs in SEE 496 A3.3 The SEE API 499 A3.4 SEE Implementation Details 500 A3.5 How Code Gets Loaded in Erlang 508

Index 511