How a Clojure pet project turned into a full-blown cloud-computing web-app

Berlin, February 2, 2010

Presentation, Release 0.1

Hamburg Lispers @ Berlin Lispers Meetup

‘How a Clojure pet project turned into a

full-blown cloud computing web app.

Or: What are the differences between a Clojure web app and one written in Java or Common Lisp?’

© freiheit.comBerlin Lispers MeetupPresentation, Rel. 0.1

Thanks to Franz Lisp and the newthinking store here in Berlin for making this meetup possible!

2


Hi, I am Stefan Richter, CTO and founder of freiheit.com technologies.

3


Why am I here?- We are working on a Clojure web-app- This app runs on Google App Engine- We use this app in our own organization- We want to provide the app as a software service- Why did we choose Clojure for such a mission critical project?- We want to share our experiences with you:

- How Clojure made us more productive in developing web applications (and how much fun it is!)- What are the differences to Java and Common Lisp when we talk about web development?

4


But first things first:What kind of app is this?

5


I have to tell a little story first ...

6


My company freiheit.com has been developing large internet systems for more than 10 years.

7


We are about 70 people.Only 7 are non-hackers.So 90% are hackers.

8


We have no project managers.The programmers ‘manage’ their projects themselves.

9


We are very efficient.We are very well organized.How does this work?

10


We have strong principles on how to plan and deliver software. We do this in a way which is natural to hackers.All principles are supported by internal software systems.

11


Note: The way we work is ‘agile.’But I would not call it that because I hate all this ‘agile’ hype.We have been working like this since 1999.We know what we are doing.

12


So our principles have been forged not only into our culture, but also into our internal ‘project management’ systems.

13


Our main system is ‘Captain Feature,’ which provides task-management and measures for self- and team management.

14


The ‘Captain,’ as we fondly call him, is an intranet app, written in Java, over 8 years old.

15


In 2008 I started a pet project to replace the old ‘Captain’ with new software written in Common Lisp.

16


I was the only programmer on this project.But I got very far.I built a nice web-app using sbcl, hunchentoot, cl-who, cl-sql, JavaScript and CSS.

17

© freiheit.comBerlin Lispers MeetupPresentation, Rel. 0.1 18


We thought:Our tools are so useful to us, why don’t we share them with other programmers?

19


This would mean:Provide our internal systems as external services.

20


We don‘t want to own hardware.And we don‘t want to operate our own server farm.Not even a virtual one.

21


So we checked Google App Enginebecause it promisedzero maintenance and a price model based on the resources you actually use.

22


Problem: App Engine only runs Python and Java.

But we wanted Lisp.

23


Clojure is a Lisp and it runs on the JVM.If it runs on the JVM, it runs on App Engine, too!

24


First, we started to design and implement a ground-breaking to-do manager.The plan was to later build the new ‘Captain’on top of this to do manager infrastructure.

25


We call this to-do manager‘TheDeadline’.

26


‘TheDeadline’ is an intelligent to-do manager that behaves like a real, human personal assistant. It supports you. I doesn’t annoy you. It helps you achieve your goals.

27


I showed a screenshot of an early version of ‘TheDeadline’ in a lightning talk at the European Common Lisp Meeting 2009:

28



And this is how it looks today. (We release a new version every week. It is changing fast! Maybe I can give you a demo later):

30



Our vision? The dream of every hacker on this planet!

32


‘We want to eliminate all human project management work on this planet.’

33


Don’t get me wrong:We want to eliminate the ‘project management work’ not the ‘project managers.’;)

34


‘Instead we provide AI-based systems that help people (especially hackers) to become more productive and help them to manage themselves!’

35


We use AI for Intelligence Amplification and not to replace humans.

36


Sounds crazy?!

37


To say it in the words of Mike Muir of Suicidal Tendencies:‘I’m not crazy. You’re the one that’s crazy.’

38


With so many projects still failing today, we urgently need intelligent software for project management ...

39


With an AI system doing 80% of the work that a human assistant would do for you, you can focus on the things that AI can’t do (today).And: ‘TheDeadline’ never sleeps and doesn’t need feeding.

40


Here is why I don’t like the popular to do manager on the web: They are just list management apps with no intelligence at all. They don’t help you. They just store your to-dos.

41


‘TheDeadline’ is currently in Closed Beta. Public Beta starts soon.(Sign-up link at the end of the presentation.)

42


Now that you understand why we did what we did, we can start with the technical part ... :)

43


Most people praise Clojure because of the concurrency features. Lots of stuff has been written about it. So, I won’t talk about this.(You can’t start a thread in App Engine anyway ...)

44


So how does Clojure improve your life as a programmer, especially in web development?!

45


Three things that make your life much easier:1. Structs vs. objects2. Lisp syntax (data = code)3. Macros (extending the language)

46


Of course first-class/higher-order functions, everything being a sequence, immutability, the java virtual machine, etc., are all important factors. But let us take a closer look at the data structures and code generation features ...

47


1. Structs vs. objects.

48


I used CLOS - which is cool, especially compared to Java’s OO system.But sooner or later you get the feeling that you don’t need much OO in a functional language.

49


You don’t want that much state anyway. You want to fold, transform and destructure your data in your app all the time.And: Not everything is an object.

50


BTW: Putting objects into a relational database was called ‘impedance mismatch’ in the 1990’s. Today it is called ‘persistence mapping.’But this doesn’t fix the problems.

51


Did you ever ask yourself why not all data structures are represented in the same style? Especially when you use maps in your code, send JSON to your client and store data in a key-value store?

52


This is what Clojure offers:

53


The following examples are taken from „Programming Clojure,“ Stuart Halloway, Pragmatic Programmers Series.Cool book. Buy it! :)


You define a struct. You instantiate structs. Same structs share their keys. They behave like hash tables (maps). They are immutable. They are human-readable.

55


The Clojure structs combined with multimethods are a really smart idea. It’s like in CLOS: Classes and methods are separated. But I think structs are a more readable and more flexible concept.

56



You can even specialize a method on more than one class/type.(Which is unknown in the Java world).

58


Important: You don’t have to use multimethods. But sometimes they help to reduce the number of different function names in your namespace.

59


2. Lisp syntax (data = code)

60


Most people say they are afraid of the parenthesis.Maybe that is why 31% of the JAX mag readers think that Scala has the most potential of the JVM languages. Only 15% bet on Clojure ...

61


Lisp syntax is easy to learn because there are almost no syntax rules. Your programs are essentially expression trees built from function calls.Code generation is an integral part of the language.

62


This is a function call:(+ 3 2)This is data:‘(+ 3 2)You can translate data into a function call at runtime and execute it!

63


So, functions generate functions. You generalize functions and parameterize them by passing other functions to them. There is no boilerplate code. Just small functional building blocks put together.

64


3. Macros (extending the language)

65


Macros are Lisp programs that generate other Lisp programs.

66


Note: This is not the same as self-modifying assembler code! ;)

67


The Lisp syntax is the precondition to make language extension very easy!(Yes, this parenthesis.)

68


BTW: Scala has ‘no parenthesis’, which is why it is so bloated with syntactic sugar ...

69


Why would I want to extend the language?

70


In other languages, you define an API to implement a subsystem for a specific domain, like database access or html templating. This API is then encapsulated in a library.

71


To use such an API, you normally build an integration layer between your application code and the API. You need this abstraction, but it also requires extra work.

72


Instead, by extending the language with some kind of domain-specific language constructs, you don’t need an integration layer anymore. This layer still exists, but it is generated by the macro system.

73


Most of the time, you can change the code generation inside the macros without having to change the code that is using the macros.

74


First example: html templating. The current approach in other languages is broken. Putting your application code into the html code is wrong. And mixing html code with your app code is wrong.

75


This is why there are so many complicated templating frameworks for Java: You have to mix two different languages - Java and html.

76


But what if you could just extend your language to ‘know’ how to generate html?

77


If everything would ‘feel’ like one language?

78


This example is taken from the TheDeadline’s printer-friendly tabular to do list.You build the html by using language constructs that ‘know’ how to generate html. You put function calls into the right spots to fill in the blanks. And you just call this as a function if you want to display a to do.


Second example: Accessing the Google App Engine datastore. You can use a high-level Java interface providing Java objects or you can use a low-level interface to work directly on the key-value structures.

80


The key-value structure of the Google datastore entities is almost an exact fit to our Clojure structs! So we decided to use the low-level API to the datastore.

81


In addition to that, we wanted to have an abstraction layer to make the application itself independent from the underlying Google platform.

82


This is how you would define a book entity in Python ...

83

The following examples are taken from „Programming Google App Engine,“ Dan Sanderson, O‘ReillyCool book. Buy it! :)


This is how it is done in Java...

84

Not enough space to show all the code that is needed for this example ...


And this is how we do it in Clojure:

85

This automatically generates the code to generate, store and retrieve entities of the type book. It converts the types to and from Google App Engine types and you can also add your own functions to validate and to convert every single property to and from a custom type (not shown in this example).We will write a blog post about it and post the code:http://www.hackers-with-attitude.com)

http://www.hackers-with-attitude.com



Our Clojure data structure looks almost like the low-level key-value structure from Google’s datastore.We can work directly with it in our app. But the user of defentity doesn’t have to know how to use the datastore API at all.

86


The difference to Python:We don’t declare types, but you can define functions for :pre-save and :after-read execution for every property and for the whole entity. We think, this is much more concise and much more powerful.

87


One last example: We are using several AI techniques.

We implemented a rule-based expert system that provides the functionality of systems like CLIPS, LISA or OPS5.

88


A snippet from the famous Monkey & Banana problem:

89


We extended the language to be able to do expert system tasks.So, we are not just creating APIs. We create powerful abstractions that generate their own integration layers on the fly.

90


So, what is our conclusion? What made Clojure more fun and more productive for us in comparison with Java (or Scala)?

91


This is where you spend lots of your time in Java web development. It’s all about moving data around:

92

User Interface

Application

Database/Persistence

Generating HTML, JSON from application objectsTranslating HTTP-Requests to application objects

Translating application objects to SQL statementsTranslating SQL queries to application objects

Folding/Destructuring your

data in your app code


So you access a database, generate objects, iterate over objects, copy objects, convert objects to other objects, convert objects to JSON, to XML, etc., etc., etc.That is what you do in Java all the time...

93


There are lots of Java frameworks that help you to generate the needed code because it is almost impossible to do this by hand. (That is also why you need Eclipse at all ...)

94


But these frameworks add a lot of complexity. You need XML configuration and annotations.Java programming becomes more and more a configuration task.

95


The edit-build-deploy-run cycle is very slow. All the code that is generated by the frameworks, the dependency injections, the start of the app server ...You go through this for every change you make.

96


Our experience is that it is much better to have a simple unified data structure (struct) that you can pass around in your system and build powerful abstractions/mini-languages (with macros) to work on these data structures ...

97


... and to use a real dynamic and interactive programming environment that lets you change and test your code in a running system.

98

Interactive programming even works with the Google App Engine dev server. We posted an article in our blog explaning how to setup your environment. After that, you need ‘ant’ only for the deployment on the App Engine production servers.


One more thing: You don’t need Eclipse anymore. So you don’t have to reserve 1.0 Gigs on your machine for your editor/compiler anymore.

99


But what aboutCommon Lisp?

100


You can do the same in Common Lisp!You could implement the same features that Clojure has in Common Lisp and vice versa. That’s the power of the Lisp programming languages!

101


But nobody is doing it.The Common Lisp Standard is hardly changing anymore.

102


Cool people are writing world-class libraries for Common Lisp.But the platform itself is not changing anymore.

103


Common Lisp is still cool, but Clojure is improving very fast. It’s a modern Lisp.

104


For us, Clojure is the next evolutionary step of Common Lisp.

105


That’s why we helped fund Rich Hickey’s development efforts on Clojure with a donation.

106


So if you want to learn Clojure, you can learn lots of advanced stuff from Common Lisp books.Especially from Paul Graham’s ‘On Lisp.’

107


And it is much easier to sneak some Clojure code into the enterprise Java project your 40-person team is currently working on than to sneak in some Common Lisp code ... ;)

108


Start tomorrow!

109

Thank you.

TheDeadline Pre-Registration:https://the-deadline.appspot.com

Our corporate Blog:http://www.hackers-with-attitude.com

Looking for a job?http://www.freiheit.com

Self:http://www.smartrevolution.dehttp://www.twitter.com/smartrevolution

https://the-deadline.appspot.com

https://the-deadline.appspot.com





http://www.freiheit.com

http://www.freiheit.com

http://www.smartrevolution.de

http://www.smartrevolution.de

http://www.twitter.com/smartrevolution

http://www.twitter.com/smartrevolution

Technology

How a Clojure pet project turned into a full-blown cloud-computing web-app