Pro Meteor Book

  • View

  • Download

Embed Size (px)


Meteor Reaction

Text of Pro Meteor Book

  • Pro Meteor

    Arunoda Susiripala

    Introduction to Pro Meteor

    Pro Meteor is a guide that has been created as a reference for building aproduction-quality Meteor apps. Main focus of this guide is to show you on howto scale Meteor apps and maintain them with existing web proven technologies& services.

    The guide assumes that its reader has a basic understanding of Meteor, butfor readers that need a better introduction, Discover Meteor comes highlyrecommended.

    Good Luck & Happy Reading!

    Arunoda Susiripala

    Hacker at MeteorHacks

    Understanding Meteor Internals

    This section is an introduction to Meteors internals, and serves as the foundationfor the Pro Meteor guide.

    Meteor Internals: Meteor as a Server and a Client

    A Meteor application is seen by browsers, proxy servers, routers and othernetwork components as a typical web application. Yet Meteor is comprised oftwo main components: a part that runs inside the browser and another partthat runs inside the server. These two parts are configured to communicatewith each other in a way thats similar to modern web applications

    e.g. Gmail and Trello

    Meteor allows developers to build applications without worrying about thecomplexities of client-server connectivity.


  • Figure 1: Meteor has 2 parts, which run on the browser and the server

    Meteor Handles Three Different Types of Requests

    Underneath its surface, Meteor handles 3 types of requests. They are:

    Static Files DDP Messages HTTP Requests

    Static Files

    Static files are images and other resources inside the /public folder. Meteorserves these files automatically when the app starts.

    Additionally, Meteor minifies and concatenates all JavaScript (including tem-plates, which are pre-compiled as JavaScript) and CSS files, and serves them asstatic files.

    DDP Messages

    DDP is the protocol Meteor uses to communicate between client and server.All client-side subscription data, method calls and MongoDB operations arecommunicated as DDP messages. This is a very lightweight protocol. Thesemessages can be inspected with a handy tool called ddp-analyzer.


  • HTTP Requests

    While there is no official documentation, Meteor can handle HTTP Requestssimilar to other traditional applications. For example, file uploads to a Meteorapp are sent as HTTP Requests. See this StackOverflow question for details onhow to manually handle HTTP requests.

    Meteor has Two Types of Servers

    Although Meteor runs on a single port, internally it works as two separateservers:

    HTTP Server DDP Server

    Figure 2: Meteor Server is a combination of a HTTP Server and a DDP Server

    HTTP Server

    The HTTP server is used to serve static files and HTTP requests. Meteor usesthe connect Node.js module for this purpose.

    DDP Server

    The DDP server handles all publications, MongoDB operations and Meteormethods. Meteor uses SockJS as the transport protocol. In actuality, the DDPserver is a SockJS server customized for Meteor.


  • Future guides will detail how to scale these two servers separately.

    MongoDB and Meteor

    Its possible to scale Meteors HTTP and DDP servers by running multipleinstances of Meteor connected to the same database, but the result isnt ideal.Because of the way Meteor polls MongoDB for changes, if one Meteor instanceupdates MongoDB, it may take several seconds for other instances to detect thechange and propagate it to connected users.

    To illustrate this further, think of two Meteor instances (A and B, with theirrespective HTTP and DDP servers) serving the same chat app. In front ofthese, a reverse proxy randomly connects users to one of these instances. Whensomeone connected to server A posts a chat, users connected to server B wontsee the chat in real-time, as they would have to wait a few seconds before serverB recognizes the change and pushes the chat to its users browsers.

    In later sections, Ill show you how to configure both Meteor and MongoDB toget rid of this issue.

    This polling logic is very expensive for production use, and it wouldbe better to use a solution based around the MongoDB Oplog.Meteor 1.0 comes with a such driver, but you can use Smart Collec-tions until 1.0 comes.

    Now you have a better understanding of Meteor internals, particularly when itcomes to scaling. Lets start our journey.

    Fibers, Event Loop and Meteor

    Meteors use of Fibers allows it to do many great things. In fact, Meteorspopularity may be a direct result of its use of Fibers, though you wouldnt knowit without a deep understanding of Meteors internals.

    Its a bit hard to understand how Fibers works, and how it relates to Meteor. Butonce you do, youll have a better understanding of how Meteor works internally.

    Event Loop and Node.js

    Meteor is built on top of Node.js, so we cant forget the Event Loop. Node.js runson a single thread, and thanks to the Event Loop and event-driven programming,program execution isnt blocked by I/O activities (network and disk, mainly).


  • Instead, we provide a callback function that is run after the I/O completes, andthe rest of the program continues to run.

    Heres a psuedo-code example showing two different tasks.

    // Call functions.fetchTwitterFollowers('arunoda');createThumbnail('/tmp/files/arunoda.png', '/opt/data/arunoda.thumb.png');

    // Define functions.function fetchTwitterFollowers(username) {TwitterAPI.getProfile(username, function(){Model.setFollowers(profile.username, profile.followers, function() {console.log('profile saved!');



    function createThumbnail(imageLocation, newLocation) {File.getFile(imageLocation, function(err, fileData) {var newImage = ImageModule.resize(fileData);File.saveFile(newLocation, function() {console.log('image saved');



    Now lets see how the above two functions get executed over time.

    Tasks in fetchTwitterFollowers are marked in green, and tasksin createThumbnail are marked in orange. Dark colors show CPUtime, and light colors show I/O time.The Blue Bar shows waitTime in the queue and the Red Bar showsidleTime.


    The diagram above shows us a few interesting things. First, there is no particularprogram execution order. I/O activities can take any amount of time to complete,and they wont block the program from executing other tasks. For example,ImageModule.resize does not need to wait for Twitter.getProfile to becompleted before it can be run.

    Second, CPU-bound activities do block program execution. In the middle of thediagram you can see a blue bar where Model.setFollowers cannot get started


  • Figure 3: Understanding Event Loop


  • even though TwitterAPI.getProfile has completed. ImageModule.resize isthe reason for that. It is a CPU-bound task, so it blocks the Event Loop. Asmentioned earlier, Node.js runs in a single thread. Thats why Node.js is not thebest choice for CPU-bound activities like image processing and video conversion.

    You can also see there are three red bars indicating idleTime. If our app hadmore functionality, it could use this time to execute it.


    Now you know how the Event Loop works, and how efficient it is. But there isa problem: Callbacks. Callbacks make Node.js code difficult to reason about(some describe it as callback soup). Error handling and nested callbacks areuncomfortable to write, and their existence makes code difficult to maintain andscale. Thats why some say Node.js is hard to learn (and use).

    Luckily, several techniques exist to overcome this issue. Fibers, Promises, andGenerator-based coroutines are some of them.

    Meteor uses Fibers, and implements APIs on top of it. But before going into itany further, lets see how Fibers works. See the diagram below.

    Fibers provides an abstraction layer for the Event Loop that allows us toexecute functions (tasks) in sequence. It allows us to write asynchronous codewithout callbacks. We get the best of both worldsasynchronous efficiency withsynchronous-style coding. Behind the scenes, Fibers takes care of dealing withthe Event Loop.

    Fibers is really good if you use it correctly (Meteor does it well). Also, theoverhead caused by Fibers is negligible.

    How Meteor Uses Fibers

    Meteor abstracts Fibers with its APIs, allowing you to write your app withoutcallbacks. The best part is that you can write your code this way and becompletely oblivious to Fibers. It just works.

    Meteor creates a new Fiber for each and every request (DDP Request) madefrom the client. By default, Meteor executes one request at a time for each client,meaning one Fiber for each client at a time. But you can change that.

    Fibers is the one of the best reasons Meteor is so popular. Since it allows usto write Node.js apps without callbacks, it has attracted many developers whohated Node.js for that reason.


  • Figure 4: Understanding Fibers


  • How To Use Async Functions With Meteor

    We cant satisfy 100% of our needs with Meteors APIsometimes we need touse NPM modules to get things done. But how can we do this if we dont knowhow to use callbacks with Meteor?

    For example, say you need to use the Github NPM module to get your userspublic profile. It needs to be done inside a Meteor method, and we need toreturn the profile from the method. Okay, lets try to implement this.

    var GithubAPI = Meteor.require('github');var ghapi = new GithubAPI({version: "3.0.0"});

    Meteor.methods({getProfile: function(username) {ghapi.user.getFrom({user: username}, function(err, profile) {// How to return?


    // We need to return the profile from here.}


    We cant use callbacks like above. We