Node.jsWebDevelopmentFourthEdition

Server-sidedevelopmentwithNode10madeeasy

DavidHerron

BIRMINGHAM-MUMBAI

Node.jsWebDevelopmentFourthEditionCopyright©2018PacktPublishingAllrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishingoritsdealersanddistributors,willbeheldliableforanydamagescausedorallegedtohavebeencauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

CommissioningEditor:AmarabhaBanerjeeAcquisitionEditor:LarissaPintoContentDevelopmentEditor:GauriPradhanTechnicalEditor:LeenaPatilCopyEditor:SafisEditingProjectCoordinator:SheejalShahProofreader:SafisEditingIndexer:MariammalChettiyarGraphics:JasonMonteiroProductionCoordinator:ShraddhaFalebhai

Firstpublished:August2011

Secondedition:July2013

Thirdedition:June2016

Fourthedition:May2018

Productionreference:1240518

PublishedbyPacktPublishingLtd.LiveryPlace35LiveryStreetBirminghamB32PB,UK.

ISBN978-1-78862-685-9

www.packtpub.com

Tomymother,Herron,andtothememoryofmyfather,James,sinceIwouldnotexistwithoutthemTomypartnerMaggieforbeingmylovingpartnerthroughoutourjointlife-journey

mapt.io

Maptisanonlinedigitallibrarythatgivesyoufullaccesstoover5,000booksandvideos,aswellasindustryleadingtoolstohelpyouplanyourpersonaldevelopmentandadvanceyourcareer.Formoreinformation,pleasevisitourwebsite.

Whysubscribe?SpendlesstimelearningandmoretimecodingwithpracticaleBooksandVideosfromover4,000industryprofessionals

ImproveyourlearningwithSkillPlansbuiltespeciallyforyou

GetafreeeBookorvideoeverymonth

Maptisfullysearchable

Copyandpaste,print,andbookmarkcontent

PacktPub.comDidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusatservice@packtpub.comformoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewsletters,andreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

Contributors

AbouttheauthorDavidHerronisasoftwareengineerinSiliconValley,workingonprojectsfromanX.400e-mailservertoassistlaunchingtheOpenJDKproject,toYahoo'sNode.jsapplication-hostingplatform,andasolararrayperformancemonitoringservice.Davidwritesaboutelectricvehicles,greentechnologyonTheLongTailPipewebsite,andaboutothertopics,includingNode.js,onTechSparxwebsite.UsingNode.js,hedevelopedtheAkashaCMSstaticwebsitegenerator.

Iwishtothankmymother,Evelyn,foreverything;myfather,Jim;mysister,Patti;mybrother,Ken;mypartnerMaggieforbeingthereandencouragingme,andthemanyyearsweexpecttohavewitheachother.IwishtothankDr.KubotaoftheUniversityofKentuckyforbelievinginme,givingmemyfirstcomputingjob,andoverseeing6yearsoflearningtheartofcomputersystemmaintenance.IamgratefultoRyanDahl,thecreatorofNode.js,andthecurrentNode.jscoreteammembers.Someplatformsarejustplainhardtoworkwith,butnotNode.js.

WhatthisbookcoversChapter1,AboutNode.js,introducesyoutotheNode.jsplatform.Itcoversitsuses,thetechnologicalarchitecturechoicesinNode.js,itshistory,thehistoryofserver-sideJavaScript,whyJavaScriptshouldbeliberatedfromthebrowser,andimportantrecentadvancesintheJavaScriptscene.

Chapter2,SettingupNode.js,goesoversettingupaNode.jsdeveloperenvironment.ThisincludesinstallingNode.jsonWindows,macOS,andLinux.Importanttoolsarecovered,includingthenpmandyarnpackagemanagementsystemsandBabel,whichisusedfortranspilingmodernJavaScriptintoaformthat'srunnableonolderJavaScriptimplementations.

Chapter3,Node.jsModules,exploresthemoduleastheunitofmodularityinNode.jsapplications.WedivedeepintounderstandinganddevelopingNode.jsmodulesandusingnpmtomaintaindependencies.Welearnaboutthenewmoduleformat,ES6Modules,thatshouldsupplanttheCommonJSmoduleformatcurrentlyusedinNode.js,andarenativelysupportedinNode.js10.x.

Chapter4,HTTPServersandClients,startsexploringwebdevelopmentwithNode.js.WedevelopseveralsmallwebserverandclientapplicationsinNode.js.WeusetheFibonaccialgorithmtoexploretheeffectsofheavy-weight,long-runningcomputationsonaNode.jsapplication.Wealsolearnseveralmitigationstrategies,andhaveourfirstexperiencewithdevelopingRESTservices.

Chapter5,YourFirstExpressApplication,beginsthesectionondevelopinganote-takingapplication.Thefirststepisgettingabasicapplicationrunning.

Chapter6,ImplementingtheMobile-FirstParadigm,usesBootstrapV4to

implementresponsivewebdesign.Wetakealookatintegratingapopulariconsetsothatwecanhavepictorialbuttons,andgoovercompilingacustomBootstraptheme.

Chapter7,DataStorageandRetrieval,ensuresthatwedon'tloseournoteswhenwerestarttheapplication.Weexploreseveraldatabaseenginesandamethodtoenableeasilyswitchingbetweenthematwill.

Chapter8,MultiuserAuthenticationtheMicroserviceWay,addsuserauthenticationtothenote-takingapplication.Bothlogged-inandanonymoususerscanaccesstheapplication,withvaryingcapabilitiesbasedonrole.AuthenticationissupportedbothforlocallystoredcredentialsandforusingOAuthagainstTwitter.

Chapter9,DynamicClient/ServerInteractionwithSocket.IO,letsouruserstalkwitheachotherinrealtime.JavaScriptcodewillruninboththebrowserandtheserver,withSocket.IOprovidingtheplumbingneededforreal-timeeventexchange.Userswillseenoteschangeasthey'reeditedbyotherusersandcanleavemessages/commentsforothers.

Chapter10,DeployingNode.jsApplications,helpsusunderstandNode.jsapplicationdeployment.WelookatbothtraditionalLinuxservicedeploymentusinganetcinitscriptandusingDockerforbothlocaldevelopmentanddeploymentoncloudhostingservices.

Chapter11,UnitTestingandFunctionalTesting,takesalookatthreetestdevelopmentmodels:unittesting,RESTtesting,andfunctionaltesting.We'llusethepopularMochaandChaiframeworksforthefirsttwo,andPuppeteerforthethird.PuppeteerusesaheadlessversionofChrometosupportrunningtests.Dockerisusedtofacilitatesettingupandtearingdowntestenvironments.

Chapter12,Security,explorestechniquesandtoolsrequiredtomitigatetheriskofsecurityintrusions.IntelligentlyusingDockerisagreatfirststepifonlybecauseitcaneasilylimittheattacksurfaceofyourapplication.TheNode.jscommunityhasdevelopedasuiteoftoolsthatintegratewithExpresstoimplementseveralcriticalsecuritytechnologies.

AboutthereviewerNicholasDuffyhashadawide-rangingcareer,holdingpositionsfromanalysttobusinessintelligencearchitect,tosoftwareengineer,andevengolfprofessional.Hehasapassionforallthingsdataandsoftwareengineering,specializingincloudarchitecture,Python,andNode.js.HeisafrequentcontributortoopensourceprojectsandisalsoalifelongNewYorkMetsfan.

I'dliketothankmywife,Anne,andourboys,JackandChuck,fortheirneverending-supportinwhateverendeavorIpursue.

PacktissearchingforauthorslikeyouIfyou'reinterestedinbecominganauthorforPackt,pleasevisitauthors.packtpub.comandapplytoday.Wehaveworkedwiththousandsofdevelopersandtechprofessionals,justlikeyou,tohelpthemsharetheirinsightwiththeglobaltechcommunity.Youcanmakeageneralapplication,applyforaspecifichottopicthatwearerecruitinganauthorfor,orsubmityourownidea.

TableofContentsTitlePage

CopyrightandCredits

Node.jsWebDevelopment

FourthEdition

Dedication

PacktUpsell

Whysubscribe?

PacktPub.com

Contributors

Abouttheauthor

Aboutthereviewer

Packtissearchingforauthorslikeyou

Preface

Whothisbookisfor

Whatthisbookcovers

Togetthemostoutofthisbook

Downloadtheexamplecodefiles

Conventionsused

Getintouch

Reviews

1. AboutNode.js

ThecapabilitiesofNode.js

Server-sideJavaScript

WhyshouldyouuseNode.js?

Popularity

JavaScriptatalllevelsofthestack

LeveragingGoogle'sinvestmentinV8

Leaner,asynchronous, event-drivenmodel

Microservicearchitecture

Node.jsisstrongerforhavingsurvivedamajorschisma

ndhostilefork

Threadedversusevent-drivenarchitecture

Performanceandutilization

IsNode.jsacancerousscalabilitydisaster?

Serverutilization,thebusinessbottomline,andgreen

webhosting

EmbracingadvancesintheJavaScriptlanguage

DeployingES2015/2016/2017/2018JavaScriptcode

Node.js,themicroservicearchitecture,andeasilytestablesyste

ms

Node.jsandtheTwelve-Factorappmodel

Summary

2. SettingupNode.js

Systemrequirements

InstallingNode.jsusingpackagemanagers

InstallingonmacOSwithMacPorts

InstallingonmacOSwithHomebrew

InstallingonLinux,*BSD,orWindowsfrompackagemanag

ementsystems

InstallingNode.jsintheWindowsSubsystemfor

Linux(WSL)

Openinganadministrator-privilegedPowerShell

onWindows

InstallingtheNode.jsdistributionfromnodejs.org

InstallingfromsourceonPOSIX-likesystems

Installingprerequisites

InstallingdevelopertoolsonmacOS

InstallingfromsourceforallPOSIX-likesystems

InstallingfromsourceonWindows

InstallingmultipleNode.jsinstanceswithnvm

InstallingnvmonWindows

Nativecodemodulesandnode-gyp

Node.jsversionspolicyandwhattouse

Editorsanddebuggers

Runningandtestingcommands

Node.js'scommand-linetools

RunningasimplescriptwithNode.js

ConversiontoasyncfunctionsandthePromiseparadigm

LaunchingaserverwithNode.js

NPM –theNode.jspackagemanager

Node.js,ECMAScript2015/2016/2017,andbeyond 

UsingBabeltouseexperimentalJavaScriptfeatures

Summary

3. Node.jsModules

Definingamodule

CommonJSandES2015moduleformats

CommonJS/Node.jsmoduleformat

ES6moduleformat

JSONmodules

SupportingES6modulesonolderNode.jsversion

s

Demonstratingmodule-levelencapsulation

FindingandloadingCommonJSandJSONmodulesusingrequire

Filemodules

ModulesbakedintoNode.jsbinary

Directoriesasmodules

Moduleidentifiersandpathnames

Anexampleofapplicationdirectorystructure

FindingandloadingES6modulesusingimport

HybridCommonJS/Node.js/ES6modulescenarios

Dynamicimportswithimport()

Theimport.metafeature

npm-theNode.jspackagemanagementsystem

Thenpmpackageformat

Findingnpmpackages

Othernpmcommands

Installingannpmpackage

Installingapackagebyversionnumber

Globalpackageinstalls

Avoidingglobalmoduleinstallation

Maintainingpackagedependencieswithnpm

Automaticallyupdatingpackage.jsondependencie

s

Fixingbugsbyupdatingpackagedependencies

Packagesthatinstallcommands

ConfiguringthePATHvariabletohandlecommand

sinstalledbymodules

ConfiguringthePATHvariableonWindows

AvoidingmodificationstothePATHvariable

Updatingoutdatedpackagesyou'veinstalled

Installingpackagesfromoutsidethenpmrepository

Initializinganewnpmpackage

DeclaringNode.jsversioncompatibility

Publishingannpmpackage

Explicitlyspecifyingpackagedependencyversionnumbers

TheYarnpackagemanagementsystem

Summary

4. HTTPServersandClients

SendingandreceivingeventswithEventEmitters

JavaScriptclassesandclassinheritance

TheEventEmitterClass

TheEventEmittertheory

HTTPserverapplications

ES2015multilineandtemplatestrings

HTTPSniffer–listeningtotheHTTPconversation

Webapplicationframeworks

GettingstartedwithExpress

SettingenvironmentvariablesinWindowscmd.execommand

line

WalkingthroughthedefaultExpressapplication

TheExpressmiddleware

Middlewareandrequestpaths

Errorhandling

CalculatingtheFibonaccisequencewithanExpressapplication

Computationallyintensivecodeandthe Node.jseven

tloop

Algorithmicrefactoring

MakingHTTPClientrequests

CallingaRESTbackendservicefromanExpressapplication

ImplementingasimpleRESTserver withExpress

RefactoringtheFibonacciapplicationforREST

SomeRESTfulmodulesandframeworks

Summary

5. YourFirstExpressApplication

Promises,asyncfunctions,andExpressrouterfunctions

Promisesanderrorhandling

Flatteningourasynchronouscode

Promisesandgeneratorsbirthedasyncfunctions

ExpressandtheMVCparadigm

CreatingtheNotesapplication

YourfirstNotesmodel

UnderstandingES-2015classdefinitions

Fillingoutthein-memoryNotesmodel

TheNoteshomepage

Addinganewnote–create

Viewingnotes–read

Editinganexistingnote–update

Deletingnotes–destroy

ThemingyourExpressapplication

Scalingup–runningmultipleNotesinstances

Summary

6. ImplementingtheMobile-FirstParadigm

Problem–theNotesappisn'tmobilefriendly

Mobile-firstparadigm

UsingTwitterBootstrapontheNotesapplication

Settingitup

AddingBootstraptoapplicationtemplates

Alternativelayoutframeworks

FlexboxandCSSGrids

Mobile-firstdesignfortheNotesapplication

LayingtheBootstrapgridfoundation

ResponsivepagestructurefortheNotesapplication

Usingiconlibrariesandimprovingvisualappeal

Responsivepageheadernavigationbar

ImprovingtheNoteslistonthefrontpage

CleaninguptheNoteviewingexperience

Cleaninguptheadd/editnoteform

Cleaningupthedelete-notewindow

BuildingacustomizedBootstrap

Pre-builtcustomBootstrapthemes

Summary

7. DataStorageandRetrieval

Datastorageandasynchronouscode

Logging

RequestloggingwithMorgan

Debuggingmessages

Capturingstdoutandstderr

Uncaughtexceptions

UnhandledPromiserejections

UsingtheES6moduleformat

Rewritingapp.jsasanES6module

Rewritingbin/wwwasanES6module

RewritingmodelscodeasES6modules

RewritingroutermodulesasES6modules

Storingnotesinthefilesystem

DynamicimportofES6modules

RunningtheNotesapplicationwithfilesystemstorage

StoringnoteswiththeLevelUPdatastore

StoringnotesinSQLwithSQLite3

SQLite3databaseschema

SQLite3modelcode

RunningNoteswithSQLite3

StoringnotestheORMwaywithSequelize

SequelizemodelfortheNotesapplication

ConfiguringaSequelizedatabaseconnection

RunningtheNotesapplicationwithSequelize

StoringnotesinMongoDB

MongoDBmodelfortheNotesapplication

RunningtheNotesapplicationwithMongoDB

Summary

8. MultiuserAuthenticationtheMicroserviceWay

Creatingauserinformationmicroservice

Userinformationmodel

ARESTserverforuserinformation

Scriptstotestandadministertheuserauthentications

erver

LoginsupportfortheNotesapplication

AccessingtheuserauthenticationRESTAPI

Loginandlogoutroutingfunctions

Login/logoutchangestoapp.js

Login/logoutchangesinroutes/index.mjs

Login/logoutchangesrequiredinroutes/notes.m

js

Viewtemplatechangessupportinglogin/logout

RunningtheNotesapplicationwithuserauthent

ication

TwitterloginsupportfortheNotesapplication

RegisteringanapplicationwithTwitter

ImplementingTwitterStrategy

Securelykeepingsecretsandpasswords

TheNotesapplicationstack

Summary

9. DynamicClient/ServerInteractionwithSocket.IO

IntroducingSocket.IO

InitializingSocket.IOwithExpress

Real-timeupdatesontheNoteshomepage

TheNotesmodelasanEventEmitterclass

Real-timechangesintheNoteshomepage

Changingthehomepageandlayouttemplates

RunningNoteswithreal-timehomepageupdates

Real-timeactionwhileviewingnotes

Changingthenoteviewtemplateforreal-timea

ction

RunningNoteswithreal-timeupdateswhileview

inganote

Inter-userchatandcommentingforNotes

Datamodelforstoringmessages

AddingmessagestotheNotesrouter

Changingthenoteviewtemplateformessages

UsingaModalwindowtocomposemessages

Sending,displaying,anddeletingmessages

RunningNotesandpassingmessages

OtherapplicationsofModalwindows

Summary

10. DeployingNode.jsApplications

Notesapplicationarchitectureanddeploymentconsiderations

TraditionalLinuxNode.jsservicedeployment

Prerequisite–provisioningthedatabases

InstallingNode.jsonUbuntu

SettingupNotesanduserauthentication ontheser

ver

AdjustingTwitterauthenticationtoworkonthe

server

SettingupPM2tomanageNode.jsprocesses

Node.jsmicroservicedeployment withDocker

InstallingDockeronyourlaptop

StartingDockerwithDockerforWindows/macOS

KickingthetiresofDocker

CreatingtheAuthNetfortheuserauthenticationservice

MySQLcontainerforDocker

InitializingAuthNet

ScriptexecutiononWindows

LinkingDockercontainers

Thedb-userauthcontainer

Dockerfilefortheauthenticationservice

ConfiguringtheauthenticationserviceforDock

er

Buildingandrunningtheauthenticationservice

Dockercontainer

ExploringAuthnet

CreatingFrontNetfortheNotesapplication

MySQLcontainerfortheNotesapplication

DockerizingtheNotesapplication

ControllingthelocationofMySQLdatavolumes

Dockerdeploymentofbackgroundservices

DeployingtothecloudwithDockercompose

Dockercomposefiles

RunningtheNotesapplicationwithDockercompo

se

DeployingtocloudhostingwithDockercompose

Summary

11. UnitTestingandFunctionalTesting

Assert–thebasisoftestingmethodologies

TestingaNotesmodel

MochaandChai­–thechosentesttools

Notesmodeltestsuite

Configuringandrunningtests

MoretestsfortheNotesmodel

Testingdatabasemodels

UsingDockertomanagetestinfrastructure

DockerComposetoorchestratetestinfrastructure

ExecutingtestsunderDockerCompose

MongoDBsetupunderDockerandtestingNotesag

ainstMongoDB

TestingRESTbackendservices

Automatingtestresultsreporting

Frontendheadlessbrowsertesting withPuppeteer

SettingupPuppeteer

ImprovingtestabilityintheNotesUI

PuppeteertestscriptforNotes

Runningtheloginscenario

TheAddNotescenario

Mitigating/preventingspurioustesterrorsinPuppeteer

scripts

Configuringtimeouts

TracingeventsonthePageandthePuppeteerin

stance

Insertingpauses

AvoidingWebSocketsconflicts

Takingscreenshots

Summary

12. Security

HTTPS/TLS/SSLusingLet'sEncrypt

AssociatingadomainnamewithDocker-basedcloudhostin

g

ADockercontainertomanageLet'sEncryptSSLcertifica

tes

Cross-containermountingofLet'sEncryptdirectoriesto

thenotescontainer

AddingHTTPSsupporttoNotes

PutonyourHelmetforacross-the-boardsecurity

UsingHelmettosettheContent-Security-Policyheader

UsingHelmettosettheX-DNS-Prefetch-Controlheader

UsingHelmettosettheX-Frame-Optionsheader

UsingHelmettoremovetheX-Powered-Byheader

ImprovingHTTPSwithStrictTransportSecurity

MitigatingXSSattackswithHelmet

AddressingCross-SiteRequestForgery(CSRF)attacks

DenyingSQLinjectionattacks

Sequelizedeprecationwarningregardingoperatorinjecti

onattack

Scanningforknownvulnerabilities

Usinggoodcookiepractices

Summary

OtherBooksYouMayEnjoy

Leaveareview-letotherreadersknowwhatyouthink

PrefaceNode.jsisaserver-sideJavaScriptplatformusinganevent-driven,non-blockingI/Omodel,allowinguserstobuildfastandscalabletransaction-intensiveapplicationsrunninginrealtime.ItplaysasignificantroleinthesoftwaredevelopmentworldandliberatesJavaScriptfromthewebbrowser.WithNode.js,wecanreuseourJavaScriptskillsforgeneralsoftwaredevelopmentonalargerangeofsystems.

Itrunsatoptheultra-fastJavaScriptengineattheheartofGoogle'sChromebrowser,V8,andaddsafastandrobustlibraryofasynchronousnetworkI/Omodules.

TheprimaryfocusofNode.jsisdevelopinghighperformance,highlyscalablewebapplications,anditalsoseesawidespreaduseinotherareas.Electron,theNode.js-basedwrapperaroundtheChromeengine,isthebasisforpopulardesktopapplications,suchasAtomandVisualStudioCodeeditors,GitKraken,Postman,Etcher,andthedesktopSlackclient.Node.jsispopularfordevelopingInternetofThingsdevicesandseesatremendousadoptioninmicroservicedevelopmentandforbuildingtoolsforfrontendwebdevelopersandmore.Node.js,asalightweighthigh-performanceplatform,fitsmicroservicedevelopmentlikeaglove.

TheNode.jsplatformusesanasynchronoussingle-threadsystem,withasynchronouslyinvokedcallbackfunctions(alsoknownaseventhandlers)andaneventloop,asopposedtoatraditionalthread-basedarchitecture.

Thetheoryisthatthreadedsystemsarenotoriouslydifficulttodevelop,andthatthreadsthemselvesimposeanarchitecturalburdenonappservers.Node.js'sgoalistoprovideaneasywaytobuildscalablenetworkservers.

ThewholeNode.jsruntimeisdesignedaroundasynchronousexecution.

JavaScriptwaschosenasthelanguagebecauseanonymousfunctionsandotherlanguageelementsprovideanexcellentbaseforimplementingasynchronouscomputation.

WhothisbookisforWeassumethatyouhavesomeknowledgeofJavaScriptandpossiblyhaveexperiencewithserver-sidecodedevelopment,andthatyouarelookingforadifferentwayofdevelopingserver-sidecode.

Server-sideengineersmayfindtheconceptsbehindNode.jsrefreshing.Itoffersanewperspectiveonwebapplicationdevelopmentandadifferenttakeonserverarchitecturesfromthemonolithswedealwithinotherprogramminglanguages.JavaScriptisapowerfullanguageandNode.js'sasynchronousnatureplaystoitsstrengths.HavingJavaScriptonboththefrontendandthebackendgivesawholenewmeaning.

Developersexperiencedwithbrowser-sideJavaScriptwillfinditproductivetobringthatknowledgetoanewterritory.

Althoughourfocusisonwebapplicationdevelopment,Node.jsknowledgecanbeappliedinotherareasaswell.Assaidearlier,Node.jsiswidelyusedtodevelopmanytypesofapplications.

TogetthemostoutofthisbookThebasicrequirementistoinstallNode.jsandhaveaprogrammer-orientedtexteditor.Theeditorneednotbeanythingfancy,vi/vimwillevendoinapinch.Wewillshowyouhowtoinstalleverythingthat'sneeded.It'sallopensourcesoftwarethatcanbeeasilydownloadedfromwebsites.

Themostimportanttoolistheonebetweenyourears.

Somechaptersrequiredatabaseengines,suchasMySQLandMongoDB.

AlthoughNode.jsisacross-platformsoftwaredevelopmentplatform,somethird-partymodulesarewritteninC/C++andmustbecompiledduringinstallation.Todoso,native-codedevelopmenttoolssuchasC/C++compilersarerequired,andPythonisrequiredtorunthetool-chain.ThedetailsarecoveredinChapter2,SettingupNode.js.MicrosoftisinvolvedwiththeNode.jsprojectandtoensuredeveloperproductivitywithNode.jsonWindows.

DownloadtheexamplecodefilesYoucandownloadtheexamplecodefilesforthisbookfromyouraccountatwww.packtpub.com.Ifyoupurchasedthisbookelsewhere,youcanvisitwww.packtpub.com/supportandregistertohavethefilesemaileddirectlytoyou.

Youcandownloadthecodefilesbyfollowingthesesteps:

1. Loginorregisteratwww.packtpub.com.2. SelecttheSUPPORTtab.3. ClickonCodeDownloads&Errata.4. EnterthenameofthebookintheSearchboxandfollowthe

onscreeninstructions.

Oncethefileisdownloaded,pleasemakesurethatyouunziporextractthefolderusingthelatestversionof:

WinRAR/7-ZipforWindows

Zipeg/iZip/UnRarXforMac

7-Zip/PeaZipforLinux

ThecodebundleforthebookisalsohostedonGitHubathttps://github.com/PacktPublishing/Node.js-Web-Development-Fourth-Edition.Wealsohaveothercodebundlesfromourrichcatalogofbooksandvideosavailableathttps://github.com/PacktPublishing/.Checkthemout!

ConventionsusedThereareanumberoftextconventionsusedthroughoutthisbook.

CodeInText:Indicatescodewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandles.Hereisanexample:"ThehttpobjectencapsulatestheHTTPprotocol,anditshttp.createServermethodcreatesawholewebserver,listeningontheportspecifiedinthelistenmethod."

Ablockofcodeissetasfollows:

varhttp=require('http');

http.createServer(function(req,res){

res.writeHead(200,{'Content-Type':'text/plain'});

res.end('HelloWorld\n');

}).listen(8124,"127.0.0.1");

console.log('Serverrunningathttp://127.0.0.1:8124/');

Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

if(urlP.query['n']){

fibonacciAsync(urlP.query['n'],fibo=>{

res.end('Fibonacci'+urlP.query['n']+'='+fibo);

});

}else{

Anycommand-lineinputoroutputiswrittenasfollows:

$node--version

v8.9.1

Bold:Indicatesanewterm,animportantword,orwordsthatyousee

onscreen.Forexample,wordsinmenusordialogboxesappearinthetextlikethis.Hereisanexample:"IntheStartmenu,enterPowerShellintheapplicationssearchbox."

Warningsorimportantnotesappearlikethis.

Tipsandtricksappearlikethis.

GetintouchFeedbackfromourreadersisalwayswelcome.

Generalfeedback:Emailfeedback@packtpub.comandmentionthebooktitleinthesubjectofyourmessage.Ifyouhavequestionsaboutanyaspectofthisbook,pleaseemailusatquestions@packtpub.com.

Errata:Althoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyouhavefoundamistakeinthisbook,wewouldbegratefulifyouwouldreportthistous.Pleasevisitwww.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetails.

Piracy:IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,wewouldbegratefulifyouwouldprovideuswiththelocationaddressorwebsitename.Pleasecontactusatcopyright@packtpub.comwithalinktothematerial.

Ifyouareinterestedinbecominganauthor:Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,pleasevisitauthors.packtpub.com.

ReviewsPleaseleaveareview.Onceyouhavereadandusedthisbook,whynotleaveareviewonthesitethatyoupurchaseditfrom?Potentialreaderscanthenseeanduseyourunbiasedopiniontomakepurchasedecisions,weatPacktcanunderstandwhatyouthinkaboutourproducts,andourauthorscanseeyourfeedbackontheirbook.Thankyou!

FormoreinformationaboutPackt,pleasevisitpacktpub.com.

AboutNode.jsNode.jsisanexcitingnewplatformfordevelopingwebapplications,applicationservers,anysortofnetworkserverorclient,andgeneralpurposeprogramming.Itisdesignedforextremescalabilityinnetworkedapplicationsthroughaningeniouscombinationofserver-sideJavaScript,asynchronousI/O,andasynchronousprogramming.ItisbuiltaroundJavaScriptanonymousfunctions,andasingleexecutionthreadevent-drivenarchitecture.

Whileonlyafewyearsold,Node.jshasquicklygrowninprominenceandit'snowplayingasignificantrole.Companies,bothsmallandlarge,areusingitforlarge-scaleandsmall-scaleprojects.PayPal,forexample,hasconvertedmanyservicesfromJavatoNode.js.

TheNode.jsarchitecturedepartsfromatypicalchoicemadebyotherapplicationplatforms.WherethreadsarewidelyusedtoscaleanapplicationtofilltheCPU,Node.jseschewsthreadsbecauseoftheirinherentcomplexity.It'sclaimedthatwithsingle-threadevent-drivenarchitectures,memoryfootprintislow,throughputishigh,thelatencyprofileunderloadisbetter,andtheprogrammingmodelissimpler.TheNode.jsplatformisinaphaseofrapidgrowth,andmanyareseeingitasacompellingalternativetothetraditionalwebapplicationarchitecturesusingJava,PHP,Python,orRubyonRails.

Atitsheart,itisastandaloneJavaScriptenginewithextensionsmakingitsuitableforgeneralpurposeprogrammingandwithaclearfocusonapplicationserverdevelopment.Eventhoughwe'recomparingNode.jstoapplicationserverplatforms,itisnotanapplicationserver.Instead,Node.jsisaprogrammingrun-timeakintoPython,Go,orJavaSE.WhiletherearewebapplicationframeworksandapplicationserverswritteninNode.js,itissimplyasystemtoexecuteJavaScriptprograms.

Itisimplementedaroundanon-blockingI/OeventloopandalayeroffileandnetworkI/Olibraries,allbuiltontopoftheV8JavaScriptengine(fromtheChromewebbrowser).TherapidperformanceandfeatureimprovementsimplementedinChromequicklyflowthroughtotheNode.jsplatform.Additionally,ateamoffolksareworkingonaNode.jsimplementationthatrunsontopofMicrosoft'sChakraCoreJavaScriptengine(fromtheEdgewebbrowser).ThatwouldgivetheNode.jscommunitygreaterflexibilitybynotbeingreliantononeJavaScriptengineprovider.Visithttps://github.com/nodejs/node-chakracoretotakealookattheproject.

TheNode.jsI/OlibraryisgeneralenoughtoimplementanysortofserverexecutinganyTCPorUDPprotocol,whetherit'sdomainnamesystem(DNS),HTTP,internetrelaychat(IRC),orFTP.Whileitsupportsdevelopinginternetserversorclients,itsbiggestusecaseisinregularwebsites,inplaceoftechnologysuchasanApache/PHPorRailsstack,ortocomplementexistingwebsites.Forexample,addingreal-timechatormonitoringexistingwebsitescanbeeasilydonewiththeSocket.IOlibraryforNode.js.Itslightweight,high-performancenatureoftenseesNode.jsusedasaglueservice.

AparticularlyintriguingcombinationisdeployingsmallservicesusingDockerintocloudhostinginfrastructure.Alargeapplicationcanbedividedintowhat'snowcalledmicroservicesthatareeasilydeployedatscaleusingDocker.TheresultfitsagileprojectmanagementmethodssinceeachmicroservicecanbeeasilymanagedbyasmallteamthatcollaboratesattheboundaryoftheirindividualAPI.

ThisbookwillgiveyouanintroductiontoNode.js.Wepresumethefollowing:

Youalreadyknowhowtowritesoftware

YouarefamiliarwithJavaScript

Youknowsomethingaboutdevelopingwebapplicationsinotherlanguages

Wewillcoverthefollowingtopicsinthischapter:

AnintroductiontoNode.js

WhyyoushoulduseNode.js

ThearchitectureofNode.js

Performance,utilization,andscalabilitywithNode.js

Node.js,microservicearchitecture,andtesting

ImplementingtheTwelve-FactorAppmodelwithNode.js

Wewilldiverightintodevelopingworkingapplicationsandrecognizethatoftenthebestwaytolearnisbyrummagingaroundinworkingcode.

ThecapabilitiesofNode.jsNode.jsisaplatformforwritingJavaScriptapplicationsoutsidewebbrowsers.ThisisnottheJavaScriptwearefamiliarwithinwebbrowsers!Forexample,thereisnoDOMbuiltintoNode.js,noranyotherbrowsercapability.

BeyonditsnativeabilitytoexecuteJavaScript,thebundledmodulesprovidecapabilitiesofthissort:

Command-linetools(inshellscriptstyle)

Aninteractive-terminalstyleofprogramthatisRead-Eval-PrintLoop(REPL)

Excellentprocesscontrolfunctionstooverseechildprocesses

Abufferobjecttodealwithbinarydata

TCPorUDPsocketswithcomprehensiveevent-drivencallbacks

DNSlookup

AnHTTP,HTTPSandHTTP/2client/serverlayeredontopoftheTCPlibraryfilesystemaccess

Built-inrudimentaryunittestingsupportthroughassertions

ThenetworklayerofNode.jsislowlevelwhilebeingsimpletouse.Forexample,theHTTPmodulesallowyoutowriteanHTTPserver(orclient)usingafewlinesofcode.Thisispowerful,butitputsyou,theprogrammer,veryclosetotheprotocolrequestsandmakesyouimplementpreciselythoseHTTPheadersthatyoushouldreturninrequestresponses.

Typicalwebapplicationdevelopersdon'tneedtoworkatalowleveloftheHTTPorotherprotocols.Instead,wetendtobemoreproductive,workingwithhigher-levelinterfaces.Forexample,PHPcodersassumethatApache(orotherHTTPservers)isalreadythereprovidingtheHTTPprotocol,andthattheydon'thavetoimplementtheHTTPserverportionofthestack.Bycontrast,aNode.jsprogrammerdoesimplementanHTTPservertowhichtheirapplicationcodeisattached.

Tosimplifythesituation,theNode.jscommunityhasseveralwebapplicationframeworks,suchasExpress,providingthehigher-levelinterfacesrequiredbytypicalprogrammers.YoucanquicklyconfigureanHTTPserverwithbaked-incapabilitiessuchassessions,cookies,servingstaticfiles,andlogging,lettingdevelopersfocusontheirbusinesslogic.OtherframeworksprovideOAuth2support,orfocusonRESTAPIs,andsoon.

Node.jsisnotlimitedtowebserviceapplicationdevelopment.ThecommunityaroundNode.jshastakenitinmanyotherdirections,

Buildtools:Node.jshasbecomeapopularchoicefordevelopingcommand-linetoolsusedinsoftwaredevelopment,orcommunicatingwithserviceinfrastructure.GruntandGulparewidelyusedbyfrontenddeveloperstobuildassetsforwebsites.BabeliswidelyusedfortranspilingmodernES-2016codetorunonolderbrowsers.PopularCSSoptimizersandprocessors,suchasPostCSS,arewritteninNode.js.StaticwebsitegenerationsystemssuchasMetalsmith,Punch,andAkashaCMS,runatthecommandlineandgeneratewebsitecontentthatyouuploadtoawebserver.

WebUItesting:Puppeteergivesyoucontroloveraheadless-Chromewebbrowserinstance.Withit,youcandevelopNode.jsscriptscontrollingamodernfull-featuredwebbrowser.Typicalusecasesinvolvewebscrapingandtestingwebapplications.

Desktopapplications:BothElectronandnode-webkit(NW.js)areframeworksfordevelopingdesktopapplicationsforWindows,macOS,andLinux.TheseframeworksutilizealargechunkofChrome,wrapped

byNode.jslibraries,todevelopdesktopapplicationsusingwebUItechnologies.ApplicationsarewrittenwithmodernHTML5,CSS3,andJavaScript,andcanutilizeleading-edgewebframeworks,suchasBootstrap,React,orAngularJS.ManypopularapplicationshavebeenbuiltusingElectron,includingtheSlackdesktopclientapplication,theAtomandMicrosoftVisualCodeprogrammingeditors,thePostmanRESTclient,theGitKrakenGITclient,andEtcher,whichmakesitincrediblyeasytoburnOSimagestoflashdrivestorunonsingle-boardcomputers.

Mobileapplications:TheNode.jsforMobileSystemsprojectletsyoudevelopsmartphoneortabletcomputerapplicationsusingNode.js,forbothiOSandAndroid.Apple'sAppStorerulesprecludeincorporatingaJavaScriptenginewithJITcapabilities,meaningthatnormalNode.jscannotbeusedinaniOSapplication.ForiOSapplicationdevelopment,theprojectusesNode.js-on-ChakraCoretoskirtaroundtheAppStorerules.ForAndroidapplicationdevelopmenttheprojectusesregularNode.jsonAndroid.Atthetimeofwriting,theprojectisinanearlystageofdevelopment,butitlookspromising.

InternetofThings(IoT):Reportedly,itisaverypopularlanguageforInternet-of-Thingsprojects,andNode.jsdoesrunonmostARM-basedsingle-boardcomputers.TheclearestexampleistheNodeREDproject.Itoffersagraphicalprogrammingenvironment,lettingyoudrawprogramsbyconnectingblockstogether.Itfeatureshardware-orientedinputandoutputmechanisms,forexample,tointeractwithGeneralPurposeI/O(GPIO)pinsonRaspberryPiorBeaglebonesingle-boardcomputers.

Server-sideJavaScriptQuitscratchingyourheadalready!Ofcourseyou'redoingit,scratchingyourheadandmumblingtoyourself,"What'sabrowserlanguagedoingontheserver?"Intruth,JavaScripthasalongandlargelyunknownhistoryoutsidethebrowser.JavaScriptisaprogramminglanguage,justlikeanyotherlanguage,andthebetterquestiontoaskis"WhyshouldJavaScriptremaintrappedinsidebrowsers?".

Backinthedawnofthewebage,thetoolsforwritingwebapplicationswereatafledglingstage.SomewereexperimentingwithPerlorTCLtowriteCGIscripts,andthePHPandJavalanguageshadjustbeendeveloped.Eventhen,JavaScriptsawuseontheserverside.OneearlywebapplicationserverwasNetscape'sLiveWireserver,whichusedJavaScript.SomeversionsofMicrosoft'sASPusedJScript,theirversionofJavaScript.Amorerecentserver-sideJavaScriptprojectistheRingoJSapplicationframeworkintheJavauniverse.Java6andJava7werebothshippedwiththeRhinoJavaScriptengine.InJava8,RhinowasdroppedinfavorofthenewerNashornJavaScriptengine.

Inotherwords,JavaScriptoutsidethebrowserisnotanewthing,evenifitisuncommon.

WhyshouldyouuseNode.js?Amongthemanyavailablewebapplicationdevelopmentplatforms,whyshouldyouchooseNode.js?Therearemanystackstochoosefrom;whatisitaboutNode.jsthatmakesitriseabovetheothers?Wewillseeinthefollowingsections.

PopularityNode.jsisquicklybecomingapopulardevelopmentplatformwithadoptionbyplentyofbigandsmallplayers.OneofthoseisPayPal,whoarereplacingtheirincumbentJava-basedsystemwithonewritteninNode.js.ForPayPal'sblogpostaboutthis,visithttps://www.paypal-engineering.com/2013/11/22/node-js-at-paypal/.OtherlargeNode.jsadoptersincludeWalmart'sonlinee-commerceplatform,LinkedIn,andeBay.

AccordingtoNodeSource,Node.jsusageisgrowingrapidly(visithttps://nodesource.com/node-by-numbers).ThemeasuresincludeincreasingbandwidthfordownloadingNode.jsreleases,increasingactivityinNode.js-relatedGitHubprojects,andmore.

It'sbesttonotjustfollowthecrowdbecausethecrowdclaimstheirsoftwareplatformdoescoolthings.Node.jsdoessomecoolthings,butmoreimportantisitstechnicalmerit.

JavaScriptatalllevelsofthestackHavingthesameprogramminglanguageontheserverandclienthasbeenalong-timedreamontheweb.ThisdreamdatesbacktotheearlydaysofJava,whereJavaappletsweretobethefrontendtoserverapplicationswritteninJava,andJavaScriptwasoriginallyenvisionedasalightweightscriptinglanguageforthoseapplets.Javaneverfulfilleditshypeasaclient-sideprogramminglanguage,forvariousreasons.WeendedupwithJavaScriptastheprinciplein-browser,client-sidelanguage,ratherthanJava.Typically,thefrontendJavaScriptdeveloperswereinadifferentlanguageuniversethantheserver-sideteam,whowaslikelytobecodinginPHP,Java,Ruby,orPython.

Overtime,in-browserJavaScriptenginesbecameincrediblypowerful,lettinguswriteever-morecomplexbrowser-sideapplications.WithNode.js,wemayfinallybeabletoimplementapplicationswiththesameprogramminglanguageontheclientandserverbyhavingJavaScriptatbothendsoftheweb,inthebrowserandserver.

Acommonlanguageforfrontendandbackendoffersseveralpotentialbenefits:

Thesameprogrammingstaffcanworkonbothendsofthewire

Codecanbemigratedbetweenserverandclientmoreeasily

Commondataformats(JSON)existbetweenserverandclient

Commonsoftwaretoolsexistforserverandclient

Commontestingorqualityreportingtoolsforserverandclient

Whenwritingwebapplications,viewtemplatescanbeusedonbothsides

TheJavaScriptlanguageisverypopularduetoitsubiquityinwebbrowsers.Itcomparesfavorablyagainstotherlanguageswhilehavingmanymodern,advancedlanguageconcepts.Thankstoitspopularity,thereisadeeptalentpoolofexperiencedJavaScriptprogrammersoutthere.

LeveragingGoogle'sinvestmentinV8TomakeChromeapopularandexcellentwebbrowser,GoogleinvestedinmakingV8asuper-fastJavaScriptengine.Google,therefore,hasahugemotivationtokeeponimprovingV8.V8istheJavaScriptengineforChrome,anditcanalsobeexecutedstandalone.Node.jsisbuiltontopoftheV8JavaScriptengine.

AsNode.jsbecomesmoreimportanttotheV8team,there'sapotentialsynergyoffasterV8performancewinsasmorepeoplefocusonV8improvements.

Leaner,asynchronous,event-drivenmodelWe'llgetintothislater.TheNode.jsarchitecture,asingleexecutionthread,aningeniousevent-orientedasynchronous-programmingmodel,andafastJavaScriptengine,haslessoverheadthanthread-basedarchitectures.

MicroservicearchitectureAnewsensationinsoftwaredevelopmentisthemicroserviceidea.Microservicesarefocusedonsplittingalargewebapplicationintosmall,tightly-focusedservicesthatcanbeeasilydevelopedbysmallteams.Whiletheyaren'texactlyanewidea,they'remoreofareframingofoldclient-servercomputingmodels,themicroservicepatternfitswellwithagileprojectmanagementtechniques,andgivesusmoregranularapplicationdeployment.

Node.jsisanexcellentplatformforimplementingmicroservices.We'llgetintothislater.

Node.jsisstrongerforhavingsurvivedamajorschismandhostileforkDuring2014and2015,theNode.jscommunityfacedamajorsplitoverpolicy,direction,andcontrol.Theio.jsprojectwasahostileforkdrivenbyagroupwhowantedtoincorporateseveralfeaturesandchangewho'sinthedecision-makingprocess.TheendresultwasamergeoftheNode.jsandio.jsrepositories,anindependentNode.jsfoundationtoruntheshow,andthecommunityisworkingtogethertomoveforwardinacommondirection.

AconcreteresultofhealingthatriftistherapidadoptionofnewECMAScriptlanguagefeatures.TheV8engineisadoptingthosenewfeaturesquicklytoadvancethestateofwebdevelopment.TheNode.jsteam,inturn,isadoptingthosefeaturesasquicklyastheyshowupinV8,meaningthatPromisesandasyncfunctionsarequicklybecomingarealityforNode.jsprogrammers.

ThebottomlineisthattheNode.jscommunitynotonlysurvivedtheio.jsfork,butthecommunityandtheplatformitnurturesgrewstrongerasaresult.

Threadedversusevent-drivenarchitectureNode.js'sblisteringperformanceissaidtobebecauseofitsasynchronousevent-drivenarchitecture,anditsuseoftheV8JavaScriptengine.That'sanicethingtosay,butwhat'stherationaleforthestatement?

TheV8JavaScriptengineisamongthefastestJavaScriptimplementations.Asaresult,Chromeiswidelyusednotjusttoviewwebsitecontent,buttoruncomplexapplications.ExamplesincludeGmail,theGoogleGSuiteapplications(Docs,Slides,andsoon),imageeditorssuchasPixlr,anddrawingapplicationssuchasdraw.ioandCanva.BothAtomandMicrosoft'sVisualStudioCodeareexcellentIDE'sthatjusthappentobeimplementedinNode.jsandChromeusingElectron.ThattheseapplicationsexistandarehappilyusedbyalargenumberofpeopleistestamenttoV8'sperformance.Node.jsbenefitsfromV8performanceimprovements.

ThenormalapplicationservermodelusesblockingI/Otoretrievedata,anditusesthreadsforconcurrency.BlockingI/Ocausesthreadstowaitonresults.Thatcausesachurnbetweenthreadsastheapplicationserverstartsandstopsthethreadstohandlerequests.Eachsuspendedthread(typicallywaitingonanI/Ooperationtofinish)consumesafullstacktraceofmemory,increasingmemoryconsumptionoverhead.Threadsaddcomplexitytotheapplicationserveraswellasserveroverhead.

Node.jshasasingleexecutionthreadwithnowaitingonI/Oorcontextswitching.Instead,thereisaneventlooplookingforeventsanddispatchingthemtohandlerfunctions.Theparadigmisthatanyoperationthatwouldblockorotherwisetaketimetocompletemustusetheasynchronousmodel.Thesefunctionsaretobegivenananonymousfunctiontoactasahandlercallback,orelse(withtheadventofES2015

promises),thefunctionwouldreturnaPromise.Thehandlerfunction,orPromise,isinvokedwhentheoperationiscomplete.Inthemeantime,controlreturnstotheeventloop,whichcontinuesdispatchingevents.

AttheNode.jsinteractiveconferencein2017,IBM'sChrisBaileymadeacaseforNode.jsbeinganexcellentchoiceforhighlyscalablemicroservices.KeyperformancecharacteristicsareI/Operformance,measuredintransactionspersecond,startuptime,becausethatlimitshowquicklyyourservicecanscaleuptomeetdemand,andmemoryfootprint,becausethatdetermineshowmanyapplicationinstancescanbedeployedperserver.Node.jsexcelsonallthosemeasures;witheverysubsequentreleaseeach,iseitherimprovingorremainingfairlysteady.BaileypresentedfigurescomparingNode.jstoasimilarbenchmarkwritteninSpringBootshowingNode.jstoperformmuchbetter.Toviewhistalk,seehttps://www.youtube.com/watch?v=Fbhhc4jtGW4.

Tohelpuswrapourheadsaroundwhythiswouldbe,let'sreturntoRyanDahl,thecreatorofNode.js,andthekeyinspirationleadinghimtocreateNode.js.InhisCincodeNodeJSpresentationinMay2010,https://www.youtube.com/watch?v=M-sc73Y-zQA,Dahlaskeduswhathappenswhileexecutingalineofcodesuchasthis:

result=query('SELECT*fromdb');

//operateontheresult

Ofcourse,theprogrampausesatthatpointwhilethedatabaselayersendsthequerytothedatabase,whichdeterminestheresultandreturnsthedata.Dependingonthequery,thatpausecanbequitelong;well,afewmilliseconds,whichisaneonincomputertime.Thispauseisbadbecausethatexecutionthreadcandonothingwhilewaitingfortheresulttoarrive.Ifyoursoftwareisrunningonasingle-threadedplatform,theentireserverwouldbeblockedandunresponsive.Ifinstead,yourapplicationisrunningonathread-basedserverplatform,athreadcontextswitchisrequiredtosatisfyanyotherrequeststhatarrive.Thegreaterthenumberofoutstandingconnectionstotheserver,thegreaterthenumberofthreadcontextswitches.ContextswitchingisnotfreebecausemorethreadsrequiremorememoryperthreadstateandmoretimefortheCPUtospend

onthreadmanagementoverhead.

Simplyusinganasynchronous,event-drivenI/O,Node.jsremovesmostofthisoverheadwhileintroducingverylittleofitsown.

Usingthreadstoimplementconcurrencyoftencomeswithadmonitionssuchasthese:expensiveanderror-prone,theerror-pronesynchronizationprimitivesofJava,ordesigningconcurrentsoftwarecanbecomplexanderrorprone.Thecomplexitycomesfromtheaccesstosharedvariablesandvariousstrategiestoavoiddeadlockandcompetitionbetweenthreads.ThesynchronizationprimitivesofJavaareanexampleofsuchastrategy,andobviouslymanyprogrammersfindthemdifficulttouse.There'sthetendencytocreateframeworkssuchasjava.util.concurrenttotamethecomplexityofthreadedconcurrency,butsomemightarguethatpaperingovercomplexitydoesnotmakethingssimpler.

Node.jsasksustothinkdifferentlyaboutconcurrency.Callbacksfiredasynchronouslyfromaneventloopareamuchsimplerconcurrencymodel—simplertounderstand,simplertoimplement,simplertoreasonabout,andsimplertodebugandmaintain.

RyanDahlpointstotherelativeaccesstimeofobjectstounderstandtheneedforasynchronousI/O.Objectsinmemoryaremorequicklyaccessed(intheorderofnanoseconds)thanobjectsondiskorobjectsretrievedoverthenetwork(millisecondsorseconds).Thelongeraccesstimeforexternalobjectsismeasuredinzillionsofclockcycles,whichcanbeaneternitywhenyourcustomerissittingattheirwebbrowserreadytomoveonifittakeslongerthantwosecondstoloadthepage.

InNode.js,thequerydiscussedpreviouslywillreadasfollows:

query('SELECT*fromdb',function(err,result){

if(err)throwerr;//handleerrors

//operateonresult

});

Theprogrammersuppliesafunctionthatiscalled(hencethe

namecallbackfunction)whentheresult(orerror)isavailable.Insteadofathreadcontextswitch,thiscodereturnsalmostimmediatelytotheeventloop.Thateventloopisfreetohandleotherrequests.TheNode.jsruntimekeepstrackofthestackcontextleadingtothiscallbackfunction,andeventuallyaneventwillfirecausingthiscallbackfunctiontobecalled.

AdvancesintheJavaScriptlanguagearegivingusnewoptionstoimplementthisidea.TheequivalentcodelookslikesowhenusedwithES2015Promise's:

query('SELECT*fromdb')

.then(result=>{

//operateonresult

})

.catch(err=>{

//handleerrors

});

ThefollowingwithanES-2017asyncfunction:

try{

varresult=awaitquery('SELECT*fromdb');

//operateonresult

}catch(err){

//handleerrors

}

Allthreeofthesecodesnippetsperformthesamequerywrittenearlier.Thedifferenceisthatthequerydoesnotblocktheexecutionthread,becausecontrolpassesbacktotheeventloop.Byreturningalmostimmediatelytotheeventloop,itisfreetoserviceotherrequests.Eventually,oneofthoseeventswillbetheresponsetothequeryshownpreviously,whichwillinvokethecallbackfunction.

WiththecallbackorPromiseapproach,theresultisnotreturnedastheresultofthefunctioncall,butisprovidedtoacallbackfunctionthatwillbecalledlater.Theorderofexecutionisnotonelineafteranother,asitisinsynchronousprogramminglanguages.Instead,theorderofexecutionisdeterminedbytheorderofthecallbackfunctionexecution.

Whenusinganasyncfunction,thecodingstyleLOOKSliketheoriginalsynchronouscodeexample.Theresultisreturnedastheresultofthefunctioncall,anderrorsarehandledinanaturalmannerusingtry/catch.Theawaitkeywordintegratesasynchronousresultshandlingwithoutblockingtheexecutionthread.Alotisburiedunderthecoversoftheasync/awaitfeature,andwe'llbecoveringthismodelextensivelythroughoutthebook.

Commonly,webpagesbringtogetherdatafromdozensofsources.Eachonehasaqueryandresponseasdiscussedearlier.Usingasynchronousqueries,eachquerycanhappeninparallel,wherethepageconstructionfunctioncanfireoffdozensofqueries—nowaiting,eachwiththeirowncallback—andthengobacktotheeventloop,invokingthecallbacksaseachisdone.Becauseit'sinparallel,thedatacanbecollectedmuchmorequicklythanifthesequeriesweredonesynchronouslyoneatatime.Now,thereaderonthewebbrowserishappierbecausethepageloadsmorequickly.

PerformanceandutilizationSomeoftheexcitementoverNode.jsisduetoitsthroughput(therequestsperseconditcanserve).Comparativebenchmarksofsimilarapplications,forexample,Apache,showthatNode.jshastremendousperformancegains.

OnebenchmarkgoingaroundisthissimpleHTTPserver(borrowedfromhttps://nodejs.org/en/),whichsimplyreturnsaHelloWorldmessagedirectlyfrommemory:

varhttp=require('http');

http.createServer(function(req,res){

res.writeHead(200,{'Content-Type':'text/plain'});

res.end('HelloWorld\n');

}).listen(8124,"127.0.0.1");

console.log('Serverrunningathttp://127.0.0.1:8124/');

ThisisoneofthesimplerwebserversthatyoucanbuildwithNode.js.ThehttpobjectencapsulatestheHTTPprotocol,anditshttp.createServermethodcreatesawholewebserver,listeningontheportspecifiedinthelistenmethod.Everyrequest(whetheraGETorPOSTonanyURL)onthatwebservercallstheprovidedfunction.Itisverysimpleandlightweight.Inthiscase,regardlessoftheURL,itreturnsasimpletext/plainthatistheHelloWorldresponse.

RyanDahlshowedasimplebenchmark(https://www.youtube.com/watch?v=M-sc73Y-zQA)thatreturneda1-megabytebinarybuffer;Node.jsgave822req/sec,whileNginxgave708req/sec,fora15%improvementoverNginx.HealsonotedthatNginxpeakedatfourmegabytesmemory,whileNode.jspeakedat64megabytes.

ThekeyobservationwasthatNode.js,runninganinterpretedJIT-compiledhigh-levellanguage,wasaboutasfastasNginx,builtofhighlyoptimized

Ccode,whilerunningsimilartasks.ThatpresentationwasinMay2010,andNode.jshasimprovedhugelysincethen,asshowninChrisBailey'stalkthatwereferencedearlier.

Yahoo!searchengineerFabianFrankpublishedaperformancecasestudyofareal-worldsearchquerysuggestionwidgetimplementedwithApache/PHPandtwovariantsofNode.jsstacks(http://www.slideshare.net/FabianFrankDe/nodejs-performance-case-study).Theapplicationisapop-uppanelshowingsearchsuggestionsastheusertypesinphrases,usingaJSON-basedHTTPquery.TheNode.jsversioncouldhandleeighttimesthenumberofrequestspersecondwiththesamerequestlatency.FabianFranksaidbothNode.jsstacksscaledlinearlyuntilCPUusagehit100%.Inanotherpresentation(http://www.slideshare.net/FabianFrankDe/yahoo-scale-nodejs),hediscussedhowYahoo!AxisisrunningonManhattan+Mojitoandthevalueofbeingabletousethesamelanguage(JavaScript)andframework(YUI/YQL)onbothfrontendandbackend.

LinkedIndidamassiveoverhauloftheirmobileappusingNode.jsfortheserver-sidetoreplaceanoldRubyonRailsapp.Theswitchletthemmovefrom30serversdowntothree,andallowedthemtomergethefrontendandbackendteambecauseeverythingwaswritteninJavaScript.BeforechoosingNode.js,they'devaluatedRailswithEventMachine,PythonwithTwisted,andNode.js,choosingNode.jsforthereasonsthatwejustdiscussed.ForalookatwhatLinkedIndid,seehttp://arstechnica.com/information-technology/2012/10/a-behind-the-scenes-look-at-

linkedins-mobile-engineering/.

MostexistingadviceonNode.jsperformancetipstendstohavebeenwrittenforolderV8versionsthatusedtheCrankShaftoptimizer.TheV8teamhascompletelydumpedCrankShaft,andithasanewoptimizercalledTurboFan.Forexample,underCrankShaft,itwasslowertousetry/catch,let/const,generatorfunctions,andsoon.Therefore,commonwisdomsaidtonotusethosefeatures,whichisdepressingbecausewewanttousethenewJavaScriptfeaturesbecauseofhowmuchithasimprovedtheJavaScriptlanguage.PeterMarshall,anEngineerontheV8teamatGoogle,gaveatalkatNode.jsInteractive2017claimingthat,

underTurboFan,youshouldjustwritenaturalJavaScript.WithTurboFan,thegoalisforacross-the-boardperformanceimprovementsinV8.Toviewthepresentation,seehttps://www.youtube.com/watch?v=YqOhBezMx1o.

AtruismaboutJavaScriptisthatit'snogoodforheavycomputationwork,becauseofthenatureofJavaScript.We'llgooversomeideasrelatedtothisinthenextsection.AtalkbyMikolaLysenkoatNode.jsInteractive2016wentoversomeissueswithnumericalcomputinginJavaScript,andsomepossiblesolutions.CommonnumericalcomputinginvolveslargenumericalarraysprocessedbynumericalalgorithmsthatyoumighthavelearnedinCalculusorLinearAlgebraclasses.WhatJavaScriptlacksismulti-dimensionalarrays,andaccesstocertainCPUinstructions.Thesolutionhepresentedisalibrarytoimplementmulti-dimensionalarraysinJavaScript,alongwithanotherlibraryfullofnumericalcomputingalgorithms.Toviewthepresentation,seehttps://www.youtube.com/watch?v=1ORaKEzlnys.

ThebottomlineisthatNode.jsexcelsatevent-drivenI/Othroughput.WhetheraNode.jsprogramcanexcelatcomputationalprogramsdependsonyouringenuityinworkingaroundsomelimitationsintheJavaScriptlanguage.Abigproblemwithcomputationalprogrammingisthatitpreventstheeventloopfromexecutingand,aswewillseeinthenextsection,thatcanmakeNode.jslooklikeapoorcandidateforanything.

IsNode.jsacancerousscalabilitydisaster?InOctober2011,softwaredeveloperandbloggerTedDziubawroteablogpost(sincepulledfromhisblog)titledNode.jsisacancer,callingitascalabilitydisaster.TheexampleheshowedforproofisaCPU-boundimplementationoftheFibonaccisequencealgorithm.Whilehisargumentwasflawed,heraisedavalidpointthatNode.jsapplicationdevelopershavetoconsiderthefollowing:wheredoyouputtheheavycomputationaltasks?

AkeytomaintaininghighthroughputofNode.jsapplicationsisensuringthateventsarehandledquickly.Becauseitusesasingleexecutionthread,ifthatthreadisboggeddownwithabigcalculation,Node.jscannothandleevents,andeventthroughputwillsuffer.

TheFibonaccisequence,servingasastand-inforheavycomputationaltasks,quicklybecomescomputationallyexpensivetocalculate,especiallyforanaïveimplementationsuchasthis:

constfibonacci=exports.fibonacci=function(n){

if(n===1||n===2)return1;

elsereturnfibonacci(n-1)+fibonacci(n-2);

}

Yes,therearemanywaystocalculatefibonaccinumbersmorequickly.WeareshowingthisasageneralexampleofwhathappenstoNode.jswheneventhandlersareslow,andnottodebatethebestwaystocalculatemathematicsfunctions.Considerthisserver:

consthttp=require('http');

consturl=require('url');

constfibonacci=//asabove

http.createServer(function(req,res){

consturlP=url.parse(req.url,true);

letfibo;

res.writeHead(200,{'Content-Type':'text/plain'});

if(urlP.query['n']){

fibo=fibonacci(urlP.query['n']);

res.end('Fibonacci'+urlP.query['n']+'='+fibo);

}else{

res.end('USAGE:http://127.0.0.1:8124?n=##where##istheFibonacci

numberdesired');

}

}).listen(8124,'127.0.0.1');

console.log('Serverrunningathttp://127.0.0.1:8124');

Forsufficientlylargevaluesofn(forexample,40),theserverbecomescompletelyunresponsivebecausetheeventloopisnotrunning,andinsteadthisfunctionisblockingeventprocessingbecauseitisgrindingthroughthecalculation.

DoesthismeanthatNode.jsisaflawedplatform?No,itjustmeansthattheprogrammermusttakecaretoidentifycodewithlong-runningcomputationsanddevelopsolutions.Theseincluderewritingthealgorithmtoworkwiththeeventloop,orrewritingthealgorithmforefficiency,orintegratinganativecodelibrary,orfoistingcomputationallyexpensivecalculationsontoabackendserver.

Asimplerewritedispatchesthecomputationsthroughtheeventloop,lettingtheservercontinuetohandlerequestsontheeventloop.Usingcallbacksandclosures(anonymousfunctions),we'reabletomaintainasynchronousI/Oandconcurrencypromises:

constfibonacciAsync=function(n,done){

if(n===0)return0;

elseif(n===1||n===2)done(1);

elseif(n===3)return2;

else{

process.nextTick(function(){

fibonacciAsync(n-1,function(val1){

process.nextTick(function(){

fibonacciAsync(n-2,function(val2){

done(val1+val2);});

});

});

});

}

}

Becausethisisanasynchronousfunction,itnecessitatesasmallrefactoringoftheserver:

consthttp=require('http');

consturl=require('url');

constfibonacciAsync=//asabove

http.createServer(function(req,res){

leturlP=url.parse(req.url,true);

res.writeHead(200,{'Content-Type':'text/plain'});

if(urlP.query['n']){

fibonacciAsync(urlP.query['n'],fibo=>{

res.end('Fibonacci'+urlP.query['n']+'='+fibo);

});

}else{

res.end('USAGE:http://127.0.0.1:8124?n=##where##istheFibonacci

numberdesired');

}

}).listen(8124,'127.0.0.1');console.log('Serverrunningat

http://127.0.0.1:8124');

Dziuba'svalidpointwasn'texpressedwellinhisblogpost,anditwassomewhatlostintheflamesfollowingthatpost.Namely,thatwhileNode.jsisagreatplatformforI/O-boundapplications,itisn'tagoodplatformforcomputationallyintensiveones.

Laterinthisbook,we'llexplorethisexamplealittlemoredeeply.

Serverutilization,thebusinessbottomline,andgreenwebhostingThestrivingforoptimalefficiency(handlingmorerequestspersecond)isnotjustaboutthegeekysatisfactionthatcomesfromoptimization.Therearerealbusinessandenvironmentalbenefits.Handlingmorerequestspersecond,asNode.jsserverscando,meansthedifferencebetweenbuyinglotsofserversandbuyingonlyafewservers.Node.jspotentiallyletsyourorganizationdomorewithless.

Roughlyspeaking,themoreserversyoubuy,thegreaterthecost,andthegreatertheenvironmentalimpactofhavingthoseservers.There'sawholefieldofexpertisearoundreducingcostsandtheenvironmentalimpactofrunningwebserverfacilities,towhichthatroughguidelinedoesn'tdojustice.Thegoalisfairlyobvious—fewerservers,lowercosts,andareducedenvironmentalimpactthroughutilizingmoreefficientsoftware.

Intel'spaper,IncreasingDataCenterEfficiencywithServerPowerMeasurements(https://www.intel.com/content/dam/doc/white-paper/intel-it-data-center-efficiency-server-power-paper.pdf),givesanobjectiveframeworkforunderstandingefficiencyanddatacentercosts.Therearemanyfactors,suchasbuildings,coolingsystems,andcomputersystemdesigns.Efficientbuildingdesign,efficientcoolingsystems,andefficientcomputersystems(datacenterefficiency,datacenterdensity,andstoragedensity)canlowercostsandenvironmentalimpact.Butyoucandestroythosegainsbydeployinganinefficientsoftwarestackcompellingyoutobuymoreserversthanyouwouldifyouhadanefficientsoftwarestack.Alternatively,youcanamplifygainsfromdatacenterefficiencywithanefficientsoftwarestackthatletsyoudecreasethenumberofserversrequired.

Thistalkaboutefficientsoftwarestacksisn'tjustforaltruisticenvironmentalpurposes.Thisisoneofthosecaseswherebeinggreencanhelpyourbusinessbottomline.

EmbracingadvancesintheJavaScriptlanguageThelastcoupleofyearshavebeenanexcitingtimeforJavaScriptprogrammers.TheTC-39committeethatoverseestheECMAScriptstandardhasaddedmanynewfeatures,someofwhicharesyntacticsugar,butseveralofwhichhavepropelledusintoawholeneweraofJavaScriptprogramming.Byitself,theasync/awaitfeaturepromisesusawayoutofwhat'scalledCallbackHell,orthesituationwefindourselvesinwhennestingcallbackswithincallbacks.It'ssuchanimportantfeaturethatitshouldnecessitateabroadrethinkingoftheprevailingcallback-orientedparadigminNode.jsandtherestoftheJavaScriptecosystem.

Referbackafewpagestothis:

query('SELECT*fromdb',function(err,result){

if(err)throwerr;//handleerrors

//operateonresult

});

ThiswasanimportantinsightonRyanDahl'spart,andiswhatpropelledNode.js'spopularity.Certainactionstakealongtimetorun,suchasdatabasequeries,andshouldnotbetreatedthesameasoperationsthatquicklyretrievedatafrommemory.BecauseofthenatureoftheJavaScriptlanguage,Node.jshadtoexpressthisasynchronouscodingconstructinanunnaturalway.Theresultsdonotappearatthenextlineofcode,butinsteadappearwithinthiscallbackfunction.Further,errorshavetobehandledinanunnaturalway,insidethatcallbackfunction.

TheconventioninNode.jsisthatthefirstparametertoacallbackfunctionisanerrorindicator,andthesubsequentparametersaretheresults.Thisisausefulconventionthatyou'llfindallacrosstheNode.jslandscape.

However,itcomplicatesworkingwithresultsanderrorsbecausebothlandinaninconvenientlocation—thatcallbackfunction.Thenaturalplaceforerrorsandresultstolandisonthesubsequentline(s)ofcode.

Wedescendfurtherintocallbackhellwitheachlayerofcallbackfunctionnesting.Theseventhlayerofcallbacknestingismorecomplexthanthesixthlayerofcallbacknesting.Why?Ifnothingelse,it'sthatthespecialconsiderationsforerrorhandlingbecomeevermorecomplexascallbacksarenestedmoredeeply.

varresults=awaitquery('SELECT*fromdb');

Instead,ES2017asyncfunctionsreturnustothisverynaturalexpressionofprogrammingintent.Resultsanderrorslandinthecorrectlocation,whilepreservingtheexcellentevent-drivenasynchronousprogrammingmodelthatmadeNode.jsgreat.We'llseelaterinthebookhowthisworks.

TheTC-39committeeaddedmanymorenewfeaturestoJavaScript,suchas:

AnimprovedsyntaxforClassdeclarationsmakingobjectinheritanceandgetter/setterfunctionsverynatural.

AnewmoduleformatthatisstandardizedacrossbrowsersandNode.js.

Newmethodsforstrings,suchasthetemplatestringnotation.

Newmethodsforcollectionsandarrays—forexample,operationsformap/reduce/filter.

Theconstkeywordtodefinevariablesthatcannotbechanged,andtheletkeywordtodefinevariableswhosescopeislimitedtotheblockinwhichthey'redeclared,ratherthanhoistedtothefrontofthefunction.

Newloopingconstructs,andaniterationprotocolthatworkswiththosenewloops.

Anewkindoffunction,thearrowfunction,whichislighterweightmeaninglessmemoryandexecutiontimeimpact

ThePromiseobjectrepresentsaresultthatispromisedtobedeliveredinthefuture.Bythemselves,Promisescanmitigatethecallbackhellproblem,andtheyformpartofthebasisforasyncfunctions.

Generatorfunctionsareanintriguingwaytorepresentasynchronousiterationoverasetofvalues.Moreimportantly,theyformtheotherhalfofthebasisforasyncfunctions.

YoumayseethenewJavaScriptdescribedasES6orES2017.What'sthepreferrednametodescribetheversionofJavaScriptthatisbeingused?

ES1throughES5markedvariousphasesofJavaScript'sdevelopment.ES5wasreleasedin2009,andiswidelyimplementedinmodernbrowsers.StartingwithES6,theTC-39committeedecidedtochangethenamingconventionbecauseoftheirintentiontoaddnewlanguagefeatureseveryyear.Therefore,thelanguageversionnamenowincludestheyear,henceES2015wasreleasedin2015,ES2016wasreleasedin2016,andES2017wasreleasedin2017.

DeployingES2015/2016/2017/2018JavaScriptcodeThepinkelephantintheroomisthat,becauseofhowJavaScriptisdeliveredtotheworld,wecannotjuststartusingthelatestES2017features.InfrontendJavaScript,wearelimitedbythefactthatoldbrowsersarestillinuse.InternetExplorerversion6hasfortunatelybeenalmostcompletelyretired,buttherearestillplentyofoldbrowsersinstalledonoldercomputersthatarestillservingavalidrolefortheirowners.OldbrowsersmeanoldJavaScriptimplementations,andifwewantourcodetowork,weneedittobecompatiblewitholdbrowsers.

UsingcoderewritingtoolssuchasBabel,someofthenewfeaturescanberetrofittedtofunctiononsomeoftheolderbrowsers.FrontendJavaScriptprogrammerscanadopt(someof)thenewfeaturesatthecostofamorecomplexbuildtoolchain,andtheriskofbugsintroducedbythecoderewritingprocess.Somemaywishtodothat,whileotherswillprefertowaitawhile.

TheNode.jsworlddoesn'thavethisproblem.Node.jshasrapidlyadoptedES2015/2016/2017featuresasquicklyastheywereimplementedintheV8engine.WithNode.js8,wecannowuseasyncfunctionsasanativefeature,andmostoftheES2015/2016featuresbecameavailablewithNode.jsversion6.ThenewmoduleformatisnowsupportedinNode.jsversion10.

Inotherwords,whilefrontendJavaScriptprogrammerscanarguethattheymustwaitacoupleofyearsbeforeadoptingES2015/2016/2017features,Node.jsprogrammershavenoneedtowait.Wecansimplyusethenewfeatureswithoutneedinganycoderewritingtools.

Node.js,themicroservicearchitecture,andeasilytestablesystemsNewcapabilities,suchasclouddeploymentsystemsandDocker,makeitpossibletoimplementanewkindofservicearchitecture.Dockermakesitpossibletodefineserverprocessconfigurationinarepeatablecontainerthat'seasytodeploybythemillionsintoacloudhostingsystem.Itlendsitselfbesttosmallsingle-purposeserviceinstancesthatcanbeconnectedtogethertomakeacompletesystem.Dockerisn'ttheonlytooltohelpsimplifyclouddeployments;however,itsfeaturesarewellattunedtomodernapplicationdeploymentneeds.

Somehavepopularizedthemicroserviceconceptasawaytodescribethiskindofsystem.Accordingtothemicroservices.iowebsite,amicroserviceconsistsofasetofnarrowlyfocused,independentlydeployableservices.Theycontrastthiswiththemonolithicapplicationdeploymentpatternwhereeveryaspectofthesystemisintegratedintoonebundle(suchasasingleWARfileforaJavaEEappserver).Themicroservicemodelgivesdevelopersmuchneededflexibility.

Someadvantagesofmicroservicesareasfollows:

Eachmicroservicecanbemanagedbyasmallteam

Eachteamcanworkonitsownschedule,solongastheserviceAPIcompatibilityismaintained

Microservicescanbedeployedindependently,suchasforeasiertesting

It'seasiertoswitchtechnologystackchoices

WheredoesNode.jsfitinwiththis?Itsdesignfitsthemicroservicemodellikeaglove:

Node.jsencouragessmall,tightlyfocused,single-purposemodules

Thesemodulesarecomposedintoanapplicationbytheexcellentnpmpackagemanagementsystem

Publishingmodulesisincrediblysimple,whetherviatheNPMrepositoryoraGitURL

Node.jsandtheTwelve-FactorappmodelThroughoutthisbook,we'llcalloutaspectsoftheTwelve-FactorAppmodel,andwaystoimplementthoseideasinNode.js.Thismodelispublishedonhttp://12factor.net,andisasetofguidelinesforapplicationdeploymentinthemoderncloudcomputingera.It'snotthattheTwelve-FactorAppmodelisthebe-allandend-allofapplicationarchitectureparadigms.It'sasetofusefulideas,clearlybirthedaftermanylatenightsspentdebuggingcomplexapplications,whichofferusefulideasthatcouldsaveusallalotofeffortbyhavingeasier-to-maintainandmorereliablesystems.

Theguidelinesarestraightforward,andonceyoureadthem,theywillseemlikepurecommonsense.Asabestpractice,theTwelve-FactorAppmodelisacompellingstrategyfordeliveringthekindoffluidself-containedcloud-deployedapplicationscalledforbyourcurrentcomputingenvironment.

SummaryYoulearnedalotinthischapter.Specifically,yousawthatJavaScripthasalifeoutsidewebbrowsersandyoulearnedaboutthedifferencebetweenasynchronousandblockingI/O.WethencoveredtheattributesofNode.jsandwhereitfitsintheoverallwebapplicationplatformmarketandthreadedversusasynchronoussoftware.Lastly,wesawtheadvantagesoffastevent-drivenasynchronousI/O,coupledwithalanguagewithgreatsupportforanonymousclosures.

Ourfocusinthisbookisreal-worldconsiderationsofdevelopinganddeployingNode.jsapplications.We'llcoverasmanyaspectsaswecanofdeveloping,refining,testing,anddeployingNode.jsapplications.

Nowthatwe'vehadthisintroductiontoNode.js,we'rereadytodiveinandstartusingit.InChapter2,SettingupNode.js,we'llgooversettingupaNode.jsenvironment,solet'sgetstarted.

SettingupNode.jsBeforegettingstartedwithusingNode.js,youmustsetupyourdevelopmentenvironment.Inthefollowingchapters,we'llusethisfordevelopmentandfornon-productiondeployment.

Inthischapter,wewillcoverthefollowingtopics:

HowtoinstallNode.jsfromsourceandprepackagedbinariesonLinux,macOS,orWindows

HowtoinstallNodePackageManager(NPM)andsomepopulartools

TheNode.jsmodulesystem

Node.jsandJavaScriptlanguageimprovementsfromtheECMAScriptcommittee

Solet'sgetonwithit.

SystemrequirementsNode.jsrunsonPOSIX-likeoperatingsystems,variousUNIXderivatives(Solaris,forexample)orworkalikes(Linux,macOS,andsoon),aswellasonMicrosoftWindows.Itcanrunonmachinesbothlargeandsmall,includingthetinyARMdevicessuchastheRaspberryPimicroscaleembeddablecomputerforDIYsoftware/hardwareprojects.

Node.jsisnowavailableviapackagemanagementsystems,limitingtheneedtocompileandinstallfromsource.

BecausemanyNode.jspackagesarewritteninCorC++,youmusthaveaCcompiler(suchasGCC),Python2.7(orlater),andthenode-gyppackage.Ifyouplantouseencryptioninyournetworkingcode,youwillalsoneedtheOpenSSLcryptographiclibrary.ThemodernUNIXderivativesalmostcertainlycomewiththese,andNode.js'sconfigurescript,usedwheninstallingfromsource,willdetecttheirpresence.Ifyouneedtoinstallthem,Pythonisavailableathttp://python.organdOpenSSLisavailableathttp://openssl.org.

InstallingNode.jsusingpackagemanagersThepreferredmethodforinstallingNode.js,now,istousetheversionsavailableinpackagemanagers,suchasapt-get,orMacPorts.Packagemanagerssimplifyyourlifebyhelpingtomaintainthecurrentversionofthesoftwareonyourcomputer,ensuringtoupdatedependentpackagesasnecessary,allbytypingasimplecommandsuchasapt-getupdate.Let'sgooverthisfirst.

InstallingonmacOSwithMacPortsTheMacPortsproject(http://www.macports.org/)hasforyearsbeenpackagingalonglistofopensourcesoftwarepackagesformacOS,andtheyhavepackagedNode.js.AfteryouhaveinstalledMacPortsusingtheinstallerontheirwebsite,installingNode.jsisprettymuchthissimple:

$portsearchnodejsnpm

...

nodejs6@6.12.0(devel,net)

EventedI/OforV8JavaScript

nodejs7@7.10.1(devel,net)

EventedI/OforV8JavaScript

nodejs8@8.9.1(devel,net)

EventedI/OforV8JavaScript

nodejs9@9.2.0(devel,net)

EventedI/OforV8JavaScript

Found6ports.

--

npm4@4.6.1(devel)

nodepackagemanager

npm5@5.5.1(devel)

nodepackagemanager

Found4ports.

$sudoportinstallnodejs8npm5

..longlogofdownloadingandinstallingprerequisitesandNode

$whichnode

optlocal/bin/node

$node--version

v8.9.1

<strong>$brewupdate</strong><br/><strong>...longwaitandlotsofoutput</strong><br/><strong>$brewsearchnode</strong><br/><strong>==>Searchinglocaltaps...</strong><br/><strong>node<imgsrc="Images/902ea8ae-c8f4-4de3-a41e-a21d10704fd1.png"style="width:1.67em;height:1.67em;"width="150"height="150">libbitcoin-nodenode-buildnode@6nodeenv<strong><br/><strong>leafnodellnodenode@4nodebrewnodenv</strong><br/><strong>==>SearchingtapsonGitHub...</strong><br/><strong>caskroom/cask/node-profiler</strong><br/><strong>==>Searchingblacklisted,migratedanddeletedformulae...</strong>

<strong>$brewinstallnode</strong><br/><strong>...</strong><br/><strong>==>Installingnode</strong><br/><strong>==>Downloadinghttps://homebrew.bintray.com/bottles/node-8.9.1.el_capitan.bottle.tar.gz</strong><br/><strong>########################################################################100.0%</strong><br/><strong>==>Pouringnode-8.9.1.el_capitan.bottle.tar.gz</strong><br/><strong>==>Caveats</strong><br/><strong>Bashcompletionhasbeeninstalledto:</strong><br/><strong>usrlocal/etc/bash_completion.d</strong><br/><strong>==>Summary</strong><br/><strong><imgsrc="Images/062f1f44-c05b-4b0b-881f-b7ad78f50bc9.png"style="width:1.33em;height:1.33em;"width="150"height="150"/>usrlocal/Cellar/node/8.9.1:5,012files,49.6MB</strong>

<strong>$node--versionv8.9.1</strong>

<strong>#curl-sLhttps://deb.nodesource.com/setup_10.x|sudo-Ebash-

#sudoapt-getinstall-ynodejs#sudoapt-getinstall-ybuild-essential</strong>

TodownloadotherNode.jsversions(thisexampleshowsversion10.x),modifytheURLtosuit.

InstallingNode.jsintheWindowsSubsystemforLinux(WSL)TheWindowsSubsystemforLinux(WSL)letsyouinstallUbuntu,openSUSE,orSUSELinuxEnterpriseonWindows.AllthreeareavailableviatheStorebuiltintoWindows10.YoumayneedtoupdateyourWindowsfortheinstallationtowork.

Onceinstalled,theLinux-specificinstructionswillinstallNode.jswithintheLinuxsubsystem.

ToinstalltheWSL,seehttps://msdn.microsoft.com/en-us/commandline/wsl/install-win10.

Openinganadministrator-privilegedPowerShellonWindowsSomeofthecommandsyou'llrunwhileinstallingtoolsonWindowsaretobeexecutedinaPowerShellwindowwithelevatedprivileges.WementionthisbecausetheprocessofenablingtheWSLincludesacommandtoberuninsuchaPowerShellwindow.

Theprocessissimple:

1. IntheStartmenu,enterPowerShellintheapplicationssearchbox.2. TheresultantmenuwilllistPowerShell.3. Right-clickthePowerShellentry.4. ThecontextmenuthatcomesupwillhaveanentryRunas

Administrator.Clickonthat.

Theresultantcommandwindowwillhaveadministratorprivileges,andthetitlebarwillsayAdministrator:WindowsPowerShell.

InstallingtheNode.jsdistributionfromnodejs.orgThehttps://nodejs.org/en/websiteoffersbuilt-inbinariesforWindows,macOS,Linux,andSolaris.Wecansimplygotothewebsite,clickontheInstallbutton,andruntheinstaller.Forsystemswithpackagemanagers,suchastheoneswe'vejustdiscussed,it'spreferabletousethepackagemanagementsystem.That'sbecauseyou'llfinditeasiertostayup-to-datewiththelatestversion.But,thatdoesn'tserveallpeoplebecause:

Somewillprefertoinstallabinaryratherthandealwiththepackagemanager

Theirchosensystemdoesn'thaveapackagemanagementsystem

TheNode.jsimplementationintheirpackagemanagementsystemisout-of-date

SimplygototheNode.jswebsiteandyou'llseesomethinglikethefollowingscreenshot.ThepagedoesitsbesttodetermineyourOSandsupplytheappropriatedownload.Ifyouneedsomethingdifferent,clickontheDOWNLOADSlinkintheheaderforallpossibledownloads:

FormacOS,theinstallerisaPKGfilegivingthetypicalinstallationprocess.ForWindows,theinstallersimplytakesyouthroughthetypicalInstallWizardprocess.

Oncefinishedwiththeinstaller,youhavecommand-linetools,suchasnodeandnpm,withwhichyoucanrunNode.jsprograms.OnWindows,you'resuppliedwithaversionoftheWindowscommandshellpreconfiguredtoworknicelywithNode.js.

InstallingfromsourceonPOSIX-likesystemsInstallingtheprepackagedNode.jsdistributionsisthepreferredinstallationmethod.However,installingNode.jsfromsourceisdesirableinafewsituations:

Itcanletyouoptimizethecompilersettingsasdesired

Itcanletyoucross-compile,say,foranembeddedARMsystem

YoumightneedtokeepmultipleNode.jsbuildsfortesting

YoumightbeworkingonNode.jsitself

Nowthatyouhavethehigh-levelview,let'sgetourhandsdirtymuckingaroundinsomebuildscripts.Thegeneralprocessfollowstheusualconfigure,make,andmakeinstallroutinethatyoumayalreadyhaveperformedwithotheropensourcesoftwarepackages.Ifnot,don'tworry,we'llguideyouthroughtheprocess.

TheofficialinstallationinstructionsareintheREADME.mdcontainedwithinthesourcedistributionathttps://github.com/nodejs/node/blob/master/README.md.

InstallingprerequisitesTherearethreeprerequisites:aCcompiler,Python,andtheOpenSSLlibraries.TheNode.jscompilationprocesschecksfortheirpresenceandwillfailiftheCcompilerorPythonisnotpresent.Thespecificmethodofinstallingtheseisdependentonyouroperatingsystem.

Thesesortsofcommandswillcheckfortheirpresence:$cc--versionAppleLLVMversion7.0.2(clang-700.1.81)Target:x86_64-apple-darwin15.3.0Threadmodel:posix$pythonPython2.7.11(default,Jan82016,22:23:13)[GCC4.2.1CompatibleAppleLLVM7.0.2(clang-700.1.81)]ondarwinType"help","copyright","credits"or"license"formoreinformation.>>>

Seethisfordetails:https://github.com/nodejs/node/blob/master/BUILDING.md.

TheNode.jsbuildtoolsdonotsupportPython3.x.

InstallingdevelopertoolsonmacOSDevelopertools(suchasGCC)areanoptionalinstallationonmacOS.Fortunately,they'reeasytoacquire.

YoustartwithXcode,whichisavailableforfreethroughtheMacAppStore.SimplysearchforXcodeandclickontheGetbutton.OnceyouhaveXcodeinstalled,openaTerminalwindowandtypethefollowing:$xcode-select--install

ThisinstallstheXcodecommand-linetools:

Foradditionalinformation,visithttp://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/.

InstallingfromsourceforallPOSIX-likesystemsCompilingNode.jsfromsourcefollowsthisprocess:

1. Downloadthesourcefromhttp://nodejs.org/download.

2. Configurethesourceforbuildingusing./configure.3. Runmake,thenmakeinstall.

Thesourcebundlecanbedownloadedwithyourbrowser,orasfollows,substitutingyourpreferredversion:

$mkdirsrc

$cdsrc

$wgethttps://nodejs.org/dist/v10.0.0/node-v10.0.0.tar.gz

$tarxvfznode-v10.0.0.tar.gz

$cdnode-v10.0.0

Nowweconfigurethesourcesothatitcanbebuilt.Thisisjustlikemanyotheropensourcepackages,andtherearealonglistofoptionstocustomizethebuild:

$./configure--help

Tocausetheinstallationtolandinyourhomedirectory,runitthisway:

$./configure--prefix=$HOME/node/10.0.0

..outputfromconfigure

Ifyou'regoingtoinstallmultipleNode.jsversionssidebyside,it'susefultoputtheversionnumberinthepathlikethis.Thatway,eachversionwillsitinaseparatedirectory.It'sasimplematterofswitchingbetweenNode.jsversionsbychangingthePATHvariableappropriately:

#Onbashshell:

$exportPATH=${HOME}/node/VERSION-NUMBER/bin:${PATH}

#Oncsh

$setenvPATH${HOME}/node/VERSION-NUMBER/bin:${PATH}

AsimplerwaytoinstallmultipleNode.jsversionsisthenvmscriptdescribedlater.

IfyouwanttoinstallNode.jsinasystem-widedirectory,simplyleaveoffthe--prefixoptionanditwilldefaulttoinstallinginusrlocal.

Afteramoment,it'llstopandwilllikelyhavesuccessfullyconfiguredthesourcetreeforinstallationinyourchosendirectory.Ifthisdoesn'tsucceed,theerrormessagesthatareprintedwilldescribewhatneedstobefixed.Oncetheconfigurescriptissatisfied,youcangoontothenextstep.

Withtheconfigurescriptsatisfied,youcompilethesoftware:

$make

..alonglogofcompileroutputisprinted

$makeinstall

Ifyouareinstallingintoasystem-widedirectory,dothelaststepthiswayinstead:

$make

$sudomakeinstall

Onceinstalled,youshouldmakesurethatyouaddtheinstallationdirectorytoyourPATHvariableasfollows:

$echo'exportPATH=$HOME/node/10.0.0/bin:${PATH}'>>~/.bashrc

$.~/.bashrc

Alternatively,forcshusers,usethissyntaxtomakeanexportedenvironmentvariable:

$echo'setenvPATH$HOME/node/10.0.0/bin:${PATH}'>>~/.cshrc

$source~/.cshrc

Thisshouldresultinsomedirectories,asfollows:

$ls~/node/10.0.0/

binincludelibshare

$ls~/node/10.0.0/bin

InstallingfromsourceonWindowsTheBUILDING.mddocumentreferencedpreviouslyhasinstructions.OneusesthebuildtoolsfromVisualStudio,orelsethefullVisualStudio2017product:

VisualStudio2017:https://www.visualstudio.com/downloads/

Buildtools:https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017

Threeadditionaltoolsarerequired:

GitforWindows:http://git-scm.com/download/win

Python:https://www.python.org/

OpenSSL:https://www.openssl.org/source/andhttps://wiki.openssl.org/index.php/Binaries

Then,runtheincluded.\vcbuildscripttoperformthebuild.

InstallingmultipleNode.jsinstanceswithnvmNormally,youwon'tinstallmultipleversionsofNode.jsanddoingsoaddscomplexitytoyoursystem.ButifyouarehackingonNode.jsitself,oraretestingyoursoftwareagainstdifferentNode.jsreleases,youmaywanttohavemultipleNode.jsinstallations.Themethodtodosoisasimplevariationonwhatwe'vealreadydiscussed.

Earlier,whilediscussingbuildingNode.jsfromsource,wenotedthatonecaninstallmultipleNode.jsinstancesinseparatedirectories.It'sonlynecessarytobuildfromsourceifyouneedacustomizedNode.jsbuild,andmostfolkswillbesatisfiedwithpre-builtNode.jsbinaries.They,too,canbeinstalledintoseparatedirectories.

ToswitchbetweenNode.jsversionsissimplyamatterofchangingthePATHvariable(onPOSIXsystems),asfollows,usingthedirectorywhereyouinstalledNode.js:

$exportPATH=usrlocal/node/VERSION-NUMBER/bin:${PATH}

Itstartstobealittletedioustomaintainthisafterawhile.Foreachrelease,youhavetosetupNode.js,NPM,andanythird-partymodulesyoudesireinyourNode.jsinstallation.Also,thecommandshowntochangeyourPATHisnotquiteoptimal.InventiveprogrammershavecreatedseveralversionmanagerstosimplifymanagingmultipleNode.js/NPMreleasesandprovidingcommandstochangeyourPATHthesmartway:

Nodeversionmanager:https://github.com/tj/n

Nodeversionmanager:https://github.com/creationix/nvm

BothmaintainmultiplesimultaneousversionsofNodeandletyoueasilyswitchbetweenversions.Installationinstructionsareavailableontheirrespectivewebsites.

Forexample,withnvm,youcanruncommandslikethese:

$nvmls

...

v6.0.0

v6.1.0

v6.2.2

v6.3.1

v6.4.0

...

v6.11.2

v7.0.0

v7.1.0

v7.10.0

v8.0.0

v8.1.3

v8.2.1

v8.5.0

v8.9.1

v8.9.3

v9.2.0

v9.4.0

v9.5.0

v9.10.1

v9.11.1

->v10.0.0

->system

node->stable(->v8.9.1)(default)

stable->8.9(->v8.9.1)(default)

iojs->N/A(default)

$nvmuse10

Nowusingnodev10.0.0(npmv5.6.0)

$node--version

v10.0.0

$nvmusev4.2

Nowusingnodev4.2.0(npmv2.14.7)

$node--version

v4.2.0

$nvminstall9

Downloadinghttps://nodejs.org/dist/v9.2.0/node-v9.2.0-darwin-x64.tar.xz...

########################################################################

100.0%

WARNING:checksumsarecurrentlydisabledfornode.jsv4.0andlater

Nowusingnodev9.2.0(npmv5.5.1)

$node--version

v9.2.0

$whichnode

/Users/david/.nvm/versions/node/v9.2.0/bin/node

$usrlocal/bin/node--version

v8.9.1

$optlocal/bin/node--version

v8.9.1

Thisdemonstratesthatyoucanhaveasystem-wideNode.jsinstalled,keepmultipleprivateNode.jsversionsmanagedbynvm,andswitchbetweenthemasneeded.WhennewNode.jsversionsarereleased,theyaresimpletoinstallwithnvmeveniftheofficialpackagedversionforyourOSdoesn'timmediatelyupdate.

InstallingnvmonWindowsUnfortunately,nvmdoesnotsupportWindows.Fortunately,acoupleofWindows-specificclonesofthenvmconceptexist:

https://github.com/coreybutler/nvm-windows

https://github.com/marcelklehr/nodist

AnotherrouteistousetheWSL.BecauseinWSLyou'reinteractingwithaLinuxcommandline,youcanusenvmitself.

Manyoftheexamplesinthisbookweretestedusingthenvm-windowsapplication.Thereareslightbehaviordifferences,butitactslargelythesameasnvmforLinuxandmacOS.Thebiggestchangeistheversionnumberspecifierinthenvmuseandnvminstallcommands.

WithnvmforLinuxandmacOSonecantypeasimpleversionnumber,likenvmuse8,anditwillautomaticallysubstitutethelatestreleaseofthenamedNode.jsversion.Withnvm-windowsthesamecommandactsasifyoutyped"nvmuse8.0.0".Inotherwords,withnvm-windowsyoumustusetheexactversionnumber.Fortunately,thelistofsupportedversionsiseasilyavailableusingthe"nvmlistavailable"command.

Nativecodemodulesandnode-gypWhilewewon'tdiscussnativecodemoduledevelopmentinthisbook,wedoneedtomakesurethattheycanbebuilt.SomemodulesintheNPMrepositoryarenativecode,andtheymustbecompiledwithaCorC++compilertobuildthecorresponding.nodefiles(the.nodeextensionisusedforbinarynative-codemodules).

Themodulewilloftendescribeitselfasawrapperforsomeotherlibrary.Forexample,thelibxsltandlibxmljsmodulesarewrappersaroundtheC/C++librariesofthesamename.ThemoduleincludestheC/C++sourcecode,andwheninstalled,ascriptisautomaticallyruntodothecompilationwithnode-gyp.

Thenode-gyptoolisacross-platformcommand-linetoolwritteninNode.jsforcompilingnativeadd-onmodulesforNode.js.We'vementionednativecodemodulesseveraltimes,anditisthistoolthatcompilesthemforusewithNode.js.

Youcaneasilyseethisinactionbyrunningthesecommands:

$mkdirtemp

$cdtemp

$npminstalllibxmljslibxslt

Thisisdoneinatemporarydirectory,soyoucandeleteitafterward.Ifyoursystemdoesnothavethetoolsinstalledtocompilenativecodemodules,you'llseeerrormessages.Otherwise,you'llseeintheoutputanode-gypexecution,followedbymanylinesoftextobviouslyrelatedtocompilingC/C++files.

Thenode-gyptoolhasprerequisitessimilartothoseforcompilingNode.jsfromsource.Namely,aC/C++compiler,aPythonenvironment,andotherbuildtoolssuchasGit.ForUnix/macOS/Linuxsystemsthoseareeasytocomeby.ForWindows,youshouldinstall:

VisualStudioBuildTools:https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017

GitforWindows:http://git-scm.com/download/win

PythonforWindows:https://www.python.org/

Normally,youwon'tneedtoworryaboutinstallingnode-gyp.That'sbecauseitisinstalledbehindthescenesaspartofNPM.That'sdonesothatNPMcanautomaticallybuildnativecodemodules.

ItsGitHubrepositorycontainsdocumentationathttps://github.com/nodejs/node-gyp.

Readingthenode-gypdocumentation,initsrepository,willgiveyouaclearerunderstandingofthecompilationprerequisitesdiscussedpreviously,aswellasofdevelopingnativecodemodules.

Node.jsversionspolicyandwhattouseWejustthrewaroundsomanydifferentNode.jsversionnumbersintheprevioussectionthatyoumayhavebecomeconfusedoverwhichversiontouse.ThisbookistargetingNode.jsversion10.x,andit'sexpectedthateverythingwe'llcoveriscompatiblewithNode.js10.xandanysubsequentrelease.

StartingwithNode.js4.x,theNode.jsteamisfollowingadual-trackapproach.Theeven-numberedreleases(4.x,6.x,8.x,andsoon)arewhatthey'recallingLongTermSupport(LTS),whiletheodd-numberedreleases(5.x,7.x,9.x,andsoon)arewherecurrentnewfeaturedevelopmentoccurs.Whilethedevelopmentbranchiskeptstable,theLTSreleasesarepositionedasbeingforproductionuseandwillreceiveupdatesforseveralyears.

Atthetimeofwriting,Node.js8.xisthecurrentLTSrelease;Node.js9.xwasjustreleasedandwilleventuallybecomeNode.js10.x,whichinturnwilleventuallybecometheLTSrelease.Forcompletedetailsaboutthereleaseschedule,refertohttps://github.com/nodejs/LTS/.

AmajorimpactofeachnewNode.jsrelease,beyondtheusualperformanceimprovementsandbugfixes,isbringinginthelatestV8JavaScriptenginerelease.Inturn,thismeansbringinginmoreoftheES-2015/2016/2017featuresastheV8teamimplementsthosefeatures.InNode.js8.x,async/awaitfunctionsarrived,andinNode.js10.xsupportforthestandardES6moduleformathasarrived.

ApracticalconsiderationiswhetheranewNode.jsreleasewillbreakyourcode.NewlanguagefeaturesarealwaysbeingaddedasV8catchesupwithECMAScript,andtheNode.jsteamsometimesmakesbreaking

changesintheNode.jsAPI.Ifyou'vetestedononeNode.jsversion,willitworkonanearlierversion?WillaNode.jschangebreaksomeassumptionswemade?

TheNPMPackageManagerhelpsusensurethatourpackagesexecuteonthecorrectNode.jsversion.Thismeansthatwecanspecifyinthepackage.jsonfile,whichwe'llexploreinChapter3,Node.jsModules,thecompatibleNode.jsversionsforapackage.

Wecanaddanentrytopackage.jsonasfollows:

engines:{

"node":">=6.x"

}

Thismeansexactlywhatitimplies—thatthegivenpackageiscompatiblewithNode.jsversion6.xorlater.

Ofcourse,yourdevelopmentmachine(s)couldhaveseveralNode.jsversionsinstalled.You'llneedtheversionyoursoftwareisdeclaredtosupport,plusanylaterversionsyouwishtoevaluate.

EditorsanddebuggersSinceNode.jscodeisJavaScript,anyJavaScript-awareeditorwillbeuseful.UnlikesomeotherlanguagesthataresocomplexthatanIDEwithcodecompletionisanecessity,asimpleprogrammingeditorisperfectlysufficientforNode.jsdevelopment.

TwoeditorsareworthcallingoutbecausetheyarewritteninNode.js:AtomandMicrosoftVisualStudioCode.

Atom(https://atom.io/)billsitselfasahackableeditorforthe21stcentury.ItisextendablebywritingNode.jsmodulesusingtheAtomAPI,andtheconfigurationfilesareeasilyeditable.Inotherwords,it'shackableinthesamewayplentyofothereditorshavebeen,goingbacktoEmacs,meaningonewritesasoftwaremoduletoaddcapabilitiestotheeditor.TheElectronframeworkwasinventedinordertobuildAtom,andElectronisasupereasywaytobuilddesktopapplicationsusingNode.js.

MicrosoftVisualStudioCode(https://code.visualstudio.com/)isalsoahackableeditor—well,thehomepagesaysextensibleandcustomizable,whichmeansthesamething—thatisalsoopensource,andisalsoimplementedinElectron.Butit'snotahollowme-tooeditor,apingAtomwhileaddingnothingofitsown.Instead,VisualStudioCodeisasolidprogrammerseditorinitsownright,bringinginterestingfunctionalitytothetable.

Asfordebuggers,thereareseveralinterestingchoices.StartingwithNode.js6.3,theinspectorprotocolmadeitpossibletousetheGoogleChromedebugger.VisualStudioCodehasabuilt-indebuggerthatalsousestheinspectorprotocol.

Forafulllistofdebuggingoptionsandtools,seehttps://nodejs.org/en/docs/guides/debugging-getting-started/.

RunningandtestingcommandsNowthatyou'veinstalledNode.js,wewanttodotwothings—verifythattheinstallationwassuccessful,andfamiliarizeyouwiththecommand-linetools.

Node.js'scommand-linetoolsThebasicinstallationofNode.jsincludestwocommands,nodeandnpm.We'vealreadyseenthenodecommandinaction.It'susedeitherforrunningcommand-linescriptsorserverprocesses.Theother,npm,isapackagemanagerforNode.js.

TheeasiestwaytoverifythatyourNode.jsinstallationworksisalsothebestwaytogethelpwithNode.js.Typethefollowingcommand:

$node--help

Usage:node[options][-escript|script.js|-][arguments]

nodeinspectscript.js[arguments]

Options:

-v,--versionprintNode.jsversion

-e,--evalscriptevaluatescript

-p,--printevaluatescriptandprintresult

-c,--checksyntaxcheckscriptwithoutexecuting

-i,--interactivealwaysentertheREPLevenifstdin

doesnotappeartobeaterminal

-r,--requiremoduletopreload(optioncanberepeated)

-scriptreadfromstdin(default;interactivemodeifatty)

--inspect[=[host:]port]activateinspectoronhost:port

(default:127.0.0.1:9229)

--inspect-brk[=[host:]port]

activateinspectoronhost:port

andbreakatstartofuserscript

--inspect-port=[host:]port

sethost:portforinspector

...manymoreoptions

Environmentvariables:

NODE_DEBUG','-separatedlistofcoremodules

thatshouldprintdebuginformation

NODE_DISABLE_COLORSsetto1todisablecolorsintheREPL

NODE_EXTRA_CA_CERTSpathtoadditionalCAcertificates

file

NODE_ICU_DATAdatapathforICU(Intlobject)data

(willextendlinked-indata)

NODE_NO_WARNINGSsetto1tosilenceprocesswarnings

NODE_NO_HTTP2setto1tosuppressthehttp2module

NODE_OPTIONSsetCLIoptionsintheenvironment

viaaspace-separatedlist

NODE_PATH':'-separatedlistofdirectories

prefixedtothemodulesearchpath

NODE_PENDING_DEPRECATIONsetto1toemitpendingdeprecation

warnings

NODE_REPL_HISTORYpathtothepersistentREPLhistory

file

NODE_REDIRECT_WARNINGSwritewarningstopathinsteadof

stderr

OPENSSL_CONFloadOpenSSLconfigurationfromfile

Documentationcanbefoundathttps://nodejs.org/

NotethatthereareoptionsforbothNode.jsandV8(notshowninthepreviouscommandline).RememberthatNode.jsisbuiltontopofV8;ithasitsownuniverseofoptionsthatlargelyfocusondetailsofbytecodecompilationorgarbagecollectionandheapalgorithms.Enternode--v8-optionstoseethefulllistofthem.

Onthecommandline,youcanspecifyoptions,asinglescriptfile,andalistofargumentstothatscript.We'lldiscussscriptargumentsfurtherinthenextsection,RunningasimplescriptwithNode.js.

RunningNode.jswithnoargumentsplopsyouintoaninteractiveJavaScriptshell:

$node

>console.log('Hello,world!');

Hello,world!

undefined

AnycodeyoucanwriteinaNode.jsscriptcanbewrittenhere.ThecommandinterpretergivesagoodTerminal-orienteduserexperienceandisusefulforinteractivelyplayingwithyourcode.Youdoplaywithyourcode,don'tyou?Good!

RunningasimplescriptwithNode.jsNow,let'sseehowtorunscriptswithNode.js.It'squitesimple;let'sstartbyreferringtothehelpmessageshownpreviously.Thecommand-linepatternisjustascriptfilenameandsomescriptarguments,whichshouldbefamiliartoanyonewhohaswrittenscriptsinotherlanguages.

CreatingandeditingNode.jsscriptscanbedonewithanytexteditorthatdealswithplaintextfiles,suchasVI/VIM,Emacs,Notepad++,Atom,VisualStudioCode,Jedit,BBEdit,TextMate,orKomodo.It'shelpfulifit'saprogrammer-orientededitor,ifonlyforthesyntaxcoloring.

Forthisandotherexamplesinthisbook,itdoesn'ttrulymatterwhereyouputthefiles.However,forthesakeofneatness,youcanstartbymakingadirectorynamednode-web-devinthehomedirectoryofyourcomputer,andinsidethatcreatingonedirectoryperchapter(forexample,chap02andchap03).

First,createatextfilenamedls.jswiththefollowingcontent:

constfs=require('fs');

constutil=require('util');

constfs_readdir=util.promisify(fs.readdir);

(async()=>{

constfiles=awaitfs_readdir('.');

for(letfnoffiles){

console.log(fn);

}

})().catch(err=>{console.error(err);});

Next,runitbytypingthefollowingcommand:

$nodels.js

ls.js

ThisisapalecheapimitationoftheUnixlscommand(asifyoucouldn'tfigurethatoutfromthename).ThereaddirfunctionisacloseanalogtotheUnixreaddirsystemcall(typeman3readdirinaTerminalwindowtolearnmore)andisusedtolistthefilesinadirectory.

Wehavewrittenthisusinganinlineasyncfunction,theawaitkeyword,andanES2015for..ofloop.Usingutil.promisify,wecanconvertanycallback-orientedfunctionsoitreturnsaPromise,sothatthePromiseplayswellwiththeawaitkeyword.

Bydefaultfsmodulefunctionsusethecallbackparadigm,asdoesmostNode.jsmodules.Butwithinasyncfunctionsitismoreconvenientiffunctionsinsteadreturnpromises.Usingutil.promisifywecanmakeitso.

Thisscriptishardcodedtolistfilesinthecurrentdirectory.Thereallscommandtakesadirectoryname,solet'smodifythescriptalittle.

Command-lineargumentslandinaglobalarraynamedprocess.argv.Thereforewecanmodifyls.js,copyingitasls2.js,asfollowstoseehowthisarrayworks:

constfs=require('fs');

constutil=require('util');

constfs_readdir=util.promisify(fs.readdir);

(async()=>{

vardir='.';

if(process.argv[2])dir=process.argv[2];

constfiles=awaitfs_readdir(dir);

for(letfnoffiles){

console.log(fn);

}

})().catch(err=>{console.error(err);});

Youcanrunitasfollows:

$pwd

/Users/David/chap02

$nodels2..

chap01

chap02

$nodels2

app.js

ls.js

ls2.js

Wesimplycheckedifacommand-lineargumentwaspresent,if(process.argv[2]).Ifitwas,weoverrodethevalueofthedirvariable,dir=process.argv[2],andwethenusedthatasthereaddirargument.

Ifyougiveitanonexistentdirectorypathname,anerrorwillbethrownandprintedusingthecatchclause.Thatlookslikeso:

$nodels2.js/nonexistent

{Error:ENOENT:nosuchfileordirectory,scandir'/nonexistent'

errno:-2,

code:'ENOENT',

syscall:'scandir',

path:'/nonexistent'}

ConversiontoasyncfunctionsandthePromiseparadigmIntheprevioussectionwediscussedtheutil.promisifyanditsabilitytoconvertacallback-orientedfunctionintoonethatreturnsaPromise.ThelatterplaywellwithinasyncfunctionsandthereforeitispreferableforfunctionstoreturnaPromise.

Tobemoreprecise,util.promisifyistobegivenafunctionthatusestheerror-first-callbackparadigm.Thelastargumentofsuchfunctionsisacallbackfunctionwhosefirstargumentisinterpretedasanerrorindicator,hencethephraseerror-first-callback.Whatutil.promisifyreturnsisanotherfunctionthatreturnsaPromise.

ThePromiseservesthesamepurposeastheerror-first-callback.Ifanerrorisindicated,thePromiseresolvestotherejectedstatus,whileifsuccessisindicatedthePromiseresolvestoasuccessstatus.Asweseeintheseexamples,withinanasyncfunctionthePromiseishandledverynicely.

TheNode.jsecosystemhasalargebodyoffunctionsusingtheerror-first-callback.ThecommunityhasbegunaconversionprocesswherefunctionswillreturnaPromise,andpossiblyalsotakeanerror-first-callbackforAPIcompatibility.

OneofthenewfeaturesinNode.js10isanexampleofsuchaconversion.Withinthefsmoduleisasubmodule,namedfs.promises,withthesameAPIbutproducingPromiseobjects.Wecouldrewritethepreviousexampleasso:constfs=require('fs').promises;(async()=>{vardir='.';if(process.argv[2])dir=process.argv[2];constfiles=awaitfs.readdir(dir);

for(letfnoffiles){console.log(fn);}})().catch(err=>{console.error(err);});

Asyoucansee,thefunctionsinthefs.promisesmodulereturnsaPromisewithoutrequiringacallbackfunction.Thenewprogram,whichyoucansaveasls2-promises.js,isrunasso:

$nodels2-promises.js

(node:40329)ExperimentalWarning:Thefs.promisesAPIisexperimental

app.js

ls.js

ls2-promises.js

ls2.js

TheAPIiscurrentlyinanexperimentalstateandthereforewe'reshownthiswarning.

Anotherchoiceisa3rdpartymodule,fs-extra.ThismodulehasanextendedAPIbeyondthestandardfsmodule.OntheonehanditsfunctionsreturnaPromiseifnocallbackfunctionisprovided,orelseinvokesthecallback.Inadditionitincludesseveralusefulfunctions.

Intherestofthisbookwewillbeusingfs-extrabecauseofthoseadditionalfunctions.Fordocumentationofthemodule,see:https://www.npmjs.com/package/fs-extra.

LaunchingaserverwithNode.jsManyscriptsthatyou'llrunareserverprocesses.We'llberunninglotsofthesescriptslateron.Sincewe'restillinthedualmodeofverifyingtheinstallationandfamiliarizingyouwithusingNode.js,wewanttorunasimpleHTTPserver.Let'sborrowthesimpleserverscriptontheNode.jshomepage(http://nodejs.org).

Createafilenamedapp.jscontainingthefollowing:

consthttp=require('http');

http.createServer(function(req,res){

res.writeHead(200,{'Content-Type':'text/plain'});

res.end('Hello,World!\n');

}).listen(8124,'127.0.0.1');

console.log('Serverrunningathttp://127.0.0.1:8124');

Runitasfollows:

$nodeapp.js

Serverrunningathttp://127.0.0.1:8124

ThisisthesimplestofwebserversyoucanbuildwithNode.js.Ifyou'reinterestedinhowitworks,flipforwardtoChapter4,HTTPServersandClients;Chapter5,YourFirstExpressApplication;andChapter6,ImplementingtheMobile-FirstParadigm.Forthemoment,justvisithttp://127.0.0.1:8124inyourbrowsertoseetheHello,World!message:

Aquestiontoponderiswhythisscriptdidnotexitwhenls.jsdidexit.Inbothcases,executionofthescriptreachestheendofthescript;theNode.jsprocessdoesnotexitinapp.js,whileinls.jsitdoes.

Thereasonisthepresenceofactiveeventlisteners.Node.jsalwaysstartsupaneventloop,andinapp.js,thelistenfunctioncreatesaneventlistenerthatimplementstheHTTPprotocol.Thiseventlistenerkeepsapp.jsrunninguntilyoudosomethingsuchastypingCtrl+CintheTerminalwindow.Inls.js,thereisnothingthatcreatesalong-runningeventlistener,sowhenls.jsreachestheendofitsscript,thenodeprocesswillexit.

NPM–theNode.jspackagemanagerNode.jsbyitselfisaprettybasicsystem,beingaJavaScriptinterpreterwithafewinterestingasynchronousI/Olibraries.OneofthethingsthatmakesNode.jsinterestingistherapidlygrowingecosystemofthird-partymodulesforNode.js.

AtthecenterofthatecosystemisNPM.WhileNode.jsmodulescanbedownloadedassourceandassembledmanuallyforusewithNode.jsprograms,that'stediousandit'sdifficulttoimplementarepeatablebuildprocess.NPMgivesusasimplerway;NPMisthedefactostandardpackagemanagerforNode.jsanditgreatlysimplifiesdownloadingandusingthesemodules.WewilltalkaboutNPMatlengthinthenextchapter.

Thesharp-eyedwillhavenoticedthatnpmisalreadyinstalledviaalltheinstallationmethodsdiscussedpreviously.Inthepast,npmwasinstalledseparately,buttodayitisbundledwithNode.js.

Nowthatwehavenpminstalled,let'stakeitforaquickspin.Thehexyprogramisautilityforprintinghexdumpsoffiles.That'savery1970thingtodo,butisstillextremelyuseful.Itservesourpurposerightnowingivingussomethingtoquicklyinstallandtryout:

$npminstall-ghexy

optlocal/bin/hexy->optlocal/lib/node_modules/hexy/bin/hexy_cmd.js

+hexy@0.2.10

added1packagein1.107s

Addingthe-gflagmakesthemoduleavailableglobally,irrespectiveofthepresent-working-directoryofyourcommandshell.Aglobalinstallismostusefulwhenthemoduleprovidesacommand-lineinterface.Whena

packageprovidesacommand-linescript,npmsetsthatup.Foraglobalinstall,thecommandisinstalledcorrectlyforusebyallusersofthecomputer.

DependingonhowNode.jsisinstalledforyou,thatmayneedtoberunwithsudo:

$sudonpminstall-ghexy

Onceitisinstalled,you'llbeabletorunthenewly–installedprogramthisway:

$hexy--width12ls.js

00000000:636f6e7374206673203d2072const.fs.=.r

0000000c:657175697265282766732729equire('fs')

00000018:3b0a636f6e7374207574696c;.const.util

00000024:203d20726571756972652827.=.require('

00000030:7574696c27293b0a636f6e73util');.cons

0000003c:742066735f72656164646972t.fs_readdir

00000048:203d207574696c2e70726f6d.=.util.prom

00000054:69736966792866732e726561isify(fs.rea

00000060:64646972293b0a0a28617379ddir);..(asy

0000006c:6e63202829203d3e207b0a20nc.().=>.{..

00000078:20636f6e73742066696c6573.const.files

00000084:203d2061776169742066735f.=.await.fs_

00000090:7265616464697228272e2729readdir('.')

0000009c:3b0a2020666f722028666e20;...for.(fn.

000000a8:6f662066696c657329207b0aof.files).{.

000000b4:20202020636f6e736f6c652e....console.

000000c0:6c6f6728666e293b0a20207dlog(fn);...}

000000cc:0a7d2928292e636174636828.})().catch(

000000d8:657272203d3e207b20636f6eerr.=>.{.con

000000e4:736f6c652e6572726f722865sole.error(e

000000f0:7272293b207d293brr);.});

Again,we'llbedoingadeepdiveintoNPMinthenextchapter.ThehexyutilityisbothaNode.jslibraryandascriptforprintingouttheseold-stylehexdumps.

Node.js,ECMAScript2015/2016/2017,andbeyondIn2015,theECMAScriptcommitteereleasedalong-awaitedmajorupdateoftheJavaScriptlanguage.TheupdatebroughtinmanynewfeaturestoJavaScript,suchasPromises,arrowfunctions,andClassobjects.Thelanguageupdatesetthestageforimprovements.sincethatshoulddramaticallyimproveourabilitytowriteclean,understandableJavaScriptcode.

Thebrowsermakersareaddingthosemuch-neededfeatures,meaningtheV8engineisaddingthosefeaturesaswell.ThesefeaturesaremakingtheirwayintoNode.jsstartingwithversion4.x.

TolearnaboutthecurrentstatusofES-2015inNode.js,visithttps://nodejs.org/en/docs/es6/.

Bydefault,onlytheES-2015/2016/2017featuresthatV8considersstableareenabledbyNode.js.Furtherfeaturescanbeenabledwithcommand-lineoptions.Thealmost-completefeaturesareenabledwiththe--es_stagingoption.Thewebsitedocumentationgivesmoreinformation.

TheNodegreenwebsite(http://node.green/)hasatablelistingthestatusofalonglistoffeaturesinNode.jsversions.

TheES2017languagespecispublishedat:https://www.ecma-international.org/publications/standards/Ecma-262.htm.

TheTC-39committeedoesitsworkonGitHubhttps://github.com/tc39.

TheES-2015featuresmakeabigimprovementintheJavaScriptlanguage.Onefeature,thePromiseclass,shouldmeanafundamentalrethinkingofcommonidiomsinNode.jsprogramming.InES-2017,apairofnew

keywords,asyncandawait,willsimplifywritingasynchronouscodeinNode.js,anditshouldencouragetheNode.jscommunitytofurtherrethinkthecommonidiomsoftheplatform.

There'salonglistofnewJavaScriptfeatures,butlet'squicklygoovertwoofthemthatwe'lluseextensively.

Thefirstisalighter-weightfunctionsyntaxcalledthearrowfunction:

fs.readFile('file.txt','utf8',(err,data)=>{

if(err)...;//dosomethingwiththeerror

else...;//dosomethingwiththedata

});

Thisismorethanthesyntacticsugarofreplacingthefunctionkeywordwiththefatarrow.Arrowfunctionsarelighter-weightaswellasbeingeasiertoread.Thelighterweightcomesatthecostofchangingthevalueofthisinsidethearrowfunction.Inregularfunctions,thishasauniquevalueinsidethefunction.Inanarrowfunction,thishasthesamevalueasthescopecontainingthearrowfunction.Thismeansthat,whenusinganarrowfunction,wedon'thavetojumpthroughhoopstobringthisintothecallbackfunctionbecausethisisthesameatbothlevelsofthecode.

ThenextfeatureisthePromiseclass,whichisusedfordeferredandasynchronouscomputations.DeferredcodeexecutiontoimplementasynchronousbehaviorisakeyparadigmforNode.js,anditrequirestwoidiomaticconventions:

Thelastargumenttoanasynchronousfunctionisacallbackfunction,whichiscalledwhenanasynchronousexecutionistobeperformed

Thefirstargumenttothecallbackfunctionisanerrorindicator

Whileconvenient,theseconventionsresultedinmultilayercodepyramidsthatcanbedifficulttounderstandandmaintain:

doThis(arg1,arg2,(err,result1,result2)=>{

if(err)...;

else{

//dosomework

doThat(arg2,arg3,(err2,results)=>{

if(err2)...;

else{

doSomethingElse(arg5,err=>{

if(err)..;

else..;

});

}

});

}

});

Dependingonhowmanystepsarerequiredforaspecifictask,acodepyramidcangetquitedeep.Promiseswillletusunravelthecodepyramidandimprovereliability,becauseerrorhandlingismorestraightforwardandeasilycapturesallerrors.

APromiseclassiscreatedasfollows:

functiondoThis(arg1,arg2){

returnnewPromise((resolve,reject)=>{

//executesomeasynchronouscode

if(errorIsDetected)returnreject(errorObject);

//Whentheprocessisfinishedcallthis:

resolve(result1,result2);

});

}

Ratherthanpassinginacallbackfunction,thecallerreceivesaPromiseobject.Whenproperlyutilized,theprecedingpyramidcanbecodedasfollows:

doThis(arg1,arg2)

.then(result=>{

//Thiscanreceiveonlyonevalue,henceto

//receivemultiplevaluesrequiresanobjectorarray

returndoThat(arg2,arg3);

})

.then((results)=>{

returndoSomethingElse(arg5);

})

.then(()=>{

//doafinalsomething

})

.catch(err=>{

//errorslandhere

});

ThisworksbecausethePromiseclasssupportschainingifathenfunctionreturnsaPromiseobject.

Theasync/awaitfeatureimplementsthepromiseofthePromiseclasstosimplifyasynchronouscoding.Thisfeaturebecomesactivewithinanasyncfunction:

asyncfunctionmumble(){

//asyncmagichappenshere

}

Anasyncarrowfunctionisasfollows:

constmumble=async()=>{

//asyncmagichappenshere

};

It'susedasso:

asyncfunctiondoSomething(arg1,arg2,arg3,arg4,arg5){

var{result1,result2}=awaitdoThis(arg1,arg2);

varresults=awaitdoThat(arg2,arg3);

awaitdoSomethingElse(arg5);

//doafinalsomething

returnfinalResult;

}

Isn'tthisabreathoffreshaircomparedtothenestedstructurewestartedwith?

TheawaitkeywordisusedwithaPromise.ItautomaticallywaitsforthePromisetoresolve.IfthePromiseresolvessuccessfullythenthevalueisreturned,andifitresolveswithanerrorthenthaterroristhrown.Bothhandlingresultsandthrowingerrorsarehandledinthenaturalmanner.

ThisexamplealsoshowsanotherES2015feature:destructuring.Thefieldsofanobjectcanbeextractedusingthefollowing:

var{value1,value2}={

value1:"Value1",value2:"Value2",value3:"Value3"

};

Wehaveanobjectwiththreefields,butextractonlytwoofthefields.

UsingBabeltouseexperimentalJavaScriptfeaturesTheBabeltranspiler(http://babeljs.io/)isagreatwaytousecutting-edgeJavaScriptfeaturesonolderimplementations.ThewordtranspilemeansBabelrewritesJavaScriptcodeintootherJavaScriptcode,specificallytorewriteES-2015orES-2016featurestoolderJavaScriptcode.BabelconvertsJavaScriptsourcetoanabstractsyntaxtree,thenmanipulatesthattreetorewritethecodeusingolderJavaScriptfeatures,andthenwritesthattreetoaJavaScriptsourcecodefile.

Putanotherway,BabelrewritesJavaScriptcodeintoJavaScriptcode,applyingdesiredtransformationssuchasconvertingES2015/2016featuresintoES5codethatcanruninawebbrowser.

ManyuseBabeltoexperimentwithnewJavaScriptfeatureproposalsworkingtheirwaythroughtheTC-39committee.OthersuseBabeltousenewJavaScriptfeaturesinprojectsonJavaScriptenginesthatdonotsupportthosefeatures.

TheNodeGreenwebsitemakesitclearthatNode.jssupportsprettymuchalloftheES2015/2016/2017features.Therefore,asapracticalmatter,wenolongerneedtouseBabelforNode.jsprojects.

Forwebbrowsers,thereisamuchlongertimelagbetweenasetofECMAScriptfeaturesandwhenwecanreliablyusethosefeaturesinbrowser-sidecode.It'snotthatthewebbrowsermakersareslowinadoptingnewfeatures,becausetheGoogle,Mozilla,andMicrosoftteamsareproactiveaboutadoptingthelatestfeatures.Apple'sSafariteamseemsslowtoadoptnewfeatures,unfortunately.What'sslower,however,isthe

penetrationofnewbrowsersintothefleetofcomputersinthefield.

Therefore,modernJavaScriptprogrammersneedtofamiliarizethemselveswithBabel.

We'renotreadytoshowexamplecodeforthesefeatures,butwecangoaheadanddocumentthesetupoftheBabeltool.Forfurtherinformationonsetupdocumentation,visithttp://babeljs.io/docs/setup/,andthenclickontheCLIbutton.

TogetabriefintroductiontoBabel,we'lluseittotranspilethescriptswesawearliertorunonNode.js6.x.Inthosescriptsweusedasyncfunctions,whicharenotsupportedinNode.js6.x.

Inthedirectorycontainingls.jsandls2.js,typethesecommands:

$npminstallbabel-cli\

babel-plugin-transform-es2015-modules-commonjs\

babel-plugin-transform-async-to-generator

ThisinstallstheBabelsoftware,alongwithacoupleoftransformationplugins.Babelhasapluginsystemsothatyouenablethetransformationsrequiredbyyourproject.OurprimarygoalinthisexampleisconvertingtheasyncfunctionsshownearlierintoGeneratorfunctions.GeneratorsareanewsortoffunctionintroducedwithES2015,whichformthefoundationforimplementationofasyncfunctions.

BecauseNode.js6.xdoesnothaveutil.promisify,weneedtomakeonesubstitution:

//constfs_readdir=util.promisify(fs.readdir);

constfs_readdir=dir=>{

returnnewPromise((resolve,reject)=>{

fs.readdir(dir,(err,fileList)=>{

if(err)reject(err);

elseresolve(fileList);

});

});

};

Thisstructureismoreorlesswhattheutil.promisifyfunctiondoes.

Next,createafilenamed.babelrccontainingthefollowing:

{

"plugins":[

"transform-es2015-modules-commonjs",

"transform-async-to-generator"

]

}

ThisfileinstructsBabeltousethenamedtransformationpluginsthatweinstalledearlier.

Becauseweinstalledbabel-cli,ababelcommandisinstalledsuchthatwecantypethefollowing:

$./node_modules/.bin/babel-help

Totranspileyourcode,runthefollowingcommand:

$./node_modules/.bin/babells2.js-ols2-babel.js

Thiscommandtranspilesthenamedfile,producinganewfile.Thenewfileisasfollows:

'usestrict';

function_asyncToGenerator(fn){returnfunction(){vargen=

fn.apply(this,arguments);returnnewPromise(function(resolve,reject){

functionstep(key,arg){try{varinfo=gen[key](arg);varvalue=

info.value;}catch(error){reject(error);return;}if(info.done){

resolve(value);}else{returnPromise.resolve(value).then(function(value)

{step("next",value);},function(err){step("throw",err);});}}return

step("next");});};}

constfs=require('fs');

constutil=require('util');

//constfs_readdir=util.promisify(fs.readdir);

constfs_readdir=dir=>{

returnnewPromise((resolve,reject)=>{

fs.readdir(dir,(err,fileList)=>{

if(err)reject(err);

elseresolve(fileList);

});

});

};

_asyncToGenerator(function*(){

vardir='.';

if(process.argv[2])dir=process.argv[2];

constfiles=yieldfs_readdir(dir);

for(letfnoffiles){

console.log(fn);

}

})().catch(err=>{

console.error(err);

});

Thiscodeisn'tmeanttobeeasytoreadbyhumans.Instead,it'smeantthatyouedittheoriginalsourcefile,andthenconvertitforyourtargetJavaScriptengine.ThemainthingtonoticeisthatthetranspiledcodeusesaGeneratorfunctioninplaceoftheasyncfunction,andtheyieldkeywordinplaceoftheawaitkeyword.The_asyncToGeneratorfunctionimplementsfunctionalitysimilartoasyncfunctions.

Thetranspiledscriptisrunasfollows:

$nodels2-babel

.babelrc

app.js

babel

ls.js

ls2-babel.js

ls2.js

node_modules

Inotherwords,itrunsthesameastheasyncversion,butonanolderNode.jsrelease.

SummaryYoulearnedalotinthischapteraboutinstallingNode.js,usingitscommand-linetools,andrunningaNode.jsserver.Wealsobreezedpastalotofdetailsthatwillbecoveredlaterinthebook,sobepatient.

Specifically,wecovereddownloadingandcompilingtheNode.jssourcecode,installingNode.jseitherfordevelopmentuseinyourhomedirectoryorfordeploymentinsystemdirectoriesandinstallingNPM—thedefactostandardpackagemanagerusedwithNode.js.WealsosawhowtorunNode.jsscriptsorNode.jsservers.WethentookalookatthenewfeaturesinES-2015/2016/2017.Finally,wesawhowtouseBabeltoimplementthosefeaturesinyourcode.

Nowthatwe'veseenhowtosetupthebasicsystem,we'rereadytostartworkingonimplementingapplicationswithNode.js.First,youmustlearnthebasicbuildingblocksofNode.jsapplicationsandmodules,whichwewillcoverinthenextchapter.

Node.jsModulesBeforewritingNode.jsapplications,youmustlearnaboutNode.jsmodulesandpackages.Modulesandpackagesarethebuildingblocksforbreakingdownyourapplicationintosmallerpieces.

Inthischapter,wewillcoverthefollowingtopics:

Definingamodule

TheCommonJSandES2015modulespecifications

UsingES2015/2016/2017codingpracticesinNode.js

UsingtheES6moduleformatinNode.jscode

UnderstandinghowNode.jsfindsmodules

Thenpmpackagemanagementsystem

So,let'sgetonwithit.

DefiningamoduleModulesarethebasicbuildingblocksforconstructingNode.jsapplications.ANode.jsmoduleencapsulatesfunctions,hidingdetailsinsideawell-protectedcontainer,andexposinganexplicitly-declaredlistoffunctions.

Therearetwomoduleformatsthatwemustconsider:

ThetraditionalNode.jsformatbasedontheCommonJSstandardhasbeenusedsinceNode.jswascreated.

WithES2015/2016anewformat,ES6Modules,hasbeendefinedwithanewimportkeyword.ES6moduleswillbe(oris)supportedinallJavaScriptimplementations.

BecauseES6modulesarenowthestandardmoduleformat,theNode.jsTechnicalSteeringCommittee(TSC)iscommittedtofirst-classsupportforES6modules.

Wehavealreadyseenmodulesinactioninthepreviouschapter.EveryJavaScriptfileweuseinNode.jsisitselfamodule.It'stimetoseewhattheyareandhowtheywork.We'llstartwithCommonJSmodulesandthenquicklybringinES6modules.

Inthels.jsexampleinChapter2,SettingupNode.js,wewrotethefollowingcodetopullinthefsmodule,givingusaccesstoitsfunctions:

constfs=require('fs');

Therequirefunctionsearchesforthenamedmodule,loadingthemodule

definitionintotheNode.jsruntime,andmakingitsfunctionsavailable.Inthiscase,thefsobjectcontainsthecode(anddata)exportedbythefsmodule.ThefsmoduleispartoftheNode.jscoreandprovidesfilesystemfunctions.

Bydeclaringfsasconst,wehavealittlebitofassuranceagainstmakingcodingmistakesthatwouldmodifytheobjectholdingthemodulereference.

IneveryNode.jsmodule,theexportsobjectwithinthemoduleistheinterfaceexportedtoothercode.Anythingassignedtoafieldoftheexportsobjectisavailabletootherpiecesofcode,andeverythingelseishidden.Bytheway,thisobjectisactuallymodule.exports.Theexportsobjectisanaliasformodule.exports.

Therequirefunctionandmodule.exportsobjectsbothcomefromtheCommonJSspecification.ES6moduleshavesimilarconcepts,butadifferentimplementation.

Let'slookatabriefexampleofthisbeforedivingintothedetails.Ponderoverthesimple.jsmodule:

varcount=0;

exports.next=function(){return++count;};

exports.hello=function(){

return"Hello,world!";

};

Wehaveonevariable,count,whichisnotattachedtotheexportsobject,andafunction,next,whichisattached.Now,let'suseit:

$node

>consts=require('./simple');

undefined

>s.hello();

'Hello,world!'

>s.next();

1

>s.next();

2

>s.next();

3

>console.log(s.count);

undefined

undefined

>

Theexportsobjectinthemoduleistheobjectthatisreturnedbyrequire('./simple').Therefore,eachcalltos.nextcallsthenextfunctioninsimple.js.Eachreturns(andincrements)thevalueofthelocalvariable,count.Anattempttoaccesstheprivatefield,count,showsit'sunavailablefromoutsidethemodule.

Toreiteratetherule:

Anything(functionsorobjects)assignedasafieldofexports(asknownasmodule.exports)isavailabletoothercodeoutsidethemodule

Objectsnotassignedtoexportsarenotavailabletocodeoutsidethemodule,unlessthemoduleexportsthoseobjectsviaanothermechanism

ThisishowNode.jssolvestheglobalobjectproblemofbrowser-basedJavaScript.Thevariablesthatlooklikethey'reglobalvariablesareonlyglobaltothemodulecontainingthatvariable.Thesevariablesarenotvisibletoanyothercode.

Nowthatwe'vegotatasteformodules,let'stakeadeeperlook.

CommonJSandES2015moduleformatsNode.js'smoduleimplementationisstronglyinspiredby,butnotidenticalto,theCommonJSmodulespecification.ThedifferencesbetweenthemmightonlybeimportantifyouneedtosharecodebetweenNodeandotherCommonJSsystems.

AmongthechangesinES2015isastandardmoduleformatmeantforuseeverywhere.Ithassomeinterestingfeatures,andbyexistingeverywhereitshouldadvancethestateofJavaScript.SinceitisincompatiblewiththeCommonJS/Node.jsmodulesystem,adoptingES2015modulesinNode.jsmeansreworkingourpracticesandacceptednorms.

Asapracticalmatter,Node.jsprogrammerswillbedealingwithbothmoduleformatsforsometimeduringatransitionperiod.Ourlong-termgoalshouldbetoadoptES2015modulesacrosstheboard.TheNode.jsplatformisslatedtobringinsupportforES2015modulesinNode.js10.AsofNode.js8.5,thefeatureisavailablebysettingacommand-lineflag.

CommonJS/Node.jsmoduleformatWe'vealreadyseenacoupleofexamplesofthismoduleformat,withthesimple.jsexample,andtheprogramsweexaminedinChapter2,SettingupNode.js.Solet'stakeacloserlook.

CommonJSmodulesarestoredinfileswiththeextension.js.

LoadingaCommonJSmoduleisasynchronousoperation.Thatmeansthatwhentherequire('modulename')functioncallreturns,themodulehasbeenlocatedandcompletelyreadintomemoryandisreadytogo.Themoduleiscachedinmemorysothatsubsequentrequire('modulename')callsreturnimmediately,andallreturntheexactsameobject.

Node.jsmodulesprovideasimpleencapsulationmechanismtohideimplementationdetailswhileexposinganAPI.Modulesaretreatedasiftheywerewrittenasfollows:

(function(){...contentsofmodulefile...})();

Thus,everythingwithinthemoduleiscontainedwithinananonymousprivatenamespacecontext.Thisishowtheglobalobjectproblemisresolved;everythinginamodulethatlooksglobalisactuallycontainedwithinthisprivatecontext.

ObjectsandfunctionscanbeexposedfromaCommonJSmodulebymeansoftwofreevariablesNode.jsinsertsintothisprivatecontext:moduleandexports:

Themoduleobjectcontainsseveralfieldsthatyoumightfinduseful.

RefertotheonlineNode.jsdocumentationfordetails.

Theexportsobjectisanaliasofthemodule.exportsfield.Thismeansthatthefollowingtwolinesofcodeareequivalent:

exports.funcName=function(arg,arg1){...};

module.exports.funcName=function(arg,arg2){..};

Yourcodecanbreakthealiasbetweenthetwoifyoudothis:

exports=function(arg,arg1){...};

Donotdothat,becauseexportswillnolongerbeequivalenttomodule.exports.Ifyourintentistoassignasingleobjectorfunctiontobewhat'sreturnedbyrequire,dothisinstead:

module.exports=function(arg,arg1){...};

Somemodulesdoexportasinglefunctionbecausethat'showthemoduleauthorenvisioneddeliveringthedesiredfunctionality.

TheNode.jspackageformatisderivedfromtheCommonJSmodulesystem(http://commonjs.org).Whendeveloped,theCommonJSteamaimedtofillagapintheJavaScriptecosystem.Atthattime,therewasnostandardmodulesystem,makingittrickiertopackageJavaScriptapplications.Therequirefunction,theexportsobject,andotheraspectsofNode.jsmodulescomedirectlyfromtheCommonJSModules/1.0spec.

ES6moduleformatES6modulesareanewmoduleformatdesignedforallJavaScriptenvironments.WhileNode.jshashadagoodmodulesystemforitswholeexistence,browser-sideJavaScripthasnot.Thatleftthebrowser-sidecommunitywitheitherrelyingonthe<script>tag,orusingnon-standardizedsolutions.Forthatmatter,traditionalNode.jsmoduleswereneverstandardized,outsideoftheCommonJSeffort.Therefore,ES6modulesstandtobeabigimprovementfortheentireJavaScriptworld,bygettingeveryoneonthesamepagewithacommonmoduleformatandmechanisms.

ThesideeffectisthattheNode.jscommunityneedstostartlookingat,learningabout,andadoptingtheES2015moduleformat.

ES6modulesarereferredtobyNode.jswiththeextension.mjs.Whenitcametoimplementingthenewmoduleformat,theNode.jsteamdeterminedthattheycouldnotsupportbothCommonJSandES6moduleswiththe.jsextension.The.mjsextensionwasdecidedasthesolution,andyoumayseetongue-in-cheekreferencestoMichaelJacksonScriptforthisfileextension.

OneinterestingdetailisthatES6modulesloadasynchronously.ThismaynothaveanimpactonNode.jsprogrammers,exceptthatthisispartoftherationalebehindrequiringthenew.mjsextension.

Createafilenamedsimple2.mjsinthesamedirectoryasthesimple.jsexamplethatwelookedatearlier:

varcount=0;

exportfunctionnext(){return++count;}

functionsquared(){returnMath.pow(count,2);}

exportfunctionhello(){

return"Hello,world!";

}

exportdefaultfunction(){returncount;}

exportconstmeaning=42;

exportletnocount=-1;

export{squared};

ES6itemsexportedfromamodulearedeclaredwiththeexportkeyword.Thiskeywordcanbeputinfrontofanytop-leveldeclaration,suchasvariable,function,orclassdeclarations:

exportfunctionnext(){..}

Theeffectofthisissimilartothefollowing:

module.exports.next=function(){..}

Theintentofbothisessentiallythesame:tomakeafunction,orotherobject,availabletocodeoutsidethemodule.Astatementsuchasexportfunctionnext()isanamedexport,meaningtheexportedthinghasaname,andthatcodeoutsidethemoduleusesthatnametoaccesstheobject.Asweseehere,namedexportscanbefunctionsorobjects,andtheymayalsobeclassdefinitions.

Usingexportdefaultcanbedoneoncepermodule,andisthedefaultexportfromthemodule.Thedefaultexportiswhatcodeoutsidethemoduleaccesseswhenusingthemoduleobjectitself,ratherthanwhenusingoneoftheexportsfromthemodule.

Youcanalsodeclaresomething,suchasthesquaredfunction,andthenexportitlater.

Nowlet'sseehowtousethisES2015module.Createasimpledemo.mjsfilewiththefollowing:

import*assimple2from'./simple2.mjs';

console.log(simple2.hello());

console.log(`${simple2.next()}${simple2.squared()}`);

console.log(`${simple2.next()}${simple2.squared()}`);

console.log(`${simple2.default()}${simple2.squared()}`);

console.log(`${simple2.next()}${simple2.squared()}`);

console.log(`${simple2.next()}${simple2.squared()}`);

console.log(`${simple2.next()}${simple2.squared()}`);

console.log(simple2.meaning);

Theimportstatementdoeswhatitmeans:itimportsobjectsexportedfromamodule.ThisversionoftheimportstatementismostsimilartoatraditionalNode.jsrequirestatement,meaningthatitcreatesanobjectthroughwhichyouaccesstheobjectsexportedfromthemodule.

Thisishowthecodeexecutes:

$node--experimental-modulessimpledemo.mjs

(node:63937)ExperimentalWarning:TheESMmoduleloaderisexperimental.

Hello,world!

11

24

24

39

416

525

42

AsofNode.js8.5,thenewmoduleformatisavailablebehindanoptionflagasshownhere.You'realsopresentedwiththisnicewarningthatit'sanexperimentalfeature.Accessingthedefaultexportisaccomplishedbyaccessingthefieldnameddefault.Accessinganexportedvalue,suchasthemeaningfield,isdonewithoutparenthesesbecauseitisavalueandnotafunction.

Nowtoseeadifferentwaytoimportobjectsfromamodule,createanotherfile,namedsimpledemo2.mjs,containingthefollowing:

import{

defaultassimple,hello,next

}from'./simple2.mjs';

console.log(hello());

console.log(next());

console.log(next());

console.log(simple());

console.log(next());

console.log(next());

console.log(next());

Inthiscase,eachimportedobjectisitsownthingratherthanbeingattachedtoanotherobject.Insteadofwritingsimple2.next(),yousimplywritenext().Theasclauseisawaytodeclareanalias,ifnothingelsesoyoucanusethedefaultexport.Wealreadyusedanasclauseearlier,anditcanbeusedinotherinstanceswhereyouwishtoprovideanaliasforthevaluebeingexportedorimported.

Node.jsmodulescanbeusedfromES2015.mjscode.Createafilenamedls.mjs,containingthefollowing:

import_fsfrom'fs';

constfs=_fs.promises;

importutilfrom'util';

(async()=>{

constfiles=awaitfs.readdir('.');

for(letfnoffiles){

console.log(fn);

}

})().catch(err=>{console.error(err);});

Youcannot,however,requireanES2015moduleintoregularNode.jscode.ThelookupalgorithmforES2015modulesisdifferent,andaswementionedearlier,ES2015modulesareloadedasynchronously.

Anotherwrinkleishandlingthefs.promisessubmodule.Weareusingthatsubmoduleintheexample,buthow?Thisimportstatementdoesnotwork:

import{promisesasfs}from'fs';

Thisfailsasso:

$node--experimental-modulesls.mjs

(node:45186)ExperimentalWarning:TheESMmoduleloaderisexperimental.

file:///Volumes/Extra/book-4th/chap03/ls.mjs:1

import{promisesasfs}from'fs';

^^^^^^^^

SyntaxError:Therequestedmodule'fs'doesnotprovideanexportnamed

'promises'

atModuleJob._instantiate(internal/modules/esm/module_job.js:89:21)

Thatleavesuswiththisconstruct:

import_fsfrom'fs';

constfs=_fs.promises;

Executingthescriptgivesthefollowing:

$node--experimental-modulesls.mjs

(node:65359)ExperimentalWarning:TheESMmoduleloaderisexperimental.

(node:37671)ExperimentalWarning:Thefs.promisesAPIisexperimental

ls.mjs

module1.js

module2.js

simple.js

simple2.mjs

simpledemo.mjs

simpledemo2.mjs

ThelastthingtonoteaboutES2015modulecodeisthatimportandexportstatementsmustbetop-levelcode.Evenputtinganexportinsideasimpleblocklikethis:

{

exportconstmeaning=42;

}

Resultsinanerror:

$node--experimental-modulesbadexport.mjs

(node:67984)ExperimentalWarning:TheESMmoduleloaderisexperimental.

SyntaxError:Unexpectedtokenexport

atModuleJob.loaders.set[asmoduleProvider]

(internal/loader/ModuleRequest.js:32:13)

at<anonymous>

WhilethereareafewmoredetailsaboutES2015modules,thesearetheirmostimportantattributes.

JSONmodulesNode.jssupportsusingrequire('pathto/file-name.json')toimportaJSONfile.Itisequivalenttothiscode:

constfs=require('fs');

module.exports=JSON.parse(

fs.readFileSync('pathto/file-name.json','utf8'));

Thatis,theJSONfileisreadsynchronously,andthetextisparsedasJSON.Theresultantobjectisavailableastheobjectexportedfromthemodule.Createafilenameddata.json,containingthefollowing:{"hello":"Hello,world!","meaning":42}

Nowcreateafilenamedshowdata.js,containingthefollowing:

constutil=require('util');

constdata=require('./data');

console.log(util.inspect(data));

Itwillexecuteasfollows:

$nodeshowdata.js

{hello:'Hello,world!',meaning:42}

Theutil.inspectfunctionisausefulwaytopresentanobjectinaneasy-to-readfashion.

SupportingES6modulesonolderNode.jsversionsWhilesupportforES6modulesarrivedasanexperimentalfeatureinNode.js8.5,therearetwowaystousethesemodulesonearlierNode.jsimplementations.

OnemethodistousetheBabeltranspilertorewriteES6codesoitcanexecuteonolderNode.jsversions.Foranexample,seehttps://blog.revillweb.com/using-es2015-es6-modules-with-babel-6-3ffc0870095b.

ThebettermethodistheesmpackageintheNode.jsregistry.Simplydothefollowing:

$nvminstall6

Downloadingandinstallingnodev6.14.1...

Downloadinghttps://nodejs.org/dist/v6.14.1/node-v6.14.1-darwin-x64.tar.xz...

########################################################################

100.0%

Computingchecksumwithshasum-a256

Checksumsmatched!

Nowusingnodev6.14.1(npmv3.10.10)

$nvmuse6

Nowusingnodev6.14.1(npmv3.10.10)

$npminstallesm

...npmoutput

$node--requireesmsimpledemo.mjs

Hello,world!

11

24

24

39

416

525

42

Tousethismodule,onesimplyinvokesrequire('esm')once,andES6

modulesareretrofittedintoNode.js.The--requireflagautomaticallyloadsthenamedmodule.Withoutrewritingthecode,wecanselectivelyusetheesmmodulewiththisthecommand-lineoption.

ThisexampledemonstratesretrofittingES6modulesintoolderNode.jsreleases.Tosuccessfullyexecutethels.mjsexamplewemusthavesupportforasync/awaitfunctions,andarrowfunctions.SinceNode.js6.xdoesnotsupporteither,thels.mjsexamplewillfail,andwillnecessitaterewritingsuchcode:

$node--version

v6.14.1

$node-resmls.mjs

UsersDavid/chap03/ls.mjs:5

(async()=>{

^

SyntaxError:Unexpectedtoken(

atexports.runInThisContext(vm.js:53:16)

atModule._compile(module.js:373:25)

Formoreinformation,see:https://medium.com/web-on-the-edge/es-modules-in-node-today-32cff914e4b.Thatarticledescribesanolderreleaseoftheesmmodule,atthetimenamed@std/esm.

Demonstratingmodule-levelencapsulationAkeyattributeofmodulesisencapsulation.Theobjectsthatarenotexportedfromthemoduleareprivatetothemodule,andcannotbeaccessedfromcodeoutsidethemodule.Toreiterate,modulesaretreatedasiftheywerewrittenasfollows:

(function(){...contentsofmodulefile...})();

ThisJavaScriptidiomdefinesananonymousprivatescope.Anythingdeclaredwithinthatscopecannotbeaccessedbycodeoutsidethescope.Thatis,unlesssomecodemakesobjectreferencesavailabletoothercodeoutsidethisprivatescope.That'swhatthemodule.exportsobjectdoes:itisamechanismforthemoduleauthortoexposeobjectreferencesfromthemodule.Othercodecanthenaccessresourcesinsidethemoduleinacontrolledfashion.

Thetop-levelvariablesinsideamodulelookliketheyexistintheglobalscope.InsteadofbeingtrulyGlobal,they'resafelyprivatetothemoduleandarecompletelyinaccessibletoothercode.

Let'stakealookatapracticaldemonstrationofthatencapsulation.Createafilenamedmodule1.js,containingthefollowing:

constA="valueA";

constB="valueB";

exports.values=function(){

return{A:A,B:B};

}

Then,createafilenamedmodule2.js,containingthefollowing:

constutil=require('util');

constA="adifferentvalueA";

constB="adifferentvalueB";

constm1=require('./module1');

console.log(`A=${A}B=${B}values=${util.inspect(m1.values())}`);

console.log(`${m1.A}${m1.B}`);

constvals=m1.values();

vals.B="somethingcompletelydifferent";

console.log(util.inspect(vals));

console.log(util.inspect(m1.values()));

Then,runitasfollows(youmusthaveNode.jsalreadyinstalled):

$nodemodule2.js

A=adifferentvalueAB=adifferentvalueBvalues={A:'valueA',B:'value

B'}

undefinedundefined

{A:'valueA',B:'somethingcompletelydifferent'}

{A:'valueA',B:'valueB'}

Thisartificialexampledemonstratesencapsulationofthevaluesinmodule1.jsfromthoseinmodule2.js.TheAandBvaluesinmodule1.jsdon'toverwriteAandBinmodule2.jsbecausethey'reencapsulatedwithinmodule1.js.Thevaluesfunctioninmodule1.jsdoesallowcodeinmodule2.jsaccesstothevalues;however,module2.jscannotdirectlyaccessthosevalues.Wecanmodifytheobjectmodule2.jsreceivedfrommodule1.js.Butdoingsodoesnotchangethevalueswithinmodule1.js.

FindingandloadingCommonJSandJSONmodulesusingrequireWehavetalkedaboutseveraltypesofmodules:CommonJS,JSON,ES2015,andnativecodemodules.AllbuttheES2015modulesareloadedusingtherequirefunction.Thatfunctionhasaverypowerfulandflexiblealgorithmforlocatingmoduleswithinadirectoryhierarchy.Thisalgorithm,coupledwiththenpmpackagemanagementsystem,givestheNode.jsplatformalotofpowerandflexibility.

FilemodulesTheCommonJSandES2015moduleswe'vejustlookedatarewhattheNode.jsdocumentationdescribesasafilemodule.Suchmodulesarecontainedwithinasinglefile,whosefilenameendswith.js,.mjs,.json,or.node.ThelatterarecompiledfromCorC++sourcecode,orevenotherlanguagessuchasRust,whiletheformerareofcoursewritteninJavaScriptorJSON.

We'vealreadylookedatseveralexamplesofusingthesemodules,aswellasthedifferencebetweentheCommonJSformattraditionallyusedinNode.js,andthenewES2015modulesthatarenowsupported.

ModulesbakedintoNode.jsbinarySomemodulesarepre-compiledintotheNode.jsbinary.ThesearethecoreNode.jsmodulesdocumentedontheNode.jswebsiteathttps://nodejs.org/api/index.html.

TheystartoutassourcecodewithintheNode.jsbuildtree.Thebuildprocesscompilesthemintothebinarysothatthemodulesarealwaysavailable.

DirectoriesasmodulesAmodulecancontainawholedirectorystructurefullofstuff.Stuffhereisatechnicaltermreferringtointernalfilemodules,datafiles,templatefiles,documentation,tests,assets,andmore.Oncestoredwithinaproperlyconstructeddirectorystructure,Node.jswilltreattheseasamodulethatsatisfiesarequire('moduleName')call.

Thismaybealittleconfusingbecausethewordmoduleisbeingoverloadedwithtwomeanings.Insomecases,amoduleisafile,andinothercases,amoduleisadirectorycontainingoneormorefilemodules.

Inmostcases,adirectory-as-modulecontainsapackage.jsonfile.Thisfilecontainsdataaboutthemodule(knownaspackage)thatNode.jsuseswhileloadingthemodule.TheNode.jsruntimerecognizesthesetwofields:{name:"myAwesomeLibrary",main:"./lib/awesome.js"}

Ifthispackage.jsonfileisinadirectorynamedawesomelib,thenrequire('./awesomelib')willloadthefilemodulein./awesomelib/lib/awesome.js.

Ifthereisnopackage.json,thenNode.jswilllookforeitherindex.jsorindex.node.Insuchacase,require('./awesomelib')willloadthefilemodulein./awesomelib/index.js.

Ineithercase,thedirectorymodulecaneasilycontainotherfilemodules.Themodulethat'sinitiallyloadedwouldsimplyuserequire('./anotherModule')oneormoretimestoloadother,privatemodules.

Thenpmpackagemanagementsystemcanrecognizealotmoredatainthepackage.jsonfile.Thatincludesthepackagename,itsauthor,thehomepageURL,theissue-queueURL,packagedependencies,andmore.We'llgooverthislater.

ModuleidentifiersandpathnamesGenerallyspeaking,themodulenameisapathname,butwiththefileextensionremoved.Earlier,whenwewroterequire('./simple'),Node.jsknewtoadd.jstothefilenameandloadinsimple.js.Similarly,Node.jswouldrecognizesimple.jsonorsimple.nodeasthefilenamelegitimatelysatisfyingrequire('./simple').

Therearethreetypesofmoduleidentifiers:relative,absolute,andtop-level:

Relativemoduleidentifiers:Thesebeginwith./or../andabsoluteidentifiersbeginwith/.ThemodulenameisidenticalwithPOSIXfilesystemsemantics.Theresultantpathnameisinterpretedrelativetothelocationofthefilebeingexecuted.Thatis,amoduleidentifierbeginningwith./islookedforinthecurrentdirectory,whereasonestartingwith../islookedforintheparentdirectory.

Absolutemoduleidentifiers:Thesebeginwith/andare,ofcourse,lookedforintherootofthefilesystem,butthisisnotarecommendedpractice.

Top-levelmoduleidentifiers:Thesebeginwithnoneofthosestringsandarejustthemodulename,orelsemodule-name/path/to/module.Thesemustbestoredinanode_modulesdirectory,andtheNode.jsruntimehasanicelyflexiblealgorithmforlocatingthecorrectnode_modulesdirectory:

Inthecaseofmodule-name/path/to/modulespecifiers,whatwillbeloadedisamodulepath/to/modulewithinthetop-levelmodulenamedmodule-name

Thebaked-inmodulesarespecifiedusingtop-levelmodulenames

Thesearchbeginsinthedirectorycontainingthefilecallingrequire().Ifthatdirectorycontainsanode_modulesdirectory,whichthencontainseitheramatchingdirectorymoduleoramatchingfilemodule,thenthesearchissatisfied.Ifthelocalnode_modulesdirectorydoesnotcontainasuitablemodule,ittriesagainintheparentdirectory,anditwillcontinueupwardinthefilesystemuntiliteitherfindsasuitablemoduleoritreachestherootdirectory.

Thatis,witharequirecallinhomedavid/projects/notes/foo.js,thefollowingdirectorieswillbeconsulted:

homedavid/projects/notes/node_modules

homedavid/projects/node_modules

homedavid/node_modules

homenode_modules

/node_modules

Ifthemoduleisnotfoundthroughthissearch,thereareglobalfoldersinwhichmodulescanbelocated.ThefirstisspecifiedintheNODE_PATHenvironmentvariable.Thisisinterpretedasacolon-delimitedlistofabsolutepathssimilartothePATHenvironmentvariable.OnWindows,theelementsofNODE_PATHareofcourseseparatedbysemicolons.Node.jswillsearchthosedirectoriesforamatchingmodule.

TheNODE_PATHapproachisnotrecommended,becauseofsurprisingbehaviorwhichcanhappenifpeopleareunawarethatthisvariablemustbeset.IfaspecificmodulelocatedinaspecificdirectoryreferencedinNODE_PATHisrequiredforproperfunction,andthevariableisnotset,theapplicationwilllikelyfail.AstheTwelve-FactorApplicationmodelsuggests,itisbestforalldependenciestobeexplicitlydeclared,andwithNode.jsthatmeanslistingalldependenciesinthepackage.jsonsothatnpmoryarncanmanagethedependencies.

Thisvariablewasimplementedbeforethemoduleresolutionalgorithmjustdescribedwasfinalized.Becauseofthatalgorithm,NODE_PATHislargelyunnecessary.

Therearethreeadditionallocationsthatcanholdmodules:

$HOME/.node_modules

$HOME/.node_libraries

$PREFIX/lib/node

Inthiscase,$HOMEiswhatyouexpect,theuser'shomedirectory,and$PREFIXisthedirectorywhereNode.jsisinstalled.

Somearebeginningtorecommendagainstusingglobalmodules.Therationaleisthedesireforrepeatabilityanddeployability.Ifyou'vetestedanapp,andallitscodeisconvenientlylocatedwithinadirectorytree,youcancopythattreefordeploymenttoothermachines.But,whatiftheappdependedonsomeotherfilethatwasmagicallyinstalledelsewhereonthesystem?Willyouremembertodeploysuchfiles?

AnexampleofapplicationdirectorystructureLet'stakealookatthefilesystemstructureofatypicalNode.jsExpressapplication:

ThisisanExpressapplication(we'llstartusingExpressinChapter5,YourFirstExpressApplication)containingafewmodulesinstalledinthenode_modulesdirectory.Oneofthose,Express,hasitsownnode_modulesdirectorycontainingacoupleofmodules.

Forapp.jstoloadmodels-sequelize/notes.js,itusesthefollowingrequirecall:

constnotesModel=require('./models-sequelize/notes');

Thisisarelativemoduleidentifier,wherethepathnameisresolvedrelativetothedirectorycontainingthefilemakingthereference.

Usethefollowingcodetodothereverseinmodels-sequelize/notes.js:

constapp=require('../app');

Again,thisisarelativemoduleidentifier,thistimeresolvedrelativetothesubdirectorycontainingmodels-sequelize/notes.js.

Anyreferencetoatop-levelmoduleidentifierwillfirstlookinthenode_modulesdirectoryshownhere.Thisdirectoryispopulatedfromthedependencieslistedinthepackage.json,aswe'llseeinafewpages:

constexpress=require('express');

constfavicon=require('serve-favicon');

constlogger=require('morgan');

constcookieParser=require('cookie-parser');

constbodyParser=require('body-parser');

AllofthesearetypicalmodulesincludedinanExpressapplication.Mostofthemarereadilyvisibleinthescreenshotshownearlier.What'sloadedisthemainfileinthecorrespondingsubdirectoryofnode_modules,forexample,node_modules/express/index.js.

ButtheapplicationcannotdirectlyreferencethedependenciesoftheExpressmodulethatareinitsinternalnode_modulesdirectory.Themodulesearchalgorithmonlymovesupwardinthefilesystem;itdoesnotdescendintosubsidiarydirectorytrees.

Onesideeffectoftheupwardsearchdirectionisthehandlingofconflictingdependencies.

Supposetwomodules(modulesAandB)listedadependencyonthesamemodule(C)?Inthenormalcase,thetwodependenciesonmoduleCcouldbehandledbythesameinstanceofthatmodule.Aswe'llseeinafewpages,npm'sdependencylistinpackage.jsoncanuselooseorpreciseversionnumberreferences.DependingonthecurrentversionnumberformoduleC,modulesAandBmay,ormaynot,beinagreementastowhichversiontouse.Iftheydonotagree,npmcanarrangethemoduleinstallationsuchthatbothmoduleAandBgettheversionofmoduleCtheydependon,withouteithersteppingontheother.IfbothareagreeablewiththesamemoduleCinstance,onlyonecopywillbeinstalled,butiftheydisagreethennpmwillinstalltwocopies.ThetwocopieswillbelocatedsuchthatthemodulesearchalgorithmwillcauseeachmoduletofindthecorrectversionofmoduleC.

Let'stryaconcreteexampletoclarifywhatwasjustsaid.Inthescreenshotearlier,youseetwoinstancesofthecookiemodule.Wecanusenpmtoqueryforallreferencestothismodule:

$npmlscookie

notes@0.0.0UsersDavid/chap05/notes

├─┬cookie-parser@1.3.5

│└──cookie@0.1.3

└─┬express@4.13.4

└──cookie@0.1.5

Thissaysthecookie-parsermoduledependsonversion0.1.3ofcookie,whileExpressdependsonversion0.1.5.Howdoesnpmavoidproblemswiththesetwoconflictingversions?Byputtingoneinsidethenode_modulesdirectoryinsidetheexpressmodule.Thisway,whenExpressreferstothismodule,itwillusethe0.1.5instanceinitsownnode_modulesdirectory,whilethecookie-parsermodulewillusethe0.1.3instanceinthetop-levelnode_modulesdirectory.

FindingandloadingES6modulesusingimportTheimportstatementisusedtoloadES6modules,anditonlyworksinsideanES6module.BecauseES6modulesareloadedasynchronously,therequire()statementcannotloadES6modules.Aswesaidearlier,ES6modulesarerecognizedbyNode.jsbythe.mjsextension.TheECMAScriptTC-39committeehas(orplansto)officiallyregisterthatfileextensionwiththerecognizedauthoritiessothatregulartoolswillrecognizebothfileextensionsasJavaScript.

ThemodulespecifieronehandstotheimportstatementisinterpretedasaURL.Forthetimebeing,Node.jswillonlyacceptfile:URLbecauseofthesecurityimplicationsofloadingmodulesovertheInternet.Becauseit'saURL,somecharacterssuchas:,?,#,or%mustreceivespecialtreatment.Forexample:

import'./foo?search';

import'./foo#hash';

Thesearevalidmodulespecifierswhere?searchand#hashhavethesortofmeaningyou'dexpectinaURL.SolongasNode.jsonlysupportsfile:URLforimportstatements,wecannotmakeuseofthatfeature,butwehavetokeepitinmindandavoidusingthesestringsinmoduleURL.

OnecaninstallcustommoduleloaderhooksthatcouldconceivablyusethoseURLpartsforsomepurpose.

Themodulesearchalgorithmissimilartowhatwedescribedforrequire.Ifthespecifierbeginswith./,../,or/,thespecifierisinterpretedasapathname.Otherwise,itisinterpretedasatop-levelmodulesimilartotherequirestatement,withonebigdifference.Theimportstatementwillnot

searchforaglobalmodule.Thisisfrownedon,butifonemustuseaglobalmodule,thatcanbeaccomplishedwithasymboliclink.

Fordocumentation,seehttps://nodejs.org/api/esm.html.

HybridCommonJS/Node.js/ES6modulescenariosWe'vegoneovertheformatforCommonJS/Node.jsmodules,theformatforES6modules,andthealgorithmforlocatingandimportingboth.Thelastthingtocoveristhosehybridsituationswhereourcodewillusebothmoduleformatsatthesametime.

Asapracticalmatter,ES6modulesareverynewtotheNode.jsplatform,andthereforewehavealargebodyofexistingcodewrittenasCommonJS/Node.jsmodules.ManytoolsintheNode.jsmarkethaveimplementationdependenciesontheCommonJSformat.Thismeanswe'llbefacingsituationswhereES6moduleswillneedtouseCommonJSmodules,andviceversa:

CommonJSmoduleloadsotherCommonJSmoduleswithrequire()

CommonJSmodulecannotloadES6modules—exceptfortwomethods:

Dynamicimport,alsoknownasimport(),canloadanES6moduleasanasynchronousoperation

The@std/esmpackagesuppliesarequire()functionwithonethatcanloadES6modulesasanasynchronousoperation

ES6modulesloadotherES6moduleswithimport,withthefull

semanticsoftheimportstatement

ES6modulesloadCommonJSmodulesusingimport

Therefore,outofthebox,threeofthescenariosaredirectlysupported.Thefourthissupportedwithaworkaroundmodule.

WhenanES6moduleloadsaCommonJSmodule,itsmodule.exportsobjectisexposedasthedefaultexportofthemodule.Thismeansyourcodeusesthispattern:

importcjsModulefrom'common-js-module';

...

cjsModule.functionName();

ThisisextremelysimilartousingaCommonJSmoduleinanotherCommonJSmodule.Youaresimplytransliteratingtherequire()callintoanimportstatement.

Dynamicimportswithimport()ES6modulesdonotcoveralltherequirementstofullyreplaceNode.js/CommonJSmodules.OneofthemissingcapabilitiesisbeingaddressedwiththeDynamicImportfeaturecurrentlyonitswaythroughtheTC-39committee.

SupportfordynamicimportslandedinNode.js9.7.Seethedocumentationat:https://github.com/tc39/proposal-dynamic-import.

We'llusedynamicimportstosolveanissueinChapter7,DataStorageandRetrieval,aboutdynamicallychoosingthemoduletoload.Innormalusageoftherequire()statement,canuseasimplestringliteraltospecifythemodulename.Butitisalsopossibletouseastringliteraltocomputethemodulename,likeso:

//Node.jsdynamicallydeterminedmoduleloading

constmoduleName=require(`../models/${process.env.MODEL_NAME}`);

WeusedthistechniqueinearliereditionsofthisbooktodynamicallychoosebetweenseveralimplementationsofthesamemodelAPI.TheES6importstatementdoesnotsupportanythingbutasimplestringliteral,andthereforecannotcomputethemodulespecifierlikethisexample.

Withdynamicimports,wehaveanimport()functionwherethemodulespecifierisaregularstring,lettingusmakeasimilardynamicchoiceofmodule.Unliketherequire()function,whichissynchronous,import()isasynchronous,andreturnsaPromise.Hence,it'snotadirectreplacementforrequire()inthatit'snotterriblyusefulasatop-levelfunction.You'llseehowtouseitinChapter7,DataStorageandRetrieval.

PerhapsthemostimportantfeatureitbringsisthatCommonJSmodules

canuseimport()toloadanES6module.

Theimport.metafeatureAnothernewfeature,import.meta,ismakingitswaythroughtheTC-39committee,andisbeingimplementedforNode.js10.x.ItisanobjectexistingwithinthescopeofanES6moduleprovidingsomemetadataaboutthemodule.Seehttps://github.com/tc39/proposal-import-meta.

Apartialimplementation,supportingjustimport.meta.url,haslandedintheNode.jssource.Itsuserequiresthe--harmony-import-metacommand-lineflag.Thecontentofimport.meta.urlisafullyqualifiedfile:URLforthecurrentmodule,suchasfile:///Users/david/chap10/notes/app.mjs.

WherethisbecomesimportantisthatES6modulesdonotsupportthe__dirname,__filename,andotherglobalvariablesusedhistoricallyinNode.jsmodules.The__dirnamevariableisroutinelyusedtoreadinresourcedatafromfilessittinginthepackagedirectory.Itisintendedthatforsuchcases,oneparsesthedirectorynameoutofimport.meta.url.

npm-theNode.jspackagemanagementsystemAsdescribedinChapter2,SettingupNode.js,npmisapackagemanagementanddistributionsystemforNode.js.Ithasbecomethedefactostandardfordistributingmodules(packages)forusewithNode.js.Conceptually,it'ssimilartotoolssuchasapt-get(Debian),rpm/yum(RedHat/Fedora),MacPorts(macOS),CPAN(Perl),orPEAR(PHP).ItspurposeispublishinganddistributingNode.jspackagesovertheInternetusingasimplecommand-lineinterface.Withnpm,youcanquicklyfindpackagestoservespecificpurposes,downloadthem,installthem,andmanagepackagesyou'vealreadyinstalled.

ThenpmapplicationextendsonthepackageformatforNode.js,whichinturnislargelybasedontheCommonJSpackagespecification.Itusesthesamepackage.jsonfilethat'ssupportednativelybyNode.js,butwithadditionalfieldstobuildinadditionalfunctionality.

ThenpmpackageformatAnnpmpackageisadirectorystructurewithapackage.jsonfiledescribingthepackage.Thisisexactlywhatwasreferredtoearlierasadirectorymodule,exceptthatnpmrecognizesmanymorepackage.jsontagsthanNode.jsdoes.Thestartingpointfornpm'spackage.jsonaretheCommonJSPackages/1.0specification.Thedocumentationfornpm'spackage.jsonimplementationisaccessedusingthefollowingcommand:

$npmhelpjson

Abasicpackage.jsonfileisasfollows:

{"name":"packageName",

"version":"1.0",

"main":"mainModuleName",

"modules":{

"mod1":"lib/mod1",

"mod2":"lib/mod2"

}

}

ThefileisinJSONformat,which,asaJavaScriptprogrammer,youshouldbefamiliarwith.

Themostimportanttagsarenameandversion.ThenamewillappearinURLsandcommandnames,sochooseonethat'ssafeforboth.Ifyoudesiretopublishapackageinthepublicnpmrepository,it'shelpfultocheckwhetheraparticularnameisalreadybeingusedathttp://npmjs.comorwiththefollowingcommand:

$npmsearchpackageName

Themaintagistreatedthesameaswediscussedintheprevioussectionondirectorymodules.Itdefineswhichfilemodulewillbereturnedwheninvokingrequire('packageName').Packagescancontainmanymoduleswithinthemselvesandtheycanbelistedinthemoduleslist.

Packagescanbebundledastar-gziparchives(tarballs),especiallytosendthemovertheinternet.

Apackagecandeclaredependenciesonotherpackages.Thatway,npmcanautomaticallyinstallothermodulesrequiredbythemodulebeinginstalled.Dependenciesaredeclaredasfollows:

"dependencies":{

"foo":"1.0.0-2.x.x",

"bar":">=1.0.2<2.1.2"

}

Thedescriptionandkeywordfieldshelppeoplefindthepackagewhensearchinginannpmrepository(https://www.npmjs.com/).Ownershipofapackagecanbedocumentedinthehomepage,author,orcontributorsfields:

"description":"Mywonderfulpackagethatwalksdogs",

"homepage":"http://npm.dogs.org/dogwalker/",

"author":"dogwhisperer@dogs.org"

Somenpmpackagesprovideexecutableprogramsmeanttobeintheuser'sPATH.Thesearedeclaredusingthebintag.It'samapofcommandnamestothescriptthatimplementsthatcommand.Thecommandscriptsareinstalledintothedirectorycontainingthenodeexecutableusingthenamegiven:

bin:{

'nodeload.js':'./nodeload.js',

'nl.js':'./nl.js'

},

Thedirectoriestagdescribesthepackagedirectorystructure.Thelib

directoryisautomaticallyscannedformodulestoload.Thereareotherdirectorytagsforbinaries,manuals,anddocumentation:

directories:{lib:'./lib',bin:'./bin'},

Thescripttagsarescriptcommandsrunatvariouseventsinthelifecycleofthepackage.Theseeventsincludeinstall,activate,uninstall,update,andmore.Formoreinformationaboutscriptcommands,usethefollowingcommand:

$npmhelpscripts

We'vealreadyusedthescriptsfeaturewhenshowinghowtosetupBabel.We'llusetheselaterforautomatingthebuild,test,andexecutionprocesses.

Thiswasonlyatasteofthenpmpackageformat;seethedocumentation(npmhelpjson)formore.

FindingnpmpackagesBydefault,npmmodulesareretrievedovertheinternetfromthepublicpackageregistrymaintainedonhttp://npmjs.com.Ifyouknowthemodulename,itcanbeinstalledsimplybytypingthefollowing:

$npminstallmoduleName

Butwhatifyoudon'tknowthemodulename?Howdoyoudiscovertheinterestingmodules?Thewebsitehttp://npmjs.compublishesasearchableindexofthemodulesinthatregistry.

Thenpmpackagealsohasacommand-linesearchfunctiontoconsultthesameindex:

Ofcourse,uponfindingamodule,it'sinstalledasfollows:

$npminstallacoustid

Afterinstallingamodule,youmaywanttoseethedocumentation,whichwouldbeonthemodule'swebsite.Thehomepagetaginpackage.jsonliststhatURL.Theeasiestwaytolookatthepackage.jsonfileiswiththenpmviewcommand,asfollows:

$npmviewakasharender

...

{name:'akasharender',

description:'RenderingsupportforgeneratingstaticHTMLwebsites

orEPUBeBooks',

'dist-tags':{latest:'0.6.15'},

versions:

['0.0.1',

...

author:'DavidHerron<david@davidherron.com>

(http://davidherron.com)',

repository:{type:'git',url:

'git://github.com/akashacms/akasharender.git'},

homepage:'http://akashacms.com/akasharender/toc.html',

...

}

Youcanusenpmviewtoextractanytagfrompackage.json,likethefollowing,whichletsyouviewjustthehomepagetag:

$npmviewakasharenderhomepage

http://akashacms.org/akasharender/toc.html

Otherfieldsinthepackage.jsoncanbeviewedbysimplygivingthedesiredtagname.

<strong>$npmhelp<command>Thehelptextwillbeshownonyourscreen.

Or,seethewebsite:http://docs.npmjs.com</strong>

<strong>$npminstallexpress

homedavid/projects/notes/

-express@4.13.4

...</strong>

Thenamedmoduleisinstalledinnode_modulesinthecurrentdirectory.Thespecificversioninstalleddependsonanyversionnumberlistedonthecommandline,asweseeinthenextsection.

InstallingapackagebyversionnumberVersionnumbermatchinginnpmispowerfulandflexible.Thesamesortofversionspecifiersusedinpackage.jsondependenciescanalsobeusedwiththenpminstallcommand:

$npminstallpackage-name@tag

$npminstallpackage-name@version

$npminstallpackage-name@version-range

Thelasttwoarewhattheysoundlike.Youcanspecifyexpress@4.16.2totargetapreciseversion,orexpress@">4.1.0<5.0"totargetarangeofExpressV4versions.

Theversionmatchspecifiersincludethesechoices:

Exactversionmatch:1.2.3

AtleastversionN:>1.2.3

UptoversionN:<1.2.3

Betweentworeleases:>=1.2.3<1.3.0

The@tagattributeisasymbolicnamesuchas@latest,@stable,or@canary.Thepackageownerassignsthesesymbolicnamestospecificversionnumbers,andcanreassignthemasdesired.Theexceptionis@latest,whichisupdatedwheneveranewreleaseofthepackageispublished.

Formoredocumentation,runthesecommands:npmhelpjsonandnpmhelpnpm-dist-tag.

GlobalpackageinstallsInsomeinstancesyouwanttoinstallamoduleglobally,sothatitcanbeusedfromanydirectory.Forexample,theGruntorGulpbuildtoolsarewidelyuseful,andconceivablyyouwillfinditusefulifthesetoolsareinstalledglobally.Simplyaddthe-goption:

$npminstall-ggrunt-cli

Ifyougetanerror,andyou'reonaUnix-likesystem(Linux/Mac),youmayneedtorunthiswithsudo:

$sudonpminstall-ggrunt-cli

Aglobalinstallismostimportantforthosepackageswhichinstallexecutablecommands.We'llgetintothisshortly.

Ifalocalpackageinstalllandsinnode_modules,wheredoesaglobalpackageinstallland?OnaUnix-likesystemitlandsinPREFIX/lib/node_modules,andonWindowsitlandsinPREFIX/node_modules.InthiscasePREFIXmeansthedirectorywhereNode.jsisinstalled.Youcaninspectthelocationofthisdirectorylikeso:

$npmconfiggetprefix

/Users/david/.nvm/versions/node/v8.9.1

ThealgorithmusedbyNode.jsfortherequirefunctionautomaticallysearchesthisdirectoryforpackagesifthepackageisnotfoundelsewhere.

RememberthatES6modulesdonotsupportglobalpackages.

AvoidingglobalmoduleinstallationSomeintheNode.jscommunitynowfrownoninstallingapackageglobally.OnerationaleexistsintheTwelveFactormodel.Namely,asoftwareprojectismorereliableifallitsdependenciesareexplicitlydeclared.IfabuildtoolsuchasGruntisrequired,butisnotexplicitlydeclaredinpackage.json,theusersoftheapplicationwouldhavetoreceiveinstructionstoinstallGrunt,andtheywouldhavetofollowthoseinstructions.

Usersbeingusers,theymightskipovertheinstructions,failtoinstallthedependency,andthencomplaintheapplicationdoesn'twork.Surelymostofushavedonethatonceortwice.

It'srecommendedtoavoidthispotentialproblembyinstallingeverythinglocallyviaonemechanism—thenpminstallcommand.

MaintainingpackagedependencieswithnpmAswementionedearlier,thenpminstallcommandbyitselfinstallsthepackageslistedinthedependenciessectionofpackage.json.Thisiseasyandconvenient.Simplybylistingallthedependencies,it'squickandeasytoinstallthedependenciesrequiredforusingthepackage.Whathappensisnpmlooksinpackage.jsonforthedependenciesordevDependenciesfield,anditwillautomaticallyinstallthementionedpackages.

Youcanmanagethedependenciesmanuallybyeditingpackage.json.Oryoucanusenpmtoassistyouwitheditingthedependencies.Youcanaddanewdependencylikeso:

$npminstallakasharender--save

Inresponse,npmwilladdadependenciestagtopackage.json:

"dependencies":{

"akasharender":"^0.6.15"

}

Now,whenyourapplicationisinstalled,npmwillautomaticallyalsoinstallthatpackagealongwithanydependencieslistedbythatpackage.

ThedevDependenciesaremodulesusedduringdevelopment.Thatfieldisinitializedthesameasabove,butwiththe--save-devflag.

Bydefault,whenannpminstallisrun,moduleslistedinbothdependenciesanddevDependenciesareinstalled.Ofcourse,thepurposeforhavingtwolistsistonotinstallthedevDependenciesinsomecases:

$npminstall--production

ThisinstallsonlythemoduleslistedindependenciesandnoneofthedevDependenciesmodules.

IntheTwelve-Factorapplicationmodel,it'ssuggestedthatweexplicitlyidentifythedependenciesrequiredbytheapplication.Thiswaywecanreliablybuildourapplication,knowingthatwe'vetestedagainstaspecificsetofdependenciesthatwe'vecarefullyidentified.Byinstallingexactlythedependenciesagainstwhichtheapplicationhasbeentested,wehavemoreconfidenceintheapplication.OntheNode.jsplatform,npmgivesusthisdependenciessection,includingaflexiblemechanismtodeclarecompatiblepackageversionsbytheirversionnumber.

<strong>$npmconfigsetsavefalse</strong>

Thenpmconfigcommandsupportsalonglistofsettableoptionsfortuningbehaviorofnpm.Seenpmhelpconfigforthedocumentation,andnpmhelp7configforthelistofoptions.

FixingbugsbyupdatingpackagedependenciesBugsexistineverypieceofsoftware.AnupdatetotheNode.jsplatformmaybreakanexistingpackage,asmightanupgradetopackagesusedbytheapplication.Yourapplicationmaytriggerabuginapackageituses.Intheseandothercases,fixingtheproblemmightbeassimpleasupdatingapackagedependencytoalater(orearlier)version.

Firstidentifywhethertheproblemexistsinthepackageorinyourcode.Afterdeterminingit'saprobleminanotherpackage,investigatewhetherthepackagemaintainershavealreadyfixedthebug.IsthepackagehostedonGitHuboranotherservicewithapublicissuequeue?Lookforanopenissueonthisproblem.Thatinvestigationwilltellyouwhethertoupdatethepackagedependencytoalaterversion.Sometimes,itwilltellyoutoreverttoanearlierversion;forexample,ifthepackagemaintainerintroducedabugthatdoesn'texistinanearlierversion.

Sometimes,youwillfindthatthepackagemaintainersareunpreparedtoissueanewrelease.Insuchacase,youcanforktheirrepositoryandcreateapatchedversionoftheirpackage.

Oneapproachtofixingthisproblemispinningthepackageversionnumbertoonethat'sknowntowork.Youmightknowthatversion6.1.2wasthelastreleaseagainstwhichyourapplicationfunctioned,andthatstartingwithversion6.2.0yourapplicationbreaks.Hence,inpackage.json:"dependencies":{"module1":"6.1.2"}

Thisfreezesyourdependencytothespecificversionnumber.You'refree,then,totakeyourtimeupdatingyourcodetoworkagainstlaterreleasesof

thatmodule.Onceyourcodeisupdated,ortheupstreamprojectisupdated,changethedependencyappropriately.

Anotherapproachistohostaversionofthepackagesomewhereoutsideofthenpmrepository.Thisiscoveredinalatersection.

PackagesthatinstallcommandsSomepackagesinstallcommand-lineprograms.Asideeffectofinstallingsuchpackagesisanewcommandthatyoucantypeattheshellpromptoruseinshellscripts.AnexampleisthehexyprogramthatwebrieflyusedinChapter2,SettingupNode.js.AnotherexampleisthewidelyusedGruntorGulpbuildtools.

Thepackage.jsonfileinsuchpackagesspecifiesthecommand-linetoolsthatareinstalled.Thecommandcanbeinstalledtooneoftwoplaces:

GlobalInstall:Itisinstalledeithertoadirectorysuchasusrlocal,ortothebindirectorywhereNode.jswasinstalled.Thenpmbin-gcommandtellsyoutheabsolutepathnameforthisdirectory.

LocalInstall:Tonode_modules/.bininthepackagewherethemoduleisbeinginstalled.Thenpmbincommandtellsyoutheabsolutepathnameforthisdirectory.

Torunthecommand,simplytypethecommandnameatashellprompt.Exceptthere'salittlebitofconfigurationrequiredtomakethatsimple.

ConfiguringthePATHvariabletohandlecommandsinstalledbymodulesTypingthefullpathnameisnotauser-friendlyrequirementtoexecutethecommand.Wewanttousethecommandsinstalledbymodules,andwewantasimpleprocessfordoingso.Meaning,wemustaddanappropriatevalueinthePATHvariable,butwhatisit?

Forglobalpackageinstallations,theexecutablelandsinadirectorythatisprobablyalreadyinyourPATHvariable,likeusrbinorusrlocal/bin.Localpackageinstallationsarewhatrequirespecialhandling.Thefullpathforthenode_modules/.bindirectoryvariesforeachproject,andobviouslyitwon'tworktoaddthefullpathforeverynode_modules/.bindirectorytoyourPATH.

Adding./node_modules/.bintothePATHvariable(or,onWindows,.\node_modules\.bin)worksgreat.AnytimeyourshellisintherootofaNode.jsproject,itwillautomaticallyfindlocally-installedcommandsfromNode.jspackages.

Howwedothisdependsonthecommandshellyouuse,andyouroperatingsystem.

OnaUnix-likesystemthecommandshellsarebashandcsh.YourPATHvariablewouldbesetupinoneoftheseways:

$exportPATH=./node_modules/.bin:${PATH}#bash

$setenvPATH./node_modules/.bin:${PATH}#csh

Thenextstepisaddingthecommandtoyourloginscriptssothevariableisalwaysset.Onbash,addthecorrespondinglinetoyour~/.bashrc,andon

cshaddittoyour~/.cshrc.

ConfiguringthePATHvariableonWindowsOnWindows,thistaskishandledthroughasystem-widesettingspanel:

ThispaneoftheSystemPropertiespanelisfoundbysearchingforPATHintheWindowsSettingsscreen.ClickontheEnvironmentVariablesbutton,thenselectthePathvariable,andfinallyclickontheEditbutton.InthescreenhereclicktheNewbuttontoaddanentrytothisvariable,andenter.\node_modules\.binasshown.You'llhavetorestartanyopencommandshellwindows.Onceyoudo,theeffectwillbeasshownpreviously.

AvoidingmodificationstothePATHvariableWhatifyoudon'twanttoaddthesevariablestoyourPATHatalltimes?Thenpm-pathmodulemaybeofinterest.ThisisasmallprogramthatcomputesthecorrectPATHvariableforyourshellandoperatingsystem.Seethepackageathttps://www.npmjs.com/package/npm-path.

Updatingoutdatedpackagesyou'veinstalledThecodercodes,updatingtheirpackage,leavingyouintheirdustunlessyoukeepup.

Tofindoutifyourinstalledpackagesareoutofdate,usethefollowingcommand:$npmoutdated

Thereportshowsthecurrentnpmpackages,thecurrently-installedversion,aswellasthecurrentversioninthenpmrepository.Updatingtheoutdatedpackagesisverysimple:$npmupdateexpress$npmupdate

InstallingpackagesfromoutsidethenpmrepositoryAsawesomeasthenpmrepositoryis,wedon'twanttopusheverythingwedothroughtheirservice.Thisisespeciallytrueforinternaldevelopmentteamswhocannotpublishtheircodeforalltheworldtosee.Whileyoucanrentorinstallaprivatenpmrepository,there'sanotherway.Packagescanbeinstalledfromotherlocations.Detailsaboutthisareinnpmhelppackage.jsoninthedependenciessection.Someexamplesare:

URL:YoucanspecifyanyURLthatdownloadsatarball,thatis,a.tar.gzfile.Forexample,GitHuborGitLabrepositoriescaneasilyexporttarballURL.SimplygototheReleasestabtofindthem.

GitURL:Similarly,anyGitrepositorycanbeaccessedwiththerightURL.Forexample:

$npminstallgit+ssh://user@hostname:project.git#tag--save

GitHubShortcut:ForGitHubrepositoriesyoucanlistjusttherepositoryspecifier,suchasexpressjs/express.Atagoracommitcanbereferencedusingexpressjs/express#tag-name.

Localfilesystem:YoucaninstallfromalocaldirectoryusingaURLlikethis:file:../../path/to/dir.

InitializinganewnpmpackageIfyouwanttocreateanewpackage,youcancreatethepackage.jsonfilebyhandoryoucangetnpm'shelp.Thenpminitcommandleadsyouthroughalittledialogtogetstartingvaluesforthepackage.jsonfile.

Onceyougetthroughthequestions,thepackage.jsonfileiswrittentodisk.

Expecttohavetoeditthatfileconsiderablybeforepublishingtothenpmrepository.Afewfieldshelpgiveagoodimpressiontofolkslookingatthepackagelistingonnpmjs.com:

Linktothehomepage,andissuequeueURL

Keywords,soitcanbelinkedwithothersimilarpackages

Agooddescriptionthathelpsfolksunderstandthepurpose

AgoodREADME.mdfilesofolkscanreadsomedocumentationrightaway

"engines":{

"node":">=8.x"

}

This,ofcourse,usesthesameversionnumbermatchingschemediscussedearlier.

PublishingannpmpackageAllthosepackagesinthenpmrepositorycamefrompeoplelikeyouwithanideaofabetterwayofdoingsomething.Itisveryeasytogetstartedwithpublishingpackages.Onlinedocscanbefoundathttps://docs.npmjs.com/getting-started/publishing-npm-packages.

Youfirstusethenpmaddusercommandtoregisteryourselfwiththenpmrepository.Youcanalsosignupwiththewebsite.Next,youloginusingthenpmlogincommand.

Finally,whilesittinginthepackagerootdirectory,usethenpmpublishcommand.Then,standbacksothatyoudon'tgetstampededbythecrushofthrongingfans.Or,maybenot.Therearealmost600,000packagesintherepository,withalmost400packagesaddedeveryday.Togetyourstostandout,youwillrequiresomemarketingskill,whichisanothertopicbeyondthescopeofthisbook.

ExplicitlyspecifyingpackagedependencyversionnumbersOnefeatureoftheTwelve-Factormethodologyissteptwo,explicitlydeclaringyourdependencies.We'vealreadytouchedonthis,butit'sworthreiteratingandtoseeingnpmmakesthiseasytoaccomplish.

SteponeoftheTwelve-Factormethodologyisensuringthatyourapplicationcodeischeckedintoasourcecoderepository.Youprobablyalreadyknowthis,andevenhavethebestofintentionstoensurethateverythingischeckedin.WithNode.js,eachmoduleshouldhaveitsownrepositoryratherthanputtingeverysinglelastpieceofcodeinonerepository.

Eachmodulecanthenprogressonitsowntimeline.Abreakageinonemoduleiseasytobackoutbychangingtheversiondependencyinpackage.json.

ThisgetsustoTwelve-Factorsteptwo.Therearetwoaspectsofthisstep,oneofwhichisthepackageversioningthatwediscussedpreviously.Thenextisexplicitlydeclaringversionnumbers,whichcanbedeclaredindependenciesanddevDependenciessectionsofpackage.json.Thisensuresthateveryoneontheteamisonthesamepage,developingagainstthesameversionsofthesamemodules.Whenit'stimetodeploytotesting,staging,orproductionservers,andthedeploymentscriptrunsnpminstallornpmupdate,thecodewilluseaknownversionofthemodulethateveryonetestedagainst.

Thelazywayofdeclaringdependenciesisputting*intheversionfield.Thatusesthelatestversioninthenpmrepository.Maybethiswillwork,untilonedaythemaintainersofthatpackageintroduceabug.You'lltypenpmupdate,andallofasuddenyourcodedoesn'twork.You'llheadoverto

theGitHubsiteforthepackage,lookintheissuequeue,andpossiblyseethatothershavealreadyreportedtheproblemyou'reseeing.Someofthemwillsaythatthey'vepinnedonthepreviousreleaseuntilthisbugisfixed.Whatthatmeansistheirpackage.jsonfiledoesnotdependon*forthelatestversion,butonaspecificversionnumberbeforethebugwascreated.

Don'tdothelazything,dothesmartthing.

Theotheraspectofexplicitlydeclaringdependenciesistonotimplicitlydependonglobalpackages.Earlier,wesaidthatsomeintheNode.jscommunitycautionagainstinstallingmodulesintheglobaldirectories.Thismightseemlikeaneasyshortcuttosharingcodebetweenapplications.Justinstallitglobally,andyoudon'thavetoinstallthecodeineachapplication.

But,doesn'tthatmakedeploymentharder?Willthenewteammemberbeinstructedonallthespecialfilestoinstallhereandtheretomaketheapplicationrun?Willyouremembertoinstallthatglobalmoduleonalldestinationmachines?

ForNode.js,thatmeanslistingallthemoduledependenciesinpackage.json,andthentheinstallationinstructionsaresimplynpminstall,followedperhapsbyeditingaconfigurationfile.

TheYarnpackagemanagementsystemAspowerfulasnpmis,itisnottheonlypackagemanagementsystemforNode.js.BecausetheNode.jscoreteamdoesnotdictateapackagemanagementsystem,theNode.jscommunityisfreetorolluptheirsleevesanddevelopanysystemtheyfeelbest.Thatthevastmajorityofususenpmisatestamenttoitsvalueandusefulness.Butthereisacompetitor.

Yarn(seehttps://yarnpkg.com/en/)isacollaborationbetweenengineersatFacebook,Google,andseveralothercompanies.TheyproclaimthatYarnisultrafast,ultra-secure(byusingchecksumsofeverything),andultrareliable(byusingayarn-lock.jsonfiletorecordprecisedependencies).

Insteadofrunningtheirownpackagerepository,Yarnrunsontopofnpm'spackagerepositoryatnpmjs.com.ThismeansthattheNode.jscommunityisnotforkedbyYarn,butenhancedbyhavinganimprovedpackagemanagementtool.

ThenpmteamrespondedtoYarninnpm@5(alsoknownasnpmversion5)byimprovingperformance,andbyintroducingapackage-lock.jsonfiletoimprovereliability.Thenpmteamhaveannouncedadditionalimprovementsinnpm@6.

Yarnhasbecomeverypopularandiswidelyrecommendedovernpm.Theyperformextremelysimilarfunctions,andtheperformanceisnotthatdifferenttonpm@5.Thecommand-lineoptionsarewordeddifferently.AnimportantbenefitYarnbringstotheNode.jscommunityisthatcompetitionbetweenYarnandnpmseemstobebreedingfasteradvancesinNode.jspackagemanagement.

Togetyoustarted,thesearethemostimportantcommands:

yarnadd:Addsapackagetouseinyourcurrentpackage

yarninit:Initializesthedevelopmentofapackage

yarninstall:Installsallthedependenciesdefinedinapackage.jsonfile

yarnpublish:Publishesapackagetoapackagemanager

yarnremove:Removesanunusedpackagefromyourcurrentpackage

Runningyarnbyitselfdoestheyarninstallbehavior.ThereareseveralothercommandsinYarn,andyarnhelpwilllistthemall.

SummaryYoulearnedalotinthischapteraboutmodulesandpackagesforNode.js.

Specifically,wecoveredimplementingmodulesandpackagesforNode.js,managinginstalledmodulesandpackages,andsawhowNode.jslocatesmodules.

Nowthatyou'velearnedaboutmodulesandpackages,we'rereadytousethemtobuildapplications,whichisthetopicofthenextchapter.

HTTPServersandClientsNowthatyou'velearnedaboutNode.jsmodules,it'stimetoputthisknowledgetoworkbybuildingasimpleNode.jswebapplication.Inthischapter,we'llkeeptoasimpleapplication,enablingustoexplorethreedifferentapplicationframeworksforNode.js.Inlaterchapters,we'llbuildsomemorecomplexapplications,butbeforewecanwalk,wemustlearntocrawl.

Wewillcoverthefollowingtopicsinthischapter:

EventEmitters

ListeningtoHTTPeventsandtheHTTPServerobject

HTTPrequestrouting

ES2015templatestrings

Buildingasimplewebapplicationwithnoframeworks

TheExpressapplicationframework

Expressmiddlewarefunctions

Howtodealwithcomputationallyintensivecode

TheHTTPClientobject

CreatingasimpleRESTservicewithExpress

SendingandreceivingeventswithEventEmittersEventEmittersareoneofthecoreidiomsofNode.js.IfNode.js'scoreideaisanevent-drivenarchitecture,emittingeventsfromanobjectisoneoftheprimarymechanismsofthatarchitecture.AnEventEmitterisanobjectthatgivesnotifications—events—atdifferentpointsinitslifecycle.Forexample,anHTTPServerobjectemitseventsconcerningeachstageofthestartup/shutdownoftheServerobject,andasHTTPrequestsaremadefromHTTPclients.

ManycoreNode.jsmodulesareEventEmitters,andEventEmittersareanexcellentskeletontoimplementasynchronousprogramming.EventEmittershavenothingtodowithwebapplicationdevelopment,buttheyaresomuchpartoftheNode.jswoodworkthatyoumayskipovertheirexistence.

Inthischapter,we'llworkwiththeHTTPServerandHTTPClientobjects.BotharesubclassesoftheEventEmitterclass,andrelyonittosendeventsforeachstepoftheHTTPprotocol.

JavaScriptclassesandclassinheritanceBeforegettingstartedontheEventEmitterclass,weneedtotakealookatanotheroftheES2015features:classes.TheJavaScriptlanguagehasalwayshadobjects,andaconceptofaclasshierarchy,butnothingsoformalasinotherlanguages.TheES2015classobjectbuildsontheexistingprototype-basedinheritancemodel,butwithasyntaxlookingverymuchlikeclassdefinitionsinotherlanguages.

Forexample,considerthisclasswe'llbeusinglaterinthebook:

classNote{

constructor(key,title,body){

this._key=key;

this._title=title;

this._body=body;

}

getkey(){returnthis._key;}

gettitle(){returnthis._title;}

settitle(newTitle){returnthis._title=newTitle;}

getbody(){returnthis._body;}

setbody(newBody){returnthis._body=newBody;}

}

Onceyou'vedefinedtheclass,youcanexporttheclassdefinitiontoothermodules:

module.exports.Note=classNote{..}#inCommonJSmodules

exportclassNote{..}#inES6modules

Thefunctionsmarkedwithgetorsetkeywordsaregettersandsetters,usedlikeso:

varaNote=newNote("key","TheRaininSpain","Fallsmainlyonthe

plain");

varkey=aNote.key;

vartitle=aNote.title;

aNote.title="TheRaininSpain,whichmademewanttocrywithjoy";

Newinstancesofaclassarecreatedwithnew.Youaccessagetterorsetterfunctionasifitisasimplefieldontheobject.Behindthescenes,thegetter/setterfunctionisinvoked.

Theprecedingimplementationisnotthebestbecausethe_titleand_bodyfieldsarepubliclyvisible,andthereisnodatahidingorencapsulation.We'llgooverabetterimplementationlater.

Onetestswhetheragivenobjectisofacertainclassbyusingtheinstanceofoperator:

if(anotherNoteinstanceofNote){

...it'saNote,soactonitasaNote

}

Finally,youdeclareasubclassusingtheextendsoperator,similartowhat'sdoneinotherlanguages:

classLoveNoteextendsNote{

constructor(key,title,body,heart){

super(key,title,body);

this._heart=heart;

}

getheart(){returnthis._heart;}

setheart(newHeart){returnthis._heart=newHeart;}

}

Inotherwords,theLoveNoteclasshasallthefieldsofNote,plusthisnewfieldnamedheart.

TheEventEmitterClassTheEventEmitterobjectisdefinedintheeventsmoduleofNode.js.DirectlyusingtheEventEmitterclassmeansperformingrequire('events').Inmostcases,you'llbeusinganexistingobjectthatusesEventEmitterinternallyandyouwon'trequirethismodule.ButtherearecaseswhereneedsdictateimplementinganEventEmittersubclass.

Createafilenamedpulser.jscontainingthefollowingcode:

constEventEmitter=require('events');

classPulserextendsEventEmitter{

start(){

setInterval(()=>{

console.log(`${newDate().toISOString()}>>>>pulse`);

this.emit('pulse');

console.log(`${newDate().toISOString()}<<<<pulse`);

},1000);

}

}

module.exports=Pulser;

ThisdefinesaPulserclass,whichinheritsfromEventEmitter.InolderNode.jsreleases,thiswouldrequireusingutil.inherits,butthenewclassobjectmakessubclassingmuchsimpler.

Anotherthingtoexamineishowthis.emitinthecallbackfunctionreferstothePulserobject.BeforetheES2015arrowfunction,whenourcallbacksusedaregularfunction,thiswouldnothavereferredtothePulserobject.Instead,itwouldhavereferredtosomeotherobjectrelatedtothesetIntervalfunction.Becauseitisanarrowfunction,thethisinsidethearrowfunctionisthesamethisasintheouterfunction.

Ifyouneededtouseafunctionratherthananarrowfunction,thistrick

wouldwork:

classPulserextendsEventEmitter{

start(){

varself=this;

setInterval(function(){

self.emit(...);

});

}

}

What'sdifferentistheassignmentofthistoself.Thevalueofthisinsidethefunctionisdifferent,butthevalueofselfremainsthesameineveryenclosedscope.Thiswidely-usedtrickislessnecessarynowthatwehavearrowfunctions.

IfyouwantasimpleEventEmitter,butwithyourownclassname,thebodyoftheextendedclasscanbeempty:

classHeartBeatextendsEventEmitter{}

constbeatMaker=newHeartBeat();

ThepurposeofthePulserclassissendingatimedevent,onceasecond,toanylisteners.ThestartmethodusessetIntervaltokickoffrepeatedcallbackexecution,scheduledforeverysecond,callingemittosendthepulseeventstoanylisteners.

Now,let'sseehowtousethePulserobject.Createanewfile,calledpulsed.js,containing:

constPulser=require('./pulser');

//InstantiateaPulserobject

constpulser=newPulser();

//Handlerfunction

pulser.on('pulse',()=>{

console.log(`${newDate().toISOString()}pulsereceived`);

});

//Startitpulsing

pulser.start();

Here,wecreateaPulserobjectandconsumeitspulseevents.Callingpulser.on('pulse')setsupconnectionsforthepulseeventstoinvokethecallbackfunction.Itthencallsthestartmethodtogettheprocessgoing.

Enterthisintoafileandnamethefilepulsed.js.Whenyourunit,youshouldseethefollowingoutput:

$nodepulsed.js

2017-12-03T06:24:10.272Z>>>>pulse

2017-12-03T06:24:10.275Zpulsereceived

2017-12-03T06:24:10.276Z<<<<pulse

2017-12-03T06:24:11.279Z>>>>pulse

2017-12-03T06:24:11.279Zpulsereceived

2017-12-03T06:24:11.279Z<<<<pulse

2017-12-03T06:24:12.281Z>>>>pulse

2017-12-03T06:24:12.281Zpulsereceived

2017-12-03T06:24:12.282Z<<<<pulse

ThatgivesyoualittlepracticalknowledgeoftheEventEmitterclass.Let'snowlookatitsoperationaltheory.

TheEventEmittertheoryWiththeEventEmitterclass,yourcodeemitseventsthatothercodecanreceive.It'sawayofconnectingtwoseparatedsectionsofyourprogram,kindoflikehowquantumentanglementmeanstwoelectronscancommunicatewitheachotherfromanydistance.Seemssimpleenough.

Theeventnamecanbeanythingthatmakessensetoyou,andyoucandefineasmanyeventnamesasyoulike.Eventnamesaredefinedsimplybycalling.emitwiththeeventname.There'snothingformaltodoandnoregistryofeventnames.Simplymakingacallto.emitisenoughtodefineaneventname.

Byconvention,theeventnameerrorindicateserrors.

Anobjectsendseventsusingthe.emitfunction.Eventsaresenttoanylistenersthathaveregisteredtoreceiveeventsfromtheobject.Theprogramregisterstoreceiveaneventbycallingthatobject's.onmethod,givingtheeventnameandaneventhandlerfunction.

Thereisnocentraldistributionpointforallevents.Instead,eachinstanceofanEventEmitterobjectmanagesitsownsetoflistenersanddistributesitseventstothoselisteners.

Often,itisrequiredtosenddataalongwithanevent.Todoso,simplyaddthedataasargumentstothe.emitcall,asfollows:

this.emit('eventName',data1,data2,..);

Whentheprogramreceivesthatevent,thedataappearsasargumentstothecallbackfunction.Yourprogramwouldlistentosuchaneventasfollows:

emitter.on('eventName',(data1,data2,...theArgs)=>{

//actonevent

});

Thereisnohandshakingbetweeneventreceiversandtheeventsender.Thatis,theeventsendersimplygoesonwithitsbusiness,anditgetsnonotificationsaboutanyeventsreceived,anyactiontaken,oranyerrorthatoccurred.

Inthisexample,weusedanotheroftheES2015features,therestoperator,shownhereas...theArgs.Therestoperatorcatchesanynumberofremainingfunctionparametersintoanarray.SinceEventEmittercanpassalonganynumberofparameters,andtherestoperatorcanautomaticallyreceiveanynumberofparameters,it'samatchmadeinheaven,orelseintheTC-39committee.

HTTPserverapplicationsTheHTTPserverobjectisthefoundationofallNode.jswebapplications.TheobjectitselfisveryclosetotheHTTPprotocol,anditsuserequiresknowledgeofthatprotocol.Inmostcases,you'llbeabletouseanapplicationframeworksuchasExpressthathidestheHTTPprotocoldetails,allowingtheprogrammertofocusonbusinesslogic.

WealreadysawasimpleHTTPserverapplicationinChapter2,SettingupNode.js,whichisasfollows:

consthttp=require('http');

http.createServer((req,res)=>{

res.writeHead(200,{'Content-Type':'text/plain'});

res.end('Hello,World!\n');

}).listen(8124,'127.0.0.1');

console.log('Serverrunningathttp://127.0.0.1:8124');

Thehttp.createServerfunctioncreatesanhttp.Serverobject.BecauseitisanEventEmitter,thiscanbewritteninanotherwaytomakethatfactexplicit:

consthttp=require('http');

constserver=http.createServer();

server.on('request',(req,res)=>{

res.writeHead(200,{'Content-Type':'text/plain'});

res.end('Hello,World!\n');

});

server.listen(8124,'127.0.0.1');

console.log('Serverrunningathttp://127.0.0.1:8124');

Therequesteventtakesafunction,whichreceivesrequestandresponseobjects.Therequestobjecthasdatafromthewebbrowser,whiletheresponseobjectisusedtogatherthedatatobesentintheresponse.Thelistenfunctioncausestheservertostartlisteningandarrangingtodispatchaneventforeveryrequestarrivingfromawebbrowser.

Now,let'slookatsomethingmoreinterestingwithdifferentactionsbasedontheURL.

Createanewfile,namedserver.js,containingthefollowingcode:

consthttp=require('http');

constutil=require('util');

consturl=require('url');

constos=require('os');

constserver=http.createServer();

server.on('request',(req,res)=>{

varrequrl=url.parse(req.url,true);

if(requrl.pathname===''){

res.writeHead(200,{'Content-Type':'texthtml'});

res.end(

`<html><head><title>Hello,world!</title></head>

<body><h1>Hello,world!</h1>

<p><ahref='osinfo'>OSInfo<a></p>

</body></html>`);

}elseif(requrl.pathname==="osinfo"){

res.writeHead(200,{'Content-Type':'texthtml'});

res.end(

`<html><head><title>OperatingSystemInfo</title></head>

<body><h1>OperatingSystemInfo</h1>

<table>

<tr><th>TMPDir</th><td>${os.tmpdir()}</td></tr>

<tr><th>HostName</th><td>${os.hostname()}</td></tr>

<tr><th>OSType</th><td>${os.type()}${os.platform()}${os.arch()}

${os.release()}</td></tr>

<tr><th>Uptime</th><td>${os.uptime()}${util.inspect(os.loadavg())}</td></tr>

<tr><th>Memory</th><td>total:${os.totalmem()}free:${os.freemem()}</td>

</tr>

<tr><th>CPU's</th><td><pre>${util.inspect(os.cpus())}</pre></td></tr>

<tr><th>Network</th><td><pre>${util.inspect(os.networkInterfaces())}</pre>

</td></tr>

</table>

</body></html>`);

}else{

res.writeHead(404,{'Content-Type':'text/plain'});

res.end("badURL"+req.url);

}

});

server.listen(8124);

console.log('listeningtohttp://localhost:8124');

Torunit,typethefollowingcommand:

$nodeserver.js

listeningtohttp://localhost:8124

ThisapplicationismeanttobesimilartoPHP'ssysinfofunction.Node'sosmoduleisconsultedtoprovideinformationabouttheserver.Thisexamplecaneasilybeextendedtogatherotherpiecesofdataabouttheserver:

Acentralpartofanywebapplicationisthemethodofroutingrequeststorequesthandlers.Therequestobjecthasseveralpiecesofdataattachedtoit,twoofwhichareusefulforroutingrequests:therequest.urlandrequest.methodfields.

Inserver.js,weconsulttherequest.urldatatodeterminewhichpagetoshow,afterparsing(usingurl.parse)toeasethedigestionprocess.Inthis

case,wecandoasimplecomparisonofthepathnametodeterminewhichhandlermethodtouse.

SomewebapplicationscareabouttheHTTPverb(GET,DELETE,POST,andsoon)usedandmustconsulttherequest.methodfieldoftherequestobject.Forexample,POSTisfrequentlyusedforFORMsubmissions.

ThepathnameportionoftherequestURLisusedtodispatchtherequesttothecorrecthandler.Whilethisroutingmethod,basedonsimplestringcomparison,willworkforasmallapplication,it'llquicklybecomeunwieldy.LargerapplicationswillusepatternmatchingtousepartoftherequestURLtoselecttherequesthandlerfunctionandotherpartstoextractrequestdataoutoftheURL.We'llseethisinactionwhilelookingatExpresslaterintheGettingstartedwithExpresssection.

AsearchforaURLmatchinthenpmrepositoryturnsupseveralpromisingpackagesthatcouldbeusedtoimplementrequestmatchingandrouting.AframeworklikeExpresshasthiscapabilityalreadybakedinandtested.

IftherequestURLisnotrecognized,theserversendsbackanerrorpageusinga404resultcode.Theresultcodeinformsthebrowseraboutthestatusoftherequest,wherea200codemeanseverythingisfine,anda404codemeanstherequestedpagedoesn'texist.Thereare,ofcourse,manyotherHTTPresponsecodes,eachwiththeirownmeaning.

ES2015multilineandtemplatestringsThepreviousexampleshowedtwoofthenewfeaturesintroducedwithES2015,multilineandtemplatestrings.Thefeatureismeanttosimplifyourlifewhilecreatingtextstrings.

Theexistingstringrepresentationsusesinglequotesanddoublequotes.Templatestringsaredelimitedwiththebacktickcharacterthat'salsoknownasthegraveaccent:

`templatestringtext`

BeforeES2015,onewaytoimplementamultilinestringwasthisconstruct:

["<html><head><title>Hello,world!</title></head>",

"<body><h1>Hello,world!</h1>",

"<p><ahref='osinfo'>OSInfo<a></p>",

"</body></html>"]

.join('\n')

Yes,thatwasthecodeusedinthesameexampleinpreviousversionsofthisbook.ThisiswhatwecandowithES2015:

`<html><head><title>Hello,world!</title></head>

<body><h1>Hello,world!</h1>

<p><ahref='osinfo'>OSInfo<a></p>

</body></html>`

Thisismoresuccinctandstraightforward.Theopeningquoteisonthefirstline,theclosingquoteonthelastline,andeverythinginbetweenis

partofourstring.

Therealpurposeofthetemplatestringsfeatureissupportingstringswherewecaneasilysubstitutevaluesdirectlyintothestring.Mostotherprogramminglanguagessupportthisability,andnowJavaScriptdoestoo.

Pre-ES2015,aprogrammercouldhavewrittencodelikethis:

[...

"<tr><th>OSType</th><td>{ostype}{osplat}{osarch}{osrelease}</td></tr>"

...].join('\n')

.replace("{ostype}",os.type())

.replace("{osplat}",os.platform())

.replace("{osarch}",os.arch())

.replace("{osrelease}",os.release())

Again,thisisextractedfromthesameexampleinpreviousversionsofthisbook.Withtemplatestrings,thiscanbewrittenasfollows:

`...<tr><th>OSType</th><td>${os.type()}${os.platform()}${os.arch()}

${os.release()}</td></tr>...`

Withinatemplatestring,thepartwithinthe${..}bracketsisinterpretedasanexpression.Itcanbeasimplemathematicalexpression,avariablereference,or,asinthiscase,afunctioncall.

Thelastthingtomentionisamatterofindentation.Innormalcoding,oneindentsalongargumentlisttothesamelevelasthecontainingfunctioncall.But,forthesemultilinestringexamples,thetextcontentisflushwithcolumnzero.What'sup?

Thismayimpedethereadabilityofyourcode,soit'sworthweighingcodereadabilityagainstanotherissue:excesscharactersintheHTMLoutput.TheblankswewouldusetoindentthecodeforreadabilitywillbecomepartofthestringandwillbeoutputintheHTML.Bymakingthecodeflushwithcolumnzero,wedon'taddexcessblankstotheoutputatthecostofsomecodereadability.

Thisapproachalsocarriesasecurityrisk.Haveyouverifiedthedataissafe?Thatitwillnotformthebasisofasecurityattack?Inthiscase,we'redealingwithsimplestringsandnumberscomingfromasafedatasource.ThereforethiscodeisassafeastheNode.jsruntime.Whataboutuser-suppliedcontent,andtheriskthatanefarioususermightsupplyinsecurecontentimplantingsomekindofmalwareintotargetcomputers?

Forthisandmanyotherreasons,itisoftensafertouseanexternaltemplateengine.ApplicationslikeExpressmakeiteasytodoso.

HTTPSniffer–listeningtotheHTTPconversationTheeventsemittedbytheHTTPServerobjectcanbeusedforadditionalpurposesbeyondtheimmediatetaskofdeliveringawebapplication.ThefollowingcodedemonstratesausefulmodulethatlistenstoalltheHTTPServerevents.Itcouldbeausefuldebuggingtool,whichalsodemonstrateshowHTTPserverobjectsoperate.

Node.js'sHTTPServerobjectisanEventEmitterandtheHTTPSniffersimplylistenstoeveryserverevent,printingoutinformationpertinenttoeachevent.

Whatwe'reabouttodois:

1. Createamodule,httpsniffer,thatprintsinformationaboutHTTPrequests.

2. Addthatmoduletotheserver.jsscriptwejustcreated.3. RerunthatservertoviewatraceofHTTPactivity.

Createafilenamedhttpsniffer.jscontainingthefollowingcode:

constutil=require('util');

consturl=require('url');

consttimestamp=()=>{returnnewDate().toISOString();}

exports.sniffOn=function(server){

server.on('request',(req,res)=>{

console.log(`${timestamp()}e_request`);

console.log(`${timestamp()}${reqToString(req)}`);

});

server.on('close',errno=>{console.log(`${timestamp()}e_close

${errno}`);});

server.on('checkContinue',(req,res)=>{

console.log(`${timestamp()}e_checkContinue`);

console.log(`${timestamp()}${reqToString(req)}`);

res.writeContinue();

});

server.on('upgrade',(req,socket,head)=>{

console.log(`${timestamp()}e_upgrade`);

console.log(`${timestamp()}${reqToString(req)}`);

});

server.on('clientError',()=>{console.log(`${timestamp()}

e_clientError`);});

};

constreqToString=exports.reqToString=(req)=>{

varret=`req${req.method}${req.httpVersion}${req.url}`+'\n';

ret+=JSON.stringify(url.parse(req.url,true))+'\n';

varkeys=Object.keys(req.headers);

for(vari=0,l=keys.length;i<l;i++){

varkey=keys[i];

ret+=`${i}${key}:${req.headers[key]}`+'\n';

}

if(req.trailers)ret+=util.inspect(req.trailers)+'\n';

returnret;

};

Thatwasalotofcode!ButthekeytoitisthesniffOnfunction.WhengivenanHTTPServerobject,itusesthe.onfunctiontoattachlistenerfunctionsthatprintdataabouteachemittedevent.ItgivesafairlydetailedtraceofHTTPtrafficonanapplication.

Inordertouseit,simplyinsertthiscodejustbeforethelistenfunctioninserver.js:

require('./httpsniffer').sniffOn(server);

server.listen(8124);

console.log('listeningtohttp://localhost:8124');

Withthisinplace,runtheserveraswedidearlier.Youcanvisithttp://localhost:8124/inyourbrowserandseethefollowingconsoleoutput:

$nodeserver.js

listeningtohttp://localhost:8124

2017-12-03T19:21:33.162Zrequest

2017-12-03T19:21:33.162ZrequestGET1.1/

{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname

":null,"hash":null,"search":"","query":{},"pathname":"/","path":"","href":""}

0host:localhost:8124

1upgrade-insecure-requests:1

2accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

3user-agent:Mozilla/5.0(Macintosh;IntelMacOSX10_11_6)

AppleWebKit/604.3.5(KHTML,likeGecko)Version/11.0.1Safari/604.3.5

4accept-language:en-us

5accept-encoding:gzip,deflate

6connection:keep-alive

{}

2017-12-03T19:21:42.154Zrequest

2017-12-03T19:21:42.154ZrequestGET1.1/osinfo

{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname

":null,"hash":null,"search":"","query":

{},"pathname":"/osinfo","path":"osinfo","href":"osinfo"}

0host:localhost:8124

1connection:keep-alive

2upgrade-insecure-requests:1

3accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

4user-agent:Mozilla/5.0(Macintosh;IntelMacOSX10_11_6)

AppleWebKit/604.3.5(KHTML,likeGecko)Version/11.0.1Safari/604.3.5

5referer:http://localhost:8124/

6accept-language:en-us

7accept-encoding:gzip,deflate

{}

YounowhaveatoolforsnoopingonHTTPServerevents.Thissimpletechniqueprintsadetailedlogofeventdata.ThepatterncanbeusedforanyEventEmitterobject.YoucanusethistechniqueasawaytoinspecttheactualbehaviorofEventEmitterobjectsinyourprogram.

WebapplicationframeworksTheHTTPServerobjectisveryclosetotheHTTPprotocol.Whilethisispowerfulinthesamewaythatdrivingastickshiftcargivesyoulow-levelcontroloverthedrivingexperience,typicalwebapplicationprogrammingisbetterdoneatahigherlevel.Doesanybodyuseassemblylanguagetowritewebapplications?It'sbettertoabstractawaytheHTTPdetailsandconcentrateonyourapplication.

TheNode.jsdevelopercommunityhasdevelopedquiteafewapplicationframeworkstohelpwithdifferentaspectsofabstractingawayHTTPprotocoldetails.Ofthem,Expressisthemostpopular,andKoa(http://koajs.com/)shouldbeconsideredbecauseitwasdevelopedbythesameteamandhasfullyintegratedsupportforasyncfunctions.

TheExpressJSWikihasalistofframeworksbuiltontopofExpressJS,ortoolsthatworkwithit.Thisincludestemplateengines,middlewaremodules,andmore.TheExpressJSWikiislocatedathttps://github.com/expressjs/express/wiki.

Onereasontouseawebframeworkisthattheyoftenprovidethebestpracticesusedinwebapplicationdevelopmentforover20years.Theusualbestpracticesincludethefollowing:

ProvidingapageforbadURLs(the404page)

ScreeningURLsandformsforanyinjectedscriptingattacks

Supportingtheuseofcookiestomaintainsessions

Loggingrequestsforbothusagetrackinganddebugging

Authentication

Handlingstaticfiles,suchasimages,CSS,JavaScript,orHTML

Providingcachecontrolheaderstocachingproxies

Limitingthingssuchaspagesizeorexecutiontime

WebframeworkshelpyouinvestyourtimeinthetaskwithoutgettinglostinthedetailsofimplementingHTTPprotocol.Abstractingawaydetailsisatime-honoredwayforprogrammerstobemoreefficient.Thisisespeciallytruewhenusingalibraryorframeworkprovidingprepackagedfunctionsthattakecareofthedetails.

GettingstartedwithExpressExpressisperhapsthemostpopularNode.jswebappframework.It'ssopopularthatit'spartoftheMEANStackacronym.MEANreferstoMongoDB,ExpressJS,AngularJS,andNode.js.ExpressisdescribedasbeingSinatra-like,referringtoapopularRubyapplicationframework,andthatitisn'tanopinionatedframework,meaningtheframeworkauthorsdon'timposetheiropinionsaboutstructuringanapplication.ThismeansExpressisnotatallstrictabouthowyourcodeisstructured;youjustwriteitthewayyouthinkisbest.

YoucanvisitthehomepageforExpressathttp://expressjs.com/.

Shortly,we'llimplementasimpleapplicationtocalculateFibonaccinumbersusingExpress,andinlaterchapters,we'lldoquiteabitmorewithExpress.We'llalsoexplorehowtomitigatetheperformanceproblemsfromcomputationallyintensivecodewediscussedearlier.

Asofwritingthisbook,Express4.16isthecurrentversion,andExpress5isinAlphatesting.AccordingtotheExpressJSwebsite,thereareveryfewdifferencesbetweenExpress4andExpress5.

Let'sstartbyinstallingtheexpress-generator.Whilewecanjuststartwritingsomecode,theexpress-generatorprovidesablankstartingapplication.We'lltakethatandmodifyit.

Installitusingthefollowingcommands:

$mkdirfibonacci

$cdfibonacci

$npminstallexpress-generator@4.x

ThisisdifferentfromthesuggestedinstallationmethodontheExpress

website,whichwastousethe-gtagforaglobalinstall.We'realsousinganexplicitversionnumbertoensurecompatibility.Asofwritingthisbook,express-generator@5.xdoesnotexist.Whenitdoesexist,oneshouldbeabletousethe5.xversionwiththefollowinginstructions.

Earlier,wediscussedhowmanynowrecommendagainstinstallingmodulesglobally.IntheTwelve-Factormodel,it'sstronglyrecommendedtonotinstallglobaldependencies,andthat'swhatwe'redoing.

Theresultisthatanexpresscommandisinstalledinthe./node_modules/.bindirectory:

$lsnode_modules/.bin/

express

Runtheexpresscommandlikeso:

$./node_modules/.bin/express--help

Usage:express[options][dir]

Options:

-h,--helpoutputusageinformation

-V,--versionoutputtheversionnumber

-e,--ejsaddejsenginesupport(defaultstojade)

--hbsaddhandlebarsenginesupport

-H,--hoganaddhogan.jsenginesupport

-c,--css<engine>addstylesheet<engine>support

(less|stylus|compass|sass)(defaultstoplaincss)

--gitadd.gitignore

-f,--forceforceonnon-emptydirectory

Weprobablydon'twanttotype./node_modules/.bin/expresseverytimeweruntheexpress-generatorapplicationor,forthatmatter,anyoftheotherapplicationsthatprovidecommand-lineutilities.ReferbacktothediscussioninChapter3,Node.jsModulesaboutaddingthatdirectorytothePATHvariable.

Nowthatyou'veinstalledexpress-generatorinthefibonaccidirectory,useit

tosetuptheblankframeworkapplication:

$./node_modules/.bin/express--view=hbs--git.

destinationisnotempty,continue?[y/N]y

create:.

create:./package.json

create:./app.js

create:./.gitignore

create:./public

create:./routes

create:./routes/index.js

create:./routes/users.js

create:./views

create:./views/index.hbs

create:./views/layout.hbs

create:./views/error.hbs

create:./bin

create:./bin/www

create:./public/javascripts

create:./public/images

create:./public/stylesheets

create:./public/stylesheets/style.css

installdependencies:

$cd.&&npminstall

runtheapp:

$DEBUG=fibonacci:*npmstart

$npmuninstallexpress-generator

added83packagesandremoved5packagesin4.104s

Thiscreatedabunchoffilesforus,whichwe'llwalkthroughinaminute.Thenode_modulesdirectorystillhastheexpress-generatormodule,whichisnownotuseful.Wecanjustleaveitthereandignoreit,orwecanaddittothedevDependenciesofthepackage.jsonitgenerated.Alternatively,wecanuninstallitasshownhere.

Thenextthingtodoisruntheblankapplicationinthewaywe'retold.Thecommandshown,npmstart,reliesonasectionofthesuppliedpackage.jsonfile:

"scripts":{

"start":"node./bin/www"

},

Thenpmtoolsupportsscriptsthatarewaystoautomatevarioustasks.We'llusethiscapabilitythroughoutthebooktodovariousthings.WhentheTwelve-FactorApplicationmodelsuggestsautomatingallyouradministrativetasks,thenpmscriptsfeatureisanexcellentmechanismtodoso.MostnpmscriptsarerunwiththenpmrunscriptNamecommand,butthestartcommandisexplicitlyrecognizedbynpmandcanberunasshownpreviously.

Thestepsare:

1. Installthedependenciesnpminstall.2. Starttheapplicationusingnpmstart.3. Optionallymodifypackage.jsontoalwaysrunwithdebugging.

Toinstallthedependencies,andruntheapplication,typethesecommands:

$npminstall

$DEBUG=fibonacci:*npmstart

>fibonacci@0.0.0startUsersDavid/chap04/fibonacci

>node./bin/www

fibonacci:serverListeningonport3000+0ms

SettingtheDEBUGvariablethiswayturnsonsomedebuggingoutput,whichincludesthismessageaboutlisteningonport3000.Otherwise,wearen'ttoldthisinformation.Thissyntaxiswhat'susedintheBashshelltorunacommandwithanenvironmentvariable.Ifyougetanerrortryrunningjust"npmstart"thenreadthenextsection.

Wecanmodifythesuppliednpmstartscripttoalwaysruntheappwithdebuggingenabled.Changethescriptssectiontothefollowing:

"scripts":{

"start":"DEBUG=fibonacci:*node./bin/www"

},

Sincetheoutputsaysitislisteningonport3000,wedirectourbrowsertohttp://localhost:3000/andseethefollowingoutput:

SettingenvironmentvariablesinWindowscmd.execommandlineIfyou'reonWindowsthepreviousexamplemayhavefailedwithanerrorthatDEBUGisnotaknowncommand.TheproblemisthattheWindowsshell,thecmd.exeprogram,doesnotsupporttheBashcommand-linestructure.

AddingVARIABLE=valueatthebeginningofacommand-lineisspecifictosomeshells,likeBash,onLinuxandmacOS.Itsetsthatenvironmentvariableonlyforthecommand-linebeingexecuted,andisaveryconvenientwaytotemporarilyoverrideenvironmentvariablesforaspecificcommand.

Clearlyasolutionisrequiredifyourpackage.jsonistobeusableacrossdifferentoperatingsystems.

Thebestsolutionappearstobethecross-envpackageinthenpmrepository,see:https://www.npmjs.com/package/cross-envWiththispackageinstalled,commandsinthescriptssectioninpackage.jsoncansetenvironmentvariablesjustasinBashonLinux/macOS.Theusagelookslikeso:

"scripts":{

"start":"cross-envDEBUG=fibonacci:*node./bin/www"

},

"dependencies":{

...

"cross-env":"5.1.x"

}

Thenthecommandisexecutedasso:

C:\Users\david\Documents\chap04\fibonacci>npminstall

...outputfrominstallingpackages

C:\Users\david\Documents\chap04\fibonacci>npmrunstart

>fibonacci@0.0.0startC:\Users\david\Documents\chap04\fibonacci

>cross-envDEBUG=fibonacci:*node./bin/www

fibonacci:serverListeningonport3000+0ms

GET/30490.597ms--

GET/stylesheets/style.css30414.480ms--

GET/fibonacci20084.726ms-503

GET/stylesheets/style.css3044.465ms--

GET/fibonacci?fibonum=225001069.049ms-327

GET/stylesheets/style.css3042.601ms--

WalkingthroughthedefaultExpressapplicationWehaveaworking,blankExpressapplication;let'slookatwhatwasgeneratedforus.We'redoingthistofamiliarizeourselveswithExpressbeforedivingintostartcodingourFibonacciapplication.

Becauseweusedthe--view=hbsoption,thisapplicationissetuptousetheHandlebars.jstemplateengine.HandlebarswasbuiltontopofMustache,andwasoriginallydesignedforuseinthebrowser;formoreinformationseeitshomepageathttp://handlebarsjs.com/.TheversionshownherehasbeenpackagedforusewithExpress,andisdocumentedathttps://github.com/pillarjs/hbs.

Generallyspeaking,atemplateenginemakesitpossibletoinsertdataintogeneratedwebpages.TheExpressJSWikihasalistoftemplateenginesforExpresshttps://github.com/expressjs/express/wiki#template-engines.

Theviewsdirectorycontainstwofiles,error.hbsandindex.hbs.ThehbsextensionisusedforHandlebarsfiles.Anotherfile,layout.hbs,isthedefaultpagelayout.Handlebarshasseveralwaystoconfigurelayouttemplatesandevenpartials(snippetsofcodewhichcanbeincludedanywhere).

Theroutesdirectorycontainstheinitialroutingsetup,thatis,thecodetohandlespecificURLs.We'llmodifytheselater.

Thepublicdirectorywillcontainassetsthattheapplicationdoesn'tgenerate,butaresimplysenttothebrowser.What'sinitiallyinstalledisaCSSfile,public/stylesheets/style.css.

Thepackage.jsonfilecontainsourdependenciesandothermetadata.

Thebindirectorycontainsthewwwscriptthatwesawearlier.That'saNode.jsscript,whichinitializestheHTTPServerobjects,startsitlisteningonaTCPport,andcallsthelastfilewe'lldiscuss,app.js.ThesescriptsinitializeExpress,hookuptheroutingmodules,anddootherthings.

There'salotgoingoninthewwwandapp.jsscripts,solet'sstartwiththeapplicationinitialization.Let'sfirsttakealookatacoupleoflinesinapp.js:

varexpress=require('express');

...

varapp=express();

...

module.exports=app;

Thismeansthatapp.jsisamodulethatexportstheobjectreturnedbytheexpressmodule.Itdoesn'tstarttheHTTPserverobject,however.

Now,let'sturntothewwwscript.Thefirstthingtoseeisthatitstartswiththisline:

#!/usr/bin/envnode

ThisisaUnix/Linuxtechniquetomakeacommandscript.Itsaystorunthefollowingasascriptusingthenodecommand.Inotherwords,wehaveNode.jscodeandwe'reinstructingtheoperatingsystemtoexecutethatcodeusingtheNode.jsruntime:

$ls-lbin/www

-rwx------1davidstaff1595Feb51970bin/www

Wecanalsoseethatthescriptwasmadeexecutablebyexpress-generator.

Itcallstheapp.jsmoduleasfollows:

varapp=require('../app');

...

varport=normalizePort(process.env.PORT||'3000');

app.set('port',port);

...

varserver=http.createServer(app);

...

server.listen(port);

server.on('error',onError);

server.on('listening',onListening);

Weseewhereport3000comesfrom;it'saparametertothenormalizePortfunction.WealsoseethatsettingthePORTenvironmentvariablewilloverridethedefaultport3000.Andfinally,weseethattheHTTPServerobjectiscreatedhere,andistoldtousetheapplicationinstancecreatedinapp.js.Tryrunningthefollowingcommand:

$PORT=4242DEBUG=fibonacci:*npmstart

Theapplicationnowtellsyouthatit'slisteningonport4242,whereyoucanponderthemeaningoflife.

Theappobjectisnextpassedtohttp.createServer().AlookintheNode.jsdocumentationtellsusthisfunctiontakesarequestListener,whichissimplyafunctionthattakestherequestandresponseobjectswe'veseenpreviously.Therefore,theappobjectissuchafunction.

Finally,thewwwscriptstartstheserverlisteningontheportwespecified.

Let'snowwalkthroughapp.jsinmoredetail:

app.set('views',path.join(__dirname,'views'));

app.set('viewengine','hbs');

ThistellsExpresstolookfortemplatesintheviewsdirectoryandtousetheEJStemplatingengine.

Theapp.setfunctionisusedforsettingapplicationproperties.It'llbeusefultobrowsetheAPIdocumentationaswegothrough(http://expressjs.com/en/4x/api.html).

Nextisaseriesofapp.usecalls:

app.use(logger('dev'));

app.use(bodyParser.json());

app.use(bodyParser.urlencoded({extended:false}));

app.use(cookieParser());

app.use(express.static(path.join(__dirname,'public')));

app.use('',routes);

app.use('users',users);

Theapp.usefunctionmountsmiddlewarefunctions.ThisisanimportantpieceofExpressjargonwewilldiscussshortly.Atthemoment,let'ssaythatmiddlewarefunctionsareexecutedduringtheprocessingofroutes.Thismeansallthefeaturesnamedhereareenabledinapp.js:

LoggingisenabledusingtheMorganrequestlogger.Visithttps://www.npmjs.com/package/morganforitsdocumentation.

Thebody-parsermodulehandlesparsingHTTPrequestbodies.Visithttps://www.npmjs.com/package/body-parserforitsdocumentation.

Thecookie-parsermoduleisusedtoparseHTTPcookies.Visithttps://www.npmjs.com/package/cookie-parserforitsdocumentation.

Astaticfilewebserverisconfiguredtoservetheassetfilesinthepublicdirectory.

Tworoutermodules,routesandusers,tosetupwhichfunctionshandlewhichURLs.

TheExpressmiddlewareLet'sroundoutthewalkthroughofapp.jsbydiscussingwhatmiddlewarefunctionsdoforourapplication.Wehaveanexampleattheendofthescript:

app.use(function(req,res,next){

varerr=newError('Notfound');

err.status=404;

next(err);

});

Thecommentsayscatch404andforwardtoerrorhandler.Asyouprobablyknow,anHTTP404statusmeanstherequestedresourcewasnotfound.Weneedtotelltheusertheirrequestwasn'tsatisfied,andmaybeshowthemapictureofaflockofbirdspullingawhaleoutoftheocean.Thisisthefirststepindoingso.Beforegettingtothelaststepofreportingthiserror,youmustlearnhowmiddlewareworks.

Wedohaveamiddlewarefunctionrightinfrontofus.Refertoitsdocumentationathttp://expressjs.com/en/guide/writing-middleware.html.

Middlewarefunctionstakethreearguments.Thefirsttwo,requestandresponse,areequivalenttotherequestandresponseoftheNode.jsHTTPrequestobject.However,Expressexpandstheobjectswithadditionaldataandcapabilities.Thelast,next,isacallbackfunctioncontrollingwhentherequest-responsecycleends,anditcanbeusedtosenderrorsdownthemiddlewarepipeline.

Theincomingrequestgetshandledbythefirstmiddlewarefunction,thenthenext,thenthenext,andsoon.Eachtimetherequestistobepasseddownthechainofmiddlewarefunctions,thenextfunctioniscalled.Ifnextiscalledwithanerrorobject,asshownhere,anerrorisbeingsignaled.

Otherwise,thecontrolsimplypassestothenextmiddlewarefunctioninthechain.

Whathappensifnextisnotcalled?TheHTTPrequestwillhangbecausenoresponsehasbeengiven.Amiddlewarefunctiongivesaresponsewhenitcallsfunctionsontheresponseobject,suchasres.sendorres.render.

Forexample,considertheinclusionofapp.js:

app.get('/',function(req,res){res.send('HelloWorld!');});

Thisdoesnotcallnext,butinsteadcallsres.send.Thisisthecorrectmethodofendingtherequest-responsecycle,bysendingaresponse(res.send)totherequest.Ifneithernextnorres.sendiscalled,therequestnevergetsaresponse.

Hence,amiddlewarefunctiondoesoneofthefollowingfourthings:

Executesitsownbusinesslogic.Therequestloggermiddlewareshownearlierisanexample.

Modifiestherequestorresponseobjects.Boththebody-parserandcookie-parserdoso,lookingfordatatoaddtotherequestobject.

Callsnexttoproceedtothenextmiddlewarefunctionorelsesignalsanerror.

Sendsaresponse,endingthecycle.

Theorderingofmiddlewareexecutiondependsontheorderthey'readdedtotheappobject.Thefirstaddedisexecutedfirst,andsoon.

MiddlewareandrequestpathsWe'veseentwokindsofmiddlewarefunctionssofar.Inone,thefirstargumentisthehandlerfunction.Intheother,thefirstargumentisastringcontainingaURLsnippet,andthesecondargumentisthehandlerfunction.

What'sactuallygoingonisapp.usehasanoptionalfirstargument:thepaththemiddlewareismountedon.ThepathisapatternmatchagainsttherequestURL,andthegivenfunctionistriggerediftheURLmatchesthepattern.There'sevenamethodtosupplynamedparametersintheURL:

app.use('userprofile/:id',function(req,res,next){

userProfiles.lookup(req.params.id,(err,profile)=>{

if(err)returnnext(err);

//dosomethingwiththeprofile

//Suchasdisplayittotheuser

res.send(profile.display());

});

});

Thispathspecificationhasapattern,:id,andthevaluewilllandinreq.params.id.Inthisexample,we'resuggestingauserprofilesservice,andthatforthisURLwewanttodisplayinformationaboutthenameduser.

AnotherwaytouseamiddlewarefunctionisonaspecificHTTPrequestmethod.Withapp.use,anyrequestwillbematched,butintruth,GETrequestsaresupposedtobehavedifferentlytoPOSTrequests.Youcallapp.METHODwhereMETHODmatchesoneoftheHTTPrequestverbs.Thatis,app.getmatchestheGETmethod,app.postmatchesPOST,andsoon.

Finally,wegettotherouterobject.ThisisakindofmiddlewareusedexplicitlyforroutingrequestsbasedontheirURL.Takealookatroutes/users.js:

varexpress=require('express');

varrouter=express.Router();

router.get('/',function(req,res,next){

res.send('respondwitharesource');

});

module.exports=router;

Wehaveamodulewhoseexportsobjectisarouter.Thisrouterhasonlyoneroute,butitcanhaveanynumberofroutesyouthinkisappropriate.

Backinapp.js,thisisaddedasfollows:

app.use('/users',users);

Allthefunctionswediscussedfortheappobjectapplytotherouterobject.Iftherequestmatches,therouterisgiventherequestforitsownchainofprocessingfunctions.AnimportantdetailisthattherequestURLprefixisstrippedwhentherequestispassedtotherouterinstance.

You'llnoticethattherouter.getinusers.jsmatches'/'andthatthisrouterismountedon'/users'.Ineffect,thatrouter.getmatches/usersaswell,butbecausetheprefixwasstripped,itspecifies'/'instead.Thismeansaroutercanbemountedondifferentpathprefixeswithouthavingtochangetherouterimplementation.

ErrorhandlingNow,wecanfinallygetbacktothegeneratedapp.js,the404Errorpagenotfound,andanyothererrorstheapplicationmightwanttoshowtotheuser.

Amiddlewarefunctionindicatesanerrorbypassingavaluetothenextfunctioncall.OnceExpressseesanerror,itwillskipanyremainingnon-errorrouting,anditwillonlypassittoerrorhandlersinstead.Anerrorhandlerfunctionhasadifferentsignaturethanwhatwesawearlier.

Inapp.js,whichwe'reexamining,thisisourerrorhandler:

app.use(function(err,req,res,next){

res.status(err.status||500);

res.render('error',{

message:err.message,

error:{}

});

});

Errorhandlerfunctionstakefourparameters,witherraddedtothefamiliarreq,res,andnext.Forthishandler,weuseres.statustosettheHTTPresponsestatuscode,andweuseres.rendertoformatanHTMLresponseusingtheviews/error.hbstemplate.Theres.renderfunctiontakesdata,renderingitwithatemplatetoproduceHTML.

Thismeansanyerrorinourapplicationwilllandhere,bypassinganyremainingmiddlewarefunctions.

CalculatingtheFibonaccisequencewithanExpressapplicationTheFibonaccinumbersaretheintegersequence:0,1,1,2,3,5,8,13,21,34,...

Eachentryinthelististhesumoftheprevioustwoentriesinthelist.Thesequencewasinventedin1202byLeonardoofPisa,whowasalsoknownasFibonacci.OnemethodtocalculateentriesintheFibonaccisequenceistherecursivealgorithmweshowedearlier.WewillcreateanExpressapplicationthatusestheFibonacciimplementationandthenexploreseveralmethodstomitigateperformanceproblemsincomputationallyintensivealgorithms.

Let'sstartwiththeblankapplicationwecreatedinthepreviousstep.WehadyounamethatapplicationFibonacciforareason.Wewerethinkingahead.

Inapp.js,makethefollowingchangestothetopportionofthefile:

constexpress=require('express');

consthbs=require('hbs');

constpath=require('path');

constfavicon=require('serve-favicon');

constlogger=require('morgan');

constcookieParser=require('cookie-parser');

constbodyParser=require('body-parser');

constindex=require('./routes/index');

constfibonacci=require('./routes/fibonacci');

constapp=express();

//viewenginesetup

app.set('views',path.join(__dirname,'views'));

app.set('viewengine','hbs');

hbs.registerPartials(path.join(__dirname,'partials'));

//uncommentafterplacingyourfaviconin/public

//app.use(favicon(path.join(__dirname,'public','favicon.ico')));

app.use(logger('dev'));

app.use(bodyParser.json());

app.use(bodyParser.urlencoded({extended:false}));

app.use(cookieParser());

app.use(express.static(path.join(__dirname,'public')));

app.use('/',index);

app.use('/fibonacci',fibonacci);

Mostofthisiswhatexpress-generatorgaveus.Thevarstatementshavebeenchangedtoconst,forthatlittleteensybitofextracomfort.Weexplicitlyimportedthehbsmodulesowecoulddosomeconfiguration.AndweimportedaroutermoduleforFibonacci,whichwe'llseeinaminute.

FortheFibonacciapplication,wedon'tneedtosupportusers,andthereforedeletedthatroutingmodule.Thefibonaccimodule,whichwe'llshownext,servestoqueryanumberforwhichwe'llcalculatetheFibonaccinumber.

Inthetop-leveldirectory,createafile,math.js,containingthisextremelysimplisticFibonacciimplementation:

exports.fibonacci=function(n){

if(n===0)return0;

elseif(n===1||n===2)return1;

elsereturnexports.fibonacci(n-1)+exports.fibonacci(n-2);

};

Intheviewsdirectory,lookatthefilenamedlayout.hbswhichexpress-generatorcreated:

<!DOCTYPEhtml>

<html>

<head>

<title>{{title}}</title>

<linkrel='stylesheet'href='stylesheetsstyle.css'/>

</head>

<body>

{{{body}}}

</body>

</html>

Thisfilecontainsthestructurewe'lluseforHTMLpages.GoingbytheHandlebarssyntax,weseethat{{title}}appearswithintheHTMLtitletag.Itmeanswhenwecallres.render,weshouldsupplyatitleattribute.The{{{body}}}tagiswheretheviewtemplatecontentlands.

Changeviews/index.hbstojustcontainthefollowing:

<h1>{{title}}</h1>

{{>navbar}}

Thisservesasthefrontpageofourapplication.Itwillbeinsertedinplaceof{{{body}}}inlayout.hbs.Themarker,{{>navbar}},referstoapartialnamednavbar.Earlier,weconfiguredadirectorynamedpartialstoholdpartials.Nowlet'screateafile,partials/navbar.html,containing:

<divclass='navbar'>

<p><ahref=''>home<a>|<ahref='fibonacci'>Fibonacci's<a></p>

</div>

Thiswillserveasanavigationbarthat'sincludedoneverypage.

Createafile,views/fibonacci.hbs,containingthefollowingcode:

<h1>{{title}}</h1>

{{>navbar}}

{{#iffiboval}}

<p>Fibonaccifor{{fibonum}}is{{fiboval}}</p>

<hr/>

{{/if}}

<p>Enteranumbertoseeits'Fibonaccinumber</p>

<formname='fibonacci'action='/fibonacci'method='get'>

<inputtype='text'name='fibonum'/>

<inputtype='submit'value='Submit'/>

</form>

Rememberthatthefilesinviewsaretemplatesintowhichdataisrendered.TheyservetheViewaspectoftheModel-View-Controller(MVC)paradigm,hencethedirectoryname.

Intheroutesdirectory,deletetheuser.jsmodule.ItisgeneratedbytheExpressframework,butwewillnotuseitinthisapplication.

Inroutes/index.js,changetherouterfunctiontothefollowing:

/*GEThomepage.*/

router.get('/',function(req,res,next){

res.render('index',{title:"WelcometotheFibonacciCalculator"});

});

Theanonymousobjectpassedtores.rendercontainsthedatavaluesweprovidetothelayoutandviewtemplates.

Then,finally,intheroutesdirectory,createafilenamedfibonacci.jscontainingthefollowingcode:

constexpress=require('express');

constrouter=express.Router();

constmath=require('../math');

router.get('',function(req,res,next){

if(req.query.fibonum){

/Calculatedirectlyinthisserver

res.render('fibonacci',{

title:"CalculateFibonaccinumbers",

fibonum:req.query.fibonum,

fiboval:math.fibonacci(req.query.fibonum)

});

}else{

res.render('fibonacci',{

title:"CalculateFibonaccinumbers",

fiboval:undefined

});

}

});

module.exports=router;

Thepackage.jsonisalreadysetupsowecanusenpmstarttorunthescript

andalwayshavedebuggingmessagesenabled.Andnowwe'rereadytodoso:

$npmstart

>fibonacci@0.0.0startUsersdavid/chap04/fibonacci

>DEBUG=fibonacci:*node./bin/www

fibonacci:serverListeningonport3000+0ms

Asitsuggests,youcanvisithttp://localhost:3000/andseewhatwehave:

Thispageisrenderedfromtheviews/index.hbstemplate.SimplyclickontheFibonacci'slinktogotothenextpage,whichisofcourserenderedfromtheviews/fibonacci.hbstemplate.Onthatpage,you'llbeabletoenteranumber,clickontheSubmitbutton,andgetananswer(hint:pickanumberbelow40ifyouwantyouranswerinareasonableamountoftime):

Let'swalkthroughtheapplicationtodiscusshowitworks.

Therearetworoutesinapp.js:theroutefor/,whichishandledbyroutes/index.js,andtheroutefor/fibonacci,whichishandledbyroutes/fibonacci.js.

Theres.renderfunctionrendersthenamedtemplateusingtheprovideddatavaluesandemitstheresultasanHTTPresponse.Forthehomepageofthisapplication,therenderingcode(routes/index.js)andtemplate(views/index.hbs)aren'tmuch,anditisontheFibonaccipagewherealltheactionishappening.

Theviews/fibonacci.hbstemplatecontainsaforminwhichtheuserentersanumber.BecauseitisaGETform,whentheuserclicksontheSubmitbutton,thebrowserwillissueanHTTPGETonthe/fibonacciURL.WhatdistinguishesoneGETon/fibonaccifromanotheriswhethertheURLcontainsaqueryparameternamedfibonum.Whentheuserfirstentersthepage,thereisnofibonumandhencenothingtocalculate.AftertheuserhasenteredanumberandclickedonSubmit,thereisafibonumandsomethingtocalculate.

Expressautomaticallyparsesthequeryparameters,makingthemavailable

asreq.query.Thatmeansroutes/fibonacci.jscanquicklycheckwhetherthereisafibonum.Ifthereis,itcallsthefibonaccifunctiontocalculatethevalue.

Earlier,weaskedyoutoenteranumberlessthan40.Goaheadandenteralargernumber,suchas50,butgotakeacoffeebreakbecausethisisgoingtotakeawhiletocalculate.Orproceedontoreadingthenextsectionwherewestarttodiscussuseofcomputationallyintensivecode.

ComputationallyintensivecodeandtheNode.jseventloopThisFibonacciexampleispurposelyinefficienttodemonstrateanimportantconsiderationforyourapplications.WhathappenstotheNode.jseventloopwhenrunninglongcomputations?Toseetheeffect,opentwobrowserwindows,eachopenedtotheFibonaccipage.Inone,enterthenumber55orgreater,andintheother,enter10.Notethatthesecondwindowfreezes,andifyouleaveitrunninglongenough,theanswerwilleventuallypopupinbothwindows.What'shappeningistheNode.jseventloopisblockedfromprocessingeventsbecausetheFibonaccialgorithmisrunninganddoesnoteveryieldtotheeventloop.

SinceNode.jshasasingleexecutionthread,processingrequestsdependonrequesthandlersquicklyreturningtotheeventloop.Normally,theasynchronouscodingstyleensuresthattheeventloopexecutesregularly.

Thisistrueevenforrequeststhatloaddatafromaserverhalfwayaroundtheglobe,becausetheasynchronousI/Oisnon-blockingandcontrolisquicklyreturnedtotheeventloop.ThenaïveFibonaccifunctionwechosedoesn'tfitintothismodelbecauseit'salong-runningblockingoperation.ThistypeofeventhandlerpreventsthesystemfromprocessingrequestsandstopsNode.jsfromdoingwhatit'smeanttodo,namelytobeablisteringlyfastwebserver.

Inthiscase,thelong-response-timeproblemisobvious.ResponsetimequicklyescalatestothepointwhereyoucantakeavacationtoTibetandperhapsgetreincarnatedasallamainPeruduringthetimeittakestorespondwiththeFibonaccinumber!

Toseethismoreclearly,createafilenamedfibotimes.jscontainingthefollowingcode:

constmath=require('./math');

constutil=require('util');

for(varnum=1;num<80;num++){

letnow=newDate().toISOString();

console.log(`${now}Fibonaccifor${num}=${math.fibonacci(num)}`);

}

Nowrunit.Youwillgetthefollowingoutput:

$nodefibotimes.js

2017-12-10T23:04:42.342ZFibonaccifor1=1

2017-12-10T23:04:42.345ZFibonaccifor2=1

2017-12-10T23:04:42.345ZFibonaccifor3=2

2017-12-10T23:04:42.345ZFibonaccifor4=3

2017-12-10T23:04:42.345ZFibonaccifor5=5

...

2017-12-10T23:04:42.345ZFibonaccifor10=55

2017-12-10T23:04:42.345ZFibonaccifor11=89

2017-12-10T23:04:42.345ZFibonaccifor12=144

2017-12-10T23:04:42.345ZFibonaccifor13=233

2017-12-10T23:04:42.345ZFibonaccifor14=377

...

2017-12-10T23:04:44.072ZFibonaccifor40=102334155

2017-12-10T23:04:45.118ZFibonaccifor41=165580141

2017-12-10T23:04:46.855ZFibonaccifor42=267914296

2017-12-10T23:04:49.723ZFibonaccifor43=433494437

2017-12-10T23:04:54.218ZFibonaccifor44=701408733

...

2017-12-10T23:06:07.531ZFibonaccifor48=4807526976

2017-12-10T23:07:08.056ZFibonaccifor49=7778742049

^C

Thisquicklycalculatesthefirst40orsomembersoftheFibonaccisequence,butafterthe40thmember,itstartstakingacoupleofsecondsperresultandquicklydegradesfromthere.Itisuntenabletoexecutecodeofthissortonasingle-threadedsystemthatreliesonaquickreturntotheeventloop.Awebservicecontainingsuchcodewouldgivepoorperformancetotheusers.

TherearetwogeneralwaystosolvethisprobleminNode.js:

Algorithmicrefactoring:Perhaps,liketheFibonaccifunctionwechose,oneofyouralgorithmsissuboptimalandcanberewrittentobefaster.Or,ifnotfaster,itcanbesplitintocallbacksdispatchedthroughtheeventloop.We'lllookatonesuchmethodinamoment.

Creatingabackendservice:CanyouimagineabackendserverdedicatedtocalculatingFibonaccinumbers?Okay,maybenot,butit'squitecommontoimplementbackendserverstooffloadworkfromfrontendservers,andwewillimplementabackendFibonacciserverattheendofthischapter.

AlgorithmicrefactoringToprovethatwehaveanartificialproblemonourhands,hereisamuchmoreefficientFibonaccifunction:

exports.fibonacciLoop=function(n){

varfibos=[];

fibos[0]=0;

fibos[1]=1;

fibos[2]=1;

for(vari=3;i<=n;i++){

fibos[i]=fibos[i-2]+fibos[i-1];

}

returnfibos[n];

}

Ifwesubstituteacalltomath.fibonacciLoopinplaceofmath.fibonacci,thefibotimesprogramrunsmuchfaster.Eventhisisn'tthemostefficientimplementation;forexample,asimpleprewiredlookuptableismuchfasteratthecostofsomememory.

Editfibotimes.jsasfollowsandrerunthescript.Thenumberswillflybysofastyourheadwillspin:

for(varnum=1;num<8000;num++){

letnow=newDate().toISOString();

console.log(`${now}Fibonaccifor${num}=${math.fibonacciLoop(num)}`);

}

Somealgorithmsaren'tsosimpletooptimizeandstilltakealongtimetocalculatetheresult.Inthissection,we'reexploringhowtohandleinefficientalgorithms,andthereforewillstickwiththeinefficientFibonacciimplementation.

Itispossibletodividethecalculationintochunksandthendispatchthe

computationofthosechunksthroughtheeventloop.Addthefollowingcodetomath.js:

exports.fibonacciAsync=function(n,done){

if(n===0)done(undefined,0);

elseif(n===1||n===2)done(undefined,1);

else{

setImmediate(()=>{

exports.fibonacciAsync(n-1,(err,val1)=>{

if(err)done(err);

elsesetImmediate(()=>{

exports.fibonacciAsync(n-2,(err,val2)=>{

if(err)done(err);

elsedone(undefined,val1+val2);

});

});

});

});

}

};

Thisconvertsthefibonaccifunctionfromasynchronousfunctiontoatraditionalcallback-orientedasynchronousfunction.We'reusingsetImmediateateachstageofthecalculationtoensuretheeventloopexecutesregularlyandthattheservercaneasilyhandleotherrequestswhilechurningawayonacalculation.Itdoesnothingtoreducethecomputationrequired;thisisstillthesilly,inefficientFibonaccialgorithm.Allwe'vedoneisspreadthecomputationthroughtheeventloop.

Infibotimes.js,wecanusethis:

constmath=require('./math');

constutil=require('util');

(async()=>{

for(varnum=1;num<8000;num++){

awaitnewPromise((resolve,reject)=>{

math.fibonacciAsync(num,(err,fibo)=>{

if(err)reject(err);

else{

letnow=newDate().toISOString();

console.log(`${now}Fibonaccifor${num}=

${fibo}`);

resolve();

}

})

})

}

})().catch(err=>{console.error(err);});

Thisversionoffibotimes.jsexecutesthesame,wesimplytypenodefibotimes.However,usingfibonacciAsyncwillrequirechangesintheserver.

Becauseit'sanasynchronousfunction,wewillneedtochangeourroutercode.Createanewfile,namedroutes/fibonacci-async1.js,containingthefollowing:

constexpress=require('express');

constrouter=express.Router();

constmath=require('../math');

router.get('/',function(req,res,next){

if(req.query.fibonum){

//Calculateusingasync-awarefunction,inthisserver

math.fibonacciAsync(req.query.fibonum,(err,fiboval)=>{

res.render('fibonacci',{

title:"CalculateFibonaccinumbers",

fibonum:req.query.fibonum,

fiboval:fiboval

});

});

}else{

res.render('fibonacci',{

title:"CalculateFibonaccinumbers",

fiboval:undefined

});

}

});

module.exports=router;

Thisisthesameasearlier,justrewrittenforanasynchronousFibonaccicalculation.

Inapp.js,makethischangetotheapplicationwiring:

//constfibonacci=require('./routes/fibonacci');

constfibonacci=require('./routes/fibonacci-async1');

Withthischange,theservernolongerfreezeswhencalculatingalargeFibonaccinumber.Thecalculationofcoursestilltakesalongtime,butatleastotherusersoftheapplicationaren'tblocked.

Youcanverifythisbyagainopeningtwobrowserwindowsintheapplication.Enter60inonewindow,andintheotherstartrequestingsmallerFibonaccinumbers.Unlikewiththeoriginalfibonaccifunction,usingfibonacciAsyncallowsbothwindowstogiveanswers,thoughifyoureallydidenter60inthefirstwindowyoumightaswelltakethatthree-monthvacationtoTibet:

It'suptoyou,andyourspecificalgorithms,tochoosehowtobestoptimizeyourcodeandtohandleanylong-runningcomputationsyoumayhave.

MakingHTTPClientrequestsThenextwaytomitigatecomputationallyintensivecodeistopushthecalculationtoabackendprocess.Toexplorethatstrategy,we'llrequestcomputationsfromabackendFibonacciserver,usingtheHTTPClientobjecttodoso.However,beforewelookatthat,let'sfirsttalkingeneralaboutusingtheHTTPClientobject.

Node.jsincludesanHTTPClientobject,usefulformakingHTTPrequests.IthasthecapabilitytoissueanykindofHTTPrequest.Inthissection,we'llusetheHTTPClientobjecttomakeHTTPrequestssimilartocallingaRepresentationalStateTransfer(REST)webservice.

Let'sstartwithsomecodeinspiredbythewgetorcurlcommandstomakeHTTPrequestsandshowtheresults.Createafilenamedwget.jscontainingthiscode:

consthttp=require('http');

consturl=require('url');

constutil=require('util');

constargUrl=process.argv[2];

constparsedUrl=url.parse(argUrl,true);

//Theoptionsobjectispassedtohttp.request

//tellingittheURLtoretrieve

constoptions={

host:parsedUrl.hostname,

port:parsedUrl.port,

path:parsedUrl.pathname,

method:'GET'

};

if(parsedUrl.search)options.path+="?"+parsedUrl.search;

constreq=http.request(options);

//Invokedwhentherequestisfinished

req.on('response',res=>{

console.log('STATUS:'+res.statusCode);

console.log('HEADERS:'+util.inspect(res.headers));

res.setEncoding('utf8');

res.on('data',chunk=>{console.log('BODY:'+chunk);});

res.on('error',err=>{console.log('RESPONSEERROR:'+err);});

});

//Invokedonerrors

req.on('error',err=>{console.log('REQUESTERROR:'+err);});

req.end();

Youcanrunthescriptasfollows:

$nodewget.jshttp://example.com

STATUS:200

HEADERS:{'accept-ranges':'bytes',

'cache-control':'max-age=604800',

'content-type':'text/html',

date:'Sun,10Dec201723:40:44GMT',

etag:'"359670651"',

expires:'Sun,17Dec201723:40:44GMT',

'last-modified':'Fri,09Aug201323:54:35GMT',

server:'ECS(rhv/81A7)',

vary:'Accept-Encoding',

'x-cache':'HIT',

'content-length':'1270',

connection:'close'}

BODY:<!doctypehtml>

<html>

...

There'smoreintheprintout,namelytheHTMLofthepageathttp://example.com/.Thepurposeofwget.jsistomakeanHTTPrequestandshowyouvoluminousdetailsoftheresponse.AnHTTPrequestisinitiatedwiththehttp.requestmethod,asfollows:

varhttp=require('http');

varoptions={

host:'example.com',

port:80,

path:null,

method:'GET'

};

varrequest=http.request(options);

request.on('response',response=>{

...

});

Theoptionsobjectdescribestherequesttomake,andthecallbackfunctioniscalledwhentheresponsearrives.Theoptionsobjectisfairlystraightforward,withthehost,port,andpathfieldsspecifyingtheURLbeingrequested.ThemethodfieldmustbeoneoftheHTTPverbs(GET,PUT,POST,andsoon).YoucanalsoprovideaheadersarrayfortheheadersintheHTTPrequest.Forexample,youmightneedtoprovideacookie:

varoptions={

headers:{'Cookie':'..cookievalue'}

};

TheresponseobjectisitselfanEventEmitter,whichemitsthedataanderrorevents.Thedataeventiscalledasdataarrives,andtheerroreventis,ofcourse,calledonerrors.

TherequestobjectisaWritableStream,whichisusefulforHTTPrequestscontainingdata,suchasPUTorPOST.Thismeanstherequestobjecthasawritefunctionthatwritesdatatotherequester.ThedataformatinanHTTPrequestisspecifiedbythestandardMultipurposeInternetMailExtensions(MIME)originallycreatedtogiveusbetteremail.Around1992,theWWWcommunityworkedwiththeMIMEstandardcommitteewhichwasdevelopingaformatformultipart,multi-media-richelectronicmail.Receivingfancy-lookingemailissocommonplacetodaythatonemightnotbeawarethatemailusedtobeplaintext.MIME-typesweredevelopedtodescribetheformatofeachpieceofdata,andtheWWWcommunityadoptedthisforuseontheweb.HTMLformswillpostwithaContent-Typeofmultipart/form-data,forexample.

CallingaRESTbackendservicefromanExpressapplicationNowthatwe'veseenhowtomakeHTTPclientrequests,wecanlookathowtomakeaRESTqueryinsideanExpresswebapplication.WhatthateffectivelymeansistomakeanHTTPGETrequesttoabackendserver,whichrespondswiththeFibonaccinumberrepresentedbytheURL.Todoso,we'llrefactortheFibonacciapplicationtomakeaFibonacciserverthatiscalledfromtheapplication.WhilethisisoverkillforcalculatingFibonaccinumbers,itletsuslookatthebasicsofimplementingamultitierapplicationstackinExpress.

Inherently,callingaRESTserviceisanasynchronousoperation.ThatmeanscallingtheRESTservicewillinvolveafunctioncalltoinitiatetherequestandacallbackfunctiontoreceivetheresponse.RESTservicesareaccessedoverHTTP,sowe'llusetheHTTPclientobjecttodoso.

ImplementingasimpleRESTserverwithExpressWhileExpresshasapowerfultemplatingsystem,makingitsuitablefordeliveringHTMLwebpagestobrowsers,itcanalsobeusedtoimplementasimpleRESTservice.TheparameterizedURLsweshowedearlier(/user/profile/:id)canactlikeparameterstoaRESTcall.AndExpressmakesiteasytoreturndataencodedinJSON.

Now,createafilenamedfiboserver.jscontainingthiscode:

constmath=require('./math');

constexpress=require('express');

constlogger=require('morgan');

constapp=express();

app.use(logger('dev'));

app.get('fibonacci:n',(req,res,next)=>{

math.fibonacciAsync(Math.floor(req.params.n),(err,val)=>{

if(err)next('FIBOSERVERERROR'+err);

elseres.send({n:req.params.n,result:val});

});

});

app.listen(process.env.SERVERPORT);

Thisisastripped-downExpressapplicationthatgetsrighttothepointofprovidingaFibonaccicalculationservice.TheonerouteitsupportshandlestheFibonaccicomputationusingthesamefunctionswe'vealreadyworkedwith.

Thisisthefirsttimewe'veseenres.sendused.It'saflexiblewaytosendresponseswhichcantakeanarrayofheadervalues(fortheHTTPresponseheader),andanHTTPstatuscode.Asusedhere,itautomaticallydetectstheobject,formatsitasJSONtext,andsendsitwiththecorrectContent-Type.

Inpackage.json,addthistothescriptssection:

"server":"SERVERPORT=3002node./fiboserver"

ThisautomateslaunchingourFibonacciservice.

Notethatwe'respecifyingtheTCP/IPportviaanenvironmentvariableandusingthatvariableintheapplication.ThisisanotheraspectoftheTwelve-Factorapplicationmodel:toputconfigurationdataintheenvironment.

Now,let'srunit:

$npmrunserver

>fibonacci@0.0.0serverUsersDavid/chap04/fibonacci

>SERVERPORT=3002node./fiboserver

Then,inaseparatecommandwindow,wecanusethecurlprogramtomakesomerequestsagainstthisservice:

$curl-fhttp://localhost:3002fibonacci10

{"n":"10","result":55}

$curl-fhttp://localhost:3002fibonacci11

{"n":"11","result":89}

$curl-fhttp://localhost:3002fibonacci12

{"n":"12","result":144}

Overinthewindowwheretheserviceisrunning,we'llseealogofGETrequestsandhowlongeachtooktoprocess:

$npmrunserver

>fibonacci@0.0.0serverUsersDavid/chap04/fibonacci

>SERVERPORT=3002node./fiboserver

GETfibonacci102000.393ms-22

GETfibonacci112000.647ms-22

GETfibonacci122000.772ms-23

Now,let'screateasimpleclientprogram,fiboclient.js,toprogrammaticallycalltheFibonacciservice:

consthttp=require('http');

[

"fibonacci30","fibonacci20","fibonacci10",

"fibonacci9","fibonacci8","fibonacci7",

"fibonacci6","fibonacci5","fibonacci4",

"fibonacci3","fibonacci2","fibonacci1"

].forEach(path=>{

console.log(`${newDate().toISOString()}requesting${path}`);

varreq=http.request({

host:"localhost",

port:process.env.SERVERPORT,

path:path,

method:'GET'

},res=>{

res.on('data',chunk=>{

console.log(`${newDate().toISOString()}BODY:${chunk}`);

});

});

req.end();

});

Then,inpackage.json,addthistothescriptssection:

"scripts":{

"start":"node./bin/www",

"server":"SERVERPORT=3002node./fiboserver",

"client":"SERVERPORT=3002node./fiboclient"

}

Thenruntheclientapp:

$npmrunclient

>fibonacci@0.0.0clientUsersDavid/chap04/fibonacci

>SERVERPORT=3002node./fiboclient

2017-12-11T00:41:14.857Zrequestingfibonacci30

2017-12-11T00:41:14.864Zrequestingfibonacci20

2017-12-11T00:41:14.865Zrequestingfibonacci10

2017-12-11T00:41:14.865Zrequestingfibonacci9

2017-12-11T00:41:14.866Zrequestingfibonacci8

2017-12-11T00:41:14.866Zrequestingfibonacci7

2017-12-11T00:41:14.866Zrequestingfibonacci6

2017-12-11T00:41:14.866Zrequestingfibonacci5

2017-12-11T00:41:14.866Zrequestingfibonacci4

2017-12-11T00:41:14.866Zrequestingfibonacci3

2017-12-11T00:41:14.867Zrequestingfibonacci2

2017-12-11T00:41:14.867Zrequestingfibonacci1

2017-12-11T00:41:14.884ZBODY:{"n":"9","result":34}

2017-12-11T00:41:14.886ZBODY:{"n":"10","result":55}

2017-12-11T00:41:14.891ZBODY:{"n":"6","result":8}

2017-12-11T00:41:14.892ZBODY:{"n":"7","result":13}

2017-12-11T00:41:14.893ZBODY:{"n":"8","result":21}

2017-12-11T00:41:14.903ZBODY:{"n":"3","result":2}

2017-12-11T00:41:14.904ZBODY:{"n":"4","result":3}

2017-12-11T00:41:14.905ZBODY:{"n":"5","result":5}

2017-12-11T00:41:14.910ZBODY:{"n":"2","result":1}

2017-12-11T00:41:14.911ZBODY:{"n":"1","result":1}

2017-12-11T00:41:14.940ZBODY:{"n":"20","result":6765}

2017-12-11T00:41:18.200ZBODY:{"n":"30","result":832040}

We'rebuildingourwaytowardaddingtheRESTservicetothewebapplication.Atthispoint,we'veprovedseveralthings,oneofwhichistheabilitytocallaRESTserviceinourprogram.

Wealsoinadvertentlydemonstratedanissuewithlong-runningcalculations.You'llnoticetherequestsweremadefromthelargesttothesmallest,buttheresultsappearedinaverydifferentorder.Why?It'sbecauseoftheprocessingtimeforeachrequest,andtheinefficientalgorithmwe'reusing.Thecomputationtimeincreasesenoughtoensurethatthelargerrequestvaluesrequireenoughprocessingtimetoreversetheorder.

Whathappensisthatfiboclient.jssendsallitsrequestsrightaway,andtheneachonewaitsfortheresponsetoarrive.BecausetheserverisusingfibonacciAsync,itwillworkoncalculatingallresponsessimultaneously.Thevaluesthatarequickesttocalculatearetheonesthatwillbereadyfirst.Astheresponsesarriveintheclient,thematchingresponsehandlerfires,andinthiscase,theresultprintstotheconsole.Theresultswillarrivewhenthey'rereadyandnotamillisecondsooner.

RefactoringtheFibonacciapplicationforRESTNowthatwe'veimplementedaREST-basedserver,wecanreturntotheFibonacciapplication,applyingwhatwe'velearnedtoimproveit.Wewillliftsomeofthecodefromfiboclient.jsandtransplantitintotheapplicationtodothis.Createanewfile,routes/fibonacci-rest.js,withthefollowingcode:

constexpress=require('express');

constrouter=express.Router();

consthttp=require('http');

constmath=require('../math');

router.get('/',function(req,res,next){

if(req.query.fibonum){

varhttpreq=http.request({

host:"localhost",

port:process.env.SERVERPORT,

path:"fibonacci"+Math.floor(req.query.fibonum),

method:'GET'

});

httpreq.on('response',response=>{

response.on('data',chunk=>{

vardata=JSON.parse(chunk);

res.render('fibonacci',{

title:"CalculateFibonaccinumbers",

fibonum:req.query.fibonum,

fiboval:data.result

});

});

response.on('error',err=>{next(err);});

});

httpreq.on('error',err=>{next(err);});

httpreq.end();

}else{

res.render('fibonacci',{

title:"CalculateFibonaccinumbers",

fiboval:undefined

});

}

});

module.exports=router;

Inapp.js,makethischange:

constindex=require('./routes/index');

//constfibonacci=require('./routes/fibonacci');

//constfibonacci=require('./routes/fibonacci-async1');

//constfibonacci=require('./routes/fibonacci-await');

constfibonacci=require('./routes/fibonacci-rest');

Then,inpackage.json,changethescriptsentrytothefollowing:

"scripts":{

"start":"DEBUG=fibonacci:*node./bin/www",

"startrest":"DEBUG=fibonacci:*SERVERPORT=3002node./bin/www",

"server":"DEBUG=fibonacci:*SERVERPORT=3002node./fiboserver",

"client":"DEBUG=fibonacci:*SERVERPORT=3002node./fiboclient"

},

HowcanwehavethesamevalueforSERVERPORTforallthreescriptsentries?Theansweristhatthevariableisuseddifferentlyindifferentplaces.Instartrest,thatvariableisusedinroutes/fibonacci-rest.jstoknowatwhichporttheRESTserviceisrunning.Likewise,inclient,fiboclient.jsusesthatvariableforthesamepurpose.Finally,inserver,thefiboserver.jsscriptusestheSERVERPORTvariabletoknowwhichporttolistenon.

Instartandstartrest,novalueisgivenforPORT.Inbothcases,bin/wwwdefaultstoPORT=3000ifitisnotspecified.

Inonecommandwindow,startthebackendserver,andintheother,starttheapplication.Openabrowserwindowasbefore,andmakeafewrequests.Youshouldseeoutputsimilartothis:

$npmrunserver

>fibonacci@0.0.0serverUsersDavid/chap04/fibonacci

>DEBUG=fibonacci:*SERVERPORT=3002node./fiboserver

GETfibonacci3420021124.036ms-27

GETfibonacci122001.578ms-23

GETfibonacci162006.600ms-23

GETfibonacci2020033.980ms-24

GETfibonacci282001257.514ms-26

Theoutputlikethisfortheapplication:

$npmrunstartrest

>fibonacci@0.0.0startrestUsersDavid/chap04/fibonacci

>DEBUG=fibonacci:*SERVERPORT=3002node./bin/www

fibonacci:serverListeningonport3000+0ms

GET/fibonacci?fibonum=3420021317.792ms-548

GET/stylesheets/style.css30420.952ms--

GET/fibonacci?fibonum=12304109.516ms--

GET/stylesheets/style.css3040.465ms--

GET/fibonacci?fibonum=1620083.067ms-544

GET/stylesheets/style.css3040.900ms--

GET/fibonacci?fibonum=20200221.842ms-545

GET/stylesheets/style.css3040.778ms--

GET/fibonacci?fibonum=282001428.292ms-547

GET/stylesheets/style.css30419.083ms--

Becausewehaven'tchangedthetemplates,thescreenwilllookexactlyasitdidearlier.

Wemayrunintoanotherproblemwiththissolution.TheasynchronousimplementationofourinefficientFibonaccialgorithmmaycausetheFibonacciserviceprocesstorunoutofmemory.IntheNode.jsFAQ,https://github.com/nodejs/node/wiki/FAQ,it'ssuggestedtousethe--max_old_space_sizeflag.You'daddthisinpackage.jsonasfollows:

"server":"SERVERPORT=3002node./fiboserver--max_old_space_size5000",

However,theFAQalsosaysthatifyou'rerunningintomaximummemoryspaceproblems,yourapplicationshouldprobablyberefactored.Thisgets

backtoourpointseveralpagesagothatthereareseveralapproachestoaddressingperformanceproblems,oneofwhichisthealgorithmicrefactoringofyourapplication.

WhygotothetroubleofdevelopingthisRESTserverwhenwecouldjustdirectlyusefibonacciAsync?

WecannowpushtheCPUloadforthisheavyweightcalculationtoaseparateserver.DoingsowouldpreserveCPUcapacityonthefrontendserversoitcanattendtowebbrowsers.GPUco-processorsarenowwidelyusedfornumericalcomputingandcanbeaccessedviaasimplenetworkAPI.Theheavycomputationcanbekeptseparate,andyoucanevendeployaclusterofbackendserverssittingbehindaloadbalancer,evenlydistributingrequests.Decisionslikethisaremadeallthetimetocreatemultitiersystems.

Whatwe'vedemonstratedisthatit'spossibletoimplementsimplemultitierRESTservicesinafewlinesofNode.jsandExpress.ThewholeexercisegaveusachancetothinkaboutcomputationallyintensivecodeinNode.js.

SomeRESTfulmodulesandframeworksHereareafewavailablepackagesandframeworkstoassistyourREST-basedprojects:

Restify(>http://restify.com/):Thisoffersbothclient-sideandserver-sideframeworksforbothendsofRESTtransactions.Theserver-sideAPIissimilartoExpress.

Loopback(http://loopback.io/):ThisisanofferingfromStrongLoop,thecurrentsponsoroftheExpressproject.Itoffersalotoffeaturesandis,ofcourse,builtontopofExpress.

SummaryYoulearnedalotinthischapteraboutNode'sHTTPsupport,implementingwebapplications,andevenRESTserviceimplementation.

Nowwecanmoveontoimplementingamorecompleteapplication:onefortakingnotes.WewillusetheNotesapplicationforseveralupcomingchaptersasavehicletoexploretheExpressapplicationframework,databaseaccess,deploymenttocloudservicesoronyourownserver,anduserauthentication.

Inthenextchapter,wewillbuildthebasicinfrastructure.

YourFirstExpressApplicationNowthatwe'vegotourfeetwetbuildinganExpressapplicationforNode.js,let'sworkonanapplicationthatperformsausefulfunction.Theapplicationwe'llbuildwillkeepalistofnotes,anditwillletusexploresomeaspectsofarealapplication.

Inthischapter,we'llonlybuildthebasicinfrastructureoftheapplication,andinthelaterchapters,we'llextendtheapplicationconsiderably.

Thetopicscoveredinthischapterincludes

UsingPromisesandasyncfunctionsinExpressrouterfunctions

ApplyingtheMVCparadigmtoExpressapplications

BuildinganExpressapplication

JavaScriptClassdefinitions

ImplementingtheCRUDparadigm

Handlebarstemplates

Promises,asyncfunctions,andExpressrouterfunctionsBeforewegetintodevelopingourapplication,wemusttakeadeeperlookatapairofnewES-2015/2016/2017featuresthatcollectivelyrevolutionizeJavaScriptprogramming:ThePromiseclassandasyncfunctions.Bothareusedfordeferredandasynchronouscomputationandcanmakeintenselynestedcallbackfunctionsathingofthepast:

APromiserepresentsanoperationthathasn'tcompletedyetbutisexpectedtobecompletedinthefuture.We'veseenPromisesinuse.The.thenor.catchfunctionsareinvokedwhenthepromisedresult(orerror)isavailable.

Generatorfunctionsareanewkindoffunctionthatcanbepausedandresumed,andcanreturnresultsfromthemiddleofthefunction.

Thosetwofeaturesweremixedwithanother,theiterationprotocol,alongwithsomenewsyntax,tocreateasyncfunctions.

Themagicofasyncfunctionsisthatwecanwriteasynchronouscodeasifit'ssynchronouscode.It'sstillasynchronouscode,meaninglong-runningrequesthandlerswon'tblocktheeventloop.Thecodelookslikethesynchronouscodewe'dwriteinotherlanguages.Onestatementfollowsanother,theerrorsarethrownasexceptions,andtheresultslandonthenextlineofcode.Promiseandasyncfunctionsaresomuchofanimprovementthatit'sextremelycompellingfortheNode.jscommunitytoswitchparadigms,meaningrewritinglegacycallback-orientedAPIs.

Overtheyears,severalotherapproacheshavebeenusedtomanageasynchronouscode,andyoumaycomeacrosscodeusingtheseothertechniques.BeforethePromiseobjectwasstandardized,atleasttwoimplementationswereavailable:Bluebird(http://bluebirdjs.com/)andQ(https://www.npmjs.com/package/q).Useofanon-standardPromiselibraryshouldbecarefullyconsidered,sincethereisvalueinmaintainingcompatibilitywiththestandardPromiseobject.

ThePyramidofDoomisnamedaftertheshapethecodetakesafterafewlayersofnesting.Anymultistageprocesscanquicklyescalatetocodenested15levelsdeep.Considerthefollowingexample:

router.get('pathto/something',(req,res,next)=>{

doSomething(arg1,arg2,(err,data1)=>{

if(err)returnnext(err);

doAnotherThing(arg3,arg2,data1,(err2,data2)=>{

if(err2)returnnext(err2);

somethingCompletelyDifferent(arg1,arg42,(err3,data3)=>{

if(err3)returnnext(err3);

doSomethingElse((err4,data4)=>{

if(err4)returnnext(err4);

res.render('page',{data});

});

});

});

});

});

Rewritingthisasanasyncfunctionwillmakethismuchclearer.Togetthere,weneedtoexaminethefollowingideas:

UsingPromisestomanageasynchronousresults

GeneratorfunctionsandPromises

asyncfunctions

WegenerateaPromisethisway:

exports.asyncFunction=function(arg1,arg2){

returnnewPromise((resolve,reject)=>{

//performsometaskorcomputationthat'sasynchronous

//foranyerrordetected:

if(errorDetected)returnreject(dataAboutError);

//Whenthetaskisfinished

resolve(theResult);

});

};

NotethatasyncFunctionisanasynchronousfunction,butitdoesnottakeacallback.Instead,itreturnsaPromiseobject,andtheasynchronouscodeisexecutedwithinacallbackpassedtothePromiseclass.

Yourcodemustindicatethestatusoftheasynchronousoperationviatheresolveandrejectfunctions.Asimpliedbythefunctionnames,rejectindicatesanerroroccurredandresolveindicatesasuccessresult.Yourcallerthenusesthefunctionasfollows:

asyncFunction(arg1,arg2)

.then((result)=>{

//theoperationsucceeded

//dosomethingwiththeresult

returnnewResult;

})

.catch(err=>{

//anerroroccurred

});

Thesystemisfluidenoughthatthefunctionpassedina.thencanreturnsomething,suchasanotherPromise,andyoucanchainthe.thencallstogether.Thevaluereturnedina.thenhandler(ifany)becomesanewPromiseobject,andinthiswayyoucanconstructachainof.thenand.catchcallstomanageasequenceofasynchronousoperations.

Asequenceofasynchronousoperationswouldbeimplementedasachainof.thenfunctions,aswewillseeinthenextsection.

notes.read(req.query.key).then(note=>{returnfilterNote(note);}).then(note=>{returnswedishChefSpeak(note);}).then(note=>{

res.render('noteview',{

title:note?note.title:"",notekey:req.query.key,note:note

});

})

.catch(err=>{next(err);});

Thereareseveralplaceswhereerrorscanoccurinthislittlebitofcode.Thenotes.readfunctionhasseveralpossiblefailuremodes:thefilterNotefunctionmightwanttoraiseanalarmifitdetectsacross-sitescriptingattack.TheSwedishchefcouldbeonstrike.Therecouldbeafailureinres.renderorthetemplatebeingused.Butwehaveonlyonewaytocatchandreporterrors.Arewemissingsomething?

ThePromiseclassautomaticallycaptureserrors,sendingthemdownthechainofoperationsattachedtothePromise.IfthePromiseclasshasanerroronitshands,itskipsoverthe.thenfunctionsandwillinsteadinvokethefirst.catchfunctionitfinds.Inotherwords,usinginstancesofPromiseprovidesahigherassuranceofcapturingandreportingerrors.Withtheolderconvention,errorreportingwastrickier,anditwaseasytoforgettoaddcorrecterrorhandling.

FlatteningourasynchronouscodeTheproblembeingaddressedisthatasynchronouscodinginJavaScriptresultsinthePyramidofDoom.Toexplain,let'sreiteratetheexampleRyanDahlgaveastheprimaryNode.jsidiom:

db.query('SELECT..etc..',function(err,resultSet){

if(err){

//Instead,errorsarrivehere

}else{

//Instead,resultsarrivehere

}

});

//WeWANTtheerrorsorresultstoarrivehere

Thegoalwastoavoidblockingtheeventloopwithalongoperation.DeferringtheprocessingofresultsorerrorsusingcallbackfunctionswasanexcellentsolutionandisthefoundingidiomofNode.js.Theimplementationofcallbackfunctionsledtothispyramid-shapedproblem.Namely,thatresultsanderrorslandinthecallback.Ratherthandeliveringthemtothenextlineofcode,theerrorsandresultsareburied.

Promiseshelpflattenthecodesothatitnolongertakesapyramidalshape.Theyalsocaptureerrors,ensuringdeliverytoausefullocation.Butthoseerrorsandresultsarestillburiedinsideananonymousfunctionanddonotgetdeliveredtothenextlineofcode.

Further,usingPromisesresultsinalittlebitofboilerplatecodethatobscurestheprogrammersintent.It'slessboilerplatethanwithregularcallbackfunctions,buttheboilerplateisstillthere.

Fortunately,theECMAScriptcommitteekeptworkingontheproblem.

PromisesandgeneratorsbirthedasyncfunctionsGeneratorsandtheassociatedIterationProtocolarealargetopic,whichwewillbrieflycover.

TheIterationProtocoliswhat'sbehindthenewfor..ofloop,andsomeothernewloopingconstructs.Theseconstructscanbeusedwithanythingproducinganiterator.Formoreaboutboth,seehttps://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols.

Ageneratorisakindoffunctionwhichcanbestoppedandstartedusingtheyieldkeyword.Generatorsproduceaniteratorwhosevaluesarewhateverisgiventotheyieldstatement.Formoreonthis,seehttps://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Generator.

Considerthis:

$catgen.js

function*gen(){

yield1;

yield2;

yield3;

yield4;

}

for(letgofgen()){

console.log(g);

}

$nodegen.js

1

2

3

4

Theyieldstatementcausesageneratorfunctiontopauseandtoprovidethe

valuegiventothenextcallonitsnextfunction.Thenextfunctionisn'texplicitlyseenhere,butiswhatcontrolstheloop,andispartoftheiterationprotocol.Insteadoftheloop,trycallinggen().next()severaltimes:

vargeniter=gen();

console.log(geniter.next());

console.log(geniter.next());

console.log(geniter.next());

You'llseethis:

$nodegen.js

{value:1,done:false}

{value:2,done:false}

{value:3,done:false}

TheIterationprotocolsaystheiteratorisfinishedwhendoneistrue.Inthiscase,wedidn'tcallitenoughtotriggertheendstateoftheiterator.

WheregeneratorsbecameinterestingiswhenusedwithfunctionsthatreturnaPromise.ThePromiseiswhat'smadeavailablethroughtheiterator.ThecodeconsumingtheiteratorcanwaitonthePromisetogetitsvalue.Aseriesofasynchronousoperationscouldbeinsidethegeneratorandinvokedinaniterablefashion.

Withthehelpofanextrafunction,ageneratorfunctionalongwithPromise-returningasynchronousfunctionscanbeaverynicewaytowriteasynchronouscode.WesawanexampleofthisinChapter2,SettingupNode.js,whileexploringBabel.Babelhasaplugintorewriteasyncfunctionsintoageneratoralongwithahelperfunction,andwetookalookatthetranspiledcodeandthehelperfunction.Thecolibrary(https://www.npmjs.com/package/co)isapopularhelperfunctionforimplementingasynchronouscodingingenerators.Createafilenamed2files.js:

constfs=require('fs-extra');

constco=require('co');

constutil=require('util');

co(function*(){

vartexts=[

yieldfs.readFile('hello.txt','utf8'),

yieldfs.readFile('goodbye.txt','utf8')

];

console.log(util.inspect(texts));

});

Thenrunitlikeso:

$node2files.js

['Hello,world!\n','Goodbye,world!\n']

Normally,fs.readFilesendsitsresulttoacallbackfunction,andwe'dbuildalittlepyramid-shapedpieceofcodetoperformthistask.Thefs-extramodulecontainsimplementationsofallfunctionsfromthebuilt-infsmodulebutchangedtoreturnaPromiseinsteadofacallbackfunction.Therefore,eachfs.readFileshownhereisreturningaPromisethat'sresolvedwhenthefilecontentisfullyreadintomemory.WhatcodoesisitmanagesthedanceofwaitingforthePromisetoberesolved(orrejected),andreturnsthevalueofthePromise.Therefore,withtwosuitabletextfileswehavetheresultshownfromexecuting2files.js.

Theimportantthingisthatthecodeisverycleanandreadable.Wearen'tcaughtupinboilerplatecoderequiredtomanageasynchronousoperations.Theintentoftheprogrammerisprettyclear.

asyncfunctionstakethatsamecombinationofgeneratorsandPromisesanddefineastandardizedsyntaxintheJavaScriptlanguage.Createafilenamed2files-async.js:

constfs=require('fs-extra');

constutil=require('util');

asyncfunctiontwofiles(){

vartexts=[

awaitfs.readFile('hello.txt','utf8'),

awaitfs.readFile('goodbye.txt','utf8')

];

console.log(util.inspect(texts));

}

twofiles().catch(err=>{console.error(err);});

Thenrunitlikeso:

$node2files-async.js

['Hello,world!\n','Goodbye,world!\n']

Clean.Readable.Theintentoftheprogrammerisclear.Nodependencyonanadd-onlibrary,withsyntaxbuilt-intotheJavaScriptlanguage.Mostimportantly,everythingishandledinanaturalway.Errorsareindicatednaturallybythrowingexceptions.Theresultsofanasynchronousoperationnaturallyappearastheresultoftheoperation,withtheawaitkeywordfacilitatingthedeliveryofthatresult.

Toseetherealadvantage,let'sreturntothePyramidofDoomexamplefromearlier:

router.get('pathto/something',async(req,res,next)=>{

try{

letdata1=awaitdoSomething(req.query.arg1,req.query.arg2);

letdata2=awaitdoAnotherThing(req.query.arg3,req.query.arg2,

data1);

letdata3=awaitsomethingCompletelyDifferent(req.query.arg1,

req.query.arg42);

letdata4=awaitdoSomethingElse();

res.render('page',{data1,data2,data3,data4});

}catch(err){

next(err);

}

});

Otherthanthetry/catch,thisexamplebecameverycleancomparedtoitsformasacallbackpyramid.Alltheasynchronouscallbackboilerplateiserased,andtheintentoftheprogrammershinesclearly.

Whywasthetry/catchneeded?Normally,anasyncfunctioncatchesthrownerrors,automaticallyreportingthemcorrectly.ButsincethisexampleiswithinanExpressrouterfunction,we'relimitedbyitscapabilities.Expressdoesn'tknowhowtorecognizeanasyncfunction,andthereforeitdoesnotknowtolookforthethrownerrors.Instead,we'rerequiredtocatchthem

andcallnext(err).

Thisimprovementisonlyforcodeexecutinginsideanasyncfunction.CodeoutsideanasyncfunctionstillrequirescallbacksorPromisesforasynchronouscoding.Further,thereturnvalueofanasyncfunctionisaPromise.

Refertotheofficialspecificationofasyncfunctionsathttps://tc39.github.io/ecmascript-asyncawait/fordetails.

ExpressandtheMVCparadigmExpressdoesn'tenforceanopiniononhowyoushouldstructuretheModel,View,andControllermodulesofyourapplication,orwhetheryoushouldfollowanykindofMVCparadigmatall.Aswelearnedinthepreviouschapter,theblankapplicationcreatedbytheExpressGeneratorprovidestwoaspectsoftheMVCmodel:

Theviewsdirectorycontainstemplatefiles,controllingthedisplayportion,correspondingtotheView.

TheroutesdirectorycontainscodeimplementingtheURLsrecognizedbytheapplicationandcoordinatingtheresponsetoeachURL.Thiscorrespondstothecontroller.

Thisleavesyouwonderingwheretoputcodecorrespondingtothemodel.Modelsholdtheapplicationdata,changingitasinstructedbythecontrollerandsupplyingdatarequestedbyViewcode.Ataminimum,theModelcodeshouldbeinseparatemodulesfromtheControllercode.Thisistoensureacleanseparationofconcerns,forexample,toeasetheunittestingofeach.

Theapproachwe'lluseistocreateamodelsdirectoryasasiblingoftheviewsandroutesdirectories.Themodelsdirectorywillholdmodulesforstoringthenotesandrelateddata.TheAPIofthemodulesinthemodelsdirectorywillprovidefunctionstocreate,read,update,ordeletedataitemsCreate,Read,Update,andDeleteorDestroy(CRUDmodel)andotherfunctionsnecessaryfortheViewcodetodoitsthing.

TheCRUDmodel(create,read,update,destroy)isthefourbasicoperationsofpersistentdatastorage.TheNotesapplicationisstructuredas

aCRUDapplicationtodemonstrateimplementingeachoftheseoperations.

We'llusefunctionsnamedcreate,read,update,anddestroytoimplementeachofthebasicoperations.

We'reusingtheverbdestroyratherthandelete,becausedeleteisareservedwordinJavaScript.

<strong>$mkdirnotes</strong><br/><strong>$cdnotes</strong><br/><strong>$npminstallexpress-generator@4.x</strong><br/><strong>$./node_modules/.bin/express--view=hbs--git.</strong><br/><strong>destinationisnotempty,continue?[y/N]y</strong><br/><br/><strong>create:.</strong><br/><strong>create:./package.json</strong><br/><strong>create:./app.js</strong><br/><strong>create:./.gitignore</strong><br/><strong>create:./public</strong><br/><strong>create:./routes</strong><br/><strong>create:./routes/index.js</strong><br/><strong>create:./routes/users.js</strong><br/><strong>create:./views</strong><br/><strong>create:./views/index.hbs</strong><br/><strong>create:./views/layout.hbs</strong><br/><strong>create:./views/error.hbs</strong><br/><strong>create:./bin</strong><br/><strong>create:./bin/www</strong><br/><strong>create:./public/stylesheets</strong><br/><strong>create:./public/stylesheets/style.css</strong><br/><br/><strong>installdependencies:</strong><br/><strong>$cd.&&npminstall</strong><br/><br/><strong>runtheapp:</strong><br/><strong>$DEBUG=notes:*npmstart</strong><br/><br/><strong>create:./public/javascripts</strong><br/><strong>create:./public/images</strong><br/><strong>$npminstall</strong><br/><strong>added82packagesandremoved5packagesin97.188s</strong><br/><strong>$npmuninstallexpress-generator</strong><br/><strong>uptodatein8.325s</strong>

Ifyouwish,youcanrunnpmstartandviewtheblankapplicationinyourbrowser.Instead,let'smoveontosettingupthecode.

YourfirstNotesmodelCreateadirectorynamedmodels,asasiblingoftheviewsandroutesdirectories.

Then,createafilenamedNote.jsinthatdirectory,andputthiscodeinit:

constnotekey=Symbol('key');

constnotetitle=Symbol('title');

constnotebody=Symbol('body');

module.exports=classNote{

constructor(key,title,body){

this[notekey]=key;

this[notetitle]=title;

this[notebody]=body;

}

getkey(){returnthis[notekey];}

gettitle(){returnthis[notetitle];}

settitle(newTitle){this[notetitle]=newTitle;}

getbody(){returnthis[notebody];}

setbody(newBody){this[notebody]=newBody;}

};

Thisdefinesanewclass,Note,forusewithinourNotesapplication.Theintentistoholddatarelatedtonotesbeingexchangedbetweenusersofourapplication.

UnderstandingES-2015classdefinitionsThissortofobjectclassdefinitionisnewtoJavaScriptwithES-2015.ItsimplifiesdefiningclassesoverpreviousmethodsandbringsJavaScriptclassdefinitionsclosertothesyntaxinotherlanguages.Underthehood,JavaScriptclassesstilluseprototype-basedinheritance,butwithasimplersyntax,andthecoderdoesn'tevenhavetothinkabouttheobjectprototype.

Wecanreliablydeterminewhetheranobjectisanotewiththeinstanceofoperator:

$node

>constNote=require('./Note');

>typeofNote

'function'

>constaNote=newNote('foo','TheRainInSpain','Fallsmainlyonthe

plain');

>varnotNote={}

>notNoteinstanceofNote

false

>aNoteinstanceofNote

true

>typeofaNote

'object'

Thisshowsustheclearestmethodtoidentifyanobjectiswiththeinstanceofoperator.ThetypeofoperatorinformsusNoteisafunction(becauseoftheprototype-basedinheritancebehindthescenes),andthataninstanceoftheNoteclassisanobject.Withinstanceof,wecaneasilydeterminewhetheranobjectisaninstanceofagivenclass.

WiththeNoteclass,wehaveusedSymbolinstancestoprovideasmallmeasureofdatahiding.JavaScriptclassesdon'tprovideadata-hiding

mechanism—youcan'tlabelafieldprivateasyoucaninJava,forexample.It'susefultoknowhowtohideimplementationdetails.Thisisanimportantattributeofobject-orientedprogramming,becauseit'susefultohavethefreedomtochangetheimplementationatwill.Andthere'stheissueofcontrollingwhichcodecanmanipulatetheobject'sfields.

First,wedeclaredgetterandsetterfunctionstoprovideaccesstothevalues.Wewentovernormalgetter/setterusageinChapter4,HTTPServersandClients.

Accesstoagetter-basedfieldisbyusingthenameoftheproperty,andnotbycallingafunction-aNote.titleandnotaNote.title().Itlookslikeyou'reaccessinganobjectpropertybyassigningavalueoraccessingthevalue.Inactuality,thefunctiondefinedintheclassisexecutedoneveryaccess.Youcandefinearead-onlypropertybyonlyimplementingagetter,andnosetter,aswedidwiththekeyfield.

Therearesignificantdifferencesbetweentheprecedingandsimplydefininganonymousobjects:

{

key:'foo',title:'TheRaininSpain',

body:'Fallsmainlyontheplain'

}

WewritecodelikethatinJavaScriptallthetime.It'seasy,it'squick,andit'saveryfluidwaytosharedatabetweenfunctions.Butthere'snomeasureofhidingimplementationdetails,andnoclearidentificationofobjecttype.

IntheNoteclass,wecouldhaveusedthisconstructormethod:

classNote{

constructor(key,title,body){

this.key=key;

this.title=title;

this.body=body;

}

}

That'seffectivelythesameastheanonymousobject,inthatnodetailshavebeenhiddenandnocontrolisimplementedintermsofwhichcodecandowhattoobjectinstances.Theonlyadvantageoverananonymousobjectisusingtheinstanceofoperatortoidentifyobjectinstances.

ThemethodwechoseusestheSymbolclass,whichisalsonewwithES-2015.ASymbolisanopaqueobjectwithtwomainusecases:

Generatinguniquekeystouseaspropertyfields—asinthepreviousNoteclass

SymbolicidentifiersthatyoucanuseforconceptslikeCOLOR_RED

YoudefineaSymbolthroughafactorymethodthatgeneratesSymbolinstances:

>letsymfoo=Symbol('foo')

EachtimeyouinvoketheSymbolfactorymethod,anewanduniqueinstanceiscreated.Forexample,Symbol('foo')===Symbol('foo')isfalse,asissymfoo===Symbol('foo'),becauseanewinstanceiscreatedoneachsideoftheequalityoperator.However,symfoo===symfooistrue,becausetheyarethesameinstance.

Whatthismeansinpracticeisthatifwetryadirectapproachtoaccessafield,itfails:

>aNote[Symbol('title')]

undefined

RememberthateachtimeweusetheSymbolfactorymethodwegetanewinstance.ThenewinstanceofSymbol('title')isnotthesameinstanceused

withintheNote.jsmodule.

ThebottomlineisthatusingSymbolobjectsforthefieldsprovidesasmallmeasureofimplementationhiding.

Fillingoutthein-memoryNotesmodelCreateafilenamednotes-memory.jsinthemodelsdirectory,withthiscode:

constNote=require('./Note');

varnotes=[];

exports.update=exports.create=asyncfunction(key,title,body){

notes[key]=newNote(key,title,body);

returnnotes[key];

};

exports.read=asyncfunction(key){

if(notes[key])returnnotes[key];

elsethrownewError(`Note${key}doesnotexist`);

};

exports.destroy=asyncfunction(key){

if(notes[key]){

deletenotes[key];

}elsethrownewError(`Note${key}doesnotexist`);

};

exports.keylist=asyncfunction(){returnObject.keys(notes);};

exports.count=asyncfunction(){returnnotes.length;};

exports.close=asyncfunction(){}

Thisisasimplein-memorydatastorethat'sfairlyself-explanatory.ThekeyforeachNoteinstanceisusedastheindextoanarray,whichinturnholdstheNoteinstance.Simple,fast,andeasytoimplement.Itdoesnotsupportanylong-termdatapersistence.Anydatastoredinthismodelwilldisappearwhentheserveriskilled.

Wehaveusedasyncfunctionsbecauseinthefuturewe'llbestoringdatainthefilesystemorindatabases.Therefore,weneedanasynchronousAPI.

Thecreateandupdatefunctionsarebeinghandledbythesamefunction.AtthisstageoftheNotesapplication,thecodeforboththesefunctionscanbeexactlythesamebecausetheyperformtheexactsameoperation.Later,whenweadddatabasesupporttoNotes,thecreateandupdatefunctionswillneedtobedifferent.Forexample,inaSQLdatamodel,createwouldbeimplementedwithanINSERTINTOcommand,whileupdatewouldbeimplementedwithanUPDATEcommand.

TheNoteshomepageWe'regoingtomodifythestarterapplicationtosupportcreating,editing,updating,viewing,anddeletingnotes.Let'sstartbyfixingupthehomepage.Itshouldshowalistofnotes,andthetopnavigationbarshouldlinktoanADDNotepagesothatwecanalwaysaddanewnote.

Whilewewillbemodifyingthegeneratedapp.js,itneedsnomodificationtosupportthehomepage.Theselinesofcodearerelatedtothehomepage:

constindex=require('./routes/index');

..

app.use('/',index);

Additionally,tosupportHandlebarstemplatesapp.jsrequiresthesechanges:

consthbs=require('hbs');

...

app.set('viewengine','hbs');

hbs.registerPartials(path.join(__dirname,'partials'));

We'llputHandlebarspartialsinadirectory,partials,whichisasiblingto

theviewsdirectory.Changeroutes/index.jstothis:

constexpress=require('express');

constrouter=express.Router();

constnotes=require('../models/notes-memory');

/*GEThomepage.*/

router.get('/',async(req,res,next)=>{

letkeylist=awaitnotes.keylist();

letkeyPromises=keylist.map(key=>{

returnnotes.read(key)

});

letnotelist=awaitPromise.all(keyPromises);

res.render('index',{title:'Notes',notelist:notelist});

});

module.exports=router;

Thisgathersdataaboutthenotesthatwe'llbedisplayingonthehomepage.Bydefault,we'llshowasimpletableofnotetitles.Wedoneedtotalkaboutthetechnique.

ThePromise.allfunctionexecutesanarrayofPromises.ThePromisesareevaluatedinparallel,allowingourcodetopotentiallymakeparallelrequeststoaservice.Thisshouldexecutemorequicklythanmakingtherequestsoneatatimesequentially.

Wecouldhavewrittenasimpleforlooplikeso:

letkeylist=awaitnotes.keylist();

letnotelist=[];

for(keyofkeylist){

letnote=awaitnotes.read(keylist);

notelist.push({key:note.key,title:note.title});

}

Whilesimplertoread,thenotesareretrievedoneatatimewithnoopportunitytooverlapreadoperations.

ThePromisearrayisconstructedusingthemapfunction.Withmap,oneiteratesoveranarraytoproduceanewarray.Inthiscase,thenewarraycontainsthePromisesgeneratedbythenotes.readfunctioncalls.

BecausewewroteawaitPromise.all,thenotelistarraywillbecompletelyfilledwiththecorrectdataonceallthePromisessucceed.IfanyPromisefails—isrejected,inotherwords—anexceptionwillbethrowninstead.Whatwe'vedoneisenqueuealistofasynchronousoperationsandneatlywaitedforthemalltofinish.

Thenotelistarrayisthenpassedintotheviewtemplateswe'reabouttowrite.

Startwithviews/layout.hbs,containing:

<!DOCTYPEhtml>

<html>

<head>

<title>{{title}}</title>

<linkrel='stylesheet'href='stylesheetsstyle.css'/>

</head>

<body>

{{>header}}

{{{body}}}

</body>

</html>

Thisisthegeneratedfile,withtheadditionofapartialforthepageheader.Wealreadydeclaredpartialstoliveinthepartialsdirectory.Createpartials/header.hbs,containing:

<header>

<h1>{{title}}</h1>

<divclass='navbar'>

<p><ahref='/'>Home</a>|<ahref='notesadd'>ADDNote</a></p>

</div>

</header>

Changeviews/index.hbstothis:

{{#eachnotelist}}

<ul>

<li>{{key}}:

<ahref="notesview?key={{key}}">{{title}}</a>

</li>

</ul>

{{/each}}

Thissimplystepsthroughthearrayofnotedataandformatsasimplelisting.EachitemlinkstothenotesviewURLwithakeyparameter.Wehaveyettolookatthatcode,butthisURLwillobviouslydisplaythenote.

AnotherthingofnoteisthatnoHTMLforthelistisgeneratedifthenotelistisempty.

There'sofcourseawholelotmorethatcouldbeputintothis.Forexample,it'seasytoaddjQuerysupporttoeverypagejustbyaddingtheappropriatescripttagshere.

Wenowhaveenoughwrittentoruntheapplication;let'sviewthehomepage:

$DEBUG=notes:*npmstart

>notes@0.0.0startUsersDavid/chap05/notes

>node./bin/www

notes:serverListeningonport3000+0ms

GET/20087.300ms-308

GETstylesheetsstyle.css20027.744ms-111

Ifwevisithttp://localhost:3000,wewillseethefollowingpage:

Becausetherearen'tanynotes(yet),there'snothingtoshow.ClickingontheHomelinkjustrefreshesthepage.ClickingontheADDNotelink

throwsanerrorbecausewehaven't(yet)implementedthatcode.Thisshowsthattheprovidederrorhandlerinapp.jsisperformingasexpected.

Addinganewnote–createNow,let'slookatcreatingnotes.Becausetheapplicationdoesn'thavearouteconfiguredforthenotesaddURL,wemustaddone.Todothat,weneedacontrollerforthenotes.

Inapp.js,makethefollowingchanges.

Commentouttheselines:

//varusers=require('./routes/users');

..

//app.use('/users',users);

Atthisstage,theNotesapplicationdoesnotsupportusers,andtheseroutesarenotrequired.Thatwillchangeinafuturechapter.

Whatwereallyneedtodoisaddcodeforthenotescontroller:

//constusers=require('./routes/users');

constnotes=require('./routes/notes');

..

//app.use('users',users);

app.use('notes',notes);

Now,we'lladdaControllermodulecontainingthenotesrouter.Createafilenamedroutes/notes.js,withthiscontent:

constutil=require('util');

constexpress=require('express');

constrouter=express.Router();

constnotes=require('../models/notes-memory');

//AddNote.

router.get('/add',(req,res,next)=>{

res.render('noteedit',{

title:"AddaNote",

docreate:true,

notekey:"",note:undefined

});

});

module.exports=router;

TheresultingnotesaddURLcorrespondstothelinkinpartials/header.hbs.

Intheviewsdirectory,addatemplatenamednoteedit.hbs,containingthefollowing:

<formmethod='POST'action='notessave'>

<inputtype='hidden'name='docreate'value='<%=

docreate?"create":"update"%>'>

<p>Key:

{{#ifdocreate}}

<inputtype='text'name='notekey'value=''/>

{{else}}

{{#ifnote}}{{notekey}}{{/if}}

<inputtype='hidden'name='notekey'

value='{{#ifnote}}{{notekey}}{{/if}}'/>

{{/if}}

</p>

<p>Title:<inputtype='text'name='title'

value='{{#ifnote}}{{note.title}}{{/if}}'><p>

<br/><textarearows=5cols=40name='body'>

{{#ifnote}}{{note.body}}{{/if}}

</textarea>

<br/><inputtype='submit'value='Submit'/>

</form>

We'llbereusingthistemplatetosupportbotheditingnotesandcreatingnewones.

Noticethatthenoteandnotekeyobjectspassedtothetemplateareemptyinthiscase.Thetemplatedetectsthisconditionandensurestheinputareasareempty.Additionally,aflag,docreate,ispassedinsothattheformrecordswhetheritisbeingusedtocreateorupdateanote.Atthispoint,we'readdinganewnote,sononoteobjectexists.Thetemplatecodeisbeingwrittendefensivelytonotthrowerrors.

ThistemplateisaformthatwillPOSTitsdatatothenotessaveURL.Ifyouweretoruntheapplicationatthistime,itwouldgiveyouanerrormessagebecausenorouteisconfiguredforthatURL.

TosupportthenotessaveURL,addthistoroutes/notes.js:

//SaveNote(update)

router.post('/save',async(req,res,next)=>{

varnote;

if(req.body.docreate==="create"){

note=awaitnotes.create(req.body.notekey,

req.body.title,req.body.body);

}else{

note=awaitnotes.update(req.body.notekey,

req.body.title,req.body.body);

}

res.redirect('notesview?key='+req.body.notekey);

});

BecausethisURLwillalsobeusedforbothcreatingandupdatingnotes,itneedstodetectthedocreateflagandcalltheappropriatemodeloperation.

ThemodelreturnsaPromiseforbothnotes.createandnotes.update.Ofcourse,wemustcallthecorrespondingModelfunctionbasedonthedocreateflag.

ThisisaPOSToperationhandler.BecauseofthebodyParsermiddleware,theformdataisaddedtothereq.bodyobject.Thefieldsattachedtoreq.bodycorresponddirectlytoelementsintheHTMLform.

Now,wecanruntheapplicationagainandusetheAddaNoteform:

ButuponclickingontheSubmitbutton,wegetanerrormessage.Thereisn'tanything,yet,implementingthenotesviewURL.

YoucanmodifytheURLinthelocationboxtorevisithttp://localhost:3000,andyou'llseesomethinglikethefollowingscreenshotonthehomepage:

Thenoteisactuallythere;wejustneedtoimplementnotesview.Let'sgetonwiththat.

Viewingnotes–readNowthatwe'velookedathowtocreatenotes,weneedtomoveontoreadingthem.ThismeansimplementingcontrollerlogicandviewtemplatesforthenotesviewURL.

Toroutes/notes.js,addthisrouterfunction:

//ReadNote(read)

router.get('/view',async(req,res,next)=>{

varnote=awaitnotes.read(req.query.key);

res.render('noteview',{

title:note?note.title:"",

notekey:req.query.key,note:note

});

});

Becausethisrouteismountedonarouterhandling/notes,thisroutehandlesnotesview.

Ifnotes.readsuccessfullyreadsthenote,itisrenderedwiththenoteviewtemplate.Ifsomethinggoeswrong,we'llinsteaddisplayanerrortotheuserthroughExpress.

Totheviewsdirectory,addthenoteview.hbstemplate,referencedbythiscode:{{#ifnote}}<h3>{{note.title}}</h3>{{/if}}{{#ifnote}}<p>{{note.body}}</p>{{/if}}<p>Key:{{notekey}}</p>{{#ifnotekey}}<hr/><p><ahref="notesdestroy?key={{notekey}}">Delete</a>|<ahref="notesedit?key={{notekey}}">Edit</a></p>{{/if}}

Asexpected,withthiscode,theapplicationcorrectlyredirectstonotesview,andwecanseeourhandiwork.Also,asexpected,clickingoneithertheDeleteorEditlinkswillgiveyouanerror,becausethecodehasn'tbeenimplemented.

Thisisstraightforward:takingdataoutofthenoteobjectanddisplayingusingHTML.Atthebottomaretwolinks,onetonotesdestroytodeletethenoteandtheothertonotesedittoeditit.

Thecodeforneitheroftheseexistsatthemoment.Butthatwon'tstopusfromgoingaheadandexecutingtheapplication:

Editinganexistingnote–updateNowthatwe'velookedatthecreateandreadoperations,let'slookathowtoupdateoreditanote.

Toroutes/notes.js,addthisrouterfunction:

//Editnote(update)

router.get('/edit',async(req,res,next)=>{

varnote=awaitnotes.read(req.query.key);

res.render('noteedit',{

title:note?("Edit"+note.title):"AddaNote",

docreate:false,

notekey:req.query.key,note:note

});

});

We'rereusingthenoteedit.ejstemplate,becauseitcanbeusedforbothcreateandupdate/editoperations.Noticethatwepassfalsefordocreate,informingthetemplatethatitistobeusedforediting.

Inthiscase,wefirstretrievethenoteobjectandthenpassitthroughtothetemplate.Thisway,thetemplateissetupforeditingratherthannotecreation.WhentheuserclicksontheSubmitbutton,we'llendupinthesamenotessaveroutehandlershownintheprecedingscreenshot.Italreadydoestherightthing:callingthenotes.updatemethodinthemodelratherthannotes.create.

Becausethat'sallweneeddo,wecangoaheadandreruntheapplication:

ClickontheSubmitbuttonhere,andyouwillberedirectedtothenotesviewscreenandwillthenbeabletoreadthenewlyeditednote.Backtothenotesviewscreen:we'vejusttakencareoftheEditlink,buttheDeletelinkstillproducesanerror.

Deletingnotes–destroyNow,let'slookathowtoimplementthenotesdestroyURLtodeletenotes.

Toroutes/notes.js,addthefollowingrouterfunction:

//AsktoDeletenote(destroy)

router.get('/destroy',async(req,res,next)=>{

varnote=awaitnotes.read(req.query.key);

res.render('notedestroy',{

title:note?note.title:"",

notekey:req.query.key,note:note

});

});

Destroyinganoteisasignificantstepifonlybecausethere'snotrashcantoretrieveitfromifwemakeamistake.Therefore,wewanttoasktheuserwhetherthey'resuretheywanttodeletethatnote.Inthiscase,weretrievethenoteandthenrenderthefollowingpage,displayingaquestiontoensuretheydowanttodeletethenote.

Totheviewsdirectory,addanotedestroy.hbstemplate:<formmethod='POST'action='notesdestroy/confirm'><inputtype='hidden'name='notekey'value='{{#ifnote}}{{notekey}}{{/if}}'><p>Delete{{note.title}}?</p><br/><inputtype='submit'value='DELETE'/><ahref="notesview?key={{#ifnote}}{{notekey}}{{/if}}">Cancel</a></form>

Thisisasimpleform,askingtheusertoconfirmbyclickingonthebutton.TheCancellinkjustsendsthembacktothenotesviewpage.ClickingontheSubmitbuttongeneratesaPOSTrequestonthenotesdestroy/confirmURL.

Nowthateverythingisworkingintheapplication,youcanclickonanybuttonorlinkandkeepallthenotesyouwant.

ThatURLneedsarequesthandler.Addthiscodetoroutes/notes.js://Reallydestroynote(destroy)router.post('destroyconfirm',async(req,res,next)=>{awaitnotes.destroy(req.body.notekey);res.redirect('/');});

Thiscallsthenotes.destroyfunctioninthemodel.Ifitsucceeds,thebrowserisredirectedtothehomepage.Ifnot,anerrormessageisshowntotheuser.Rerunningtheapplication,wecannowviewitinaction:

ThemingyourExpressapplicationTheExpressteamhasdoneadecentjobofmakingsureExpressapplicationslookokayoutofthegate.OurNotesapplicationwon'twinanydesignawards,butatleastitisn'tugly.There'salotofwaystoimproveit,nowthatthebasicapplicationisrunning.Let'stakeaquicklookattheminganExpressapplication.InChapter6,ImplementingtheMobile-FirstParadigm,we'lltakeadeeperdive,focusingonthatall-importantgoalofaddressingthemobilemarket.

Ifyou'rerunningtheNotesapplicationusingtherecommendedmethod,npmstart,anicelogofactivityisbeingprintedinyourconsolewindow.Oneofthoseisthefollowing:

GETstylesheetsstyle.css3040.702ms--

Thisisduetothislineofcodethatweputinlayout.hbs:

<linkrel='stylesheet'href='stylesheetsstyle.css'/>

ThisfilewasautogeneratedforusbytheExpressGeneratorattheoutsetanddroppedinsidethepublicdirectory.ThepublicdirectoryismanagedbytheExpressstaticfileserver,usingthislineinapp.js:

app.use(express.static(path.join(__dirname,'public')));

Let'sopenpublicstylesheetsstyle.cssandtakealook:

body{

padding:50px;

font:14px"LucidaGrande",Helvetica,Arial,sans-serif;

}

a{

color:#00B7FF;

}

Somethingthatleapsoutisthattheapplicationcontenthasalotofwhitespaceatthetopandleft-handsidesofthescreen.Thereasonisthatbodytagshavethepadding:50pxstyle.Changingitisquickbusiness.

SincethereisnocachingintheExpressstaticfileserver,wecansimplyedittheCSSfileandreloadthepage,andtheCSSwillbereloadedaswell.It'spossibletoturnoncache-controlheadersandETagsgeneration,asyouwoulddoforaproductionwebsite.LookintheonlineExpressdocumentationfordetails.

Itinvolvesalittlebitofwork:

body{

padding:5px;

..

}

..

header{

background:#eeeeee;

padding:5px;

}

Asaresult,we'llhavethis:

We'renotgoingtowinanydesignawardswiththiseither,butthere'sthebeginningofsomebrandingandthemingpossibilities.

Generallyspeaking,thewaywe'vestructuredthepagetemplates,applyingasite-widethemeisjustamatterofaddingappropriatecodetolayout.hbsalongwithappropriatestylesheetsandotherassets.Manyofthemodernthemingframeworks,suchasTwitter'sBootstrap,serveupCSSandJavaScriptfilesoutofaCDNserver,makingitincrediblyeasytoincorporateintoasitedesign.

ForjQuery,refertohttp://jquery.com/download/.

Google'sHostedLibrariesserviceprovidesalonglistoflibraries,hostedonGoogle'sCDNinfrastructure.Refertohttps://developers.google.com/speed/libraries/.

Whileit'sstraightforwardtousethird-partyCDNstohosttheseassets,it'ssafertohostthemyourself.Notonlydoyoutakeresponsibilityfor

bandwidthconsumptionofyourapplication,butyou'recertainofnotbeingaffectedbyanyoutagesofthird-partyservices.AsreliableasGooglemightbe,theirservicecangodown,andifthatmeansjQueryandBootstrapdoesn'tload,yourcustomerwillthinkyoursiteisbroken.Butifthosefilesareloadedfromthesameserverasyourapplication,thereliabilityofdeliveringthosefileswillexactlyequalthereliabilityofyourapplication.

InChapter6,ImplementingtheMobile-FirstParadigm,wewilllookatasimplemethodtoaddthosefront-endlibrariestoyourapplication.

Scalingup–runningmultipleNotesinstancesNowthatwe'vegotourselvesarunningapplication,you'llhaveplayedaroundabitandcreated,read,updated,anddeletedmanynotes.

Supposeforamomentthisisn'tatoyapplication,butonethatisinterestingenoughtodrawamillionusersaday.Servingahighloadtypicallymeansaddingservers,loadbalancers,andmanyotherthings.Acorepartistohavemultipleinstancesoftheapplicationrunningatthesametimetospreadtheload.

Let'sseewhathappenswhenyourunmultipleinstancesoftheNotesapplicationatthesametime.

Thefirstthingistomakesuretheinstancesareondifferentports.Inbin/www,you'llseethatsettingthePORTenvironmentvariablecontrolstheportbeingused.IfthePORTvariableisnotset,itdefaultstohttp://localhost:3000,orwhatwe'vebeenusingallalong.

Let'sopenuppackage.jsonandaddtheselinestothescriptssection:

"scripts":{

"start":"DEBUG=notes:*node./bin/www",

"server1":"DEBUG=notes:*PORT=3001node./bin/www",

"server2":"DEBUG=notes:*PORT=3002node./bin/www"},

Theserver1scriptrunsonPORT3001,whiletheserver2scriptrunsonPORT3002.Isn'titnicetohaveallthisdocumentedinoneplace?

Then,inonecommandwindow,runthis:

$npmrunserver1

>notes@0.0.0server1UsersDavid/chap05/notes

>DEBUG=notes:*PORT=3001node./bin/www

notes:serverListeningonport3001+0ms

Inanothercommandwindow,runthis:

$npmrunserver2

>notes@0.0.0server2UsersDavid/chap05/notes

>DEBUG=notes:*PORT=3002node./bin/www

notes:serverListeningonport3002+0ms

ThisgivesustwoinstancesoftheNotesapplication.Usetwobrowserwindowstovisithttp://localhost:3001andhttp://localhost:3002.Enteracoupleofnotes,andyoumightseesomethinglikethis:

Aftereditingandaddingsomenotes,yourtwobrowserwindowscouldlookliketheprecedingscreenshot.Thetwoinstancesdonotsharethesamedatapool.Eachisinsteadrunninginitsownprocessandmemoryspace.Youaddanoteinone,anditdoesnotshowintheotherscreen.

Additionally,becausethemodelcodedoesnotpersistdataanywhere,thenotesarenotsaved.YoumighthavewrittenthegreatestNode.jsprogrammingbookofalltime,butassoonastheapplicationserverrestarts,it'sgone.

Typically,yourunmultipleinstancesofanapplicationtoscaleperformance.That'stheoldthrowmoreserversatittrick.Forthistowork,thedataofcoursemustbeshared,andeachinstancemustaccessthesamedatasource.Typically,thisinvolvesadatabase.Andwhenitcomestouseridentityinformation,itmightevenentailarmedguards.

Holdon—we'llgettodatabaseimplementationshortly.Beforethat,we'llcovermobile-firstdevelopment.

SummaryWe'vecomealongwayinthischapter.

WestartedwiththePyramidofDoomandhowthePromiseobjectandasyncfunctionscanhelpusatameasynchronouscode.We'llbeusingthesetechniquesallthroughthisbook.

WequicklymovedontowritingthefoundationofarealapplicationwithExpress.Atthemoment,itkeepsitsdatainmemory,butithasthebasicfunctionalityofwhatwillbecomeanote-takingapplicationsupportingreal-timecollaborativecommentingonthenotes.

Inthenextchapter,we'lldipourtoesinthewaterofresponsive,mobile-friendlywebdesign.Duetothegrowingpopularityofmobilecomputingdevices,it'sbecomenecessarytoaddressmobiledevicesfirstbeforedesktopcomputerusers.Inordertoreachthosemillionsofusersaday,theNotesapplicationusersneedagooduserexperiencewhenusingtheirsmartphone.

Infollowingchapters,we'llkeepgrowingthecapabilitiesoftheNotesapplication,startingwithdatabasestoragemodels.

ImplementingtheMobile-FirstParadigmNowthatourfirstExpressapplicationisusable,weactonthemantraofthisageofsoftwaredevelopment:mobile-first.Mobiledevices,whethertheybesmartphones,tabletcomputers,automobiledashboards,refrigeratordoors,orbathroommirrors,aretakingovertheworld.

Anotherissueismobile-firstindexing,meaningthatsearchenginesarestartingtopreferenceindexingthemobileversionofawebsite.Searchenginessofarconcentratedonindexingthedesktopversionofwebsites,butthegrowingpopularityofmobiledevicesmeanssearchengineresultsareskewedawayfromwhatfolksareusing.Googlesaysitisnotfairtomobileusersifthesearchresult,whichwasderivedfromthedesktopversion,doesnotmatchthemobileversionofawebsite.ForGoogle'stake,includingtechnicaltipsonthemarkuptouse,seehttp://webmasters.googleblog.com/2017/12/getting-your-site-ready-for-mobile.html.

Theprimaryconsiderationsindesigningformobilesarethesmallscreensizes,thetouch-orientedinteraction,thatthere'snomouse,andthesomewhatdifferentuserinterfaceexpectations.WiththeNotesapplication,ouruserinterfaceneedsaremodest,andthelackofamousedoesn'tmakeanydifferencetous.

Inthischapter,wewon'tdomuchNode.jsdevelopment.Instead,we'll:

Modifythetemplatesforbettermobilepresentation

EditCSSandSASSfilestocustomizethestyle

LearnaboutBootstrap4,apopularframeworkforresponsiveUIdesign

Bydoingso,we'lldipourtoesinthewaterofwhatitmeanstobeafullstackwebengineer.

Problem–theNotesappisn'tmobilefriendlyLet'sstartbyquantifyingtheproblem.Weneedtoexplorehowwell(ornot)theapplicationbehavesonamobiledevice.Thisissimpletodo:

1. StarttheNotesapplication.DeterminetheIPaddressofthehostsystem.

2. Usingyourmobiledevice,connecttotheserviceusingtheIPaddress,andbrowsearoundtheNotesapplication,puttingitthroughitspaces,notinganydifficulties.

Anotherwaytoapproachthisistouseyourdesktopbrowser,resizingittobeverynarrow.TheChromeDevToolsalsoincludesamobiledeviceemulator.Eitherway,youcanmimicthesmallscreensizeofasmartphoneonyourdesktop.

Toseearealuserinterfaceproblemonamobilescreen,editviews/noteedit.ejsandchangethisline:

<br/><textarearows=5cols=80name='body'>

{{#ifnote}}{{note.body}}{{/if}}

</textarea>

What'schangedisthecols=80parameter.Wewantthistextareaelementtobeoverlylargesothatyoucanexperiencehowanon-responsivewebappappearsonamobiledevice.Viewtheapplicationonamobiledeviceandyou'llseesomethinglikeoneofthescreensinthisscreenshot:

ViewinganoteworkswellonaniPhone6,butthescreenforediting/addinganoteisnotgood.Thetextentryareaissowidethatitrunsoffthesideofthescreen.EventhoughinteractionwithFORMelementsworkwell,it'sclumsy.Ingeneral,browsingtheNotesapplicationgivesanacceptablemobileuserexperiencethatdoesn'tsuckandwon'tmakeourusersgiveravereviews.

Mobile-firstparadigmMobiledeviceshaveasmallerscreen,aregenerallytouchoriented,andhavedifferentuserexperienceexpectationsthanadesktopcomputer.

Toaccommodatesmallerscreens,weuseresponsivewebdesigntechniques.Thismeansdesigningtheapplicationtoaccommodatethescreensizeandensuringwebsitesprovideoptimalviewingandinteractionacrossawiderangeofdevices.Techniquesincludechangingfontsizes,rearrangingelementsonthescreen,usingcollapsibleelementsthatopenwhentouched,andresizingimagesorvideostofitavailablespace.Thisiscalledresponsivebecausetheapplicationrespondstodevicecharacteristicsbymakingthesechanges.

Bymobile-first,wemeanthatyoudesigntheapplicationfirsttoworkwellonamobiledeviceandthenmoveontodeviceswithlargerscreens.It'saboutprioritizingmobiledevicesfirst.

Theprimarytechniqueisusingmediaqueriesinstylesheetstodetectdevicecharacteristics.Eachmediaquerysectiontargetsarangeofdevices,usingCSSdeclarationtoappropriatelyrestylecontent.

Let'sconsultaconcreteexample.TheTwentyTwelvethemeforWordpresshasastraightforwardresponsivedesignimplementation.It'snotbuiltwithanyframework,soyoucanseeclearlyhowthemechanismworks,andthestylesheetissmallenoughtobeeasilydigestible.RefertoitssourcecodeintheWordpressrepositoryathttps://themes.svn.wordpress.org/twentytwelve/1.9/style.css.

Thestylesheetstartswithanumberofresets,wherethestylesheetoverridessometypicalbrowserstylesettingswithcleardefaults.Then,thebulkofthestylesheetdefinesstylingformobiledevices.TowardthebottomofthestylesheetisasectionlabeledMediaquerieswhere,forcertainsizedscreens,thestylesdefinedformobiledevicesareoverridden

toworkondeviceswithlargerscreens.

Itdoesthiswiththefollowingtwomediaqueries:

@mediascreenand(min-width:600px){/*Screensabove600pxwidth}

@mediascreenand(min-width:960px){Screensabove960pxwidth*/}

Thefirstsegmentofthestylesheetconfiguresthepagelayoutforalldevices.Next,foranybrowserviewportatleast600pxwide,reconfigurethepagetodisplayonthelargerscreen.Then,foranybrowserviewportatleast960pxwide,reconfigureitagain.Thestylesheethasafinalmediaquerytocoverprintdevices.

Thesewidthsarewhat'scalledabreakpoint.Thosethresholdviewportwidthsarewherethedesignchangesitselfaround.Youcanseebreakpointsinactionbygoingtoanyresponsivewebsite,thenresizingthebrowserwindow.Watchhowthedesignjumpsatcertainsizes.Thosearethebreakpointschosenbytheauthorofthatwebsite.

There'sawiderangeofdifferingopinionsaboutthebeststrategytochooseyourbreakpoints.Doyoutargetspecificdevicesordoyoutargetgeneralcharacteristics?TheTwentyTwelvethemedidfairlywellonmobiledevicesusingonlytwoviewport-sizemediaqueries.TheCSS-Tricksbloghaspostedanextensivelistofspecificmediaqueriesforeveryknowndevice,whichisavailableathttps://css-tricks.com/snippets/css/media-queries-for-standard-devices/.

Weshouldatleasttargetthesedevices:

Small:ThisincludesiPhone5SE.

Medium:Thiscanrefertotabletcomputersorthelargersmartphones.

Large:Thisincludeslargertabletcomputersorthesmallerdesktopcomputers.

Extra-large:Thisreferstolargerdesktopcomputersandotherlargescreens.

Landscape/portrait:Youmaywanttocreateadistinctionbetweenlandscapemodeandportraitmode.Switchingbetweenthetwoofcoursechangesviewportwidth,possiblypushingitpastabreakpoint.However,yourapplicationmayneedtobehavedifferentlyinthetwomodes.

Enoughwiththetheory;let'sgetbacktoourcode.

UsingTwitterBootstrapontheNotesapplicationBootstrapisamobile-firstframeworkconsistingofHTML5,CSS3,andJavaScriptcodeprovidingacomprehensivesetofworldclass,responsivewebdesigncomponents.ItwasdevelopedbyengineersatTwitterandthenreleasedtotheworldinAugust2011.

Theframeworkincludescodetoretrofitmodernfeaturesontoolderbrowsers,aresponsive12-columngridsystem,andalonglistofcomponents(someusingJavaScript)forbuildingwebapplicationsandwebsites.It'smeanttoprovideastrongfoundationonwhichtobuildyourapplication.

Refertohttp://getbootstrap.comformoredetails.

SettingitupThefirststepistoduplicatethecodeyoucreatedinthepreviouschapter.If,forexample,youcreatedadirectorynamedchap05/notes,thencreateonenamedchap06/notesfromthecontentofchap05/notes.

Now,weneedtogoaboutaddingBootstrap'scodeintheNotesapplication.TheBootstrapwebsitesuggestsloadingtherequiredCSSandJavaScriptfilesoutoftheBootstrap(andjQuery)publicCDN.Whilethat'seasytodo,wewon'tdothisfortworeasons:

Itviolatestheprincipleofkeepingalldependencieslocaltotheapplicationandnotrelyingonglobaldependencies

Itpreventsusfromgeneratingacustomtheme

Instead,we'llinstallalocalcopyofBootstrap.ThereareseveralwaystoinstallBootstrap.Forexample,theBootstrapwebsiteoffersadownloadableTAR/GZIParchive(tarball).Thebetterapproachisanautomateddependencymanagementtool.

ThemoststraightforwardchoiceisusingBootstrap(https://www.npmjs.com/package/bootstrap),popper.js(https://www.npmjs.com/package/popper.js),andjQuery(https://www.npmjs.com/package/jquery)packagesinthenpmrepository.ThesepackagesprovidenoNode.jsmodules,andinsteadarefrontendcodedistributedthroughnpm.

Weinstallthepackagesusingthefollowingcommand:

$npminstallbootstrap@4.1.x--save

npmnoticecreatedalockfileaspackage-lock.json.Youshouldcommitthis

file.

npmWARNbootstrap@4.1.0requiresapeerofjquery@1.9.1-3butnoneis

installed.Youmustinstallpeerdependenciesyourself.

npmWARNbootstrap@4.1.0requiresapeerofpopper.js@^1.14.0butnoneis

installed.Youmustinstallpeerdependenciesyourself.

+bootstrap@4.1.0

added1packagein1.026s

$npminstalljquery@1.9.x--save

+jquery@1.9.1

$npminstallpopper.js@1.14.x--save

+popper.js@1.14.0

Asweseehere,whenweinstallBootstrap,ithelpfullytellsusthecorrespondingversionsofjQueryandpopper.jstouse.Therefore,wedutifullyinstallthoseversions.What'smostimportantistoseewhatgotdownloaded:

$lsnode_modules/bootstrap/dist/*

...directorycontents

$lsnode_modules/jquery/

...directorycontents

$lsnode_modules/popper.js/dist

...directorycontents

WithineachofthesedirectoriesaretheCSSandJavaScriptfilesthataremeanttobeusedinthebrowser.Moreimportantly,thesefilesarelocatedinagivendirectorywhosepathnameisknown—specifically,thedirectorieswejustinspected.Let'sseehowtoconfigureourExpressapptousethosethreepackagesonthebrowserside.

AddingBootstraptoapplicationtemplatesOntheBootstrapwebsite,theygivearecommendedHTMLstructure.We'llbeinterpolatingfromtheirrecommendationtouseBootstrapcodeprovidedthroughtheCDNtoinsteadusethelocalcopiesofBootstrap,jQuery,andPopperthatwejustinstalled.RefertotheGettingstartedpageathttp://getbootstrap.com/docs/4.0/getting-started/introduction/.

Whatwe'lldoismodifyviews/layout.hbstomatchtheirrecommendedtemplate:

<!doctypehtml>

<htmllang="en">

<head>

<title>{{title}}</title>

<metacharset="utf-8">

<metaname="viewport"

content="width=device-width,initial-scale=1,shrink-to-

fit=no">

<linkrel="stylesheet"

href="assetsvendor/bootstrap/css/bootstrap.min.css">

<linkrel='stylesheet'href='assetsstylesheets/style.css'/>

</head>

<body>

{{>header}}

{{{body}}}

<!--jQueryfirst,thenPopper.js,thenBootstrapJS-->

<scriptsrc="assetsvendor/jquery/jquery.min.js"></script>

<scriptsrc="assetsvendor/popper.js/popper.min.js"></script>

<scriptsrc="assetsvendor/bootstrap/js/bootstrap.min.js"></script>

</body>

</html>

ThisislargelythetemplateshownontheBootstrapsite,incorporatingthepreviouscontentofviews/layout.hbs.Ourownstylesheetisloadedfollowing

theBootstrapstylesheet,givingustheopportunitytooverrideanythinginBootstrapwewanttochange.What'sdifferentisthatinsteadofloadingBootstrap,popper.js,andjQuerypackagesfromtheirrespectiveCDNs,weusethepathassetsvendor/product-nameinstead.

ThisisthesameasrecommendedontheBootstrapwebsiteexcepttheURLspointtoourownsiteratherthanrelyingonthepublicCDN.

ThisassetsvendorURLisnotcurrentlyrecognizedbytheNotesapplication.Toaddthissupport,editapp.jstoaddtheselines:

app.use(express.static(path.join(__dirname,'public')));

app.use('assetsvendor/bootstrap',express.static(

path.join(__dirname,'node_modules','bootstrap','dist')));

app.use('assetsvendor/jquery',express.static(

path.join(__dirname,'node_modules','jquery')));

app.use('assetsvendor/popper.js',express.static(

path.join(__dirname,'node_modules','popper.js','dist')));

Withinthepublicdirectory,wehavealittlehouse-keepingtodo.Whenexpress-generatorsetuptheinitialproject,itgeneratedpublic/images,public/javascripts,andpublic/stylesheetsdirectories.Wewanteachtobewithinthe/assetsdirectory,sodothis:

$mkdirpublic/assets

$mvpublic/images/public/javascripts/public/stylesheets/publicassets

Wenowhaveourassetfiles,includingBootstrap,popper.js,andjQuery,allavailabletotheNotesapplicationunderthe/assetsdirectory.ThepagelayoutreferstotheseassetsandshouldgiveusthedefaultBootstraptheme:

$npmstart

>notes@0.0.0start/Users/David/chap06/notes

>DEBUG=notes:*node./bin/www

notes:serverListeningonport3000+0ms

GET/200306.660ms-883

GET/stylesheets/style.css404321.057ms-2439

GETassetsstylesheets/style.css200160.371ms-165

GETassetsvendor/bootstrap/js/bootstrap.min.js200157.459ms-50564

GETassetsvendor/popper.js/popper.min.js200769.508ms-18070

GETassetsvendor/jquery/jquery.min.js200777.988ms-92629

GETassetsvendor/bootstrap/css/bootstrap.min.css200788.028ms-127343

Theon-screendifferencesareminor,butthisistheproofnecessarythattheCSSandJavaScriptfilesforBootstraparebeingloaded.Wehaveaccomplishedthefirstmajorgoal—usingamodern,mobile-friendlyframeworktoimplementamobile-firstdesign.

AlternativelayoutframeworksBootstrapisn'ttheonlyJavaScript/CSSframeworkprovidingaresponsivelayoutandusefulcomponents.We'reusingBootstrapinthisprojectbecauseofitspopularity.Theseframeworksareworthyofalook:

Pure.css(https://purecss.io/):AresponsiveCSSframeworkwithanemphasisonasmallcodefootprint.

PicnicCSS(https://picnicss.com/):AresponsiveCSSframeworkemphasizingsmallsizeandbeauty.

Shoelace(https://shoelace.style/):ACSSframeworkemphasizingusingfutureCSS,meaningitusesCSSconstructsattheleadingedgeofCSSstandardization.Sincemostbrowsersdon'tsupportthosefeatures,cssnext(http://cssnext.io/)isusedtoretrofitthatsupport.ShoelaceusesagridlayoutsystembasedonBootstrap'sgrid.

PaperCSS(https://www.getpapercss.com/):AninformalCSSframeworkwhichlookslikeitwashanddrawn.

Foundation(https://foundation.zurb.com/):Self-describedasthemostadvancedresponsivefrontendframeworkintheworld.

Base(http://getbase.org/):AlightweightmodernCSSframework.

HTML5Boilerplate(https://html5boilerplate.com/)isanextremelyusefulbasisfromwhichtocodetheHTMLandotherassets.ItcontainsthecurrentbestpracticesfortheHTMLcodeinwebpages,aswellastoolsto

normalizeCSSsupportandconfigurationfilesforseveralwebservers.

FlexboxandCSSGridsOthernewtechnologiesimpactingwebapplicationdevelopmentaretwonewCSSlayoutmethodologies.TheCSS3committeehasbeenworkingonseveralfronts,includingpagelayout.

Inthedistantpast,weusednestedHTMLtablesforpagelayout.Thatisabadmemorythatwedon'thavetorevisit.Morerecently,we'vebeenusingaboxmodelusingDIVs,andevenattimesusingabsoluteorrelativeplacementtechniques.Allthesetechniqueshavebeensuboptimalinseveralways,somemorethanothers.

Onepopularlayouttechniqueistodividethehorizontalspaceintocolumnsandassignacertainnumberofcolumnstoeachthingonthepage.Withsomeframeworks,wecanevenhavenestedDIVs,eachwiththeirownsetofcolumns.Bootstrap3,andothermodernframeworks,usedthatlayouttechnique.

ThetwonewCSSlayoutmethodologies,Flexbox(https://en.wikipedia.org/wiki/CSS_flex-box_layout)andCSSGrids(https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout),areasignificantimprovementoverallpreviousmethodologies.Wearementioningthesetechnologiesbecausethey'rebothworthyofattention.Botharesomewhatearlyintheiradoptioncurve—they'vebeenstandardizedbycommitteesandadoptedinthelatestbrowsers,butofcoursetherearealotofoldbrowsersinthefield.

WithBootstrap4,theBootstrapteamchosetogowithFlexbox.Therefore,underthehoodareFlexboxCSSconstructs.

Mobile-firstdesignfortheNotesapplicationWe'velearnedaboutthebasicsofresponsivedesignandBootstrap,andwehookedtheBootstrapframeworkintoourapplication.Nowwe'rereadytolauncharedesignoftheapplicationsothatitworkswellonmobiledevices.

LayingtheBootstrapgridfoundationBootstrapusesa12-columngridsystemtocontrollayout,givingapplicationsaresponsivemobile-firstfoundationonwhichtobuild.Itautomaticallyscalescomponentsastheviewportchangessizeorshape.Themethodrelieson<div>elementswithclassestodescribetheroleeach<div>playsinthelayout.

Thebasiclayoutpatternisasfollows:

<divclass="container-fluid">

<divclass="row">

<divclass="col-sm-3">Column1content</div><!--25%-->

<divclass="col-sm-9">Column2content</div><!--75%-->

</div>

<divclass="row">

<divclass="col-sm-3">Column1content</div><!--25%-->

<divclass="col-sm-6">Column2content</div><!--50%-->

<divclass="col-sm-3">Column3content</div><!--25%-->

</div>

</div>

Theoutermostlayeristhe.containeror.container-fluidelement.Containersprovideameanstocenterorhorizontallypadthecontent.Containersmarkedas.container-fluidactasiftheyhavewidth:100%,meaningtheyexpandtofillthehorizontalspace.

A.rowiswhatitsoundslike,a"row".Technically,arowisawrapperforcolumns.Containersarewrappersforrows,androwsarewrappersforcolumns,andcolumnscontainthestuffdisplayedtoourusers.Gotthat?

Columnsaremarkedwithvariationsofthe.colclass.Withthebasiccolumnclass,.col,thecolumnsdivideequallyintotheavailablespace.

Youcanspecifyanumericalcolumncounttoassigndifferentwidthstoeachcolumn.Bootstrapsupportsupto12numberedcolumns,henceeachrowintheexampleaddsupto12columns.

Youcanalsospecifyabreakpointtowhichthecolumnapplies:

Usingcol-xstargetsextra-smalldevices(smartphones,<576px)

Usingcol-smtargetssmalldevices(>=576px)

Usingcol-mdtargetsmediumdevices(>=768px)

Usingcol-lgtargetslargedevices(>=992px)

Usingcol-xltargetsextra-largedevices(>=1200px)

Specifyingabreakpoint,forexamplecol-sm,meansthatitappliestodevicesmatchingthatbreakpoint,orlarger.Hence,intheexampleshownearlier,thecolumndefinitionsappliedtocol-sm,col-md,col-lg,andcol-xldevices,butnottocol-xsdevices.

Thecolumncountisappendedtotheclassname.Thatmeansusingcol-#whennottargetingabreakpoint,forexample,col-4,orcol-{breakpoint}-#whentargetingabreakpoint,forexample,col-md-4.Ifthecolumnsadduptomorethan12,thecolumnsbeyondthetwelfthcolumnwraparoundtobecomeanewrow.Thewordautocanbeusedinsteadofanumericalcolumncounttosizethecolumntothenaturalwidthofitscontent.

It'spossibletomixandmatchtotargetmultiplebreakpoints:

<divclass="container-fluid">

<divclass="row">

<divclass="col-xs-9col-md-3col-lg-6">Column1content</div>

<divclass="col-xs-3col-md-9col-lg-6">Column2content</div>

</div>

...

</div>

Thisdeclaresthreedifferentlayouts,oneforextra-smalldevices,anotherformediumdevices,andthelastforlargedevices.ThisgivesusenoughtostartmodifyingtheNotesapplication.Thegridsystemcandoalotmore.Fordetails,seethedocumentation:http://getbootstrap.com/docs/4.0/layout/grid/.

<!DOCTYPEhtml>

<html>

<head>..headerStuff</head>

<body>

..pageHeader

..maincontent<br/>..bottomOfPageStuff

</body>

</html>

Thepagecontentthereforehastwovisiblerows:theheaderandthemaincontent.AtthebottomofthepageareinvisiblethingsliketheJavaScriptfilesforBootstrapandjQuery.

Nochangeisrequiredinviews/layout.hbs.Onemightthinkthecontainer-fluidwrapperwouldbeinthatfile,withtherowsandcolumnsspecifiedintheothertemplates.Instead,we'lldoitinthetemplatestogiveusthemostlayoutfreedom.

UsingiconlibrariesandimprovingvisualappealTheworldaroundusisn'tconstructedofwords,butinsteadthings.Hence,apictorialstyle,asicons,shouldhelpcomputersoftwaretobemorecomprehensible.Givingagooduserexperienceshouldmakeourusersrewarduswithmorelikesintheappstore.

Thereareseveraliconlibrariesthatcanbeusedinawebsite.TheBootstrapteamhasacuratedlistathttp://getbootstrap.com/docs/4.1/extend/icons/.Forthisproject,we'lluseFeatherIcons(https://feathericons.com/)anditsconvenientlyavailablenpmpackage,https://www.npmjs.com/package/feathericons.

Inpackage.json,addthistothedependencies:

"feathericons":">=4.5.x"

Thenrunnpminstalltodownloadthenewpackage.Youcantheninspectthedownloadedpackageandseethat./node_modules/feathericons/dist/feather.jscontainsbrowser-sidecode,makingiteasytousetheicons.

Wemakethatdirectoryavailablebymountingitinapp.js,justaswedidforBootstrapandjQuerylibraries.Addthiscodetoapp.js:

app.use('assetsvendor/feathericons',express.static(

path.join(__dirname,'node_modules','feathericons','dist')));

Goingbythedocumentation,wemustputthisatthebottomofviews/layout.hbstoenablefeathericonssupport:

<scriptsrc="assetsvendor/feathericons/feather.js"></script>

<script>

feather.replace();

</script>

Touseoneoftheicons,useadata-featherattributespecifyingoneoftheiconnames,likeso:

<idata-feather="circle"></i>

What'simportantisthedata-featherattribute,whichtheFeatherIconslibraryusestoidentifytheSVGfiletouse.TheFeatherIconslibrarycompletelyreplacestheelementwhereitfoundthedata-featherattribute.Therefore,ifyouwanttheicontobeaclickablelink,it'snecessarytowraptheicondefinitionwithan<a>tag,ratherthanaddingdata-feathertothe<a>tag.Thenextsectionshowsanexample.

ResponsivepageheadernavigationbarTheheadersectionwedesignedbeforecontainsapagetitleandalittlenavigationbar.Bootstraphasseveralwaystospiffthisup,andevengiveusaresponsivenavigationbarwhichneatlycollapsestoamenuonsmalldevices.

Inviews/pageHeader.ejs,makethischange:

<headerclass="page-header">

<h1>{{title}}</h1>

<navclass="navbarnavbar-expand-mdnavbar-darkbg-dark">

<aclass="navbar-brand"href=''><idata-feather="home"><i></a>

<buttonclass="navbar-toggler"type="button"

data-toggle="collapse"data-target="#navbarSupportedContent"

aria-controls="navbarSupportedContent"

aria-expanded="false"aria-label="Togglenavigation">

<spanclass="navbar-toggler-icon"></span>

</button>

<divclass="collapsenavbar-collapse"id="navbarSupportedContent">

<divclass="navbar-navcol">

{{#ifbreadcrumb}}

<aclass="nav-itemnav-link"href='{{breadcrumb.url}}'>

{{breadcrumb.title}}</a>

{{/if}}

</div>

<aclass="nav-itemnav-linkbtnbtn-lightcol-auto"

href='notesadd'>ADDNote</a>

</div>

</nav>

</header>

Addingclass="page-header"informsBootstrapthisis,well,thepageheader.Withinthatwehavethe<h1>headerasbefore,givingthepagetitle,andthenaresponsiveBootstrapnavbar.

Bydefaultthenavbarisexpanded—meaningthecomponentsinsidethenavbararevisible—becauseofthenavbar-expand-mdclass.Thisnavbarisusinganavbar-togglerbuttonwhichgovernstheresponsivenessofthenavbar.Bydefault,thisbuttonishiddenandthebodyofthenavbarisvisible.Ifthescreenissmallenough,thenavbar-togglerisswitchedsoit'svisible,thebodyofthenavbarisinvisible,andwhenclickingonthenow-visiblenavbar-toggler,amenudropsdowncontainingthebodyofthenavbar:

Wechosethefeathericonshomeiconbecauseitsaysgohome.It'sintendedthatthemiddleportionofthenavbarwillcontainabreadcrumbaswenavigatearoundtheNotesapplication.

TheADDNotebuttonisgluedtotheright-hand-sidewithalittleFlexboxmagic.ThecontainerisaFlexbox,meaningwecanusetheBootstrapclassestocontrolthespaceconsumedbyeachitem.Thebreadcrumbareaisemptyinthiscase,butthe<div>thatwouldcontainitisthereanddeclaredwithclass="col",meaningthatittakesupacolumnunit.TheADDNotebuttonis,ontheotherhand,declaredwithclass="col-auto",meaningittakesuponlytheroomrequiredforitself.Itistheemptybreadcrumbareathatwillexpandtofillthespace,whiletheADDNotebuttonfillsonlyitsownspace,andisthereforepushedovertotheside.

Becauseit'sthesameapplication,thefunctionalityallworks;we'resimplyworkingonthepresentation.We'veaddedafewnotesbutthepresentationofthelistonthefrontpageleavesalottobedesired.Thesmallsizeofthetitleisnotverytouch-friendly,sinceitdoesn'tpresentalargetargetarea

forafingertip.Andcanyouexplainwhythenotekeyvaluehastobedisplayedonthehomepage?

ImprovingtheNoteslistonthefrontpageThecurrenthomepagehasasimpletextlistthat'snotterriblytouch-friendlyandshowingthekeyatthefrontofthelinemightbeinexplicabletotheuser.Let'sfixthis.

Editviews/index.hbsandmakethischange:

<divclass="container-fluid">

<divclass="row">

<divclass="col-12btn-group-vertical"role="group">

{{#eachnotelist}}

<aclass="btnbtn-lgbtn-blockbtn-outline-dark"

href="notesview?key={{key}}">{{title}}</a>

{{/each}}

</div>

</div>

</div>

Thefirstchangeistoswitchawayfromusingalistandtouseaverticalbuttongroup.Bymakingthetextlinkslookandbehavelikebuttons,we'reimprovingtheuserinterface,especiallyitstouchfriendliness.Wechosethebtn-outline-darkbuttonstylebecauseitlooksgoodintheuserinterface.Weuselargebuttons(btn-lg)thatfillthewidthofthecontainer(btn-block).

Weeliminatedshowingthenotekeytotheuser.Thisinformationdoesn'taddanythingtotheuserexperience:

Thisisbeginningtotakeshape,withadecent-lookinghomepagethathandlesresizingverynicelyandistouchfriendly.

There'sstillsomethingmoretodowiththis,sincetheheaderareaistakingupafairamountofspace.Weshouldalwaysfeelfreetorethinkaplanaswelookatintermediateresults.Earlier,wecreatedonedesignfortheheaderarea,butonreflectionthatdesignlookstobetoolarge.Theintentionhadbeentoinsertabreadcrumbjusttotherightofthehomeicon,andtoleavethe<h1>titleatthetopoftheheaderarea.Butthisistakingupverticalspaceandwecantightenuptheheaderandpossiblyimprovetheappearance.

Editpartials/header.hbsandreplaceitwiththefollowing:

<headerclass="page-header">

<navclass="navbarnavbar-expand-mdnavbar-darkbg-dark">

<aclass="navbar-brand"href='/'><idata-feather="home"></i></a>

<buttonclass="navbar-toggler"type="button"

data-toggle="collapse"data-target="#navbarSupportedContent"

aria-controls="navbarSupportedContent"

aria-expanded="false"

aria-label="Togglenavigation">

<spanclass="navbar-toggler-icon"></span>

</button>

<divclass="collapsenavbar-collapse"id="navbarSupportedContent">

<spanclass="navbar-textcol">{{title}}</span>

<aclass="nav-itemnav-linkbtnbtn-lightcol-auto"href='notesadd'>ADD

Note</a>

</div>

</nav>

</header>

Thisremovesthe<h1>tagatthetopoftheheaderarea,immediatelytighteningthepresentation.

Withinthenavbar-collapsearea,we'vereplacedwhathadbeenintendedasthebreadcrumb,withasimplenavbar-textcomponent.TokeeptheADDNotebuttongluedtotheright,we'remaintainingtheclass="col"andclass="col-auto"settings:

Whichheaderareadesignisbetter?That'sagoodquestion.Sincebeautyisintheeyeofthebeholder,bothdesignsareprobablyequallygood.Whatwehavedemonstratedistheeasewithwhichwecanupdatethedesignbyeditingthetemplatefiles.

CleaninguptheNoteviewingexperienceViewingaNoteisn'tbad,buttheuserexperiencecanbeimproved.Theuserdoesnotneedtoseethenotekey,forexample.Additionally,Bootstraphasnicer-lookingbuttonswecanuse.

Inviews/noteview.hbs,makethesechanges:

<divclass="container-fluid">

<divclass="row"><divclass="col-xs-12">

{{#ifnote}}<h3>{{note.title}}</h3>{{/if}}

{{#ifnote}}<p>{{note.body}}</p>{{/if}}

<p>Key:{{notekey}}</p>

</div></div>

{{#ifnotekey}}

<divclass="row"><divclass="col-xs-12">

<divclass="btn-group">

<aclass="btnbtn-outline-dark"

href="notesdestroy?key={{notekey}}"

role="button">Delete</a>

<aclass="btnbtn-outline-dark"

href="notesedit?key={{notekey}}"

role="button">Edit</a>

</div>

</div></div>

{{/if}}

</div>

Wehavedeclaredtworows,onefortheNote,andanotherforbuttonstoactontheNote.Botharedeclaredtoconsumeall12columns,andthereforetakeupthefullavailablewidth.Thebuttonsareagaincontainedwithinabuttongroup:

Dowereallyneedtoshowthenotekeytotheuser?We'llleaveitthere,butthat'sanopenquestionfortheuserexperienceteam.Otherwise,we'veimprovedthenote-readingexperience.

Cleaninguptheadd/editnoteformThenextmajorglaringproblemistheformforaddingandeditingnotes.Aswesaidearlier,it'seasytogetthetextinputareatooverflowasmallscreen.Ontheotherhand,Bootstraphasextensivesupportformakingnice-lookingformsthatworkwellonmobiledevices.

Changetheforminviews/noteedit.hbstothis:

<formmethod='POST'action='notessave'>

<divclass="container-fluid">

{{#ifdocreate}}

<inputtype='hidden'name='docreate'value="create">

{{else}}

<inputtype='hidden'name='docreate'value="update">

{{/if}}

<divclass="form-grouprowalign-items-center">

<labelfor="notekey"class="col-1col-form-label">Key</label>

{{#ifdocreate}}

<divclass="col">

<inputtype='text'class="form-control"

placeholder="notekey"name='notekey'value=''/>

</div>

{{else}}

{{#ifnote}}

<spanclass="input-group-text">{{notekey}}</span>

{{/if}}

<inputtype='hidden'name='notekey'

value='{{#ifnote}}{{notekey}}{{/if}}'/>

{{/if}}

</div>

<divclass="form-grouprow">

<labelfor="title"class="col-1col-form-label">Title</label>

<divclass="col">

<inputtype="text"class="form-control"

id='title'name='title'placeholder="notetitle"

value='{{#ifnote}}{{note.title}}{{/if}}'>

</div>

</div>

<divclass="form-grouprow">

<textareaclass="form-control"name='body'

rows="5">{{#ifnote}}{{note.body}}{{/if}}</textarea>

</div>

<buttontype="submit"class="btnbtn-default">Submit</button>

</div>

</form>

There'salotgoingonhere.Whatwe'vedoneisreorganizetheformsoBootstrapcandotherightthingswithit.Thefirstthingtonoteisthatwehaveseveralinstancesofthis:

<divclass="form-grouprow">..</div>

Thesearecontainedwithinacontainer-fluid,meaningthatwe'vesetupthreerowsintheform.

Bootstrapusesform-groupelementstoaddstructuretoforms,andtoencourageproperuseof<label>elements,alongwithotherformelements.It'sgoodpracticetousea<label>withevery<input>toimproveassistivebehaviorinthebrowser,ratherthanifyousimplyleftsomedanglingtext.

Everyformelementhasclass="form-control".Bootstrapusesthistoidentifythecontrolssoitcanaddstylingandbehavior.

Bydefault,Bootstrapformatsform-groupelementssothelabelappearsonanotherlinefromtheinputcontrol.Notethatwe'veaddedclass="col-1"tothelabelsandclass="col"tothe<div>wrappingtheinput.Thisdeclarestwocolumns,thefirstconsumingonecolumnunitandtheotherconsumingtheremainder.

Theplaceholder='key'attributeputssampletextinanotherwiseemptytextinputelement.Itdisappearsassoonastheusertypessomethingandisanexcellentwaytoprompttheuserwithwhat'sexpected.

Finally,wechangedtheSubmitbuttontobeaBootstrapbutton.These

looknice,andBootstrapmakessurethattheyworkgreat:

TheresultlooksgoodandworkswellontheiPhone.Itautomaticallysizesitselftowhateverscreenit'son.Everythingbehavesnicely.Inthisscreenshot,we'veresizedthewindowsmallenoughtocausethenavbartocollapse.Clickingontheso-calledhamburgericonontheright(thethreehorizontallines)causesthenavbarcontentstopopupasamenu.

Cleaningupthedelete-notewindowThewindowusedtoverifythedesiretodeleteaNotedoesn'tlookbad,butitcanbeimproved.

Editviews/notedestroy.hbstocontainthefollowing:

<formmethod='POST'action='notesdestroy/confirm'>

<divclass="container-fluid">

<inputtype='hidden'name='notekey'value='{{#ifnote}}{{notekey}}{{/if}}'>

<pclass="form-text">Delete{{note.title}}?</p>

<divclass="btn-group">

<buttontype="submit"value='DELETE'

class="btnbtn-outline-dark">DELETE</button>

<aclass="btnbtn-outline-dark"

href="notesview?key={{#ifnote}}{{notekey}}{{/if}}"

role="button">

Cancel</a>

</div>

</div>

</form>

We'vereworkedeverythingtouseBootstrapformgoodness.ThequestionaboutdeletingtheNoteiswrappedwithclass="form-text"sothatBootstrapcandisplaythisproperly.

Thebuttonsarewrappedwithclass="btn-group"asbefore.Thebuttonshaveexactlythesamestylingasonotherscreens,givingaconsistentlookacrosstheapplication:

ThereisanissuethatthetitletextinthenavbardoesnotusethewordDelete.Inroutes/notes.js,wecanmakethischange:

//AsktoDeletenote(destroy)

router.get('/destroy',async(req,res,next)=>{

varnote=awaitnotes.read(req.query.key);

res.render('notedestroy',{

title:note?`Delete${note.title}`:"",

notekey:req.query.key,note:note

});

});

Whatwe'vedoneischangedthetitleparameterpassedtothetemplate.We'ddonethisinthenoteseditroutehandlerandseeminglymisseddoingsointhishandler.

BuildingacustomizedBootstrapOnereasontouseBootstrapisthatyoucaneasilybuildacustomizedversion.StylesheetsarebuiltusingSASS,whichisoneoftheCSSpreprocessorstosimplifyCSSdevelopment.InBootstrap'scode,onefile(scss/_variables.scss)containsvariablesusedthroughouttherestofBootstrap's.scssfiles.ChangeonevariableanditcanautomaticallyaffecttherestofBootstrap.

Earlier,weoverrodeacoupleofBootstrapbehaviorswithourcustomCSSfile,public/stylesheets/style.css.Thisisaneasywaytochangeacoupleofspecificthings,butitdoesn'tworkforlarge-scalechangestoBootstrap.SeriousBootstrapcustomizationrequiresgeneratingacustomizedBootstrapbuild.

TheofficialdocumentationontheBootstrapwebsite(http://getbootstrap.com/docs/4.1/getting-started/build-tools/)isusefulforreferenceonthebuildprocess.

Ifyou'vefollowedthedirectionsgivenearlier,youhaveadirectory,chap06/notes,containingtheNotesapplicationsourcecode.Createadirectorynamedchap06notestheme,withinwhichwe'llsetupacustomBootstrapbuildprocess.

AsstudentsoftheTwelveFactorApplicationmodel,we'llbeusingapackage.jsoninthatdirectorytoautomatethebuildprocess.Thereisn'tanyNode.jscodeinvolved;npmisalsoaconvenienttooltoautomatethesoftwarebuildprocesses.

Tostart,downloadtheBootstrapsourcetreefromhttps://github.com/twbs/bootstrap.WhiletheBootstrapnpmpackageincludesSASSsourcefiles,it

isn'tsufficienttobuildBootstrap,andthereforeonemustdownloadthesourcetree.WhatwedoisnavigatetotheGitHubrepository,clickontheReleasestab,andselecttheURLforthemostrecentrelease.

Withtheme/package.jsoncontainingthisscriptssection:

{

"scripts":{

"download":"wget-O-

https://github.com/twbs/bootstrap/archive/v4.1.0.tar.gz|tarxvfz-",

"postdownload":"cdbootstrap-4.1.0&&npminstall"

}

}

Typethiscommand:

$npmrundownload

Thisdownloadsthetar-gzip(tarball)archivefromtheBootstraprepositoryandimmediatelyunpacksit.IfyouareonWindows,itwillbeeasiesttorunthatscriptinWindowsSubsystemforLinuxtoexecutethesecommands.Afterdownloadingandunpackingthearchive,thepostdownloadsteprunsnpminstallinthedirectory.TheBootstrapteamusestheirpackage.json,notonlytotrackallthedependenciesrequiredtobuildBootstrap,buttodrivethebuildprocess.

ThenpminstallforBootstrapwilltakealongtime,sobepatient.

ThismuchonlyinstallsthetoolsnecessarytobuildBootstrap.BuildingtheBootstrapdocumentationrequiresinstallingadditionalRuby-basedtools(Jekyllandsomeplugins).

TobuildBootstrap,let'saddthefollowinglinestothescriptssectioninourtheme/package.json:

"scripts":{

...

"build":"cdbootstrap-4.1.0&&npmrundist",

"watch":"cdbootstrap-4.1.0&&npmrunwatch"

...

}

Obviouslyyou'llneedtoadjustthedirectorynameastheBootstrapprojectissuesnewreleases.IntheBootstrapsourcetree,runningnpmrundist

buildsBootstrap,whilenpmrunwatchsetsupanautomatedprocesstoscan

forchangedfilesandrebuildsBootstrapuponchanginganyfile.Byaddingtheselinestoourtheme/package.json,wecanstartthisintheterminal

anditautomaticallyrerunsthebuildasneeded.

Nowrunabuildwiththiscommand:

$npmrunbuild

Thebuiltfileslandinthetheme/bootstrap-4.1.0/distdirectory.Thecontentofthatdirectorywillmatchthecontentsofthecorrespondingnpmpackage.

Incaseithasn'tbeenobviousallalong—thereareBootstrapversionnumbersembeddedintheseURLsandfileordirectorynames.AsnewBootstrapreleasesareissued,youmustadjustthepathnamestomatchthecurrentversionnumber.

Beforeproceeding,let'stakealookaroundtheBootstrapsourcetree.ThescssdirectorycontainstheSASSsourcethatwillbecompiledintotheBootstrapCSSfiles.TogenerateacustomizedBootstrapbuildwillrequireafewmodificationsinthatdirectory.

Thebootstrap-4.1.0/scss/bootstrap.scssfilecontains@importdirectivestopullinallBootstrapcomponents.Thefilebootstrap-4.1.0/scss/_variables.scsscontainsdefinitionsusedintheremainderoftheBootstrapSASSsource.Editing,oroverriding,thesevalueswillchangethelookofwebsitesusingtheresultingBootstrapbuild.

Forexample,thesedefinitionsdeterminethemaincolorvalues:

$white:#fff!default;

$gray-100:#f8f9fa!default;

...

$gray-800:#343a40!default;

...

$blue:#007bff!default;

...

$red:#dc3545!default;

$orange:#fd7e14!default;

$yellow:#ffc107!default;

$green:#28a745!default;

...

$primary:$blue!default;

$secondary:$gray-600!default;

$success:$green!default;

$info:$cyan!default;

$warning:$yellow!default;

$danger:$red!default;

$light:$gray-100!default;

$dark:$gray-800!default;

ThesearesimilartoCSSstatements.The!defaultattributedesignatesthesevaluesasthedefault.Any!defaultvaluescanbeoverriddenwithoutediting_values.scss.

Createafile,theme/_custom.scss,containingthefollowing:

$white:#fff!default;

$gray-900:#212529!default;

$body-bg:$gray-900!default;

$body-color:$white!default;

Thisreversesthevaluesforthe$body-bgand$body-colorsettingsin_variables.scss.TheNotesappwillnowusewhitetextonadarkbackground,ratherthanthedefaultwhitebackgroundwithdarktext.Becausethesedeclarationsdonotuse!default,they'lloverridethevaluesin_variables.scss.

Thenmakeacopyofscss/bootstrap.scssinthethemedirectoryandmodifyit,likeso:

@import"custom";

@import"functions";

@import"variables";

...

We'reimportingthe_custom.scssfilewejustcreated.Finally,addthislinetothescriptssectionoftheme/package.json:

"prebuild":"cp_custom.scssbootstrap.scssbootstrap-4.1.0/scss",

Withthatinplace,beforebuildingBootstrapthesetwofileswillbecopiedinplace.Next,rebuildBootstrap:

$npmrunbuild

>@prebuildUsersDavid/chap06notestheme

>cp_custom.scssbootstrap.scssbootstrap-4.1.0/scss

>@buildUsersDavid/chap06notestheme

>cdbootstrap-4.1.0&&npmrundist

...

Whilethat'sbuilding,let'smodifynotes/app.jstomountthebuilddirectory:

//app.use('assetsvendor/bootstrap',express.static(

//path.join(__dirname,'node_modules','bootstrap','dist')));

app.use('assetsvendor/bootstrap',express.static(

path.join(__dirname,'theme','bootstrap-4.1.0','dist')));

Whatwe'vedoneisswitchfromtheBootstrapinnode_modulestowhatwejustbuiltinthethemedirectory.TheBootstrapversionnumbershowsuphere,sothismustalsobeupdatedasnewBootstrapreleasesareadopted.

Thenreloadtheapplication,andyou'llseethefollowing:

Togetexactlythis,youmayneedtomakeachangeinthetemplates.TheButtonelementsweusedearlierhavethebtn-outline-darkclass,whichworkswellonalightbackground.Thebackgroundisnowdarkandthesebuttonsneedtouselightcoloring.

Tochangethebuttons,inviews/index.hbsmakethischange:

<aclass="btnbtn-lgbtn-blockbtn-outline-light"

href="notesview?key={{key}}">{{title}}</a>

Makeasimilarchangeinviews/noteview.hbs:

<aclass="btnbtn-outline-light"href="notesdestroy?key={{notekey}}"

role="button">Delete</a>

<aclass="btnbtn-outline-light"href="notesedit?key={{notekey}}"

role="button">Edit</a>

That'scool,wecannowreworktheBootstrapcolorschemeanywaywewant.Don'tshowthistoyouruserexperienceteam,becausethey'llthrowafit.Wedidthistoprovethepointthatwecanedit_custom.scssandchangetheBootstraptheme.

PrebuiltcustomBootstrapthemesIfallthisistoocomplicatedforyou,severalwebsitesprovideprebuiltBootstrapthemes,orelsesimplifiedtoolstogenerateaBootstrapbuild.Togetourfeetwet,let'sdownloadathemefromBootswatch(https://bootswatch.com/).ThisisbothacollectionoffreeandopensourcethemesandabuildsystemforgeneratingcustomBootstrapthemes(https://github.com/thomaspark/bootswatch/).

Let'susetheMintythemefromBootswatchtoexploretheneededchanges.Youcandownloadthethemefromthewebsiteoraddthefollowingtothescriptssectionofpackage.json:

"dl-minty":"mkdir-pminty&&npmrundl-minty-css&&npmrundl-minty-min-

css",

"dl-minty-css":"wgethttps://bootswatch.com/4/minty/bootstrap.css-O

minty/bootstrap.css",

"dl-minty-min-css":"wgethttps://bootswatch.com/4/minty/bootstrap.min.css-O

minty/bootstrap.min.css"

ThiswilldownloadtheprebuiltCSSfilesforourchosentheme.Inpassing,noticethattheBootswatchwebsiteoffers_variables.scssand_bootswatch.scssfileswhichshouldbeusablewithaworkflowsimilartowhatweimplementedintheprevioussection.TheGitHubrepositorymatchingtheBootswatchwebsitehasacompletebuildprocedureforbuildingcustomthemes.

Performthedownload:

$npmrundl-minty

>notes@0.0.0dl-mintyUsersDavid/chap06/notes

>mkdir-pminty&&npmrundl-minty-css&&npmrundl-minty-min-css

>notes@0.0.0dl-minty-cssUsersDavid/chap06/notes

>wgethttps://bootswatch.com/4/minty/bootstrap.css-Ominty/bootstrap.css

>notes@0.0.0dl-minty-min-cssUsersDavid/chap06/notes

>wgethttps://bootswatch.com/4/minty/bootstrap.min.css-O

minty/bootstrap.min.css

Inapp.jswewillneedtochangetheBootstrapmountstoseparatelymounttheJavaScriptandCSSfiles.Usethefollowing:

//app.use('assetsvendor/bootstrap',express.static(

//path.join(__dirname,'node_modules','bootstrap','dist')));

//app.use('assetsvendor/bootstrap',express.static(

//path.join(__dirname,'theme','bootstrap-4.0.0','dist')));

app.use('assetsvendor/bootstrap/js',express.static(

path.join(__dirname,'node_modules','bootstrap','dist','js')));

app.use('assetsvendor/bootstrap/css',express.static(

path.join(__dirname,'minty')));

Insteadofonemountforvendorbootstrap,wenowhavetwomountsforeachofthesubdirectories.Simplymakethevendorbootstrap/cssmountpointtoadirectorycontainingtheCSSfilesyoudownloadedfromthethemeprovider.

BecauseMintyisalight-coloredtheme,thebuttonsneedtousethedarkstyle.Wehadearlierchangedthebuttonstousealightstylebecauseofthedarkbackground.Wemustnowswitchfrombtn-outline-lightbacktobtn-outline-dark.Inpartials/header.hbs,thecolorschemerequiresachangetothenavbarcontent:

<divclass="collapsenavbar-collapse"id="navbarSupportedContent">

<spanclass="navbar-texttext-darkcol">{{title}}</span>

<aclass="nav-itemnav-linkbtnbtn-darkcol-auto"href='notesadd'>ADD

Note</a>

</div>

Weselectedtext-darkandbtn-darkclassestoprovidesomecontrastagainstthebackground.

Re-runtheapplicationandyou'llseesomethinglikethis:

SummaryThepossibilitiesforusingBootstrapareendless.Whilewecoveredalotofmaterial,weonlytouchedthesurface,andwecouldhavedonemuchmoretotheNotesapplication.

YoulearnedwhattheTwitterBootstrapframeworkcando.Bootstrap'sgoalistomakemobile-responsivedevelopmenteasy.WeusedBootstraptomakegreatimprovementstothewaytheNotesapplooksandfeels.WecustomizedBootstrap,dippingourtoesintogeneratingacustomtheme.

Now,wewanttogetbacktowritingNode.jscode.WeleftoffChapter5,YourFirstExpressApplication,withtheproblemofpersistencesothattheNotesapplicationcanbestoppedandrestartedwithoutlosingournotes.InChapter7,DataStorageandRetrieval,we'lldiveintousingdatabasestostoreourdata.

TogiveourselvessomeexperiencewiththeES6Moduleformat,we'llrewritetheNotesapplicationaccordingly.

DataStorageandRetrievalIntheprevioustwochapters,webuiltasmallandsomewhatusefulapplicationforstoringnotes,andthenmadeitworkonmobiledevices.Whiletheapplicationworksreasonablywell,itdoesn'tstorethosenotesanywhereonalong-termbasis,meaningthenotesarelostwhenyoustoptheserverand,ifyourunmultipleinstancesofNotes,eachinstancehasitsownsetofnotes.Thetypicalnextstepistointroduceadatabasetier.

Inthischapter,wewilllookatdatabasesupportinNode.js,sotheuserseesthesamesetofnotesforanyNotesinstanceaccessed,andtoreliablystorenotesforlong-termretrieval.

We'llstartwiththeNotesapplicationcodeusedinthepreviouschapter.Westartedwithasimple,in-memorydatamodelusinganarraytostorethenotes,andthenmadeitmobilefriendly.Inthischapter,wewill:

Discoverloggingoperationalanddebugginginformation

BeginusingtheES6moduleformat

ImplementdatapersistenceforNotesobjectsusingseveraldatabaseengines

Let'sgetstarted!

Thefirststepistoduplicatethecodefromthepreviouschapter.Forinstance,ifyouwereworkinginchap06/notes,duplicatethattobechap07/notes.

DatastorageandasynchronouscodeBydefinition,externaldatastoragesystemsrequireasynchronouscodeintheNode.jsarchitecture.Theaccesstimetoretrievedatafromdisk,fromanotherprocess,orfromadatabase,alwaystakessufficienttimetorequiredeferredexecution.

TheexistingNotesdatamodelisanin-memorydatastore.Intheory,in-memorydataaccessdoesnotrequireasynchronouscodeandtherefore,theexistingmodelmodulecouldhaveusedregularfunctionsratherthanasyncfunctions.

WeknewthatNotesmustmovetousingdatabases,andwouldrequireanasynchronousAPItoaccessNotesdata.Forthatreason,theexistingNotesmodelAPIusesasyncfunctionssothatinthischapter,wecanpersistNotedatatodatabases.

LoggingBeforewegetintodatabases,wehavetoaddressoneoftheattributesofahigh-qualitysoftwaresystem:managingloggedinformation,includingnormalsystemactivity,systemerrors,anddebugginginformation.Logsgiveusaninsightintothebehaviorofthesystem.Howmuchtrafficisitgetting?Ifit'sawebsite,whichpagesarepeoplehittingthemost?Howmanyerrorsoccurandofwhatkind?Doattacksoccur?Aremalformedrequestsbeingsent?

Logmanagementisalsoanissue.Logrotationmeansregularlymovingthelogfileoutoftheway,tostartwithafreshlogfile.Youshouldprocessloggeddatatoproducereports.Ahighpriorityonscreeningforsecurityvulnerabilitiesisamust.

TheTwelveFactorapplicationmodelsuggestssimplysendinglogginginformationtotheconsole,andthensomeothersoftwaresystemcapturesthatoutputanddirectsittoaloggingservice.Followingtheiradvicecanreducesystemcomplexitybyhavingfewerthingsthatcanbreak.Inalaterchapter,we'llusePM2forthatpurpose.

Let'sfirstcompleteatourofinformationloggingasitstandsrightnowinNotes.

WhenweusedtheExpressGeneratortoinitiallycreatetheNotesapplication,itconfiguredanactivityloggingsystemusingmorgan:

constlogger=require('morgan');

..

app.use(logger('dev'));

ThisiswhatprintstherequestsontheTerminalwindow.Visithttps://github.com/expressjs/morganformoreinformation.

Internally,ExpressusestheDebugpackagefordebuggingtraces.YoucanturntheseonusingtheDEBUGenvironmentvariable.Weshouldtrytousethispackageinourapplicationcode.Formoreinformation,visithttps://www.npmjs.com/package/debug.

Finally,theapplicationmightgenerateuncaughtexceptions.TheuncaughtExceptionerrorneedstobecaptured,logged,anddealtwithappropriately.

RequestloggingwithMorganTheMorganpackagehastwogeneralareasforconfiguration:

Logformat

Loglocation

Asitstands,Notesusesthedevformat,whichisdescribedasaconcisestatusoutputmeantfordevelopers.Thiscanbeusedtologwebrequestsasawaytomeasurewebsiteactivityandpopularity.TheApachelogformatalreadyhasalargeecosystemofreportingtoolsand,sureenough,Morgancanproducelogfilesinthisformat.

Tochangetheformat,simplychangethislineinapp.js:

app.use(logger(process.env.REQUEST_LOG_FORMAT||'dev'));

ThenrunNotesasfollows:

$REQUEST_LOG_FORMAT=commonnpmstart

>notes@0.0.0startUsersdavid/chap07/notes

>node./bin/www

::1--[12/Feb/2016:05:51:21+0000]"GETHTTP1.1"304-

::1--[12/Feb/2016:05:51:21+0000]"GET

vendorbootstrap/css/bootstrap.min.cssHTTP/1.1"304-

::1--[12/Feb/2016:05:51:21+0000]"GETstylesheetsstyle.cssHTTP/1.1"304

-

::1--[12/Feb/2016:05:51:21+0000]"GETvendorbootstrap/js/bootstrap.min.js

HTTP/1.1"304-

Toreverttothepreviousloggingoutput,simplydonotsetthisenvironmentvariable.Ifyou'velookedatApacheaccesslogs,thislogging

formatwilllookfamiliar.The::1notationatthebeginningofthelineisIPV6notationforthelocalhost,whichyoumaybemorefamiliarwithas127.0.0.1.

Wecandeclarevictoryonrequestloggingandmoveontodebuggingmessages.However,let'slookatloggingthistoafiledirectly.Whileit'spossibletocapturestdoutthroughaseparateprocess,MorganisalreadyinstalledinNotesanditdoesprovidethecapabilitytodirectitsoutputtoafile.

TheMorgandocumentationsuggeststhis:

//createawritestream(inappendmode)

varaccessLogStream=fs.createWriteStream(__dirname+'access.log',{flags:

'a'})

/setupthelogger

app.use(morgan('combined',{stream:accessLogStream}));

Butthishasaproblem;it'simpossibletoperformlogrotationwithoutkillingandrestartingtheserver.Instead,we'llusetheirrotating-file-streampackage.

First,installthepackage:

$npminstallrotating-file-stream--save

Thenweaddthiscodetoapp.js:

constfs=require('fs-extra');

...

constrfs=require('rotating-file-stream');

varlogStream;

//Logtoafileifrequested

if(process.env.REQUEST_LOG_FILE){

(async()=>{

letlogDirectory=path.dirname(process.env.REQUEST_LOG_FILE);

awaitfs.ensureDir(logDirectory);

logStream=rfs(process.env.REQUEST_LOG_FILE,{

size:'10M',//rotateevery10MegaByteswritten

interval:'1d',//rotatedaily

compress:'gzip'//compressrotatedfiles

});

})().catch(err=>{console.error(err);});

}

..

app.use(logger(process.env.REQUEST_LOG_FORMAT||'dev',{

stream:logStream?logStream:process.stdout

}));

Here,we'reusinganenvironmentvariable,REQUEST_LOG_FILE,tocontrolwhethertosendthelogtostdoutortoafile.Thelogcangointoadirectory,andthecodewillautomaticallycreatethatdirectoryifitdoesn'texist.Byusingrotating-file-stream(https://www.npmjs.com/package/rotating-file-stream),we'reguaranteedtohavelogfilerotationwithnoextrasystemsrequired.

Thefs-extramoduleisbeingusedbecauseitaddsPromise-basedfunctionstothefsmodule(https://www.npmjs.com/package/fs-extra).Inthiscase,fs.ensureDirchecksifthenameddirectorystructureexistsand,ifnot,thedirectorypathiscreated.

<strong>$DEBUG=express:*npmstart</strong>

constdebug=require('debug')('module-name');..

debug('somemessage');..

debug(`gotfile${fileName}`);

CapturingstdoutandstderrImportantmessagescanbeprintedtoprocess.stdoutorprocess.stderr,whichcanbelostifyoudon'tcapturethatoutput.TheTwelveFactormodelsuggestsusingasystemfacilitytocapturetheseoutputstreams.WithNotes,we'llusePM2forthatpurpose,whichwe'llcoverinChapter10,DeployingNode.jsApplications.

Thelogbookmodule(https://github.com/jpillora/node-logbook)offerssomeusefulcapabilitiesintermofnotonlycapturingprocess.stdoutandprocess.stderr,butsendingthatoutputtousefulplaces.

consterror=require('debug')('notes:error');

process.on('uncaughtException',function(err){

error("I'vecrashed!!!-"+(err.stack||err));});

..

if(app.get('env')==='development'){

app.use(function(err,req,res,next){

//util.log(err.message);res.status(err.status||500);error((err.status||500)+''+error.message);res.render('error',{

message:err.message,error:err});

});

}

..

app.use(function(err,req,res,next){

//util.log(err.message);res.status(err.status||500);error((err.status||500)+''+error.message);res.render('error',{

message:err.message,error:{}

});

});

Thedebugpackagehasaconventionwe'refollowing.Foranapplicationwithseveralmodules,alldebuggerobjectsshouldusethenamingpatternapp-name:module-name.Inthiscase,weusednotes:errorthatwillbeusedforallerrormessages.Wecouldalsousenotes:memory-modelornotes:mysql-modelfordebuggingdifferentmodels.

Whileweweresettingupthehandlerforuncaughtexceptions,itisalsoagoodideatoadderrorloggingintotheerrorhandlers.

UnhandledPromiserejectionsUsingPromiseandasyncfunctionsautomaticallychannelserrorsinausefuldirection.ErrorswillcauseaPromisetoflipintoarejectedstate,whichmusteventuallybehandledina.catchmethod.Sincewe'reallhuman,we'reboundtoforgettoensurethatallcodepathshandletheirrejectedPromise's.

Currently,Node.jsprintsthefollowingwarningifitdetectsanunhandledPromiserejection:(node:4796)UnhandledPromiseRejectionWarning:Unhandledpromiserejection

ThewarninggoesontosaythatthedefaulthandlerforunhandledPromiserejectionhasbeendeprecatedandthatsuchPromiserejectionswillcrashtheNodeprocessratherthanprintthismessage.Thebuilt-inprocessmoduledoesemitaneventinthiscase,soit'seasyenoughtoaddahandler:importutilfrom'util';...process.on('unhandledRejection',(reason,p)=>{error(`UnhandledRejectionat:${util.inspect(p)}reason:${reason}`);});

Attheminimum,wecanprintanerrormessagesuchasthefollowing:

notes:errorUnhandledRejectionat:Promise{

notes:error<rejected>TypeError:model(...).keylistisnotafunction

...fullstacktrace

}reason:TypeError:model(...).keylistisnotafunction+3s

UsingtheES6moduleformatWewrotetheNotesapplicationusingCommonJSmodules,thetraditionalNode.jsmoduleformat.Whiletheapplicationcouldcontinueusingthatformat,theJavaScriptcommunityhaschosentoswitchtoES6modulesinbothbrowserandNode.jscode,andthereforeit'simportanttoswitchES6modulessowecanallgetonboardwithacommonmoduleformat.Let'srewritetheapplicationusingES6modules,andthenwriteES6modulesforanythingnewweadd.

Thechangesrequiredarelargetoreplacerequirestatementswithimportstatements,andrenamingfilesfromfoo.jstofoo.mjs.Let'sgetstarted.

Rewritingapp.jsasanES6moduleLet'sstartwithapp.js,changingitsnametoapp.mjs:

$mvapp.jsapp.mjs

Changetheblockofrequirestatementsatthetoptothefollowing:

importfsfrom'fs-extra';

importurlfrom'url';

importexpressfrom'express';

importhbsfrom'hbs';

importpathfrom'path';

importutilfrom'util';

importfaviconfrom'serve-favicon';

importloggerfrom'morgan';

importcookieParserfrom'cookie-parser';

importbodyParserfrom'body-parser';

importDBGfrom'debug';

constdebug=DBG('notes:debug');

consterror=DBG('notes:error');

import{routerasindex}from'./routes/index';

//constusers=require('./routes/users');

import{routerasnotes}from'./routes/notes';

//Workaroundforlackof__dirnameinES6modules

const__dirname=path.dirname(newURL(import.meta.url).pathname);

constapp=express();

importrfsfrom'rotating-file-stream';

Then,atthebottomofthescript,makethischange:

exportdefaultapp;

Let'stalkalittleabouttheworkaroundmentionedhere.TherewereseveralglobalvariablesautomaticallyinjectedbyNode.jsintoCommonJSmodules.ThosevariablesarenotsupportedbyES6modules.ThecriticalvariableforNotesis__dirname,whichisusedinapp.mjsinseveralplaces.ThecodechangeshownhereincludesaworkaroundbasedonabrandnewJavaScriptfeaturethatisavailablestartinginNode.js10.x,theimport.meta.urlvariable.

Theimport.metaobjectismeanttoinjectusefulinformationintoanES6module.Asthenameimplies,theimport.meta.urlvariablecontainstheURLdescribingwherethemodulewasloadedfrom.ForNode.js,atthistime,ES6modulescanonlybeloadedfromafile://URLonthelocalfilesystem.Thatmeans,ifweextractthepathnameofthatURL,wecaneasilycalculatethedirectorycontainingthemodule,asshownhere.

Whythissolution?Whynotuseapathnamebeginningwith./?Theansweristhata./filenameisevaluatedrelativetotheprocess'scurrentworkingdirectory.ThatdirectoryisusuallydifferentfromthedirectorycontainingtheNode.jsmodulebeingexecuted.ThereforeitismorethanconvenientthattheNode.jsteamhasaddedtheimport.meta.urlfeature.

Thepatternfollowedinmostcasesisthischange:

constmoduleName=require('moduleName');//inCommonJSmodules

importmoduleNamefrom'moduleName';//inES6modules

RememberthatNode.jsusesthesamemodulelookupalgorithminbothES6andCommonJSmodules.ANode.jsrequirestatementissynchronous,meaningthatbythetimerequirefinishes,ithasexecutedthemoduleandisreturningitsmodule.exports.Bycontrast,anES6moduleisasynchronous,meaningthemodulemaynothavefinishedloading,andyoucanimportjusttherequiredbitsofthemodule.

MostofthemoduleimportsshownhereareforregularNode.jsmodulesinstalledinthenode_modulesdirectory,mostofwhichareCommonJSmodules.TheruleforusingimportwithaCommonJSmoduleisthatthemodule.exportsobjectistreatedasifitwerethedefaultexport.The

importstatementshownearliernamethedefaultexport(orthemodule.exportsobject)asshownintheimportstatement.ForaCommonJSmoduleimportedthisway,youthenuseitasyouwouldinaCommonJScontext,moduleName.functionName().

Theusageofthedebugmoduleiseffectivelythesame,butiscodeddifferently.IntheCommonJScontext,we'retoldtousethatmoduleasfollows:

constdebug=require('debug')('notes:debug');

consterror=require('debug')('notes:error');

Inotherwords,themodule.exportsofthismoduleisafunction,whichweimmediatelyinvoke.Thereisn'tasyntaxforES6modulestousethedebugmoduleinthatfashion.Therefore,wehadtobreakitapartasshown,andexplicitlycallthatfunction.

Thefinalpointofdiscussionistheimportofthetworoutermodules.Itwasfirstattemptedtohavethosemodulesexporttherouterasthedefaultvalue,butExpressthrewanerrorinthatcase.Instead,we'llrewritethesemodulestoexportrouterasanamedexportandthenusethatnamedexportasshownhere.

Rewritingbin/wwwasanES6moduleRememberthatbin/wwwisascriptusedtolaunchtheapplication.ItiswrittenasaCommonJSscript,butbecauseapp.mjsisnowanES6module,bin/wwwalsomustberewrittenasanES6module.ACommonJSmodulecannot,atthetimeofwriting,importanES6module.

Changethefilename:

$mvbin/wwwbin/www.mjs

Then,atthetop,changetherequirestatementstoimportstatements:

importappfrom'../app.mjs';

importDBGfrom'debug';

constdebug=DBG('notes:server-debug');

consterror=DBG('notes:server-error');

importhttpfrom'http';

We'vealreadydiscussedeverythinghereexceptthatapp.mjsexportsitsappobjectasthedefaultexport.Therefore,weuseitasshownhere.

RewritingmodelscodeasES6modulesThemodelsdirectorycontainstwomodules:Note.jsdefinestheNoteclass,andnotes-memory.jscontainsanin-memorydatamodel.BothareeasytoconverttoES6modules.

Changethefilenames:

$cdmodels

$mvNote.jsNote.mjs

$mvnotes-memory.jsnotes-memory.mjs

InNote.mjs,simplymakethefollowingchange:

exportdefaultclassNote{

...

}

ThismakestheNoteclassthedefaultexport.

Then,innotes-memory.mjs,makethefollowingchange:

importNotefrom'./Note';

varnotes=[];

asyncfunctioncrupdate(key,title,body){

notes[key]=newNote(key,title,body);

returnnotes[key];

}

exportfunctioncreate(key,title,body){returncrupdate(key,title,body);

}

exportfunctionupdate(key,title,body){returncrupdate(key,title,body);

}

exportasyncfunctionread(key){

if(notes[key])returnnotes[key];

elsethrownewError(`Note${key}doesnotexist`);

}

exportasyncfunctiondestroy(key){

if(notes[key]){

deletenotes[key];

}elsethrownewError(`Note${key}doesnotexist`);

}

exportasyncfunctionkeylist(){returnObject.keys(notes);}

exportasyncfunctioncount(){returnnotes.length;}

exportasyncfunctionclose(){}

Thisisastraightforwardtransliterationofassigningfunctionstomodule.exportstousingnamedexports.

BydefiningtheNoteclassasthedefaultexportoftheNote.mjsmodule,itimportsnicelyintoanymoduleusingthatclass.

RewritingroutermodulesasES6modulesTheroutesdirectorycontainstworoutermodules.Asitstands,eachroutermodulecreatesarouterobject,addsroutefunctionstothatobject,andthenassignsittothemodule.exportsfield.Thatsuggestsweshouldexporttherouterasthedefaultexport,butaswesaidearlier,thatdidn'tworkoutright.Instead,we'llexportrouterasanamedexport.

Changethefilenames:

$cdroutes

$mvindex.jsindex.mjs

$mvnotes.jsnotes.mjs

Then,atthetopofeach,changetherequirestatementblocktothefollowing:

importutilfrom'util';

importexpressfrom'express';

import*asnotesfrom'../models/notes-memory';

exportconstrouter=express.Router();

Itwillbethesameinbothfiles.Then,atthebottomofeachfile,deletethelineassigningroutertomodule.exports.

Let'sturntoapp.mjsandchangehowtheroutermodulesareimported.

Becauserouterisanamedexport,bydefaultyou'dimporttherouterobject,inapp.mjs,asfollows:

import{router}from'./routes/index';

Butthenwe'dhaveaconflictsincebothmodulesdefinearouterobject.Instead,wechangedthenameofthisobjectusinganasclause:

import{routerasindex}from'./routes/index';

import{routerasnotes}from'./routes/notes';

Therouterobjectfromeachmoduleishencegivenasuitablename.

StoringnotesinthefilesystemThefilesystemisanoftenoverlookeddatabaseengine.Whilefilesystemsdon'thavethesortofqueryfeaturessupportedbydatabaseengines,theyareareliableplacetostorefiles.Thenotesschemaissimpleenoughthatthefilesystemcaneasilyserveasitsdatastoragelayer.

Let'sstartbyaddingafunctiontoNote.mjs:

exportdefaultclassNote{

...

getJSON(){

returnJSON.stringify({

key:this.key,title:this.title,body:this.body

});

}

staticfromJSON(json){

vardata=JSON.parse(json);

varnote=newNote(data.key,data.title,data.body);

returnnote;

}

}

JSONisagetter,whichmeansitgetsthevalueoftheobject.Inthiscase,thenote.JSONattribute/getter,noparentheses,willsimplygiveustheJSONrepresentationoftheNote.We'llusethislaterforwritingtoJSONfiles.

fromJSONisastaticfunction,orfactorymethod,toaidinconstructingNoteobjectsifwehaveaJSONstring.ThedifferenceisthatJSONisassociatedwithaninstanceoftheNoteclass,whilefromJSONisassociatedwiththeclassitself.Thetwocanbeusedasfollows:

constnote=newNote("key","title","body");

constjson=note.JSON;//producesJSONtext

constnewnote=Note.fromJSON(json);//producesnewNoteinstance

Now,let'screateanewmodule,models/notes-fs.mjs,toholdthefilesystemmodel:

importfsfrom'fs-extra';

importpathfrom'path';

importutilfrom'util';

importNotefrom'./Note';

importDBGfrom'debug';

constdebug=DBG('notes:notes-fs');

consterror=DBG('notes:error-fs');

asyncfunctionnotesDir(){

constdir=process.env.NOTES_FS_DIR||"notes-fs-data";

awaitfs.ensureDir(dir);

returndir;

}

functionfilePath(notesdir,key){returnpath.join(notesdir,`${key}.json`);

}

asyncfunctionreadJSON(notesdir,key){

constreadFrom=filePath(notesdir,key);

vardata=awaitfs.readFile(readFrom,'utf8');

returnNote.fromJSON(data);

}

ThenotesDirfunctionwillbeusedthroughoutnotes-fstoensurethatthedirectoryexists.Tomakethissimple,we'reusingthefs-extramodulebecauseitaddsPromise-basedfunctionstothefsmodule(https://www.npmjs.com/package/fs-extra).Inthiscase,fs.ensureDirverifieswhetherthenameddirectorystructureexists,and,ifnot,thedirectorypathiscreated.

Theenvironmentvariable,NOTES_FS_DIR,configuresadirectorywithinwhichtostorenotes.We'llhaveonefilepernoteandstorethenoteasJSON.Ifnoenvironmentvariableisspecified,we'llfallbackonusingnotes-fs-dataasthedirectoryname.

Becausewe'readdinganotherdependency:

$npminstallfs-extra--save

Thefilenameforeachdatafileisthekeywith.jsonappended.Thatgivesonelimitationthatfilenamescannotcontainthe/character,sowetestforthatusingthefollowingcode:

asyncfunctioncrupdate(key,title,body){

varnotesdir=awaitnotesDir();

if(key.indexOf('/')>=0)

thrownewError(`key${key}cannotcontain'/'`);

varnote=newNote(key,title,body);

constwriteTo=filePath(notesdir,key);

constwriteJSON=note.JSON;

awaitfs.writeFile(writeTo,writeJSON,'utf8');

returnnote;

}

exportfunctioncreate(key,title,body){returncrupdate(key,title,body);

}

exportfunctionupdate(key,title,body){returncrupdate(key,title,body);

}

Asisthecasewiththenotes-memorymodule,thecreateandupdatefunctionsusetheexactsamecode.ThenotesDirfunctionisusedtoensurethatthedirectoryexists,thenwecreateaNoteobject,andthenwritethedatatoafile.

Noticehowthecodeisverystraightforwardbecauseoftheasyncfunction.Wearen'tcheckingforerrorsbecausethey'llbeautomaticallycaughtbytheasyncfunctionandbubbleouttoourcaller:

exportasyncfunctionread(key){

varnotesdir=awaitnotesDir();

varthenote=awaitreadJSON(notesdir,key);

returnthenote;

}

UsingreadJSON,readthefilefromthedisk.ItalreadygeneratestheNoteobject,soallwehavetodoisreturnthatobject:

exportasyncfunctiondestroy(key){

varnotesdir=awaitnotesDir();

awaitfs.unlink(filePath(notesdir,key));

}

Thefs.unlinkfunctiondeletesourfile.Becausethismoduleusesthefilesystem,deletingthefileisallthat'snecessarytodeletethenoteobject:

exportasyncfunctionkeylist(){

varnotesdir=awaitnotesDir();

varfilez=awaitfs.readdir(notesdir);

if(!filez||typeoffilez==='undefined')filez=[];

varthenotes=filez.map(asyncfname=>{

varkey=path.basename(fname,'.json');

varthenote=awaitreadJSON(notesdir,key);

returnthenote.key;

});

returnPromise.all(thenotes);

}

ThecontractforkeylististoreturnaPromisethatwillresolvetoanarrayofkeysforexistingnoteobjects.Sincethey'restoredasindividualfilesinthenotesdir,wehavetoreadeveryfileinthatdirectorytoretrieveitskey.

Array.mapconstructsanewarrayfromanexistingarray,namelythearrayoffilenamesreturnedbyfs.readdir.Eachentryintheconstructedarrayistheasyncfunction,whichreadstheNote,returningthekey:

exportasyncfunctioncount(){

varnotesdir=awaitnotesDir();

varfilez=awaitfs.readdir(notesdir);

returnfilez.length;

}

exportasyncfunctionclose(){}

Countingthenumberofnotesissimplyamatterofcountingthenumberoffilesinnotesdir.

DynamicimportofES6modulesBeforewestartmodifyingtherouterfunctions,wehavetoconsiderhowtoaccountformultiplemodels.Wecurrentlyhavetwomodulesforourdatamodel,notes-memoryandnotes-fs,andwe'llbeimplementingseveralmorebytheendofthischapter.Wewillneedasimplemethodtoselectbetweenthemodelbeingused.

Thereareseveralpossiblewaystodothis.Forexample,inaCommonJSmodule,it'spossibletodothefollowing:

constpath=require('path');

constnotes=require(process.env.NOTES_MODEL

?path.join('..',process.env.NOTES_MODEL)

:'../models/notes-memory');

Thisletsussetanenvironmentvariable,NOTES_MODEL,toselectthemoduletouseforthedatamodel.

Thisapproachdoesnotworkwiththeregularimportstatement,becausethemodulenameinanimportstatementcannotbesuchanexpression.TheDynamicImportfeaturenowinNode.jsdoesofferamechanismsimilartothesnippetjustshown.

Dynamicimportisanimport()functionthatreturnsaPromisethatwillresolvetotheimportedmodule.Asafunction-returning-a-Promise,import()won'tbeusefulastop-levelcodeinthemodule.But,considerthis:

varNotesModule;

asyncfunctionmodel(){

if(NotesModule)returnNotesModule;

NotesModule=awaitimport(`../models/notes-${process.env.NOTES_MODEL}`);

returnNotesModule;

}

exportasyncfunctioncreate(key,title,body){

return(awaitmodel()).create(key,title,body);

}

exportasyncfunctionupdate(key,title,body){

return(awaitmodel()).update(key,title,body);

}

exportasyncfunctionread(key){return(awaitmodel()).read(key);}

exportasyncfunctiondestroy(key){return(awaitmodel()).destroy(key);}

exportasyncfunctionkeylist(){return(awaitmodel()).keylist();}

exportasyncfunctioncount(){return(awaitmodel()).count();}

exportasyncfunctionclose(){return(awaitmodel()).close();}

Savethatmoduleinafile,models/notes.mjs.ThismoduleimplementsthesameAPIaswe'lluseforallNotesmodelmodules.Themodel()functionisthekeytodynamicallyselectinganotesmodelimplementationbasedonanenvironmentvariable.

ThisisanasyncfunctionandthereforeitsreturnvalueisaPromise.ThevalueofthatPromiseistheselectedmodule,asloadedbyimport().Becauseimport()returnsaPromise,weuseawaittoknowwhetheritloadedcorrectly.

EveryAPImethodfollowsthispattern:

exportasyncfunctionmethodName(args){

return(awaitmodel()).methodName(args);

}

Becausemodel()returnsaPromise,it'smostsuccincttouseanasyncfunctionanduseawaittoresolvethePromise.OncethePromiseisresolved,wesimplycallthemethodNamefunctionandgoaboutourbusiness.Otherwise,thoseAPImethodfunctionswouldbeasfollows:

exportfunctionmethodName(args){

returnmodel().then(notes=>{returnnotes.methodName(args);});

}

Thetwoimplementationsareequivalent,andit'sclearwhichisthemoresuccinct.

WithallthisawaitingonPromise'sreturnedfromasyncfunctions,it'sworthdiscussingtheoverhead.Theworstcaseisonthefirstcalltomodel(),becausetheselectednotesmodelhasnotbeenloaded.Thefirsttimearound,thecallflowgoesasfollows:

TheAPImethodcallsmodel(),whichcallsimport(),thenawait'sthemoduletofinishloading

TheAPImethodawait'sthePromisereturnedfrommodel(),gettingthemoduleobject,anditthencallstheAPIfunction

Thecallerisalsousingawaittoreceivethefinalresult

Thefirsttimearound,thetimeisdominatedbywaitingonimport()tofinishloadingthemodule.Onsubsequentcalls,themodulehasalreadybeenloadedandthefirststepistosimplyformaresolvedPromisecontainingthemodule.TheAPImethodcanthenquicklygetonwithdelegatingtotheactualAPImethod.

Tousethis,inroutes/index.mjs,andinroutes/notes.mjs,wemakethischange:

importutilfrom'util';

importexpressfrom'express';

import*asnotesfrom'../models/notes';

exportconstrouter=express.Router();

RunningtheNotesapplicationwithfilesystemstorageInpackage.json,addthistothescriptssection:

"start-fs":"DEBUG=notes:*NOTES_MODEL=fsnode--experimental-modules

./bin/www.mjs",

Whenyouputtheseentriesinpackage.json,makesurethatyouusecorrectJSONsyntax.Inparticular,ifyouleaveacommaattheendofthescriptssection,itwillfailtoparseandnpmwillthrowupanerrormessage.

Withthiscodeinplace,wecannowruntheNotesapplicationasfollows:

$DEBUG=notes:*npmrunstart-fs

>notes@0.0.0start-fsUsersdavid/chap07/notes

>NOTES_MODEL=models/notes-fsnode--experimental-modules./bin/www.mjs

notes:serverListeningonport3000+0ms

notes:fs-modelkeylistdir=notes-fs-datafiles=[]+4s

Thenwecanusetheapplicationathttp://localhost:3000asbefore.BecausewedidnotchangeanytemplateorCSSfiles,theapplicationwilllookexactlyasyouleftitattheendofChapter6,ImplementingtheMobile-FirstParadigm.

Becausedebuggingisturnedonfornotes:*,we'llseealogofwhatevertheNotesapplicationisdoing.It'seasytoturnthisoffsimplybynotsettingtheDEBUGvariable.

YoucannowkillandrestarttheNotesapplicationandseetheexactsamenotes.Youcanalsoeditthenotesatthecommandlineusingregulartexteditorssuchasvi.Youcannowstartmultipleserversondifferentportsandseeexactlythesamenotes:

"server1":"NOTES_MODEL=fsPORT=3001node--experimental-

modules./bin/www.mjs",

"server2":"NOTES_MODEL=fsPORT=3002node--experimental-

modules./bin/www.mjs",

Thenyoustartserver1andserver2inseparatecommandwindowsaswedidinChapter5,YourFirstExpressApplication.Then,visitthetwoserversinseparatebrowserwindows,andyouwillseethatbothbrowserwindowsshowthesamenotes.

Thefinalcheckistocreateanotewherethekeyhasa/character.Rememberthatthekeyisusedtogeneratethefilenamewherewestorethenote,andthereforethekeycannotcontaina/character.Withthebrowseropen,clickonADDNoteandenteranote,ensuringthatyouusea/characterinthekeyfield.OnclickingtheSubmitbutton,you'llseeanerrorsayingthatthisisn'tallowed.

StoringnoteswiththeLevelUPdatastoreTogetstartedwithactualdatabases,let'slookatanextremelylightweight,small-footprintdatabaseengine:LevelUP.ThisisaNode.js-friendlywrapperaroundtheLevelDBenginedevelopedbyGoogle,whichisnormallyusedinwebbrowsersforlocaldatapersistence.Itisanon-indexed,NoSQLdatastoredesignedoriginallyforuseinbrowsers.TheNode.jsmodule,Level,usestheLevelDBAPI,andsupportsmultiplebackends,includingLevelDOWN,whichintegratestheC++LevelDBdatabaseintoNode.js.

Visithttps://www.npmjs.com/package/levelforinformationonthemodule.Thelevelpackageautomaticallysetsupthelevelupandleveldownpackages.

Toinstallthedatabaseengine,runthiscommand:

$npminstalllevel@2.1.x--save

Then,startcreatingthemodels/notes-level.mjsmodule:

importfsfrom'fs-extra';

importpathfrom'path';

importutilfrom'util';

importNotefrom'./Note';

importlevelfrom'level';

importDBGfrom'debug';

constdebug=DBG('notes:notes-level');

consterror=DBG('notes:error-level');

vardb;

asyncfunctionconnectDB(){

if(typeofdb!=='undefined'||db)returndb;

db=awaitlevel(

process.env.LEVELDB_LOCATION||'notes.level',{

createIfMissing:true,

valueEncoding:"json"

});

returndb;

}

Thelevelmodulegivesusadbobjectthroughwhichtointeractwiththedatabase.We'restoringthatobjectasaglobalwithinthemoduleforeaseofuse.Ifthedbobjectisset,wecanjustreturnitimmediately.Otherwise,weopenthedatabaseusingcreateIfMissingtogoaheadandcreatethedatabaseifneeded.

Thelocationofthedatabasedefaultstonotes.levelinthecurrentdirectory.TheenvironmentvariableLEVELDB_LOCATIONcanbeset,asthenameimplies,tospecifythedatabaselocation:

asyncfunctioncrupdate(key,title,body){

constdb=awaitconnectDB();

varnote=newNote(key,title,body);

awaitdb.put(key,note.JSON);

returnnote;

}

exportfunctioncreate(key,title,body){

returncrupdate(key,title,body);

}

exportfunctionupdate(key,title,body){

returncrupdate(key,title,body);

}

Callingdb.puteithercreatesanewdatabaseentry,orreplacesanexistingone.Therefore,bothupdateandcreatearesettobethesamefunction.WeconverttheNotetoJSONsoitcanbeeasilystoredinthedatabase:

exportasyncfunctionread(key){

constdb=awaitconnectDB();

varnote=Note.fromJSON(awaitdb.get(key));

returnnewNote(note.key,note.title,note.body);

}

ReadingaNoteiseasy:justcalldb.getanditretrievesthedata,whichmustbedecodedfromtheJSONrepresentation.

Noticethatdb.getanddb.putdidnottakeacallbackfunction,andthatweuseawaittogettheresultsvalue.Thefunctionsexportedbylevelcantakeacallbackfunction,inwhichthecallbackwillbeinvoked.Alternatively,ifnocallbackfunctionisprovided,thelevelfunctionwillinsteadreturnaPromiseforcompatibilitywithasyncfunctions:

exportasyncfunctiondestroy(key){

constdb=awaitconnectDB();

awaitdb.del(key);

}

Thedb.destroyfunctiondeletesarecordfromthedatabase:

exportasyncfunctionkeylist(){

constdb=awaitconnectDB();

varkeyz=[];

awaitnewPromise((resolve,reject)=>{

db.createKeyStream()

.on('data',data=>keyz.push(data))

.on('error',err=>reject(err))

.on('end',()=>resolve(keyz));

});

returnkeyz;

}

exportasyncfunctioncount(){

constdb=awaitconnectDB();

vartotal=0;

awaitnewPromise((resolve,reject)=>{

db.createKeyStream()

.on('data',data=>total++)

.on('error',err=>reject(err))

.on('end',()=>resolve(total));

});

returntotal;

}

exportasyncfunctionclose(){

var_db=db;

db=undefined;

return_db?_db.close():undefined;

}

ThecreateKeyStreamfunctionusesanevent-orientedinterfacesimilartotheStreamsAPI.Itwillstreamthrougheverydatabaseentry,emittingeventsasitgoes.Adataeventisemittedforeverykeyinthedatabase,whiletheendeventisemittedattheendofthedatabase,andtheerroreventisemittedonerrors.Theeffectisthatthere'snosimplewaytopresentthisasasimplePromise.Instead,weinvokecreateKeyStream,letitrunitscourse,collectingdataasitgoes.WehavetowrapitinsideaPromiseobject,andcallresolveontheendevent.

Thenweaddthistopackage.jsoninthescriptssection:

"start-level":"DEBUG=notes:*NOTES_MODEL=levelnode--experimental-modules

./bin/www.mjs",

Finally,youcanruntheNotesapplication:

$DEBUG=notes:*npmrunstart-level

>notes@0.0.0startUsersdavid/chap07/notes

>node./bin/www

notes:serverListeningonport3000+0ms

Theprintoutintheconsolewillbethesame,andtheapplicationwillalsolookthesame.Youcanputitthroughitspacesandseethateverythingworkscorrectly.

Sinceleveldoesnotsupportsimultaneousaccesstoadatabasefrommultipleinstances,youwon'tbeabletousethemultipleNotesapplicationscenario.Youwill,however,beabletostopandrestarttheapplicationatwillwithoutlosinganynotes.

StoringnotesinSQLwithSQLite3Togetstartedwithmorenormaldatabases,let'sseehowtouseSQLfromNode.js.First,we'lluseSQLite3,alightweight,simple-to-set-updatabaseengineeminentlysuitableformanyapplications.

Tolearnaboutthatdatabaseengine,visithttp://www.sqlite.org/.

TolearnabouttheNode.jsmodule,visithttps://github.com/mapbox/node-sqlite3/wiki/APIorhttps://www.npmjs.com/package/sqlite3.

TheprimaryadvantageofSQLite3isthatitdoesn'trequireaserver;itisaself-contained,no-set-up-requiredSQLdatabase.

Thefirststepistoinstallthemodule:$npminstallsqlite3@3.x--save

SQLite3databaseschemaNext,weneedtomakesurethatourdatabaseisconfigured.We'reusingthisSQLtabledefinitionfortheschema(savethisasmodels/schema-sqlite3.sql):CREATETABLEIFNOTEXISTSnotes(notekeyVARCHAR(255),titleVARCHAR(255),bodyTEXT);

Howdoweinitializethisschemabeforewritingsomecode?Onewayistoensurethatthesqlite3packageisinstalledthroughyouroperatingsystempackagemanagementsystem,suchasusingapt-getonUbuntu/Debian,andMacPortsonmacOS.Onceit'sinstalled,youcanrunthefollowingcommand:$sqlite3chap07.sqlite3SQLiteversion3.21.02017-10-2418:55:49Enter".help"forusagehints.sqlite>CREATETABLEIFNOTEXISTSnotes(...>notekeyVARCHAR(255),...>titleVARCHAR(255),...>bodyTEXT...>);sqlite>.schemanotesCREATETABLEnotes(notekeyVARCHAR(255),titleVARCHAR(255),bodyTEXT);sqlite>^D$ls-lchap07.sqlite3-rwx------1davidstaff8192Jan1420:40chap07.sqlite3

Whilewecandothat,theTwelveFactorapplicationmodelsayswemust

automateanyadministrativeprocessesinthisway.Tothatend,weshouldinsteadwritealittlescripttorunanSQLoperationonSQLite3andusethattoinitializethedatabase.

Fortunately,thesqlite3commandoffersusawaytodothis.Addthefollowingtothescriptssectionofpackage.json:"sqlite3-setup":"sqlite3chap07.sqlite3--initmodels/schema-sqlite3.sql",

Runthesetupscript:

$npmrunsqlite3-setup

>notes@0.0.0sqlite3-setupUsersdavid/chap07/notes

>sqlite3chap07.sqlite3--initmodels/schema-sqlite3.sql

--Loadingresourcesfrommodels/schema-sqlite3.sql

SQLiteversion3.10.22016-01-2015:27:19

Enter".help"forusagehints.

sqlite>.schemanotes

CREATETABLEnotes(

notekeyVARCHAR(255),

titleVARCHAR(255),

bodyTEXT

);

sqlite>^D

WecouldhavewrittenasmallNode.jsscripttodothis,andit'seasytodoso.However,byusingthetoolsprovidedbythepackage,wehavelesscodetomaintaininourownproject.

SQLite3modelcodeNow,wecanwritecodetousethisdatabaseintheNotesapplication.

Createmodels/notes-sqlite3.mjsfile:

importutilfrom'util';

importNotefrom'./Note';

importsqlite3from'sqlite3';

importDBGfrom'debug';

constdebug=DBG('notes:notes-sqlite3');

consterror=DBG('notes:error-sqlite3');

vardb;//storethedatabaseconnectionhere

asyncfunctionconnectDB(){

if(db)returndb;

vardbfile=process.env.SQLITE_FILE||"notes.sqlite3";

awaitnewPromise((resolve,reject)=>{

db=newsqlite3.Database(dbfile,

sqlite3.OPEN_READWRITE|sqlite3.OPEN_CREATE,

err=>{

if(err)returnreject(err);

resolve(db);

});

});

returndb;

}

ThisservesthesamepurposeastheconnectDBfunctioninnotes-level.mjs:tomanagethedatabaseconnection.Ifthedatabaseisnotopen,it'llgoaheadanddoso,andevenmakesurethatthedatabasefileiscreated(ifitdoesn'texist).Butifitisalreadyopen,itisimmediatelyreturned:

exportasyncfunctioncreate(key,title,body){

vardb=awaitconnectDB();

varnote=newNote(key,title,body);

awaitnewPromise((resolve,reject)=>{

db.run("INSERTINTOnotes(notekey,title,body)"+

"VALUES(?,?,?);",[key,title,body],err=>{

if(err)returnreject(err);

resolve(note);

});

});

returnnote;

}

exportasyncfunctionupdate(key,title,body){

vardb=awaitconnectDB();

varnote=newNote(key,title,body);

awaitnewPromise((resolve,reject)=>{

db.run("UPDATEnotes"+

"SETtitle=?,body=?WHEREnotekey=?",

[title,body,key],err=>{

if(err)returnreject(err);

resolve(note);

});

});

returnnote;

}

Theseareourcreateandupdatefunctions.Aspromised,wearenowjustifiedindefiningtheNotesmodeltohaveseparatefunctionsforcreateandupdateoperations,becausetheSQLstatementforeachisdifferent.

Callingdb.runexecutesanSQLquery,givingustheopportunitytoinsertparametersintothequerystring.

Thesqlite3moduleusesaparametersubstitutionparadigmthat'scommoninSQLprogramminginterfaces.TheprogrammerputstheSQLqueryintoastring,andthenplacesaquestionmarkineachplacewheretheaimistoinsertavalueintothequerystring.Eachquestionmarkinthequerystringhastomatchavalueinthearrayprovidedbytheprogrammer.Themoduletakescareofencodingthevaluescorrectlysothatthequerystringisproperlyformatted,whilepreventingSQLinjectionattacks.

Thedb.runfunctionsimplyrunstheSQLqueryitisgiven,anddoesnotretrieveanydata.Becausethesqlite3moduledoesn'tproduceanykindofPromise,wehavetowrapfunctioncallsinaPromiseobject:

exportasyncfunctionread(key){

vardb=awaitconnectDB();

varnote=awaitnewPromise((resolve,reject)=>{

db.get("SELECT*FROMnotesWHEREnotekey=?",[key],(err,row)=>{

if(err)returnreject(err);

constnote=newNote(row.notekey,row.title,row.body);

resolve(note);

});

});

returnnote;

}

Toretrievedatausingthesqlite3module,youusethedb.get,db.all,ordb.eachfunctions.Thedb.getfunctionusedherereturnsonlythefirstrowoftheresultset.Thedb.allfunctionreturnsallrowsoftheresultsetatonce,whichcanbeaproblemforavailablememoryiftheresultsetislarge.Thedb.eachfunctionretrievesonerowatatime,whilestillallowingprocessingoftheentireresultset.

FortheNotesapplication,usingdb.gettoretrieveanoteissufficientbecausethereisonlyonenotepernotekey.Therefore,ourSELECTquerywillreturnatmostonerowanyway.Butwhatifyourapplicationwillseemultiplerowsintheresultset?We'llseewhattodoaboutthatinaminute.

Bytheway,thisreadfunctionhasabuginit.Seeifyoucanspottheerror.We'llreadmoreaboutthisinChapter11,UnitTestingandFunctionalTesting,whenourtestingeffortsuncoverthebug:

exportasyncfunctiondestroy(key){

vardb=awaitconnectDB();

returnawaitnewPromise((resolve,reject)=>{

db.run("DELETEFROMnotesWHEREnotekey=?;",[key],err=>{

if(err)returnreject(err);

resolve();

});

});

}

Todestroyanote,wesimplyexecutetheDELETEFROMstatement:

exportasyncfunctionkeylist(){

vardb=awaitconnectDB();

varkeyz=awaitnewPromise((resolve,reject)=>{

varkeyz=[];

db.all("SELECTnotekeyFROMnotes",(err,rows)=>{

if(err)returnreject(err);

resolve(rows.map(row=>row.notekey));

});

});

returnkeyz;

}

Thedb.allfunctionretrievesallrowsoftheresultset.

Thecontractforthisfunctionistoreturnanarrayofnotekeys.Therowsobjectisanarrayofresultsfromthedatabasethatcontainsthedatawearetoreturn,butinadifferentformat.Therefore,weusethemapfunctiontoconvertthearrayintotheformatrequiredtofulfillthecontract:

exportasyncfunctioncount(){

vardb=awaitconnectDB();

varcount=awaitnewPromise((resolve,reject)=>{

db.get("selectcount(notekey)ascountfromnotes",(err,row)

=>{

if(err)returnreject(err);

resolve(row.count);

});

});

returncount;

}

exportasyncfunctionclose(){

var_db=db;

db=undefined;

return_db?newPromise((resolve,reject)=>{

_db.close(err=>{

if(err)reject(err);

elseresolve();

});

}):undefined;

}

WecansimplyuseSQLtocountthenumberofnotesforus.Inthiscase,db.getreturnsarowwithasinglecolumn,count,whichisthevaluewewant

toreturn.

RunningNoteswithSQLite3Finally,we'rereadytoruntheNotesapplicationwithSQLite3.Addthefollowingcodetothescriptssectionofpackage.json:"start-sqlite3":"SQLITE_FILE=chap07.sqlite3NOTES_MODEL=sqlite3node--experimental-modules./bin/www.mjs",

RuntheNotesapplication:

$DEBUG=notes:*npmrunstart-sqlite3

>notes@0.0.0start-sqlite3Usersdavid/chap07/notes

>SQLITE_FILE=chap07.sqlite3NOTES_MODEL=models/notes-sqlite3node

./bin/www.mjs

notes:serverListeningonport3000+0ms

notes:sqlite3-modelOpenedSQLite3databasechap07.sqlite3+5s

Youcannowbrowsetheapplicationathttp://localhost:3000,andrunitthroughitspacesasbefore.

BecauseSQLite3supportssimultaneousaccessfrommultipleinstances,youcanrunthemultiserverexamplebyaddingthistothescriptssectionofpackage.json:"server1-sqlite3":"SQLITE_FILE=chap07.sqlite3NOTES_MODEL=sqlite3PORT=3001node./bin/www.mjs","server2-sqlite3":"SQLITE_FILE=chap07.sqlite3NOTES_MODEL=sqlite3PORT=3002node./bin/www.mjs",

Then,runeachoftheseinseparatecommandWindows,asbefore.

Becausewestillhaven'tmadeanychangestotheViewtemplatesorCSSfiles,theapplicationwilllookthesameasbefore.

Ofcourse,youcanusethesqlitecommand,orotherSQLite3client

applications,toinspectthedatabase:

$sqlite3chap07.sqlite3

SQLiteversion3.10.22016-01-2015:27:19

Enter".help"forusagehints.

sqlite>select*fromnotes;

hithere|HiThere||hotherewhatthere

himom|HiMom||Thisiswherewesaythanks

StoringnotestheORMwaywithSequelizeThereareseveralpopularSQLdatabaseengines,suchasPostgreSQL,MySQL(https://www.npmjs.com/package/mysql),andMariaDB(https://www.npmjs.com/package/mariasql).CorrespondingtoeachareNode.jsclientmodulessimilarinnaturetothesqlite3modulethatwejustused.TheprogrammerisclosetotheSQL,whichcanbegoodinthesamewaythatdrivingastickshiftcarisfun.Butwhatifwewantahigher-levelviewofthedatabasesothatwecanthinkintermsofobjectsratherthanrowsofadatabasetable?ObjectRelationMapping(ORM)systemsprovidesuchahigher-levelinterfaceandevenoffertheabilitytousethesamedatamodelwithseveraldatabases.

TheSequelizemodule(http://www.sequelizejs.com/)isPromise-based,offersstrong,well-developedORMfeatures,andcanconnectwithSQLite3,MySQL,PostgreSQL,MariaDB,andMSSQL.BecauseSequelizeisPromise-based,itwillfitnaturallywiththePromise-basedapplicationcodewe'rewriting.

AprerequisitetomostSQLdatabaseenginesishavingaccesstoadatabaseserver.Intheprevioussection,weskirtedaroundthatissuebyusingSQLite3,whichrequiresnodatabaseserversetup.Whileit'spossibletoinstalladatabaseserveronyourlaptop,wewanttoavoidthecomplexityofdoingso,andwilluseSequelizetomanageanSQLite3database.We'llalsoseethatit'ssimplyamatterofaconfigurationfiletorunthesameSequelizecodeagainstahosteddatabasesuchasMySQL.InChapter10,DeployingNode.jsApplications,we'lllearnhowtouseDockertoeasilysetupanyservice,includingdatabaseservers,onourlaptopanddeploytheexactsameconfigurationtoaliveserver.MostwebhostingprovidersofferMySQLorPostgreSQLaspartoftheservice.

Beforewestartonthecode,let'sinstalltwomodules:

$npminstallsequelize@4.31.x--save

$npminstalljs-yaml@3.10.x--save

ThefirstobviouslyinstallstheSequelizepackage.Thesecond,js-yaml,isinstalledsothatwecanimplementaYAML-formattedfiletostoretheSequelizeconnectionconfiguration.YAMLisahuman-readabledataserializationlanguage,whichsimplymeansYAMLisaneasy-to-usetextfileformattodescribedataobjects.PerhapsthebestplacetolearnaboutYAMLisitsWikipediapageathttps://en.wikipedia.org/wiki/YAML.

SequelizemodelfortheNotesapplicationLet'screateanewfile,models/notes-sequelize.mjs:

importfsfrom'fs-extra';

importutilfrom'util';

importjsyamlfrom'js-yaml';

importNotefrom'./Note';

importSequelizefrom'sequelize';

importDBGfrom'debug';

constdebug=DBG('notes:notes-sequelize');

consterror=DBG('notes:error-sequelize');

varSQNote;

varsequlz;

asyncfunctionconnectDB(){

if(typeofsequlz==='undefined'){

constYAML=awaitfs.readFile(process.env.SEQUELIZE_CONNECT,'utf8');

constparams=jsyaml.safeLoad(YAML,'utf8');

sequlz=newSequelize(params.dbname,params.username,

params.password,params.params);

}

if(SQNote)returnSQNote.sync();

SQNote=sequlz.define('Note',{

notekey:{type:Sequelize.STRING,primaryKey:true,unique:

true},

title:Sequelize.STRING,

body:Sequelize.TEXT

});

returnSQNote.sync();

}

Thedatabaseconnectionisstoredinthesequlzobject,andisestablishedbyreadingaconfigurationfile(we'llgooverthisfilelater),andinstantiatingaSequelizeinstance.Thedatamodel,SQNote,describesourobjectstructuretoSequelizesothatitcandefinecorrespondingdatabasetable(s).

IfSQNoteisalreadydefined,wesimplyreturnit,otherwisewedefineandreturnSQNote.

TheSequelizeconnectionparametersarestoredinaYAMLfilewespecifyintheSEQUELIZE_CONNECTenvironmentvariable.ThelinenewSequelize(..)opensthedatabaseconnection.Theparametersobviouslycontainanyneededdatabasename,username,password,andotheroptionsrequiredtoconnectwiththedatabase.

Thelinesequlz.defineiswherewedefinethedatabaseschema.InsteadofdefiningtheschemaastheSQLcommandtocreatethedatabasetable,we'regivingahigh-leveldescriptionofthefieldsandtheircharacteristics.Sequelizemapstheobjectattributesintocolumnsintables.

We'retellingSequelizetocallthisschemaNote,butwe'reusingaSQNotevariabletorefertothatschema.That'sbecausewealreadydefinedNoteasaclasstorepresentnotes.Toavoidaclashofnames,we'llkeepusingtheNoteclass,anduseSQNotetointeractwithSequelizeaboutthenotesstoredinthedatabase.

Onlinedocumentationcanbefoundatthefollowinglocations:Sequelizeclass:http://docs.sequelizejs.com/en/latest/api/sequelize/.Definingmodels:http://docs.sequelizejs.com/en/latest/api/model/.

Addthesefunctionstomodels/notes-sequelize.mjs:

exportasyncfunctioncreate(key,title,body){

constSQNote=awaitconnectDB();

constnote=newNote(key,title,body);

awaitSQNote.create({notekey:key,title:title,body:body});

returnnote;

}

exportasyncfunctionupdate(key,title,body){

constSQNote=awaitconnectDB();

constnote=awaitSQNote.find({where:{notekey:key}})

if(!note){thrownewError(`Nonotefoundfor${key}`);}else{

awaitnote.updateAttributes({title:title,body:body});

returnnewNote(key,title,body);

}

}

ThereareseveralwaystocreateanewobjectinstanceinSequelize.Thesimplestistocallanobject'screatefunction(inthiscase,SQNote.create).Thatfunctioncollapsestogethertwootherfunctions,build(tocreatetheobject),andsave(towriteittothedatabase).

Updatinganobjectinstanceisalittledifferent.First,wemustretrieveitsentryfromthedatabaseusingthefindoperation.Thefindoperationisgivenanobjectspecifyingthequery.Usingfind,weretrieveoneinstance,whereasthefindAlloperationretrievesallmatchinginstances.

FordocumentationonSequelizequeries,visithttp://docs.sequelizejs.com/en/latest/docs/querying/.

LikemostorallotherSequelizefunctions,SQNote.findreturnsaPromise.Therefore,insideanasyncfunction,weawaittheresultoftheoperation.

Theupdateoperationrequirestwosteps,thefirstbeingtofindthecorrespondingobjecttoreaditinfromthedatabase.Oncetheinstanceisfound,wecanupdateitsvaluessimplywiththeupdateAttributesfunction:

exportasyncfunctionread(key){

constSQNote=awaitconnectDB();

constnote=awaitSQNote.find({where:{notekey:key}})

if(!note){thrownewError(`Nonotefoundfor${key}`);}else{

returnnewNote(note.notekey,note.title,note.body);

}

}

Toreadanote,weusethefindoperationagain.Thereisthepossibilityofanemptyresult,andwehavetothrowanerrortomatch.

ThecontractforthisfunctionistoreturnaNoteobject.ThatmeanstakingthefieldsretrievedusingSequelizeandusingthattocreateaNoteobject:

exportasyncfunctiondestroy(key){

constSQNote=awaitconnectDB();

constnote=awaitSQNote.find({where:{notekey:key}})

returnnote.destroy();

}

Todestroyanote,weusethefindoperationtoretrieveitsinstance,andthencallitsdestroy()method:

exportasyncfunctionkeylist(){

constSQNote=awaitconnectDB();

constnotes=awaitSQNote.findAll({attributes:['notekey']});

returnnotes.map(note=>note.notekey);

}

BecausethekeylistfunctionactsonallNoteobjects,weusethefindAlloperation.Wequeryforthenotekeyattributeonallnotes.We'regivenanarrayofobjectswithafieldnamednotekey,andweusethe.mapfunctiontoconvertthisintoanarrayofthenotekeys:

exportasyncfunctioncount(){

constSQNote=awaitconnectDB();

constcount=awaitSQNote.count();

returncount;

}

exportasyncfunctionclose(){

if(sequlz)sequlz.close();

sequlz=undefined;

SQNote=undefined;

}

Forthecountfunction,wecanjustusethecount()methodtocalculatetheneededresult.

ConfiguringaSequelizedatabaseconnectionSequelizesupportsthesameAPIonseveralSQLdatabaseengines.ThedatabaseconnectionisinitializedusingparametersontheSequelizeconstructor.TheTwelveFactorApplicationmodelsuggeststhatconfigurationdatasuchasthisshouldbekeptoutsidethecodeandinjectedusingenvironmentvariablesorasimilarmechanism.Whatwe'lldoisuseaYAML-formattedfiletostoretheconnectionparameters,specifyingthefilenamewithanenvironmentvariable.

TheSequelizelibrarydoesnotdefineanysuchfileforstoringconnectionparameters.Butit'ssimpleenoughtodevelopsuchafile.Let'sdoso.

TheAPIfortheSequelizeconstructoris:constructor(database:String,username:String,password:String,options:Object).

IntheconnectDBfunction,wewrotetheconstructorasfollows:

sequlz=newSequelize(params.dbname,params.username,params.password,

params.params);

Thisfile,namedmodels/sequelize-sqlite.yaml,provideswithusasimplemappingthatlookslikethisforanSQLite3database:

dbname:notes

username:

password:

params:

dialect:sqlite

storage:notes-sequelize.sqlite3

TheYAMLfileisadirectmappingtotheSequelizeconstructor

parameters.Thedbname,username,andpasswordfieldsinthisfilecorresponddirectlytotheconnectioncredentials,andtheparamsobjectgivesadditionalparameters.Therearemany,many,possibleattributestouseintheparamsfield,andyoucanreadaboutthemintheSequelizedocumentationathttp://docs.sequelizejs.com/manual/installation/usage.html.

ThedialectfieldtellsSequelizewhatkindofdatabasetouse.ForanSQLitedatabase,thedatabasefilenameisgiveninthestoragefield.

Let'sfirstuseSQLite3,becausenofurthersetupisrequired.Afterthat,we'llgetadventurousandreconfigureourSequelizemoduletouseMySQL.

Ifyoualreadyhaveadifferentdatabaseserveravailable,it'ssimpletocreateacorrespondingconfigurationfile.ForaplausibleMySQLdatabaseonyourlaptop,createanewfile,suchasmodels/sequelize-mysql.yaml,containingsomethinglikethefollowingcode:

dbname:notes

username:..username

password:..password

params:

host:localhost

port:3306

dialect:mysql

Thisisstraightforward.Theusernameandpasswordmustcorrespondtothedatabasecredentials,whilehostandportwillspecifywherethedatabaseishosted.Setthedatabasedialectandotherconnectioninformation,andyou'regoodtogo.

TouseMySQL,youwillneedtoinstallthebaseMySQLdriversothatSequelizecanuseMySQL:

$npminstallmysql@2.x--save

RunningwithSequelizeagainstotherdatabasesitsupports,suchas

PostgreSQL,isjustassimple.Justcreateaconfigurationfile,installtheNode.jsdriver,andinstall/configurethedatabaseengine.

RunningtheNotesapplicationwithSequelizeNowwecangetreadytoruntheNotesapplicationusingSequelize.WecanrunthisagainstbothSQLite3andMySQL,butlet'sstartwithSQLite.Addthisentrytothescriptsentryinpackage.json:

"start-sequelize":"SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=sequelizenode--experimental-modules./bin/www.mjs"

Thenrunitasfollows:

$DEBUG=notes:*npmrunstart-sequelize

>notes@0.0.0start-sequelizeUsersdavid/chap07/notes

>SEQUELIZE_CONNECT=models/sequelize-sqlite.yamlNOTES_MODEL=sequelizenode-

-experimental-modules./bin/www.mjs

notes:serverListeningonport3000+0ms

Asbefore,theapplicationlooksexactlythesamebecausewe'venotchangedtheViewtemplatesorCSSfiles.Putitthroughitspacesandeverythingshouldwork.

WithSequelize,multipleNotesapplicationinstancesisassimpleasaddingtheselinestothescriptssectionofpackage.json,andthenstartingbothinstancesasbefore:

"server1-sequelize":"SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=sequelizePORT=3001node--experimental-modules./bin/www.mjs",

"server2-sequelize":"SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=sequelizePORT=3002node--experimental-modules./bin/www.mjs",

Youwillbeabletostartbothinstances,useseparatebrowserwindowstovisitbothinstances,andseethattheyshowthesamesetofnotes.

ToreiterateusingtheSequelize-basedmodelonagivendatabaseserver:

1. Installandprovisionthedatabaseserverinstance,orelsegettheconnectionparametersforanalready-provisioneddatabaseserver.

2. InstallthecorrespondingNode.jsdriver.3. WriteaYAMLconfigurationfilecorrespondingtotheconnection

parameters.4. Createnewscriptsentriesinpackage.jsontoautomatestartingNotes

againstthatdatabase.

StoringnotesinMongoDBMongoDBiswidelyusedwithNode.jsapplications,asignofwhichisthepopularMEANacronym:MongoDB(orMySQL),Express,Angular,andNode.js.MongoDBisoneoftheleadingNoSQLdatabases.Itisdescribedasascalable,high-performance,opensource,document-orienteddatabase.ItusesJSON-styledocumentswithnopredefined,rigidschemaandalargenumberofadvancedfeatures.Youcanvisittheirwebsiteformoreinformationanddocumentationathttp://www.mongodb.org.

DocumentationontheNode.jsdriverforMongoDBcanbefoundathttps://www.npmjs.com/package/mongodbandhttp://mongodb.github.io/node-mongodb-native/.

MongooseisapopularORMforMongoDB(http://mongoosejs.com/).Inthissection,we'llusethenativeMongoDBdriverinstead,butMongooseisaworthyalternative.

YouwillneedarunningMongoDBinstance.Thecompose.io(https://www.compose.io/)andScaleGrid.io(https://scalegrid.io/)hostedserviceprovidersofferhostedMongoDBservices.Nowadays,itisstraightforwardtohostMongoDBasaDockercontaineraspartofasystembuiltofotherDockercontainers.We'lldothisinChapter11,UnitTestingandFunctionalTesting.

It'spossibletosetupatemporaryMongoDBinstancefortestingon,say,yourlaptop.Itisavailableinalltheoperatingsystempackagemanagementsystems,andtheMongoDBwebsitehasinstructions(https://docs.mongodb.org/manual/installation/).

Onceinstalled,it'snotnecessarytosetupMongoDBasabackgroundservice.Instead,youcanrunacoupleofsimplecommandstogetaMongoDBinstancerunningintheforegroundofacommandwindow,whichyoucankillandrestartanytimeyoulike.

Inonecommandwindow,runthefollowing:

$mkdirdata

$mongod--dbpathdata

Inanothercommandwindow,youcantestitasfollows:

$mongo

MongoDBshellversion:3.0.8

connectingto:test

WelcometotheMongoDBshell.

Forinteractivehelp,type"help".

Formorecomprehensivedocumentation,see

http://docs.mongodb.org/

Questions?Trythesupportgroup

http://groups.google.com/group/mongodb-user

>db.foo.save({a:1});

WriteResult({"nInserted":1})

>db.foo.find();

{"_id":ObjectId("56c0c98673f65b7988a96a77"),"a":1}

>

bye

Thissavesadocumentinthecollectionnamedfoo.Thesecondcommandfindsalldocumentsinfoo,printingthemoutforyou.The_idfieldisaddedbyMongoDBandservesasadocumentidentifier.Thisisusefulfortestinganddebugging.Forarealdeployment,yourMongoDBservermustbeproperlyinstalledonaserver.SeetheMongoDBdocumentationfortheseinstructions.

MongoDBmodelfortheNotesapplicationNowthatyou'veprovedyouhaveaworkingMongoDBserver,let'sgettowork.

InstallingtheNode.jsdriverisassimpleasrunningthefollowingcommand:

$npminstallmongodb@3.x--save

Nowcreateanewfile,models/notes-mongodb.mjs:

importutilfrom'util';

importNotefrom'./Note';

importmongodbfrom'mongodb';

constMongoClient=mongodb.MongoClient;

importDBGfrom'debug';

constdebug=DBG('notes:notes-mongodb');

consterror=DBG('notes:error-mongodb');

varclient;

asyncfunctionconnectDB(){

if(!client)client=awaitMongoClient.connect(process.env.MONGO_URL);

return{

db:client.db(process.env.MONGO_DBNAME),

client:client

};

}

TheMongoClientclassisusedtoconnectwithaMongoDBinstance.TherequiredURL,whichwillbespecifiedthroughanenvironmentvariable,usesastraightforwardformat:mongodb://localhost/.Thedatabasenameisspecifiedviaanotherenvironmentvariable.

Documentationforthecorrespondingobjectscanbefoundathttp://mongodb.github.io/node-mongodb-native/2.2/api/MongoClient.html

forMongoClientandhttp://mongodb.github.io/node-mongodb-native/2.2/api/Db.htmlforDb

Thiscreatesthedatabaseclient,andthenopensthedatabaseconnection.BothobjectsarereturnedfromconnectDBinananonymousobject.ThegeneralpatternforMongoDBoperationsisasfollows:

(async()=>{

constclient=awaitMongoClient.connect(process.env.MONGO_URL);

constdb=client.db(process.env.MONGO_DBNAME);

//performdatabaseoperationsusingdbobject

client.close();

})();

Therefore,ourmodelmethodsrequirebothclientanddbobjects,becausetheywilluseboth.Let'sseehowthat'sdone:

exportasyncfunctioncreate(key,title,body){

const{db,client}=awaitconnectDB();

constnote=newNote(key,title,body);

constcollection=db.collection('notes');

awaitcollection.insertOne({notekey:key,title,body});

returnnote;

}

exportasyncfunctionupdate(key,title,body){

const{db,client}=awaitconnectDB();

constnote=newNote(key,title,body);

constcollection=db.collection('notes');

awaitcollection.updateOne({notekey:key},{$set:{title,body}});

returnnote;

}

Weretrievedbandclientintoindividualvariablesusingadestructuringassignment.

MongoDBstoresalldocumentsincollections.Acollectionisagroupofrelateddocuments,andacollectionisanalogoustoatableinarelationaldatabase.ThismeanscreatinganewdocumentorupdatinganexistingonestartsbyconstructingitasaJavaScriptobject,andthenaskingMongoDBtosavethatobjecttothedatabase.MongoDBautomaticallyencodesthe

objectintoitsinternalrepresentation.

Thedb.collectionmethodgivesusaCollectionobjectwithwhichwecanmanipulatethenamedcollection.Seeitsdocumentationathttp://mongodb.github.io/node-mongodb-native/2.2/api/Collection.html.

Asthemethodnameimplies,insertOneinsertsonedocumentintothecollection.Likewise,theupdateOnemethodfirstfindsadocument(inthiscase,bylookingupthedocumentwiththematchingnotekeyfield),andthenchangesfieldsinthedocumentasspecified.

You'llseethatthesemethodsreturnaPromise.ThemongodbdriversupportsbothcallbacksandPromises.Manymethodswillinvokethecallbackfunctionifoneisprovided,otherwiseitreturnsaPromisethatwilldelivertheresultsorerrors.And,ofcourse,sincewe'reusingasyncfunctions,theawaitkeywordmakesthissoclean.

Furtherdocumentationcanbefoundatthefollowinglinks:Insert:https://docs.mongodb.org/getting-started/node/insert/.Update:https://docs.mongodb.org/getting-started/node/update/.

Next,let'slookatreadinganotefromMongoDB:

exportasyncfunctionread(key){

const{db,client}=awaitconnectDB();

constcollection=db.collection('notes');

constdoc=awaitcollection.findOne({notekey:key});

constnote=newNote(doc.notekey,doc.title,doc.body);

returnnote;

}

Themongodbdriversupportsseveralvariantsoffindoperations.Inthiscase,theNotesapplicationensuresthatthereisexactlyonedocumentmatchingagivenkey.Therefore,wecanusethefindOnemethod.Asthenameimplies,findOnewillreturnthefirstmatchingdocument.

TheargumenttofindOneisaquerydescriptor.Thissimplequerylooksfordocumentswhosenotekeyfieldmatchestherequestedkey.Anemptyquerywill,ofcourse,matchalldocumentsinthecollection.Youcanmatch

againstotherfieldsinasimilarway,andthequerydescriptorcandomuchmore.Fordocumentationonqueries,visithttps://docs.mongodb.org/getting-started/node/query/.

TheinsertOnemethodweusedearlieralsotookthesamekindofquerydescriptor.

Inordertosatisfythecontractforthisfunction,wecreateaNoteobjectandthenreturnittothecaller.Hence,wecreateaNoteusingthedataretrievedfromthedatabase:

exportasyncfunctiondestroy(key){

const{db,client}=awaitconnectDB();

constcollection=db.collection('notes');

awaitcollection.findOneAndDelete({notekey:key});

}

OneofthefindvariantsisfindOneAndDelete.Asthenameimplies,itfindsonedocumentmatchingthequerydescriptor,andthendeletesthatdocument:

exportasyncfunctionkeylist(){

const{db,client}=awaitconnectDB();

constcollection=db.collection('notes');

constkeyz=awaitnewPromise((resolve,reject)=>{

varkeyz=[];

collection.find({}).forEach(

note=>{keyz.push(note.notekey);},

err=>{

if(err)reject(err);

elseresolve(keyz);

}

);

});

returnkeyz;

}

Here,we'reusingthebasefindoperationandgivingitanemptyquerysothatitmatcheseverydocument.Whatwe'retoreturnisanarraycontainingthenotekeyforeverydocument.

AllofthefindoperationsreturnaCursorobject.Thedocumentationcanbefoundathttp://mongodb.github.io/node-mongodb-native/2.1/api/Cursor.html.

TheCursorobjectis,asthenameimplies,apointerintoaresultsetfromaquery.Ithasanumberofusefulfunctionsrelatedtooperatingonaresultset.Forexample,youcanskipthefirstfewitemsintheresults,orlimitthesizeoftheresultset,orperformthefilterandmapoperations.

TheCursor.forEachmethodtakestwocallbackfunctions.Thefirstiscalledoneveryelementintheresultset.Inthiscase,wecanusethattorecordjustthenotekeyinanarray.Thesecondcallbackiscalledafterallelementsintheresultsethavebeenprocessed.Weusethistoindicatesuccessorfailure,andtoreturnthekeyzarray.

BecauseforEachusesthispattern,itdoesnothaveanoptionforsupplyingaPromise,andwehavetocreatethePromiseourselves,asshownhere:

exportasyncfunctioncount(){

const{db,client}=awaitconnectDB();

constcollection=db.collection('notes');

constcount=awaitcollection.count({});

returncount;

}

exportasyncfunctionclose(){

if(client)client.close();

client=undefined;

}

Thecountmethodtakesaquerydescriptorand,asthenameimplies,countsthenumberofmatchingdocuments.

RunningtheNotesapplicationwithMongoDBNowthatwehaveourMongoDBmodel,wecangetreadytorunNoteswithit.

Bynowyouknowthedrill;addthistothescriptssectionofpackage.json:"start-mongodb":"MONGO_URL=mongodb://localhost/MONGO_DBNAME=chap07NOTES_MODEL=mongodbnode--experimental-modules./bin/www.mjs",

TheMONGO_URLenvironmentvariableistheURLtoconnectwithyourMongoDBdatabase.

YoucanstarttheNotesapplicationasfollows:

$DEBUG=notes:*npmrunstart-mongodb

>notes@0.0.0start-mongodbUsersdavid/chap07/notes

>MONGO_URL=mongodb://localhost/MONGO_DBNAME=chap07NOTES_MODEL=mongodbnode

--experimental-modules./bin/www

notes:serverListeningonport3000+0ms

Youcanbrowsetheapplicationathttp://localhost:3000andputitthroughitspaces.Youcankillandrestarttheapplication,andyournoteswillstillbethere.

Addthistothescriptssectionofpackage.json:"server1-mongodb":"MONGO_URL=mongodb://localhost/MONGO_DBNAME=chap07NOTES_MODEL=mongodbPORT=3001node--experimental-modules./bin/www.mjs","server2-mongodb":"MONGO_URL=mongodb://localhost/MONGO_DBNAME=chap07NOTES_MODEL=mongodbPORT=3002

node--experimental-modules./bin/www.mjs",

YouwillbeabletostarttwoinstancesoftheNotesapplication,andseethatbothsharethesamesetofnotes.

SummaryInthischapter,wewentthrougharealwhirlwindofdifferentdatabasetechnologies.Whilewelookedatthesamesevenfunctionsoverandover,it'susefultobeexposedtothevariousdatastoragemodelsandwaysofgettingthingsdone.Evenso,weonlytouchedthesurfaceofoptionsforaccessingdatabasesanddatastorageenginesfromNode.js.

Byabstractingthemodelimplementationscorrectly,wewereabletoeasilyswitchdatastorageengineswhilenotchangingtherestoftheapplication.Wedidskiparoundtheissueofsettingupdatabaseservers.Aspromised,we'llgettothatinChapter10,DeployingNode.jsApplications,whenweexploreproductiondeploymentofNode.jsapplications.

Byfocusingthemodelcodeonthepurposeofstoringdata,boththemodelsandtheapplicationshouldbeeasiertotest.Theapplicationcanbetestedwithamockdatamodulethatprovidesknownpredictablenotesthatcanbecheckedpredictably.We'lllookatthisinmoredepthinChapter11,UnitTestingandFunctionalTesting.

Inthenextchapter,we'llfocusonauthenticatingourusersusingOAuth2.

MultiuserAuthenticationtheMicroserviceWayNowthatourNotesapplicationcansaveitsdatainadatabase,wecanthinkaboutthenextphaseofmakingthisarealapplication,namelyauthenticatingourusers.

It'ssonaturaltologintoawebsitetouseitsservices.Wedoiteveryday,andweeventrustbankingandinvestmentorganizationstosecureourfinancialinformationthroughloginproceduresonawebsite.HTTPisastatelessprotocol,andawebapplicationcannottellmuchaboutoneHTTPrequestversusanother.BecauseHTTPisstateless,HTTPrequestsdonotnativelyknowwhethertheuserdrivingthewebbrowserisloggedin,theuser'sidentity,orevenwhethertheHTTPrequestwasinitiatedbyahumanbeing.

Thetypicalmethodforuserauthenticationistosendacookietothebrowsercontainingatokentocarryuseridentity.Thecookieneedstocontaindataidentifyingthebrowserandwhetherthatbrowserisloggedin.Thecookiewillthenbesentwitheveryrequest,lettingtheapplicationtrackwhichuseraccountisassociatedwiththebrowser.

WithExpress,thebestwaytodothisiswiththeexpress-sessionmiddleware.Itstoresdataasacookieandlooksforthatdataoneverybrowserrequest.Itiseasytoconfigure,butisnotacompletesolutionforuserauthentication.Thereareseveraladd-onmodulesthathandleuserauthentication,andsomeevensupportauthenticatingusersagainstthird-partywebsites,suchasFacebookorTwitter.

Onepackageappearstobeleadingthepackinuserauthentication–Passport(http://passportjs.org/).Itsupportsalonglistofservicesagainstwhichtoauthenticate,makingiteasytodevelopawebsitethatletsusers

signupwithcredentialsfromanotherwebsite,forexample,Twitter.Another,express-authentication(https://www.npmjs.com/package/express-authentication),billsitselfastheopinionatedalternativetoPassport.

WewillusePassporttoauthenticateusersagainstbothalocallystoredusercredentialsdatabaseandusingOAuth2toauthenticateagainstaTwitteraccount.We'llalsotakethisasanopportunitytoexploreREST-basedmicroserviceimplementationwithNode.js.

Inthischapter,we'lldiscussthefollowingthreeaspectsofthisphase:

Creatingamicroservicetostoreuserprofile/authenticationdata.

Userauthenticationwithalocallystoredpassword.

UsingOAuth2tosupportauthenticationviathird-partyservices.Specifically,we'lluseTwitterasathird-partyauthenticationservice.

Let'sgetstarted!

Thefirstthingtodoisduplicatethecodeusedforthepreviouschapter.Forexample,ifyoukeptthatcodeinchap07/notes,createanewdirectory,chap08/notes.

CreatingauserinformationmicroserviceWecouldimplementuserauthenticationandaccountsbysimplyaddingausermodel,andafewroutesandviewstotheexistingNotesapplication.Whileitwouldbeaccomplishable,isthiswhatwewoulddoinareal-worldproductionapplication?

Considerthehighvalueofuseridentityinformation,andthesuper-strongneedforrobustandreliableuserauthentication.Websiteintrusionshappenregularly,anditseemstheitemmostfrequentlystolenisuseridentities.

Canyoudesignandbuildauserauthenticationsystemwiththerequiredlevelofsecurity?Onethatisprobablysafeagainstallkindsofintruders?

Aswithsomanyothersoftwaredevelopmentproblems,it'sbesttouseapre-existingauthenticationlibrary,preferablyonewithalongtrackrecord,wheresignificantbugshavebeenfixedalready.

Anotherissueisarchitecturalchoicestopromotesecurity.Bugswilloccurandthetalentedmiscreantswillbreakin.Wallingofftheuserinformationdatabaseisanexcellentideatolimittherisk.

Keepingauserinformationdatabaseenablesyoutoauthenticateyourusers,presentuserprofiles,helpusersconnectwitheachother,andsoforth.Thoseareusefulservicestooffertowebsiteusers,buthowcanyoulimittheriskthatdatawillfallintothewronghands?

Inthischapter,we'lldevelopauserauthenticationmicroservice.Theplanistoeventuallysegregatethatserviceintoawell-protectedbarricadedarea.Thismimicsanarchitecturalchoicemadebysomesites,tostrictlycontrolAPIandevenphysicalaccesstotheuserinformationdatabase,

implementingasmanytechnologicalbarriersaspossibleagainstunapprovedaccess.

Microservicesare,ofcourse,notapanacea,meaningweshouldn'ttrytoforce-fiteveryapplicationintothemicroservicebox.Byanalogy,microservicesareliketheUnixphilosophyofsmalltoolseachdoingonethingwell,whichwemix/match/combineintolargertools.Anotherwordforthisiscomposability.Whilewecanbuildalotofusefulsoftwaretoolswiththatphilosophy,doesitworkforapplicationssuchasPhotoshoporLibreOffice?Whilecomposingasystemoutofsingle-purposetoolsishighlyflexible,onelosestheadvantagesgainedbytightintegrationofcomponents.

ThefirstquestioniswhethertouseaREST-serviceorientedframework,codetheRESTapplicationonbareNode.js,orwhat?YoucouldimplementRESTservicesonthebuilt-inhttpmodule.Theadvantageofusinganapplicationframeworkistheframeworkauthorswillhavealreadybaked-inalotofbestpracticesandbugfixingandsecuritymeasures.Express,forexample,iswidelyused,verypopular,andcaneasilybeusedforRESTservices.ThereareotherframeworksmorealignedwithdevelopingRESTservices,andwe'lluseoneofthem–Restify(http://restify.com/).

Theuserauthenticationserverwillrequiretwomodules:

UsingRestify,implementingtheRESTinterface

AdatamodelusingSequelizetostoreuserdataobjectsinanSQLdatabase

Totesttheservice,we'llwriteacoupleofsimplescriptsforadministeringuserinformationinthedatabase.Wewon'tbeimplementinganadministrativeuserinterfaceintheNotesapplication,andwillrelyonthescriptstoadministertheusers.Asasideeffect,we'llhaveatooltorunacoupleofsimpletestsagainsttheuserservice.

Afterthisserviceisfunctioningcorrectly,we'llsetaboutmodifyingtheNotesapplicationtoaccessuserinformationfromtheservice,whileusingPassporttohandleauthentication.

ThefirststepiscreatinganewdirectorytoholdtheUserInformationmicroservice.ThisshouldbeasiblingdirectorytotheNotesapplication.Ifyoucreatedadirectorynamedchap08/notestoholdtheNotesapplication,thencreateadirectorynamedchap08/userstoholdthemicroservice.

Thenrunthefollowingcommands:

$cdusers

$npminit

..answerquestions

..name-user-auth-server

$npminstalldebug@^2.6.xfs-extra@^5.xjs-yaml@^3.10.x\

restify@^6.3.xrestify-clients@^1.5.xsequelize@^4.31.x\

sqlite3@^3.1.x--save

Thisgetsusreadytostartcoding.We'llusethedebugmoduleforloggingmessages,js-yamltoreadtheSequelizeconfigurationfile,restifyforitsRESTframework,andsequelize/mysql/sqlite3fordatabaseaccess.

UserinformationmodelWe'llbestoringtheuserinformationusingaSequelize-basedmodelinanSQLdatabase.Aswegothroughthis,ponderaquestion:shouldweintegratethedatabasecodedirectlyintotheRESTAPIimplementation?Doingsowouldreducetheuserinformationmicroservicetoonemodule,withdatabasequeriesmingledwithRESThandlers.ByseparatingtheRESTservicefromthedatastoragemodel,wehavethefreedomtoadoptotherdatastoragesystemsbesidesSequelize/SQL.Further,thedatastoragemodelcouldconceivablybeusedinwaysotherthantheRESTservice.

Createanewfilenamedusers-sequelize.mjsinusers,containingthefollowing:

importSequelizefrom"sequelize";

importjsyamlfrom'js-yaml';

importfsfrom'fs-extra';

importutilfrom'util';

importDBGfrom'debug';

constlog=DBG('users:model-users');

consterror=DBG('users:error');

varSQUser;

varsequlz;

asyncfunctionconnectDB(){

if(SQUser)returnSQUser.sync();

constyamltext=awaitfs.readFile(process.env.SEQUELIZE_CONNECT,

'utf8');

constparams=awaitjsyaml.safeLoad(yamltext,'utf8');

if(!sequlz)sequlz=newSequelize(params.dbname,params.username,

params.password,

params.params);

//ThesefieldslargelycomefromthePassport/PortableContacts

schema.

//Seehttp://www.passportjs.org/docs/profile

//

//TheemailsandphotosfieldsarearraysinPortableContacts.

//We'dneedtosetupadditionaltablesforthose.

//

//ThePortableContacts"id"fieldmapstothe"username"field

here

if(!SQUser)SQUser=sequlz.define('User',{

username:{type:Sequelize.STRING,unique:true},

password:Sequelize.STRING,

provider:Sequelize.STRING,

familyName:Sequelize.STRING,

givenName:Sequelize.STRING,

middleName:Sequelize.STRING,

emails:Sequelize.STRING(2048),

photos:Sequelize.STRING(2048)

});

returnSQUser.sync();

}

AswithourSequelize-basedmodelforNotes,weuseaYAMLfiletostoreconnectionconfiguration.We'reevenusingthesameenvironmentvariable,SEQUELIZE_CONNECT.

Whatisthebeststorageserviceforuserauthenticationdata?ByusingSequelize,wehaveourpickofSQLdatabasestochoosefrom.WhileNoSQLdatabasesarealltherage,isthereanyadvantagetousingonetostoreuserauthenticationdata?Nope.AnSQLserverwilldothejobjustfine,andSequelizeallowsusthefreedomofchoice.

It'stemptingtosimplifytheoverallsystembyusingthesamedatabaseinstancetostorenotesanduserinformation,andtouseSequelizeforboth.Butwe'vechosentosimulateasecuredserverforuserdata.Thatcallsforthedatatobeinseparatedatabaseinstances,preferablyonseparateservers.Ahighlysecureapplicationdeploymentmightputtheuserinformationserviceoncompletelyseparateservers,perhapsinaphysicallyisolateddatacenter,withcarefullyconfiguredfirewalls,andtheremightevenbearmedguardsatthedoor.

Theuserprofileschemashownhereisderivedfromthenormalizedprofile

providedbyPassport;refertohttp://www.passportjs.org/docs/profileformoreinformation.Passportwillharmonizeinformationgivenbythird-partyservicesintoasingleobjectdefinition.Tosimplifyourcode,we'resimplyusingtheschemadefinedbyPassport:

exportasyncfunctioncreate(username,password,provider,familyName,

givenName,middleName,emails,photos){

constSQUser=awaitconnectDB();

returnSQUser.create({

username,password,provider,

familyName,givenName,middleName,

emails:JSON.stringify(emails),photos:JSON.stringify(photos)

});

}

exportasyncfunctionupdate(username,password,provider,familyName,

givenName,middleName,emails,photos){

constuser=awaitfind(username);

returnuser?user.updateAttributes({

password,provider,

familyName,givenName,middleName,

emails:JSON.stringify(emails),

photos:JSON.stringify(photos)

}):undefined;

}

Ourcreateandupdatefunctionstakeuserinformationandeitheraddanewrecordorupdateanexistingrecord:

exportasyncfunctionfind(username){

constSQUser=awaitconnectDB();

constuser=awaitSQUser.find({where:{username:username}});

constret=user?sanitizedUser(user):undefined;

returnret;

}

Thisletsuslookupauserinformationrecord,andwereturnasanitizedversionofthatdata.

RememberthatSequelizereturnsaPromiseobject.Becausethisisexecutedinsideanasyncfunction,theawaitkeywordwillresolvethePromise,causinganyerrortobethrownorresultstobeprovidedasthereturnvalue.Inturn,asyncfunctionsreturnaPromisetothecaller.

Becausewe'resegregatingtheuserdatafromtherestoftheNotesapplication,wewanttoreturnasanitizedobjectratherthantheactualSQUserobject.WhatiftherewassomeinformationleakagebecausewesimplysenttheSQUserobjectbacktothecaller?ThesanitizedUserfunction,shownlater,createsananonymousobjectwithexactlythefieldswewantexposedtotheothermodules:

exportasyncfunctiondestroy(username){

constSQUser=awaitconnectDB();

constuser=awaitSQUser.find({where:{username:username}});

if(!user)thrownewError('Didnotfindrequested'+username+'to

delete');

user.destroy();

}

Thisletsussupportdeletinguserinformation.WedothisaswedidfortheNotesSequelizemodel,byfirstfindingtheuserobjectandthencallingitsdestroymethod:

exportasyncfunctionuserPasswordCheck(username,password){

constSQUser=awaitconnectDB();

constuser=awaitSQUser.find({where:{username:username}});

if(!user){

return{check:false,username:username,message:"Couldnot

finduser"};

}elseif(user.username===username&&user.password===

password){

return{check:true,username:user.username};

}else{

return{check:false,username:username,message:"Incorrect

password"};

}

}

Thisletsussupportthecheckingofuserpasswords.Thethreeconditionstohandleareasfollows:

Whetherthere'snosuchuser

Whetherthepasswordsmatched

Whethertheydidnotmatch

Theobjectwereturnletsthecallerdistinguishbetweenthosecases.Thecheckfieldindicateswhethertoallowthisusertobeloggedin.Ifcheckisfalse,there'ssomereasontodenytheirrequesttologin,andthemessageiswhatshouldbedisplayedtotheuser:

exportasyncfunctionfindOrCreate(profile){

constuser=awaitfind(profile.id);

if(user)returnuser;

returnawaitcreate(profile.id,profile.password,profile.provider,

profile.familyName,profile.givenName,

profile.middleName,

profile.emails,profile.photos);

}

Thiscombinestwoactionsinonefunction:first,toverifywhetherthenameduserexistsand,ifnot,tocreatethatuser.Primarily,thiswillbeusedwhileauthenticatingagainstthird-partyservices:

exportasyncfunctionlistUsers(){

constSQUser=awaitconnectDB();

constuserlist=awaitSQUser.findAll({});

returnuserlist.map(user=>sanitizedUser(user));

}

Listtheexistingusers.ThefirststepisusingfindAlltogiveusthelistoftheusersasanarrayofSQUserobjects.Thenwesanitizethatlistsowedon'texposeanydatathatwedon'twantexposed:

exportfunctionsanitizedUser(user){

varret={

id:user.username,username:user.username,

provider:user.provider,

familyName:user.familyName,givenName:user.givenName,

middleName:user.middleName,

emails:JSON.parse(user.emails),

photos:JSON.parse(user.photos)

};

try{

ret.emails=JSON.parse(user.emails);

}catch(e){ret.emails=[];}

try{

ret.photos=JSON.parse(user.photos);

}catch(e){ret.photos=[];}

returnret;

}

Thisisourutilityfunctiontoensureweexposeacarefullycontrolledsetofinformationtothecaller.Withthisservice,we'reemulatingasecureduserinformationservicethat'swalledofffromotherapplications.Aswesaidearlier,thisfunctionreturnsananonymoussanitizedobjectwhereweknowexactlywhat'sintheobject.

It'sveryimportanttodecodetheJSONstringweputintothedatabase.RememberthatwestoredtheemailsandphotosdatausingJSON.stringifyinthedatabase.UsingJSON.parse,wedecodethosevalues,justlikeaddinghotwatertoinstantcoffeeproducesadrinkablebeverage.

ARESTserverforuserinformationWearebuildingourwaytowardsintegratinguserinformationandauthenticationintotheNotesapplication.ThenextstepistowraptheuserdatamodelwejustcreatedintoaRESTserver.Afterthat,we'llcreateacoupleofscriptssothatwecanaddsomeusers,performotheradministrativetasks,andgenerallyverifythattheserviceworks.Finally,we'llextendtheNotesapplicationwithloginandlogoutsupport.

Inthepackage.jsonfile,changethemaintagtothefollowinglineofcode:

"main":"user-server.mjs",

Thencreateafilenameduser-server.mjs,containingthefollowingcode:

importrestifyfrom'restify';

importutilfrom'util';

importDBGfrom'debug';

constlog=DBG('users:service');

consterror=DBG('users:error');

import*asusersModelfrom'./users-sequelize';

varserver=restify.createServer({

name:"User-Auth-Service",

version:"0.0.1"

});

server.use(restify.plugins.authorizationParser());

server.use(check);

server.use(restify.plugins.queryParser());

server.use(restify.plugins.bodyParser({

mapParams:true

}));

ThecreateServermethodcantakealonglistofconfigurationoptions.Thesetwomaybeusefulforidentifyinginformation.

AswithExpressapplications,theserver.usecallsinitializewhatExpresswouldcallmiddlewarefunctions,butwhichRestifycallshandlerfunctions.ThesearecallbackfunctionswhoseAPIisfunction(req,res,next).AswithExpress,thesearetherequestandresponseobjects,andnextisafunctionwhich,whencalled,carriesexecutiontothenexthandlerfunction.

UnlikeExpress,everyhandlerfunctionmustcallthenextfunction.InordertotellRestifytostopprocessingthroughhandlers,thenextfunctionmustbecalledasnext(false).Callingnextwithanerrorobjectalsocausestheexecutiontoend,andtheerrorissentbacktotherequestor.

Thehandlerfunctionslistedheredotwothings:authorizerequestsandhandleparsingparametersfromboththeURLandthepostrequestbody.TheauthorizationParserfunctionlooksforHTTPbasicauthheaders.ThecheckfunctionisshownlaterandemulatestheideaofanAPItokentocontrolaccess.

Refertohttp://restify.com/docs/plugins-api/formoreinformationonthebuilt-inhandlersavailableinRestify.

Addthistouser-server.mjs:

//Createauserrecord

server.post('/create-user',async(req,res,next)=>{

try{

varresult=awaitusersModel.create(

req.params.username,req.params.password,

req.params.provider,

req.params.familyName,req.params.givenName,

req.params.middleName,

req.params.emails,req.params.photos);

res.send(result);

next(false);

}catch(err){res.send(500,err);next(false);}

});

AsforExpress,theserver.VERBfunctionsletusdefinethehandlersforspecificHTTPactions.ThisroutehandlesaPOSTon/create-user,and,asthenameimplies,thiswillcreateauserbycallingtheusersModel.createfunction.

AsaPOSTrequest,theparametersarriveinthebodyoftherequestratherthanasURLparameters.BecauseofthemapParamsflagonthebodyParamshandler,theargumentspassedintheHTTPbodyareaddedtoreq.params.

WesimplycallusersModel.createwiththeparameterssenttous.Whencompleted,theresultobjectshouldbeauserobject,whichwesendbacktotherequestorusingres.send:

//Updateanexistinguserrecord

server.post('update-user:username',async(req,res,next)=>{

try{

varresult=awaitusersModel.update(

req.params.username,req.params.password,

req.params.provider,

req.params.familyName,req.params.givenName,

req.params.middleName,

req.params.emails,req.params.photos);

res.send(usersModel.sanitizedUser(result));

next(false);

}catch(err){res.send(500,err);next(false);}

});

The/update-userrouteishandledinasimilarway.However,wehaveputtheusernameparameterontheURL.LikeExpress,RestifyletsyouputnamedparametersintheURLlikeasfollows.Suchnamedparametersarealsoaddedtoreq.params.

WesimplycallusersModel.updatewiththeparameterssenttous.That,too,returnsanobjectwesendbacktothecallerwithres.send:

//Findauser,ifnotfoundcreateonegivenprofileinformation

server.post('/find-or-create',async(req,res,next)=>{

log('find-or-create'+util.inspect(req.params));

try{

varresult=awaitusersModel.findOrCreate({

id:req.params.username,username:req.params.username,

password:req.params.password,provider:

req.params.provider,

familyName:req.params.familyName,givenName:

req.params.givenName,

middleName:req.params.middleName,

emails:req.params.emails,photos:req.params.photos

});

res.send(result);

next(false);

}catch(err){res.send(500,err);next(false);}

});

ThishandlesourfindOrCreateoperation.Wesimplydelegatethistothemodelcode,asdonepreviously.

Asthenameimplies,we'lllooktoseewhetherthenameduseralreadyexistsand,ifso,simplyreturnthatuser,otherwiseitwillbecreated:

//Findtheuserdata(doesnotreturnpassword)

server.get('find:username',async(req,res,next)=>{

try{

varuser=awaitusersModel.find(req.params.username);

if(!user){

res.send(404,newError("Didnotfind"+

req.params.username));

}else{

res.send(user);

}

next(false);

}catch(err){res.send(500,err);next(false);}

});

Here,wesupportlookinguptheuserobjectfortheprovidedusername.

Iftheuserwasnotfound,thenwereturna404statuscodebecauseitindicatesaresourcethatdoesnotexist.Otherwise,wesendtheobjectthatwasretrieved:

//Delete/destroyauserrecord

server.del('destroy:username',async(req,res,next)=>{

try{

awaitusersModel.destroy(req.params.username);

res.send({});

next(false);

}catch(err){res.send(500,err);next(false);}

});

ThisishowwedeleteauserfromtheNotesapplication.TheDELHTTPverbismeanttobeusedtodeletethingsonaserver,makingitthenaturalchoiceforthisfunctionality:

//Checkpassword

server.post('/passwordCheck',async(req,res,next)=>{

try{

awaitusersModel.userPasswordCheck(

req.params.username,req.params.password);

res.send(check);

next(false);

}catch(err){res.send(500,err);next(false);}

});

Thisisanotheraspectofkeepingthepasswordsolelywithinthisserver.Thepasswordcheckisperformedbythisserver,ratherthanintheNotesapplication.WesimplycalltheusersModel.userPasswordCheckfunctionshownearlierandsendbacktheobjectitreturns:

//Listusers

server.get('/list',async(req,res,next)=>{

try{

varuserlist=awaitusersModel.listUsers();

if(!userlist)userlist=[];

res.send(userlist);

next(false);

}catch(err){res.send(500,err);next(false);}

});

Then,finally,ifrequired,wesendalistofNotesapplicationusersbacktotherequestor.Incasenolistofusersisavailable,weatleastsendanemptyarray:

server.listen(process.env.PORT,"localhost",function(){

log(server.name+'listeningat'+server.url);

});

//MimicAPIKeyauthentication.

varapiKeys=[{

user:'them',

key:'D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF'

}];

functioncheck(req,res,next){

if(req.authorization){

varfound=false;

for(letauthofapiKeys){

if(auth.key===req.authorization.basic.password

&&auth.user===req.authorization.basic.username){

found=true;

break;

}

}

if(found)next();

else{

res.send(401,newError("Notauthenticated"));

next(false);

}

}else{

res.send(500,newError('NoAuthorizationKey'));

next(false);

}

}

AswiththeNotesapplication,welistentotheportnamedinthePORTenvironmentvariable.Byexplicitlylisteningonlyonlocalhost,we'lllimitthescopeofsystemsthatcanaccesstheuserauthenticationserver.Inarealdeployment,wemighthavethisserverbehindafirewallwithatightlistofhostsystemsallowedtohaveaccess.

Thislastfunction,check,implementsauthenticationfortheRESTAPIitself.Thisisthehandlerfunctionweaddedearlier.

ItrequiresthecallertoprovidecredentialsontheHTTPrequestusingthebasicauthheaders.TheauthorizationParserhandlerlooksforthisandgivesittousonthereq.authorization.basicobject.Thecheckfunctionsimplyverifiesthatthenameduserandpasswordcombinationexistsinthelocalarray.

ThisismeanttomimicassigninganAPIkeytoanapplication.Thereareseveralwaysofdoingso;thisisjustone.

ThisapproachisnotlimitedtojustauthenticatingusingHTTPbasicauth.TheRestifyAPIletsuslookatanyheaderintheHTTPrequest,meaningwecouldimplementanykindofsecuritymechanismwelike.Thecheckfunctioncouldimplementsomeothersecuritymethod,withtherightcode.

Becauseweaddedcheckwiththeinitialsetofserver.usehandlers,itiscalledoneveryrequest.Therefore,everyrequesttothisservermustprovidetheHTTPbasicauthcredentialsrequiredbythischeck.

ThisstrategyisgoodifyouwanttocontrolaccesstoeverysinglefunctioninyourAPI.Fortheuserauthenticationservice,that'sprobablyagoodidea.SomeRESTservicesintheworldhavecertainAPIfunctionsthatareopentotheworldandothersprotectedbyanAPItoken.Toimplementthat,thecheckfunctionshouldnotbeconfiguredamongtheserver.usehandlers.Instead,itshouldbeaddedtotheappropriateroutehandlersasfollows:

server.get('requesturl',authHandler,(req,res,next)=>{

..

});

SuchanauthHandlerwouldbecodedsimilarlytoourcheckfunction.Afailuretoauthenticateisindicatedbysendinganerrorcodeandusingnext(false)toendtheroutingfunctionchain.

Wenowhavethecompletecodefortheuserauthenticationserver.ItdefinesseveralrequestURLs,andforeach,thecorrespondingfunctionintheusermodeliscalled.

NowweneedaYAMLfiletoholdthedatabasecredentials,socreatesequelize-sqlite.yaml,containingthefollowingcode:

dbname:users

username:

password:

params:

dialect:sqlite

storage:users-sequelize.sqlite3

SincethisisSequelize,it'seasytoswitchtootherdatabaseenginessimplybysupplyingadifferentconfigurationfile.RememberthatthefilenameofthisconfigurationfilemustappearintheSEQUELIZE_CONNECTenvironmentvariable.

Finally,package.jsonshouldlookasfollows:

{

"name":"user-auth-server",

"version":"0.0.1",

"description":"",

"main":"user-server.js",

"scripts":{

"start":"DEBUG=users:*PORT=3333SEQUELIZE_CONNECT=sequelize-sqlite.yaml

node--experimental-modulesuser-server"

},

"author":"",

"license":"ISC",

"engines":{

"node":">=8.9"

},

"dependencies":{

"debug":"^2.6.9",

"fs-extra":"^5.x",

"js-yaml":"^3.10.x",

"mysql":"^2.15.x",

"restify":"^6.3.x",

"restify-clients":"^1.5.x",

"sqlite3":"^3.1.x",

"sequelize":"^4.31.x"

}

}

Weconfigurethisservertolistenonport3333usingthedatabasecredentialswejustgaveandwithdebuggingoutputfortheservercode.

Youcannowstarttheuserauthenticationserver:

$npmstart

>user-auth-server@0.0.1startUsersdavid/chap08/users

>DEBUG=users:*PORT=3333SEQUELIZE_CONNECT=sequelize-mysql.yamlnodeuser-

server

users:serverUser-Auth-Servicelisteningathttp://127.0.0.1:3333+0ms

Butwedon'thaveanywaytointeractwiththisserver,yet.

ScriptstotestandadministertheuserauthenticationserverTogiveourselvesassurancethattheuserauthenticationserverworks,let'swriteacoupleofscriptstoexercisetheAPI.Becausewe'renotgoingtotakethetimetowriteanadministrativebackendtotheNotesapplication,thesescriptswillletusaddanddeleteuserswhoareallowedaccesstoNotes.Thesescriptswilllivewithintheuserauthenticationserverpackagedirectory.

TheRestifypackagesupportscodingRESTservers.FortheRESTclients,we'reusingacompanionlibrary,restify-clients,whichhasbeenspunoutofRestify.

Createafilenamedusers-add.js,containingthefollowingcode:

'usestrict';

constutil=require('util');

constrestify=require('restify-clients');

varclient=restify.createJsonClient({

url:'http://localhost:'+process.env.PORT,

version:'*'

});

client.basicAuth('them','D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF');

client.post('/create-user',{

username:"me",password:"w0rd",provider:"local",

familyName:"Einarrsdottir",givenName:"Ashildr",middleName:"",

emails:[],photos:[]

},

(err,req,res,obj)=>{

if(err)console.error(err.stack);

elseconsole.log('Created'+util.inspect(obj));

});

ThisisthebasicstructureofaRestifyclient.WecreatetheClientobject–wehaveachoicebetweentheJsonClient,asusedhere,theStringClient,andtheHttpClient.TheHTTPbasicAuthcredentialsareeasytoset,asshownhere.

Thenwemaketherequest,inthiscaseaPOSTrequeston/create-user.BecauseitisaPOSTrequest,theobjectwespecifyhereisformattedbyRestifyintoHTTPPOSTbodyparameters.Aswesawearlier,theserverhasthebodyParserhandlerfunctionconfigured,whichconvertsthosebodyparametersintothereq.paramobject.

IntheRestifyclient,asfortheRestifyserver,weusethevariousHTTPmethodsbycallingclient.METHOD.BecauseitisaPOSTrequest,weuseclient.post.Whentherequestfinishes,thecallbackfunctionisinvoked.

Beforerunningthesescripts,starttheauthenticationserverinonewindowusingthefollowingcommand:

$npmstart

Nowrunthetestscriptusingthefollowingcommand:

$PORT=3333nodeusers-add.js

Created{id:1,username:'me',password:'w0rd',provider:'local',

familyName:'Einarrsdottir',givenName:'Ashildr',

middleName:'',

emails:'[]',photos:'[]',

updatedAt:'2016-02-24T02:34:41.661Z',

createdAt:'2016-02-24T02:34:41.661Z'}

Wecaninspectourhandiworkusingthefollowingcommand:

$sqlite3users-sequelize.sqlite3

SQLiteversion3.10.22016-01-2015:27:19

Enter".help"forusagehints.

sqlite>.schemausers

CREATETABLE`Users`(`id`INTEGERPRIMARYKEYAUTOINCREMENT,`username`

VARCHAR(255)UNIQUE,`password`VARCHAR(255),`provider`VARCHAR(255),

`familyName`VARCHAR(255),`givenName`VARCHAR(255),`middleName`

VARCHAR(255),`emails`VARCHAR(2048),`photos`VARCHAR(2048),`createdAt`

DATETIMENOTNULL,`updatedAt`DATETIMENOTNULL,UNIQUE(`username`));

sqlite>select*fromusers;

2|me|w0rd|local|Einarrsdottir|Ashildr||[]|[]|2018-01-2105:34:56.629

+00:00|2018-01-2105:34:56.629+00:00

sqlite>^D

Nowlet'swriteascript,users-find.js,tolookupagivenuser:

'usestrict';

constutil=require('util');

constrestify=require('restify-clients');

varclient=restify.createJsonClient({

url:'http://localhost:'+process.env.PORT,

version:'*'

});

client.basicAuth('them','D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF');

client.get('find'+process.argv[2],

(err,req,res,obj)=>{

if(err)console.error(err.stack);

elseconsole.log('Found'+util.inspect(obj));

});

Thissimplycallsthe/findURL,specifyingtheusernamethattheusersuppliesasacommand-lineargument.Notethatthegetoperationdoesnottakeanobjectfullofparameters.Instead,anyparameterswouldbeaddedtotheURL.

It'srunasfollows:

$PORT=3333nodeusers-find.jsme

Found{username:'me',provider:'local',

familyName:'Einarrsdottir',givenName:'Ashildr',

middleName:'',

emails:'[]',photos:'[]'}

Similarly,wecanwritescriptsagainsttheotherRESTfunctions.Butwe

needtogetonwiththerealgoalofintegratingthisintotheNotesapplication.

LoginsupportfortheNotesapplicationNowthatwehaveprovedthattheuserauthenticationserviceisworking,wecansetuptheNotesapplicationtosupportuserlogins.We'llbeusingPassporttosupportlogin/logout,andtheauthenticationservertostoretherequireddata.

Amongtheavailablepackages,Passportstandsoutforsimplicityandflexibility.ItintegratesdirectlywiththeExpressmiddlewarechain,andthePassportcommunityhasdevelopedhundredsofso-calledStrategymodulestohandleauthenticationagainstalonglistofthird-partyservices.Seehttp://www.passportjs.org/forinformationanddocumentation.

AccessingtheuserauthenticationRESTAPIThefirststepistocreateauserdatamodelfortheNotesapplication.Ratherthanretrievingdatafromdatafilesoradatabase,itwilluseRESTtoquerytheserverwejustcreated.Wecouldhavecreatedusermodelcodethatdirectlyaccessesthedatabasebut,forreasonsalreadydiscussed,we'vedecidedtosegregateuserauthenticationintoaseparateservice.

LetusnowturntotheNotesapplication,whichyoumayhavestoredaschap08/notes.We'llbemodifyingtheapplication,firsttoaccesstheuserauthenticationRESTAPI,andthentousePassportforauthorizationandauthentication.

Forthetest/adminscriptsthatwecreatedearlier,weusedtherestify-clientsmodule.Thatpackageisacompaniontotherestifylibrary,whererestifysupportstheserversideoftheRESTprotocolandrestify-clientssupportstheclientside.Theirnamesmightgiveawaythepurpose.

Howevernicetherestify-clientslibraryis,itdoesn'tsupportaPromise-orientedAPI,asisrequiredtoplaywellwithasyncfunctions.Anotherlibrary,superagent,doessupportaPromise-orientedAPI,playswellinasyncfunctions,andthereisacompaniontothatpackage,Supertest,that'susefulinunittesting.We'lluseSupertestinChapter11,UnitTestingandFunctionalTesting,whenwetalkaboutunittesting.Fordocumentation,seehttps://www.npmjs.com/package/superagent:

$npminstallsuperagent@^3.8.x

Createanewfile,models/users-superagent.mjs,containingthefollowingcode:

importrequestfrom'superagent';

importutilfrom'util';

importurlfrom'url';

constURL=url.URL;

importDBGfrom'debug';

constdebug=DBG('notes:users-superagent');

consterror=DBG('notes:error-superagent');

functionreqURL(path){

constrequrl=newURL(process.env.USER_SERVICE_URL);

requrl.pathname=path;

returnrequrl.toString();

}

ThereqURLfunctionreplacestheconnectXYZZYfunctionsthatwewroteinearliermodules.Withsuperagent,wedon'tleaveaconnectionopentotheservice,butopenanewconnectiononeachrequest.ThecommonthingtodoistoformulatetherequestURL.TheuserisexpectedtoprovideabaseURL,suchashttp://localhost:3333/,intheUSER_SERVICE_URLenvironmentvariable.ThisfunctionmodifiesthatURL,usingthenewWHATWGURLsupportinNode.js,touseagivenURLpath:

exportasyncfunctioncreate(username,password,

provider,familyName,givenName,middleName,

emails,photos){

varres=awaitrequest

.post(reqURL('/create-user'))

.send({

username,password,provider,

familyName,givenName,middleName,emails,photos

})

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth('them','D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF');

returnres.body;

}

exportasyncfunctionupdate(username,password,

provider,familyName,givenName,middleName,

emails,photos){

varres=awaitrequest

.post(reqURL(`/update-user/${username}`))

.send({

username,password,provider,

familyName,givenName,middleName,emails,photos

})

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth('them','D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF');

returnres.body;

}

Theseareourcreateandupdatefunctions.Ineachcase,theytakethedataprovided,constructananonymousobject,andPOSTittotheserver.

ThesuperagentlibraryusesanAPIstylewhereonechainstogethermethodcallstoconstructarequest.Thechainofmethodcallscanendina.thenor.endclause,eitherofwhichtakeacallbackfunction.Butleaveoffboth,anditwillreturnaPromise.

Allthroughthislibrary,we'llusethe.authclausetosetuptherequiredauthenticationkey.

Theseanonymousobjectsarealittledifferentthanusual.We'reusinganewES-2015featureherethatwehaven'tdiscussedsofar.RatherthanspecifyingtheobjectfieldsusingthefieldName:fieldValuenotation,ES-2015givesustheoptiontoshortenthiswhenthevariablenameusedforfieldValuematchesthedesiredfieldName.Inotherwords,wecanjustlistthevariablenames,andthefieldnamewillautomaticallymatchthevariablename.

Inthiscase,we'vepurposelychosenvariablenamesfortheparameterstomatchfieldnamesoftheobjectwithparameternamesusedbytheserver.Bydoingso,wecanusethisshortenednotationforanonymousobjects,andourcodeisalittlecleanerbyusingconsistentvariablenamesfrombeginningtoend:

exportasyncfunctionfind(username){

varres=awaitrequest

.get(reqURL(`/find/${username}`))

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth('them','D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF');

returnres.body;

}

Ourfindoperationletsuslookupuserinformation:

exportasyncfunctionuserPasswordCheck(username,password){

varres=awaitrequest

.post(reqURL(`/passwordCheck`))

.send({username,password})

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth('them','D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF');

returnres.body;

}

We'resendingtherequesttocheckpasswordstotheserver.

Apointaboutthismethodisusefultonote.ItcouldhavetakentheparametersintheURL,insteadoftherequestbodyasisdonehere.ButsincerequestURLareroutinelyloggedtofiles,puttingtheusernameandpasswordparametersintheURLmeansuseridentityinformationwouldbeloggedtofilesandpartofactivityreports.Thatwouldobviouslybeaverybadchoice.Puttingthoseparametersintherequestbodynotonlyavoidsthatbadresult,butifanHTTPSconnectiontotheservicewereused,thetransactionwouldbeencrypted:

exportasyncfunctionfindOrCreate(profile){

varres=awaitrequest

.post(reqURL('/find-or-create'))

.send({

username:profile.id,password:profile.password,

provider:profile.provider,

familyName:profile.familyName,

givenName:profile.givenName,

middleName:profile.middleName,

emails:profile.emails,photos:profile.photos

})

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth('them','D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF');

returnres.body;

}

ThefindOrCreatefunctioneitherdiscoverstheuserinthedatabase,orcreatesanewuser.TheprofileobjectwillcomefromPassport,buttakecarefulnoteofwhatwedowithprofile.id.ThePassportdocumentationsaysitwillprovidetheusernameintheprofile.idfield.Butwewantto

storeitasusername,instead:

exportasyncfunctionlistUsers(){

varres=awaitrequest

.get(reqURL('/list'))

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth('them','D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF');

returnres.body;

}

Finally,wecanretrievealistofusers.

LoginandlogoutroutingfunctionsWhatwe'vebuiltsofarisauserdatamodel,withaRESTAPIwrappingthatmodeltocreateourauthenticationinformationservice.Then,withintheNotesapplication,wehaveamodulethatrequestsuserdatafromthisserver.Asofyet,nothingintheNotesapplicationknowsthatthisusermodelexists.Thenextstepistocreatearoutingmoduleforlogin/logoutURLsandtochangetherestofNotestouseuserdata.

Theroutingmoduleiswhereweusepassporttohandleuserauthentication.Thefirsttaskistoinstalltherequiredmodules:

$npminstallpassport@^0.4.xpassport-local@1.x--save

Thepassportmodulegivesustheauthenticationalgorithms.Tosupportdifferentauthenticationmechanisms,thepassportauthorshavedevelopedseveralstrategyimplementations.Theauthenticationmechanisms,orstrategies,correspondtothevariousthird-partyservicesthatsupportauthentication,suchasusingOAuth2toauthenticateagainstservicessuchasFacebook,Twitter,orGitHub.

TheLocalStrategyauthenticatessolelyusingdatastoredlocaltotheapplication,forexample,ouruserauthenticationinformationservice.

Let'sstartbycreatingtheroutingmodule,routes/users.mjs:

importpathfrom'path';

importutilfrom'util';

importexpressfrom'express';

importpassportfrom'passport';

importpassportLocalfrom'passport-local';

constLocalStrategy=passportLocal.Strategy;

import*asusersModelfrom'../models/users-superagent';

import{sessionCookieName}from'../app';

exportconstrouter=express.Router();

importDBGfrom'debug';

constdebug=DBG('notes:router-users');

consterror=DBG('notes:error-users');

Thisbringsinthemodulesweneedforthe/usersrouter.ThisincludesthetwopassportmodulesandtheREST-baseduserauthenticationmodel.

Inapp.mjs,wewillbeaddingsessionsupportsoouruserscanloginandlogout.Thatreliesonstoringacookieinthebrowser,andthecookienameisfoundinthisvariableexportedfromapp.mjs.We'llbeusingthatcookieinamoment:

exportfunctioninitPassport(app){

app.use(passport.initialize());

app.use(passport.session());

}

exportfunctionensureAuthenticated(req,res,next){

try{

//req.userissetbyPassportinthedeserializefunction

if(req.user)next();

elseres.redirect('userslogin');

}catch(e){next(e);}

}

TheinitPassportfunctionwillbecalledfromapp.mjs,anditinstallsthePassportmiddlewareintotheExpressconfiguration.We'lldiscusstheimplicationsofthislaterwhenwegettoapp.mjschanges,butPassportusessessionstodetectwhetherthisHTTPrequestisauthenticatedornot.Itlooksateveryrequestcomingintotheapplication,looksforcluesaboutwhetherthisbrowserisloggedinornot,andattachesdatatotherequestobjectasreq.user.

TheensureAuthenticatedfunctionwillbeusedbyotherroutingmodulesandistobeinsertedintoanyroutedefinitionthatrequiresanauthenticated

logged-inuser.Forexample,editingordeletinganoterequirestheusertobeloggedin,andthereforethecorrespondingroutesinroutes/notes.mjsmustuseensureAuthenticated.Iftheuserisnotloggedin,thisfunctionredirectsthemtousersloginsothattheycandoso:

outer.get('/login',function(req,res,next){

try{

res.render('login',{title:"LogintoNotes",user:req.user,});

}catch(e){next(e);}

});

router.post('/login',

passport.authenticate('local',{

successRedirect:'',/SUCCESS:Gotohomepage

failureRedirect:'login',//FAIL:Gotouserlogin

})

);

Becausethisrouterismountedon/users,alltheserouteswillhave/userprepended.Theusersloginroutesimplyshowsaformrequestingausernameandpassword.Whenthisformissubmitted,welandinthesecondroutedeclaration,withaPOSTonuserslogin.IfpassportdeemsthisasuccessfulloginattemptusingLocalStrategy,thenthebrowserisredirectedtothehomepage.Otherwise,itisredirectedtotheusersloginpage:

router.get('/logout',function(req,res,next){

try{

req.session.destroy();

req.logout();

res.clearCookie(sessionCookieName);

res.redirect('/');

}catch(e){next(e);}

});

WhentheuserrequeststologoutofNotes,theyaretobesenttouserslogout.We'llbeaddingabuttontotheheadertemplateforthispurpose.Thereq.logoutfunctioninstructsPassporttoerasetheirlogincredentials,andtheyarethenredirectedtothehomepage.

Thisfunctiondeviatesfromwhat'sinthePassportdocumentation.There,

wearetoldtosimplycallreq.logout.Butcallingonlythatfunctionsometimesresultsintheusernotbeingloggedout.It'snecessarytodestroythesessionobject,andtoclearthecookie,inordertoensurethattheuserisloggedout.Thecookienameisdefinedinapp.mjs,andweimportedsessionCookieNameforthisfunction:

passport.use(newLocalStrategy(

async(username,password,done)=>{

try{

varcheck=awaitusersModel.userPasswordCheck(username,

password);

if(check.check){

done(null,{id:check.username,username:check.username});

}else{

done(null,false,check.message);

}

}catch(e){done(e);}

}

));

HereiswherewedefineourimplementationofLocalStrategy.Inthecallbackfunction,wecallusersModel.userPasswordCheck,whichmakesaRESTcalltotheuserauthenticationservice.Rememberthatthisperformsthepasswordcheckandthenreturnsanobjectindicatingwhetherthey'reloggedinornot.

Asuccessfulloginisindicatedwhencheck.checkistrue.Forthiscase,wetellPassporttouseanobjectcontainingtheusernameinthesessionobject.Otherwise,wehavetwowaystotellPassportthattheloginattemptwasunsuccessful.Inonecase,weusedone(null,false)toindicateanerrorloggingin,andpassalongtheerrormessageweweregiven.Intheothercase,we'llhavecapturedanexception,andpassalongthatexception.

You'llnoticethatPassportusesacallback-styleAPI.Passportprovidesadonefunction,andwearetocallthatfunctionwhenweknowwhat'swhat.Whileweuseanasyncfunctiontomakeacleanasynchronouscalltothebackendservice,Passportdoesn'tknowhowtogrokthePromisethatwouldbereturned.Therefore,wehavetothrowatry/catcharoundthefunctionbodytocatchanythrownexception:

passport.serializeUser(function(user,done){

try{

done(null,user.username);

}catch(e){done(e);}

});

passport.deserializeUser(async(username,done)=>{

try{

varuser=awaitusersModel.find(username);

done(null,user);

}catch(e){done(e);}

});

Theprecedingfunctionstakecareofencodinganddecodingauthenticationdataforthesession.Allweneedtoattachtothesessionistheusername,aswedidinserializeUser.ThedeserializeUserobjectiscalledwhileprocessinganincomingHTTPrequestandiswherewelookuptheuserprofiledata.Passportwillattachthistotherequestobject.

Login/logoutchangestoapp.jsWehaveafewchangesrequiredinapp.mjs,someofwhichwe'vealreadytouchedon.WedidcarefullyisolatethePassportmoduledependenciestoroutes/users.mjs.Thechangesrequiredinapp.mjssupportthecodeinroutes/users.mjs.

It'snowtimetouncommentalinewetoldyoutocommentoutwaybackinChapter5,YourFirstExpressApplication.Theimportsfortheroutingmoduleswillnowlookasfollows:

import{routerasindex}from'./routes/index';

import{routerasusers,initPassport}from'./routes/users';

import{routerasnotes}from'./routes/notes';

TheUserroutersupportsthe/loginand/logoutURL'saswellasusingPassportforauthentication.WeneedtocallinitPassportforalittlebitofinitialization:

importsessionfrom'express-session';

importsessionFileStorefrom'session-file-store';

constFileStore=sessionFileStore(session);

exportconstsessionCookieName='notescookie.sid';

BecausePassportusessessions,weneedtoenablesessionsupportinExpress,andthesemodulesdoso.Thesession-file-storemodulesavesoursessiondatatodisksothatwecankillandrestarttheapplicationwithoutlosingsessions.It'salsopossibletosavesessionstodatabaseswithappropriatemodules.AfilesystemsessionstoreissuitableonlywhenallNotesinstancesarerunningonthesameservercomputer.Foradistributeddeploymentsituation,you'llneedtouseasessionstorethatrunsonanetwork-wideservice,suchasadatabase.

We'redefiningsessionCookieNameheresoitcanbeusedinmultipleplaces.Bydefault,express-sessionusesacookienamedconnect.sidtostorethesessiondata.Asasmallmeasureofsecurity,it'susefulwhenthere'sapublisheddefaulttouseadifferentnon-defaultvalue.Anytimeweusethedefaultvalue,it'spossiblethatanattackermightknowasecurityflawdependingonthatdefault.

Usethefollowingcommandtoinstallthemodules:

$npminstallexpress-session@1.15.xsession-file-store@1.2.x--save

ExpressSessionsupport,includingallthevariousSessionStoreimplementations,isdocumentedonitsGitHubprojectpageathttps://github.com/expressjs/session.

Addthisinapp.mjs:

app.use(session({

store:newFileStore({path:"sessions"}),

secret:'keyboardmouse',

resave:true,

saveUninitialized:true,

name:sessionCookieName

}));

initPassport(app);

Hereweinitializethesessionsupport.ThefieldnamedsecretisusedtosignthesessionIDcookie.Thesessioncookieisanencodedstringthatisencryptedinpartusingthissecret.IntheExpressSessiondocumentation,theysuggestthestringkeyboardcatforthesecret.But,intheory,whatifExpresshasavulnerability,suchthatknowingthissecretcanmakeiteasiertobreakthesessionlogiconyoursite?Hence,wechoseadifferentstringforthesecretjusttobealittledifferentandperhapsalittlemoresecure.

Similarly,thedefaultcookienameusedbyexpress-sessionisconnect.sid.Here'swherewechangethecookienametoanon-defaultname.

TheFileStorewillstoreitssessiondatarecordsinadirectorynamedsessions.Thisdirectorywillbeauto-createdasneeded:

app.use('',index);

app.use('users',users);

app.use('/notes',notes);

TheprecedingarethethreeroutersusedintheNotesapplication.

Login/logoutchangesinroutes/index.mjsThisroutermodulehandlesthehomepage.Itdoesnotrequiretheusertobeloggedin,butwewanttochangethedisplayalittleiftheyareloggedin:

router.get('/',async(req,res,next)=>{

try{

letkeylist=awaitnotes.keylist();

letkeyPromises=keylist.map(key=>{returnnotes.read(key)});

letnotelist=awaitPromise.all(keyPromises);

res.render('index',{

title:'Notes',notelist:notelist,

user:req.user?req.user:undefined

});

}catch(e){next(e);}

});

Rememberthatweensuredthatreq.userhastheuserprofiledata,whichwedidindeserializeUser.Wesimplycheckforthisandmakesuretoaddthatdatawhenrenderingtheviewstemplate.

We'llbemakingsimilarchangestomostoftheotherroutedefinitions.Afterthat,we'llgooverthechangestotheviewtemplatesinwhichweusereq.usertoshowthecorrectbuttonsoneachpage.

Login/logoutchangesrequiredinroutes/notes.mjsThechangesrequiredherearemoresignificant,butstillstraightforward:

import{ensureAuthenticated}from'./users';

WeneedtousetheensureAuthenticatedfunctiontoprotectcertainroutesfrombeingusedbyuserswhoarenotloggedin.NoticehowES6modulesletusimportjustthefunction(s)werequire.Sincethatfunctionisintheuserroutermodule,weneedtoimportitfromthere:

router.get('/add',ensureAuthenticated,(req,res,next)=>{

try{

res.render('noteedit',{

title:"AddaNote",

docreate:true,notekey:"",

user:req.user,note:undefined

});

}catch(e){next(e);}

});

ThefirstthingweaddedistocallusersRouter.ensureAuthenticatedintheroutedefinition.Iftheuserisnotloggedin,they'llredirectto/users/login,thankstothatfunction.

Becausewe'veensuredthattheuserisauthenticated,weknowthatreq.userwillalreadyhavetheirprofileinformation.Wecanthensimplypassittotheviewtemplate.

Fortheotherroutes,weneedtomakesimilarchanges:

router.post('/save',ensureAuthenticated,(req,res,next)=>{

..

});

The/saverouterequiresonlythischangetocallensureAuthenticatedtomakesurethattheuserisloggedin:

router.get('/view',(req,res,next)=>{

try{

varnote=awaitnotes.read(req.query.key);

res.render('noteview',{

title:note?note.title:"",

notekey:req.query.key,

user:req.user?req.user:undefined,

note:note

});

}catch(e){next(e);}

});

Forthisroute,wedon'trequiretheusertobeloggedin.Wedoneedtheuser'sprofileinformation,ifany,senttotheviewtemplate:

router.get('/edit',ensureAuthenticated,(req,res,next)=>{

try{

varnote=awaitnotes.read(req.query.key);

res.render('noteedit',{

title:note?("Edit"+note.title):"AddaNote",

docreate:false,

notekey:req.query.key,

user:req.user?req.user:undefined,

note:note

});

}catch(e){next(e);}

});

router.get('/destroy',ensureAuthenticated,(req,res,next)=>{

try{

varnote=awaitnotes.read(req.query.key);

res.render('notedestroy',{

title:note?`Delete${note.title}`:"",

notekey:req.query.key,

user:req.user?req.user:undefined,

note:note

});

}catch(e){next(e);}

});

router.post('/destroy/confirm',ensureAuthenticated,(req,res,next)=>{

..

});

Fortheseroutes,werequiretheusertobeloggedin.Inmostcases,weneedtosendthereq.uservaluetotheviewtemplate.

Viewtemplatechangessupportinglogin/logoutSofar,we'vecreatedabackenduserauthenticationservice,aRESTmodule,toaccessthatservice,aroutermoduletohandleroutesrelatedtologginginandoutofthewebsite,andchangesinapp.mjstousethosemodules.We'realmostready,butwe'vegotanumberofoutstandingchangestomakeinthetemplates.We'repassingthereq.userobjecttoeverytemplatebecauseeachonemustbechangedtoaccommodatewhethertheuserisloggedinornot.

Inpartials/header.hbs,makethefollowingadditions:

...

{{#ifuser}}

<divclass="collapsenavbar-collapse"

id="navbarSupportedContent">

<spanclass="navbar-texttext-darkcol">{{title}}</span>

<aclass="btnbtn-darkcol-auto"href="userslogout">

LogOut<spanclass="badgebadge-light">{{user.username}}

</span></a>

<aclass="nav-itemnav-linkbtnbtn-darkcol-auto"

href='notesadd'>

ADDNote</a>

</div>

{{else}}

<divclass="collapsenavbar-collapse"id="navbarLogIn">

<aclass="btnbtn-primary"href="userslogin">Login</a>

</div>

{{/if}}

...

Whatwe'redoinghereiscontrollingwhichbuttonstodisplayatthetopofthescreendependingonwhethertheuserisloggedinornot.Theearlierchangesensurethattheuservariablewillbeundefinediftheuserisloggedout,otherwiseitwillhavetheuserprofileobject.Therefore,it'ssufficient

tochecktheuservariableasshownheretorenderdifferentuserinterfaceelements.

Alogged-outuserdoesn'tgettheADDNotebutton,andgetsaLoginbutton.Otherwise,theusergetsanADDNotebuttonandaLogOutbutton.TheLoginbuttontakestheusertouserslogin,whiletheLogOutbuttontakesthemtouserslogout.Bothofthosearehandledinroutes/users.js,andperformtheexpectedfunction.

TheLogOutbuttonhasaBootstrapbadgecomponentdisplayingtheusername.Thisaddsalittlevisualsplotch,inwhichwe'llputtheusernamethat'sloggedin.Aswe'llseelater,itwillserveasavisualcuetotheuserastotheiridentity.

Weneedtocreateviews/login.hbs:

<divclass="container-fluid">

<divclass="row">

<divclass="col-12btn-group-vertical"role="group">

<formmethod='POST'action='userslogin'>

<divclass="form-group">

<labelfor="username">Username:</label>

<inputclass="form-control"type='text'id='username'

name='username'value=''placeholder='UserName'/>

</div>

<divclass="form-group">

<labelfor="password">Password:</label>

<inputclass="form-control"type='password'id='password'

name='password'value=''placeholder='Password'/>

</div>

<buttontype="submit"class="btnbtn-default">Submit</button>

</form>

</div>

</div>

</div>

ThisisasimpleformdecoratedwithBootstrapgoodnesstoaskfortheusernameandpassword.Whensubmitted,itcreatesaPOSTrequesttouserslogin,whichinvokesthedesiredhandlertoverifytheloginrequest.

ThehandlerforthatURLwillstartthePassport'sprocesstodecidewhethertheuserisauthenticatedornot.

Inviews/notedestroy.hbs,wewanttodisplayamessageiftheuserisnotloggedin.Normally,theformtocausethenotetobedeletedisdisplayed,butiftheuserisnotloggedin,wewanttoexplainthesituation:

<formmethod='POST'action='notesdestroy/confirm'>

<divclass="container-fluid">

{{#ifuser}}

<inputtype='hidden'name='notekey'value='{{#ifnote}}{{notekey}}

{{/if}}'>

<pclass="form-text">Delete{{note.title}}?</p>

<divclass="btn-group">

<buttontype="submit"value='DELETE'

class="btnbtn-outline-dark">DELETE</button>

<aclass="btnbtn-outline-dark"

href="notesview?key={{#ifnote}}{{notekey}}{{/if}}"

role="button">Cancel</a>

</div>

{{else}}

{{>not-logged-in}}

{{/if}}

</div>

</form>

That'sstraightforward;iftheuserisloggedin,displaytheform,otherwisedisplaythemessageinpartials/not-logged-in.hbs.Wedetermineourapproachbasedontheuservariable.

Wecouldputsomethinglikethisinpartials/not-logged-in.hbs:

<divclass="jumbotron">

<h1>NotLoggedIn</h1>

<p>Youarerequiredtobeloggedinforthisaction,butyouarenot.

Youshouldnotseethismessage.It'sabugifthismessageappears.

</p>

<p><aclass="btnbtn-primary"href="userslogin">Login</a></p>

</div>

Inviews/noteedit.hbs,weneedasimilarchange:

..

<divclass="row"><divclass="col-xs-12">

{{#ifuser}}

..

{{else}}

{{>not-logged-in}}

{{/if}}

</div></div>

..

Thatis,atthebottomweaddasegmentthat,fornon-logged-inusers,pullsinthenot-logged-inpartial.

TheBootstrapjumbotroncomponentmakesaniceandlargetextdisplaythatstandsoutnicely,andwillcatchtheviewer'sattention.However,theusershouldneverseethisbecauseeachofthosetemplatesisusedonlywhenwe'vepreverifiedthattheuserisloggedin.

Amessagesuchasthisisusefulasacheckagainstbugsinyourcode.Supposethatweslippedupandfailedtoproperlyensurethattheseformsweredisplayedonlytologged-inusers.Supposethatwehadotherbugsthatdidn'tchecktheformsubmissiontoensureit'srequestedonlybyalogged-inuser.Fixingthetemplateinthiswayisanotherlayerofpreventionagainstdisplayingformstouserswhoarenotallowedtousethatfunctionality.

RunningtheNotesapplicationwithuserauthenticationNowwe'rereadytoruntheNotesapplicationandtryourhandatlogginginandout.

Weneedtochangethescriptssectionofpackage.jsonasfollows:

"scripts":{

"start":"DEBUG=notes:*SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=sequelizeUSER_SERVICE_URL=http://localhost:3333node--

experimental-modules./bin/www.mjs",

"dl-minty":"mkdir-pminty&&npmrundl-minty-css&&npmrundl-

minty-min-css",

"dl-minty-css":"wgethttps://bootswatch.com/4/minty/bootstrap.css

-Ominty/bootstrap.css",

"dl-minty-min-css":"wget

https://bootswatch.com/4/minty/bootstrap.min.css-Ominty/bootstrap.min.css"

},

Inthepreviouschapters,webuiltupquiteafewcombinationsofmodelsanddatabasesforrunningtheNotesapplication.Thisleavesuswithone,configuredtousetheSequelizemodelforNotes,usingtheSQLite3database,andtousethenewuserauthenticationservicethatwewroteearlier.Wecansimplifythescriptssectionbydeletingthoseotherconfigurations.AlltheotherNotesdatamodelsarestillavailablejustbysettingtheenvironmentvariablesappropriately.

TheUSER_SERVICE_URLneedstomatchtheportnumberthatwedesignatedforthatservice.

Inonewindow,starttheuserauthenticationserviceasfollows:

$cdusers

$npmstart

>user-auth-server@0.0.1startUsersdavid/chap08/users

>DEBUG=users:*PORT=3333SEQUELIZE_CONNECT=sequelize-sqlite.yamlnodeuser-

server

users:serverUser-Auth-Servicelisteningathttp://127.0.0.1:3333

+0ms

Then,inanotherwindow,starttheNotesapplication:

$cdnotes

$DEBUG=notes:*npmstart

>notes@0.0.0startUsersdavid/chap08/notes

>SEQUELIZE_CONNECT=models/sequelize-sqlite.yamlNOTES_MODEL=models/notes-

sequelizeUSERS_MODEL=models/users-rest

USER_SERVICE_URL=http://localhost:3333node./bin/www

notes:serverListeningonport3000+0ms

You'llbegreetedwiththefollowing:

Noticethenewbutton,Login,andthelackofanADDNotebutton.We'renotloggedin,andthereforepartials/header.hbsisriggedtoshowonlytheLoginbutton.

ClickontheLoginbutton,andyouwillseetheloginscreen:

Thisisourloginformfromviews/login.hbs.Youcannowlogin,createanoteorthree,andyoumightendupwiththefollowingonthehomepage:

YounowhavebothLogOutandADDNotebuttons.

You'llnoticethattheLogOutbuttonhastheusername(me)shown.Aftersomethoughtandconsideration,thisseemedthemostcompactwaytoshowwhethertheuserisloggedinornot,andwhichuserisloggedin.

Thismightdrivetheuserexperienceteamnuts,andyouwon'tknowwhetherthisuserinterfacesdesignworksuntilit'stestedwithusers,butit'sgoodenoughforourpurposeatthemoment.

TwitterloginsupportfortheNotesapplicationIfyouwantyourapplicationtohitthebigtime,it'sagreatideatoallowuserstoregisterusingthird-partycredentials.WebsitesallovertheinternetallowyoutologinusingFacebook,Twitter,oraccountsfromotherservices.Doingsoremoveshurdlestoprospectiveuserssigningupforyourservice.Passportmakesitextremelyeasytodothis.

SupportingTwitterrequiresinstallingTwitterStrategy,registeringanewapplicationwithTwitter,andaddingacoupleofroutesintoroutes/user.mjsandasmallchangeinpartials/header.hbs.Integratingotherthird-partyservicesrequiressimilarsteps.

RegisteringanapplicationwithTwitterTwitter,aswitheveryotherthird-partyservice,usesOAuthtohandleauthenticationandrequiresanauthenticationkeytowritesoftwareusingtheirAPI.It'stheirservice,soyouhavetoplaybytheirrules,ofcourse.

ToregisteranewapplicationwithTwitter,gotohttps://apps.twitter.com/.ThenyouclickontheCreateNewAppbutton.Sincewehaven'tdeployedtheNotesapplicationtoaregularserverand,moreimportantly,thereisn'tavaliddomainnamefortheapplication,wehavetogiveTwittertheconfigurationrequiredfortestingonourlocallaptop.

EveryserviceofferingOAuth2authenticationhasanadministrativebackendforregisteringnewapplications.Thecommonpurposeistodescribetheapplicationtotheservicesothattheservicecancorrectlyrecognizetheapplicationwhenrequestsaremadeusingtheauthenticationtokens.Thenormalsituationisthattheapplicationisdeployedtoaregularserver,andisaccessedthroughadomainnamesuchasMyNotes.info.We'vedoneneitherasofthismoment.

Atthetimeofwriting,therearefourpiecesofinformationrequestedbytheTwittersign-upprocess:

Name:Thisistheapplicationname,anditcanbeanythingyoulike.ItwouldbegoodformtousetestinthenameincaseTwitter'sstaffdecidetodosomevalidation.

Description:Descriptivephrase,andagainitcanbeanythingyoulike.Again,itwouldbegoodformto,atthistime,describeitasa

testapplication.

Website:Thiswouldbeyourdesireddomainname.Here,thehelptexthelpfullysuggestsIfyoudon'thaveaURLyet,justputaplaceholderherebutremembertochangeitlater.

CallbackURL:ThisistheURLtoreturntoaftersuccessfulauthentication.Sincewedon'thaveapublicURLtosupply,thisiswherewespecifyavaluereferringtoyourlaptop.It'sbeenfoundthathttp://localhost:3000worksjustfine.macOSusershaveanotheroptionbecauseofthe.localdomainname,whichisautomaticallyassignedtotheirlaptop.Allalong,wecouldhaveusedaURLsimilartothistoaccesstheNotesapplicationathttp://MacBook-Pro-2.local:3000/.

ItwasfoundbyattemptingthisprocedurewithdifferentservicesthatFacebook(andother)servicesarenotlenientabouttestapplicationshostedonlaptops.AtleastTwitteriskeenfordeveloperstoconfigureatestapplicationontheirlaptop.Passport'sotherOAuth-basedstrategieswillworksimilarlyenoughtoTwitter,sotheknowledgewe'regainingwilltransfertothoseotherauthenticationstrategies.

Thelastthingtonoticeistheextremelysensitivenatureoftheauthenticationkeys.It'sbadformtochecktheseintoasourcecoderepositoryorotherwiseputtheminaplacewhereanybodycanaccessthekey.We'lltacklethisissueinChapter12,Security.

Twitterdoeschangethesignuppagefromtimetotime,butitshouldlooksomethinglikethefollowing:

ImplementingTwitterStrategyAswithmanywebapplications,wehavedecidedtoallowouruserstologinusingTwittercredentials.TheOAuth2protocoliswidelyusedforthispurposeandisthebasisforauthenticatingononewebsiteusingcredentialsmaintainedbyanotherwebsite.

Theapplicationregistrationprocessyoujustfollowedatapps.twitter.comgeneratedforyouapairofAPIkeys,aconsumerkey,and,consumersecret.ThesekeysarepartoftheOAuthprotocol,andwillbesuppliedbyanyOAuthserviceyouregisterwith,andthekeysshouldbetreatedwiththeutmostcare.ThinkofthemastheusernameandpasswordyourserviceusestoaccesstheOAuth-basedservice(Twitteretal).Themorepeoplewhocanseethesekeys,themorelikelyamiscreantcanseethemandthencausetrouble.AnybodywiththosesecretscanwriteaccesstheserviceAPIasiftheyareyou.

DozensofStrategypackagesforvariousthird-partyservicesareavailablewithinthePassportecosystem.Let'sinstallthepackagerequiredtouseTwitterStrategy:

$npminstallpassport-twitter@1.x--save

Inroutes/users.mjs,let'sstartmakingsomechanges:

importpassportTwitterfrom'passport-twitter';

constTwitterStrategy=passportTwitter.Strategy;

Tobringinthepackagewejustinstalled,addthefollowing:

consttwittercallback=process.env.TWITTER_CALLBACK_HOST

?process.env.TWITTER_CALLBACK_HOST

:"http://localhost:3000";

passport.use(newTwitterStrategy({

consumerKey:process.env.TWITTER_CONSUMER_KEY,

consumerSecret:process.env.TWITTER_CONSUMER_SECRET,

callbackURL:`${twittercallback}/usersauthtwitter/callback`

},

asyncfunction(token,tokenSecret,profile,done){

try{

done(null,awaitusersModel.findOrCreate({

id:profile.username,username:profile.username,password:"",

provider:profile.provider,familyName:profile.displayName,

givenName:"",middleName:"",

photos:profile.photos,emails:profile.emails

}));

}catch(err){done(err);}

}));

ThisregistersTwitterStrategywithpassport,arrangingtocalltheuserauthenticationserviceasusersregisterwiththeNotesapplication.ThiscallbackfunctioniscalledwhenuserssuccessfullyauthenticateusingTwitter.

WedefinedtheusersModel.findOrCreatefunctionspecificallytohandleuserregistrationfromthird-partyservicessuchasTwitter.Itstaskistolookfortheuserdescribedintheprofileobjectand,ifthatuserdoesnotexist,toautocreatethatuseraccountinNotes.

TheconsumerKeyandconsumerSecretvaluesaresuppliedbyTwitter,afteryou'veregisteredyourapplication.ThesesecretsareusedintheOAuthprotocolasproofofidentitytoTwitter.

ThecallbackURLsettingintheTwitterStrategyconfigurationisaholdoverfromTwitter'sOAuth1-basedAPIimplementation.InOAuth1,thecallbackURLwaspassedaspartoftheOAuthrequest.SinceTwitterStrategyusesTwitter'sOAuth1service,wehavetosupplytheURLhere.We'llseeinamomentwherethatURLisimplementedinNotes.

ThecallbackURL,consumerKey,andconsumerSecretareallinjectedusingenvironmentvariables.Itistempting,becauseoftheconvenience,tojustputthosekeysinthesourcecode.But,howwidelydistributedisyour

sourcecode?IntheSlackAPIdocumentation(https://api.slack.com/docs/oauth-safety),we'rewarnedDonotdistributeclientsecretsinemail,distributednativeapplications,client-sideJavaScript,orpubliccoderepositories.

InChapter10,DeployingNode.jsApplications,we'llputthesekeysintoaDockerfile.That'snotentirelysecurebecausetheDockerfilewillalsobecommittedtoasourcerepositorysomewhere.

ItwasfoundwhiledebuggingthattheprofileobjectsuppliedbytheTwitterStrategydidnotmatchthedocumentationonthepassportwebsite.Therefore,wehavemappedtheobjectactuallysuppliedbypassportintosomethingthatNotescanuse:

router.get('authtwitter',passport.authenticate('twitter'));

TostarttheuserlogginginwithTwitter,we'llsendthemtothisURL.RememberthatthisURLisreally/usersauthtwitter,and,inthetemplates,we'llhavetousethatURL.Whenthisiscalled,thepassportmiddlewarestartstheuserauthenticationandregistrationprocessusingTwitterStrategy.

Oncetheuser'sbrowservisitsthisURL,theOAuthdancebegins.It'scalledadancebecausetheOAuthprotocolinvolvescarefullydesignedredirectsbetweenseveralwebsites.PassportsendsthebrowserovertothecorrectURLatTwitter,whereTwitteraskstheuserwhethertheyagreetoauthenticateusingTwitter,andthenTwitterredirectstheuserbacktoyourcallbackURL.Alongtheway,specifictokensarepassedbackandforthinaverycarefullydesigneddancebetweenwebsites.

OncetheOAuthdanceconcludes,thebrowserlandshere:

router.get('authtwitter/callback',

passport.authenticate('twitter',{successRedirect:'/',

failureRedirect:'userslogin'}));

ThisroutehandlesthecallbackURL,anditcorrespondstothecallbackURL

settingconfiguredearlier.Dependingonwhetheritindicatesasuccessfulregistrationornot,passportwillredirectthebrowsertoeitherthehomepageorbacktotheusersloginpage.

Becauserouterismountedon/user,thisURLisactually/userauthtwitter/callback.Therefore,thefullURLtouseinconfiguringtheTwitterStrategy,andtosupplytoTwitter,ishttp://localhost:3000/userauthtwitter/callback

IntheprocessofhandlingthecallbackURL,Passportwillinvokethecallbackfunctionshownearlier.BecauseourcallbackusestheusersModel.findOrCreatefunction,theuserwillbeautomaticallyregisteredifnecessary.

We'realmostready,butweneedtomakeacoupleofsmallchangeselsewhereinNotes.

Inpartials/header.hbs,makethefollowingchangestothecode:

...

{{else}}

<divclass="collapsenavbar-collapse"id="navbarLogIn">

<spanclass="navbar-texttext-darkcol"></span>

<aclass="nav-itemnav-linkbtnbtn-darkcol-auto"href="userslogin">

Login</a>

<aclass="nav-itemnav-linkbtnbtn-darkcol-auto"

href="/usersauthtwitter">

<imgwidth="15px"

src="assetsvendor/twitter/Twitter_Social_Icon_Rounded_Square_Color.png"/>

LoginwithTwitter</a>

</div>

{{/if}}

Thisaddsanewbuttonthat,whenclicked,takestheuserto/usersauthtwitter,which,ofcourse,kicksofftheTwitterauthenticationprocess.

TheimagebeingusedisfromtheofficialTwitterbrandassetspageathttps://about.twitter.com/company/brand-assets.Twitterrecommendsusingthese

brandingassetsforaconsistentlookacrossallservicesusingTwitter.Downloadthewholesetandthenpickoneyoulike.FortheURLshownhere,placethechosenimageinadirectorynamedpublicassetsvendor/twitter.Noticethatweforcethesizetobesmallenoughforthenavigationbar.

Withthesechanges,we'rereadytotrylogginginwithTwitter.

StarttheNotesapplicationserverasdonepreviously:

$npmstart

>notes@0.0.0startUsersDavid/chap08/notes

>DEBUG=notes:*SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=sequelizeUSER_SERVICE_URL=http://localhost:3333node--

experimental-modules./bin/www.mjs

(node:42095)ExperimentalWarning:TheESMmoduleloaderisexperimental.

notes:server-debugListeningonport3000+0ms

Thenuseabrowsertovisithttp://localhost:3000:

Noticethenewbutton.Itlooksaboutright,thankstohavingusedtheofficialTwitterbrandingimage.Thebuttonisalittlelarge,somaybeyouwanttoconsultadesigner.Obviously,adifferentdesignisrequiredifyou'regoingtosupportdozensofauthenticationservices.

Clickingonthisbuttontakesthebrowserto/usersauthtwitter,whichstartsPassportrunningtheOAuth2protocoltransactionsnecessarytoauthenticate.Andthen,onceyou'reloggedinwithTwitter,you'llseesomethinglikethefollowingscreenshot:

We'renowloggedin,andnoticethatourNotesusernameisthesameasourTwitterusername.Youcanbrowsearoundtheapplicationandcreate,edit,ordeletenotes.Infact,youcandothistoanynoteyoulike,evenonescreatedbyothers.That'sbecausewedidnotcreateanysortofaccesscontrolorpermissionssystem,andthereforeeveryuserhascompleteaccesstoeverynote.That'safeaturetoputonthebacklog.

Byusingmultiplebrowsersorcomputers,youcansimultaneouslyloginasdifferentusers,oneuserperbrowser.

YoucanrunmultipleinstancesoftheNotesapplicationbydoingwhatwedidearlier:

"scripts":{

"start":"SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=models/notes-sequelizeUSERS_MODEL=models/users-rest

USER_SERVICE_URL=http://localhost:3333node./bin/www",

"start-server1":"SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=models/notes-sequelizeUSERS_MODEL=models/users-rest

USER_SERVICE_URL=http://localhost:3333PORT=3000node./bin/www",

"start-server2":"SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=models/notes-sequelizeUSERS_MODEL=models/users-rest

USER_SERVICE_URL=http://localhost:3333PORT=3002node./bin/www",

"dl-minty":"mkdir-pminty&&npmrundl-minty-css&&npmrundl-minty-

min-css",

"dl-minty-css":"wgethttps://bootswatch.com/4/minty/bootstrap.css-O

minty/bootstrap.css",

"dl-minty-min-css":"wget

https://bootswatch.com/4/minty/bootstrap.min.css-Ominty/bootstrap.min.css"

},

Then,inonecommandwindow,runthefollowingcommand:

$npmrunstart-server1

>notes@0.0.0start-server1UsersDavid/chap08/notes

>DEBUG=notes:*SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=sequelizeUSER_SERVICE_URL=http://localhost:3333PORT=3000node-

-experimental-modules./bin/www.mjs

(node:43591)ExperimentalWarning:TheESMmoduleloaderisexperimental.

notes:server-debugListeningonport3000+0ms

Inanothercommandwindow,runthefollowingcommand:

$npmrunstart-server2

>notes@0.0.0start-server2UsersDavid/chap08/notes

>DEBUG=notes:*SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=sequelizeUSER_SERVICE_URL=http://localhost:3333PORT=3002node-

-experimental-modules./bin/www.mjs

(node:43755)ExperimentalWarning:TheESMmoduleloaderisexperimental.

notes:server-debugListeningonport3002+0ms

Aspreviously,thisstartstwoinstancesoftheNotesserver,eachwithadifferentvalueinthePORTenvironmentvariable.Inthiscase,eachinstancewillusethesameuserauthenticationservice.Asshownhere,you'llbeabletovisitthetwoinstancesathttp://localhost:3000andhttp://localhost:3002.And,aspreviously,you'llbeabletostartandstoptheserversasyouwish,seethesamenotesineach,andseethatthenotesareretainedafterrestartingtheserver.

Anotherthingtotryistofiddlewiththesessionstore.Oursessiondatais

beingstoredinthesessionsdirectory.Thesearejustfilesinthefilesystem,andwecantakealook:

$ls-lsessions/

total32

-rw-r--r--1davidwheel139Jan2519:28-QOS7eX8ZBAfmK9CCV8Xj8v-

3DVEtaLK.json

-rw-r--r--1davidwheel139Jan2521:30

T7VT4xt3_e9BiU49OMC6RjbJi6xB7VqG.json

-rw-r--r--1davidwheel223Jan2519:27ermh-

7ijiqY7XXMnA6zPzJvsvsWUghWm.json

-rw-r--r--1davidwheel139Jan2521:23

uKzkXKuJ8uMN_ROEfaRSmvPU7NmBc3md.json

$catsessions/T7VT4xt3_e9BiU49OMC6RjbJi6xB7VqG.json

{"cookie":

{"originalMaxAge":null,"expires":null,"httpOnly":true,"path":"/"},"__lastAcce

ss":1516944652270,"passport":{"user":"7genblogger"}}

ThisisafterlogginginusingaTwitteraccount;youcanseethattheTwitteraccountnameisstoredhereinthesessiondata.

Whatifyouwanttoclearasession?It'sjustafileinthefilesystem.Deletingthesessionfileerasesthesession,andtheuser'sbrowserwillbeforcefullyloggedout.

Thesessionwilltimeoutiftheuserleavestheirbrowseridleforlongenough.Oneofthesession-file-storeoptions,ttl,controlsthetimeoutperiod,whichdefaultsto3,600seconds(anhour).Withatimed-outsession,theapplicationrevertstoalogged-outstate.

SecurelykeepingsecretsandpasswordsWe'vecautionedseveraltimesabouttheimportanceofsafelyhandlinguseridentificationinformation.Theintentiontosafelyhandlethatdataisonething,butitisimportanttofollowthroughandactuallydoso.Whilewe'reusingafewgoodpracticessofar,asitstands,theNotesapplicationwouldnotwithstandanykindofsecurityaudit:

Userpasswordsarekeptincleartextinthedatabase

TheauthenticationtokensforTwitteretal,areinthesourcecodeincleartext

TheauthenticationserviceAPIkeyisnotacryptographicallysecureanything,it'sjustacleartextUUID

Ifyoudon'trecognizethephrasecleartext,itsimplymeansunencrypted.Anyonecouldreadthetextofuserpasswordsortheauthenticationtokens.It'sbesttokeepbothencryptedtoavoidinformationleakage.

Keepthisissueinthebackofyourmindbecausewe'llrevisittheseandothersecurityissuesinChapter12,Security.

TheNotesapplicationstackDidyounoticeearlierwhenwesaidruntheNotesapplicationstack?It'stimetoexplaintothemarketingteamwhat'smeantbythatphrase.They'llperhapsneedtoputanarchitecturediagramonmarketingbrochuresandthelike.It'salsousefulfordeveloperslikeustotakeastepbackanddrawapictureofwhatwe'vecreated,orareplanningtocreate.

Here'sthesortofdiagramthatanengineermightdrawtoshowthemarketingteamthesystemdesign.Themarketingteamwill,ofcourse,hireagraphicsartisttocleanitup:

TheboxlabeledNotesApplicationisthepublic-facingcodeimplementedbythetemplatesandtheroutermodules.Ascurrentlyconfigured,it'svisiblefromourlaptoponport3000.Itcanuseoneofseveraldatastorageservices.ItcommunicateswiththebackendUserAuthenticationServiceoverport3333(ascurrentlyconfigured).InChapter10,DeployingNode.jsApplications,we'llbeexpandingthispictureabitaswelearnhowtodeployonarealserver.

SummaryYou'vecoveredalotofgroundinthischapter,lookingatnotonlyuserauthenticationinExpressapplications,butalsomicroservicedevelopment.

Specifically,youcoveredsessionmanagementinExpress,usingPassportforuserauthentication,includingTwitter/OAuth,usingroutermiddlewaretolimitaccess,creatingaRESTservicewithRestify,andwhentocreateamicroservice.

Inthenextchapter,we'lltaketheNotesapplicationtoanewlevel-semi-real-timecommunicationbetweenapplicationusers.Todothis,we'llwritesomebrowser-sideJavaScriptandexplorehowtheSocket.iopackagecanletussendmessagesbetweenusers.

DynamicClient/ServerInteractionwithSocket.IOTheoriginaldesignmodelofthewebissimilartothewaythatmainframesworkedinthe1970s.Bothold-schooldumbterminals,suchastheIBM3270,andwebbrowsers,followarequest-responseparadigm.Theusersendsarequestandthefar-offcomputersendsaresponsescreen.Whilewebbrowserscanshowmorecomplexinformationthanold-schooldumbterminals,theinteractionpatterninbothcasesisabackandforthofuserrequests,eachresultinginascreenofdatasentbytheserverscreenafterscreenor,inthecaseofwebbrowsers,pageafterpage.

Incaseyou'rewonderingwhatthishistorylessonisabout,thatrequest-responseparadigmisevidentintheNode.jsHTTPServerAPI,asshowninthefollowingcode:

http.createServer(function(request,response){

...handlerequest

}).listen();

Theparadigmcouldn'tbemoreexplicitthanthis.Therequestandtheresponsearerightthere.

Thefirstwebbrowserswereanadvancementovertext-baseduserinterfaces,withHTMLmixingimages,andtextwithvaryingcolors,fonts,andsizes.AsCSScamealong,HTMLimproved,iframesallowedembeddedmediaofallkinds,andJavaScriptimproved,sowehaveaquitedifferentparadigm.Thewebbrowserisstillsendingrequestsandreceivingapageofdata,butthatdatacanbequitecomplexand,moreimportantly,JavaScriptaddsinteractivity.

Onenewtechniqueiskeepinganopenconnectiontotheserverfor

continualdataexchangebetweenserverandclient.Thischangeinthewebapplicationmodeliscalled,bysome,thereal-timeweb.Insomecases,websiteskeepanopenconnectiontothewebbrowser,withreal-timeupdatestowebpagesbeingonegoal.

Someobservethattraditionalwebapplicationscanuntruthfullydisplaytheirdata;thatis,iftwopeoplearelookingatapage,andonepersoneditsthatpage,thatperson'sbrowserwillupdatewiththecorrectcopyofthepage,whiletheotherbrowserisnotupdated.Thetwobrowsersshowdifferentversionsofthepage,oneofwhichisuntruthful.Thesecondbrowsercanevenshowapagethatnolongerexists,iftheuseratthefirstbrowserdeletesthatpage.Somethinkitwouldbebetteriftheotherperson'sbrowserisrefreshedtoshowthenewcontentassoonasthepageisedited.

Thisisonepossibleroleofthereal-timeweb;pagesthatupdatethemselvesaspagecontentchanges.Allkindsofsystemssupportreal-timeinteractivitybetweenfolksonthesamewebsite.Whetherit'sseeingFacebookcommentspopupasthey'rewritten,orcollaborativelyediteddocuments,there'sanewinteractivityparadigmontheweb.

We'reabouttoimplementthisbehaviorintheNotesapplication.

OneoftheoriginalpurposesforinventingNode.jswastosupportthereal-timeweb.TheCometapplicationarchitecture(CometisrelatedtoAJAX,andbothhappentobenamesofhouseholdcleaningproducts)involvesholdingtheHTTPconnectionopenforalongtime,withdataflowingbackandforthbetweenbrowserandserveroverthatchannel.ThetermCometwasintroducedbyAlexRussellinhisblogin2006(http://infrequently.org/2006/03/comet-low-latency-data-for-the-browser/)asageneraltermforthearchitecturalpatterntoimplementthisreal-time,two-waydataexchangebetweenclientandserver.ThatblogpostcalledforthedevelopmentofaprogrammingplatformverysimilartoNode.js.

Tosimplifythetask,we'llleanontheSocket.IOlibrary(http://socket.io/).Thislibrarysimplifiestwo-waycommunicationbetweenthebrowserandserver,andcansupportavarietyofprotocolswithfallbacktoold-school

webbrowsers.

We'llbecoveringthefollowingtopics:

Real-timecommunicationsinmodernwebbrowsers

TheSocket.IOlibrary

IntegratingSocket.IOwithanExpressapplicationtosupportreal-timecommunication

Userexperienceforreal-timecommunication

IntroducingSocket.IOTheaimofSocket.IOistomakereal-timeappspossibleineverybrowserandmobiledevice.Itsupportsseveraltransportprotocols,choosingthebestoneforthespecificbrowser.

IfyouweretoimplementyourapplicationwithWebSockets,itwouldbelimitedtothemodernbrowserssupportingthatprotocol.BecauseSocket.IOfallsbackonsomanyalternateprotocols(WebSockets,Flash,XHR,andJSONP),itsupportsawiderangeofwebbrowsers,includingsomeoldcruftybrowsers.

Astheapplicationauthor,youdon'thavetoworryaboutthespecificprotocolSocket.IOusesinagivenbrowser.Instead,youcanimplementthebusinesslogicandthelibrarytakescareofthedetailsforyou.

Socket.IOrequiresthataclientlibrarymakeitswayintothebrowser.Thatlibraryisprovided,andiseasytoinstantiate.You'llbewritingcodeonboththebrowsersideandserversideusingsimilarSocket.IOAPIsateachend.

ThemodelthatSocket.IOprovidesissimilartotheEventEmitterobject.Theprogrammerusesthe.onmethodtolistenforeventsandthe.emitmethodtosendthem.TheemittedeventsaresentbetweenthebrowserandtheserverwiththeSocket.IOlibrarytakingcareofsendingthembackandforth.

InformationaboutSocket.IOisavailableathttps://socket.io/.

InitializingSocket.IOwithExpressSocket.IOworksbywrappingitselfaroundanHTTPServerobject.ThinkbacktoChapter4,HTTPServersandClients,wherewewroteamodulethathookedintoHTTPServermethodssothatwecouldspyonHTTPtransactions.TheHTTPSnifferattachesalistenertoeveryHTTPeventtoprintouttheevents.Butwhatifyouusedthatideatodorealwork?Socket.IOusesasimilarconcept,listeningtoHTTPrequestsandrespondingtospecificonesbyusingtheSocket.IOprotocoltocommunicatewithclientcodeinthebrowser.

Togetstarted,let'sfirstmakeaduplicateofthecodefromthepreviouschapter.Ifyoucreatedadirectorynamedchap08forthatcode,createanewdirectorynamedchap09andcopythesourcetreethere.

Wewon'tmakechangestotheuserauthenticationmicroservice,butwewilluseitforuserauthentication,ofcourse.

IntheNotessourcedirectory,installthesenewmodules:

$npminstallsocket.io@2.xpassport.socketio@3.7.x--save

Wewillincorporateuserauthenticationwiththepassportmodule,usedinChapter8,MultiuserAuthenticationtheMicroserviceWay,intosomeofthereal-timeinteractionswe'llimplement.

ToinitializeSocket.IO,wemustdosomemajorsurgeryonhowtheNotesapplicationisstarted.Sofar,weusedthebin/www.mjsscriptalongwithapp.mjs,witheachscripthostingdifferentstepsoflaunchingNotes.Socket.IOinitializationrequiresthatthesestepsoccurinadifferentordertowhatwe'vebeendoing.Therefore,wemustmergethesetwoscriptsinto

one.Whatwe'lldoiscopythecontentofthebin/www.mjsscriptintoappropriatesectionsofapp.mjs,andfromthere,we'lluseapp.mjstolaunchNotes.

Atthebeginningofapp.mjs,addthistotheimportstatements:

importhttpfrom'http';

importpassportSocketIofrom'passport.socketio';

importsessionfrom'express-session';

importsessionFileStorefrom'session-file-store';

constFileStore=sessionFileStore(session);

exportconstsessionCookieName='notescookie.sid';

constsessionSecret='keyboardmouse';

constsessionStore=newFileStore({path:"sessions"});

Thepassport.socketiomoduleintegratesSocket.IOwithPassportJS-baseduserauthentication.We'llconfigurethissupportshortly.TheconfigurationforsessionmanagementisnowsharedbetweenSocket.IO,Express,andPassport.Theselinescentralizethatconfigurationtooneplaceinapp.mjs,sowecanchangeitoncetoaffecteveryplaceit'sneeded.

UsethistoinitializetheHTTPServerobject:

constapp=express();

exportdefaultapp;

constserver=http.createServer(app);

importsocketiofrom'socket.io';

constio=socketio(server);

io.use(passportSocketIo.authorize({

cookieParser:cookieParser,

key:sessionCookieName,

secret:sessionSecret,

store:sessionStore

}));

Thismovestheexportdefaultapplinefromthebottomofthefiletothislocation.Doesn'tthislocationmakemoresense?

TheioobjectisourentrypointintotheSocket.IOAPI.WeneedtopassthisobjecttoanycodethatneedstousethatAPI.Itwon'tbeenoughtosimplyrequirethesocket.iomoduleinothermodulesbecausetheioobjectiswhatwrapstheserverobject.Instead,we'llbepassingtheioobjectintowhatevermodulesaretouseit.

Theio.usefunctioninstallsinSocket.IOfunctionssimilartoExpressmiddleware.Inthiscase,weintegratePassportauthenticationintoSocket.IO:

varport=normalizePort(process.env.PORT||'3000');

app.set('port',port);

server.listen(port);

server.on('error',onError);

server.on('listening',onListening);

Thiscodeiscopiedfrombin/www.mjs,andsetsuptheporttolistento.Itreliesonthreefunctionsthatwillalsobecopiedintoapp.mjsfrombin/www.mjs:

app.use(session({

store:sessionStore,

secret:sessionSecret,

resave:true,

saveUninitialized:true,

name:sessionCookieName

}));

initPassport(app);

ThischangestheconfigurationofExpresssessionsupporttomatchtheconfigurationvariableswesetupearlier.It'sthesamevariablesusedwhensettinguptheSocket.IOsessionintegration,meaningthey'rebothonthesamepage.

UsethistoinitializeSocket.IOcodeintheroutermodules:

app.use('/',index);

app.use('/users',users);

app.use('/notes',notes);

indexSocketio(io);

//notesSocketio(io);

Thisiswherewepasstheioobjectintomodulesthatmustuseit.ThisissothattheNotesapplicationcansendmessagestothewebbrowsersaboutchangesinNotes.Whatthatmeanswillbeclearerinasecond.What'srequiredisanalogoustotheExpressrouterfunctions,andthereforethecodetosend/receivemessagesfromSocket.IOclientswillalsobelocatedintheroutermodules.

Wehaven'twritteneitherofthesefunctionsyet(havepatience).Tosupportthis,weneedtomakeachangeinanimportstatementatthetop:

import{socketioasindexSocketio,routerasindex}from'./routes/index';

Eachroutermodulewillexportafunctionnamedsocketio,whichwe'llhavetorenameasshownhere.Thisfunctioniswhatwillreceivetheioobject,andhandleanySocket.IO-basedcommunications.Wehaven'twrittenthesefunctionsyet.

Then,attheendofapp.mjs,we'llcopyintheremainingcodefrombin/www.mjssotheHTTPServerstartslisteningonourselectedport:

functionnormalizePort(val){

varport=parseInt(val,10);

if(isNaN(port)){//namedpipe

returnval;

}

if(port>=0){//portnumber

returnport;

}

returnfalse;

}

/**

*EventlistenerforHTTPserver"error"event.

*/

functiononError(error){

if(error.syscall!=='listen'){throwerror;}

varbind=typeofport==='string'?'Pipe'+port:'Port'+

port;

//handlespecificlistenerrorswithfriendlymessages

switch(error.code){

case'EACCES':

console.error(bind+'requireselevatedprivileges');

process.exit(1);

break;

case'EADDRINUSE':

console.error(bind+'isalreadyinuse');

process.exit(1);

break;

default:

throwerror;

}

}

/**

*EventlistenerforHTTPserver"listening"event.

*/

functiononListening(){

varaddr=server.address();

varbind=typeofaddr==='string'?'pipe'+addr:'port'+addr.port;

debug('Listeningon'+bind);

}

Then,inpackage.json,wemuststartapp.mjsratherthanbin/www.mjs:

"scripts":{

"start":"DEBUG=notes:*SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=sequelizeUSER_SERVICE_URL=http://localhost:3333node--

experimental-modules./app",

"start-server1":"DEBUG=notes:*SEQUELIZE_CONNECT=models/sequelize-

sqlite.yamlNOTES_MODEL=sequelizeUSER_SERVICE_URL=http://localhost:3333

PORT=3000node--experimental-modules./app",

"start-server2":"DEBUG=notes:*SEQUELIZE_CONNECT=models/sequelize-

sqlite.yamlNOTES_MODEL=sequelizeUSER_SERVICE_URL=http://localhost:3333

PORT=3002node--experimental-modules./app",

...

},

Atthispoint,youcandeletebin/www.mjsifyoulike.Youcanalsotry

startingtheserver,butit'llfailbecausetheindexSocketiofunctiondoesnotexistyet.

Real-timeupdatesontheNoteshomepageThegoalwe'reworkingtowardsisfortheNoteshomepagetoautomaticallyupdatethelistofnotes,asnotesareeditedordeleted.Whatwe'vedonesofaristorestructuretheapplicationstartupsothatSocket.IOisinitializedintheNotesapplication.There'snochangeofbehavioryet,exceptthatitwillcrashduetoamissingfunction.

TheapproachisfortheNotesmodelclassestosendmessageswheneveranoteiscreated,updated,ordeleted.Intherouterclasses,we'lllistentothosemessages,thensendalistofnotetitlestoallbrowsersattachedtotheNotesapplication.

WheretheNotesmodelsofarhasbeenapassiverepositoryofdocuments,itnowneedstoemiteventstoanyinterestedparties.Thisisthelistenerpatternand,intheory,therewillbecodethatisinterestedinknowingwhennotesarecreated,edited,ordestroyed.Atthemoment,we'llusethatknowledgetoupdatetheNoteshomepage,buttherearemanypotentialotherusesofthatknowledge.

TheNotesmodelasanEventEmitterclassTheEventEmitteristheclassthatimplementslistenersupport.Let'screateanewmodule,models/notes-events.mjs,containingthefollowing:

importEventEmitterfrom'events';

classNotesEmitterextendsEventEmitter{

noteCreated(note){this.emit('notecreated',note);}

noteUpdate(note){this.emit('noteupdate',note);}

noteDestroy(data){this.emit('notedestroy',data);}

}

exportdefaultnewNotesEmitter();

ThismodulemaintainsthelistenerstoNotes-relatedeventsforus.We'vecreatedasubclassofEventEmitterbecauseitalreadyknowshowtomanagethelisteners.Aninstanceofthatobjectisexportedasthedefaultexport.

Let'snowupdatemodels/notes.mjstousenotes-eventstoemitevents.Becausewehaveasinglemodule,notes.mjs,todispatchcallstotheindividualNotesmodels,thismoduleprovidesakeypointatwhichwecanintercepttheoperationsandsendevents.Otherwise,we'dhavetointegratetheevent-sendingcodeintoeveryNotesmodel.

import_eventsfrom'./notes-events';

exportconstevents=_events;

Weneedtoimportthismoduleforusehere,butalsoexportitsothatothermodulescanalsoemitNotesevents.Bydoingthis,anothermodulethatimportsNotescancallnotes.events.functiontousethenotes-eventsmodule.

Thistechniqueiscalledre-exporting.Sometimes,youneedtoexporta

functionfrommoduleAthatisactuallydefinedinmoduleB.ModuleAthereforeimportsthefunctionfrommoduleB,addingittoitsexports.

Thenwedoalittlerewritingofthesefunctions:

exportasyncfunctioncreate(key,title,body){

constnote=awaitmodel().create(key,title,body);

_events.noteCreated(note);

returnnote;

}

exportasyncfunctionupdate(key,title,body){

constnote=awaitmodel().update(key,title,body);

_events.noteUpdate(note);

returnnote;

}

exportasyncfunctiondestroy(key){

awaitmodel().destroy(key);

_events.noteDestroy({key});

returnkey;

}

ThecontractfortheNotesmodelfunctionsisthattheyreturnaPromise,andthereforeourcallerwillbeusingawaittoresolvethePromise.Therearethreesteps:

1. Callthecorrespondingfunctioninthecurrentmodelclass,andawaititsresult

2. Sendthecorrespondingmessagetoourlisteners3. Returnthevalue,andbecausethisisanasyncfunction,thevalue

willbereceivedasaPromisethatfulfillsthecontractforthesefunctions

Real-timechangesintheNoteshomepageTheNotesmodelnowsendseventsasNotesarecreated,updated,ordestroyed.Forthistobeuseful,theeventsmustbedisplayedtoourusers.Makingtheeventsvisibletoourusersmeansthecontrollerandviewportionsoftheapplicationmustconsumethoseevents.

Let'sstartmakingchangestoroutes/index.mjs:

router.get('/',async(req,res,next)=>{

try{

letnotelist=awaitgetKeyTitlesList();

res.render('index',{

title:'Notes',notelist:notelist,

user:req.user?req.user:undefined

});

}catch(e){next(e);}

});

Weneedtoreusepartoftheoriginalroutingfunction,touseitinanotherfunction.Therefore,we'vepulledcodethatusedtobeinthisblockintoanewfunction,getKeyTitlesList:

asyncfunctiongetKeyTitlesList(){

constkeylist=awaitnotes.keylist();

varkeyPromises=keylist.map(key=>{

returnnotes.read(key).then(note=>{

return{key:note.key,title:note.title};

});

});

returnPromise.all(keyPromises);

};

Thisportionoftheoriginalroutingfunctionisnowitsownfunction.It

generatesanarrayofitemscontainingthekeyandtitleforallexistingNotes,usingPromise.alltomanagetheprocessofreadingeverything.

exportfunctionsocketio(io){

varemitNoteTitles=async()=>{

constnotelist=awaitgetKeyTitlesList()

io.of('/home').emit('notetitles',{notelist});

};

notes.events.on('notecreated',emitNoteTitles);

notes.events.on('noteupdate',emitNoteTitles);

notes.events.on('notedestroy',emitNoteTitles);

};

Hereisthesocketiofunctionwediscussedwhilemodifyingapp.mjs.Wereceivetheioobject,thenuseittoemitanotestitleseventtoallconnectedbrowsers.

Theio.of('/namespace')methodrestrictswhateverfollowstothegivennamespace.Inthiscase,we'reemittinganotestitlemessagetothe/homenamespace.

Theio.ofmethoddefineswhatSocket.IOcallsanamespace.NamespaceslimitthescopeofmessagessentthroughSocket.IO.Thedefaultnamespaceis/,andnamespaceslooklikepathnames,inthatthey'reaseriesofslash-separatednames.Aneventemittedintoanamespaceisdeliveredtoanysocketlisteningtothatnamespace.

Thecode,inthiscase,isfairlystraightforward.Itlistenstotheeventswejustimplemented,notecreated,noteupdate,andnotedestroy.Foreachoftheseevents,itemitsanevent,notetitles,containingthelistofnotekeysandtitles.

That'sit!

AsNotesarecreated,updated,anddestroyed,weensurethatthehomepagewillberefreshedtomatch.Thehomepagetemplate,views/index.hbs,willrequirecodetoreceivethateventandrewritethepagetomatch.

ChangingthehomepageandlayouttemplatesSocket.IOrunsonbothclientandserver,withthetwocommunicatingbackandforthovertheHTTPconnection.ThisrequiresloadingtheclientJavaScriptlibraryintotheclientbrowser.EachpageoftheNotesapplicationinwhichweseektoimplementSocket.IOservicesmustloadtheclientlibraryandhavecustomclientcodeforourapplication.

EachpageinNoteswillrequireadifferentSocket.IOclientimplementation,sinceeachpagehasdifferentrequirements.ThisaffectshowweloadJavaScriptcodeinNotes.

Initially,wesimplyputJavaScriptcodeatthebottomoflayout.hbs,becauseeverypagerequiredthesamesetofJavaScriptmodules.Butnowwe'veidentifiedtheneedforadifferentsetofJavaScriptoneachpage.Furthermore,someoftheJavaScriptneedstobeloadedfollowingtheJavaScriptcurrentlyloadedatthebottomoflayout.hbs.Specifically,jQueryisloadedcurrentlyinlayout.hbs,butwewanttousejQueryintheSocket.IOclientstoperformDOMmanipulationsoneachpage.Therefore,sometemplaterefactoringisrequired.

Createafile,partials/footerjs.hbs,containing:

<!--jQueryfirst,thenPopper.js,thenBootstrapJS-->

<scriptsrc="assetsvendor/jquery/jquery.min.js"></script>

<scriptsrc="assetsvendor/popper.js/umd/popper.min.js"></script>

<scriptsrc="assetsvendor/bootstrap/js/bootstrap.min.js"></script>

<scriptsrc="assetsvendor/feather-icons/feather.js"></script>

<script>

feather.replace()

</script>

Thishadbeenatthebottomofviews/layout.hbs.Wenowneedtomodifythatfileasfollows:

<body>

{{>header}}

{{{body}}}

</body>

Then,atthebottomofeverytemplate(error.hbs,index.hbs,login.hbs,notedestroy.hbs,noteedit.hbs,andnoteview.hbs),addthisline:

{{>footerjs}}

Sofar,thishasn'tchangedwhatwillbeloadedinthepages,becausefooterjscontainsexactlywhatwasalreadyatthebottomoflayout.hbs.ButitgivesusthefreedomtoloadSocket.IOclientcodeafterthescriptsinfooterjsareloaded.

Inviews/index.hbsaddthisatthebottom,afterthefooterjspartial:

{{>footerjs}}

<scriptsrc="socket.iosocket.io.js"></script>

<script>

$(document).ready(function(){

varsocket=io('/home');

socket.on('notetitles',function(data){

varnotelist=data.notelist;

$('#notetitles').empty();

for(vari=0;i<notelist.length;i++){

notedata=notelist[i];

$('#notetitles')

.append('<aclass="btnbtn-lgbtn-blockbtn-outline-dark"

href="/notes/view?key='+

notedata.key+'">'+notedata.title+'</a>');

}

});

});

</script>

ThefirstlineiswhereweloadtheSocket.IOclientlibrary.You'llnoticethatweneversetupanyExpressroutetohandlethe/socket.ioURL.Instead,theSocket.IOlibrarydidthatforus.

Becausewe'vealreadyloadedjQuery(tosupportBootstrap),wecaneasilyensurethatthiscodeisexecutedoncethepageisfullyloadedusing$(document).ready.

Thiscodefirstconnectsasocketobjecttothe/homenamespace.ThatnamespaceisbeingusedforeventsrelatedtotheNoteshomepage.Wethenlistenforthenotetitlesevents,forwhichsomejQueryDOMmanipulationerasesthecurrentlistofNotesandrendersanewlistonthescreen.

That'sit.Ourcodeinroutes/index.mjslistenedtovariouseventsfromtheNotesmodel,and,inresponse,sentanotetitleseventtothebrowser.Thebrowsercodetakesthatlistofnoteinformationandredrawsthescreen.

Youmightnoticethatourbrowser-sideJavaScriptisnotusingES-2015/2016/2017features.Thiscodewould,ofcourse,becleanerifweweretodoso.Howcanweknowwhetherourvisitorsuseabrowsermodernenoughforthoselanguagefeatures?WecoulduseBabeltotranspileES-2015/2016/2017codeintoES5codecapableofrunningonanybrowser.However,itmaybeausefultrade-offtostillwriteES5codeinthebrowser.

RunningNoteswithreal-timehomepageupdatesWenowhaveenoughimplementedtoruntheapplicationandseesomereal-timeaction.

Asyoudidearlier,starttheuserinformationmicroserviceinonewindow:

$npmstart

>user-auth-server@0.0.1startUsersdavid/chap09/users

>DEBUG=users:*PORT=3333SEQUELIZE_CONNECT=sequelize-sqlite.yamlnode--

experimental-modulesuser-server

(node:11866)ExperimentalWarning:TheESMmoduleloaderisexperimental.

users:serviceUser-Auth-Servicelisteningathttp://127.0.0.1:3333+0ms

Then,inanotherwindow,starttheNotesapplication:

$npmstart

>notes@0.0.0startUsersdavid/chap09/notes

>DEBUG=notes:*SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml

NOTES_MODEL=sequelizeUSER_SERVICE_URL=http://localhost:3333node--

experimental-modules./app

(node:11998)ExperimentalWarning:TheESMmoduleloaderisexperimental.

notes:debug-INDEXListeningonport3000+0ms

Then,inabrowserwindow,gotohttp://localhost:3000andlogintotheNotesapplication.Toseethereal-timeeffects,openmultiplebrowserwindows.IfyoucanuseNotesfrommultiplecomputers,thendothataswell.

Inonebrowserwindow,startcreatinganddeletingnotes,whileleavingtheotherbrowserwindowsviewingthehomepage.Createanote,anditshouldshowupimmediatelyonthehomepageintheotherbrowserwindows.Deleteanoteanditshoulddisappearimmediatelyaswell.

Real-timeactionwhileviewingnotesIt'scoolhowwecannowseereal-timechangesinapartoftheNotesapplication.Let'sturntothenotesviewpagetoseewhatwecando.Whatcomestomindisthisfunctionality:

Updatethenoteifsomeoneelseeditsit

Redirecttheviewertothehomepageifsomeoneelsedeletesthenote

Allowuserstoleavecommentsonthenote

Forthefirsttwofeatures,wecanrelyontheexistingeventscomingfromtheNotesmodel.Thethirdfeaturewillrequireamessagingsubsystem,sowe'llgettothatlaterinthischapter.

Inroutes/notes.mjs,addthistotheendofthemodule:

exportfunctionsocketio(io){

notes.events.on('noteupdate',newnote=>{

io.of('view').emit('noteupdate',newnote);

});

notes.events.on('notedestroy',data=>{

io.of('view').emit('notedestroy',data);

});

};

Atthetopofapp.mjs,makethischange:

import{socketioasindexSocketio,routerasindex}from'./routes/index';

import{routerasusers,initPassport}from'./routes/users';

import{socketioasnotesSocketio,routerasnotes}from'./routes/notes';

Uncommentthatlineofcodeinapp.mjsbecausewe'venowimplementedthefunctionwesaidwe'dgettolater:

indexSocketio(io);

notesSocketio(io);

ThissetsuptheNotesapplicationtosendnoteupdateandnotedestroymessageswhennotesareupdatedordestroyed.Thedestinationisthe/viewnamespace.We'llneedtomakeacorrespondingmodificationtothenoteviewtemplatesoitdoestherightthing.Thismeansanybrowserviewinganynoteintheapplicationwillconnecttothisnamespace.Everysuchbrowserwillreceiveeventsaboutanynotebeingchanged,eventhosenotesthatarenotbeingviewed.Thismeansthattheclientcodewillhavetocheckthekey,andonlytakeactioniftheeventreferstothenotebeingdisplayed.

Changingthenoteviewtemplateforreal-timeactionAswedidearlier,inordertomaketheseeventsvisibletotheuser,wemustnotonlyaddclientcodetothetemplate,views/noteview.hbs;weneedacoupleofsmallchangestothetemplate:

<divclass="container-fluid">

<divclass="row"><divclass="col-xs-12">

{{#ifnote}}<h3id="notetitle">{{note.title}}</h3>{{/if}}

{{#ifnote}}<divid="notebody">{{note.body}}</div>{{/if}}

<p>Key:{{notekey}}</p>

</div></div>

{{#ifuser}}

{{#ifnotekey}}

<divclass="row"><divclass="col-xs-12">

<divclass="btn-group">

<aclass="btnbtn-outline-dark"

href="notesdestroy?key={{notekey}}"

role="button">Delete</a>

<acletemplate,views/noteview.hb

ass="btnbtn-outline-dark"

href="notesedit?key={{notekey}}"

role="button">Edit</a>

</div>

</div></div>

{{/if}}

{{/if}}

</div>

{{>footerjs}}

{{#ifnotekey}}

<scriptsrc="socket.iosocket.io.js"></script>

<script>

$(document).ready(function(){

io('/view').on('noteupdate',function(note){

if(note.key==="{{notekey}}"){

$('h3#notetitle').empty();

$('h3#notetitle').text(note.title);

$('#notebody').empty();

$('#notebody').text(note.body);

}

});

io('/view').on('notedestroy',function(data){

if(data.key==="{{notekey}}"){

window.location.href="/";

}

});

});

</script>

{{/if}}

Weconnecttothe/viewnamespacewherethemessagesaresent.Asnoteupdateornotedestroymessagesarrive,wecheckthekeytoseewhetheritmatchesthekeyforthenotebeingdisplayed.

Atechniqueisusedherethat'simportanttounderstand.WehavemixedJavaScriptexecutedontheserver,withJavaScriptexecutedinthebrowser.Wemustcomparethenotekeyreceivedbytheclientcodeagainstthenotekeyforthenotebeingviewedbythispage.Thelatternotekeyvalueisknownontheserver,whiletheformerisknownintheclient.

Rememberthatcodewithinthe{{..}}delimitersisinterpretedbytheHandlebarstemplateengineontheserver.Considerthefollowing:

if(note.key==="{{notekey}}"){

..

}

Thiscomparisonisbetweenthenotekeyvalueinthebrowser,whicharrivedinsidethemessagefromtheserver,andthenotekeyvariableontheserver.Thatvariablecontainsthekeyofthenotebeingdisplayed.Therefore,inthiscase,weareabletoensurethesecodesnippetsareexecutedonlyforthenotebeingshownonthescreen.

Forthenoteupdateevent,wetakethenewnotecontentanddisplayitonthescreen.Forthistowork,wehadtoaddid=attributestotheHTMLsowecouldusejQueryselectorsinmanipulatingtheDOM.

Forthenotedestroyevent,wesimplyredirectthebrowserwindowbacktothehomepage.Thenotebeingviewedhasbeendeleted,andthere'snopointtheusercontinuingtolookatanotethatnolongerexists.

RunningNoteswithreal-timeupdateswhileviewinganoteAtthispoint,youcannowreruntheNotesapplicationandtrythisout.

LaunchtheuserauthenticationserverandtheNotesapplicationasbefore.Then,inthebrowser,openmultiplewindowsontheNotesapplication.Thistime,haveoneviewingthehomepage,andtwoviewinganote.Inoneofthosewindows,editthenotetomakeachange,andseethetextchangeonboththehomepageandthepageviewingthenote.

Thendeletethenote,andwatchitdisappearfromthehomepage,andthebrowserwindowthathadviewedthenoteisnowonthehomepage.

Inter-userchatandcommentingforNotesThisiscool!Wenowhavereal-timeupdatesinNotesasweeditdeleteorcreatenotes.Let'snowtakeittothenextlevelandimplementsomethingakintointer-userchatting.

It'spossibletopivotourNotesapplicationconceptandtakeitinthedirectionofasocialnetwork.Inthemajorityofsuchnetworks,userspostthings(notes,pictures,videos,andsoon),andotheruserscommentonthosethings.Donewell,thesebasicelementscandevelopalargecommunityofpeoplesharingnoteswitheachother.WhiletheNotesapplicationiskindofatoy,it'snottooterriblyfarfrombeingabasicsocialnetwork.Commentingthewaywewilldonowisatinystepinthatdirection.

Oneachnotepage,we'llhaveanareatodisplaymessagesfromNotesusers.Eachmessagewillshowtheusername,atimestamp,andtheirmessage.We'llalsoneedamethodforuserstopostamessage,andwe'llalsoallowuserstodeletemessages.

Eachofthoseoperationswillbeperformedwithoutrefreshingthescreen.Instead,coderunninginsidethewebpagewillsendcommandsto/fromtheserverandtakeactionsdynamically.

Let'sgetstarted.

DatamodelforstoringmessagesWeneedtostartbyimplementingadatamodelforstoringmessages.ThebasicfieldsrequiredareauniqueID,theusernameofthepersonsendingthemessage,thenamespacethemessageissentto,theirmessage,andfinallyatimestampforwhenthemessagewassent.Asmessagesarereceivedordeleted,eventsmustbeemittedfromthemodelsowecandotherightthingonthewebpage.

ThismodelimplementationwillbewrittenforSequelize.Ifyoupreferadifferentstoragesolution,youcan,byallmeans,re-implementthesameAPIonotherdatastoragesystems.

Createanewfile,models/messages-sequelize.mjs,containingthefollowing:

importSequelizefrom'sequelize';

importjsyamlfrom'js-yaml';

importfsfrom'fs-extra';

importutilfrom'util';

importEventEmitterfrom'events';

classMessagesEmitterextendsEventEmitter{}

importDBGfrom'debug';

constdebug=DBG('notes:model-messages');

consterror=DBG('notes:error-messages');

varSQMessage;

varsequlz;

exportconstemitter=newMessagesEmitter();

ThissetsupthemodulesbeingusedandalsoinitializestheEventEmitterinterface.We'realsoexportingtheEventEmitterasemittersoothermodules

canuseit:

asyncfunctionconnectDB(){

if(typeofsequlz==='undefined'){

constyamltext=await

fs.readFile(process.env.SEQUELIZE_CONNECT,'utf8');

constparams=jsyaml.safeLoad(yamltext,'utf8');

sequlz=newSequelize(params.dbname,

params.username,params.password,params.params);

}

if(SQMessage)returnSQMessage.sync();

SQMessage=sequlz.define('Message',{

id:{type:Sequelize.INTEGER,autoIncrement:true,primaryKey:

true},

from:Sequelize.STRING,

namespace:Sequelize.STRING,

message:Sequelize.STRING(1024),

timestamp:Sequelize.DATE

});

returnSQMessage.sync();

}

Thisdefinesourmessageschemainthedatabase.We'llusethesamedatabasethatweusedforNotes,butthemessageswillbestoredintheirowntable.

Theidfieldwon'tbesuppliedbythecaller;instead,itwillbeautogenerated.BecauseitisanautoIncrementfield,eachmessagethat'saddedwillbeassignedanewidnumberbythedatabase:

exportasyncfunctionpostMessage(from,namespace,message){

constSQMessage=awaitconnectDB();

constnewmsg=awaitSQMessage.create({

from,namespace,message,timestamp:newDate()

});

vartoEmit={

id:newmsg.id,from:newmsg.from,

namespace:newmsg.namespace,message:newmsg.message,

timestamp:newmsg.timestamp

};

emitter.emit('newmessage',toEmit);

}

Thisistobecalledwhenauserpostsanewcomment/message.Wefirststoreitinthedatabase,andthenweemitaneventsayingthemessagewascreated:

exportasyncfunctiondestroyMessage(id,namespace){

constSQMessage=awaitconnectDB();

constmsg=awaitSQMessage.find({where:{id}});

if(msg){

msg.destroy();

emitter.emit('destroymessage',{id,namespace});

}

}

Thisistobecalledwhenauserrequeststhatamessageshouldbedeleted.WithSequelize,wemustfirstfindthemessageandthendeleteitbycallingitsdestroymethod.Oncethat'sdone,weemitamessagesayingthemessagewasdestroyed:

exportasyncfunctionrecentMessages(namespace){

constSQMessage=awaitconnectDB();

constmessages=SQMessage.findAll({

where:{namespace},order:['timestamp'],limit:20

});

returnmessages.map(message=>{

return{

id:message.id,from:message.from,

namespace:message.namespace,message:message.message,

timestamp:message.timestamp

};

});

}

Whilethisismeanttobecalledwhenviewinganote,itisgeneralizedtoworkforanySocket.IOnamespace.Itfindsthemostrecent20messagesassociatedwiththegivennamespaceandreturnsacleaned-uplisttothecaller.

AddingmessagestotheNotesrouterNowthatwecanstoremessagesinthedatabase,let'sintegratethisintotheNotesroutermodule.

Inroutes/notes.mjs,addthistotheimportstatements:

import*asmessagesfrom'../models/messages-sequelize';

Ifyouwishtoimplementadifferentdatastoragemodelformessages,you'llneedtochangethisimportstatement.Youshouldconsiderusinganenvironmentvariabletospecifythemodulename,aswe'vedoneelsewhere:

//Saveincomingmessagetomessagepool,thenbroadcastit

router.post('/make-comment',ensureAuthenticated,async(req,res,next)=>{

try{

awaitmessages.postMessage(req.body.from,

req.body.namespace,req.body.message);

res.status(200).json({});

}catch(err){

res.status(500).end(err.stack);

}

});

//Deletetheindicatedmessage

router.post('/del-message',ensureAuthenticated,async(req,res,next)=>{

try{

awaitmessages.destroyMessage(req.body.id,req.body.namespace);

res.status(200).json({});

}catch(err){

res.status(500).end(err.stack);

}

});

Thispairofroutes,notesmake-commentandnotesdel-message,isusedtopostanewcommentordeleteanexistingone.Eachcallsthecorrespondingdatamodelfunctionandthensendsanappropriateresponsebacktothecaller.

RememberthatpostMessagestoresamessageinthedatabase,andthenitturnsaroundandemitsthatmessagetootherbrowsers.Likewise,destroyMessagedeletesthemessagefromthedatabase,thenemitsamessagetootherbrowserssayingthatthemessagehasbeendeleted.Finally,theresultsfromrecentMessageswillreflectthecurrentsetofmessagesinthedatabase.

BothofthesewillbecalledbyAJAXcodeinthebrowser:

module.exports.socketio=function(io){

io.of('/view').on('connection',function(socket){

//'cb'isafunctionsentfromthebrowser,towhichwe

//sendthemessagesforthenamednote.

socket.on('getnotemessages',(namespace,cb)=>{

messages.recentMessages(namespace).then(cb)

.catch(err=>console.error(err.stack));

});

});

messages.emitter.on('newmessage',newmsg=>{

io.of('/view').emit('newmessage',newmsg);

});

messages.emitter.on('destroymessage',data=>{

io.of('/view').emit('destroymessage',data);

});

..

};

ThisistheSocket.IOgluecode,whichwewilladdtothecodewelookedatearlier.

ThegetnotemessagesmessagefromthebrowserrequeststhelistofmessagesforthegivenNote.ThiscallstherecentMessagesfunctioninthemodel.ThisusesafeatureofSocket.IOwheretheclientcanpassacallbackfunction,andserver-sideSocket.IOcodecaninvokethatcallback,givingitsomedata.

Wealsolistentothenewmessageanddestroymessagemessagesemittedbythemessagesmodel,sendingcorrespondingmessagestothebrowser.Thesearesentusingthemethoddescribedearlier.

ChangingthenoteviewtemplateformessagesWeneedtodivebackintoviews/noteview.hbswithmorechangessothatwecanview,create,anddeletemessages.Thistime,wewilladdalotofcode,includingusingaBootstrapmodalpopuptogetthemessage,severalAJAXcallstocommunicatewiththeserver,and,ofcourse,moreSocket.IOstuff.

UsingaModalwindowtocomposemessagesTheBootstrapframeworkhasaModalcomponentthatservesasimilarpurposetoModaldialogsindesktopapplications.YoupopuptheModal,itpreventsinteractionwithotherpartsofthewebpage,youenterstuffintofieldsintheModal,andthenclickabuttontomakeitclose.

ThisnewsegmentofcodereplacestheexistingsegmentdefiningtheEditandDeletebuttons,inviews/noteview.hbs:

{{#ifuser}}

{{#ifnotekey}}

<divclass="row"><divclass="col-xs-12">

<divclass="btn-group">

<aclass="btnbtn-outline-dark"href="notesdestroy?key=

{{notekey}}"

role="button">Delete</a>

<aclass="btnbtn-outline-dark"href="notesedit?key=

{{notekey}}"

role="button">Edit</a>

<buttontype="button"class="btnbtn-outline-dark"

data-toggle="modal"

data-target="#notes-comment-modal">Comment</button>

</div>

</div></div>

<divid="noteMessages"></div>

{{/if}}

{{/if}}

Thisaddssupportforpostingcommentsonanote.TheuserwillseeaModalpop-upwindowinwhichtheywritetheircomment.We'llshowthecodefortheModallater.

WeaddedanewbuttonlabeledCommentthattheuserwillclicktostart

theprocessofpostingamessage.ThisbuttonisconnectedtotheModalbywayoftheelementIDspecifiedinthedata-targetattribute.TheIDwillmatchtheoutermostdivwrappingtheModal.ThisstructureofdivelementsandclassnamesarefromtheBootstrapwebsiteathttp://getbootstrap.com/docs/4.0/components/modal/.

Let'saddthecodefortheModalatthebottomofviews/noteview.hbs.

{{>footerjs}}

{{#ifnotekey}}

{{#ifuser}}

<divclass="modalfade"id="notes-comment-modal"tabindex="-1"

role="dialog"aria-labelledby="noteCommentModalLabel"aria-hidden="true">

<divclass="modal-dialogmodal-dialog-centered"role="document">

<divclass="modal-content">

<divclass="modal-header">

<buttontype="button"class="close"data-dismiss="modal"aria-

label="Close">

<spanaria-hidden="true">&times;</span>

</button>

<h4class="modal-title"id="noteCommentModalLabel">Leavea

Comment</h4>

</div>

<divclass="modal-body">

<formmethod="POST"id="submit-comment"class="well"data-async

data-target="#rating-modal"action="notesmake-comment">

<inputtype="hidden"name="from"value="{{user.id}}">

<inputtype="hidden"name="namespace"value="/view-

{{notekey}}">

<inputtype="hidden"name="key"value="{{notekey}}">

<fieldset>

<divclass="form-group">

<labelfor="noteCommentTextArea">

YourExcellentThoughts,Please</label>

<textareaid="noteCommentTextArea"name="message"

class="form-control"rows="3"></textarea>

</div>

<divclass="form-group">

<divclass="col-sm-offset-2col-sm-10">

<buttonid="submitNewComment"type="submit"class="btn

btn-default">

MakeComment</button>

</div>

</div>

</fieldset>

</form>

</div>

</div>

</div>

</div>

{{/if}}

{{/if}}

ThekeyportionofthisistheHTMLformcontainedwithinthediv.modal-bodyelement.It'sastraightforward,normalBootstrap,augmentedformwithanormalSubmitbuttonatthebottom.Afewhiddeninputelementsareusedtopassextrainformationinsidetherequest.

WiththeHTMLsetupthisway,BootstrapwillensurethatthisModalistriggeredwhentheuserclicksontheCommentbutton.TheusercanclosetheModalbyclickingontheClosebutton.Otherwise,it'suptoustoimplementcodetohandletheformsubmissionusingAJAXsothatitdoesn'tcausethepagetoreload.

Sending,displaying,anddeletingmessagesNotethatthesecodesnippetsarewrappedwith{{#if}}statements,sothatcertainuserinterfaceelementsaredisplayedonlytosufficientlyprivilegedusers.Auserthatisn'tloggedincertainlyshouldn'tbeabletopostamessage.

NowwehavealotofSocket.IOcodetoadd:

{{#ifnotekey}}

{{#ifuser}}

<script>

$(document).ready(function(){...});

{{/if}}

{{/if}}

There'sanothercodesectiontohandlethenoteupdateandnotedestroymessages.Thisnewsectionhastodowithmessagesthatmanagethecomments.

Weneedtohandletheformsubmissionforpostinganewcomment,gettherecentmessageswhenfirstviewinganote,listenforeventsfromtheserveraboutnewmessagesordeletedmessages,renderthemessagesonthescreen,andhandlerequeststodeleteamessage:

$(document).ready(function(){

io('view').emit('getnotemessages','view-{{notekey}}',function(msgs){

$('#noteMessages').empty();

if(msgs.length>0){

msgs.forEach(function(newmsg){

$('#noteMessages').append(formatMessage(newmsg));

});

$('#noteMessages').show();

connectMsgDelButton();

}else$('#noteMessages').hide();

});

varconnectMsgDelButton=function(){

$('.message-del-button').on('click',function(event){

$.post('notesdel-message',{

id:$(this).data("id"),

namespace:$(this).data("namespace")

},

function(response){});

event.preventDefault();

});

};

varformatMessage=function(newmsg){

return'<divid="notemessage-'+newmsg.id+'"class="card">'

+'<divclass="card-body">'

+'<h5class="card-title">'+newmsg.from+'</h5>'

+'<divclass="card-text">'+newmsg.message

+'<smallstyle="display:block">'+newmsg.timestamp

+'</small></div>'

+'<buttontype="button"class="btnbtn-primarymessage-

del-button"data-id="'

+newmsg.id+'"data-namespace="'+newmsg.namespace+'">'

+'Delete</button>'

+'</div>'

+'</div>';

};

io('/view').on('newmessage',function(newmsg){

if(newmsg.namespace==='/view-{{notekey}}'){

$('#noteMessages').prepend(formatMessage(newmsg));

connectMsgDelButton();

}

});

io('/view').on('destroymessage',function(data){

if(data.namespace==='/view-{{notekey}}'){

$('#noteMessages#notemessage-'+data.id).remove();

}

});

$('form#submit-comment').submit(function(event){

//Abortanypendingrequest

if(request){request.abort();}

var$form=$('form#submit-comment');

var$target=$($form.attr('data-target'));

varrequest=$.ajax({

type:$form.attr('method'),

url:$form.attr('action'),

data:$form.serialize()

});

request.done(function(response,textStatus,jqXHR){});

request.fail(function(jqXHR,textStatus,errorThrown){

alert("ERROR"+jqXHR.responseText);

});

request.always(function(){

//Reenabletheinputs

$('#notes-comment-modal').modal('hide');

});

event.preventDefault();

});

});

Thecodewithin$('form#submit-comment').submithandlestheformsubmissionforthecommentform.BecausewealreadyhavejQueryavailable,wecanuseitsAJAXsupporttoPOSTarequesttotheserverwithoutcausingapagereload.

Usingevent.preventDefault,weensurethatthedefaultactiondoesnotoccur.FortheFORMsubmission,thatmeansthebrowserpagedoesnotreload.WhathappensisanHTTPPOSTissenttonotesmake-commentwithadatapayloadconsistingofthevaluesoftheform'sinputelements.Includedinthosevaluesarethreehiddeninputs,from,namespace,andkey,providingusefulidentificationdata.

Ifyourefertothenotesmake-commentroutedefinition,thiscallsmessagesModel.postMessagetostorethemessageinthedatabase.Thatfunctionthenpostsanevent,newmessage,whichourserver-sidecodeforwardstoanybrowserthat'sconnectedtothenamespace.Shortlyafterthat,anewmessageeventwillarriveinbrowsers.

Thenewmessageeventaddsamessageblock,usingtheformatMessagefunction.TheHTMLforthemessageisprependedto#noteMessages.

Whenthepageisfirstloaded,wewanttoretrievethecurrentmessages.Thisiskickedoffwithio('/view').emit('getnotemessages',...Thisfunction,asthenameimplies,sendsagetnotemessagesmessagetotheserver.Weshowedtheimplementationoftheserver-sidehandlerforthismessageearlier,inroutes/notes.mjs.

Ifyouremember,wesaidthatSocket.IOsupportstheprovisionofacallbackfunctionthatiscalledbytheserverinresponsetoanevent.You

simplypassafunctionasthelastparametertoa.emitcall.Thatfunctionismadeavailableattheotherendofthecommunication,tobecalledwhenappropriate.Tomakethisclear,wehaveacallbackfunctiononthebrowserbeinginvokedbyserver-sidecode.

Inthiscase,theserver-sidecallsourcallbackfunctionwithalistofmessages.Themessagelistarrivesintheclient-sidecallbackfunction,whichdisplaystheminthe#noteMessagesarea.ItusesjQueryDOMmanipulationtoeraseanyexistingmessages,thenrenderseachmessageintothemessagesareausingtheformatMessagefunction.

Themessagedisplaytemplate,informatMessage,isstraightforward.ItusesaBootstrapcardtogiveanicevisualeffect.And,thereisabuttonfordeletingmessages.

InformatMessagewecreatedaDeletebuttonforeachmessage.Thosebuttonsneedaneventhandler,andtheeventhandlerissetupusingtheconnectMsgDelButtonfunction.Inthiscase,wesendanHTTPPOSTrequesttonotesdel-message.WeagainusethejQueryAJAXsupporttopostthatHTTPrequest.

Thenotesdel-messagerouteinturncallsmessagesModel.destroyMessagetodothedeed.Thatfunctionthenemitsanevent,destroymessage,whichgetssentbacktothebrowser.Asyouseehere,thedestroymessageeventhandlercausesthecorrespondingmessagetoberemovedusingjQueryDOMmanipulation.Wewerecarefultoaddanid=attributetoeverymessagetomakeremovaleasy.

Sincetheflipsideofdestructioniscreation,weneedtohavethenewmessageeventhandlersittingnexttothedestroymessageeventhandler.ItalsousesjQueryDOMmanipulationtoinsertthenewmessageintothe#noteMessagesarea.

Whileenteringamessage,theModallookslikethis:

RunningNotesandpassingmessagesThatwasalotofcode,butwenowhavetheabilitytocomposemessages,displaythemonthescreen,anddeletethem,allwithnopagereloads.

Youcanruntheapplicationaswedidearlier,firststartingtheuserauthenticationserverinonecommand-linewindowandtheNotesapplicationinanother:

Trythiswithmultiplebrowserwindowsviewingthesamenoteordifferentnotes.Thisway,youcanverifythatnotesshowuponlyonthecorrespondingnotewindow.

OtherapplicationsofModalwindowsWeusedaModalandsomeAJAXcodetoavoidonepagereloadduetoaformsubmission.IntheNotesapplication,asitstands,asimilartechniquecouldbeusedwhencreatinganewnote,editingexistingnotes,anddeletingexistingnotes.Ineachcase,wewoulduseaModal,someAJAXcodetohandletheformsubmission,andsomejQuerycodetoupdatethepagewithoutcausingareload.

Butwait,that'snotall.Considerthesortofdynamicreal-timeuserinterfacewizardryonthepopularsocialnetworks.Imaginewhateventsand/orAJAXcallsarerequired.

WhenyouclickonanimageinTwitter,itpopsup,youguessedit,aModalwindowtoshowalargerversionoftheimage.TheTwitterComposenewTweetwindowisalsoaModalwindow.FacebookusesmanydifferentModalwindows,suchaswhensharingapost,reportingaspampost,orwhiledoingalotofotherthingsFacebook'sdesignersdeemtorequireapop-upwindow.

Socket.IO,aswe'veseen,givesusarichfoundationofeventspassingbetweenserverandclientthatcanbuildmultiuser,multichannelcommunicationexperiencesforyourusers.

SummaryWhilewecamealongwayinthischapter,maybeFacebookdoesn'thaveanythingtofearfromthebabystepswetooktowardconvertingtheNotesapplicationintoasocialnetwork.Thischaptergaveustheopportunitytoexploresomereallycooltechnologyforpseudoreal-timecommunicationbetweenbrowsersessions.

Lookupthetechnicaldefinitionforthephraserealtimeandyou'llseethereal-timewebisnottrulyrealtime.Theactualmeaningofrealtimeinvolvessoftwarewithstricttimeboundariesthatmustrespondtoeventswithinaspecifiedtimeconstraint.Real-timesoftwareistypicallyusedinembeddedsystemstorespondtobuttonpresses,forapplicationsasdiverseasjunkfooddispensersandmedicaldevicesinintensivecareunits.Eattoomuchjunkfoodandyoucouldendupinintensivecare,andbeservedbyreal-timesoftwareinbothcases.Tryandrememberthedistinctionbetweendifferentmeaningsforthisphrase.

Inthischapter,youlearnedaboutusingSocket.IOforpseudoreal-timewebexperiences,usingtheEventEmitterclasstosendmessagesbetweenpartsofanapplication,jQuery,AJAX,andotherbrowser-sideJavaScripttechnologies,whileavoidingpagereloadswhilemakingAJAXcalls.

Inthenextchapter,wewilllookintoNode.jsapplicationdeploymentonrealservers.Runningcodeonourlaptopiscool,buttohitthebigtime,theapplicationneedstobeproperlydeployed.

DeployingNode.jsApplicationsNowthattheNotesapplicationisfairlycomplete,it'stimetothinkabouthowtodeployittoarealserver.We'vecreatedaminimalimplementationofthecollaborativenoteconceptthatworksfairlywell.Togrow,Notesmustescapeourlaptopandliveonarealserver.ThegoalistolookatdeploymentmethodsforNode.jsapplications.

Inthischapter,wewillcoverthefollowingtopics:

TraditionalLSB-compliantNode.jsdeployment

UsingPM2toimprovereliability

DeploymenttoaVirtualPrivateServer(VPS)provider

MicroservicedeploymentwithDocker(wehavefourdistinctservicestodeploy)

DeploymenttoaDockerhostingprovider

Thefirsttaskistoduplicatethesourcecodefromthepreviouschapter.It'ssuggestedyoucreateanewdirectory,chap10,asasiblingofthechap09directory,andcopyeverythingfromchap09tochap10.

NotesapplicationarchitectureanddeploymentconsiderationsBeforewegetintodeployingtheNotesapplication,weneedtoreviewitsarchitecture.TodeploytheNotesapplication,wemustunderstandwhatwe'replanningtodo.Wehavesegmentedtheservicesintotwogroups,asshowninthefollowingdiagram:

Theuserauthenticationservershouldbethemoresecureportionofthe

system.Onourlaptop,weweren'tabletocreatetheenvisionedprotectivewallaroundthatservice,butwe'reabouttoimplementsuchprotection.

Onestrategytoenhancesecurityistoexposeasfewportsaspossible.Thatreducestheso-calledattacksurface,simplifyingourworkinhardeningtheapplicationagainstsecuritybugs.WiththeNotesapplication,wehaveexactlyoneporttoexpose,theHTTPservicethroughwhichusersaccesstheapplication.Theotherports,thetwoforMySQLserversandtheuserauthenticationserviceport,shouldbehidden.

Internally,theNotesapplicationneedstoaccessboththeNotesdatabaseandtheuserauthenticationservice.Thatservice,inturn,needstoaccesstheuserauthenticationdatabase.Ascurrentlyenvisaged,noserviceoutsidetheNotesapplicationrequiresaccesstoeitherdatabaseortotheauthenticationservice.

Implementationofthissegmentationrequireseithertwoorthreesubnets,dependingonthelengthsyouwishtogoto.Thefirst,FrontNet,containstheNotesapplicationanditsdatabase.Thesecond,AuthNet,containstheauthenticationserviceanditsdatabase.AthirdpossiblesubnetwouldcontaintheNotesandauthenticationservices.Thesubnetconfigurationmustlimitthehostswithaccesstothesubnet,andcreateasecuritywallbetweensubnets.

TraditionalLinuxNode.jsservicedeploymentTraditionalLinux/Unixserverapplicationdeploymentusesaninitscripttomanagebackgroundprocesses.Theyaretostarteverytimethesystembootsandcleanlyshutdownwhenthesystemishalted.Whileit'sasimplemodel,thespecificsofthisvarywidelyfromoneoperatingsystem(OS)toanother.

Acommonmethodisfortheinitprocesstomanagebackgroundprocessesusingshellscriptsintheetcinit.ddirectory.OtherOSesuseotherprocessmanagers,suchasupstartorlaunchd.

TheNode.jsprojectitselfdoesnotincludeanyscriptstomanageserverprocessesonanyOS.Node.jsismorelikeaconstructionkit,withthepiecesandpartstobuildservers,andisnotacompletepolishedserverframeworkitself.ImplementingacompletewebservicebasedonNode.jsmeanscreatingthescriptingtointegratewithprocessmanagementonyourOS.It'suptoustodevelopthosescripts.

Webserviceshavetobe:

Reliable:Forexample,toauto-restartwhentheserverprocesscrashes

Manageable:Meaningitintegrateswellwithsystemmanagementpractices

Observable:Meaningtheadministratormustbeabletogetstatusandactivityinformationfromtheservice

Todemonstratewhat'sinvolved,let'susePM2toimplementbackgroundserverprocessmanagementforNotes.PM2detectsthesystemtypeandcanautomaticallyintegrateitselfwiththeprocessmanagementsystem.ItwillcreateanLSB-styleinitscript(http://wiki.debian.org/LSBInitScripts),orotherscripts,asrequiredbytheprocessmanagementsystemonyourserver.

Forthisdeployment,we'llsetupasingleUbuntu17.10server.YoushouldprovisionaVirtualPrivateServer(VPS)fromahostingprovideranddoallinstallationandconfigurationthere.Rentingasmallmachineinstancefromoneofthemajorprovidersforthetimeneededtogothroughthischapterwillonlycostacoupleofdollars.

YoucanalsodothetasksinthissectionusingVirtualBoxonyourlaptop.SimplyinstallDebianorUbuntuasavirtualmachineinVirtualBox,thenfollowtheinstructionsinthissection.Itwon'tbequitethesameasusingaremoteVPShostingprovider,butdoesnotrequirerentingaserver.

BoththeNotesanduserauthenticationserviceswillbeonthatserver,alongwithasingleMySQLinstance.WhileourgoalisastrongseparationbetweenFrontNetandAuthNet,withtwoMySQLinstances,wewon'tdosoatthistime.

Prerequisite–provisioningthedatabasesTheLinuxpackagemanagementsystemdoesn'tallowustoinstalltwoMySQLinstances.Instead,weimplementseparationinthesameMySQLinstancebyusingseparatedatabaseswithdifferentusernamesandaccessprivilegesforeachdatabase.

ThefirststepistoensurethatMySQLisinstalledonyourserver.ForUbuntu,DigitalOceanhasafairlygoodtutorial:https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-14-

04.WhiletheUbuntuversionforthattutorialisold,theinstructionsarestillaccurateenough.

TheMySQLservermustsupportTCPconnectionsfromlocalhost.Edittheconfigurationfile,etcmysql/my.cnf,tohavethefollowingline:

bind-address=127.0.0.1

ThislimitsMySQLserverconnectionstotheprocessesontheserver.Amiscreantwouldhavetobreakintotheservertoaccessyourdatabase.Nowthatourdatabaseserverisavailable,let'ssetuptwodatabases.

Inthechap10/notes/modelsdirectory,createafilenamedmysql-create-db.sqlcontainingthefollowing:

CREATEDATABASEnotes;

CREATEUSER'notes'@'localhost'IDENTIFIEDBY'notes';

GRANTALLPRIVILEGESONnotes.*TO'notes'@'localhost'WITHGRANTOPTION;

Inthechap10/usersdirectory,createafilenamedmysql-create-db.sqlcontainingthefollowing:

CREATEDATABASEuserauth;

CREATEUSER'userauth'@'localhost'IDENTIFIEDBY'userauth';

GRANTALLPRIVILEGESONuserauth.*TO'userauth'@'localhost'WITHGRANT

OPTION;

Wecan'trunthosescriptsontheserver,becausetheNotesapplicationhasnotyetbeencopiedtotheserver.Whenthat'saccomplished,we'llrunthescriptsas:

$mysql-uroot-p<chap10/users/mysql-create-db.sql

$mysql-uroot-p<chap10/notes/models/mysql-create-db.sql

Thiswillcreatethetwodatabases,notesanduserauth,withassociatedusernamesandpasswords.Eachusercanaccessonlytheirassociateddatabase.Later,we'llsetupNotesandtheuserauthenticationservicewithYAMLconfigurationfilestoaccessthesedatabases.

<strong>$curl-sLhttps://deb.nodesource.com/setup_10.x|sudo-Ebash-</strong><br/><strong>$sudoapt-getupdate

$sudoapt-getinstall-ynodejsbuild-essential</strong>

We'veseenthisbefore,sosubstitutetheNode.jsdesiredversionnumberintheURL.InstallingthiswaymeansthatasnewNode.jsreleasesareissued,upgradesareeasilyaccomplishedwiththenormalpackagemanagementprocedures.

SettingupNotesanduserauthenticationontheserverBeforecopyingtheNotesanduserauthenticationcodetothisserver,let'sdoalittlecodingtoprepareforthemove.WeknowthattheNotesandauthenticationservicesmustaccesstheMySQLinstanceonlocalhostusingtheusernamesandpasswordsgivenearlier.

Usingtheapproachwe'vefollowedsofar,thismeansapairofYAMLfilesforSequelizeparameters,andchangingenvironmentvariablesinthepackage.jsonfilestomatch.

Createachap10/notes/models/sequelize-server-mysql.yamlfilecontaining:

dbname:notes

username:notes

password:notes12345

params:

host:localhost

port:3306

dialect:mysql

ItwasdiscoveredduringtestingthatasimplepasswordsuchasnoteswasnotacceptabletotheMySQLserver,andthatalongerpasswordwasrequired.Inchap10/notes/package.json,addthefollowinglinetothescriptssection:

"on-server":"SEQUELIZE_CONNECT=models/sequelize-server-mysql.yaml

NOTES_MODEL=sequelizeUSER_SERVICE_URL=http://localhost:3333PORT=3000node-

-experimental-modules./app",

Thencreateachap10/users/sequelize-server-mysql.yamlfilecontainingthefollowingcodethefollowingcode:

dbname:userauth

username:userauth

password:userauth

params:

host:localhost

port:3306

dialect:mysql

Thepasswordsshownintheseconfigurationfilesobviouslywillnotpassanysecurityaudits.

Inchap10/users/package.json,addthefollowinglinetothescriptssection:

"on-server":"PORT=3333SEQUELIZE_CONNECT=sequelize-server-mysql.yamlnode--

experimental-modules./user-server",

Thisconfigurestheauthenticationservicetoaccessthedatabasesjustcreated.

Nowweneedtoselectaplaceontheservertoinstalltheapplicationcode:

#ls/opt

Thisemptydirectorylookstobeasgoodaplaceasany.Simplyuploadchap10/notesandchap10/userstoyourpreferredlocation.Beforeuploading,removethenode_modulesdirectoryinbothdirectories.That'sbothtosavetimeontheupload,andbecauseofthesimplefactthatanynative-codemodulesinstalledonyourlaptopwillbeincompatiblewiththeserver.Onyourlaptop,youmightrunacommandlikethis:

$rsync--archive--verbose./root@159.89.145.190:opt

UsetheactualIPaddressordomainnameassignedtotheserverbeingused.

Youshouldendupwithsomethinglikethefollowing:

#ls/opt

notesusers

Then,ineachdirectory,runthesecommands:

#rm-rfnode_modules

#npminstall

We'rerunningthesecommandsasrootratherthanauserIDthatcanusethesudocommand.Themachineofferedbythechosenhostingprovider(DigitalOcean)isconfiguredsousersloginasroot.OtherVPShostingproviderswillprovidemachineswhereyouloginasaregularuserandthenusesudotoperformprivilegedoperations.Asyoureadtheseinstructions,payattentiontothecommandpromptweshow.We'vefollowedtheconventionwhere$isusedforcommandsrunasaregularuserand#isusedforcommandsrunasroot.Ifyou'rerunningasaregularuser,andneedtorunarootcommand,thenrunthecommandwithsudo.

Thesimplestwayofdoingthisistojustdeletethewholenode_modulesdirectoryandthenletnpminstalldoitsjob.RememberthatwesetupthePATHenvironmentvariablethefollowingway:

#exportPATH=./node_modules/.bin:${PATH}

Youcanplacethisintheloginscript(.bashrc,.cshrc,andsoon)onyourserversoit'sautomaticallyenabled.

Finally,youcannowruntheSQLscriptswrittenearliertosetupthedatabaseinstances:

#mysql-uroot-p<users/mysql-create-db.sql

#mysql-uroot-p<notes/models/mysql-create-db.sql

Thenyoushouldbeabletostartuptheservicesbyhandtocheckthateverythingisworkingcorrectly.TheMySQLinstancehasalreadybeentested,sowejustneedtostarttheuserauthenticationandNotesservices:

#cdoptusers

#DEBUG=users:*npmrunon-server

>userauth-server@0.0.1on-serveroptusers

>PORT=3333SEQUELIZE_CONNECT=sequelize-server-mysql.yamlnode--

experimental-modules./user-server

(node:9844)ExperimentalWarning:TheESMmoduleloaderisexperimental.

ThenlogintotheserveronanotherTerminalsessionandrunthefollowing:

#cdoptusers/

#PORT=3333nodeusers-add.js

Created{id:1,username:'me',password:'w0rd',provider:'local',

familyName:'Einarrsdottir',givenName:'Ashildr',middleName:'',

emails:'[]',photos:'[]',

updatedAt:'2018-02-02T00:43:16.923Z',createdAt:'2018-02-

02T00:43:16.923Z'}

#PORT=3333nodeusers-list.js

List[{id:'me',username:'me',provider:'local',

familyName:'Einarrsdottir',givenName:'Ashildr',middleName:'',

emails:'[]',photos:'[]'}]

Theprecedingcommandbothteststhatthebackenduserauthenticationserviceisfunctioningandgivesusauseraccountwecanusetologin.Theusers-listcommanddemonstratesthatitworks.

Youmaygetanerror:

users:error/create-userError:Pleaseinstallmysql2packagemanually

ThisisgeneratedinsideofSequelize.Themysql2driverisanalternateMySQLdriver,implementedinpureJavaScript,andincludessupportforreturningPromisesforsmoothusageinasyncfunctions.Ifyoudogetthismessage,goaheadandinstallthepackageandremembertoaddthisdependencytoyourpackage.json.

NowwecanstarttheNotesservice:

#cd../notes

#npmrunon-server

>notes@0.0.0on-serveroptnotes

>SEQUELIZE_CONNECT=models/sequelize-server-mysql.yamlNOTES_MODEL=sequelize

USER_SERVICE_URL=http://localhost:3333PORT=3000node--experimental-modules

./app

(node:9932)ExperimentalWarning:TheESMmoduleloaderisexperimental.

Thenwecanuseourwebbrowsertoconnecttotheapplication.Sinceyouprobablydonothaveadomainnameassociatedwiththisserver,NotescanbeaccessedviatheIPaddressoftheserver,suchashttp://159.89.145.190:3000/.

Intheseexamples,we'reusingtheIPaddressoftheVPSusedtotesttheinstructionsinthissection.TheIPaddressyouusewill,ofcourse,bedifferent.

Bynow,youknowthatthedrillforverifyingNotesisworking.Createafewnotes,openafewbrowserwindows,seethatreal-timenotificationswork,andsoon.Onceyou'resatisfiedthatNotesisworkingontheserver,killtheprocessesandmoveontothenextsection,wherewe'llsetthisuptorunwhentheserverstarts.

AdjustingTwitterauthenticationtoworkontheserverTheTwitterapplicationwesetupforNotespreviouslywon'tworkbecausetheauthenticationURLisincorrectfortheserver.Fornow,wecanloginusingtheuserprofilecreatedpreviously.IfyouwanttoseeOAuthworkwithTwitter,gotoapps.twitter.comandreconfiguretheapplicationtousetheIPaddressofyourserver.

Byhostingsomewhereotherthanourlaptop,theTwittercallbackURLmustpointtothecorrectlocation.Thedefaultvaluewashttp://localhost:3000foruseonourlaptop.ButwenowneedtousetheIPaddressfortheserver.Innotes/package.json,addthefollowingenvironmentvariabletotheon-serverscript:

TWITTER_CALLBACK_HOST=http://159.89.145.190:3000

UsetheactualIPaddressordomainnameassignedtotheserverbeingused.Inarealdeployment,we'llhaveadomainnametousehere.

SettingupPM2tomanageNode.jsprocessesTherearemanywaystomanageserverprocesses,toensurerestartsiftheprocesscrashes,andsoon.We'llusePM2(http://pm2.keymetrics.io/)becauseit'soptimizedforNode.jsprocesses.Itbundlesprocessmanagementandmonitoringintooneapplication.

Let'screateadirectory,init,inwhichtousePM2.ThePM2websitesuggestsyouinstallthetoolgloballybut,asstudentsoftheTwelveFactorApplicationmodel,werecognizeit'sbesttouseexplicitlydeclareddependenciesandavoidglobalunmanageddependencies.

Createapackage.jsonfilecontaining:

{

"name":"pm2deploy",

"version":"1.0.0",

"scripts":{

"start":"pm2startecosystem.json",

"stop":"pm2stopecosystem.json",

"restart":"pm2restartecosystem.json",

"status":"pm2status",

"save":"pm2save",

"startup":"pm2startup"

},

"dependencies":{

"pm2":"^2.9.3"

}

}

InstallPM2usingnpminstallasusual.

InnormalPM2usage,welaunchscriptswithpm2startscript-name.js.Wecouldmakeanetcinitscriptwhichdoesthat,butPM2alsosupportsafile

namedecosystem.jsonthatcanbeusedtomanageaclusterofprocesses.Wehavetwoprocessestomanagetogether,theuser-facingNotesapplication,andtheuserauthenticationserviceonthebackend.

Createafilenamedecosystem.jsoncontainingthefollowing:

{

"apps":[

{

"name":"UserAuthentication",

"script":"user-server.mjs",

"cwd":"optusers",

"node_args":"--experimental-modules",

"env":{

"PORT":"3333",

"SEQUELIZE_CONNECT":"sequelize-server-mysql.yaml"

},

"env_production":{"NODE_ENV":"production"}

},

{

"name":"Notes",

"script":"app.mjs",

"cwd":"optnotes",

"node_args":"--experimental-modules",

"env":{

"PORT":"3000",

"SEQUELIZE_CONNECT":"models/sequelize-server-mysql.yaml",

"NOTES_MODEL":"sequelize",

"USER_SERVICE_URL":"http://localhost:3333",

"TWITTER_CONSUMER_KEY":"..",

"TWITTER_CONSUMER_SECRET":"..",

"TWITTER_CALLBACK_HOST":"http://45.55.37.74:3000"

},

"env_production":{"NODE_ENV":"production"}

}

]

}

Thisfiledescribesthedirectoriescontainingbothservices,thescripttoruneachservice,thecommand-lineoptions,andtheenvironmentvariablestouse.It'sthesameinformationthatisinthepackage.jsonscripts,butspelledoutmoreclearly.AdjustTWITTER_CALLBACK_HOSTfortheIPaddressoftheserver.Fordocumentation,seehttp://pm2.keymetrics.io/docs/usage/appli

cation-declaration/.

Wethenstarttheserviceswithnpmrunstart,whichlookslikethefollowingonthescreen:

YoucanagainnavigateyourbrowsertotheURLforyourserver,suchashttp://159.89.145.190:3000,andcheckthatNotesisworking.Oncestarted,someusefulcommandsareasfollows:

#pm2list

#pm2describe1

#pm2logs1

Thesecommandsletyouquerythestatusoftheservices.

Thepm2monitcommandgivesyouapseudo-graphicalmonitorofsystemactivity.Fordocumentation,

seehttp://pm2.keymetrics.io/docs/usage/monitoring/.

Thepm2logscommandaddressestheapplicationlogmanagementissueweraisedelsewhere.Activitylogsshouldbetreatedasaneventstream,andshouldbecapturedandmanagedappropriately.WithPM2,theoutputisautomaticallycaptured,canbeviewed,andthelogfilescanberotatedandpurged.Seehttp://pm2.keymetrics.io/docs/usage/log-management/fordocumentation.

Ifwerestarttheserver,theseprocessesdon'tstartwiththeserver.Howdowehandlethat?It'sverysimplebecausePM2cangenerateaninitscriptforus:

#pm2save

[PM2]Savingcurrentprocesslist...

[PM2]Successfullysavedinroot.pm2/dump.pm2

#pm2startup

[PM2]InitSystemfound:systemd

Platformsystemd

Template

[Unit]

Description=PM2processmanager

Documentation=https://pm2.keymetrics.io/

After=network.target

...moreoutputisprinted

Thepm2savecommandsavesthecurrentstate.Whateverservicesarerunningatthattimewillbesavedandmanagedbythegeneratedstartupscript.

Thenextstepistogeneratethestartupscript,usingthepmstartupcommand.PM2supportsgeneratingstartupscriptsonseveralOSes,butwhenrunthisway,itautodetectsthesystemtypeandgeneratesthecorrectstartupscript.Italsoinstallsthestartupscript,andstartsitrunning.Seethedocumentationathttp://pm2.keymetrics.io/docs/usage/startup/formoreinformation.

Ifyoulookcloselyattheoutput,someusefulcommandswillbeprinted.

Thedetailswillvarybasedonyouroperatingsystem,becauseeachoperatingsystemhasitsowncommandsformanagingbackgroundprocesses.Inthiscase,theinstallationisgearedtousethesystemctlcommand,asverifiedbythisoutput:

Commandlist

['systemctlenablepm2-root',

'systemctlstartpm2-root',

'systemctldaemon-reload',

'systemctlstatuspm2-root']

[PM2]Writinginitconfigurationinetcsystemd/system/pm2-root.service

[PM2]Makingscriptbootingatstartup...

...

[DONE]

>>>Executingsystemctlstartpm2-root

[DONE]

>>>Executingsystemctldaemon-reload

[DONE]

>>>Executingsystemctlstatuspm2-root

Youarefreetorunthesecommandsyourself:

#systemctlstatuspm2-root

●pm2-root.service-PM2processmanager

Loaded:loaded(etcsystemd/system/pm2-root.service;enabled;vendor

preset:enabled)

Active:active(running)sinceFri2018-02-0222:27:45UTC;29minago

Docs:https://pm2.keymetrics.io/

Process:738ExecStart=optinit/node_modules/pm2/bin/pm2resurrect

(code=exited,status=0/SUCCESS)

MainPID:873(PM2v2.9.3:God)

Tasks:30(limit:4915)

Memory:171.6M

CPU:11.528s

CGroup:system.slicepm2-root.service

├─873PM2v2.9.3:GodDaemon(root.pm2)

├─895nodeoptusers/user-server.mjs

└─904nodeoptnotes/app.mjs

ToverifythatPM2startstheservicesasadvertised,rebootyourserver,thenusePM2tocheckthestatus:

Thefirstthingtonoticeisthatuponinitiallyloggingintotherootaccount,thepm2statuscommandisnotavailable.WeinstalledPM2locallytooptinit,andthecommandisonlyavailableinthatdirectory.

Aftergoingtothatdirectory,wecannowrunthecommandandseethestatus.RemembertosetthecorrectIPaddressordomainnameintheTWITTER_CALLBACK_HOSTenvironmentvariable.Otherwise,logginginwithTwitterwillfail.

WenowhavetheNotesapplicationunderafairlygoodmanagementsystem.Wecaneasilyupdateitscodeontheserverandrestarttheservice.Iftheservicecrashes,PM2willautomaticallyrestartit.Logfilesareautomaticallykeptforourperusal.

PM2alsosupportsdeploymentfromthesourceonourlaptop,whichwecanpushtostagingorproductionenvironments.Tosupportthis,wemustadddeploymentinformationtotheecosystem.jsonfileandthenrunthepm2deploycommandtopushthecodetotheserver.SeethePM2websiteformoreinformation:http://pm2.keymetrics.io/docs/usage/deployment/.

WhilePM2doesagoodjobatmanagingserverprocesses,thesystem

we'vedevelopedisinsufficientforaninternet-scaleservice.WhatiftheNotesapplicationweretobecomeaviralhitandsuddenlyweneedtodeployamillionserversspreadaroundtheplanet?Deployingandmaintainingserversoneatatime,likethis,isnotscalable.

Wealsoskippedoverimplementingthearchitecturaldecisionsatthebeginning.Puttingtheuserauthenticationdataonthesameserverisasecurityrisk.Wewanttodeploythatdataonadifferentserver,undertightersecurity.

Inthenextsection,we'llexploreanewsystem,Docker,thatsolvestheseproblemsandmore.

Node.jsmicroservicedeploymentwithDockerDocker(http://docker.com)isthenewattractioninthesoftwareindustry.Interestistakingofflikecrazy,spawningmanyprojects,oftenwithnamescontainingpunsaroundshippingcontainers.

Itisdescribedasanopenplatformfordistributedapplicationsfordevelopersandsysadmins.ItisdesignedaroundLinuxcontainerizationtechnologyandfocusesondescribingtheconfigurationofsoftwareonanyvariantofLinux.

Dockerautomatestheapplicationdeploymentwithinsoftwarecontainers.ThebasicconceptsofLinuxcontainersdatebacktochrootjail'sfirstimplementationinthe1970s,andothersystemssuchasSolarisZones.TheDockerimplementationcreatesalayerofsoftwareisolationandvirtualizationbasedonLinuxcgroups,kernelnamespaces,andunion-capablefilesystems,whichblendtogethertomakeDockerwhatitis.Thatwassomeheavygeek-speak,solet'stryasimplerexplanation.

ADockercontainerisarunninginstantiationofaDockerimage.AnimageisagivenLinuxOSandapplicationconfigurationdesignedbydevelopersforwhateverpurposetheyhaveinmind.DevelopersdescribeanimageusingaDockerfile.TheDockerfileisafairlysimple-to-writescriptshowingDockerhowtobuildanimage.Dockerimagesaredesignedtobecopiedtoanyserver,wheretheimageisinstantiatedasaDockercontainer.

Arunningcontainerwillmakeyoufeellikeyou'reinsideavirtualserverrunningonavirtualmachine.ButDockercontainerizationisverydifferentfromavirtualmachinesystemsuchasVirtualBox.TheprocessesrunninginsidethecontainerareactuallyrunningonthehostOS.The

containerizationtechnology(cgroups,kernelnamespaces,andsoon)createtheillusionofrunningontheLinuxvariantspecifiedintheDockerfile,evenifthehostOSiscompletelydifferent.YourhostOScouldbeUbuntuandthecontainerOScouldbeFedoraorOpenSUSE;Dockermakesitallwork.

Bycontrast,withVirtualMachinesoftware(VirtualBox,andVMWare,amongothers),you'reusingwhatfeelslikearealcomputer.ThereisavirtualBIOSandvirtualizedsystemhardware,andyoumustinstallafull-fledgedguestOS.Youmustfolloweveryritualofcomputerownership,includingsecuringlicensesifit'saclosedsourcesystemsuchasWindows.

WhileDockerisprimarilytargetedatx86flavorsofLinux,itisavailableonseveralARM-basedOSes,aswellasotherprocessors.YoucanevenrunDockeronsingle-boardcomputers,suchasRaspberryPis,forhardware-orientedInternetofThingsprojects.OperatingsystemssuchasResin.IOareoptimizedtosolelyrunDockercontainers.

TheDockerecosystemcontainsmanytools,andtheirnumberisquicklyincreasing.Forourpurposes,we'llbefocusingonthefollowingthreespecifictools:

Dockerengine:Thisisthecoreexecutionsystemthatorchestrateseverything.ItrunsonaLinuxhostsystem,exposinganetwork-basedAPIthatclientapplicationsusetomakeDockerrequests,suchasbuilding,deploying,andrunningcontainers.

Dockermachine:ThisisaclientapplicationperformingfunctionsaroundprovisioningDockerEngineinstancesonhostcomputers.

Dockercompose:Thishelpsyoudefine,inasinglefile,amulticontainerapplication,withallitsdependenciesdefined.

WiththeDockerecosystem,youcancreateawholeuniverseofsubnets

andservicestoimplementyourdreamapplication.Thatuniversecanrunonyourlaptoporbedeployedtoaglobe-spanningnetworkofcloud-hostingfacilitiesaroundtheworld.Thesurfaceareathroughwhichmiscreantscanattackisstrictlydefinedbythedeveloper.Amulticontainerapplicationwillevenlimitaccesssostronglybetweenservicesthatmiscreantswhodomanagetobreakintoacontainerwillfinditdifficulttobreakoutofthecontainer.

UsingDocker,we'llfirstdesignonourlaptopthesystemshowninthepreviousdiagram.Thenwe'llmigratethatsystemtoaDockerinstanceonaserver.

InstallingDockeronyourlaptopThebestplacetolearnhowtoinstallDockeronyourlaptopistheDockerdocumentationwebsite.Whatwe'relookingforistheDockerCommunityEdition(CE).ThereistheDockerEnterpriseEdition(EE),withmorefeaturesandsomeopportunitiestopaysupportfees:

macOSinstallation–https://docs.docker.com/docker-for-mac/install/

Windowsinstallation–https://docs.docker.com/docker-for-windows/install/

Ubuntuinstallation–https://docs.docker.com/install/linux/docker-ce/ubuntu/

Instructionsareavailableforseveralotherdistros.SomeusefulpostinstallLinuxinstructionsareathttps://docs.docker.com/install/linux/linux-postinstall/

BecauseDockerrunsonLinux,itdoesnotrunnativelyonmacOSorWindows.InstallationoneitherOSrequiresinstallingLinuxinsideavirtualmachineandthenrunningDockertoolswithinthatvirtualLinuxmachine.Thedayswhenyouhadtohandcraftthatsetupyourselfarelonggone.TheDockerteamhasmadethiseasybydevelopingeasy-to-useDockerapplicationsforMacandWindows.TheDockerforWindowsandDockerforMacbundlespackagetheDockertoolsandlightweightvirtualmachinesoftware.Theresultisverylightweight,andtheDockercontainerscanbeleftrunninginthebackgroundwithlittleimpact.

YoumayfindreferencestoDockerToolboxasthemethodtoinstallDockeronmacOS.Thatapplicationislonggone,andhasbeenreplacedbyDockerforWindowsandDockerforMac.

StartingDockerwithDockerforWindows/macOSTostartDockerforWindowsorMacisverysimple.Yousimplyfindanddouble-clickontheapplicationicon.Itlaunchesaswouldanyothernativeapplication.Whenstarted,itmanagesavirtualmachine(notVirtualBox)withinwhichisaLinuxinstancerunningtheDockerEngine.OnmacOS,amenubariconshowsupwithwhichyoucontrolDocker.app,andonWindows,aniconisavailableinthesystemtray.

TherearesettingsavailablesothatDockerautomaticallylauncheseverytimeyoustartyourlaptop.

Onboth,theCPUmustsupportVirtualization.BundledinsideDockerforWindowsandDockerforMacisanultra-lightweighthypervisor,which,inturn,requiresvirtualizationsupportfromtheCPU.

ForWindows,thismayrequireBIOSconfiguration.Seehttps://docs.docker.com/docker-for-windows/troubleshoot/#virtualization-must-be-enabled.

ForMac,thisrequireshardwarefrom2010ornewer,withIntel’shardwaresupportformemorymanagementunit(MMU)virtualization,includingExtendedPageTables(EPT)andUnrestrictedMode.Youcancheckforthissupportbyrunningsysctlkern.hv_support.ItalsorequiresmacOS10.11orlater.

KickingthetiresofDockerWiththesetupaccomplished,wecanusethelocalDockerinstancetocreateDockercontainers,runafewcommands,and,ingeneral,learnhowtousethisamazingsystem.

Asinsomanysoftwarejourneys,thisonestartswithsayingHelloWorld:

$dockerrunhello-world

Unabletofindimage'hello-world:latest'locally

latest:Pullingfromlibrary/hello-world

ca4f61b1923c:Pullcomplete

Digest:

sha256:66ef312bbac49c39a89aa9bcc3cb4f3c9e7de3788c944158df3ee0176d32b751

Status:Downloadednewerimageforhello-world:latest

HellofromDocker!

Thismessageshowsthatyourinstallationappearstobeworkingcorrectly.

Togeneratethismessage,Dockertookthefollowingsteps:

1.TheDockerclientcontactedtheDockerdaemon.

2.TheDockerdaemonpulledthe"hello-world"imagefromtheDockerHub.

(amd64)

3.TheDockerdaemoncreatedanewcontainerfromthatimagewhichrunsthe

executablethatproducestheoutputyouarecurrentlyreading.

4.TheDockerdaemonstreamedthatoutputtotheDockerclient,whichsent

it

toyourterminal.

Totrysomethingmoreambitious,youcanrunanUbuntucontainerwith:

$dockerrun-itubuntubash

Shareimages,automateworkflows,andmorewithafreeDockerID:

https://cloud.docker.com/

Formoreexamplesandideas,visit:

https://docs.docker.com/engine/userguide/

ThedockerruncommanddownloadsaDockerimage,namedonthe

commandline,initializesaDockercontainerfromthatimage,andthenrunsthatcontainer.Inthiscase,theimage,namedhello-world,wasnotpresentonthelocalcomputerandhadtobedownloadedandinitialized.Oncethatwasdone,thehello-worldcontainerwasexecutedanditprintedouttheseinstructions.

Youcanqueryyourcomputertoseethatwhilethehello-worldcontainerhasexecutedandfinished,itstillexists:

ThedockerpscommandliststherunningDockercontainers.Asweseehere,thehello-worldcontainerisnolongerrunning,butwiththe-aswitch,dockerpsalsoshowsthosecontainersthatexistbutarenotcurrentlyrunning.WealsoseethatthiscomputerhasaNextcloudinstanceinstalledalongwithitsassociateddatabase.

Whenyou'redoneusingacontainer,youcancleanupwiththefollowingcommand:

$dockerrmboring_lumiere

boring_lumiere

Thenameboring_lumiereisthecontainernameautomaticallygeneratedbyDocker.Whiletheimagenamewashello-world,that'snotthecontainername.Dockergeneratedthecontainernamesoyouhaveamoreuser-friendlyidentifierforthecontainersthanthehexIDshowninthecontainerIDcolumn.Whencreatingacontainer,it'seasytospecifyanycontainernameyoulike.

CreatingtheAuthNetfortheuserauthenticationserviceWithallthattheoryspinningaroundourheads,it'stimetodosomethingpractical.Let'sstartbysettinguptheuserauthenticationservice.Inthediagramshownearlier,thiswillbetheboxlabeledAuthNetcontainingaMySQLinstanceandtheauthenticationserver.

MySQLcontainerforDockerTofindpubliclyavailableDockerimages,gotohttps://hub.docker.com/andsearch.You'llfindmanyDockerimagesreadytogo.Forexample,Nextcloud,anditsassociateddatabase,wasshownearlierinstalledalongsidethehello-worldapplicationwhenwekickedthetires.Bothareavailablefromtheirrespectiveprojectteamsandit'ssimply(moreorless)amatteroftypingdockerrunnextcloudtoinstallandrunthecontainers.TheprocessofinstallingNextcloud,anditsassociateddatabase,aswellasmanyotherpackagedapplications,suchasGitLab,isverysimilartowhatwe'reabouttodotobuildAuthNet,sotheskillsyou'reabouttolearnareverypractical.

JustforMySQL,thereareover11,000containersavailable.Fortunately,thetwocontainersprovidedbytheMySQLteamareverypopularandeasytouse.Themysql/mysql-serverimageisalittleeasiertoconfigure,solet'susethat.

ADockerimagenamecanbespecified,alongwithatagthatisusuallythesoftwareversionnumber.Inthiscase,we'llusemysql/mysql-server:5.7,wheremysql/mysql-serveristhecontainername,and5.7isthetag.MySQL5.7isthecurrentGArelease.Downloadtheimageasfollows:

$dockerpullmysql/mysql-server:5.7

5.7:Pullingfrommysql/mysql-server

4040fe120662:Pullcomplete

d049aa45d358:Pullcomplete

a6c7ed00840d:Pullcomplete

853789d8032e:Pullcomplete

Digest:

sha256:1b4c7c24df07fa89cdb7fe1c2eb94fbd2c7bd84ac14bd1779e3dec79f75f37c5

Status:Downloadednewerimageformysql/mysql-server:5.7

Thisdownloadedfourimagesintotal,becausethisimageisbuiltontopof

threeotherimages.We'llseelaterhowthatworkswhenwelearnhowtobuildaDockerfile.

Acontainercanbestartedusingthisimageasfollows:

$dockerrun--name=mysql--envMYSQL_ROOT_PASSWORD=f00barmysql/mysql-

server:5.7

[Entrypoint]MySQLDockerImage5.7.21-1.1.4

[Entrypoint]Initializingdatabase

[Entrypoint]Databaseinitialized

...

[Entrypoint]ignoringdocker-entrypoint-initdb.d*

[Entrypoint]Servershutdown

[Entrypoint]MySQLinitprocessdone.Readyforstartup.

[Entrypoint]StartingMySQL5.7.21-1.1.4

Westartedthisserviceintheforeground.Thecontainernameismysql.Wesetanenvironmentvariable,which,inturn(accordingtotheimagedocumentation),initializestherootpasswordasshown.Inanotherwindow,wecangetintothecontainerandruntheMySQLclientasfollows:

$dockerexec-itmysqlmysql-uroot-p

Enterpassword:

WelcometotheMySQLmonitor.Commandsendwith;or\g.

YourMySQLconnectionidis4

Serverversion:5.7.21MySQLCommunityServer(GPL)

Copyright(c)2000,2018,Oracleand/oritsaffiliates.Allrightsreserved.

OracleisaregisteredtrademarkofOracleCorporationand/orits

affiliates.Othernamesmaybetrademarksoftheirrespective

owners.

Type'help;'or'\h'forhelp.Type'\c'toclearthecurrentinput

statement.

mysql>showdatabases;

+--------------------+

|Database|

+--------------------+

|information_schema|

|mysql|

|performance_schema|

|sys|

+--------------------+

4rowsinset(0.00sec)

mysql>

Thedockerexeccommandletsyourunprogramsinsidethecontainer.The-itoptionsaysthecommandisruninteractively,onanassignedterminal.Substitutebashformysql,andyouhaveaninteractivebashcommandshell.

Thismysqlcommandinstanceisrunninginsidethecontainer.Thecontainerisconfiguredbydefaulttonotexposeanyexternalport,andithasadefaultmy.cnffile.

Thedatabasefilesarelockedinsidethecontainer.Assoonasthatcontainerisdeleted,thedatabasewillgoaway.Dockercontainersaremeanttobeephemeral,beingcreatedanddestroyedasneeded,whiledatabasesaremeanttobepermanent,withlifetimesmeasuredindecadessometimes.

Inotherwords,it'scoolthatwecaneasilyinstallandlaunchaMySQLinstance.Butthereareseveraldeficiencies:

Accesstothedatabasefromothersoftware

Storingthedatabasefilesoutsidethecontainerforalongerlifespan

Customconfiguration,becausedatabaseadminslovetotweakthesettings

ItneedstobeconnectedtoAuthNetalongwiththeuserauthenticationservice

Beforeproceeding,let'scleanup.InaTerminalwindow,type:

$dockerstopmysql

mysql

$dockerrmmysql

mysql

Thisclosesoutandcleansupthecontainers.And,toreiteratethepointmadeearlier,thedatabaseinthatcontainerwentaway.Ifthatdatabasecontainedcriticalinformation,youjustlostitwithnochancetorecoverthedata.

InitializingAuthNetDockersupportscreatingvirtualbridgenetworksbetweencontainers.RememberthataDockercontainerhasmanyofthefeaturesofaninstalledLinuxOS.EachcontainercanhaveitsownIPaddress(es)andexposedports.DockersupportscreatingwhatamountstobeingavirtualEthernetsegment,calledabridgenetwork.Thesenetworkslivesolelywithinthehostcomputerand,bydefault,arenotreachablebyanythingoutsidethehostcomputer.

ADockerbridgenetwork,therefore,hasstrictlylimitedaccess.AnyDockercontainerattachedtoabridgenetworkcancommunicatewithothercontainersattachedtothatnetwork.Thecontainersfindeachotherbyhostname,andDockerincludesanembeddedDNSservertosetupthehostnamesrequired.ThatDNSserverisconfiguredtonotrequiredotsindomainnames,meaningthattheDNS/hostnameofeachcontainerissimplythecontainername,ratherthansomethingsuchascontainer-name.service.ThispolicyofusinghostnamestoidentifycontainersisDocker'simplementationofservicediscovery.

Createadirectorynamedauthnetasasiblingtotheusersandnotesdirectories.We'llbeworkingonAuthNetinthatdirectory.

Createafile,buildauthnet.sh,containingthefollowing:

dockernetworkcreate--driverbridgeauthnet

Typethefollowing:

$sh-xbuildauthnet.sh

+dockernetworkcreate--driverbridgeauthnet

3021e2069278c2acb08d94a2d31507a43f089db1c02eecc97792414b498eb785

ThiscreatesaDockerbridgenetwork.

ScriptexecutiononWindowsExecutingscriptsonWindowsisdifferentbecauseitusesPowerShellratherthanbash,andalargenumberofotherconsiderations.Forthis,andthescriptswhichfollow,makethesechanges.

Powershellscriptfilenamesmustendwiththe.ps1extension.Formostofthesescripts,that'sallthatisrequiredbecausethescriptsaresosimple.Toexecutethescript,simplytype.\scriptname.ps1inthePowershellwindow.Inotherwords,onWindows,thescriptjustshownmustbenamedbuildauthnet.ps1,andisexecutedas.\buildauthnet.ps1.

Toexecutethescripts,youmayneedtochangethePowershellExecutionPolicy:

PSC:\Users\david\chap10\authnet>Get-ExecutionPolicy

Restricted

PSC:\Users\david\chap10\authnet>Set-ExecutionPolicyUnrestricted

Obviously,therearesecurityconsiderationswiththischange,sochangetheExecutionPolicybackwhenyou'redone.

AsimplermethodonWindowsistosimplypastethesecommandsintoaPowerShellwindow.

LinkingDockercontainersIntheolderdaysofDocker,weweretoldtolinkcontainersusingthe--linkoption.Withthatoption,Dockerwouldcreateentriesinetchostssothatonecontainercanrefertoanothercontainerbyitshostname.ThatoptionalsoarrangedaccesstoTCPportsandvolumesbetweenlinkedcontainers.Thisallowedthecreationofmulticontainerservices,usingprivateTCPportsforcommunicationthatexposednothingtoprocessesoutsidethecontainers.

Today,wearetoldthatthe--linkoptionisalegacyfeature,andthatinsteadweshouldusebridgenetworks.Inthischapter,we'llfocussolelyonusingbridgenetworks.

Youcanlistthenetworksasfollows:

$dockernetworkls

NETWORKIDNAMEDRIVERSCOPE

3021e2069278authnetbridgelocal

Lookatdetailsaboutthenetworkwiththiscommand:

$dockernetworkinspectauthnet

...muchJSONoutput

Atthemoment,thiswon'tshowanycontainersattachedtoauthnet.Theoutputshowsthenetworkname,theIPrangeofthisnetwork,thedefaultgateway,andotherusefulnetworkconfigurationinformation.Sincenothingisconnectedtothenetwork,let'sgetstartedwithbuildingtherequiredcontainers.

Thedb-userauthcontainerNowthatwehaveanetwork,wecanstartconnectingcontainerstothatnetwork.Andthenwe'llexplorethecontainerstoseehowprivatetheyare.

Createascript,startdb.sh,containing:

dockerrun--namedb-userauth--envMYSQL_RANDOM_ROOT_PASSWORD=true\

--envMYSQL_USER=userauth--envMYSQL_PASSWORD=userauth\

--envMYSQL_DATABASE=userauth\

--volume`pwd`/my.cnf:etcmy.cnf\

--volume`pwd`/../userauth-data:varlib/mysql\

--networkauthnetmysql/mysql-server:5.7

OnWindows,youwillneedtonamethescriptstartdb.ps1instead,andputthetextallononelineratherthanextendthelineswithbackslashes.And,thevolumemountedonvarlib/mysqlmustbecreatedseparately.Usethesecommandsinstead:

dockervolumecreatedb-userauth-volume

dockerrun--namedb-userauth--envMYSQL_RANDOM_ROOT_PASSWORD=true--env

MYSQL_USER=userauth--envMYSQL_PASSWORD=userauth--env

MYSQL_DATABASE=userauth--volume$PSScriptRoot\my.cnf:/etc/my.cnf--volume

db-userauth-volume:/var/lib/mysql--networkauthnetmysql/mysql-server:5.7

Whenrun,thecontainerwillbenameddb-userauth.Togivealittlebitofsecurity,therootpasswordhasbeenrandomized.We'veinsteaddefinedadatabasenameduserauth,accessedbyausernameduserauth,usingthepassworduserauth.That'snotexactlysecure,sofeelfreetochoosebetternamesandpasswords.Thecontainerisattachedtotheauthnetnetwork.

Therearetwo--volumeoptionsthatwemusttalkabout.InDockerese,avolumeisathinginsideacontainerthatcanbemountedfromoutside

thecontainer.Inthiscase,we'redefiningavolume,userauth-data,inthehostfilesystemtobemountedasvarlib/mysqlinsidethecontainer.And,we'redefiningalocalmy.cnffiletobeusedasetcmy.cnfinsidethecontainer.

FortheWindowsversion,wehavetwochangestothe--volumemounts.Wespecifythemountforetcmy.cnfas$PSScriptRoot\my.cnf:etcmy.cnf,becausethat'showyoureferencealocalfileinPowershell.

Forvarlib/mysql,wereferencedaseparatelycreatedvolume.Thevolumeiscreatedusingthevolumecreatecommand,andwiththatcommandthereisnoopportunitytocontrolthelocationofthevolume.It'simportantthatthevolumelivesoutsidethecontainer,sothatthedatabasefilessurvivethedestruction/creationcycleforthiscontainer.

Takentogether,thosesettingsmeanthedatabasefilesandtheconfigurationfileliveoutsidethecontainerandwillthereforeexistbeyondthelifetimeofonespecificcontainer.Togetthemy.cnf,youwillhavetorunthecontaineroncewithoutthe--volume`pwd`/my.cnf:etcmy.cnfoptionsoyoucancopythedefaultmy.cnffileintotheauthnetdirectory.

Runthescriptoncewithoutthatoption:

$shstartdb.sh

...muchoutput

[Entrypoint]GENERATEDROOTPASSWORD:UMyh@q]@j4qijyj@wK4s4SkePIkq

...muchoutput

Theoutputissimilartowhatwesawearlier,butforthisnewlinegivingtherandomizedpassword:

$dockernetworkinspectauthnet

Thiswilltellyouthedb-userauthcontainerisattachedtoauthnet:

$dockerexec-itdb-userauthmysql-uuserauth-p

Enterpassword:

WelcometotheMySQLmonitor.Commandsendwith;or\g.

...muchoutput

mysql>showdatabases;

+--------------------+

|Database|

+--------------------+

|information_schema|

|userauth|

+--------------------+

2rowsinset(0.00sec)

mysql>useuserauth;

Databasechanged

mysql>showtables;

Emptyset(0.00sec)

Weseeourdatabasehasbeencreatedandit'sempty.Butwedidthissowecouldgrabthemy.cnffile:

$dockercpdb-userauth:etcmy.cnf.

$ls

my.cnfmysql-datastartdb.sh

Thedockercpcommandisusedforcopyingfilesinandoutofcontainers.Ifyou'veusedscp,thesyntaxwillbefamiliar.

Onceyouhavethemy.cnffile,there'sabigpileofsettingchangesyoumightwanttomake.Thefirstspecificchangetomakeiscommentingoutthelinereadingsocket=varlib/mysql/mysql.sock,andthesecondisaddingalinereadingbind-address=0.0.0.0.ThepurposewiththesechangesistoconfiguretheMySQLservicetolistenonaTCPportratherthanaUnixdomainsocket.ThismakesitpossibletocommunicatewiththeMySQLservicefromoutsidethecontainer.Theresultwouldbe:

#socket=varlib/mysql/mysql.sock

bind-address=0.0.0.0

Nowstopthedb-userauthservice,andremovethecontainer,aswedidearlier.Editthestartdbscripttoenablethelinemountingetcmy.cnfintothecontainer,andthenrestartthecontainer:

$dockerstopdb-userauth

db-userauth

$dockerrmdb-userauth

db-userauth

$sh./startdb.sh

[Entrypoint]MySQLDockerImage5.7.21-1.1.4

[Entrypoint]StartingMySQL5.7.21-1.1.4

Now,ifweinspecttheauthnetnetwork,weseethefollowing:

$dockernetworkinspectauthnet

"Name":"authnet",

...

"Subnet":"172.18.0.0/16",

"Gateway":"172.18.0.1"

...

"Containers":{

"Name":"db-userauth",

"MacAddress":"02:42:ac:12:00:02",

"IPv4Address":"172.18.0.2/16",

...

Inotherwords,theauthnetnetworkhasthenetworknumber172.18.0.0/16,andthedb-userauthcontainerwasassigned172.18.0.2.Thislevelofdetailisrarelyimportant,butitisusefulonourfirsttimethroughtocarefullyexaminethesetupsoweunderstandwhatwe'redealingwith:

#catetcresolv.conf

searchattlocal.net

nameserver127.0.0.11

optionsndots:0

Aswesaidearlier,thereisaDNSserverrunningwithintheDockerbridgenetworksetup,anddomainnameresolutionisconfiguredtousenodots.That'ssoDockercontainernamesaretheDNShostnameforthecontainer:

#mysql-hdb-userauth-uuserauth-p

Enterpassword:

WelcometotheMySQLmonitor.Commandsendwith;or\g.

YourMySQLconnectionidis33

Serverversion:5.7.21MySQLCommunityServer(GPL)

AccesstheMySQLserverusingthecontainernameasthehostname.

DockerfilefortheauthenticationserviceIntheusersdirectory,createafilenamedDockerfilecontainingthefollowing:

FROMnode:10

ENVDEBUG="users:*"

ENVPORT="3333"

ENVSEQUELIZE_CONNECT="sequelize-docker-mysql.yaml"

ENVREST_LISTEN="0.0.0.0"

RUNmkdir-p/userauth

COPYpackage.jsonsequelize-docker-mysql.yaml.mjs.jsuserauth

WORKDIR/userauth

RUNapt-getupdate-y\

&&apt-get-yinstallcurlpythonbuild-essentialgitca-certificates\

&&npminstall--unsafe-perm

EXPOSE3333

CMDnpmrundocker

Dockerfilesdescribetheinstallationofanapplicationonaserver.Seehttps://docs.docker.com/engine/reference/builder/fordocumentation.TheydocumentassemblyofthebitsinaDockercontainerimage,andtheinstructionsinaDockerfileareusedtobuildaDockerimage.

TheFROMcommandspecifiesapre-existingimagefromwhichtoderiveagivenimage.Wetalkedaboutthisearlier;youcanbuildaDockercontainerstartingfromanexistingimage.TheofficialNode.jsDockerimage(https://hub.docker.com/_/node/)we'reusingisderivedfromdebian:jessie.Therefore,commandsavailablewithinthecontainerarewhatDebianoffers,andweuseapt-gettoinstallmorepackages.WeuseNode.js10becauseitsupportsES6modulesandtheotherfeatureswe've

beenusing.

TheENVcommandsdefineenvironmentvariables.Inthiscase,we'reusingthesameenvironmentvariablesdefinedwithintheuserauthenticationservice,exceptwehaveanewREST_LISTENvariable.We'lltakealookatthatshortly.

TheRUNcommandsarewhereweruntheshellcommandsrequiredtobuildthecontainer.Thefirstthingistomakea/userauthdirectorythatwillcontaintheservicesourcecode.TheCOPYcommandcopiesfilesintothatdirectory.Andthenwe'llneedtorunannpminstallsothatwecanruntheservice.ButfirstweusetheWORKDIRcommandtomovethecurrentworkingdirectoryinto/userauthsothatthenpminstallisruninthecorrectplace.WealsoinstalltherequisiteDebianpackagessothatanynativecodeNode.jspackagescanbeinstalled.

It'srecommendedthatyoualwayscombineapt-getupdatewithapt-getinstallinthesamecommandline,likethis,becauseoftheDockerbuildcache.Whenrebuildinganimage,Dockerstartswiththefirstchangedline.Byputtingthosetwotogether,youensurethatapt-getupdateisexecutedanytimeyouchangethelistofpackagestobeinstalled.Foracompletediscussion,seethedocumentationathttps://docs.docker.com/develop/develop-images/dockerfile_best-practices/.

Attheendofthiscommandisnpminstall--unsafe-perm.Theissuehereisthatthesecommandsarebeingrunasroot.Normally,whennpmisrunasroot,itchangesitsuserIDtoanonprivilegeduser.Thiscancausefailure,however,andthe--unsafe-permoptionpreventschangingtheuserID.

TheEXPOSEcommandinformsDockerthatthecontainerlistensonthenamedTCPport.Thisdoesnotexposetheportbeyondthecontainer.

Finally,theCMDcommanddocumentstheprocesstolaunchwhenthecontainerisexecuted.TheRUNcommandsareexecutedwhilebuildingthecontainer,whileCMDsayswhat'sexecutedwhenthecontainerstarts.

WecouldhaveinstalledPM2inthecontainer,thenusedaPM2commandtolaunchtheservice.ButDockerisabletofulfillthesamefunction,becauseitsupportsautomaticallyrestartingacontaineriftheserviceprocessdies.We'llseehowtodothislater.

ConfiguringtheauthenticationserviceforDockerWe'reusingadifferentfileforSEQUELIZE_CONNECT.Createanewfilenamedusers/sequelize-docker-mysql.yamlcontainingthefollowing:

dbname:userauth

username:userauth

password:userauth

params:

host:db-userauth

port:3306

dialect:mysql

Thedifferenceisthatinsteadoflocalhostasthedatabasehost,weusedb-userauth.Earlier,weexploredthedb-userauthcontaineranddeterminedthatwasthehostnameofthecontainer.Byusingdb-userauthinthisfile,theauthenticationservicewillusethedatabaseinthecontainer.

NowweneedtotakecareoftheenvironmentvariablenamedREST_LISTEN.Previously,theauthenticationserverhadlistenedonlytohttp://localhost:3333.We'ddonethisforsecuritypurposes,thatis,tolimitwhichprocessescouldconnecttotheservice.UnderDocker,weneedtoconnecttothisservicefromoutsideitscontainersothatothercontainerscanconnecttothisservice.Therefore,itmustlistentoconnectionsfromoutsidethelocalhost.Inusers-server.mjs,weneedtomakethefollowingchange:

server.listen(process.env.PORT,

process.env.REST_LISTEN?process.env.REST_LISTEN:"localhost",

()=>{log(server.name+'listeningat'+server.url);});

Thatis,iftheREST_LISTENvariableexists,theRESTserveristoldtolistento

whateveritsays,otherwisetheserviceistolistentolocalhost.WiththeenvironmentvariableintheDockerfile,theauthenticationservicewilllistentotheworld(0.0.0.0).Arewethrowingcautiontothewindandabrogatingourfiduciarydutyinkeepingthesacredtrustofstoringallthisuseridentificationinformation?No.Bepatient.We'lldescribemomentarilyhowtoconnectthisserviceanditsdatabasetoAuthNetandwillpreventaccesstoAuthNetbyanyotherprocess.

BuildingandrunningtheauthenticationserviceDockercontainerInusers/package.jsonaddthefollowinglinetothescriptssection:

"docker":"node--experimental-modules./user-server",

"docker-build":"dockerbuild-tnode-web-development/userauth."

Previously,we'veputtheconfigurationenvironmentvariablesintopackage.json.Inthiscase,theconfigurationenvironmentvariablesareintheDockerfile.ThismeansweneedawaytoruntheserverwithnoenvironmentvariablesotherthanthoseintheDockerfile.Withthisscriptsentry,wecandonpmrundockerandthentheDockerfileenvironmentvariableswillsupplyallconfiguration.

Wecanbuildtheauthenticationserviceasfollows:

$npmrundocker-build

>userauth-server@0.0.1docker-buildUsersdavid/chap10/users

>dockerbuild-tnode-web-development/userauth.

SendingbuildcontexttoDockerdaemon33.8MB

Step1/11:FROMnode:9.5

--->a696309517c6

Step2/11:ENVDEBUG="users:*"

--->Usingcache

--->f8cc103432e8

Step3/11:ENVPORT="3333"

--->Usingcache

--->39b24b8b554e

...moreoutput

ThedockerbuildcommandbuildsacontainerfromaDockerfile.Aswesaidearlier,theprocessbeginswiththeimagedefinedintheFROMcommand.Thenthebuildproceedsstepbystep,andtheoutputshowsliterallyeachstepasitisexecuted.

Thencreateascript,authnet/startserver.sh,oronWindowscallitstartserver.ps1,containingthefollowingcommand:

dockerrun-it--nameuserauth--net=authnetnode-web-development/userauth

Thislaunchesthenewlybuiltcontainer,givingitthenameuserauth,attachingittoauthnet:

$sh-xstartserver.sh

+dockerrun-it--nameuserauth--net=authnetnode-web-development/userauth

>userauth-server@0.0.1docker/userauth

>node--experimental-modules./user-server

(node:17)ExperimentalWarning:TheESMmoduleloaderisexperimental.

users:serviceUserAuth-Servicelisteningathttp://0.0.0.0:3333+0ms

Thatstartstheuserauthenticationservice.OnWindows,startitas.\startserver.ps1.Youshouldrecallthatit'saRESTservice,andthereforerunningitthroughitspacesisdonewithusers-add.jsandtheotherscripts.But,sincewedidnotexposeapublicportfromtheservicewemustrunthosescriptsfrominsidethecontainer.

Wedeterminewhetheracontainerexposesapublicportinoneoftwoways.Theeasiestisrunningdockerps-aandviewingthecontainerlistingdetails.ThereisacolumnmarkedPORTS,andforuserauthwesee3333/tcp.ThisisasideeffectoftheEXPOSEcommandintheDockerfile.Ifthatportwereexposed,itwouldappearinthePORTScolumnas0.0.0.0:3333->3333/tcp.Rememberthegoalfortheuserauthcontainer,andauthnetoverall,wasthatitwouldnotbepubliclyaccessiblebecauseofsecurityconcerns.

ExploringAuthnetLet'sexplorewhatwejustcreated:

$dockernetworkinspectauthnet

ThisprintsoutalargeJSONobjectdescribingthenetwork,anditsattachedcontainers,whichwe'velookedatbefore.Ifallwentwell,we'llseetherearenowtwocontainersattachedtoauthnetwherethere'dpreviouslybeenonlyone.

Let'sgointotheuserauthcontainerandpokearound:

$dockerexec-ituserauthbash

root@a29d833287bf:/userauth#ls

node_modulesuser-server.mjsusers-list.js

package-lock.jsonusers-add.jsusers-sequelize.mjs

package.jsonusers-delete.js

sequelize-docker-mysql.yamlusers-find.js

The/userauthdirectoryisinsidethecontainerandisexactlythefilesplacedinthecontainerusingtheCOPYcommand,plustheinstalledfilesinnode_modules:

root@a29d833287bf:/userauth#PORT=3333nodeusers-list.js

List[]

root@a29d833287bf:/userauth#PORT=3333nodeusers-add.js

Created{id:1,username:'me',password:'w0rd',provider:'local',

familyName:'Einarrsdottir',givenName:'Ashildr',

middleName:'',emails:'[]',photos:'[]',

updatedAt:'2018-02-05T01:54:53.320Z',createdAt:'2018-02-

05T01:54:53.320Z'}

root@a29d833287bf:/userauth#PORT=3333nodeusers-list.js

List[{id:'me',username:'me',provider:'local',

familyName:'Einarrsdottir',givenName:'Ashildr',middleName:'',

emails:'[]',photos:'[]'}]

Ourtestofaddingausertotheauthenticationserviceworks:

root@a29d833287bf:/userauth#ps-eafw

UIDPIDPPIDCSTIMETTYTIMECMD

root10001:52pts/000:00:00binsh-cnpmrundocker

root91001:52pts/000:00:00npm

root199001:52pts/000:00:00sh-cnode--experimental-

modules./user-server

root2019001:52pts/000:00:01node--experimental-modules

./user-server

root300001:54pts/100:00:00bash

root7030001:57pts/100:00:00ps-eafw

root@a29d833287bf:/userauth#pingdb-userauth

PINGdb-userauth(172.18.0.2):56databytes

64bytesfrom172.18.0.2:icmp_seq=0ttl=64time=0.105ms

64bytesfrom172.18.0.2:icmp_seq=1ttl=64time=0.077ms

^C---db-userauthpingstatistics---

2packetstransmitted,2packetsreceived,0%packetloss

round-tripmin/avg/max/stddev=0.077/0.091/0.105/0.000ms

root@a29d833287bf:/userauth#pinguserauth

PINGuserauth(172.18.0.3):56databytes

64bytesfrom172.18.0.3:icmp_seq=0ttl=64time=0.132ms

64bytesfrom172.18.0.3:icmp_seq=1ttl=64time=0.095ms

^C---userauthpingstatistics---

2packetstransmitted,2packetsreceived,0%packetloss

Theprocesslistingisinterestingtostudy.ProcessPID1isthenpmrundockercommandintheDockerfile.Processesproceedfromtheretothenodeprocessrunningtheactualserver.

Apingcommandprovesthetwocontainersareavailableashostnamesmatchingthecontainernames.

Then,youcanlogintothedb-userauthcontainerandinspectthedatabase:

$dockerexec-itdb-userauthbash

bash-4.2#mysql-uuserauth-p

Enterpassword:

WelcometotheMySQLmonitor.Commandsendwith;or\g.

...

mysql>useuserauth

Databasechanged

mysql>showtables;

+--------------------+

|Tables_in_userauth|

+--------------------+

|Users|

+--------------------+

1rowinset(0.00sec)

mysql>select*fromUsers;

+----+----------+----------+----------+---------------+-----------+--...

|id|username|password|provider|familyName|givenName|...

+----+----------+----------+----------+---------------+-----------+--...

|1|me|w0rd|local|Einarrsdottir|Ashildr|...

+----+----------+----------+----------+---------------+-----------+--...

1rowinset(0.00sec)

WehavesuccessfullyDockerizedtheuserauthenticationserviceintwocontainers,db-userauthanduserauth.We'vepokedaroundtheinsidesofarunningcontainerandfoundsomeinterestingthings.But,ourusersneedthefantasticNotesapplicationtoberunning,andwecan'taffordtorestonourlaurels.

CreatingFrontNetfortheNotesapplicationWehavethebackhalfofoursystemsetupinaDockercontainer,aswellastheprivatebridgenetworktoconnectthebackendcontainers.Wenowneedtosetupanotherprivatebridgenetwork,frontnet,andattachtheotherhalfofoursystemtothatnetwork.

Createadirectory,frontnet,whichiswherewe'lldevelopthetoolstobuildandrunthatnetwork.Inthatdirectory,createafile,buildfrontnet.sh,oronWindows,buildfrontnet.ps1,containing:dockernetworkcreate--driverbridgefrontnet

Let'sgoaheadandcreatethefrontnetbridgenetwork:

$sh-xbuildfrontnet.sh

+dockernetworkcreate--driverbridgefrontnet

f3df227d4bfff57bc7aed1e096a2ad16f6cebce4938315a54d9386a42d1ae3ed

$dockernetworkls

NETWORKIDNAMEDRIVERSCOPE

3021e2069278authnetbridgelocal

f3df227d4bfffrontnetbridgelocal

We'llproceedfromheresimilarlytohowauthnetwascreated.However,wecanworkmorequicklybecausewe'vealreadygoneoverthebasics.

MySQLcontainerfortheNotesapplicationFromtheauthnetdirectory,copythemy.cnfandstartdb.shfilesintothefrontnetdirectory.Themy.cnffilecanprobablybeusedunmodified,butwehaveafewchangestomaketothestartdb.shfile:

dockerrun--namedb-notes--envMYSQL_RANDOM_ROOT_PASSWORD=true\

--envMYSQL_USER=notes--envMYSQL_PASSWORD=notes12345\

--envMYSQL_DATABASE=notes\

--volume`pwd`/my.cnf:/etc/my.cnf\

--volume`pwd`/../notes-data:/var/lib/mysql\

--networkfrontnetmysql/mysql-server:5.7

OnWindows,namethefilestartdb.ps1containingthis:

dockervolumecreatenotes-data-volume

dockerrun--namedb-notes--envMYSQL_RANDOM_ROOT_PASSWORD=true--env

MYSQL_USER=notes--envMYSQL_PASSWORD=notes12345--envMYSQL_DATABASE=notes-

-volume$PSScriptRoot\my.cnf:/etc/my.cnf--volumenotes-data-

volume:/var/lib/mysql--networkfrontnetmysql/mysql-server:5.7

Thechangesaresimplesubstitutionstotransliteratefromuserauthtonotes.Andthenrunit:

$mkdir../notes-data

$sh-xstartdb.sh

+pwd

+pwd

+dockerrun--namedb-notes--envMYSQL_RANDOM_ROOT_PASSWORD=true--env

MYSQL_USER=notes--envMYSQL_PASSWORD=notes12345--envMYSQL_DATABASE=notes-

-volumehomedavid/nodewebdev/nodeweb-development-code-4th-

edition/chap10/frontnet/my.cnf:/etc/my.cnf--volume

homedavid/nodewebdev/nodeweb-development-code-4th-

edition/chap10/frontnet/../notes-data:/var/lib/mysql--networkfrontnet

mysql/mysql-server:5.7

[Entrypoint]MySQLDockerImage5.7.21-1.1.4

[Entrypoint]Initializingdatabase

[Entrypoint]Databaseinitialized

[Entrypoint]GENERATEDROOTPASSWORD:3kZ@q4hBItYGYj3Mes!AdiP83Nol

[Entrypoint]ignoringdocker-entrypoint-initdb.d*

[Entrypoint]Servershutdown

[Entrypoint]MySQLinitprocessdone.Readyforstartup.

[Entrypoint]StartingMySQL5.7.21-1.1.4

ForWindows,simplyrun.\startdb.ps1.

Thisdatabasewillbeavailableatthedb-notesdomainnameonfrontnet.Becauseit'sattachedtofrontnet,itwon'tbereachablebycontainersconnectedtoauthnet.

$dockerexec-ituserauthbash

root@0a2009334b79:/userauth#pingdb-notes

ping:unknownhost

Sincedb-notesisonadifferentnetworksegment,we'veachievedseparation.

DockerizingtheNotesapplicationInthenotesdirectory,createafilenamedDockerfilecontainingthefollowing:

FROMnode:10

ENVDEBUG="notes:*,messages:*"

ENVSEQUELIZE_CONNECT="models/sequelize-docker-mysql.yaml"

ENVNOTES_MODEL="sequelize"

ENVUSER_SERVICE_URL="http://userauth:3333"

ENVPORT="3000"

ENVNOTES_SESSIONS_DIR="/sessions"

#ENVTWITTER_CONSUMER_KEY="..."

#ENVTWITTER_CONSUMER_SECRET="..."

#UsethislinewhentheTwitterCallbackURL

#hastobeotherthanlocalhost:3000

#ENVTWITTER_CALLBACK_HOST=http://45.55.37.74:3000

RUNmkdir-pnotesappnotesapp/mintynotesapppartialsnotesapppublic

notesapproutesnotesappthemenotesappviews

COPYminty/notesappminty/

COPYmodels/*.mjsmodels/sequelize-docker-mysql.yamlnotesappmodels/

COPYpartials/notesapppartials/

COPYpublic/notesapppublic/

COPYroutes/notesapproutes/

COPYtheme/notesapptheme/

COPYviews/notesappviews/

COPYapp.mjspackage.jsonnotesapp

WORKDIR/notesapp

RUNapt-getupdate-y\

&&apt-get-yinstallcurlpythonbuild-essentialgitca-certificates\

&&npminstall--unsafe-perm

#Uncommenttobuildthethemedirectory

#WORKDIRnotesapptheme

#RUNnpmrundownload&&npmrunbuild&&npmrunclean

WORKDIR/notesapp

VOLUME/sessions

EXPOSE3000

CMDnode--experimental-modules./app

ThisissimilartotheDockerfileweusedfortheauthenticationservice.We'reusingtheenvironmentvariablesfromnotes/package.json,plusanewone,andthere'sacoupleofnewtricksinvolvedhere,solet'stakealook.

ThemostobviouschangeisthenumberofCOPYcommands.TheNotesapplicationisalotmoreinvolvedgiventhenumberofsubdirectoriesfulloffilesthatmustbeinstalled.Westartbycreatingthetop-leveldirectoriesoftheNotesapplicationdeploymenttree.Then,onebyone,wecopyeachsubdirectoryintoitscorrespondingsubdirectoryinthecontainerfilesystem.

InaCOPYcommand,thetrailingslashonthedestinationdirectoryisimportant.Why?Becausethedocumentationsaysthatthetrailingslashisimportant.

Thebigquestionis:WhyusemultipleCOPYcommandssuchasthis?Thiswouldhavebeentriviallysimple:

COPY./notesapp

But,itisimportanttoavoidcopyingthenode_modulesdirectoryintothecontainer.Thecontainernode_modulesmustbebuiltinsidethecontainer,becausethecontaineroperatingsystemisalmostcertainlydifferenttothehostoperatingsystem.Anynativecodemodulesmustbebuiltforthecorrectoperatingsystem.Thatconstraintledtothequestionofconciselycopyingspecificfilestothedestination.

We'vedevelopedaprocesstobuildaBootstrap4theme,whichwedevelopedinChapter6,ImplementingtheMobile-FirstParadigm.IfyouhaveaBootstrap4themetobuild,simplyuncommentthecorrespondinglinesintheDockerfile.Thoselinesmovetheworkingdirectoryto

notesappthemeandthenrunthescriptstobuildthetheme.Anewscriptisrequiredintheme/package.jsontoremovethetheme/node_modulesdirectoryafterthethemehasbeenbuilt:

"scripts":{

...

"clean":"rm-rfbootstrap-4.0.0/node_modules"

...

}

WealsohaveanewSEQUELIZE_CONNECTfile.Createmodels/sequelize-docker-mysql.yamlcontainingthefollowing:

dbname:notes

username:notes

password:notes12345

params:

host:db-notes

port:3306

dialect:mysql

Thiswillaccessadatabaseserveronthedb-notesdomainnameusingthenameddatabase,username,andpassword.

NoticethattheUSER_SERVICE_URLvariablenolongeraccessestheauthenticationserviceatlocalhost,butatuserauth.TheuserauthdomainnameiscurrentlyonlyadvertisedbytheDNSserveronAuthNet,buttheNotesserviceisonFrontNet.Thismeanswe'llhavetoconnecttheuserauthcontainertotheFrontNetbridgenetworksothatitsnameisknownthereaswell.We'llgettothatinaminute.

InChapter8,wediscussedtheneedtoprotecttheAPIkeyssuppliedbyTwitter.

Wedidn'twanttocommitthekeysinthesourcecode,buttheyhavetogosomewhere.PlaceholdersareintheDockerfileforspecifyingTWITTER_CONSUMER_KEYandTWITTER_CONSUMER_SECRET.

ThevalueforTWITTER_CALLBACK_HOSTneedstoreflectwhereNotesisdeployed.Rightnow,itisstillonyourlaptop,butbytheendofthechapter,itwillbedeployedtotheserver,and,atthattime,itwillneedtheIPaddressordomainnameoftheserver.

AnewvariableisNOTES_SESSIONS_DIRandthematchingVOLUMEdeclaration.IfweweretorunmultipleNotesinstances,theycouldsharesessiondatabysharingthisvolume.

SupportingtheNOTES_SESSIONS_DIRvariablerequiresonechangeinapp.mjs:

constsessionStore=newFileStore({

path:process.env.NOTES_SESSIONS_DIR?

process.env.NOTES_SESSIONS_DIR:"sessions"

});

Insteadofahardcodeddirectoryname,wecanuseanenvironmentvariabletodefinethelocationwheresessiondataisstored.Alternatively,therearesessionStoreimplementationsforvariousserverssuchasREDIS,enablingsessiondatasharingbetweencontainersonseparatehostsystems.

Innotes/package.json,addthesescripts:

"scripts":{

...

"docker":"node--experimental-modules./app",

"docker-build":"dockerbuild-tnode-web-development/notes."

...

}

Asfortheauthenticationserver,thisletsusbuildthecontainerandthen,withinthecontainer,wecanruntheservice.

Nowwecanbuildthecontainerimage:

$npmrundocker-build

>notes@0.0.0docker-buildUsersdavid/chap10/notes

>dockerbuild-tnode-web-development/notes.

SendingbuildcontexttoDockerdaemon76.27MB

Step1/22:FROMnode:9.5

--->a696309517c6

Step2/22:ENVDEBUG="notes:*,messages:*"

--->Usingcache

--->8628ecad9fa4

Next,inthefrontnetdirectory,createafilenamedstartserver.sh,or,onWindows,startserver.ps1:

dockerrun-it--namenotes--net=frontnet-p3000:3000node-web-

development/notes

Unliketheauthenticationservice,theNotesapplicationcontainermustexportaporttothepublic.Otherwise,thepublicwillneverbeabletoenjoythiswonderfulcreationwe'rebuilding.The-poptionishowweinstructDockertoexposeaport.

ThefirstnumberisaTCPportnumberpublishedfromthecontainer,andthesecondnumberistheTCPportinsidethecontainer.Generallyspeaking,thisoptionmapsaportinsidethecontainertoonereachablebythepublic.

Thenrunitasfollows:

$sh-xstartserver.sh

+dockerrun-it--namenotes--net=frontnet-p3000:3000node-web-

development/notes

(node:6)ExperimentalWarning:TheESMmoduleloaderisexperimental.

notes:debug-INDEXListeningonport3000+0ms

OnWindows,run.\startserver.ps1.

Atthispoint,wecanconnectourbrowsertohttp://localhost:3000andstartusingtheNotesapplication.Butwe'llquicklyrunintoaproblem:

Theuserexperienceteamisgoingtoscreamaboutthisuglyerrormessage,soputitonyourbacklogtogenerateaprettiererrorscreen.Forexample,aflockofbirdspullingawhaleoutoftheoceanispopular.

ThiserrormeansthatNotescannotaccessanythingatthehostnameduserauth.Thathostdoesexist,becausethecontainerisrunning,butit'snotonfrontnet,andisnotreachablefromthenotescontainer.Namely:

$dockerexec-itnotesbash

root@125a196c3fd5:/notesapp#pinguserauth

ping:unknownhost

root@125a196c3fd5:/notesapp#pingdb-notes

PINGdb-notes(172.19.0.2):56databytes

64bytesfrom172.19.0.2:icmp_seq=0ttl=64time=0.136ms

^C---db-notespingstatistics---

1packetstransmitted,1packetsreceived,0%packetloss

round-tripmin/avg/max/stddev=0.136/0.136/0.136/0.000ms

root@125a196c3fd5:/notesapp#

IfyouinspectFrontNetandAuthNet,you'llseethecontainersattachedtoeachdonotoverlap:

$dockernetworkinspectfrontnet

$dockernetworkinspectauthnet

Inthearchitecturediagramatthebeginningofthechapter,weshowedaconnectionbetweenthenotesanduserauthcontainers.Theconnectionisrequiredsonotescanauthenticateitsusers.Butthatconnectiondoesnot

exist,yet.

Unfortunately,asimplechangetostartserver.sh(startserver.ps1)doesnotwork:

dockerrun-it--namenotes--net=authnet--net=frontnet-p3000:3000node-

web-development/notes

Whileitisconceptuallysimpletospecifymultiple--netoptionswhenstartingacontainer,Dockerdoesnotsupportthis.Itsilentlyacceptsthecommandasshown,butonlyconnectsthecontainertothelastnetworkmentionedintheoptions.Instead,Dockerrequiresthatyoutakeasecondsteptoattachthecontainertoasecondnetwork:

$dockernetworkconnectauthnetnotes

Withnootherchange,theNotesapplicationwillnowallowyoutologinandstartaddingandeditingnotes.

Thereisaglaringarchitecturequestionstaringatus.Doweconnecttheuserauthservicetofrontnet,ordoweconnectthenotesservicetoauthnet?Toverifythateitherdirectionsolvestheproblem,runthesecommands:

$dockernetworkdisconnectauthnetnotes

$dockernetworkconnectfrontnetuserauth

Thefirsttimearound,weconnectednotestoauthnet,thenwedisconnecteditfromauthnet,andthenconnecteduserauthtofrontnet.Thatmeanswetriedbothcombinationsand,asexpected,inbothcasesnotesanduserauthwereabletocommunicate.

Thisisaquestionforsecurityexpertssincetheconsiderationistheattackvectorsavailabletoanyintruders.SupposeNoteshasasecurityholeallowinganinvadertogainaccess.Howdowelimitwhatisreachablevia

thathole?

Theprimaryobservationisthatbyconnectingnotestoauthnet,notesnotonlyhasaccesstouserauth,butalsotodb-userauth:

$dockernetworkdisconnectfrontnetuserauth

$dockernetworkconnectauthnetnotes

$dockerexec-itnotesbash

root@7fce818e9a4d:/notesapp#pinguserauth

PINGuserauth(172.18.0.3):56databytes

64bytesfrom172.18.0.3:icmp_seq=0ttl=64time=0.103ms

^C---userauthpingstatistics---

1packetstransmitted,1packetsreceived,0%packetloss

round-tripmin/avg/max/stddev=0.103/0.103/0.103/0.000ms

root@7fce818e9a4d:/notesapp#pingdb-userauth

PINGdb-userauth(172.18.0.2):56databytes

64bytesfrom172.18.0.2:icmp_seq=0ttl=64time=0.201ms

^C---db-userauthpingstatistics---

1packetstransmitted,1packetsreceived,0%packetloss

round-tripmin/avg/max/stddev=0.201/0.201/0.201/0.000ms

root@7fce818e9a4d:/notesapp#

Thissequencereconnectsnotestoauthnet,anddemonstratestheabilitytoaccessboththeuserauthanddb-userauthcontainers.Therefore,asuccessfulinvadercouldaccessthedb-userauthdatabase,aresultwewantedtoprevent.Ourdiagramatthebeginningshowednosuchconnectionbetweennotesanddb-userauth.

GiventhatourgoalforusingDockerwastolimittheattackvectors,wehaveacleardistinctionbetweenthetwocontainer/networkconnectionsetups.Attachinguserauthtofrontnetlimitsthenumberofcontainersthatcanaccessdb-userauth.Foranintrudertoaccesstheuserinformationdatabase,theymustfirstbreakintonotes,andthenbreakintouserauth.Unless,thatis,ouramateurattemptatasecurityauditisflawed.

ControllingthelocationofMySQLdatavolumesThedb-userauthanddb-notesDockerfilescontainVOLUMEvarlib/mysql,andwhenwestartedthecontainers,wegave--volumeoptions,assigningahostdirectoryforthatcontainerdirectory:

dockerrun--namedb-notes\

...

--volume`pwd`/../notes-data:varlib/mysql\

...

Wecaneasilyseethisconnectsahostdirectory,soitappearswithinthecontaineratthatlocation.SimplyinspectingthehostdirectorywithtoolssuchaslsshowsthatfilesarecreatedinthatdirectorycorrespondingtoaMySQLdatabase.

TheVOLUMEinstructioninstructsDockertocreateadirectoryoutsidethecontainerandtomapthatdirectorysothatit'smountedinsidethecontaineronthenamedpath.TheVOLUMEinstructionbyitselfdoesn'tcontrolthedirectorynameonthehostcomputer.Ifno--volumeoptionisgiven,Dockerstillarrangesforthecontentofsaiddirectorytobekeptoutsidethecontainer.That'suseful,andatleastthedataisavailableoutsidethecontainer,butyouhaven'tcontrolledthelocation.

Ifwerestartthedb-notescontainerwithoutusingthe--volumeoptionforvarlib/mysql,wecaninspectthecontainertodiscoverwhereDockerputthevolume:

$dockerinspect--format'{{json.Mounts}}'db-notes

[{"Type":"bind",

"Source":"Usersdavid/chap10/frontnet/my.cnf","Destination":"etcmy.cnf",

"Mode":"","RW":true,"Propagation":"rprivate"},

{"Type":"volume","Name":"39f9a80b49e3ecdebc7789de7b7dd2366c400ee7fbfedd6e4df1

8f7e60bad409",

"Source":"varlib/docker/volumes/39f9a80b49e3ecdebc7789de7b7dd2366c400ee7fbfed

d6e4df18f7e60bad409/_data","Destination":"varlib/mysql",

"Driver":"local","Mode":"","RW":true,"Propagation":""}]

That'snotexactlyauser-friendlypathname,butyoucansnoopintothatdirectoryandseethatindeedtheMySQLdatabaseisstoredthere.Thesimplestwaytouseauser-friendlypathnameforavolumeiswiththe--volumeoptionsweshowedearlier.

Anotheradvantagewehaveistoeasilyswitchdatabases.Forexample,wecouldtestNoteswithpre-cookedtestdatabasesfullofnoteswritteninSwahili(notes-data-swahili),Romanian(notes-data-romanian),German(notes-data-german)andEnglish(notes-data-english).Eachtestdatabasecouldbestoredinthenameddirectory,andtestingagainstthespecificlanguageisassimpleasrunningthenotescontainerwithdifferent--volumeoptions.

Inanycase,ifyourestartthenotescontainerwiththe--volumeoption,youcaninspectthecontainerandseethedirectoryismountedonthedirectoryyouspecified:

$dockerinspect--format'{{json.Mounts}}'db-notes

[{"Type":"bind",

"Source":"Usersdavid/chap10/frontnet/my.cnf","Destination":"etcmy.cnf",

"Mode":"","RW":true,"Propagation":"rprivate"},

{"Type":"bind",

"Source":"Usersdavid/chap10/notes-data","Destination":"varlib/mysql",

"Mode":"","RW":true,"Propagation":"rprivate"}]

Withthe--volumeoptions,wehavecontrolledthelocationofthehostdirectorycorrespondingtothecontainerdirectory.

Thelastthingtonoteisthatcontrollingthelocationofsuchdirectoriesmakesiteasiertomakebackupsandtakeotheradministrativeactionswiththatdata.

DockerdeploymentofbackgroundservicesWiththescriptswe'vewrittensofar,theDockercontainerisrunintheforeground.Thatmakesiteasiertodebugtheservicesinceyouseetheerrors.Foraproductiondeployment,weneedtheDockercontainerdetachedfromtheterminal,andanassurancethatitwillrestartitselfautomatically.Thosetwoattributesaresimpletoimplement.

Simplychangethispattern:$dockerrun-it...

Tothispattern:

$dockerrun--detach--restartalways...

The-itoptioniswhatcausestheDockercontainertorunintheforeground.UsingtheseoptionscausestheDockercontainertorundetachedfromyourterminal,andiftheserviceprocessdies,thecontainerwillautomaticallyrestart.

DeployingtothecloudwithDockercomposeThisiscoolthatwecancreateencapsulatedinstantiationsofthesoftwareserviceswe'vecreated.ButthepromisewastousetheDockerizedapplicationfordeploymentoncloudservices.Inotherwords,weneedtotakeallthislearningandapplyittothetaskofdeployingNotesonapublicinternetserverwithafairlyhighdegreeofsecurity.

We'vedemonstratedthat,withDocker,Notescanbedecomposedintofourcontainersthathaveahighdegreeofisolationfromeachother,andfromtheoutsideworld.

Thereisanotherglaringproblem:ourprocessintheprevioussectionwaspartlymanual,partlyautomated.Wecreatedscriptstolauncheachportionofthesystem,whichisagoodpracticeaccordingtotheTwelveFactorApplicationmodel.ButwedidnotautomatetheentireprocesstobringupNotesandtheauthenticationservices.Noristhissolutionscalablebeyondonemachine.

Let'sstartwiththelastissuefirst—scalability.WithintheDockerecosystem,severalDockerorchestratorservicesareavailable.AnOrchestratorautomaticallydeploysandmanagesDockercontainersoveragroupofmachines.SomeexamplesofDockerOrchestratorsareDockerSwarm(whichisbuiltintotheDockerCLI),Kubernetes,CoreOSFleet,andApacheMesos.Thesearepowerfulsystemsabletoautomaticallyincrease/decreaseresourcesasneeded,tomovecontainersfromonehosttoanother,andmore.Wementionthesesystemsforyourfurtherstudyasyourneedsgrow.

Dockercompose(https://docs.docker.com/compose/overview/)willsolvetheotherproblemswe'veidentified.Itletsuseasilydefineandrunseveral

Dockercontainerstogetherasacompleteapplication.ItusesaYAMLfile,docker-compose.yml,todescribethecontainers,theirdependencies,thevirtualnetworks,andthevolumes.Whilewe'llbeusingittodescribethedeploymentontoasinglehostmachine,Dockercomposecanbeusedformultimachinedeployments,especiallywhencombinedwithDockerSwarm.UnderstandingDockercomposewillprovideabasisuponwhichtounderstand/usetheothertools,suchasSwarmorKubernetes.

Dockermachine(https://docs.docker.com/machine/overview/)isatoolforinstallingDockerEngineonvirtualhosts,eitherlocalorremote,andformanagingDockercontainersonthosehosts.We'llbeusingthistoprovisionaserveronacloudhostingservice,andpushcontainersintothatserver.ItcanalsobeusedtoprovisionavirtualhostonyourlaptopwithinaVirtualBoxinstance.

Beforeproceeding,ensureDockercomposeandDockermachineareinstalled.Ifyou'veinstalledDockerforWindowsorDockerforMac,bothareinstalledalongwitheverythingelse.OnLinux,youmustinstallbothseparatelybyfollowingtheinstructionsatthelinksgivenearlier.

DockercomposefilesLet'sstartbycreatingadirectory,compose,asasiblingtotheusersandnotesdirectories.Inthatdirectory,createafilenameddocker-compose.yml:

version:'3'

services:

db-userauth:

image:"mysql/mysql-server:5.7"

container_name:db-userauth

command:["mysqld","--character-set-server=utf8mb4",

"--collation-server=utf8mb4_unicode_ci",

"--bind-address=0.0.0.0"]

expose:

-"3306"

networks:

-authnet

volumes:

-db-userauth-data:/var/lib/mysql

-../authnet/my.cnf:/etc/my.cnf

environment:

MYSQL_RANDOM_ROOT_PASSWORD:"true"

MYSQL_USER:userauth

MYSQL_PASSWORD:userauth

MYSQL_DATABASE:userauth

restart:always

userauth:

build:../users

container_name:userauth

depends_on:

-db-userauth

networks:

-authnet

-frontnet

restart:always

db-notes:

image:"mysql/mysql-server:5.7"

container_name:db-notes

command:["mysqld","--character-set-server=utf8mb4",

"--collation-server=utf8mb4_unicode_ci",

"--bind-address=0.0.0.0"]

expose:

-"3306"

networks:

-frontnet

volumes:

-db-notes-data:/var/lib/mysql

-../frontnet/my.cnf:/etc/my.cnf

environment:

MYSQL_RANDOM_ROOT_PASSWORD:"true"

MYSQL_USER:notes

MYSQL_PASSWORD:notes12345

MYSQL_DATABASE:notes

restart:always

notes:

build:../notes

container_name:notes

restart:always

depends_on:

-db-notes

networks:

-frontnet

ports:

-"3000:3000"

restart:always

networks:

frontnet:

driver:bridge

authnet:

driver:bridge

volumes:

db-userauth-data:

db-notes-data:

That'sthedescriptionoftheentireNotesdeployment.It'satafairlyhighlevelofabstraction,roughlyequivalenttotheoptionsonthecommand-linetoolswe'veusedsofar.FurtherdetailsarelocatedinsidetheDockerfiles,whicharereferencedfromthiscomposefile.

Theversionlinesaysthatthisisaversion3composefile.Theversionnumberisinspectedbythedocker-composecommand,soitcancorrectlyinterpretitscontent.Thefulldocumentationisworthreadingathttps://doc

s.docker.com/compose/compose-file/.

Therearethreemajorsectionsusedhere:services,volumes,andnetworks.Theservicessectiondescribesthecontainersbeingused,thenetworkssectiondescribesthenetworks,andthevolumessectiondescribesthevolumes.Thecontentofeachsectionmatchestheintent/purposeofthecommandsweranearlier.Theinformationwe'vealreadydealtwithisallhere,justrearranged.

Therearetwodatabasecontainers,db-userauthanddb-notes.BothreferencetheDockerhubimageusingtheimagetag.Forthedatabases,wedidnotcreateaDockerfile,butinsteadbuiltdirectlyfromtheDockerhubimage.Thesamehappenshereinthecomposefile.

Fortheuserauthandnotescontainers,wecreatedaDockerfile.Thedirectorycontainingthatfileisreferencedbythebuildtag.Tobuildthecontainer,docker-composelooksforafilenamedDockerfileinthenameddirectory.Therearemoreoptionsforthebuildtag,whicharediscussedintheofficialdocumentation.

Thecontainer_nameattributeisequivalenttothe--nameattributeandspecifiesauser-friendlynameforthecontainer.WemustspecifythecontainernameinordertospecifythecontainerhostnameinordertodoDocker-styleservicediscovery.

ThecommandtagoverridestheCMDtagintheDockerfile.We'vespecifiedthisforthetwodatabasecontainers,sowecaninstructMySQLtobindtoIPaddress0.0.0.0.Eventhoughwedidn'tcreateaDockerfileforthedatabasecontainers,thereisaDockerfilecreatedbytheMySQLmaintainers.

Thenetworksattributeliststhenetworkstowhichthiscontainermustbeconnectedandisexactlyequivalenttothe--netargument.Eventhoughthedockercommanddoesn'tsupportmultiple--netoptions,wecanlistmultiplenetworksinthecomposefile.Inthiscase,thenetworksarebridgenetworks.Aswedidearlier,thenetworksthemselvesmustbecreatedseparately,andinacomposefile,that'sdoneinthenetworkssection.

Eachofthenetworksinoursystemisabridgenetwork.Thisfactisdescribedinthecomposefile.

Theexposeattributedeclareswhichportsareexposedfromthecontainer,andisequivalenttotheEXPOSEtag.Theexposedportsarenotpublishedoutsidethehostmachine,however.Theportsattributedeclarestheportsthataretobepublished.Intheportsdeclaration,wehavetwoportnumbers:thefirstbeingthepublishedportnumberandthesecondbeingtheportnumberinsidethecontainer.Thisisexactlyequivalenttothe-poptionusedearlier.

Thenotescontainerhasafewenvironmentvariables,suchasTWITTER_CONSUMER_KEYandTWITTER_CONSUMER_SECRET,thatyoumayprefertostoreinthisfileratherthanintheDockerfile.

Thedepends_onattributeletsuscontrolthestartuporder.Acontainerthatdependsonanotherwillwaittostartuntilthedepended-uponcontainerisrunning.

Thevolumesattributedescribesmappingsofacontainerdirectorytoahostdirectory.Inthiscase,we'vedefinedtwovolumenames,db-userauth-dataanddb-notes-data,andthenusedthemforthevolumemapping.Toexplorethevolumes,startwiththiscommand:

$dockervolumels

DRIVERVOLUMENAME

...

localcompose_db-notes-data

localcompose_db-userauth-data

...

Thevolumenamesarethesameasinthecomposefile,butwithcompose_tackedonthefront.

Youcaninspectthevolumelocationusingthedockercommandline:

$dockervolumeinspectcompose_db-notes-data

$dockervolumeinspectcompose_db-userauth-data

Ifit'spreferable,youcanspecifyapathnameinthecomposefile:

db-auth:

..

volumes:

#-db-userauth-data:/var/lib/mysql

-../userauth-data:/var/lib/mysql

db-notes:

..

volumes:

#-db-notes-data:/var/lib/mysql

-../notes-data:/var/lib/mysql

Thisisthesameconfigurationwemadeearlier.Itusestheuserauth-dataandnotes-datadirectoriesfortheMySQLdatafilesfortheirrespectivedatabasecontainers.

Theenvironmenttagdescribestheenvironmentvariablesthatwillbereceivedbythecontainer.Asbefore,environmentvariablesshouldbeusedtoinjectconfigurationdata.

Therestartattributecontrolswhathappensif,orwhen,thecontainerdies.Whenacontainerstarts,itrunstheprogramnamedintheCMDinstruction,andwhenthatprogramexits,thecontainerexits.Butwhatifthatprogramismeanttorunforever,shouldn'tDockerknowitshouldrestarttheprocess?Wecoulduseabackgroundprocesssupervisor,suchasSupervisordorPM2.But,wecanalsousetheDockerrestartoption.

Therestartattributecantakeoneofthefollowingfourvalues:

no–donotrestart

on-failure:count–restartuptoNtimes

always–alwaysrestart

unless-stopped–startthecontainerunlessitwasexplicitlystopped

RunningtheNotesapplicationwithDockercomposeOnWindows,we'reabletorunthecommandsinthissectionunchanged.

Beforedeployingthistoaserver,let'srunitonourlaptopusingdocker-compose:

$dockerstopdb-notesuserauthdb-authnotesapp

db-notes

userauth

db-auth

notesapp

$dockerrmdb-notesuserauthdb-authnotesapp

db-notes

userauth

db-auth

notesapp

Wefirstneededtostopanddeletetheexistingcontainers.Becausethecomposefilewantstolaunchcontainerswiththesamenamesaswe'dbuiltearlier,wealsohavetoremovetheexistingcontainers:

$docker-composebuild

Buildingdb-auth

..lotsofoutput

$docker-composeup

Creatingdb-auth

Recreatingcompose_db-notes_1

Recreatingcompose_userauth_1

Recreatingcompose_notesapp_1

Attachingtodb-auth,db-notes,userauth,notesapp

Oncethat'sdone,wecanbuildthecontainers,docker-composebuild,andthenstartthemrunning,docker-composeup.

Thefirsttestistoexecuteashellinuserauthtorunouruserdatabasescript:

$dockerexec-ituserauthbash

root@9972adbbdbb3:/userauth#PORT=3333nodeusers-add.js

Created{id:2,

username:'me',password:'w0rd',provider:'local',

familyName:'Einarrsdottir',givenName:'Ashildr',middleName:'',

emails:'[]',photos:'[]',

updatedAt:'2018-02-07T02:24:04.257Z',createdAt:'2018-02-

07T02:24:04.257Z'}

root@9972adbbdbb3:/userauth#

Nowthatwe'veprovedthattheauthenticationservicewillwork,and,bytheway,createdauseraccount,youshouldbeabletobrowsetotheNotesapplicationandrunitthroughitspaces.

Youcanalsotrypingingdifferentcontainerstoensurethattheapplicationnetworktopologyhasbeencreatedcorrectly.

IfyouuseDockercommand-linetoolstoexploretherunningcontainersandnetworks,you'llseetheyhavenewnames.Thenewnamesaresimilartotheoldnames,butprefixedwiththestringcompose_.ThisisasideeffectofusingDockercompose.

Bydefault,docker-composeattachestothecontainerssothatloggingoutputisprintedontheTerminal.Outputfromallfourcontainerswillbeintermingledtogether.Thankfully,eachlineisprependedbythecontainername.

Whenyou'redonetestingthesystem,simplytypeCTRL+ContheTerminal:

^CGracefullystopping...(pressCtrl+Cagaintoforce)

Stoppingdb-userauth...done

Stoppinguserauth...done

Stoppingdb-notes...done

Stoppingnotes...done

ToavoidrunningwiththecontainersattachedtotheTerminal,usethe-doption.ThissaystodetachfromtheTerminalandruninthebackground.

Analternatewaytobringdownthesystemdescribedinthecomposefileiswiththedocker-composedowncommand.

Theupcommandbuilds,recreates,andstartsthecontainers.Thebuildstepcanbehandledseparatelyusingthedocker-composebuildcommand.Likewise,startingandstoppingthecontainerscanbehandledseparatelybyusingthedocker-composestartanddocker-compose-stopcommands.

Inallcases,yourcommandshellshouldbeinthedirectorycontainingthedocker-compose.ymlfile.That'sthedefaultnameforthisfile.Thiscanbeoverriddenwiththe-foptiontospecifyadifferentfilename.

DeployingtocloudhostingwithDockercomposeWe'veverifiedonourlaptopthattheservicesdescribedbythecomposefileworkasintended.Launchingthecontainersisnowautomated,fixingoneoftheissueswenamedearlier.It'snowtimetoseehowtodeploytoacloud-hostingprovider.ThisiswhereweturntoDockermachine.

DockermachinecanbeusedtoprovisionDockerinstancesinsideaVirtualBoxhostonyourlaptop.Whatwe'llbedoingisprovisioningaDockersystemonDigitalOcean.Thedocker-machinecommandcomeswithdriverssupportingalonglistofcloud-hostingproviders.It'seasytoadapttheinstructionsshownhereforotherproviders,simplybysubstitutingadifferentdriver.

AftersigningupforaDigitalOceanaccount,clickontheAPIlinkinthedashboard.WeneedanAPItokentograntdocker-machineaccesstotheaccount.Gothroughtheprocessofcreatingatokenandsaveawaythetokenstringyou'regiven.TheDockerwebsitehasatutorialathttps://docs.docker.com/machine/examples/ocean/.

Withthetokeninhand,typethefollowing:

$docker-machinecreate--driverdigitalocean--digitalocean-size2gb\

--digitalocean-access-tokenTOKEN-FROM-PROVIDER\

sandbox

Runningpre-createchecks...

Creatingmachine...

(sandbox)CreatingSSHkey...

(sandbox)CreatingDigitalOceandroplet...

(sandbox)WaitingforIPaddresstobeassignedtotheDroplet...

Waitingformachinetoberunning,thismaytakeafewminutes...

Detectingoperatingsystemofcreatedinstance...

WaitingforSSHtobeavailable...

Detectingtheprovisioner...

Provisioningwithubuntu(systemd)...

InstallingDocker...

Copyingcertstothelocalmachinedirectory...

Copyingcertstotheremotemachine...

SettingDockerconfigurationontheremotedaemon...

CheckingconnectiontoDocker...

Dockerisupandrunning!

ToseehowtoconnectyourDockerClienttotheDockerEnginerunningonthis

virtualmachine,run:docker-machineenvsandbox

Thedigitaloceandriveris,aswesaidearlier,usedwithDigitalOcean.TheDockerwebsitehasalistofdriversathttps://docs.docker.com/machine/drivers/.

Alotofinformationisprintedhereaboutthingsbeingsetup.Themostimportantisthemessageattheend.AseriesofenvironmentvariablesareusedtotellthedockercommandwheretoconnecttotheDockerEngineinstance.Asthemessagessay,run:docker-machineenvsandbox:

$docker-machineenvsandbox

exportDOCKER_TLS_VERIFY="1"

exportDOCKER_HOST="tcp://45.55.37.74:2376"

exportDOCKER_CERT_PATH="homedavid/.docker/machine/machines/sandbox"

exportDOCKER_MACHINE_NAME="sandbox"

#Runthiscommandtoconfigureyourshell:

#eval$(docker-machineenvsandbox)

That'stheenvironmentvariablesusedtoaccesstheDockerhostwejustcreated.Youshouldalsogotoyourcloud-hostingproviderdashboardandseethatthehosthasbeencreated.Thiscommandalsogivesussomeinstructionstofollow:

$eval$(docker-machineenvsandbox)

$docker-machinels

NAMEACTIVEDRIVERSTATEURLSWARM

DOCKERERRORS

sandbox*digitaloceanRunningtcp://45.55.37.74:2376

v18.01.0-ce

ThisshowsthatwehaveaDockerEngineinstancerunninginahostatour

chosencloud-hostingprovider.

Oneinterestingtestatthispointistorundockerps-aonthisTerminal,andthentorunitinanotherTerminalthatdoesnothavetheseenvironmentvariables.Thatshouldshowthecloudhosthasnocontainersatall,whileyourlocalmachinemayhavesomecontainers(dependingonwhatyoucurrentlyhaverunning):

$dockerrunhello-world

Unabletofindimage'hello-world:latest'locally

latest:Pullingfromlibrary/hello-world

ca4f61b1923c:Pullcomplete

Digest:

sha256:66ef312bbac49c39a89aa9bcc3cb4f3c9e7de3788c944158df3ee0176d32b751

Status:Downloadednewerimageforhello-world:latest

...

$dockerimages

REPOSITORYTAGIMAGEIDCREATED

SIZE

hello-worldlatestf2a91732366c2monthsago

1.85kB

Here,we'veverifiedthatwecanlaunchacontainerontheremotehost.

Thenextstepistobuildourcontainersforthenewmachine.Becausewe'veswitchedtheenvironmentvariablestopointtothenewserver,thesecommandscauseactiontohappenthereratherthaninsideourlaptop:

$docker-composebuild

db-userauthusesanimage,skipping

db-notesusesanimage,skipping

Buildingnotes

Step1/22:FROMnode:9.5

9.5:Pullingfromlibrary/node

f49cf87b52c1:Pullcomplete

7b491c575b06:Pullcomplete

b313b08bab3b:Pullcomplete

51d6678c3f0e:Pullcomplete

...

Becausewechangedtheenvironmentvariables,thebuildoccursonthe

sandboxmachineratherthanonourlaptop,aspreviously.

ThiswilltakeawhilebecausetheDockerimagecacheontheremotemachineisempty.Additionally,buildingthenotesappanduserauthcontainerscopiestheentiresourcetreetotheserverandrunsallbuildstepsontheserver.

Thebuildmayfailifthedefaultmemorysizeis500MB,thedefaultonDigitalOceanatthetimeofwriting.Ifso,thefirstthingtotryisresizingthememoryonthehosttoatleast2GB.

Oncethebuildisfinished,launchthecontainersontheremotemachine:

$docker-composeup

Creatingnotes...done

Recreatingdb-userauth...done

Recreatingdb-notes...done

Creatingnotes...

Attachingtodb-userauth,db-notes,userauth,notes

Oncethecontainersstart,youshouldtesttheuserauthcontaineraswe'vedonepreviously.Unfortunately,thefirsttimeyoudothis,thatcommandwillfail.Theproblemistheselinesinthedocker-compose.yml:

-../authnet/my.cnf:etcmy.cnf

...

-../frontnet/my.cnf:etcmy.cnf

Inthiscase,thebuildoccursontheremotemachine,andthedocker-machinecommanddoesnotcopythenamedfiletotheserver.Hence,whenDockerattemptstostartthecontainer,itisunabletodosobecausethatvolumemountcannotbesatisfiedbecausethefileissimplynotthere.This,then,meanssomesurgeryondocker-compose.yml,andtoaddtwonewDockerfiles.

First,makethesechangestodocker-compose.yml:

...

db-userauth:

build:../authnet

container_name:db-userauth

networks:

-authnet

volumes:

-db-userauth-data:varlib/mysql

restart:always

...

db-notes:

build:../frontnet

container_name:db-notes

networks:

-frontnet

volumes:

-db-notes-data:varlib/mysql

restart:always

InsteadofbuildingthedatabasecontainersfromaDockerimage,we'renowbuildingthemfromapairofDockerfiles.NowwemustcreatethosetwoDockerfiles.

Inauthnet,createafilenamedDockerfilecontainingthefollowing:

FROMmysql/mysql-server:5.7

EXPOSE3306

COPYmy.cnfetc

ENVMYSQL_RANDOM_ROOT_PASSWORD="true"

ENVMYSQL_USER=userauth

ENVMYSQL_PASSWORD=userauth

ENVMYSQL_DATABASE=userauth

CMD["mysqld","--character-set-server=utf8mb4",\

"--collation-server=utf8mb4_unicode_ci","--bind-address=0.0.0.0"]

Thiscopiescertainsettingsfromwhathadbeenthedb-userauthdescriptionindocker-compose.yml.TheimportantthingisthatwenowCOPYthemy.cnffileratherthanuseavolumemount.

Infrontnet,createaDockerfilecontainingthefollowing:

FROMmysql/mysql-server:5.7

EXPOSE3306

COPYmy.cnfetc

ENVMYSQL_RANDOM_ROOT_PASSWORD="true"

ENVMYSQL_USER=notes

ENVMYSQL_PASSWORD=notes12345

ENVMYSQL_DATABASE=notes

CMD["mysqld","--character-set-server=utf8mb4",\

"--collation-server=utf8mb4_unicode_ci","--bind-address=0.0.0.0"]

Thisisthesame,butwithafewcriticalvalueschanged.

Aftermakingthesechanges,wecannowbuildthecontainers,andlaunchthem:

$docker-composebuild

...muchoutput

$docker-composeup--force-recreate

...muchoutput

Nowthatwehaveaworkingbuild,andcanbringupthecontainers,let'sinspectthemandverifyeverythingworks.

Executeashellinuserauthtotestandsetuptheuserdatabase:

$dockerexec-ituserauthbash

root@931dd2a267b4:/userauth#PORT=3333nodeusers-list.js

List[{id:'me',username:'me',provider:'local',

familyName:'Einarrsdottir',givenName:'Ashildr',middleName:'',

emails:'[]',photos:'[]'}]

Asmentionedpreviously,thisverifiesthattheuserauthserviceworks,thattheremotecontainersaresetup,andthatwecanproceedtousingtheNotesapplication.

Thequestionis:What'stheURLtouse?Theserviceisnotonlocalhost,becauseit'sontheremoteserver.Wedon'thaveadomainnameassigned,butthereisanIPaddressfortheserver.

Runthefollowingcommand:

$docker-machineipsandbox

45.55.37.74

DockertellsyoutheIPaddress,whichyoushoulduseasthebasisoftheURL.Hence,inyourbrowser,visithttp://IP-ADDRESS:3000

WithNotesdeployedtotheremoteserver,youshouldcheckoutallthethingswe'velookedatpreviously.Thebridgenetworksshouldexist,asshownpreviously,withthesamelimitedaccessbetweencontainers.Theonlypublicaccessshouldbeport3000onthenotescontainer.

RemembertosettheTWITTER_CALLBACK_HOSTenvironmentvariableappropriatelyforyourserver.

Becauseourdatabasecontainersmountavolumetostorethedata,let'sseewherethatvolumelandedontheserver:

$dockervolumels

DRIVERVOLUMENAME

localcompose_db-notes-data

localcompose_db-userauth-data

Thosearetheexpectedvolumes,oneforeachcontainer:

$dockervolumeinspectcompose_db-notes-data

[

{

"CreatedAt":"2018-02-07T06:30:06Z",

"Driver":"local",

"Labels":{

"com.docker.compose.project":"compose",

"com.docker.compose.volume":"db-notes-data"

},

"Mountpoint":"varlib/docker/volumes/compose_db-notes-

data/_data",

"Name":"compose_db-notes-data",

"Options":{},

"Scope":"local"

}

]

Thosearethedirectories,butthey'renotlocatedonourlaptop.Instead,

they'reontheremoteserver.Accessingthesedirectoriesmeansloggingintotheremoteservertotakealook:

$docker-machinesshsandbox

WelcometoUbuntu16.04.3LTS(GNU/Linux4.4.0-112-genericx86_64)

*Documentation:https://help.ubuntu.com

*Management:https://landscape.canonical.com

*Support:https://ubuntu.com/advantage

GetcloudsupportwithUbuntuAdvantageCloudGuest:

http://www.ubuntu.com/business/services/cloud

4packagescanbeupdated.

0updatesaresecurityupdates.

Lastlogin:WedFeb704:00:292018from108.213.68.139

root@sandbox:~#

Fromthispoint,youcaninspectthedirectoriescorrespondingtothesevolumesandseethattheyindeedcontainMySQLconfigurationanddatafiles:

root@sandbox:~#lsvarlib/docker/volumes/compose_db-notes-data/_data

auto.cnfclient-key.pemib_logfile1mysql.sock.lock

public_key.pem

ca-key.pemib_buffer_poolibtmp1notesserver-

cert.pem

ca.pemibdata1mysqlperformance_schemaserver-

key.pem

client-cert.pemib_logfile0mysql.sockprivate_key.pemsys

You'llalsofindthattheDockercommand-linetoolswillwork.Theprocesslistisespeciallyinteresting:

Lookcloselyatthisandyouseeaprocesscorrespondingtoeverycontainerinthesystem.Theseprocessesarerunninginthehostoperatingsystem.Dockercreateslayersofconfiguration/containmentaroundthoseprocessestocreatetheappearancethattheprocessisrunningunderadifferentoperatingsystem,andwithvarioussystem/networkconfigurationfiles,asspecifiedinthecontainerscreenshot.

TheclaimedadvantageDockerhasovervirtualizationapproaches,suchasVirtualBox,isthatDockerisverylightweight.WeseerightherewhyDockerislightweight:thereisnovirtualizationlayer,thereisonlyacontainerizationprocess(docker-containerd-shim).

Onceyou'resatisfiedthatNotesisworkingontheremoteserver,youcanshutitdownandremoveitasfollows:

$docker-composestop

Stoppingnotesapp...done

Stoppinguserauth...done

Stoppingdb-notes...done

Stoppingdb-auth...done

Thisshutsdownallthecontainersatonce:

$docker-machinestopsandbox

Stopping"sandbox"...

Machine"sandbox"wasstopped.

Thisshutsdowntheremotemachine.Thecloud-hostingproviderdashboardwillshowthattheDroplethasstopped.

Atthispoint,youcangoaheadanddeletetheDockermachineinstanceaswell,ifyoulike:

$docker-machinermsandbox

Abouttoremovesandbox

Areyousure?(y/n):y

Successfullyremovedsandbox

And,ifyou'retrulycertainyouwanttodeletethemachine,theprecedingcommanddoesthedeed.Assoonasyoudothis,themachinewillbeerasedfromyourcloud-hostingproviderdashboard.

SummaryThischapterhasbeenquiteajourney.Wewentfromanapplicationthatexistedsolelyonourlaptop,toexploringtwowaystodeployNode.jsapplicationstoaproductionserver.

WestartedbyreviewingtheNotesapplicationarchitectureandhowthatwillaffectdeployment.Thatenabledyoutounderstandwhatyouhadtodoforserverdeployment.

ThenyoulearnedthetraditionalwaytodeployservicesonLinuxusinganinitscript.PM2isausefultoolformanagingbackgroundprocessesinsuchanenvironment.Youalsolearnedhowtoprovisionaremoteserverusingavirtualmachinehostingservice.

ThenyoutookalongtripintothelandofDocker,anewandexcitingsystemfordeployingservicesonmachines.YoulearnedhowtowriteaDockerfilesothatDockerknowshowtoconstructaserviceimage.YoulearnedseveralwaystodeployDockerimagesonalaptoporonaremoteserver.Andyoulearnedhowtodescribeamulti-containerapplicationusingDockercompose.

You'realmostreadytowrapupthisbook.You'velearnedalotalongtheway;therearetwofinalthingstocover.

Inthenextchapter,wewilllearnaboutbothunittestingandfunctionaltesting.Whileacoreprincipleoftest-drivendevelopmentistowritetheunittestsbeforewritingtheapplication,we'vedoneittheotherwayaroundandputthechapteraboutunittestingattheendofthisbook.That'snottosayunittestingisunimportant,becauseitisextremelyimportant.

Inthefinalchapter,we'llexplorehowtohardenourapplication,andapplicationinfrastructure,againstattackers.

UnitTestingandFunctionalTestingUnittestinghasbecomeaprimarypartofgoodsoftwaredevelopmentpractice.Itisamethodbywhichindividualunitsofsourcecodearetestedtoensureproperfunctioning.Eachunitistheoreticallythesmallesttestablepartofanapplication.InaNode.jsapplication,youmightconsidereachmoduleasaunit.

Inunittesting,eachunitistestedseparately,isolatingtheunitundertestasmuchaspossiblefromotherpartsoftheapplication.Ifatestfails,youwouldwantittobeduetoabuginyourcoderatherthanabuginthepackagethatyourcodehappenstouse.Acommontechniqueistousemockobjectsormockdatatoisolateindividualpartsoftheapplicationfromoneanother.

Functionaltesting,ontheotherhand,doesn'ttrytotestindividualcomponents,butinsteaditteststhewholesystem.Generallyspeaking,unittestingisperformedbythedevelopmentteam,andfunctionaltestingisperformedbyaQualityAssurance(QA)orQualityEngineering(QE)team.Bothtestingmodelsareneededtofullycertifyanapplication.Ananalogymightbethatunittestingissimilartoensuringthateachwordinasentenceiscorrectlyspelled,whilefunctionaltestingensuresthattheparagraphcontainingthatsentencehasagoodstructure.

Inthischapter,we'llcover:

Assertionsasthebasisofsoftwaretests

TheMochaunittestingframeworkandtheChaiassertionslibrary

Usingteststofindbugsandfixingthebug

UsingDockertomanagetestinfrastructure

TestingaRESTbackendservice

UItestinginarealwebbrowserusingPuppeteer

ImprovingUItestabilitywithelementIDattributes

Assert–thebasisoftestingmethodologiesNode.jshasausefulbuilt-intestingtool,theassertmodule.Itsfunctionalityissimilartoassertlibrariesinotherlanguages.Namely,it'sacollectionoffunctionsfortestingconditions,andiftheconditionsindicateanerror,theassertfunctionthrowsanexception.

Atitssimplest,atestsuiteisaseriesofassertcallstovalidatethebehaviorofathingbeingtested.Forexample,atestsuitecouldinstantiatetheuserauthenticationservice,thenmakeanAPIcall,usingassertmethodstovalidatetheresult,thenmakeanotherAPIcall,validatingitsresults,andsoon.

Consideracodesnippetlikethis,whichyoucouldsaveinafilenameddeleteFile.js:

constfs=require('fs');

exports.deleteFile=function(fname,callback){

fs.stat(fname,(err,stats)=>{

if(err)callback(newError(`thefile${fname}doesnotexist`));

else{

fs.unlink(fname,err2=>{

if(err)callback(newError(`couldnotdelete${fname}`));

elsecallback();

});

}

});

};

Thefirstthingtonoticeisthiscontainsseverallayersofasynchronouscallbackfunctions.Thatpresentsacoupleofchallenges:

Capturingerrorsfromdeepinsideacallback,toensurethetestscenariofails

Detectingconditionswherethecallbacksarenevercalled

Thefollowingisanexampleofusingassertfortesting.Createafilenamedtest-deleteFile.jscontainingthefollowing:

constfs=require('fs');

constassert=require('assert');

constdf=require('./deleteFile');

df.deleteFile("no-such-file",(err)=>{

assert.throws(

function(){if(err)throwerr;},

function(error){

if((errorinstanceofError)

&&doesnotexist.test(error)){

returntrue;

}elsereturnfalse;

},

"unexpectederror"

);

});

Thisiswhat'scalledanegativetestscenario,inthatit'stestingwhetherrequestingtodeleteanonexistentfilethrowsanerror.

Ifyouarelookingforaquickwaytotest,theassertmodulecanbeusefulwhenusedthisway.Ifitrunsandnomessagesareprinted,thenthetestpasses.But,diditcatchtheinstanceofthedeleteFilecallbackneverbeingcalled?

$nodetest-deleteFile.js

Theassertmoduleisusedbymanyofthetestframeworksasacoretoolforwritingtestcases.Whatthetestframeworksdoiscreateafamiliartestsuiteandtestcasestructuretoencapsulateyourtestcode.

TherearemanystylesofassertionlibrariesavailableinNode.js.Laterinthischapter,we'llusetheChaiassertionlibrary(http://chaijs.com/)whichgivesyouachoicebetweenthreedifferentassertionstyles(should,expect,andassert).

TestingaNotesmodelLet'sstartourunittestingjourneywiththedatamodelswewrotefortheNotesapplication.Becausethisisunittesting,themodelsshouldbetestedseparatelyfromtherestoftheNotesapplication.

InthecaseofmostoftheNotesmodels,isolatingtheirdependenciesimpliescreatingamockdatabase.Areyougoingtotestthedatamodelortheunderlyingdatabase?Mockingoutadatabasemeanscreatingafakedatabaseimplementation,whichdoesnotlooklikeaproductiveuseofourtime.Youcanarguethattestingadatamodelisreallyabouttestingtheinteractionbetweenyourcodeandthedatabase,thatmockingoutthedatabasemeansnottestingthatinteraction,andthereforeweshouldtestourcodeagainstthedatabaseengineusedinproduction.

Withthatlineofreasoninginmind,we'llskipmockingoutthedatabase,andinsteadrunthetestsagainstadatabasecontainingtestdata.Tosimplifylaunchingthetestdatabase,we'lluseDockertostartandstopaversionoftheNotesapplicationstackthat'ssetupfortesting.

MochaandChai–thechosentesttoolsIfyouhaven'talreadydoneso,duplicatethesourcetreetouseinthischapter.Forexample,ifyouhadadirectorynamedchap10,createonenamedchap11containingeverythingfromchap10.

Inthenotesdirectory,createanewdirectorynamedtest.

Mocha(http://mochajs.org/)isoneofmanytestframeworksavailableforNode.js.Asyou'llseeshortly,ithelpsuswritetestcasesandtestsuites,anditprovidesatestresultsreportingmechanism.ItwaschosenoverthealternativesbecauseitsupportsPromises.ItfitsverywellwiththeChaiassertionlibrarymentionedearlier.And,we'llneedtouseES6modulesfromtestsuiteswritteninCommonJS,andthereforewemustusetheesmmodule.

Youmayfindreferencestoanearlier@std/esmmodule.Thatmodulehasbeendeprecated,withesmputinitsplace.

Whileinthenotes/testdirectory,typethistoinstallMocha,Chai,andesm:$npminit...answerthequestionstocreatepackage.json$npminstallmocha@5.xchai@4.1.xesm--save

NotesmodeltestsuiteBecausewehaveseveralNotesmodels,thetestsuiteshouldrunagainstanymodel.WecanwritetestsusingtheNotesmodelAPIwedeveloped,andanenvironmentvariableshouldbeusedtodeclarethemodeltotest.

Becausewe'vewrittentheNotesapplicationusingES6modules,wehaveasmallchallengetoovercome.MochaonlysupportsrunningtestsinCommonJSmodules,andNode.js(asofthiswriting)doesnotsupportloadinganES6modulefromaCommonJSmodule.AnES6modulecanuseimporttoloadaCommonJSmodule,butaCommonJSmodulecannotuserequiretoloadanES6module.Therearevarioustechnicalreasonsbehindthis,thebottomlineisthatwe'relimitedinthisway.

BecauseMocharequiresthattestsbeCommonJSmodules,we'reinthepositionofhavingtoloadanES6moduleintoaCommonJSmodule.Amodule,esm,existswhichallowsthatcombinationtowork.Ifyou'llreferback,weinstalledthatmoduleintheprevioussection.Let'sseehowtouseit.

Inthetestdirectory,createafilenamedtest-model.jscontainingthisastheoutershellofthetestsuite:'usestrict';require=require("esm")(module,{"esm":"js"});constassert=require('chai').assert;constmodel=require('../models/notes');describe("ModelTest",function(){..});

ThesupporttoloadES6modulesisenabledbytherequire('esm')statementshownhere.Itreplacesthestandardrequirefunctionwithonefromtheesmmodule.ThatparameterlistattheendenablesthefeaturetoloadES6modulesinaCommonJSmodule.Onceyou'vedonethis,yourCommonJSmodulecanloadanES6moduleasevidencedbyrequire('../models/notes')acoupleoflineslater.

TheChailibrarysupportsthreeflavorsofassertions.We'reusingtheassertstylehere,butit'seasytouseadifferentstyleifyouprefer.FortheotherstylessupportedbyChai,seehttp://chaijs.com/guide/styles/.

Chai'sassertionsincludeaverylonglistofusefulassertionfunctions,seehttp://chaijs.com/api/assert/.

TheNotesmodeltotestmustbeselectedwiththeNOTES_MODELenvironmentvariable.Forthemodelsthatalsoconsultenvironmentvariables,we'llneedtosupplythatconfigurationaswell.

WithMocha,atestsuiteiscontainedwithinadescribeblock.Thefirstargumentisdescriptivetext,whichyouusetotailorthepresentationoftestresults.

Ratherthanmaintainingaseparatetestdatabase,wecancreateoneontheflywhileexecutingtests.Mochahaswhatarecalledhooks,whicharefunctionsexecutedbeforeoraftertestcaseexecution.Thehookfunctionsletyou,thetestsuiteauthor,setupandteardownrequiredconditionsforthetestsuitetooperateasdesired.Forexample,tocreateatestdatabasewithknowntestcontent:describe("ModelTest",function(){beforeEach(asyncfunction(){try{constkeyz=awaitmodel.keylist();for(letkeyofkeyz){awaitmodel.destroy(key);}awaitmodel.create("n1","Note1","Note1");awaitmodel.create("n2","Note2","Note2");awaitmodel.create("n3","Note3","Note3");}catch(e){console.error(e);throwe;}});..});

ThisdefinesabeforeEachhook,whichisexecutedbeforeeverytestcase.

Theotherhooksarebefore,after,beforeEach,andafterEach.Theeachhooksaretriggeredbeforeoraftereachtestcaseexecution.

Thisismeanttobeacleanup/preparationstepbeforeeverytest.ItusesourNotesAPItofirstdeleteallnotesfromthedatabase(ifany)andthencreateasetofnewnoteswithknowncharacteristics.Thistechniquesimplifiestestsbyensuringthatwehaveknownconditionstotestagainst.

Wealsohaveasideeffectoftestingthemodel.keylistandmodel.createmethods.

InMocha,aseriesoftestcasesareencapsulatedwithadescribeblock,andwrittenusinganitblock.Thedescribeblockismeanttodescribethatgroupoftests,andtheitblockisforcheckingassertionsonaspecificaspectofthethingbeingtested.Youcannestthedescribeblocksasdeeplyasyoulike:describe("checkkeylist",function(){it("shouldhavethreeentries",asyncfunction(){constkeyz=awaitmodel.keylist();assert.exists(keyz);assert.isArray(keyz);assert.lengthOf(keyz,3);});it("shouldhavekeysn1n2n3",asyncfunction(){constkeyz=awaitmodel.keylist();assert.exists(keyz);assert.isArray(keyz);assert.lengthOf(keyz,3);for(letkeyofkeyz){assert.match(key,n[123],"correctkey");}});it("shouldhavetitlesNode#",asyncfunction(){constkeyz=awaitmodel.keylist();assert.exists(keyz);assert.isArray(keyz);assert.lengthOf(keyz,3);varkeyPromises=keyz.map(key=>model.read(key));

constnotez=awaitPromise.all(keyPromises);for(letnoteofnotez){assert.match(note.title,Note[123],"correcttitle");}});});

TheideaistocallNotesAPIfunctions,thentotesttheresultstocheckwhethertheymatchedtheexpectedresults.

Thisdescribeblockiswithintheouterdescribeblock.Thedescriptionsgiveninthedescribeanditblocksareusedtomakethetestreportmorereadable.Theitblockformsapseudo-sentencealongthelinesofit(thethingbeingtested)shoulddothisorthat.

ItisimportantwithMochatonotusearrowfunctionsinthedescribeanditblocks.Bynow,youwillhavegrownfondofarrowfunctionsbecauseofhowmucheasiertheyaretowrite.But,MochacallsthesefunctionswithathisobjectcontainingusefulfunctionsforMocha.Becausearrowfunctionsavoidsettingupathisobject,Mochawouldbreak.

EventhoughMocharequiresregularfunctionsforthedescribeanditblocks,wecanusearrowfunctionswithinthosefunctions.

HowdoesMochaknowwhetherthetestcodepasses?Howdoesitknowwhenthetestfinishes?Thissegmentofcodeshowsoneofthethreemethods.

Generally,Mochaislookingtoseeifthefunctionthrowsanexception,orwhetherthetestcasetakestoolongtoexecute(atimeoutsituation).Ineithercase,Mochawillindicateatestfailure.That'sofcoursesimpletodeterminefornon-asynchronouscode.But,Node.jsisallaboutasynchronouscode,andMochahastwomodelsfortestingasynchronouscode.Inthefirst(notseenhere),Mochapassesinacallbackfunction,andthetestcodeistocallthecallbackfunction.Inthesecond,asseenhere,itlooksforaPromisebeingreturnedbythetestfunction,anddeterminespass/failonwhetherthePromiseisintheresolveorrejectstate.

Inthiscase,we'reusingasyncfunctions,becausetheyautomaticallyreturnaPromise.Withinthefunctions,we'recallingasynchronousfunctions

usingawait,ensuringanythrownexceptionisindicatedasarejectedPromise.

Anotheritemtonoteisthequestionaskedearlier:whatifthecallbackfunctionwe'retestingisnevercalled?Or,whatifaPromiseisneverresolved?Mochastartsatimerandifthetestcasedoesnotfinishbeforethetimerexpires,Mochafailsthetestcase.

ConfiguringandrunningtestsWehavemoreteststowrite,butlet'sfirstgetsetuptorunthetests.Thesimplestmodeltotestisthein-memorymodel.Let'saddthistothescriptssectionofnotes/test/package.json:

"test-notes-memory":"NOTES_MODEL=memorymochatest-model",

Toinstalldependencies,wemustrunnpminstallinboththenotes/testandnotesdirectories.Thatwayboththedependenciesforthetestcode,andthedependenciesforNotes,areinstalledintheircorrectplace.

Then,wecanrunitasfollows:

$npmruntest-notes-memory

>notes-test@1.0.0test-notes-memoryUsersdavid/chap11/notes/test

>NOTES_MODEL=memorymochatest-model

ModelTest

checkkeylist

√shouldhavethreeentries

√shouldhavekeysn1n2n3

√shouldhavetitlesNode#

3passing(18ms)

Themochacommandisusedtorunthetestsuite.

Thestructureoftheoutputfollowsthestructureofthedescribeanditblocks.Youshouldsetupthedescriptivetextstringssoitreadsnicely.

MoretestsfortheNotesmodelThatwasn'tenoughtotestmuch,solet'sgoaheadandaddsomemoretests:

describe("readnote",function(){

it("shouldhavepropernote",asyncfunction(){

constnote=awaitmodel.read("n1");

assert.exists(note);

assert.deepEqual({key:note.key,title:note.title,body:

note.body},{

key:"n1",title:"Note1FAIL",body:"Note1"

});

});

it("Unknownnoteshouldfail",asyncfunction(){

try{

constnote=awaitmodel.read("badkey12");

assert.notExists(note);

thrownewError("shouldnotgethere");

}catch(err){

//thisisexpected,sodonotindicateerror

assert.notEqual(err.message,"shouldnotgethere");

}

});

});

describe("changenote",function(){

it("afterasuccessfulmodel.update",asyncfunction(){

constnewnote=awaitmodel.update("n1","Note1title

changed","Note1bodychanged");

constnote=awaitmodel.read("n1");

assert.exists(note);

assert.deepEqual({key:note.key,title:note.title,body:

note.body},{

key:"n1",title:"Note1titlechanged",body:"Note1body

changed"

});

});

});

describe("destroynote",function(){

it("shouldremovenote",asyncfunction(){

awaitmodel.destroy("n1");

constkeyz=awaitmodel.keylist();

assert.exists(keyz);

assert.isArray(keyz);

assert.lengthOf(keyz,2);

for(letkeyofkeyz){

assert.match(key,n[23],"correctkey");

}

});

it("shouldfailtoremoveunknownnote",asyncfunction(){

try{

awaitmodel.destroy("badkey12");

thrownewError("shouldnotgethere");

}catch(err){

//thisisexpected,sodonotindicateerror

assert.notEqual(err.message,"shouldnotgethere");

}

});

});

after(function(){model.close();});

});

Noticethatforthenegativetests–wherethetestpassesifanerroristhrown–werunitinatry/catchblock.ThethrownewErrorlineineachcaseshouldnotexecutebecausetheprecedingcodeshouldthrowanerror.Therefore,wecancheckifthemessageinthatthrownerroristhemessagewhicharrives,andfailthetestifthat'sthecase.

Now,thetestreport:

$npmruntest-notes-memory

>notes-test@1.0.0test-notes-memoryUsersdavid/chap11/notes/test

>NOTES_MODEL=memorymochatest-model

ModelTest

checkkeylist

√shouldhavethreeentries

√shouldhavekeysn1n2n3

√shouldhavetitlesNode#

readnote

√shouldhavepropernote

√Unknownnoteshouldfail

changenote

√afterasuccessfulmodel.update

destroynote

√shouldremovenote

√shouldfailtoremoveunknownnote

8passing(17ms)

Intheseadditionaltests,wehaveacoupleofnegativetests.Ineachtestthatweexpecttofail,wesupplyanotekeythatweknowisnotinthedatabase,andwethenensurethatthemodelgivesusanerror.

TheChaiAssertionsAPIincludessomeveryexpressiveassertions.Inthiscase,we'veusedthedeepEqualmethodwhichdoesadeepcomparisonoftwoobjects.Inourcase,itlookslikethis:

assert.deepEqual({key:note.key,title:note.title,body:note.body},{

key:"n1",title:"Note1",body:"Note1"

});

Thisreadsnicelyinthetestcode,butmoreimportantlyareportedtestfailurelooksverynice.Sincethesearecurrentlypassing,tryintroducinganerrorbychangingoneoftheexpectedvaluestrings.Uponrerunningthetest,you'llsee:

ModelTest

checkkeylist

√shouldhavethreeentries

√shouldhavekeysn1n2n3

√shouldhavetitlesNode#

readnote

1)shouldhavepropernote

√Unknownnoteshouldfail

changenote

√afterasuccessfulmodel.update

destroynote

√shouldremovenote

√shouldfailtoremoveunknownnote

7passing(42ms)

1failing

1)ModelTest

readnote

shouldhavepropernote:

AssertionError:expected{Object(key,title,...)}todeeply

equal{Object(key,title,...)}

+expected-actual

{

"body":"Note1"

"key":"n1"

-"title":"Note1"

+"title":"Note1FAIL"

}

atContext.<anonymous>(test-model.js:53:16)

at<anonymous>

Atthetopisthestatusreportofeachtestcase.Foronetest,insteadofacheckmarkisanumber,andthenumbercorrespondstothereporteddetailsatthebottom.Mochapresentstestfailuresthiswaywhenthespecreporterisused.Mochasupportsothertestreportformats,someofwhichproducedatathatcanbesentintoteststatusreportingsystems.Formoreinformation,seehttps://mochajs.org/#reporters.

Inthiscase,thefailurewasdetectedbyadeepEqualmethod,whichpresentsthedetectedobjectinequalityinthisway.

TestingdatabasemodelsThatwasgood,butweobviouslywon'trunNotesinproductionwiththein-memoryNotesmodel.Thismeansthatweneedtotestalltheothermodels.

TestingtheLevelUPandfilesystemmodelsiseasy,justaddthistothescriptssectionofpackage.json:"test-notes-levelup":"NOTES_MODEL=levelupmocha","test-notes-fs":"NOTES_MODEL=fsmocha",

Thenrunthefollowingcommand:$npmruntest-notes-fs$npmruntest-notes-levelup

Thiswillproduceasuccessfultestresult.

ThesimplestdatabasetotestisSQLite3,sinceitrequireszerosetup.WehavetwoSQLite3modelstotest,let'sstartwithnotes-sqlite3.js.Addthefollowingtothescriptssectionofpackage.json:"test-notes-sqlite3":"rm-fchap11.sqlite3&&sqlite3chap11.sqlite3--init../models/chap07.sql<devnull&&NOTES_MODEL=sqlite3SQLITE_FILE=chap11.sqlite3mochatest-model",

Thiscommandsequenceputsthetestdatabaseinthechap11.sqlite3file.Itfirstinitializesthatdatabaseusingthesqlite3command-linetool.Notethatwe'veconnecteditsinputtodevnullbecausethesqlite3commandwillpromptforinputotherwise.Then,itrunsthetestsuitepassinginenvironmentvariablesrequiredtorunagainsttheSQLite3model.

Runningthetestsuitedoesfindtwoerrors:$npmruntest-notes-sqlite3

>notestest@1.0.0test-notes-sqlite3Usersdavid/chap11/notes/test>rm-fchap11.sqlite3&&sqlite3chap11.sqlite3--init

../models/chap07.sql<devnull&&NOTES_MODEL=sqlite3SQLITE_FILE=chap11.sqlite3mochatest-model

ModelTestcheckkeylist√shouldhavethreeentries√shouldhavekeysn1n2n3√shouldhavetitlesNode#readnote√shouldhavepropernote1)Unknownnoteshouldfailchangenote√afterasuccessfulmodel.update(114ms)destroynote√shouldremovenote(103ms)2)shouldfailtoremoveunknownnote

6passing(6s)2failing

1)ModelTestreadnoteUnknownnoteshouldfail:UncaughtTypeError:Cannotreadproperty'notekey'ofundefinedatStatement.db.get(/home/david/nodewebdev/nodeweb-development-code-4th-edition/chap11/notes/models/notes-sqlite3.mjs:64:39)

2)ModelTestdestroynoteshouldfailtoremoveunknownnote:

AssertionError:expected'shouldnotgethere'tonotequal'shouldnotgethere'+expected-actual

Thefailingtestcallsmodel.read("badkey12"),akeywhichweknowdoesnot

exist.Writingnegativetestspaidoff.Thefailinglineofcodeatmodels/notes-sqlite3.mjs(line64)readsasfollows:constnote=newNote(row.notekey,row.title,row.body);

It'seasyenoughtoinsertconsole.log(util.inspect(row));justbeforethisandlearnthat,forthefailingcall,SQLite3gaveusundefinedforrow,explainingtheerrormessage.

Thetestsuitecallsthereadfunctionmultipletimeswithanotekeyvaluethatdoesexist.Obviously,whengivenaninvalidnotekeyvalue,thequerygivesanemptyresultssetandSQLite3invokesthecallbackwithboththeundefinederrorandtheundefinedrowvalues.Thisiscommonbehaviorfordatabasemodules.Anemptyresultsetisn'tanerror,andthereforewereceivednoerrorandanundefinedrow.

Infact,wesawthisbehaviorearlierwithmodels/notes-sequelize.mjs.Theequivalentcodeinmodels/notes-sequelize.mjsdoestherightthing,andithasacheck,whichwecanadapt.Let'srewritethereadfunctioninmodels/notes-sqlite.mjstothis:exportasyncfunctionread(key){vardb=awaitconnectDB();varnote=awaitnewPromise((resolve,reject)=>{db.get("SELECT*FROMnotesWHEREnotekey=?",[key],(err,row)=>{if(err)returnreject(err);if(!row){reject(newError(`Nonotefoundfor${key}`));}else{constnote=newNote(row.notekey,row.title,row.body);resolve(note);}});});returnnote;}

Thisissimple,wejustcheckwhetherrowisundefinedand,ifso,throwanerror.Whilethedatabasedoesn'tseeanemptyresultssetasanerror,Notesdoes.Furthermore,Notesalreadyknowshowtodealwithathrownerror

Ifweinspecttheothermodels,they'rethrowingerrorsforanonexistentkey.InSQL,itobviouslyisnotanerrorifthisSQL(frommodels/notes-sqlite3.mjs)doesnotdeleteanything:db.run("DELETEFROMnotesWHEREnotekey=?;",...);

Unfortunately,thereisn'taSQLoptiontomakethisSQLstatementfailifitdoesnotdeleteanyrecords.Therefore,wemustaddachecktoseeifarecordexists.Namely:exportasyncfunctiondestroy(key){constdb=awaitconnectDB();constnote=awaitread(key);returnawaitnewPromise((resolve,reject)=>{db.run("DELETEFROMnotesWHEREnotekey=?;",[key],err=>{if(err)returnreject(err);resolve();});});}Therefore,wereadthenoteandasabyproductweverifythenoteexists.Ifthenotedoesn'texist,readwillthrowanerror,andtheDELETEoperationwillnotevenrun.ThesearethebugswereferredtoinChapter7,DataStorageandRetrieval.Wesimplyforgottocheckfortheseconditionsinthis

inthiscase.Makethischangeandthatparticulartestcasepasses.

Thereisasecondsimilarerrorinthedestroylogic.Thetesttodestroyanonexistentnotefailstoproduceanerroratthisline:

awaitmodel.destroy("badkey12");

particularmodel.Thankfully,ourdiligenttestingcaughttheproblem.Atleast,that'sthestorytotellthemanagersratherthantellingthemthatweforgottocheckforsomethingwealreadyknewcouldhappen.Nowthatwe'vefixedmodels/notes-sqlite3.mjs,let'salsotestmodels/notes-sequelize.mjsusingtheSQLite3database.Todothis,weneedaconnectionobjecttospecifyintheSEQUELIZE_CONNECTvariable.Whilewecanreusetheexistingone,let'screateanewone.Createafilenamedtest/sequelize-sqlite.yamlcontainingthis:dbname:notestestusername:password:params:dialect:sqlitestorage:notestest-sequelize.sqlite3logging:falseThisway,wedon'toverwritetheproductiondatabaseinstancewithourtestsuite.Sincethetestsuitedestroysthedatabaseittests,itmustberunagainstadatabasewearecomfortabledestroying.TheloggingparameterturnsoffthevoluminousoutputSequelizeproducessothatwecanreadthetestresultsreport.Addthefollowingtothescriptssectionofpackage.json:"test-notes-sequelize-sqlite":"NOTES_MODEL=sequelizeSEQUELIZE_CONNECT=sequelize-sqlite.yamlmochatest-model"Thenrunthetestsuite:$npmruntest-notes-sequelize-sqlite

..

8passing(2s)

Wepasswithflyingcolors!We'vebeenabletoleveragethesametestsuiteagainstmultipleNotesmodels.Weevenfoundtwobugsinonemodel.But,wehavetwotestconfigurationsremainingtotest.Ourtestresultsmatrixreadsasfollows:models-fs:PASSmodels-memory:PASSmodels-levelup:PASSmodels-sqlite3:2failures,nowfixedmodels-sequelize:withSQLite3:PASS

models-sequelize:withMySQL:untestedmodels-mongodb:untestedThetwountestedmodelsbothrequirethesetupofadatabaseserver.Weavoidedtestingthesecombinations,butourmanagerwon'tacceptthatexcusebecausetheCEOneedstoknowwe'vecompletedthetestcycles.Notesmustbetestedinasimilarconfigurationtotheproductionenvironment.Inproduction,we'llbeusingaregulardatabaseserver,ofcourse,withMySQLorMongoDBbeingtheprimarychoices.Therefore,weneedawaythatincursalowoverheadtoruntestsagainstthosedatabases.Testingagainsttheproductionconfigurationmustbesoeasythatweshouldfeelnoresistanceindoingso,toensurethattestsarerunoftenenoughtomakethedesiredimpact.Fortunately,we'vealreadyhadexperienceofatechnologythatsupportseasilycreatinganddestroyingthedeploymentinfrastructure.Hello,Docker!

UsingDockertomanagetestinfrastructureOneadvantageDockergivesistheabilitytoinstalltheproductionenvironmentonourlaptop.It'sthenveryeasytopushthesameDockersetuptothecloud-hostingenvironmentforstagingorproductiondeployment.

Whatwe'lldointhissectionisdemonstratereusingtheDockerComposeconfigurationdefinedpreviouslyfortestinfrastructure,andtoautomateexecutingtheNotestestsuiteinsidethecontainersusingashellscript.Generallyspeaking,it'simportanttoreplicatetheproductionenvironmentwhenrunningtests.Dockercanmakethisaneasythingtodo.

UsingDocker,we'llbeabletoeasilytestagainstadatabase,andhaveasimplemethodforstartingandstoppingatestversionofourproductionenvironment.Let'sgetstarted.

DockerComposetoorchestratetestinfrastructureWehadagreatexperienceusingDockerComposetoorchestrateNotesapplicationdeployment.Thewholesystem,withfourindependentservices,iseasilydescribedincompose/docker-compose.yml.Whatwe'lldoisduplicatetheComposefile,thenmakeacoupleofsmallchangesrequiredtosupporttestexecution.

Let'sstartbymakinganewdirectory,test-compose,asasiblingtothenotes,users,andcomposedirectories.Copycompose/docker-compose.ymltothenewlycreatedtest-composedirectory.We'llbemakingseveralchangestothisfileandacoupleofsmallchangestotheexistingDockerfiles.

Wewanttochangethecontainerandnetworknamessoourtestinfrastructuredoesn'tclobbertheproductioninfrastructure.We'llconstantlydeleteandrecreatethetestcontainers,soastokeepthedevelopershappy,we'llleavedevelopmentinfrastructurealoneandperformtestingonseparateinfrastructure.Bymaintainingseparatetestcontainersandnetworks,ourtestscriptscandoanythingtheylikewithoutdisturbingthedevelopmentorproductioncontainers.

Considerthischangetothedb-authanddb-notescontainers:

db-userauth-test:

build:../authnet

container_name:db-userauth-test

networks:

-authnet-test

environment:

MYSQL_RANDOM_ROOT_PASSWORD:"true"

MYSQL_USER:userauth-test

MYSQL_PASSWORD:userauth-test

MYSQL_DATABASE:userauth-test

volumes:

-db-userauth-test-data:/var/lib/mysql

restart:always

..

db-notes-test:

build:../frontnet

container_name:db-notes-test

networks:

-frontnet-test

environment:

MYSQL_RANDOM_ROOT_PASSWORD:"true"

MYSQL_USER:notes-test

MYSQL_PASSWORD:notes12345

MYSQL_DATABASE:notes-test

volumes:

-db-notes-test-data:/var/lib/mysql

restart:always

Thisisthesameasearlier,butwith-testappendedtocontainerandnetworknames.

That'sthefirstchangewemustmake,append-testtoeverycontainerandnetworknameintest-compose/docker-compose.yml.Everythingwe'lldowithtestswillrunoncompletelyseparatecontainers,hostnames,andnetworksfromthoseofthedevelopmentinstance.

Thischangewillaffectthenotes-testanduserauth-testservicesbecausethedatabaseserverhostnamesarenowdb-auth-testanddb-notest-test.Thereareseveralenvironmentvariablesorconfigurationfilestoupdate.

Anotherconsiderationistheenvironmentvariablesrequiredtoconfiguretheservices.Previously,wedefinedallenvironmentvariablesintheDockerfiles.It'sextremelyusefultoreusethoseDockerfilessoweknowwe'retestingthesamedeploymentasisusedinproduction.Butweneedtotweaktheconfigurationsettingstomatchthetestinfrastructure.

Thedatabaseconfigurationshownhereisanexample.ThesameDockerfilesareused,butwealsodefineenvironmentvariablesintest-compose/docker-compose.yml.Asyoumightexpect,thisoverridestheDockerfileenvironmentvariableswiththevaluessethere:

userauth-test:

build:../users

container_name:userauth-test

depends_on:

-db-userauth-test

networks:

-authnet-test

-frontnet-test

environment:

DEBUG:""

NODE_ENV:"test"

SEQUELIZE_CONNECT:"sequelize-docker-test-mysql.yaml"

HOST_USERS_TEST:"localhost"

restart:always

volumes:

-./reports-userauth:/reports

..

notes-test:

build:../notes

container_name:notes-test

depends_on:

-db-notes-test

networks:

-frontnet-test

ports:

-"3000:3000"

restart:always

environment:

NODE_ENV:"test"

SEQUELIZE_CONNECT:"test/sequelize-mysql.yaml"

USER_SERVICE_URL:"http://userauth-test:3333"

volumes:

-./reports-notes:/reports

...

networks:

frontnet-test:

driver:bridge

authnet-test:

driver:bridge

volumes:

db-userauth-test-data:

db-notes-test-data:

Again,wechangedthecontainerandnetworknamestoappend-test.WemovedsomeoftheenvironmentvariablesfromDockerfiletotest-compose/docker-compose.yml.Finally,weaddedsomedatavolumestomount

hostdirectoriesinsidethecontainer.

Anotherthingtodoistosetupdirectoriestostoretestcode.AcommonpracticeinNode.jsprojectsistoputtestcodeinthesamedirectoryastheapplicationcode.Earlierinthischapter,wedidso,implementingasmalltestsuiteinthenotes/testdirectory.Asitstands,notes/Dockerfiledoesnotcopythatdirectoryintothecontainer.Thetestcodemustexistinthecontainertoexecutethetests.Anotherissueisit'shelpfultonotdeploytestcodeinproduction.

Whatwecandoistoensurethattest-compose/docker-compose.ymlmountsnotes/testintothecontainer:

notes-test:

...

volumes:

-./reports-notes:/reports

-../notes/test:/notesapp/test

Thisgivesusthebestofbothworlds.

Thetestcodeisinnotes/testwhereitbelongs

Thetestcodeisnotcopiedintotheproductioncontainer

Intestmode,thetestdirectoryappearswhereitbelongs

WehaveacoupleofconfigurationfilesremainingfortheSequelizedatabaseconnectiontosetup.

Fortheuserauth-testcontainer,theSEQUELIZE_CONNECTvariablenowreferstoaconfigurationfilethatdoesnotexist,thankstooverridingthevariableinuser/Dockerfile.Let'screatethatfileastest-compose/userauth/sequelize-docker-mysql.yaml,containingthefollowing:

dbname:userauth-test

username:userauth-test

password:userauth-test

params:

host:db-userauth-test

port:3306

dialect:mysql

Thevaluesmatchthevariablespassedtothedb-userauth-testcontainer.Thenwemustensurethisconfigurationfileismountedintotheuserauth-testcontainer:

userauth-test:

...

volumes:

-./reports-userauth:/reports

-./userauth/sequelize-docker-test-mysql.yaml:/userauth/sequelize-

docker-test-mysql.yaml

Fornotes-testwehaveaconfigurationfile,test/sequelize-mysql.yaml,toputinthenotes/testdirectory:

dbname:notes-test

username:notes-test

password:notes12345

params:

host:db-notes-test

port:3306

dialect:mysql

logging:false

Again,thismatchestheconfigurationvariablesindb-notes-test.Intest-compose/docker-compose.yml,wemountthatfileintothecontainer.

ExecutingtestsunderDockerComposeNowwe'rereadytoexecutesomeofthetestsinsideacontainer.We'veusedaDockerComposefiletodescribethetestenvironmentfortheNotesapplication,usingthesamearchitectureasintheproductionenvironment.Thetestscriptsandconfigurationhasbeeninjectedintothecontainers.Thequestionis,howdoweautomatetestexecution?

Thetechniquewe'lluseistorunashellscript,andusedockerexec-ittoexecutecommandstorunthetestscripts.Thisissomewhatautomated,andwithsomemoreworkitcanbefullyautomated.

Intest-compose,let'smakeashellscriptcalledrun.sh(onWindows,run.ps1):

docker-composestop

docker-composebuild

docker-composeup--force-recreate-d

dockerps

dockernetworkls

sleep20

dockerexec-it--workdirnotesapptest-eDEBUG=notes-testnpminstall

dockerexec-it--workdirnotesapptest-eDEBUG=notes-testnpmruntest-

notes-memory

dockerexec-it--workdirnotesapptest-eDEBUG=notes-testnpmruntest-

notes-fs

dockerexec-it--workdirnotesapptest-eDEBUG=notes-testnpmruntest-

notes-levelup

dockerexec-it--workdirnotesapptest-eDEBUG=notes-testnpmruntest-

notes-sqlite3

dockerexec-it--workdirnotesapptest-eDEBUG=notes-testnpmruntest-

notes-sequelize-sqlite

dockerexec-it--workdirnotesapptest-eDEBUG=notes-testnpmruntest-

notes-sequelize-mysql

docker-composestop

It'scommonpracticetoruntestsoutofacontinuousintegrationsystemsuchasJenkins.Continuousintegrationsystemsautomaticallyrunbuildsortestsagainstsoftwareproducts.Thebuildandtestresultsdataisusedtoautomaticallygeneratestatuspages.Visithttps://jenkins.io/index.html,whichisagoodstartingpointforaJenkinsjob.

Thatmakesthefirstrealsteptobuildingthecontainers,followedbybringingthemup.Thescriptsleepsforafewsecondstogivethecontainerstimetofullyinstantiatethemselves.

Thesubsequentcommandsallfollowaparticularpatternthatisimportanttounderstand.Thecommandsareexecutedinthenotesapptestdirectorythankstothe--workdiroption.RememberthatdirectoryisinjectedintothecontainerbytheDockerComposefile.

Using-eDEBUG=we'vedisabledtheDEBUGoptions.Ifthoseoptionsareset,we'dhaveexcessunwantedoutputinthetestresults,sousingthisoptionensuresthatdebuggingoutputdoesn'toccur.

Nowthatyouunderstandtheoptions,youcanseethatthesubsequentcommandsareallexecutedinthetestdirectoryusingthepackage.jsoninthatdirectory.Itstartsbyrunningnpminstall,andthenrunningeachofthescenariosinthetestmatrix.

Torunthetests,simplytype:

$sh-xrun.sh

That'sgood,we'vegotmostofourtestmatrixautomatedandprettywellsquaredaway.ThereisaglaringholeinthetestmatrixandpluggingthatholewillletusseehowtosetupMongoDBunderDocker.

MongoDBsetupunderDockerandtestingNotesagainstMongoDBInChapter7,DataStorageandRetrieval,wedevelopedMongoDBsupportforNotes,andsincethenwe'vefocusedonSequelize.Tomakeupforthatslight,let'smakesureweatleasttestourMongoDBsupport.TestingonMongoDBwouldsimplyrequiredefiningacontainerfortheMongoDBdatabaseandalittlebitofconfiguration.

Visithttps://hub.docker.com/_/mongo/fortheofficialMongoDBcontainer.You'llbeabletoretrofitthistoallowdeployingtheNotesapplicationrunningonMongoDB.

Addthistotest-compose/docker-compose.yml:

db-notes-mongo-test:

image:mongo:3.6-jessie

container_name:db-notes-mongo-test

networks:

-frontnet-test

volumes:

-./db-notes-mongo:/data/db

That'sallthat'srequiredtoaddaMongoDBcontainertoaDockerComposefile.We'veconnectedittofrontnetsothatthenotes(notes-test)containercanaccesstheservice.

Theninnotes/test/package.jsonweaddalinetofacilitaterunningtestsonMongoDB:

"test-notes-mongodb":"MONGO_URL=mongodb://db-notes-mongo-test/

MONGO_DBNAME=chap11-testNOTES_MODEL=mongodbmocha--no-timeoutstest-model"

SimplybyaddingtheMongoDBcontainertofrontnet-test,thedatabaseisavailableattheURLshownhere.Hence,it'ssimpletonowrunthetestsuiteusingtheNotesMongoDBmodel.

The--no-timeoutsoptionwasnecessarytoavoidaspuriouserrorwhiletestingthesuiteagainstMongoDB.ThisoptioninstructsMochatonotcheckwhetheratestcaseexecutiontakestoolong.

Thefinalrequirementistoaddthislineinrun.sh(orrun.ps1forWindows):

dockerexec-it--workdirnotesapptest-eDEBUG=notes-testnpmruntest-

notes-mongodb

That,then,ensuresMongoDBistestedduringeverytestrun.

Wecannowreporttothemanagerthefinaltestresultsmatrix:

models-fs:PASS

models-memory:PASS

models-levelup:PASS

models-sqlite3:Twofailures,nowfixed,PASS

models-sequelizewithSQLite3:PASS

models-sequelizewithMySQL:PASS

models-mongodb:PASS

Themanagerwilltellyou"goodjob"andthenrememberthattheModelsareonlyaportionoftheNotesapplication.We'velefttwoareascompletelyuntested:

TheRESTAPIfortheuserauthenticationservice

Functionaltestingoftheuserinterface

Let'sgetonwiththosetestingareas.

TestingRESTbackendservicesIt'snowtimetoturnourattentiontotheuserauthenticationservice.We'vementionedtestsofthisservice,sayingthatwe'llgettothemlater.Wehaddevelopedsomescriptsforadhoctesting,whichhavebeenusefulallalong.Butlaterisnow,andit'stimetogetcrackingonsomerealtests.

There'saquestionofwhichtooltousefortestingtheauthenticationservice.Mochadoesagoodjoboforganizingaseriesoftestcases,andweshouldreuseithere.ButthethingwehavetotestisaRESTservice.Thecustomerofthisservice,theNotesapplication,usesitthroughtheRESTAPI,givingusaperfectrationalizationtotestattheRESTinterface.OuradhocscriptsusedtheSuperAgentlibrarytosimplifymakingRESTAPIcalls.Therehappenstobeacompanionlibrary,SuperTest,thatismeantforRESTAPItesting.Readitsdocumentationhere:https://www.npmjs.com/package/supertest.

We'vealreadymadethetest-compose/userauthdirectory.Inthatdirectory,createafilenamedtest.js:

'usestrict';

constassert=require('chai').assert;

constrequest=require('supertest')(process.env.URL_USERS_TEST);

constutil=require('util');

consturl=require('url');

constURL=url.URL;

constauthUser='them';

constauthKey='D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF';

describe("UsersTest",function(){

...Testcodefollows

});

ThissetsupMochaandtheSuperTestclient.TheURL_USERS_TESTenvironmentvariablespecifiesthebaseURLoftheservertorunthetestagainst.You'llalmostcertainlybeusinghttp://localhost:3333giventheconfigurationwe'veusedearlierinthebook.SuperTestinitializesitselfalittledifferentlytoSuperAgent.TheSuperTestmoduleexposesafunctionthatwecallwiththeURL_USERS_TESTenvironmentvariable,thenweuseTHATrequestobjectthroughouttherestofthescripttomakeRESTAPIrequests.

Thisvariablewasalreadysetintest-compose/docker-compose.ymlwiththerequiredvalue.TheotherthingofimportanceisapairofvariablestostoretheauthenticationuserIDandkey:

beforeEach(asyncfunction(){

awaitrequest.post('/create-user')

.send({

username:"me",password:"w0rd",provider:"local",

familyName:"Einarrsdottir",givenName:"Ashildr",

middleName:"",

emails:[],photos:[]

})

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth(authUser,authKey);

});

afterEach(asyncfunction(){

awaitrequest.delete('destroyme')

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth(authUser,authKey);

});

Ifyouremember,thebeforeEachfunctionisrunimmediatelybeforeeverytestcase,andafterEachisrunafterward.ThesefunctionsusetheRESTAPItocreateourtestuserbeforerunningthetest,andthenafterwardtodestroythetestuser.Thatwayourtestscanassumethisuserwillexist:

describe("Listuser",function(){

it("listcreatedusers",asyncfunction(){

constres=awaitrequest.get('/list')

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth(authUser,authKey);

assert.exists(res.body);

assert.isArray(res.body);

assert.lengthOf(res.body,1);

assert.deepEqual(res.body[0],{

username:"me",id:"me",provider:"local",

familyName:"Einarrsdottir",givenName:"Ashildr",

middleName:"",

emails:[],photos:[]

});

});

});

Now,wecanturntotestingsomeAPImethods,suchasthe/listoperation.

Wehavealreadyguaranteedthatthereisanaccount,inthebeforeEachmethod,so/listshouldgiveusanarraywithoneentry.

ThisfollowsthegeneralpatternforusingMochatotestaRESTAPImethod.First,weuseSuperTest'srequestobjecttocalltheAPImethod,andawaititsresult.Oncewehavetheresult,weuseassertmethodstovalidateitiswhatisexpected:

describe("finduser",function(){

it("findcreatedusers",asyncfunction(){

constres=awaitrequest.get('findme')

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth(authUser,authKey);

assert.exists(res.body);

assert.isObject(res.body);

assert.deepEqual(res.body,{

username:"me",id:"me",provider:"local",

familyName:"Einarrsdottir",givenName:"Ashildr",

middleName:"",

emails:[],photos:[]

});

});

it("failtofindnonexistentusers",asyncfunction(){

varres;

try{

res=awaitrequest.get('findnonExistentUser')

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth(authUser,authKey);

}catch(e){

return;//Testisokayinthiscase

}

assert.exists(res.body);

assert.isObject(res.body);

assert.deepEqual(res.body,{});

});

});

Wearecheckingthe/findoperationintwoways:

Lookingfortheaccountweknowexists–failureisindicatediftheuseraccountisnotfound

Lookingfortheoneweknowdoesnotexist–failureisindicatedifwereceivesomethingotherthananerrororanemptyobject

Addthistestcase:

describe("deleteuser",function(){

it("deletenonexistentusers",asyncfunction(){

varres;

try{

res=awaitrequest.delete('destroynonExistentUser')

.set('Content-Type','application/json')

.set('Acccept','application/json')

.auth(authUser,authKey);

}catch(e){

return;//Testisokayinthiscase

}

assert.exists(res);

assert.exists(res.error);

assert.notEqual(res.status,200);

});

});

Finally,weshouldcheckthe/destroyoperation.Wealreadycheckthis

operationintheafterEachmethod,wherewedestroyaknownuseraccount.Weneedtoalsoperformthenegativetestandverifyitsbehavioragainstanaccountweknowdoesnotexist.

Thedesiredbehavioristhateitheranerroristhrown,ortheresultshowsanHTTPstatusindicatinganerror.Infact,thecurrentauthenticationservercodegivesa500statuscodealongwithsomeotherinformation.

Intest-compose/docker-compose.yml,weneedtoinjectthisscript,test.js,intotheuserauth-testcontainer.We'lladdthathere:

userauth-test:

...

volumes:

-./reports-userauth:/reports

-./userauth/sequelize-docker-test-mysql.yaml:/userauth/sequelize-docker-

test-mysql.yaml

-./userauth/test.js:/userauth/test.js

Wehaveatestscript,andhaveinjectedthatscriptintothedesiredcontainer(userauth-test).Thenextstepistoautomaterunningthistest.Onewayistoaddthistorun.sh(akarun.ps1onWindows):

dockerexec-it-eDEBUG=userauth-testnpminstallsupertestmochachai

dockerexec-it-eDEBUG=userauth-test./node_modules/.bin/mochatest.js

Now,ifyouruntherun.shtestscriptyou'llseetherequiredpackagesgetinstalled,andthenthistestsuiteexecution.

AutomatingtestresultsreportingIt'scoolwehaveautomatedtestexecution,andMochamakesthetestresultslooknicewithallthosecheckmarks.Whatifthemanagementwantsagraphoftestfailuretrendsovertime?Ortherecouldbeanynumberofreasonstoreporttestresultsasdataratherthanauser-friendlyprintoutontheconsole.

Mochauseswhat'scalledaReportertoreporttestresults.AMochaReporterisamodulethatprintsdatainwhateverformatitsupports.InformationisontheMochawebsite:https://mochajs.org/#reporters.

Youwillfindthecurrentlistofavailablereporterslikeso:

#mocha--reporters

dot-dotmatrix

doc-htmldocumentation

spec-hierarchicalspeclist

json-singlejsonobject

progress-progressbar

list-spec-stylelisting

tap-test-anything-protocol

...

Thenyouuseaspecificreporterlikeso:

#mocha--reportertaptest

1..4

ok1UsersTestListuserlistcreatedusers

ok2UsersTestfinduserfindcreatedusers

ok3UsersTestfinduserfailtofindnonexistentusers

ok4UsersTestdeleteuserdeletenonexistentusers

#tests4

#pass4

#fail0

TestAnythingProtocol(TAP)isawidelyusedtestresultsformat,increasingthepossibilityoffindinghigherlevelreportingtools.Obviously,thenextstepwouldbetosavetheresultsintoafilesomewhere,aftermountingahostdirectoryintothecontainer.

FrontendheadlessbrowsertestingwithPuppeteerAbigcostareaintestingismanualuserinterfacetesting.Therefore,awiderangeoftoolshavebeendevelopedtoautomaterunningtestsattheHTTPlevel.SeleniumisapopulartoolimplementedinJava,forexample.IntheNode.jsworld,wehaveafewinterestingchoices.Thechai-httpplugintoChaiwouldletusinteractattheHTTPlevelwiththeNotesapplication,whilestayingwithinthenow-familiarChaienvironment.

However,forthissection,we'llusePuppeteer(https://github.com/GoogleChrome/puppeteer).Thistoolisahigh-levelNode.jsmoduletocontrolaheadlessChromeorChromiumbrowser,usingtheDevToolsprotocol.Thatprotocolallowstoolstoinstrument,inspect,debug,andprofileChromiumorChrome.

Puppeteerismeanttobeageneralpurposetestautomationtool,andhasastrongfeaturesetforthatpurpose.Becauseit'seasytomakewebpagescreenshotswithPuppeteer,itcanalsobeusedinascreenshotservice.

BecausePuppeteeriscontrollingarealwebbrowser,youruserinterfacetestswillbeveryclosetolivebrowsertestingwithouthavingtohireahumantodothework.BecauseitusesaheadlessversionofChrome,novisiblebrowserwindowwillshowonyourscreen,andtestscanberuninthebackground,instead.AdownsidetothisattractivestoryisthatPuppeteeronlyworksagainstChrome.MeaningthatanautomatedtestagainstChromedoesnottestyourapplicationagainstotherbrowsers,suchasOperaorFirefox.

SettingupPuppeteerLet'sfirstsetupthedirectoryandinstallthepackages:

$mkdirtest-compose/notesui

$cdtest-compose/notesui

$npminit

...answerthequestions

$npminstallpuppeteer@1.1.xmocha@5.xchai@4.1.x--save

Duringinstallation,you'llseethatPuppeteercausesthedownloadofChromiumlikeso:

DownloadingChromiumr497674-92.5Mb[====================]100%0.0s

ThepuppeteermodulewilllaunchthatChromiuminstanceasneeded,managingitasabackgroundprocess,andcommunicatingwithitusingtheDevToolsprotocol.

Inthescriptwe'reabouttowrite,weneedauseraccountthatwecanusetologinandperformsomeactions.Fortunately,wealreadyhaveascripttosetupatestaccount.Inusers/package.json,addthislinetothescriptssection:"setupuser":"PORT=3333nodeusers-add",

We'reabouttowritethistestscript,butlet'sfinishthesetup,thefinalbitofwhichisaddingtheselinestorun.sh:

dockerexec-ituserauth-testnpmrunsetupuser

dockerexec-itnotesapp-testnpmruntest-docker-ui

Whenexecuted,thesetwolinesensurethatthetestuserissetup,anditthenrunstheuserinterfacetests.

ImprovingtestabilityintheNotesUIWhiletheNotesapplicationdisplayswellinthebrowser,howdowewritetestsoftwaretodistinguishonepagefromanother?Thekeyrequirementisfortestscriptstoinspectthepage,determinewhichpageisbeingdisplayed,andreadthedataonthepage.ThatmeanseachHTMLelementmustbeeasilyaddressableusingaCSSselector.

WhiledevelopingtheNotesapplication,weforgottodothat,andtheSoftwareQualityEngineering(SQE)managerhasrequestedourassistance.Atstakeisthetestingbudget,whichwillbestretchedfurtherthemoretheSQEteamcanautomatetheirtests.

Allthat'snecessaryistoaddafewidorclassattributestoHTMLelementstoimprovetestability.Withafewidentifiers,andacommitmenttomaintainthoseidentifiers,theSQEteamcanwriterepeatabletestscriptstovalidatetheapplication.

Innotes/partials/header.hbs,changetheselines:

...

<aid="btnGoHome"class="navbar-brand"href='/'>

...

{{#ifuser}}

...

<aclass="nav-itemnav-linkbtnbtn-darkcol-auto"id="btnLogout"

href="userslogout">...</a>

<aclass="nav-itemnav-linkbtnbtn-darkcol-auto"id="btnAddNote"

href='notesadd'>...</a>

{{else}}

...

<aclass="nav-itemnav-linkbtnbtn-darkcol-auto"id="btnloginlocal"

href="userslogin">..</a>

<aclass="nav-itemnav-linkbtnbtn-darkcol-auto"

id="btnLoginTwitter"href="usersauth/twitter">...</a>

...

{{/if}}

...

Innotes/views/index.hbs,makethesechanges:

<divid="notesHomePage"class="container-fluid">

<divclass="row">

<divid="notetitles"class="col-12btn-group-vertical"role="group">

{{#eachnotelist}}

<aid="{{key}}"class="btnbtn-lgbtn-blockbtn-outline-dark"

href="notesview?key={{key}}">...</a>

{{/each}}

</div>

</div>

</div>

Innotes/views/login.hbs,makethesechanges:

<divid="notesLoginPage"class="container-fluid">

...

<formid="notesLoginForm"method='POST'action='userslogin'>

...

<buttontype="submit"id="formLoginBtn"class="btnbtn-

default">Submit</button>

</form>

...

</div>

Innotes/views/notedestroy.hbs,makethesechanges:

<formid="formDestroyNote"method='POST'action='notesdestroy/confirm'>

...

<buttonid="btnConfirmDeleteNote"type="submit"value='DELETE'

class="btnbtn-outline-dark">DELETE</button>

...

</form>

Innotes/views/noteedit.hbs,makethesechanges:

<formid="formAddEditNote"method='POST'action='notessave'>

...

<buttonid='btnSave'type="submit"class="btnbtn-default">Submit</button>

...

</form>

Innotes/views/noteview.hbs,makethesechanges:

<divid="noteView"class="container-fluid">

...

<pid="showKey">Key:{{notekey}}</p>

...

<aid="btnDestroyNote"class="btnbtn-outline-dark"

href="notesdestroy?key={{notekey}}"role="button">...</a>

<aid="btnEditNote"class="btnbtn-outline-dark"

href="notesedit?key={{notekey}}"role="button">...</a>

<buttonid="btnComment"type="button"class="btnbtn-outline-dark"

data-toggle="modal"data-target="#notes-comment-modal">...</button>

...

</div>

Whatwe'vedoneisaddid=attributestoselectedelementsinthetemplates.WecannoweasilywriteCSSselectorstoaddressanyelement.TheengineeringteamcanalsostartusingtheseselectorsinUIcode.

PuppeteertestscriptforNotesIntest-compose/notesui,createafilenameduitest.jscontainingthefollowing:

constpuppeteer=require('puppeteer');

constassert=require('chai').assert;

constutil=require('util');

const{URL}=require('url');

describe('Notes',function(){

this.timeout(10000);

letbrowser;

letpage;

before(asyncfunction(){

browser=awaitpuppeteer.launch({slomo:500});

page=awaitbrowser.newPage();

awaitpage.goto(process.env.NOTES_HOME_URL);

});

after(asyncfunction(){

awaitpage.close();

awaitbrowser.close();

});

});

ThisisthestartofaMochatestsuite.Inthebeforefunction,wesetupPuppeteerbylaunchingaPuppeteerinstance,startinganewPageobject,andtellingthatPagetogototheNotesapplicationhomepage.ThatURLispassedinusingthenamedenvironmentvariable.

It'susefultofirstthinkaboutscenarioswemightwanttoverifywiththeNotesapplications:

LogintotheNotesapplication

Addanotetotheapplication

Viewanaddednote

Deleteanaddednote

Logout

Andsoon

Here'scodeforanimplementationoftheLoginscenario:

describe('Login',function(){

before(asyncfunction(){...});

it('shouldclickonloginbutton',asyncfunction(){

constbtnLogin=awaitpage.waitForSelector('#btnloginlocal');

awaitbtnLogin.click();

});

it('shouldfillinloginform',asyncfunction(){

constloginForm=awaitpage.waitForSelector('#notesLoginPage

#notesLoginForm');

awaitpage.type('#notesLoginForm#username',"me");

awaitpage.type('#notesLoginForm#password',"w0rd");

awaitpage.click('#formLoginBtn');

});

it('shouldreturntohomepage',asyncfunction(){

consthome=awaitpage.waitForSelector('#notesHomePage');

constbtnLogout=awaitpage.waitForSelector('#btnLogout');

constbtnAddNote=awaitpage.$('#btnAddNote');

assert.exists(btnAddNote);

});

after(asyncfunction(){...});

});

ThistestsequencehandlestheLoginScenario.ItshowsyouafewofthePuppeteerAPImethods.DocumentationofthefullAPIisathttps://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md.ThePageobjectencapsulatestheequivalentofabrowsertabinChrome/Chromium.

ThewaitForSelectorfunctiondoeswhatitsays–itwaitsuntilanHTMLelementmatchingtheCSSselectorappears,anditwillwaitoveroneor

morepagerefreshes.Thereareseveralvariantsofthisfunctiontoallowwaitingforseveralkindsofthings.ThisfunctionreturnsaPromise,makingitworthourtimetouseasyncfunctionsinourtestcode.ThePromisewillresolvetoanElementHandle,whichisawrapperaroundanHTMLelement,orelsethrowanexception,whichwouldconvenientlymakethetestfail.

Thenamedelement,#btnloginlocal,isinpartials/header.hbs,andwillshowuponlywhenauserisnotloggedin.Hence,wewillhavedeterminedthatthebrowseriscurrentlydisplayingaNotespage,andthatitisnotloggedin.

Theclickmethoddoeswhatitsuggests,andcausesamousebuttonclickonthereferencedHTMLelement.Ifyouwanttoemulateatap,suchasforamobiledevice,thereisatapmethodforthatpurpose.

Thenextstageofthetestsequencepicksupfromthatclick.ThebrowsershouldhavegonetotheLoginpage,andthereforethisCSSselectorshouldbecomevalid:#notesLoginPage#notesLoginForm.WhatwedonextistypetextforourtestuserIDandpasswordintothecorrespondingformelements,andthenclickontheLogInbutton.

Thenextteststagepicksupfromthere,andthebrowsershouldbeonthehomepageasdeterminedbythisCSSselector:#notesHomePage.Ifwewereloggedinsuccessfully,thepageshouldhaveLogOut(#btnLogout)andADDNotebuttons(#btnAddNote).

Inthiscase,we'veusedadifferentfunction,$,tocheckiftheADDNotebuttonexists.Unlikethewaitfunctions,$simplyqueriesthecurrentpagewithoutwaiting.IfthenamedCSSSelectorisnotinthecurrentpage,itsimplyreturnsnullratherthanthrowinganexception.Therefore,todeterminethattheelementexists,weuseassert.existsratherthanrelyingonthethrownexception.

RunningtheloginscenarioNowthatwehaveonetestscenarioentered,let'sgiveitawhirl.Inonewindow,starttheNotestestinfrastructure:

$cdtest-compose

$docker-composeup--force-rebuild

Theninanotherwindow:

$dockerexec-ituserauthbash

userauth#PORT=3333node./users-add.js

userauth#exit

$cdtest-compare/notesui

$NOTES_HOME_URL=http://localhost:3000mocha--no-timeoutsuitest.js

Notes

Login

√shouldclickonloginbutton

√shouldfillinloginform(72ms)

√shouldreturntohomepage(1493ms)

3passing(3s)

TheNOTES_HOME_URLvariableiswhatthescriptlooksfortodirecttheChromiumbrowsertousetheNotesapplication.Torunthetests,weshoulduseDockerComposetolaunchthetestinfrastructure,andthenensurethetestuserisinstalledintheuserdatabase.

TheAddNotescenarioAddthistouitest.js:

describe('AddNote',function(){

//before(asyncfunction(){...});

it('shouldseeAddNotebutton',asyncfunction(){

constbtnAddNote=awaitpage.waitForSelector('#btnAddNote');

awaitbtnAddNote.click();

});

it('shouldfillinAddNoteform',asyncfunction(){

constformAddEditNote=await

page.waitForSelector('#formAddEditNote');

awaitpage.type('#notekey','key42');

awaitpage.type('#title','Hello,world!');

awaitpage.type('#body','Loremipsumdolor');

awaitpage.click('#btnSave');

});

it('shouldviewnote',asyncfunction(){

awaitpage.waitForSelector('#noteView');

constshownKey=awaitpage.$eval('#showKey',el=>

el.innerText);

assert.exists(shownKey);

assert.isString(shownKey);

assert.include(shownKey,'key42');

constshownTitle=awaitpage.$eval('#notetitle',el=>

el.innerText);

assert.exists(shownTitle);

assert.isString(shownTitle);

assert.include(shownTitle,'Hello,world!');

constshownBody=awaitpage.$eval('#notebody',el=>

el.innerText);

assert.exists(shownBody);

assert.isString(shownBody);

assert.include(shownBody,'Loremipsumdolor');

});

it('shouldgotohomepage',asyncfunction(){

awaitpage.waitForSelector('#btnGoHome');

awaitpage.goto(process.env.NOTES_HOME_URL);

//awaitpage.click('#btnGoHome');

awaitpage.waitForSelector('#notesHomePage');

consttitles=awaitpage.$('#notetitles');

assert.exists(titles);

constkey42=awaitpage.$('#key42');

assert.exists(key42);

constbtnLogout=awaitpage.$('#btnLogout');

assert.exists(btnLogout);

constbtnAddNote=awaitpage.$('#btnAddNote');

assert.exists(btnAddNote);

});

//after(asyncfunction(){...});

});

Thisisamoreinvolvedscenario,inwhichwe:

ClickontheADDNotebutton

Waitforthenoteeditscreentoshowup

FillinthetextforthenoteandclicktheSavebutton

Validatethenoteviewpagetoensurethat'scorrect

Validatethehomepagetoensurethat'scorrect.

MostofthisisusingthesamePuppeteerfunctionsasbefore,butwithacoupleofadditions.

The$evalfunctionlooksfortheelementmatchingtheCSSselector,andinvokesthecallbackfunctiononthatelement.Ifnoelementisfoundanerroristhrowninstead.Asusedhere,weareretrievingthetextfromcertainelementsonthescreen,andvalidatingthatitmatcheswhatthetestenteredasthenote.That'sanend-to-endtestofaddingandretrievingnotes.

Thenextdifferenceisusinggotoinsteadofclickingon#btnGoHome.

Asyouaddtestscenariostothetestscript,you'llfinditeasyforPuppeteertohaveaspurioustimeout,orfortheloginprocesstomysteriouslynotwork,orotherspuriouserrors.

Ratherthangoovertheremainingscenarios,we'llspendthenextsectiondiscussinghowtomitigatesuchissues.Butfirstweneedtoprovethescenariodoesworkevenifwehavetorunthetest10timestogetthisresult:

$NOTES_HOME_URL=http://localhost:3000./node_modules/.bin/mocha--no-

timeoutsuitest3.js

Notes

Login

√shouldclickonloginbutton(50ms)

√shouldfillinloginform(160ms)

√shouldreturntohomepage(281ms)

AddNote

√shouldseeAddNotebutton

√shouldfillinAddNoteform(1843ms)

√shouldviewnote

√shouldgotohomepage(871ms)

7passing(5s)

Mitigating/preventingspurioustesterrorsinPuppeteerscriptsThegoalistofullyautomatethetestrun,inordertoavoidhavingtohireahumantobabysitthetestexecutionandspendtimererunningtestsbecauseofspuriouserrors.Todoso,thetestsneedtoberepeatablewithoutanyspuriouserrors.Puppeteerisacomplexsystem–thereisaNode.jsmodulecommunicatingwithaChromiuminstancerunningHeadlessinthebackground–anditseemseasyfortimingissuestocauseaspuriouserror.

ConfiguringtimeoutsBothMochaandPuppeteerallowyoutosettimeoutvalues,andalongtimeoutvaluecanavoidtriggeringanerror,ifsomeactionsimplyrequiresalongtimetorun.Atthetopofthetestsuite,weusedthisMochafunction:

this.timeout(10000);

Thatgives10secondsforeverytestcase.Ifyouwanttousealongertimeout,increasethatnumber.

Thepuppeteer.launchfunctioncantakeatimeoutvalueinitsoptionsobject.Bydefault,Puppeteerusesa30-secondtimeoutonmostoperations,andtheyalltakeanoptionsobjectwithasettingtochangethattimeoutperiod.Inthiscase,we'veaddedtheslowMooptiontoslowdownoperationsonthebrowser.

TracingeventsonthePageandthePuppeteerinstanceAnotherusefultacticistogenerateatraceofwhathappenedsoyoucanpuzzleaway.Insertingconsole.logstatementsistediousandmakesyourcodelookalittleugly.Puppeteeroffersacoupleofmethodstotracetheactionsandtodynamicallyturnofftracing.

Inuitest.js,addthiscode:functionframeEvent(evtname,frame){console.log(`${evtname}${frame.url()}${frame.title()}`);}

functionignoreURL(url){if(url.match(/\/assets\//)===null&&url.match(/\/socket.io\//)===null&&url.match(/fonts.gstatic.com/)===null&&url.match(/fonts.googleapis.com/)===null){returnfalse;}else{returntrue;}}...before(asyncfunction(){browser=awaitpuppeteer.launch({slomo:500});page=awaitbrowser.newPage();page.on('console',msg=>{console.log(`${msg.type()}${msg.text()}${msg.args().join('')}`);});page.on('error',err=>{console.error(`pageERROR${err.stack}`);

});page.on('pageerror',err=>{console.error(`pagePAGEERROR${err.stack}`);});page.on('request',req=>{if(ignoreURL(req.url()))return;console.log(`pagerequest${req.method()}${req.url()}`);});page.on('response',async(res)=>{if(ignoreURL(res.url()))return;console.log(`pageresponse${res.status()}${res.url()}`);});page.on('frameattached',async(frame)=>frameEvent('frameattached',awaitframe));page.on('framedetached',async(frame)=>frameEvent('framedetached',awaitframe));page.on('framenavigated',async(frame)=>frameEvent('framenavigated',awaitframe));awaitpage.goto(process.env.NOTES_HOME_URL);});...

Thatis,thePageobjectoffersseveraleventlistenersinwhichwecanoutputdetailsaboutvariousevents,includingHTTPrequestsandresponses.WecanevenprintouttheHTMLtextoftheresponse.TheignoreURLfunctionletsussuppressafewselectURLssowe'renotinundatedwithunimportantrequestsandresponses.

YoucantracePuppeteeritselfusingitsDEBUGenvironmentvariable.SeetheREADMEformoreinformation:https://github.com/GoogleChrome/puppeteer.

InsertingpausesItcanbeusefultoinsertalongpauseatcertainpointstogivethebrowsertimetodosomething.Trythisfunction:functionwaitFor(timeToWait){returnnewPromise(resolve=>{setTimeout(()=>{resolve(true);},timeToWait);});};

ThisishowweimplementtheequivalentofasleepfunctionusingPromises.UsingsetTimeOutthisway,alongwithatimeoutvalue,simplycausesadelayforthegivennumberofmilliseconds.

Tousethisfunction,simplyinsertthisintothetestscenarios:

awaitwaitFor(3000);

Avariantonthisistowaitforthingstofullyrenderinthebrowser.Forexample,youmighthaveseenapausebeforetheHomeiconintheupper-leftcornerfullyrenders.Thatpausecancausespuriouserrors,andthisfunctioncanwaituntilthatbuttonfullyrendersitself:asyncfunctionwaitForBtnGoHome(){returnpage.waitForSelector('#btnGoHome');}

Touseit:

awaitwaitForBtnGoHome();

Ifyoudon'twanttomaintainthisextrafunction,it'seasyenoughtoaddthewaitForSelectorcallintoyourtestcasesinstead.

AvoidingWebSocketsconflictsAnerror,Cannotfindcontextwithspecifiedidundefined,canbethrownbyPuppeteer.AccordingtoanissueinthePuppeteerissuequeue,thiscanarisefromunplannedinteractionsbetweenPuppeteerandWebSockets:https://github.com/GoogleChrome/puppeteer/issues/1325ThisissueinturnaffectstheSocket.IOsupportintheNotesapplication,andthereforeitmaybeusefultodisableSocket.IOsupportduringtestruns.

It'sfairlysimpletoallowdisablingofSocket.IO.Inapp.mjs,addthisexportedfunction:

exportfunctionenableSocketio(){

varret=true;

constenv=process.env.NOTES_DISABLE_SOCKETIO;

if(!env||env!=='true'){

ret=true;

}

returnret;

}

Thislooksforanenvironmentvariabletocausethefunctiontoreturntrueorfalse.

Inroutes/index.mjsandroutes/notes.mjs,addthisline:

import{enableSocketio,sessionCookieName}from'../app';

Wedothistoimporttheprecedingfunction.ItalsodemonstratessomeoftheflexibilitywegetfromES6Modules,becausewecanimportjusttherequiredfunctions.

Inroutes/index.mjsandroutes/notes.mjs,foreveryrouterfunctionthatcallsres.rendertosendresults,usetheenableSocketiofunctionasso:

res.render('view-name',{

...

enableSocketio:enableSocketio()

});

Hence,we'veimportedthefunctionandforeveryviewwepassenableSocketioasdatatotheviewtemplate.

Inviews/index.hbsandviews/noteview.hbs,wehaveasectionofJavaScriptcodetoimplementSocketIO-basedsemi-real-timefeatures.Surroundeachsuchsectionlikeso:

{{#ifenableSocketio}}

...JavaScriptcodeforSocketIOsupport

{{/if}}

Byeliminatingtheclient-sideSocketIOcode,weensuretheuserinterfacedoesnotopenaconnectiontotheSocketIOservice.ThepointofthisexercisewastoavoidusingWebSocketstoavoidissueswithPuppeteer.

Similarly,inviews/noteview.hbssupportdisablingtheCommentbuttonlikeso:

{{#ifenableSocketio}}

<buttonid="btnComment"type="button"class="btnbtn-outline-dark"

data-toggle="modal"data-target="#notes-comment-

modal">Comment</button>

{{/if}}

Thefinalstepwouldbetosettheenvironmentvariable,NOTES_DISABLE_SOCKETIO,intheDockerComposefile.

TakingscreenshotsOneofPuppeteer'scorefeaturesistotakescreenshots,eitherasPNGorPDFfiles.Inourtestscripts,wecantakescreenshotstotrackwhatwasonthescreenatanygiventimeduringthetest.Forexample,iftheLoginscenariospuriouslyfailstologin,wecanseethatinthescreenshots:awaitpage.screenshot({type:'png',path:`./screen/login-01-start.png`});

Simplyaddcodesnippetslikethisthroughoutyourtestscript.Thefilenameshownherefollowsaconventionwherethefirstsegmentnamesthetestscenario,thenumberisasequencenumberwithinthetestscenario,andthelastdescribesthestepwithinthetestscenario.

Takingscreenshotsalsoprovidesanotherstageofvalidation.Youmaywanttodovisualvalidationofyourapplicationaswell.ThepixelmatchmodulecancomparetwoPNGfiles,andthereforeasetofso-calledGoldenImagescanbemaintainedforcomparisonduringtestruns.

ForanexampleofusingPuppeteerthisway,see:https://meowni.ca/posts/2017-puppeteer-tests/.

SummaryWe'vecoveredalotofterritoryinthischapter,lookingatthreedistinctareasoftesting:unittesting,RESTAPItesting,andUIfunctionaltests.Ensuringthatanapplicationiswelltestedisanimportantstepontheroadtosoftwaresuccess.Ateamthatdoesnotfollowgoodtestingpracticesisoftenboggeddownwithfixingregressionafterregression.

We'vetalkedaboutthepotentialsimplicityofsimplyusingtheassertmodulefortesting.Whilethetestframeworks,suchasMocha,providegreatfeatures,wecangoalongwaywithasimplescript.

Thereisaplacefortestframeworks,suchasMocha,ifonlytoregularizeourtestcases,andtoproducetestresultsreports.WeusedMochaandChaiforthis,andthesetoolswerequitesuccessful.Weevenfoundacoupleofbugswithasmalltestsuite.

Whenstartingdowntheunittestingroad,onedesignconsiderationismockingoutdependencies.Butit'snotalwaysagooduseofourtimetoreplaceeverydependencywithamockversion.

Toeasetheadministrativeburdenofrunningtests,weusedDockertoautomatesettingupandtearingdownthetestinfrastructure.JustasDockerwasusefulinautomatingdeploymentoftheNotesapplication,it'salsousefulinautomatingtestinfrastructuredeployment.

Finally,wewereabletotesttheNoteswebuserinterfaceinarealwebbrowser.Wecan'ttrustthatunittestingwillfindeverybug;somebugswillonlyshowupinthewebbrowser.Evenso,we'veonlytouchedthebeginningofwhatcouldbetestedinNotes.

Inthisbook,we'vecoveredthegamutofNode.jsdevelopment,givingyouastrongfoundationfromwhichtostartdevelopingNode.jsapplications.

Inthenextchapter,we'llexploreanothercriticalarea–security.We'llstartbyusingHTTPStoencryptandauthenticateuseraccesstoNotes.And,we'lluseseveralNode.jspackagestolimitthechanceofsecurityintrusions.

SecurityWe'recomingtotheendofthisjourneyoflearningNode.js.Butthereisoneimportanttopiclefttodiscuss:security.

Cybersecurityofficialsaroundtheworldhavebeenclamoringforgreatersecurityontheinternet.Insomecases,vastbotnetshavebeenbuilt,thankstoweaksecurityimplementation,whichareweaponizedtobludgeonwebsitesorcommitothermayhem.Inothercases,rampantidentitytheftfromsecurityintrusionsareafinancialthreattousall.Almosteveryday,thenewsincludesmorerevelationsofcybersecurityproblems.

In2016,theUS-CERTissuedseveralwarningsofvulnerabilitiesinInternetofThings(IoT)devices,suchassecuritycamerasorWi-Firouters.Byexploitingvulnerabilitiesinthedevices,attackerswereabletoinjectattacksoftwareintothesedevices.TheresultwasaslavedbotnetofhundredsofthousandsofIoTdevices,whichweredeployedtosendmassiveDistributedDenialofService(DDOS)attacksagainstspecificwebsites.

Companieslargeandsmallhavesufferedsecuritybreaches.Theattackersgenerallymakeoffwithuseridentityinformation,whichiswhy,inChapter10,DeployingNode.jsApplications,wewerecarefultosegmenttheuserdatabaseintoanisolatedcontainer.Themorelayersofsecurityyouputaroundcriticalsystems,thelesslikelyattackerscangetin.

Generallyspeaking,theinternethastransitionedfrombeingtheexperimentalplaygroundofcomputerresearchersinthe1980stobecomingacentralfacetofglobalsociety.Allkindsofcriticalactivitiesarebeingconductedovertheinternet.

Forexample,theelectricalutilitiesandelectricgridoperatorsareresearchingthetransitionofelectricitygridcontrolfromprivatenetworks

totheinternet.AninfluencingcauseistheincreaseinInternetofThings-baseddevicesfordistributedenergyresources,suchassmartthermostats,smartlighting,solararrays,andenergystoragedevices.Theindustryisrevisitingthecommunicationsprotocolsusedforcontrollingthesesystems,andmovingtowardsinternettechnologies.ThisincludestheIEEE2030.5standardusingHTTPSwithNSA-gradeencryption,securedwithspecificallyconstructeddigitalcertificatesidentifyingdevicesandservices,alongwithREST-basedcommunicationprotocols.

Becauseofthecriticalroletheinternetnowplaysinallourlives,itisimportantforallsoftwaredeveloperstoaddresssecurityissuesintheirproducts.

Securityshouldn'tbeanafterthought,justastestingshouldnotbeanafterthought.Bothareincrediblyimportant,ifonlytokeepyourcompanyfromgettinginthenewsforthewrongreasons.

EventhoughtheNotesapplicationisasimpletoyapplication,we'vebeenusingittoexploreproductiondeploymentissues.Let'snowturntousingNotestoexploretheimplementationofgoodsecuritypractices.Wewillcoverthefollowingtopics:

ImplementingHTTPS/SSLinanExpressapplication

AutomationofSSLcertificaterenewalwithLet'sEncrypt

UsingtheHelmetlibrarytoimplementheadersforContentSecurityPolicy,DNSPrefetchControl,FrameOptions,StrictTransportSecurity,andmitigatingXSSattacks

Preventingcross-siterequestforgeryattacksagainstforms

SQLInjectionattacks

Pre-deploymentscanningforpackageswithknownvulnerabilities

Ifyouhaven'tyetdoneso,duplicatetheChapter11,UnitTestingandFunctionalTestingsourcetree,whichyoumayhavecalledchap11,tomakeaChapter12,Securitysourcetree,whichyoucancallchap12.

Expresshasanexcellentsecurityresourcepageathttps://expressjs.com/en/advanced/best-practice-security.html.

HTTPS/TLS/SSLusingLet'sEncryptSecuringyourwebsiteusingHTTPSisbecomingincreasinglyimportant.ThebrowsermakershavestartedissuingwarningsforHTTP-onlywebsites,forexample,andthesearchenginesaredowngradingsuchsitesinsearchrankings.Privacyconcernsdictatetheencryptionofalltrafficsentovertheinternet.Phishingattacksluringvictimstofakewebsitesfilledwithmalwaredictatewehaveamechanismtorobustlyidentifywebsiteownership.

HTTPSissimplyHTTPrunthroughTLS/SSL,whichisaninternetprotocolforencryptedconnections.TheencryptionkeysarestoredinatrustedPublicKeyInfrastructure(PKI)withencryptedcertificatesforvalidatingwebsites.Withacorrectlyissuedcertificate,HTTPSvalidatesthedomainsothatourusershavesomeassurancethey'vevisitedavalidwebsite,andthattheirdatatransfersareencryptedtoprevent(casual)eavesdropping.Thatgreenbuttoninthebrowserlocationbarismeanttoreassureourvisitorsthatthey'resafe.

Foryears,acquiringtherequiredSSLcertificateswasamanualprocessrequiringsignificantfeepaymentstoaCertificateAuthoritycompany.CertificateAuthorities(CA)arepartofthewholePublicKeyInfrastructure(PKI)behindtheSSLcertificatesusedbytheHTTPSprotocol.TheInternetPKIusesahierarchyofCAs,withhigherlevelCAscertifyingtheend-userCAs.HoweverdesirableitisforeverywebsitetobeprotectedwithHTTPS,themanualprocess,andthecostofacquiringSSLcertificates,meantfolkswerediscouragedfromdeployingHTTPSwebsites.Theinternetwasthereforemuchlesssecurethanitcouldhavebeen.

ThatchangedwiththeadventofLet'sEncrypt,anon-profitorganization

offeringfreeSSLcertificates.Mostimportantly,theprocessiscompletelyautomatedandeasytosetupanduse.ThereisnownoreasontonothaveHTTPSsupportout-of-the-box.

Whatwe'reabouttodoistoimplementHTTPSusingtheLet'sEncrypttoolsintheDockerinfrastructurefortheNotesapplication.Thisrequiresafewsteps,noneofwhicharedifficult:

DeployingNotestoacloudserveraswedidinChapter10,DeployingNode.jsApplications,withtheadditionofassociatingarealdomainnamewiththedeployment

Addinganewcontainercontainingcertbot,thecommand-linetoolforLet'sEncrypt,anduseittomanageregisteringandrenewingSSLcertificates

ConfiguringtheDockerinfrastructuretocross-mountdirectoriesfromthecertbotcontainertothenotescontainer,sothatNoteshastheSSLcertificates

ImplementtheHTTPServerobjectinNotes

Let'sgetstarted.

AssociatingadomainnamewithDocker-basedcloudhostingLet'sstartbydeployingourNotesapplicationtocloudhostingaswedidinChapter10,DeployingNode.jsApplications,butwithafewchanges.

Onechangeistousearealdomainnamethistime.Thisrequiresgoingtoadomainnameregistrarsuchaspairdomains.comtoregisteradomain.Someweb-hostingprovidersalsoofferdomainnameregistration,howeverit'sgenerallybetterifyourdomainnameisregisteredseparatelyfromthehostingprovider.Goaheadandregisteradomain.fooblebartz.comseemstobeavailable,forexample.

ThenextstepistoassociatethedomainnamewithavirtualservertowhichwecandeployDockercontainers.Again,we'llturntoDigitalOceanasanexampleserviceforhostingtheapplication,andshowhowtoconfigureadomainname.

Theexactmethodtoassociateadomainnamewillvarydependingonthehostingprovider.Typically,thehostingproviderwillaskyoutoassigntheNSrecordsforthedomaintolisttheDNSserversoperatedbythehostingprovider.SuchhostingproviderswillgiveyoualistofNSserverhostnames.OncetheNSrecordsareassignedtothehostingprovider,youcanusethehostingprovider'sdashboardtoconfigureyourdomain.

IntheDigitalOceandashboard,clickonNetworkingandthenDomains.Inthatpanel,youcanenterthedomainnameyou'veregistered:

ClickontheAddDomainbuttonandthedashboardwilltransitionyoutoanewscreeninstructingyoutoconfigurethedomainnamewiththreeNSrecords.YoumustthencopythoseNSrecordstothedomainregistrarwebsite,whereyou'llenterthemlikeso:

Now,wemustcreateaDockerhostonDigitalOcean(oryourchosencloud-hostingprovider)likeso:

$docker-machinecreate--driverdigitalocean\

--digitalocean-size2gb\

--digitalocean-access-tokenDIGITALOCEAN-API-TOKEN\

notes-https

...

$eval$(docker-machineenvnotes-https)

Thisgivesusavirtualserver,whichwecaninspectintheDigitalOcean

dashboard.IfyouthennavigatetotheNetworking/Domainspanel,thedomaincanbeassociatedwiththeserver:

Shortly,yourdomainnamewillbeproperlyassociatedwiththeserver:

$pingevsoul.com

PINGevsoul.com(159.65.179.28)56(84)bytesofdata.

64bytesfrom159.65.179.28(159.65.179.28):icmp_seq=1ttl=49time=83.0ms

64bytesfrom159.65.179.28(159.65.179.28):icmp_seq=2ttl=49time=85.1ms

^C

---evsoul.compingstatistics---

2packetstransmitted,2received,0%packetloss,time3001ms

rttmin/avg/max/mdev=83.050/83.655/85.184/0.930ms

$dig-tanyevsoul.com

;<<>>DiG9.10.3-P4-Ubuntu<<>>-tanyevsoul.com

...

;;ANSWERSECTION:

evsoul.com.1800INSOAns1.digitalocean.com.hostmaster.evsoul.com.

15190921201080036006048001800

evsoul.com.3502INA159.65.179.28

evsoul.com.1800INNSns1.digitalocean.com.

evsoul.com.1800INNSns2.digitalocean.com.

evsoul.com.1800INNSns3.digitalocean.com.

...

ButbecausenothingontheserverwillanswertoanyHTTPport,itcannotbevisitedbyawebbrowser.Withthecommandshellonourlaptopstillassociatedwiththevirtualserver,let'sdeployNotestotheserversowecanhavevisitors:

$docker-composebuild

...

$docker-composeup--force-recreate-d

Oncethat'sdone,youcanvisittheNotesapplicationviayourdomainname.

TherearethreethingstodotoenableloggingintotheNotesservice.Oneistoruntheusers-addscriptintheuserauthcontainer,asyouwillhavedonesomanytimesalready.Theotheristomakethischangeinnotes/Dockerfile:

ENVTWITTER_CALLBACK_HOST=http://evsoul.com

Substituteyourchosendomainnamehere.ThischangeenablesusingTwittertologintotheNotesapplication.

Thelastchangeisincompose/docker-compose.yml:

notes:

build:../notes

container_name:notes

depends_on:

-db-notes

networks:

-frontnet

ports:

-"80:3000"

restart:always

environment:

-NODE_ENV="production"

Inotherwords,it'stimeforNotestobeonport80likeanyHTTPservice.

Atthispoint,youwillbeabletoviewtheNotesapplicationusing

http://DOMAIN,andtologinusingeitherthelocalusername,orusingTwitter.Otherthanaddingthedomainname,thisisexactlywhatwehadinChapter10,DeployingNode.jsApplications.

What'snextisto:

UseLet'sEncrypttosetupSSLcertificates

ModifyNotestousethosecertificates

RedirectHTTPtraffictotheNotesHTTPSport

EnsurewecandeploytoHTTPorHTTPSasneeded,sincewedon'tneedHTTPonthedevelopers'laptops

ADockercontainertomanageLet'sEncryptSSLcertificatesYouacquireanSSLcertificatefromLet'sEncryptusinganACMEclient.ACMEisaprotocolinventedconcurrentlywiththeLet'sEncryptserviceforfetchingSSLcertificatesfromaprovider.TheprimaryACMEclientisCertbot,acommand-linetoolthathelpsyouregisteradomainwithLet'sEncrypt,andwhichautomatesrenewalofLet'sEncryptSSLcertificates.ForCertbotdocumentation,seehttps://certbot.eff.org/.

Thecontainerwe'reabouttoimplementissetuptomakeiteasytoregisteradomainwithLet'sEncrypt,andthentoautomatecertificaterenewal.It'saverysimplecontainer,consistingofacrondaemon,acrontabentry,andthecertbottool.TheonlyCPUconsumptionoccursaboutonceadaywhenthecronjobrunstoattemptcertificaterenewal.

Createadirectory,certbot,andwithinthatdirectoryaDockerfile:

FROMdebian:jessie

#Installcron,certbot,bash,plusanyotherdependencies

RUNapt-getupdate&&apt-getinstall-ycronbashwget

RUNmkdir-pwebrootsevsoul.com/.well-known&&mkdir-p/scripts

WORKDIR/scripts

RUNwgethttps://dl.eff.org/certbot-auto

RUNchmoda+x./certbot-auto

#Runcertbot-autosothatitinstallsitself

RUNscriptscertbot-auto-ncertificates

#webrootsDOMAIN.TLD/.well-known/...filesgohere

VOLUME/webroots

VOLUMEetcletsencrypt

#ThisinstallsaCrontabentrywhich

#runs"certbotrenew"onthe2ndand7thdayofeachweekat03:22AM

#

#cron(8)saystheDebiancrondaemonreadsthefilesinetccron.d,

#mergingintothedatafrometccrontab,touseasthesystem-widecronjobs

#

#RUNecho"22032,7rootscriptscertbot-autorenew">etccron.d/certbot

CMD["cron","-f"]

Thissetsuptwodirectorystructures,/webrootsandetcletsencrypt,whichareexposedfromthecontainerusingtheVOLUMEcommand.ThesedirectoriescontainadministrativefilesusedbycertbotintheprocessofregisteringorrenewingSSLcertificateswiththeLet'sEncryptservice.

TheDockerfilealsoinstallscertbot-auto,whichisreallycertbotbutwithadifferentname.Runningcertbot-auto-ncertificatesisrequiredsothatcertbot-autocaninstallitsdependenciesinthecontainer.Thecertificatescommandliststhecertificatesexistingonthelocalmachine,buttherearenone,andinsteadit'sexecutedforthesideeffectofinstallingthosedependencies.

AnotherfeatureofthisDockerfileistoautomaterenewalofSSLcertificates.Thecertbot-autorenewcommandchecksallcertificatesstoredonthismachinetodetermineifanyrequirerenewal.Ifanydo,arequestisautomaticallymadewithLet'sEncrypttorenewthecertificates.

Asconfigured,thisrenewalattemptwillrunat03:22AMonthe2ndand7thdayoftheweek.

ThelasttaskwehavetohandleisregisteringwithLet'sEncryptforSSLcertificates.Inthecertbotdirectorycreateashellscript,register,orforWindowscallitregister.ps1,containing:

#!/bin/sh

scriptscertbot-autocertonly--webroot-wwebroots$1-d$1

ThisishowweregisteradomainwithLet'sEncrypt.Thescripttakesoneargument,thedomainnametoberegistered.

The--webrootoptionsayswewanttousethewebrootauthenticationalgorithm.WhatthismeansisLet'sEncryptverifiesthatyouownthedomaininquestionbyrequestingaspecificURLonthedomain.Fordocumentation,seehttps://certbot.eff.org/docs/using.html#webroot.

AnexamplevalidationrequestbytheLet'sEncryptservicemightbehttp://example.com/.well-known/acme-challenge/HGr8U1IeTW4kY_Z6UIyaakzOkyQgPr_7ArlLgtZE8S

X.

Inthecertbotcontainer,we'vesaidthis.well-knowndirectorywillexistinthewebrootsDOMAIN-NAME/.well-knowndirectory.Thepurposeofthe-woptionissocertbot-autoknowsthisdirectorylocation,andthe-doptiontellsitthedomainnametoregister.

Whenwerunthecertbot-autocertonlycommand,Let'sEncrypthandsbackachallengefilewhichisinstalledinthedirectorytreespecifiedinthe-wargument.ItmeansthatwhenLet'sEncryptretrievestheURLshownpreviously,itwillfetchthechallengefile,andthereforevalidatethatyoudoindeedcontrolthegivendomainname.

Asitstands,thatdirectoryisnotvisibletoanythingthatwouldsatisfythevalidationrequest.

Forthistoworkasintended,thewebrootsDOMAIN-NAME/.well-knowndirectoryhastobevisibletotheNotesapplication,suchthatNotescansatisfytheURLrequestLet'sEncryptwillmake.Let'sseehowtodothis.

Cross-containermountingofLet'sEncryptdirectoriestothenotescontainerThepurposeofthecertbotcontaineristomanageLet'sEncryptSSLcertificates.ThosecertificatesaretobeusedbytheNotesapplicationtoconfigureanHTTPSserver.It'srequiredthatthecertificates,andthechallengefiles,bevisibleinsidethenotescontainer.

Insteadofcreatingaseparatecontainer,wecouldhaveinsteadintegratedcertbot-autointothenotescontainer.Butthatwouldhavepreventedscalingthenumberofnotescontainers.Wecan'thaveeachnotesinstancerunningacertbot-autoscripttogeneratecertificates.Instead,thecertificatemanagementprocessesmustinsteadbecentralized.Hence,wedevelopedthecertbotcontainer.

What'srequiredisforthewebrootsDOMAIN-NAME/.well-knowndirectoryinthecertbotcontainertobevisiblesomewhereinthenotescontainer.

Tosetthisup,wehavetwochangesrequiredincompose/docker-compose.yml.First,weaddthisstanzaforthecertbotcontainer:

certbot:

build:../certbot

container_name:certbot

networks:

-frontnet

restart:always

volumes:

-certbot-webroot-evsoul:webrootsevsoul.com/.well-known

-certbot-letsencrypt:etcletsencrypt

ThisbuildsacontainerimagefromtheDockerfiledescribedinthe

previoussection.Asfarasitgoes,thisisprettystraightforwardexceptfortheentriesinthevolumessection.Thoseentriesassociatethedirectoriesshownherewithnamedvolumesweneedtodefineelsewhereindocker-compose.yml.

Whatwe'redoingisattachingthesametwovolumestoboththecertbotandnotescontainers.Inthisexample,we'remountingthosevolumestospecificdirectoriesincertbot.

Thetwonamedvolumesaredeclaredlikeso:

volumes:

...

certbot-webroot-evsoul:

certbot-letsencrypt:

Thenwemustmakeasimilarchangetothenotescontainer:

notes:

build:../notes

container_name:notes

depends_on:

-db-notes

networks:

-frontnet

ports:

-"80:3000"

-"443:3443"

restart:always

volumes:

-certbot-webroot-evsoul:notesapppublic/.well-known

-certbot-letsencrypt:etcletsencrypt

We'vemadetwochanges,thefirstofwhichistoaddaTCPportexportfortheHTTPSport(port443).Andthesecondistomountthetwonamedvolumestoappropriatelocationsinthenotescontainer.Thesearethesamenamedvolumesdeclaredearlier.We'resimplymountingthevolumesinplacesthatmakesenseforthenotescontainer.

TheSSLcertificatescouldexistatanylocationinthenotescontainer,but

etcletsencryptisasgoodalocationasany.What'snecessaryisthattheNotescodebeabletoreadthecertificates.

Puttingthe.well-knowndirectoryundernotesapppublicmeansthatNoteswillautomaticallyserveanyfileinthatdirectorytoanyservicemakingarequest.That'sbecauseofthestaticmiddlewarealreadyconfiguredinapp.mjs.

Whatthissetsupisapairofdirectoriesthatarevisibleintwocontainers.Eithercontainercanwriteafileineitherdirectory,andthefilewillautomaticallyshowupintheothercontainer.

Whenweruntheregisterscriptinthecertbotcontainer,itwillwritethechallengefileprovidedbyLet'sEncryptintothewebrootsevsoul.com/.well-knowndirectorytree.Thatsamedirectoryisvisibleinthenotescontainerasnotesapppublic/.well-known,andthereforeNoteswillautomaticallyservethechallengefilejustasitservestheCSS,JavaScript,andimagefilesinthatdirectorytree.

Theprocesslookslikethis:

$dockerexec-itcertbotbash

root@05b095690414:/scripts#sh./registerevsoul.com

Savingdebuglogtovarlog/letsencrypt/letsencrypt.log

Pluginsselected:Authenticatorwebroot,InstallerNone

Enteremailaddress(usedforurgentrenewalandsecuritynotices)(Enter'c'

to

cancel):ENTERYOUREMAILADDRESSHERE

---------------------------------------------------------------

PleasereadtheTermsofServiceat

https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf.Youmust

agreeinordertoregisterwiththeACMEserverat

https://acme-v01.api.letsencrypt.org/directory

---------------------------------------------------------------

(A)gree/(C)ancel:a

---------------------------------------------------------------

WouldyoubewillingtoshareyouremailaddresswiththeElectronicFrontier

Foundation,afoundingpartneroftheLet'sEncryptprojectandthenon-

profit

organizationthatdevelopsCertbot?We'dliketosendyouemailaboutEFFand

ourworktoencrypttheweb,protectitsusersanddefenddigitalrights.

--------------------------------------------------------------

(Y)es/(N)o:n

Obtaininganewcertificate

Performingthefollowingchallenges:

http-01challengeforevsoul.com

Usingthewebrootpathwebrootsevsoul.comforallunmatcheddomains.

Waitingforverification...

Cleaningupchallenges

IMPORTANTNOTES:

-Congratulations!Yourcertificateandchainhavebeensavedat:

etcletsencrypt/live/evsoul.com/fullchain.pem

Yourkeyfilehasbeensavedat:

etcletsencrypt/live/evsoul.com/privkey.pem

Yourcertwillexpireon2018-05-22.Toobtainanewortweaked

versionofthiscertificateinthefuture,simplyruncertbot-auto

again.Tonon-interactivelyrenewallofyourcertificates,run

"certbot-autorenew"

-YouraccountcredentialshavebeensavedinyourCertbot

configurationdirectoryatetcletsencrypt.Youshouldmakea

securebackupofthisfoldernow.Thisconfigurationdirectorywill

alsocontaincertificatesandprivatekeysobtainedbyCertbotso

makingregularbackupsofthisfolderisideal.

-IfyoulikeCertbot,pleaseconsidersupportingourworkby:

DonatingtoISRGLet'sEncrypt:https:/letsencrypt.org/donate

DonatingtoEFF:https://eff.org/donate-le

root@05b095690414:/scripts#

Afterrunningthecommand,it'sagoodideatoinspecttheetcletsencryptdirectorystructuretoseewhat'sthere.Thecontentsofthesedirectoriesmustbetreatedwithcare,sinceitincludestheprivatekeyscertifyingyourdomain.Thewholepointofaprivatekeyencryptionsystemisthattheprivatekeyisstrictlycontrolled,whilethepublickeyisgiventoanyone.

Thefilesendingwiththe.pemextensionarePEM-encodedcertificates.PrivacyEnhancedMail(PEM),whichwasanearlyattempttodevelopasecureencryptedemailsystemfortheinternet.Whilethatprojectfailed,thePEMcontainerformatlivesonandiswidelyusedforSSLcertificates.

AddingHTTPSsupporttoNotesNowthatwehaveaprocesstoregisterwithLet'sEncrypt,andrenewtheSSLcertificateswereceive,let'slookataddingHTTPSsupporttotheNotesapplication.ThetaskisfairlysimplesincetheNode.jsplatformprovidesanHTTPSserveralongsidetheHTTPserverobjectwe'veusedallalong.

We'llrequirethesechangestonotes/app.mjs:

importhttpfrom'http';

importhttpsfrom'https';

...

constUSEHTTPS=process.env.NOTES_USE_HTTPS

&&(typeofprocess.env.NOTES_USE_HTTPS==='string')

&&(process.env.NOTES_USE_HTTPS==='true');

constCERTSDIR=process.env.NOTES_CERTS_DIR;

constoptions=USEHTTPS?{

key:fs.readFileSync(`${CERTSDIR}/privkey1.pem`),

cert:fs.readFileSync(`${CERTSDIR}/fullchain1.pem`),

ca:fs.readFileSync(`${CERTSDIR}/chain1.pem`)

}:{};

constserver=http.createServer(app);

constserverSSL=USEHTTPS?https.createServer(options,app):undefined;

importsocketiofrom'socket.io';

constio=socketio(USEHTTPS?serverSSL:server,options);

...

varport=normalizePort(process.env.PORT||'3000');

app.set('port',port);

...

server.listen(port);

server.on('error',onError);

server.on('listening',onListening);

if(USEHTTPS){

serverSSL.listen(3443);

serverSSL.on('error',onError);

serverSSL.on('listening',onListening);

}

Inotherwords,we'veimportedthehttpsmodulealongsidethehttpmodule,thenwereadintheSSLcertificatesrequiredtoinitializeHTTPSsupport,thencreatedanHTTPSserverobjectusingthosecertificates,andfinallyconfiguredittolistenonport3443.TheHTTPSsupportdependsonthevalueoftheNOTES_USE_HTTPSenvironmentvariable.Ifthatvariableexistsandisequaltotrue,thentheUSEHTTPSvariableissettotrue,andtheHTTPSsupportisturnedon.

Thisleadsustoagainmodifycompose/docker-compose.ymltomatch:

notes:

build:../notes

container_name:notes

depends_on:

-db-notes

networks:

-frontnet

ports:

-"80:3000"

-"443:3443"

restart:always

environment:

-NOTES_USE_HTTPS=true

-NOTES_CERTS_DIR=/etc/letsencrypt/archive/evsoul.com

volumes:

-certbot-webroot-evsoul:/notesapp/public/.well-known

-certbot-letsencrypt:/etc/letsencrypt

BoththeHTTPandHTTPSportsareexposedfromthecontainer,andwehavesomeenvironmentvariablesfortheconfigurationsettings.

TheTWITTER_CALLBACK_URLenvironmentvariableneedstobeupdatedtohttps://DOMAIN.ThesiteisnowhostedonHTTPS,andthereforeTwittershouldredirectouruserstotheHTTPSsite.

Withallthissetup,youshouldnowbeabletovisittheNotesapplication,eitherashttp://DOMAINorhttps://DOMAIN.IfyoutellyourcustomerstosimplyusetheHTTPSversionofyourwebsite,doesthatmeanyou'redone?

No.Thesearchenginesroutinelydowngradesitesforhostingduplicatecontent,andhavingbothanHTTPandHTTPSversionofyourwebsiteisduplicatecontent.Furthermore,thebrowsermakersaremovingquicklytowardswarningbrowserusersthatHTTPwebsitesareinsecure,whileHTTPSwebsitesaresafe.Inotherwords,it'sveryimportanttoredirectanyHTTPconnectionstotheHTTPSversionofyourwebsite.

Onemethodtodothisiswiththeexpress-force-sslpackageforExpress(https://www.npmjs.com/package/express-force-ssl).Asthepackagenameimplies,itintegrateswithanExpressapplication(suchasNotes)andforcesthebrowsertoredirecttotheHTTPSversionofthewebsite.

But,wehaveothersecurityfishtofry.UsingHTTPSsolvesonlypartofthesecurityproblem.Inthenextsection,we'lllookatHelmet,atoolforExpressapplicationstosetmanysecurityoptionsintheHTTPheaders.HelmetincludesatooltorequirebrowserstousetheHTTPSversionofthewebsite,andwe'llalsoshowhowtouseexpress-force-sslatthesametime.

Beforewego,headtotheQualysSSLLabstestpageforSSLimplementations.Thisservicewillexamineyourwebsite,especiallytheSSLcertificates,andgiveyouascore.UsingthestepsinthissectionwillgiveyouanAscore.Toexamineyourscore,seehttps://www.ssllabs.com/ssltest/.

PutonyourHelmetforacross-the-boardsecurityHelmet(https://www.npmjs.com/package/helmet)isnotasecuritysilverbullet(doHelmet'sauthorsthinkwe'retryingtoprotectagainstvampires?).Insteaditisatoolkitforsettingvarioussecurityheadersandtakingotherprotectivemeasures.

Inthenotesdirectory,installthepackagelikeso:

$npminstallhelmet--save

Thenaddthistonotes/app.mjs:

importhelmetfrom'helmet';

...

constapp=express();

exportdefaultapp;

app.use(helmet());

That'senoughformostapplications.UsingHelmetout-of-the-boxprovidesareasonablesetofdefaultsecurityoptions.Wecouldbedonewiththissectionrightnow,exceptthatit'susefultoexaminecloselywhatHelmetdoes,anditsoptions.

Helmetisactuallyaclusterof12modulesforapplyingseveralsecuritytechniques.Eachofthetechniquescanbeindividuallyenabledordisabled,andmanyhaveconfigurationsettingstomake.

UsingHelmettosettheContent-Security-PolicyheaderTheContent-Security-Policy(CSP)headercanhelptoprotectagainstinjectedmaliciousJavaScriptandotherfiletypes.

WewouldberemisstonotpointoutaglaringproblemwithservicessuchastheNotesapplication.Ouruserscouldenteranycodetheylike,andanimproperly-behavingapplicationwillsimplydisplaythatcode.SuchapplicationscanbeavectorforJavaScriptinjectionattacksamongotherthings.

Totrythisout,editanoteandentersomethinglike:

<scriptsrc="http://example.com/malicious.js"></script>

ClicktheSavebutton,andyou'llseethiscodedisplayedastext.AdangerousversionofNoteswouldinsteadinsertthe<script>taginthenotesviewpagesuchthatthemaliciousJavaScriptwouldbeloadedandcauseaproblemforourvisitors.Instead,the<script>tagisencodedassafeHTMLsoitsimplyshowsupastextonthescreen.Wedidn'tdoanythingspecialforthatbehavior,Handlebarsdidthatforus.

Actually,it'salittlemoreinterestingifwelookattheHandlebarsdocumentationhttp://handlebarsjs.com/expressions.html,welearnaboutthisdistinction:

{{encodedAsHtml}}

{{{notEncodedAsHtml}}}

InHandlebars,avalueappearinginatemplateusingtwocurlybraces

({{encoded}})isencodedusingHTMLcoding.Forthepreviousexample,theanglebracketisencodedas&lt;andsoonfordisplay,renderingthatJavaScriptcodeasneutraltextratherthanasHTMLelements.Ifinsteadyouusethreecurlybraces({{{notEncoded}}}),thevalueisnotencodedandisinsteadpresentedasis.ThemaliciousJavaScriptwouldbeexecutedinyourvisitor'sbrowser,causingproblemsforyourusers.

InNotes,ifwewantedouruserstoenterHTMLandhaveitdisplayedastheHTML,thenviews/noteview.hbswouldrequirethiscode:

{{#ifnote}}<divid="notebody">{{{note.body}}}</div>{{/if}}

Most(orall)templateenginesincludethispatternfordisplayingvalues.ThetemplatedevelopershaveachoicebetweenencodinganygivenvaluewithHTMLcodesordisplayingitasis.

ReturningtoHelmet'ssupportfortheheader,it'susefultohavestrictcontroloverthelocationsfromwhichthebrowsercandownloadfiles.Theissuenamed,ourusersenteringmaliciousJavaScriptcodeisonlyonerisk.SupposeamaliciousactorbrokeinandmodifiedthetemplatestoincludemaliciousJavaScriptcode?

Withtheheader,wecantellthebrowserthatJavaScriptcancomeonlyfromourownserverandGoogle'sCDN,andeverythingelseistoberejected.ThatmaliciousJavaScriptthat'sloadedfrompiratesden.netwon'trun.WecouldevenletourusersenterHTMLwithsomecomfortthatanymaliciousJavaScriptreferencedfromathird-partywebsitewon'trun.

ToseethedocumentationforthisHelmetmodule,seehttps://helmetjs.github.io/docs/csp/.

Therearealonglistofoptions.Forinstance,youcancausethebrowsertoreportanyviolationsbacktoyourserver,inwhichcaseyou'llneedtoimplementaroutehandlerfor/report-violation.ThissnippetissufficientforNotes:

app.use(helmet.contentSecurityPolicy({

directives:{

defaultSrc:["'self'"],

scriptSrc:["'self'","'unsafe-inline'"],

styleSrc:["'self'",'fonts.googleapis.com'],

fontSrc:["'self'",'fonts.gstatic.com'],

connectSrc:["'self'",'wss://evsoul.com']

}

}));

Forbetterorforworse,theNotesapplicationimplementsonesecuritybestpractice—allCSSandJavaScriptfilesareloadedfromthesameserverastheapplication.Therefore,forthemostpart,wecanusethe'self'policy.Thereareseveralexceptions:

scriptSrc:DefineswhereweareallowedtoloadJavaScript.WedouseinlineJavaScriptinnoteview.hbsandindex.hbs,whichmustbeallowed.

styleSrc,fontSrc:We'reloadingCSSfilesfromboththelocalserver,andfromGoogleFonts.

connectSrc:TheWebSocketschannelusedbySocket.IOisdeclaredhere.

app.use(helmet.dnsPrefetchControl({allow:false}));//ortrue

app.use(helmet.frameguard({action:'deny'}));

Thissettingdeniesallsuch<iframe>content.

UsingHelmettoremovetheX-Powered-ByheaderTheX-Powered-Byheadercangivemaliciousactorsaclueaboutthesoftwarestackinuse,informingthemofattackalgorithmsthatarelikelytosucceed.TheHidePoweredBysubmoduleforHelmetsimplyremovesthatheader.

Expresscandisablethisfeatureonitsown:

app.disable('x-powered-by')

OryoucanuseHelmettodoso:

app.use(helmet.hidePoweredBy())

ImprovingHTTPSwithStrictTransportSecurityHavingimplementedHTTPSsupport,wearen'tcompletelydone.Aswesaidearlier,it'snecessaryforouruserstousetheHTTPSversionofNotes,butasitstandstheycanstillusetheHTTPversion.TheStrictTransportSecurityheadernotifiesthebrowserthatitshouldusetheHTTPSversionofthesite.Sincethat'ssimplyanotification,it'salsonecessarytoimplementaredirectfromtheHTTPtoHTTPSversionofNotes.

WesettheStrict-Transport-Securitylikeso:constsixtyDaysInSeconds=5184000app.use(helmet.hsts({maxAge:sixtyDaysInSeconds}));

ThistellsthebrowsertostickwiththeHTTPSversionofthesiteforthenext60days,andnevervisittheHTTPversion.

And,aslongaswe'reonthisissue,let'sgoaheadanduseexpress-force-ssltoimplementtheredirect.Afteraddingadependencytothatpackageinpackage.json,addthisinapp.mjs:importforceSSLfrom'express-force-ssl';...app.use(forceSSL);app.use(bodyParser.json());

MitigatingXSSattackswithHelmetXSSattacksattempttoinjectJavaScriptcodeintowebsiteoutput.Withmaliciouscodeinjectedintoanotherwebsite,theattackercanaccessinformationtheyotherwisecouldnotretrieve.TheX-XSS-ProtectionheaderpreventscertainXSSattacks,butnotallofthem.

app.use(helmet.xssFilter());

AddressingCross-SiteRequestForgery(CSRF)attacksCSRFattacksaresimilartoXSSattacksinthatbothoccuracrossmultiplesites.InaCSRFattack,malicioussoftwareforgesabogusrequestonanothersite.Topreventsuchanattack,CSRFtokensaregeneratedforeachpageview,areincludedashiddenvaluesinHTMLFORMs,andthencheckedwhentheFORMissubmitted.Amismatchonthetokenscausestherequesttobedenied.

ThecsurfpackageisdesignedtobeusedwithExpresshttps://www.npmjs.com/package/csurfInthenotesdirectory,runthis:

$npminstallcsurf--save

Theninstallthemiddlewarelikeso:

importcsrffrom'csurf';

...

app.use(cookieParser());

app.use(csrf({cookie:true}));

ThecsurfmiddlewaremustbeinstalledfollowingthecookieParsermiddleware.

Next,foreverypagethatincludesaFORM,wemustgenerateandsendatokenwiththepage.Thatrequirestwothings,intheres.rendercallwegeneratethetoken,andthenintheviewtemplateweincludethetokenasahiddenINPUTonanyforminthepage.We'regoingtobetouchingonseveralfileshere,solet'sgetstarted.

Inroutes/notes.mjs,addthefollowingasaparametertotheres.rendercallfor

the/add,/edit,/view,and/destroyroutes:

csrfToken:req.csrfToken()

Likewise,dothesameforthe/loginrouteinroutes/users.mjs.ThisaddsthegeneratedCSRFtokentotheparameterssenttothetemplate.

Theninviews/noteedit.hbsandviews/notedestroy.hbs,addthefollowing:

{{#ifuser}}

<inputtype="hidden"name="_csrf"value="{{csrfToken}}">

...

{{/if}}

Inviews/login.hbs,makethesameadditionbutwithoutthe{{#ifuser}}instruction.

Inviews/noteview.hbs,there'saformforsubmittingcomments.Makethischange:

<formid="submit-comment"class="well"data-asyncdata-target="#rating-modal"

action="notesmake-comment"method="POST">

<inputtype="hidden"name="_csrf"value="{{csrfToken}}">

...

</form>

This<input>tagrenderstheCSRFtokenintotheFORM.WhentheFORMissubmitted,thecsurfmiddlewarechecksitforcorrectnessandrejectsanythatdonotmatch.

DenyingSQLinjectionattacksSQLinjectionisanotherlargeclassofsecurityexploits,wheretheattackerputsSQLcommandsintoinputdata.Seehttps://www.xkcd.com/327/foranexample.

Thesqlinjectionpackagescansquerystrings,requestbodyparameters,androuteparametersforSQLcode.

Installwith:$npminstallsqlinjection--save

Theninstallitinapp.mjs:importsqlinjectionfrom'sqlinjection';...app.use(sqlinjection);

SequelizedeprecationwarningregardingoperatorinjectionattackYoumayhaveseenthisdeprecationwarningprintedbyNotes:

sequelizedeprecatedStringbasedoperatorsarenowdeprecated.Pleaseuse

Symbolbasedoperatorsforbettersecurity,readmoreat

http://docs.sequelizejs.com/manual/tutorial/querying.html#operators

NowhereinNotesareweusingSequelizestring-basedoperators,andthereforethiswouldseemtobeaspuriouserrormessage.Inactuality,itisarealissuewithpotentialsimilartoanSQLinjectionattack.

Thisissuequeueentryhasanin-depthdiscussionofthesecurityproblem:https://github.com/sequelize/sequelize/issues/8417withmoredetailsinthedocumentationathttp://docs.sequelizejs.com/manual/tutorial/querying.html#operators-security.

Namely,aquerysuchasthis:

db.Token.findOne({

where:{token:req.query.token}

});

Issusceptibletoaninjection-styleattackthatwouldsubvertthequery.Wedohavecodelikethisinnotes/models/notes-sequelize.mjs,thereforethisissueshouldbeaddressed.

Fortunately,thesolutionissimplyamatterofdisablingthestringaliasesfortheOperators.IntheSequelizeconfigurationfileswedefined,we

disablethealiaseslikeso:

dbname:users

username:

password:

params:

dialect:sqlite

storage:users-sequelize.sqlite3

operatorAliases:false

ThischangeshouldbemadeineverySequelizeconfigurationfile.

ScanningforknownvulnerabilitiesThensppackage(https://www.npmjs.com/package/nsp)scansapackage.jsonornpm-shrinkwrap.json,lookingforknownvulnerabilities.Thecompanybehindthatpackagekeepsalistofsuchpackages,whicharequeriedbythensppackage.

Startingwithnpmversion6,thensppackagefunctionalityhasbeenfoldedintonpmitselfasthenpmauditcommand.Itisacommand-linetoolyourunlikeso:

$npminstallnsp

$./node_modules/.bin/nspcheck

(+)3vulnerabilitiesfound

┌────────────┬───────────────────────────────────────────────────────────────

─────┐

││RegularExpressionDenialofService

│

├────────────┼───────────────────────────────────────────────────────────────

─────┤

│Name│mime

│

├────────────┼───────────────────────────────────────────────────────────────

─────┤

│CVSS│7.5(High)

│

├────────────┼───────────────────────────────────────────────────────────────

─────┤

│Installed│1.3.4

│

├────────────┼───────────────────────────────────────────────────────────────

─────┤

│Vulnerable│<1.4.1||>2.0.0<2.0.3

│

├────────────┼───────────────────────────────────────────────────────────────

─────┤

│Patched│>=1.4.1<2.0.0||>=2.0.3

│

├────────────┼───────────────────────────────────────────────────────────────

─────┤

│Path│notes@0.0.0>express@4.15.5>send@0.15.6>mime@1.3.4

│

├────────────┼───────────────────────────────────────────────────────────────

─────┤

│MoreInfo│https://nodesecurity.io/advisories/535

│

└────────────┴───────────────────────────────────────────────────────────────

─────┘

...moreoutput

Thisreportsaysthemimepackagecurrentlyinstalledisvulnerable.TheVulnerablelinesayswhichversionshavetheknownproblem,andthePatchedlinesayswhichreleasesaresafe.TheMoreInfolinetellsyouwheretogetmoreinformation.

Accordingtothenpmteam,npmversion6willincludethisfeatureasabaked-incapability.Seehttps://blog.npmjs.org/post/173260195980/announcing-npm6

If,asistrueinthiscase,you'reunsurewherethepackageisbeingused,trythis:

$npmlsmime

notes@0.0.0Usersdavid/chap12/notes

├─┬express@4.15.5

│└─┬send@0.15.6

│└──mime@1.3.4

└─┬superagent@3.8.2

└──mime@1.6.0

Therearetwoversionsofthemimemodulebeingused,oneofwhichisvulnerablegoingbytheversionnumbersshownpreviously.Thisisgoodinformation,buthowdoyouuseit?Intheory,youshouldsimplychangethedependenciestouseasafeversionofthepackage.

Inthiscase,wehaveaprobleminthatourpackage.jsondidnotcausethemimepackagetobeinstalled.Instead,it'sthesendpackage,whichinturnwasrequestedbyExpress,whichisresponsible.We'reatthemercyofthat

packagemaintainertoupdatetheirdependencies.

Fortunately,there'sanewerversionofExpressavailablewhichdoesindeedupdatethedependencyonthesendpackage,whichinturnupdatesitsdependencyonthemimepackage:

$npmlsmime

notes@0.0.0Usersdavid/chap12/notes

├─┬express@4.16.2

│└─┬send@0.16.1

│└──mime@1.4.1

└─┬superagent@3.8.2

└──mime@1.6.0

Simplyupdatingthedependenciesfixedtheproblem.Butwenowhaveanadministrativetaskthat,accordingtotheTwelveFactorApplicationmodel,wemustautomate.

Onewaytoautomatethisisbyfirstaddingnsptothepackage.jsondependenciestoinstallitinsideNotes.Thensppackagesaysitisbestinstalledglobally,butthatwouldbeanimplicitdependency.TheTwelveFactorApplicationmodelsuggestsit'sbettertohaveexplicitdependencies,andthereforeit'sbesttolistnspinthepackage.json.Onceyou'veinstallednsp,addthissteptotheDockerfile:

WORKDIR/notesapp

RUNnpminstall--unsafe-perm

RUN./node_modules/.bin/nspcheck

BuildingtheDockercontainerwillnowgivethisresult:

Step21/25:RUN./node_modules/.bin/nspcheck

--->Runningin9dedf22ec1f9

(+)2vulnerabilitiesfound

...

ERROR:Service'notes'failedtobuild:Thecommand'binsh-c

./node_modules/.bin/nspcheck'returnedanon-zerocode:1

Inotherwords,wehaveasimplemechanismtonotuseaservicewhose

dependencieshaveknownvulnerabilities.

UsinggoodcookiepracticesSomenutritionistssayeatingtoomanysweets,suchascookies,isbadforyourhealth.Webcookies,however,arewidelyusedformanypurposesincludingrecordingwhetherabrowserisloggedinornot.

IntheNotesapplication,we'realreadyusingsomegoodpractices:

We'reusinganExpresssessioncookienamedifferentfromthedefaultshowninthedocumentation

TheExpresssessioncookiesecretisnotthedefaultshowninthedocumentation

Takentogether,anattackercan'texploitanyknownvulnerabilitystemmingfromusingdefaultvalues.Allkindsofsoftwareproductsshowdefaultpasswordsorotherdefaults.Thosedefaultscouldbesecurityvulnerabilities,andthereforeit'sbesttonotusethedefaults.Forexample,thedefaultRaspberryPilogin/passwordispiandraspberry.Whilethat'scute,anyRaspbian-basedIoTdevicethat'sleftwiththedefaultlogin/passwordissusceptible.

Butthere'sabitmorewecandotomakethesinglecookiewe'reusing,theExpresssessioncookie,moresecure.

Thepackagehasafewoptionsavailable,seehttps://www.npmjs.com/package/express-session:

app.use(session({

store:sessionStore,

secret:sessionSecret,

resave:true,

saveUninitialized:true,

name:sessionCookieName,

secure:true,

maxAge:26060*1000//2hours

}));

Theseareadditionalattributesthatlookuseful.ThesecureattributerequiresthatcookiesbesentONLYoverHTTPSconnections.ThisensuresthecookiedataisencryptedbyHTTPSencryption.ThemaxAgeattributesetsanamountoftimethatcookiesarevalid,expressedasmilliseconds.

SummaryInthischapter,we'vecoveredanextremelyimportanttopic,applicationsecurity.ThankstothehardworkoftheNode.jsandExpresscommunities,we'vebeenabletotightenthesecuritysimplybyaddingafewbitsofcodehereandtheretoconfiguresecuritymodules.We'veevenworkedouthowtopreventthesystemfrombeingbuilt,ifit'susingpackageswithknownvulnerabilities.

EnablingHTTPSmeansourusershavebetterassuranceofsecurity.TheSSLcertificateisameasureofauthenticitythatprotectsagainstman-in-the-middlesecurityattacks,andthedataisencryptedfortransmissionacrosstheinternet.Withalittlebitofwork,wewereabletosetupasystemtoacquire,andcontinuouslyrenew,freeSSLcertificatesfromtheLet'sEncryptservice.

Thehelmetpackageprovidesasuiteoftoolstosetsecurityheadersthatinstructwebbrowsersonhowtotreatourcontent.Thesesettingspreventormitigatewholeclassesofsecuritybugs.Withthecsurfpackage,we'reabletopreventcross-siterequestforgeryattacks.

ThesefewstepsareagoodstartforsecuringtheNotesapplication.But,donotstopatthesemeasures,becausethereisanever-endingsetofsecurityholes.

AllYahooemployeesaretrainedinsecuritypractices,Yahoo'sinternalnetworkhaswelldefinedroutingandotherprotectionssurroundingtheall-importantuserdatabase,andYahoo'sengineeringstaffissaltedwithpeoplecarryingthejobtitleParanoid,taskedwithconstantlyscrutinizingsystemsforpotentialsecurityvulnerabilities.Yetwithallthatwork,Yahoostillsufferedthelargestbreachofuseridentitydatainthehistoryoftheinternet.Thelessonisthatnoneofuscanneglectthesecurityoftheapplicationswedeploy.

Overthecourseofthisbook,we'vecomealongway.Overall,thejourneyhasbeentoexaminethemajorlifecyclestepsrequiredtodevelopanddeployaNode.jswebapplication.

WestartedbylearningthebasicsofNode.js,andhowtousetheplatformtodevelopsimpleservices.Throughoutthebook,we'velearnedhowadvancedJavaScriptfeaturessuchasasyncfunctionsandES6modulesareusedinNode.jsapplications.Tostoreourdata,welearnedhowtouseseveraldatabaseengines,andamethodologytomakeiteasytoswitchbetweenengines.

Mobile-firstdevelopmentisextremelyimportantintoday'senvironment,andtofulfillthatgoal,welearnedhowtousetheBootstrapframework.

Real-timecommunicationisexpectedonawidevarietyofwebsites,becauseadvancedJavaScriptcapabilitiesmeanswecannowoffermoreinteractiveservicesinourwebapplications.Tofulfillthatgoal,welearnedhowtousetheSocket.IOreal-timecommunicationsframework.

Deployingapplicationservicestocloudhostingiswidelyused,bothforsimplifyingsystemsetup,andtoscaleservicestomeetthedemandsofouruserbase.Tofulfillthatgoal,welearnedtouseDocker.WenotonlyusedDockerforproductiondeployment,butfordeployingatestinfrastructure,withinwhichwecanrununittestsandfunctionaltests.AndwelearnedhowtoimplementHTTPSsupportbydevelopingacustomDockercontainercontainingLet'sEncrypttoolstoregisterandrenewSSLcertificates.

OtherBooksYouMayEnjoyIfyouenjoyedthisbook,youmaybeinterestedintheseotherbooksbyPackt:

Node.jsDesignPatterns-SecondEditionMarioCasciaro,LucianoMammino

ISBN:978-1-78328-732-1

Designandimplementaseriesofserver-sideJavaScriptpatternssoyouunderstandwhyandwhentoapplythemindifferentusecasescenarios

Becomecomfortablewithwritingasynchronouscodebyleveragingconstructssuchascallbacks,promises,generatorsandtheasync-awaitsyntax

IdentifythemostimportantconcernsandapplyuniquetrickstoachievehigherscalabilityandmodularityinyourNode.js

application

Untangleyourmodulesbyorganizingandconnectingthemcoherently

Reusewell-knowntechniquestosolvecommondesignandcodingissues

ExplorethelatesttrendsinUniversalJavaScript,learnhowtowritecodethatrunsonbothNode.jsandthebrowserandleverageReactanditsecosystemtoimplementuniversalapplications

NodeCookbook-ThirdEditionDavidMarkClements,MatthiasBuus,MatteoCollina,PeterElger

ISBN:978-1-78588-124-4

DebugNode.jsprograms

WriteandpublishyourownNode.jsmodules

DetailedcoverageofNode.jscoreAPI's

UsewebframeworkssuchasExpress,HapiandKoafor

acceleratedwebapplicationdevelopment

ApplyNode.jsstreamsforlow-footprintdataprocessing

Fast-trackperformanceknowledgeandoptimizationabilities

Persistencestrategies,includingdatabaseintegrationswithMongoDB,MySQL/MariaDB,Postgres,Redis,andLevelDB

Applycritical,essentialsecurityconcepts

UseNodewithbest-of-breeddeploymenttechnologies:Docker,KubernetesandAWS

Leaveareview-letotherreadersknowwhatyouthinkPleaseshareyourthoughtsonthisbookwithothersbyleavingareviewonthesitethatyouboughtitfrom.IfyoupurchasedthebookfromAmazon,pleaseleaveusanhonestreviewonthisbook'sAmazonpage.Thisisvitalsothatotherpotentialreaderscanseeanduseyourunbiasedopiniontomakepurchasingdecisions,wecanunderstandwhatourcustomersthinkaboutourproducts,andourauthorscanseeyourfeedbackonthetitlethattheyhaveworkedwithPackttocreate.Itwillonlytakeafewminutesofyourtime,butisvaluabletootherpotentialcustomers,ourauthors,andPackt.Thankyou!