1.1

1.2

1.2.1

1.2.1.1

1.2.1.2

1.2.1.3

1.2.1.4

1.2.2

1.2.3

1.2.4

1.3

1.4

1.4.1

1.4.2

1.4.3

1.4.4

1.4.4.1

1.4.4.2

1.4.4.3

1.5

1.5.1

1.5.2

1.5.2.1

1.5.2.2

1.5.3

1.5.3.1

1.5.3.2

1.5.4

1.5.4.1

1.5.4.2

1.5.4.3

1.5.5

1.5.6

1.5.6.1

1.5.6.2

1.5.7

1.5.7.1

1.5.7.2

1.5.7.3

1.5.7.4

1.6

TableofContentsIntroduction

GettingStarted

Installing

Linux

MacOSX

Windows

Compiling

Authentication

AccessingtheWebInterface

ComingfromSQL

Highlights

Scalability

Architecture

Datamodels

Limitations

Datacentertodatacenterreplication

Introduction

Applicability

Requirements

Datamodels&modeling

Concepts

Databases

WorkingwithDatabases

NotesaboutDatabases

Collections

CollectionMethods

DatabaseMethods

Documents

BasicsandTerminology

CollectionMethods

DatabaseMethods

Graphs,Vertices&Edges

Views

DatabaseMethods

ViewMethods

NamingConventions

DatabaseNames

CollectionNames

DocumentKeys

AttributeNames

Indexing

1

1.6.1

1.6.2

1.6.3

1.6.4

1.6.4.1

1.6.4.2

1.6.4.3

1.6.4.4

1.6.4.5

1.6.4.6

1.7

1.7.1

1.7.1.1

1.7.1.2

1.7.2

1.7.2.1

1.7.3

1.7.3.1

1.7.3.2

1.7.4

1.7.5

1.8

1.8.1

1.8.1.1

1.9

1.9.1

1.9.2

1.9.3

1.9.4

1.9.5

1.9.6

1.9.7

1.9.7.1

1.9.7.2

1.9.7.3

1.9.7.4

1.9.8

1.9.9

1.9.9.1

1.9.9.1.1

1.9.9.1.2

1.9.9.2

1.9.9.2.1

1.9.9.2.2

IndexBasics

Whichindextousewhen

IndexUtilization

WorkingwithIndexes

HashIndexes

Skiplists

Persistent

FulltextIndexes

GeoIndexes

VertexCentricIndexes

Graphs

GeneralGraphs

GraphManagement

GraphFunctions

SmartGraphs

SmartGraphManagement

Traversals

UsingTraversalObjects

ExampleData

WorkingwithEdges

Pregel

Views

ArangoSearch

Analyzers

FoxxMicroservices

Ataglance

Gettingstarted

Servicemanifest

Servicecontext

Configuration

Dependencies

Routers

Endpoints

Middleware

Request

Response

UsingGraphQL

Sessionsmiddleware

Sessionstorages

Collectionstorage

JWTstorage

Sessiontransports

Cookietransport

Headertransport

2

1.9.10

1.9.11

1.9.12

1.9.13

1.9.14

1.9.14.1

1.9.14.2

1.9.14.3

1.9.14.4

1.9.14.5

1.9.14.5.1

1.9.14.5.2

1.9.14.5.3

1.9.14.5.4

1.9.14.5.5

1.9.14.5.6

1.9.14.6

1.9.14.7

1.9.14.8

1.9.15

1.9.16

1.9.17

1.9.17.1

1.9.17.2

1.9.17.3

1.10

1.10.1

1.10.2

1.10.3

1.10.4

1.10.5

1.11

1.11.1

1.11.2

1.11.2.1

1.11.2.2

1.11.2.3

1.11.2.3.1

1.11.2.3.2

1.11.2.3.3

1.11.2.3.4

1.11.3

1.11.3.1

1.11.3.2

Servingfiles

Writingtests

CrossOrigin

Scriptsandqueuedjobs

Migrating2.xservices

Migratingfrompre-2.8

manifest.json

applicationContext

RepositoriesandModels

Controllers

Requestcontext

Errorhandling

Before/After/Around

Requestobject

Responseobject

DependencyInjection

Sessions

AuthandOAuth2

FoxxQueries

Legacycompatibilitymode

Usermanagement

Relatedmodules

Authentication

OAuth1.0a

OAuth2.0

Transactions

Transactioninvocation

Passingparameters

Lockingandisolation

Durability

Limitations

Deployment

Singleinstance

Cluster

Cluster:Mesos,DC/OS

Cluster:Generic&Docker

AdvancedTopics

StandaloneAgency

Cluster:Localtestsetups

Cluster:Processes

Cluster:Docker

MultipleDatacenters

Cluster

Kafka&Zookeeper

3

1.11.3.3

1.11.3.4

1.11.3.5

1.12

1.12.1

1.12.1.1

1.12.1.2

1.12.1.3

1.12.1.4

1.12.1.5

1.12.1.6

1.12.1.7

1.12.1.8

1.12.1.9

1.12.2

1.12.2.1

1.12.2.2

1.12.2.3

1.12.3

1.12.4

1.12.5

1.12.6

1.12.7

1.12.7.1

1.12.8

1.12.8.1

1.12.8.2

1.12.8.3

1.12.8.4

1.12.8.5

1.12.8.6

1.12.8.7

1.12.8.8

1.12.8.9

1.12.8.10

1.12.8.11

1.12.8.12

1.12.9

1.12.10

1.12.11

1.12.11.1

1.12.11.2

1.12.12

1.12.12.1

ArangoSyncMaster

ArangoSyncWorkers

Prometheus&Grafana

Administration

WebInterface

Dashboard

Cluster

Collections

Document

Queries

Graphs

Services

Users

Logs

ArangoDBShell

ShellOutput

Configuration

Details

Arangoimport

Arangodump

Arangorestore

Arangoexport

ManagingUsers

InArangosh

ServerConfiguration

OperatingSystemConfiguration

ManagingEndpoints

SSLConfiguration

LDAPOptions

LoggingOptions

GeneralOptions

Write-AheadLogOptions

CompactionOptions

ClusterOptions

RocksDBEngineOptions

HashCacheOptions

AsynchronousTasks

Durability

Encryption

Auditing

Configuration

Events

Replication

AsynchronousReplication

4

1.12.12.1.1

1.12.12.1.2

1.12.12.1.3

1.12.12.1.4

1.12.12.1.5

1.12.12.2

1.12.12.2.1

1.12.12.2.2

1.12.12.3

1.12.13

1.12.14

1.12.15

1.12.15.1

1.12.15.2

1.12.15.3

1.12.15.4

1.12.15.5

1.12.15.6

1.12.15.7

1.12.15.8

1.12.15.9

1.12.15.10

1.13

1.13.1

1.13.2

1.13.3

1.13.4

1.13.5

1.14

1.14.1

1.15

1.15.1

1.16

1.16.1

1.16.2

1.17

1.17.1

1.17.2

1.17.3

1.17.4

1.17.5

1.17.6

1.17.7

1.17.8

Components

Per-DatabaseSetup

Server-LevelSetup

SyncingCollections

ReplicationLimitations

SynchronousReplication

Implementation

Configuration

SatelliteCollections

Datacentertodatacenterreplication

Sharding

Upgrading

Upgradingto3.3

Upgradingto3.2

Upgradingto3.1

Upgradingto3.0

Upgradingto2.8

Upgradingto2.6

Upgradingto2.5

Upgradingto2.4

Upgradingto2.3

Upgradingto2.2

Troubleshooting

arangod

EmergencyConsole

DatafileDebugger

Arangobench

Datacentertodatacenterreplication

Monitoring

Datacentertodatacenterreplication

Security

Datacentertodatacenterreplication

Architecture

Write-aheadlog

StorageEngines

Releasenotes

WhatsNewin3.3

Incompatiblechangesin3.3

WhatsNewin3.2

KnownIssuesin3.2

Incompatiblechangesin3.2

WhatsNewin3.1

Incompatiblechangesin3.1

WhatsNewin3.0

5

1.17.9

1.17.10

1.17.11

1.17.12

1.17.13

1.17.14

1.17.15

1.17.16

1.17.17

1.17.18

1.17.19

1.17.20

1.17.21

1.17.22

1.17.23

1.18

1.18.1

1.18.1.1

1.18.1.2

1.18.1.3

1.18.2

1.18.2.1

1.18.2.2

1.18.2.3

1.18.2.4

1.18.2.5

1.18.2.6

1.18.2.7

1.18.2.8

1.18.2.9

1.18.3

1.18.3.1

1.18.3.1.1

1.18.3.1.2

1.18.3.1.3

1.18.3.1.4

1.18.3.2

1.18.3.2.1

1.18.3.2.2

1.18.3.2.3

1.18.4

1.18.5

Incompatiblechangesin3.0

WhatsNewin2.8

Incompatiblechangesin2.8

WhatsNewin2.7

Incompatiblechangesin2.7

WhatsNewin2.6

Incompatiblechangesin2.6

WhatsNewin2.5

Incompatiblechangesin2.5

WhatsNewin2.4

Incompatiblechangesin2.4

WhatsNewin2.3

Incompatiblechangesin2.3

WhatsNewin2.2

WhatsNewin2.1

Appendix

References

db

collection

cursor

JavaScriptModules

@arangodb

console

crypto

fs

request

actions

queries

Write-aheadlog

TaskManagement

Deprecated

SimpleQueries

Pagination

ModificationQueries

GeoQueries

FulltextQueries

Actions

DeliveringHTMLPages

JsonObjects

Modifying

Errorcodesandmeanings

Glossary

6

ArangoDBv3.4.milestone0DocumentationWelcometotheArangoDBdocumentation!

NewandeagertotryoutArangoDB?Startrightawaywithourbeginner'sguide:GettingStarted

Thedocumentationisorganizedinfourhandbooks:

ThismanualdescribesArangoDBanditsfeaturesindetailforyouasauser,developerandadministrator.TheAQLhandbookexplainsArangoDB'squerylanguageAQL.TheHTTPhandbookdescribestheinternalAPIofArangoDBthatisusedtocommunicatewithclients.Ingeneral,theHTTPhandbookwillbeofinteresttodriverdevelopers.Ifyouuseanyoftheexistingdriversforthelanguageofyourchoice,youcanskipthishandbook.Ourcookbookwithrecipesforspecificproblemsandsolutions.

Featuresareillustratedwithinteractiveusageexamples;youcancut'n'pastethemintoarangoshtotrythemout.TheHTTPREST-APIfordriverdevelopersisdemonstratedwithcut'n'pasterecepiesintendedtobeusedwiththecURL.Driversmayprovidetheirownexamplesbasedonthese.jsbasedexamplestoimproveunderstandeabilityfortheirrespectiveusers,i.e.forthejavadriversomeofthesamplesarere-implemented.

Overview

ArangoDBisanativemulti-model,open-sourcedatabasewithflexibledatamodelsfordocuments,graphs,andkey-values.BuildhighperformanceapplicationsusingaconvenientSQL-likequerylanguageorJavaScriptextensions.UseACIDtransactionsifyourequirethem.Scalehorizontallyandverticallywithafewmouseclicks.

Keyfeaturesinclude:

installingArangoDBonaclusterisaseasyasinstallinganapponyourmobileFlexibledatamodeling:modelyourdataascombinationofkey-valuepairs,documentsorgraphs-perfectforsocialrelationsPowerfulquerylanguage(AQL)toretrieveandmodifydataUseArangoDBasanapplicationserverandfuseyourapplicationanddatabasetogetherformaximalthroughputTransactions:runqueriesonmultipledocumentsorcollectionswithoptionaltransactionalconsistencyandisolationReplicationandSharding:setupthedatabaseinamaster-slaveconfigurationorspreadbiggerdatasetsacrossmultipleserversConfigurabledurability:lettheapplicationdecideifitneedsmoredurabilityormoreperformanceNo-nonsensestorage:ArangoDBusesallofthepowerofmodernstoragehardware,likeSSDandlargecachesJavaScriptforall:nolanguagezoo,youcanuseonelanguagefromyourbrowsertoyourback-endArangoDBcanbeeasilydeployedasafault-tolerantdistributedstatemachine,whichcanserveastheanimalbrainofdistributedappliancesItisopensource(ApacheLicense2.0)

Community

IfyouhavequestionsregardingArangoDB,Foxx,drivers,orthisdocumentationdon'thesitatetocontactuson:

GitHubforissuesandmisbehaviororpullrequestsGoogleGroupsfordiscussionsaboutArangoDBingeneralortoannounceyournewFoxxAppStackOverflowforquestionsaboutAQL,usagescenariosetc.Slack,ourcommunitychat

Whenreportingissues,pleasedescribe:

theenvironmentyourunArangoDBintheArangoDBversionyouusewhetheryou'reusingFoxxtheclientyou'reusing

Introduction

7

whichpartsofthedocumentationyou'reworkingwith(link)whatyouexpecttohappenwhatisactuallyhappening

Wewillrespondassoonaspossible.

Introduction

8

Gettingstarted

Overview

Thisbeginner'sguidewillmakeyoufamiliarwithArangoDB.Wewillcoverhowto

installandrunalocalArangoDBserverusethewebinterfacetointeractwithitstoreexampledatainthedatabasequerythedatabasetoretrievethedataagaineditandremoveexistingdata

Installation

Headtoarangodb.com/download,selectyouroperatingsystemanddownloadArangoDB.Youmayalsofollowtheinstructionsonhowtoinstallwithapackagemanager,ifavailable.

IfyouinstalledabinarypackageunderLinux,theserverisautomaticallystarted.

IfyouinstalledArangoDBusinghomebrewunderMacOSX,starttheserverbyrunning/usr/local/sbin/arangod.

IfyouinstalledArangoDBunderWindowsasaservice,theserverisautomaticallystarted.Otherwise,runthearangod.exelocatedintheinstallationfolder'sbindirectory.YoumayhavetorunitasadministratortograntitwritepermissionstoC:\ProgramFiles.

Formorein-depthinformationonhowtoinstallArangoDB,aswellasavailablestartupparameters,installationinaclusterandsoon,seeInstalling.

ArangoDBofferstwostorageengines:MMFilesandRocksDB.Choosetheonewhichsuitsyourneedsbestintheinstallationprocessoronfirststartup.

Securingtheinstallation

Thedefaultinstallationcontainsonedatabase_systemandausernamedroot.

DebianbasedpackagesandtheWindowsinstallerwillaskforapasswordduringtheinstallationprocess.Red-Hatbasedpackageswillsetarandompassword.Forallotherinstallationpackagesyouneedtoexecute

shell>arango-secure-installation

Thiswillaskedforarootpasswordandsetsthispassword.

Webinterface

Theserveritself(arangod)speaksHTTP/REST,butyoucanusethegraphicalwebinterfacetokeepitsimple.There'salsoarangosh,asynchronousshellforinteractionwiththeserver.Ifyou'readeveloper,youmightprefertheshellovertheGUI.Itdoesnotprovidefeatureslikesyntaxhighlightinghowever.

WhenyoustartusingArangoDBinyourproject,youwilllikelyuseanofficialorcommunity-madedriverwritteninthesamelanguageasyourproject.Driversimplementaprogramminginterfacethatshouldfeelnaturalforthatprogramminglanguage,anddoallthetalkingtotheserver.Therefore,youcanmostcertainlyignoretheHTTPAPIunlessyouwanttowriteadriveryourselforexplicitlywanttousetherawinterface.

Togetfamiliarwiththedatabasesystemyoucanevenputdriversasideandusethewebinterface(codenameAardvark)forbasicinteraction.Thewebinterfacewillbecomeavailableshortlyafteryoustartedarangod.Youcanaccessitinyourbrowserathttp://localhost:8529-ifnot,pleaseseeTroubleshooting.

GettingStarted

9

Bydefault,authenticationisenabled.Thedefaultuserisroot.Dependingontheinstallationmethodused,theinstallationprocesseitherpromptedfortherootpasswordorthedefaultrootpasswordisempty(seeabove).

Nextyouwillbeaskedwhichdatabasetouse.Everyserverinstancecomeswitha_systemdatabase.Selectthisdatabasetocontinue.

GettingStarted

10

Youshouldthenbepresentedthedashboardwithserverstatisticslikethis:

Foramoredetaileddescriptionoftheinterface,seeWebInterface.

Databases,collectionsanddocuments

Databasesaresetsofcollections.Collectionsstorerecords,whicharereferredtoasdocuments.CollectionsaretheequivalentoftablesinRDBMS,anddocumentscanbethoughtofasrowsinatable.Thedifferenceisthatyoudon'tdefinewhatcolumns(orratherattributes)therewillbeinadvance.Everydocumentinanycollectioncanhavearbitraryattributekeysandvalues.Documentsinasinglecollectionwilllikelyhaveasimilarstructureinpracticehowever,butthedatabasesystemitselfdoesnotimposeitandwilloperatestableandfastnomatterhowyourdatalookslike.

Readmoreinthedata-modelconceptschapter.

Fornow,youcanstickwiththedefault_systemdatabaseandusethewebinterfacetocreatecollectionsanddocuments.StartbyclickingtheCOLLECTIONSmenuentry,thentheAddCollectiontile.Giveitaname,e.g.users,leavetheothersettingsunchanged(wewantittobeadocumentcollection)andSaveit.Anewtilelabeledusersshouldshowup,whichyoucanclicktoopen.

TherewillbeNodocumentsyet.Clickthegreencirclewiththewhiteplusontheright-handsidetocreateafirstdocumentinthiscollection.Adialogwillaskyoufora_key.YoucanleavethefieldblankandclickCreatetoletthedatabasesystemassignanautomaticallygenerated(unique)key.Notethatthe_keypropertyisimmutable,whichmeansyoucannotchangeitoncethedocumentiscreated.Whatyoucanuseasdocumentkeyisdescribedinthenamingconventions.

Anautomaticallygeneratedkeycouldbe"9883"(_keyisalwaysastring!),andthedocument_idwouldbe"users/9883"inthatcase.Asidefromafewsystemattributes,thereisnothinginthisdocumentyet.Let'saddacustomattributebyclickingtheicontotheleftof(emptyobject),thenAppend.Twoinputfieldswillbecomeavailable,FIELD(attributekey)andVALUE(attributevalue).Typenameaskeyandyournameasvalue.Appendanotherattribute,nameitageandsetittoyourage.ClickSavetopersistthechanges.IfyouclickonCollection:usersatthetopontheright-handsideoftheArangoDBlogo,thedocumentbrowserwillshowthedocumentsintheuserscollectionandyouwillseethedocumentyoujustcreatedinthelist.

Queryingthedatabase

GettingStarted

11

TimetoretrieveourdocumentusingAQL,ArangoDB'squerylanguage.Wecandirectlylookupthedocumentwecreatedviathe_id,buttherearealsootheroptions.ClicktheQUERIESmenuentrytobringupthequeryeditorandtypethefollowing(adjustthedocumentIDtomatchyourdocument):

RETURNDOCUMENT("users/9883")

ThenclickExecutetorunthequery.Theresultappearsbelowthequeryeditor:

[

{

"_key":"9883",

"_id":"users/9883",

"_rev":"9883",

"age":32,

"name":"JohnSmith"

}

]

Asyoucansee,theentiredocumentincludingthesystemattributesisreturned.DOCUMENT()isafunctiontoretrieveasingledocumentoralistofdocumentsofwhichyouknowthe_keysor_ids.Wereturntheresultofthefunctioncallasourqueryresult,whichisourdocumentinsideoftheresultarray(wecouldhavereturnedmorethanoneresultwithadifferentquery,butevenforasingledocumentasresult,westillgetanarrayatthetoplevel).

Thistypeofqueryiscalleddataaccessquery.Nodataiscreated,changedordeleted.Thereisanothertypeofquerycalleddatamodificationquery.Let'sinsertaseconddocumentusingamodificationquery:

INSERT{name:"KatieFoster",age:27}INTOusers

Thequeryisprettyself-explanatory:theINSERTkeywordtellsArangoDBthatwewanttoinsertsomething.Whattoinsert,adocumentwithtwoattributesinthiscase,followsnext.Thecurlybraces{}signifydocuments,orobjects.Whentalkingaboutrecordsinacollection,wecallthemdocuments.EncodedasJSON,wecallthemobjects.Objectscanalsobenested.Here'sanexample:

{

"name":{

"first":"Katie",

"last":"Foster"

}

}

INTOisamandatorypartofeveryINSERToperationandisfollowedbythecollectionnamethatwewanttostorethedocumentin.Notethattherearenoquotemarksaroundthecollectionname.

Ifyourunabovequery,therewillbeanemptyarrayasresultbecausewedidnotspecifywhattoreturnusingaRETURNkeyword.Itisoptionalinmodificationqueries,butmandatoryindataaccessqueries.EvenwithRETURN,thereturnvaluecanstillbeanemptyarray,e.g.ifthespecifieddocumentwasnotfound.Despitetheemptyresult,theabovequerystillcreatedanewuserdocument.Youcanverifythiswiththedocumentbrowser.

Let'saddanotheruser,butreturnthenewlycreateddocumentthistime:

INSERT{name:"JamesHendrix",age:69}INTOusers

RETURNNEW

NEWisapseudo-variable,whichreferstothedocumentcreatedbyINSERT.Theresultofthequerywilllooklikethis:

[

{

"_key":"10074",

"_id":"users/10074",

"_rev":"10074",

"age":69,

"name":"JamesHendrix"

}

]

GettingStarted

12

Nowthatwehave3usersinourcollection,howtoretrievethemallwithasinglequery?Thefollowingdoesnotwork:

RETURNDOCUMENT("users/9883")

RETURNDOCUMENT("users/9915")

RETURNDOCUMENT("users/10074")

TherecanonlybeasingleRETURNstatementhereandasyntaxerrorisraisedifyoutrytoexecuteit.TheDOCUMENT()functionoffersasecondarysignaturetospecifymultipledocumenthandles,sowecoulddo:

RETURNDOCUMENT(["users/9883","users/9915","users/10074"])

Anarraywiththe_idsofall3documentsispassedtothefunction.Arraysaredenotedbysquarebrackets[]andtheirelementsareseparatedbycommas.

Butwhatifweaddmoreusers?Wewouldhavetochangethequerytoretrievethenewlyaddedusersaswell.Allwewanttosaywithourqueryis:"Foreveryuserinthecollectionusers,returntheuserdocument".WecanformulatethiswithaFORloop:

FORuserINusers

RETURNuser

Itexpressestoiterateovereverydocumentinusersandtouseuserasvariablename,whichwecanusetorefertothecurrentuserdocument.Itcouldalsobecalleddoc,uorahuacatlguacamole,thisisuptoyou.Itisadvisabletouseashortandself-descriptivenamehowever.

Theloopbodytellsthesystemtoreturnthevalueofthevariableuser,whichisasingleuserdocument.Alluserdocumentsarereturnedthisway:

[

{

"_key":"9915",

"_id":"users/9915",

"_rev":"9915",

"age":27,

"name":"KatieFoster"

},

{

"_key":"9883",

"_id":"users/9883",

"_rev":"9883",

"age":32,

"name":"JohnSmith"

},

{

"_key":"10074",

"_id":"users/10074",

"_rev":"10074",

"age":69,

"name":"JamesHendrix"

}

]

Youmayhavenoticedthattheorderofthereturneddocumentsisnotnecessarilythesameastheywereinserted.Thereisnoorderguaranteedunlessyouexplicitlysortthem.WecanaddaSORToperationveryeasily:

FORuserINusers

SORTuser._key

RETURNuser

Thisdoesstillnotreturnthedesiredresult:James(10074)isreturnedbeforeJohn(9883)andKatie(9915).Thereasonisthatthe_keyattributeisastringinArangoDB,andnotanumber.Theindividualcharactersofthestringsarecompared.1islowerthan9andtheresultistherefore"correct".Ifwewantedtousethenumericalvalueofthe_keyattributesinstead,wecouldconvertthestringtoanumberanduseitforsorting.Therearesomeimplicationshowever.Wearebetteroffsortingsomethingelse.Howabouttheage,indescendingorder?

GettingStarted

13

FORuserINusers

SORTuser.ageDESC

RETURNuser

Theuserswillbereturnedinthefollowingorder:James(69),John(32),Katie(27).InsteadofDESCfordescendingorder,ASCcanbeusedforascendingorder.ASCisthedefaultthoughandcanbeomitted.

Wemightwanttolimittheresultsettoasubsetofusers,basedontheageattributeforexample.Let'sreturnusersolderthan30only:

FORuserINusers

FILTERuser.age>30

SORTuser.age

RETURNuser

ThiswillreturnJohnandJames(inthisorder).Katie'sageattributedoesnotfulfillthecriterion(greaterthan30),sheisonly27andthereforenotpartoftheresultset.Wecanmakeheragetoreturnheruserdocumentagain,usingamodificationquery:

UPDATE"9915"WITH{age:40}INusers

RETURNNEW

UPDATEallowstopartiallyeditanexistingdocument.ThereisalsoREPLACE,whichwouldremoveallattributes(exceptfor_keyand_id,whichremainthesame)andonlyaddthespecifiedones.UPDATEontheotherhandonlyreplacesthespecifiedattributesandkeepseverythingelseas-is.

TheUPDATEkeywordisfollowedbythedocumentkey(oradocument/objectwitha_keyattribute)toidentifywhattomodify.TheattributestoupdatearewrittenasobjectaftertheWITHkeyword.INdenotesinwhichcollectiontoperformthisoperationin,justlikeINTO(bothkeywordsareactuallyinterchangablehere).ThefulldocumentwiththechangesappliedisreturnedifweusetheNEWpseudo-variable:

[

{

"_key":"9915",

"_id":"users/9915",

"_rev":"12864",

"age":40,

"name":"KatieFoster"

}

]

IfweusedREPLACEinstead,thenameattributewouldbegone.WithUPDATE,theattributeiskept(thesamewouldapplytoadditionalattributesifwehadthem).

LetusrunourFILTERqueryagain,butonlyreturntheusernamesthistime:

FORuserINusers

FILTERuser.age>30

SORTuser.age

RETURNuser.name

Thiswillreturnthenamesofall3users:

[

"JohnSmith",

"KatieFoster",

"JamesHendrix"

]

Itiscalledaprojectionifonlyasubsetofattributesisreturned.Anotherkindofprojectionistochangethestructureoftheresults:

FORuserINusers

RETURN{userName:user.name,age:user.age}

GettingStarted

14

Thequerydefinestheoutputformatforeveryuserdocument.TheusernameisreturnedasuserNameinsteadofname,theagekeepstheattributekeyinthisexample:

[

{

"userName":"JamesHendrix",

"age":69

},

{

"userName":"JohnSmith",

"age":32

},

{

"userName":"KatieFoster",

"age":40

}

]

Itisalsopossibletocomputenewvalues:

FORuserINusers

RETURNCONCAT(user.name,"'sageis",user.age)

CONCAT()isafunctionthatcanjoinelementstogethertoastring.Weuseitheretoreturnastatementforeveryuser.Asyoucansee,theresultsetdoesnotalwayshavetobeanarrayofobjects:

[

"JamesHendrix'sageis69",

"JohnSmith'sageis32",

"KatieFoster'sageis40"

]

Nowlet'sdosomethingcrazy:foreverydocumentintheuserscollection,iterateoveralluserdocumentsagainandreturnuserpairs,e.g.JohnandKatie.Wecanusealoopinsidealoopforthistogetthecrossproduct(everypossiblecombinationofalluserrecords,3*3=9).Wedon'twantpairingslikeJohn+Johnhowever,solet'seliminatethemwithafiltercondition:

FORuser1INusers

FORuser2INusers

FILTERuser1!=user2

RETURN[user1.name,user2.name]

Weget6pairings.PairslikeJames+JohnandJohn+Jamesarebasicallyredundant,butfairenough:

[

["JamesHendrix","JohnSmith"],

["JamesHendrix","KatieFoster"],

["JohnSmith","JamesHendrix"],

["JohnSmith","KatieFoster"],

["KatieFoster","JamesHendrix"],

["KatieFoster","JohnSmith"]

]

Wecouldcalculatethesumofbothagesandcomputesomethingnewthisway:

FORuser1INusers

FORuser2INusers

FILTERuser1!=user2

RETURN{

pair:[user1.name,user2.name],

sumOfAges:user1.age+user2.age

}

WeintroduceanewattributesumOfAgesandaddupbothagesforthevalue:

[

GettingStarted

15

{

"pair":["JamesHendrix","JohnSmith"],

"sumOfAges":101

},

{

"pair":["JamesHendrix","KatieFoster"],

"sumOfAges":109

},

{

"pair":["JohnSmith","JamesHendrix"],

"sumOfAges":101

},

{

"pair":["JohnSmith","KatieFoster"],

"sumOfAges":72

},

{

"pair":["KatieFoster","JamesHendrix"],

"sumOfAges":109

},

{

"pair":["KatieFoster","JohnSmith"],

"sumOfAges":72

}

]

Ifwewantedtopost-filteronthenewattributetoonlyreturnpairswithasumlessthan100,weshoulddefineavariabletotemporarilystorethesum,sothatwecanuseitinaFILTERstatementaswellasintheRETURNstatement:

FORuser1INusers

FORuser2INusers

FILTERuser1!=user2

LETsumOfAges=user1.age+user2.age

FILTERsumOfAges<100

RETURN{

pair:[user1.name,user2.name],

sumOfAges:sumOfAges

}

TheLETkeywordisfollowedbythedesignatedvariablename(sumOfAges),thenthere'sa=symbolandthevalueoranexpressiontodefinewhatvaluethevariableissupposedtohave.Were-useourexpressiontocalculatethesumhere.WethenhaveanotherFILTERtoskiptheunwantedpairingsandmakeuseofthevariablewedeclaredbefore.Wereturnaprojectionwithanarrayoftheusernamesandthecalculatedage,forwhichweusethevariableagain:

[

{

"pair":["JohnSmith","KatieFoster"],

"sumOfAges":72

},

{

"pair":["KatieFoster","JohnSmith"],

"sumOfAges":72

}

]

Protip:whendefiningobjects,ifthedesiredattributekeyandthevariabletousefortheattributevaluearethesame,youcanuseashorthandnotation:{sumOfAges}insteadof{sumOfAges:sumOfAges}.

Finally,let'sdeleteoneoftheuserdocuments:

REMOVE"9883"INusers

ItdeletestheuserJohn(_key:"9883").Wecouldalsoremovedocumentsinaloop(samegoesforINSERT,UPDATEandREPLACE):

FORuserINusers

FILTERuser.age>=30

REMOVEuserINusers

GettingStarted

16

Thequerydeletesalluserswhoseageisgreaterthanorequalto30.

Howtocontinue

ThereisalotmoretodiscoverinAQLandmuchmorefunctionalitythatArangoDBoffers.Continuereadingtheotherchaptersandexperimentwithatestdatabasetofosteryourknowledge.

IfyouwanttowritemoreAQLqueriesrightnow,havealookhere:

DataQueries:dataaccessandmodificationqueriesHigh-leveloperations:detaileddescriptionsofFOR,FILTERandmoreoperationsnotshowninthisintroductionFunctions:areferenceofallprovidedfunctions

ArangoDBprogramsTheArangoDBpackagecomeswiththefollowingprograms:

arangod:TheArangoDBdatabasedaemon.ThisserverprogramisintendedtorunasadaemonprocessandtoservethevariousclientsconnectiontotheserverviaTCP/HTTP.

arangosh:TheArangoDBshell.Aclientthatimplementsaread-eval-printloop(REPL)andprovidesfunctionstoaccessandadministratetheArangoDBserver.

arangoimport:AbulkimporterfortheArangoDBserver.ItsupportsJSONandCSV.

arangodump:AtooltocreatebackupsofanArangoDBdatabaseinJSONformat.

arangorestore:AtooltoloaddataofabackupbackintoanArangoDBdatabase.

arango-dfdb:AdatafiledebuggerforArangoDB.ItisprimarilyintendedtobeusedduringdevelopmentofArangoDB.

arangobench:Abenchmarkandtesttool.Itcanbeusedforperformanceandserverfunctiontesting.

GettingStarted

17

InstallingFirstofall,downloadandinstallthecorrespondingRPMorDebianpackageorusehomebrewonMacOSX.Youcanfindpackagesforvariousoperationsystemsatourinstallsection,includinginstallersforWindows.

Howtodothatindetailisdescribedinthesubchaptersofthissection.

Onhowtosetupacluster,checkouttheDeploymentchapter.

Installing

18

LinuxVisittheofficialArangoDBinstallpageanddownloadthecorrectpackageforyourLinuxdistribution.Youcanfindbinarypackagesforthemostcommondistributionsthere.Followtheinstructionstouseyourfavoritepackagemanagerforthemajordistributions.AftersettinguptheArangoDBrepositoryyoucaneasilyinstallArangoDBusingyum,aptitude,urpmiorzypper.Debianbasedpackageswillaskforapasswordduringinstallation.ForanunattendedinstallationforDebian,seebelow.Red-Hatbasedpackageswillsetarandompasswordduringinstallation.Forotherdistributionsortochangethepassword,runarango-secure-installationtosetarootpassword.Alternatively,seeCompilingifyouwanttobuildArangoDByourself.Startupthedatabaseserver.

Normally,thisisdonebyexecutingthefollowingcommand:

unix>/etc/init.d/arangodstart

Itwillstarttheserver,anddothataswellatsystemboottime.

Tostoptheserveryoucanusethefollowingcommand:

unix>/etc/init.d/arangodstop

TheexactcommandsdependonyourLinuxdistribution.Youmayrequirerootprivilegestoexecutethesecommands.

LinuxMint

PleaseusethecorrespondingUbuntuorDebianpackages.

UnattendedInstallation

Debianbasedpackagewillaskforapasswordduringinstallation.Forunattendedinstallation,youcansetthepasswordusingthedebconfhelpers.

echoarangodb3arangodb3/passwordpasswordNEWPASSWORD|debconf-set-selections

echoarangodb3arangodb3/password_againpasswordNEWPASSWORD|debconf-set-selections

Thecommandsshouldbeexecutedpriortotheinstallation.

Red-Hatbasedpackageswillsetarandompasswordduringinstallation.Ifyouwanttoforceapassword,execute

ARANGODB_DEFAULT_ROOT_PASSWORD=NEWPASSWORDarango-secure-installation

Thecommandshouldbeexecutedaftertheinstallation.

Non-StandardInstallationIfyoucompiledArangoDBfromsourceanddidnotuseanyinstallationpackage–orusingnon-defaultlocationsand/ormultipleArangoDBinstancesonthesamehost–youmaywanttostarttheserverprocessmanually.Youcandosobyinvokingthearangodbinaryfromthecommandlineasshownbelow:

unix>/usr/local/sbin/arangod/tmp/vocbase

20ZZ-XX-YYT12:37:08Z[8145]INFOusingbuilt-inJavaScriptstartupfiles

20ZZ-XX-YYT12:37:08Z[8145]INFOArangoDB(version1.x.y)isreadyforbusiness

20ZZ-XX-YYT12:37:08Z[8145]INFOHaveFun!

Linux

19

Tostopthedatabaseservergracefully,youcaneitherpressCTRL-CorbysendtheSIGINTsignaltotheserverprocess.Onmanysystemsthiscanbeachievedwiththefollowingcommand:

unix>kill-2`pidofarangod`

Onceyoustartedtheserver,thereshouldbearunninginstanceofarangod-theArangoDBdatabaseserver.

unix>psauxw|fgreparangod

arangodb145360.10.6530726423464s002S1:21pm0:00.18/usr/local/sbin/arangod

Ifthereisnosuchprocess,checkthelogfile/var/log/arangodb/arangod.logforerrors.Ifyouseealogmessagelike

2012-12-03T11:35:29Z[12882]ERRORDatabasedirectoryversion(1)islowerthanserverversion(1.2).

2012-12-03T11:35:29Z[12882]ERRORItseemslikeyouhaveupgradedtheArangoDBbinary.Ifthisiswhatyouwantedtodo,pleas

erestartwiththe--database.auto-upgradeoptiontoupgradethedatainthedatabasedirectory.

2012-12-03T11:35:29Z[12882]FATALDatabaseversioncheckfailed.Pleasestarttheserverwiththe--database.auto-upgradeopti

on

makesuretostarttheserveroncewiththe--database.auto-upgradeoption.

Notethatyoumayhavetoenableloggingfirst.Ifyoustarttheserverinashell,youshouldseeerrorsloggedthereaswell.

Linux

20

MacOSXThepreferredmethodforinstallingArangoDBunderMacOSXishomebrew.However,incaseyouarenotusinghomebrew,weprovideacommand-lineapporgraphicalappwhichcontainsalltheexecutables.

Homebrew

Ifyouareusinghomebrew,thenyoucaninstallthelatestreleasedstableversionofArangoDBusingbrewasfollows:

brewinstallarangodb

ThiswillinstallthecurrentstableversionofArangoDBandalldependencieswithinyourHomebrewtree.Notethattheserverwillbeinstalledas:

/usr/local/sbin/arangod

Youcanstarttheserverbyrunningthecommand/usr/local/sbin/arangod&.

Configurationfileislocatedat

/usr/local/etc/arangodb3/arangod.conf

TheArangoDBshellwillbeinstalledas:

/usr/local/bin/arangosh

YoucanuninstallArangoDBusing:

brewuninstallarangodb

However,incaseyoustartedArangoDBusingthelaunchctl,youneedtounloaditbeforeuninstallingtheserver:

launchctlunload~/Library/LaunchAgents/homebrew.mxcl.arangodb.plist

ThenremovetheLaunchAgent:

rm~/Library/LaunchAgents/homebrew.mxcl.arangodb.plist

Note:IfthelatestArangoDBVersionisnotshowninhomebrew,youalsoneedtoupdatehomebrew:

brewupdate

Knownissues

Performance-theLLVMdeliveredasofMacOSXElCapitanbuildsslowbinaries.UseGCCinstead,untilthisissuehasbeenfixedbyApple.theCommandlineargumentparsingdoesn'tacceptblanksinfilenames;theCLIversionbelowdoes.ifyouneedtochangeserverendpointwhilestartinghomebrewversion,youcaneditarangod.conffileanduncommentlinewithendpointneeded,e.g.:

[server]

endpoint=tcp://0.0.0.0:8529

MacOSX

21

GraphicalApp

Incaseyouarenotusinghomebrew,wealsoprovideagraphicalapp.Youcandownloaditfromhere.

ChooseMacOSX.DownloadandinstalltheapplicationArangoDBinyourapplicationfolder.

CommandlineAppIncaseyouarenotusinghomebrew,wealsoprovideacommand-lineapp.Youcandownloaditfromhere.

ChooseMacOSX.DownloadandinstalltheapplicationArangoDB-CLIinyourapplicationfolder.

Startingtheapplicationwillstarttheserverandopenaterminalwindowshowingyouthelog-file.

ArangoDBserverhasbeenstarted

Thedatabasedirectoryislocatedat

'/Applications/ArangoDB-CLI.app/Contents/MacOS/opt/arangodb/var/lib/arangodb'

Thelogfileislocatedat

'/Applications/ArangoDB-CLI.app/Contents/MacOS/opt/arangodb/var/log/arangodb/arangod.log'

Youcanaccesstheserverusingabrowserat'http://127.0.0.1:8529/'

orstarttheArangoDBshell

'/Applications/ArangoDB-CLI.app/Contents/MacOS/arangosh'

Switchingtolog-filenow,killingthiswindowswillNOTstoptheserver.

2013-10-27T19:42:04Z[23840]INFOArangoDB(version1.4.devel[darwin])isreadyforbusiness.Havefun!

Notethatitispossibletoinstallboth,thehomebrewversionandthecommand-lineapp.Youshould,however,edittheconfigurationfilesofoneversionandchangetheportused.

MacOSX

22

WindowsThedefaultinstallationdirectoryisC:\ProgramFiles\ArangoDB-3.x.x.Duringtheinstallationprocessyoumaychangethis.InthefollowingdescriptionwewillassumethatArangoDBhasbeeninstalledinthelocation<ROOTDIR>.

Youhavetobecarefulwhenchoosinganinstallationdirectory.Youneedeitherwritepermissiontothisdirectoryoryouneedtomodifytheconfigurationfilefortheserverprocess.InthelattercasethedatabasedirectoryandtheFoxxdirectoryhavetobewritablebytheuser.

Single-andMultiuserInstallation

TherearetwomainmodesfortheinstallerofArangoDB.Theinstallerletsyouselect:

multiuserinstallation(default;adminprivilegesrequired)WillinstallArangoDBasservice.singleuserinstallationAllowtoinstallArangodbasnormaluser.Requiresmanualstartingofthedatabaseserver.

CheckBoxesThecheckboxesallowyoutochoseweatheryouwantto:

chosecustominstallpathsdoanautomaticupgradekeepanbackupofyourdataaddexecutablestopathcreateadesktopicon

ornot.

CustomInstallPaths

Thischeckboxcontrolsifyouwillbeabletooverridethedefaultpathsfortheinstallationinsubsequentsteps.

Thedefaultinstallationpathsare:

MultiUserDefault:

Installation:C:\ProgramFiles\ArangoDB-3.x.xDataBase:C:\ProgramData\ArangoDBFoxxService:C:\ProgramData\ArangoDB-apps

SingleUserDefault:

Installation:C:\Users\\\AppData\Local\ArangoDB-3.x.xDataBase:C:\Users\\\AppData\Local\ArangoDBFoxxService:C:\Users\\\AppData\Local\ArangoDB-apps

Wearenotusingtheroamingpartoftheuser'sprofile,becausedoingsoavoidsthedatabeingsyncedtothewindowsdomaincontroller.

AutomaticUpgrade

Ifthischeckboxisselectedtheinstallerwillattempttoperformanautomaticupdate.FormoreinformationpleaseseeUpgradingfromPreviousVersion.

KeepBackup

Selectthistocreateabackupofyourdatabasedirectoryduringautomaticupgrade.Thebackupwillbecreatednexttoyourcurrentdatabasedirectorysuffixedbyatimestamp.

Windows

23

AddtoPath

Selectthistoaddthebinarydirectorytoyoursystem'spath(multiuserinstallation)oruser'spath(singleuserinstallation).

DesktopIcon

SelectifyouwanttheinstallertocreateDesktopIconsthatletyou:

accessthewebintefacestartthecommandlineclient(arangosh)startthedatabaseserver(singleuserinstallationonly)

UpgradingfromPreviousVersionIfyouareupgradingArangoDBfromanearlierversionyouneedtocopyyourolddatabasedirectorytothenewdefaultpaths.Upgradingwillkeepyourolddata,passwordandchoiceofstorageengineasitis.SwitchingtotheRocksDBstorageenginerequiresaexportandreimportofyourdata.

StartingIfyouinstalledArangoDBformultipleusers(asaservice)itisautomaticallystarted.OtherwiseyouneedtousethelinkthatwascreatedonyouDesktopifyouchosetolettheinstallercreatedesktopiconsor

theexecutablearangod.exelocatedin<ROOTDIR>\bin.Thiswillusetheconfigurationfilearangod.conflocatedin<ROOTDIR>\etc\arangodb,whichyoucanadjusttoyourneedsandusethedatadirectory<ROOTDIR>\var\lib\arangodb.Thisistheplacewhereallyourdata(databasesandcollections)willbestoredbydefault.

Pleasechecktheoutputofthearangod.exeexecutablebeforegoingon.Iftheserverstartedsuccessfully,youshouldseealineArangoDBisreadyforbusiness.Havefun!attheendofitsoutput.

Wenowwishtocheckthattheinstallationisworkingcorrectlyandtodothiswewillbeusingtheadministrationwebinterface.Executearangod.exeifyouhavenotalreadydoneso,thenopenupyourwebbrowserandpointittothepage:

http://127.0.0.1:8529/

AdvancedStarting

Ifyouwanttoprovideourownstartscripts,youcansettheenvironmentvariableARANGODB_CONFIG_PATH.Thisvariableshouldpointtoadirectorycontainingtheconfigurationfiles.

UsingtheClient

ToconnecttoanalreadyrunningArangoDBserverinstance,thereisashellarangosh.exelocatedin<ROOTDIR>\bin.Thisstartsashellwhichcanbeused–amongstotherthings–toadministerandqueryalocalorremoteArangoDBserver.

Notethatarangosh.exedoesNOTstartaseparateserver,itonlystartstheshell.Touseityoumusthaveaserverrunningsomewhere,e.g.byusingthearangod.exeexecutable.

arangosh.exeusesconfigurationfromthefilearangosh.conflocatedin<ROOTDIR>\etc\arangodb\.Pleaseadjustthistoyourneedsifyouwanttousedifferentconnectionsettingsetc.

Uninstalling

TouninstalltheArangoserverapplicationyoucanusethewindowscontrolpanel(asyouwouldnormallyuninstallanapplication).Notehowever,thatanydatafilescreatedbytheArangoserverwillremainaswellasthe<ROOTDIR>directory.Tocompletetheuninstallationprocess,removethedatafilesandthe<ROOTDIR>directorymanually.

Windows

24

LimitationsforCygwin

PleasenotesomeimportantlimitationswhenrunningArangoDBunderCygwin:StartingArangoDBcanbestartedfromoutofaCygwinterminal,butpressingCTRL-Cwillforcefullykilltheserverprocesswithoutgivingitachancetohandlethekillsignal.Inthiscase,aregularservershutdownisnotpossible,whichmayleaveafileLOCKaroundintheserver'sdatadirectory.ThisfileneedstoberemovedmanuallytomakeArangoDBstartagain.Additionally,asArangoDBdoesnothaveachancetohandlethekillsignal,theservercannotforcefullyflushanydatatodiskonshutdown,leadingtopotentialdataloss.WhenstartingArangoDBfromaCygwinterminalitmightalsohappenthatnoerrorsareprintedintheterminaloutput.StartingArangoDBfromanMS-DOScommandpromptdoesnotimposetheselimitationsandisthusthepreferredmethod.

PleasenotethatArangoDBusesUTF-8asitsinternalencodingandthatthesystemconsolemustsupportaUTF-8codepage(65001)andfont.ItmaybenecessarytomanuallyswitchtheconsolefonttoafontthatsupportsUTF-8.

Windows

25

CompilingArangoDBfromscratchThefollowingsectionsdescribehowtocompileandbuildtheArangoDBfromscratch.ArangoDBwillcompileonmostLinuxandMacOSXsystems.WeassumethatyouusetheGNUC/C++compilerorclang/clang++tocompilethesource.ArangoDBhasbeentestedwiththesecompilers,butshouldbeabletocompilewithanyPosix-compliant,C++11-enabledcompiler.PleaseletusknowwhetheryousuccessfullycompileditwithanotherC/C++compiler.

Bydefault,cloningthegithubrepositorywillcheckoutdevel.ThisversioncontainsthedevelopmentversionoftheArangoDB.UsethisbranchifyouwanttomakechangestotheArangoDBsource.

OnWindowsyoufirstneedtoallowandenablesymlinksforyouruser.

PleasecheckoutthecookbookonhowtocompileArangoDB.

Compiling

26

AuthenticationArangoDBallowstorestrictaccesstodatabasestocertainusers.Allusersofthesystemdatabaseareconsideredadministrators.Duringinstallationadefaultuserrootiscreated,whichhasaccesstoalldatabases.

Youshouldcreateadatabaseforyourapplicationtogetherwithauserthathasaccessrightstothisdatabase.SeeManagingUsers.

Usethearangoshtocreateanewdatabaseanduser.

arangosh>db._createDatabase("example");

arangosh>varusers=require("@arangodb/users");

arangosh>users.save("root@example","password");

arangosh>users.grantDatabase("root@example","example");

Youcannowconnecttothenewdatabaseusingtheuserroot@example.

shell>arangosh--server.username"root@example"--server.databaseexample

Authentication

27

AccessingtheWebInterfaceArangoDBcomeswithabuilt-inwebinterfaceforadministration.ThewebinterfacecanbeaccessedviatheURL:

http://127.0.0.1:8529

Ifeverythingworksasexpected,youshouldseetheloginview:

FormoreinformationontheArangoDBwebinterface,seeWebInterface

AccessingtheWebInterface

28

ComingfromSQLIfyouworkedwitharelationaldatabasemanagementsystem(RDBMS)suchasMySQL,MariaDBorPostgreSQL,youwillbefamiliarwithitsquerylanguage,adialectofSQL(StructuredQueryLanguage).

ArangoDB'squerylanguageiscalledAQL.Therearesomesimilaritiesbetweenbothlanguagesdespitethedifferentdatamodelsofthedatabasesystems.ThemostnotabledifferenceisprobablytheconceptofloopsinAQL,whichmakesitfeelmorelikeaprogramminglanguage.Itsuitestheschema-lessmodelmorenaturalandmakesthequerylanguageverypowerfulwhileremainingeasytoreadandwrite.

TogetstartedwithAQL,havealookatourdetailedcomparisonofSQLandAQL.ItwillalsohelpyoutotranslateSQLqueriestoAQLwhenmigratingtoArangoDB.

Howdobrowsevectorstranslateintodocumentqueries?

IntraditionalSQLyoumayeitherfetchallcolumnsofatablerowbyrow,usingSELECT*FROMtable,orselectasubsetofthecolumns.Thelistoftablecolumnstofetchiscommonlycalledcolumnlistorbrowsevector:

SELECTcolumnA,columnB,columnZFROMtable

Sincedocumentsaren'ttwo-dimensional,andneitherdoyouwanttobelimitedtoreturningtwo-dimensionallists,therequirementsforaquerylanguagearehigher.AQListhusalittlebitmorecomplexthanplainSQLatfirst,butoffersmuchmoreflexibilityinthelongrun.Itletsyouhandlearbitrarilystructureddocumentsinconvenientways,mostlyleanedonthesyntaxusedinJavaScript.

Composingthedocumentstobereturned

TheAQLRETURNstatementreturnsoneitemperdocumentitishanded.Youcanreturnthewholedocument,orjustpartsofit.GiventhatoneDocumentisadocument(retrievedlikeLEToneDocument=DOCUMENT("myusers/3456789")forinstance),itcanbereturnedas-islikethis:

RETURNoneDocument

[

{

"_id":"myusers/3456789",

"_key":"3456789"

"_rev":"14253647",

"firstName":"John",

"lastName":"Doe",

"address":{

"city":"Gotham",

"street":"RoadToNowhere1"

},

"hobbies":[

{name:"swimming",howFavorite:10},

{name:"biking",howFavorite:6},

{name:"programming",howFavorite:4}

]

}

]

Returnthehobbiessub-structureonly:

RETURNoneDocument.hobbies

[

[

{name:"swimming",howFavorite:10},

{name:"biking",howFavorite:6},

{name:"programming",howFavorite:4}

ComingfromSQL

29

]

]

Returnthehobbiesandtheaddress:

RETURN{

hobbies:oneDocument.hobbies,

address:oneDocument.address

}

[

{

hobbies:[

{name:"swimming",howFavorite:10},

{name:"biking",howFavorite:6},

{name:"programming",howFavorite:4}

],

address:{

"city":"Gotham",

"street":"RoadToNowhere1"

}

}

]

Returnthefirsthobbyonly:

RETURNoneDocument.hobbies[0].name

[

"swimming"

]

Returnalistofallhobbystrings:

RETURN{hobbies:oneDocument.hobbies[*].name}

[

{hobbies:["swimming","biking","porgramming"]}

]

MorecomplexarrayandobjectmanipulationscanbedoneusingAQLfunctionsandoperators.

ComingfromSQL

30

Highlights

Version3.3

EnterpriseEdition

DatacentertoDatacenterReplication:ReplicatetheentirestructureandcontentofanArangoDBclusterasynchronouslytoanotherclusterinadifferentdatacenterwithArangoSync.Multi-datacentersupportmeansyoucanfallbacktoareplicaofyourclusterincaseofadisasterinonedatacenter.

EncryptedBackups:ArangodumpcancreatebackupsencryptedwithasecretkeyusingAES256blockcipher.

AllEditions

Server-levelReplication:Inadditiontoper-databasereplication,thereisnowanadditionalglobalApplier.Starttheglobalreplicationontheslaveonceandallcurrentandfuturedatabaseswillbereplicatedfromthemastertotheslaveautomatically.

AsynchronousFailover:Makeasingleserverinstanceresilientwithasecondserverinstance,oneasmasterandtheotherasasynchronouslyreplicatingslave,withautomaticfailovertotheslaveifthemastergoesdown.

AlsoseeWhat'sNewin3.3.

Version3.2

RocksDBStorageEngine:YoucannowuseasmuchdatainArangoDBasyoucanfitonyourdisk.Plus,youcanenjoyperformanceboostsonwritesbyhavingonlydocument-levellocks

Pregel:WeimplementeddistributedgraphprocessingwithPregeltodiscoverhiddenpatterns,identifycommunitiesandperformin-depthanalyticsoflargegraphdatasets.

Fault-TolerantFoxx:TheFoxxmanagementinternalshavebeenrewrittenfromthegrounduptomakesuremulti-coordinatorclustersetupsalwayskeeptheirservicesinsyncandnewcoordinatorsarefullyinitializedevenwhenallexistingcoordinatorsareunavailable.

Enterprise:Workingwithsomeofourlargestcustomers,we’veaddedfurthersecurityandscalabilityfeaturestoArangoDBEnterpriselikeLDAPintegration,EncryptionatRest,andthebrandnewSatelliteCollections.

AlsoseeWhat'sNewin3.2.

Version3.1SmartGraphs:Scalewithgraphstoaclusterandstayperformant.WithSmartGraphsyoucanusethe"smartness"ofyourapplicationlayertoshardyourgraphefficientlytoyourmachinesandlettraversalsrunlocally.

EncryptionControl:ChooseyourlevelofSSLencryption

Auditing:KeepadetailedlogofalltheimportantthingsthathappenedinArangoDB.

AlsoseeWhat'sNewin3.1.

Version3.0

self-organizingclusterwithsynchronousreplication,master/mastersetup,sharednothingarchitecture,clustermanagementagency.

Deeplyintegrated,nativeAQLgraphtraversal

Highlights

31

VelocyPackasnewinternalbinarystorageformataswellasforintermediateAQLvalues.

PersistentindexesviaRocksDBsuitableforsortingandrangequeries.

Foxx3.0:overhauledJSframeworkfordata-centricmicroservices

SignificantlyimprovedWebInterface

AlsoseeWhat'sNewin3.0.

Highlights

32

ScalabilityArangoDBisadistributeddatabasesupportingmultipledatamodels,andcanthusbescaledhorizontally,thatis,byusingmanyservers,typicallybasedoncommodityhardware.Thisapproachnotonlydeliversperformanceaswellascapacityimprovements,butalsoachievesresiliencebymeansofreplicationandautomaticfail-over.Furthermore,onecanbuildsystemsthatscaletheircapacitydynamicallyupanddownautomaticallyaccordingtodemand.

OnecanalsoscaleArangoDBvertically,thatis,byusingeverlargerservers.ThereisnobuiltinlimitationinArangoDB,forexample,theserverwillautomaticallyusemorethreadsifmoreCPUsarepresent.

However,scalingverticallyhasthedisadvantagethatthecostsgrowfasterthanlinearwiththesizeoftheserver,andnoneoftheresilienceanddynamicalcapabilitiescanbeachievedinthisway.

InthischapterweexplainthedistributedarchitectureofArangoDBanddiscussitsscalabilityfeaturesandlimitations:

ArangoDB'sdistributedarchitectureDifferentdatamodelsandscalabilityLimitations

Scalability

33

Architecture

TheclusterarchitectureofArangoDBisaCPmaster/mastermodelwithnosinglepointoffailure.With"CP"wemeanthatinthepresenceofanetworkpartition,thedatabaseprefersinternalconsistencyoveravailability.With"master/master"wemeanthatclientscansendtheirrequeststoanarbitrarynode,andexperiencethesameviewonthedatabaseregardless."Nosinglepointoffailure"meansthattheclustercancontinuetoserverequests,evenifonemachinefailscompletely.

Inthisway,ArangoDBhasbeendesignedasadistributedmulti-modeldatabase.Thissectiongivesashortoutlineontheclusterarchitectureandhowtheabovefeaturesandcapabilitiesareachieved.

StructureofanArangoDBcluster

AnArangoDBclusterconsistsofanumberofArangoDBinstanceswhichtalktoeachotheroverthenetwork.Theyplaydifferentroles,whichwillbeexplainedindetailbelow.Thecurrentconfigurationoftheclusterisheldinthe"Agency",whichisahighly-availableresilientkey/valuestorebasedonanoddnumberofArangoDBinstancesrunningRaftConsensusProtocol.

ForthevariousinstancesinanArangoDBclusterthereare4distinctroles:Agents,Coordinators,PrimaryandSecondaryDBservers.Inthefollowingsectionswewillshedlightoneachofthem.NotethatthetasksforallrolesrunthesamebinaryfromthesameDockerimage.

Agents

OneormultipleAgentsformtheAgencyinanArangoDBcluster.TheAgencyisthecentralplacetostoretheconfigurationinacluster.Itperformsleaderelectionsandprovidesothersynchronizationservicesforthewholecluster.WithouttheAgencynoneoftheothercomponentscanoperate.

Whilegenerallyinvisibletotheoutsideitistheheartofthecluster.Assuch,faulttoleranceisofcourseamusthavefortheAgency.ToachievethattheAgentsareusingtheRaftConsensusAlgorithm.ThealgorithmformallyguaranteesconflictfreeconfigurationmanagementwithintheArangoDBcluster.

AtitscoretheAgencymanagesabigconfigurationtree.Itsupportstransactionalreadandwriteoperationsonthistree,andotherserverscansubscribetoHTTPcallbacksforallchangestothetree.

Coordinators

Coordinatorsshouldbeaccessiblefromtheoutside.Thesearetheonestheclientstalkto.TheywillcoordinateclustertaskslikeexecutingqueriesandrunningFoxxservices.Theyknowwherethedataisstoredandwilloptimizewheretorunusersuppliedqueriesorpartsthereof.Coordinatorsarestatelessandcanthuseasilybeshutdownandrestartedasneeded.

PrimaryDBservers

PrimaryDBserversaretheoneswherethedataisactuallyhosted.Theyhostshardsofdataandusingsynchronousreplicationaprimarymayeitherbeleaderorfollowerforashard.

Theyshouldnotbeaccessedfromtheoutsidebutindirectlythroughthecoordinators.Theymayalsoexecutequeriesinpartorasawholewhenaskedbyacoordinator.

Secondaries

SecondaryDBserversareasynchronousreplicasofprimaries.Ifoneisusingonlysynchronousreplication,onedoesnotneedsecondariesatall.Foreachprimary,therecanbeoneormoresecondaries.Sincethereplicationworksasynchronously(eventualconsistency),thereplicationdoesnotimpedetheperformanceoftheprimaries.Ontheotherhand,theirreplicaofthedatacanbeslightlyoutofdate.Thesecondariesareperfectlysuitableforbackupsastheydon'tinterferewiththenormalclusteroperation.

ClusterID

Everynon-AgencyArangoDBinstanceinaclusterisassignedauniqueIDduringitsstartup.UsingitsIDanodeisidentifiablethroughoutthecluster.AllclusteroperationswillcommunicateviathisID.

Architecture

34

Sharding

UsingtherolesoutlinedaboveanArangoDBclusterisabletodistributedatainsocalledshardsacrossmultipleprimaries.Fromtheoutsidethisprocessisfullytransparentandassuchweachievethegoalsofwhatothersystemscall"master-masterreplication".InanArangoDBclusteryoutalktoanycoordinatorandwheneveryoureadorwritedataitwillautomaticallyfigureoutwherethedataisstored(read)ortobestored(write).TheinformationabouttheshardsissharedacrossthecoordinatorsusingtheAgency.

AlsoseeShardingintheAdministrationchapter.

Manysensibleconfigurations

Thisarchitectureisveryflexibleandthusallowsmanyconfigurations,whicharesuitablefordifferentusagescenarios:

1. ThedefaultconfigurationistorunexactlyonecoordinatorandoneprimaryDBserveroneachmachine.Thisachievestheclassicalmaster/mastersetup,sincethereisaperfectsymmetrybetweenthedifferentnodes,clientscanequallywelltalktoanyoneofthecoordinatorsandallexposethesameviewtothedatastore.

2. OnecandeploymorecoordinatorsthanDBservers.ThisisasensibleapproachifoneneedsalotofCPUpowerfortheFoxxservices,becausetheyrunonthecoordinators.

3. OnecandeploymoreDBserversthancoordinatorsifmoredatacapacityisneededandthequeryperformanceisthelesserbottleneck

4. Onecandeployacoordinatoroneachmachinewhereanapplicationserver(e.g.anode.jsserver)runs,andtheAgentsandDBserversonaseparatesetofmachineselsewhere.Thisavoidsanetworkhopbetweentheapplicationserverandthedatabaseandthusdecreaseslatency.Essentially,thismovessomeofthedatabasedistributionlogictothemachinewheretheclientruns.

Theseforshallsufficefornow.TheimportantpieceofinformationhereisthatthecoordinatorlayercanbescaledanddeployedindependentlyfromtheDBserverlayer.

Replication

ArangoDBofferstwowaysofdatareplicationwithinacluster,synchronousandasynchronous.Inthissectionweexplainsomedetailsandhighlighttheadvantagesanddisadvantagesrespectively.

Synchronousreplicationwithautomaticfail-over

Synchronousreplicationworksonaper-shardbasis.Oneconfiguresforeachcollection,howmanycopiesofeachshardarekeptinthecluster.Atanygiventime,oneofthecopiesisdeclaredtobethe"leader"andallotherreplicasare"followers".WriteoperationsforthisshardarealwayssenttotheDBserverwhichhappenstoholdtheleadercopy,whichinturnreplicatesthechangestoallfollowersbeforetheoperationisconsideredtobedoneandreportedbacktothecoordinator.Readoperationsareallservedbytheserverholdingtheleadercopy,thisallowstoprovidesnapshotsemanticsforcomplextransactions.

IfaDBserverfailsthatholdsafollowercopyofashard,thentheleadercannolongersynchronizeitschangestothatfollower.Afterashorttimeout(3seconds),theleadergivesuponthefollower,declaresittobeoutofsync,andcontinuesservicewithoutthefollower.Whentheserverwiththefollowercopycomesback,itautomaticallyresynchronizesitsdatawiththeleaderandsynchronousreplicationisrestored.

IfaDBserverfailsthatholdsaleadercopyofashard,thentheleadercannolongerserveanyrequests.ItwillnolongersendaheartbeattotheAgency.Therefore,asupervisionprocessrunningintheRaftleaderoftheAgency,cantakethenecessaryaction(after15secondsofmissingheartbeats),namelytopromoteoneoftheserversthatholdin-syncreplicasoftheshardtoleaderforthatshard.ThisinvolvesareconfigurationintheAgencyandleadstothefactthatcoordinatorsnowcontactadifferentDBserverforrequeststothisshard.Serviceresumes.Theothersurvivingreplicasautomaticallyresynchronizetheirdatawiththenewleader.WhentheDBserverwiththeoriginalleadercopycomesback,itnoticesthatitnowholdsafollowerreplica,resynchronizesitsdatawiththenewleaderandorderisrestored.

Allsharddatasynchronizationsaredoneinanincrementalway,suchthatresynchronizationsarequick.Thistechnologyallowstomoveshards(followerandleaderones)betweenDBserverswithoutserviceinterruptions.Therefore,anArangoDBclustercanmoveallthedataonaspecificDBservertootherDBserversandthenshutdownthatserverinacontrolledway.ThisallowstoscaledownanArangoDBclusterwithoutserviceinterruption,lossoffaulttoleranceordataloss.Furthermore,onecanre-balancethedistributionoftheshards,eithermanuallyorautomatically.

AlltheseoperationscanbetriggeredviaaREST/JSONAPIorviathegraphicalwebUI.Allfail-overoperationsarecompletelyhandledwithintheArangoDBcluster.

Architecture

35

Obviously,synchronousreplicationinvolvesacertainincreasedlatencyforwriteoperations,simplybecausethereisonemorenetworkhopwithintheclusterforeveryrequest.Thereforetheusercansetthereplicationfactorto1,whichmeansthatonlyonecopyofeachshardiskept,therebyswitchingoffsynchronousreplication.Thisisasuitablesettingforlessimportantoreasilyrecoverabledataforwhichlowlatencywriteoperationsmatter.

Asynchronousreplicationwithautomaticfail-over

Asynchronousreplicationworksdifferently,inthatitisorganizedusingprimaryandsecondaryDBservers.Eachsecondaryserverreplicatesallthedataheldonaprimarybypollinginanasynchronousway.Thisprocesshasverylittleimpactontheperformanceoftheprimary.Thedisadvantageisthatthereisadelaybetweentheconfirmationofawriteoperationthatissenttotheclientandtheactualreplicationofthedata.Ifthemasterserverfailsduringthisdelay,thencommittedandconfirmeddatacanbelost.

Nevertheless,wealsoofferautomaticfail-overwiththissetup.Contrarytothesynchronouscase,herethefail-overmanagementisdonefromoutsidetheArangoDBcluster.InafutureversionwemightmovethismanagementintothesupervisionprocessintheAgency,butasofnow,themanagementisdoneviatheMesosframeworkschedulerforArangoDB(seebelow).

ThegranularityofthereplicationisawholeArangoDBinstancewithalldatathatresidesonthatinstance,whichmeansthatyouneedtwiceasmanyinstancesaswithoutasynchronousreplication.Synchronousreplicationismoreflexibleinthatrespect,youcanhavesmallerandlargerinstances,andifonefails,thedatacanberebalancedacrosstheremainingones.

Microservicesandzeroadministation

ThedesignandcapabilitiesofArangoDBaregearedtowardsusageinmodernmicroservicearchitecturesofapplications.WiththeFoxxservicesitisveryeasytodeployadatacentricmicroservicewithinanArangoDBcluster.

Inaddition,onecandeploymultipleinstancesofArangoDBwithinthesameproject.Onepartoftheprojectmightneedascalabledocumentstore,anothermightneedagraphdatabase,andyetanothermightneedthefullpowerofamulti-modeldatabaseactuallymixingthevariousdatamodels.Thereareenormousefficiencybenefitstobereapedbybeingabletouseasingletechnologyforvariousrolesinaproject.

TosimplifyliveofthedevopsinsuchascenariowetryasmuchaspossibletouseazeroadministrationapproachforArangoDB.ArunningArangoDBclusterisresilientagainstfailuresandessentiallyrepairsitselfincaseoftemporaryfailures.Seethenextsectionforfurthercapabilitiesinthisdirection.

ApacheMesosintegration

Forthedistributedsetup,weusetheApacheMesosinfrastructurebydefault.ArangoDBisafullycertifiedpackageforDC/OSandcanthusbedeployedessentiallywithafewmouseclicksorasinglecommand,onceyouhaveanexistingDC/OScluster.ButevenonaplainApacheMesosclusteronecandeployArangoDBviaMarathonwithasingleAPIcallandsomeJSONconfiguration.

Theadvantageofthisapproachisthatwecannotonlyimplementtheinitialdeployment,butalsothelatermanagementofautomaticreplacementoffailedinstancesandthescalingoftheArangoDBcluster(triggeredmanuallyorevenautomatically).SinceallmanipulationsareeitherviathegraphicalwebUIorviaJSON/RESTcalls,onecanevenimplementauto-scalingveryeasily.

ADC/OSclusterisaverynaturalenvironmenttodeploymicroservicearchitectures,sinceitissoconvenienttodeployvariousservices,includingpotentiallymultipleArangoDBclusterinstanceswithinthesameDC/OScluster.Thebuilt-inservicediscoverymakesitextremelysimpletoconnectthevariousmicroservicesandMesosautomaticallytakescareofthedistributionanddeploymentofthevarioustasks.

SeetheDeploymentchapteranditssubsectionsforinstructions.

ItispossibletodeployanArangoDBclusterbysimplylaunchingabunchofDockercontainerswiththerightcommandlineoptionstolinkthemup,orevenonasinglemachinestartingmultipleArangoDBprocesses.Inthatcase,synchronousreplicationwillworkwithinthedeployedArangoDBcluster,andautomaticfail-overinthesensethatthedutiesofafailedserverwillautomaticallybeassignedtoanother,survivingone.However,sincetheArangoDBclustercannotwithinitselflaunchadditionalinstances,replacementoffailednodesisnotautomaticandscalingupanddownhastobemanagedmanually.Thisiswhywedonotrecommendthissetupforproductiondeployment.

Architecture

36

Differentdatamodelsandscalability

InthissectionwediscussscalabilityinthecontextofthedifferentdatamodelssupportedbyArangoDB.

Key/valuepairs

Thekey/valuestoredatamodelistheeasiesttoscale.InArangoDB,thisisimplementedinthesensethatadocumentcollectionalwayshasaprimarykey_keyattributeandintheabsenceoffurthersecondaryindexesthedocumentcollectionbehaveslikeasimplekey/valuestore.

Theonlyoperationsthatarepossibleinthiscontextaresinglekeylookupsandkey/valuepairinsertionsandupdates.If_keyistheonlyshardingattributethentheshardingisdonewithrespecttotheprimarykeyandalltheseoperationsscalelinearly.Iftheshardingisdoneusingdifferentshardkeys,thenalookupofasinglekeyinvolvesaskingallshardsandthusdoesnotscalelinearly.

Documentstore

Forthedocumentstorecaseeveninthepresenceofsecondaryindexesessentiallythesameargumentsapply,sinceanindexforashardedcollectionissimplythesameasalocalindexforeachshard.Therefore,singledocumentoperationsstillscalelinearlywiththesizeofthecluster,unlessaspecialshardingconfigurationmakeslookupsorwriteoperationsmoreexpensive.

ForadeeperanalysisofthistopicseethisblogpostinwhichgoodlinearscalabilityofArangoDBforsingledocumentoperationsisdemonstrated.

Complexqueriesandjoins

TheAQLquerylanguageallowscomplexqueries,usingmultiplecollections,secondaryindexesaswellasjoins.Inparticularwiththelatter,scalingcanbeachallenge,sinceifthedatatobejoinedresidesondifferentmachines,alotofcommunicationhastohappen.TheAQLqueryexecutionengineorganizesadatapipelineacrosstheclustertoputtogethertheresultsinthemostefficientway.Thequeryoptimizerisawareoftheclusterstructureandknowswhatdataiswhereandhowitisindexed.Therefore,itcanarriveataninformeddecisionaboutwhatpartsofthequeryoughttorunwhereinthecluster.

Nevertheless,forcertaincomplicatedjoins,therearelimitsastowhatcanbeachieved.

Graphdatabase

Graphdatabasesareparticularlygoodatqueriesongraphsthatinvolvepathsinthegraphofanaprioriunknownlength.Forexample,findingtheshortestpathbetweentwoverticesinagraph,orfindingallpathsthatmatchacertainpatternstartingatagivenvertexaresuchexamples.

However,iftheverticesandedgesalongtheoccurringpathsaredistributedacrossthecluster,thenalotofcommunicationisnecessarybetweennodes,andperformancesuffers.Toachievegoodperformanceatscale,itisthereforenecessarytogetthedistributionofthegraphdataacrosstheshardsintheclusterright.Mostofthetime,theapplicationdevelopersandusersofArangoDBknowbest,howtheirgraphsarastructured.Therefore,ArangoDBallowsuserstospecify,accordingtowhichattributesthegraphdataissharded.Ausefulfirststepisusuallytomakesurethattheedgesoriginatingatavertexresideonthesameclusternodeasthevertex.

Datamodels

37

Limitations

ArangoDBhasnobuilt-inlimitationstohorizontalscalability.ThecentralresilientAgencywilleasilysustainhundredsofDBserversandcoordinators,andtheusualdatabaseoperationsworkcompletelydecentrallyanddonotrequireassistanceoftheAgency.

Likewise,thesupervisionprocessintheAgencycaneasilydealwithlotsofservers,sinceallitsactivitiesarenotperformancecritical.

Obviously,anArangoDBclusterislimitedbytheavailableresourcesofCPU,memory,diskandnetworkbandwidthandlatency.

Limitations

38

DatacentertodatacenterreplicationThischapterintroducesArangoDB'sdatacentertodatacenterreplication(DC2DC).

Forfurtherinformationaboutdatacentertodatacenterreplication,pleaserefertothefollowingsections:

DeploymentAdministrationTroubleshootingMonitoringSecurity

Datacentertodatacenterreplication

39

IntroductionAtsomepointinthegrowsofadatabase,therecomesaneedforreplicatingitacrossmultipledatacenters.

Reasonsforthatcanbe:

FallbackincaseofadisasterinonedatacenterRegionalavailabilitySeparationofconcerns

Andmanymore.

Startingfromversion3.3,ArangoDBsupportsdatacentertodatacenterreplication,viatheArangoSynctool.

ArangoDB'sdatacentertodatacenterreplicationisasolutionthatenablesyoutoasynchronouslyreplicatetheentirestructureandcontentinanArangoDBClusterinoneplacetoaClusterinanotherplace.Typicallyitisusedfromonedatacentertoanother.Itisnotasolutionforreplicatingsingleserverinstances.

ThereplicationdonebyArangoSyncinasynchronous.Thatmeansthatwhenaclientiswritingdataintothesourcedatacenter,itwillconsidertherequestfinishedbeforethedatahasbeenreplicatedtotheotherdatacenter.Thetimeneededtocompletelyreplicatechangestotheotherdatacenteristypicallyintheorderofseconds,butthiscanvarysignificantlydependingonload,network&computercapacity.

ArangoSyncperformsreplicationinasingledirectiononly.ThatmeansthatyoucanreplicatedatafromclusterAtoclusterBorfromclusterBtoclusterA,butneveratthesametime.Datamodifiedinthedestinationclusterwillbelost!

Replicationisacompletelyautonomousprocess.Onceitisconfigureditisdesignedtorun24/7withoutfrequentmanualintervention.Thisdoesnotmeanthatitrequiresnomaintenanceorattentionatall.Aswithanydistributedsystemsomeattentionisneededtomonitoritsoperationandkeepitsecure(e.g.certificate&passwordrotation).

Onceconfigured,ArangoSyncwillreplicatebothstructureanddataofanentirecluster.Thismeansthatthereisnoneedtomakeadditionalconfigurationchangeswhenadding/removingdatabasesorcollections.Alsometadatasuchasusers,foxxapplication&jobsareautomaticallyreplicated.

Introduction

40

Whentouseit...andwhennotThedatacentertodatacenterreplicationisagoodsolutioninallcaseswhereyouwanttoreplicatedatafromoneclustertoanotherwithouttherequirementthatthedataisavailableimmediatelyintheothercluster.

Thedatacentertodatacenterreplicationisnotagoodsolutionwhenoneofthefollowingapplies:

YouwanttoreplicatedatafromclusterAtoclusterBandfromclusterBtoclusterAatthesametime.Youneedsynchronousreplicationbetween2clusters.ThereisnonetworkconnectionbetweenclusterAandB.Youwantcompletecontroloverwhichdatabase,collection&documentsarereplicateandwhichnot.

Applicability

41

RequirementsTousedatacentertodatacenterreplicationyouneedthefollowing:

Twodatacenters,eachrunninganArangoDBEnterprisecluster,version3.3orhigher.Anetworkconnectionbetweenbothdatacenterswithaccessibleendpointsforseveralcomponents(seeindividualcomponentsfordetails).TLScertificatesforArangoSyncmasterinstances(canbeself-signed).TLScertificatesforKafkabrokers(canbeself-signed).Optional(butrecommended)TLScertificatesforArangoDBclusters(canbeself-signed).ClientcertificatesCAforArangoSyncmasters(typicallyself-signed).ClientcertificatesforArangoSyncmasters(typicallyself-signed).Atleast2instancesoftheArangoSyncmasterineachdatacenter.OneinstancesoftheArangoSyncworkeroneverymachineineachdatacenter.

Note:Inseveralplacesyouwillneeda(x509)certificate.TheCertificatessectionprovidesmoreguidanceforcreatingandrenewingthesecertificates.

Besidestheabovelist,youprobablywanttousethefollowing:

Anorchestratortokeepallcomponentsrunning,e.g.systemd.Alogfilecollectorforcentralizedcollection&accesstothelogsofallcomponents.Ametricscollector&viewingsolutionsuchasPrometheus+Grafana.

Requirements

42

Datamodels&modelingThischapterintroducesArangoDB'scoreconceptsandcovers

itsdatamodel(ordatamodelsrespectively),theterminologyusedthroughoutthedatabasesystemandinthisdocumentation,aswellasaspectstoconsiderwhenmodelingyourdatatostrikeabalancebetweennaturaldatastructuresandgreatperformance

Youwillalsofindusageexamplesonhowtointeractwiththedatabasesystemusingarangosh,e.g.howtocreateanddropdatabases/collections,orhowtosave,update,replaceandremovedocuments.Youcandoallthisusingthewebinterfaceaswellandmaythereforeskipthesesectionsasbeginner.

Datamodels&modeling

43

Concepts

DatabaseInteraction

ArangoDBisadatabasethatservesdocumentstoclients.ThesedocumentsaretransportedusingJSONviaaTCPconnection,usingtheHTTPprotocol.ARESTAPIisprovidedtointeractwiththedatabasesystem.

ThewebinterfacethatcomeswithArangoDB,calledAardvark,providesgraphicaluserinterfacethatiseasytouse.Aninteractiveshell,calledArangosh,isalsoshipped.Inaddition,therearesocalleddriversthatmakeiteasytousethedatabasesysteminvariousenvironmentsandprogramminglanguages.AllthesetoolsusetheHTTPinterfaceoftheserverandremovethenecessitytorollownlow-levelcodeforbasiccommunicationinmostcases.

DatamodelThedocumentsyoucanstoreinArangoDBcloselyfollowtheJSONformat,althoughtheyarestoredinabinaryformatcalledVelocyPack.Adocumentcontainszeroormoreattributes,eachoftheseattributeshavingavalue.Avaluecaneitherbeanatomictype,i.e.number,string,booleanornull,oracompoundtype,i.e.anarrayorembeddeddocument/object.Arraysandsub-objectscancontainallofthesetypes,whichmeansthatarbitrarilynesteddatastructurescanberepresentedinasingledocument.

Documentsaregroupedintocollections.Acollectioncontainszeroormoredocuments.Ifyouarefamiliarwithrelationaldatabasemanagementsystems(RDBMS)thenitissafetocomparecollectionstotablesanddocumentstorows.ThedifferenceisthatinatraditionalRDBMS,youhavetodefinecolumnsbeforeyoucanstorerecordsinatable.Suchdefinitionsarealsoknownasschemas.ArangoDBisschema-less,whichmeansthatthereisnoneedtodefinewhatattributesadocumentcanhave.Everysingledocumentcanhaveacompletelydifferentstructureandstillbestoredtogetherwithotherdocumentsinasinglecollection.Inpractice,therewillbecommondenominatorsamongthedocumentsinacollection,butthedatabasesystemitselfdoesn'tforceyoutolimityourselftoacertaindatastructure.

Therearetwotypesofcollections:documentcollection(alsoreferedtoasvertexcollectionsinthecontextofgraphs)aswellasedgecollections.Edgecollectionsstoredocumentsaswell,buttheyincludetwospecialattributes,_fromand_to,whichareusedtocreaterelationsbetweendocuments.Usually,twodocuments(vertices)storedindocumentcollectionsarelinkedbyadocument(edge)storedinanedgecollection.ThisisArangoDB'sgraphdatamodel.Itfollowsthemathematicalconceptofadirected,labeledgraph,exceptthatedgesdon'tjusthavelabels,butarefull-blowndocuments.

Collectionsexistinsideofdatabases.Therecanbeoneormanydatabases.Differentdatabasesareusuallyusedformultitenantsetups,asthedatainsidethem(collections,documentsetc.)isisolatedfromoneanother.Thedefaultdatabase_systemisspecial,becauseitcannotberemoved.Databaseusersaremanagedinthisdatabase,andtheircredentialsarevalidforalldatabasesofaserverinstance.

Similarlydatabasesmayalsocontainviewentities.AViewinitssimplestformcanbeseenasaread-onlyarrayorcollectionofdocuments.Theviewconceptquitecloselymatchesasimilarlynamedconceptavailableinmostrelationaldatabasemanagementsystems(RDBMS).Eachviewentityusuallymapssomeimplementationspecificdocumenttransformation,(possiblyidentity),ontodocumentsfromzeroormorecollections.

DataRetrieval

Queriesareusedtofilterdocumentsbasedoncertaincriteria,tocomputenewdata,aswellastomanipulateordeleteexistingdocuments.Queriescanbeassimpleasa"querybyexample"orascomplexas"joins"usingmanycollectionsortraversinggraphstructures.TheyarewrittenintheArangoDBQueryLanguage(AQL).

Cursorsareusedtoiterateovertheresultofqueries,sothatyougeteasilyprocessablebatchesinsteadofonebighunk.

Indexesareusedtospeedupsearches.Therearevarioustypesofindexes,suchashashindexesandgeoindexes.

Concepts

44

HandlingDatabasesThisisanintroductiontomanagingdatabasesinArangoDBfromwithinJavaScript.

WhenyouhaveanestablishedconnectiontoArangoDB,thecurrentdatabasecanbechangedexplicitlyusingthedb._useDatabase()method.Thiswillswitchtothespecifieddatabase(provideditexistsandtheusercanconnecttoit).Fromthispointon,anyfollowingactioninthesameshellorconnectionwillusethespecifieddatabase,unlessotherwisespecified.

Note:Ifthedatabaseischanged,clientdriversneedtostorethecurrentdatabasenameontheirside,too.ThisisbecauseconnectionsinArangoDBdonotcontainanystateinformation.AllstateinformationiscontainedintheHTTPrequest/responsedata.

Toconnecttoaspecificdatabaseafterarangoshhasstartedusethecommanddescribedabove.Itisalsopossibletospecifyadatabasenamewheninvokingarangosh.Forthispurpose,usethecommand-lineparameter--server.database,e.g.

>arangosh--server.databasetest

Pleasenotethatcommands,actions,scriptsorAQLqueriesshouldneveraccessmultipledatabases,eveniftheyexist.TheonlyintendedandsupportedwayinArangoDBistouseonedatabaseatatimeforacommand,anaction,ascriptoraquery.Operationsstartedinonedatabasemustnotswitchthedatabaselaterandcontinueoperatinginanother.

Databases

45

WorkingwithDatabases

DatabaseMethods

ThefollowingmethodsareavailabletomanagedatabasesviaJavaScript.Pleasenotethatseveralofthesemethodscanbeusedfromthe_systemdatabaseonly.

Name

returnthedatabasenamedb._name()

Returnsthenameofthecurrentdatabaseasastring.

Examples

arangosh>require("@arangodb").db._name();

_system

ID

returnthedatabaseiddb._id()

Returnstheidofthecurrentdatabaseasastring.

Examples

arangosh>require("@arangodb").db._id();

1

Path

returnthepathtodatabasefilesdb._path()

Returnsthefilesystempathofthecurrentdatabaseasastring.

Examples

arangosh>require("@arangodb").db._path();

/tmp/arangosh_3GEDY1/tmp-21277-3063631881/data/databases/database-1

isSystem

returnthedatabasetypedb._isSystem()

Returnswhetherthecurrentlyuseddatabaseisthe_systemdatabase.Thesystemdatabasehassomespecialprivilegesandproperties,forexample,databasemanagementoperationssuchascreateordropcanonlybeexecutedfromwithinthisdatabase.Additionally,the_systemdatabaseitselfcannotbedropped.

UseDatabase

changethecurrentdatabasedb._useDatabase(name)

Changesthecurrentdatabasetothedatabasespecifiedbyname.Notethatthedatabasespecifiedbynamemustalreadyexist.

Changingthedatabasemightbedisallowedinsomecontexts,forexampleserver-sideactions(includingFoxx).

WorkingwithDatabases

46

Whenperformingthiscommandfromarangosh,thecurrentcredentials(usernameandpassword)willbere-used.Thesecredentialsmightnotbevalidtoconnecttothedatabasespecifiedbyname.Additionally,thedatabaseonlybeaccessedfromcertainendpointsonly.Inthiscase,switchingthedatabasemightnotwork,andtheconnection/sessionshouldbeclosedandrestartedwithdifferentusernameandpasswordcredentialsand/orendpointdata.

ListDatabases

returnthelistofallexistingdatabasesdb._databases()

Returnsthelistofalldatabases.Thismethodcanonlybeusedfromwithinthe_systemdatabase.

CreateDatabase

createanewdatabasedb._createDatabase(name,options,users)

Createsanewdatabasewiththenamespecifiedbyname.Therearerestrictionsfordatabasenames(seeDatabaseNames).

Notethatevenifthedatabaseiscreatedsuccessfully,therewillbenochangeintothecurrentdatabasetothenewdatabase.Changingthecurrentdatabasemustexplicitlyberequestedbyusingthedb._useDatabasemethod.

Theoptionsattributecurrentlyhasnomeaningandisreservedforfutureuse.

Theoptionalusersattributecanbeusedtocreateinitialusersforthenewdatabase.Ifspecified,itmustbealistofuserobjects.Eachuserobjectcancontainthefollowingattributes:

username:theusernameasastring.Thisattributeismandatory.passwd:theuserpasswordasastring.Ifnotspecified,thenitdefaultstoanemptystring.active:abooleanflagindicatingwhethertheuseraccountshouldbeactiveornot.Thedefaultvalueistrue.extra:anoptionalJSONobjectwithextrauserinformation.ThedatacontainedinextrawillbestoredfortheuserbutnotbeinterpretedfurtherbyArangoDB.

Ifnoinitialusersarespecified,adefaultuserrootwillbecreatedwithanemptystringpassword.ThisensuresthatthenewdatabasewillbeaccessibleviaHTTPafteritiscreated.

Youcancreateusersinadatabaseifnoinitialuserisspecified.Switchintothenewdatabase(usernameandpasswordmustbeidenticaltothecurrentsession)andaddormodifyuserswiththefollowingcommands.

require("@arangodb/users").save(username,password,true);

require("@arangodb/users").update(username,password,true);

require("@arangodb/users").remove(username);

Alternatively,youcanspecifyuserdatadirectly.Forexample:

db._createDatabase("newDB",{},[{username:"newUser",passwd:"123456",active:true}])

Thosemethodscanonlybeusedfromwithinthe_systemdatabase.

DropDatabase

dropanexistingdatabasedb._dropDatabase(name)

Dropsthedatabasespecifiedbyname.Thedatabasespecifiedbynamemustexist.

Note:Droppingdatabasesisonlypossiblefromwithinthe_systemdatabase.The_systemdatabaseitselfcannotbedropped.

Databasesaredroppedasynchronously,andwillbephysicallyremovedifallclientshavedisconnectedandreferenceshavebeengarbage-collected.

Enginestatistics

retrievestatisticsrelatedtothestorageengine-rocksdbdb._engineStats()

Returnssomestatisticsrelatedtostorageengineactivity,includingfiguresaboutdatasize,cacheusage,etc.

WorkingwithDatabases

47

Note:CurrentlythisonlyproducesusefuloutputfortheRocksDBengine.

WorkingwithDatabases

48

NotesaboutDatabasesPleasekeepinmindthateachdatabasecontainsitsownsystemcollections,whichneedtobesetupwhenadatabaseiscreated.Thiswillmakethecreationofadatabasetakeawhile.

Replicationisconfiguredonaper-databaselevel,meaningthatanyreplicationloggingorapplyingforanewdatabasemustbeconfiguredexplicitlyafteranewdatabasehasbeencreated.

Foxxapplicationsarealsoavailableonlyinthecontextofthedatabasetheyhavebeeninstalledin.AnewdatabasewillonlyprovideaccesstothesystemapplicationsshippedwithArangoDB(thatisthewebinterfaceatthemoment)andnootherFoxxapplicationsuntiltheyareexplicitlyinstalledfortheparticulardatabase.

NotesaboutDatabases

49

JavaScriptInterfacetoCollectionsThisisanintroductiontoArangoDB'sinterfaceforcollectionsandhowtohandlecollectionsfromtheJavaScriptshellarangosh.ForotherlanguagesseethecorrespondinglanguageAPI.

Themostimportantcallisthecalltocreateanewcollection.

AddressofaCollection

AllcollectionsinArangoDBhaveauniqueidentifierandauniquename.ArangoDBinternallyusesthecollection'suniqueidentifiertolookupcollections.Thisidentifier,however,ismanagedbyArangoDBandtheuserhasnocontroloverit.Inordertoallowuserstousetheirownnames,eachcollectionalsohasauniquenamewhichisspecifiedbytheuser.Toaccessacollectionfromtheuserperspective,thecollectionnameshouldbeused,i.e.:

Collection

db._collection(collection-name)

Acollectioniscreatedbya"db._create"call.

Forexample:Assumethatthecollectionidentifieris7254820andthenameisdemo,thenthecollectioncanbeaccessedas:

db._collection("demo")

Ifnocollectionwithsuchanameexists,thennullisreturned.

Thereisashort-cutthatcanbeusedfornon-systemcollections:

Collectionname

db.collection-name

Thiscallwilleitherreturnthecollectionnameddb.collection-nameorcreateanewonewiththatnameandasetofdefaultproperties.

Note:Creatingacollectionontheflyusingdb.collection-nameisnotrecommendanddoesnotworkinarangosh.Tocreateanewcollection,pleaseuse

Create

db._create(collection-name)

Thiscallwillcreateanewcollectioncalledcollection-name.ThismethodisadatabasemethodandisdocumentedindetailatDatabaseMethods

Synchronousreplication

StartinginArangoDB3.0,thedistributedversionofferssynchronousreplication,whichmeansthatthereistheoptiontoreplicatealldataautomaticallywithintheArangoDBcluster.Thisisconfiguredforshardedcollectionsonapercollectionbasisbyspecifyinga"replicationfactor"whenthecollectioniscreated.Areplicationfactorofkmeansthataltogetherkcopiesofeachshardarekeptintheclusteronkdifferentservers,andarekeptinsync.Thatis,everywriteoperationisautomaticallyreplicatedonallcopies.

Thisisorganisedusingaleader/followermodel.Atalltimes,oneoftheserversholdingreplicasforashardis"theleader"andallothersare"followers",thisconfigurationisheldintheAgency(seeScalabilityfordetailsoftheArangoDBclusterarchitecture).Everywriteoperationissenttotheleaderbyoneofthecoordinators,andthenreplicatedtoallfollowersbeforetheoperationisreportedtohavesucceeded.Theleaderkeepsarecordofwhichfollowersarecurrentlyinsync.Incaseofnetworkproblemsorafailureofafollower,aleadercanandwilldropafollowertemporarilyafter3seconds,suchthatservicecanresume.Induecourse,thefollowerwillautomaticallyresynchronizewiththeleadertorestoreresilience.

Collections

50

Ifaleaderfails,theclusterAgencyautomaticallyinitiatesafailoverroutineafteraround15seconds,promotingoneofthefollowerstoleader.Theotherfollowers(andtheformerleader,whenitcomesback),automaticallyresynchronizewiththenewleadertorestoreresilience.Usually,thiswholefailoverprocedurecanbehandledtransparentlyforthecoordinator,suchthattheusercodedoesnotevenseeanerrormessage.

Obviously,thisfaulttolerancecomesatacostofincreasedlatency.Eachwriteoperationneedsanadditionalnetworkroundtripforthesynchronousreplicationofthefollowers,butallreplicationoperationstoallfollowershappenconcurrently.Thisis,whythedefaultreplicationfactoris1,whichmeansnoreplication.

Fordetailsonhowtoswitchonsynchronousreplicationforacollection,seethedatabasemethoddb._create(collection-name)inthesectionaboutDatabaseMethods.

Collections

51

CollectionMethods

Drop

dropsacollectioncollection.drop(options)

Dropsacollectionandallitsindexesanddata.Inordertodropasystemcollection,anoptionsobjectwithattributeisSystemsettotruemustbespecified.

Note:droppingacollectioninacluster,whichisprototypeforsharinginothercollectionsisprohibited.Inordertobeabletodropsuchacollection,alldependentcollectionsmustbedroppedfirst.

Examples

arangosh>col=db.example;

[ArangoCollection16878,"example"(typedocument,statusloaded)]

arangosh>col.drop();

arangosh>col;

[ArangoCollection16878,"example"(typedocument,statusdeleted)]

arangosh>col=db._example;

[ArangoCollection16889,"_example"(typedocument,statusloaded)]

arangosh>col.drop({isSystem:true});

arangosh>col;

[ArangoCollection16889,"_example"(typedocument,statusdeleted)]

Truncate

truncatesacollectioncollection.truncate()

Truncatesacollection,removingalldocumentsbutkeepingallitsindexes.

Examples

Truncatesacollection:

arangosh>col=db.example;

arangosh>col.save({"Hello":"World"});

arangosh>col.count();

arangosh>col.truncate();

arangosh>col.count();

showexecutionresults

Properties

getsorsetsthepropertiesofacollectioncollection.properties()Returnsanobjectcontainingallcollectionproperties.

waitForSync:Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.journalSize:Thesizeofthejournalinbytes.ThisoptionismeaningfulfortheMMFilesstorageengineonly.isVolatile:IftruethenthecollectiondatawillbekeptinmemoryonlyandArangoDBwillnotwriteorsyncthedatatodisk.ThisoptionismeaningfulfortheMMFilesstorageengineonly.keyOptions(optional)additionaloptionsforkeygeneration.ThisisaJSONarraycontainingthefollowingattributes(note:someoftheattributesareoptional):

type:thetypeofthekeygeneratorusedforthecollection.allowUserKeys:ifsettotrue,thenitisallowedtosupplyownkeyvaluesinthe_keyattributeofadocument.Ifsettofalse,

CollectionMethods

52

thenthekeygeneratorwillsolelyberesponsibleforgeneratingkeysandsupplyingownkeyvaluesinthe_keyattributeofdocumentsisconsideredanerror.increment:incrementvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.offset:initialoffsetvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.

indexBuckets:numberofbucketsintowhichindexesusingahashtablearesplit.Thedefaultis16andthisnumberhastobeapowerof2andlessthanorequalto1024.ThisoptionismeaningfulfortheMMFilesstorageengineonly.Forverylargecollectionsoneshouldincreasethistoavoidlongpauseswhenthehashtablehastobeinitiallybuiltorresized,sincebucketsareresizedindividuallyandcanbeinitiallybuiltinparallel.Forexample,64mightbeasensiblevalueforacollectionwith100000000documents.Currently,onlytheedgeindexrespectsthisvalue,butotherindextypesmightfollowinfutureArangoDBversions.Changes(seebelow)areappliedwhenthecollectionisloadedthenexttime.Inaclustersetup,theresultwillalsocontainthefollowingattributes:numberOfShards:thenumberofshardsofthecollection.shardKeys:containsthenamesofdocumentattributesthatareusedtodeterminethetargetshardfordocuments.replicationFactor:determineshowmanycopiesofeachshardarekeptondifferentDBServers.collection.properties(properties)Changesthecollectionproperties.propertiesmustbeanobjectwithoneormoreofthefollowingattribute(s):waitForSync:Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.journalSize:Thesizeofthejournalinbytes.ThisoptionismeaningfulfortheMMFilesstorageengineonly.indexBuckets:Seeabove,changesareonlyappliedwhenthecollectionisloadedthenexttime.ThisoptionismeaningfulfortheMMFilesstorageengineonly.replicationFactor:ChangethenumberofshardcopieskeptondifferentDBServers,validvaluesareintegernumbersintherangeof1-10(Clusteronly)Note:itisnotpossibletochangethejournalsizeafterthejournalordatafilehasbeencreated.Changingthisparameterwillonlyeffectnewlycreatedjournals.Alsonotethatyoucannotlowerthejournalsizetolessthensizeofthelargestdocumentalreadystoredinthecollection.Note:someothercollectionproperties,suchastype,isVolatile,orkeyOptionscannotbechangedoncethecollectioniscreated.

Examples

Readallproperties

arangosh>db.example.properties();

showexecutionresultsChangeaproperty

arangosh>db.example.properties({waitForSync:true});

showexecutionresults

Figures

returnsthefiguresofacollectioncollection.figures()

Returnsanobjectcontainingstatisticsaboutthecollection.Note:Retrievingthefigureswillalwaysloadthecollectionintomemory.

alive.count:Thenumberofcurrentlyactivedocumentsinalldatafilesandjournalsofthecollection.Documentsthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.alive.size:Thetotalsizeinbytesusedbyallactivedocumentsofthecollection.Documentsthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.dead.count:Thenumberofdeaddocuments.Thisincludesdocumentversionsthathavebeendeletedorreplacedbyanewerversion.Documentsdeletedorreplacedthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.dead.size:Thetotalsizeinbytesusedbyalldeaddocuments.dead.deletion:Thetotalnumberofdeletionmarkers.Deletionmarkersonlycontainedinthewrite-aheadlogarenotreportinginthisfigure.datafiles.count:Thenumberofdatafiles.datafiles.fileSize:Thetotalfilesizeofdatafiles(inbytes).journals.count:Thenumberofjournalfiles.journals.fileSize:Thetotalfilesizeofthejournalfiles(inbytes).

CollectionMethods

53

compactors.count:Thenumberofcompactorfiles.compactors.fileSize:Thetotalfilesizeofthecompactorfiles(inbytes).shapefiles.count:Thenumberofshapefiles.Thisvalueisdeprecatedandkeptforcompatibilityreasonsonly.Thevaluewillalwaysbe0sinceArangoDB2.0andhigher.shapefiles.fileSize:Thetotalfilesizeoftheshapefiles.Thisvalueisdeprecatedandkeptforcompatibilityreasonsonly.Thevaluewillalwaysbe0inArangoDB2.0andhigher.shapes.count:Thetotalnumberofshapesusedinthecollection.Thisincludesshapesthatarenotinuseanymore.Shapesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.shapes.size:Thetotalsizeofallshapes(inbytes).Thisincludesshapesthatarenotinuseanymore.Shapesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.attributes.count:Thetotalnumberofattributesusedinthecollection.Note:thevalueincludesdataofattributesthatarenotinuseanymore.Attributesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.attributes.size:Thetotalsizeoftheattributedata(inbytes).Note:thevalueincludesdataofattributesthatarenotinuseanymore.Attributesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.indexes.count:Thetotalnumberofindexesdefinedforthecollection,includingthepre-definedindexes(e.g.primaryindex).indexes.size:Thetotalmemoryallocatedforindexesinbytes.lastTick:Thetickofthelastmarkerthatwasstoredinajournalofthecollection.Thismightbe0ifthecollectiondoesnotyethaveajournal.uncollectedLogfileEntries:Thenumberofmarkersinthewrite-aheadlogforthiscollectionthathavenotbeentransferredtojournalsordatafiles.documentReferences:ThenumberofreferencestodocumentsindatafilesthatJavaScriptcodecurrentlyholds.Thisinformationcanbeusedfordebuggingcompactionandunloadissues.waitingFor:Anoptionalstringvaluethatcontainsinformationaboutwhichobjecttypeisattheheadofthecollection'scleanupqueue.Thisinformationcanbeusedfordebuggingcompactionandunloadissues.compactionStatus.time:Thepointintimethecompactionforthecollectionwaslastexecuted.Thisinformationcanbeusedfordebuggingcompactionissues.compactionStatus.message:Theactionthatwasperformedwhenthecompactionwaslastrunforthecollection.Thisinformationcanbeusedfordebuggingcompactionissues.

Note:collectiondatathatarestoredinthewrite-aheadlogonlyarenotreportedintheresults.Whenthewrite-aheadlogiscollected,documentsmightbeaddedtojournalsanddatafilesofthecollection,whichmaymodifythefiguresofthecollection.AlsonotethatwaitingForandcompactionStatusmaybeemptywhencalledonacoordinatorinacluster.

Additionally,thefilesizesofcollectionandindexparameterJSONfilesarenotreported.Thesefilesshouldnormallyhaveasizeofafewbyteseach.PleasealsonotethatthefileSizevaluesarereportedinbytesandreflectthelogicalfilesizes.Somefilesystemsmayuseoptimisations(e.g.sparsefiles)sothattheactualphysicalfilesizeissomewhatdifferent.Directoriesandsub-directoriesmayalsorequirespaceinthefilesystem,butthisspaceisnotreportedinthefileSizeresults.

Thatmeansthatthefiguresreporteddonotreflecttheactualdiskusageofthecollectionwith100%accuracy.TheactualdiskusageofacollectionisnormallyslightlyhigherthanthesumofthereportedfileSizevalues.StillthesumofthefileSizevaluescanstillbeusedasalowerboundapproximationofthediskusage.

Examples

arangosh>db.demo.figures()

showexecutionresults

Load

loadsacollectioncollection.load()

Loadsacollectionintomemory.

Note:clustercollectionsareloadedatalltimes.

Examples

arangosh>col=db.example;

CollectionMethods

54

[ArangoCollection16998,"example"(typedocument,statusloaded)]

arangosh>col.load();

arangosh>col;

[ArangoCollection16998,"example"(typedocument,statusloaded)]

Revision

returnstherevisionidofacollectioncollection.revision()

Returnstherevisionidofthecollection

Therevisionidisupdatedwhenthedocumentdataismodified,eitherbyinserting,deleting,updatingorreplacingdocumentsinit.

Therevisionidofacollectioncanbeusedbyclientstocheckwhetherdatainacollectionhaschangedorifitisstillunmodifiedsinceapreviousfetchoftherevisionid.

Therevisionidreturnedisastringvalue.Clientsshouldtreatthisvalueasanopaquestring,andonlyuseitforequality/non-equalitycomparisons.

Path

returnsthephysicalpathofthecollectioncollection.path()

Thepathoperationreturnsastringwiththephysicalstoragepathforthecollectiondata.

Note:thismethodwillreturnnothingmeaningfulinacluster.Inasingle-serverArangoDB,thismethodwillonlyreturnmeaningfuldatafortheMMFilesengine.

Checksum

calculatesachecksumforthedatainacollectioncollection.checksum(withRevisions,withData)

Thechecksumoperationcalculatesanaggregatehashvalueforalldocumentkeyscontainedincollectioncollection.

IftheoptionalargumentwithRevisionsissettotrue,thentherevisionidsofthedocumentsarealsoincludedinthehashcalculation.

IftheoptionalargumentwithDataissettotrue,thenalluser-defineddocumentattributesarealsochecksummed.Includingthedocumentdatainchecksummingwillmakethecalculationslower,butismoreaccurate.

ThechecksumcalculationalgorithmchangedinArangoDB3.0,sochecksumsfrom3.0andearlierversionsforthesamedatawilldiffer.

Note:thismethodisnotavailableinacluster.

Unload

unloadsacollectioncollection.unload()

Startsunloadingacollectionfrommemory.Notethatunloadingisdeferreduntilallqueryhavefinished.

Note:clustercollectionscannotbeunloaded.

Examples

arangosh>col=db.example;

[ArangoCollection7475,"example"(typedocument,statusloaded)]

arangosh>col.unload();

arangosh>col;

[ArangoCollection7475,"example"(typedocument,statusunloaded)]

Rename

renamesacollectioncollection.rename(new-name)

CollectionMethods

55

Renamesacollectionusingthenew-name.Thenew-namemustnotalreadybeusedforadifferentcollection.new-namemustalsobeavalidcollectionname.Formoreinformationonvalidcollectionnamespleaserefertothenamingconventions.

Ifrenamingfailsforanyreason,anerroristhrown.Ifrenamingthecollectionsucceeds,thenthecollectionisalsorenamedinallgraphdefinitionsinsidethe_graphscollectioninthecurrentdatabase.

Note:thismethodisnotavailableinacluster.

Examples

arangosh>c=db.example;

[ArangoCollection17116,"example"(typedocument,statusloaded)]

arangosh>c.rename("better-example");

arangosh>c;

[ArangoCollection17116,"better-example"(typedocument,statusloaded)]

Rotate

rotatesthecurrentjournalofacollectioncollection.rotate()

Rotatesthecurrentjournalofacollection.Thisoperationmakesthecurrentjournalofthecollectionaread-onlydatafilesoitmaybecomeacandidateforgarbagecollection.Ifthereiscurrentlynojournalavailableforthecollection,theoperationwillfailwithanerror.

Note:thismethodisspecificfortheMMFilesstorageengine,andthereitisnotavailableinacluster.

Note:pleasenotethatyouneedappropriateuserpermissionstoexecutethis.

TodotherenamecollectionsinfirstplaceyouneedtohaveadministrativerightsonthedatabaseTohaveaccesstotheresultingrenamedcollectionyoueitherneedtohaveaccesstoallcollectionsofthatdatabase(*)oramainsystemadministratorhastogiveyouaccesstothenewlynamedone.

CollectionMethods

56

DatabaseMethods

Collection

returnsasinglecollectionornulldb._collection(collection-name)

Returnsthecollectionwiththegivennameornullifnosuchcollectionexists.

db._collection(collection-identifier)

Returnsthecollectionwiththegivenidentifierornullifnosuchcollectionexists.Accessingcollectionsbyidentifierisdiscouragedforendusers.Endusersshouldaccesscollectionsusingthecollectionname.

Examples

Getacollectionbyname:

arangosh>db._collection("demo");

[ArangoCollection102,"demo"(typedocument,statusloaded)]

Getacollectionbyid:

arangosh>db._collection(123456);

[ArangoCollection123456,"demo"(typedocument,statusloaded)]

Unknowncollection:

arangosh>db._collection("unknown");

null

Create

createsanewdocumentoredgecollectiondb._create(collection-name)

Createsanewdocumentcollectionnamedcollection-name.Ifthecollectionnamealreadyexistsorifthenameformatisinvalid,anerroristhrown.Formoreinformationonvalidcollectionnamespleaserefertothenamingconventions.

db._create(collection-name,properties)

propertiesmustbeanobjectwiththefollowingattributes:

waitForSync(optional,defaultfalse):Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.

journalSize(optional,defaultisaconfigurationparameter:Themaximalsizeofajournalordatafile.Notethatthisalsolimitsthemaximalsizeofasingleobject.Mustbeatleast1MB.

isSystem(optional,defaultisfalse):Iftrue,createasystemcollection.Inthiscasecollection-nameshouldstartwithanunderscore.Endusersshouldnormallycreatenon-systemcollectionsonly.APIimplementorsmayberequiredtocreatesystemcollectionsinveryspecialoccasions,butnormallyaregularcollectionwilldo.

isVolatile(optional,defaultisfalse):Iftruethenthecollectiondataiskeptin-memoryonlyandnotmadepersistent.Unloadingthecollectionwillcausethecollectiondatatobediscarded.Stoppingorre-startingtheserverwillalsocausefulllossofdatainthecollection.Thecollectionitselfwillremainhowever(onlythedataisvolatile).SettingthisoptionwillmaketheresultingcollectionbeslightlyfasterthanregularcollectionsbecauseArangoDBdoesnotenforceanysynchronizationtodiskanddoesnotcalculateanyCRCchecksumsfordatafiles(astherearenodatafiles).ThisoptionismeaningfulfortheMMFilesstorageengineonly.

keyOptions(optional):additionaloptionsforkeygeneration.Ifspecified,thenkeyOptionsshouldbeaJSONobjectcontainingthefollowingattributes(note:someofthemareoptional):

type:specifiesthetypeofthekeygenerator.Thecurrentlyavailablegeneratorsaretraditionalandautoincrement.(note:autoincrementiscurrentlyonlysupportedfornon-shardedcollections)

DatabaseMethods

57

allowUserKeys:ifsettotrue,thenitisallowedtosupplyownkeyvaluesinthe_keyattributeofadocument.Ifsettofalse,thenthekeygeneratorwillsolelyberesponsibleforgeneratingkeysandsupplyingownkeyvaluesinthe_keyattributeofdocumentsisconsideredanerror.increment:incrementvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.offset:initialoffsetvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.

numberOfShards(optional,defaultis1):inacluster,thisvaluedeterminesthenumberofshardstocreateforthecollection.Inasingleserversetup,thisoptionismeaningless.

shardKeys(optional,defaultis["_key"]):inacluster,thisattributedetermineswhichdocumentattributesareusedtodeterminethetargetshardfordocuments.Documentsaresenttoshardsbasedonthevaluestheyhaveintheirshardkeyattributes.Thevaluesofallshardkeyattributesinadocumentarehashed,andthehashvalueisusedtodeterminethetargetshard.Notethatvaluesofshardkeyattributescannotbechangedonceset.Thisoptionismeaninglessinasingleserversetup.

Whenchoosingtheshardkeys,onemustbeawareofthefollowingrulesandlimitations:InashardedcollectionwithmorethanonesharditisnotpossibletosetupauniqueconstraintonanattributethatisnottheoneandonlyshardkeygiveninshardKeys.Thisisbecauseenforcingauniqueconstraintwouldotherwisemakeaglobalindexnecessaryorneedextensivecommunicationforeverysinglewriteoperation.Furthermore,if_keyisnottheoneandonlyshardkey,thenitisnotpossibletosetthe_keyattributewheninsertingadocument,providedthecollectionhasmorethanoneshard.Again,thisisbecausethedatabasehastoenforcetheuniqueconstraintonthe_keyattributeandthiscanonlybedoneefficientlyifthisistheonlyshardkeybydelegatingtotheindividualshards.

replicationFactor(optional,defaultis1):inacluster,thisattributedetermineshowmanycopiesofeachshardarekeptondifferentDBServers.Thevalue1meansthatonlyonecopy(nosynchronousreplication)iskept.Avalueofkmeansthatk-1replicasarekept.AnytwocopiesresideondifferentDBServers.Replicationbetweenthemissynchronous,thatis,everywriteoperationtothe"leader"copywillbereplicatedtoall"follower"replicas,beforethewriteoperationisreportedsuccessful.

Ifaserverfails,thisisdetectedautomaticallyandoneoftheserversholdingcopiestakeover,usuallywithoutanerrorbeingreported.

distributeShardsLikedistributetheshardsofthiscollectioncloningthesharddistributionofanother.

WhenusingtheEnterpriseversionofArangoDBthereplicationFactormaybesetto"satellite"makingthecollectionlocallyjoinableoneverydatabaseserver.ThisreducesthenumberofnetworkhopsdramaticallywhenusingjoinsinAQLatthecostsofreducedwriteperformanceonthesecollections.

db._create(collection-name,properties,type)

Specifiestheoptionaltypeofthecollection,itcaneitherbedocumentoredge.Ondefaultitisdocument.Insteadofgivingatypeyoucanalsousedb._createEdgeCollectionordb._createDocumentCollection.

db._create(collection-name,properties[,type],options)

Asanoptionalthird(ifthetypestringisbeingomitted)orfourthparameteryoucanspecifyanoptionaloptionsmapthatcontrolshowtheclusterwillcreatethecollection.Theseoptionsareonlyrelevantatcreationtimeandwillnotbepersisted:

waitForSyncReplication(default:true)Whenenabledtheserverwillonlyreportsuccessbacktotheclientifallreplicashavecreatedthecollection.Settofalseifyouwantfasterserverresponsesanddon'tcareaboutfullreplication.

enforceReplicationFactor(default:true)Whenenabledwhichmeanstheserverwillcheckifthereareenoughreplicasavailableatcreationtimeandbailoutotherwise.Settofalsetodisablethisextracheck.

Examples

Withdefaults:

arangosh>c=db._create("users");

arangosh>c.properties();

showexecutionresultsWithproperties:

arangosh>c=db._create("users",{waitForSync:true,

........>journalSize:1024*1204});

DatabaseMethods

58

arangosh>c.properties();

showexecutionresultsWithakeygenerator:

arangosh>db._create("users",

........>{keyOptions:{type:"autoincrement",offset:10,increment:5}});

arangosh>db.users.save({name:"user1"});

arangosh>db.users.save({name:"user2"});

arangosh>db.users.save({name:"user3"});

showexecutionresultsWithaspecialkeyoption:

arangosh>db._create("users",{keyOptions:{allowUserKeys:false}});

arangosh>db.users.save({name:"user1"});

arangosh>db.users.save({name:"user2",_key:"myuser"});

arangosh>db.users.save({name:"user3"});

showexecutionresultscreatesanewedgecollectiondb._createEdgeCollection(collection-name)

Createsanewedgecollectionnamedcollection-name.Ifthecollectionnamealreadyexistsanerroristhrown.ThedefaultvalueforwaitForSyncisfalse.

db._createEdgeCollection(collection-name,properties)

propertiesmustbeanobjectwiththefollowingattributes:

waitForSync(optional,defaultfalse):Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.journalSize(optional,defaultis"configurationparameter"):Themaximalsizeofajournalordatafile.Notethatthisalsolimitsthemaximalsizeofasingleobjectandmustbeatleast1MB.

createsanewdocumentcollectiondb._createDocumentCollection(collection-name)

Createsanewdocumentcollectionnamedcollection-name.Ifthedocumentnamealreadyexistsanderroristhrown.

AllCollections

returnsallcollectionsdb._collections()

Returnsallcollectionsofthegivendatabase.

Examples

arangosh>db._collections();

showexecutionresults

CollectionName

selectsacollectionfromthevocbasedb.collection-name

Returnsthecollectionwiththegivencollection-name.Ifnosuchcollectionexists,createacollectionnamedcollection-namewiththedefaultproperties.

Examples

arangosh>db.example;

[ArangoCollection16678,"example"(typedocument,statusloaded)]

DatabaseMethods

59

Drop

dropsacollectiondb._drop(collection)

Dropsacollectionandallitsindexesanddata.

db._drop(collection-identifier)

Dropsacollectionidentifiedbycollection-identifierwithallitsindexesanddata.Noerroristhrownifthereisnosuchcollection.

db._drop(collection-name)

Dropsacollectionnamedcollection-nameandallitsindexes.Noerroristhrownifthereisnosuchcollection.

db._drop(collection-name,options)

Inordertodropasystemcollection,onemustspecifyanoptionsobjectwithattributeisSystemsettotrue.Otherwiseitisnotpossibletodropsystemcollections.

Note:clustercollection,whichareprototypesforcollectionswithdistributeShardsLikeparameter,cannotbedropped.

Examples

Dropsacollection:

arangosh>col=db.example;

[ArangoCollection16773,"example"(typedocument,statusloaded)]

arangosh>db._drop(col);

arangosh>col;

[ArangoCollection16773,"example"(typedocument,statusloaded)]

Dropsacollectionidentifiedbyname:

arangosh>col=db.example;

[ArangoCollection16784,"example"(typedocument,statusloaded)]

arangosh>db._drop("example");

arangosh>col;

[ArangoCollection16784,"example"(typedocument,statusdeleted)]

Dropsasystemcollection

arangosh>col=db._example;

[ArangoCollection16795,"_example"(typedocument,statusloaded)]

arangosh>db._drop("_example",{isSystem:true});

arangosh>col;

[ArangoCollection16795,"_example"(typedocument,statusdeleted)]

Truncate

truncatesacollectiondb._truncate(collection)

Truncatesacollection,removingalldocumentsbutkeepingallitsindexes.

db._truncate(collection-identifier)

Truncatesacollectionidentifiedbycollection-identified.Noerroristhrownifthereisnosuchcollection.

db._truncate(collection-name)

Truncatesacollectionnamedcollection-name.Noerroristhrownifthereisnosuchcollection.

Examples

Truncatesacollection:

DatabaseMethods

60

arangosh>col=db.example;

arangosh>col.save({"Hello":"World"});

arangosh>col.count();

arangosh>db._truncate(col);

arangosh>col.count();

showexecutionresultsTruncatesacollectionidentifiedbyname:

arangosh>col=db.example;

arangosh>col.save({"Hello":"World"});

arangosh>col.count();

arangosh>db._truncate("example");

arangosh>col.count();

showexecutionresults

DatabaseMethods

61

DocumentsThisisanintroductiontoArangoDB'sinterfaceforworkingwithdocumentsfromtheJavaScriptshellarangoshorinJavaScriptcodeintheserver.ForotherlanguagesseethecorrespondinglanguageAPI.

BasicsandTerminology:sectiononthebasicapproachCollectionMethods:detailedAPIdescriptionforcollectionobjectsDatabaseMethods:detailedAPIdescriptionfordatabaseobjects

Documents

62

BasicsandTerminologyDocumentsinArangoDBareJSONobjects.Theseobjectscanbenested(toanydepth)andmaycontainlists.Eachdocumenthasauniqueprimarykeywhichidentifiesitwithinitscollection.Furthermore,eachdocumentisuniquelyidentifiedbyitsdocumenthandleacrossallcollectionsinthesamedatabase.Differentrevisionsofthesamedocument(identifiedbyitshandle)canbedistinguishedbytheirdocumentrevision.Anytransactiononlyeverseesasinglerevisionofadocument.Forexample:

{

"_id":"myusers/3456789",

"_key":"3456789",

"_rev":"14253647",

"firstName":"John",

"lastName":"Doe",

"address":{

"street":"RoadToNowhere1",

"city":"Gotham"

},

"hobbies":[

{name:"swimming",howFavorite:10},

{name:"biking",howFavorite:6},

{name:"programming",howFavorite:4}

]

}

Alldocumentscontainspecialattributes:thedocumenthandleisstoredasastringin_id,thedocument'sprimarykeyin_keyandthedocumentrevisionin_rev.Thevalueofthe_keyattributecanbespecifiedbytheuserwhencreatingadocument._idand_keyvaluesareimmutableoncethedocumenthasbeencreated.The_revvalueismaintainedbyArangoDBautomatically.

DocumentHandle

Adocumenthandleuniquelyidentifiesadocumentinthedatabase.Itisastringandconsistsofthecollection'snameandthedocumentkey(_keyattribute)separatedby/.

DocumentKey

Adocumentkeyuniquelyidentifiesadocumentinthecollectionitisstoredin.Itcanandshouldbeusedbyclientswhenspecificdocumentsarequeried.Thedocumentkeyisstoredinthe_keyattributeofeachdocument.ThekeyvaluesareautomaticallyindexedbyArangoDBinacollection'sprimaryindex.Thuslookingupadocumentbyitskeyisafastoperation.The_keyvalueofadocumentisimmutableoncethedocumenthasbeencreated.Bydefault,ArangoDBwillauto-generateadocumentkeyifno_keyattributeisspecified,andusetheuser-specified_keyotherwise.Thegenerated_keyisguaranteedtobeuniqueinthecollectionitwasgeneratedfor.Thisalsoappliestoshardedcollectionsinacluster.Itcan'tbeguaranteedthatthe_keyisuniquewithinadatabaseoracrossawholenodeorinstancehowever.

Thisbehaviorcanbechangedonaper-collectionlevelbycreatingcollectionswiththekeyOptionsattribute.

UsingkeyOptionsitispossibletodisallowuser-specifiedkeyscompletely,ortoforceaspecificregimeforauto-generatingthe_keyvalues.

DocumentRevision

AsArangoDBsupportsMVCC(MultipleVersionConcurrencyControl),documentscanexistinmorethanonerevision.ThedocumentrevisionistheMVCCtokenusedtospecifyaparticularrevisionofadocument(identifiedbyits_id).Itisastringvaluethatcontained(uptoArangoDB3.0)anintegernumberandisuniquewithinthelistofdocumentrevisionsforasingledocument.InArangoDB>=3.1the_revstringsareinfacttimestamps.TheyusethelocalclockoftheDBserverthatactuallywritesthedocumentandhavemillisecondaccuracy.Actually,a"HybridLogicalClock"isused(forthisconceptseethispaper).

Withinonesharditisguaranteedthattwodifferentdocumentrevisionshaveadifferent_revstring,eveniftheyarewritteninthesamemillisecond,andthatthesestampsareascending.

BasicsandTerminology

63

Notehoweverthatdifferentserversinyourclustermighthaveaclockskew,andthereforebetweendifferentshardsorevenbetweendifferentcollectionsthetimestampsarenotguaranteedtobecomparable.

TheHybridLogicalClockfeaturedoesonethingtoaddressthisissue:WheneveramessageissentfromsomeserverAinyourclustertoanotheroneB,itisensuredthatanytimestamptakenonBafterthemessagehasarrivedisgreaterthananytimestamptakenonAbeforethemessagewassent.Thisensuresthatifthereissome"causality"betweeneventsondifferentservers,timestampsincreasefromcausetoeffect.Adirectconsequenceofthisisthatsometimesaserverhastotaketimestampsthatseemtocomefromthefutureofitsownclock.Itwillhoweverstillproduceeverincreasingtimestamps.Iftheclockskewissmall,thenyourtimestampswillrelativelyaccuratelydescribethetimewhenthedocumentrevisionwasactuallywritten.

ArangoDBuses64bitunsignedintegervaluestomaintaindocumentrevisionsinternally.Atthisstageweintentionallydonotdocumenttheexactformatoftherevisionvalues.Whenreturningdocumentrevisionstoclients,ArangoDBwillputthemintoastringtoensuretherevisionisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreattherevisionreturnedbyArangoDBasanopaquestringwhentheystoreoruseitlocally.ThiswillallowArangoDBtochangetheformatofrevisionslaterifthisshouldberequired(ashashappenedwith3.1withtheHybridLogicalClock).Clientscanuserevisionstoperformsimpleequality/non-equalitycomparisons(e.g.tocheckwhetheradocumenthaschangedornot),buttheyshouldnotuserevisionidstoperformgreater/lessthancomparisonswiththemtocheckifadocumentrevisionisolderthanoneanother,evenifthismightworkforsomecases.

Documentrevisionscanbeusedtoconditionallyquery,update,replaceordeletedocumentsinthedatabase.Inordertofindaparticularrevisionofadocument,youneedthedocumenthandleorkey,andthedocumentrevision.

MultipleDocumentsinasingleCommand

BeginningwithArangoDB3.0thebasicdocumentAPIhasbeenextendedtohandlenotonlysingledocumentsbutmultipledocumentsinasinglecommand.Thisiscrucialforperformance,inparticularintheclustersituation,inwhichasinglerequestcaninvolvemultiplenetworkhopswithinthecluster.Anotheradvantageisthatitreducestheoverheadofindividualnetworkroundtripsbetweentheclientandtheserver.ThegeneralideatoperformmultipledocumentoperationsinasinglecommandistouseJSONarraysofobjectsintheplaceofasingledocument.Asaconsequence,documentkeys,handlesandrevisionsforpreconditionshavetobesuppliedembeddedintheindividualdocumentsgiven.Multipledocumentoperationsarerestrictedtoasingledocumentoredgecollection.SeetheAPIdescriptionsforcollectionobjectsfordetails.NotethattheAPIfordatabaseobjectsdonotoffertheseoperations.

BasicsandTerminology

64

CollectionMethods

All

collection.all()

Fetchesalldocumentsfromacollectionandreturnsacursor.YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.

Examples

UsetoArraytogetalldocumentsatonce:

arangosh>db.five.save({name:"one"});

arangosh>db.five.save({name:"two"});

arangosh>db.five.save({name:"three"});

arangosh>db.five.save({name:"four"});

arangosh>db.five.save({name:"five"});

arangosh>db.five.all().toArray();

showexecutionresultsUselimittorestrictthedocuments:

arangosh>db.five.save({name:"one"});

arangosh>db.five.save({name:"two"});

arangosh>db.five.save({name:"three"});

arangosh>db.five.save({name:"four"});

arangosh>db.five.save({name:"five"});

arangosh>db.five.all().limit(2).toArray();

showexecutionresults

Querybyexamplecollection.byExample(example)

Fetchesalldocumentsfromacollectionthatmatchthespecifiedexampleandreturnsacursor.

YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.

Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.Ifyouuse

{"a":{"c":1}}

asexample,thenyouwillfindalldocuments,suchthattheattributeacontainsadocumentoftheform{c:1}.Forexamplethedocument

{"a":{"c":1},"b":1}

willmatch,butthedocument

{"a":{"c":1,"b":1}}

willnot.

However,ifyouuse

{"a.c":1}

CollectionMethods

65

thenyouwillfindalldocuments,whichcontainasub-documentinathathasanattributecofvalue1.Boththefollowingdocuments

{"a":{"c":1},"b":1}

and

{"a":{"c":1,"b":1}}

willmatch.

collection.byExample(path1,value1,...)

Asalternativeyoucansupplyanarrayofpathsandvalues.

Examples

UsetoArraytogetalldocumentsatonce:

arangosh>db.users.save({name:"Gerhard"});

arangosh>db.users.save({name:"Helmut"});

arangosh>db.users.save({name:"Angela"});

arangosh>db.users.all().toArray();

arangosh>db.users.byExample({"_id":"users/20"}).toArray();

arangosh>db.users.byExample({"name":"Gerhard"}).toArray();

arangosh>db.users.byExample({"name":"Helmut","_id":"users/15"}).toArray();

showexecutionresultsUsenexttoloopoveralldocuments:

arangosh>db.users.save({name:"Gerhard"});

arangosh>db.users.save({name:"Helmut"});

arangosh>db.users.save({name:"Angela"});

arangosh>vara=db.users.byExample({"name":"Angela"});

arangosh>while(a.hasNext())print(a.next());

showexecutionresults

FirstExample

collection.firstExample(example)

Returnssomedocumentofacollectionthatmatchesthespecifiedexample.Ifnosuchdocumentexists,nullwillbereturned.Theexamplehastobespecifiedaspathsandvalues.SeebyExamplefordetails.

collection.firstExample(path1,value1,...)

Asalternativeyoucansupplyanarrayofpathsandvalues.

Examples

arangosh>db.users.firstExample("name","Angela");

showexecutionresults

Range

collection.range(attribute,left,right)

CollectionMethods

66

Returnsalldocumentsfromacollectionsuchthattheattributeisgreaterorequalthanleftandstrictlylessthanright.

YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.

Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.

Note:therangesimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionwithinaspecificrangeistouseanAQLqueryasfollows:

FORdocIN@@collection

FILTERdoc.value>=@left&&doc.value<@right

LIMIT@skip,@limit

RETURNdoc

Examples

UsetoArraytogetalldocumentsatonce:

arangosh>db.old.ensureIndex({type:"skiplist",fields:["age"]});

arangosh>db.old.save({age:15});

arangosh>db.old.save({age:25});

arangosh>db.old.save({age:30});

arangosh>db.old.range("age",10,30).toArray();

showexecutionresults

Closedrange

collection.closedRange(attribute,left,right)

Returnsalldocumentsofacollectionsuchthattheattributeisgreaterorequalthanleftandlessorequalthanright.

YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.

Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.

Note:theclosedRangesimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionwithinaspecificrangeistouseanAQLqueryasfollows:

FORdocIN@@collection

FILTERdoc.value>=@left&&doc.value<=@right

LIMIT@skip,@limit

RETURNdoc

Examples

UsetoArraytogetalldocumentsatonce:

arangosh>db.old.ensureIndex({type:"skiplist",fields:["age"]});

arangosh>db.old.save({age:15});

arangosh>db.old.save({age:25});

arangosh>db.old.save({age:30});

arangosh>db.old.closedRange("age",10,30).toArray();

showexecutionresults

Any

collection.any()

Returnsarandomdocumentfromthecollectionornullifnoneexists.

Note:thismethodisexpensivewhenusingtheRocksDBstorageengine.

CollectionMethods

67

Count

collection.count()

Returnsthenumberoflivingdocumentsinthecollection.

Examples

arangosh>db.users.count();

0

toArray

collection.toArray()

Convertsthecollectionintoanarrayofdocuments.NeverusethiscallinaproductionenvironmentasitwillbasicallycreateacopyofyourcollectioninRAMwhichwilluseresourcesdependingonthenumberandsizeofthedocumentsinyourcollecion.

Documentcollection.document(object)

Thedocumentmethodfindsadocumentgivenanobjectobjectcontainingthe_idor_keyattribute.Themethodreturnsthedocumentifitcanbefound.Ifbothattributesaregiven,the_idtakesprecedence,itisanerror,ifthecollectionpartofthe_iddoesnotmatchthecollection.

Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.Anerrorisalsothrownifnodocumentexistswiththegiven_idor_keyvalue.

Pleasenotethatifthemethodisexecutedonthearangodserver(e.g.frominsideaFoxxapplication),animmutabledocumentobjectwillbereturnedforperformancereasons.Itisnotpossibletochangeattributesofthisimmutableobject.Toupdateorpatchthereturneddocument,itneedstobecloned/copiedintoaregularJavaScriptobjectfirst.Thisisnotnecessaryifthedocumentmethodiscalledfromoutofarangoshorfromanyotherclient.

collection.document(document-handle)

Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.Norevisioncanbespecifiedinthiscase.

collection.document(document-key)

Asbefore.Insteadofobjectadocument-keycanbepassedasfirstargument.

collection.document(array)

Thisvariantallowstoperformtheoperationonawholearrayofarguments.Thebehaviorisexactlyasifdocumentwouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.

Examples

Returnsthedocumentforadocument-handle:

arangosh>db.example.document("example/2873916");

showexecutionresultsReturnsthedocumentforadocument-key:

arangosh>db.example.document("2873916");

showexecutionresultsReturnsthedocumentforanobject:

arangosh>db.example.document({_id:"example/2873916"});

CollectionMethods

68

showexecutionresultsReturnsthedocumentforanarrayoftwokeys:

arangosh>db.example.document(["2873916","2873917"]);

showexecutionresultsAnerrorisraisedifthedocumentisunknown:

arangosh>db.example.document("example/4472917");

[ArangoError1202:documentnotfound]

Anerrorisraisedifthehandleisinvalid:

arangosh>db.example.document("");

[ArangoError1205:illegaldocumenthandle]

Changesin3.0from2.8:

documentcannowquerymultipledocumentswithonecall.

Exists

checkswhetheradocumentexistscollection.exists(object)

Theexistsmethoddetermineswhetheradocumentexistsgivenanobjectobjectcontainingthe_idor_keyattribute.Ifbothattributesaregiven,the_idtakesprecedence,itisanerror,ifthecollectionpartofthe_iddoesnotmatchthecollection.

Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.

Insteadofreturningthefounddocumentoranerror,thismethodwillonlyreturnanobjectwiththeattributes_id,_keyand_rev,orfalseifnodocumentwiththegiven_idor_keyexists.Itcanthusbeusedforeasyexistencechecks.

Thismethodwillthrowanerrorifusedimproperly,e.g.whencalledwithanon-documenthandle,anon-document,orwhenacross-collectionrequestisperformed.

collection.exists(document-handle)

Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.

collection.exists(document-key)

Asbefore.Insteadofobjectadocument-keycanbepassedasfirstargument.

collection.exists(array)

Thisvariantallowstoperformtheoperationonawholearrayofarguments.Thebehaviorisexactlyasifexistswouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,theoperationstopsimmediatelyreturningonlyanerrorobject.

Changesin3.0from2.8:

Inthecaseofarevisionmismatchexistsnowthrowsanerrorinsteadofsimplyreturningfalse.Thisistomakeitpossibletotellthedifferencebetweenarevisionmismatchandanon-existingdocument.

existscannowquerymultipledocumentswithonecall.

LookupByKeyscollection.documents(keys)

CollectionMethods

69

Looksupthedocumentsinthespecifiedcollectionusingthearrayofkeysprovided.Alldocumentsforwhichamatchingkeywasspecifiedinthekeysarrayandthatexistinthecollectionwillbereturned.Keysforwhichnodocumentcanbefoundintheunderlyingcollectionareignored,andnoexceptionwillbethrownforthem.

Thismethodisdeprecatedinfavourofthearrayvariantofdocument.

Examples

arangosh>keys=[];

arangosh>for(vari=0;i<10;++i){

........>db.example.insert({_key:"test"+i,value:i});

........>keys.push("test"+i);

........>}

arangosh>db.example.documents(keys);

showexecutionresults

Insert

collection.insert(data)

Createsanewdocumentinthecollectionfromthegivendata.Thedatamustbeanobject.Theattributes_idand_revareignoredandareautomaticallygenerated.Auniquevaluefortheattribute_keywillbeautomaticallygeneratedifnotspecified.Ifspecified,theremustnotbeadocumentwiththegiven_keyinthecollection.

Themethodreturnsadocumentwiththeattributes_id,_keyand_rev.Theattribute_idcontainsthedocumenthandleofthenewlycreateddocument,theattribute_keythedocumentkeyandtheattribute_revcontainsthedocumentrevision.

collection.insert(data,options)

Createsanewdocumentinthecollectionfromthegivendataasabove.Theoptionaloptionsparametermustbeanobjectandcanbeusedtospecifythefollowingoptions:

waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.silent:Ifthisflagissettotrue,themethoddoesnotreturnanyoutput.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.

Note:sinceArangoDB2.2,insertisanaliasforsave.

collection.insert(array)

collection.insert(array,options)

Thesetwovariantsallowtoperformtheoperationonawholearrayofarguments.Thebehaviorisexactlyasifinsertwouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.

Changesin3.0from2.8:

TheoptionssilentandreturnNewarenew.Themethodcannowinsertmultipledocumentswithonecall.

Examples

arangosh>db.example.insert({Hello:"World"});

arangosh>db.example.insert({Hello:"World"},{waitForSync:true});

showexecutionresults

arangosh>db.example.insert([{Hello:"World"},{Hello:"there"}])

CollectionMethods

70

arangosh>db.example.insert([{Hello:"World"},{}],{waitForSync:true});

showexecutionresults

Replace

collection.replace(selector,data)

Replacesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idor_keyattribute.Theremustbeadocumentwiththat_idor_keyinthecurrentcollection.Thisdocumentisthenreplacedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.

Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowreplaced)document.

Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.

collection.replace(selector,data,options)

Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:

waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.

collection.replace(document-handle,data)

collection.replace(document-handle,data,options)

Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisionpreconditionistested.

collection.replace(document-key,data)

collection.replace(document-key,data,options)

Asbefore.Insteadofselectoradocument-keycanbepassedasfirstargument.Norevisionpreconditionistested.

collection.replace(selectorarray,dataarray)

collection.replace(selectorarray,dataarray,options)

Thesetwovariantsallowtoperformtheoperationonawholearrayofselector/datapairs.Thetwoarraysgivenasselectorarrayanddataarraymusthavethesamelength.Thebehaviorisexactlyasifreplacewouldhavebeencalledonallrespectivemembersofthetwoarraysandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.

Examples

Createandupdateadocument:

arangosh>a1=db.example.insert({a:1});

arangosh>a2=db.example.replace(a1,{a:2});

arangosh>a3=db.example.replace(a1,{a:3});

showexecutionresultsUseadocumenthandle:

arangosh>a1=db.example.insert({a:1});

CollectionMethods

71

arangosh>a2=db.example.replace("example/3903044",{a:2});

showexecutionresults

Changesin3.0from2.8:

Theoptionssilent,returnNewandreturnOldarenew.Themethodcannowreplacemultipledocumentswithonecall.

Update

collection.update(selector,data)

Updatesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idor_keyattribute.Theremustbeadocumentwiththat_idor_keyinthecurrentcollection.Thisdocumentisthenpatchedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.

Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowupdated)document.

Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.

collection.update(selector,data,options)

Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:

waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.keepNull:TheoptionalkeepNullparametercanbeusedtomodifythebehaviorwhenhandlingnullvalues.Normally,nullvaluesarestoredinthedatabase.BysettingthekeepNullparametertofalse,thisbehaviorcanbechangedsothatallattributesindatawithnullvalueswillberemovedfromthetargetdocument.mergeObjects:Controlswhetherobjects(notarrays)willbemergedifpresentinboththeexistingandthepatchdocument.Ifsettofalse,thevalueinthepatchdocumentwilloverwritetheexistingdocument'svalue.Ifsettotrue,objectswillbemerged.Thedefaultistrue.

collection.update(document-handle,data)

collection.update(document-handle,data,options)

Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisionpreconditionistested.

collection.update(document-key,data)

collection.update(document-key,data,options)

Asbefore.Insteadofselectoradocument-keycanbepassedasfirstargument.Norevisionpreconditionistested.

collection.update(selectorarray,dataarray)

collection.update(selectorarray,dataarray,options)

Thesetwovariantsallowtoperformtheoperationonawholearrayofselector/datapairs.Thetwoarraysgivenasselectorarrayanddataarraymusthavethesamelength.Thebehaviorisexactlyasifupdatewouldhavebeencalledonallrespectivemembersofthetwoarraysandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.

Examples

Createandupdateadocument:

CollectionMethods

72

arangosh>a1=db.example.insert({"a":1});

arangosh>a2=db.example.update(a1,{"b":2,"c":3});

arangosh>a3=db.example.update(a1,{"d":4});

arangosh>a4=db.example.update(a2,{"e":5,"f":6});

arangosh>db.example.document(a4);

arangosh>a5=db.example.update(a4,{"a":1,c:9,e:42});

arangosh>db.example.document(a5);

showexecutionresultsUseadocumenthandle:

arangosh>a1=db.example.insert({"a":1});

arangosh>a2=db.example.update("example/18612115",{"x":1,"y":2});

showexecutionresultsUsethekeepNullparametertoremoveattributeswithnullvalues:

arangosh>db.example.insert({"a":1});

arangosh>db.example.update("example/19988371",

........>{"b":null,"c":null,"d":3});

arangosh>db.example.document("example/19988371");

arangosh>db.example.update("example/19988371",{"a":null},false,false);

arangosh>db.example.document("example/19988371");

arangosh>db.example.update("example/19988371",

........>{"b":null,"c":null,"d":null},false,false);

arangosh>db.example.document("example/19988371");

showexecutionresultsPatchingarrayvalues:

arangosh>db.example.insert({"a":{"one":1,"two":2,"three":3},

........>"b":{}});

arangosh>db.example.update("example/20774803",{"a":{"four":4},

........>"b":{"b1":1}});

arangosh>db.example.document("example/20774803");

arangosh>db.example.update("example/20774803",{"a":{"one":null},

........>"b":null},

........>false,false);

arangosh>db.example.document("example/20774803");

showexecutionresults

Changesin3.0from2.8:

Theoptionssilent,returnNewandreturnOldarenew.Themethodcannowupdatemultipledocumentswithonecall.

Remove

collection.remove(selector)

Removesadocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idor_keyattribute.Theremustbeadocumentwiththat_idor_keyinthecurrentcollection.Thisdocumentisthenremoved.

Themethodreturnsadocumentwiththeattributes_id,_keyand_rev.Theattribute_idcontainsthedocumenthandleoftheremoveddocument,theattribute_revcontainsthedocumentrevisionoftheremoveddocument.

CollectionMethods

73

Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.

collection.remove(selector,options)

Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:

waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.

collection.remove(document-handle)

collection.remove(document-handle,options)

Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisioncheckisperformed.

collection.remove(document-key)

collection.remove(document-handle,options)

Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisioncheckisperformed.

collection.remove(selectorarray)

collection.remove(selectorarray,options)

Thesetwovariantsallowtoperformtheoperationonawholearrayofselectors.Thebehaviorisexactlyasifremovewouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.

Examples

Removeadocument:

arangosh>a1=db.example.insert({a:1});

arangosh>db.example.document(a1);

arangosh>db.example.remove(a1);

arangosh>db.example.document(a1);

showexecutionresultsRemoveadocumentwithaconflict:

arangosh>a1=db.example.insert({a:1});

arangosh>a2=db.example.replace(a1,{a:2});

arangosh>db.example.remove(a1);

arangosh>db.example.remove(a1,true);

arangosh>db.example.document(a1);

showexecutionresults

Changesin3.0from2.8:

Themethodnowreturnsnotonlytruebutinformationabouttheremoveddocument(s).TheoptionssilentandreturnOldarenew.Themethodcannowremovemultipledocumentswithonecall.

RemoveByKeys

collection.removeByKeys(keys)

CollectionMethods

74

Looksupthedocumentsinthespecifiedcollectionusingthearrayofkeysprovided,andremovesalldocumentsfromthecollectionwhosekeysarecontainedinthekeysarray.Keysforwhichnodocumentcanbefoundintheunderlyingcollectionareignored,andnoexceptionwillbethrownforthem.

Themethodwillreturnanobjectcontainingthenumberofremoveddocumentsintheremovedsub-attribute,andthenumberofnot-removed/ignoreddocumentsintheignoredsub-attribute.

Thismethodisdeprecatedinfavourofthearrayvariantofremove.

Examples

arangosh>keys=[];

arangosh>for(vari=0;i<10;++i){

........>db.example.insert({_key:"test"+i,value:i});

........>keys.push("test"+i);

........>}

arangosh>db.example.removeByKeys(keys);

showexecutionresults

RemoveByExamplecollection.removeByExample(example)

Removesalldocumentsmatchinganexample.

collection.removeByExample(document,waitForSync)

TheoptionalwaitForSyncparametercanbeusedtoforcesynchronizationofthedocumentdeletionoperationtodiskevenincasethatthewaitForSyncflaghadbeendisabledfortheentirecollection.Thus,thewaitForSyncparametercanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.

collection.removeByExample(document,waitForSync,limit)

Theoptionallimitparametercanbeusedtorestrictthenumberofremovalstothespecifiedvalue.Iflimitisspecifiedbutlessthanthenumberofdocumentsinthecollection,itisundefinedwhichdocumentsareremoved.

Examples

arangosh>db.example.removeByExample({Hello:"world"});

1

ReplaceByExample

collection.replaceByExample(example,newValue)

Replacesalldocumentsmatchinganexamplewithanewdocumentbody.TheentiredocumentbodyofeachdocumentmatchingtheexamplewillbereplacedwithnewValue.Thedocumentmeta-attributes_id,_keyand_revwillnotbereplaced.

collection.replaceByExample(document,newValue,waitForSync)

TheoptionalwaitForSyncparametercanbeusedtoforcesynchronizationofthedocumentreplacementoperationtodiskevenincasethatthewaitForSyncflaghadbeendisabledfortheentirecollection.Thus,thewaitForSyncparametercanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.

collection.replaceByExample(document,newValue,waitForSync,limit)

Theoptionallimitparametercanbeusedtorestrictthenumberofreplacementstothespecifiedvalue.Iflimitisspecifiedbutlessthanthenumberofdocumentsinthecollection,itisundefinedwhichdocumentsarereplaced.

CollectionMethods

75

Examples

arangosh>db.example.save({Hello:"world"});

arangosh>db.example.replaceByExample({Hello:"world"},{Hello:"mars"},false,5);

showexecutionresults

UpdateByExamplecollection.updateByExample(example,newValue)

Partiallyupdatesalldocumentsmatchinganexamplewithanewdocumentbody.SpecificattributesinthedocumentbodyofeachdocumentmatchingtheexamplewillbeupdatedwiththevaluesfromnewValue.Thedocumentmeta-attributes_id,_keyand_revcannotbeupdated.

Partialupdatecouldalsobeusedtoappendnewfields,iftherewerenooldfieldwithsamename.

collection.updateByExample(document,newValue,keepNull,waitForSync)

TheoptionalkeepNullparametercanbeusedtomodifythebehaviorwhenhandlingnullvalues.Normally,nullvaluesarestoredinthedatabase.BysettingthekeepNullparametertofalse,thisbehaviorcanbechangedsothatallattributesindatawithnullvalueswillberemovedfromthetargetdocument.

TheoptionalwaitForSyncparametercanbeusedtoforcesynchronizationofthedocumentreplacementoperationtodiskevenincasethatthewaitForSyncflaghadbeendisabledfortheentirecollection.Thus,thewaitForSyncparametercanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.

collection.updateByExample(document,newValue,keepNull,waitForSync,limit)

Theoptionallimitparametercanbeusedtorestrictthenumberofupdatestothespecifiedvalue.Iflimitisspecifiedbutlessthanthenumberofdocumentsinthecollection,itisundefinedwhichdocumentsareupdated.

collection.updateByExample(document,newValue,options)

Usingthisvariant,theoptionsfortheoperationcanbepassedusinganobjectwiththefollowingsub-attributes:

keepNullwaitForSynclimitmergeObjects

Examples

arangosh>db.example.save({Hello:"world",foo:"bar"});

arangosh>db.example.updateByExample({Hello:"world"},{Hello:"foo",World:"bar"},

false);

arangosh>db.example.byExample({Hello:"foo"}).toArray()

showexecutionresults

Collectiontypecollection.type()

Returnsthetypeofacollection.Possiblevaluesare:

2:documentcollection3:edgecollection

GettheVersionofArangoDBdb._version()

CollectionMethods

76

Returnstheserverversionstring.Notethatthisisnottheversionofthedatabase.

Examples

arangosh>require("@arangodb").db._version();

3.4.milestone0

Edges

Edgesarenormaldocumentsthatalwayscontaina_fromanda_toattribute.Therefore,youcanusethedocumentmethodstooperateonedges.Thefollowingmethods,however,arespecifictoedges.

edge-collection.edges(vertex)

Theedgesoperatorfindsalledgesstartingfrom(outbound)orendingin(inbound)vertex.

edge-collection.edges(vertices)

Theedgesoperatorfindsalledgesstartingfrom(outbound)orendingin(inbound)adocumentfromvertices,whichmustbealistofdocumentsordocumenthandles.

arangosh>db._create("vertex");

arangosh>db._createEdgeCollection("relation");

arangosh>varmyGraph={};

arangosh>myGraph.v1=db.vertex.insert({name:"vertex1"});

arangosh>myGraph.v2=db.vertex.insert({name:"vertex2"});

arangosh>myGraph.e1=db.relation.insert(myGraph.v1,myGraph.v2,

........>{label:"knows"});

arangosh>db._document(myGraph.e1);

arangosh>db.relation.edges(myGraph.e1._id);

showexecutionresultsedge-collection.inEdges(vertex)

Theedgesoperatorfindsalledgesendingin(inbound)vertex.

edge-collection.inEdges(vertices)

Theedgesoperatorfindsalledgesendingin(inbound)adocumentfromvertices,whichmustalistofdocumentsordocumenthandles.

Examples

arangosh>db._create("vertex");

arangosh>db._createEdgeCollection("relation");

arangosh>myGraph.v1=db.vertex.insert({name:"vertex1"});

arangosh>myGraph.v2=db.vertex.insert({name:"vertex2"});

arangosh>myGraph.e1=db.relation.insert(myGraph.v1,myGraph.v2,

........>{label:"knows"});

arangosh>db._document(myGraph.e1);

arangosh>db.relation.inEdges(myGraph.v1._id);

arangosh>db.relation.inEdges(myGraph.v2._id);

showexecutionresultsedge-collection.outEdges(vertex)

Theedgesoperatorfindsalledgesstartingfrom(outbound)vertices.

edge-collection.outEdges(vertices)

Theedgesoperatorfindsalledgesstartingfrom(outbound)adocumentfromvertices,whichmustalistofdocumentsordocumenthandles.

Examples

CollectionMethods

77

arangosh>db._create("vertex");

arangosh>db._createEdgeCollection("relation");

arangosh>myGraph.v1=db.vertex.insert({name:"vertex1"});

arangosh>myGraph.v2=db.vertex.insert({name:"vertex2"});

arangosh>myGraph.e1=db.relation.insert(myGraph.v1,myGraph.v2,

........>{label:"knows"});

arangosh>db._document(myGraph.e1);

arangosh>db.relation.outEdges(myGraph.v1._id);

arangosh>db.relation.outEdges(myGraph.v2._id);

showexecutionresults

Misc

collection.iterate(iterator,options)

Iteratesoversomeelementsofthecollectionandapplythefunctioniteratortotheelements.Thefunctionwillbecalledwiththedocumentasfirstargumentandthecurrentnumber(startingwith0)assecondargument.

optionsmustbeanobjectwiththefollowingattributes:

limit(optional,defaultnone):useatmostlimitdocuments.

probability(optional,defaultall):anumberbetween0and1.Documentsarechosenwiththisprobability.

Examples

arangosh>for(i=-90;i<=90;i+=10){

........>for(j=-180;j<=180;j+=10){

........>db.example.save({name:"Name/"+i+"/"+j,

........>home:[i,j],

........>work:[-i,-j]});

........>}

........>}

........>

arangosh>db.example.ensureIndex({type:"geo",fields:["home"]});

arangosh>items=db.example.getIndexes().map(function(x){returnx.id;});

........>db.example.index(items[1]);

showexecutionresults

CollectionMethods

78

DatabaseMethods

Document

db._document(object)

The_documentmethodfindsadocumentgivenanobjectobjectcontainingthe_idattribute.Themethodreturnsthedocumentifitcanbefound.

Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.Anerrorisalsothrownifnodocumentexistswiththegiven_id.

Pleasenotethatifthemethodisexecutedonthearangodserver(e.g.frominsideaFoxxapplication),animmutabledocumentobjectwillbereturnedforperformancereasons.Itisnotpossibletochangeattributesofthisimmutableobject.Toupdateorpatchthereturneddocument,itneedstobecloned/copiedintoaregularJavaScriptobjectfirst.Thisisnotnecessaryifthe_documentmethodiscalledfromoutofarangoshorfromanyotherclient.

db._document(document-handle)

Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.Norevisioncanbespecifiedinthiscase.

Examples

Returnsthedocument:

arangosh>db._document("example/12345");

showexecutionresults

Existsdb._exists(object)

The_existsmethoddetermineswhetheradocumentexistsgivenanobjectobjectcontainingthe_idattribute.

Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.

Insteadofreturningthefounddocumentoranerror,thismethodwillonlyreturnanobjectwiththeattributes_id,_keyand_rev,orfalseifnodocumentwiththegiven_idor_keyexists.Itcanthusbeusedforeasyexistencechecks.

Thismethodwillthrowanerrorifusedimproperly,e.g.whencalledwithanon-documenthandle,anon-document,orwhenacross-collectionrequestisperformed.

db._exists(document-handle)

Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.

Changesin3.0from2.8:

Inthecaseofarevisionmismatch_existsnowthrowsanerrorinsteadofsimplyreturningfalse.Thisistomakeitpossibletotellthedifferencebetweenarevisionmismatchandanon-existingdocument.

Replace

db._replace(selector,data)

Replacesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idattribute.Theremustbeadocumentwiththat_idinthecurrentdatabase.Thisdocumentisthenreplacedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.

Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowreplaced)document.

DatabaseMethods

79

Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.

collection.replace(selector,data,options)

Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:

waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.

db._replace(document-handle,data)

db._replace(document-handle,data,options)

Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisionpreconditionistested.

Examples

Createandreplaceadocument:

arangosh>a1=db.example.insert({a:1});

arangosh>a2=db._replace(a1,{a:2});

arangosh>a3=db._replace(a1,{a:3});

showexecutionresults

Changesin3.0from2.8:

Theoptionssilent,returnNewandreturnOldarenew.

Update

db._update(selector,data)

Updatesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idattribute.Theremustbeadocumentwiththat_idinthecurrentdatabase.Thisdocumentisthenpatchedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.

Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowupdated)document.

Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.

db._update(selector,data,options)

Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:

waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.

DatabaseMethods

80

silent:Ifthisflagissettotrue,nooutputisreturned.keepNull:TheoptionalkeepNullparametercanbeusedtomodifythebehaviorwhenhandlingnullvalues.Normally,nullvaluesarestoredinthedatabase.BysettingthekeepNullparametertofalse,thisbehaviorcanbechangedsothatallattributesindatawithnullvalueswillberemovedfromthetargetdocument.mergeObjects:Controlswhetherobjects(notarrays)willbemergedifpresentinboththeexistingandthepatchdocument.Ifsettofalse,thevalueinthepatchdocumentwilloverwritetheexistingdocument'svalue.Ifsettotrue,objectswillbemerged.Thedefaultistrue.

db._update(document-handle,data)

db._update(document-handle,data,options)

Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisionpreconditionistested.

Examples

Createandupdateadocument:

arangosh>a1=db.example.insert({a:1});

arangosh>a2=db._update(a1,{b:2});

arangosh>a3=db._update(a1,{c:3});

showexecutionresults

Changesin3.0from2.8:

Theoptionssilent,returnNewandreturnOldarenew.

Removedb._remove(selector)

Removesadocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idattribute.Theremustbeadocumentwiththat_idinthecurrentdatabase.Thisdocumentisthenremoved.

Themethodreturnsadocumentwiththeattributes_id,_keyand_rev.Theattribute_idcontainsthedocumenthandleoftheremoveddocument,theattribute_revcontainsthedocumentrevisionoftheremovedeocument.

Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.

db._remove(selector,options)

Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:

waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.

db._remove(document-handle)

db._remove(document-handle,options)

Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisioncheckisperformed.

Examples

Removeadocument:

arangosh>a1=db.example.insert({a:1});

DatabaseMethods

81

arangosh>db._remove(a1);

arangosh>db._remove(a1);

arangosh>db._remove(a1,{overwrite:true});

showexecutionresultsRemovethedocumentintherevisiona1withaconflict:

arangosh>a1=db.example.insert({a:1});

arangosh>a2=db._replace(a1,{a:2});

arangosh>db._remove(a1);

arangosh>db._remove(a1,{overwrite:true});

arangosh>db._document(a1);

showexecutionresultsRemoveadocumentusingnewsignature:

arangosh>db.example.insert({_key:"11265325374",a:1});

arangosh>db.example.remove("example/11265325374",

........>{overwrite:true,waitForSync:false})

showexecutionresults

Changesin3.0from2.8:

Themethodnowreturnsnotonlytruebutinformationabouttheremoveddocument(s).TheoptionssilentandreturnOldarenew.

DatabaseMethods

82

Graphs,Vertices&EdgesGraphs,vertices&edgesaredefinedintheGraphschapterindetails.

Relatedblogposts:

Graphsindatamodeling-istheemperornaked?IndexFreeAdjacencyorHybridIndexesforGraphDatabases

Graphs,Vertices&Edges

83

JavaScriptInterfacetoViewsThisisanintroductiontoArangoDB'sinterfaceforviewsandhowtohandleviewsfrontheJavaScriptshellarangosh.ForotherlanguagesseethecorrespondinglanguageAPI.

AddressofaView

AllviewsinArangoDBhaveauniqueidentifierandauniquename.ArangoDBinternallyusestheview'suniqueidentifiertolookupviews.Thisidentifier,however,ismanagedbyArangoDBandtheuserhasnocontroloverit.Inordertoallowuserstousetheirownnames,eachviewalsohasauniquenamewhichisspecifiedbytheuser.Toaccessaviewfromtheuserperspective,theviewnameshouldbeused,i.e.:

View

db._view(view-name)

Aviewiscreatedbya"db._createView"call.Thereturnedobjectmaythenbeusedviatheexposedmethods.

Forexample:Assumethattheviewidentifieris7254820andthenameisdemo,thentheviewcanbeaccessedas:

db._view("demo")

Ifnoviewwithsuchanameexists,thennullisreturned.

Createdb._createView(view-name,view-type,view-properties)

Thiscallwillcreateanewviewcalledview-name.ThismethodisadatabasemethodandisdocumentedindetailinDatabaseMethods

ViewTypes

Thecurrentlysupportedviewimplementationis:arangosearchasdescribedinArangoSearchView.

Views

84

DatabaseMethods

View

db._view(view-name)

Returnstheviewwiththegivennameornullifnosuchviewexists.

db._view(view-identifier)

Returnstheviewwiththegivenidentifierornullifnosuchviewexists.Accessingviewsbyidentifierisdiscouragedforendusers.Endusersshouldaccessviewsusingtheviewname.

Examples

Getaviewbyname:

arangosh>db._view("demo");

null

Getaviewbyid:

arangosh>db._view(123456);

[ArangoView123456,"demo"]

Unknownview:

arangosh>db._view("unknown");

null

Createdb._createView(view-name,view-type,view-properties)

view-typemustbeoneofthesupportedViewTypes

view-propertiesviewconfigurationspecifictoeachview-type

Createsanewviewnamedview-nameoftypeview-typewithpropertiesview-properties.Iftheviewnamealreadyexistsorifthenameformatisinvalid,anerroristhrown.Formoreinformationonvalidviewnamespleaserefertothenamingconventions.

Examples

Createaview:

arangosh>v=db._createView("example",\<view-type\>,\<view-properties\>);

arangosh>v.properties();

arangosh>db._dropView("example");

AllViewsdb._views()

Returnsallviewsofthegivendatabase.

Examples

Queryviews:

arangosh>db._createView("example",\<view-type\>,\<view-properties\>);

arangosh>db._views();

arangosh>db._dropView("example");

DatabaseMethods

85

Drop

db._dropView(view-name)

Dropsaviewnamedview-nameandallitsdata.Noerroristhrownifthereisnosuchview.

db._dropView(view-identifier)

Dropsaviewidentifiedbyview-identifierwithallitsdata.Noerroristhrownifthereisnosuchview.

Examples

Dropaview:

arangosh>db._createView("example",\<view-type\>,\<view-properties\>);

arangosh>v=db._view("example");

arangosh>db._dropView("example");

arangosh>v;

DatabaseMethods

86

ViewMethods

Drop

view.drop()

Dropsaviewandallitsdata.

Examples

Dropaview:

arangosh>db._createView("example",\<view-type\>,\<view-properties\>);

arangosh>v=db._view("example");

arangosh>v.drop();

arangosh>v;

QueryName

view.name()

Returnsthenameoftheview.

Examples

Getviewname:

arangosh>db._createView("example",\<view-type\>,\<view-properties\>);

arangosh>v=db._view("example");

arangosh>v.name();

arangosh>db._dropView("example");

ModifyNameview.rename(new-name)

Renamesaviewusingthenew-name.Thenew-namemustnotalreadybeusedbyadifferentview.new-namemustalsobeavalidviewname.Formoreinformationonvalidviewnamespleaserefertothenamingconventions.

Ifrenamingfailsforanyreason,anerroristhrown.

Examples

arangosh>db._createView("example",\<view-type\>,\<view-properties\>);

arangosh>v=db._view("example");

arangosh>v.name();

arangosh>v.rename("example-renamed");

arangosh>v.name();

arangosh>db._dropView("example-renamed");

QueryType

view.type()

Returnsthetypeoftheview.

Examples

Getviewtype:

arangosh>db._createView("example",\<view-type\>,\<view-properties\>);

arangosh>v=db._view("example");

arangosh>v.type();

arangosh>db._dropView("example");

ViewMethods

87

QueryProperties

view.properties()

Returnsthepropertiesoftheview.TheformatoftheresultisspecifictoeachofthesupportedViewTypes.

Examples

Getviewproperties:

arangosh>db._createView("example",\<view-type\>,\<view-properties\>);

arangosh>v=db._view("example");

arangosh>v.properties();

arangosh>db._dropView("example");

ModifyPropertiesview.properties(view-property-modification)

Modifiesthepropertiesoftheview.TheformatoftheresultisspecifictoeachofthesupportedViewTypes.

Examples

Modifyviewproperties:

arangosh>db._createView("example",\<view-type\>,\<view-properties\>);

arangosh>v=db._view("example");

arangosh>v.properties(\<view-property-modification\>);

arangosh>db._dropView("example");

ViewMethods

88

NamingConventionsinArangoDBThefollowingnamingconventionsshouldbefollowedbyuserswhencreatingdatabases,collectionsanddocumentsinArangoDB.

NamingConventions

89

DatabaseNamesArangoDBwillalwaysstartupwithadefaultdatabase,named_system.UserscancreateadditionaldatabasesinArangoDB,providedthedatabasenamesconformtothefollowingconstraints:

Databasenamesmustonlyconsistofthelettersatoz(bothloweranduppercaseallowed),thenumbers0to9,andtheunderscore(_)ordash(-)symbolsThisalsomeansthatanynon-ASCIIdatabasenamesarenotallowedDatabasenamesmustalwaysstartwithaletter.Databasenamesstartingwithanunderscoreareconsideredtobesystemdatabases,andusersshouldnotcreateordeletethoseThemaximumallowedlengthofadatabasenameis64bytesDatabasenamesarecase-sensitive

DatabaseNames

90

CollectionNamesUserscanpicknamesfortheircollectionsasdesired,providedthefollowingnamingconstraintsarenotviolated:

Collectionnamesmustonlyconsistofthelettersatoz(bothinloweranduppercase),thenumbers0to9,andtheunderscore(_)ordash(-)symbols.Thisalsomeansthatanynon-ASCIIcollectionnamesarenotallowedUser-definedcollectionnamesmustalwaysstartwithaletter.Systemcollectionnamesmuststartwithanunderscore.AllcollectionnamesstartingwithanunderscoreareconsideredtobesystemcollectionsthatareforArangoDB'sinternaluseonly.SystemcollectionnamesshouldnotbeusedbyendusersfortheirowncollectionsThemaximumallowedlengthofacollectionnameis64bytesCollectionnamesarecase-sensitive

CollectionNames

91

DocumentKeysUserscandefinetheirownkeysfordocumentstheysave.Thedocumentkeywillbesavedalongwithadocumentinthe_keyattribute.Userscanpickkeyvaluesasrequired,providedthatthevaluesconformtothefollowingrestrictions:

Thekeymustbeastringvalue.Numerickeysarenotallowed,butanynumericvaluecanbeputintoastringandcanthenbeusedasdocumentkey.Thekeymustbeatleast1byteandatmost254byteslong.Emptykeysaredisallowedwhenspecified(thoughitmaybevalidtocompletelyomitthe_keyattributefromadocument)Itmustconsistofthelettersa-z(loweroruppercase),thedigits0-9oranyofthefollowingpunctuationcharacters:_-:.@()+,=;$!*'%Anyothercharacters,especiallymulti-byteUTF-8sequences,whitespaceorpunctuationcharacterscannotbeusedinsidekeyvaluesThekeymustbeuniquewithinthecollectionitisused

Keysarecase-sensitive,i.e.myKeyandMyKEYareconsideredtobedifferentkeys.

Specifyingadocumentkeyisoptionalwhencreatingnewdocuments.Ifnodocumentkeyisspecifiedbytheuser,ArangoDBwillcreatethedocumentkeyitselfaseachdocumentisrequiredtohaveakey.

Therearenoguaranteesabouttheformatandpatternofauto-generateddocumentkeysotherthantheaboverestrictions.Clientsshouldthereforetreatauto-generateddocumentkeysasopaquevaluesandnotrelyontheirformat.

Thecurrentformatforgeneratedkeysisastringcontainingnumericdigits.Thenumericvaluesreflectchronologicaltimeinthesensethat_keyvaluesgeneratedlaterwillcontainhighernumbersthan_keyvaluesgeneratedearlier.Buttheexactvaluethatwillbegeneratedbytheserverisnotpredictable.Notethatifyousortonthe_keyattribute,stringcomparisonwillbeused,whichmeans"100"islessthan"99"etc.

DocumentKeys

92

AttributeNamesUserscanpickattributenamesfordocumentattributesasdesired,providedthefollowingattributenamingconstraintsarenotviolated:

AttributenamesstartingwithanunderscoreareconsideredtobesystemattributesforArangoDB'sinternaluse.SuchattributenamesarealreadyusedbyArangoDBforspecialpurposes:

_idisusedtocontainadocument'shandle_keyisusedtocontainadocument'suser-definedkey_revisusedtocontainthedocument'srevisionnumberInedgecollections,the

_from_to

attributesareusedtoreferenceotherdocuments.

Moresystemattributesmaybeaddedinthefuturewithoutfurthernoticesoendusersshouldtrytoavoidusingtheirownattributenamesstartingwithunderscores.

Theoretically,attributenamescanincludepunctuationandspecialcharactersasdesired,providedthenameisavalidUTF-8string.Formaximumportability,specialcharactersshouldbeavoidedthough.Forexample,attributenamesmaycontainthedotsymbol,butthedothasaspecialmeaninginJavaScriptandalsoinAQL,sowhenusingsuchattributenamesinoneoftheselanguages,theattributenameneedstobequotedbytheenduser.Overallitmightbebettertouseattributenameswhichdon'trequireanyquoting/escapinginalllanguagesused.Thisincludeslanguagesusedbytheclient(e.g.Ruby,PHP)iftheattributesaremappedtoobjectmembersthere.

Attributenamesstartingwithanat-mark(@)willneedtobeenclosedinbacktickswhenusedinanAQLquerytotellthemapartfrombindvariables.Thereforewedonotencouragetheuseofattributesstartingwithat-marks,thoughtheywillworkwhenusedproperly.ArangoDBdoesnotenforcealengthlimitforattributenames.However,longattributenamesmayusemorememoryinresultsetsetc.Thereforetheuseoflongattributenamesisdiscouraged.Attributenamesarecase-sensitive.Attributeswithemptynames(anemptystring)aredisallowed.

AttributeNames

93

HandlingIndexesThisisanintroductiontoArangoDB'sinterfaceforindexesingeneral.Therearespecialsectionsfor

IndexBasics:IntroductiontoallindextypesWhichindextousewhen:IndextypeandoptionsadviserIndexUtilization:HowArangoDBusesindexesWorkingwithIndexes:Howtohandleindexesprogrammaticallyusingthedbobject

HashIndexesSkiplistsPersistentIndexesFulltextIndexesGeo-spatialIndexesVertex-centricIndexes

Indexing

94

Indexbasics

Indexesallowfastaccesstodocuments,providedtheindexedattribute(s)areusedinaquery.WhileArangoDBautomaticallyindexessomesystemattributes,usersarefreetocreateextraindexesonnon-systemattributesofdocuments.

User-definedindexescanbecreatedoncollectionlevel.Mostuser-definedindexescanbecreatedbyspecifyingthenamesoftheindexattributes.Someindextypesallowindexingjustoneattribute(e.g.fulltextindex)whereasotherindextypesallowindexingmultipleattributesatthesametime.

Thesystemattributes_id,_key,_fromand_toareautomaticallyindexedbyArangoDB,withouttheuserbeingrequiredtocreateextraindexesforthem._idand_keyarecoveredbyacollection'sprimarykey,and_fromand_toarecoveredbyanedgecollection'sedgeindexautomatically.

Usingthesystemattribute_idinuser-definedindexesisnotpossible,butindexing_key,_rev,_from,and_tois.

ArangoDBprovidesthefollowingindextypes:

PrimaryIndex

Foreachcollectiontherewillalwaysbeaprimaryindexwhichisahashindexforthedocumentkeys(_keyattribute)ofalldocumentsinthecollection.Theprimaryindexallowsquickselectionofdocumentsinthecollectionusingeitherthe_keyor_idattributes.ItwillbeusedfromwithinAQLqueriesautomaticallywhenperformingequalitylookupson_keyor_id.

Therearealsodedicatedfunctionstofindadocumentgivenits_keyor_idthatwillalwaysmakeuseoftheprimaryindex:

db.collection.document("<document-key>");

db._document("<document-id>");

Astheprimaryindexisanunsortedhashindex,itcannotbeusedfornon-equalityrangequeriesorforsorting.

Theprimaryindexofacollectioncannotbedroppedorchanged,andthereisnomechanismtocreateuser-definedprimaryindexes.

EdgeIndex

Everyedgecollectionalsohasanautomaticallycreatededgeindex.Theedgeindexprovidesquickaccesstodocumentsbyeithertheir_fromor_toattributes.Itcanthereforebeusedtoquicklyfindconnectionsbetweenvertexdocumentsandisinvokedwhentheconnectingedgesofavertexarequeried.

EdgeindexesareusedfromwithinAQLwhenperformingequalitylookupson_fromor_tovaluesinanedgecollections.Therearealsodedicatedfunctionstofindedgesgiventheir_fromor_tovaluesthatwillalwaysmakeuseoftheedgeindex:

db.collection.edges("<from-value>");

db.collection.edges("<to-value>");

db.collection.outEdges("<from-value>");

db.collection.outEdges("<to-value>");

db.collection.inEdges("<from-value>");

db.collection.inEdges("<to-value>");

Internally,theedgeindexisimplementedasahashindex,whichstorestheunionofall_fromand_toattributes.Itcanbeusedforequalitylookups,butnotforrangequeriesorforsorting.Edgeindexesareautomaticallycreatedforedgecollections.Itisnotpossibletocreateuser-definededgeindexes.However,itispossibletofreelyusethe_fromand_toattributesinuser-definedindexes.

Anedgeindexcannotbedroppedorchanged.

HashIndex

Ahashindexcanbeusedtoquicklyfinddocumentswithspecificattributevalues.Thehashindexisunsorted,soitsupportsequalitylookupsbutnorangequeriesorsorting.

IndexBasics

95

Ahashindexcanbecreatedononeormultipledocumentattributes.Ahashindexwillonlybeusedbyaqueryifallindexattributesarepresentinthesearchcondition,andifallattributesarecomparedusingtheequality(==)operator.HashindexesareusedfromwithinAQLandseveralqueryfunctions,e.g.byExample,firstExampleetc.

Hashindexescanoptionallybedeclaredunique,thendisallowingsavingthesamevalue(s)intheindexedattribute(s).Hashindexescanoptionallybesparse.

Thedifferenttypesofhashindexeshavethefollowingcharacteristics:

uniquehashindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Tryingtoinsertadocumentwiththesamekeyvalueasanalreadyexistingdocumentwillleadtoauniqueconstraintviolation.

Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Akeyvalueofnullmayonlyoccuronceintheindex,sothistypeofindexcannotbeusedforoptionalattributes.

Theuniqueoptioncanalsobeusedtoensurethatnoduplicateedgesarecreated,byaddingacombinedindexforthefields_fromand_totoanedgecollection.

unique,sparsehashindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Documentsinwhichatleastoneoftheindexattributesisnotsetorhasavalueofnullarenotincludedintheindex.Thistypeofindexcanbeusedtoensurethattherearenoduplicatekeysinthecollectionfordocumentswhichhavetheindexedattributesset.Astheindexwillexcludedocumentsforwhichtheindexedattributesarenullornotset,itcanbeusedforoptionalattributes.

non-uniquehashindex:alldocumentsinthecollectionwillbeindexed.Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Duplicatekeyvaluescanoccuranddonotleadtouniqueconstraintviolations.

non-unique,sparsehashindex:onlythosedocumentswillbeindexedthathavealltheindexedattributessettoavalueotherthannull.Itcanbeusedforoptionalattributes.

Theamortizedcomplexityoflookup,insert,update,andremovaloperationsinuniquehashindexesisO(1).

Non-uniquehashindexeshaveanamortizedcomplexityofO(1)forinsert,update,andremovaloperations.Thatmeansnon-uniquehashindexescanbeusedonattributeswithlowcardinality.

Ifahashindexiscreatedonanattributethatismissinginallormanyofthedocuments,thebehaviorisasfollows:

iftheindexissparse,thedocumentsmissingtheattributewillnotbeindexedandnotuseindexmemory.Thesedocumentswillnotinfluencetheupdateorremovalperformancefortheindex.

iftheindexisnon-sparse,thedocumentsmissingtheattributewillbecontainedintheindexwithakeyvalueofnull.

Hashindexessupportindexingarrayvaluesiftheindexattributenameisextendedwitha[*].

SkiplistIndex

Askiplistisasortedindexstructure.Itcanbeusedtoquicklyfinddocumentswithspecificattributevalues,forrangequeriesandforreturningdocumentsfromtheindexinsortedorder.SkiplistswillbeusedfromwithinAQLandseveralqueryfunctions,e.g.byExample,firstExampleetc.

Skiplistindexeswillbeusedforlookups,rangequeriesandsortingonlyifeitherallindexattributesareprovidedinaquery,orifaleftmostprefixoftheindexattributesisspecified.

Forexample,ifaskiplistindexiscreatedonattributesvalue1andvalue2,thefollowingfilterconditionscanusetheindex(note:the<=and>=operatorsareintentionallyomittedhereforthesakeofbrevity):

FILTERdoc.value1==...

FILTERdoc.value1<...

FILTERdoc.value1>...

FILTERdoc.value1>...&&doc.value1<...

FILTERdoc.value1==...&&doc.value2==...

FILTERdoc.value1==...&&doc.value2>...

FILTERdoc.value1==...&&doc.value2>...&&doc.value2<...

IndexBasics

96

Inordertouseaskiplistindexforsorting,theindexattributesmustbespecifiedintheSORTclauseofthequeryinthesameorderastheyappearintheindexdefinition.Skiplistindexesarealwayscreatedinascendingorder,buttheycanbeusedtoaccesstheindexedelementsinbothascendingordescendingorder.However,foracombinedindex(anindexonmultipleattributes)thisrequiresthatthesortordersinasinglequeryasspecifiedintheSORTclausemustbeeitherallascending(optionallyommittedasascendingisthedefault)oralldescending.

Forexample,iftheskiplistindexiscreatedonattributesvalue1andvalue2(inthisorder),thenthefollowingsortsclausescanusetheindexforsorting:

SORTvalue1ASC,value2ASC(anditsequivalentSORTvalue1,value2)SORTvalue1DESC,value2DESC

SORTvalue1ASC(anditsequivalentSORTvalue1)SORTvalue1DESC

Thefollowingsortclausescannotmakeuseoftheindexorder,andrequireanextrasortstep:

SORTvalue1ASC,value2DESC

SORTvalue1DESC,value2ASC

SORTvalue2(anditsequivalentSORTvalue2ASC)SORTvalue2DESC(becausefirstindexedattributevalue1isnotusedinsortclause)

Note:thelattertwosortclausescannotusetheindexbecausethesortclausedoesnotrefertoaleftmostprefixoftheindexattributes.

Skiplistscanoptionallybedeclaredunique,disallowingsavingthesamevalueintheindexedattribute.Theycanbesparseornon-sparse.

Thedifferenttypesofskiplistindexeshavethefollowingcharacteristics:

uniqueskiplistindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Tryingtoinsertadocumentwiththesamekeyvalueasanalreadyexistingdocumentwillleadtoauniqueconstraintviolation.

Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Akeyvalueofnullmayonlyoccuronceintheindex,sothistypeofindexcannotbeusedforoptionalattributes.

unique,sparseskiplistindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Documentsinwhichatleastoneoftheindexattributesisnotsetorhasavalueofnullarenotincludedintheindex.Thistypeofindexcanbeusedtoensurethattherearenoduplicatekeysinthecollectionfordocumentswhichhavetheindexedattributesset.Astheindexwillexcludedocumentsforwhichtheindexedattributesarenullornotset,itcanbeusedforoptionalattributes.

non-uniqueskiplistindex:alldocumentsinthecollectionwillbeindexed.Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Duplicatekeyvaluescanoccuranddonotleadtouniqueconstraintviolations.

non-unique,sparseskiplistindex:onlythosedocumentswillbeindexedthathavealltheindexedattributessettoavalueotherthannull.Itcanbeusedforoptionalattributes.

Theoperationalamortizedcomplexityforskiplistindexesislogarithmicallycorrelatedwiththenumberofdocumentsintheindex.

Skiplistindexessupportindexingarrayvaluesiftheindexattributenameisextendedwitha[*]`.

PersistentIndex

Thepersistentindexisasortedindexwithpersistence.Theindexentriesarewrittentodiskwhendocumentsarestoredorupdated.Thatmeanstheindexentriesdonotneedtoberebuiltfromthecollectiondatawhentheserverisrestartedortheindexedcollectionisinitiallyloaded.Thususingpersistentindexesmayreducecollectionloadingtimes.

Thepersistentindextypecanbeusedforsecondaryindexesatthemoment.Thatmeansthepersistentindexcurrentlycannotbemadetheonlyindexforacollection,becausetherewillalwaysbethein-memoryprimaryindexforthecollectioninaddition,andpotentiallymoreindexes(suchastheedgesindexforanedgecollection).

TheindeximplementationisusingtheRocksDBengine,anditprovideslogarithmiccomplexityforinsert,update,andremoveoperations.Asthepersistentindexisnotanin-memoryindex,itdoesnotstorepointersintotheprimaryindexasallthein-memoryindexesdo,butinsteaditstoresadocument'sprimarykey.Toretrieveadocumentviaapersistentindexviaanindexvaluelookup,therewillthereforebe

IndexBasics

97

anadditionalO(1)lookupintotheprimaryindextofetchtheactualdocument.

Asthepersistentindexissorted,itcanbeusedforpointlookups,rangequeriesandsortingoperations,butonlyifeitherallindexattributesareprovidedinaquery,orifaleftmostprefixoftheindexattributesisspecified.

GeoIndex

Userscancreateadditionalgeoindexesononeormultipleattributesincollections.Ageoindexisusedtofindplacesonthesurfaceoftheearthfast.

Thegeoindexstorestwo-dimensionalcoordinates.Itcanbecreatedoneithertwoseparatedocumentattributes(latitudeandlongitude)orasinglearrayattributethatcontainsbothlatitudeandlongitude.Latitudeandlongitudemustbenumericvalues.

Thegeoindexprovidesoperationstofinddocumentswithcoordinatesnearesttoagivencomparisoncoordinate,andtofinddocumentswithcoordinatesthatarewithinaspecifiableradiusaroundacomparisoncoordinate.

ThegeoindexisusedviadedicatedfunctionsinAQL,thesimplequeriesfunctionsanditisimplicitlyappliedwheninAQLaSORTorFILTERisusedwiththedistancefunction.Otherwiseitwillnotbeusedforothertypesofqueriesorconditions.

FulltextIndex

Afulltextindexcanbeusedtofindwords,orprefixesofwordsinsidedocuments.Afulltextindexcanbecreatedonasingleattributeonly,andwillindexallwordscontainedindocumentsthathaveatextualvalueinthatattribute.Onlywordswitha(specifiable)minimumlengthareindexed.Wordtokenizationisdoneusingthewordboundaryanalysisprovidedbylibicu,whichistakingintoaccounttheselectedlanguageprovidedatserverstart.Wordsareindexedintheirlower-casedform.Theindexsupportscompletematchqueries(fullwords)andprefixqueries,plusbasiclogicaloperationssuchasand,orandnotforcombiningpartialresults.

Thefulltextindexissparse,meaningitwillonlyindexdocumentsforwhichtheindexattributeissetandcontainsastringvalue.Additionally,onlywordswithaconfigurableminimumlengthwillbeincludedintheindex.

ThefulltextindexisusedviadedicatedfunctionsinAQLorthesimplequeries,butwillnotbeenabledforothertypesofqueriesorconditions.

Indexingattributesandsub-attributes

Top-levelaswellasnestedattributescanbeindexed.Forattributesatthetoplevel,theattributenamesalonearerequired.Toindexasinglefield,passanarraywithasingleelement(stringoftheattributekey)tothefieldsparameteroftheensureIndex()method.Tocreateacombinedindexovermultiplefields,simplyaddmorememberstothefieldsarray:

//{name:"Smith",age:35}

db.posts.ensureIndex({type:"hash",fields:["name"]})

db.posts.ensureIndex({type:"hash",fields:["name","age"]})

Toindexsub-attributes,specifytheattributepathusingthedotnotation:

//{name:{last:"Smith",first:"John"}}

db.posts.ensureIndex({type:"hash",fields:["name.last"]})

db.posts.ensureIndex({type:"hash",fields:["name.last","name.first"]})

Indexingarrayvalues

Ifanindexattributecontainsanarray,ArangoDBwillstoretheentirearrayastheindexvaluebydefault.Accessingindividualmembersofthearrayviatheindexisnotpossiblethisway.

Tomakeanindexinserttheindividualarraymembersintotheindexinsteadoftheentirearrayvalue,aspecialarrayindexneedstobecreatedfortheattribute.Arrayindexescanbesetuplikeregularhashorskiplistindexesusingthecollection.ensureIndex()function.Tomakeahashorskiplistindexanarrayindex,theindexattributenameneedstobeextendedwith[*]whencreatingtheindexandwhenfilteringinanAQLqueryusingtheINoperator.

Thefollowingexamplecreatesanarrayhashindexonthetagsattributeinacollectionnamedposts:

IndexBasics

98

db.posts.ensureIndex({type:"hash",fields:["tags[*]"]});

db.posts.insert({tags:["foobar","baz","quux"]});

ThisarrayindexcanthenbeusedforlookingupindividualtagsvaluesfromAQLqueriesviatheINoperator:

FORdocINposts

FILTER'foobar'INdoc.tags

RETURNdoc

Itispossibletoaddthearrayexpansionoperator[*],butitisnotmandatory.Youmayuseittoindicatethatanarrayindexisused,itispurelycosmetichowever:

FORdocINposts

FILTER'foobar'INdoc.tags[*]

RETURNdoc

ThefollowingFILTERconditionswillnotusethearrayindex:

FILTERdoc.tagsANY=='foobar'

FILTERdoc.tagsANYIN'foobar'

FILTERdoc.tagsIN'foobar'

FILTERdoc.tags=='foobar'

FILTER'foobar'==doc.tags

Itisalsopossibletocreateanindexonsubattributesofarrayvalues.Thismakessenseiftheindexattributeisanarrayofobjects,e.g.

db.posts.ensureIndex({type:"hash",fields:["tags[*].name"]});

db.posts.insert({tags:[{name:"foobar"},{name:"baz"},{name:"quux"}]});

Thefollowingquerywillthenusethearrayindex(thisdoesrequirethearrayexpansionoperator):

FORdocINposts

FILTER'foobar'INdoc.tags[*].name

RETURNdoc

Ifyoustoreadocumenthavingthearraywhichdoescontainelementsnothavingthesubattributesthisdocumentwillalsobeindexedwiththevaluenull,whichinArangoDBisequaltoattributenotexisting.

ArangoDBsupportscreatingarrayindexeswithasingle[*]operatorperindexattribute.Forexample,creatinganindexasfollowsisnotsupported:

db.posts.ensureIndex({type:"hash",fields:["tags[*].name[*].value"]});

Arrayvalueswillautomaticallybede-duplicatedbeforebeinginsertedintoanarrayindex.Forexample,ifthefollowingdocumentisinsertedintothecollection,theduplicatearrayvaluebarwillbeinsertedonlyonce:

db.posts.insert({tags:["foobar","bar","bar"]});

Thisisdonetoavoidredudantstorageofthesameindexvalueforthesamedocument,whichwouldnotprovideanybenefit.

Ifanarrayindexisdeclaredunique,thede-duplicationofarrayvalueswillhappenbeforeinsertingthevaluesintotheindex,sotheaboveinsertoperationwithtwoidenticalvaluesbarwillnotnecessarilyfail

Itwillalwaysfailiftheindexalreadycontainsaninstanceofthebarvalue.However,ifthevaluebarisnotalreadypresentintheindex,thenthede-duplicationofthearrayvalueswilleffectivelyleadtobarbeinginsertedonlyonce.

Toturnoffthededuplicationofarrayvalues,itispossibletosetthededuplicateattributeonthearrayindextofalse.Thedefaultvaluefordeduplicateistruehowever,sode-duplicationwilltakeplaceifnotexplicitlyturnedoff.

db.posts.ensureIndex({type:"hash",fields:["tags[*]"],deduplicate:false});

IndexBasics

99

//willfailnow

db.posts.insert({tags:["foobar","bar","bar"]});

Ifanarrayindexisdeclaredandyoustoredocumentsthatdonothaveanarrayatthespecifiedattributethisdocumentwillnotbeinsertedintheindex.Hencethefollowingobjectswillnotbeindexed:

db.posts.ensureIndex({type:"hash",fields:["tags[*]"]});

db.posts.insert({something:"else"});

db.posts.insert({tags:null});

db.posts.insert({tags:"thisisnoarray"});

db.posts.insert({tags:{content:[1,2,3]}});

Anarrayindexisabletoindexexplicitnullvalues.Whenqueriedfornullvalues,itwillonlyreturnthosedocumentshavingexplicitlynullstoredinthearray,itwillnotreturnanydocumentsthatdonothavethearrayatall.

db.posts.ensureIndex({type:"hash",fields:["tags[*]"]});

db.posts.insert({tags:null})//Willnotbeindexed

db.posts.insert({tags:[]})//Willnotbeindexed

db.posts.insert({tags:[null]});//Willbeindexedfornull

db.posts.insert({tags:[null,1,2]});//Willbeindexedfornull,1and2

Declaringanarrayindexassparsedoesnothaveaneffectonthearraypartoftheindex,thisinparticularmeansthatexplicitnullvaluesarealsoindexedinthesparseversion.Ifanindexiscombinedfromanarrayandanormalattributethesparsitywillapplyfortheattributee.g.:

db.posts.ensureIndex({type:"hash",fields:["tags[*]","name"],sparse:true});

db.posts.insert({tags:null,name:"alice"})//Willnotbeindexed

db.posts.insert({tags:[],name:"alice"})//Willnotbeindexed

db.posts.insert({tags:[1,2,3]})//Willnotbeindexed

db.posts.insert({tags:[1,2,3],name:null})//Willnotbeindexed

db.posts.insert({tags:[1,2,3],name:"alice"})

//Willbeindexedfor[1,"alice"],[2,"alice"],[3,"alice"]

db.posts.insert({tags:[null],name:"bob"})

//Willbeindexedfor[null,"bob"]

PleasenotethatfilteringusingarrayindexesonlyworksfromwithinAQLqueriesandonlyifthequeryfiltersontheindexedattributeusingtheINoperator.Theothercomparisonoperators(==,!=,>,>=,<,<=,ANY,ALL,NONE)currentlycannotusearrayindexes.

Vertexcentricindexes

Asmentionedabove,themostimportantindexesforgraphsaretheedgeindexes,indexingthe_fromand_toattributesofedgecollections.Theyprovideveryquickaccesstoalledgesoriginatinginorarrivingatagivenvertex,whichallowstoquicklyfindallneighboursofavertexinagraph.

Inmanycasesonewouldliketorunmorespecificqueries,forexamplefindingamongsttheedgesoriginatinginagivenvertexonlythosewiththe20latesttimestamps.Exactlythisisachievedwith"vertexcentricindexes".Inasensethesearelocalizedindexesforanedgecollection,whichsitateverysinglevertex.

Technically,theyareimplementedinArangoDBasindexes,whichsortthecompleteedgecollectionfirstby_fromandthenbyotherattributes.Ifweforexamplehaveaskiplistindexontheattributes_fromandtimestampofanedgecollection,wecananswertheabovequestionveryquicklywithasinglerangelookupintheindex.

SinceArangoDB3.0onecancreatesortedindexes(type"skiplist"and"persistent")thatindexthespecialedgeattributes_fromor_toandadditionallyotherattributes.SinceArangoDB3.1,theseareusedingraphtraversals,whenappropriateFILTERstatementsarefoundbytheoptimizer.

Forexample,tocreateavertexcentricindexoftheabovetype,youwouldsimplydo

db.edges.ensureIndex({"type":"skiplist","fields":["_from","timestamp"]});

Then,querieslike

IndexBasics

100

FORv,e,pIN1..1OUTBOUND"V/1"edges

FILTERe.timestampALL>="2016-11-09"

RETURNp

willbeconsiderablyfasterincasetherearemanyedgesoriginatinginvertex"V/1"butonlyfewwitharecenttimestamp.

IndexBasics

101

WhichIndextousewhen

ArangoDBautomaticallyindexesthe_keyattributeineachcollection.Thereisnoneedtoindexthisattributeseparately.Pleasenotethatadocument's_idattributeisderivedfromthe_keyattribute,andisthusimplicitlyindexed,too.

ArangoDBwillalsoautomaticallycreateanindexon_fromand_toinanyedgecollection,meaningincomingandoutgoingconnectionscanbedeterminedefficiently.

Indextypes

Userscandefineadditionalindexesononeormultipledocumentattributes.SeveraldifferentindextypesareprovidedbyArangoDB.Theseindexeshavedifferentusagescenarios:

hashindex:providesquickaccesstoindividualdocumentsif(andonlyif)allindexedattributesareprovidedinthesearchquery.Theindexwillonlybeusedforequalitycomparisons.Itdoesnotsupportrangequeriesandcannotbeusedforsorting.

Thehashindexisagoodcandidateifallormostqueriesontheindexedattribute(s)areequalitycomparisons.TheuniquehashindexprovidesanamortizedcomplexityofO(1)forinsert,update,removeandlookupoperations.Thenon-uniquehashindexprovidesO(1)inserts,updatesandremoves,andwillallowlookingupdocumentsbyindexvaluewithamortizedO(n)complexity,withnbeingthenumberofdocumentswiththatindexvalue.

Anon-uniquehashindexonanoptionaldocumentattributeshouldbedeclaredsparsesothatitwillnotindexdocumentsforwhichtheindexattributeisnotset.

skiplistindex:skiplistskeeptheindexedvaluesinanorder,sotheycanbeusedforequalitylookups,rangequeriesandforsorting.Forhighselectivityattributes,skiplistindexeswillhaveahigheroverheadthanhashindexes.Forlowselectivityattributes,skiplistindexeswillbemoreefficientthannon-uniquehashindexes.

Additionally,skiplistindexesallowmoreusecases(e.g.rangequeries,sorting)thanhashindexes.Furthermore,theycanbeusedforlookupsbasedonaleftmostprefixoftheindexattributes.

persistentindex:apersistentindexbehavesmuchlikethesortedskiplistindex,exceptthatallindexvaluesarepersistedondiskanddonotneedtoberebuiltinmemorywhentheserverisrestartedortheindexedcollectionisreloaded.Theoperationsinapersistentindexhavelogarithmiccomplexity,butoperationshavemayhaveahigherconstantfactorthantheoperationsinaskiplistindex,becausethepersistentindexmayneedtomakeextraroundtripstotheprimaryindextofetchtheactualdocuments.

Apersistentindexcanbeusedforequalitylookups,rangequeriesandforsorting.Forhighselectivityattributes,persistentindexeswillhaveahigheroverheadthanskiplistorhashindexes.

Persistentindexesallowmoreusecases(e.g.rangequeries,sorting)thanhashindexes.Furthermore,theycanbeusedforlookupsbasedonaleftmostprefixoftheindexattributes.Incontrasttothein-memoryskiplistindexes,persistentindexesdonotneedtoberebuiltin-memorysotheydon'tinfluencetheloadingtimeofcollectionsasotherin-memoryindexesdo.

geoindex:thegeoindexprovidedbyArangoDBallowssearchingfordocumentswithinaradiusaroundatwo-dimensionalearthcoordinate(point),ortofinddocumentswithareclosesttoapoint.Documentcoordinatescaneitherbespecifiedintwodifferentdocumentattributesorinasingleattribute,e.g.

{"latitude":50.9406645,"longitude":6.9599115}

or

{"coords":[50.9406645,6.9599115]}

GeoindexeswillbeinvokedviaspecialfunctionsorAQLoptimization.TheoptimizationcanbetriggeredwhenacollectionwithgeoindexisenumeratedandaSORTorFILTERstatementisusedinconjunctionwiththedistancefunction.

fulltextindex:afulltextindexcanbeusedtoindexallwordscontainedinaspecificattributeofalldocumentsinacollection.Onlywordswitha(specifiable)minimumlengthareindexed.Wordtokenizationisdoneusingthewordboundaryanalysisprovidedbylibicu,whichistakingintoaccounttheselectedlanguageprovidedatserverstart.

Whichindextousewhen

102

Theindexsupportscompletematchqueries(fullwords)andprefixqueries.Fulltextindexeswillonlybeinvokedviaspecialfunctions.

Sparsevs.non-sparseindexes

Hashindexesandskiplistindexescanoptionallybecreatedsparse.Asparseindexdoesnotcontaindocumentsforwhichatleastoneoftheindexattributeisnotsetorcontainsavalueofnull.

Assuchdocumentsareexcludedfromsparseindexes,theymaycontainfewerdocumentsthantheirnon-sparsecounterparts.Thisenablesfasterindexingandcanleadtoreducedmemoryusageincasetheindexedattributedoesoccuronlyinsome,butnotalldocumentsofthecollection.Sparseindexeswillalsoreducethenumberofcollisionsinnon-uniquehashindexesincasenon-existingoroptionalattributesareindexed.

Inordertocreateasparseindex,anobjectwiththeattributesparsecanbeaddedtotheindexcreationcommands:

db.collection.ensureIndex({type:"hash",fields:["attributeName"],sparse:true});

db.collection.ensureIndex({type:"hash",fields:["attributeName1","attributeName2"],sparse:true});

db.collection.ensureIndex({type:"hash",fields:["attributeName"],unique:true,sparse:true});

db.collection.ensureIndex({type:"hash",fields:["attributeName1","attributeName2"],unique:true,sparse:true});

db.collection.ensureIndex({type:"skiplist",fields:["attributeName"],sparse:true});

db.collection.ensureIndex({type:"skiplist",fields:["attributeName1","attributeName2"],sparse:true});

db.collection.ensureIndex({type:"skiplist",fields:["attributeName"],unique:true,sparse:true});

db.collection.ensureIndex({type:"skiplist",fields:["attributeName1","attributeName2"],unique:true,sparse:true});

Whennotexplicitlyset,thesparseattributedefaultstofalsefornewindexes.Otherindexesthanhashandskiplistdonotsupportsparsity.

Assparseindexesmayexcludesomedocumentsfromthecollection,theycannotbeusedforalltypesofqueries.Sparsehashindexescannotbeusedtofinddocumentsforwhichatleastoneoftheindexedattributeshasavalueofnull.Forexample,thefollowingAQLquerycannotuseasparseindex,evenifonewascreatedonattributeattr:

FORdocIncollection

FILTERdoc.attr==null

RETURNdoc

Ifthelookupvalueisnon-constant,asparseindexmayormaynotbeused,dependingontheothertypesofconditionsinthequery.Iftheoptimizercansafelydeterminethatthelookupvaluecannotbenull,asparseindexmaybeused.Whenuncertain,theoptimizerwillnotmakeuseofasparseindexinaqueryinordertoproducecorrectresults.

Forexample,thefollowingqueriescannotuseasparseindexonattrbecausetheoptimizerwillnotknowbeforehandwhetherthevalueswhicharecomparedtodoc.attrwillincludenull:

FORdocIncollection

FILTERdoc.attr==SOME_FUNCTION(...)

RETURNdoc

FORotherINotherCollection

FORdocIncollection

FILTERdoc.attr==other.attr

RETURNdoc

Sparseskiplistindexescanbeusedforsortingiftheoptimizercansafelydetectthattheindexrangedoesnotincludenullforanyoftheindexattributes.

Notethatifyouintendtousejoinsitmaybeclevertousenon-sparsityandmaybeevenuniquenessforthatattribute,elseallitemscontainingthenullvaluewillmatchagainsteachotherandthusproducelargeresults.

Whichindextousewhen

103

IndexUtilizationInmostcasesArangoDBwilluseasingleindexpercollectioninagivenquery.AQLqueriescanusemorethanoneindexpercollectionwhenmultipleFILTERconditionsarecombinedwithalogicalORandthesecanbecoveredbyindexes.AQLquerieswilluseasingleindexpercollectionwhenFILTERconditionsarecombinedwithlogicalAND.

Creatingmultipleindexesondifferentattributesofthesamecollectionmaygivethequeryoptimizermorechoiceswhenpickinganindex.Creatingmultipleindexesondifferentattributescanalsohelpinspeedingupdifferentqueries,withFILTERconditionsondifferentattributes.

Itisoftenbeneficialtocreateanindexonmorethanjustoneattribute.Byaddingmoreattributestoanindex,anindexcanbecomemoreselectiveandthusreducethenumberofdocumentsthatqueriesneedtoprocess.

ArangoDB'sprimaryindexes,edgesindexesandhashindexeswillautomaticallyprovideselectivityestimates.Indexselectivityestimatesareprovidedinthewebinterface,thegetIndexes()returnvalueandintheexplain()outputforagivenquery.

Themoreselectiveanindexis,themoredocumentsitwillfilteronaverage.Theindexselectivityestimatesarethereforeusedbytheoptimizerwhencreatingqueryexecutionplanswhentherearemultipleindexestheoptimizercanchoosefrom.Theoptimizerwillthenselectacombinationofindexeswiththelowestestimatedtotalcost.Ingeneral,theoptimizerwillpicktheindexeswiththehighestestimatedselectivity.

Sparseindexesmayormaynotbepickedbytheoptimizerinaquery.Assparseindexesdonotcontainnullvalues,theywillnotbeusedforqueriesiftheoptimizercannotsafelydeterminewhetheraFILTERconditionincludesnullvaluesfortheindexattributes.Theoptimizerpolicyistoproducecorrectresults,regardlessofwhetherorwhichindexisusedtosatisfyFILTERconditions.Ifitisunsureaboutwhetherusinganindexwillviolatethepolicy,itwillnotmakeuseoftheindex.

Troubleshooting

WhenindoubtaboutwhetherandwhichindexeswillbeusedforexecutingagivenAQLquery,clicktheExplainbuttoninthewebinterfaceintheQueriesvieworusetheexplain()methodforthestatementasfollows(fromtheArangoShell):

varquery="FORdocINcollectionFILTERdoc.value>42RETURNdoc";

varstmt=db._createStatement(query);

stmt.explain();

Theexplain()commandwillreturnadetailedJSONrepresentationofthequery'sexecutionplan.TheJSONexplainoutputisintendedtobeusedbycode.Togetahuman-readableandmuchmorecompactexplanationofthequery,thereisanexplainertool:

varquery="FORdocINcollectionFILTERdoc.value>42RETURNdoc";

require("@arangodb/aql/explainer").explain(query);

Ifanyoftheexplainmethodsshowsthataqueryisnotusingindexes,thefollowingstepsmayhelp:

checkiftheattributenamesinthequeryarecorrectlyspelled.Inaschema-freedatabase,documentsinthesamecollectioncanhavevaryingstructures.Thereisnosuchthingasanon-existingattributeerror.Aquerythatreferstoattributenamesnotpresentinanyofthedocumentswillnotreturnanerror,andobviouslywillnotbenefitfromindexes.

checkthereturnvalueofthegetIndexes()methodforthecollectionsusedinthequeryandvalidatethatindexesareactuallypresentontheattributesusedinthequery'sfilterconditions.

ifindexesarepresentbutnotusedbythequery,theindexesmayhavethewrongtype.Forexample,ahashindexwillonlybeusedforequalitycomparisons(i.e.==)butnotforothercomparisontypessuchas<,<=,>,>=.Additionallyhashindexeswillonlybeusedifalloftheindexattributesareusedinthequery'sFILTERconditions.AskiplistindexwillonlybeusedifatleastitsfirstattributeisusedinaFILTERcondition.Ifadditionallyoftheskiplistindexattributesarespecifiedinthequery(fromleft-to-right),theymayalsobeusedandallowtofiltermoredocuments.

usingindexedattributesasfunctionparametersorinarbitraryexpressionswilllikelyleadtotheindexontheattributenotbeingused.Forexample,thefollowingquerieswillnotuseanindexonvalue:

IndexUtilization

104

FORdocINcollectionFILTERTO_NUMBER(doc.value)==42RETURNdoc

FORdocINcollectionFILTERdoc.value-1==42RETURNdoc

Inthesecasesthequeriesshouldberewrittensothatonlytheindexattributeispresentononesideoftheoperator,oradditionalfiltersandindexesshouldbeusedtorestricttheamountofdocumentsotherwise.

certainAQLfunctionssuchasWITHIN()orFULLTEXT()doutilizeindexesinternally,buttheiruseisnotmentionedinthequeryexplanationforfunctionsingeneral.Thesefunctionswillraisequeryerrors(atruntime)ifnosuitableindexispresentforthecollectioninquestion.

thequeryoptimizerwillingeneralpickoneindexpercollectioninaquery.ItcanpickmorethanoneindexpercollectioniftheFILTERconditioncontainsmultiplebranchescombinedwithlogicalOR.Forexample,thefollowingqueriescanuseindexes:

FORdocINcollectionFILTERdoc.value1==42||doc.value1==23RETURNdoc

FORdocINcollectionFILTERdoc.value1==42||doc.value2==23RETURNdoc

FORdocINcollectionFILTERdoc.value1<42||doc.value2>23RETURNdoc

ThetwoORsinthefirstquerywillbeconvertedtoanINlist,andifthereisasuitableindexonvalue1,itwillbeused.Thesecondqueryrequirestwoseparateindexesonvalue1andvalue2andwillusethemifpresent.Thethirdquerycanusetheindexesonvalue1andvalue2whentheyaresorted.

IndexUtilization

105

WorkingwithIndexes

IndexIdentifiersandHandles

Anindexhandleuniquelyidentifiesanindexinthedatabase.Itisastringandconsistsofthecollectionnameandanindexidentifierseparatedbya/.Theindexidentifierpartisanumericvaluethatisauto-generatedbyArangoDB.

Aspecificindexofacollectioncanbeaccessedusingitsindexhandleorindexidentifierasfollows:

db.collection.index("<index-handle>");

db.collection.index("<index-identifier>");

db._index("<index-handle>");

Forexample:Assumethattheindexhandle,whichisstoredinthe_idattributeoftheindex,isdemo/362549736andtheindexwascreatedinacollectionnameddemo.Thenthisindexcanbeaccessedas:

db.demo.index("demo/362549736");

Becausetheindexhandleisuniquewithinthedatabase,youcanleaveoutthecollectionandusetheshortcut:

db._index("demo/362549736");

CollectionMethods

Listingallindexesofacollection

returnsinformationabouttheindexesgetIndexes()

Returnsanarrayofallindexesdefinedforthecollection.

Notethat_keyimplicitlyhasanindexassignedtoit.

arangosh>db.test.ensureHashIndex("hashListAttribute",

........>"hashListSecondAttribute.subAttribute");

arangosh>db.test.getIndexes();

showexecutionresults

Creatinganindex

IndexesshouldbecreatedusingthegeneralmethodensureIndex.Thismethodobsoletesthespecializedindex-specificmethodsensureHashIndex,ensureSkiplist,ensureUniqueConstraintetc.

ensuresthatanindexexistscollection.ensureIndex(index-description)

Ensuresthatanindexaccordingtotheindex-descriptionexists.Anewindexwillbecreatedifnoneexistswiththegivendescription.

Theindex-descriptionmustcontainatleastatypeattribute.Otherattributesmaybenecessary,dependingontheindextype.

typecanbeoneofthefollowingvalues:

hash:hashindexskiplist:skiplistindexfulltext:fulltextindexgeo1:geoindex,withoneattributegeo2:geoindex,withtwoattributes

WorkingwithIndexes

106

sparsecanbetrueorfalse.

Forhash,andskiplistthesparsitycanbecontrolled,fulltextandgeoaresparsebydefinition.

uniquecanbetrueorfalseandissupportedbyhashorskiplist

Callingthismethodreturnsanindexobject.WhetherornottheindexobjectexistedbeforethecallisindicatedinthereturnattributeisNewlyCreated.

deduplicatecanbetrueorfalseandissupportedbyarrayindexesoftypehashorskiplist.Itcontrolswhetherinsertingduplicateindexvaluesfromthesamedocumentintoauniquearrayindexwillleadtoauniqueconstrainterrorornot.Thedefaultvalueistrue,soonlyasingleinstanceofeachnon-uniqueindexvaluewillbeinsertedintotheindexperdocument.Tryingtoinsertavalueintotheindexthatalreadyexistsintheindexwillalwaysfail,regardlessofthevalueofthisattribute.

Examples

arangosh>db.test.ensureIndex({type:"hash",fields:["a"],sparse:true});

arangosh>db.test.ensureIndex({type:"hash",fields:["a","b"],unique:true});

showexecutionresults

Droppinganindex

dropsanindexcollection.dropIndex(index)

Dropstheindex.Iftheindexdoesnotexist,thenfalseisreturned.Iftheindexexistedandwasdropped,thentrueisreturned.Notethatyoucannotdropsomespecialindexes(e.g.theprimaryindexofacollectionortheedgeindexofanedgecollection).

collection.dropIndex(index-handle)

Sameasabove.Insteadofanindexanindexhandlecanbegiven.

arangosh>db.example.ensureSkiplist("a","b");

arangosh>varindexInfo=db.example.getIndexes();

arangosh>indexInfo;

arangosh>db.example.dropIndex(indexInfo[0])

arangosh>db.example.dropIndex(indexInfo[1].id)

arangosh>indexInfo=db.example.getIndexes();

showexecutionresults

LoadIndexesintoMemory

LoadsallindexesofthiscollectionintoMemory.collection.loadIndexesIntoMemory()

Thisfunctiontriestocacheallindexentriesofthiscollectionintothemainmemory.Thereforeititeratesoverallindexesofthecollectionandstorestheindexedvalues,nottheentiredocumentdata,inmemory.Alllookupsthatcouldbefoundinthecachearemuchfasterthanlookupsnotstoredinthecachesoyougetaniceperformanceboost.Itisalsoguaranteedthatthecacheisconsistentwiththestoreddata.

ForthetimebeingthisfunctionisonlyusefulonRocksDBstorageengine,asinMMFilesengineallindexesareinmemoryanyways.

OnRocksDBthisfunctionhonorsallmemorylimits,iftheindexesyouwanttoloadaresmallerthanyourmemorylimitthisfunctionguaranteesthatmostindexvaluesarecached.Iftheindexislargerthanyourmemorylimitthisfunctionwillfillupvaluesuptothislimitandforthetimebeingthereisnowaytocontrolwhichindexesofthecollectionshouldhavepriorityoverothers.

arangosh>db.example.loadIndexesIntoMemory();

{

"result":true

}

WorkingwithIndexes

107

DatabaseMethods

Fetchinganindexbyhandle

findsanindexdb._index(index-handle)

Returnstheindexwithindex-handleornullifnosuchindexexists.

arangosh>db.example.ensureIndex({type:"skiplist",fields:["a","b"]});

arangosh>varindexInfo=db.example.getIndexes().map(function(x){returnx.id;});

arangosh>indexInfo;

arangosh>db._index(indexInfo[0])

arangosh>db._index(indexInfo[1])

showexecutionresults

Droppinganindex

dropsanindexdb._dropIndex(index)

Dropstheindex.Iftheindexdoesnotexist,thenfalseisreturned.Iftheindexexistedandwasdropped,thentrueisreturned.

db._dropIndex(index-handle)

Dropstheindexwithindex-handle.

arangosh>db.example.ensureIndex({type:"skiplist",fields:["a","b"]});

arangosh>varindexInfo=db.example.getIndexes();

arangosh>indexInfo;

arangosh>db._dropIndex(indexInfo[0])

arangosh>db._dropIndex(indexInfo[1].id)

arangosh>indexInfo=db.example.getIndexes();

showexecutionresults

Revalidatingwhetheranindexisused

findsanindex

Soyou'vecreatedanindex,andsinceitsmaintainanceisn'tforfree,youdefinitelywanttoknowwhetheryourquerycanutilizeit.

Youcanuseexplaintoverifywhetherskiplistsorhashindexesareused(ifyouomitcolors:falseyouwillgetnicecolorsinArangoShell):

arangosh>varexplain=require("@arangodb/aql/explainer").explain;

arangosh>db.example.ensureIndex({type:"skiplist",fields:["a","b"]});

arangosh>explain("FORdocINexampleFILTERdoc.a<23RETURNdoc",{colors:false});

showexecutionresults

WorkingwithIndexes

108

HashIndexes

IntroductiontoHashIndexes

Itispossibletodefineahashindexononeormoreattributes(orpaths)ofadocument.ThishashindexisthenusedinqueriestolocatedocumentsinO(1)operations.Ifthehashindexisunique,thennotwodocumentsareallowedtohavethesamesetofattributevalues.

Creatinganewdocumentorupdatingadocumentwillfailiftheuniquenessisviolated.Iftheindexisdeclaredsparse,adocumentwillbeexcludedfromtheindexandnouniquenesscheckswillbeperformedifanyindexattributevalueisnotsetorhasavalueofnull.

AccessingHashIndexesfromtheShell

UniqueHashIndexes

Ensuresthatauniqueconstraintexists:collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"],unique:true})

Createsauniquehashindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Alldocumentsinthecollectionmustdifferintermsoftheindexedattributes.Creatinganewdocumentorupdatinganexistingdocumentwillwillfailiftheattributeuniquenessisviolated.

Tocreateasparseuniqueindex,setthesparseattributetotrue:

collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"],unique:true,sparse:true})

Incasethattheindexwassuccessfullycreated,theindexidentifierisreturned.

Non-existingattributeswilldefaulttonull.Inasparseindexalldocumentswillbeexcludedfromtheindexforwhichallspecifiedindexattributesarenull.Suchdocumentswillnotbetakenintoaccountforuniquenesschecks.

Inanon-sparseindex,alldocumentsregardlessofnull-attributeswillbeindexedandwillbetakenintoaccountforuniquenesschecks.

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

arangosh>db.test.ensureIndex({type:"hash",fields:["a","b.c"],unique:true});

arangosh>db.test.save({a:1,b:{c:1}});

arangosh>db.test.save({a:1,b:{c:1}});

arangosh>db.test.save({a:1,b:{c:null}});

arangosh>db.test.save({a:1});

showexecutionresults

Non-uniqueHashIndexes

Ensuresthatanon-uniquehashindexexists:collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"]})

Createsanon-uniquehashindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Tocreateasparseuniqueindex,setthesparseattributetotrue:

collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"],sparse:true})

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

arangosh>db.test.ensureIndex({type:"hash",fields:["a"]});

arangosh>db.test.save({a:1});

HashIndexes

109

arangosh>db.test.save({a:1});

arangosh>db.test.save({a:null});

showexecutionresults

HashArrayIndexes

Ensuresthatahasharrayindexexists(non-unique):collection.ensureIndex({type:"hash",fields:["field1[*]",...,"fieldn[*]"]})

Createsanon-uniquehasharrayindexfortheindividualelementsofthearrayattributesfield1[*],...fieldn[*]foundinthedocuments.Atleastoneattributepathhastobegiven.Theindexalwaystreatstheindexedarraysassparse.

Itispossibletocombinearrayindexingwithstandardindexing:collection.ensureIndex({type:"hash",fields:["field1[*]","field2"]})

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

arangosh>db.test.ensureIndex({type:"hash",fields:["a[*]"]});

arangosh>db.test.save({a:[1,2]});

arangosh>db.test.save({a:[1,3]});

arangosh>db.test.save({a:null});

showexecutionresults

Ensureuniquenessofrelationsinedgecollections

Itispossibletocreatesecondaryindexesusingtheedgeattributes_fromand_to,startingwithArangoDB3.0.Acombinedindexoverbothfieldstogetherwiththeuniqueoptionenabledcanbeusedtopreventduplicaterelationsfrombeingcreated.

Forexample,adocumentcollectionvertsmightcontainverticeswiththedocumenthandlesverts/A,verts/Bandverts/C.Relationsbetweenthesedocumentscanbestoredinanedgecollectionedgesforinstance.Now,youmaywanttomakesurethatthevertexverts/Aisneverlinkedtoverts/Bbyanedgemorethanonce.Thiscanbeachievedbyaddingaunique,non-sparsehashindexforthefields_fromand_to:

db.edges.ensureIndex({type:"hash",fields:["_from","_to"],unique:true});

Creatinganedge{_from:"verts/A",_to:"verts/B"}inedgeswillbeaccepted,butonlyonce.AnotherattempttostoreanedgewiththerelationA→Bwillberejectedbytheserverwithauniqueconstraintviolatederror.Thisincludesupdatestothe_fromand_tofields.

NotethataddingarelationB→Aisstillpossible,soisA→AandB→B,becausetheyarealldifferentrelationsinadirectedgraph.Eachonecanonlyoccuroncehowever.

HashIndexes

110

Skiplists

IntroductiontoSkiplistIndexes

ThisisanintroductiontoArangoDB'sskiplists.

Itispossibletodefineaskiplistindexononeormoreattributes(orpaths)ofdocuments.Thisskiplististhenusedinqueriestolocatedocumentswithinagivenrange.Iftheskiplistisdeclaredunique,thennotwodocumentsareallowedtohavethesamesetofattributevalues.

Creatinganewdocumentorupdatingadocumentwillfailiftheuniquenessisviolated.Iftheskiplistindexisdeclaredsparse,adocumentwillbeexcludedfromtheindexandnouniquenesscheckswillbeperformedifanyindexattributevalueisnotsetorhasavalueofnull.

AccessingSkiplistIndexesfromtheShell

UniqueSkiplistIndex

Ensuresthatauniqueskiplistindexexists:collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"],unique:true})

Createsauniqueskiplistindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Alldocumentsinthecollectionmustdifferintermsoftheindexedattributes.Creatinganewdocumentorupdatinganexistingdocumentwillfailiftheattributeuniquenessisviolated.

Tocreateasparseuniqueindex,setthesparseattributetotrue:

collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"],unique:true,sparse:true})

Inasparseindexalldocumentswillbeexcludedfromtheindexthatdonotcontainatleastoneofthespecifiedindexattributesorthathaveavalueofnullinanyofthespecifiedindexattributes.Suchdocumentswillnotbeindexed,andnotbetakenintoaccountforuniquenesschecks.

Inanon-sparseindex,thesedocumentswillbeindexed(fornon-presentindexedattributes,avalueofnullwillbeused)andwillbetakenintoaccountforuniquenesschecks.

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

arangosh>db.ids.ensureIndex({type:"skiplist",fields:["myId"],unique:true});

arangosh>db.ids.save({"myId":123});

arangosh>db.ids.save({"myId":456});

arangosh>db.ids.save({"myId":789});

arangosh>db.ids.save({"myId":123});

showexecutionresults

arangosh>db.ids.ensureIndex({type:"skiplist",fields:["name.first","name.last"],

unique:true});

arangosh>db.ids.save({"name":{"first":"hans","last":"hansen"}});

arangosh>db.ids.save({"name":{"first":"jens","last":"jensen"}});

arangosh>db.ids.save({"name":{"first":"hans","last":"jensen"}});

arangosh>db.ids.save({"name":{"first":"hans","last":"hansen"}});

showexecutionresults

Non-uniqueSkiplistIndex

Skiplists

111

Ensuresthatanon-uniqueskiplistindexexists:collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"]})

Createsanon-uniqueskiplistindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Tocreateasparsenon-uniqueindex,setthesparseattributetotrue.

collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"],sparse:true})

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

arangosh>db.names.ensureIndex({type:"skiplist",fields:["first"]});

arangosh>db.names.save({"first":"Tim"});

arangosh>db.names.save({"first":"Tom"});

arangosh>db.names.save({"first":"John"});

arangosh>db.names.save({"first":"Tim"});

arangosh>db.names.save({"first":"Tom"});

showexecutionresults

SkiplistArrayIndex

Ensuresthataskiplistarrayindexexists(non-unique):collection.ensureIndex({type:"skiplist",fields:["field1[*]",...,"fieldn[*]"]})

Createsanon-uniqueskiplistarrayindexfortheindividualelementsofthearrayattributesfield1[*],...fieldn[*]foundinthedocuments.Atleastoneattributepathhastobegiven.Theindexalwaystreatstheindexedarraysassparse.

Itispossibletocombinearrayindexingwithstandardindexing:collection.ensureIndex({type:"skiplist",fields:["field1[*]","field2"]})

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

arangosh>db.test.ensureIndex({type:"skiplist",fields:["a[*]"]});

arangosh>db.test.save({a:[1,2]});

arangosh>db.test.save({a:[1,3]});

arangosh>db.test.save({a:null});

showexecutionresults

Querybyexampleusingaskiplistindex

Constructsaquery-by-exampleusingaskiplistindex:collection.byExample(example)

Selectsalldocumentsfromthecollectionthatmatchthespecifiedexampleandreturnsacursor.Askiplistindexwillbeusedifpresent.

YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.

Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.Ifyouuse

{"a":{"c":1}}

asexample,thenyouwillfindalldocuments,suchthattheattributeacontainsadocumentoftheform{c:1}.Forexamplethedocument

{"a":{"c":1},"b":1}

willmatch,butthedocument

{"a":{"c":1,"b":1}}

Skiplists

112

willnot.

However,ifyouuse

{"a.c":1},

thenyouwillfindalldocuments,whichcontainasub-documentinathathasanattributecofvalue1.Boththefollowingdocuments

{"a":{"c":1},"b":1}

and

{"a":{"c":1,"b":1}}

willmatch.

Skiplists

113

Persistentindexes

IntroductiontoPersistentIndexes

ThisisanintroductiontoArangoDB'spersistentindexes.

Itispossibletodefineapersistentindexononeormoreattributes(orpaths)ofdocuments.Theindexisthenusedinqueriestolocatedocumentswithinagivenrange.Iftheindexisdeclaredunique,thennotwodocumentsareallowedtohavethesamesetofattributevalues.

Creatinganewdocumentorupdatingadocumentwillfailiftheuniquenessisviolated.Iftheindexisdeclaredsparse,adocumentwillbeexcludedfromtheindexandnouniquenesscheckswillbeperformedifanyindexattributevalueisnotsetorhasavalueofnull.

AccessingPersistentIndexesfromtheShell

ensuresthatauniquepersistentindexexistscollection.ensureIndex({type:"persistent",fields:["field1",...,"fieldn"],unique:true})

Createsauniquepersistentindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Alldocumentsinthecollectionmustdifferintermsoftheindexedattributes.Creatinganewdocumentorupdatinganexistingdocumentwillwillfailiftheattributeuniquenessisviolated.

Tocreateasparseuniqueindex,setthesparseattributetotrue:

collection.ensureIndex({type:"persistent",fields:["field1",...,"fieldn"],unique:true,sparse:true})

Inasparseindexalldocumentswillbeexcludedfromtheindexthatdonotcontainatleastoneofthespecifiedindexattributesorthathaveavalueofnullinanyofthespecifiedindexattributes.Suchdocumentswillnotbeindexed,andnotbetakenintoaccountforuniquenesschecks.

Inanon-sparseindex,thesedocumentswillbeindexed(fornon-presentindexedattributes,avalueofnullwillbeused)andwillbetakenintoaccountforuniquenesschecks.

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

arangosh>db.ids.ensureIndex({type:"persistent",fields:["myId"],unique:true});

arangosh>db.ids.save({"myId":123});

arangosh>db.ids.save({"myId":456});

arangosh>db.ids.save({"myId":789});

arangosh>db.ids.save({"myId":123});

showexecutionresults

arangosh>db.ids.ensureIndex({type:"persistent",fields:["name.first","name.last"],

unique:true});

arangosh>db.ids.save({"name":{"first":"hans","last":"hansen"}});

arangosh>db.ids.save({"name":{"first":"jens","last":"jensen"}});

arangosh>db.ids.save({"name":{"first":"hans","last":"jensen"}});

arangosh>db.ids.save({"name":{"first":"hans","last":"hansen"}});

showexecutionresultsensuresthatanon-uniquepersistentindexexistscollection.ensureIndex({type:"persistent",fields:["field1",...,"fieldn"]})

Createsanon-uniquepersistentindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Persistent

114

Tocreateasparseuniqueindex,setthesparseattributetotrue.

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

arangosh>db.names.ensureIndex({type:"persistent",fields:["first"]});

arangosh>db.names.save({"first":"Tim"});

arangosh>db.names.save({"first":"Tom"});

arangosh>db.names.save({"first":"John"});

arangosh>db.names.save({"first":"Tim"});

arangosh>db.names.save({"first":"Tom"});

showexecutionresults

Querybyexampleusingapersistentindex

constructsaquery-by-exampleusingapersistentindexcollection.byExample(example)

Selectsalldocumentsfromthecollectionthatmatchthespecifiedexampleandreturnsacursor.Apersistentindexwillbeusedifpresent.

YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.

Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.Ifyouuse

{"a":{"c":1}}

asexample,thenyouwillfindalldocuments,suchthattheattributeacontainsadocumentoftheform{c:1}.Forexamplethedocument

{"a":{"c":1},"b":1}

willmatch,butthedocument

{"a":{"c":1,"b":1}}

willnot.

However,ifyouuse

{"a.c":1},

thenyouwillfindalldocuments,whichcontainasub-documentinathathasanattributecofvalue1.Boththefollowingdocuments

{"a":{"c":1},"b":1}

and

{"a":{"c":1,"b":1}}

willmatch.

PersistentIndexesandServerLanguageTheorderofindexentriesinpersistentindexesadherestotheconfiguredserverlanguage.If,however,theserverisrestartedwithadifferentlanguagesettingaswhenthepersistentindexwascreated,notalldocumentsmaybereturnedanymoreandthesortorderofthosewhicharereturnedcanbewrong(wheneverthepersistentindexisconsulted).

Tofixpersistentindexesafteralanguagechange,deleteandre-createthem.Skiplistindexesarenotaffected,becausetheyarenotpersistedandautomaticallyrebuiltoneveryserverstart.

Persistent

115

Persistent

116

FulltextindexesThisisanintroductiontoArangoDB'sfulltextindexes.

IntroductiontoFulltextIndexes

Afulltextindexcanbeusedtofindwords,orprefixesofwordsinsidedocuments.

Afulltextindexcanbedefinedononeattributeonly,andwillincludeallwordscontainedindocumentsthathaveatextualvalueintheindexattribute.SinceArangoDB2.6theindexwillalsoincludewordsfromtheindexattributeiftheindexattributeisanarrayofstrings,oranobjectwithstringvaluemembers.

Forexample,givenafulltextindexonthetranslationsattributeandthefollowingdocuments,thensearchingforлисаusingthefulltextindexwouldreturnonlythefirstdocument.SearchingfortheindexfortheexactstringFoxwouldreturnthefirsttwodocuments,andsearchingforprefix:Foxwouldreturnallthreedocuments:

{translations:{en:"fox",de:"Fuchs",fr:"renard",ru:"лиса"}}

{translations:"FoxistheEnglishtranslationoftheGermanwordFuchs"}

{translations:["ArangoDB","document","database","Foxx"]}

Notethatdeepernestedobjectsareignored.Forexample,afulltextindexontranslationswouldindexFuchs,butnotfox,giventhefollowingdocumentstructure:

{translations:{en:{US:"fox"},de:"Fuchs"}

Ifyouneedtosearchacrossmultiplefieldsand/ornestedobjects,youmaywriteallthestringsintoaspecialattribute,whichyouthencreatetheindexon(itmightbenecessarytocleanthestringsfirst,e.g.removelinebreaksandstripcertainwords).

Iftheindexattributeisneitherastring,anobjectoranarray,itscontentswillnotbeindexed.Whenindexingthecontentsofanarrayattribute,anarraymemberwillonlybeincludedintheindexifitisastring.Whenindexingthecontentsofanobjectattribute,anobjectmembervaluewillonlybeincludedintheindexifitisastring.Otherdatatypesareignoredandnotindexed.

Currently,fulltextindexesarenotyetsupportedwiththeRocksDBstorageengine.ThusthefunctionFULLTEXT()willbeunavailablewhenusingthisstorageengine.Tousefulltextindexes,pleaseusetheMMFilesstorageengineforthetimebeing.

AccessingFulltextIndexesfromtheShell

Ensuresthatafulltextindexexists:

collection.ensureIndex({type:"fulltext",fields:["field"],minLength:minLength})

Createsafulltextindexonalldocumentsonattributefield.

Fulltextindexesareimplicitlysparse:alldocumentswhichdonothavethespecifiedfieldattributeorthathaveanon-qualifyingvalueintheirfieldattributewillbeignoredforindexing.

Onlyasingleattributecanbeindexed.Specifyingmultipleattributesisunsupported.

TheminimumlengthofwordsthatareindexedcanbespecifiedviatheminLengthparameter.WordsshorterthanminLengthcharacterswillnotbeindexed.minLengthhasadefaultvalueof2,butthisvaluemightbechangedinfutureversionsofArangoDB.Itisthusrecommendedtoexplicitlyspecifythisvalue.

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetailsisreturned.

arangosh>db.example.ensureIndex({type:"fulltext",fields:["text"],minLength:3});

arangosh>db.example.save({text:"thequickbrown",b:{c:1}});

arangosh>db.example.save({text:"quickbrownfox",b:{c:2}});

arangosh>db.example.save({text:"brownfoxjums",b:{c:3}});

arangosh>db.example.save({text:"foxjumpsover",b:{c:4}});

FulltextIndexes

117

arangosh>db.example.save({text:"jumpsoverthe",b:{c:5}});

arangosh>db.example.save({text:"overthelazy",b:{c:6}});

arangosh>db.example.save({text:"thelazydog",b:{c:7}});

arangosh>db._query("FORdocumentINFULLTEXT(example,'text','the')RETURNdocument");

showexecutionresultsLooksupafulltextindex:

collection.lookupFulltextIndex(attribute,minLength)

Checkswhetherafulltextindexonthegivenattributeattributeexists.

FulltextAQLFunctions

FulltextAQLfunctionsaredetailedinFulltextfunctions.

FulltextIndexes

118

GeoIndexes

IntroductiontoGeoIndexes

ThisisanintroductiontoArangoDB'sgeoindexes.

AQL'sgeographicfeaturesaredescribedinGeofunctions.

ArangoDBusesHilbertcurvestoimplementgeo-spatialindexes.Seethisblogfordetails.

Ageo-spatialindexassumesthatthelatitudeisbetween-90and90degreeandthelongitudeisbetween-180and180degree.Ageoindexwillignorealldocumentswhichdonotfulfilltheserequirements.

AccessingGeoIndexesfromtheShell

ensuresthatageoindexexistscollection.ensureIndex({type:"geo",fields:["location"]})

Createsageo-spatialindexonalldocumentsusinglocationaspathtothecoordinates.Thevalueoftheattributehastobeanarraywithatleasttwonumericvalues.Thearraymustcontainthelatitude(firstvalue)andthelongitude(secondvalue).

Alldocuments,whichdonothavetheattributepathorhaveanon-conformingvalueinitareexcludedfromtheindex.

Ageoindexisimplicitlysparse,andthereisnowaytocontrolitssparsity.

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

Tocreateageoindexonanarrayattributethatcontainslongitudefirst,setthegeoJsonattributetotrue.ThiscorrespondstotheformatdescribedinRFC7946Position

collection.ensureIndex({type:"geo",fields:["location"],geoJson:true})

Tocreateageo-spatialindexonalldocumentsusinglatitudeandlongitudeasseparateattributepaths,twopathsneedtobespecifiedinthefieldsarray:

collection.ensureIndex({type:"geo",fields:["latitude","longitude"]})

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

Examples

Createageoindexforanarrayattribute:

arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});

arangosh>for(i=-90;i<=90;i+=10){

........>for(j=-180;j<=180;j+=10){

........>db.geo.save({name:"Name/"+i+"/"+j,loc:[i,j]});

........>}

........>}

arangosh>db.geo.count();

arangosh>db.geo.near(0,0).limit(3).toArray();

arangosh>db.geo.near(0,0).count();

showexecutionresultsCreateageoindexforahasharrayattribute:

arangosh>db.geo2.ensureIndex({type:"geo",fields:["location.latitude",

"location.longitude"]});

arangosh>for(i=-90;i<=90;i+=10){

........>for(j=-180;j<=180;j+=10){

........>db.geo2.save({name:"Name/"+i+"/"+j,location:{latitude:i,

longitude:j}});

GeoIndexes

119

........>}

........>}

arangosh>db.geo2.near(0,0).limit(3).toArray();

showexecutionresultsUseGeoIndexwithAQLSORTstatement:

arangosh>db.geoSort.ensureIndex({type:"geo",fields:["latitude","longitude"]});

arangosh>for(i=-90;i<=90;i+=10){

........>for(j=-180;j<=180;j+=10){

........>db.geoSort.save({name:"Name/"+i+"/"+j,latitude:i,longitude

:j});

........>}

........>}

arangosh>varquery="FORdocingeoSortSORTDISTANCE(doc.latitude,doc.longitude,0,0)

LIMIT5RETURNdoc"

arangosh>db._explain(query,{},{colors:false});

arangosh>db._query(query);

showexecutionresultsUseGeoIndexwithAQLFILTERstatement:

arangosh>db.geoFilter.ensureIndex({type:"geo",fields:["latitude","longitude"]});

arangosh>for(i=-90;i<=90;i+=10){

........>for(j=-180;j<=180;j+=10){

........>db.geoFilter.save({name:"Name/"+i+"/"+j,latitude:i,

longitude:j});

........>}

........>}

arangosh>varquery="FORdocingeoFilterFILTERDISTANCE(doc.latitude,doc.longitude,

0,0)<2000RETURNdoc"

arangosh>db._explain(query,{},{colors:false});

arangosh>db._query(query);

showexecutionresultsconstructsageoindexselectioncollection.geo(location-attribute)Looksupageoindexdefinedonattributelocation_attribute.Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.Thisisusefulforcollectionswithmultipledefinedgeoindexes.collection.geo(location_attribute,true)Looksupageoindexonacompoundattributelocation_attribute.Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.collection.geo(latitude_attribute,longitude_attribute)Looksupageoindexdefinedonthetwoattributeslatitude_attributeandlongitude-attribute.Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.Note:thegeosimplequeryhelperfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforrunninggeoqueriesistousetheirAQLequivalents.

Examples

Assumeyouhavealocationstoredaslistintheattributehomeandadestinationstoredintheattributework.Thenyoucanusethegeooperatortoselectwhichgeo-spatialattributes(andthuswhichindex)touseinanearquery.

arangosh>for(i=-90;i<=90;i+=10){

........>for(j=-180;j<=180;j+=10){

........>db.complex.save({name:"Name/"+i+"/"+j,

........>home:[i,j],

........>work:[-i,-j]});

........>}

GeoIndexes

120

........>}

........>

arangosh>db.complex.near(0,170).limit(5);

arangosh>db.complex.ensureIndex({type:"geo",fields:["home"]});

arangosh>db.complex.near(0,170).limit(5).toArray();

arangosh>db.complex.geo("work").near(0,170).limit(5);

arangosh>db.complex.ensureIndex({type:"geo",fields:["work"]});

arangosh>db.complex.geo("work").near(0,170).limit(5).toArray();

showexecutionresultsconstructsanearqueryforacollectioncollection.near(latitude,longitude)Thereturnedlistissortedaccordingtothedistance,withthenearestdocumenttothecoordinate(latitude,longitude)comingfirst.Ifthereareneardocumentsofequaldistance,documentsarechosenrandomlyfromthissetuntilthelimitisreached.Itispossibletochangethelimitusingthelimitoperator.Inordertousethenearoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.Note:neardoesnotsupportnegativeskips.//However,youcanstilluselimitfollowedtoskip.collection.near(latitude,longitude).limit(limit)Limitstheresulttolimitdocumentsinsteadofthedefault100.Note:Unlikewithmultipleexplicitlimits,limitwillraisetheimplicitdefaultlimitimposedbywithin.collection.near(latitude,longitude).distance()Thiswilladdanattributedistancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.collection.near(latitude,longitude).distance(name)Thiswilladdanattributenametoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.Note:thenearsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthenearoperatoristousetheAQLNEARfunctioninanAQLqueryasfollows:

FORdocINNEAR(@@collection,@latitude,@longitude,@limit)

RETURNdoc

Examples

Togetthenearesttwolocations:

arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});

arangosh>for(vari=-90;i<=90;i+=10){

........>for(varj=-180;j<=180;j+=10){

........>db.geo.save({

........>name:"Name/"+i+"/"+j,

........>loc:[i,j]});

........>}}

arangosh>db.geo.near(0,0).limit(2).toArray();

showexecutionresultsIfyouneedthedistanceaswell,thenyoucanusethedistanceoperator:

arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});

arangosh>for(vari=-90;i<=90;i+=10){

........>for(varj=-180;j<=180;j+=10){

........>db.geo.save({

........>name:"Name/"+i+"/"+j,

........>loc:[i,j]});

........>}}

arangosh>db.geo.near(0,0).distance().limit(2).toArray();

showexecutionresultsconstructsawithinqueryforacollectioncollection.within(latitude,longitude,radius)Thiswillfindalldocumentswithinagivenradiusaroundthecoordinate(latitude,longitude).Thereturnedarrayissortedbydistance,beginningwiththenearestdocument.Inordertousethewithinoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesfor

GeoIndexes

121

thedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.collection.within(latitude,longitude,radius).distance()Thiswilladdanattribute_distancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.collection.within(latitude,longitude,radius).distance(name)Thiswilladdanattributenametoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.Note:thewithinsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthewithinoperatoristousetheAQLWITHINfunctioninanAQLqueryasfollows:

FORdocINWITHIN(@@collection,@latitude,@longitude,@radius,@distanceAttributeName)

RETURNdoc

Examples

Tofindalldocumentswithinaradiusof2000kmuse:

arangosh>for(vari=-90;i<=90;i+=10){

........>for(varj=-180;j<=180;j+=10){

........>db.geo.save({name:"Name/"+i+"/"+j,loc:[i,j]});}}

arangosh>db.geo.within(0,0,2000*1000).distance().toArray();

showexecutionresultsensuresthatageoindexexistscollection.ensureIndex({type:"geo",fields:["location"]})

SinceArangoDB2.5,thismethodisanaliasforensureGeoIndexsincegeoindexesarealwayssparse,meaningthatdocumentsthatdonotcontaintheindexattributesorhavenon-numericvaluesintheindexattributeswillnotbeindexed.ensureGeoConstraintisdeprecatedandensureGeoIndexshouldbeusedinstead.

Theindexdoesnotprovideauniqueoptionbecauseofitslimitedusability.Itwouldpreventidenticalcoordinatesfrombeinginsertedonly,butevenaslightlydifferentlocation(like1inchor1cmoff)wouldbeuniqueagainandnotconsideredaduplicate,althoughitprobablyshould.Thedesiredthresholdfordetectingduplicatesmayvaryforeveryproject(includinghowtocalculatethedistanceeven)andneedstobeimplementedontheapplicationlayerasneeded.YoucanwriteaFoxxserviceforthispurposeandmakeuseoftheAQLgeofunctionstofindnearbycoordinatessupportedbyageoindex.

GeoIndexes

122

VertexCentricIndexes

IntroductiontoVertexCentricIndexes

InArangoDBtherearespecialindicesdesignedtospeedupgraphoperations,especiallyifthegraphcontainssupernodes(verticesthathaveanexceptionallyhighamountofconnectededges).Theseindicesarecalledvertexcentricindexesandcanbeusedinadditiontotheexistingedgeindex.

Motivation

Theideaofthisindexistoindexacombinationofavertex,thedirectionandanyarbitrarysetofotherattributesontheedges.Totakeanexample,ifwehaveanattributecalledtypeontheedges,wecanuseanoutboundvertex-centricindexonthisattributetofindalledgesattachedtoavertexwithagiventype.Thefollowingqueryexamplecouldbenefitfromsuchanindex:

FORv,e,pIN3..5OUTBOUND@startGRAPH@graphName

FILTERp.edges[*].typeALL=="friend"

RETURNv

Usingthebuilt-inedge-indexArangoDBcanfindthelistofalledgesattachedtothevertexfast,butstillithastowalkthroughthislistandcheckifallofthemhavetheattributetype=="friend".Usingavertex-centricindexwouldallowArangoDBtofindalledgesforthevertexhavingtheattributetype=="friend"inthesametimeandcansavetheiterationtoverifythecondition.

IndexcreationAvertex-centriccanbeeitherofthefollowingtypes:

HashIndexSkiplistIndexPersistentIndex

Andiscreatedusingtheircreationoperations.Howeverinthelistoffieldsusedtocreatetheindexwehavetoincludeeither_fromor_to.Letusagainexplainthisbyanexample.Assumewewanttocreateanhash-basedoutboundvertex-centricindexontheattributetype.Thiscanbecreatedwiththefollowingway:

arangosh>db.collection.ensureIndex({type:"hash",fields:["_from","type"]})

showexecutionresultsAlloptionsthataresupportedbytherespectiveindexesaresupportedbythevertex-centricindexaswell.

Indexusage

TheAQLoptimizercandecidetouseavertex-centricwheneversuitable,howeveritisnotguaranteedthatthisindexisused,theoptimizermayestimatethatanotherindexisassumedtobebetter.Theoptimizerwillconsiderthistypeofindexesonexplicitfilteringof_fromrespectively_to:

FORedgeINcollection

FILTERedge._from=="vertices/123456"ANDedge.type=="friend"

RETURNedge

andduringpatternmatchingqueries:

FORv,e,pIN3..5OUTBOUND@startGRAPH@graphName

FILTERp.edges[*].typeALL=="friend"

RETURNv

VertexCentricIndexes

123

VertexCentricIndexes

124

ArangoDBGraphs

FirstStepswithGraphs

AGraphconsistsofverticesandedges.Edgesarestoredasdocumentsinedgecollections.Avertexcanbeadocumentofadocumentcollectionorofanedgecollection(soedgescanbeusedasvertices).Whichcollectionsareusedwithinanamedgraphisdefinedviaedgedefinitions.Anamedgraphcancontainmorethanoneedgedefinition,atleastoneisneeded.Graphsallowyoutostructureyourmodelsinlinewithyourdomainandgroupthemlogicallyincollectionsandgivingyouthepowertoquerytheminthesamegraphqueries.

Newtographs?Takeourfreegraphcourseforfreshersandgetfromzeroknowledgetoadvancedquerytechniques.

Comingfromarelationalbackground-what'sagraph?

InSQLyoucommonlyhavetheconstructofarelationtabletostoren:mrelationsbetweentwodatatables.Anedgecollectionissomewhatsimilartotheserelationtables;vertexcollectionsresemblethedatatableswiththeobjectstoconnect.WhilesimplegraphquerieswithfixednumberofhopsviatherelationtablemaybedoableinSQLwithseveralnestedjoins,graphdatabasescanhandleanarbitrarynumberofthesehopsoveredgecollections-thisiscalledtraversal.Alsoedgesinoneedgecollectionmaypointtoseveralvertexcollections.Itscommontohaveattributesattachedtoedges,i.e.alabelnamingthisinterconnection.Edgeshaveadirection,withtheirrelations_fromand_topointingfromonedocumenttoanotherdocumentstoredinvertexcollections.Inqueriesyoucandefineinwhichdirectionstheedgerelationsmaybefollowed(OUTBOUND:_from→_to,INBOUND:_from←_to,ANY:_from↔_to).

NamedGraphs

NamedgraphsarecompletelymanagedbyArangoDB,andthusalsovisibleinthewebinterface.TheyusethefullspectrumofArangoDB'sgraphfeatures.Youmayaccessthemviaseveralinterfaces.

AQLGraphOperationswithseveralflavors:AQLTraversalsonbothnamedandanonymousgraphsAQLShortestPathonbothnamedandanonymousgraph

JavaScriptGeneralGraphimplementation,asyoumayuseitinFoxxServicesGraphManagement;creating&manipualatinggraphdefinitions;inserting,updatinganddeletingverticesandedgesintographsGraphFunctionsforworkingwithedgesandvertices,toanalyzethemandtheirrelations

JavaScriptSmartGraphimplementation,forscalablegraphsSmartGraphManagement;creating&manipualatingSmartGraphdefinitions;DifferencestoGeneralGraph

RESTfulGeneralGraphinterfaceusedtoimplementgraphmanagementinclientdrivers

Manipulatingcollectionsofnamedgraphswithregulardocumentfunctions

Theunderlyingcollectionsofthenamedgraphsarestillaccessibleusingthestandardmethodsforcollections.Howeverthegraphmoduleaddsanadditionallayerontopofthesecollectionsgivingyouthefollowingguarantees:

AllmodificationsareexecutedtransactionalIfyoudeleteavertexalledgeswillbedeleted,youwillneverhavelooseendsIfyouinsertanedgeitischeckediftheedgematchestheedgedefinitions,youredgecollectionswillonlycontainvalidedges

TheseguaranteesarelostifyouaccessthecollectionsinanyotherwaythanthegraphmoduleorAQL,soifyoudeletedocumentsfromyourvertexcollectionsdirectly,theedgespointingtothemwillberemaininplace.

Anonymousgraphs

Sometimesyoumaynotneedallthepowersofnamedgraphs,butsomeofitsbitsmaybevaluabletoyou.YoumayuseanonymousgraphsinthetraversalsandintheWorkingwithEdgeschapter.Anonymousgraphsdon'thaveedgedefinitionsdescribingwhichvertexcollectionisconnectedbywhichedgecollection.Thegraphmodelhastobemaintainedintheclientsidecode.Thisgivesyoumorefreedomthanthestrictnamedgraphs.

Graphs

125

AQLGraphOperationsareavailableforboth,namedandanonymousgraphs:AQLTraversalsAQLShortestPath

Whentochooseanonymousornamedgraphs?

Asnotedabove,namedgraphsensuregraphintegrity,bothwheninsertingorremovingedgesorvertices.Soyouwon'tencounterdanglingedges,evenifyouusethesamevertexcollectioninseveralnamedgraphs.Thisinvolvesmoreoperationsinsidethedatabasewhichcomeatacost.Thereforeanonymousgraphsmaybefasterinmanyoperations.Sothisquestionmaybenarroweddownto:'CanIaffordtheadditionaleffortordoIneedthewarrantyforintegrity?'.

Multipleedgecollectionsvs.FILTERsonedgedocumentattributes

Ifyouwanttoonlytraverseedgesofaspecifictype,therearetwowaystoachievethis.Thefirstwouldbeanattributeintheedgedocument-i.e.type,whereyouspecifyadifferentiatorfortheedge-i.e."friends","family","married"or"workmates",soyoucanlaterFILTERe.type="friends"ifyouonlywanttofollowthefriendedges.

Anotherway,whichmaybemoreefficientinsomecases,istousedifferentedgecollectionsfordifferenttypesofedges,soyouhavefriend_edges,family_edges,married_edgesandworkmate_edgesascollectionnames.Youcanthenconfigureseveralnamedgraphsincludingasubsetoftheavailableedgeandvertexcollections-oryouuseanonymousgraphqueries,whereyouspecifyalistofedgecollectionstotakeintoaccountinthatquery.Toonlyfollowfriendedges,youwouldspecifyfriend_edgesassoleedgecollection.

Bothapproacheshaveadvantagesanddisadvantages.FILTERoperationsonedeattributeswilldocomparisonsoneachtraversededge,whichmaybecomeCPU-intense.Whennotfindingtheedgesinthefirstplacebecauseofthecollectioncontainingthemisnottraversedatall,therewillneverbeareasontoactualycheckfortheirtypeattributewithFILTER.

Themultipleedgecollectionsapproachislimitedbythenumberofcollectionsthatcanbeusedsimultaneouslyinonequery.EverycollectionusedinaqueryrequiressomeresourcesinsideofArangoDBandthenumberisthereforelimitedtocaptheresourcerequirements.Youmayalsohaveconstraintsonotheredgeattributes,suchasahashindexwithauniqueconstraint,whichrequiresthedocumentstobeinasinglecollectionfortheuniquenessguarantee,anditmaythusnotbepossibletostorethedifferenttypesofedgesinmultipleedegcollections.

So,ifyouredgeshaveaboutadozendifferenttypes,it'sokaytochoosethecollectionapproach,otherwisetheFILTERapproachispreferred.YoucanstilluseFILTERoperationsonedgesofcourse.YoucangetridofaFILTERonthetypewiththeformerapproach,everythingelsecanstaythesame.

WhichpartofmydataisanEdgeandwhichaVertex?

Themainobjectsinyourdatamodel,suchasusers,groupsorarticles,areusuallyconsideredtobevertices.Foreachtypeofobject,adocumentcollection(alsocalledvertexcollection)shouldstoretheindividualentities.Entitiescanbeconnectedbyedgestoexpressandclassifyrelationsbetweenvertices.Itoftenmakessensetohaveanedgecollectionperrelationtype.

ArangoDBdoesnotrequireyoutostoreyourdataingraphstructureswithedgesandvertices,youcanalsodecidetoembedattributessuchaswhichgroupsauserispartof,or_idsofdocumentsinanotherdocumentinsteadofconnectingthedocumentswithedges.Itcanbeameaningfulperformanceoptimizationfor1:nrelationships,ifyourdataisnotfocusedonrelationsandyoudon'tneedgraphtraversalwithvaryingdepth.Itusuallymeanstointroducesomeredundancyandpossiblyinconsistenciesifyouembeddata,butitcanbeanacceptabletradeoff.

Vertices

Let'ssaywehavetwovertexcollections,UsersandGroups.DocumentsintheGroupscollectioncontaintheattributesoftheGroup,i.e.whenitwasfounded,itssubject,aniconURLandsoon.Usersdocumentscontainthedataspecifictoauser-likeallnames,birthdays,AvatarURLs,hobbies...

Edges

Wecanuseanedgecollectiontostorerelationsbetweenusersandgroups.Sincemultipleusersmaybeinanarbitrarynumberofgroups,thisisanm:nrelation.TheedgecollectioncanbecalledUsersInGroupswithi.e.oneedgewith_frompointingtoUsers/Johnand_topointingtoGroups/BowlingGroupHappyPin.ThismakestheuserJohnamemberofthegroupBowlingGroupHappyPin.

Graphs

126

Attributesofthisrelationmaycontainqualifierstothisrelation,likethepermissionsofJohninthisgroup,thedatewhenhejoinedthegroupetc.

Soroughlyput,ifyouusedocumentsandtheirattributesinasentence,nounswouldtypicallybevertices,verbsbecometheedges.Youcanseethisintheknowsgraphbelow:

AliceknowsBob,whointermknowsCharlie.

Advantagesofthisapproach

Graphsgiveyoutheadvantageofnotjustbeingabletohaveafixednumberofm:nrelationsinarow,butanarbitrarynumber.Edgescanbetraversedinbothdirections,soit'seasytodetermineallgroupsauserisin,butalsotofindoutwhichmembersacertaingrouphas.Userscouldalsobeinterconnectedtocreateasocialnetwork.

Usingthegraphdatamodel,dealingwithdatathathaslotsofrelationsstaysmanageableandcanbequeriedinveryflexibleways,whereasitwouldcauseheadachetohandleitinarelationaldatabasesystem.

Backupandrestore

Forsureyouwanttohavebackupsofyourgraphdata,youcanuseArangodumptocreatethebackup,andArangorestoretorestoreabackupintoanewArangoDB.Youshouldhowevernotethat:

youneedthesystemcollection_graphsifyoubackupnamedgraphs.youneedtobackupthecompletesetofalledgeandvertexcollectionsyourgraphconsistsof.Partialdump/restoremaynotwork.

Managinggraphs

Bydefaultyoushouldusetheinterfaceyourdriverprovidestomanagegraphs.

Thisisi.e.documentedinGraphs-SectionoftheArangoDBJavadriver.

ExampleGraphs

ArangoDBcomeswithasetofeasilygraspablegraphsthatareusedtodemonstratetheAPIs.Youcanusetheaddsamplestabinthecreategraphwindowinthewebinterface,orloadthemodule@arangodb/graph-examples/example-graphinarangoshanduseittocreateinstancesofthesegraphsinyourArangoDB.Onceyou'vecreatedthem,youcaninspecttheminthewebinterface-whichwasusedtocreatethepicturesbelow.

Youcaneasilylookintotheinnardsofthisscriptforreferenceabouthowtomanagegraphsprogramatically.

TheKnows_Graph

Graphs

127

Asetofpersonsknowingeachother:

Theknowsgraphconsistsofonevertexcollectionpersonsconnectedviaoneedgecollectionknows.ItwillcontainfivepersonsAlice,Bob,Charlie,DaveandEve.Wewillhavethefollowingdirectedrelations:

AliceknowsBobBobknowsCharlieBobknowsDaveEveknowsAliceEveknowsBob

Thisishowwecreateit,inspectitsverticesandedges,anddropitagain:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("knows_graph");

arangosh>db.persons.toArray()

arangosh>db.knows.toArray();

arangosh>examples.dropGraph("knows_graph");

showexecutionresults

TheSocialGraph

Asetofpersonsandtheirrelations:

Graphs

128

Thisexamplehasfemaleandmalepersonsasverticesintwovertexcollections-femaleandmale.Theedgesaretheirconnectionsintherelationedgecollection.Thisishowwecreateit,inspectitsverticesandedges,anddropitagain:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>db.female.toArray()

arangosh>db.male.toArray()

arangosh>db.relation.toArray()

arangosh>examples.dropGraph("social");

showexecutionresults

TheCityGraph

Asetofeuropeancities,andtheirfictionaltravelingdistancesasconnections:

Graphs

129

Theexamplehasthecitiesasverticesinseveralvertexcollections-germanCityandfrenchCity.Theedgesaretheirinterconnectionsinseveraledgecollectionsfrench/german/internationalHighway.Thisishowwecreateit,inspectitsedgesandvertices,anddropitagain:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("routeplanner");

arangosh>db.frenchCity.toArray();

arangosh>db.germanCity.toArray();

arangosh>db.germanHighway.toArray();

arangosh>db.frenchHighway.toArray();

arangosh>db.internationalHighway.toArray();

arangosh>examples.dropGraph("routeplanner");

showexecutionresults

TheTraversalGraph

Thisgraphwasdesignedtodemonstratefiltersintraversals.Ithassomelabelstofilteronit.

Graphs

130

Theexamplehasallitsverticesinthecirclescollection,andanedgesedgecollectiontoconnectthem.Circleshaveuniquenumericlabels.Edgeshavetwobooleanattributes(theFalsealwaysbeingfalse,theTruthalwaysbeingtrue)andalabelsortingB-Dtotheleftside,G-Ktotherightside.LeftandrightsidesplitintoPaths-atBandGwhichareeachdirectneighboursoftheroot-nodeA.StartingfromAthegraphhasadepthof3onallitspaths.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("traversalGraph");

arangosh>db.circles.toArray();

arangosh>db.edges.toArray();

arangosh>examples.dropGraph("traversalGraph");

showexecutionresults

TheWorldGraph

Theworldcountrygraphstructuresitsnodeslikethat:world→continent→country→capital.Insomecasesedgedirectionsaren'tforward(thereforeitwillbedisplayeddisjunctinthegraphviewer).Ithastwowaysofcreatingit.Oneusingthenamedgraphutilities(worldCountry),onewithout(worldCountryUnManaged).Itisusedtodemonstraterawtraversaloperations.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("worldCountry");

arangosh>db.worldVertices.toArray();

arangosh>db.worldEdges.toArray();

arangosh>examples.dropGraph("worldCountry");

arangosh>varg=examples.loadGraph("worldCountryUnManaged");

arangosh>examples.dropGraph("worldCountryUnManaged");

showexecutionresults

Graphs

131

Cookbookexamples

TheabovereferencedchaptersdescribethevariousAPIsofArangoDBsgraphenginewithsmallexamples.Ourcookbookhassomemorereallifeexamples:

TraversingagraphinfulldepthUsinganexamplevertexwiththejavadriverRetrievingdocumentsfromArangoDBwithoutknowingthestructureUsingacustomvisitorfromnode.jsAQLExampleQueriesonanActorsandMoviesDatabase

Highervolumegraphexamples

Alloftheaboveexamplesarerathersmallsotheyareeasiertocomprehendandcandemonstratethewaythefunctionalityworks.Therearehoweverseveraldatasetsfreelyavailableonthewebthatarealotbigger.Wecollectedsomeofthemwithimportscriptssoyoumayplayaroundwiththem.AnotherhugegraphisthePokecsocialnetworkfromSlovakiathatweusedforperformancetestingonseveraldatabases;Youwillfindimportingscriptsetc.inthisblogpost.

Graphs

132

GraphsThischapterdescribesthegeneral-graphmodule.Itallowsyoutodefineagraphthatisspreadacrossseveraledgeanddocumentcollections.Thisallowsyoutostructureyourmodelsinlinewithyourdomainandgroupthemlogicallyincollectionsgivingyouthepowertoquerytheminthesamegraphqueries.Thereisnoneedtoincludethereferencedcollectionswithinthequery,thismodulewillhandleitforyou.

ThreeStepstocreateagraph

Createagraph

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>vargraph=graph_module._create("myGraph");

arangosh>graph;

{[Graph]

}

Addsomevertexcollections

arangosh>graph._addVertexCollection("shop");

arangosh>graph._addVertexCollection("customer");

arangosh>graph._addVertexCollection("pet");

arangosh>graph;

showexecutionresultsDefinerelationsontheGraph

arangosh>varrel=graph_module._relation("isCustomer",["shop"],["customer"]);

arangosh>graph._extendEdgeDefinitions(rel);

arangosh>graph;

showexecutionresults

GeneralGraphs

133

GraphManagementThischapterdescribesthejavascriptinterfaceforcreatingandmodifyingnamedgraphs.Inordertocreateanonemptygraphthefunctionalitytocreateedgedefinitionshastobeintroducedfirst:

EdgeDefinitions

Anedgedefinitionisalwaysadirectedrelationofagraph.Eachgraphcanhavearbitrarymanyrelationsdefinedwithintheedgedefinitionsarray.

Initializethelist

Createalistofedgedefinitionstoconstructagraph.

graph_module._edgeDefinitions(relation1,relation2,...,relationN)

Thelistofedgedefinitionsofagraphcanbemanagedbythegraphmoduleitself.Thisfunctionistheentrypointforthemanagementandwillreturnthecorrectlist.

Parameters

relationX(optional)Anobjectrepresentingadefinitionofonerelationinthegraph

Examples

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>directed_relation=graph_module._relation("lives_in","user","city");

arangosh>undirected_relation=graph_module._relation("knows","user","user");

arangosh>edgedefinitions=graph_module._edgeDefinitions(directed_relation,

undirected_relation);

showexecutionresults

Extendthelist

Extendthelistofedgedefinitionstoconstructagraph.

graph_module._extendEdgeDefinitions(edgeDefinitions,relation1,relation2,...,relationN)

Inordertoaddmoreedgedefinitionstothegraphbeforecreatingthisfunctioncanbeusedtoaddmoredefinitionstotheinitiallist.

Parameters

edgeDefinitions(required)Alistofrelationdefinitionobjects.relationX(required)Anobjectrepresentingadefinitionofonerelationinthegraph

Examples

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>directed_relation=graph_module._relation("lives_in","user","city");

arangosh>undirected_relation=graph_module._relation("knows","user","user");

arangosh>edgedefinitions=graph_module._edgeDefinitions(directed_relation);

arangosh>edgedefinitions=graph_module._extendEdgeDefinitions(undirected_relation);

showexecutionresults

Relation

Defineadirectedrelation.

GraphManagement

134

graph_module._relation(relationName,fromVertexCollections,toVertexCollections)

TherelationNamedefinesthenameofthisrelationandreferencestotheunderlyingedgecollection.ThefromVertexCollectionsisanArrayofdocumentcollectionsholdingthestartvertices.ThetoVertexCollectionsisanArrayofdocumentcollectionsholdingthetargetvertices.RelationsareonlyallowedinthedirectionfromanycollectioninfromVertexCollectionstoanycollectionintoVertexCollections.

Parameters

relationName(required)Thenameoftheedgecollectionwheretheedgesshouldbestored.Willbecreatedifitdoesnotyetexist.fromVertexCollections(required)Oneoralistofcollectionnames.Sourceverticesfortheedgeshavetobestoredinthesecollections.Collectionswillbecreatediftheydonotexist.toVertexCollections(required)Oneoralistofcollectionnames.Targetverticesfortheedgeshavetobestoredinthesecollections.Collectionswillbecreatediftheydonotexist.

Examples

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>graph_module._relation("has_bought",["Customer","Company"],["Groceries",

"Electronics"]);

showexecutionresults

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>graph_module._relation("has_bought","Customer","Product");

showexecutionresults

CreateagraphAfterhavingintroducededgedefinitionsagraphcanbecreated.

Createagraph

graph_module._create(graphName,edgeDefinitions,orphanCollections)

Thecreationofagraphrequiresthenameofthegraphandadefinitionofitsedges.

Foreverytypeofedgedefinitionaconveniencemethodexiststhatcanbeusedtocreateagraph.Optionallyalistofvertexcollectionscanbeadded,whicharenotusedinanyedgedefinition.Thesecollectionsarereferredtoasorphancollectionswithinthischapter.Allcollectionsusedwithinthecreationprocessarecreatediftheydonotexist.

Parameters

graphName(required)UniqueidentifierofthegraphedgeDefinitions(optional)ListofrelationdefinitionobjectsorphanCollections(optional)Listofadditionalvertexcollectionnames

Examples

Createanemptygraph,edgedefinitionscanbeaddedatruntime:

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>graph=graph_module._create("myGraph");

{[Graph]

}

Createagraphusinganedgecollectionedgesandasinglevertexcollectionvertices

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>varedgeDefinitions=[{collection:"edges","from":["vertices"],"to":[

"vertices"]}];

GraphManagement

135

arangosh>graph=graph_module._create("myGraph",edgeDefinitions);

{[Graph]

"edges":[ArangoCollection18560,"edges"(typeedge,statusloaded)],

"vertices":[ArangoCollection18555,"vertices"(typedocument,statusloaded)]

}

Createagraphwithedgedefinitionsandorphancollections:

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>graph=graph_module._create("myGraph",

........>[graph_module._relation("myRelation",["male","female"],["male","female"])],

["sessions"]);

showexecutionresults

CompleteExampletocreateagraph

ExampleCall:

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>varedgeDefinitions=graph_module._edgeDefinitions();

arangosh>graph_module._extendEdgeDefinitions(edgeDefinitions,

graph_module._relation("friend_of","Customer","Customer"));

arangosh>graph_module._extendEdgeDefinitions(

........>edgeDefinitions,graph_module._relation(

........>"has_bought",["Customer","Company"],["Groceries","Electronics"]));

arangosh>graph_module._create("myStore",edgeDefinitions);

showexecutionresultsalternativecall:

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>varedgeDefinitions=graph_module._edgeDefinitions(

........>graph_module._relation("friend_of",["Customer"],["Customer"]),

graph_module._relation(

........>"has_bought",["Customer","Company"],["Groceries","Electronics"]));

arangosh>graph_module._create("myStore",edgeDefinitions);

showexecutionresults

Listavailablegraphs

Listallgraphs.

graph_module._list()

Listsallgraphnamesstoredinthisdatabase.

Examples

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>graph_module._list();

[]

Loadagraph

GraphManagement

136

Getagraph

graph_module._graph(graphName)

Agraphcanberetrievedbyitsname.

Parameters

graphName(required)Uniqueidentifierofthegraph

Examples

Getagraph:

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>graph=graph_module._graph("social");

showexecutionresults

Removeagraph

Removeagraph

graph_module._drop(graphName,dropCollections)

Agraphcanbedroppedbyitsname.Thiscandropallcollectionscontainedinthegraphaslongastheyarenotusedwithinothergraphs.Todropthecollectionsonlybelongingtothisgraph,theoptionalparameterdrop-collectionshastobesettotrue.

Parameters

graphName(required)UniqueidentifierofthegraphdropCollections(optional)Defineifcollectionsshouldbedropped(default:false)

Examples

Dropagraphandkeepcollections:

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>graph_module._drop("social");

true

arangosh>db._collection("female");

[ArangoCollection18682,"female"(typedocument,statusloaded)]

arangosh>db._collection("male");

[ArangoCollection18685,"male"(typedocument,statusloaded)]

arangosh>db._collection("relation");

[ArangoCollection18688,"relation"(typeedge,statusloaded)]

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>graph_module._drop("social",true);

true

arangosh>db._collection("female");

null

arangosh>db._collection("male");

null

arangosh>db._collection("relation");

null

ModifyagraphdefinitionduringruntimeAfteryouhavecreatedangraphitsdefinitionisnotimmutable.Youcanstilladd,deleteormodifyedgedefinitionsandvertexcollections.

GraphManagement

137

Extendtheedgedefinitions

Addanotheredgedefinitiontothegraph

graph._extendEdgeDefinitions(edgeDefinition)

Extendstheedgedefinitionsofagraph.Ifanorphancollectionisusedinthisedgedefinition,itwillberemovedfromtheorphanage.Iftheedgecollectionoftheedgedefinitiontoaddisalreadyusedinthegraphorusedinadifferentgraphwithdifferentfromand/ortocollectionsanerroristhrown.

Parameters

edgeDefinition(required)Therelationdefinitiontoextendthegraph

Examples

arangosh>vargraph_module=require("@arangodb/general-graph")

arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);

arangosh>vared2=graph_module._relation("myEC2",["myVC1"],["myVC3"]);

arangosh>vargraph=graph_module._create("myGraph",[ed1]);

arangosh>graph._extendEdgeDefinitions(ed2);

Modifyanedgedefinition

Modifyanrelationdefinition

graph_module._editEdgeDefinition(edgeDefinition)

Editsonerelationdefinitionofagraph.Theedgedefinitionusedasargumentwillreplacetheexistingedgedefinitionofthegraphwhichhasthesamecollection.VertexCollectionsofthereplacededgedefinitionthatarenotusedinthenewdefinitionwilltransformtoanorphan.Orphansthatareusedinthisnewedgedefinitionwillbedeletedfromthelistoforphans.Othergraphswiththesameedgedefinitionwillbemodified,too.

Parameters

edgeDefinition(required)Theedgedefinitiontoreplacetheexistingedgedefinitionwiththesameattributecollection.

Examples

arangosh>vargraph_module=require("@arangodb/general-graph")

arangosh>varoriginal=graph_module._relation("myEC1",["myVC1"],["myVC2"]);

arangosh>varmodified=graph_module._relation("myEC1",["myVC2"],["myVC3"]);

arangosh>vargraph=graph_module._create("myGraph",[original]);

arangosh>graph._editEdgeDefinitions(modified);

Deleteanedgedefinition

Deleteonerelationdefinition

graph_module._deleteEdgeDefinition(edgeCollectionName,dropCollection)

Deletesarelationdefinitiondefinedbytheedgecollectionofagraph.Ifthecollectionsdefinedintheedgedefinition(collection,from,to)arenotusedinanotheredgedefinitionofthegraph,theywillbemovedtotheorphanage.

Parameters

edgeCollectionName(required)Nameofedgecollectionintherelationdefinition.dropCollection(optional)Defineiftheedgecollectionshouldbedropped.Defaultfalse.

Examples

Removeanedgedefinitionbutkeeptheedgecollection:

arangosh>vargraph_module=require("@arangodb/general-graph")

GraphManagement

138

arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);

arangosh>vared2=graph_module._relation("myEC2",["myVC1"],["myVC3"]);

arangosh>vargraph=graph_module._create("myGraph",[ed1,ed2]);

arangosh>graph._deleteEdgeDefinition("myEC1");

arangosh>db._collection("myEC1");

[ArangoCollection25989,"myEC1"(typeedge,statusloaded)]

Removeanedgedefinitionanddroptheedgecollection:

arangosh>vargraph_module=require("@arangodb/general-graph")

arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);

arangosh>vared2=graph_module._relation("myEC2",["myVC1"],["myVC3"]);

arangosh>vargraph=graph_module._create("myGraph",[ed1,ed2]);

arangosh>graph._deleteEdgeDefinition("myEC1",true);

arangosh>db._collection("myEC1");

null

ExtendvertexCollections

Eachgraphcanhaveanarbitraryamountofvertexcollections,whicharenotpartofanyedgedefinitionofthegraph.Thesecollectionsarecalledorphancollections.Ifthegraphisextendedwithanedgedefinitionusingoneoftheorphans,itwillberemovedfromthesetoforphancollectionautomatically.

Addavertexcollection

Addavertexcollectiontothegraph

graph._addVertexCollection(vertexCollectionName,createCollection)

Addsavertexcollectiontothesetoforphancollectionsofthegraph.Ifthecollectiondoesnotexist,itwillbecreated.Ifitisalreadyusedbyanyedgedefinitionofthegraph,anerrorwillbethrown.

Parameters

vertexCollectionName(required)Nameofvertexcollection.createCollection(optional)Iftruethecollectionwillbecreatedifitdoesnotexist.Default:true.

Examples

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);

arangosh>vargraph=graph_module._create("myGraph",[ed1]);

arangosh>graph._addVertexCollection("myVC3",true);

Gettheorphanedcollections

Getallorphancollections

graph._orphanCollections()

Returnsallvertexcollectionsofthegraphthatarenotusedinanyedgedefinition.

Examples

arangosh>vargraph_module=require("@arangodb/general-graph")

arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);

arangosh>vargraph=graph_module._create("myGraph",[ed1]);

arangosh>graph._addVertexCollection("myVC3",true);

arangosh>graph._orphanCollections();

GraphManagement

139

[

"myVC3"

]

Removeavertexcollection

Removeavertexcollectionfromthegraph

graph._removeVertexCollection(vertexCollectionName,dropCollection)

Removesavertexcollectionfromthegraph.Onlycollectionsnotusedinanyrelationdefinitioncanberemoved.Optionallythecollectioncanbedeleted,ifitisnotusedinanyothergraph.

Parameters

vertexCollectionName(required)Nameofvertexcollection.dropCollection(optional)Iftruethecollectionwillbedroppedifitisnotusedinanyothergraph.Default:false.

Examples

arangosh>vargraph_module=require("@arangodb/general-graph")

arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);

arangosh>vargraph=graph_module._create("myGraph",[ed1]);

arangosh>graph._addVertexCollection("myVC3",true);

arangosh>graph._addVertexCollection("myVC4",true);

arangosh>graph._orphanCollections();

arangosh>graph._removeVertexCollection("myVC3");

arangosh>graph._orphanCollections();

showexecutionresults

ManiuplatingVertices

Saveavertex

CreateanewvertexinvertexCollectionName

graph.vertexCollectionName.save(data)

Parameters

data(required)JSONdataofvertex.

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>graph.male.save({name:"Floyd",_key:"floyd"});

showexecutionresults

Replaceavertex

ReplacesthedataofavertexincollectionvertexCollectionName

graph.vertexCollectionName.replace(vertexId,data,options)

Parameters

vertexId(required)_idattributeofthevertexdata(required)JSONdataofvertex.

GraphManagement

140

options(optional)Seecollectiondocumentation

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>graph.male.save({neym:"Jon",_key:"john"});

arangosh>graph.male.replace("male/john",{name:"John"});

showexecutionresults

Updateavertex

UpdatesthedataofavertexincollectionvertexCollectionName

graph.vertexCollectionName.update(vertexId,data,options)

Parameters

vertexId(required)_idattributeofthevertexdata(required)JSONdataofvertex.options(optional)Seecollectiondocumentation

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>graph.female.save({name:"Lynda",_key:"linda"});

arangosh>graph.female.update("female/linda",{name:"Linda",_key:"linda"});

showexecutionresults

Removeavertex

RemovesavertexincollectionvertexCollectionName

graph.vertexCollectionName.remove(vertexId,options)

Additionallyremovesallingoingandoutgoingedgesofthevertexrecursively(seeedgeremove).

Parameters

vertexId(required)_idattributeofthevertexoptions(optional)Seecollectiondocumentation

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>graph.male.save({name:"Kermit",_key:"kermit"});

arangosh>db._exists("male/kermit")

arangosh>graph.male.remove("male/kermit")

arangosh>db._exists("male/kermit")

showexecutionresults

ManipulatingEdges

Saveanewedge

GraphManagement

141

CreatesanedgefromvertexfromtovertextoincollectionedgeCollectionName

graph.edgeCollectionName.save(from,to,data,options)

Parameters

from(required)_idattributeofthesourcevertexto(required)_idattributeofthetargetvertexdata(required)JSONdataoftheedgeoptions(optional)Seecollectiondocumentation

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>graph.relation.save("male/bob","female/alice",{type:"married",_key:

"bobAndAlice"});

showexecutionresultsIfthecollectionsoffromandtoarenotdefinedinanedgedefinitionofthegraph,theedgewillnotbestored.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>graph.relation.save(

........>"relation/aliceAndBob",

........>"female/alice",

........>{type:"married",_key:"bobAndAlice"});

[ArangoError1906:invalidedgebetweenrelation/aliceAndBobandfemale/alice.Doesn't

conformtoanyedgedefinition]

Replaceanedge

ReplacesthedataofanedgeincollectionedgeCollectionName.Notethat_fromand_toaremandatory.

graph.edgeCollectionName.replace(edgeId,data,options)

Parameters

edgeId(required)_idattributeoftheedgedata(required)JSONdataoftheedgeoptions(optional)Seecollectiondocumentation

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>graph.relation.save("female/alice","female/diana",{typo:"nose",_key:

"aliceAndDiana"});

arangosh>graph.relation.replace("relation/aliceAndDiana",{type:"knows",_from:

"female/alice",_to:"female/diana"});

showexecutionresults

Updateanedge

UpdatesthedataofanedgeincollectionedgeCollectionName

graph.edgeCollectionName.update(edgeId,data,options)

Parameters

GraphManagement

142

edgeId(required)_idattributeoftheedgedata(required)JSONdataoftheedgeoptions(optional)Seecollectiondocumentation

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>graph.relation.save("female/alice","female/diana",{type:"knows",_key:

"aliceAndDiana"});

arangosh>graph.relation.update("relation/aliceAndDiana",{type:"quarreled",_key:

"aliceAndDiana"});

showexecutionresults

Removeanedge

RemovesanedgeincollectionedgeCollectionName

graph.edgeCollectionName.remove(edgeId,options)

Ifthisedgeisusedasavertexbyanotheredge,theotheredgewillberemoved(recursively).

Parameters

edgeId(required)_idattributeoftheedgeoptions(optional)Seecollectiondocumentation

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>graph.relation.save("female/alice","female/diana",{_key:"aliceAndDiana"});

arangosh>db._exists("relation/aliceAndDiana")

arangosh>graph.relation.remove("relation/aliceAndDiana")

arangosh>db._exists("relation/aliceAndDiana")

showexecutionresults

Connectedges

Getallconnectingedgesbetween2groupsofverticesdefinedbytheexamples

graph._getConnectingEdges(vertexExample,vertexExample2,options)

Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.

Parameters

vertexExample1(optional)SeeDefinitionofexamplesvertexExample2(optional)SeeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:

edgeExamples:Filtertheedges,seeDefinitionofexamplesedgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.vertex1CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredontheintermediatevertexsteps.vertex2CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredontheintermediatevertexsteps.

Examples

Arouteplannerexample,allconnectingedgesbetweencapitals.

GraphManagement

143

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._getConnectingEdges({isCapital:true},{isCapital:true});

[]

GraphManagement

144

GraphFunctionsThischapterdescribesvariousfunctionsonagraph.Alotoftheseacceptavertex(oredge)exampleasparameterasdefinedinthenextsection.

ExampleswillexplaintheAPIonthethecitygraph:

DefinitionofexamplesFormanyofthefollowingfunctionsexamplescanbepassedinasaparameter.Examplesareusedtofiltertheresultsetforobjectsthatmatchtheconditions.Theseexamplescanhavethefollowingvalues:

null,thereisnomatchingexecutedallfoundresultsarevalid.Astring,onlyresultsarereturned,which_idequalthevalueofthestringAnexampleobject,definingasetofattributes.Onlyresultshavingtheseattributesarematched.Alistcontainingexampleobjectsand/orstrings.Allresultsmatchingatleastoneoftheelementsinthelistarereturned.Getverticesfromedges.

Getvertexfromofanedge

Getthesourcevertexofanedge

graph._fromVertex(edgeId)

Returnsthevertexdefinedwiththeattribute_fromoftheedgewithedgeIdasits_id.

Parameters

edgeId(required)_idattributeoftheedge

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

GraphFunctions

145

arangosh>vargraph=examples.loadGraph("social");

arangosh>varany=require("@arangodb").db.relation.any();

arangosh>graph._fromVertex("relation/"+any._key);

showexecutionresults

Getvertextoofanedge

Getthetargetvertexofanedge

graph._toVertex(edgeId)

Returnsthevertexdefinedwiththeattribute_tooftheedgewithedgeIdasits_id.

Parameters

edgeId(required)_idattributeoftheedge

Examples

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("social");

arangosh>varany=require("@arangodb").db.relation.any();

arangosh>graph._toVertex("relation/"+any._key);

showexecutionresults

_neighbors

Getallneighborsoftheverticesdefinedbytheexample

graph._neighbors(vertexExample,options)

Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.ThecomplexityofthismethodisO(n*m^x)withnbeingtheverticesdefinedbytheparametervertexExamplex,mtheaverageamountofneighborsandxthemaximaldepths.HencethedefaultcallwouldhaveacomplexityofO(n*m);

Parameters

vertexExample(optional)SeeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:

direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).edgeExamples:Filtertheedges,seeDefinitionofexamplesneighborExamples:Filtertheneighborvertices,seeDefinitionofexamplesedgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.vertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredontheintermediatevertexsteps.minDepth:Definestheminimalnumberofintermediatestepstoneighbors(defaultis1).maxDepth:Definesthemaximalnumberofintermediatestepstoneighbors(defaultis1).

Examples

Arouteplannerexample,allneighborsofcapitals.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._neighbors({isCapital:true});

showexecutionresultsArouteplannerexample,alloutboundneighborsofHamburg.

GraphFunctions

146

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._neighbors('germanCity/Hamburg',{direction:'outbound',maxDepth:2});

showexecutionresults

_commonNeighbors

Getallcommonneighborsoftheverticesdefinedbytheexamples.

graph._commonNeighbors(vertex1Example,vertex2Examples,optionsVertex1,optionsVertex2)

Thisfunctionreturnstheintersectionofgraph_module._neighbors(vertex1Example,optionsVertex1)andgraph_module._neighbors(vertex2Example,optionsVertex2).Forparameterdocumentationsee_neighbors.

ThecomplexityofthismethodisO(n*m^x)withnbeingthemaximalamountofverticesdefinedbytheparametersvertexExamples,mtheaverageamountofneighborsandxthemaximaldepths.HencethedefaultcallwouldhaveacomplexityofO(n*m);

Examples

Arouteplannerexample,allcommonneighborsofcapitals.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._commonNeighbors({isCapital:true},{isCapital:true});

showexecutionresultsArouteplannerexample,allcommonoutboundneighborsofHamburgwithanyotherlocationwhichhaveamaximaldepthof2:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._commonNeighbors(

........>'germanCity/Hamburg',

........>{},

........>{direction:'outbound',maxDepth:2},

........>{direction:'outbound',maxDepth:2});

showexecutionresults

_countCommonNeighborsGettheamountofcommonneighborsoftheverticesdefinedbytheexamples.

graph._countCommonNeighbors(vertex1Example,vertex2Examples,optionsVertex1,optionsVertex2)

Similarto_commonNeighborsbutreturnscountinsteadoftheelements.

Examples

Arouteplannerexample,allcommonneighborsofcapitals.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>varexample={isCapital:true};

arangosh>varoptions={includeData:true};

arangosh>graph._countCommonNeighbors(example,example,options,options);

showexecutionresults

GraphFunctions

147

Arouteplannerexample,allcommonoutboundneighborsofHamburgwithanyotherlocationwhichhaveamaximaldepthof2:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>varoptions={direction:'outbound',maxDepth:2,includeData:true};

arangosh>graph._countCommonNeighbors('germanCity/Hamburg',{},options,options);

showexecutionresults

_commonPropertiesGettheverticesofthegraphthatsharecommonproperties.

graph._commonProperties(vertex1Example,vertex2Examples,options)

Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertex1Exampleandvertex2Example.

ThecomplexityofthismethodisO(n)withnbeingthemaximalamountofverticesdefinedbytheparametersvertexExamples.

Parameters

vertex1Examples(optional)Filterthesetofsourcevertices,seeDefinitionofexamples

vertex2Examples(optional)Filterthesetofverticescomparedto,seeDefinitionofexamples

options(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:vertex1CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbesearchedforsourcevertices.vertex2CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbesearchedforcomparevertices.ignoreProperties:Oneoralistofattributenamesofadocumentthatshouldbeignored.

Examples

Arouteplannerexample,alllocationswiththesameproperties:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._commonProperties({},{});

showexecutionresultsArouteplannerexample,allcitieswhichsharesamepropertiesexceptforpopulation.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._commonProperties({},{},{ignoreProperties:'population'});

showexecutionresults

_countCommonPropertiesGettheamountofverticesofthegraphthatsharecommonproperties.

graph._countCommonProperties(vertex1Example,vertex2Examples,options)

Similarto_commonPropertiesbutreturnscountinsteadoftheobjects.

Examples

Arouteplannerexample,alllocationswiththesameproperties:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

GraphFunctions

148

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._countCommonProperties({},{});

showexecutionresultsArouteplannerexample,allGermancitieswhichsharesamepropertiesexceptforpopulation.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._countCommonProperties({},{},{vertex1CollectionRestriction:

'germanCity',

........>vertex2CollectionRestriction:'germanCity',ignoreProperties:'population'});

showexecutionresults

_paths

The_pathsfunctionreturnsallpathsofagraph.

graph._paths(options)

Thisfunctiondeterminesallavailablepathsinagraph.

ThecomplexityofthismethodisO(n*n*m)withnbeingtheamountofverticesinthegraphandmtheaverageamountofconnectededges;

Parameters

options(optional)Anobjectcontainingoptions,seebelow:direction:Thedirectionoftheedges.Possiblevaluesareany,inboundandoutbound(default).followCycles(optional):Ifsettotruethequeryfollowscyclesinthegraph,defaultisfalse.minLength(optional):Definestheminimallengthapathmusthavetobereturned(defaultis0).maxLength(optional):Definesthemaximallengthapathmusthavetobereturned(defaultis10).

Examples

Returnallpathsofthegraph"social":

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("social");

arangosh>g._paths();

showexecutionresultsReturnallinboundpathsofthegraph"social"withamaximallengthof1andaminimallengthof2:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("social");

arangosh>g._paths({direction:'inbound',minLength:1,maxLength:2});

showexecutionresults

_shortestPath

The_shortestPathfunctionreturnsallshortestpathsofagraph.

graph._shortestPath(startVertexExample,endVertexExample,options)

GraphFunctions

149

Thisfunctiondeterminesallshortestpathsinagraph.Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforstartandendvertex.Thelengthofapathisbydefaulttheamountofedgesfromonestartvertextoanendvertex.Theoptionweightallowstheusertodefineanedgeattributerepresentingthelength.

Parameters

startVertexExample(optional)AnexampleforthedesiredstartVertices(seeDefinitionofexamples).endVertexExample(optional)AnexampleforthedesiredendVertices(seeDefinitionofexamples).options(optional)Anobjectcontainingoptions,seebelow:

direction:Thedirectionoftheedgesasastring.Possiblevaluesareoutbound,inboundandany(default).edgeCollectionRestriction:Oneormultipleedgecollectionnames.Onlyedgesfromthesecollectionswillbeconsideredforthepath.startVertexCollectionRestriction:Oneormultiplevertexcollectionnames.Onlyverticesfromthesecollectionswillbeconsideredasstartvertexofapath.endVertexCollectionRestriction:Oneormultiplevertexcollectionnames.Onlyverticesfromthesecollectionswillbeconsideredasendvertexofapath.weight:Thenameoftheattributeoftheedgescontainingthelengthasastring.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedaslength.IfnodefaultissuppliedthedefaultwouldbepositiveInfinitysothepathcouldnotbecalculated.

Examples

Arouteplannerexample,shortestpathfromallgermantoallfrenchcities:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("routeplanner");

arangosh>g._shortestPath({},{},{weight:'distance',endVertexCollectionRestriction:

'frenchCity',

........>startVertexCollectionRestriction:'germanCity'});

showexecutionresultsArouteplannerexample,shortestpathfromHamburgandColognetoLyon:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("routeplanner");

arangosh>g._shortestPath([{_id:'germanCity/Cologne'},{_id:'germanCity/Munich'}],

'frenchCity/Lyon',

........>{weight:'distance'});

showexecutionresults

_distanceTo

The_distanceTofunctionreturnsallpathsandtheredistancewithinagraph.

graph._distanceTo(startVertexExample,endVertexExample,options)

Thisfunctionisawrapperofgraph._shortestPath.Itdoesnotreturntheactualpathbutonlythedistancebetweentwovertices.

Examples

Arouteplannerexample,shortestdistancefromallgermantoallfrenchcities:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("routeplanner");

arangosh>g._distanceTo({},{},{weight:'distance',endVertexCollectionRestriction:

'frenchCity',

........>startVertexCollectionRestriction:'germanCity'});

GraphFunctions

150

showexecutionresultsArouteplannerexample,shortestdistancefromHamburgandColognetoLyon:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("routeplanner");

arangosh>g._distanceTo([{_id:'germanCity/Cologne'},{_id:'germanCity/Munich'}],

'frenchCity/Lyon',

........>{weight:'distance'});

showexecutionresults

_absoluteEccentricityGettheeccentricityoftheverticesdefinedbytheexamples.

graph._absoluteEccentricity(vertexExample,options)

Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.

Parameters

vertexExample(optional)Filterthevertices,seeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:

direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).edgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.startVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredforsourcevertices.endVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredfortargetvertices.weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheeccentricitycannotbecalculated.

Examples

Arouteplannerexample,theabsoluteeccentricityofalllocations.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._absoluteEccentricity({});

showexecutionresultsArouteplannerexample,theabsoluteeccentricityofalllocations.Thisconsiderstheactualdistances.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._absoluteEccentricity({},{weight:'distance'});

showexecutionresultsArouteplannerexample,theabsoluteeccentricityofallcitiesregardingonlyoutboundpaths.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._absoluteEccentricity({},{startVertexCollectionRestriction:

'germanCity',

........>direction:'outbound',weight:'distance'});

showexecutionresults

GraphFunctions

151

_eccentricity

Getthenormalizedeccentricityoftheverticesdefinedbytheexamples.

graph._eccentricity(vertexExample,options)

Similarto_absoluteEccentricitybutreturnsanormalizedresult.

Examples

Arouteplannerexample,theeccentricityofalllocations.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._eccentricity();

showexecutionresultsArouteplannerexample,theweightedeccentricity.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._eccentricity({weight:'distance'});

showexecutionresults

_absoluteCloseness

Gettheclosenessoftheverticesdefinedbytheexamples.

graph._absoluteCloseness(vertexExample,options)

Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.

Parameters

vertexExample(optional)Filterthevertices,seeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:

direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).edgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.startVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredforsourcevertices.endVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredfortargetvertices.weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheclosenesscannotbecalculated.

Examples

Arouteplannerexample,theabsoluteclosenessofalllocations.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._absoluteCloseness({});

showexecutionresultsArouteplannerexample,theabsoluteclosenessofalllocations.Thisconsiderstheactualdistances.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

GraphFunctions

152

arangosh>graph._absoluteCloseness({},{weight:'distance'});

showexecutionresultsArouteplannerexample,theabsoluteclosenessofallGermanCitiesregardingonlyoutboundpaths.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._absoluteCloseness({},{startVertexCollectionRestriction:'germanCity',

........>direction:'outbound',weight:'distance'});

showexecutionresults

_closeness

Getthenormalizedclosenessofgraphsvertices.

graph._closeness(options)

Similarto_absoluteClosenessbutreturnsanormalizedvalue.

Examples

Arouteplannerexample,thenormalizedclosenessofalllocations.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._closeness();

showexecutionresultsArouteplannerexample,theclosenessofalllocations.Thisconsiderstheactualdistances.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._closeness({weight:'distance'});

showexecutionresultsArouteplannerexample,theclosenessofallcitiesregardingonlyoutboundpaths.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._closeness({direction:'outbound',weight:'distance'});

showexecutionresults

_absoluteBetweennessGetthebetweennessofallverticesinthegraph.

graph._absoluteBetweenness(vertexExample,options)

Parameters

vertexExample(optional)Filterthevertices,seeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:

direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthis

GraphFunctions

153

defaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencethebetweenesscannotbecalculated.

Examples

Arouteplannerexample,theabsolutebetweennessofalllocations.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._absoluteBetweenness({});

showexecutionresultsArouteplannerexample,theabsolutebetweennessofalllocations.Thisconsiderstheactualdistances.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._absoluteBetweenness({weight:'distance'});

{

}

Arouteplannerexample,theabsolutebetweennessofallcitiesregardingonlyoutboundpaths.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._absoluteBetweenness({direction:'outbound',weight:'distance'});

{

}

_betweenness

Getthenormalizedbetweennessofgraphsvertices.

graph_module._betweenness(options)

Similarto_absoluteBetweenessbutreturnsnormalizedvalues.

Examples

Arouteplannerexample,thebetweennessofalllocations.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._betweenness();

showexecutionresultsArouteplannerexample,thebetweennessofalllocations.Thisconsiderstheactualdistances.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._betweenness({weight:'distance'});

showexecutionresultsArouteplannerexample,thebetweennessofallcitiesregardingonlyoutboundpaths.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._betweenness({direction:'outbound',weight:'distance'});

GraphFunctions

154

showexecutionresults

_radiusGettheradiusofagraph.

`

Parameters

options(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheradiuscannotbecalculated.

Examples

Arouteplannerexample,theradiusofthegraph.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._radius();

1

Arouteplannerexample,theradiusofthegraph.Thisconsiderstheactualdistances.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._radius({weight:'distance'});

1

Arouteplannerexample,theradiusofthegraphregardingonlyoutboundpaths.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._radius({direction:'outbound',weight:'distance'});

1

_diameter

Getthediameterofagraph.

graph._diameter(graphName,options)

Parameters

options(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheradiuscannotbecalculated.

Examples

GraphFunctions

155

Arouteplannerexample,thediameterofthegraph.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._diameter();

1

Arouteplannerexample,thediameterofthegraph.Thisconsiderstheactualdistances.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._diameter({weight:'distance'});

1

Arouteplannerexample,thediameterofthegraphregardingonlyoutboundpaths.

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._diameter({direction:'outbound',weight:'distance'});

1

GraphFunctions

156

SmartGraphsThisfeatureisonlyavailableintheEnterpriseEdition.

Thischapterdescribesthesmart-graphmodule.Itenablesyoutomanagegraphsatscale,itwillgiveavastperformancebenefitforallgraphsshardedinanArangoDBCluster.Onasingleserverthisfeatureispointless,henceitisonlyavailableinaclustermode.IntermsofqueryingthereisnodifferencebetweensmartandGeneralGraphs.Theformerareatransparentreplacementforthelatter.SoforqueryingthegraphpleaserefertoAQLGraphOperationsandGraphFunctionssections.TheoptimizeriscleverenoughtoidentifyifweareonaSmartGraphornot.

Thedifferenceisonlyinthemanagementsection:creatingandmodifyingtheunderlyingcollectionsofthegraph.ForadetailedAPIreferencepleaserefertoSmartGraphManagement.

Whatmakesagraphsmart?

Mostgraphshaveonefeaturethatdividestheentiregraphintoseveralsmallersubgraphs.Thesesubgraphshavealargeamountofedgesthatonlyconnectverticesinthesamesubgraphandonlyhavefewedgesconnectingverticesfromothersubgraphs.Examplesforthesegraphsare:

SocialNetworks

Typicallythefeaturehereistheregion/countryuserslivein.Everyusertypicalliyhasmorecontactsinthesameregion/countrythenshehasinotherregions/countries

TransportSystems

Forthosealsothefeatureistheregion/country.Youhavemanylocaltransportionbutonlyfewaccrosscountries.

E-Commerce

Inthiscaseprobablythecategoryofproductsisagoodfeature.Oftenproductsofthesamecategoryareboughttogether.

Ifthisfeatureisknown,SmartGraphscanmakeuseifit.WhencreatingaSmartGraphyouhavetodefineasmartAttribute,whichisthenameofanattributestoredineveryvertex.Thegraphwillthanbeautomaticallyshardedinsuchawaythatallverticeswiththesamevaluearestoredonthesamephysicalmachine,alledgesconnectingverticeswithidenticalsmartAttributevaluesarestoredonthismachineaswell.Duringquerytimethequeryoptimizerandthequeryexecutorbothknowforeverydocumentexactlywhereitisstoredandcantherebyminimizenetworkoverhead.Everythingthatcanbecomputedlocallywillbecomputedlocally.

BenefitsofSmartGraphs

Becauseoftheabovedescribedguaranteedsharding,theperformanceofqueriesthatonlycoveronesubgraphhaveaperformancealmostequaltoanonlylocalcomputation.Queriesthatcovermorethanonesubgraphrequiresomenetworkoverhead.Themoresubgraphsaretouchedthemorenetworkcostwillapply.HowevertheoverallperformanceisneverworsethanthesamequeryonaGeneralGraph.

Gettingstarted

FirstofallSmartGraphscannotuseexistingcollections,whenswitchingtoSmartGraphfromanexistingdatasetyouhavetoimportthedataintoafreshSmartGraph.Thisswitchcanbeeasilyachievedwitharangodumpandarangorestore.TheonlythingyouhavetochangeinthispipelineisthatyoucreatethenewcollectionswiththeSmartGraphbeforestartingarangorestore.

Createagraph

IncomparisontoGeneralGraphwehavetoaddmoreoptionswhencreatingthegraph.ThetwooptionssmartGraphAttributeandnumberOfShardsarerequiredandcannotbemodifedlater.

arangosh>vargraph_module=require("@arangodb/smart-graph");

arangosh>vargraph=graph_module._create("myGraph",[],[],{smartGraphAttribute:"region",numberOfShards:9});

arangosh>graph;

[SmartGraphmyGraphEdgeDefinitions:[]VertexCollections:[]]

Addsomevertexcollections

SmartGraphs

157

ThisisagainidenticaltoGeneralGraph.Themodulewillsetupcorrectshardingforallthesecollections.Note:Thecollectionshavetobenew.

arangosh>graph._addVertexCollection("shop");

arangosh>graph._addVertexCollection("customer");

arangosh>graph._addVertexCollection("pet");

arangosh>graph;

[SmartGraphmyGraphEdgeDefinitions:[]VertexCollections:["shop","customer","pet"]]

DefinerelationsontheGraph

arangosh>varrel=graph_module._relation("isCustomer",["shop"],["customer"]);

arangosh>graph._extendEdgeDefinitions(rel);

arangosh>graph;

[SmartGraphmyGraphEdgeDefinitions:["isCustomer:[shop]->[customer]"]VertexCollections:["pet"]]

SmartGraphs

158

SmartGraphManagementThischapterdescribestheJavaScriptinterfaceforcreatingandmodifyingSmartGraphs.AtfirstyouhavetonotethateverySmartGraphisaspecializedversionofaGeneralGraph,whichmeansalloftheGeneralGraphfunctionalityisavailableonaSmartGraphaswell.Themajordifferenceofbothmodulesishandlingoftheunderlyingcollections,theGeneralGraphdoesnotenforceormaintainanyshardingofthecollectionsandcanthereforcombinearbitrarysetsofexistingcollections.SmartGraphsenforceandrelyonaspecialshardingoftheunderlyingcollectionsandhencecanonlyworkwithcollectionsthatarecreatedthroughtheSmartGraphitself.ThisalsomeansthatSmartGraphscannotbeoverlapping,acollectioncaneitherbeshardedforoneSmartGraphorfortheother.IfyouneedtomakesurethatallqueriescanbeexecutedwithSmartGraphperformance,justcreateonelargeSmartGraphcoveringeverythingandqueryitstatingthesubsetofedgecollectionsexplicitly.TogenerallyunderstandtheconceptofthismodulepleasereadthechapteraboutGeneralGraphManagementfirst.Inthefollowingwewillonlydescribetheoverloadedfunctionality.Everythingelseworksidenticalinbothmodules.

Createagraph

AlsoSmartGraphsrequireedgerelationstobecreated,theformatoftherelationsisidentical.TheonlydifferenceisthatallcollectionsusedwithintherelationstocreateanewSmartGraphcannotexistyet.TheyhavetobecreatedbytheGraphinordertoenforcethecorrectsharding.

Createagraph

graph_module._create(graphName,edgeDefinitions,orphanCollections,smartOptions)

ThecreationofagraphrequiresthenameandsomeSmartGraphoptions.DuetotheAPIedgeDefinitionsandorphanCollectionshavetobegiven,butbothcanbeemptyarraysandcanbecreatedlater.TheedgeDefinitionscanbecreatedusingtheconveniencemethod_relationknownfromthegeneral-graphmodule,whichisalsoavailablehere.orphanCollectionsagainisjustalistofadditionalvertexcollectionswhicharenotyetconnectedviaedgesbutshouldfollowthesameshardingtobeconnectedlateron.Allcollectionsusedwithinthecreationprocessarenewlycreated.Theprocesswillfailifoneofthemalreadyexists.Allnewlycreatedcollectionswillimmediatelybedroppedagaininthefailedcase.

Parameters

graphName(required)UniqueidentifierofthegraphedgeDefinitions(required)Listofrelationdefinitionobjects,maybeemptyorphanCollections(required)Listofadditionalvertexcollectionnames,maybeemptysmartOptions(required)AJSONobjecthavingthefollowingkeys:

numberOfShards(required)Thenumberofshardsthatwillbecreatedforeachcollection.Tomaintainthecorrectshardingallcollectionsneedanidenticalnumberofshards.Thiscannotbemodifiedaftercreationofthegraph.smartGraphAttribute(required)Theattributethatwillbeusedforsharding.Allverticesarerequiredtohavethisattributesetandithastobeastring.Edgesderivetheattributefromtheirconnectedvertices.

Examples

Createanemptygraph,edgedefinitionscanbeaddedatruntime:

arangosh>vargraph_module=require("@arangodb/smart-graph");

arangosh>vargraph=graph_module._create("myGraph",[],[],{smartGraphAttribute:"region",numberOfShards:9});

[SmartGraphmyGraphEdgeDefinitions:[]VertexCollections:[]]

Createagraphusinganedgecollectionedgesandasinglevertexcollectionvertices

arangosh>vargraph_module=require("@arangodb/smart-graph");

arangosh>varedgeDefinitions=[graph_module._relation("edges","vertices","vertices")];

arangosh>vargraph=graph_module._create("myGraph",edgeDefinitions,[],{smartGraphAttribute:"region",numberOfShards:9});

[SmartGraphmyGraphEdgeDefinitions:["edges:[vertices]->[vertices]"]VertexCollections:[]]

Createagraphwithedgedefinitionsandorphancollections:

arangosh>vargraph_module=require("@arangodb/smart-graph");

arangosh>varedgeDefinitions=[graph_module._relation("myRelation",["male","female"],["male","female"])];

SmartGraphManagement

159

arangosh>vargraph=graph_module._create("myGraph",edgeDefinitions,["sessions"],{smartGraphAttribute:"region",numberOfSh

ards:9});

[GraphmyGraphEdgeDefinitions:[

"myRelation:[female,male]->[female,male]"

]VertexCollections:[

"sessions"

]]

ModifyagraphdefinitionduringruntimeAfteryouhavecreatedaSmartGraphitsdefinitionisalsonotimmutable.Youcanstilladdorremoverelations.ThisisagainidenticaltoGeneralGraphs.Howeverthereisoneimportantdifference:Youcanonlyaddcollectionsthateitherdonotexist,orthathavebeencreatedbythisgraphearlier.Thelatercanbeachievedifyouforexampleremoveanorphancollectionfromthisgraph,withoutdroppingthecollectionitself.Thanaftersometimeyoudecidetoadditagain,itcanbeused.Thisisbecausetheenforcedshardingisstillappliedtothisvertexcollection,henceitissuitabletobeaddedagain.

Removeavertexcollection

Removeavertexcollectionfromthegraph

graph._removeVertexCollection(vertexCollectionName,dropCollection)

InmostcasesthisfunctionworksidenticallytotheGeneralGraphone.Butthereisonespecialcase:Thefirstvertexcollectionaddedtothegraph(eitherorphanorwithinarelation)definestheshardingforallcollectionswithinthegraph.Thiscollectioncanneverberemovedfromthegraph.

Parameters

vertexCollectionName(required)Nameofvertexcollection.dropCollection(optional)Iftruethecollectionwillbedroppedifitisnotusedinanyothergraph.Default:false.

Examples

Thefollowingexampleshowsthatyoucannotdroptheinitialcollection.Youhavetodropthecompletegraph.Ifyoujustwanttogetridofthedatatruncateit.

arangosh>vargraph_module=require("@arangodb/smart-graph")

arangosh>varrelation=graph_module._relation("edges","vertices","vertices");

arangosh>vargraph=graph_module._create("myGraph",[relation],["other"],{smartGraphAttribute:"region",numberOfShards:

9});

arangosh>graph._orphanCollections();

[

"other"

]

arangosh>graph._deleteEdgeDefinition("edges");

arangosh>graph._orphanCollections();

[

"vertices",

"other"

]

arangosh>graph._removeVertexCollection("other");

arangosh>graph._orphanCollections();

[

"vertices"

]

arangosh>graph._removeVertexCollection("vertices");

ArangoError4002:cannotdropthissmartcollection

SmartGraphManagement

160

TraversalsArangoDBprovidesseveralwaystoquerygraphdata.Verysimpleoperationscanbecomposedwiththelow-leveledgemethodsedges,inEdges,andoutEdgesforedgecollections.Theseworkonnamedandanonymousgraphs.Formorecomplexoperations,ArangoDBprovidespredefinedtraversalobjects.

AlsoTraversalshavebeenaddedtoAQL.PleasereadthechapteraboutAQLtraversersalsbeforeyoucontinuereadinghere.MostofthetraversalcasesarecoveredbyAQLandwillbeexecutedinanoptimizedway.OnlyifthelogicforyouristoocomplextobedefinedusingAQLfiltersyoucanusethetraversalobjectdefinedherewhichgivesyoucompleteprogrammaticaccesstothedata.

Foranyofthefollowingexamples,we'llbeusingtheexamplecollectionsvande,populatedwithcontinents,countriesandcapitalsdatalistedbelow(seeExampleData).

StartingfromScratchArangoDBprovidestheedges,inEdges,andoutEdgesmethodsforedgecollections.Thesemethodscanbeusedtoquicklydetermineifavertexisconnectedtoothervertices,andwhich.ThisfunctionalitycanbeexploitedtowriteverysimplegraphqueriesinJavaScript.

Forexample,todeterminewhichedgesarelinkedtotheworldvertex,wecanuseinEdges:

db.e.inEdges('v/world').forEach(function(edge){

require("@arangodb").print(edge._from,"->",edge.type,"->",edge._to);

});

inEdgeswillgiveusallingoingedgesforthespecifiedvertexv/world.TheresultisaJavaScriptarraythatwecaniterateoverandprinttheresults:

v/continent-africa->is-in->v/world

v/continent-south-america->is-in->v/world

v/continent-asia->is-in->v/world

v/continent-australia->is-in->v/world

v/continent-europe->is-in->v/world

v/continent-north-america->is-in->v/world

Note:edges,inEdges,andoutEdgesreturnanarrayofedges.Ifwewanttoretrievethelinkedvertices,wecanuseeachedges'_fromand_toattributesasfollows:

db.e.inEdges('v/world').forEach(function(edge){

require("@arangodb").print(db._document(edge._from).name,"->",edge.type,"->",db._document(edge._to).name);

});

Weareusingthedocumentmethodfromthedbobjecttoretrievetheconnectedverticesnow.

Whilethismaybesufficientforone-levelgraphoperations,writingatraversalbyhandmaybecometoocomplexformulti-leveltraversals.

Traversals

161

GettingstartedTouseatraversalobject,wefirstneedtorequirethetraversalmodule:

vartraversal=require("@arangodb/graph/traversal");

varexamples=require("@arangodb/graph-examples/example-graph.js");

examples.loadGraph("worldCountry");

Wethenneedtosetupaconfigurationforthetraversalanddetermineatwhichvertextostartthetraversal:

varconfig={

datasource:traversal.generalGraphDatasourceFactory("worldCountry"),

strategy:"depthfirst",

order:"preorder",

filter:traversal.visitAllFilter,

expander:traversal.inboundExpander,

maxDepth:1

};

varstartVertex=db._document("v/world");

Note:ThestartVertexneedstobeadocument,notonlyadocumentid.

Wecanthencreateatraverserandstartthetraversalbycallingitstraversemethod.Notethattraverseneedsaresultobject,whichitcanmodifyinplace:

varresult={

visited:{

vertices:[],

paths:[]

}

};

vartraverser=newtraversal.Traverser(config);

traverser.traverse(result,startVertex);

Finally,wecanprintthecontentsoftheresultsobject,limitedtothevisitedvertices.Wewillonlyprintthenameandtypeofeachvisitedvertexforbrevity:

require("@arangodb").print(result.visited.vertices.map(function(vertex){

returnvertex.name+"("+vertex.type+")";

}));

Thefullscript,whichincludesallstepscarriedoutsofaristhus:

vartraversal=require("@arangodb/graph/traversal");

varconfig={

datasource:traversal.generalGraphDatasourceFactory("worldCountry"),

strategy:"depthfirst",

order:"preorder",

filter:traversal.visitAllFilter,

expander:traversal.inboundExpander,

maxDepth:1

};

varstartVertex=db._document("v/world");

varresult={

visited:{

vertices:[],

paths:[]

}

};

vartraverser=newtraversal.Traverser(config);

traverser.traverse(result,startVertex);

UsingTraversalObjects

162

require("@arangodb").print(result.visited.vertices.map(function(vertex){

returnvertex.name+"("+vertex.type+")";

}));

Theresultisanarrayofverticesthatwerevisitedduringthetraversal,startingatthestartvertex(i.e.v/worldinourexample):

[

"World(root)",

"Africa(continent)",

"Asia(continent)",

"Australia(continent)",

"Europe(continent)",

"NorthAmerica(continent)",

"SouthAmerica(continent)"

]

Note:Theresultislimitedtoverticesdirectlyconnectedtothestartvertex.WeachievedthisbysettingthemaxDepthattributeto1.Notsettingitwouldreturnthefullarrayofvertices.

TraversalDirection

Fortheexamplescontainedinthismanual,we'llbestartingthetraversalsatvertexv/world.Verticesinourgraphareconnectedlikethis:

v/world<-is-in<-continent(Africa)<-is-in<-country(Algeria)<-is-in<-capital(Algiers)

Togetanymeaningfulresults,wemusttraversethegraphininboundorder.Thismeans,we'llbefollowingallincomingedgesoftoavertex.Inthetraversalconfiguration,wehavespecifiedthisviatheexpanderattribute:

varconfig={

...

expander:traversal.inboundExpander

};

Forothergraphs,wemightwanttotraverseviatheoutgoingedges.Forthis,wecanusetheoutboundExpander.ThereisalsoananyExpander,whichwillfollowbothoutgoingandincomingedges.Thisshouldbeusedwithcareandthetraversalshouldalwaysbelimitedtoamaximumnumberofiterations(e.g.usingthemaxIterationsattribute)inordertoterminateatsomepoint.

Toinvokethedefaultoutboundexpanderforagraph,simplyusethepredefinedfunction:

varconfig={

...

expander:traversal.outboundExpander

};

Pleasenotetheoutboundexpanderwillnotproduceanyoutputfortheexamplesifwestillstartthetraversalatthev/worldvertex.

Still,wecanusetheoutboundexpanderifwestartsomewhereelseinthegraph,e.g.

vartraversal=require("@arangodb/graph/traversal");

varconfig={

datasource:traversal.generalGraphDatasourceFactory("world_graph"),

strategy:"depthfirst",

order:"preorder",

filter:traversal.visitAllFilter,

expander:traversal.outboundExpander

};

varstartVertex=db._document("v/capital-algiers");

varresult={

visited:{

vertices:[],

paths:[]

}

};

UsingTraversalObjects

163

vartraverser=newtraversal.Traverser(config);

traverser.traverse(result,startVertex);

require("@arangodb").print(result.visited.vertices.map(function(vertex){

returnvertex.name+"("+vertex.type+")";

}));

Theresultis:

[

"Algiers(capital)",

"Algeria(country)",

"Africa(continent)",

"World(root)"

]

whichconfirmsthatnowwe'regoingoutbound.

TraversalStrategy

Depth-firsttraversals

Thevisitationorderofverticesisdeterminedbythestrategyandorderattributessetintheconfiguration.Wechosedepthfirstandpreorder,meaningthetraverserwillvisiteachvertexbeforehandlingconnectededges(pre-order),anddescendintoanyconnectededgesbeforeprocessingotherverticesonthesamelevel(depth-first).

Let'sremovethemaxDepthattributenow.We'llnowbegettingallvertices(directlyandindirectlyconnectedtothestartvertex):

varconfig={

datasource:traversal.generalGraphDatasourceFactory("world_graph"),

strategy:"depthfirst",

order:"preorder",

filter:traversal.visitAllFilter,

expander:traversal.inboundExpander

};

varresult={

visited:{

vertices:[],

paths:[]

}

};

vartraverser=newtraversal.Traverser(config);

traverser.traverse(result,startVertex);

require("@arangodb").print(result.visited.vertices.map(function(vertex){

returnvertex.name+"("+vertex.type+")";

}));

Theresultwillbealongerarray,assembledindepth-first,pre-orderorder.Foreachcontinentfound,thetraverserwilldescendintolinkedcountries,andthenintothelinkedcapital:

[

"World(root)",

"Africa(continent)",

"Algeria(country)",

"Algiers(capital)",

"Angola(country)",

"Luanda(capital)",

"Botswana(country)",

"Gaborone(capital)",

"BurkinaFaso(country)",

"Ouagadougou(capital)",

...

]

UsingTraversalObjects

164

Let'sswitchtheorderattributefrompreordertopostorder.Thiswillmakethetraverservisitverticesafterallconnectedverticeswerevisited(i.e.mostdistantverticeswillbeemittedfirst):

[

"Algiers(capital)",

"Algeria(country)",

"Luanda(capital)",

"Angola(country)",

"Gaborone(capital)",

"Botswana(country)",

"Ouagadougou(capital)",

"BurkinaFaso(country)",

"Bujumbura(capital)",

"Burundi(country)",

"Yaounde(capital)",

"Cameroon(country)",

"N'Djamena(capital)",

"Chad(country)",

"Yamoussoukro(capital)",

"Coted'Ivoire(country)",

"Cairo(capital)",

"Egypt(country)",

"Asmara(capital)",

"Eritrea(country)",

"Africa(continent)",

...

]

Breadth-firsttraversals

Ifwegobacktopreorder,butchangethestrategytobreadth-firstandre-runthetraversal,we'llseethatthereturnorderchanges,anditemsonthesamelevelwillbereturnedadjacently:

[

"World(root)",

"Africa(continent)",

"Asia(continent)",

"Australia(continent)",

"Europe(continent)",

"NorthAmerica(continent)",

"SouthAmerica(continent)",

"BurkinaFaso(country)",

"Burundi(country)",

"Cameroon(country)",

"Chad(country)",

"Algeria(country)",

"Angola(country)",

...

]

Note:Theorderofitemsreturnedforthesamelevelisundefined.Thisisbecausethereisnonaturalorderofedgesforavertexwithmultipleconnectededges.Toexplicitlysettheorderforedgesonthesamelevel,youcanspecifyanedgecomparatorfunctionwiththesortattribute:

varconfig={

...

sort:function(l,r){returnl._key<r._key?1:-1;}

...

};

Theargumentslandrareedgedocuments.Thiswilltraverseedgesofthesamevertexinbackward_keyorder:

[

"World(root)",

"SouthAmerica(continent)",

"NorthAmerica(continent)",

"Europe(continent)",

"Australia(continent)",

"Asia(continent)",

UsingTraversalObjects

165

"Africa(continent)",

"Ecuador(country)",

"Colombia(country)",

"Chile(country)",

"Brazil(country)",

"Bolivia(country)",

"Argentina(country)",

...

]

Note:Thisattributeonlyworksfortheusualexpanderstraversal.inboundExpander,traversal.outboundExpander,traversal.anyExpanderandtheircorresponding"WithLabels"variants.Ifyouareusingcustomexpandersyouhavetoorganizethesortingwithinthespecifiedexpander.

WritingCustomVisitors

Sofarwehaveusedmuchofthetraverser'sdefaultfunctions.Thetraverserisveryconfigurableandmanyofthedefaultfunctionscanbeoverriddenwithcustomfunctionality.

Forexample,wehavebeenusingthedefaultvisitorfunction(whichisalwaysusediftheconfigurationdoesnotcontainthevisitorattribute).Thedefaultvisitorfunctioniscalledforeachvertexinatraversal,andwillpushitintotheresult.Thisisthereasonwhytheresultvariablelookeddifferentafterthetraversal,andneededtobeinitializedbeforethetraversalwasstarted.

Notethatthedefaultvisitor(namedtrackingVisitor)willaddeveryvisitedvertexintotheresult,includingthefullpathsfromthestartvertex.Thisisusefulforlearninganddebuggingpurposes,butshouldbeavoidedinproductionbecauseitmightproduce(andcopy)hugeamountsofdata.Instead,onlythosedatashouldbecopiedintotheresultthatareactuallynecessary.

Thetraversercomeswiththefollowingpredefinedvisitors:

trackingVisitor:thisisthedefaultvisitor.Itwillcopyalldataofeachvisitedvertexplusthefullpathinformationintotheresult.Thiscanbeslowiftheresultsetishugeorverticescontainalotofdata.countingVisitor:thisisaverylightweightvisitor:allitdoesisincreaseacounterintheresultforeachvertexvisited.Vertexdataandpathswillnotbecopiedintotheresult.doNothingVisitor:ifnoactionshallbecarriedoutwhenavertexisvisited,thisvisitorcanbeemployed.Itwillnotdoanythingandwillthusbefast.Itcanbeusedforperformancecomparisonswithothervisitors.

Wecanalsowriteourownvisitorfunctionifwewantto.Thegeneralfunctionsignatureforvisitorfunctionsisasfollows:

varconfig={

...

visitor:function(config,result,vertex,path,connected){...}

};

Note:theconnectedparametervaluewillonlybesetifthetraversalorderissettopreorder-expander.Otherwise,thisparameterwon'tbesetbythetraverser.

Visitorfunctionsarenotexpectedtoreturnanyvalues.Instead,theycanmodifytheresultvariable(e.g.bypushingthecurrentvertexintoit),ordoanythingelse.Forexample,wecancreateasimplevisitorfunctionthatonlyprintsinformationaboutthecurrentvertexaswetraverse:

varconfig={

datasource:traversal.generalGraphDatasourceFactory("world_graph"),

strategy:"depthfirst",

order:"preorder",

filter:traversal.visitAllFilter,

expander:traversal.inboundExpander,

visitor:function(config,result,vertex,path){

require("@arangodb").print("visitingvertex",vertex.name);

}

};

vartraverser=newtraversal.Traverser(config);

traverser.traverse(undefined,startVertex);

Towriteavisitorthatincrementsacountereachtimeavertexisvisited,wecouldwritethefollowingcustomvisitor:

UsingTraversalObjects

166

config.visitor=function(config,result,vertex,path,connected){

if(!result){

result={};

}

if(!result.hasOwnProperty('count')){

result.count=0;

}

++result.count;

}

Notethatsuchvisitorisalreadypredefined(it'sthecountingVisitordescribedabove).Itcanbeusedasfollows:

config.visitor=traversal.countingVisitor;

Anotherexampleofavisitorisonethatcollectsthe_idvaluesofallverticesvisited:

config.visitor=function(config,result,vertex,path,connected){

if(!result){

result={};

}

if(!result.hasOwnProperty("visited")){

result.visited={vertices:[]};

}

result.visited.vertices.push(vertex._id);

}

Whenthetraversalorderissettopreorder-expander,thetraverserwillpassafifthparametervalueintothevisitorfunction.Thisparametercontainstheconnectededgesofthevisitedvertexasanarray.Thiscanbehandybecauseinthiscasethevisitorwillgetallinformationaboutthevertexandtheconnectededgestogether.

Forexample,thefollowingvisitorcanbeusedtoprintonlyleafnodes(thatdonothaveanyfurtherconnectededges):

config.visitor=function(config,result,vertex,path,connected){

if(connected&&connected.length===0){

require("@arangodb").print("foundaleaf-node:",vertex);

}

}

Notethatforthisvisitortowork,thetraversalorderattributeneedstobesettothevaluepreorder-expander.

FilteringVerticesandEdges

FilteringVertices

Sofarwehaveprintedorreturnedallverticesthatwerevisitedduringthetraversal.Thisisnotalwaysrequired.Iftheresultshallberestricttojustspecificvertices,wecanuseafilterfunctionforvertices.Itcanbedefinedbysettingthefilterattributeofatraversalconfiguration,e.g.:

varconfig={

filter:function(config,vertex,path){

if(vertex.type!=='capital'){

return'exclude';

}

}

}

Theabovefilterfunctionwillexcludeallverticesthatdonothaveatypevalueofcapital.Thefilterfunctionwillbecalledforeachvertexfoundduringthetraversal.Itwillreceivethetraversalconfiguration,thecurrentvertex,andthefullpathfromthetraversalstartvertextothecurrentvertex.Thepathconsistsofanarrayofedges,andanarrayofvertices.Wecouldalsofiltereverythingbutcapitalsby

UsingTraversalObjects

167

checkingthelengthofthepathfromthestartvertextothecurrentvertex.Capitalswillhaveadistanceof3fromthev/worldstartvertex(capital→is-in→country→is-in→continent→is-in→world):

varconfig={

...

filter:function(config,vertex,path){

if(path.edges.length<3){

return'exclude';

}

}

}

Note:Ifafilterfunctionreturnsnothing(orundefined),thecurrentvertexwillbeincluded,andallconnectededgeswillbefollowed.Ifafilterfunctionreturnsexcludethecurrentvertexwillbeexcludedfromtheresult,andallstillallconnectededgeswillbefollowed.Ifafilterfunctionreturnsprune,thecurrentvertexwillbeincluded,butnoconnectededgeswillbefollowed.

Forexample,thefollowingfilterfunctionwillnotdescendintoconnectededgesofcontinents,limitingthedepthofthetraversal.Still,continentverticeswillbeincludedintheresult:

varconfig={

...

filter:function(config,vertex,path){

if(vertex.type==='continent'){

return'prune';

}

}

}

Itisalsopossibletocombineexcludeandprunebyreturninganarraywithbothvalues:

return['exclude','prune'];

FilteringEdges

Itispossibletoexcludecertainedgesfromthetraversal.Tofilteronedges,afilterfunctioncanbedefinedviatheexpandFilterattribute.TheexpandFilterisafunctionwhichiscalledforeachedgeduringatraversal.

Itwillreceivethecurrentedge(edgevariable)andthevertexwhichtheedgeconnectsto(inthedirectionofthetraversal).Italsoreceivesthecurrentpathfromthestartvertexuptothecurrentvertex(excludingthecurrentedgeandthevertextheedgepointsto).

Ifthefunctionreturnstrue,theedgewillbefollowed.Ifthefunctionreturnsfalse,theedgewillnotbefollowed.Hereisaverysimplecustomedgefilterfunctionimplementation,whichsimplyincludesedgesifthe(edges)pathlengthislessthan1,andwillexcludeanyotheredges.Thiswilleffectivelyterminatethetraversalafterthefirstlevelofedges:

varconfig={

...

expandFilter:function(config,vertex,edge,path){

return(path.edges.length<1);

}

};

WritingCustomExpanders

Theedgesconnectedtoavertexaredeterminedbytheexpander.Sofarwehaveusedadefaultexpander(thedefaultinboundexpandertobeprecise).Thedefaultinboundexpandersimplyenumeratesallconnectedingoingedgesforavertex,basedontheedgecollectionspecifiedinthetraversalconfiguration.

Thereisalsoadefaultoutboundexpander,whichwillenumerateallconnectedoutgoingedges.Finally,thereisananyexpander,whichwillfollowbothingoingandoutgoingedges.

Ifconnectededgesmustbedeterminedinsomedifferentfashionforwhateverreason,acustomexpandercanbewrittenandregisteredbysettingtheexpanderattributeoftheconfiguration.Theexpanderfunctionsignatureisasfollows:

UsingTraversalObjects

168

varconfig={

...

expander:function(config,vertex,path){...}

}

Itistheexpander'sresponsibilitytoreturnalledgesandverticesdirectlyconnectedtothecurrentvertex(whichispassedviathevertexvariable).Thefullpathfromthestartvertexuptothecurrentvertexisalsosuppliedviathepathvariable.Anexpanderisexpectedtoreturnanarrayofobjects,whichneedtohaveanedgeandavertexattributeeach.

Note:Ifyouwanttorelyonaparticularorderinwhichtheedgesaretraversed,youhavetosorttheedgesreturnedbyyourexpanderwithinthecodeoftheexpander.Thefunctionstogetoutbound,inboundoranyedgesfromavertexdonotguaranteeanyparticularorder!

Acustomimplementationofaninboundexpandercouldlooklikethis(thisisanon-deterministicexpander,whichrandomlydecideswhetherornottoincludeconnectededges):

varconfig={

...

expander:function(config,vertex,path){

varconnected=[];

vardatasource=config.datasource;

datasource.getInEdges(vertex._id).forEach(function(edge){

if(Math.random()>=0.5){

connected.push({edge:edge,vertex:(edge._from)});

}

});

returnconnected;

}

};

Acustomexpandercanalsobeusedasanedgefilterbecauseithasfullcontroloverwhichedgeswillbereturned.

Followingaretwoexamplesofcustomexpandersthatpickedgesbasedonattributesoftheedgesandtheconnectedvertices.

Findingtheconnectededges/verticesbasedonanattributewhenintheconnectedvertices.Thegoalistofollowtheedgethatleadstothevertexwiththehighestvalueinthewhenattribute:

varconfig={

...

expander:function(config,vertex,path){

vardatasource=config.datasource;

//determinealloutgoingedges

varoutEdges=datasource.getOutEdges(vertex);

if(outEdges.length===0){

return[];

}

vardata=[];

outEdges.forEach(function(edge){

data.push({edge:edge,vertex:datasource.getInVertex(edge)});

});

//sortoutgoingverticesaccordingto"when"attributevalue

data.sort(function(l,r){

if(l.vertex.when===r.vertex.when){

return0;

}

return(l.vertex.when<r.vertex.when?1:-1);

});

//pickfirstvertexfound(withhighest"when"attributevalue)

return[data[0]];

}

...

};

Findingtheconnectededges/verticesbasedonanattributewhenintheedgeitself.Thegoalistopicktheoneedge(outofpotentiallymany)thathasthehighestwhenattributevalue:

UsingTraversalObjects

169

varconfig={

...

expander:function(config,vertex,path){

vardatasource=config.datasource;

//determinealloutgoingedges

varoutEdges=datasource.getOutEdges(vertex);

if(outEdges.length===0){

return[];//returnanemptyarray

}

//sortalloutgoingedgesaccordingto"when"attribute

outEdges.sort(function(l,r){

if(l.when===r.when){

return0;

}

return(l.when<r.when?-1:1);

});

//returnfirstedge(theonewithhighest"when"value)

varedge=outEdges[0];

try{

varv=datasource.getInVertex(edge);

return[{edge:edge,vertex:v}];

}

catch(e){}

return[];

}

...

};

HandlingUniqueness

Graphsmaycontaincycles.Tobeontopofwhathappenswhenatraversalencountersavertexoranedgeithasalreadyvisited,thereareconfigurationoptions.

Thedefaultconfigurationistovisiteveryvertex,regardlessofwhetheritwasalreadyvisitedinthesametraversal.However,edgeswillbydefaultonlybefollowediftheyarenotalreadypresentinthecurrentpath.

Imaginethefollowinggraphwhichcontainsacycle:

A->B->C->A

WhenthetraversalfindstheedgefromCtoA,itwillbydefaultfollowit.Thisisbecausewehavenotseenthisedgeyet.ItwillalsovisitvertexAagain.Thisisbecausebydefaultallverticeswillbevisited,regardlessofwhetheralreadyvisitedornot.

However,thetraversalwillnotagainfollowingtheoutgoingedgefromAtoB.ThisisbecausewealreadyhavetheedgefromAtoBinourcurrentpath.

Thesedefaultsettingswillpreventinfinitetraversals.

Toadjusttheuniquenessforvisitingvertices,therearethefollowingoptionsforuniqueness.vertices:

"none":alwaysvisitavertices,regardlessofwhetheritwasalreadyvisitedornot"global":visitavertexonlyifitwasnotvisitedinthetraversal"path":visitavertexifitisnotincludedinthecurrentpath

Toadjusttheuniquenessforfollowingedges,therearethefollowingoptionsforuniqueness.edges:

"none":alwaysfollowanedge,regardlessofwhetheritwasfollowedbefore"global":followanedgeonlyifitwasn'tfollowedinthetraversal"path":followanedgeifitisnotincludedinthecurrentpath

Notethatuniquenesscheckingwillhavesomeeffectonbothruntimeandmemoryusage.Forexample,whenuniquenesschecksaresetto"global" ,arraysofvisitedverticesandedgesmustbekeptinmemorywhilethetraversalisexecuted.Globaluniquenessshouldthusonlybeusedwhenatraversalisexpectedtovisitfewnodes.

UsingTraversalObjects

170

Intermsofruntime,turningoffuniquenesschecks(bysettingbothoptionsto"none")isthebestchoice,butitisonlysafeforgraphsthatdonotcontaincycles.Whenuniquenesschecksaredeactivatedinagraphwithcycles,thetraversalmightnotabortinasensibleamountoftime.

Optimizations

Thereareafewoptionsformakingatraversalrunfaster.

Thebestoptionistomaketheamountofvisitedverticesandfollowededgesassmallaspossible.Thiscanbeachievedbywritingcustomfilterandexpanderfunctions.Suchfunctionsshouldonlyincludeverticesofinterest,andonlyfollowedgesthatmightbeinteresting.

TraversaldepthcanalsobeboundedwiththeminDepthandmaxDepthoptions.

Anotherwaytospeeduptraversalsistowriteacustomvisitorfunction.Thedefaultvisitorfunction(trackingVisitor)willcopyeveryvisitedvertexintotheresult.Ifverticescontainlotsofdata,thismightbeexpensive.Itisthereforerecommendedtoonlycopysuchdataintotheresultthatisactuallyneeded.Thedefaultvisitorfunctionwillalsocopythefullpathtothevisiteddocumentintotheresult.Thisisevenmoreexpensiveandshouldbeavoidedifpossible.

Ifthegoalofatraversalistoonlycountthenumberofvisitedvertices,theprefabcountingVisitorwillbemuchmoreefficientthanthedefaultvisitor.

Forgraphsthatareknowntonotcontainanycycles,uniquenesschecksshouldbeturnedoff.Thiscanachievedviatheuniquenessconfigurationoptions.Notethatuniquenesschecksshouldnotbeturnedoffforgraphsthatareknowncontaincyclesorifthereisnoinformationaboutthegraph'sstructure.

Bydefault,atraversalwillonlyprocessalimitednumberofvertices.Thisisprotecttheuserfromunintentionallyrunanever-endingtraversalonagraphwithcyclicdata.HowmanyverticeswillbeprocessedatmostisdeterminedbythemaxIterationsconfigurationoption.IfatraversalhitsthecapspecifiedbymaxIterations,itwillabortandthrowatoomanyiterationsexception.Ifthiserrorisencountered,themaxIterationsvalueshouldbeincreasedifitismadesurethattheothertraversalconfigurationparametersaresaneandthetraversalwillabortnaturallyatsomepoint.

Finally,thebuildVerticesconfigurationoptioncanbesettofalsetoavoidlookingupandfullyconstructingvertexdata.Ifallthat'sneededfromverticesarethe_idor_keyattributes,thebuildverticesoptioncanbesettofalse.Ifvisitor,filterorexpandFilterfunctionsneedtoaccessothervertexattributes,theoptionshouldnotbechanged.

ConfigurationOverview

Thissectionsummarizestheconfigurationattributesforthetraversalobject.Theconfigurationcanconsistofthefollowingattributes:

visitor:visitorfunctionforvertices.Itwillbecalledforallnon-excludedvertices.Thegeneralvisitorfunctionsignatureisfunction(config,result,vertex,path).Ifthetraversalorderispreorder-expander,theconnectingedgesofthevisitedvertexwillbepassedasthefifthparameter,extendingthefunctionsignatureto:function(config,result,vertex,path,edges).

Visitorfunctionsarenotexpectedtoreturnvalues,buttheymaymodifytheresultvariableasneeded(e.g.bypushingvertexdataintotheresult).

expander:expanderfunctionthatisresponsibleforreturningedgesandverticesdirectlyconnectedtoavertex.Thefunctionsignatureisfunction(config,vertex,path).Theexpanderfunctionisrequiredtoreturnanarrayofconnectionobjects,consistingofanedgeandvertexattributeeach.Iftherearenoconnectingedges,theexpanderisexpectedtoreturnanemptyarray.filter:vertexfilterfunction.Thefunctionsignatureisfunction(config,vertex,path).Itmayreturnoneofthefollowingvalues:

undefined:vertexwillbeincludedintheresultandconnectededgeswillbetraversed"exclude":vertexwillnotbeincludedintheresultandconnectededgeswillbetraversed"prune":vertexwillbeincludedintheresultbutconnectededgeswillnotbetraversed["prune" ,"exclude" ]:vertexwillnotbeincludedintheresultandconnectededgeswillnotbereturned

expandFilter:filterfunctionappliedoneachedge/vertexcombinationdeterminedbytheexpander.Thefunctionsignatureisfunction(config,vertex,edge,path).Thefunctionshouldreturntrueiftheedge/vertexcombinationshouldbeprocessed,andfalseifitshouldbeignored.sort:afilterfunctiontodeterminetheorderinwhichconnectededgesareprocessed.Thefunctionsignatureisfunction(l,r).Thefunctionisrequiredtoreturnoneofthefollowingvalues:

-1iflshouldhaveasortvaluelessthanr

UsingTraversalObjects

171

1iflshouldhaveahighersortvaluethanr0iflandrhavethesamesortvalue

strategy:determinesthevisitationstrategy.Possiblevaluesaredepthfirstandbreadthfirst.order:determinesthevisitationorder.Possiblevaluesarepreorder,postorder,andpreorder-expander.preorder-expanderisthesameaspreorder,exceptthatthesignatureofthevisitorfunctionwillchangeasdescribedabove.itemOrder:determinestheorderinwhichconnectionsreturnedbytheexpanderwillbeprocessed.Possiblevaluesareforwardandbackward.maxDepth:ifsettoavaluegreaterthan0,thiswilllimitthetraversaltothismaximumdepth.minDepth:ifsettoavaluegreaterthan0,allverticesfoundonalevelbelowtheminDepthlevelwillnotbeincludedintheresult.maxIterations:themaximumnumberofiterationsthatthetraversalisallowedtoperform.Itissensibletosetthisnumbersounboundedtraversalswillterminateatsomepoint.uniqueness:anobjectthatdefineshowrepeatedvisitationsofverticesshouldbehandled.Theuniquenessobjectcanhaveasub-attributevertices,andasub-attributeedges.Eachsub-attributecanhaveoneofthefollowingvalues:

"none":nouniquenessconstraints"path":elementisexcludedifitisalreadycontainedinthecurrentpath.Thissettingmaybesensibleforgraphsthatcontaincycles(e.g.A→B→C→A)."global":elementisexcludedifitwasalreadyfound/visitedatanypointduringthetraversal.

buildVertices:thisattributecontrolswhetherverticesencounteredduringthetraversalwillbelookedupinthedatabaseandwillbemadeavailabletovisitor,filter,andexpandFilterfunctions.Bydefault,verticeswillbelookedupandmadeavailable.However,therearesomespecialusecaseswhenfullyconstructingvertexobjectsisnotnecessaryandcanbeavoided.Forexample,ifatraversalismeanttoonlycountthenumberofvisitedverticesbutdonotreadanydatafromvertices,thisoptionmightbesettotrue.

UsingTraversalObjects

172

ExampleDataThefollowingexamplesalluseavertexcollectionvandanedgecollectione.Thevertexcollectionvcontainscontinents,countries,andcapitals.Theedgecollectionecontainsconnectionsbetweencontinentsandcountries,andbetweencountriesandcapitals.

Tosetupthecollectionsandpopulatethemwithinitialdata,thefollowingscriptwasused:

db._create("v");

db._createEdgeCollection("e");

//vertices:rootnode

db.v.save({_key:"world",name:"World",type:"root"});

//vertices:continents

db.v.save({_key:"continent-africa",name:"Africa",type:"continent"});

db.v.save({_key:"continent-asia",name:"Asia",type:"continent"});

db.v.save({_key:"continent-australia",name:"Australia",type:"continent"});

db.v.save({_key:"continent-europe",name:"Europe",type:"continent"});

db.v.save({_key:"continent-north-america",name:"NorthAmerica",type:"continent"});

db.v.save({_key:"continent-south-america",name:"SouthAmerica",type:"continent"});

//vertices:countries

db.v.save({_key:"country-afghanistan",name:"Afghanistan",type:"country",code:"AFG"});

db.v.save({_key:"country-albania",name:"Albania",type:"country",code:"ALB"});

db.v.save({_key:"country-algeria",name:"Algeria",type:"country",code:"DZA"});

db.v.save({_key:"country-andorra",name:"Andorra",type:"country",code:"AND"});

db.v.save({_key:"country-angola",name:"Angola",type:"country",code:"AGO"});

db.v.save({_key:"country-antigua-and-barbuda",name:"AntiguaandBarbuda",type:"country",code:"ATG"});

db.v.save({_key:"country-argentina",name:"Argentina",type:"country",code:"ARG"});

db.v.save({_key:"country-australia",name:"Australia",type:"country",code:"AUS"});

db.v.save({_key:"country-austria",name:"Austria",type:"country",code:"AUT"});

db.v.save({_key:"country-bahamas",name:"Bahamas",type:"country",code:"BHS"});

db.v.save({_key:"country-bahrain",name:"Bahrain",type:"country",code:"BHR"});

db.v.save({_key:"country-bangladesh",name:"Bangladesh",type:"country",code:"BGD"});

db.v.save({_key:"country-barbados",name:"Barbados",type:"country",code:"BRB"});

db.v.save({_key:"country-belgium",name:"Belgium",type:"country",code:"BEL"});

db.v.save({_key:"country-bhutan",name:"Bhutan",type:"country",code:"BTN"});

db.v.save({_key:"country-bolivia",name:"Bolivia",type:"country",code:"BOL"});

db.v.save({_key:"country-bosnia-and-herzegovina",name:"BosniaandHerzegovina",type:"country",code:"BIH"});

db.v.save({_key:"country-botswana",name:"Botswana",type:"country",code:"BWA"});

db.v.save({_key:"country-brazil",name:"Brazil",type:"country",code:"BRA"});

db.v.save({_key:"country-brunei",name:"Brunei",type:"country",code:"BRN"});

db.v.save({_key:"country-bulgaria",name:"Bulgaria",type:"country",code:"BGR"});

db.v.save({_key:"country-burkina-faso",name:"BurkinaFaso",type:"country",code:"BFA"});

db.v.save({_key:"country-burundi",name:"Burundi",type:"country",code:"BDI"});

db.v.save({_key:"country-cambodia",name:"Cambodia",type:"country",code:"KHM"});

db.v.save({_key:"country-cameroon",name:"Cameroon",type:"country",code:"CMR"});

db.v.save({_key:"country-canada",name:"Canada",type:"country",code:"CAN"});

db.v.save({_key:"country-chad",name:"Chad",type:"country",code:"TCD"});

db.v.save({_key:"country-chile",name:"Chile",type:"country",code:"CHL"});

db.v.save({_key:"country-colombia",name:"Colombia",type:"country",code:"COL"});

db.v.save({_key:"country-cote-d-ivoire",name:"Coted'Ivoire",type:"country",code:"CIV"});

db.v.save({_key:"country-croatia",name:"Croatia",type:"country",code:"HRV"});

db.v.save({_key:"country-czech-republic",name:"CzechRepublic",type:"country",code:"CZE"});

db.v.save({_key:"country-denmark",name:"Denmark",type:"country",code:"DNK"});

db.v.save({_key:"country-ecuador",name:"Ecuador",type:"country",code:"ECU"});

db.v.save({_key:"country-egypt",name:"Egypt",type:"country",code:"EGY"});

db.v.save({_key:"country-eritrea",name:"Eritrea",type:"country",code:"ERI"});

db.v.save({_key:"country-finland",name:"Finland",type:"country",code:"FIN"});

db.v.save({_key:"country-france",name:"France",type:"country",code:"FRA"});

db.v.save({_key:"country-germany",name:"Germany",type:"country",code:"DEU"});

db.v.save({_key:"country-people-s-republic-of-china",name:"People'sRepublicofChina",type:"country",code:"CHN"});

//vertices:capitals

db.v.save({_key:"capital-algiers",name:"Algiers",type:"capital"});

db.v.save({_key:"capital-andorra-la-vella",name:"AndorralaVella",type:"capital"});

db.v.save({_key:"capital-asmara",name:"Asmara",type:"capital"});

db.v.save({_key:"capital-bandar-seri-begawan",name:"BandarSeriBegawan",type:"capital"});

db.v.save({_key:"capital-beijing",name:"Beijing",type:"capital"});

db.v.save({_key:"capital-berlin",name:"Berlin",type:"capital"});

db.v.save({_key:"capital-bogota",name:"Bogota",type:"capital"});

db.v.save({_key:"capital-brasilia",name:"Brasilia",type:"capital"});

ExampleData

173

db.v.save({_key:"capital-bridgetown",name:"Bridgetown",type:"capital"});

db.v.save({_key:"capital-brussels",name:"Brussels",type:"capital"});

db.v.save({_key:"capital-buenos-aires",name:"BuenosAires",type:"capital"});

db.v.save({_key:"capital-bujumbura",name:"Bujumbura",type:"capital"});

db.v.save({_key:"capital-cairo",name:"Cairo",type:"capital"});

db.v.save({_key:"capital-canberra",name:"Canberra",type:"capital"});

db.v.save({_key:"capital-copenhagen",name:"Copenhagen",type:"capital"});

db.v.save({_key:"capital-dhaka",name:"Dhaka",type:"capital"});

db.v.save({_key:"capital-gaborone",name:"Gaborone",type:"capital"});

db.v.save({_key:"capital-helsinki",name:"Helsinki",type:"capital"});

db.v.save({_key:"capital-kabul",name:"Kabul",type:"capital"});

db.v.save({_key:"capital-la-paz",name:"LaPaz",type:"capital"});

db.v.save({_key:"capital-luanda",name:"Luanda",type:"capital"});

db.v.save({_key:"capital-manama",name:"Manama",type:"capital"});

db.v.save({_key:"capital-nassau",name:"Nassau",type:"capital"});

db.v.save({_key:"capital-n-djamena",name:"N'Djamena",type:"capital"});

db.v.save({_key:"capital-ottawa",name:"Ottawa",type:"capital"});

db.v.save({_key:"capital-ouagadougou",name:"Ouagadougou",type:"capital"});

db.v.save({_key:"capital-paris",name:"Paris",type:"capital"});

db.v.save({_key:"capital-phnom-penh",name:"PhnomPenh",type:"capital"});

db.v.save({_key:"capital-prague",name:"Prague",type:"capital"});

db.v.save({_key:"capital-quito",name:"Quito",type:"capital"});

db.v.save({_key:"capital-saint-john-s",name:"SaintJohn's",type:"capital"});

db.v.save({_key:"capital-santiago",name:"Santiago",type:"capital"});

db.v.save({_key:"capital-sarajevo",name:"Sarajevo",type:"capital"});

db.v.save({_key:"capital-sofia",name:"Sofia",type:"capital"});

db.v.save({_key:"capital-thimphu",name:"Thimphu",type:"capital"});

db.v.save({_key:"capital-tirana",name:"Tirana",type:"capital"});

db.v.save({_key:"capital-vienna",name:"Vienna",type:"capital"});

db.v.save({_key:"capital-yamoussoukro",name:"Yamoussoukro",type:"capital"});

db.v.save({_key:"capital-yaounde",name:"Yaounde",type:"capital"});

db.v.save({_key:"capital-zagreb",name:"Zagreb",type:"capital"});

//edges:continent->world

db.e.save("v/continent-africa","v/world",{type:"is-in"});

db.e.save("v/continent-asia","v/world",{type:"is-in"});

db.e.save("v/continent-australia","v/world",{type:"is-in"});

db.e.save("v/continent-europe","v/world",{type:"is-in"});

db.e.save("v/continent-north-america","v/world",{type:"is-in"});

db.e.save("v/continent-south-america","v/world",{type:"is-in"});

//edges:country->continent

db.e.save("v/country-afghanistan","v/continent-asia",{type:"is-in"});

db.e.save("v/country-albania","v/continent-europe",{type:"is-in"});

db.e.save("v/country-algeria","v/continent-africa",{type:"is-in"});

db.e.save("v/country-andorra","v/continent-europe",{type:"is-in"});

db.e.save("v/country-angola","v/continent-africa",{type:"is-in"});

db.e.save("v/country-antigua-and-barbuda","v/continent-north-america",{type:"is-in"});

db.e.save("v/country-argentina","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-australia","v/continent-australia",{type:"is-in"});

db.e.save("v/country-austria","v/continent-europe",{type:"is-in"});

db.e.save("v/country-bahamas","v/continent-north-america",{type:"is-in"});

db.e.save("v/country-bahrain","v/continent-asia",{type:"is-in"});

db.e.save("v/country-bangladesh","v/continent-asia",{type:"is-in"});

db.e.save("v/country-barbados","v/continent-north-america",{type:"is-in"});

db.e.save("v/country-belgium","v/continent-europe",{type:"is-in"});

db.e.save("v/country-bhutan","v/continent-asia",{type:"is-in"});

db.e.save("v/country-bolivia","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-bosnia-and-herzegovina","v/continent-europe",{type:"is-in"});

db.e.save("v/country-botswana","v/continent-africa",{type:"is-in"});

db.e.save("v/country-brazil","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-brunei","v/continent-asia",{type:"is-in"});

db.e.save("v/country-bulgaria","v/continent-europe",{type:"is-in"});

db.e.save("v/country-burkina-faso","v/continent-africa",{type:"is-in"});

db.e.save("v/country-burundi","v/continent-africa",{type:"is-in"});

db.e.save("v/country-cambodia","v/continent-asia",{type:"is-in"});

db.e.save("v/country-cameroon","v/continent-africa",{type:"is-in"});

db.e.save("v/country-canada","v/continent-north-america",{type:"is-in"});

db.e.save("v/country-chad","v/continent-africa",{type:"is-in"});

db.e.save("v/country-chile","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-colombia","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-cote-d-ivoire","v/continent-africa",{type:"is-in"});

db.e.save("v/country-croatia","v/continent-europe",{type:"is-in"});

db.e.save("v/country-czech-republic","v/continent-europe",{type:"is-in"});

db.e.save("v/country-denmark","v/continent-europe",{type:"is-in"});

db.e.save("v/country-ecuador","v/continent-south-america",{type:"is-in"});

ExampleData

174

db.e.save("v/country-egypt","v/continent-africa",{type:"is-in"});

db.e.save("v/country-eritrea","v/continent-africa",{type:"is-in"});

db.e.save("v/country-finland","v/continent-europe",{type:"is-in"});

db.e.save("v/country-france","v/continent-europe",{type:"is-in"});

db.e.save("v/country-germany","v/continent-europe",{type:"is-in"});

db.e.save("v/country-people-s-republic-of-china","v/continent-asia",{type:"is-in"});

//edges:capital->country

db.e.save("v/capital-algiers","v/country-algeria",{type:"is-in"});

db.e.save("v/capital-andorra-la-vella","v/country-andorra",{type:"is-in"});

db.e.save("v/capital-asmara","v/country-eritrea",{type:"is-in"});

db.e.save("v/capital-bandar-seri-begawan","v/country-brunei",{type:"is-in"});

db.e.save("v/capital-beijing","v/country-people-s-republic-of-china",{type:"is-in"});

db.e.save("v/capital-berlin","v/country-germany",{type:"is-in"});

db.e.save("v/capital-bogota","v/country-colombia",{type:"is-in"});

db.e.save("v/capital-brasilia","v/country-brazil",{type:"is-in"});

db.e.save("v/capital-bridgetown","v/country-barbados",{type:"is-in"});

db.e.save("v/capital-brussels","v/country-belgium",{type:"is-in"});

db.e.save("v/capital-buenos-aires","v/country-argentina",{type:"is-in"});

db.e.save("v/capital-bujumbura","v/country-burundi",{type:"is-in"});

db.e.save("v/capital-cairo","v/country-egypt",{type:"is-in"});

db.e.save("v/capital-canberra","v/country-australia",{type:"is-in"});

db.e.save("v/capital-copenhagen","v/country-denmark",{type:"is-in"});

db.e.save("v/capital-dhaka","v/country-bangladesh",{type:"is-in"});

db.e.save("v/capital-gaborone","v/country-botswana",{type:"is-in"});

db.e.save("v/capital-helsinki","v/country-finland",{type:"is-in"});

db.e.save("v/capital-kabul","v/country-afghanistan",{type:"is-in"});

db.e.save("v/capital-la-paz","v/country-bolivia",{type:"is-in"});

db.e.save("v/capital-luanda","v/country-angola",{type:"is-in"});

db.e.save("v/capital-manama","v/country-bahrain",{type:"is-in"});

db.e.save("v/capital-nassau","v/country-bahamas",{type:"is-in"});

db.e.save("v/capital-n-djamena","v/country-chad",{type:"is-in"});

db.e.save("v/capital-ottawa","v/country-canada",{type:"is-in"});

db.e.save("v/capital-ouagadougou","v/country-burkina-faso",{type:"is-in"});

db.e.save("v/capital-paris","v/country-france",{type:"is-in"});

db.e.save("v/capital-phnom-penh","v/country-cambodia",{type:"is-in"});

db.e.save("v/capital-prague","v/country-czech-republic",{type:"is-in"});

db.e.save("v/capital-quito","v/country-ecuador",{type:"is-in"});

db.e.save("v/capital-saint-john-s","v/country-antigua-and-barbuda",{type:"is-in"});

db.e.save("v/capital-santiago","v/country-chile",{type:"is-in"});

db.e.save("v/capital-sarajevo","v/country-bosnia-and-herzegovina",{type:"is-in"});

db.e.save("v/capital-sofia","v/country-bulgaria",{type:"is-in"});

db.e.save("v/capital-thimphu","v/country-bhutan",{type:"is-in"});

db.e.save("v/capital-tirana","v/country-albania",{type:"is-in"});

db.e.save("v/capital-vienna","v/country-austria",{type:"is-in"});

db.e.save("v/capital-yamoussoukro","v/country-cote-d-ivoire",{type:"is-in"});

db.e.save("v/capital-yaounde","v/country-cameroon",{type:"is-in"});

db.e.save("v/capital-zagreb","v/country-croatia",{type:"is-in"});

ExampleData

175

Edges,Identifiers,HandlesThisisanintroductiontoArangoDB'sinterfaceforedges.Edgesmaybeusedingraphs.HereweworkwithedgesfromtheJavaScriptshellarangosh.ForotherlanguagesseethecorrespondinglanguageAPI.

Agraphdatamodelalwaysconsistsofatleasttwocollections:therelationsbetweenthenodesinthegraphsarestoredinan"edgescollection",thenodesinthegrapharestoredindocumentsinregularcollections.

EdgesinArangoDBarespecialdocuments.Inadditiontothesystemattributes_key,_idand_rev,theyhavetheattributes_fromand_to,whichcontaindocumenthandles,namelythestart-pointandtheend-pointoftheedge.

Example:

the"edge"collectionstorestheinformationthatacompany'sreceptionissub-unittotheservicesunitandtheservicesunitissub-unittotheCEO.Youwouldexpressthisrelationshipwiththe_fromand_toattributesthe"normal"collectionstoresallthepropertiesaboutthereception,e.g.that20peopleareworkingthereandtheroomnumberetc_fromisthedocumenthandleofthelinkedvertex(incomingrelation)_toisthedocumenthandleofthelinkedvertex(outgoingrelation)

Edgecollectionsarespecialcollectionsthatstoreedgedocuments.Edgedocumentsareconnectiondocumentsthatreferenceotherdocuments.Thetypeofacollectionmustbespecifiedwhenacollectioniscreatedandcannotbechangedafterwards.

Tochangeedgeendpointsyouwouldneedtoremoveolddocument/edgeandinsertnewone.Otherfieldscanbeupdatedasindefaultcollection.

WorkingwithEdges

Edgesarenormaldocumentsthatalwayscontaina_fromanda_toattribute.

WorkingwithEdges

176

DistributedIterativeGraphProcessing(Pregel)Distributedgraphprocessingenablesyoutodoonlineanalyticalprocessingdirectlyongraphsstoredintoarangodb.Thisisintendedtohelpyougainanalyticalinsightsonyourdata,withouthavingtouseexternalprocessingsytems.ExamplesofalgorithmstoexecutearePageRank,VertexCentrality,VertexCloseness,ConnectedComponents,CommunityDetection.Thissystemisnotusefulfortypicalonlinequeries,whereyoujustdoworkonasmallsetofvertices.ThesekindoftasksarebettersuitedforAQL.

TheprocessingsysteminsideArangoDBisbasedon:Pregel:ASystemforLarge-ScaleGraphProcessing–Malewiczetal.(Google)2010Thisconceptenablesustoperformdistributedgraphprocessing,withouttheneedfordistributedgloballocking.

PrerequisitesIfyouarerunningasingleArangoDBinstanceinsingle-servermode,therearenorequirementsregardingthemodelingofyourdata.Allyouneedisatleastonevertexcollectionandoneedgecollection.Notethattheperformancemaybebetter,ifthenumberofyourshards/collectionsmatchesthenumberofCPUcores.

WhenyouuseArangoDBCommunityeditioninclustermode,youmightneedtomodelyourcollectionsinacertainwaytoensurecorrectresults.Formoreinformationseethenextsection.

RequirementsforCollectionsinaCluster(NonSmartGraph)Toenableiterativegraphprocessingforyourdata,youwillneedtoensurethatyourvertexandedgecollectionsareshardedinaspecificway.

ThepregelcomputingmodelrequiresalledgestobepresentontheDBServerwherethevertexdocumentidentifiedbythe_fromvalueislocated.Thismeansthevertexcollectionsneedtobeshardedby'_key'andtheedgecollectionwillneedtobeshardedafteranattributewhichalwayscontainsthe'_key'ofthevertex.

Ourimplementationcurrentlyrequireseveryedgecollectiontobeshardedaftera"vertex"attributes,additionallyyouwillneedtospecifythekeydistributeShardsLikeandanequalnumberofshardsoneverycollection.OnlyiftheserequirementsaremetcanArangoDBplacetheedgesandverticescorrectly.

Forexampleyoumightcreateyourcollectionslikethis:

//Createmainvertexcollection:

db._create("vertices",{

shardKeys:['_key'],

numberOfShards:8

});

//Optionallycreatearbitraryadditionalvertexcollections

db._create("additonal",{

distributeShardsLike:"vertices",

numberOfShards:8

});

//Create(oneormore)edge-collections:

db._createEdgeCollection("edges",{

shardKeys:['vertex'],

distributeShardsLike:"vertices",

numberOfShards:8

});

Youwillneedtoensurethatedgedocumentscontainthepropervaluesintheirshardingattribute.Foravertexdocumentwiththefollowingcontent{_key:"A",value:0}thecorrespondingedgedocumentswouldhavelooklikethis:

{_from:"vertices/A",_to:"vertices/B",vertex:"A"}

{_from:"vertices/A",_to:"vertices/C",vertex:"A"}

{_from:"vertices/A",_to:"vertices/D",vertex:"A"}

...

Pregel

177

ThiswillensurethatoutgoingedgedocumentswillbeplacedonthesameDBServerasthevertex.Withoutthecorrectplacementoftheedges,thepregelgraphprocessingsystemwillnotworkcorrectly,becauseedgeswillnotloadcorrectly.

ArangoshAPI

StartinganAlgorithmExecution

ThepregelAPIisaccessiblethroughthe@arangodb/pregelpackage.Tostartanexecutionyouneedtospecifythealgorithmnameandthevertexandedgecollections.Alternativelyyoucanspecifyanamedgraph.Additionallyyoucanspecifycustomparameterswhichvaryforeachalgorithm.ThestartmethodwillalwaysauniqueIDwhichcanbeusedtointeractwiththealgorithmandlateron.

Thebelowversionofthestartmethodcanbeusedfornamedgraphs:

varpregel=require("@arangodb/pregel");

varparams={};

varexecution=pregel.start("<algorithm>","<yourgraph>",params);

Paramsneedstobeanobject,thevalidkeysarementionedbelowinthesectionAlgorithms

Alternativelyyoumightwanttospecifythevertexandedgecollectionsdirectly.Thecall-syntaxofthestart``methodchangesinthiscase.ThesecondargumentmustbeanobjectwiththekeysvertexCollectionsandedgeCollections`.

varpregel=require("@arangodb/pregel");

varparams={};

varexecution=pregel.start("<algorithm>",{vertexCollections:["vertices"],edgeCollections:["edges"]},{});

Thelastargumentisstilltheparameterobject.Seebelowforalistofalgorithmsandparameters.

StatusofanAlgorithmExecution

Thecodereturnedbythepregel.start(...)methodcanbeusedtotrackthestatusofyouralgorithm.

varexecution=pregel.start("sssp","demograph",{source:"vertices/V"});

varstatus=pregel.status(execution);

Theresultwilltellyouthecurrentstatusofthealgorithmexecution.Itwilltellyouthecurrentstateoftheexecution,thecurrentglobalsuperstep,theruntime,theglobalaggregatorvaluesaswellasthenumberofsendandreceivedmessages.

Validvaluesforthestatefieldinclude:

"running"algorithmisstillrunning"done":Theexecutionisdone,theresultmightnotbewrittenbackintothecollectionyet."canceled":Theexecutionwaspermanentlycanceled,eitherbytheuserorbyanerror."inerror":Theexeuctionisinanerrorstate.ThiscanbecausedbyprimaryDBServersbeingnotreachableorbeingnonresponsive.Theexecutionmightrecoverlater,orswitchto"canceled"ifitwasnotabletorecoversuccessfuly"recovering":Theexecutionisactivelyrecovering,willswitchbackto"running"iftherecoverywassuccessful

Theobjectreturnedbythestatusmethodmightforexamplelooksomethinglikethis:

{

"state":"running",

"gss":12,

"totalRuntime":123.23,

"aggregators":{

"converged":false,

"max":true,

"phase":2

},

"sendCount":3240364978,

"receivedCount":3240364975

}

Pregel

178

CancelinganExecution/Discardingresults

Tocancelanexecutionwhichisstillrunnning,anddiscardanyintermediareresultsyoucanusethecancelmethod.Thiswillimmediatlyfreeallmemorytakenupbytheexecution,andwillmakeyouloseallintermediarydata.

Youmightgetinconsistentresultsifyoucancelanexecutionwhileitisalreadyinit'sdonestate.Thedataiswrittenmulti-threadedintoallcollectionshardsatonce,thismeanstherearemultipletransactionssimultaniously.Atransactionmightalreadybecommitedwhenyoucanceltheexecutionjob,thereforeyoumightseetheresultinyourcollection.Thisdoesnotapplyifyouconfiguredtheexecutiontonotwritedataintothecollection.

//startasinglesourceshortestpathjob

varexecution=pregel.start("sssp","demograph",{source:"vertices/V"});

pregel.cancel(execution);

AQLintegrationArangoDBsupportsretrievingtemporarypregelresultsthroughtheArangoDBquerylanguage(AQL).Whenourgraphprocessingsubsystemfinishesexecutinganalgorithm,theresultcaneitherbewrittenbackintothedatabaseorkeptinmemory.InbothcasestheresultcanbequeriedviaAQL.Ifthedatawasnotwrittentothedatabasestoreitisonlyheldtemporarily,untiltheusercallsthecancelmethodForexampleausermightwanttoqueryonlynodeswiththemostrankfromtheresultsetofaPageRankexecution.

FORvINPREGEL_RESULT(<handle>)

FILTERv.value>=0.01

RETURNv._key

AvailableAlgorithmsThereareanumberofgeneralparameterswhichapplytoalmostallalgorithms:

store:Isperdefaulttrue,thepregelenginewillwriteresultsbacktothedatabase.ifthevalueisfalsethenyoucanquerytheresultsviaAQLk,seeAQLintegration.maxGSS:Maximumnumberofglobaliterationsforthisalgorithmparallelism:Numberofparellelthreadstouseperworker.Doesnotinfluencethenumberofthreadsusedtoload

orstoredatafromthedatabase(thisdependsonthenumberofshards).

async:Algorithmswichsupportasyncmode,willrunwithoutsynchronizedglobaliterations,mightleadtoperformanceincreasesifyouhaveloadimbalances.resultField:Mostalgorithmswillwritetheresultintothisfield

PageRank

PageRankisawellknownalgorithmtorankdocumentsinagraph.Thealgorithmwillrununtiltheexecutionconverges.Specifyacustomthresholdwiththeparameterthreshold,torunforafixednumberofiterationsusethemaxGSSparameter.

varpregel=require("@arangodb/pregel");

pregel.start("pagerank","graphname",{maxGSS:100,threshold:0.00000001})

Single-SourceShortestPath

Calculatesthedistanceofeachvertextoacertainshortestpath.Thealgorithmwillrununtilitconverges,theiterationsareboundbythediameter(thelongestshortestpath)ofyourgraph.

varpregel=require("@arangodb/pregel");

pregel.start("sssp","graphname",{source:"vertices/1337"})

ConnectedComponents

Pregel

179

Therearetwoalgorithmstofindconnectedcomponentsinagraph.Tofindweaklyconnectedcomponents(WCC)youcanusethealgorithmnamed"connectedcomponents",tofindstronglyconnectedcomponents(SCC)youcanusethealgorithmnamed"scc".BothalgorithmwillassignacomponentIDtoeachvertex.

Aweaklyconnectedcomponentsmeansthatthereexistapathfromeveryvertexpairinthatcomponent.WCCisaverysimpleandfastalgorithm,whichwillonlyworkcorrectlyonundirectedgraphs.Yourresultsondirectedgraphsmayvary,dependingonhowconnectedyourcomponentsare.

InthecaseofSCCacomponentmeanseveryvertexisreachablefromanyothervertexinthesamecomponent.ThealgorithmismorecomplexthantheWCCalgorithmandrequiresmoreRAM,becauseeachvertexneedstostoremuchmorestate.ConsiderusingWCCifyouthinkyourdatamaybesuitableforit.

varpregel=require("@arangodb/pregel");

//weaklyconnectedcomponents

pregel.start("connectedcomponents","graphname")

//stronglyconnectedcomponents

pregel.start("scc","graphname")

Hyperlink-InducedTopicSearch(HITS)

HITSisalinkanalysisalgorithmthatratesWebpages,developedbyJonKleinberg(Thealgorithmisalsoknownashubsandauthorities).

TheideabehindHubsandAuthoritiescomesfromthetypicalstructureoftheweb:Certainwebsitesknownashubs,serveaslargedirectoriesthatarenotactuallyauthoritativeontheinformationthattheyhold.Thesehubsareusedascompilationsofabroadcatalogofinformationthatleadsusersdirecttootherauthoritativewebpages.Thealgorithmassignseachvertextwoscores:Theauthority-scoreandthehub-score.Theauthorityscorerateshowmanygoodhubspointtoaparticularvertex(orwebpage),thehubscorerateshowgood(authoritative)theverticespointedtoare.Formoreseehttps://en.wikipedia.org/wiki/HITS_algorithm

Ourversionofthealgorithmconvergesafteracertainamountoftime.Theparameterthresholdcanbeusedtosetalimitfortheconvergence(measuredasmaximumabsolutedifferenceofthehubandauthorityscoresbetweenthecurrentandlastiteration)Whenyouspecifytheresultfieldname,thehubscorewillbestoredin"_hub"andtheauthorityscorein"_auth".Thealgorithmcanbeexecutedlikethis:

varpregel=require("@arangodb/pregel");

varhandle=pregel.start("hits","yourgraph",{threshold:0.00001,resultField:"score"});

VertexCentrality

Centralitymeasureshelpidentifythemostimportantverticesinagraph.Theycanbeusedinawiderangeofapplications:Forexampletheycanbeusedtoidentifyinfluencersinsocialnetworks,ormiddle-meninterroristnetworks.Therearevariousdefinitionsforcentrality,thesimplestonebeingthevertexdegree.Thesedefinitionswerenotdesignedwithscalabilityinmind.Itisprobablyimpossibletodiscoveranefficientalgorithmwhichcomputestheminadistributedway.Fortunatelytherearescalablesubstitutionsavailable,whichshouldbeequallyusableformostusecases.

Pregel

180

EffectiveCloseness

Acommondefinitionsofcentralityistheclosenesscentrality(orcloseness).Theclosenessofavertexinagraphistheinverseaveragelengthoftheshortestpathbetweenthevertexandallothervertices.Forverticesx,yandshortestdistanced(y,x)itisdefinedas

EffectiveClosenessapproximatestheclosenessmeasure.Thealgorithmworksbyiterativelyestimatingthenumberofshortestpathspassingthrougheachvertex.Thescorewillapproximatesthetherealclosenessscore,sinceitisnotpossibletoactuallycountallshortestpathsduetothehorrendousO(n^2d)memoryrequirements.ThealgorithmisfromthepaperCentralitiesinLargeNetworks:AlgorithmsandObservations(UKanget.al.2011)*

ArangoDBsimplementationapproximatesthenumberofshortestpathineachiterationbyusingaHyperLogLogcounterwith64buckets.Thisshouldworkwellonlargegraphsandonsmalleronesaswell.ThememoryrequirementsshouldbeO(n*d)wherenisthenumberofverticesanddthediameterofyourgraph.Eachvertexwillstoreacounterforeachiterationofthealgorithm.Thealgorithmcanbeusedlikethis

constpregel=require("@arangodb/pregel");

consthandle=pregel.start("effectivecloseness","yourgraph",{resultField:"closeness"});

LineRank

Anothercommonmeasureisthebetweenness*centrality:Itmeasuresthenumberoftimesavertexispartofshortestpathsbetweenanypairsofvertices.Foravertexvbetweennessisdefinedas

Wheretheσrepresentsthenumberofshortestpathsbetweenxandy,andσ(v)representsthenumberofpathsalsopassingthroughavertexv.Byintuitionavertexwithhigherbetweenesscentralitywillhavemoreinformationpassingthroughit.

LineRankapproximatestherandomwalkbetweennessofeveryvertexinagraph.Thisistheprobabilitythatsomeonestartingonanarbitaryvertex,willvisitthisnodewhenherandomlychoosesedgestovisit.Thealgoruthmessentiallybuildsalinegraphoutofyourgraph(switchestheverticesandedges),andthencomputesascoresimilartoPageRank.Thiscanbeconsideredascalableequivalenttovertexbetweeness,whichcanbeexecuteddistributedlyinArangoDB.ThealgorithmisfromthepaperCentralitiesinLargeNetworks:AlgorithmsandObservations(UKanget.al.2011)

Pregel

181

constpregel=require("@arangodb/pregel");

consthandle=pregel.start("linerank","yourgraph",{"resultField":"rank"});

CommunityDetection

Graphsbasedonrealworldnetworksoftenhaveacommunitystructure.Thismeansitispossibletofindgroupsofverticessuchthateacheachvertexgroupisinternallymoredenselyconnectedthanoutsidethegroup.Thishasmanyapplicationswhenyouwanttoanalyzeyournetworks,forexampleSocialnetworksincludecommunitygroups(theoriginoftheterm,infact)basedoncommonlocation,interests,occupation,etc.

LabelPropagation

LabelPropagationcanbeusedtoimplementcommunitydetectiononlargegraphs.Theideaisthateachvertexshouldbeinthecommunitythatmostofhisneighboursarein.WeiterativelydeteminethisbyfirstassigningrandomCommunityID's.Theneachitertation,avertexwillsendit'scurrentcommunityIDtoallhisneighborvertices.TheneachvertexadoptsthecommunityIDhereceivedmostfrequentlyduringtheiteration.

Thealgorithmrunsuntilitconverges,whichlikelyneverreallyhappensonlargegraphs.Thereforeyouneedtospecifyamaximumiterationboundwhichsuitsyou.Thedefaultboundis500iterations,whichislikelytoolargeforyourapplication.Shouldworkbestonundirectedgraphs,resultsondirectedgraphsmightvarydependingonthedensityofyourgraph.

constpregel=require("@arangodb/pregel");

consthandle=pregel.start("labelpropagation","yourgraph",{maxGSS:100,resultField:"community"});

Speaker-ListenerLabelPropagation

TheSpeaker-listenerLabelPropagation(SLPA)canbeusedtoimplementcommunitydetection.Itworkssimilartothelabelpropagationalgorithm,butnoweverynodeadditionallyaccumulatesamemoryofobservedlabels(insteadofforgettingallbutonelabel).

Beforethealgorithmrun,everyvertexisinitializedwithanuniqueID(theinitialcommunitylabel).Duringtherunthreestepsareexecutedforeachvertex:

1. Currentvertexisthelistenerallotherverticesarespeakers2. Eachspeakersendsoutalabelfrommemory,wesendoutarandomlabelwithaprobabilityproportionaltothenumberoftimesthe

vertexobservedthelabel3. Thelistenerremembersoneofthelabels,wealwayschoosethemostfrequentlyobservedlabel

constpregel=require("@arangodb/pregel");

consthandle=pregel.start("slpa","yourgraph",{maxGSS:100,resultField:"community"});

YoucanalsoexecuteSLPAwiththemaxCommunitiesparametertolimitthenumberofouputcommunities.Internallythealgorithmwillstillkeepthememoryofalllabels,buttheoutputisreducedtojusthenmostfrequentlyobservedlabels.

constpregel=require("@arangodb/pregel");

consthandle=pregel.start("slpa","yourgraph",{maxGSS:100,resultField:"community",maxCommunities:1});

//checkthestatusperiodicallyforcompletion

pregel.status(handle);

Pregel

182

ViewsavailableinArangoDBArangoSearch

Views

183

BringingthepowerofIResearchtoArangoDB

WhatisArangoSearch

ArangoSearchisanativelyintegratedAQLextensionmakinguseoftheIResearchlibrary.

Arangosearchallowsoneto:

joindocumentslocatedindifferentcollectionstooneresultlistfilterdocumentsbasedonAQLbooleanexpressionsandfunctionssorttheresultsetbasedonhowcloselyeachdocumentmatchedthefilter

Aconceptofvalue'analysis'thatismeanttobreakupagivenvalueintoasetofsub-valuesinternallytiedtogetherbymetadatawhichinfluencesboththefilterandsortstagestoprovidethemostappropriatematchforthespecifiedconditions,similartoqueriestowebsearchengines.

Inplaintermsthismeansausercanforexample:

requestdocumentswherethe'body'attributebestmatches'aquickbrownfox'requestdocumentswherethe'dna'attributebestmatchesaDNAsubsequencerequestdocumentswherethe'name'attributebestmatchesgenderetc...(viacustomanalyzersdescribedinthenextsection)

TheIResearchLibrary

IResearchsacross-platformopensourceindexingandsearchingenginewritteninC++,optimizedforspeedandmemoryfootprint,withsourceavailablefrom:https://github.com/iresearch-toolkit/iresearch

IResearchisaframeworkforindexing,filteringandsortingofdata.Theindexingstagecantreateachdataitemasanatomorusecustom'analyzers'tobreakthedataitemintosub-atomicpiecestiedtogetherwithinternallytrackedmetadata.

TheIResearchframeworkingeneralcanbefurtherextendedatruntimewithcustomimplementationsofanalyzers(usedduringtheindexingandfilteringstages)andscorers(usedduringthesortingstage)allowingfullcontroloverthebehaviouroftheengine.

ArangoSearchScorers:

ArangoSearchaccessesscorersdirectlybytheirinternalnames.Thename(inupper-case)ofthescoreristhefunctionnametobeusedinthe'SORT'section.Functionarguments,(excludingthefirstargument),areserializedasastringrepresentationofaJSONarrayandpasseddirectlytothecorrespondingscorer.ThefirstargumenttoanyscorerfunctionisthereferencetothecurrentdocumentemittedbytheFORstatement,i.e.itwouldbe'doc'forthisstatement:

FORdocINVIEWsomeView

IResearchprovidesa'bm25'scorerimplementingtheBM25algorithm.Thisscoreroptionallytakes'k'and'b'positionalparameters.

Theuserisabletorunquerieswiththesaidscorer,e.g.

SORTBM25(doc,1.2,0.75)

ThefunctionargumentswillthenbeserializedintoaJSONrepresentation:

[1.2,0.75]

andpassedtothescorerimplementation.

SimilarlyanadministratormaychoosetodeployacustomDNAanalyzer'DnaRank'.

Theuseristhenimmediatelyabletorunquerieswiththesaidscorer,e.g.

ArangoSearch

184

SORTDNARANK(doc,123,456,"abc",{"def","ghi"})

ThefunctionargumentswillthenbeserializedintoaJSONrepresentation:

[123,456,"abc",{"def","ghi"}]

andpassedtothescorerimplementation.

Runtime-pluggingfunctionalityforscoresisnotavaiableinArangoDBatthispointintime,soArangoDBcomeswithafewdefault-initializedscores:

attribute-nameorderresultsbasedonthevalueofattribute-name

BM25orderresultsbasedontheBM25algorithm

TFIDForderresultsbasedontheTFIDFalgorithm

ArangoSearchismuchmorethanafulltextsearch

Butfulltextsearchingisasubsetofitsavailablefunctionality,supportedviathe'text'analyzerand'tfidf'/'bm25'scorers,withoutimpacttoperformancewhenspecifyingdocumentsfromdifferentcollectionsorfilteringonmultipledocumentattributes.

Viewdatasource

TheIResearchfunctionalityisexposedtoArangoDBviathetheArangoSearchviewAPIbecausetheArangoSearchviewismerelyanidentitytransformationappliedontodocumentsstoredinlinkedcollectionsofthesameArangoDBdatabase.InplaintermsanArangoSearchviewonlyallowsfilteringandsortingofdocumentslocatedincollectionsofthesamedatabase.Thematchingdocumentsthemselvesarereturnedas-isfromtheircorrespondingcollections.

LinkstoArangoDBcollections

AconceptofanArangoDBcollection'link'isintroducedtoallowspecifyingwhichArangoDBcollectionsagivenArangoSearchViewshouldqueryfordocumentsandhowthesedocumentsshouldbequeried.

AnArangoSearchLinkisauni-directionalconnectionfromanArangoDBcollectiontoanArangoSearchviewdescribinghowdatacomingfromthesaidcollectionshouldbemadeavailableinthegivenview.EachArangoSearchLinkinanArangoSearchviewisuniquelyidentifiedbythenameoftheArangoDBcollectionitlinksto.AnArangoSearchviewmayhavezeroormorelinks,eachtoadistinctArangoDBcollection.SimilarlyanArangoDBcollectionmaybereferencedvialinksbyzeroormoredistinctArangoSearchviews.InplaintermsanygivenArangoSearchviewmaybelinkedtoanygivenArangoDBcollectionofthesamedatabasewithzerooratmostonelink.However,anyArangoSearchviewmaybelinkedtomultipledistinctArangoDBcollectionsandsimilarlyanyArangoDBcollectionmaybereferencedbymultipleArangoSearchviews.

ToconfigureanArangoSearchviewforconsiderationofdocumentsfromagivenArangoDBcollectionalinkdefinitionmustbeaddedtothepropertiesofthesaidArangoSearchviewdefiningthelinkparametersasperthesectionViewdefinition/modification.

Analyzers:

TosimplifyquerysyntaxArangoSearchprovidesaconceptofnamedanalyzerswhicharemerelyaliasesfortype+configurationofIResearchanalyzers.ManagementofnamedanalyzersisexposedviabothREST,GUIandJavaScriptAPIs,e.g.

Viewdefinition/modification

AnArangoSearchviewisconfiguredviaanobjectcontainingasetofview-specificconfigurationdirectivesandamapoflink-specificconfigurationdirectives.

Duringviewcreationthefollowingdirectivesapply:

id:(optional)thedesiredviewidentifiername:(required)theviewnametype:\thevalue"arangosearch"anyofthedirectivesfromthesectionViewproperties

ArangoSearch

185

Duringviewmodificationthefollowingdirectivesapply:

links:(optional)amappingofcollection-name/collection-identifiertooneof:linkcreation-linkdefinitionasperthesectionLinkpropertieslinkremoval-JSONkeywordnull(i.e.nullifyalinkifpresent)anyofthedirectivesfromthesectionmodifiableviewproperties

Viewproperties(modifiable)

commit:(optional;default:usedefaultsforallvalues)configureArangoSearchViewcommitpolicyforsingle-iteminserts/removals,e.g.whenaddingremovingdocumentsfromalinkedArangoDBcollection

cleanupIntervalStep:(optional;default:10;todisableuse:0)waitatleastthismanycommitsbetweenremovingunusedfilesintheArangoSearchdatadirectoryforthecasewheretheconsolidationpoliciesmergesegmentsoften(i.e.alotofcommit+consolidate),alowervaluewillcausealotofdiskspacetobewastedforthecasewheretheconsolidationpoliciesrarelymergesegments(i.e.fewinserts/deletes),ahighervaluewillimpactperformancewithoutanyaddedbenefits

commitIntervalMsec:(optional;default:60000;todisableuse:0)waitatleastcountmillisecondsbetweencommittingviewdatastorechangesandmakingdocumentsvisibletoqueriesforthecasewheretherearealotofinserts/updates,alowervaluewillcausetheviewnottoaccountforthem,(unlitcommit),andmemoryusagewouldcontinuetogrowforthecasewherethereareafewinserts/updates,ahighervaluewillimpactperformanceandwastediskspaceforeachcommitcallwithoutanyaddedbenefits

commitTimeoutMsec:(optional;default:5000;todisableuse:0)trytocommitasmuchaspossiblebeforecountmillisecondsforthecasewheretherearealotofinserts/updates,alowervaluewillcauseadelayintheviewaccountingforthem,dueskippingofsomecommitsforthecasewheretherearealotofinserts/updates,ahighervaluewillcausehighermemoryconsumptionbetweencommitsduetoaccumulationofdocumentmodificationswhileacommitisinprogress

consolidate:(optional;default:none)aper-policymappingofthresholdsintherange[0.0,1.0]todeterminedatastoresegmentmergecandidates,ifspecifiedthenonlythelistedpoliciesareused,keysareanyof:

bytes:(optional;fordefaultvaluesuseanemptyobject:{})

intervalStep:(optional,default:10;todisableuse:0)applyconsolidationpolicywitheveryNthcommit

threshold:(optional;default:0.85)consolidateIFF{threshold}>segment_bytes/(all_segment_bytes/#segments)

bytes_accum:(optional;fordefaultvaluesuse:{})

intervalStep:(optional;default:10;todisableuse:0)applyconsolidationpolicywitheveryNthcommit

threshold:(optional;default:0.85)consolidateIFF{threshold}>(segment_bytes+sum_of_merge_candidate_segment_bytes)/all_segment_bytes

count:(optional;fordefaultvaluesuse:{})

intervalStep:(optional;default:10;todisableuse:0)applyconsolidationpolicywitheveryNthcommit

threshold:(optional;default:0.85)consolidateIFF{threshold}>segment_docs{valid}/(all_segment_docs{valid}/#segments)

fill:(optional)ifspecified,useemptyobjectfordefaultvalues,i.e.{}

intervalStep:(optional;default:10;todisableuse:0)applyconsolidationpolicywitheveryNthcommit

threshold:(optional;default:0.85)consolidateIFF{threshold}>#segment_docs{valid}/(#segment_docs{valid}+#segment_docs{removed})

dataPath:(default:\/arangosearch-\)thefilesystempathwheretostorepersistedviewmetadata

locale:(optional;default:C)thedefaultlocaleusedfororderingprocessedattributenames

threadsMaxIdle:(optional;default:5)maximumidlenumberofthreadsforsingle-runtasksforthecasewheretherearealotofshort-livedasynchronoustasks,alowervaluewillcausealotofthreadcreation/deletioncallsforthecasewheretherearenoshort-livedasynchronoustasks,ahighervaluewillonlywastememory

ArangoSearch

186

threadsMaxTotal:(optional;default:5)maximumtotalnumberofthreads(>0)forsingle-runtasksforthecasewheretherearealotofparallelizabletasksandanabundanceofresources,alowervaluewouldlimitperformanceforthecasewheretherearelimitedresourcesCPU/memory,ahighervaluewillnegativelyimpactperformance

Viewproperties(unmodifiable)

collections:aninternallytrackedlistofcollectionidentifierswhichwereexplicitlyaddedtothecurrentviewbytheuserviaview'link'propertymodificationthelistmayhaveno-longervalididentifiersiftheuserdidnotexplicitlydropthelinkforthesaidcollectionidentifierfromthecurrentviewinvalidcollectionidentifiersareremovedduringviewpropertymodificationamongotherthingsusedforacquiringcollectionlocksintransactions(i.e.duringaviewquerynodocumentswillbereturnedforcollectionsnotinthislist)andgeneratingviewproperties'links'list

Linkproperties

analyzers:(optional;default:['identity'])alistofanalyzers,bynameasdefinedviatheAnalyzers,thatshouldbeappliedtovaluesofprocesseddocumentattributes

fields:(optional;default:{})anobject{attribute-name:[Linkproperties]}offieldsthatshouldbeprocessedateachlevelofthedocumenteachkeyspecifiesthedocumentattributetobeprocessed,thevalueofincludeAllFieldsisalsoconsultedwhenselectingfieldstobeprocessedeachvaluespecifiestheLinkpropertiesdirectivestobeusedwhenprocessingthespecifiedfield,aLinkpropertiesvalueof{}denotesinheritanceofall(exceptfields)directivesfromthecurrentlevel

includeAllFields:(optional;default:false)iftruethenprocessalldocumentattributes(ifnotexplicitlyspecifiedthenprocessthefieldswithdefaultLinkpropertiesdirectives,i.e.{}),otherwiseonlyconsiderattributesmentionedinfields

trackListPositions:(optional;default:false)iftruethenforarrayvaluestrackthevaluepositioninthearray,e.g.whenqueryingfortheinput:{attr:['valueX','valueY','valueZ']}theusermustspecify:doc.attr[1]=='valueY'otherwiseallvaluesinanarrayaretreatedasequalalternatives,e.g.whenqueryingfortheinput:{attr:['valueX','valueY','valueZ']}theusermustspecify:doc.attr=='valueY'

ArangoSearch

187

Analyzers:

TosimplifyquerysyntaxArangoSearchprovidesaconceptofnamedanalyzerswhicharemerelyaliasesfortype+configurationofIResearchanalyzers.ManagementofnamedanalyzersisexposedviabothREST,GUIandJavaScriptAPIs,e.g.

db._globalSettings("iresearch.analyzers")

AuserthenmerelyusestheseanalyzernamesinArangoSearchviewconfigurationsandAQLqueries,e.g.

ArangoSearchprovidesa'text'analyzertoanalyzehumanreadabletext.Arequiredconfigurationparameterforthistypeofanalyzeris'locale'usedtospecifythelanguageusedforanalysis.

TheArangoDBadministratormaythensetupanamedanalyzer'text_des':

{

"name":"text_des",

"type":"text",

"properties":{

"locale":"de-ch"

}

}

Theuseristhenimmediatelyabletorunquerieswiththesaidanalyzer,e.g.

FILTERdoc.descriptionINTOKENS('EinbraunerFuchsspringt','text_des')

SimilarlyanadministratormaychoosetodeployacustomDNAanalyzer'DnaSeq':

{

"name":"dna",

"type":"DnaSeq",

"properties":"use-human-config"

}

Theuseristhenimmediatelyabletorunquerieswiththesaidanalyzer,e.g.

FILTERdoc.dnaINTOKENS('ACGTCGTATGCACTGA','DnaSeq')

Toalimiteddegreetheconceptof'analysis'isevenavailableinnon-IResearchAQL,e.g.theTOKENS(...)functionwillutilizethepowerofIResearchtobreakupavalueintoanAQLarraythatcanbeusedanywhereintheAQLquery.

Inplaintermsthismeansausercanmatchadocumentattributewhenitsvaluematchesatleastonevalueformaset,(yesthisisindependentofdoc),e.g.tomatchdocswith'word==quick'OR'word==brown'OR'word==fox'

FORdocINsomeCollection

FILTERdoc.wordINTOKENS('aquickbrownfox','text_en')

RETRUNdoc

Runtime-pluggingfunctionalityforanalyzersisnotavaiableinArangoDBatthispointintime,soArangoDBcomeswithafewdefault-initializedanalyzers:

identitytreatthevalueasanatom

text_detokenizethevalueintocase-insensitivewordstemsaspertheGermanlocale,donotdiscardanyanystopwords

text_entokenizethevalueintocase-insensitivewordstemsaspertheEnglishlocale,donotdiscardanyanystopwords

text_estokenizethevalueintocase-insensitivewordstemsaspertheSpanishlocale,donotdiscardanyanystopwords

text_fitokenizethevalueintocase-insensitivewordstemsaspertheFinnishlocale,donotdiscardanyanystopwords

text_frtokenizethevalueintocase-insensitivewordstemsaspertheFrenchlocale,donotdiscardanyanystopwords

text_ittokenizethevalueintocase-insensitivewordstemsaspertheItalianlocale,donotdiscardanyanystopwords

text_nltokenizethevalueintocase-insensitivewordstemsaspertheDutchlocale,donotdiscardanyanystopwords

text_notokenizethevalueintocase-insensitivewordstemsaspertheNorwegianlocale,donotdiscardanyanystopwords

Analyzers

188

text_pttokenizethevalueintocase-insensitivewordstemsasperthePortugueselocale,donotdiscardanyanystopwords

text_rutokenizethevalueintocase-insensitivewordstemsaspertheRussianlocale,donotdiscardanyanystopwords

text_svtokenizethevalueintocase-insensitivewordstemsaspertheSwedishlocale,donotdiscardanyanystopwords

text_zhtokenizethevalueintowordstemsaspertheChineselocale

Analyzers

189

FoxxTraditionally,server-sideprojectshavebeendevelopedasstandaloneapplicationsthatguidethecommunicationbetweentheclient-sidefrontendandthedatabasebackend.Thishasledtoapplicationsthatwereeitherdevelopedassinglemonolithsorthatduplicateddataaccessanddomainlogicacrossallservicesthathadtoaccessthedatabase.Additionally,toolstoabstractawaytheunderlyingdatabasecallscouldincuralotofnetworkoverheadwhenusingremotedatabaseswithoutcarefuloptimization.

ArangoDBallowsapplicationdeveloperstowritetheirdataaccessanddomainlogicasmicroservicesrunningdirectlywithinthedatabasewithnativeaccesstoin-memorydata.TheFoxxmicroserviceframeworkmakesiteasytoextendArangoDB'sownRESTAPIwithcustomHTTPendpointsusingmodernJavaScriptrunningonthesameV8engineyouknowfromNode.jsandtheGoogleChromewebbrowser.

Unliketraditionalapproachestostoringlogicinthedatabase(likestoredprocedures),thesemicroservicescanbewrittenasregularstructuredJavaScriptapplicationsthatcanbeeasilydistributedandversioncontrolled.Dependingonyourproject'sneedsFoxxcanbeusedtobuildanythingfromoptimizedRESTendpointsperformingcomplexdataaccesstoentirestandaloneapplicationsrunningdirectlyinsidethedatabase.

FoxxMicroservices

190

FoxxataglanceEachFoxxserviceisdefinedbyaJSONmanifestspecifyingtheentrypoint,anyscriptsdefinedbytheservice,possibleconfigurationoptionsandFoxxdependencies,aswellasothermetadata.Withinaservice,theseoptionsareexposedastheservicecontext.

AttheheartoftheFoxxframeworkliestheFoxxRouterwhichisusedtodefineHTTPendpoints.AservicecanaccessthedatabaseeitherdirectlyfromitscontextusingprefixedcollectionsortheArangoDBdatabaseAPI.

WhileFoxxisprimarilydesignedtobeusedtoaccessthedatabaseitself,ArangoDBalsoprovidesanAPItomakeHTTPrequeststoexternalservices.

Scriptscanbeusedtoperformone-offtasks,whichcanalsobescheduledtobeperformedasynchronouslyusingthebuilt-injobqueue.

Finally,FoxxservicescanbeinstalledandmanagedovertheWeb-UIorthroughArangoDBsHTTPAPI.

HowdoesitworkFoxxservicesconsistofJavaScriptcoderunningintheV8JavaScriptruntimeembeddedinsideArangoDB.EachserviceismountedineachavailableV8context(thenumberofcontextscanbeadjustedintheArangoDBconfiguration).Incomingrequestsaredistributedaccrossthesecontextsautomatically.

Ifyou'recomingfromanotherJavaScriptenvironmentlikeNode.jsthisissimilartorunningmultipleNode.jsprocessesbehindaloadbalancer:youshouldnotrelyonserver-sidestate(otherthanthedatabaseitself)betweendifferentrequestsasthereisnowayofmakingsureconsecutiverequestswillbehandledinthesamecontext.

BecausetheJavaScriptcodeisrunninginsidethedatabaseanotherdifferenceisthatallFoxxandArangoDBAPIsarepurelysynchronousandshouldbeconsideredblocking.Thisisespeciallyimportantfortransactions,whichinArangoDBcanexecutearbitrarycodebutmayhavetolockentirecollections(effectivelypreventinganydatatobewritten)untilthecodehascompleted.

Forinformationonhowthisaffectsinteroperabilitywiththird-partyJavaScriptmoduleswrittenforotherJavaScriptenvironmentsseethechapterondependencies.

DevelopmentmodeDevelopmentmodeallowsyoutomakechangestodeployedservicesin-placedirectlyonthedatabaseserver'sfilesystemwithoutdownloadingandre-uploadingtheservicebundle.Additionallyerrormessageswillcontainstacktraces.

YoucantoggledevelopmentmodeonandoffintheservicesettingstabofthewebinterfaceorusingtheHTTPAPI.Onceactivatedtheservice'sfilesystempathwillbeshownintheinfotab.

Onceenabledtheservice'ssourcefilesandmanifestwillbere-evaluated,andthesetupscript(ifpresent)re-executed,everytimearouteoftheserviceisaccessed,effectivelyre-deployingtheserviceoneveryrequest.Asthenameindicatesthisisintendedtobeusedstrictlyduringdevelopmentandismostdefinitelyabadideaonproductionservers.Theadditionalinformationexposedduringdevelopmentmodemayincludefilesystempathsandpartsoftheservice'ssourcecode.

Alsonotethatifyouareservingstaticfilesaspartofyourservice,accessingthesefilesfromabrowsermayalsotriggerare-deploymentoftheservice.Finally,makingHTTPrequeststoaservicerunningindevelopmentmodefromwithintheservice(i.e.usingthe@arangodb/requestmoduletoaccesstheserviceitself)isprobablynotagoodideaeither.

Bewareofdeletingthedatabasetheserviceisdeployedon:itwillerasethesourcefilesoftheservicealongwiththecollections.Youshouldbackupthecodeyouworkedonindevelopmentbeforedoingthattoavoidlosingyourprogress.

FoxxstoreTheFoxxstoreprovidesaccesstoanumberofready-to-useofficialandcommunity-maintainedFoxxservicesyoucaninstallwithasingleclick,includingexampleservicesandwrappersforexternalSaaStoolsliketransactionale-mailservices,bugloggersoranalyticstrackers.

YoucanfindtheFoxxstoreinthewebinterfacebyusingtheAddServicebuttonintheservicelist.

Ataglance

191

Cluster-Foxx

WhenrunningArangoDBinaclustertheFoxxserviceswillrunoneachcoordinator.Installing,upgradinganduninstallingservicesonanycoordinatorwillautomaticallydistributetheservicetotheothercoordinators,makingdeploymentsaseasyasinsingle-servermode.However,thismeanstherearesomelimitations:

Youshouldavoidanykindoffilesystemstatebeyondthedeployedservicebundleitself.Don'twritedatatothefilesystemorencodeanyexpectationsofthefilesystemstateotherthanthefilesintheservicefolderthatwereinstalledaspartoftheservice(e.g.fileuploadsorcustomlogfiles).

Additionally,thedevelopmentmodewillleadtoaninconsistentstateoftheclusteruntilitisdisabled.WhileaserviceisrunningindevelopmentmodeyoucanmakechangestotheserviceonthefilesystemofanycoordinatorandseethemreflectedinrealtimejustlikewhenrunningArangoDBasasingleserver.Howeverthechangesmadeononecoordinatorwillnotbereflectedacrossothercoordinatorsuntilthedevelopmentmodeisdisabled.Whendisablingthedevelopmentmodeforaservice,thecoordinatorwillcreateanewbundleanddistributeitacrosstheservicelikeamanualupgradeoftheservice.

Forthesereasonsitisstronglyrecommendednottousedevelopmentmodeinaclusterwithmultiplecoordinatorsunlessyouaresurethatnorequestsorchangeswillbemadetoothercoordinatorswhileyouaremodifyingtheservice.Usingdevelopmentmodeinaproductionclusterisextremelyunsafeandhighlydiscouraged.

Ataglance

192

GettingStartedWe'regoingtostartwithanemptyfolder.Thiswillbetherootfolderofourservices.Youcannameitsomethingcleverbutforthecourseofthisguidewe'llassumeit'scalledthenameofyourservice:getting-started.

Firstweneedtocreateamanifest.Createanewfilecalledmanifest.jsonandaddthefollowingcontent:

{

"engines":{

"arangodb":"^3.0.0"

}

}

ThisjusttellsArangoDBtheserviceiscompatiblewithversions3.0.0andlater(allthewayuptobutnotincluding4.0.0),allowingolderversionsofArangoDBtounderstandthatthisservicelikelywon'tworkforthemandnewerversionswhatbehaviortoemulateshouldtheystillsupportit.

Thelittlehattotheleftoftheversionnumberisnotatypo,it'scalleda"caret"andindicatestheversionrange.Foxxusessemanticversioning(alsocalled"semver")formostofitsversionhandling.Youcanfindoutmoreabouthowsemverworksattheofficialsemverwebsite.

Nextwe'llneedtospecifyanentrypointtoourservice.ThisistheJavaScriptfilethatwillbeexecutedtodefineourservice'sHTTPendpoints.Wecandothisbyaddinga"main"fieldtoourmanifest:

{

"engines":{

"arangodb":"^3.0.0"

},

"main":"index.js"

}

That'sallweneedinourmanifestfornow,solet'snextcreatetheindex.jsfile:

'usestrict';

constcreateRouter=require('@arangodb/foxx/router');

constrouter=createRouter();

module.context.use(router);

Thefirstlinecausesourfiletobeinterpretedusingstrictmode.AllexamplesintheArangoDBdocumentationassumestrictmode,soyoumightwanttofamiliarizeyourselfwithitifyouhaven'tencountereditbefore.

Thesecondlineimportsthe@arangodb/foxx/routermodulewhichprovidesafunctionforcreatingnewFoxxrouters.We'reusingthisfunctiontocreateanewrouterobjectwhichwe'llbeusingforourservice.

Themodule.contextistheso-calledFoxxcontextorservicecontext.ThisvariableisavailableinallfilesthatarepartofyourFoxxserviceandprovidesaccesstoFoxxAPIsspecifictothecurrentservice,liketheusemethod,whichtellsFoxxtomounttherouterinthisservice(andtoexposeitsroutestoHTTP).

Nextlet'sdefinearoutethatprintsagenericgreeting:

//continued

router.get('/hello-world',function(req,res){

res.send('HelloWorld!');

})

.response(['text/plain'],'Agenericgreeting.')

.summary('Genericgreeting')

.description('Printsagenericgreeting.');

Therouterprovidesthemethodsget,post,etccorrespondingtoeachHTTPverbaswellasthecatch-allall.ThesemethodsindicatethatthegivenrouteshouldbeusedtohandleincomingrequestswiththegivenHTTPverb(oranymethodwhenusingall).

Gettingstarted

193

Thesemethodstakeanoptionalpath(ifomitted,itdefaultsto"/")aswellasarequesthandler,whichisafunctiontakingthereq(request)andres(response)objectstohandletheincomingrequestandgeneratetheoutgoingresponse.IfyouhaveusedtheexpressframeworkinNode.js,youmayalreadybefamiliarwithhowthisworks,otherwisecheckoutthechapteronroutes.

Theobjectreturnedbytherouter'smethodsprovidesadditionalmethodstoattachmetadataandvalidationtotheroute.We'reusingsummaryanddescriptiontodocumentwhattheroutedoes--thesearen'tstrictlynecessarybutgiveussomeniceauto-generateddocumentation.Theresponsemethodletsusadditionallydocumenttheresponsecontenttypeandwhattheresponsebodywillrepresent.

Tryitout

AtthispointyoucanuploadtheservicefolderasaziparchivefromthewebinterfaceusingtheServicestab.

ClickAddServicethenpicktheZipoptioninthedialog.Youwillneedtoprovideamountpath,whichistheURLprefixatwhichtheservicewillbemounted(e.g./getting-started).

Onceyouhavepickedtheziparchiveusingthefilepicker,theuploadshouldbeginimmediatelyandyourserviceshouldbeinstalled.OtherwisepresstheInstallbuttonandwaitforthedialogtodisappearandtheservicetoshowupintheservicelist.

Clickanywhereonthecardwithyourmountpathonthelabeltoopentheservice'sdetails.

IntheAPIdocumentationyoushouldseetheroutewedefinedearlier(/hello-world)withthewordGETnexttoitindicatingtheHTTPmethoditsupportsandthesummaryweprovidedontheright.Byclickingontheroute'spathyoucanopenthedocumentationfortheroute.

Notethatthedescriptionweprovidedappearsinthegenerateddocumentationaswellasthedescriptionweaddedtotheresponse(whichshouldcorrectlyindicatethecontenttypetext/plain,i.e.plaintext).

ClicktheTryitout!buttontosendarequesttotherouteandyoushouldseeanexamplerequestwiththeservice'sresponse:"HelloWorld!".

Congratulations!Youhavejustcreated,installedandusedyourfirstFoxxservice.

Parametervalidation

Let'saddanotherroutethatprovidesamorepersonalizedgreeting:

//continued

constjoi=require('joi');

router.get('/hello/:name',function(req,res){

res.send(`Hello${req.pathParams.name}`);

})

.pathParam('name',joi.string().required(),'Nametogreet.')

.response(['text/plain'],'Apersonalizedgreeting.')

.summary('Personalizedgreeting')

.description('Printsapersonalizedgreeting.');

ThefirstlineimportsthejoimodulefromnpmwhichcomesbundledwithArangoDB.JoiisavalidationlibrarythatisusedthroughoutFoxxtodefineschemasandparametertypes.

Note:Youcanbundleyourownmodulesfromnpmbyinstallingtheminyourservicefolderandmakingsurethenode_modulesfolderisincludedinyourziparchive.Formoreinformationseethesectiononmoduledependenciesinthechapterondependencies.

ThepathParammethodallowsustospecifyparametersweareexpectinginthepath.Thefirstargumentcorrespondstotheparameternameinthepath,thesecondargumentisajoischematheparameterisexpectedtomatchandthefinalargumentservestodescribetheparameterintheAPIdocumentation.

ThepathparametersareaccessiblefromthepathParamspropertyoftherequestobject.We'reusingatemplatestringtogeneratetheserver'sresponsecontainingtheparameter'svalue.

Gettingstarted

194

NotethatrouteswithpathparametersthatfailtovalidatefortherequestURLwillbeskippedasiftheywouldn'texist.Thisallowsyoutodefinemultipleroutesthatareonlydistinguishedbytheschemasoftheirpathparameters(e.g.aroutetakingonlynumericparametersandonetakinganystringasafallback).

Let'stakethisfurtherandcreatearoutethattakesaJSONrequestbody:

//continued

router.post('/sum',function(req,res){

constvalues=req.body.values;

res.send({

result:values.reduce(function(a,b){

returna+b;

},0)

});

})

.body(joi.object({

values:joi.array().items(joi.number().required()).required()

}).required(),'Valuestoaddtogether.')

.response(joi.object({

result:joi.number().required()

}).required(),'Sumoftheinputvalues.')

.summary('Addupnumbers')

.description('Calculatesthesumofanarrayofnumbervalues.');

Notethatweusedposttodefinethisrouteinsteadofget(whichdoesnotsupportrequestbodies).TryingtosendaGETrequesttothisroute'sURL(intheabsenceofagetrouteforthesamepath)willresultinFoxxrespondingwithanappropriateerrorresponse,indicatingthesupportedHTTPmethods.

AsthisroutenotonlyexpectsaJSONobjectasinputbutalsorespondswithaJSONobjectasoutputweneedtodefinetwoschemas.Wedon'tstrictlyneedaresponseschemabutithelpsdocumentingwhattherouteshouldbeexpectedtorespondwithandwillshowupintheAPIdocumentation.

Becausewe'repassingaschematotheresponsemethodwedon'tneedtoexplicitlytellFoxxwearesendingaJSONresponse.ThepresenceofaschemaintheabsenceofacontenttypealwaysimplieswewantJSON.Thoughwecouldjustadd["application/json"]asanadditionalargumentaftertheschemaifwewantedtomakethismoreexplicit.

Thebodymethodworksthesamewayastheresponsemethodexcepttheschemawillbeusedtovalidatetherequestbody.Iftherequestbodycan'tbeparsedasJSONordoesn'tmatchtheschema,Foxxwillrejecttherequestwithanappropriateerrorresponse.

Creatingcollections

TherealpowerofFoxxcomesfrominteractingwiththedatabaseitself.Inordertobeabletouseacollectionfromwithinourservice,weshouldfirstmakesurethatthecollectionactuallyexists.Therightplacetocreatecollectionsyourserviceisgoingtouseisinasetupscript,whichFoxxwillexecuteforyouwheninstallingorupdatingtheservice.

Firstcreateanewfoldercalled"scripts"intheservicefolder,whichwillbewhereourscriptsaregoingtolive.Forsimplicity'ssake,oursetupscriptwillliveinafilecalledsetup.jsinsidethatfolder:

//continued

'usestrict';

constdb=require('@arangodb').db;

constcollectionName='myFoxxCollection';

if(!db._collection(collectionName)){

db._createDocumentCollection(collectionName);

}

Thescriptusesthedbobjectfromthe@arangodbmodule,whichletsusinteractwiththedatabasetheFoxxservicewasinstalledinandthecollectionsinsidethatdatabase.Becausethescriptmaybeexecutedmultipletimes(i.e.wheneverweupdatetheserviceorwhentheserverisrestarted)weneedtomakesurewedon'taccidentallytrytocreatethesamecollectiontwice(whichwouldresultinanexception);wedothatbyfirstcheckingwhetheritalreadyexistsbeforecreatingit.

Gettingstarted

195

The_collectionmethodlooksupacollectionbynameandreturnsnullifnocollectionwiththatnamewasfound.The_createDocumentCollectionmethodcreatesanewdocumentcollectionbyname(_createEdgeCollectionalsoexistsandworksanalogouslyforedgecollections).

Note:Becausewehavehardcodedthecollectionname,multiplecopiesoftheserviceinstalledalongsideeachotherinthesamedatabasewillsharethesamecollection.Becausethismaynotalwaysbewhatyouwant,theFoxxcontextalsoprovidesthecollectionNamemethodwhichappliesamountpointspecificprefixtoanygivencollectionnametomakeituniquetotheservice.Italsoprovidesthecollectionmethod,whichbehavesalmostexactlylikedb._collectionexceptitalsoappliestheprefixbeforelookingthecollectionup.

Nextweneedtotellourserviceaboutthescriptbyaddingittothemanifestfile:

{

"engines":{

"arangodb":"^3.0.0"

},

"main":"index.js",

"scripts":{

"setup":"scripts/setup.js"

}

}

Theonlythingthathaschangedisthatweaddeda"scripts"fieldspecifyingthepathofthesetupscriptwejustwrote.

Gobacktothewebinterfaceandupdatetheservicewithournewcode,thenchecktheCollectionstab.Ifeverythingworkedright,youshouldseeanewcollectioncalled"myFoxxCollection".

AccessingcollectionsLet'sexpandourservicebyaddingafewmoreroutestoourindex.js:

//continued

constdb=require('@arangodb').db;

consterrors=require('@arangodb').errors;

constfoxxColl=db._collection('myFoxxCollection');

constDOC_NOT_FOUND=errors.ERROR_ARANGO_DOCUMENT_NOT_FOUND.code;

router.post('/entries',function(req,res){

constdata=req.body;

constmeta=foxxColl.save(req.body);

res.send(Object.assign(data,meta));

})

.body(joi.object().required(),'Entrytostoreinthecollection.')

.response(joi.object().required(),'Entrystoredinthecollection.')

.summary('Storeanentry')

.description('Storesanentryinthe"myFoxxCollection"collection.');

router.get('/entries/:key',function(req,res){

try{

constdata=foxxColl.document(req.pathParams.key);

res.send(data)

}catch(e){

if(!e.isArangoError||e.errorNum!==DOC_NOT_FOUND){

throwe;

}

res.throw(404,'Theentrydoesnotexist',e);

}

})

.pathParam('key',joi.string().required(),'Keyoftheentry.')

.response(joi.object().required(),'Entrystoredinthecollection.')

.summary('Retrieveanentry')

.description('Retrievesanentryfromthe"myFoxxCollection"collectionbykey.');

We'reusingthesaveanddocumentmethodsofthecollectionobjecttostoreandretrievedocumentsinthecollectionwecreatedinoursetupscript.Becausewedon'tcarewhatthedocumentslooklikeweallowanyattributesontherequestbodyandjustacceptanobject.

Gettingstarted

196

BecausethekeywillbeautomaticallygeneratedbyArangoDBwhenonewasn'tspecifiedintherequestbody,we'reusingObject.assigntoapplytheattributesofthemetadataobjectreturnedbythesavemethodtothedocumentbeforereturningitfromourfirstroute.

Thedocumentmethodreturnsadocumentinacollectionbyits_keyor_id.HoweverwhennomatchingdocumentexistsitthrowsanArangoErrorexception.BecausewewanttoprovideamoredescriptiveerrormessagethanArangoDBdoesoutofthebox,weneedtohandlethaterrorexplicitly.

AllArangoErrorexceptionshaveatruthyattributeisArangoErrorthathelpsyourecognizingtheseerrorswithouthavingtoworryaboutinstanceofchecks.TheyalsoprovideanerrorNumandanerrorMessage.Ifyouwanttocheckforspecificerrorsyoucanjustimporttheerrorsobjectfromthe@arangodbmoduleinsteadofhavingtomemorizenumericerrorcodes.

Insteadofdefiningourownresponselogicfortheerrorcasewejustuseres.throw,whichmakestheresponseobjectthrowanexceptionFoxxcanrecognizeandconverttotheappropriateserverresponse.WealsopassalongtheexceptionitselfsoFoxxcanprovidemorediagnosticinformationwhenwewantitto.

Wecouldextendthepostroutetosupportarraysofobjectsaswell,eachobjectfollowingacertainschema:

//storeschemainvariabletomakeitre-usable,see.body()

constdocSchema=joi.object().required().keys({

name:joi.string().required(),

age:joi.number().required()

}).unknown();//allowadditionalattributes

router.post('/entries',function(req,res){

constmultiple=Array.isArray(req.body);

constbody=multiple?req.body:[req.body];

letdata=[];

for(vardocofbody){

constmeta=foxxColl.save(doc);

data.push(Object.assign(doc,meta));

}

res.send(multiple?data:data[0]);

})

.body(joi.alternatives().try(

docSchema,

joi.array().items(docSchema)

),'Entryorentriestostoreinthecollection.')

.response(joi.alternatives().try(

joi.object().required(),

joi.array().items(joi.object().required())

),'Entryorentriesstoredinthecollection.')

.summary('Storeentryorentries')

.description('Storeasingleentryormultipleentriesinthe"myFoxxCollection"collection.');

WritingdatabasequeriesStoringandretrievingentriesisfine,butrightnowwehavetomemorizeeachkeywhenwecreateanentry.Let'saddaroutethatgivesusalistofthekeysofallentriessowecanusethosetolookanentryupindetail.

ThenaïveapproachwouldbetousethetoArray()methodtoconverttheentirecollectiontoanarrayandjustreturnthat.Butwe'reonlyinterestedinthekeysandtheremightpotentiallybesomanyentriesthatfirstretrievingeverysingledocumentmightgetunwieldy.Let'swriteashortAQLquerytodothisinstead:

//continued

constaql=require('@arangodb').aql;

router.get('/entries',function(req,res){

constkeys=db._query(aql`

FORentryIN${foxxColl}

RETURNentry._key

`);

res.send(keys);

})

.response(joi.array().items(

joi.string().required()

Gettingstarted

197

).required(),'Listofentrykeys.')

.summary('Listentrykeys')

.description('Assemblesalistofkeysofentriesinthecollection.');

Herewe'reusingtwonewthings:

The_querymethodexecutesanAQLqueryintheactivedatabase.

Theaqltemplatestringhandlerallowsustowritemulti-lineAQLqueriesandalsohandlesqueryparametersandcollectionnames.InsteadofhardcodingthenameofthecollectionwewanttouseinthequerywecansimplyreferencethefoxxCollvariablewedefinedearlier--itrecognizesthevalueasanArangoDBcollectionobjectandknowswearespecifyingacollectionratherthanaregularvalueeventhoughAQLdistinguishesbetweenthetwo.

Note:Ifyouaren'tusedtoJavaScripttemplatestringsandtemplatestringhandlersjustthinkofaqlasafunctionthatreceivesthemultilinestringsplitatevery${}expressionaswellasanarrayofthevaluesofthoseexpressions--that'sactuallyallthereistoit.

Alternatively,here'saversionwithouttemplatestrings(noticehowmuchcleanertheaqlversionwillbeincomparisonwhenyouhavemultiplevariables):

constkeys=db._query(

'FORentryIN@@collRETURNentry._key',

{'@coll':foxxColl.name()}

);

Nextsteps

YounowknowhowtocreateaFoxxservicefromscratch,howtohandleuserinputandhowtoaccessthedatabasefromwithinyourFoxxservicetostore,retrieveandquerydatayoustoreinsideArangoDB.ThisshouldallowyoutobuildmeaningfulAPIsforyourownapplicationsbuttherearemanymorethingsyoucandowithFoxx:

Needtogofaster?Turnondevelopmentmodeandhackonyourcoderightontheserver.

Concernedaboutsecurity?Youcouldaddauthenticationtoyourservicetoprotectaccesstothedatabeforeitevenleavesthedatabase.

Writingasinglepageapp?YoucouldstoresomebasicassetsrightinsideyourFoxxservice.

Needtointegrateexternalservices?YoucanmakeHTTPrequestsfrominsideFoxxandusequeuedjobstoperformthatworkinthebackground.

Tiredofreinventingthewheel?Learnaboutdependencies.

Everythingbroken?Youcanwriteteststomakesureyourlogicremainssound.

Gettingstarted

198

ManifestfilesEveryservicecomeswithamanifest.jsonfileprovidingmetadata.Thefollowingfieldsareallowedinmanifests:

configuration:Object(optional)

Anobjectdefiningtheconfigurationoptionsthisservicerequires.

defaultDocument:string(optional)

Ifspecified,the/(root)routeoftheservicewillautomaticallyredirecttothegivenrelativepath,e.g.:

"defaultDocument":"index.html"

ThiswouldhavethesameeffectascreatingthefollowingrouteinJavaScript:

constcreateRouter=require('@arangodb/foxx/router');

constindexRouter=createRouter();

indexRouter.all('/',function(req,res){

res.redirect('index.html');

});

module.context.use(indexRouter);

Note:Asof3.0.0thisfieldcansafelybeomitted;thevaluenolongerdefaultsto"index.html".

dependencies:Object(optional)andprovides:Object(optional)

Objectsspecifyingotherservicesthisservicehasasdependenciesandwhatdependenciesitcanprovidetootherservices.

engines:Object(optional)

AnobjectindicatingthesemanticversionrangesofArangoDB(orcompatibleenvironments)theservicewillbecompatiblewith,e.g.:

"engines":{

"arangodb":"^3.0.0"

}

ThisshouldcorrectlyindicatetheminimumversionofArangoDBtheservicehasbeentestedagainst.FoxxmaintainsastrictsemanticversioningpolicyasofArangoDB3.0.0soitisgenerallysafetousesemverranges(e.g. 3.0.0tomatchanyversiongreaterorequalto3.0.0andbelow4.0.0)formaximumcompatibility.

files:Object(optional)

Anobjectdefiningfileassetsservedbythisservice.

lib:string(Default:".")

TherelativepathtotheFoxxJavaScriptfilesintheservice,e.g.:

"lib":"lib"

Thiswouldresultinthemainentrypoint(seebelow)andotherJavaScriptpathsbeingresolvedasrelativetothelibfolderinsidetheservicefolder.

main:string(optional)

Therelativepathtothemainentrypointofthisservice(relativetolib,seeabove),e.g.:

"main":"index.js"

ThiswouldresultinFoxxloadingandexecutingthefileindex.jswhentheserviceismountedorstarted.

Servicemanifest

199

Note:whileitistechnicallypossibletoomitthisfield,youwilllikelywanttoprovideanentrypointtoyourserviceasthisistheonlywaytoexposeHTTProutesorexportaJavaScriptAPI.

scripts:Object(optional)

Anobjectdefiningnamedscriptsprovidedbythisservice,whichcaneitherbeuseddirectlyorasqueuedjobsbyotherservices.

tests:stringorArray<string>(optional)

ApathorlistofpathsofJavaScripttestsprovidedforthisservice.

Additionallymanifestscanprovidethefollowingmetadata:

author:string(optional)

Thefullnameoftheauthoroftheservice(i.e.you).Thiswillbeshowninthewebinterface.

contributors:Array<string>(optional)

Alistofnamesofpeoplethathavecontributedtothedevelopmentoftheserviceinsomeway.Thiswillbeshowninthewebinterface.

description:string(optional)

Ahuman-readabledescriptionoftheservice.Thiswillbeshowninthewebinterface.

keywords:Array<string>(optional)

Alistofkeywordsthathelpcategorizethisservice.ThisisusedbytheFoxxStoreinstallerstoorganizeservices.

license:string(optional)

Astringidentifyingthelicenseunderwhichtheserviceispublished,ideallyintheformofanSPDXlicenseidentifier.Thiswillbeshowninthewebinterface.

name:string(optional)

ThenameoftheFoxxservice.AllowedcharactersareA-Z,0-9,theASCIIhyphen(-)andunderscore(_)characters.Thenamemustnotstartwithanumber.Thiswillbeshowninthewebinterface.

thumbnail:string(optional)

Thefilenameofathumbnailthatwillbeusedalongsidetheserviceinthewebinterface.ThisshouldbeaJPEGorPNGimagethatlooksgoodatsizes50x50and160x160.

version:string(optional)

TheversionnumberoftheFoxxservice.Theversionnumbermustfollowthesemanticversioningformat.Thiswillbeshowninthewebinterface.

Examples

{

"name":"example-foxx-service",

"version":"3.0.0-dev",

"license":"MIT",

"description":"Anexampleservicewitharelativelyfull-featuredmanifest.",

"thumbnail":"foxx-icon.png",

"keywords":["demo","service"],

"author":"ArangoDBGmbH",

"contributors":[

"AlanPlum<alan@arangodb.example>"

],

"lib":"dist",

"main":"entry.js",

"defaultDocument":"welcome.html",

"engines":{

"arangodb":"^3.0.0"

},

"files":{

Servicemanifest

200

"welcome.html":"assets/index.html",

"hello.jpg":"assets/hello.jpg"

"world.jpg":{

"path":"assets/world.jpg",

"type":"image/jpeg",

"gzip":false

}

},

"tests":"dist/**.spec.js"

}

Servicemanifest

201

FoxxservicecontextTheservicecontextprovidesaccesstomethodsandattributesthatarespecifictoagivenservice.InaFoxxservicethecontextisgenerallyavailableasthemodule.contextvariable.Withinarouter'srequesthandlertherequestandresponseobjects'contextattributealsoprovideaccesstothecontextoftheservicetheroutewasmountedin(whichmaybedifferentfromtheonetheroutehandlerwasdefinedin).

Examples

//inservice/my-foxx-1

constcreateRouter=require('@arangodb/foxx/router');

constrouter=createRouter();

//Seethechapterondependenciesformoreinfoon

//howexportsanddependenciesworkacrossservices

module.exports={routes:router};

router.get(function(req,res){

module.context.mount==='/my-foxx-1';

req.context.mount==='/my-foxx-2';

res.write('Hellofrommy-foxx-1');

});

//inservice/my-foxx-2

constcreateRouter=require('@arangodb/foxx/router');

constrouter2=createRouter();

module.context.use(router2);

router2.post(function(req,res){

module.context.mount==='/my-foxx-2';

req.context.mount==='/my-foxx-2';

res.write('Hellofrommy-foxx-2');

});

constrouter1=module.context.dependencies.myFoxx1.routes;

module.context.use(router1);

Theservicecontextspecifiesthefollowingproperties:

argv:any

Anyargumentspassedinifthecurrentfilewasexecutedasascriptorqueuedjob.

basePath:string

Thefilesystempathoftheservice,i.e.thefolderinwhichtheservicewasinstalledtobyArangoDB.

baseUrl:string

ThebaseURLoftheservice,relativetotheArangoDBserver,e.g./_db/_system/my-foxx.

collectionPrefix:string

TheprefixthatwillbeusedbycollectionandcollectionNametoderivethenamesofservice-specificcollections.Thisisderivedfromtheservice'smountpoint,e.g./my-foxxbecomesmy_foxx.

configuration:Object

Configurationoptionsfortheservice.

dependencies:Object

Configureddependenciesfortheservice.

isDevelopment:boolean

Indicateswhethertheserviceisrunningindevelopmentmode.

Servicecontext

202

isProduction:boolean

TheinverseofisDevelopment.

manifest:Object

Theparsedmanifestfileoftheservice.

mount:string

Themountpointoftheservice,e.g./my-foxx.

apiDocumentationmodule.context.apiDocumentation([options]):Function

DEPRECATED

CreatesarequesthandlerthatservestheAPIdocumentation.

Note:ThismethodhasbeendeprecatedinArangoDB3.1andreplacedwiththemorestraightforwardcreateDocumentationRoutermethodprovidingthesamefunctionality.

Arguments

SeecreateDocumentationRouterbelow.

Examples

//ServetheAPIdocsforthecurrentservice

router.get('/docs/*',module.context.apiDocumentation());

//Notethatthepathmustendwithawildcard

//andtheroutemustuseHTTPGET.

createDocumentationRoutermodule.context.createDocumentationRouter([options]):Router

CreatesarouterthatservestheAPIdocumentation.

Note:Theroutercanbemountedlikeanyotherchildrouter(seeexamplesbelow).

Arguments

options:Object(optional)

Anobjectwithanyofthefollowingproperties:

mount:string(Default:module.context.mount)

Themountpathoftheservicetoservethedocumentationof.

indexFile:string(Default:"index.html")

FilenameoftheHTMLfileservingtheAPIdocumentation.

swaggerRoot:string(optional)

FullpathofthefoldercontainingtheSwaggerassetsandtheindexFile.DefaultstotheSwaggerassetsusedbythewebinterface.

before:Function(optional)

Afunctionthatwillbeexecutedbeforearequestishandled.

Ifthefunctionreturnsfalsetherequestwillnotbeprocessedanyfurther.

Ifthefunctionreturnsanobject,itsattributeswillbeusedtooverridetheoptionsforthecurrentrequest.

Servicecontext

203

Anyotherreturnvaluewillbeignored.

Ifoptionsisafunctionitwillbeusedasthebeforeoption.

IfoptionsisastringitwillbeusedastheswaggerRootoption.

ReturnsaFoxxrouter.

Examples

//ServetheAPIdocsforthecurrentservice

router.use('/docs',module.context.createDocumentationRouter());

//--or--

//ServetheAPIdocsfortheservicetherouterismountedin

router.use('/docs',module.context.createDocumentationRouter(function(req){

return{mount:req.context.mount};

}));

//--or--

//ServetheAPIdocsonlyforusersauthenticatedwithArangoDB

router.use('/docs',module.context.createDocumentationRouter(function(req,res){

if(req.suffix==='swagger.json'&&!req.arangoUser){

res.throw(401,'Notauthenticated');

}

}));

collectionmodule.context.collection(name):ArangoCollection|null

PassesthegivennametocollectionName,thenlooksupthecollectionwiththeprefixedname.

Arguments

name:string

Unprefixednameoftheservice-specificcollection.

Returnsacollectionornullifnocollectionwiththeprefixednameexists.

collectionNamemodule.context.collectionName(name):string

PrefixesthegivennamewiththecollectionPrefixforthisservice.

Arguments

name:string

Unprefixednameoftheservice-specificcollection.

Returnstheprefixedname.

Examples

module.context.mount==='/my-foxx'

module.context.collectionName('doodads')==='my_foxx_doodads'

filemodule.context.file(name,[encoding]):Buffer|string

PassesthegivennametofileName,thenloadsthefilewiththeresultingname.

Servicecontext

204

Arguments

name:string

Nameofthefiletoload,relativetothecurrentservice.

encoding:string(optional)

Encodingofthefile,e.g.utf-8.Ifomittedthefilewillbeloadedasarawbufferinsteadofastring.

Returnsthefile'scontents.

fileNamemodule.context.fileName(name):string

Resolvesthegivenfilenamerelativetothecurrentservice.

Arguments

name:string

Nameofthefile,relativetothecurrentservice.

Returnstheabsolutefilepath.

usemodule.context.use([path],router):Endpoint

Mountsagivenrouterontheservicetoexposetherouter'sroutesontheservice'smountpoint.

Arguments

path:string(Default:"/")

Pathtomounttherouterat,relativetotheservice'smountpoint.

router:Router|Middleware

Arouterormiddlewaretomount.

ReturnsanEndpointforthegivenrouterormiddleware.

Note:Mountingservicesatruntime(e.g.withinrequesthandlersorqueuedjobs)isnotsupported.

Servicecontext

205

FoxxconfigurationFoxxservicescandefineconfigurationparameterstomakethemmorere-usable.

Theconfigurationobjectmapsnamestoconfigurationparameters:

Thekeyisthenameunderwhichtheparameterwillbeavailableontheservicecontext'sconfigurationproperty.

Thevalueisaparameterdefinition.

Thekeyshouldbeavalididentifierfollowingthecase-insensitiveformat/^[_$a-z][-_$a-z0-9]*$/.

Theparameterdefinitioncanhavethefollowingproperties:

description:string

Humanreadabledescriptionoftheparameter.

type:string(Default:"string")

Typeoftheconfigurationparameter.Supportedvaluesare:

"integer"or"int":anyfiniteintegernumber.

"boolean"or"bool":thevaluestrueorfalse.

"number":anyfinitedecimalorintegernumber.

"string":anystringvalue.

"json":anywell-formedJSONvalue.

"password":likestringbutwillbedisplayedasamaskedinputfieldinthewebfrontend.

default:any

Defaultvalueoftheconfigurationparameter.

required:(Default:true)

Whethertheparameterisrequired.

Iftheconfigurationhasparametersthatdonotspecifyadefaultvalue,youneedtoconfiguretheservicebeforeitbecomesactive.InthemeantimeafallbackservicelicationwillbemountedthatrespondstoallrequestswithaHTTP500statuscodeindicatingaserver-sideerror.

TheconfigurationparametersofamountedservicecanbeadjustedfromthewebinterfacebyclickingtheConfigurationbuttonintheservicedetails.

Examples

"configuration":{

"currency":{

"description":"Currencysymboltouseforpricesintheshop.",

"default":"$",

"type":"string"

},

"secretKey":{

"description":"Secretkeytouseforsigningsessiontokens.",

"type":"password"

}

}

Configuration

206

DependencymanagementTherearetwothingscommonlycalled"dependencies"inFoxx:

Moduledependencies,i.e.dependenciesonexternalJavaScriptmodules(e.g.fromthepublicnpmregistry)

Foxxdependencies,i.e.dependenciesbetweenFoxxservices

Let'slookattheminmoredetail:

Moduledependencies

Youcanusethenode_modulesfoldertobundlethird-partyFoxx-compatiblenpmandNode.jsmoduleswithyourFoxxservice.Typicallythisisachievedbyaddingapackage.jsonfiletoyourprojectspecifyingnpmdependenciesusingthedependenciesattributeandinstallingthemwiththenpmcommand-linetool.

Makesuretoincludetheactualnode_modulesfolderinyourFoxxservicebundleasArangoDBwillnotdoanythingspecialtoinstallthesedependencies.AlsokeepinmindthatbundlingextraneousmoduleslikedevelopmentdependenciesmaybloatthefilesizeofyourFoxxservicebundle.

Compatibilitycaveats

UnlikeJavaScriptinbrowsersorNode.js,theJavaScriptenvironmentinArangoDBissynchronous.ThismeansanymodulesthatdependonasynchronousbehaviourlikepromisesorsetTimeoutwillnotbehavecorrectlyinArangoDBorFoxx.AdditionallyunlikeNode.jsArangoDBdoesnotsupportnativeextensions.AllmoduleshavetobeimplementedinpureJavaScript.

WhileArangoDBprovidesalotofcompatibilitycodetosupportmoduleswrittenforNode.js,someNode.jsbuilt-inmodulescannotbeprovidedbyArangoDB.ForacloserlookattheNode.jsmodulesArangoDBdoesordoesnotprovidecheckouttheappendixonJavaScriptmodules.

Alsonotethattheserestrictionsnotonlyapplyonthemodulesyouwishtoinstallbutalsothedependenciesofthosemodules.Asaruleofthumb:moduleswrittentoworkinNode.jsandthebrowserthatdonotrelyonasyncbehaviourshouldgenerallywork;modulesthatrelyonnetworkorfilesystemI/Oormakeheavyuseofasyncbehaviourmostlikelywillnot.

Foxxdependencies

Foxxdependenciescanbedeclaredinaservice'smanifestusingtheprovidesanddependenciesfields:

providesliststhedependenciesagivenserviceprovides,i.e.whichAPIsitclaimstobecompatiblewith

dependenciesliststhedependenciesagivenserviceuses,i.e.whichAPIsitsdependenciesneedtobecompatiblewith

Adependencynameshouldgenerallyusethesameformatasanamespaced(org-scoped)NPMmodule,e.g.@foxx/sessions.Thedependencynameshouldfolloweitherthecase-insensitiveformat/^@[_$a-z][-_$a-z0-9]*(\/[_$a-z][-_$a-z0-9]*)*$/or/^[_$a-z][-_$a-z0-9]*$/.

DependencynamesrefertotheexternalJavaScriptAPIofaserviceratherthanspecificservicesimplementingthoseAPIs.Somedependencynamesdefinedbyofficiallymaintainedservicesare:

@foxx/auth(version1.0.0)@foxx/api-keys(version1.0.0)@foxx/bugsnag(versions1.0.0and2.0.0)@foxx/mailgun(versions1.0.0and2.0.0)@foxx/postageapp(versions1.0.0and2.0.0)@foxx/postmark(versions1.0.0and2.0.0)@foxx/sendgrid(versions1.0.0and2.0.0)@foxx/oauth2(versions1.0.0and2.0.0)@foxx/segment-io(versions1.0.0and2.0.0)

Dependencies

207

@foxx/sessions(versions1.0.0and2.0.0)@foxx/users(versions1.0.0,2.0.0and3.0.0)

Aprovidesdefinitionmapseachprovideddependency'snametotheprovidedversion:

"provides":{

"@foxx/auth":"1.0.0"

}

Adependenciesdefinitionmapsthelocalaliasofagivendependencyagainstitsnameandthesupportedversionrange(eitherasaJSONobjectorashorthandstring):

"dependencies":{

"mySessions":"@foxx/sessions:^2.0.0",

"myAuth":{

"name":"@foxx/auth",

"version":"^1.0.0",

"description":"Thisdescriptionisentirelyoptional.",

"required":false

}

}

Dependenciescanbeconfiguredfromthewebinterfaceinaservice'ssettingstabusingtheDependenciesbutton.

Thelocalaliasshouldbeavalididentifierfollowingthecase-insensitiveformat/^[_$a-z][-_$a-z0-9]*$/.

Thevalueforeachdependencyshouldbethedatabase-relativemountpathoftheservice(includingtheleadingslash).Inordertobeusableasthedependencyofanotherservicebothservicesneedtobemountedinthesamedatabase.Aservicecanbeusedtoprovidemultipledependenciesforthesameservice(aslongastheexpectedJavaScriptAPIsdon'tconflict).

Aservicethathasunconfiguredrequireddependenciescannotbeuseduntilallofitsdependencieshavebeenconfigured.

Itispossibletospecifythemountpathofaservicethatdoesnotactuallydeclarethedependencyasprovided.Thereiscurrentlynovalidationbeyondthemanifestformats.

Whenaserviceusesanothermountedserviceasadependencythedependency'smainentryfile'sexportsobjectbecomesavailableinthemodule.context.dependenciesobjectoftheotherservice:

Examples

ServiceAandServiceBaremountedinthesamedatabase.ServiceBhasadependencywiththelocalalias"greeter".ThedependencyisconfiguredtousethemountpathofServiceA.

//EntryfileofServiceA

module.exports={

sayHi(){

return'Hello';

}

};

//SomewhereinServiceB

constgreeter=module.context.dependencies.greeter;

res.write(greeter.sayHi());

Dependencies

208

RoutersconstcreateRouter=require('@arangodb/foxx/router');

RoutersletyoudefineroutesthatextendArangoDB'sHTTPAPIwithcustomendpoints.

RoutersneedtobemountedusingtheusemethodofaservicecontexttoexposetheirHTTProutesataservice'smountpath.

Youcanpassroutersbetweenservicesmountedinthesamedatabaseasdependencies.Youcanevennestrouterswithineachother.

CreatingaroutercreateRouter():Router

Thisreturnsanew,cleanrouterobjectthathasnotyetbeenmountedintheserviceandcanbeexportedlikeanyotherobject.

Requesthandlersrouter.get([path],[...middleware],handler,[name]):Endpoint

router.post([path],[...middleware],handler,[name]):Endpoint

router.put([path],[...middleware],handler,[name]):Endpoint

router.patch([path],[...middleware],handler,[name]):Endpoint

router.delete([path],[...middleware],handler,[name]):Endpoint

router.all([path],[...middleware],handler,[name]):Endpoint

Thesemethodsletyouspecifyroutesontherouter.TheallmethoddefinesaroutethatwillmatchanysupportedHTTPverb,theothermethodsdefineroutesthatonlymatchtheHTTPverbwiththesamename.

Arguments

path:string(Default:"/")

ThepathoftherequesthandlerrelativetothebasepaththeRouterismountedat.Ifomitted,therequesthandlerwillhandlerequeststothebasepathoftheRouter.Forinformationondefiningdynamicroutesseethesectiononpathparametersinthechapteronrouterendpoints.

middleware:Function(optional)

Zeroormoremiddlewarefunctionsthattakethefollowingarguments:

req:Request

Anincomingserverrequestobject.

res:Response

Anoutgoingserverresponseobject.

next:Function

Acallbackthatpassescontrolovertothenextmiddlewarefunctionandreturnswhenthatfunctionhascompleted.

Ifatruthyargumentispassed,thatargumentwillbethrownasanerror.

Ifthereisnonextmiddlewarefunction,thehandlerwillbeinvokedinstead(seebelow).

handler:Function

Afunctionthattakesthefollowingarguments:

req:Request

Anincomingserverrequestobject.

Routers

209

res:Response

Anoutgoingserverresponse.

name:string(optional)

AnamethatcanbeusedtogenerateURLsfortheendpoint.Formoreinformationseethereversemethodoftherequestobject.

ReturnsanEndpointfortheroute.

Examples

Simpleindexroute:

router.get(function(req,res){

res.set('content-type','text/plain');

res.write('HelloWorld!');

});

RestrictingaccesstoauthenticatedArangoDBusers:

router.get('/secrets',function(req,res,next){

if(req.arangoUser){

next();

}else{

res.throw(404,'Secrets?Whatsecrets?');

}

},function(req,res){

res.download('allOurSecrets.zip');

});

Multiplemiddlewarefunctions:

functioncounting(req,res,next){

if(!req.counter)req.counter=0;

req.counter++;

next();

req.counter--;

}

router.get(counting,counting,counting,function(req,res){

res.json({counter:req.counter});//{"counter":3}

});

Mountingchildroutersandmiddlewarerouter.use([path],middleware,[name]):Endpoint

Theusemethodletsyoumountachildrouterormiddlewareatagivenpath.

Arguments

path:string(optional)

ThepathofthemiddlewarerelativetothebasepaththeRouterismountedat.Ifomitted,themiddlewarewillhandlerequeststothebasepathoftheRouter.Forinformationondefiningdynamicroutesseethesectiononpathparametersinthechapteronrouterendpoints.

middleware:Router|Middleware

Anunmountedrouterobjectoramiddleware.

name:string(optional)

AnamethatcanbeusedtogenerateURLsforendpointsofthisrouter.Formoreinformationseethereversemethodoftherequestobject.HasnoeffectifhandlerisaMiddleware.

ReturnsanEndpointforthemiddlewareorchildrouter.

Routers

210

Routers

211

EndpointsEndpointsarereturnedbytheuse,allandHTTPverb(e.g.get,post)methodsofroutersaswellastheusemethodoftheservicecontext.Theycanbeusedtoattachmetadatatomountedroutes,middlewareandchildroutersthataffectshowrequestsandresponsesareprocessedorprovidesAPIdocumentation.

Endpointsshouldonlybeusedtoinvokethefollowingmethods.Endpointmethodscanbechainedtogether(eachmethodreturnstheendpointitself).

headerendpoint.header(name,[schema],[description]):this

Definesarequestheaderrecognizedbytheendpoint.Anyadditionalnon-definedheaderswillbetreatedasoptionalstringvalues.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisheaderdefinitionunlessoverridden.

Arguments

name:string

Nameoftheheader.Thisshouldbeconsideredcaseinsensitiveasallheadernameswillbeconvertedtolowercase.

schema:Schema(optional)

Aschemadescribingtheformatoftheheadervalue.Thiscanbeajoischemaoranythingthathasacompatiblevalidatemethod.

Thevalueofthisheaderwillbesettothevaluepropertyofthevalidationresult.Avalidationfailurewillresultinanautomatic400(BadRequest)errorresponse.

description:string(optional)

AhumanreadablestringthatwillbeshownintheAPIdocumentation.

Returnstheendpoint.

Examples

router.get(/*...*/)

.header('arangoVersion',joi.number().min(30000).default(30000));

pathParamendpoint.pathParam(name,[schema],[description]):this

Definesapathparameterrecognizedbytheendpoint.Pathparametersareexpectedtobefilledaspartoftheendpoint'smountpath.Anyadditionalnon-definedpathparameterswillbetreatedasoptionalstringvalues.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisparameterdefinitionunlessoverridden.

Arguments

name:string

Nameoftheparameter.

schema:Schema(optional)

Aschemadescribingtheformatoftheparameter.Thiscanbeajoischemaoranythingthathasacompatiblevalidatemethod.

Endpoints

212

Thevalueofthisparameterwillbesettothevaluepropertyofthevalidationresult.Avalidationfailurewillresultintheroutefailingtomatchandbeingignored(resultingina404(NotFound)errorresponseifnootherroutesmatch).

description:string(optional)

AhumanreadablestringthatwillbeshownintheAPIdocumentation.

Returnstheendpoint.

Examples

router.get('/some/:num/here',/*...*/)

.pathParam('num',joi.number().required());

queryParamendpoint.queryParam(name,[schema],[description]):this

Definesaqueryparameterrecognizedbytheendpoint.Anyadditionalnon-definedqueryparameterswillbetreatedasoptionalstringvalues.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisparameterdefinitionunlessoverridden.

Arguments

name:string

Nameoftheparameter.

schema:Schema(optional)

Aschemadescribingtheformatoftheparameter.Thiscanbeajoischemaoranythingthathasacompatiblevalidatemethod.

Thevalueofthisparameterwillbesettothevaluepropertyofthevalidationresult.Avalidationfailurewillresultinanautomatic400(BadRequest)errorresponse.

description:string(optional)

AhumanreadablestringthatwillbeshownintheAPIdocumentation.

Returnstheendpoint.

Examples

router.get(/*...*/)

.queryParam('num',joi.number().required());

bodyendpoint.body([model],[mimes],[description]):this

Definestherequestbodyrecognizedbytheendpoint.Therecanonlybeonerequestbodydefinitionperendpoint.ThedefinitionwillalsobeshownintheroutedetailsintheAPIdocumentation.

Intheabsenceofarequestbodydefinition,therequestobject'sbodypropertywillbeinitializedtotheunprocessedrawBodybuffer.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisbodydefinitionunlessoverridden.Iftheendpointisamiddleware,therequestbodywillonlybeparsedonce(i.e.theMIMEtypesoftheroutematchingthesamerequestwillbeignoredbutthebodywillstillbevalidatedagain).

Arguments

model:Model|Schema|null(optional)

Amodelorjoischemadescribingtherequestbody.Avalidationfailurewillresultinanautomatic400(BadRequest)errorresponse.

Endpoints

213

IfthevalueisamodelwithafromClientmethod,thatmethodwillbeappliedtotheparsedrequestbody.

Ifthevalueisaschemaoramodelwithaschema,theschemawillbeusedtovalidatetherequestbodyandthevaluepropertyofthevalidationresultoftheparsedrequestbodywillbeusedinsteadoftheparsedrequestbodyitself.

IfthevalueisamodeloraschemaandtheMIMEtypehasbeenomitted,theMIMEtypewilldefaulttoJSONinstead.

Ifthevalueisexplicitlysettonull,norequestbodywillbeexpected.

Ifthevalueisanarraycontainingexactlyonemodelorschema,therequestbodywillbetreatedasanarrayofitemsmatchingthatmodelorschema.

mimes:Array<string>(optional)

AnarrayofMIMEtypestheroutesupports.

Commonnon-mimealiaseslike"json"or"html"arealsosupportedandwillbeexpandedtotheappropriateMIMEtype(e.g."application/json"and"text/html").

IftheMIMEtypeisrecognizedbyFoxxtherequestbodywillbeparsedintotheappropriatestructurebeforebeingvalidated.CurrentlyonlyJSON,application/x-www-form-urlencodedandmultipartformatsaresupportedinthisway.

IftheMIMEtypeindicatedintherequestheadersdoesnotmatchanyofthesupportedMIMEtypes,thefirstMIMEtypeinthelistwillbeusedinstead.

Failuretoparsetherequestbodywillresultinanautomatic400(BadRequest)errorresponse.

description:string(optional)

AhumanreadablestringthatwillbeshownintheAPIdocumentation.

Returnstheendpoint.

Examples

router.post('/expects/some/json',/*...*/)

.body(

joi.object().required(),

'ThisimpliesJSON.'

);

router.post('/expects/nothing',/*...*/)

.body(null);//Nobodyallowed

router.post('/expects/some/plaintext',/*...*/)

.body(['text/plain'],'Thisbodywillbeastring.');

responseendpoint.response([status],[model],[mimes],[description]):this

Definesaresponsebodyfortheendpoint.Whenusingtheresponseobject'ssendmethodintherequesthandlerofthisroute,thedefinitionwiththematchingstatuscodewillbeusedtogeneratetheresponsebody.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisresponsedefinitionunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.

Arguments

status:number|string(Default:200or204)

HTTPstatuscodetheresponseappliesto.Ifastringisprovidedinsteadofanumericstatuscodeitwillbeusedtolookupanumericstatuscodeusingthestatusesmodule.

model:Model|Schema|null(optional)

Amodelorjoischemadescribingtheresponsebody.

Endpoints

214

IfthevalueisamodelwithaforClientmethod,thatmethodwillbeappliedtothedatapassedtoresponse.sendwithintherouteiftheresponsestatuscodematches(butalsoifnostatuscodehasbeenset).

Ifthevalueisaschemaoramodelwithaschema,theactualschemawillnotbeusedtovalidatetheresponsebodyandonlyservestodocumenttheresponseinmoredetailintheAPIdocumentation.

IfthevalueisamodeloraschemaandtheMIMEtypehasbeenomitted,theMIMEtypewilldefaulttoJSONinstead.

Ifthevalueisexplicitlysettonullandthestatuscodehasbeenomitted,thestatuscodewilldefaultto204("nocontent")insteadof200.

Ifthevalueisanarraycontainingexactlyonemodelorschema,theresponsebodywillbeanarrayofitemsmatchingthatmodelorschema.

mimes:Array<string>(optional)

AnarrayofMIMEtypestheroutemightrespondwithforthisstatuscode.

Commonnon-mimealiaseslike"json"or"html"arealsosupportedandwillbeexpandedtotheappropriateMIMEtype(e.g."application/json"and"text/html").

Whenusingtheresponse.sendmethodtheresponsebodywillbeconvertedtotheappropriateMIMEtypeifpossible.

description:string(optional)

Ahuman-readablestringthatbrieflydescribestheresponseandwillbeshownintheendpoint'sdetaileddocumentation.

Returnstheendpoint.

Examples

//Thisexampleonlyprovidesdocumentation

//andimpliesagenericJSONresponsebody.

router.get(/*...*/)

.response(

joi.array().items(joi.string()),

'Alistofdoodadidentifiers.'

);

//Noresponsebodywillbeexpectedhere.

router.delete(/*...*/)

.response(null,'Thedoodadnolongerexists.');

//Anendpointcandefinemultipleresponsetypes

//fordifferentstatuscodes--butnevermorethan

//oneforeachstatuscode.

router.post(/*...*/)

.response('found','Thedoodadislocatedelsewhere.')

.response(201,['text/plain'],'Thedoodadwascreatedsohereisahaiku.');

//Heretheresponsebodywillbesetto

//thequerystring-encodedresultof

//FormModel.forClient({some:'data'})

//becausethestatuscodedefaultsto200.

router.patch(function(req,res){

//...

res.send({some:'data'});

})

.response(FormModel,['application/x-www-form-urlencoded'],'OMG.');

//Inthiscasetheresponsebodywillbesetto

//SomeModel.forClient({some:'data'})because

//thestatuscodehasbeensetto201before.

router.put(function(req,res){

//...

res.status(201);

res.send({some:'data'});

})

.response(201,SomeModel,'Somethingamazinghappened.');

error

Endpoints

215

endpoint.error(status,[description]):this

Documentsanerrorstatusfortheendpoint.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethiserrordescriptionunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.

ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.

Arguments

status:number|string

HTTPstatuscodefortheerror(e.g.400for"badrequest").Ifastringisprovidedinsteadofanumericstatuscodeitwillbeusedtolookupanumericstatuscodeusingthestatusesmodule.

description:string(optional)

Ahuman-readablestringthatbrieflydescribestheerrorconditionandwillbeshownintheendpoint'sdetaileddocumentation.

Returnstheendpoint.

Examples

router.get(function(req,res){

//...

res.throw(403,'Validationerroratx.y.z');

})

.error(403,'Indicatesthatavalidationhasfailed.');

summaryendpoint.summary(summary):this

Addsashortdescriptiontotheendpoint'sAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethissummaryunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.

ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.

Arguments

summary:string

Ahuman-readablestringthatbrieflydescribestheendpointandwillappearnexttotheendpoint'spathinthedocumentation.

Returnstheendpoint.

Examples

router.get(/*...*/)

.summary('Listalldiscombobulateddoodads')

descriptionendpoint.description(description):this

Addsalongdescriptiontotheendpoint'sAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisdescriptionunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.

ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.

Arguments

Endpoints

216

description:string

Ahuman-readablestringthatdescribestheendpointindetailandwillbeshownintheendpoint'sdetaileddocumentation.

Returnstheendpoint.

Examples

//The"dedent"libraryhelpsformatting

//multi-linestringsbyadjustingindentation

//andremovingleadingandtrailingblanklines

constdd=require('dedent');

router.post(/*...*/)

.description(dd`

Thisroutediscombobulatesthedoodadsby

frobnicatingthemoxieoftherequestbody.

`)

deprecatedendpoint.deprecated([deprecated]):this

Markstheendpointasdeprecated.

Iftheendpointisachildrouter,allroutesofthatrouterwillalsobemarkedasdeprecated.Iftheendpointisamiddleware,thismethodhasnoeffect.

ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.

Arguments

deprecated:boolean(Default:true)

Whethertheendpointshouldbemarkedasdeprecated.Ifsettofalsetheendpointwillbeexplicitlymarkedasnotdeprecated.

Returnstheendpoint.

Examples

router.get(/*...*/)

.deprecated();

tagendpoint.tag(...tags):this

MarkstheendpointwiththegiventagsthatwillbeusedtogrouprelatedroutesinthegeneratedAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillalsobemarkedwiththetags.Iftheendpointisamiddleware,thismethodhasnoeffect.

ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.

Arguments

tags:string

Oneormorestringsthatwillbeusedtogrouptheendpoint'sroutes.

Returnstheendpoint.

Examples

router.get(/*...*/)

.tag('auth','restricted');

Endpoints

217

Endpoints

218

MiddlewareMiddlewareinFoxxreferstofunctionsthatcanbemountedlikeroutesandcanmanipulatetherequestandresponseobjectsbeforeandaftertherouteitselfisinvoked.Theycanalsobeusedtocontrolaccessortoprovidecommonlogiclikeloggingetc.Unlikeroutes,middlewareismountedwiththeusemethodlikearouter.

Insteadofafunctiontheusemethodcanalsoacceptanobjectwitharegisterfunctionthatwilltakeaparameterendpoint,themiddlewarewillbemountedatandreturnstheactualmiddlewarefunction.Thisallowsmanipulatingtheendpointbeforecreatingthemiddleware(e.g.todocumentheaders,requestbodies,pathparametersorqueryparameters).

Examples

RestrictaccesstoArangoDB-authenticatedusers:

module.context.use(function(req,res,next){

if(!req.arangoUser){

res.throw(401,'NotauthenticatedwithArangoDB');

}

next();

});

Anytruthyargumentpassedtothenextfunctionwillbethrownasanerror:

module.context.use(function(req,res,next){

leterr=null;

if(!req.arangoUser){

err=newError('Thisshouldneverhappen');

}

next(err);//throwsiftheerrorwasset

})

Trivialloggingmiddleware:

module.context.use(function(req,res,next){

conststart=Date.now();

try{

next();

}finally{

console.log(`Handledrequestin${Date.now()-start}ms`);

}

});

Morecomplexexampleforheader-basedsessions:

constsessions=module.context.collection('sessions');

module.context.use({

register(endpoint){

endpoint.header('x-session-id',joi.string().optional(),'ThesessionID.');

returnfunction(req,res,next){

constsid=req.get('x-session-id');

if(sid){

try{

req.session=sessions.document(sid);

}catch(e){

deletereq.headers['x-session-id'];

}

}

next();

if(req.session){

if(req.session._rev){

sessions.replace(req.session,req.session);

res.set('x-session-id',req.session._key);

}else{

constmeta=sessions.save(req.session);

res.set('x-session-id',meta._key);

}

Middleware

219

}

};

}

});

Middleware

220

RequestobjectsTherequestobjectspecifiesthefollowingproperties:

arangoUser:string|null

TheauthenticatedArangoDBusernameusedtomaketherequest.ThisvalueisonlysetifauthenticationisenabledinArangoDBandtherequestsetanauthorizationheaderArangoDBwasabletoverify.YouarestronglyencouragedtoimplementyourownauthenticationlogicforyourownservicesbutthispropertycanbeusefulifyouneedtointegratewithArangoDB'sownauthenticationmechanisms.

arangoVersion:number

Thenumericvalueofthex-arango-versionheaderorthenumericversionoftheArangoDBserver(e.g.30102forversion3.1.2)ifnovalidheaderwasprovided.

baseUrl:string

Root-relativebaseURLoftheservice,i.e.theprefix"/_db/"followedbythevalueofdatabase.

body:any

Theprocessedandvalidatedrequestbodyforthecurrentroute.Ifnobodyhasbeendefinedforthecurrentroute,thevaluewillbeidenticaltorawBody.

FordetailsonhowrequestbodiescanbeprocessedandvalidatedbyFoxxseethebodymethodoftheendpointobject.

context:Context

Theservicecontextinwhichtherouterwasmounted(ratherthanthecontextinwhichtheroutewasdefined).

database:string

Thenameofthedatabaseinwhichtherequestisbeinghandled,e.g."_system".

headers:object

Therawheadersobject.

FordetailsonhowrequestheaderscanbevalidatedbyFoxxseetheheadermethodoftheendpointobject.

hostname:string

Thehostname(domainname)indicatedintherequestheaders.

Defaultstothehostnameportion(i.e.excludingtheport)oftheHostheaderandfallsbacktothelisteningaddressoftheserver.

method:string

TheHTTPverbusedtomaketherequest,e.g."GET".

originalUrl:string

Root-relativeURLoftherequest,i.e.pathfollowedbytherawqueryparameters,ifany.

path:string

Database-relativepathoftherequestURL(notincludingthequeryparameters).

pathParams:object

Anobjectmappingthenamesofpathparametersofthecurrentroutetotheirvalidatedvalues.

FordetailsonhowpathparameterscanbevalidatedbyFoxxseethepathParammethodoftheendpointobject.

port:number

Theportindicatedintherequestheaders.

Request

221

Defaultstotheportportion(i.e.excludingthehostname)oftheHostheaderandfallsbacktothelisteningportortheappropriatedefaultport(443forHTTPSor80forHTTP,dependingonsecure)iftheheaderonlyindicatesahostname.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettotheportportionoftheX-Forwarded-Hostheader(orapproriatedefaultport)ifpresent.

protocol:string

Theprotocolusedfortherequest.

Defaultsto"https"or"http"dependingonwhetherArangoDBisconfiguredtouseSSLornot.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettothevalueoftheX-Forwarded-Protoheaderifpresent.

queryParams:object

Anobjectmappingthenamesofqueryparametersofthecurrentroutetotheirvalidatedvalues.

FordetailsonhowqueryparameterscanbevalidatedbyFoxxseethequeryParammethodoftheendpointobject.

rawBody:Buffer

Theraw,unparsed,unvalidatedrequestbodyasabuffer.

remoteAddress:string

TheIPoftheclientthatmadetherequest.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettothefirstIPlistedintheX-Forwarded-Forheaderifpresent.

remoteAddresses:Array<string>

AlistcontainingtheIPaddressesusedtomaketherequest.

DefaultstothevalueofremoteAddresswrappedinanarray.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettothelistofIPsspecifiedintheX-Forwarded-Forheaderifpresent.

remotePort:number

Thelisteningportoftheclientthatmadetherequest.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettotheportspecifiedintheX-Forwarded-Portheaderifpresent.

secure:boolean

Whethertherequestwasmadeoverasecureconnection(i.e.HTTPS).

Thisissettofalsewhenprotocolis"http"andtruewhenprotocolis"https".

suffix:string

Thetrailingpathrelativetothecurrentrouteifthecurrentrouteendsinawildcard(e.g./something/*).

Note:StartingwithArangoDB3.2ispassedintotheserviceas-is,i.e.percentageescapesequenceslike%2Fwillnolongerbeunescaped.Alsonotethatthesuffixmaycontainpathsegmentslike..whichmayhavespecialmeaningifthesuffixisusedtobuildfilesystempaths.

trustProxy:boolean

Indicateswhethertherequestwasmadeusingatrustedproxy.Iftheoriginserver'saddresswasspecifiedintheArangoDBconfigurationusing--frontend.trusted-proxyortheservice'strustProxysettingisenabled,thiswillbetrue,otherwiseitwillbefalse.

url:string

TheURLoftherequest.

Request

222

xhr:boolean

WhethertherequestindicatesitwasmadewithinabrowserusingAJAX.

ThisissettotrueiftheX-Requested-Withheaderispresentandisacase-insensitivematchforthevalue"xmlhttprequest".

NotethatthisvaluedoesnotguaranteewhethertherequestwasmadefrominsideabrowserorwhetherAJAXwasusedandismerelyaconventionestablishedbyJavaScriptframeworkslikejQuery.

acceptsreq.accepts(types):string|false

req.accepts(...types):string|false

req.acceptsCharsets(charsets):string|false

req.acceptsCharsets(...charsets):string|false

req.acceptsEncodings(encodings):string|false

req.acceptsEncodings(...encodings):string|false

req.acceptsLanguages(languages):string|false

req.acceptsLanguages(...languages):string|false

Thesemethodswrapthecorrespondingcontentnegotiationmethodsoftheacceptsmoduleforthecurrentrequest.

Examples

if(req.accepts(['json','html'])==='html'){

//ClientexplicitlyprefersHTMLoverJSON

res.write('<h1>ClientprefersHTML</h1>');

}else{

//OtherwisejustsendJSON

res.json({success:true});

}

cookiereq.cookie(name,options):string|null

Getsthevalueofacookiebyname.

Arguments

name:string

Nameofthecookie.

options:object(optional)

Anobjectwithanyofthefollowingproperties:

secret:string(optional)

Secretthatwasusedtosignthecookie.

Ifasecretisspecified,thecookie'ssignatureisexpectedtobepresentinasecondcookiewiththesamenameandthesuffix.sig.Otherwisethesignature(ifpresent)willbeignored.

algorithm:string(Default:"sha256")

Algorithmthatwasusedtosignthecookie.

Ifastringispassedinsteadofanoptionsobjectitwillbeinterpretedasthesecretoption.

Returnsthevalueofthecookieornullifthecookieisnotsetoritssignatureisinvalid.

Request

223

get/headerreq.get(name):string

req.header(name):string

Getsthevalueofaheaderbyname.Youcanvalidaterequestheadersusingtheheadermethodoftheendpoint.

Arguments

name:string

Nameoftheheader.

Returnstheheadervalue.

isreq.is(types):string

req.is(...types):string

Thismethodwrapsthe(requestbody)contenttypedetectionmethodofthetype-ismoduleforthecurrentrequest.

Examples

consttype=req.is('html','application/xml','application/*+xml');

if(type===false){//nomatch

handleDefault(req.rawBody);

}elseif(type==='html'){

handleHtml(req.rawBody);

}else{//isXML

handleXml(req.rawBody);

}

jsonreq.json():any

AttemptstoparsetherawrequestbodyasJSONandreturnstheresult.

Itisgenerallymoreusefultodefinearequestbodyontheendpointandusethereq.bodypropertyinstead.

Returnsundefinediftherequestbodyisempty.MaythrowaSyntaxErrorifthebodycouldnotbeparsed.

makeAbsolutereq.makeAbsolute(path,[query]):string

Resolvesthegivenpathrelativetothereq.context.service'smountpathtoafullURL.

Arguments

path:string

Thepathtoresovle.

query:string|object

AstringorobjectwithqueryparameterstoaddtotheURL.

ReturnstheformattedabsoluteURL.

paramsreq.param(name):any

Request

224

Arguments

Looksupaparameterbyname,preferringpathParamsoverqueryParams.

It'sprobablybetterstyletousethereq.pathParamsorreq.queryParamsobjectsdirectly.

name:string

Nameoftheparameter.

Returnsthe(validated)valueoftheparameter.

rangereq.range([size]):Ranges|number

Thismethodwrapstherangeheaderparsingmethodoftherange-parsermoduleforthecurrentrequest.

Arguments

size:number(Default:Infinity)

Lengthofthesatisfiablerange(e.g.numberofbytesinthefullresponse).Ifpresent,rangesexceedingthesizewillbeconsideredunsatisfiable.

ReturnsundefinediftheRangeheaderisabsent,-2iftheheaderispresentbutmalformed,-1iftherangeisinvalid(e.g.startoffsetislargerthanendoffset)orunsatisfiableforthegivensize.

Otherwisereturnsanarrayofobjectswiththepropertiesstartandendvaluesforeachrange.Thearrayhasanadditionalpropertytypeindicatingtherequestrangetype.

Examples

console.log(req.headers.range);//"bytes=40-80"

constranges=req.range(100);

console.log(ranges);//[{start:40,end:80}]

console.log(ranges.type);//"bytes"

reversereq.reverse(name,[params]):string

LooksuptheURLofanamedrouteforthegivenparameters.

Arguments

name:string

Nameoftheroutetolookup.

params:object(optional)

Anobjectcontainingvaluesforthe(pathorquery)parametersoftheroute.

ReturnstheURLoftherouteforthegivenparameters.

Examples

router.get('/items/:id',function(req,res){

/*...*/

},'getItemById');

router.post('/items',function(req,res){

//...

consturl=req.reverse('getItemById',{id:createdItem._key});

res.set('location',req.makeAbsolute(url));

});

Request

225

Request

226

ResponseobjectsTheresponseobjectspecifiesthefollowingproperties:

body:Buffer|string

Responsebodyasastringorbuffer.Canbesetdirectlyorusingsomeoftheresponsemethods.

context:Context

Theservicecontextinwhichtherouterwasmounted(ratherthanthecontextinwhichtheroutewasdefined).

headers:object

Therawheadersobject.

statusCode:number

Statuscodeoftheresponse.Defaultsto200(bodysetandnotanemptystringorbuffer)or204(otherwise)ifnotchangedfromundefined.

attachmentres.attachment([filename]):this

Setsthecontent-dispositionheadertoindicatetheresponseisadownloadablefilewiththegivenname.

Note:Thisdoesnotactuallymodifytheresponsebodyoraccessthefilesystem.TosendafilefromthefilesystemseethedownloadorsendFilemethods.

Arguments

filename:string(optional)

Nameofthedownloadablefileintheresponsebody.

Ifpresent,theextensionofthefilenamewillbeusedtosettheresponsecontent-typeifithasnotyetbeenset.

Returnstheresponseobject.

cookieres.cookie(name,value,[options]):this

Setsacookiewiththegivenname.

Arguments

name:string

Nameofthecookie.

value:string

Valueofthecookie.

options:object(optional)

Anobjectwithanyofthefollowingproperties:

ttl:number(optional)

Timetoliveofthecookieinseconds.

algorithm:string(Default:"sha256")

Algorithmthatwillbeusedtosignthecookie.

Response

227

secret:string(optional)

Secretthatwillbeusedtosignthecookie.

Ifasecretisspecified,thecookie'ssignaturewillbestoredinasecondcookiewiththesameoptions,thesamenameandthesuffix.sig.Otherwisenosignaturewillbeadded.

path:string(optional)

Pathforwhichthecookieshouldbeissued.

domain:string(optional)

Domainforwhichthecookieshouldbeissued.

secure:boolean(Default:false)

Whetherthecookieshouldbemarkedassecure(i.e.HTTPS/SSL-only).

httpOnly:boolean(Default:false)

WhetherthecookieshouldbemarkedasHTTP-only(ratherthanalsoexposingittoclient-sidecode).

Ifastringispassedinsteadofanoptionsobjectitwillbeinterpretedasthesecretoption.

Ifanumberispassedinsteadofanoptionsobjectitwillbeinterpretedasthettloption.

Returnstheresponseobject.

downloadres.download(path,[filename]):this

Theequivalentofcallingres.attachment(filename).sendFile(path).

Arguments

path:string

Pathtothefileonthelocalfilesystemtobesentastheresponsebody.

filename:string(optional)

Filenametoindicateinthecontent-dispositionheader.

Ifomittedthepathwillbeusedinstead.

Returnstheresponseobject.

getHeaderres.getHeader(name):string

Getsthevalueoftheheaderwiththegivenname.

Arguments

name:string

Nameoftheheadertoget.

Returnsthevalueoftheheaderorundefined.

jsonres.json(data):this

SetstheresponsebodytotheJSONstringvalueofthegivendata.

Response

228

Arguments

data:any

Thedatatobeusedastheresponsebody.

Returnstheresponseobject.

redirectres.redirect([status],path):this

Redirectstheresponsebysettingtheresponselocationheaderandstatuscode.

Arguments

status:number|string(optional)

Responsestatuscodetoset.

Ifthestatuscodeisthestringvalue"permanent"itwillbetreatedasthevalue301.

Ifthestatuscodeisastringitwillbeconvertedtoanumericstatuscodeusingthestatusesmodulefirst.

Ifthestatuscodeisomittedbuttheresponsestatushasnotalreadybeenset,theresponsestatuswillbesetto302.

path:string

URLtosetthelocationheaderto.

Returnstheresponseobject.

removeHeaderres.removeHeader(name):this

Removestheheaderwiththegivennamefromtheresponse.

Arguments

name:string

Nameoftheheadertoremove.

Returnstheresponseobject.

sendres.send(data,[type]):this

Setstheresponsebodytothegivendatawithrespecttotheresponsedefinitionfortheresponse'scurrentstatuscode.

Arguments

data:any

Thedatatobeusedastheresponsebody.Willbeconvertedaccordingtheresponsedefinitionfortheresponse'scurrentstatuscode(or200)inthefollowingway:

IfthedataisanArangoDBresultset,itwillbeconvertedtoanarrayfirst.

IftheresponsedefinitionspecifiesamodelwithaforClientmethod,thatmethodwillbeappliedtothedatafirst.Ifthedataisanarrayandtheresponsedefinitionhasthemultipleflagset,themethodwillbeappliedtoeachentryindividuallyinstead.

Finallythedatawillbeprocessedbytheresponsetypehandlertoconvertheresponsebodytoastringorbuffer.

type:string(Default:"auto")

Response

229

Content-typeoftheresponsebody.

Ifsetto"auto"thefirstMIMEtypespecifiedintheresponsedefinitionfortheresponse'scurrentstatuscode(or200)willbeusedinstead.

Ifsetto"auto"andnoresponsedefinitionexists,theMIMEtypewillbedeterminedthefollowingway:

IfthedataisabuffertheMIMEtypewillbesettobinary(application/octet-stream).

IfthedataisanobjecttheMIMEtypewillbesettoJSONandthedatawillbeconvertedtoaJSONstring.

OtherwisetheMIMEtypewillbesettoHTMLandthedatawillbeconvertedtoastring.

Returnstheresponseobject.

sendFileres.sendFile(path,[options]):this

Sendsafilefromthelocalfilesystemastheresponsebody.

Arguments

path:string

Pathtothefileonthelocalfilesystemtobesentastheresponsebody.

Ifnocontent-typeheaderhasbeensetyet,theextensionofthefilenamewillbeusedtosetthevalueofthatheader.

options:object(optional)

Anobjectwithanyofthefollowingproperties:

lastModified:boolean(optional)

Ifsettotrueorifnolast-modifiedheaderhasbeensetyetandthevalueisnotsettofalsethelast-modifiedheaderwillbesettothemodificationdateofthefileinmilliseconds.

Returnstheresponseobject.

Examples

//Sendthefile"favicon.ico"fromthisservice'sfolder

res.sendFile(module.context.fileName('favicon.ico'));

sendStatusres.sendStatus(status):this

Sendsaplaintextresponseforthegivenstatuscode.Theresponsestatuswillbesettothegivenstatuscode,theresponsebodywillbesettothestatusmessagecorrespondingtothatstatuscode.

Arguments

status:number|string

Responsestatuscodetoset.

Ifthestatuscodeisastringitwillbeconvertedtoanumericstatuscodeusingthestatusesmodulefirst.

Returnstheresponseobject.

setHeader/setres.setHeader(name,value):this

res.set(name,value):this

Response

230

res.set(headers):this

Setsthevalueoftheheaderwiththegivenname.

Arguments

name:string

Nameoftheheadertoset.

value:string

Valuetosettheheaderto.

headers:object

Headerobjectmappingheadernamestovalues.

Returnstheresponseobject.

statusres.status(status):this

Setstheresponsestatustothegivenstatuscode.

Arguments

status:number|string

Responsestatuscodetoset.

Ifthestatuscodeisastringitwillbeconvertedtoanumericstatuscodeusingthestatusesmodulefirst.

Returnstheresponseobject.

throwres.throw(status,[reason],[options]):void

ThrowsanHTTPexceptionforthegivenstatus,whichwillbehandledbyFoxxtoservetheappropriateJSONerrorresponse.

Arguments

status:number|string

Responsestatuscodetoset.

Ifthestatuscodeisastringitwillbeconvertedtoanumericstatuscodeusingthestatusesmodulefirst.

Ifthestatuscodeisinthe500-range(500-599),itsstacktracewillalwaysbeloggedasifitwereanunhandledexception.

Ifdevelopmentmodeisenabled,theerror'sstacktracewillbeloggedasawarningifthestatuscodeisinthe400-range(400-499)orasaregularmessageotherwise.

reason:string(optional)

Messagefortheexception.

Ifomitted,thestatusmessagecorrespondingtothestatuscodewillbeusedinstead.

options:object(optional)

Anobjectwithanyofthefollowingproperties:

cause:Error(optional)

Causeoftheexceptionthatwillbeloggedaspartoftheerror'sstacktrace(recursively,iftheexceptionalsohasacausepropertyandsoon).

Response

231

extra:object(optional)

AdditionalpropertiesthatwillbeaddedtotheerrorresponsebodygeneratedbyFoxx.

Ifdevelopmentmodeisenabled,anexceptionpropertywillbeaddedtothisvaluecontainingtheerrormessageandastacktracepropertywillbeaddedcontaininganarraywitheachlineoftheerror'sstacktrace.

Ifanerrorispassedinsteadofanoptionsobjectitwillbeinterpretedasthecauseoption.Ifnoreasonwasprovidedtheerror'smessagewillbeusedasthereasoninstead.

Returnsnothing.

typeres.type([type]):string

Setstheresponsecontent-typetothegiventypeifprovidedorreturnsthepreviouslysetcontent-type.

Arguments

type:string(optional)

Content-typeoftheresponsebody.

Unlikeres.set('content-type',type)fileextensionscanbeprovidedasvaluesandwillbetranslatedtothecorrespondingMIMEtype(e.g.jsonbecomesapplication/json).

Returnsthecontent-typeoftheresponsebody.

varyres.vary(names):this

res.vary(...names):this

Thismethodwrapsthevaryheadermanipulationmethodofthevarymoduleforthecurrentresponse.

Thegivennameswillbeaddedtotheresponse'svaryheaderifnotalreadypresent.

Returnstheresponseobject.

Examples

res.vary('user-agent');

res.vary('cookie');

res.vary('cookie');//duplicateswillbeignored

//--or--

res.vary('user-agent','cookie');

//--or--

res.vary(['user-agent','cookie']);

writeres.write(data):this

Appendsthegivendatatotheresponsebody.

Arguments

data:string|Buffer

Datatoappend.

Response

232

Ifthedataisabuffertheresponsebodywillbeconvertedtoabufferfirst.

Iftheresponsebodyisabufferthedatawillbeconvertedtoabufferfirst.

IfthedataisanobjectitwillbeconvertedtoaJSONstringfirst.

Ifthedataisanyothernon-stringvalueitwillbeconvertedtoastringfirst.

Returnstheresponseobject.

Response

233

UsingGraphQLinFoxxconstcreateGraphQLRouter=require('@arangodb/foxx/graphql');

Foxxbundlesthegraphql-syncmodule,whichisasynchronouswrapperfortheofficialJavaScriptGraphQLreferenceimplementation,toallowwritingGraphQLschemasdirectlyinsideFoxx.

Additionallythe@arangodb/foxx/graphqlletsyoucreateroutersforservingGraphQLrequests,whichcloselymimicksthebehaviouroftheexpress-graphqlmodule.

Formoreinformationongraphql-syncseethegraphql-jsAPIreference(notethatgraphql-syncalwaysusesrawvaluesinsteadofwrappingtheminpromises).

ForanexampleofaGraphQLschemainFoxxthatresolvesfieldsusingthedatabaseseetheGraphQLexampleservice(alsoavailablefromtheFoxxstore).

Examples

constgraphql=require('graphql-sync');

constgraphqlSchema=newgraphql.GraphQLSchema({

//...

});

//Mountingagraphqlendpointdirectlyinaservice:

module.context.use('/graphql',createGraphQLRouter({

schema:graphqlSchema,

graphiql:true

}));

//Orattheservice'srootURL:

module.context.use(createGraphQLRouter({

schema:graphqlSchema,

graphiql:true

}));

//Orinsideanexistingrouter:

router.get('/hello',function(req,res){

res.write('Helloworld!');

});

router.use('/graphql',createGraphQLRouter({

schema:graphqlSchema,

graphiql:true

}));

Note:ArangoDBaimsforstabilitywhichmeansbundleddependencieswillgenerallynotbeupdatedasquicklyastheirmaintainersmakeupdatesavailableonGitHuborNPM.StartingwithArangoDB3.2,ifyouwanttouseanewerreleaseofgraphql-syncthantheonebundledwithyourtargetversionofArangoDB,youcanprovideyourownversionofthelibrarybypassingitviathegraphqloption:

constgraphql=require('graphql-sync');

constgraphqlSchema=newgraphql.Schema({

//...

});

module.context.use(createGraphQLRouter({

schema:graphqlSchema,

graphiql:true,

graphql:graphql

}))

ThismakessureFoxxusesthegraphql-syncmodulebundledinyourservice'snode_modulesfolder(ifavailable)insteadofthebuilt-inversion.Ifyoufindgraphql-syncitselflaggingbehindtheofficialgraphqlmodule,consideropeninganissueonitsGitHubrepository.

CreatingaroutercreateGraphQLRouter(options):Router

UsingGraphQL

234

ThisreturnsanewrouterobjectwithPOSTandGETroutesforservingGraphQLrequests.

Arguments

options:object

Anobjectwithanyofthefollowingproperties:

schema:GraphQLSchema

AGraphQLSchemaobjectfromgraphql-sync.

context:any(optional)

TheGraphQLcontextthatwillbepassedtothegraphql()functionfromgraphql-synctohandleGraphQLqueries.

rootValue:object(optional)

TheGraphQLrootvaluethatwillbepassedtothegraphql()functionfromgraphql-synctohandleGraphQLqueries.

pretty:boolean(Default:false)

Iftrue,JSONresponseswillbepretty-printed.

formatError:Function(optional)

Afunctionthatwillbeusedtoformaterrorsproducedbygraphql-sync.Ifomitted,theformatErrorfunctionfromgraphql-syncwillbeusedinstead.

validationRules:Array<any>(optional)

AdditionalvalidationrulesqueriesmustsatisfyinadditiontothosedefinedintheGraphQLspec.

graphiql:boolean(Default:false)

Iftrue,theGraphiQLexplorerwillbeservedwhenloadeddirectlyfromabrowser.

graphql:object(optional)

Ifyouneedtouseyourowncopyofthegraphql-syncmoduleinsteadoftheonebundledwithArangoDB,hereyoucanpassitindirectly.

IfaGraphQLSchemaobjectispassedinsteadofanoptionsobjectitwillbeinterpretedastheschemaoption.

Generatedroutes

TherouterhandlesGETandPOSTrequeststoitsrootpathandacceptsthefollowingparameters,whichcanbeprovidedeitherasqueryparametersorasthePOSTrequestbody:

query:string

AGraphQLquerythatwillbeexecuted.

variables:object|string(optional)

AnobjectorastringcontainingaJSONobjectwithruntimevaluestouseforanyGraphQLqueryvariables.

operationName:string(optional)

Iftheprovidedquerycontainsmultiplenamedoperations,thisspecifieswhichoperationshouldbeexecuted.

raw:boolean(Default:false)

ForcesaJSONresponseevenifgraphiqlisenabledandtherequestwasmadeusingabrowser.

ThePOSTrequestbodycanbeprovidedasJSONorasquerystringusingapplication/x-www-form-urlencoded.Arequestbodypassedasapplication/graphqlwillbeinterpretedasthequeryparameter.

UsingGraphQL

235

SessionMiddlewareconstsessionMiddleware=require('@arangodb/foxx/sessions');

ThesessionmiddlewareaddsthesessionandsessionStoragepropertiestotherequestobjectanddealswithserializinganddeserializingthesessionaswellasextractingsessionidentifiersfromincomingrequestsandinjectingthemintooutgoingresponses.

Examples

//Createasessionmiddleware

constsessions=sessionsMiddleware({

storage:module.context.collection('sessions'),

transport:['header','cookie']

});

//Firstenablethemiddlewareforthisservice

module.context.use(sessions);

//Nowmounttheroutersthatusethesession

constrouter=createRouter();

module.context.use(router);

router.get('/',function(req,res){

res.send(`Hello${req.session.uid||'anonymous'}!`);

},'hello');

router.post('/login',function(req,res){

req.session.uid=req.body;

req.sessionStorage.save(req.session);

res.redirect(req.reverse('hello'));

});

.body(['text/plain'],'Username');

CreatingasessionmiddlewaresessionMiddleware(options):Middleware

Createsasessionmiddleware.

Arguments

options:Object

Anobjectwiththefollowingproperties:

storage:Storage

Storagethatwillbeusedtopersistthesessions.

ThestorageisalsoexposedasthesessionStorageonallrequestobjectsandasthestoragepropertyofthemiddleware.

IfastringorcollectionispassedinsteadofaStorage,itwillbeusedtocreateaCollectionStorage.

transport:Transport|Array<Transport>

Transportorarrayoftransportsthatwillbeusedtoextractthesessionidentifiersfromincomingrequestsandinjectthemintooutgoingresponses.Whenattemptingtoextractasessionidentifier,thetransportswillbeusedintheorderspecifieduntilamatchisfound.Wheninjecting(orclearing)sessionidentifiers,alltransportswillbeinvoked.

Thetransportsarealsoexposedasthetransportpropertyofthemiddleware.

Ifthestring"cookie"ispassedinsteadofaTransport,theCookieTransportwillbeusedwiththedefaultsettingsinstead.

Ifthestring"header"ispassedinsteadofaTransport,theHeaderTransportwillbeusedwiththedefaultsettingsinstead.

autoCreate:boolean(Default:true)

Ifenabledthesessionstorage'snewmethodwillbeinvokedtocreateanemptysessionwheneverthetransportfailedtoreturnasessionfortheincomingrequest.Otherwisethesessionwillbeinitializedasnull.

Sessionsmiddleware

236

Returnsthesessionmiddleware.

Sessionsmiddleware

237

SessionStoragesSessionstoragesareusedbythesessionsmiddlewaretopersistsessionsacrossrequests.SessionstoragesmustimplementthefromClientandforClientmethodsandcanoptionallyimplementthenewmethod.

Thebuilt-insessionstoragesgenerallyprovidethefollowingattributes:

uid:string(Default:null)

Auniqueidentifierindicatingtheactiveuser.

created:number(Default:Date.now())

Thenumerictimestampofwhenthesessionwascreated.

data:any(Default:null)

Arbitrarydatatopersistedinthesession.

newstorage.new():Session

Generatesanewsessionobjectrepresentinganemptysession.Theemptysessionobjectshouldnotbepersistedunlessnecessary.Thereturnvaluewillbeexposedbythemiddlewareasthesessionpropertyoftherequestobjectifnosessionidentifierwasreturnedbythesessiontransportsandauto-creationisnotexplicitlydisabledinthesessionmiddleware.

Examples

new(){

return{

uid:null,

created:Date.now(),

data:null

};

}

fromClientstorage.fromClient(sid):Session|null

Resolvesordeserializesasessionidentifiertoasessionobject.

Arguments

sid:string

Sessionidentifiertoresolveordeserialize.

Returnsasessionobjectrepresentingthesessionwiththegivensessionidentifierthatwillbeexposedbythemiddlewareasthesessionpropertyoftherequestobject.Thismethodwillonlybecalledifanyofthesessiontransportsreturnedasessionidentifier.Ifthesessionidentifierisinvalidorexpired,themethodshouldreturnanullvaluetoindicatenomatchingsession.

Examples

fromClient(sid){

returndb._collection('sessions').firstExample({_key:sid});

}

forClientstorage.forClient(session):string|null

Sessionstorages

238

Derivesasessionidentifierfromthegivensessionobject.

Arguments

session:Session

Sessiontoderiveasessionidentifierfrom.

Returnsasessionidentifierforthesessionrepresentedbythegivensessionobject.Thismethodwillbecalledwiththesessionpropertyoftherequestobjectunlessthatpropertyisempty(e.g.null).

Examples

forClient(session){

if(!session._key){

constmeta=db._collection('sessions').save(session);

returnmeta._key;

}

db._collection('sessions').replace(session._key,session);

returnsession._key;

}

Sessionstorages

239

CollectionSessionStorageconstcollectionStorage=require('@arangodb/foxx/sessions/storages/collection');

Thecollectionsessionstoragepersistssessionstoacollectioninthedatabase.

CreatingastoragecollectionStorage(options):Storage

CreatesaStoragethatcanbeusedinthesessionsmiddleware.

Arguments

options:Object

Anobjectwiththefollowingproperties:

collection:ArangoCollection

Thecollectionthatshouldbeusedtopersistthesessions.Ifastringispassedinsteadofacollectionitisassumedtobethefullyqualifiednameofacollectioninthecurrentdatabase.

ttl:number(Default:60*60)

Thetimeinsecondssincethelastupdateuntilasessionwillbeconsideredexpired.

pruneExpired:boolean(Default:false)

Whetherexpiredsessionsshouldberemovedfromthecollectionwhentheyareaccessedinsteadofsimplybeingignored.

autoUpdate:boolean(Default:true)

Whethersessionsshouldbeupdatedinthecollectioneverytimetheyareaccessedtokeepthemfromexpiring.Disablingthisoptionwillimproveperformancebutmeansyouwillhavetotakecareofkeepingyoursessionsaliveyourself.

Ifastringorcollectionispassedinsteadofanoptionsobject,itwillbeinterpretedasthecollectionoption.

prunestorage.prune():Array<string>

Removesallexpiredsessionsfromthecollection.ThismethodshouldbecalledevenifthepruneExpiredoptionisenabledtocleanupabandonedsessions.

Returnsanarrayofthekeysofallsessionsthatwereremoved.

savestorage.save(session):Session

Saves(replaces)thegivensessionobjectinthecollection.Thismethodneedstobeinvokedexplicitlyaftermakingchangestothesessionorthechangeswillnotbepersisted.Assignsanew_keytothesessionifitpreviouslydidnothaveone.

Arguments

session:Session

Asessionobject.

Returnsthemodifiedsession.

clear

Sessionstorages

240

storage.clear(session):boolean

Removesthesessionfromthecollection.Hasnoeffectifthesessionwasalreadyremovedorhasnotyetbeensavedtothecollection(i.e.hasno_key).

Arguments

session:Session

Asessionobject.

Returnstrueifthesessionwasremovedorfalseifithadnoeffect.

Sessionstorages

241

JWTSessionStorageconstjwtStorage=require('@arangodb/foxx/sessions/storages/jwt');

TheJWTsessionstorageconvertssessionstoandfromJSONWebTokens.

Examples

//PassinasecuresecretfromtheFoxxconfiguration

constsecret=module.context.configuration.jwtSecret;

constsessions=sessionsMiddleware({

storage:jwtStorage(secret),

transport:'header'

});

module.context.use(sessions);

CreatingastoragejwtStorage(options):Storage

CreatesaStoragethatcanbeusedinthesessionsmiddleware.

Note:whilethe"none"algorithm(i.e.nosignature)issupportedthisdummyalgorithmprovidesnosecurityandallowsclientstomakearbitrarymodificationstothepayloadandshouldnotbeusedunlessyouarecertainyouspecificallyneedit.

Arguments

options:Object

Anobjectwiththefollowingproperties:

algorithm:string(Default:"HS512")

Thealgorithmtouseforsigningthetoken.

Supportedvalues:

"HS256"(HMAC-SHA256)"HS384"(HMAC-SHA384)"HS512"(HMAC-SHA512)"none"(nosignature)

secret:string

Thesecrettouseforsigningthetoken.

Thisfieldisforbiddenwhenusingthe"none"algorithmbutrequiredotherwise.

ttl:number(Default:3600)

Themaximumlifetimeofthetokeninseconds.Youmaywanttokeepthisshortasanewtokenisgeneratedoneveryrequestallowingclientstorefreshtokensautomatically.

verify:boolean(Default:true)

Ifsettofalsethesignaturewillnotbeverifiedbutstillgenerated(unlessusingthe"none"algorithm).

maxExp:number(Default:Infinity)

LargestvaluethatwillbeacceptedinanincomingJWTexp(expiration)field.

Ifastringispassedinsteadofanoptionsobjectitwillbeinterpretedasthesecretoption.

Sessionstorages

242

SessionTransportsSessiontransportsareusedbythesessionsmiddlewaretostoreandretrievesessionidentifiersinrequestsandresponses.Sessiontransportsmustimplementthegetand/orsetmethodsandcanoptionallyimplementtheclearmethod.

gettransport.get(request):string|null

Retrievesasessionidentifierfromarequestobject.

Ifpresentthismethodwillautomaticallybeinvokedforeachtransportuntilatransportreturnsasessionidentifier.

Arguments

request:Request

Requestobjecttoextractasessionidentifierfrom.

Returnsthesessionidentifierornullifthetransportcannotfindasessionidentifierintherequest.

Examples

get(req){

returnreq.get('x-session-id')||null;

}

settransport.set(response,sid):void

Attachesasessionidentifiertoaresponseobject.

Ifpresentthismethodwillautomaticallybeinvokedattheendofarequestregardlessofwhetherthesessionwasmodifiedornot.

Arguments

response:Response

Responseobjecttoattachasessionidentifierto.

sid:string

Sessionidentifiertoattachtotheresponse.

Returnsnothing.

Examples

set(res){

res.set('x-session-id',value);

}

cleartransport.clear(response):void

Attachesapayloadindicatingthatthesessionhasbeenclearedtotheresponseobject.Thiscanbeusedtoclearasessioncookiewhenthesessionhasbeendestroyed(e.g.duringlogout).

Ifpresentthismethodwillautomaticallybeinvokedinsteadofsetwhenthereq.sessionattributewasremovedbytheroutehandler.

Arguments

Sessiontransports

243

response:Response

Responseobjecttoremovethesessionidentifierfrom.

Returnsnothing.

Sessiontransports

244

CookieSessionTransportconstcookieTransport=require('@arangodb/foxx/sessions/transports/cookie');

Thecookietransportstoressessionidentifiersincookiesontherequestandresponseobject.

Examples

//PassinasecuresecretfromtheFoxxconfiguration

constsecret=module.context.configuration.cookieSecret;

constsessions=sessionsMiddleware({

storage:module.context.collection('sessions'),

transport:cookieTransport({

name:'FOXXSESSID',

ttl:60*60*24*7,//oneweekinseconds

algorithm:'sha256',

secret:secret

})

});

module.context.use(sessions);

CreatingatransportcookieTransport([options]):Transport

CreatesaTransportthatcanbeusedinthesessionsmiddleware.

Arguments

options:Object(optional)

Anobjectwiththefollowingproperties:

name:string(Default:"sid")

Thenameofthecookie.

ttl:number(optional)

Cookielifetimeinseconds.NotethatthisdoesnotaffectthestorageTTL(i.e.howlongthesessionitselfisconsideredvalid),justhowlongthecookieshouldbestoredbytheclient.

algorithm:string(optional)

Thealgorithmusedtosignandverifythecookie.Ifnoalgorithmisspecified,thecookiewillnotbesignedorverified.Seethecookiemethodontheresponseobject.

secret:string(optional)

Secrettouseforthesignedcookie.Willbeignoredifnoalgorithmisprovided.

path:string(optional)

Pathforwhichthecookieshouldbeissued.

domain:string(optional)

Domainforwhichthecookieshouldbeissued.

secure:boolean(Default:false)

Whetherthecookieshouldbemarkedassecure(i.e.HTTPS/SSL-only).

httpOnly:boolean(Default:false)

WhetherthecookieshouldbemarkedasHTTP-only(ratherthanalsoexposingittoclient-sidecode).

Ifastringispassedinsteadofanoptionsobject,itwillbeinterpretedasthenameoption.

Sessiontransports

245

Sessiontransports

246

HeaderSessionTransportconstheaderTransport=require('@arangodb/foxx/sessions/transports/header');

Theheadertransportstoressessionidentifiersinheadersontherequestandresponseobjects.

Examples

constsessions=sessionsMiddleware({

storage:module.context.collection('sessions'),

transport:headerTransport('X-FOXXSESSID')

});

module.context.use(sessions);

CreatingatransportheaderTransport([options]):Transport

CreatesaTransportthatcanbeusedinthesessionsmiddleware.

Arguments

options:Object(optional)

Anobjectwiththefollowingproperties:

name:string(Default:X-Session-Id)

Nameoftheheaderthatcontainsthesessionidentifier(notcasesensitive).

Ifastringispassedinsteadofanoptionsobject,itwillbeinterpretedasthenameoption.

Sessiontransports

247

StaticfileassetsThemostflexiblewaytoservefilesinyourFoxxserviceistosimplypassthemthroughinyourrouterusingthecontextobject'sfileNamemethodandtheresponseobject'ssendFilemethod:

router.get('/some/filename.png',function(req,res){

constfilePath=module.context.fileName('some-local-filename.png');

res.sendFile(filePath);

});

Whileallowingforgreatercontrolofhowthefileshouldbesenttotheclientandwhoshouldbeabletoaccessit,doingthisforallyourstaticassetscangettedious.

AlternativelyyoucanspecifyfileassetsthatshouldbeservedbyyourFoxxservicedirectlyintheservicemanifestusingthefilesattribute:

"files":{

"/some/filename.png":{

"path":"some-local-filename.png",

"type":"image/png",

"gzip":false

},

"/favicon.ico":"bookmark.ico",

"/static":"my-assets-folder"

}

Eachentryinthefilesattributecanrepresenteitherasinglefileoradirectory.Whenservingentiredirectories,thekeyactsasaprefixandrequeststothatprefixwillberesolvedwithinthegivendirectory.

Options

path:string

Therelativepathofthefileorfolderwithintheservice.

type:string(optional)

TheMIMEcontenttypeofthefile.Defaultstoanintelligentguessbasedonthefilename'sextension.

gzip:boolean(Default:false)

Ifsettotruethefilewillbeservedwithgzip-encodingifsupportedbytheclient.Thiscanbeusefulwhenservingtextfileslikeclient-sideJavaScript,CSSorHTML.

Ifastringisprovidedinsteadofanobject,itwillbeinterpretedasthepathoption.

Servingfiles

248

WritingtestsFoxxprovidesoutoftheboxsupportforrunningtestsagainstaninstalledserviceusingtheMochatestrunner.

TestfileshavefullaccesstotheservicecontextandallArangoDBAPIsbutlikescriptscannotdefineFoxxroutes.

Runningtests

Aninstalledservice'stestscanbeexecutedfromtheadministrativewebinterface:

1. Openthe"Services"tabofthewebinterface2. Clickontheinstalledservicetobetested3. Clickonthe"Settings"tab4. Clickontheflaskiconinthetopright5. Accepttheconfirmationdialog

Notethatrunningtestsinaproductiondatabaseisnotrecommendedandmayresultindatalossifthetestsaccessthedatabase.

Whenrunningaserviceindevelopmentmodespecialcareneedstobetakenasperformingrequeststotheservice'sownroutesaspartofthetestsuitesmayresultintestsbeingexecutedwhilethedatabaseisinaninconsistentstate,leadingtounexpectedbehaviour.

TestfilepathsInordertotellFoxxaboutfilescontainingtestsuites,oneormorepatternsneedtobespecifiedinthetestsoptionoftheservicemanifest:

{

"tests":[

"**/test_*.js",

"**/*_test.js"

]

}

Thesepatternscanbeeitherrelativefilepathsor"globstar"patternswhere

*matcheszeroormorecharactersinafilename**matcheszeroormorenesteddirectories

Forexample,giventhefollowingdirectorystructure:

++test/

|++a/

||+-a1.js

||+-a2.js

||+-test.js

|+-b.js

|+-c.js

|+-d_test.js

+-e_test.js

+-test.js

Thefollowingpatternswouldmatchthefollowingfiles:

test.js:

test.js

test/*.js:

/test/b.js

/test/c.js

/test/d_test.js

Writingtests

249

test/**/*.js:

/test/a/a1.js

/test/a/a2.js

/test/a/test.js

/test/b.js

/test/c.js

/test/d_test.js

**/test.js:

/test/a/test.js

**/*test.js:

/test/a/test.js

/test/d_test.js

/e_test.js

/test.js

Evenifmultiplepatternsmatchthesamefilethetestsinthatfilewillonlyberunonce.

Theorderoftestsisalwaysdeterminedbythefilepaths,nottheorderinwhichtheyarematchedorspecifiedinthemanifest.

TeststructureMochatestsuitescanbedefinedusingoneofthreeinterfaces:BDD,TDDorExports.

TheQUnitinterfaceofMochaisnotsupportedinArangoDB.

LikeallArangoDBcode,testcodeisalwayssynchronous.

BDDinterface

TheBDDinterfacedefinestestsuitesusingthedescribefunctionandeachtestcaseisdefinedusingtheitfunction:

'usestrict';

constassert=require('assert');

consttrueThing=true;

describe('Truethings',()=>{

it('aretrue',()=>{

assert.equal(trueThing,true);

});

});

TheBDDinterfacealsooffersthealiascontextfordescribeandspecifyforit.

Testfixturescanbehandledusingbeforeandafterforsuite-widefixturesandbeforeEachandafterEachforper-testfixtures:

describe('Falsethings',()=>{

letfalseThing;

before(()=>{

falseThing=!true;

});

it('arefalse',()=>{

assert.equal(falseThing,false);

});

});

TDDinterface

TheTDDinterfacedefinestestsuitesusingthesuitefunctionandeachtestcaseisdefinedusingthetestfunction:

'usestrict';

constassert=require('assert');

consttrueThing=true;

suite('Truethings',()=>{

test('aretrue',()=>{

Writingtests

250

assert.equal(trueThing,true);

});

});

TestfixturescanbehandledusingsuiteSetupandsuiteTeardownforsuite-widefixturesandsetupandteardownforper-testfixtures:

suite('Falsethings',()=>{

letfalseThing;

suiteSetup(()=>{

falseThing=!true;

});

test('arefalse',()=>{

assert.equal(falseThing,false);

});

});

Exportsinterface

TheExportsinterfacedefinestestcasesasmethodsofplainobjectpropertiesofthemodule.exportsobject:

'usestrict';

constassert=require('assert');

consttrueThing=true;

exports['Truethings']={

'aretrue':function(){

assert.equal(trueThing,true);

}

};

Thekeysbefore,after,beforeEachandafterEacharespecial-casedandbehavelikethecorrespondingfunctionsintheBDDinterface:

letfalseThing;

exports['Falsethings']={

before(){

falseThing=false;

},

'arefalse':function(){

assert.equal(falseThing,false);

}

};

Assertions

ArangoDBprovidestwobundledmodulestodefineassertions:

assertcorrespondstotheNode.jsassertmodule,providinglow-levelassertionsthatcanoptionallyspecifyanerrormessage.

chaiisthepopularChaiAssertionLibrary,providingbothBDDandTDDstyleassertionsusingafamiliarsyntax.

Writingtests

251

Cross-OriginResourceSharing(CORS)TouseCORSinyourFoxxservicesyoufirstneedtoconfigureArangoDBforCORS.Asof3.2Foxxwillthenautomaticallywhitelistallresponseheadersastheyareused.

IfyouwantmorecontroloverthewhitelistorareusinganolderversionofArangoDByoucansetthefollowingresponseheadersinyourrequesthandler:

access-control-expose-headers:acomma-separatedlistofresponseheaders.Thisdefaultstoalistofallheaderstheresponseisactuallyusing(butnotincludinganyaccess-controlheaders).

access-control-allow-credentials:canbesetto"false"toforbidexposingcookies.ThedefaultvaluedependsonwhetherArangoDBtruststheorigin.Seethenotesonhttp.trusted-origin.

NotethatitisnotpossibletooverridetheseheadersfortheCORSpreflightresponse.Itisthereforenotpossibletoacceptcredentialsorcookiesonlyforindividualroutes,servicesordatabases.TheoriginneedstobetrustedaccordingtothegeneralArangoDBconfiguration(seeabove).

CrossOrigin

252

FoxxscriptsandqueuedjobsFoxxletsyoudefinescriptsthatcanbeexecutedaspartoftheinstallationandremovalprocess,invokedmanuallyorscheduledtorunatalatertimeusingthejobqueue.

Toregisteryourscript,justaddascriptssectiontoyourservicemanifest:

{

...

"scripts":{

"setup":"scripts/setup.js",

"send-mail":"scripts/send-mail.js"

}

...

}

Thescriptsyoudefineinyourservicemanifestcanbeinvokedfromthewebinterfaceintheservice'ssettingspagewiththeScriptsdropdown.

Youcanalsousethescriptsasqueuedjobs:

'usestrict';

constqueues=require('@arangodb/foxx/queues');

queues.get('default').push(

{mount:'/my-service-mount-point',name:'send-mail'},

{to:'user@example.com',body:'Hello'}

);

ScriptargumentsandreturnvaluesIfthescriptwasinvokedwithanyarguments,youcanaccessthemusingthemodule.context.argvarray.

Toreturndatafromyourscript,youcanassignthedatatomodule.exportsasusual.PleasenotethatthisdatawillbeconvertedtoJSON.

Anyerrorsraisedbythescriptwillbehandleddependingonhowthescriptwasinvoked:

ifthescriptwasinvokedfromtheHTTPAPI(e.g.usingthewebinterface),itwillreturnanerrorresponseusingtheexception'sstatusCodepropertyifspecifiedor500.ifthescriptwasinvokedfromaFoxxjobqueue,thejob'sfailurecounterwillbeincrementedandthejobwillberescheduledormarkedasfailedifnoattemptsremain.

Examples

Let'ssayyouwanttodefineascriptthattakestwonumericvaluesandreturnstheresultofmultiplyingthem:

'usestrict';

constassert=require('assert');

constargv=module.context.argv;

assert.equal(argv.length,2,'Expectedexactlytwoarguments');

assert.equal(typeofargv[0],'number','Expectedfirstargumenttobeanumber');

assert.equal(typeofargv[1],'number','Expectedsecondargumenttobeanumber');

module.exports=argv[0]*argv[1];

LifecycleScriptsFoxxrecognizeslifecyclescriptsiftheyaredefinedandwillinvokethemduringtheinstallation,updateandremovalprocessoftheserviceifyouwant.

Scriptsandqueuedjobs

253

Thefollowingscriptsarecurrentlyrecognizedaslifecyclescriptsbytheirname:"setup"and"teardown".

SetupScript

ThesetupscriptwillbeexecutedwithoutargumentsduringtheinstallationofyourFoxxservice.

Thesetupscriptmaybeexecutedmorethanonceandshouldthereforebetreatedasreentrant.Runningthesamesetupscriptagainshouldnotresultinanyerrorsorduplicatedata.

Thesetupscriptistypicallyusedtocreatecollectionsyourserviceneedsorinsertseeddatalikeinitialadministrativeuseraccountsandsoon.

Examples

'usestrict';

constdb=require('@arangodb').db;

consttextsCollectionName=module.context.collectionName('texts');

//`textsCollectionName`isnowtheprefixednameofthisservice's"texts"collection.

//e.g."example_texts"iftheservicehasbeenmountedat`/example`

if(db._collection(textsCollectionName)===null){

constcollection=db._create(textsCollectionName);

collection.save({text:'entry1fromcollectiontexts'});

collection.save({text:'entry2fromcollectiontexts'});

collection.save({text:'entry3fromcollectiontexts'});

}else{

console.debug(`collection${texts}alreadyexists.Leavingituntouched.`);

}

TeardownScript

TheteardownscriptwillbeexecutedwithoutargumentsduringtheremovalofyourFoxxservice.

Itcanalsooptionallybeexecutedbeforeupgradinganservice.

Thisscripttypicallyremovesthecollectionsand/ordocumentscreatedbyyourservice'ssetupscript.

Examples

'usestrict';

constdb=require('@arangodb').db;

consttextsCollection=module.context.collection('texts');

if(textsCollection){

textsCollection.drop();

}

Queuesconstqueues=require('@arangodb/foxx/queues')

Foxxallowsdefiningjobqueuesthatletyouperformsloworexpensiveactionsasynchronously.Thesequeuescanbeusedtosende-mails,callexternalAPIsorperformotheractionsthatyoudonotwanttoperformdirectlyorwanttoretryonfailure.

enableordisabletheFoxxqueuesfeature--foxx.queuesflagIftrue,theFoxxqueueswillbeavailableandjobsinthequeueswillbeexecutedasynchronously.Thedefaultistrue.Whensettofalsethequeuemanagerwillbedisabledandanyjobsarepreventedfrombeingprocessed,whichmayreduceCPUloadabit.PleasenotethatFoxxjobqueuesaredatabase-specific.Queuesandjobsarealwaysrelativetothedatabaseinwhichtheyarecreatedoraccessed.

pollintervalforFoxxqueues--foxx.queues-poll-intervalvalueThepollintervalfortheFoxxqueuesmanager.Thevalueisspecifiedinseconds.LowervalueswillmeanmoreimmediateandmorefrequentFoxxqueuejobexecution,butwillmakethequeuethreadwakeupandquerythequeuesmoreoften.Whensettoalowvalue,thequeuethreadmightcauseCPUload.Thedefaultis1second.IfFoxx

Scriptsandqueuedjobs

254

queuesarenotusedmuch,thenthisvaluemaybeincreasedtomakethequeuesthreadwakeupless.Forthelow-levelfunctionalityseethechapteronthetaskmanagementmodule.

Creatingorupdatingaqueue

queues.create(name,[maxWorkers]):Queue

Returnsthequeueforthegivenname.Ifthequeuedoesnotexist,anewqueuewiththegivennamewillbecreated.IfaqueuewiththegivennamealreadyexistsandmaxWorkersisset,thequeue'smaximumnumberofworkerswillbeupdated.Thequeuewillbecreatedinthecurrentdatabase.

Arguments

name:string

Nameofthequeuetocreate.

maxWorkers:number(Default:1)

Themaximumnumberofworkers.

Examples

//Createaqueuewiththedefaultnumberofworkers(i.e.one)

constqueue1=queues.create("my-queue");

//Createaqueuewithagivennumberofworkers

constqueue2=queues.create("another-queue",2);

//Updatethenumberofworkersofanexistingqueue

constqueue3=queues.create("my-queue",10);

//queue1andqueue3refertothesamequeue

assertEqual(queue1,queue3);

Fetchinganexistingqueuequeues.get(name):Queue

Returnsthequeueforthegivenname.Ifthequeuedoesnotexistanexceptionisthrowninstead.

Thequeuewillbelookedupinthecurrentdatabase.

Arguments

name:string

Nameofthequeuetofetch.

Examples

Ifthequeuedoesnotyetexistanexceptionisthrown:

queues.get("some-queue");

//Error:Queuedoesnotexist:some-queue

//at...

Otherwisethequeuewillbereturned:

constqueue1=queues.create("some-queue");

constqueue2=queues.get("some-queue");

assertEqual(queue1,queue2);

Deletingaqueue

queues.delete(name):boolean

Returnstrueifthequeuewasdeletedsuccessfully.Ifthequeuedidnotexist,itreturnsfalseinstead.Thequeuewillbelookedupanddeletedinthecurrentdatabase.

Scriptsandqueuedjobs

255

Whenaqueueisdeleted,jobsonthatqueuewillnolongerbeexecuted.

Deletingaqueuewillnotdeleteanyjobsonthatqueue.

Arguments

name:string

Nameofthequeuetodelete.

Examples

constqueue=queues.create("my-queue");

queues.delete("my-queue");//true

queues.delete("my-queue");//false

Addingajobtoaqueue

queue.push(script,data,[opts]):string

Thejobwillbeaddedtothespecifiedqueueinthecurrentdatabase.

Returnsthejobid.

Arguments

script:object

Ajobtypedefinition,consistingofanobjectwiththefollowingproperties:

name:string

Nameofthescriptthatwillbeinvoked.

mount:string

Mountpathoftheservicethatdefinesthescript.

backOff:Function|number(Default:1000)

Eitherafunctionthattakesthenumberoftimesthejobhasfailedbeforeasinputandreturnsthenumberofmillisecondstowaitbeforetryingthejobagain,orthedelaytobeusedtocalculateanexponentialback-off,or0fornodelay.

maxFailures:number|Infinity(Default:0):

Numberoftimesasinglerunofajobwillbere-triedbeforeitismarkedas"failed".AnegativevalueorInfinitymeansthatthejobwillbere-triedonfailureindefinitely.

schema:Schema(optional)

Schematovalidateajob'sdataagainstbeforeenqueuingthejob.

preprocess:Function(optional)

Functiontopre-processajob's(validated)databeforeserializingitinthequeue.

data:any

Jobdataofthejob;mustbeserializabletoJSON.

opts:object(optional)

Objectwithanyofthefollowingproperties:

success:Function(optional)

Functiontobecalledafterthejobhasbeencompletedsuccessfully.

failure:Function(optional)

Functiontobecalledafterthejobhasfailedtoomanytimes.

Scriptsandqueuedjobs

256

delayUntil:number|Date(Default:Date.now())

Timestampinmilliseconds(orDateinstance)untilwhichtheexecutionofthejobshouldbedelayed.

backOff:Function|number(Default:1000)

Seescript.backOff.

maxFailures:number|Infinity(Default:0):

Seescript.maxFailures.

repeatTimes:number|Function(Default:0)

Ifsettoapositivenumber,thejobwillberepeatedthismanytimes(notcountingrecoverywhenusingmaxFailures).IfsettoanegativenumberorInfinity,thejobwillberepeatedindefinitely.Ifsetto0thejobwillnotberepeated.

repeatUntil:number|Date(optional)

Ifthejobissettoautomaticallyrepeat,thiscanbesettoatimestampinmilliseconds(orDateinstance)afterwhichthejobwillnolongerrepeat.Settingthisvaluetozero,anegativevalueorInfinityhasnoeffect.

repeatDelay:number(Default:0)

Ifthejobissettoautomaticallyrepeat,thiscanbesettoanon-negativevaluetosetthenumberofmillisecondsforwhichthejobwillbedelayedbeforeitisstartedagain.

NotethatifyoupassafunctionforthebackOffcalculation,successcallbackorfailurecallbackoptionsthefunctionwillbeserializedtothedatabaseasastringandthereforemustnotrelyonanyexternalscopeorexternalvariables.

Whenthejobissettoautomaticallyrepeat,thefailurecallbackwillonlybeexecutedwhenarunofthejobhasfailedmorethanmaxFailurestimes.NotethatifthejobfailsandmaxFailuresisset,itwillberescheduledaccordingtothebackOffuntilithaseitherfailedtoomanytimesorcompletedsuccessfullybeforebeingscheduledaccordingtotherepeatDelayagain.RecoveryattemptsbymaxFailuresdonotcounttowardsrepeatTimes.

Thesuccessandfailurecallbacksreceivethefollowingarguments:

result:any

Thereturnvalueofthescriptforthecurrentrunofthejob.

jobData:any

Thedatapassedtothismethod.

job:object

ArangoDBdocumentrepresentingthejob'scurrentstate.

Examples

Let'ssaywehaveanservicemountedat/mailerthatprovidesascriptcalledsend-mail:

'usestrict';

constqueues=require('@arangodb/foxx/queues');

constqueue=queues.create('my-queue');

queue.push(

{mount:'/mailer',name:'send-mail'},

{to:'hello@example.com',body:'Helloworld'}

);

Thiswillnotwork,becauselogwasdefinedoutsidethecallbackfunction(thecallbackmustbeserializabletoastring):

//WARNING:THISDOESNOTWORK!

'usestrict';

constqueues=require('@arangodb/foxx/queues');

constqueue=queues.create('my-queue');

constlog=require('console').log;//outsidethecallback'sfunctionscope

queue.push(

{mount:'/mailer',name:'send-mail'},

Scriptsandqueuedjobs

257

{to:'hello@example.com',body:'Helloworld'},

{success:function(){

log('Yay!');//throws'logisnotdefined'

}}

);

Here'sanexampleofajobthatwillbeexecutedevery5secondsuntiltomorrow:

'usestrict';

constqueues=require('@arangodb/foxx').queues;

constqueue=queues.create('my-queue');

queue.push(

{mount:'/mailer',name:'send-mail'},

{to:'hello@example.com',body:'Helloworld'},

{

repeatTimes:Infinity,

repeatUntil:Date.now()+(24*60*60*1000),

repeatDelay:5*1000

}

);

Fetchingajobfromthequeue

queue.get(jobId):Job

Createsaproxyobjectrepresentingajobwiththegivenjobid.

Thejobwillbelookedupinthespecifiedqueueinthecurrentdatabase.

ReturnsthejobforthegivenjobId.Propertiesofthejobobjectwillbefetchedwhenevertheyarereferencedandcannotbemodified.

Arguments

jobId:string

Theidofthejobtocreateaproxyobjectfor.

Examples

constjobId=queue.push({mount:'/logger',name:'log'},'HelloWorld!');

constjob=queue.get(jobId);

assertEqual(job.id,jobId);

Deletingajobfromthequeue

queue.delete(jobId):boolean

Deletesajobwiththegivenjobid.Thejobwillbelookedupanddeletedinthespecifiedqueueinthecurrentdatabase.

Arguments

jobId:string

Theidofthejobtodelete.

Returnstrueifthejobwasdeletedsuccessfully.Ifthejobdidnotexistitreturnsfalseinstead.

Fetchinganarrayofjobsinaqueue

Examples

constlogScript={mount:'/logger',name:'log'};

queue.push(logScript,'HelloWorld!',{delayUntil:Date.now()+50});

assertEqual(queue.pending(logScript).length,1);

//50mslater...

assertEqual(queue.pending(logScript).length,0);

assertEqual(queue.progress(logScript).length,1);

//evenlater...

assertEqual(queue.progress(logScript).length,0);

Scriptsandqueuedjobs

258

assertEqual(queue.complete(logScript).length,1);

Fetchinganarrayofpendingjobsinaqueuequeue.pending([script]):Array<string>

Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"pending",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments

script:object(optional)

Anobjectwiththefollowingproperties:

name:stringNameofthescript.

mount:stringMountpathoftheservicedefiningthescript.

Fetchinganarrayofjobsthatarecurrentlyinprogress

queue.progress([script])

Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"progress",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments

script:object(optional)

Anobjectwiththefollowingproperties:

name:stringNameofthescript.

mount:stringMountpathoftheservicedefiningthescript.

Fetchinganarrayofcompletedjobsinaqueuequeue.complete([script]):Array<string>

Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"complete",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments

script:object(optional)

Anobjectwiththefollowingproperties:

name:stringNameofthescript.

mount:stringMountpathoftheservicedefiningthescript.

Fetchinganarrayoffailedjobsinaqueue

queue.failed([script]):Array<string>

Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"failed",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments

Scriptsandqueuedjobs

259

script:object(optional)

Anobjectwiththefollowingproperties:

name:stringNameofthescript.

mount:stringMountpathoftheservicedefiningthescript.

Fetchinganarrayofalljobsinaqueue

queue.all([script]):Array<string>

Returnsanarrayofjobidsofalljobsinthegivenqueue,optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments

script:object(optional)

Anobjectwiththefollowingproperties:

name:stringNameofthescript.

mount:stringMountpathoftheservicedefiningthescript.

Abortingajobjob.abort():void

Abortsanon-completedjob.

Setsajob'sstatusto"failed"ifitisnotalready"complete",withoutcallingthejob'sonFailurecallback.

Scriptsandqueuedjobs

260

Migrating2.xservicesto3.0WhenmigratingservicesfromolderversionsofArangoDBitisgenerallyrecommendedyoumakesuretheyworkinlegacycompatibilitymode,whichcanalsoserveasastop-gapsolution.

ThischapteroutlinesthemajordifferencesintheFoxxAPIbetweenArangoDB2.8andArangoDB3.0.

Generalchanges

TheconsoleobjectinlaterversionsofArangoDB2.ximplementedaspecialFoxxconsoleAPIandwouldoptionallylogmessagestoacollection.ArangoDB3.0restorestheoriginalbehaviourwhereconsoleisthesameobjectavailablefromtheconsolemodule.

Migrating2.xservices

261

Migratingfrompre-2.8WhenmigratingfromaversionolderthanArangoDB2.8pleasenotethatstartingwithArangoDB2.8thebehaviouroftherequirefunctionmorecloselymimicsthebehaviourobservedinNode.jsandmodulebundlersforbrowsers,e.g.:

Inafile/routes/examples.js(relativetotherootfolderoftheservice):

require('./my-module')willbeattemptedtoberesolvedinthefollowingorder:

1. /routes/my-module(relativetoserviceroot)2. /routes/my-module.js(relativetoserviceroot)3. /routes/my-module.json(relativetoserviceroot)4. /routes/my-module/index.js(relativetoserviceroot)5. /routes/my-module/index.json(relativetoserviceroot)

require('lodash')willbeattemptedtoberesolvedinthefollowingorder:

1. /routes/node_modules/lodash(relativetoserviceroot)2. /node_modules/lodash(relativetoserviceroot)3. ArangoDBmodulelodash4. Nodecompatibilitymodulelodash5. BundledNPMmodulelodash

require('/abs/path')willbeattemptedtoberesolvedinthefollowingorder:

1. /abs/path(relativetofilesystemroot)2. /abs/path.js(relativetofilesystemroot)3. /abs/path.json(relativetofilesystemroot)4. /abs/path/index.js(relativetofilesystemroot)5. /abs/path/index.json(relativetofilesystemroot)

ThisbehaviourisincompatiblewiththesourcecodegeneratedbytheFoxxgeneratorinthewebinterfacebeforeArangoDB2.8.

Note:Theorg/arangodbmoduleisaliasedtothenewname@arangodbinArangoDB3.0.0andthe@arangodbmodulewasaliasedtotheoldnameorg/arangodbinArangoDB2.8.0.Eitheronewillworkin2.8and3.0butoutsideoflegacyservicesyoushoulduse@arangodbgoingforward.

FoxxqueueInArangoDB2.6FoxxintroducedanewwaytodefinequeuedjobsusingFoxxscriptstoreplacethefunction-basedjobtypedefinitionswhichwerecausingproblemswhenrestartingtheserver.Thefunction-basedjobshavebeenremovedin2.7andarenolongersupportedatall.

CoffeeScriptArangoDB3.0nolongerprovidesbuilt-insupportforCoffeeScriptsourcefiles,eveninlegacycompatibilitymode.IfyouwanttouseanalternativelanguagelikeCoffeeScript,makesuretopre-compiletherawsourcefilestoJavaScriptandusethecompiledJavaScriptfilesintheservice.

TherequestmoduleThe@arangodb/requestmodulewhenusedwiththejsonoptionpreviouslyoverwrotethestringinthebodypropertyoftheresponseobjectoftheresponsewiththeparsedJSONbody.In2.8thiswaschangedsotheparsedJSONbodyisaddedasthejsonpropertyoftheresponseobjectinadditiontooverwritingthebodyproperty.In3.0andlater(includinglegacycompatibilitymode)thebodypropertyisnolongeroverwrittenandmustusethejsonpropertyinstead.Notethatthisonlyaffectscodeusingthejsonoptionwhenmakingtherequest.

Migratingfrompre-2.8

262

BundledNPMmodules

ThebundledNPMmoduleshavebeenupgradedandmayincludebackwards-incompatiblechanges,especiallytheAPIofjoihaschangedseveraltimes.Ifindoubtyoushouldbundleyourownversionsofthesemodulestoensurespecificversionswillbeused.

Theutilitymodulelodashisnowavailableandshouldbeusedinsteadofunderscore,butbothmoduleswillcontinuetobeprovided.

Migratingfrompre-2.8

263

ManifestManyofthefieldsthatwererequiredinArangoDB2.xarenowoptionalandcanbesafelyomitted.

ToavoidcompatibilityproblemswithfutureversionsofArangoDByoushouldalwaysspecifytheenginesfield,e.g.:

{

"engines":{

"arangodb":"^3.0.0"

}

}

Controllers&exports

PreviouslyFoxxdistinguishedbetweenexportsandcontrollers,eachofwhichcouldbespecifiedasanobject.InArangoDB3.0thesehavebeenmergedintoasinglemainfieldspecifyinganentryfile.

Theeasiestwaytomigrateservicesusingmultipleexportsand/orcontrollersistocreateaseparateentryfilethatimportsthesefiles:

Old(manifest.json):

{

"exports":{

"doodads":"doodads.js",

"dingbats":"dingbats.js"

},

"controllers":{

"/doodads":"routes/doodads.js",

"/dingbats":"routes/dingbats.js",

"/":"routes/root.js"

}

}

New(manifest.json):

{

"main":"index.js"

}

New(index.js):

'usestrict';

module.context.use('/doodads',require('./routes/doodads'));

module.context.use('/dingbats',require('./routes/dingbats'));

module.context.use('/',require('./routes/root'));

module.exports={

doodads:require('./doodads'),

dingbats:require('./dingbats')

};

IndexredirectIfyoupreviouslydidnotdefinethedefaultDocumentfield,pleasenotethatinArangoDB3.0thefieldwillnolongerdefaulttothevalueindex.htmlwhenomitted:

Old:

{

//nodefaultDocument

}

manifest.json

264

New:

{

"defaultDocument":"index.html"

}

ThisalsomeansitisnolongernecessarytospecifythedefaultDocumentfieldwithanemptyvaluetopreventtheredirectandbeabletoserverequestsatthe/(root)pathofthemountpoint:

Old:

{

"defaultDocument":""

}

New:

{

//nodefaultDocument

}

AssetsTheassetsfieldisnolongersupportedinArangoDB3.0outsideoflegacycompatibilitymode.

Ifyoupreviouslyusedthefieldtoserveindividualfilesas-isyoucansimplyusethefilesfieldinstead:

Old:

{

"assets":{

"client.js":{

"files":["assets/client.js"],

"contentType":"application/javascript"

}

}

}

New:

{

"files":{

"client.js":{

"path":"assets/client.js",

"type":"application/javascript"

}

}

}

Ifyoureliedonbeingabletospecifymultiplefilesthatshouldbeconcatenated,youwillhavetousebuildtoolsoutsideofArangoDBtopreparethesefilesaccordingly.

RootelementTherootElementfieldisnolongersupportedandhasbeenremovedentirely.

IfyourcontrollersreliedonthisfieldbeingavailableyouneedtoadjustyourschemasandroutestobeabletohandlethefullJSONstructureofincomingdocuments.

Systemservices

manifest.json

265

TheisSystemfieldisnolongersupported.ThepresenceorabsenceofthefieldhadnoeffectinmostrecentversionsofArangoDB2.xandhasnowbeenremovedentirely.

manifest.json

266

TheapplicationcontextTheglobalapplicationContextvariableavailableinFoxxmoduleshasbeenreplacedwiththecontextattributeofthemodulevariable.Forconsistencyitisnowreferredtoastheservicecontextthroughoutthisdocumentation.

SomemethodsoftheservicecontexthavechangedinArangoDB3.0:

fileName()nowbehaveslikepath()didinArangoDB2.xpath()hasbeenremoved(usefileName()instead)foxxFileName()hasbeenremoved(usefileName()instead)

Additionallytheversionandnameattributeshavebeenremovedandcannowonlybeaccessedviathemanifestattribute(asmanifest.versionandmanifest.name).Notethatthecorrespondingmanifestfieldsarenowoptionalandmaybeomitted.

Theoptionsattributehasalsobeenremovedasitshouldbeconsideredanimplementationdetail.Youshouldinsteadaccessthedependenciesandconfigurationattributesdirectly.

Theinternal_prefixattribute(whichwasanaliasforbasePath)andtheinternalcommentandclearCommentsmethods(whichwereusedbythemagicaldocumentationcommentsinArangoDB2.x)havealsobeenremoved.

Theinternal_serviceattribute(whichprovidesaccesstotheserviceitself)hasbeenrenamedtoservice.

applicationContext

267

RepositoriesandmodelsPreviouslyFoxxwasheavilybuiltaroundtheconceptofrepositoriesandmodels,whichprovidedcomplexbutrarelynecessaryabstractionsontopofArangoDBcollectionsanddocuments.InArangoDB3.0thesehavebeenremovedentirely.

Repositoriesvscollections

RepositoriesmostlywrappedmethodsthatalreadyexistedonArangoDBcollectionobjectsandprimarilydealtwithconvertingbetweenplainArangoDBdocumentsandFoxxmodelinstances.InArangoDB3.0youcansimplyusethesecollectionsdirectlyandtreatdocumentsasplainJavaScriptobjects.

Old:

'usestrict';

constFoxx=require('org/arangodb/foxx');

constmyRepo=newFoxx.Repository(

applicationContext.collection('myCollection'),

{model:Foxx.Model}

);

//...

constmodels=myRepo.byExample({color:'green'});

res.json(models.map(function(model){

returnmodel.forClient();

}));

New:

'usestrict';

constmyDocs=module.context.collection('myCollection');

//...

constdocs=myDocs.byExample({color:'green'});

res.json(docs);

Schemavalidation

ThemainpurposeofmodelsinArangoDB2.xwastovalidateincomingdatausingjoischemas.InmorerecentversionsofArangoDB2.xitwasalreadypossibletopasstheseschemasdirectlyinmostplaceswhereamodelwasexpectedasanargument.Theonlydifferenceisthatschemasshouldnowbeconsideredthedefault.

IfyoupreviouslyreliedontheautomaticvalidationofFoxxmodelinstanceswhensettingattributesorinstantiatingmodelsfromuntrusteddata,youcansimplyusetheschema'svalidatemethoddirectly.

Old:

'usestrict';

constjoi=require('joi');

constmySchema={

name:joi.string().required(),

size:joi.number().required()

};

constFoxx=require('org/arangodb/foxx');

constMyModel=Foxx.Model.extend({schema:mySchema});

//...

constmodel=newMyModel(req.json());

if(!model.isValid){

res.status(400);

RepositoriesandModels

268

res.write('Badrequest');

return;

}

New:

'usestrict';

constjoi=require('joi');

//Notethisisnowwrappedinajoi.object()

constmySchema=joi.object({

name:joi.string().required(),

size:joi.number().required()

}).required();

//...

constresult=mySchema.validate(req.body);

if(result.errors){

res.status(400);

res.write('Badrequest');

return;

}

Migratingmodels

Whilemostusecasesformodelscannowbereplacedwithplainjoischemas,thereisstilltheconceptofa"model"inFoxxinArangoDB3.0althoughitisquitedifferentfromFoxxmodelsinArangoDB2.x.

AmodelinFoxxnowreferstoaplainJavaScriptobjectwithanoptionalschemaattributeandtheoptionalmethodsforClientandfromClient.Modelscanbeusedinsteadofplainjoischemastodefinerequestandresponsebodiesbuttherearenomodel"instances"inArangoDB3.0.

Old:

'usestrict';

const_=require('underscore');

constjoi=require('joi');

constFoxx=require('org/arangodb/foxx');

constMyModel=Foxx.Model.extend({

schema:{

name:joi.string().required(),

size:joi.number().required()

},

forClient(){

return_.omit(this.attributes,['_key','_id','_rev']);

}

});

//...

ctrl.get(/*...*/)

.bodyParam('body',{type:MyModel});

New:

'usestrict';

const_=require('lodash');

constjoi=require('joi');

constMyModel={

schema:joi.object({

name:joi.string().required(),

size:joi.number().required()

}).required(),

forClient(data){

return_.omit(data,['_key','_id','_rev']);

}

};

RepositoriesandModels

269

//...

router.get(/*...*/)

.body(MyModel);

TriggersWhensaving,updating,replacingordeletingmodelsinArangoDB2.xusingtherepositorymethodstherepositoryandmodelwouldfireeventsthatcouldbesubscribedtoinordertoperformside-effects.

Notethatevenin2.xtheseeventswouldnotfirewhenusingqueriesormanipulatingdocumentsinanyotherwaythanusingthespecificrepositorymethodsthatoperatedonindividualdocuments.

ThisbehaviourisnolongeravailableinArangoDB3.0butcanbeemulatedbyusinganEventEmitterdirectlyifitisnotpossibletosolvetheproblemdifferently:

Old:

'usestrict';

constFoxx=require('org/arangodb/foxx');

constMyModel=Foxx.Model.extend({

//...

},{

afterRemove(){

console.log(this.get('name'),'wasremoved');

}

});

//...

constmodel=myRepo.firstExample({name:'myName'});

myRepo.remove(model);

//->"myNamewasremovedsuccessfully"

New:

'usestrict';

constEventEmitter=require('events');

constemitter=newEventEmitter();

emitter.on('afterRemove',function(doc){

console.log(doc.name,'wasremoved');

});

//...

constdoc=myDocs.firstExample({name:'myName'});

myDocs.remove(doc);

emitter.emit('afterRemove',doc);

//->"myNamewasremovedsuccessfully"

Orsimply:

'usestrict';

functionafterRemove(doc){

console.log(doc.name,'wasremoved');

}

//...

constdoc=myDocs.firstExample({name:'myName'});

myDocs.remove(doc);

afterRemove(doc);

//->"myNamewasremovedsuccessfully"

RepositoriesandModels

270

ControllersvsroutersFoxxControllershavebeenreplacedwithrouters.Thisismorethanacosmeticchangeastherearesignificantdifferencesinbehaviour:

Controllerswereautomaticallymountedwhenthefiledefiningthemwasexecuted.Routersneedtobeexplicitlymountedusingthemodule.context.usemethod.Routerscanalsobeexported,importedandevennested.Thismakesiteasiertosplitupcomplexroutingtreesacrossmultiplefiles.

Old:

'usestrict';

constFoxx=require('org/arangodb/foxx');

constctrl=newFoxx.Controller(applicationContext);

ctrl.get('/hello',function(req,res){

//...

});

New:

'usestrict';

constcreateRouter=require('org/arangodb/foxx/router');

constrouter=createRouter();

//Ifyouareimportingthisfilefromyourentryfile("main"):

module.exports=router;

//Otherwise:module.context.use(router);

router.get('/hello',function(req,res){

//...

});

Somegeneralchangesinbehaviourthatmighttripyouup:

WhenspecifyingpathparameterswithschemasFoxxwillnowignoretherouteiftheschemadoesnotmatch(i.e./hello/foxxwillnolongermatch/hello/:numifnumspecifiesaschemathatdoesn'tmatchthevalue"foxx").Withcontrollersthiscouldpreviouslyresultinusersseeinga400(badrequest)errorwhentheyshouldinsteadbeserveda404(notfound)response.

WhenarequestismadewithanHTTPverbnotsupportedbyanendpoint,Foxxwillnowrespondwitha405(methodnotallowed)errorwithanappropriateAllowedheaderlistingthesupportedHTTPverbsforthatendpoint.

FoxxwillnolongerparseyourJSDoccommentstogenerateroutedocumentation(usethesummaryanddescriptionmethodsoftheendpointinstead).

TheapiDocumentationmethodnowlivesontheservicecontextandbehavesslightlydifferently.

ThereisnorouterequivalentfortheactivateAuthenticationandactivateSessionsmethods.Insteadyoushouldusethesessionmiddleware(seethesectiononsessionsbelow).

Thereisnodelaliasforthedeletemethodonrouters.IthasalwaysbeensafetousekeywordsasmethodnamesinFoxx,sotheuseofthisaliaswasalreadydiscouragedbefore.

TheallRoutesproxyisnolotavailableonroutersbutcaneasilybereplacedwithmiddlewareorchildrouters.

Controllers

271

TherequestcontextWhendefiningarouteonacontrollerthecontrollerwouldreturnanobjectcalledrequestcontext.Routersreturnasimilarobjectcalledendpoint.Routersalsoreturnendpointswhenmountingchildroutersormiddleware,asdoestheusemethodoftheservicecontext.

ThemaindifferencesbetweenthenewendpointsandtheobjectsreturnedbycontrollersinpreviousversionsofArangoDBare:

bodyParamisnowsimplycalledbody;itisnolongerneccessaryorpossibletogivethebodyanameandtherequestbodywillnotshowupintherequestparameters.It'salsopossibletospecifyaMIMEtype

body,queryParamandpathParamnowtakepositionargumentsinsteadofanobject.Forspecificsseetheendpointdocumentation.

notesisnowcalleddescriptionandtakesasinglestringargument.

onlyIfandonlyIfAuthenticatedarenolongeravailable;theycanbeemulatedwithmiddlewareifnecessary:

Old:

ctrl.get(/*...*/)

.onlyIf(function(req){

if(!req.user){

thrownewError('Notauthenticated!');

}

});

New:

router.use(function(req,res,next){

if(!req.arangoUser){

res.throw(403,'Notauthenticated!');

}

next();

});

router.get(/*...*/);

Controllers

272

ErrorhandlingTheerrorResponsemethodprovidedbycontrollerrequestcontextshasnoequivalentinrouterendpoints.Ifyouwanttohandlespecificerrortypeswithspecificstatuscodesyouneedtocatchthemexplicitly,eitherintherouteorinamiddleware:

Old:

ctrl.get('/puppies',function(req,res){

//Exceptionisthrownhere

})

.errorResponse(TooManyPuppiesError,400,'Somethingwentwrong!');

New:

ctrl.get('/puppies',function(req,res){

try{

//Exceptionisthrownhere

}catch(e){

if(!(einstanceofTooManyPuppiesError)){

throwe;

}

res.throw(400,'Somethingwentwrong!');

}

})

//The"error"methodmerelydocumentsthemeaning

//ofthestatuscodeandhasnoothereffect.

.error(400,'Throwniftherearetoomanypuppies.');

Notethaterrorscreatedwithhttp-errorsarestillhandledbyFoxxintelligently.Infactres.throwisjustahelpermethodforcreatingandthrowingtheseerrors.

Controllers

273

Before,afterandaroundThebefore,afterandaroundmethodscaneasilybereplacedbymiddleware:

Old:

letstart;

ctrl.before(function(req,res){

start=Date.now();

});

ctrl.after(function(req,res){

console.log('Requesthandledin',(Date.now()-start),'ms');

});

New:

router.use(function(req,res,next){

letstart=Date.now();

next();

console.log('Requesthandledin',(Date.now()-start),'ms');

});

Notethatunlikearoundmiddlewarereceivesthenextfunctionasthethirdargument(the"opts"argumenthasnoequivalent).

Controllers

274

RequestobjectsThenamesofsomeattributesoftherequestobjecthavebeenadjustedtomorecloselyalignwiththoseofthecorrespondingmethodsontheendpointobjectsandestablishedconventionsinotherJavaScriptframeworks:

req.urlParametersisnowcalledreq.pathParams

req.parametersisnowcalledreq.queryParams

req.params()isnowcalledreq.param()

req.requestTypeisnowcalledreq.method

req.compatibilityisnowcalledreq.arangoVersion

req.userisnowcalledreq.arangoUser

Someattributeshavebeenremovedorchanged:

req.cookieshasbeenremovedentirely(usereq.cookie(name))

req.requestBodyhasbeenremovedentirely(seebelow)

req.suffixisnowastringratherthananarray

Additionallythereq.serverandreq.clientattributesarenolongeravailable.Theinformationisnowexposedinawaythatcan(optionally)transparentlyhandleproxyforwardingheaders:

req.hostnamedefaultstoreq.server.address

req.portdefaultstoreq.server.port

req.remoteAddressdefaultstoclient.address

req.remotePortdefaultstoclient.port

Finally,thereq.cookiemethodnowtakesthesignedoptionsdirectly.

Old:

constsid=req.cookie('sid',{

signed:{

secret:'keyboardcat',

algorithm:'sha256'

}

});

New:

constsid=req.cookie('sid',{

secret:'keyboardcat',

algorithm:'sha256'

});

RequestbodiesThereq.bodyisnolongeramethodandnolongerautomaticallyparsesJSONrequestbodiesunlessarequestbodywasdefined.Thereq.rawBodynowcorrespondstothereq.rawBodyBufferofArangoDB2.xandisalsonolongeramethod.

Old:

ctrl.post('/',function(req,res){

constdata=req.body();

//...

});

Controllers

275

New:

router.post('/',function(req,res){

constdata=req.body;

//...

})

.body(['json']);

Orsimply:

constjoi=require('joi');

router.post('/',function(req,res){

constdata=req.body;

//...

})

.body(joi.object().optional());

Multipartrequests

Thereq.requestPartsmethodhasbeenremovedentirely.Ifyouneedtoacceptmultipartrequestbodies,youcansimplydefinetherequestbodyusingamultipartMIMEtypelikemultipart/form-data:

Old:

ctrl.post('/',function(req,res){

constparts=req.requestParts();

//...

});

New:

router.post('/',function(req,res){

constparts=req.body;

//...

})

.body(['multipart/form-data']);

Controllers

276

ResponseobjectsTheresponseobjecthasalotofnewmethodsinArangoDB3.0butotherwiseremainssimilartotheresponseobjectofpreviousversions:

Theres.sendmethodbehavesverydifferentlyfromhowthemethodwiththesamenamebehavedinArangoDB2.x:theconversionnowtakestheresponsebodydefinitionoftherouteintoaccount.Thereisanewmethodres.writethatimplementstheoldbehaviour.

Notethatconsecutivecallstores.writewillappendtotheresponsebodyratherthanreplacingitlikeres.send.

Theres.contentTypepropertyisalsonolongeravailable.IfyouwanttosettheMIMEtypeoftheresponsebodytoanexplicitvalueyoushouldsetthecontent-typeheaderinstead:

Old:

res.contentType='application/json';

res.body=JSON.stringify(results);

New:

res.set('content-type','application/json');

res.body=JSON.stringify(results);

Orsimply:

//setsthecontenttypetoJSON

//ifithasnotalreadybeenset

res.json(results);

Theres.cookiemethodnowtakesthesignedoptionsaspartoftheregularoptionsobject.

Old:

res.cookie('sid','abcdef',{

ttl:60*60,

signed:{

secret:'keyboardcat',

algorithm:'sha256'

}

});

New:

res.cookie('sid','abcdef',{

ttl:60*60,

secret:'keyboardcat',

algorithm:'sha256'

});

Controllers

277

DependencyinjectionThereisnoequivalentoftheaddInjectormethodavailableinArangoDB2.xcontrollers.Mostusecasescanbesolvedbysimplyusingplainvariablesbutifyouneedsomethingmoreflexibleyoucanalsousemiddleware:

Old:

ctrl.addInjector('magicNumber',function(){

returnMath.random();

});

ctrl.get('/',function(req,res,injected){

res.json(injected.magicNumber);

});

New:

functionmagicMiddleware(name){

return{

register(){

letmagic;

returnfunction(req,res,next){

if(!magic){

magic=Math.random();

}

req[name]=magic;

next();

};

}

};

}

router.use(magicMiddleware('magicNumber'));

router.get('/',function(req,res){

res.json(req.magicNumber);

});

Orsimply:

constmagicNumber=Math.random();

router.get('/',function(req,res){

res.json(magicNumber);

});

Controllers

278

SessionsThectrl.activateSessionsmethodandtherelatedutil-sessions-localFoxxservicehavebeenreplacedwiththeFoxxsessionsmiddleware.Itisnolongerpossibletousethebuilt-insessionstoragebutyoucansimplypassinanydocumentcollectiondirectly.

Old:

constlocalSessions=applicationContext.dependencies.localSessions;

constsessionStorage=localSessions.sessionStorage;

ctrl.activateSessions({

sessionStorage:sessionStorage,

cookie:{secret:'keyboardcat'}

});

ctrl.destroySession('/logout',function(req,res){

res.json({message:'Goodbye!'});

});

New:

constsessionMiddleware=require('@arangodb/foxx/sessions');

constcookieTransport=require('@arangodb/foxx/sessions/transports/cookie');

router.use(sessionMiddleware({

storage:module.context.collection('sessions'),

transport:cookieTransport('keyboardcat')

}));

router.post('/logout',function(req,res){

req.sessionStorage.clear(req.session);

res.json({message:'Goodbye!'});

});

Sessions

279

AuthandOAuth2Theutil-simple-authandutil-oauth2FoxxserviceshavebeenreplacedwiththeFoxxauthandFoxxOAuth2modules.Itisnolongernecessarytoinstalltheseservicesasdependenciesinordertousethefunctionality.

Old:

'usestrict';

constauth=applicationContext.dependencies.simpleAuth;

//...

constvalid=auth.verifyPassword(authData,password);

New:

'usestrict';

constcreateAuth=require('@arangodb/foxx/auth');

constauth=createAuth();//Usedefaultconfiguration

//...

constvalid=auth.verifyPassword(authData,password);

AuthandOAuth2

280

FoxxqueriesThecreateQuerymethodhasbeenremoved.ItcanbetriviallyreplacedwithplainJavaScriptfunctionsanddirectcallstothedb._querymethod:

Old:

'usestrict';

constFoxx=require('org/arangodb/foxx');

constquery=Foxx.createQuery({

query:'FORuIN_usersSORTu.userASCRETURNu[@propName]',

params:['propName'],

transform:function(results,uppercase){

return(

uppercase

?results[0].toUpperCase()

:results[0].toLowerCase()

);

}

});

query('user',true);

New:

'usestrict';

constdb=require('@arangodb').db;

constaql=require('@arangodb').aql;

functionquery(propName,uppercase){

constresults=db._query(aql`

FORuIN_users

SORTu.userASC

RETURNu[${propName}]

`);

return(

uppercase

?results[0].toUpperCase()

:results[0].toLowerCase()

);

}

query('user',true);

FoxxQueries

281

Legacycompatibilitymodefor2.8servicesArangoDB3continuestosupportFoxxserviceswrittenforArangoDB2.8byrunningtheminaspeciallegacycompatibilitymodethatprovidesaccesstosomeofthemodulesandAPIsnolongerprovidedin3.0andbeyond.

Note:LegacycompatibilitymodeisstrictlyintendedasatemporarystopgapsolutionforsupportingexistingserviceswhileupgradingtoArangoDB3.0andshouldnotbeconsideredapermanentfeatureofArangoDBorFoxx.

Inordertomarkanexistingserviceasalegacyservice,justmakesurethefollowingattributeisdefinedintheservicemanifest:

"engines":{

"arangodb":"^2.8.0"

}

ThissemanticversionrangedenotesthattheserviceisknowntoworkwithArangoDB2.8.0andsupportsallnewerversionsofArangoDBuptobutnotincluding3.0.0(noranydevelopmentversionof3.0.0andgreater).

Anysimilarversionrangethedoesnotinclude3.0.0orgreaterwillhavethesameeffect(e.g. 2.5.0willalsotriggerthelegacycompatibilitymode,aswill1.2.3,but>=2.8.0willnotasitindicatescompatibilitywithallversionsgreaterorequal2.8.0,notjustthosewithinthe2.xversionrange).

FeaturessupportedinlegacycompatibilitymodeLegacycompatibilitymodesupportstheoldmanifestformat,specifically:

mainisignoredcontrollerswillbemountedasin2.8exportswillbeexecutedasin2.8

AdditionallytheisSystemattributewillbeignoredifpresentbutdoesnotresultinawarninginlegacycompatibilitymode.

TheFoxxconsoleisavailableastheconsolepseudo-globalvariable(shadowingtheglobalconsoleobject).

TheservicecontextisavailableastheapplicationContextpseudo-globalvariableinthecontrollers,exports,scriptsandtestsasin2.8.Thefollowingadditionalpropertiesareavailableontheservicecontextinlegacycompatibilitymode:

path()isanaliasfor3.xfileName()(usingpath.jointobuildfilepaths)fileName()behavesasin2.x(usingfs.safeJointobuildfilepaths)foxxFileName()isanaliasfor2.xfileNameversionexposestheservicemanifest'sversionattributenameexposestheservicemanifest'snameattributeoptionsexposestheservice'srawoptions

Thefollowingmethodsareremovedontheservicecontextinlegacycompatibilitymode:

use()--use@arangodb/foxx/controllerinsteadapiDocumentation()--usecontroller.apiDocumentation()insteadregisterType()--notsupportedinlegacycompatibilitymode

Thefollowingmodulesthathavebeenremovedorreplacedin3.0.0areavailableinlegacycompatibilitymode:

@arangodb/foxx/authentication

@arangodb/foxx/console

@arangodb/foxx/controller

@arangodb/foxx/model

@arangodb/foxx/query

@arangodb/foxx/repository

@arangodb/foxx/schema

@arangodb/foxx/sessions

@arangodb/foxx/template_middleware

Legacycompatibilitymode

282

The@arangodb/foxxmodulealsoprovidesthesameexportsasin2.8,namely:

Controllerfrom@arangodb/foxx/controllercreateQueryfrom@arangodb/foxx/queryModelfrom@arangodb/foxx/modelRepositoryfrom@arangodb/foxx/repositorytoJSONSchemafrom@arangodb/foxx/schemagetExportsandrequireAppfrom@arangodb/foxx/managerqueuesfrom@arangodb/foxx/queues

Anyfeaturenotsupportedin2.8willalsonotworkinlegacycompatibilitymode.WhenmigratingfromanolderversionofArangoDBitisagoodideatomigratetoArangoDB2.8firstforaneasierupgradepath.

AdditionallypleasenotethedifferenceslaidoutinthechapterMigratingfrompre-2.8inthemigrationguide.

Legacycompatibilitymode

283

UsermanagementFoxxdoesnotprovideanyusermanagementoutoftheboxbutitisveryeasytorollyourownsolution:

Thesessionmiddlewareprovidesmechanismsforaddingsessionlogictoyourservice,usinge.g.acollectionorJSONWebTokenstostorethesessionsbetweenrequests.

Theauthmoduleprovidesutilitiesforbasicpasswordverificationandhashing.

Thefollowingexampleservicedemonstrateshowusermanagementcanbeimplementedusingthesebasicbuildingblocks.

Settingupthecollections

Let'ssaywewanttostoresessionsandusersincollections.Wecanusethesetupscripttomakesurethesecollectionsarecreatedbeforetheserviceismounted.

Firstaddasetupscripttoyourmanifestifitisn'talreadydefined:

"scripts":{

"setup":"scripts/setup.js"

}

Thencreatethesetupscriptwiththefollowingcontent:

'usestrict';

constdb=require('@arangodb').db;

constsessions=module.context.collectionName('sessions');

constusers=module.context.collectionName('users');

if(!db._collection(sessions)){

db._createDocumentCollection(sessions);

}

if(!db._collection(users)){

db._createDocumentCollection(users);

}

db._collection(users).ensureIndex({

type:'hash',

fields:['username'],

unique:true

});

Creatingtherouter

Thefollowingmainfiledemonstratesbasicusermanagement:

'usestrict';

constjoi=require('joi');

constcreateAuth=require('@arangodb/foxx/auth');

constcreateRouter=require('@arangodb/foxx/router');

constsessionsMiddleware=require('@arangodb/foxx/sessions');

constauth=createAuth();

constrouter=createRouter();

constusers=module.context.collection('users');

constsessions=sessionsMiddleware({

storage:module.context.collection('sessions'),

transport:'cookie'

});

module.context.use(sessions);

module.context.use(router);

router.get('/whoami',function(req,res){

Usermanagement

284

try{

constuser=users.document(req.session.uid);

res.send({username:user.username});

}catch(e){

res.send({username:null});

}

})

.description('Returnsthecurrentlyactiveusername.');

router.post('/login',function(req,res){

//Thismayreturnauserobjectornull

constuser=users.firstExample({

username:req.body.username

});

constvalid=auth.verify(

//Pretendtovalidateevenifnouserwasfound

user?user.authData:{},

req.body.password

);

if(!valid)res.throw('unauthorized');

//Logtheuserin

req.session.uid=user._key;

req.sessionStorage.save(req.session);

res.send({sucess:true});

})

.body(joi.object({

username:joi.string().required(),

password:joi.string().required()

}).required(),'Credentials')

.description('Logsaregistereduserin.');

router.post('/logout',function(req,res){

if(req.session.uid){

req.session.uid=null;

req.sessionStorage.save(req.session);

}

res.send({success:true});

})

.description('Logsthecurrentuserout.');

router.post('/signup',function(req,res){

constuser=req.body;

try{

//Createanauthenticationhash

user.authData=auth.create(user.password);

deleteuser.password;

constmeta=users.save(user);

Object.assign(user,meta);

}catch(e){

//Failedtosavetheuser

//We'llassumetheUniqueConstrainthasbeenviolated

res.throw('badrequest','Usernamealreadytaken',e);

}

//Logtheuserin

req.session.uid=user._key;

req.sessionStorage.save(req.session);

res.send({success:true});

})

.body(joi.object({

username:joi.string().required(),

password:joi.string().required()

}).required(),'Credentials')

.description('Createsanewuserandlogsthemin.');

Usermanagement

285

RelatedmodulesThesearesomeofthemodulesoutsideofFoxxyouwillfindusefulwhenwritingFoxxservices.

AdditionallytherearemodulesprovidingsomelevelofcompatibilitywithNode.jsaswellasanumberofbundledNPMmodules(likelodashandjoi).FormoreinformationonthesemodulesseetheJavaScriptmodulesappendix.

The @arangodbmodulerequire('@arangodb')

ThismoduleprovidesaccesstovariousArangoDBinternalsaswellasthreeofthemostimportantexportsnecessarytoworkwiththedatabaseinFoxx:db,aqlanderrors.

YoucanfindafulldescriptionofthismoduleintheArangoDBmoduleappendix.

The @arangodb/requestmodulerequire('@arangodb/request')

ThismoduleprovidesafunctionformakingHTTPrequeststoexternalservices.Notethatwhilethisallowscommunicatingwiththird-partyservicesitmayaffectdatabaseperformancebyblockingFoxxrequestsasArangoDBwaitsfortheremoteservicetorespond.Ifyouroutinelymakerequeststoslowexternalservicesandarenotdirectlyinterestedintheresponseitisprobablyabetterideatodelegatetheactualrequest/responsecycletoagatewayservicerunningoutsideArangoDB.

Youcanfindafulldescriptionofthismoduleintherequestmoduleappendix.

The @arangodb/general-graphmodulerequire('@arangodb/general-graph')

ThismoduleprovidesaccesstoArangoDBgraphdefinitionsandvariouslow-levelgraphoperationsinJavaScript.FormorecomplexqueriesitisprobablybettertouseAQLbutthismodulecanbeusefulinyoursetupandteardownscriptstocreateanddestroygraphdefinitions.

Formoreinformationseethechapteronthegeneralgraphmodule.

Relatedmodules

286

AuthenticationconstcreateAuth=require('@arangodb/foxx/auth');

Authenticatorsallowimplementingbasicpasswordmechanismusingsimplebuilt-inhashingfunctions.

ForafullexampleofsessionswithauthenticationandregistrationseetheexampleinthechapteronUserManagement.

CreatinganauthenticatorcreateAuth([options]):Authenticator

Createsanauthenticator.

Arguments

options:Object(optional)

Anobjectwiththefollowingproperties:

method:string(Default:"sha256")

Thehashingalgorithmtousetocreatepasswordhashes.Theauthenticatorwillbeabletoverifypasswordsagainsthashesusinganysupportedhashingalgorithm.Thisonlyaffectsnewhashescreatedbytheauthenticator.

Supportedvalues:

"md5"

"sha1"

"sha224"

"sha256"

"sha384"

"sha512"

saltLength:number(Default:16)

Lengthofthesaltsthatwillbegeneratedforpasswordhashes.

Returnsanauthenticator.

Creatingauthenticationdataobjectsauth.create(password):AuthData

Createsanauthenticationdataobjectforthegivenpasswordwiththefollowingproperties:

method:string

Themethodusedtogeneratethehash.

salt:string

Arandomsaltusedtogeneratethishash.

hash:string

Thehashstringitself.

Arguments

password:string

Apasswordtohash.

Returnstheauthenticationdataobject.

Authentication

287

Validatingpasswordsagainstauthenticationdataobjectsauth.verify([hash,[password]]):boolean

Verifiesthegivenpasswordagainstthegivenhashusingaconstanttimestringcomparison.

Arguments

hash:AuthData(optional)

Aauthenticationdataobjectgeneratedwiththecreatemethod.

password:string(optional)

Apasswordtoverifyagainstthehash.

Returnstrueifthehashmatchesthegivenpassword.Returnsfalseotherwise.

Authentication

288

OAuth1.0aconstcreateOAuth1Client=require('@arangodb/foxx/oauth1');

TheOAuth1moduleprovidesabstractionsoverOAuth1.0aproviderslikeTwitter,XINGandTumblr.

Examples

Thefollowingextendstheusermanagementexample:

constrouter=createRouter();

constoauth1=createOAuth1Client({

//We'lluseTwitterforthisexample

requestTokenEndpoint:'https://api.twitter.com/oauth/request_token',

authEndpoint:'https://api.twitter.com/oauth/authorize',

accessTokenEndpoint:'https://api.twitter.com/oauth/access_token',

activeUserEndpoint:'https://api.twitter.com/1.1/account/verify_credentials.json',

clientId:'keyboardcat',

clientSecret:'keyboardcat'

});

module.context.use('/oauth1',router);

//Seetheusermanagementexampleforsettingupthe

//sessionsandusersobjectsusedinthisexample

router.use(sessions);

router.post('/auth',function(req,res){

consturl=req.reverse('oauth1_callback');

constoauth_callback=req.makeAbsolute(url);

constrequestToken=oauth1.fetchRequestToken(oauth_callback);

if(requestToken.oauth_callback_confirmed!=='true'){

res.throw(500,'CouldnotfetchOAuthrequesttoken');

}

//Setrequesttokencookieforfiveminutes

res.cookie('oauth1_request_token',requestToken.oauth_token,{ttl:60*5});

//Redirecttotheprovider'sauthorizationURL

res.redirect(303,oauth1.getAuthUrl(requestToken.oauth_token));

});

router.get('/auth',function(req,res){

//MakesureCSRFcookiematchestheURL

constexpectedToken=req.cookie('oauth1_request_token');

if(!expectedToken||req.queryParams.oauth_token!==expectedToken){

res.throw(400,'CSRFmismatch.');

}

constauthData=oauth1.exchangeRequestToken(

req.queryParams.oauth_token,

req.queryParams.oauth_verifier

);

consttwitterToken=authData.oauth_token;

consttwitterSecret=authData.oauth_token_secret;

//Fetchtheactiveuser'sprofileinfo

constprofile=oauth1.fetchActiveUser(twitterToken,twitterSecret);

consttwitterId=profile.screen_name;

//TrytofindanexistinguserwiththeuserID

//(thisrequirestheuserscollection)

letuser=users.firstExample({twitterId});

if(user){

//UpdatethetwitterTokenifithaschanged

if(

user.twitterToken!==twitterToken||

user.twitterSecret!==twitterSecret

){

users.update(user,{twitterToken,twitterSecret});

}

}else{

//Createanewuserdocument

user={

username:`twitter:${twitterId}`,

twitterId,

twitterToken

OAuth1.0a

289

}

constmeta=users.save(user);

Object.assign(user,meta);

}

//Logtheuserin(thisrequiresthesessionmiddleware)

req.session.uid=user._key;

req.session.twitterToken=authData.twitterToken;

req.session.twitterSecret=authData.twitterSecret;

req.sessionStorage.save(req.session);

//Redirecttothedefaultroute

res.redirect(303,req.makeAbsolute('/'));

},'oauth1_callback')

.queryParam('oauth_token',joi.string().optional())

.queryParam('oauth_verifier',joi.string().optional());

CreatinganOAuth1.0aclientcreateOAuth1Client(options):OAuth1Client

CreatesanOAuth1.0aclient.

Arguments

options:Object

Anobjectwiththefollowingproperties:

requestTokenEndpoint:string

Thefully-qualifiedURLoftheprovider'sTemporaryCredentialsRequestendpoint.ThisURLisusedtofetchtheunauthenticatedtemporarycredentialsthatwillbeusedtogeneratetheauthorizationredirectfortheuser.

authEndpoint:string

Thefully-qualifiedURLoftheprovider'sResourceOwnerAuthorizationendpoint.ThisistheURLtheuserwillberedirectedtoinordertoauthorizetheOAuthconsumer(i.e.yourservice).

accessTokenEndpoint:string

Thefully-qualifiedURLoftheprovider'sTokenRequestendpoint.ThisURLisusedtoexchangetheauthenticatedtemporarycredentialsreceivedfromtheauthorizationredirectfortheactualtokencredentialsthatcanbeusedtomakerequeststotheAPIserver.

activeUserEndpoint:string(optional)

Thefully-qualifiedURLoftheprovider'sendpointforfetchingdetailsaboutthecurrentuser.

clientId:string

Theapplication'sClientID(orConsumerKey)fortheprovider.

clientSecret:string

Theapplication'sClientSecret(orConsumerSecret)fortheprovider.

signatureMethod:string(Default:"HMAC-SHA1")

ThecryptographicmethodthatwillbeusedtosignOAuth1.0arequests.Only"HMAC-SHA1-"and"PLAINTEXT"aresupportedatthistime.

Notethatmanyprovidersmaynotimplement"PLAINTEXT"asitexposestheClientSecretandoauth_token_secretinsteadofgeneratingasignature.

ReturnsanOAuth1.0aclientforthegivenprovider.

SettingupOAuth1.0aforTwitter

IfyouwanttouseTwitterastheOAuth1.0aprovider,usethefollowingoptions:

OAuth1.0a

290

requestTokenEndpoint:https://api.twitter.com/oauth/request_tokenauthEndpoint:https://api.twitter.com/oauth/authorizeaccessTokenEndpoint:https://api.twitter.com/oauth/access_tokenactiveUserEndpoint:https://api.twitter.com/1.1/account/verify_credentials.json

YoualsoneedtoobtainaclientIDandclientsecretfromTwitter:

1. CreatearegularaccountatTwitteroruseanexistingaccountyouown.2. VisittheTwitterApplicationManagementdashboardandsigninwithyourTwitteraccount.3. ClickonCreateNewAppandfollowtheinstructionsprovided.TheCallbackURLshouldmatchyouroauth_callbacklater.You

maybepromptedtoaddamobilephonenumbertoyouraccountandverifyit.4. OpentheKeysandAccessTonestab,thennotedowntheConsumerKeyandConsumerSecret.5. SettheoptionclientIdtotheConsumerKeyandtheoptionclientSecrettotheConsumerSecret.

Notethatifyouonlyneedread-onlyaccesstopublicinformation,youcanalsousetheclientIdandclientSecretdirectlywithoutOAuth1.0a.

SeeTwitterRESTAPIReferenceDocumentation.

SettingupOAuth1.0aforXING

IfyouwanttouseXINGastheOAuth1.0aprovider,usethefollowingoptions:

requestTokenEndpoint:https://api.xing.com/v1/request_tokenauthEndpoint:https://api.xing.com/v1/authorizeaccessTokenEndpoint:https://api.xing.com/v1/access_tokenactiveUserEndpoint:https://api.xing.com/v1/users/me

YoualsoneedtoobtainaclientIDandclientsecretfromXING:

1. CreatearegularaccountatXINGoruseanexistingaccountyouown.2. VisittheXINGDeveloperpageandsigninwithyourXINGaccount.3. ClickonCreateappandnotedowntheConsumerkeyandConsumersecret.4. SettheoptionclientIdtotheConsumerkeyandtheoptionclientSecrettotheConsumersecret.

SeeXINGDeveloperDocumentation.

SettingupOAuth1.0aforTumblr

IfyouwanttouseTumblrastheOAuth1.0aprovider,usethefollowingoptions:

requestTokenEndpoint:https://www.tumblr.com/oauth/request_tokenauthEndpoint:https://www.tumblr.com/oauth/authorizeaccessTokenEndpoint:https://www.tumblr.com/oauth/access_tokenactiveUserEndpoint:https://api.tumblr.com/v2/user/info

YoualsoneedtoobtainaclientIDandclientsecretfromTumblr:

1. CreatearegularaccountatTumblroruseanexistingaccountyouown.2. VisittheTumblrApplicationsdashboard.3. ClickonRegisterapplication,thenfollowtheinstructionsprovided.TheDefaultcallbackURLshouldmatchyouroauth_callback

later.4. NotedowntheOAuthConsumerKeyandSecretKey.Thesecretmaybehiddenbydefault.5. SettheoptionclientIdtotheOAuthConsumerKeyandtheoptionclientSecrettotheSecretKey.

SeeTumblrAPIDocumentation.

Fetchanunauthenticatedrequesttokenoauth1.fetchRequestToken(oauth_callback,opts)

Fetchesanoauth_tokenthatcanbeusedtocreateanauthorizationURLthatredirectstothegivenoauth_callbackonconfirmation.

OAuth1.0a

291

PerformsaPOSTresponsetotherequestTokenEndpoint.

Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.

Arguments

oauth_callback:string

Thefully-qualifiedURLofyourapplication'sOAuth1.0acallback.

opts:Object(optional)

Anobjectwithadditionalqueryparameterstoincludeintherequest.

SeeRFC5849.

Returnstheparsedresponseobject.

GettheauthorizationURLoauth1.getAuthUrl(oauth_token,opts):string

GeneratestheauthorizationURLfortheauthorizationendpoint.

Arguments

oauth_token:string

Theoauth_tokenpreviouslyreturnedbyfetchRequestToken.

opts:(optional)

AnobjectwithadditionalqueryparameterstoaddtotheURL.

SeeRFC5849.

Returnsafully-qualifiedURLfortheauthorizationendpointoftheproviderbyappendingtheoauth_tokenandanyadditionalargumentsfromoptstotheauthEndpoint.

Examples

constrequestToken=oauth1.fetchRequestToken(oauth_callback);

if(requestToken.oauth_callback_confirmed!=='true'){

thrownewError('ProvidercouldnotconfirmOAuth1.0callback');

}

constauthUrl=oauth1.getAuthUrl(requestToken.oauth_token);

Exchangeanauthenticatedrequesttokenforanaccesstokenoauth1.exchangeRequestToken(oauth_token,oauth_verifier,opts)

TakesapairofauthenticatedtemporarycredentialspassedtothecallbackURLbytheproviderandexchangesitforanoauth_tokenandoauth_token_secretthancanbeusedtoperformauthenticatedrequeststotheOAuth1.0aprovider.

PerformsaPOSTresponsetotheaccessTokenEndpoint.

Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.

Arguments

oauth_token:string

Theoauth_tokenpassedtothecallbackURLbytheprovider.

oauth_verifier:string

Theoauth_verifierpassedtothecallbackURLbytheprovider.

opts:Object(optional)

OAuth1.0a

292

Anobjectwithadditionalqueryparameterstoincludeintherequest.

SeeRFC5849.

Returnstheparsedresponseobject.

Fetchtheactiveuseroauth1.fetchActiveUser(oauth_token,oauth_token_secret,opts):Object

Fetchesdetailsoftheactiveuser.

PerformsaGETresponsetotheactiveUserEndpoint.

Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.

ReturnsnulliftheactiveUserEndpointisnotconfigured.

Arguments

oauth_token:string

AnOAuth1.0aaccesstokenasreturnedbyexchangeRequestToken.

oauth_token_secret:string

AnOAuth1.0aaccesstokensecretasreturnedbyexchangeRequestToken.

opts:Object(optional)

Anobjectwithadditionalqueryparameterstoincludeintherequest.

SeeRFC5849.

Returnstheparsedresponseobject.

Examples

constauthData=oauth1.exchangeRequestToken(oauth_token,oauth_verifier);

constuserData=oauth1.fetchActiveUser(authData.oauth_token,authData.oauth_token_secret);

Createanauthenticatedrequestobjectoauth1.createSignedRequest(method,url,parameters,oauth_token,oauth_token_secret)

CreatesarequestobjectthatcanbeusedtoperformarequesttotheOAuth1.0aproviderwiththeprovidedtokencredentials.

Arguments

method:string

HTTPmethodtherequestwilluse,e.g."POST".

url:string

Thefully-qualifiedURLoftheprovidertherequestwillbeperformedagainst.

TheURLmayoptionallycontainanynumberofqueryparameters.

parameters:string|Object|null

Anadditionalobjectorquerystringcontainingqueryparametersorbodyparametersthatwillbepartofthesignedrequest.

oauth_token:string

AnOAuth1.0aaccesstokenasreturnedbyexchangeRequestToken.

oauth_token_secret:string

OAuth1.0a

293

AnOAuth1.0aaccesstokensecretasreturnedbyexchangeRequestToken.

Returnsanobjectwiththreeproperties:

url:ThenormalizedURLwithoutanyqueryparameters.

qs:Anormalizedquerystringcontainingallparametersandqueryparameters.

headers:Anobjectcontainingthefollowingproperties:

accept:Thestring"application/json".

authorization:AnOAuthauthorizationheadercontainingallOAuthparametersandtherequestsignature.

Examples

Fetchalistoftweetsmentioning@arangodb:

constrequest=require('@arangodb/request');

constreq=oauth1.createSignedRequest(

'GET',

'https://api.twitter.com/1.1/search/tweets.json',

{q:'@arangodb'},

authData.oauth_token,

authData.oauth_token_secret

);

constres=request(req);

console.log(res.json.statuses);

Signingamorecomplexrequest:

consturl='https://api.example.com/v1/timeline?visible=public';

constparams={hello:'world',longcat:'islong'};

constreq=oauth1.createSignedRequest(

'POST',

url,//URLincludesaqueryparameterthatwillbesigned

params,//Requestbodyneedstobesignedtoo

authData.oauth_token,

authData.oauth_token_secret

);

constres=request.post(url,{

form:params,

headers:{

accept:'application/x-www-form-urlencoded',

//Authorizationheaderincludesthesignature

authorization:req.headers.authorization

}

});

console.log(res.json);

OAuth1.0a

294

OAuth2.0constcreateOAuth2Client=require('@arangodb/foxx/oauth2');

TheOAuth2moduleprovidesabstractionsoverOAuth2.0providerslikeFacebook,GitHubandGoogle.

Examples

Thefollowingextendstheusermanagementexample:

constcrypto=require('@arangodb/crypto');

constrouter=createRouter();

constoauth2=createOAuth2Client({

//We'lluseFacebookforthisexample

authEndpoint:'https://www.facebook.com/dialog/oauth',

tokenEndpoint:'https://graph.facebook.com/oauth/access_token',

activeUserEndpoint:'https://graph.facebook.com/v2.0/me',

clientId:'keyboardcat',

clientSecret:'keyboardcat'

});

module.context.use('/oauth2',router);

//Seetheusermanagementexampleforsettingupthe

//sessionsandusersobjectsusedinthisexample

router.use(sessions);

router.post('/auth',function(req,res){

constcsrfToken=crypto.genRandomAlphaNumbers(32);

consturl=req.reverse('oauth2_callback',{csrfToken});

constredirect_uri=req.makeAbsolute(url);

//SetCSRFcookieforfiveminutes

res.cookie('oauth2_csrf_token',csrfToken,{ttl:60*5});

//Redirecttotheprovider'sauthorizationURL

res.redirect(303,oauth2.getAuthUrl(redirect_uri));

});

router.get('/auth',function(req,res){

//Someproviderspasserrorsasqueryparameter

if(req.queryParams.error){

res.throw(500,`Providererror:${req.queryParams.error}`)

}

//MakesureCSRFcookiematchestheURL

constexpectedToken=req.cookie('oauth2_csrf_token');

if(!expectedToken||req.queryParams.csrfToken!==expectedToken){

res.throw(400,'CSRFmismatch.');

}

//MakesuretheURLcontainsagranttoken

if(!req.queryParams.code){

res.throw(400,'Providerdidnotpassgranttoken.');

}

//Reconstructtheredirect_uriusedforthegranttoken

consturl=req.reverse('oauth2_callback');

constredirect_uri=req.makeAbsolute(url);

//Fetchanaccesstokenfromtheprovider

constauthData=oauth2.exchangeGrantToken(

req.queryParams.code,

redirect_uri

);

constfacebookToken=authData.access_token;

//Fetchtheactiveuser'sprofileinfo

constprofile=oauth2.fetchActiveUser(facebookToken);

constfacebookId=profile.id;

//TrytofindanexistinguserwiththeuserID

//(thisrequirestheuserscollection)

letuser=users.firstExample({facebookId});

if(user){

//UpdatethefacebookTokenifithaschanged

if(user.facebookToken!==facebookToken){

users.update(user,{facebookToken});

}

}else{

OAuth2.0

295

//Createanewuserdocument

user={

username:`fb:${facebookId}`,

facebookId,

facebookToken

}

constmeta=users.save(user);

Object.assign(user,meta);

}

//Logtheuserin(thisrequiresthesessionmiddleware)

req.session.uid=user._key;

req.session.facebookToken=authData.facebookToken;

req.sessionStorage.save(req.session);

//Redirecttothedefaultroute

res.redirect(303,req.makeAbsolute('/'));

},'oauth2_callback')

.queryParam('error',joi.string().optional())

.queryParam('csrfToken',joi.string().optional())

.queryParam('code',joi.string().optional());

CreatinganOAuth2.0clientcreateOAuth2Client(options):OAuth2Client

CreatesanOAuth2.0client.

Arguments

options:Object

Anobjectwiththefollowingproperties:

authEndpoint:string

Thefully-qualifiedURLoftheprovider'sauthorizationendpoint.

tokenEndpoint:string

Thefully-qualifiedURLoftheprovider'stokenendpoint.

refreshEndpoint:string(optional)

Thefully-qualifiedURLoftheprovider'srefreshtokenendpoint.

activeUserEndpoint:string(optional)

Thefully-qualifiedURLoftheprovider'sendpointforfetchingdetailsaboutthecurrentuser.

clientId:string

Theapplication'sClientID(orAppID)fortheprovider.

clientSecret:string

Theapplication'sClientSecret(orAppSecret)fortheprovider.

ReturnsanOAuth2.0clientforthegivenprovider.

SettingupOAuth2.0forFacebook

IfyouwanttouseFacebookastheOAuth2.0provider,usethefollowingoptions:

authEndpoint:https://www.facebook.com/dialog/oauthtokenEndpoint:https://graph.facebook.com/oauth/access_tokenactiveUserEndpoint:https://graph.facebook.com/v2.0/me

YoualsoneedtoobtainaclientIDandclientsecretfromFacebook:

1. CreatearegularaccountatFacebookoruseanexistingaccountyouown.2. VisittheFacebookDeveloperspage.

OAuth2.0

296

3. ClickonAppsinthemenu,thenselectRegisterasaDeveloper(theonlyoption)andfollowtheinstructionsprovided.Youmayneedtoverifyyouraccountbyphone.

4. ClickonAppsinthemenu,thenselectCreateaNewAppandfollowtheinstructionsprovided.5. Opentheappdashboard,thennotedowntheAppIDandAppSecret.Thesecretmaybehiddenbydefault.6. ClickonSettings,thenAdvancedandenteroneormoreValidOAuthredirectURIs.Atleastoneofthemmustmatchyourredirect_urilater.Don'tforgettosaveyourchanges.

7. SettheoptionclientIdtotheAppIDandtheoptionclientSecrettotheAppSecret.

SettingupOAuth2.0forGitHub

IfyouwanttouseGitHubastheOAuth2.0provider,usethefollowingoptions:

authEndpoint:https://github.com/login/oauth/authorize?scope=usertokenEndpoint:https://github.com/login/oauth/access_tokenactiveUserEndpoint:https://api.github.com/user

YoualsoneedtoobtainaclientIDandclientsecretfromGitHub:

1. CreatearegularaccountatGitHuboruseanexistingaccountyouown.2. GotoAccountSettings>Applications>Registernewapplication.3. ProvideanauthorizationcallbackURL.Thismustmatchyourredirect_urilater.4. Fillintheotherrequireddetailsandfollowtheinstructionsprovided.5. Opentheapplicationpage,thennotedowntheClientIDandClientSecret.6. SettheoptionclientIdtotheClientIDandtheoptionclientSecrettotheClientSecret.

SettingupOAuth2.0forGoogle

IfyouwanttouseGoogleastheOAuth2.0provider,usethefollowingoptions:

authEndpoint:https://accounts.google.com/o/oauth2/auth?access_type=offline&scope=profiletokenEndpoint:https://accounts.google.com/o/oauth2/tokenactiveUserEndpoint:https://www.googleapis.com/plus/v1/people/me

YoualsoneedtoobtainaclientIDandclientsecretfromGoogle:

1. CreatearegularaccountatGoogleoruseanexistingaccountyouown.2. VisittheGoogleDevelopersConsole.3. ClickonCreateProject,thenfollowtheinstructionsprovided.4. Whenyourprojectisready,opentheprojectdashboard,thenclickonEnableanAPI.5. EnabletheGoogle+APItoallowyourapptodistinguishbetweendifferentusers.6. OpentheCredentialspageandclickCreatenewClientID,thenfollowtheinstructionsprovided.AtleastoneAuthorizedRedirectURImustmatchyourredirect_urilater.AtleastoneAuthorizedJavaScriptOriginmustmatchyourapp'sfully-qualifieddomain.

7. WhentheClientIDisready,notedowntheClientIDandClientsecret.8. SettheoptionclientIdtotheClientIDandtheoptionclientSecrettotheClientsecret.

GettheauthorizationURLoauth2.getAuthUrl(redirect_uri,args):string

GeneratestheauthorizationURLfortheauthorizationendpoint.

Arguments

redirect_uri:string

Thefully-qualifiedURLofyourapplication'sOAuth2.0callback.

args:(optional)

Anobjectwithanyofthefollowingproperties:

response_type:string(Default:"code")

OAuth2.0

297

SeeRFC6749.

Returnsafully-qualifiedURLfortheauthorizationendpointoftheproviderbyappendingtheclientIDandanyadditionalargumentsfromargstotheauthEndpoint.

Exchangeagrantcodeforanaccesstokenoauth2.exchangeGrantToken(code,redirect_uri)

Exchangesagrantcodeforanaccesstoken.

PerformsaPOSTresponsetothetokenEndpoint.

Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.

Arguments

code:string

Agrantcodereturnedbytheprovider'sauthorizationendpoint.

redirect_uri:string

TheoriginalcallbackURLwithwhichthecodewasrequested.

args:Object(optional)

Anobjectwithanyofthefollowingproperties:

grant_type:string(Default:"authorization_code")

SeeRFC6749.

Returnstheparsedresponseobject.

Fetchtheactiveuseroauth2.fetchActiveUser(access_token):Object

Fetchesdetailsoftheactiveuser.

PerformsaGETresponsetotheactiveUserEndpoint.

Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.

ReturnsnulliftheactiveUserEndpointisnotconfigured.

Arguments

access_token:string

AnOAuth2.0accesstokenasreturnedbyexchangeGrantToken.

Returnstheparsedresponseobject.

Examples

constauthData=oauth2.exchangeGrantToken(code,redirect_uri);

constuserData=oauth2.fetchActiveUser(authData.access_token);

OAuth2.0

298

TransactionsStartingwithversion1.3,ArangoDBprovidessupportforuser-definabletransactions.

TransactionsinArangoDBareatomic,consistent,isolated,anddurable(ACID).

TheseACIDpropertiesprovidethefollowingguarantees:

Theatomicityprinciplemakestransactionseithercompleteintheirentiretyorhavenoeffectatall.Theconsistencyprincipleensuresthatnoconstraintsorotherinvariantswillbeviolatedduringorafteranytransaction.Theisolationpropertywillhidethemodificationsofatransactionfromothertransactionsuntilthetransactioncommits.Finally,thedurabilitypropositionmakessurethatoperationsfromtransactionsthathavecommittedwillbemadepersistent.TheamountoftransactiondurabilityisconfigurableinArangoDB,asisthedurabilityoncollectionlevel.

Transactions

299

TransactioninvocationArangoDBtransactionsaredifferentfromtransactionsinSQL.

InSQL,transactionsarestartedwithexplicitBEGINorSTARTTRANSACTIONcommand.Followinganyseriesofdataretrievalormodificationoperations,anSQLtransactionisfinishedwithaCOMMITcommand,orrolledbackwithaROLLBACKcommand.Theremaybeclient/servercommunicationbetweenthestartandthecommit/rollbackofanSQLtransaction.

InArangoDB,atransactionisalwaysaserver-sideoperation,andisexecutedontheserverinonego,withoutanyclientinteraction.Alloperationstobeexecutedinsideatransactionneedtobeknownbytheserverwhenthetransactionisstarted.

TherearenoindividualBEGIN,COMMITorROLLBACKtransactioncommandsinArangoDB.Instead,atransactioninArangoDBisstartedbyprovidingadescriptionofthetransactiontothedb._executeTransactionJavaScriptfunction:

db._executeTransaction(description);

Thisfunctionwillthenautomaticallystartatransaction,executeallrequireddataretrievaland/ormodificationoperations,andattheendautomaticallycommitthetransaction.Ifanerroroccursduringtransactionexecution,thetransactionisautomaticallyaborted,andallchangesarerolledback.

Executetransaction

executesatransactiondb._executeTransaction(object)

Executesaserver-sidetransaction,asspecifiedbyobject.

objectmusthavethefollowingattributes:

collections:asub-objectthatdefineswhichcollectionswillbeusedinthetransaction.collectionscanhavetheseattributes:read:asinglecollectionoralistofcollectionsthatwillbeusedinthetransactioninread-onlymodewrite:asinglecollectionoralistofcollectionsthatwillbeusedinthetransactioninwriteorreadmode.

action:aJavascriptfunctionorastringwithJavascriptcodecontainingalltheinstructionstobeexecutedinsidethetransaction.Ifthecoderunsthroughsuccessfully,thetransactionwillbecommittedattheend.Ifthecodethrowsanexception,thetransactionwillberolledbackandalldatabaseoperationswillberolledback.

Additionally,objectcanhavethefollowingoptionalattributes:

waitForSync:booleanflagindicatingwhetherthetransactionisforcedtobesynchronous.lockTimeout:anumericvaluethatcanbeusedtosetatimeoutforwaitingoncollectionlocks.Ifnotspecified,adefaultvaluewillbeused.SettinglockTimeoutto0willmakeArangoDBnottimeoutwaitingforalock.params:optionalargumentspassedtothefunctionspecifiedinaction.

ThefollowingattributescanbeusedfortransactionsintheRocksDBstorageengine:

maxTransactionSize:transactionsizelimitinbytesintermediateCommitSize:maximumtotalsizeofoperationsafterwhichanintermediatecommitisperformedautomaticallyintermediateCommitCount:maximumnumberofoperationsafterwhichanintermediatecommitisperformedautomatically

Declarationofcollections

Allcollectionswhicharetoparticipateinatransactionneedtobedeclaredbeforehand.Thisisanecessitytoensureproperlockingandisolation.

Collectionscanbeusedinatransactioninwritemodeorinread-onlymode.

Ifanydatamodificationoperationsaretobeexecuted,thecollectionmustbedeclaredforuseinwritemode.Thewritemodeallowsmodifyingandreadingdatafromthecollectionduringthetransaction(i.e.thewritemodeincludesthereadmode).

Contrary,usingacollectioninread-onlymodewillonlyallowperformingreadoperationsonacollection.Anyattempttowriteintoacollectionusedinread-onlymodewillmakethetransactionfail.

Transactioninvocation

300

Collectionsforatransactionaredeclaredbyprovidingtheminthecollectionsattributeoftheobjectpassedtothe_executeTransactionfunction.Thecollectionsattributehasthesub-attributesreadandwrite:

db._executeTransaction({

collections:{

write:["users","logins"],

read:["recommendations"]

}

});

readandwriteareoptionalattributes,andonlyneedtobespecifiediftheoperationsinsidethetransactionsdemandforit.

Thecontentsofreadorwritecaneachbelistsarrayscollectionnamesorasinglecollectionname(asastring):

db._executeTransaction({

collections:{

write:"users",

read:"recommendations"

}

});

Note:Itiscurrentlyoptionaltospecifycollectionsforread-onlyaccess.Evenwithoutspecifyingthem,itisstillpossibletoreadfromsuchcollectionsfromwithinatransaction,butwithrelaxedisolation.PleaserefertoTransactionsLockingformoredetails.

Inordertomakeatransactionfailwhenanon-declaredcollectionisusedinsideforreading,theoptionalallowImplicitsub-attributeofcollectionscanbesettofalse:

db._executeTransaction({

collections:{

read:"recommendations",

allowImplicit:false/*thisdisallowsreadaccesstoothercollections

thanspecified*/

},

action:function(){

vardb=require("@arangodb").db;

returndb.foobar.toArray();/*willfailbecausedb.foobarmustnotbeaccessed

forreadinginsidethistransaction*/

}

});

ThedefaultvalueforallowImplicitistrue.Write-accessingcollectionsthathavenotbeendeclaredinthecollectionsarrayisneverpossible,regardlessofthevalueofallowImplicit.

Declarationofdatamodificationandretrievaloperations

AlldatamodificationandretrievaloperationsthataretobeexecutedinsidethetransactionneedtobespecifiedinaJavascriptfunction,usingtheactionattribute:

db._executeTransaction({

collections:{

write:"users"

},

action:function(){

//alloperationsgohere

}

});

AnyvalidJavascriptcodeisallowedinsideactionbutthecodemayonlyaccessthecollectionsdeclaredincollections.actionmaybeaJavascriptfunctionasshownabove,orastringrepresentationofaJavascriptfunction:

db._executeTransaction({

collections:{

write:"users"

},

action:"function(){doSomething();}"

});

Transactioninvocation

301

Pleasenotethatanyoperationsspecifiedinactionwillbeexecutedontheserver,inaseparatescope.Variableswillbeboundlate.AccessinganyJavaScriptvariablesdefinedontheclient-sideorinsomeotherservercontextfrominsideatransactionmaynotwork.Instead,anyvariablesusedinsideactionshouldbedefinedinsideactionitself:

db._executeTransaction({

collections:{

write:"users"

},

action:function(){

vardb=require(...).db;

db.users.save({...});

}

});

Whenthecodeinsidetheactionattributeisexecuted,thetransactionisalreadystartedandallrequiredlockshavebeenacquired.Whenthecodeinsidetheactionattributefinishes,thetransactionwillautomaticallycommit.Thereisnoexplicitcommitcommand.

Tomakeatransactionabortandrollbackallchanges,anexceptionneedstobethrownandnotcaughtinsidethetransaction:

db._executeTransaction({

collections:{

write:"users"

},

action:function(){

vardb=require("@arangodb").db;

db.users.save({_key:"hello"});

//willabortandrollbackthetransaction

throw"doh!";

}

});

Thereisnoexplicitabortorrollbackcommand.

Asmentionedearlier,atransactionwillcommitautomaticallywhentheendoftheactionfunctionisreachedandnoexceptionhasbeenthrown.Inthiscase,theusercanreturnanylegalJavaScriptvaluefromthefunction:

db._executeTransaction({

collections:{

write:"users"

},

action:function(){

vardb=require("@arangodb").db;

db.users.save({_key:"hello"});

//willcommitthetransactionandreturnthevalue"hello"

return"hello";

}

});

Customexceptions

Onemaywishtodefinecustomexceptionsinsideofatransaction.Tohavetheexceptionpropagateupwardsproperly,pleasethrowananinstanceofbaseJavaScriptErrorclassoraderivative.Tospecifyanerrornumber,includeitastheerrorNumberfield.Asanexample:

db._executeTransaction({

collections:{},

action:function(){

varerr=newError('Myerrorcontext');

err.errorNumber=1234;

throwerr;

}

});

Transactioninvocation

302

Note:Inpreviousversions,customexceptionswhichdidnothaveanError-likeformweresimplyconvertedtostringsandexposedintheexceptionfieldofthereturnederror.Thisisnolongerthecase,asithadthepotentialtoleakunwantedinformationifimproperlyused.

Examples

Thefirstexamplewillwrite3documentsintoacollectionnamedc1.Thec1collectionneedstobedeclaredinthewriteattributeofthecollectionsattributepassedtotheexecuteTransactionfunction.

Theactionattributecontainstheactualtransactioncodetobeexecuted.Thiscodecontainsalldatamodificationoperations(3inthisexample).

//setup

db._create("c1");

db._executeTransaction({

collections:{

write:["c1"]

},

action:function(){

vardb=require("@arangodb").db;

db.c1.save({_key:"key1"});

db.c1.save({_key:"key2"});

db.c1.save({_key:"key3"});

}

});

db.c1.count();//3

Abortingthetransactionbythrowinganexceptionintheactionfunctionwillrevertallchanges,soasifthetransactionneverhappened:

//setup

db._create("c1");

db._executeTransaction({

collections:{

write:["c1"]

},

action:function(){

vardb=require("@arangodb").db;

db.c1.save({_key:"key1"});

db.c1.count();//1

db.c1.save({_key:"key2"});

db.c1.count();//2

throw"doh!";

}

});

db.c1.count();//0

Theautomaticrollbackisalsoexecutedwhenaninternalexceptionisthrownatsomepointduringtransactionexecution:

//setup

db._create("c1");

db._executeTransaction({

collections:{

write:["c1"]

},

action:function(){

vardb=require("@arangodb").db;

db.c1.save({_key:"key1"});

//willthrowduplicateakeyerror,notexplicitlyrequestedbytheuser

db.c1.save({_key:"key1"});

//we'llnevergethere...

}

});

db.c1.count();//0

Transactioninvocation

303

Asrequiredbytheconsistencyprinciple,abortingorrollingbackatransactionwillalsorestoresecondaryindexestothestateattransactionstart.

Cross-collectiontransactions

There'salsothepossibilitytorunatransactionacrossmultiplecollections.Inthiscase,multiplecollectionsneedtobedeclaredinthecollectionsattribute,e.g.:

//setup

db._create("c1");

db._create("c2");

db._executeTransaction({

collections:{

write:["c1","c2"]

},

action:function(){

vardb=require("@arangodb").db;

db.c1.save({_key:"key1"});

db.c2.save({_key:"key2"});

}

});

db.c1.count();//1

db.c2.count();//1

Again,throwinganexceptionfrominsidetheactionfunctionwillmakethetransactionabortandrollbackallchangesinallcollections:

//setup

db._create("c1");

db._create("c2");

db._executeTransaction({

collections:{

write:["c1","c2"]

},

action:function(){

vardb=require("@arangodb").db;

for(vari=0;i<100;++i){

db.c1.save({_key:"key"+i});

db.c2.save({_key:"key"+i});

}

db.c1.count();//100

db.c2.count();//100

//abort

throw"doh!"

}

});

db.c1.count();//0

db.c2.count();//0

Transactioninvocation

304

PassingparameterstotransactionsArbitraryparameterscanbepassedtotransactionsbysettingtheparamsattributewhendeclaringthetransaction.Thisfeatureishandytore-usethesametransactioncodeformultiplecallsbutwithdifferentparameters.

Abasicexample:

db._executeTransaction({

collections:{},

action:function(params){

returnparams[1];

},

params:[1,2,3]

});

Theaboveexamplewillreturn2.

Someexamplethatusescollections:

db._executeTransaction({

collections:{

write:"users",

read:["c1","c2"]

},

action:function(params){

vardb=require('@arangodb').db;

vardoc=db.c1.document(params['c1Key']);

db.users.save(doc);

doc=db.c2.document(params['c2Key']);

db.users.save(doc);

},

params:{

c1Key:"foo",

c2Key:"bar"

}

});

Passingparameters

305

LockingandIsolationTransactionsneedtospecifyfromwhichcollectionstheywillreaddataandwhichcollectionstheyintendtomodify.Thiscanbedonebysettingtheread,write,orexclusiveattributesinthecollectionsattributeofthetransaction:

db._executeTransaction({

collections:{

read:"users",

write:["test","log"]

},

action:function(){

constdb=require("@arangodb").db;

db.users.toArray().forEach(function(doc){

db.log.insert({value:"removeduser:"+doc.name});

db.test.remove(doc._key);

});

}

});

writeheremeanswriteaccesstothecollection,andalsoincludesanyreadaccesses.exclusiveisasynonymforwriteintheMMFilesengine,becausebothexclusiveandwritewillacquirecollection-levellocksinthisengine.IntheRocksDBengine,exclusivemeansexclusivewriteaccesstothecollection,andwritemeans(shared)writeaccesstothecollection,whichcanbeinterleavedwithwriteaccessesbyotherconcurrenttransactions.

MMFilesengine

TheMMFilesengineusesthefollowinglockingmechanismstoserializetransactionsonthesamedata:

Allcollectionsspecifiedinthecollectionsattributearelockedintherequestedmode(readorwrite)attransactionstart.Lockingofmultiplecollectionsisperformedinalphabeticalorder.Whenatransactioncommitsorrollsback,alllocksarereleasedinreverseorder.Thelockingorderisdeterministictoavoiddeadlocks.

Whilelocksareheld,modificationsbyothertransactionstothecollectionsparticipatinginthetransactionareprevented.Atransactionwillthusseeaconsistentviewoftheparticipatingcollections'data.

Additionally,atransactionwillnotbeinterruptedorinterleavedwithanyotherongoingoperationsonthesamecollection.Thismeanseachtransactionwillruninisolation.Atransactionshouldneverseeuncommittedorrolledbackmodificationsbyothertransactions.Additionally,readsinsideatransactionarerepeatable.

Notethattheaboveistrueonlyforallcollectionsthataredeclaredinthecollectionsattributeofthetransaction.

RocksDBengine

TheRocksDBenginedoesnotlockanycollectionsparticipatinginatransactionforread.Readoperationscanruninparalleltootherreadorwriteoperationsonthesamecollections.

Forallcollectionsthatareusedinwritemode,theRocksDBenginewillinternallyacquirea(shared)readlock.Thismeansthatmanywriterscanmodifydatainthesamecollectioninparallel(andalsoruninparalleltoongoingreads).However,iftwoconcurrenttransactionsattempttomodifythesamedocumentorindexentry,therewillbeawrite-writeconflict,andoneofthetransactionswillabortwitherror1200("conflict").Itisthenuptoclientapplicationstoretrythefailedtransactionoracceptthefailure.

Inordertoguardlong-runningorcomplextransactionsagainstconcurrentoperationsonthesamedata,theRocksDBengineallowstoaccesscollectionsinexclusivemode.Exclusiveaccesseswillinternallyacquireawrite-lockonthecollections,sotheyarenotexecutedinparallelwithanyotherwriteoperations.Readoperationscanstillbecarriedoutbyotherconcurrenttransactions.

Lazilyaddingcollections

Theremightbesituationswhendeclaringallcollectionsaprioriisnotpossible,forexample,becausefurthercollectionsaredeterminedbyadynamicAQLqueryinsidethetransaction,forexampleaqueryusingAQLgraphtraversal.

Lockingandisolation

306

Inthiscase,itwouldbeimpossibletoknowbeforehandwhichcollectiontolock,andthusitislegaltonotdeclarecollectionsthatwillbeaccessedinthetransactioninread-onlymode.Accessinganon-declaredcollectioninread-onlymodeduringatransactionwilladdthecollectiontothetransactionlazily,andfetchdatafromthecollectionasusual.However,asthecollectionisaddedlazily,thereisnoisolationfromotherconcurrentoperationsortransactions.Readsfromsuchcollectionsarepotentiallynon-repeatable.

Examples:

db._executeTransaction({

collections:{

read:"users"

},

action:function(){

constdb=require("@arangodb").db;

/*ExecuteanAQLquerythattraversesagraphstartingata"users"vertex.

Itisyetunknownintowhichothercollectionsthequerymighttraverse*/

db._createStatement({

query:`FORvINANY"users/1234"connectionsRETURNv`

}).execute().toArray().forEach(function(d){

/*...*/

});

}

});

Thisautomaticlazyadditionofcollectionstoatransactionalsointroducesthepossibilityofdeadlocks.Deadlocksmayoccurifthereareconcurrenttransactionsthattrytoacquirelocksonthesamecollectionslazily.

Inordertomakeatransactionfailwhenanon-declaredcollectionisusedinsideatransactionforreading,theoptionalallowImplicitsub-attributeofcollectionscanbesettofalse:

db._executeTransaction({

collections:{

read:"users",

allowImplicit:false

},

action:function(){

/*Thebelowquerywillnowfailbecausethecollection"connections"hasnot

beenspecifiedinthelistofcollectionsusedbythetransaction*/

constdb=require("@arangodb").db;

db._createStatement({

query:`FORvINANY"users/1234"connectionsRETURNv`

}).execute().toArray().forEach(function(d){

/*...*/

});

}

});

ThedefaultvalueforallowImplicitistrue.Write-accessingcollectionsthathavenotbeendeclaredinthecollectionsarrayisneverpossible,regardlessofthevalueofallowImplicit.

Ifusers/1234hasanedgeinconnections,linkingittoanotherdocumentintheuserscollection,thenthefollowingexplicitdeclarationwillwork:

db._executeTransaction({

collections:{

read:["users","connections"],

allowImplicit:false

},

/*...*/

Iftheedgepointstoadocumentinanothercollectionhowever,thenthequerywillfail,unlessthatothercollectionisaddedtothedeclarationaswell.

Notethatifadocumenthandleisusedasstartingpointforatraversal,e.g.FORvINANY"users/not_linked"...orFORvINANY{_id:"users/not_linked"}...,thennoerrorisraisedinthecaseofthestartvertexnothavinganyedgestofollow,withallowImplicit:falseandusersnotbeingdeclaredforreadaccess.AQLonlyseesastringanddoesnotconsideritareadaccess,unlessthereareedgesconnectedtoit.FORvINANYDOCUMENT("users/not_linked")...willfailevenwithoutedges,asitisalwaysconsideredtobeareadaccesstotheuserscollection.

Lockingandisolation

307

DeadlocksandDeadlockdetection

Adeadlockisasituationinwhichtwoormoreconcurrentoperations(usertransactionsorAQLqueries)trytoaccessthesameresources(collections,documents)andneedtowaitfortheotherstofinish,butnoneofthemcanmakeanyprogress.

AgoodexampleforadeadlockistwoconcurrentlyexecutingtransactionsT1andT2thattrytoaccessthesamecollectionsbutthatneedtowaitforeachother.Inthisexample,transactionT1willwritetocollectionc1,butwillalsoreaddocumentsfromcollectionc2withoutannouncingit:

db._executeTransaction({

collections:{

write:"c1"

},

action:function(){

constdb=require("@arangodb").db;

/*writeintoc1(announced)*/

db.c1.insert({foo:"bar"});

/*someoperationherethattakeslongtoexecute...*/

/*readfromc2(unannounced)*/

db.c2.toArray();

}

});

TransactionT2announcestowriteintocollectionc2,butwillalsoreaddocumentsfromcollectionc1withoutannouncingit:

db._executeTransaction({

collections:{

write:"c2"

},

action:function(){

vardb=require("@arangodb").db;

/*writeintoc2(announced)*/

db.c2.insert({bar:"baz"});

/*someoperationherethattakeslongtoexecute...*/

/*readfromc1(unannounced)*/

db.c1.toArray();

}

});

Intheaboveexample,adeadlockwilloccuriftransactionT1andT2havebothacquiredtheirwritelocks(T1forcollectionc1andT2forcollectionc2)andarethentryingtoreadfromtheotherother(T1willreadfromc2,T2willreadfromc1).T1willthentrytoacquirethereadlockoncollectionc2,whichispreventedbytransactionT2.T2howeverwillwaitforthereadlockoncollectionc1,whichispreventedbytransactionT1.

Incaseofsuchdeadlock,therewouldbenoprogressforanyoftheinvolvedtransactions,andnoneoftheinvolvedtransactionscouldevercomplete.Thisiscompletelyundesirable,sotheautomaticdeadlockdetectionmechanisminArangoDBwillautomaticallyabortoneofthetransactionsinvolvedinsuchdeadlock.Abortingmeansthatallchangesdonebythetransactionwillberolledbackanderror29(deadlockdetected)willbethrown.

Clientcode(AQLqueries,usertransactions)thataccessesmorethanonecollectionshouldbeawareofthepotentialofdeadlocksandshouldhandletheerror29(deadlockdetected)properly,eitherbypassingtheexceptiontothecallerorretryingtheoperation.

Toavoidbothdeadlocksandnon-repeatablereads,allcollectionsusedinatransactionshouldbespecifiedinthecollectionsattributewhenknowninadvance.Incasethisisnotpossiblebecausecollectionsareaddeddynamicallyinsidethetransaction,deadlocksmayoccurandthedeadlockdetectionmaykickinandabortthetransaction.

TheRocksDBengineusesdocument-levellocksandthereforewillnothaveadeadlockproblemoncollectionlevel.Iftwoconcurrenttransactionshowevermodifythesamedocumentsorindexentries,theRocksDBenginewillsignalawrite-writeconflictandabortoneofthetransactionswitherror1200("conflict")automatically.

Lockingandisolation

308

Lockingandisolation

309

DurabilityTransactionsareexecutedinmainmemoryfirstuntilthereiseitherarollbackoracommit.Onrollback,nodatawillbewrittentodisk,buttheoperationsfromthetransactionwillbereversedinmemory.

Oncommit,allmodificationsdoneinthetransactionwillbewrittentothecollectiondatafiles.ThesewriteswillbesynchronizedtodiskifanyofthemodifiedcollectionshasthewaitForSyncpropertysettotrue,orifanyindividualoperationinthetransactionwasexecutedwiththewaitForSyncattribute.Additionally,transactionsthatmodifydatainmorethanonecollectionareautomaticallysynchronizedtodisk.Thissynchronizationisdonetonotonlyensuredurability,buttoalsoensureconsistencyincaseofaservercrash.

Thatmeansifyouonlymodifydatainasinglecollection,andthatcollectionhasitswaitForSyncpropertysettofalse,thewholetransactionwillnotbesynchronizedtodiskinstantly,butwithasmalldelay.

Thereisthusthepotentialriskoflosingdatabetweenthecommitofthetransactionandtheactual(delayed)disksynchronization.ThisisthesameaswritingintocollectionsthathavethewaitForSyncpropertysettofalseoutsideofatransaction.IncaseofacrashwithwaitForSyncsettofalse,theoperationsperformedinthetransactionwilleitherbevisiblecompletelyornotatall,dependingonwhetherthedelayedsynchronizationhadkickedinornot.

ToensuredurabilityoftransactionsonacollectionthathavethewaitForSyncpropertysettofalse,youcansetthewaitForSyncattributeoftheobjectthatispassedtoexecuteTransaction.ThiswillforceasynchronizationofthetransactiontodiskevenforcollectionsthathavewaitForSyncsettofalse:

db._executeTransaction({

collections:{

write:"users"

},

waitForSync:true,

action:function(){...}

});

Analternativeistoperformanoperationwithanexplicitsyncrequestinatransaction,e.g.

db.users.save({_key:"1234"},true);

Inthiscase,thetruevaluewillmakethewholetransactionbesynchronizedtodiskatthecommit.

Inanycase,ArangoDBwillgiveusersthechoiceofwhetherornottheywantfulldurabilityforsinglecollectiontransactions.Usingthedelayedsynchronization(i.e.waitForSyncwithavalueoffalse)willpotentiallyincreasethroughputandperformanceoftransactions,butwillintroducetheriskoflosingthelastcommittedtransactionsinthecaseofacrash.

Incontrast,transactionsthatmodifydatainmorethanonecollectionareautomaticallysynchronizedtodisk.Thiscomesatthecostofseveraldisksync.Foramulti-collectiontransaction,thecalltothe_executeTransactionfunctionwillonlyreturnafterthedataofallmodifiedcollectionshasbeensynchronizedtodiskandthetransactionhasbeenmadefullydurable.Thisnotonlyreducestheriskoflosingdataincaseofacrashbutalsoensuresconsistencyafterarestart.

Incaseofaservercrash,anymulti-collectiontransactionsthatwerenotyetcommittedorinpreparationtobecommittedwillberolledbackonserverrestart.

Formulti-collectiontransactions,therewillbeatleastonedisksyncoperationpermodifiedcollection.Multi-collectiontransactionsthushaveapotentiallyhighercostthansinglecollectiontransactions.Thereisnoconfigurationtoturnoffdisksynchronizationformulti-collectiontransactionsinArangoDB.Thedisksyncspeedofthesystemwillthusbethemostimportantfactorfortheperformanceofmulti-collectiontransactions.

Durability

310

Limitations

InGeneral

TransactionsinArangoDBhavebeendesignedwithparticularusecasesinmind.Theywillbemainlyusefulforshortandsmalldataretrievaland/ormodificationoperations.

Theimplementationisnotoptimizedforverylong-runningorveryvoluminousoperations,andmaynotbeusableforthesecases.

Onelimitationisthatatransactionoperationinformationmustfitintomainmemory.Thetransactioninformationconsistsofrecordpointers,revisionnumbersandrollbackinformation.Theactualdatamodificationoperationsofatransactionarewrittentothewrite-aheadloganddonotneedtofitentirelyintomainmemory.

Ongoingtransactionswillalsopreventthewrite-aheadlogsfrombeingfullygarbage-collected.Informationinthewrite-aheadlogfilescannotbewrittentocollectiondatafilesorbediscardedwhiletransactionsareongoing.

Toensureprogressofthewrite-aheadloggarbagecollection,transactionsshouldbekeptassmallaspossible,andbigtransactionsshouldbesplitintomultiplesmallertransactions.

TransactionsinArangoDBcannotbenested,i.e.atransactionmustnotstartanothertransaction.Ifanattemptismadetocallatransactionfrominsidearunningtransaction,theserverwillthrowerror1651(nestedtransactionsdetected).

ItisalsodisallowedtoexecuteusertransactiononsomeofArangoDB'sownsystemcollections.Thisshouldn'tbeaproblemforregularusageassystemcollectionswillnotcontainuserdataandthereisnoneedtoaccessthemfromwithinausertransaction.

Someoperationsarenotallowedinsidetransactionsingeneral:

creationanddeletionofdatabases(db._createDatabase(),db._dropDatabase())creationanddeletionofcollections(db._create(),db._drop(),db.<collection>.rename())creationanddeletionofindexes(db.<collection>.ensureIndex(),db.<collection>.dropIndex())

Ifanattemptismadetocarryoutanyoftheseoperationsduringatransaction,ArangoDBwillabortthetransactionwitherrorcode1653(disallowedoperationinsidetransaction).

Finally,allcollectionsthatmaybemodifiedduringatransactionmustbedeclaredbeforehand,i.e.usingthecollectionsattributeoftheobjectpassedtothe_executeTransactionfunction.Ifanyattemptismadetocarryoutadatamodificationoperationonacollectionthatwasnotdeclaredinthecollectionsattribute,thetransactionwillbeabortedandArangoDBwillthrowerror1652unregisteredcollectionusedintransaction.Itislegaltonotdeclareread-onlycollections,butthisshouldbeavoidedifpossibletoreducetheprobabilityofdeadlocksandnon-repeatablereads.

PleaserefertoLockingandIsolationformoredetails.

InClustersUsingasingleinstanceofArangoDB,multi-document/multi-collectionqueriesareguaranteedtobefullyACID.ThisismorethanmanyotherNoSQLdatabasesystemssupport.Inclustermode,single-documentoperationsarealsofullyACID.Multi-document/multi-collectionqueriesinaclusterarenotACID,whichisequallythecasewithcompetingdatabasesystems.TransactionsinaclusterwillbesupportedinafutureversionofArangoDBandmaketheseoperationsfullyACIDaswell.Notethatfornon-shardedcollectionsinacluster,thetransactionalpropertiesofasingleserverapply(fullyACID).

TransactionsintheRocksDBstorageengineDataofongoingtransactionsisstoredinRAM.Transactionsthatgettoobig(intermsofnumberofoperationsinvolvedorthetotalsizeofdatacreatedormodifiedbythetransaction)willbecommittedautomatically.EffectivelythismeansthatbigusertransactionsaresplitintomultiplesmallerRocksDBtransactionsthatarecommittedindividually.TheentireusertransactionwillnotnecessarilyhaveACIDpropertiesinthiscase.

ThefollowingglobaloptionscanbeusedtocontroltheRAMusageandautomaticintermediatecommitsfortheRocksDBengine:

Limitations

311

--rocksdb.max-transaction-size

Transactionsizelimit(inbytes).TransactionsstoreallkeysandvaluesinRAM,solargetransactionsruntheriskofcausingout-of-memorysitations.Thissettingallowsyoutoensurethatdoesnothappenbylimitingthesizeofanyindividualtransaction.TransactionswhoseoperationswouldconsumemoreRAMthanthisthresholdvaluewillabortautomaticallywitherror32("resourcelimitexceeded").

--rocksdb.intermediate-commit-size

Ifthesizeofalloperationsinatransactionreachesthisthreshold,thetransactioniscommittedautomaticallyandanewtransactionisstarted.Thevalueisspecifiedinbytes.

--rocksdb.intermediate-commit-count

Ifthenumberofoperationsinatransactionreachesthisvalue,thetransactioniscommittedautomaticallyandanewtransactionisstarted.

Theabovevaluescanalsobeadjustedpertransaction,bysettingthefollowingattributesinthecalltodb._executeTransaction():

maxTransactionSize:transactionsizelimitinbytesintermediateCommitSize:maximumtotalsizeofoperationsafterwhichanintermediatecommitisperformedautomaticallyintermediateCommitCount:maximumnumberofoperationsafterwhichanintermediatecommitisperformedautomatically

Limitations

312

DeploymentInthischapterwedescribevariouspossibilitiestodeployArangoDB.Inparticularfortheclustermode,therearedifferentwaysandwewanttohighlighttheiradvantagesanddisadvantages.Weevendocumentindetail,howtosetupaclusterbysimplystartingvariousArangoDBprocessesondifferentmachines,eitherdirectlyorusingDockercontainers.

SingleinstanceCluster:DC/OS,ApacheMesosandMarathonCluster:Generic&DockerMultipleDatacentersAdvancedTopics

Cluster:TestsetuponalocalmachineCluster:StartingprocessesondifferentmachinesCluster:LaunchinganArangoDBclusterusingDockercontainersAgency

Deployment

313

Singleinstancedeployment

ThelatestofficialbuildsofArangoDBforallsupportedoperatingsystemsmaybeobtainedfromhttps://www.arangodb.com/download/.

Linuxremarks

Besidestheofficialimageswhichareprovidedforthemostpopularlinuxdistributionstherearealsoavarietyofunofficialimagesprovidedbythecommunity.Wearetrackingmostofthecommunitycontributions(includingneworupdatedimages)inournewsletter:

https://www.arangodb.com/category/newsletter/

Windowsremarks

PleasenotethatArangoDBwillonlyworkon64bit.

Docker

ThesimplestwaytodeployArangoDBisusingDocker.TogetageneralunderstandingofDockerhavealookattheirexcellentdocumentation.

Authentication

TostarttheofficialDockercontaineryouwillhavetodecideonanauthenticationmethod.Otherwisethecontainerwon'tstart.

ProvideoneoftheargumentstoDockerasanenvironmentvariable.

Therearethreeoptions:

1. ARANGO_NO_AUTH=1

Disableauthenticationcompletely.Usefulforlocaltestingorforoperatinginatrustednetwork(withoutapublicinterface).

2. ARANGO_ROOT_PASSWORD=password

StartArangoDBwiththegivenpasswordforroot

3. ARANGO_RANDOM_ROOT_PASSWORD=1

LetArangoDBgeneratearandomrootpassword

Togetgoingquickly:

dockerrun-eARANGO_RANDOM_ROOT_PASSWORD=1arangodb/arangodb

ForanindepthguideaboutDockerandArangoDBpleasechecktheofficialdocumentation:https://hub.docker.com/r/arangodb/arangodb/.Notethatweareusingtheimagearangodb/arangodbherewhichisalwaysthemostcurrentone.Thereisalsothe"official"onecalledarangodbwhosedocumentationishere:https://hub.docker.com/_/arangodb/

Singleinstance

314

Cluster

Cluster

315

DistributeddeploymentusingApacheMesos

ArangoDBhasasophisticatedandyeteasywaytouseclustermode.Toleveragethefullclusterfeatureset(monitoring,scaling,automaticfailoverandautomaticreplacementoffailednodes)youhavetorunArangoDBonsomekindofclustermanagementsystem.CurrentlyArangoDBreliesonApacheMesosinthatmatter.Mesosisaclusteroperatingsystemwhichpowerssomeoftheworldsbiggestdatacentersrunningseveralthousandsofnodes.

DC/OS

DC/OSistherecommendedwaytoinstallaclusterasiteasesmuchoftheprocesstoinstallaMesoscluster.YoucandeployitveryquicklyonavarietyofcloudhostersorsetupyourownDC/OSlocally.DC/OSisasetoftoolsbuiltontopofApacheMesos.ApacheMesosisasocalled"DistributedClusterOperationSystem"andthecoreofDC/OS.ApacheMesoshastheconceptofsocalledpersistentvolumeswhichmakeitperfectlysuitableforadatabase.

Installing

FirstprepareaDC/OSclusterbygoingtohttps://dcos.ioandfollowingtheinstructionsthere.

DC/OScomeswithitsownpackagemanagement.Packagescanbeinstalledfromthesocalled"Universe".AsanofficialDC/OSpartnerArangoDBcanbeinstalledfromtherestraightaway.

1. InstallingviaDC/OSUI

i. OpenyourbrowserandgototheDC/OSadmininterfaceii. Openthe"Universe"tabiii. Locatearangodbandhit"InstallPackage"iv. Press"InstallPackage"

2. InstallingviatheDC/OScommandline

i. Installthedcoscliii. Openaterminalandissuedcosinstallarangodb

Bothoptionsareessentiallydoingthesameinthebackground.BotharestartingArangoDBwithitsdefaultoptionsset.

Toreviewthedefaultoptionsusingthewebinterfacesimplyclick"AdvancedInstallation"inthewebinterface.Thereyouwillfindalistofoptionsincludingsomeexplanation.

ToreviewthedefaultoptionsusingtheCLIfirsttypedcospackagedescribe--configarangodb.Thiswillgiveyouaflatlistofdefaultsettings.

Togetanexplanationofthevariouscommandlineoptionspleasecheckthelatestoptionshere(choosethemostrecentnumberandhavealookatconfig.json):

https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/A/arangodb

AfterinstallationDC/OSwillstartdeployingtheArangoDBclusterontheDC/OScluster.YoucanwatchArangoDBstartingonthe"Services"tabinthewebinterface.OnceitislistedashealthyclickthelinknexttoitandyoushouldseetheArangoDBwebinterface.

ArangoDBMesosframework

AssoonasArangoDBwasdeployedMesoswillkeepyourclusterrunning.ThewebinterfacehasmanymonitoringfacilitiessobesuretomakeyourselffamiliarwiththeDC/OSwebinterface.AsafaulttolerantsystemMesoswilltakecareofmostfailurescenariosautomatically.MesosdoesthatbyrunningArangoDBasasocalled"framework".ThisframeworkhasbeenspecificallybuilttokeepArangoDBrunninginahealthyconditionontheMesoscluster.Fromtimetotimeataskmightfail.TheArangoDBframeworkwillthentakecareofreschedulingthefailedtask.Asitknowsabouttheveryspecificsofeachclustertaskanditsroleitwillautomaticallytakecareofmostfailurescenarios.

Toinspectwhattheframeworkisdoinggotohttp://web-interface-url/mesosinyourbrowser.Locatethetask"arangodb"andinspectstderrinthe"Sandbox".Thiscanbeofinterestforexamplewhenaslavegotlostandtheframeworkisreschedulingthetask.

Cluster:Mesos,DC/OS

316

UsingArangoDB

TouseArangoDBasadatastoreinyourDC/OSclusteryoucanfacilitatetheservicediscoveryofDC/OS.AssumingyoudeployedastandardArangoDBclusterthemesosdnswillknowaboutarangodb.mesos.BydoingaSRVDNSrequest(checkthedocumentationofmesosdns)youcanfindouttheportwheretheinternalHAProxyofArangoDBisrunning.ThiswillofferaroundrobinloadbalancertoaccessallArangoDBcoordinators.

ScalingArangoDB

TochangethesettingsofyourArangoDBClusteraccesstheArangoDBUIandhit"Nodes".Onthescaletabyouwillhavetheabilitytoscaleyourclusterupanddown.

AfterchangingthesettingstheArangoDBframeworkwilltakecareoftherest.ScalingyourclusterupisgenerallyastraightforwardoperationasMesoswillsimplylaunchanothertaskandbedonewithit.Scalingdownisabitmorecomplicatedasthedatafirsthastobemovedtosomeotherplacesothatwillnaturallytakesomewhatlonger.

Pleasenotethatscalingoperationsmightnotalwayswork.ForexampleiftheunderlyingMesosclusteriscompletelysaturatedwithitsrunningtasksscalingupwillnotbepossible.ScalingdownmightalsofailduetotheclusternotbeingabletomoveallshardsofaDBServertoanewdestinationbecauseofsizelimitations.BesuretochecktheoutputoftheArangoDBframework.

Deinstallation

DeinstallingArangoDBisabitmoredifficultasthereismuchstatebeingkeptintheMesosclusterwhichisnotautomaticallycleanedup.Todeinstallfromthecommandlineusethefollowingoneliner:

dcosarangodbuninstall;dcospackageuninstallarangodb

Thiswillfirstcleanupthestateintheclusterandthenuninstallarangodb.

arangodb-cleanup-framework

Shouldyouforgettocleanupthestateyoucandosolaterbyusingthearangodb-cleanup-frameworkcontainer.Otherwiseyoumightnotbeabletodeployanewarangodbinstallation.

ThecleanupframeworkwillannounceitselfasanormalArangoDB.MesoswillrecognizethisandofferallpersistentvolumesitstillhasforArangoDBtothisframework.Thecleanupframeworkwillthenproperlyfreethepersistentvolumes.Finallyitwillcleanupanystateleftinzookeeper(thecentralconfigurationmanagerinaMesoscluster).

Todeploythecleanupframework,followtheinstructionsinthegithubrepository.AfterdeploymentwatchtheoutputinthesandboxoftheMesoswebinterface.Afterawhilethereshouldn'tbeanypersistentresourceoffersanymoreaseverythingwascleanedup.AfterthatyoucandeletethecleanupframeworkagainviaMarathon.

ApacheMesosandMarathon

YoucanalsoinstallArangoDBonabareApacheMesosclusterprovidedthatMarathonisrunningonit.

Doingsohasthefollowingdownsides:

1. ManualMesosclustersetup2. Youneedtoimplementyourownservicediscovery3. Youaremissingthedcoscli4. Installationanddeinstallationaretedious5. YouneedtosetupsomekindofproxytunneltoaccessArangoDBfromtheoutside6. Sparsemonitoringcapabilities

HoweverthesearethingswhichdonotinfluenceArangoDBitselfandoperatingyourclusterlikethisisfullysupported.

InstallingviaMarathon

ToinstallArangoDBviamarathonyouneedaproperconfigfile:

{

Cluster:Mesos,DC/OS

317

"id":"arangodb",

"cpus":0.25,

"mem":256.0,

"ports":[0,0,0],

"instances":1,

"args":[

"framework",

"--framework_name=arangodb",

"--master=zk://172.17.0.2:2181/mesos",

"--zk=zk://172.17.0.2:2181/arangodb",

"--user=",

"--principal=pri",

"--role=arangodb",

"--mode=cluster",

"--async_replication=true",

"--minimal_resources_agent=mem(*):512;cpus(*):0.25;disk(*):512",

"--minimal_resources_dbserver=mem(*):512;cpus(*):0.25;disk(*):1024",

"--minimal_resources_secondary=mem(*):512;cpus(*):0.25;disk(*):1024",

"--minimal_resources_coordinator=mem(*):512;cpus(*):0.25;disk(*):1024",

"--nr_agents=1",

"--nr_dbservers=2",

"--nr_coordinators=2",

"--failover_timeout=86400",

"--arangodb_image=arangodb/arangodb-mesos:3.1",

"--secondaries_with_dbservers=false",

"--coordinators_with_dbservers=false"

],

"container":{

"type":"DOCKER",

"docker":{

"image":"arangodb/arangodb-mesos-framework:3.1",

"network":"HOST"

}

},

"healthChecks":[

{

"protocol":"HTTP",

"path":"/framework/v1/health.json",

"gracePeriodSeconds":3,

"intervalSeconds":10,

"portIndex":0,

"timeoutSeconds":10,

"maxConsecutiveFailures":0

}

]

}

Carefullyreviewthesettings(especiallytheIPsandtheresources).ThenyoucandeploytoMarathon:

curl-XPOST-H"Content-Type:application/json"http://url-of-marathon/v2/apps-d@arangodb3.json

AlternativelyusethewebinterfaceofMarathontodeployArangoDB.IthasaJSONmodeandyoucanusetheaboveconfigurationfile.

DeinstallationviaMarathon

AswithDC/OSyoufirstneedtoproperlycleanupanystateleftovers.

TheeasiestistosimplydeleteArangoDBandthendeploythecleanup-framework(seesectionarangodb-cleanup-framework).

Configurationoptions

TheArangodbMesosframeworkhasatonofdifferentoptionswhicharelistedanddescribedhere:https://github.com/arangodb/arangodb-mesos-framework/tree/3.1

Cluster:Mesos,DC/OS

318

AutomaticnativeClusters

SimilarlytohowtheMesosframeworkarangesanArangoDBclusterinaDC/OSenvironmentforyou,arangodbcandothisforyouinaplainenvironment.

Byinvokingthefirstarangodbyoulaunchaprimarynode.Itwillbindanetworkport,andoutputthecommandsyouneedtocut'n'pasteintotheothernodes.Let'sreviewtheprocessofsuchastartuponthreehostsnamedh01,h02,andh03:

arangodb@h01~>arangodb--ownAddressh01:4000

2017/06/1214:59:38Startingarangodbversion0.5.0+git,build5f97368

2017/06/1214:59:38ServingasmasterwithID'52698769'onh01:4000...

2017/06/1214:59:38Waitingfor3serverstoshowup.

2017/06/1214:59:38Usethefollowingcommandstostartotherservers:

arangodb--dataDir=./db2--joinh01:4000

arangodb--dataDir=./db3--joinh01:4000

2017/06/1214:59:38Listeningon0.0.0.0:4000(h01:4000)

Soyoucutthelinesarangodb--data.dir=./db2--starter.join127.0.0.1andexecutethemfortheothernodes.Ifyourunitonanothernodeonyournetwork,replacethe--starter.join127.0.0.1bythepublicIPofthefirsthost.

arangodbh02~>arangodb--dataDir=./db2--joinh01:4000

2017/06/1214:48:50Startingarangodbversion0.5.0+git,build5f97368

2017/06/1214:48:50Contactingmasterh01:4000...

2017/06/1214:48:50Waitingfor3serverstoshowup...

2017/06/1214:48:50Listeningon0.0.0.0:4000(:4000)

arangodbh03~>arangodb--dataDir=./db3--joinh01:4000

2017/06/1214:48:50Startingarangodbversion0.5.0+git,build5f97368

2017/06/1214:48:50Contactingmasterh01:4000...

2017/06/1214:48:50Waitingfor3serverstoshowup...

2017/06/1214:48:50Listeningon0.0.0.0:4000(:4000)

Oncethetwootherprocessesjoinedthecluster,andstartedtheirArangoDBserverprocesses(thismaytakeawhiledependingonyoursystem),itwillinformyouwheretoconnecttheClusterfromaBrowser,shelloryourprogramm:

...

2017/06/1214:55:21coordinatorupandrunning.

Atthispointyoumayaccessyourclusterateithercoordinatorendpoint,http://h01:4002/,http://h02:4002/orhttp://h03:4002/.

AutomaticnativelocaltestClusters

Ifyouonlywantalocaltestcluster,youcanrunasinglestarterwiththe--starter.localargument.Itwillstarta3"machine"clusteronyourlocalPC.

arangodb--starter.local

Note.AlocalclusterisintendedonlyfortestpurposessinceafailureofasinglePCwillbringdowntheentirecluster.

AutomaticDockerClusters

ArangoDBStartercanalsobeusedtolaunchclustersbasedondockercontainers.Itsabitmorecomplicated,sinceyouneedtoprovideinformationaboutyourenvironmentthatcan'tbeautodetected.

IntheDockerworldyouneedtotakecareaboutwherepersistantdataisstored,sincecontainersareintendedtobevolatile.Weuseavolumenamedarangodb1here:

Cluster:Generic&Docker

319

dockervolumecreatearangodb1

(Youcanuseanytypeofdockervolumethatfitsyoursetupinstead.)

WethenneedtodeterminethetheIPofthedockerhostwhereyouintendtorunArangoDBstarteron.Dependingonyouroperatingsystemexecuteipaddr,ifconfigoripconfigtodetermineyourlocalipaddress.

192.168.1.0/24deveth0protokernelscopelinksrc192.168.1.32

SothisexampleusestheIP192.168.1.32:

dockerrun-it--name=adb1--rm-p8528:8528\

-varangodb1:/data\

-v/var/run/docker.sock:/var/run/docker.sock\

arangodb/arangodb-starter\

--starter.address=192.168.1.32

Itwillstartthemasterinstance,andcommandyoutostarttheslaveinstances:

Unabletofindimage'arangodb/arangodb-starter:latest'locally

latest:Pullingfromarangodb/arangodb-starter

Digest:sha256:b87d20c0b4757b7daa4cb7a9f55cb130c90a09ddfd0366a91970bcf31a7fd5a4

Status:Downloadednewerimageforarangodb/arangodb-starter:latest

2017/06/1213:26:14Startingarangodbversion0.7.1,buildf128884

2017/06/1213:26:14ServingasmasterwithID'46a2b40d'on192.168.1.32:8528...

2017/06/1213:26:14Waitingfor3serverstoshowup.

2017/06/1213:26:14Usethefollowingcommandstostartotherservers:

dockervolumecreatearangodb2&&\

dockerrun-it--name=adb2--rm-p8533:8528-varangodb2:/data\

-v/var/run/docker.sock:/var/run/docker.sockarangodb/arangodb-starter:0.7\

--starter.address=192.168.1.32--starter.join=192.168.1.32

dockervolumecreatearangodb3&&\

dockerrun-it--name=adb3--rm-p8538:8528-varangodb3:/data\

-v/var/run/docker.sock:/var/run/docker.sockarangodb/arangodb-starter:0.7\

--starter.address=192.168.1.32--starter.join=192.168.1.32

Onceyoustarttheotherinstances,itwillcontinuelikethis:

2017/05/1109:05:45Addedmaster'fc673b3b':192.168.1.32,portOffset:0

2017/05/1109:05:45Addednewpeer'e98ea757':192.168.1.32,portOffset:5

2017/05/1109:05:50Addednewpeer'eb01d0ef':192.168.1.32,portOffset:10

2017/05/1109:05:51Startingservice...

2017/05/1109:05:51Lookingforarunninginstanceofagentonport8531

2017/05/1109:05:51Startingagentonport8531

2017/05/1109:05:52Lookingforarunninginstanceofdbserveronport8530

2017/05/1109:05:52Startingdbserveronport8530

2017/05/1109:05:53Lookingforarunninginstanceofcoordinatoronport8529

2017/05/1109:05:53Startingcoordinatoronport8529

2017/05/1109:05:58agentupandrunning(version3.2.devel).

2017/05/1109:06:15dbserverupandrunning(version3.2.devel).

2017/05/1109:06:31coordinatorupandrunning(version3.2.devel).

Andatleastittellsyouwhereyoucanworkwithyourcluster:

2017/05/1109:06:31Yourclustercannowbeaccessedwithabrowserat`http://192.168.1.32:8529`or

2017/05/1109:06:31using`arangosh--server.endpointtcp://192.168.1.32:8529`.

Underthehood

Thefirstarangodbyouran(asshownabove)willbecomethemasterinyoursetup,the--starter.joinwillbetheslaves.

Cluster:Generic&Docker

320

ThemasterdetermineswhichArangoDBserverprocessestolaunchonwhichslave,andhowtheyshouldcommunicate.Itwillthenlaunchtheserverprocessesandmonitorthem.Onceithasdetectedthatthesetupiscompleteyouwillgettheprompt.Themasterwillsavethesetupforsubsequentstarts.

MorecomplicatedsetupoptionscanbefoundinArangoDBStartersReadme.

Cluster:Generic&Docker

321

AdvancedTopics

Incontrasttotheothertopicsinthischapterthatstrivetogetyousimplysetupinpreparedenvironments,Thefollowingchaptersdescribewhatsgoingonunderthehoodindetails,thecomponentsofArangoDBClusters,andhowthey'reputtogether:

RunningalocaltestsetupRunningadistributedsetupRunninginDocker

AdvancedTopics

322

LaunchingArangoDB'sstandalone"agency"

MultipleArangoDBinstancescanbedeployedasafault-tolerantdistributedstatemachine.

Whatisafault-tolerantstatemachineinthefirstplace?

Inmanyservicedeploymentsconsistingofarbitrarycomponentsdistributedovermultiplemachinesoneisfacedwiththechallengeofcreatingadependablecentralisedknowledgebaseorconfiguration.Implementationofsuchaserviceturnsouttobeoneofthemostfundamentalproblemsininformationengineering.Whileitmayseemasiftherealisationofsuchaserviceiseasilyconceivable,dependablityformulatesaparadoxononcomputernetworksperse.Ontheonehand,oneneedsadistributedsystemtoavoidasinglepointoffailure.Ontheotherhand,onehastoestablishconsensusamongthecomputersinvolved.

Consensusisthekeywordhereanditsrealisationonanetworkprovestobefarfromtrivial.Manypapersandconferenceproceedingshavediscussedandevaluatedthiskeychallenge.Twoalgorithms,historicallyfarapart,havebecomewidelypopular,namelyPaxosanditsderivativesandRaft.Discussingthemandtheirdifferences,althoughhighlyenjoyable,mustremainfarbeyondthescopeofthisdocument.Findthereferencestothemainpublicationsatthebottomofthispage.

AtArangoDB,wedecidedtoimplementRaftasitisarguablytheeasiertounderstandandthusimplement.Insimpleterms,Raftguaranteesthatalinearstreamoftransactions,isreplicatedinrealtimeamongagroupofmachinesthroughanelectedleader,whointurnmusthaveaccesstoandprojectleadershipuponanoverallmajorityofparticipatinginstances.InArangoDBweliketocalltheentiretyofthecomponentsofthereplicatedtransactionlog,thatisthemachinesandtheArangoDBinstances,whichconstitutethereplicatedlog,theagency.

Startup

Theagencymustconsistsofanoddnumberofagentsinordertobeabletoestablishanoverallmajorityandsomemeansfortheagentstobeabletofindoneanotheratstartup.

Themostobviouswaywouldbetoinformallagentsoftheaddressesandportsoftherest.Thishowever,ismoreinformationthanneeded.Forexample,itwouldsuffice,ifallagentswouldknowtheaddressandportofthenextagentinacyclicfashion.Anotherstraitforwardsolutionwouldbetoinformallagentsoftheaddressandportofsaythefirstagent.

Clearlyallcases,whichwouldformdisjunctsubsetsofagentswouldbreakorintheleastimpairthefunctionalityoftheagency.Fromthereontheagentswillgossipthemissinginformationabouttheirpeers.

Typically,oneachievesfairlyhighfault-tolerancewithlow,oddnumberofagentswhilekeepingthenecessarynetworktrafficataminimum.Itseemsthatthetypicalagencysizewillbeinrangeof3to7agents.

Thebelowcommandsstartupa3-hostagencyononephysical/logicalboxwithports8529,8530and8531fordemonstrationpurposes.Theadressofthefirstinstance,port8529,isknowntotheothertwo.Afteratmost2roundsofgossipping,thelast2agentswillhaveacompletepictureoftheirsurroundingandpersistitforthenextrestart.

./build/bin/arangod--agency.activatetrue--agency.size3--agency.my-addresstcp://localhost:8529--server.authenticationfal

se--server.endpointtcp://0.0.0.0:8529agency-8529

./build/bin/arangod--agency.activatetrue--agency.size3--agency.endpointtcp://localhost:8529--agency.my-addresstcp://loc

alhost:8530--server.authenticationfalse--server.endpointtcp://0.0.0.0:8530agency-8530

./build/bin/arangod--agency.activatetrue--agency.size3--agency.endpointtcp://localhost:8529--agency.my-addresstcp://loc

alhost:8531--server.authenticationfalse--server.endpointtcp://0.0.0.0:8531agency-8531

Theparameteragency.endpointisthekeyingredientforthesecondandthirdinstancestofindthefirstinstanceandthusformacompleteagency.Pleaserefertothetheshell-scriptscripts/startStandaloneAgency.shongithuborinthesourcedirectory.

Key-value-storeAPI

Theagencyshouldbeupandrunningwithinacoupleofseconds,duringwhichtheinstanceshavegossipedtheirwayintoknowingtheotheragentsandelectedaleader.ThepublicAPIcanbecheckedforthestateoftheconfiguration:

curl-slocalhost:8529/_api/agency/config

AdvancedTopics

323

{

"term":1,

"leaderId":"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98",

"lastCommitted":1,

"lastAcked":{

"ac129027-b440-4c4f-84e9-75c042942171":0.21,

"c54dbb8a-723d-4c82-98de-8c841a14a112":0.21,

"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98":0

},

"configuration":{

"pool":{

"ac129027-b440-4c4f-84e9-75c042942171":"tcp://localhost:8531",

"c54dbb8a-723d-4c82-98de-8c841a14a112":"tcp://localhost:8530",

"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98":"tcp://localhost:8529"

},

"active":[

"ac129027-b440-4c4f-84e9-75c042942171",

"c54dbb8a-723d-4c82-98de-8c841a14a112",

"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98"

],

"id":"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98",

"agencysize":3,

"poolsize":3,

"endpoint":"tcp://localhost:8529",

"minping":0.5,

"maxping":2.5,

"supervision":false,

"supervisionfrequency":5,

"compactionstepsize":1000,

"supervisiongraceperiod":120

}

}

Tohighlightsomedetailsintheaboveoutputlookfor"term"and"leaderId".BotharekeyinformationaboutthecurrentstateoftheRaftalgorithm.Youmayhavenotedthatthefirstelectiontermhasestablishedarandomleaderfortheagency,whoisinchargeofreplicationofthestatemachineandforallexternalreadandwriterequestsuntilsuchtimethattheprocessgetsisolatedfromtheothertwosubsequenctlylosingitsleadership.

ReadandWriteAPIs

Generally,allreadandwriteaccessesaretransactionsmoreoveranyreadandwriteaccessmayconsistofmultiplesuchtransactionsformulatedasarraysofarraysinJSONdocuments.

Readtransaction

Anagencystartedfromscratchwilldealwiththesimplestqueryasfollows:

curl-Llocalhost:8529/_api/agency/read-d'[["/"]]'

[{}]

Theaboverequestforanemptykeyvaluestorewillreturnwithanemptydocument.Theinnerarraybracketswillaggregatearesultfrommultiplesourcesinthekey-value-storewhiletheouterarraywilldelivermultiplesuchaggregatedresults.Alsonotethe-Lcurlflag,whichallowstherequesttofollowredirectstothecurrentleader.

Considerthefollowingkey-value-store:

{

"baz":12,

"corge":{

"e":2.718281828459045,

"pi":3.14159265359

},

"foo":{

"bar":"HelloWorld"

},

AdvancedTopics

324

"qux":{

"quux":"HelloWorld"

}

}

Thefollowingarrayofreadtransactionswillyield:

curl-Llocalhost:8529/_api/agency/read-d'[["/foo","/foo/bar","/baz"],["/qux"]]'

[

{

"baz":12,

"foo":{

"bar":"HelloWorld"

}

},

{

"qux":{

"quux":"HelloWorld"

}

}

]

Notethattheresultisanarrayoftworesultsforthefirstandsecondreadtransactionsfromaboveaccordingly.Alsonotethattheresultsfromthefirstreadtransactionareaggregatedinto

{

"baz":12,

"foo":{

"bar":"HelloWorld"

}

}

Theaggregationisperformedon2levels:

1. /foo/bariseliminatedasasubsetof/foo2. Theresultsfrom/fooand/bararejoined

Thewordtransactionmeansherethatitisguaranteedthatallaggregationshappeninquasi-realtimeandthatnowriteaccesscouldhavehappenedinbetween.

Btw,thesametransactiononthevirginkey-valuestorewouldproduce[{},{}]

WriteAPI:

ThewriteAPImustunfortunatelybealittlemorecomplex.MultipleroadsleadtoRome:

curl-Llocalhost:8529/_api/agency/write-d'[[{"/foo":{"op":"push","new":"bar"}}]]'

curl-Llocalhost:8529/_api/agency/write-d'[[{"/foo":{"op":"push","new":"baz"}}]]'

curl-Llocalhost:8529/_api/agency/write-d'[[{"/foo":{"op":"push","new":"qux"}}]]'

and

curl-Llocalhost:8529/_api/agency/write-d'[[{"foo":["bar","baz","qux"]}]]'

areequivalentforexampleandwillcreateandfillanarrayat/foo.Here,again,theoutermostarrayisthecontainerforthetransactionarrays.

WedocumententacompleteguideoftheAPIintheAPIsection.

AdvancedTopics

325

LaunchinganArangoDBclusterfortesting

AnArangoDBclusterconsistsofseveralrunningtasks(orserverprocesses)whichformthecluster.ArangoDBitselfwon'tstartormonitoranyofthesetasks.Soitwillneedsomekindofsupervisorwhichismonitoringandstartingthesetasks.ForproductionusagewerecommendusingApacheMesosastheclustersupervisor.

HoweverstartingaclustermanuallyispossibleandisaveryeasymethodtogetafirstimpressionofwhatanArangoDBclusterlookslike.

Theeasiestwaytostartalocalclusterfortestingpurposesistorunscripts/startLocalCluster.shfromacloneofthesourcerepositoryaftercompilingArangoDBfromsource(seeinstructionsinthefileREADME_maintainers.mdintherepository.Thiswillstart1Agency,2DBServersand1Coordinator.Tostoptheclusterissuescripts/stopLocalCluster.sh.

ThissectionwilldiscusstherequiredparametersforeveryroleinanArangoDBcluster.BesuretoreadtheArchitecturedocumentationtogetabasicunderstandingoftheunderlyingarchitectureandtheinvolvedrolesinanArangoDBcluster.

Inthefollowingsectionswewillgothroughtherelevantoptionsperrole.

Agency

Tostartupanagencyyoufirsthavetoactivateit.Thisisdonebyproviding--agency.activatetrue.

Tostartuptheagencyinitsfaulttolerantmodesetthe--agency.sizeto3.Youwillthenhavetoprovideatleast3agentsbeforetheagencywillstartoperation.

Duringinitializationtheagentshavetofindeachother.Todosoprovideatleastonecommon--agency.endpoint.Theagentswillthencoordinatestartupthemselves.Theywillannouncethemselveswiththeirexternaladdresswhichmaybespecifiedusing--agency.my-address.ThisisrequiredinbridgeddockersetupsorNATedenvironments.

Soinsummarythisiswhatyourstartupmightlooklike:

AdvancedTopics

326

arangod--server.endpointtcp://0.0.0.0:5001--agency.my-address=tcp://127.0.0.1:5001--server.authenticationfalse--agency.ac

tivatetrue--agency.size3--agency.endpointtcp://127.0.0.1:5001--agency.supervisiontrue--database.directoryagency1&

arangod--server.endpointtcp://0.0.0.0:5002--agency.my-address=tcp://127.0.0.1:5002--server.authenticationfalse--agency.ac

tivatetrue--agency.size3--agency.endpointtcp://127.0.0.1:5001--agency.supervisiontrue--database.directoryagency2&

arangod--server.endpointtcp://0.0.0.0:5003--agency.my-address=tcp://127.0.0.1:5003--server.authenticationfalse--agency.ac

tivatetrue--agency.size3--agency.endpointtcp://127.0.0.1:5001--agency.supervisiontrue--database.directoryagency3&

Ifyouarehappywithasingleagent,thensimplyuseasinglecommandlikethis:

arangod--server.endpointtcp://0.0.0.0:5001--server.authenticationfalse--agency.activatetrue--agency.size1--agency.endp

ointtcp://127.0.0.1:5001--agency.supervisiontrue--database-directoryagency1&

Furthermore,inthefollowingsectionswhen--cluster.agency-addressisusedmultipletimestospecifyallthreeagentaddresses,justuseasingleoption--cluster.agency.addresstcp://127.0.0.1:5001instead.

CoordinatorsandDBServers

Thesetworolesshareacommonsetofrelevantoptions.Firstyoushouldspecifytheroleusing--cluster.my-role.ThiscaneitherbePRIMARY(adatabaseserver)orCOORDINATOR.Furthermoreprovidetheexternalendpoint(IPandport)ofthetaskvia--cluster.my-address.

Thefollowingisafull-exampleofwhatitmightlooklike:

arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8529--cluster.my-addresstcp://127.0.0.1:8529--cluster.

my-rolePRIMARY--cluster.agency-endpointtcp://127.0.0.1:5001--cluster.agency-endpointtcp://127.0.0.1:5002--cluster.agency-

endpointtcp://127.0.0.1:5003--database.directoryprimary1&

arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8530--cluster.my-addresstcp://127.0.0.1:8530--cluster.

my-rolePRIMARY--cluster.agency-endpointtcp://127.0.0.1:5001--cluster.agency-endpointtcp://127.0.0.1:5002--cluster.agency-

endpointtcp://127.0.0.1:5003--database.directoryprimary2&

arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8531--cluster.my-addresstcp://127.0.0.1:8531--cluster.

my-roleCOORDINATOR--cluster.agency-endpointtcp://127.0.0.1:5001--cluster.agency-endpointtcp://127.0.0.1:5002--cluster.age

ncy-endpointtcp://127.0.0.1:5003--database.directorycoordinator&

Noteinparticularthattheendpointdescriptionsgivenunder--cluster.my-addressand--cluster.agency-endpointmustnotusetheIPaddress0.0.0.0becausetheymustcontainanactualaddressthatcanberoutedtothecorrespondingserver.The0.0.0.0in--server.endpointsimplymeansthattheserverbindsitselftoallavailablenetworkdeviceswithallavailableIPaddresses.

UponregisteringwiththeagencyduringstartuptheclusterwillassignanIDtoeveryserver.ThegeneratedIDwillbeprintedouttothelogorcanbeaccessedviathehttpAPIbycallinghttp://server-address/_admin/server/id.

YouhavenowlaunchedacompleteArangoDBclusterandcancontactitscoordinatorattheendpointtcp://127.0.0.1:8531,whichmeansthatyoucanreachthewebUIunderhttp://127.0.0.1:8531.

AdvancedTopics

327

LaunchinganArangoDBclusteronmultiplemachines

Essentially,onecanusethemethodfromtheprevioussectiontostartanArangoDBclusteronmultiplemachinesaswell.Theonlychangesarethatonehastoreplacealllocaladdresses127.0.0.1bytheactualIPaddressofthecorrespondingserver.

IfweassumethatyouwanttostartyouArangoDBclusteronthreedifferentmachineswithIPaddresses

192.168.1.1

192.168.1.2

192.168.1.3

thenthecommandsyouhavetouseare(youcanusehostnamesiftheycanberesolvedtoIPaddressesonallmachines):

On192.168.1.1:

sudoarangod--server.endpointtcp://0.0.0.0:5001--agency.my-addresstcp://192.168.1.1:5001--server.authenticationfalse--ag

ency.activatetrue--agency.size3--agency.supervisiontrue--database.directoryagency

On192.168.1.2:

sudoarangod--server.endpointtcp://0.0.0.0:5001--agency.my-addresstcp://192.168.1.2:5001--server.authenticationfalse--ag

ency.activatetrue--agency.size3--agency.supervisiontrue--database.directoryagency

On192.168.1.3:

sudoarangod--server.endpointtcp://0.0.0.0:5001--agency.my-addresstcp://192.168.1.3:5001--server.authenticationfalse--ag

ency.activatetrue--agency.size3--agency.endpointtcp://192.168.1.1:5001--agency.endpointtcp://192.168.1.2:5001--agency.e

ndpointtcp://192.168.1.3:5001--agency.supervisiontrue--database.directoryagency

On192.168.1.1:

sudoarangod--server.authentication=false--server.endpointtcp://0.0.0.0:8529--cluster.my-addresstcp://192.168.1.1:8529--c

luster.my-rolePRIMARY--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agency-endpointtcp://192.168.1.2:5001--clus

ter.agency-endpointtcp://192.168.1.3:5001--database.directoryprimary1&

On192.168.1.2:

sudoarangod--server.authentication=false--server.endpointtcp://0.0.0.0:8530--cluster.my-addresstcp://192.168.1.2:8530--c

luster.my-rolePRIMARY--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agency-endpointtcp://192.168.1.2:5001--clus

ter.agency-endpointtcp://192.168.1.3:5001--database.directoryprimary2&

On192.168.1.3:

arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8531--cluster.my-addresstcp://192.168.1.3:8531--cluste

r.my-roleCOORDINATOR--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agency-endpointtcp://192.168.1.2:5001--clust

er.agency-endpointtcp://192.168.1.3:5001--database.directorycoordinator&

Obviously,itwouldnolongerbenecessarytousedifferentportnumbersondifferentservers.Wehavechosentokeepallportnumbersincomparisontothelocalsetuptominimizethenecessarychanges.

Afterhavingswallowedtheselongishcommands,wehopethatyouappreciatethesimplicityofthesetupwithApacheMesosandDC/OS.

AdvancedTopics

328

ArangoDBClusterandDocker

Networking

AbitofextracarehastobeinvestedduetothewayinwhichDockerisolatesitsnetwork.Bydefaultitfullyisolatesthenetworkandbydoingsoanendpointlike--server.endpointtcp://0.0.0.0:8529willonlybindtoallinterfacesinsidetheDockercontainerwhichdoesnotincludeanyexternalinterfaceonthehostmachine.ThismaybesufficientifyoujustwanttoaccessitlocallybutincaseyouwanttoexposeittotheoutsideyoumustfacilitateDockersportforwardingusingthe-pcommandlineoption.BesuretochecktheofficialDockerdocumentation.

Tosimplymakearangodbavailableonallhostinterfacesonport8529:

dockerrun-p8529:8529-eARANGO_NO_AUTH=1arangodb

AnotherpossibilityistostartDockervianetworkmodehost.Thisispossiblebutgenerallynotrecommended.TodoitanywaychecktheDockerdocumentationfordetails.

DockerandClustertasks

TostarttheclusterviaDockerisbasicallythesameasstartinglocallyoronmultiplemachines.Howeverjustlikewiththesinglenetworkingimagewewillfacenetworkingissues.Youcansimplyusethe-pflagtomaketheindividualtaskavailableonthehostmachineoryoucoulduseDocker'slinkstoenabletaskintercommunication.

PleasenotethattherearesomeflagsthatspecifyhowArangoDBcanreachataskfromtheoutside.Theseareveryimportantandbuiltforthisexactusecase.Anexampleconfigurationmightlooklikethis:

dockerrun-eARANGO_NO_AUTH=1-p192.168.1.1:10000:8529arangodb/arangodbarangod--server.endpointtcp://0.0.0.0:8529--clust

er.my-addresstcp://192.168.1.1:10000--cluster.my-rolePRIMARY--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agen

cy-endpointtcp://192.168.1.2:5002--cluster.agency-endpointtcp://192.168.1.3:5003

ThiswillstartaprimaryDBserverwithinaDockercontainerwithanisolatednetwork.WithintheDockercontaineritwillbindtoallinterfaces(thiswillbe127.0.0.1:8529andsomeinternalDockeriponport8529).Bysupplying-p192.168.1.1:10000:8529weareestablishingaportforwardingfromourlocalIP(192.168.1.1port10000inthisexample)toport8529insidethecontainer.Withinthecommandwearetellingarangodhowitcanbereachedfromtheoutside--cluster.my-addresstcp://192.168.1.1:10000.ThisinformationwillbeforwardedtotheagencysothattheothertasksinyourclustercanseehowthisparticularDBServermaybereached.

AdvancedTopics

329

DatacentertodatacenterreplicationdeploymentThischapterdescribeshowtodeployallthecomponentsneededfordatacentertodatacenterreplication.

Forageneralintroductiontodatacentertodatacenterreplication,pleaserefertotheDatacentertodatacenterreplicationchapter.

Requirementscanbefoundinthissection.

Deploymentsteps:

ClusterKafka&ZookeeperArangoSyncMasterArangoSyncWorkersPrometheus&Grafana(optional)

MultipleDatacenters

330

ArangoDBclusterThereareseveralwaystostartanArangoDBcluster.InthissectionwewillfocusonourrecommendedwaytostartArangoDB:theArangoDBStarter.

Datacentertodatacenterreplicationrequirestherocksdbstorageengine.Theexamplesetupdescribedinthissectionwillhaverocksdbenabled.Ifyouchoosetodeploywithadifferentstrategykeepinmindtosetthestorageengine.

FortheotherpossibilitiestodeployanArangoDBclusterpleaserefertothissection.

TheStartersimplifiesthingsfortheoperatorandwillcoordinateadistributedclusterstartupacrossseveralmachinesandassignclusterrolesautomatically.

Whenstartedonseveralmachinesandenoughmachineshavejoined,theStarterswillstartAgents,sCoordinatorsandDBserversonthesemachines.

WhenrunningtheStarterwillsuperviseitschildtasks(namelyCoordinators,DBserversandAgents)andrestartthemincaseoffailure.

Tostarttheclusterusingasystemdunitfileusethefollowing:

[Unit]

Description=RuntheArangoDBStarter

After=network.target

[Service]

Restart=on-failure

EnvironmentFile=/etc/arangodb.env

EnvironmentFile=/etc/arangodb.env.local

Environment=DATADIR=/var/lib/arangodb/cluster

ExecStartPre=/usr/bin/sh-c"mkdir-p${DATADIR}"

ExecStart=/usr/bin/arangodb\

--starter.address=${PRIVATEIP}\

--starter.data-dir=${DATADIR}\

--starter.join=${STARTERENDPOINTS}\

--server.storage-engine=rocksdb\

--auth.jwt-secret=${CLUSTERSECRETPATH}

TimeoutStopSec=60

[Install]

WantedBy=multi-user.target

Notethatwesetrocksdbintheunitservicefile.

Clusterauthentication

Thecommunicationbetweentheclusternodesuseatoken(JWT)toauthenticate.Thismustbesharedbetweenclusternodes.

Sharingsecretsisobviouslyaverydelicatetopic.Theaboveworkflowassumesthattheoperatorwillputasecretinafilenamed${CLUSTERSECRETPATH}.

WerecommendtouseadedicatedsystemformanagingsecretslikeHashiCorps'VaultorthesecretmanagementofDC/OS.

Requiredports

Assoonasenoughmachineshavejoined,theStarterwillbeginstartingAgents,CoordinatorsandDBservers.

Eachofthesetasksneedsaporttocommunicate.Pleasemakesurethatthefollowingportsareavailableonallmachines:

8529forCoordinators8530forDBservers8531forAgents

TheStarteritselfwilluseport8528.

Cluster

331

Recommendeddeploymentenvironment

SincetheAgentsaresocriticaltotheavailabilityofboththeArangoDBandtheArangoSynccluster,itisrecommendedtorunAgentsondedicatedmachines.Considerthesemachines"pets".

CoordinatorsandDBServerscanbedeployedofothermachinesthatshouldbeconsidered"cattle".

Cluster

332

Kafka&ZookeeperHowtodeployzookeeperHowtodeploykafkaAccessibleports

Recommendeddeploymentenvironment

SincethekafkabrokersarereallyCPUandmemoryintensive,itisrecommendedtorunzookeeper&kakfaondedicatedmachines.

Considerthesemachines"pets".

Kafka&Zookeeper

333

ArangoSyncMasterTheArangoSyncMasterisresponsibleformanagingallsynchronization,creatingtasksandassigningthosetotheArangoSyncWorkers.Atleast2instancesmutsbedeployedineachdatacenter.Oneinstancewillbethe"leader",theotherwillbeaninactiveslave.Whentheleaderisgoneforashortwhile,oneoftheotherinstanceswilltakeover.

Withclustersofasignificantsize,thesyncmasterwillrequireasignificantsetofresources.Thereforeitisrecommendedtodeploythesyncmastersontheirownservers,equipedwithsufficientCPUpowerandmemorycapacity.

TostartanArangoSyncMasterusingasystemdservice,useaunitlikethis:

[Unit]

Description=RunArangoSyncinmastermode

After=network.target

[Service]

Restart=on-failure

EnvironmentFile=/etc/arangodb.env

EnvironmentFile=/etc/arangodb.env.local

LimitNOFILE=8192

ExecStart=/usr/sbin/arangosyncrunmaster\

--log.level=debug\

--cluster.endpoint=${CLUSTERENDPOINTS}\

--cluster.jwtSecret=${CLUSTERSECRET}\

--server.keyfile=${CERTIFICATEDIR}/tls.keyfile\

--server.client-cafile=${CERTIFICATEDIR}/client-auth-ca.crt\

--server.endpoint=https://${PUBLICIP}:${MASTERPORT}\

--server.port=${MASTERPORT}\

--master.jwtSecret=${MASTERSECRET}\

--mq.type=kafka\

--mq.kafka-addr=${KAFKAENDPOINTS}\

--mq.kafka-client-keyfile=${CERTIFICATEDIR}/kafka-client.key\

--mq.kafka-cacert=${CERTIFICATEDIR}/tls-ca.crt\

TimeoutStopSec=60

[Install]

WantedBy=multi-user.target

ThesyncmasterneedsaTLSservercertificateandaIfyouwanttheservicetocreateaTLScertificate&clientauthenticationcertificate,forauthenticatingwithArangoSyncMastersinanotherdatacenter,foreverystart,addthistotheServicesection.

ExecStartPre=/usr/bin/sh-c"mkdir-p${CERTIFICATEDIR}"

ExecStartPre=/usr/sbin/arangosynccreatetlskeyfile\

--cacert=${CERTIFICATEDIR}/tls-ca.crt\

--cakey=${CERTIFICATEDIR}/tls-ca.key\

--keyfile=${CERTIFICATEDIR}/tls.keyfile\

--host=${PUBLICIP}\

--host=${PRIVATEIP}\

--host=${HOST}

ExecStartPre=/usr/sbin/arangosynccreateclient-authkeyfile\

--cacert=${CERTIFICATEDIR}/tls-ca.crt\

--cakey=${CERTIFICATEDIR}/tls-ca.key\

--keyfile=${CERTIFICATEDIR}/kafka-client.key\

--host=${PUBLICIP}\

--host=${PRIVATEIP}\

--host=${HOST}

TheArangoSyncMastermustbereachableonaTCPport${MASTERPORT}(usedwith--server.portoption).Thisportmustbereachablefrominsidethedatacenter(bysyncworkersandoperations)andfrominsideoftheotherdatacenter(bysyncmastersintheotherdatacenter).

Recommendeddeploymentenvironment

ArangoSyncMaster

334

SincethesyncmasterscanbeCPUintensivewhenrunninglotsofdatabases&collections,itisrecommendedtorunthemondedicatedmachineswithalotofCPUpower.

Considerthesemachines"pets".

ArangoSyncMaster

335

ArangoSyncWorkersTheArangoSyncWorkerisresponsibleforexecutingsynchronizationtasks.Foroptimalperformanceatleast1workerinstancemustbeplacedoneverymachinethathasanArangoDBDBserverrunning.Thisensuresthattaskscanbeexecutedwithminimalnetworktrafficoutsideofthemachine.

SincesyncworkerswillautomaticallystoponcetheirTLSservercertificateexpires(whichissetto2yearsbydefault),itisrecommendedtorunatleast2instancesofaworkeroneverymachineinthedatacenter.Thatway,taskscanstillbeassignedinthemostoptimalway,evenwhenaworkerintemporarilydownforarestart.

TostartanArangoSyncWorkerusingasystemdservice,useaunitlikethis:

[Unit]

Description=RunArangoSyncinworkermode

After=network.target

[Service]

Restart=on-failure

EnvironmentFile=/etc/arangodb.env

EnvironmentFile=/etc/arangodb.env.local

Environment=PORT=8729

LimitNOFILE=1000000

ExecStart=/usr/sbin/arangosyncrunworker\

--log.level=debug\

--server.port=${PORT}\

--server.endpoint=https://${PRIVATEIP}:${PORT}\

--master.endpoint=${MASTERENDPOINTS}\

--master.jwtSecret=${MASTERSECRET}

TimeoutStopSec=60

[Install]

WantedBy=multi-user.target

TheArangoSyncWorkermustbereachableonaTCPport${PORT}(usedwith--server.portoption).Thisportmustbereachablefrominsidethedatacenter(bysyncmasters).

Notethelargefiledescriptorlimit.Thesyncworkerrequiresabout30filedescriptorspershard.Ifyouusehardwarewithhugeresources,andstillrunoutoffiledescriptors,youcandecidetorunmultiplesyncworkersoneachmachineinordertospreadthetasksacrossthem.

RecommendeddeploymentenvironmentThesyncworkersshouldberunonallmachinesthatalsocontainanArangoDBDBServer.Thesyncworkercanbememoryintensivewhenrunninglotsofdatabases&collections.

Considerthesemachines"cattle".

ArangoSyncWorkers

336

Prometheus&Grafana(optional)ArangoSyncprovidesmetricsinaformatsupportedbyPrometheus.WealsoprovideastandardsetofdashboardsforviewingthosemetricsinGrafana.

Ifyouwanttousethesetools,pleaserefertotheirwebsitesforinstructionsonhowtodeploythem.

Afterdeployment,youmustconfigurePrometheususingaconfigurationfilethatinstructsitaboutwhichtargetstoscrape.ForArangoSyncyoushouldconfigurescrapetargetsforallsyncmastersandallsyncworkers.Todoso,youcanuseaconfigurationsuchasthis:

global:

scrape_interval:10s#scrapetargetsevery10seconds.

scrape_configs:

#Scrapsyncmasters

-job_name:'sync_master'

scheme:'https'

bearer_token:"${MONITORINGTOKEN}"

tls_config:

insecure_skip_verify:true

static_configs:

-targets:

-"${IPMASTERA1}:8629"

-"${IPMASTERA2}:8629"

-"${IPMASTERB1}:8629"

-"${IPMASTERB2}:8629"

labels:

type:"master"

relabel_configs:

-source_labels:[__address__]

regex:${IPMASTERA1}\:8629|${IPMASTERA2}\:8629

target_label:dc

replacement:A

-source_labels:[__address__]

regex:${IPMASTERB1}\:8629|${IPMASTERB2}\:8629

target_label:dc

replacement:B

-source_labels:[__address__]

regex:${IPMASTERA1}\:8629|${IPMASTERB1}\:8629

target_label:instance

replacement:1

-source_labels:[__address__]

regex:${IPMASTERA2}\:8629|${IPMASTERB2}\:8629

target_label:instance

replacement:2

#Scrapsyncworkers

-job_name:'sync_worker'

scheme:'https'

bearer_token:"${MONITORINGTOKEN}"

tls_config:

insecure_skip_verify:true

static_configs:

-targets:

-"${IPWORKERA1}:8729"

-"${IPWORKERA2}:8729"

-"${IPWORKERB1}:8729"

-"${IPWORKERB2}:8729"

labels:

type:"worker"

relabel_configs:

-source_labels:[__address__]

regex:${IPWORKERA1}\:8729|${IPWORKERA2}\:8729

target_label:dc

replacement:A

-source_labels:[__address__]

regex:${IPWORKERB1}\:8729|${IPWORKERB2}\:8729

target_label:dc

replacement:B

-source_labels:[__address__]

Prometheus&Grafana

337

regex:${IPWORKERA1}\:8729|${IPWORKERB1}\:8729

target_label:instance

replacement:1

-source_labels:[__address__]

regex:${IPWORKERA2}\:8729|${IPWORKERB2}\:8729

target_label:instance

replacement:2

Note:Theaboveexampleassumes2datacenters,with2syncmasters&2syncworkersperdatacenter.Youhavetoreplaceall${...}variablesintheaboveconfigurationwithapplicablevaluesfromyourenvironment.

Recommendeddeploymentenvironment

Prometheuscanbeamemory&CPUintensiveprocess.ItisrecommendedtokeepthemonothermachinesthanusedtoruntheArangoDBclusterorArangoSynccomponents.

Considerthesemachines"cattle",unlessyouconfigurealertingonprometheus,inwhichcaseitisrecommendedtoconsiderthesemachines"pets".

Prometheus&Grafana

338

AdministrationMostadministrationcanbemanagedusingthearangosh.

FilesystemsAsonewouldexpectforadatabase,werecommendalocallymountedfilesystems.

NFSorsimilarnetworkfilesystemswillnotwork.

OnLinuxwerecommendtheuseofext4fs,onWindowsNTFSandonMacOSHFS+.

WerecommendtonotuseBTRFSonLinux.ItisknowntonotworkwellinconjunctionwithArangoDB.WeexperiencedthatArangoDBfaceslatencyissuesonaccessingitsdatabasefilesonBTRFSpartitions.InconjunctionwithBTRFSandAUFSwealsosawdatalossonrestart.

Administration

339

WebInterfaceArangoDBcomeswithabuilt-inwebinterfaceforadministration.Theinterfacediffersforstandaloneinstancesandclustersetups.

Standalone:

Cluster:

WebInterface

340

WebInterface

341

DashboardTheDashboardtabprovidesstatisticswhicharepolledregularlyfromtheArangoDBserver.

RequestsStatistics:

RequestspersecondRequesttypesNumberofclientconnectionsTransfersizeTransfersize(distribution)AveragerequesttimeAveragerequesttime(distribution)

SystemResources:

NumberofthreadsMemoryVirtualsizeMajorpagefaultsUsedCPUtime

Replication:

ReplicationstateTotalsTicksProgress

Dashboard

342

ClusterTheclustersectiondisplaysstatisticsaboutthegeneralclusterperformance.

Statistics:

AvailableandmissingcoordinatorsAvailableandmissingdatabaseserversMemoryusage(percent)CurrentconnectionsData(bytes)HTTP(bytes)Averagerequesttime(seconds)

Nodes

Overview

Theoverviewshowsavailableandmissingcoordinatorsanddatabaseservers.

Cluster

343

Functions:

CoordinatorDashboard:ClickonaCoordinatorwillopenastatisticsdashboard.

Information(Coordinator/Databaseservers):

NameEndpointLastHeartbeatStatusHealth

Shards

Theshardsectiondisplaysallavailableshardedcollections.

Cluster

344

Functions:

MoveShardLeader:Clickonaleaderdatabaseofashardserverwillopenamovesharddialog.Shardscanbetransferredtoallavailabledatabasservers,excepttheleadingdatabaseserveroranavailablefollower.MoveShardFollower:Clickonafollowerdatabaseofashardwillopenamovesharddialog.Shardscanbetransferredtoallavailabledatabasservers,excepttheleadingdatabaseserveroranavailablefollower.RebalanceShards:Anewdatabaseserverwillnothaveanyshards.Withtherebalancefunctionalitytheclusterwillstarttorebalanceshardsincludingemptydatabaseservers.

Information(collection):

ShardLeader(greenstate:synciscomplete)Followers

Cluster

345

CollectionsThecollectionssectiondisplaysallavailablecollections.Fromhereyoucancreatenewcollectionsandjumpintoacollectionfordetails(clickonacollectiontile).

Functions:

A:TogglefilterpropertiesB:SearchcollectionbynameD:CreatecollectionC:FilterpropertiesH:Showcollectiondetails(clicktile)

Information:

E:CollectiontypeF:Collectionstate(unloaded,loaded,...)G:Collectionname

Collection

Collections

346

Therearefourviewcategories:

1. Content:

CreateadocumentDeleteadocumentFilterdocumentsDownloaddocumentsUploaddocuments

2. Indices:

CreateindicesDeleteindices

3. Info:

Detailedcollectioninformationandstatistics4. Settings:

Configurename,journalsize,indexbuckets,waitforsyncDeletecollectionTruncatecollectionUnload/LoadcollectionSavemodifedproperties(name,journalsize,indexbuckets,waitforsync)

Additionalinformation:

Uploadformat:

I.Line-wise

{"_key":"key1",...}

{"_key":"key2",...}

II.JSONdocumentsinalist

[

{"_key":"key1",...},

{"_key":"key2",...}

Collections

347

]

Collections

348

DocumentThedocumentsectionoffersaeditorwhichletyoueditdocumentsandedgesofacollection.

Functions:

EditdocumentSavedocumentDeletedocmentSwitchbetweenTree/Code-ModeCreateanewdocument

Information:

Displays:_id,_rev,_keyproperties

Document

349

QueryViewThequeryviewoffersyouthreedifferentsubviews:

EditorRunningQueriesSlowQueryHistory

AQLQueryEditor

ThewebinterfaceoffersaAQLQueryEditor:

Theeditorissplitintotwoparts,thequeryeditorpaneandthebindparameterpane.

Theleftpaneisyourregularqueryinputfield,whereyoucaneditandthenexecuteorexplainyourqueries.Bydefault,theenteredbindparameterwillautomaticallyberecognizedandshowninthebindparametertableintherightpane,whereyoucaneasilyeditthem.

Theinputfieldsareequippedwithtypedetection.Thismeansyoudon'thavetousequotemarksaroundstring,justwritethemas-is.Numberswillbetreatedasnumbers,trueandfalseasbooleans,nullasnull-typevalue.Squarebracketscanbeusedtodefinearrays,andcurlybracesforobjects(keysandvalueshavetobesurroundedbydoublequotes).Thiswillmostlybewhatyouwant.Butifyouwanttoforcesomethingtobetreatedasstring,usequotationmarksforthevalue:

123//interpretedasnumber

"123"//interpretedasstring

["foo","bar",123,true]//interpretedasarray

['foo','bar',123,true]//interpretedasstring

Queries

350

IfyouareusedtoworkwithJSON,youmaywanttoswitchthebindparametereditortoJSONmodebyclickingontheupperrighttogglebutton.YoucantheneditthebindparametersinrawJSONformat.

CustomQueries

TosavethecurrentqueryusetheSavebuttoninthetopleftcorneroftheeditororusetheshortcut(seebelow).

BypressingtheQueriesbuttoninthetopleftcorneroftheeditoryouactivatethecustomqueriesview.Hereyoucanselectapreviouslystoredcustomqueryoroneofourqueryexamples.

Clickonaquerytitletogetacodepreview.Inaddition,thereareactionbuttonsto:

CopytoeditorExplainqueryRunqueryDeletequery

Forthebuilt-inexamplequeries,thereisonlyCopytoeditoravailable.

ToexportorimportqueriestoandfromJSONyoucanusethebuttonsontheright-handside.

Result

Queries

351

Eachqueryyouexecuteorexplainopensupanewresultbox,soyouareabletofireupmultiplequeriesandviewtheirresultsatthesametime.Everyqueryresultboxgivesyoudetailedqueryinformationandofcoursethequeryresultitself.Theresultboxescanbedismissedindividually,oraltogetherusingtheRemoveresultsbutton.ThetogglebuttoninthetoprightcornerofeachboxswitchesbackandforthbetweentheResultandAQLquerywithbindparameters.

Spotlight

Thespotlightfeatureopensupamodalview.ThereyoucanfindallAQLkeywords,AQLfunctionsandcollections(filteredbytheirtype)tohelpyoutobemoreproductiveinwritingyourqueries.Spotlightcanbeopenedbythemagicwandiconinthetoolbarorviashortcut(seebelow).

AQLEditorShortcuts

Queries

352

Ctrl/Cmd+ReturntoexecuteaqueryCtrl/Cmd+Shift+ReturntoexplainaqueryCtrl/Cmd+Shift+StosavethecurrentqueryCtrl/Cmd+Shift+CtotogglecommentsCtrl+SpacetoopenupthespotlightsearchCtrl+Cmd+ZtoundolastchangeCtrl+Cmd+Shift+Ztoredolastchange

RunningQueries

TheRunningQueriestabgivesyouacompactoverviewofallrunningqueries.Byclickingtheredminusbutton,youcanaborttheexecutionofarunningquery.

SlowQueryHistory

Queries

353

TheSlowQueryHistorytabgivesyouacompactoverviewofallpastslowqueries.

Queries

354

GraphsTheGraphstabprovidesaviewerfacilityforgraphdatastoredinArangoDB.ItallowsbrowsingArangoDBgraphsstoredinthe_graphssystemcollectionoragraphconsistingofanarbitraryvertexandedgecollection.

Pleasenotethatthegraphviewerrequirescanvas(optional:webgl)supportinyourbrowser.EspeciallyInternetExplorerbrowsersolderthanversion9arelikelytonotsupportthis.

GraphViewer

Graphs

355

TopToolbarFunctions:

Loadfullgraph(Alsonodeswithoutconnectionswillbedrawn.Usefulduringgraphmodelingsetup)TakeagraphscreenshotStartfullscreenmodeOpengraphoptionsmenu

DefaultContextMenu(mouse-clickbackground):

AddanewnodeClosevisiblecontextmenu(s)

NodeContextMenu(mouse-clicknode):

DeletenodeEditnodeExpandnode(Showallboundedges)Drawedge(Connectwithanothernode)Setasstartnode(TheGraphwillrerenderstartingtheselectednodeandgivenoptions(graphoptionsmenu))

EdgeContextMenu(mouse-clickedge):

EditedgeDeleteedge

EdgeHighlighting(right-mouse-clicknode):

Highlightalledgesconnectedtothenode(right-clickatthebackgroundwillremovehighlighting)

Graphs

356

GraphViewerOptionsGraphOptionsMenu:

Startnode(string-validnodeidorspaceseperatedlistofid's):Heartofyourgraph.Renderingandtraversingwillstartfromhere.Emptyvaluemeans:arandomstartingpointwillbeused.Layout:Differentgraphlayoutingalgoritms.Nooverlap(optimal:biggraph),forcelayout(optimal:mediumgraph),fruchtermann(optimal:littletomediumgraph).Renderer:Canvasmodeallowsediting.WebGLcurrentlyoffersonlydisplaymode(alotfasterwithmuchnodes/edges).Searchdepth(number):Searchdepthwhichisstartingfromyourstartnode.Limit(number):Limitnodescount.Ifemptyorzero,nolimitisset.

NodesOptionsMenu:

Label(string):Nodeswillbelabeledbythisattribute.Ifnodeattributeisnotfound,nolabelwillbedisplayed.AddCollectionName:Thisappendsthecollectionnametothelabel,ifitexists.ColorByCollections:Shouldnodesbecolorizedbytheircollection?Ifenabled,nodecolorandnodecolorattributewillbeignored.Color:Defaultnodecolor.ColorAttribute(string):Ifanattributeisgiven,nodeswillthenbecolorizedbytheattribute.Thissettingignoresdefaultnodecolorifset.SizeByConnections:Shouldnodesbesizedbytheiredgescount?Ifenabled,nodesizingattributewillbeignored.SizingAttribute(number):Defaultnodesize.Numericvalue>0.

EdgesOptionsMenu:

Label(string):Edgeswillbelabeledbythisattribute.Ifedgeattributeisnotfound,nolabelwillbedisplayed.AddCollectionName:Thisappendsthecollectionnametothelabel,ifitexists.ColorByCollections:Shouldedgesbecolorizedbytheircollection?Ifenabled,edgecolorandedgecolorattributewillbeignored.Color:Defaultedgecolor.ColorAttribute(string):Ifanattributeisgiven,edgeswillthenbecolorizedbytheattribute.Thissettingignoresdefaultnodecolorifset.Type:Therendereroffersmultipletypesofrendering.Theyonlydifferintheirdisplaystyle,exceptforthetype'curved'.Thecurvedtypeallowstodisplaymorethanoneedgesbetweentwonodes.

Graphs

357

Graphs

358

ServicesTheservicessectiondisplaysallinstalledfoxxapplications.Youcancreatenewservicesorgointoadetailedviewofachoosenservice.

CreateService

Therearefourdifferentpossibilitiestocreateanewservice:

1. Createserviceviazipfile2. Createserviceviagithubrepository3. CreateserviceviaofficialArangoDBstore4. Createablankservicefromscratch

Services

359

ServiceView

Thissectionoffersseveralinformationaboutaspecificservice.

Therearefourviewcategories:

1. Info:

Displaysname,shortdescription,license,version,mode(production,development)Offersabuttontogototheservicesinterface(ifavailable)

Services

360

2. Api:

DisplayAPIasSwaggerUIDisplayAPIasRAWJSON

3. Readme:

Displaystheservicesmanual(ifavailable)4. Settings:

DownloadserviceaszipfileRunservicetests(ifavailable)Runservicescripts(ifavailable)Configuredependencies(ifavailable)Changeserviceparameters(ifavailable)Changemode(production,development)ReplacetheserviceDeletetheservice

Services

361

ManagingUsersintheWebInterfaceArangoDBusersaregloballystoredinthe_systemdatabaseandcanonlybemanangedwhileloggedontothisdatabase.ThereyoucanfindtheUserssection:

General

SelectausertobringuptheGeneraltabwiththeusername,nameandactivestatus,aswellasoptionstodeletetheuserorchangethepassword.

Users

362

Permissions

SelectauserandgotothePermissionstab.Youwillseealistofdatabasesandtheircorrespondingdatabaseaccesslevelforthatuser.

Users

363

Pleasenotethatserveraccesslevelfollowsfromtheaccesslevelonthedatabase_system.Furthermore,thedefaultdatabaseaccesslevelforthisuserappearintheartificialrowwiththedatabasename*.

Belowthistableisanotheroneforthecollectioncategoryaccesslevels.Atfirst,itshowsthelistofdatabases,too.Ifyouclickonadatabase,thelistofcollectionsinthatdatabasewillbeopenandyoucanseethedefinedcollectionaccesslevelsforeachcollectionofthatdatabase(whichcanbeallunselectedwhichmeansthatnothingisexplicitlyset).Thedefaultaccesslevelsforthisuseranddatabaseappearintheartificialrowwiththecollectionname*.

AlsoseeManagingUsersaboutaccesslevels.

Users

364

LogsThelogssectiondisplaysallavailablelogentries.Logentriesarefilterablebytheirlogleveltypes.

Functions:

Filterlogentriesbyloglevel(all,info,error,warning,debug)

Information:

LoglevelDateMessage

Logs

365

ArangoDBShellIntroductionTheArangoDBshell(arangosh)isacommand-linetoolthatcanbeusedforadministrationofArangoDB,includingrunningad-hocqueries.

ThearangoshbinaryisshippedwithArangoDB.ItoffersaJavaScriptshellenvironmentprovidingaccesstotheArangoDBserver.Arangoshcanbeinvokedlikethis:

unix>arangosh

BydefaultarangoshwilltrytoconnecttoanArangoDBserverrunningonserverlocalhostonport8529.Itwillusetheusernamerootandanemptypasswordbydefault.Additionallyitwillconnecttothedefaultdatabase(_system).Allthesedefaultscanbechangedusingthefollowingcommand-lineoptions:

--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:databaseusername--server.password:passwordtousewhenconnecting--server.authentication:whetherornottouseauthentication

Forexample,toconnecttoanArangoDBserveronIP192.168.173.13onport8530withtheuserfooandusingthedatabasetest,use:

unix>arangosh\

--server.endpointtcp://192.168.173.13:8530\

--server.usernamefoo\

--server.databasetest\

--server.authenticationtrue

arangoshwillthendisplayapasswordpromptandtrytoconnecttotheserverafterthepasswordwasentered.

Tochangethecurrentdatabaseaftertheconnectionhasbeenmade,youcanusethedb._useDatabase()commandinarangosh:

arangosh>db._createDatabase("myapp");

true

arangosh>db._useDatabase("myapp");

true

arangosh>db._useDatabase("_system");

true

arangosh>db._dropDatabase("myapp");

true

Togetalistofavailablecommands,arangoshprovidesahelp()function.Callingitwilldisplayhelpfulinformation.

arangoshalsoprovidesauto-completion.Additionalinformationonavailablecommandsandmethodsisthusprovidedbytypingthefirstfewlettersofavariableandthenpressingthetabkey.Itisrecommendtotrythiswithenteringdb.(withoutpressingreturn)andthenpressingtab.

Bytheway,arangoshprovidesthedbobjectbydefault,andthisobjectcanbeusedforswitchingtoadifferentdatabaseandmanagingcollectionsinsidethecurrentdatabase.

Foralistofavailablemethodsforthedbobject,type

arangosh>db._help();

showexecutionresultsyoucanpastemultiplelinesintoarangosh,giventhefirstlineendswithanopeningbrace:

arangosh>for(vari=0;i<10;i++){

ArangoDBShell

366

........>require("@arangodb").print("Helloworld"+i+"!\n");

........>}

showexecutionresultsToloadyourownJavaScriptcodeintothecurrentJavaScriptinterpretercontext,usetheloadcommand:

require("internal").load("/tmp/test.js")//<-Linux/MacOS

require("internal").load("c:\\tmp\\test.js")//<-Windows

Exitingarangoshcanbedoneusingthekeycombination<CTRL>+Dorbytypingquit<CR>

Escaping

InAQL,escapingisdonetraditionallywiththebackslashcharacter:\.Asseenabove,thisleadstodoublebackslasheswhenspecifyingWindowspaths.Arangoshrequiresanotherlevelofescaping,alsowiththebackslashcharacter.ItaddsuptofourbackslashesthatneedtobewritteninArangoshforasingleliteralbackslash(c:\tmp\test.js):

db._query('RETURN"c:\\\\tmp\\\\test.js"')

Youcanusebindvariablestomitigatethis:

varsomepath="c:\\tmp\\test.js"

db._query(aql`RETURN${somepath}`)

ArangoDBShell

367

ArangoDBShellOutputTheArangoDBshellwillprinttheoutputofthelastevaluatedexpressionbydefault:

arangosh>42*23

966

Inordertopreventprintingtheresultofthelastevaluatedexpression,theexpressionresultcanbecapturedinavariable,e.g.

arangosh>varcalculationResult=42*23

ThereisalsotheprintfunctiontoexplicitlyprintoutvaluesintheArangoDBshell:

arangosh>print({a:"123",b:[1,2,3],c:"test"});

showexecutionresultsBydefault,theArangoDBshellusesaprettyprinterwhenJSONdocumentsareprinted.Thisensuresdocumentsareprintedinahuman-readableway:

arangosh>db._create("five")

arangosh>for(i=0;i<5;i++)db.five.save({value:i})

arangosh>db.five.toArray()

showexecutionresultsWhilethepretty-printerproducesnicelookingresults,itwillneedalotofscreenspaceforeachdocument.Sometimesamoredenseoutputmightbebetter.Inthiscase,theprettyprintercanbeturnedoffusingthecommandstop_pretty_print().

Toturnonprettyprintingagain,usethestart_pretty_print()command.

ShellOutput

368

ArangoDBShellConfigurationarangoshwilllookforauser-definedstartupscriptnamed.arangosh.rcintheuser'shomedirectoryonstartup.Thehomedirectorywilllikelybe/home/<username>/onUnix/Linux,andisdeterminedonWindowsbypeekingintotheenvironmentvariables%HOMEDRIVE%and%HOMEPATH%.

Ifthefile.arangosh.rcispresentinthehomedirectory,arangoshwillexecutethecontentsofthisfileinsidetheglobalscope.

Youcanusethistodefineyourownextravariablesandfunctionsthatyouneedoften.Forexample,youcouldputthefollowingintothe.arangosh.rcfileinyourhomedirectory:

//"var"keywordavoidedintentionally...

//otherwise"timed"wouldnotsurvivethescopeofthisscript

global.timed=function(cb){

console.time("callback");

cb();

console.timeEnd("callback");

};

Thiswillmakeafunctionnamedtimedavailableinarangoshintheglobalscope.

Youcannowstartarangoshandinvokethefunctionlikethis:

timed(function(){

for(vari=0;i<1000;++i){

db.test.save({value:i});

}

});

Pleasekeepinmindthat,ifpresent,the.arangosh.rcfileneedstocontainvalidJavaScriptcode.Ifyouwantanyvariablesintheglobalscopetosurviveyouneedtoomitthevarkeywordforthem.Otherwisethevariableswillonlybevisibleinsidethescriptitself,butnotoutside.

Configuration

369

DetailsabouttheArangoDBShellAftertheserverhasbeenstarted,youcanusetheArangoDBshell(arangosh)toadministratetheserver.Withoutanyarguments,theArangoDBshellwilltrytocontacttheserveronport8529onthelocalhost.FormoreinformationseetheArangoDBShelldocumentation.Youmightneedtosetadditionaloptions(endpoint,usernameandpassword)whenconnecting:

unix>./arangosh--server.endpointtcp://127.0.0.1:8529--server.usernameroot

TheshellwillprintitsownversionnumberandifsuccessfullyconnectedtoaservertheversionnumberoftheArangoDBserver.

Command-LineOptionsUse--helptogetalistofcommand-lineoptions:

unix>./arangosh--help

STANDARDoptions:

--audit-log<string>auditlogfiletosavecommandsandresultsto

--configuration<string>readconfigurationfile

--helphelpmessage

--max-upload-size<uint64>maximumsizeofimportchunks(inbytes)(default:500000)

--no-auto-completedisableautocompletion

--no-colorsdeactivatecolorsupport

--pager<string>outputpager(default:"less-X-R-F-L")

--pretty-printprettyprintvalues

--quietnobanner

--temp.path<string>pathfortemporaryfiles(default:"/tmp/arangodb")

--use-pagerusepager

JAVASCRIPToptions:

--javascript.check<string>syntaxcheckcodeJavaScriptcodefromfile

--javascript.execute<string>executeJavaScriptcodefromfile

--javascript.execute-string<string>executeJavaScriptcodefromstring

--javascript.startup-directory<string>startuppathscontainingtheJavaScriptfiles

--javascript.unit-tests<string>donotstartasshell,rununittestsinstead

--jslint<string>donotstartasshell,runjslintinstead

LOGGINGoptions:

--log.level<string>loglevel(default:"info")

CLIENToptions:

--server.connect-timeout<double>connecttimeoutinseconds(default:3)

--server.authentication<bool>whetherornottouseauthentication(default:true)

--server.endpoint<string>endpointtoconnectto,use'none'tostartwithoutaserver(default:"tcp://127.0

.0.1:8529")

--server.password<string>passwordtousewhenconnecting(leaveemptyforprompt)

--server.request-timeout<double>requesttimeoutinseconds(default:300)

--server.username<string>usernametousewhenconnecting(default:"root")

DatabaseWrappersThedbobjectisavailableinarangoshaswellasonarangodi.e.ifyou'reusingFoxx.Whileitsinterfaceispersistantbetweenthearangoshandthearangodimplementations,itsunderpinningisnot.ThearangodimplementationareJavaScriptwrappersaroundArangoDB'snativeC++implementation,whereasthearangoshimplementationwrapsHTTPaccessestoArangoDB'sRESTfullAPI.

Sowhilethiscodemayproducesimilarresultswhenexecutedinarangoshandarangod,thecpuusageandtimerequiredwillbereallydifferent:

for(i=0;i<100000;i++){

db.test.save({name:{first:"Jan"},count:i});

}

Sincethearangoshversionwillbedoingaround100kHTTPrequests,andthearangodversionwilldirectlywritetothedatabase.

Details

370

Using arangoshviaunixshebangmechanisms

Inunixoperatingsystemsyoucanstartscriptsbyspecifyingtheinterpreterinthefirstlineofthescript.Thisiscommonlycalledshebangorhashbang.Youcanalsodothatwitharangosh,i.e.create~/test.js:

#!/usr/bin/arangosh--javascript.execute

require("internal").print("helloworld")

db._query("FORxINtestRETURNx").toArray()

Notethatthefirstlinehastoendwithablankinordertomakeitwork.MarkitexecutabletotheOS:

#>chmoda+x~/test.js

andfinalytryitout:

#>~/test.js

Details

371

ArangoimportThismanualdescribestheArangoDBimporterarangoimport,whichcanbeusedforbulkimports.

ThemostconvenientmethodtoimportalotofdataintoArangoDBistousethearangoimportcommand-linetool.Itallowsyoutoimportdatarecordsfromafileintoanexistingdatabasecollection.

Itispossibletoimportdocumentkeyswiththedocumentsusingthe_keyattribute.Whenimportingintoanedgecollection,itismandatorythatallimporteddocumentshavethe_fromand_toattributes,andthattheycontainvalidreferences.

Let'sassumeforthefollowingexamplesyouwanttoimportuserdataintoanexistingcollectionnamed"users"ontheserver.

ImportingDataintoanArangoDBDatabase

ImportingJSON-encodedData

Let'sfurtherassumetheimportathandisencodedinJSON.We'llbeusingtheseexampleuserrecordstoimport:

{"name":{"first":"John","last":"Connor"},"active":true,"age":25,"likes":["swimming"]}

{"name":{"first":"Jim","last":"O'Brady"},"age":19,"likes":["hiking","singing"]}

{"name":{"first":"Lisa","last":"Jones"},"dob":"1981-04-09","likes":["running"]}

Toimporttheserecords,allyouneedtodoistoputthemintoafile(withonelineforeachrecordtoimport)andrunthefollowingcommand:

>arangoimport--file"data.json"--typejsonl--collection"users"

Thiswilltransferthedatatotheserver,importtherecords,andprintastatussummary.Toshowtheintermediateprogressduringtheimportprocess,theoption--progresscanbeadded.Thisoptionwillshowthepercentageoftheinputfilethathasbeensenttotheserver.Thiswillonlybeusefulforbigimportfiles.

>arangoimport--file"data.json"--typejson--collectionusers--progresstrue

Itisalsopossibletousetheoutputofanothercommandasaninputforarangoimport.Forexample,thefollowingshellcommandcanbeusedtopipedatafromthecatprocesstoarangoimport:

>catdata.json|arangoimport--file---typejson--collectionusers

Notethatyouhavetouse--file-ifyouwanttouseanothercommandasinputforarangoimport.Noprogresscanbereportedforsuchimportsasthesizeoftheinputwillbeunknowntoarangoimport.

Bydefault,theendpointtcp://127.0.0.1:8529willbeused.Ifyouwanttospecifyadifferentendpoint,youcanusethe--server.endpointoption.Youprobablywanttospecifyadatabaseuserandpasswordaswell.Youcandosobyusingtheoptions--server.usernameand--server.password.Ifyoudonotspecifyapassword,youwillbepromptedforone.

>arangoimport--server.endpointtcp://127.0.0.1:8529--server.usernameroot--file"data.json"--typejson--collection"users

"

Notethatthecollection(usersinthiscase)mustalreadyexistortheimportwillfail.Ifyouwanttocreateanewcollectionwiththeimportdata,youneedtospecifythe--create-collectionoption.Notethatbydefaultitwillcreateadocumentcollectionandnoedgecollection.

>arangoimport--file"data.json"--typejson--collection"users"--create-collectiontrue

Tocreateanedgecollectioninstead,usethe--create-collection-typeoptionandsetittoedge:

Arangoimport

372

>arangoimport--file"data.json"--collection"myedges"--create-collectiontrue--create-collection-typeedge

Whenimportingdataintoanexistingcollectionitisoftenconvenienttofirstremovealldatafromthecollectionandthenstarttheimport.Thiscanbeachievedbypassingthe--overwriteparametertoarangoimport.Ifitissettotrue,anyexistingdatainthecollectionwillberemovedpriortotheimport.Notethatanyexistingindexdefinitionsforthecollectionwillbepreservedevenif--overwriteissettotrue.

>arangoimport--file"data.json"--typejson--collection"users"--overwritetrue

AstheimportfilealreadycontainsthedatainJSONformat,attributenamesanddatatypesarefullypreserved.Ascanbeseenintheexampledata,thereisnoneedforalldatarecordstohavethesameattributenamesortypes.Recordscanbeinhomogeneous.

Pleasenotethatbydefault,arangoimportwillimportdataintothespecifiedcollectioninthedefaultdatabase(_system).Tospecifyadifferentdatabase,usethe--server.databaseoptionwheninvokingarangoimport.Ifyouwanttoimportintoanonexistentdatabaseyouneedtopass--create-databasetrue.Note--create-databasedefaultstofalse

Thetoolalsosupportsparallelimports,withmultiplethreads.Usingmultiplethreadsmayprovideaspeedup,especiallywhenusingtheRocksDBstorageengine.Tospecifythenumberofparallelthreadsusethe--threadsoption:

>arangoimport--threads4--file"data.json"--typejson--collection"users"

Notethatusingmultiplethreadsmayleadtoanon-sequentialimportoftheinputdata.Datathatappearslaterintheinputfilemaybeimportedearlierthandatathatappearsearlierintheinputfile.Thisisnormallynotaproblembutmaycauseissueswhenwhentherearedatadependenciesorduplicatesintheimportdata.Inthiscase,thenumberofthreadsshouldbesetto1.

JSONinputfileformats

Note:arangoimportsupportstwoformatswhenimportingJSONdatafromafile.Thefirstformatthatwealsousedaboveiscommonlyknownasjsonl).However,incontrasttotheJSONLspecificationitrequirestheinputfiletocontainonecompleteJSONdocumentineachline,e.g.

{"_key":"one","value":1}

{"_key":"two","value":2}

{"_key":"foo","value":"bar"}

...

SoonecouldarguethatthisisonlyasubsetofJSONL.

Theaboveformatcanbeimportedsequentiallybyarangoimport.Itwillreaddatafromtheinputfileinchunksandsenditinbatchestotheserver.Eachbatchwillbeaboutasbigasspecifiedinthecommand-lineparameter--batch-size.

AnalternativeistoputonebigJSONdocumentintotheinputfilelikethis:

[

{"_key":"one","value":1},

{"_key":"two","value":2},

{"_key":"foo","value":"bar"},

...

]

Thisformatallowslinebreakswithintheinputfileasrequired.Thedownsideisthatthewholeinputfilewillneedtobereadbyarangoimportbeforeitcansendthefirstbatch.Thismightbeaproblemiftheinputfileisbig.Bydefault,arangoimportwillallowimportingsuchfilesuptoasizeofabout16MB.

Ifyouwanttoallowyourarangoimportinstancetousemorememory,youmaywanttoincreasethemaximumfilesizebyspecifyingthecommand-lineoption--batch-size.Forexample,tosetthebatchsizeto32MB,usethefollowingcommand:

>arangoimport--file"data.json"--typejson--collection"users"--batch-size33554432

Pleasealsonotethatyoumayneedtoincreasethevalueof--batch-sizeifasingledocumentinsidetheinputfileisbiggerthanthevalueof--batch-size.

Arangoimport

373

ImportingCSVData

arangoimportalsooffersthepossibilitytoimportdatafromCSVfiles.ThiscomeshandywhenthedataathandisinCSVformatalreadyandyoudon'twanttospendtimeconvertingthemtoJSONfortheimport.

ToimportdatafromaCSVfile,makesureyourfilecontainstheattributenamesinthefirstrow.Allthefollowinglinesinthefilewillbeinterpretedasdatarecordsandwillbeimported.

TheCSVimportrequiresthedatatohaveahomogeneousstructure.Allrecordsmusthaveexactlythesameamountofcolumnsasthereareheaders.Bydefault,lineswithadifferentnumberofvalueswillnotbeimportedandtherewillbewarningsforthem.Tostillimportlineswithlessvaluesthanintheheader,thereisthe--ignore-missingoption.Ifsettotrue,linesthathaveadifferentamountoffieldswillbeimported.Inthiscaseonlythoseattributeswillbepopulatedforwhichtherearevalues.Attributesforwhichtherearenovaluespresentwillsilentlybediscarded.

Example:

"first","last","age","active","dob"

"John","Connor",25,true

"Jim","O'Brady"

With--ignore-missingthiswillproducethefollowingdocuments:

{"first":"John","last":"Connor","active":true,"age":25}

{"first":"Jim","last":"O'Brady"}

Thecellvaluescanhavedifferentdatatypesthough.Ifacelldoesnothaveanyvalue,itcanbeleftemptyinthefile.Thesevalueswillnotbeimportedsotheattributeswillnot"bethere"indocumentcreated.Valuesenclosedinquoteswillbeimportedasstrings,sotoimportnumericvalues,booleanvaluesorthenullvalue,don'tenclosethevalueinquotesinyourfile.

We'llbeusingthefollowingimportfortheCSVimport:

"first","last","age","active","dob"

"John","Connor",25,true,

"Jim","O'Brady",19,,

"Lisa","Jones",,,"1981-04-09"

Hans,dosSantos,0123,,

Wayne,Brewer,,false,

Thecommandlinetoexecutetheimportis:

>arangoimport--file"data.csv"--typecsv--collection"users"

Theabovedatawillbeimportedinto5documentswhichwilllookasfollows:

{"first":"John","last":"Connor","active":true,"age":25}

{"first":"Jim","last":"O'Brady","age":19}

{"first":"Lisa","last":"Jones","dob":"1981-04-09"}

{"first":"Hans","last":"dosSantos","age":123}

{"first":"Wayne","last":"Brewer","active":false}

Ascanbeseen,valuesleftcompletelyemptyintheinputfilewillbetreatedasabsent.Numericvaluesnotenclosedinquoteswillbetreatedasnumbers.Notethatleadingzerosinnumericvalueswillberemoved.Toimportnumberswithleadingzeros,pleaseusestrings.Theliteralstrueandfalsewillbetreatedasbooleansiftheyarenotenclosedinquotes.Othervaluesnotenclosedinquoteswillbetreatedasstrings.Anyvaluesenclosedinquoteswillbetreatedasstrings,too.

Stringvaluescontainingthequotecharacterortheseparatormustbeenclosedwithquotecharacters.Withinastring,thequotecharacteritselfmustbeescapedwithanotherquotecharacter(orwithabackslashifthe--backslash-escapeoptionisused).

Notethatthequoteandseparatorcharacterscanbeadjustedviathe--quoteand--separatorargumentswheninvokingarangoimport.Thequotecharacterdefaultstothedoublequote(").Tousealiteralquoteinastring,youcanusetwoquotecharacters.Tousebackslashforescapingquotecharacters,pleasesettheoption--backslash-escapetotrue.

Arangoimport

374

TheimportersupportsWindows(CRLF)andUnix(LF)linebreaks.Linebreaksmightalsooccurinsidevaluesthatareenclosedwiththequotecharacter.

Here'sanexampleforusingliteralquotesandnewlinesinsidevalues:

"name","password"

"Foo","r4ndom""123!"

"Bar","wow!

thisisa

multinepassword!"

"Bartholomew""Bart""Simpson","Milhouse"

Extrawhitespaceattheendofeachlinewillbeignored.Whitespaceatthestartoflinesorbetweenfieldvalueswillnotbeignored,sopleasemakesurethatthereisnoextrawhitespaceinfrontofvaluesorbetweenthem.

ImportingTSVData

Youmayalsoimporttab-separatedvalues(TSV)fromafile.Thisformatisverysimple:everylineinthefilerepresentsadatarecord.Thereisnoquotingorescaping.Thatalsomeansthattheseparatorcharacter(whichdefaultstothetabstopsymbol)mustnotbeusedanywhereintheactualdata.

AswithCSV,thefirstlineintheTSVfilemustcontaintheattributenames,andalllinesmusthaveanidenticalnumberofvalues.

Ifadifferentseparatorcharacterorstringshouldbeused,itcanbespecifiedwiththe--separatorargument.

AnexamplecommandlinetoexecutetheTSVimportis:

>arangoimport--file"data.tsv"--typetsv--collection"users"

AttributeNameTranslation

FortheCSVandTSVinputformats,attributenamescanbetranslatedautomatically.ThisisusefulincasetheimportfilehasdifferentattributenamesthanthosethatshouldbeusedinArangoDB.

Acommonusecaseistorenamean"id"columnfromtheinputfileinto"_key"asitisexpectedbyArangoDB.Todothis,specifythefollowingtranslationwheninvokingarangoimport:

>arangoimport--file"data.csv"--typecsv--translate"id=_key"

Othercommoncasesaretorenamecolumnsintheinputfileto_fromand_to:

>arangoimport--file"data.csv"--typecsv--translate"from=_from"--translate"to=_to"

Thetranslateoptioncanbespecifiedmultipletypes.Thesourceattributenameandthetargetattributemustbeseparatedwitha=.

IgnoringAttributes

FortheCSVandTSVinputformats,certainattributenamescanbeignoredonimports.InanArangoDBclustertherearecaseswherethiscancomeinhandy,whenyourdocumentsalreadycontaina_keyattributeandyourcollectionhasashardingattributeotherthan_key:Intheclusterthisconfigurationisnotsupported,becauseArangoDBneedstoguaranteetheuniquenessofthe_keyattributeinallshardsofthecollection.

>arangoimport--file"data.csv"--typecsv--remove-attribute"_key"

Thesamethingwouldapplyifyourdatacontainsan_idattribute:

>arangoimport--file"data.csv"--typecsv--remove-attribute"_id"

ImportingintoanEdgeCollection

Arangoimport

375

arangoimportcanalsobeusedtoimportdataintoanexistingedgecollection.Theimportdatamust,foreachedgetoimport,containatleastthe_fromand_toattributes.Theseindicatewhichothertwodocumentstheedgeshouldconnect.Itisnecessarythattheseattributesaresetforallrecords,andpointtovaliddocumentidsinexistingcollections.

Examples

{"_from":"users/1234","_to":"users/4321","desc":"1234isconnectedto4321"}

Note:Theedgecollectionmustalreadyexistwhentheimportisstarted.Usingthe--create-collectionflagwillnotworkbecausearangoimportwillalwaystrytocreatearegulardocumentcollectionifthetargetcollectiondoesnotexist.

Updatingexistingdocuments

Bydefault,arangoimportwilltrytoinsertalldocumentsfromtheimportfileintothespecifiedcollection.Incasetheimportfilecontainsdocumentsthatarealreadypresentinthetargetcollection(matchingisdoneviathe_keyattributes),thenadefaultarangoimportrunwillnotimportthesedocumentsandcomplainaboutuniquekeyconstraintviolations.

However,arangoimportcanbeusedtoupdateorreplaceexistingdocumentsincasetheyalreadyexistinthetargetcollection.Itprovidesthecommand-lineoption--on-duplicatetocontrolthebehaviorincaseadocumentisalreadypresentinthedatabase.

Thedefaultvalueof--on-duplicateiserror.Thismeansthatwhentheimportfilecontainsadocumentthatispresentinthetargetcollectionalready,thentryingtore-insertadocumentwiththesame_keyvalueisconsideredanerror,andthedocumentinthedatabasewillnotbemodified.

Otherpossiblevaluesfor--on-duplicateare:

update:eachdocumentpresentintheimportfilethatisalsopresentinthetargetcollectionalreadywillbeupdatedbyarangoimport.updatewillperformapartialupdateoftheexistingdocument,modifyingonlytheattributesthatarepresentintheimportfileandleavingallotherattributesuntouched.

Thevaluesofsystemattributes_id,_key,_rev,_fromand_tocannotbeupdatedorreplacedinexistingdocuments.

replace:eachdocumentpresentintheimportfilethatisalsopresentinthetargetcollectionalreadywillbereplacebyarangoimport.replacewillreplacetheexistingdocumententirely,resultinginadocumentwithonlytheattributesspecifiedintheimportfile.

Thevaluesofsystemattributes_id,_key,_rev,_fromand_tocannotbeupdatedorreplacedinexistingdocuments.

ignore:eachdocumentpresentintheimportfilethatisalsopresentinthetargetcollectionalreadywillbeignoredandnotmodifiedinthetargetcollection.

When--on-duplicateissettoeitherupdateorreplace,arangoimportwillreturnthenumberofdocumentsupdated/replacedintheupdatedreturnvalue.Whensettoanothervalue,thevalueofupdatedwillalwaysbezero.When--on-duplicateissettoignore,arangoimportwillreturnthenumberofignoreddocumentsintheignoredreturnvalue.Whensettoanothervalue,ignoredwillalwaysbezero.

Itispossibletoperformacombinationofinsertsandupdates/replaceswithasinglearangoimportrun.When--on-duplicateissettoupdateorreplace,alldocumentspresentintheimportfilewillbeinsertedintothetargetcollectionprovidedtheyarevalidanddonotalreadyexistwiththespecified_key.Documentsthatarealreadypresentinthetargetcollection(identifiedby_keyattribute)willinsteadbeupdated/replaced.

Arangoimportresultoutput

Anarangoimportimportrunwillprintoutthefinalresultsonthecommandline.Itwillshowthe

numberofdocumentscreated(created)numberofdocumentsupdated/replaced(updated/replaced,onlynon-zeroif--on-duplicatewassettoupdateorreplace,seebelow)numberofwarningsorerrorsthatoccurredontheserverside(warnings/errors)numberofignoreddocuments(onlynon-zeroif--on-duplicatewassettoignore).

Example

created:2

warnings/errors:0

updated/replaced:0

ignored:0

Arangoimport

376

ForCSVandTSVimports,thetotalnumberofinputfilelinesreadwillalsobeprinted(linesread).

arangoimportwillalsoprintoutdetailsaboutwarningsanderrorsthathappenedontheserver-side(ifany).

AttributeNamingandSpecialAttributes

AttributeswhosenamesstartwithanunderscorearetreatedinaspecialwaybyArangoDB:

theoptional_keyattributecontainsthedocument'skey.Ifspecified,thevaluemustbeformallyvalid(e.g.mustbeastringandconformtothenamingconventions).Additionally,thekeyvaluemustbeuniquewithinthecollectiontheimportisrunfor._from:whenimportingintoanedgecollection,thisattributecontainstheidofoneofthedocumentsconnectedbytheedge.Thevalueof_frommustbeasyntacticallyvaliddocumentidandthereferredcollectionmustexist._to:whenimportingintoanedgecollection,thisattributecontainstheidoftheotherdocumentconnectedbytheedge.Thevalueof_tomustbeasyntacticallyvaliddocumentidandthereferredcollectionmustexist._rev:thisattributecontainstherevisionnumberofadocument.However,therevisionnumbersaremanagedbyArangoDBandcannotbespecifiedonimport.Thusanyvalueinthisattributeisignoredonimport.

Ifyouimportvaluesinto_key,youshouldmakesuretheyarevalidandunique.

Whenimportingdataintoanedgecollection,youshouldmakesurethatallimportdocumentscan_fromand_toandthattheirvaluespointtoexistingdocuments.

Toavoidspecifyingcompletedocumentids(consistingofcollectionnamesanddocumentkeys)for_fromand_tovalues,therearetheoptions--from-collection-prefixand--to-collection-prefix.Ifspecified,thesevalueswillbeautomaticallyprependedtoeachvaluein_from(or_toresp.).Thisallowsspecifyingonlydocumentkeysinside_fromand/or_to.

Example

>arangoimport--from-collection-prefixusers--to-collection-prefixproducts...

Importingthefollowingdocumentwillthencreateanedgebetweenusers/1234andproducts/4321:

{"_from":"1234","_to":"4321","desc":"users/1234isconnectedtoproducts/4321"}

Arangoimport

377

DumpingDatafromanArangoDBdatabaseTodumpdatafromanArangoDBserverinstance,youwillneedtoinvokearangodump.Dumpscanbere-importedwitharangorestore.arangodumpcanbeinvokedbyexecutingthefollowingcommand:

unix>arangodump--output-directory"dump"

ThiswillconnecttoanArangoDBserveranddumpallnon-systemcollectionsfromthedefaultdatabase(_system)intoanoutputdirectorynameddump.Invokingarangodumpwillfailiftheoutputdirectoryalreadyexists.Thisisanintentionalsecuritymeasuretopreventyoufromaccidentallyoverwritingalreadydumpeddata.Ifyouarepositivethatyouwanttooverwritedataintheoutputdirectory,youcanusetheparameter--overwritetruetoconfirmthis:

unix>arangodump--output-directory"dump"--overwritetrue

arangodumpwillbydefaultconnecttothe_systemdatabaseusingthedefaultendpoint.Ifyouwanttoconnecttoadifferentdatabaseoradifferentendpoint,oruseauthentication,youcanusethefollowingcommand-lineoptions:

--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:username--server.password:passwordtouse(omitthisandyou'llbepromptedforthepassword)--server.authentication:whetherornottouseauthentication

Here'sanexampleofdumpingdatafromanon-standardendpoint,usingadedicateddatabasename:

unix>arangodump--server.endpointtcp://192.168.173.13:8531--server.usernamebackup--server.databasemydb--output-directory

"dump"

Whenfinished,arangodumpwillprintoutasummarylinewithsomeaggregatestatisticsaboutwhatitdid,e.g.:

Processed43collection(s),wrote408173500byte(s)intodatafiles,sent88batch(es)

Bydefault,arangodumpwilldumpbothstructuralinformationanddocumentsfromallnon-systemcollections.Toadjustthis,therearethefollowingcommand-linearguments:

--dump-data:settotruetoincludedocumentsinthedump.Settofalsetoexcludedocuments.Thedefaultvalueistrue.--include-system-collections:whetherornottoincludesystemcollectionsinthedump.Thedefaultvalueisfalse.

Forexample,toonlydumpstructuralinformationofallcollections(includingsystemcollections),use:

unix>arangodump--dump-datafalse--include-system-collectionstrue--output-directory"dump"

Torestrictthedumptojustspecificcollections,thereisisthe--collectionoption.Itcanbespecifiedmultipletimesifrequired:

unix>arangodump--collectionmyusers--collectionmyvalues--output-directory"dump"

Structuralinformationforacollectionwillbesavedinfileswithnamepattern.structure.json.EachstructurefilewillcontainsaJSONobjectwiththeseattributes:

parameters:containsthecollectionpropertiesindexes:containsthecollectionindexes

Documentdataforacollectionwillbesavedinfileswithnamepattern.data.json.Eachlineinadatafileisadocumentinsertion/updateordeletionmarker,alongsidewithsomemetadata.

StartingwithVersion2.1ofArangoDB,thearangodumptoolalsosupportssharding.Simplypointittooneofthecoordinatorsanditwillbehaveexactlyasdescribedabove,workingonshardedcollectionsinthecluster.

Arangodump

378

However,asopposedtothesingleinstancesituation,thisoperationdoesnotguaranteetodumpaconsistentsnapshotifwriteoperationshappenduringthedumpoperation.Itisthereforerecommendednottoperformanydata-modifcationoperationsontheclusterwhilstarangodumpisrunning.

Asabove,theoutputwillbeonestructuredescriptionfileandonedatafilepershardedcollection.Notethatthedatainthedatafileissortedfirstbyshardsandwithineachshardbyascendingtimestamp.Thestructuralinformationofthecollectioncontainsthenumberofshardsandtheshardkeys.

NotethattheversionofthearangodumpclienttoolneedstomatchtheversionoftheArangoDBserveritconnectsto.

Advancedclusteroptions

Startingwithversion3.1.17,collectionsmaybecreatedwithsharddistributionidenticaltoanexistingprototypicalcollection;i.e.shardsaredistributedintheverysamepatternasintheprototypecollection.Suchcollectionscannotbedumpedwithoutthereferencecollectionorarangodumpwithyieldanerror.

unix>arangodump--collectionclonedCollection--output-directory"dump"

ERRORCollectionclonedCollection'ssharddistributionisbasedonathatofcollectionprototypeCollection,whichisnotdumpe

dalong.Youmaydumpthecollectionregardlessofthemissingprototypecollectionbyusingthe--ignore-distribute-shards-lik

e-errorsparameter.

Therearetwowaystoapproachthatproblem:Solveit,i.e.dumptheprototypecollectionalong:

unix>arangodump--collectionclonedCollection--collectionprototypeCollection--output-directory"dump"

Processed2collection(s),wrote81920byte(s)intodatafiles,sent1batch(es)

Oroverridethatbehaviourtobeabletodumpthecollectionindividually.

unix>arangodump--collectionBclonedCollection--output-directory"dump"--ignore-distribute-shards-like-errors

Processed1collection(s),wrote34217byte(s)intodatafiles,sent1batch(es)

Nothatinconsequence,restoringsuchacollectionwithoutitsprototypeisaffected.arangorestore

Encryption

IntheArangoDBEnterpriseEditiontherearetheadditionalparameters:

Encryptionkeystoredinfile

--encryption.keyfilepath-of-keyfile

Thefilepath-to-keyfilemustcontaintheencryptionkey.Thisfilemustbesecured,sothatonlyarangodcanaccessit.Youshouldalsoensurethatincasesome-onestealsthehardware,hewillnotbeabletoreadthefile.Forexample,byencryption/mytmpfsorcreatingain-memoryfile-systemunder/mytmpfs.

Encryptionkeygeneratedbyaprogram

--encryption.key-generatorpath-to-my-generator

Theprogrampath-to-my-generatormustoutputtheencryptiononstandardoutputandexit.

Creatingkeys

Theencryptionkeyfilemustcontain32bytesofrandomdata.

Youcancreateitwithacommandlinethis.

ddif=/dev/randombs=1count=32of=yourSecretKeyFile

Arangodump

379

Forsecurity,itisbesttocreatethesekeysoffline(awayfromyourdatabaseservers)anddirectlystoretheminyousecretmanagementtool.

Arangodump

380

ArangorestoreToreloaddatafromadumppreviouslycreatedwitharangodump,ArangoDBprovidesthearangorestoretool.

Pleasenotethatarangorestoremustnotbeusedtocreateseveralsimilardatabaseinstancesinoneinstallation.

Thismeansifyouhaveanarangodumpoutputofdatabasea,andyoucreateaseconddatabasebonthesameinstanceofArangoDB,andrestorethedumpofaintob-dataintegritycannotbeguaranteed.

ReloadingDataintoanArangoDBdatabase

Invokingarangorestore

arangorestorecanbeinvokedfromthecommand-lineasfollows:

unix>arangorestore--input-directory"dump"

ThiswillconnecttoanArangoDBserverandreloadstructuralinformationanddocumentsfoundintheinputdirectorydump.Pleasenotethattheinputdirectorymusthavebeencreatedbyrunningarangodumpbefore.

arangorestorewillbydefaultconnecttothe_systemdatabaseusingthedefaultendpoint.Ifyouwanttoconnecttoadifferentdatabaseoradifferentendpoint,oruseauthentication,youcanusethefollowingcommand-lineoptions:

--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:username--server.password:passwordtouse(omitthisandyou'llbepromptedforthepassword)--server.authentication:whetherornottouseauthentication

Sinceversion2.6arangorestoreprovidestheoption--create-database.Settingthisoptiontotruewillcreatethetargetdatabaseifitdoesnotexist.Whencreatingthetargetdatabase,theusernameandpasswordspassedtoarangorestore(inoptions--server.usernameand--server.password)willbeusedtocreateaninitialuserforthenewdatabase.

Theoption--force-same-databaseallowsrestrictingarangorestoreoperationstoadatabasewiththesamenameasinthesourcedump's"dump.json"file.Itcanthusbeusedtopreventrestoringdataintoa"wrong"databasebyaccident.

Forexample,ifadumpwastakenfromdatabasea,andtherestoreisattemptedintodatabaseb,thenwiththe--force-same-databaseoptionsettotrue,arangorestorewillabortinstantly.

The--force-same-databaseoptionissettofalsebydefaulttoensurebackwards-compatibility.

Here'sanexampleofreloadingdatatoanon-standardendpoint,usingadedicateddatabasename:

unix>arangorestore--server.endpointtcp://192.168.173.13:8531--server.usernamebackup--server.databasemydb--input-directo

ry"dump"

Tocreatethetargetdatabasewherestoring,useacommandlikethis:

unix>arangorestore--server.usernamebackup--server.databasenewdb--create-databasetrue--input-directory"dump"

arangorestorewillprintoutitsprogresswhilerunning,andwillendwithalineshowingsomeaggregatestatistics:

Processed2collection(s),read2256byte(s)fromdatafiles,sent2batch(es)

Bydefault,arangorestorewillre-createallnon-systemcollectionsfoundintheinputdirectoryandloaddataintothem.Ifthetargetdatabasealreadycontainscollectionswhicharealsopresentintheinputdirectory,theexistingcollectionsinthedatabasewillbedroppedandre-createdwiththedatafoundintheinputdirectory.

Arangorestore

381

Thefollowingparametersareavailabletoadjustthisbehavior:

--create-collection:settotruetocreatecollectionsinthetargetdatabase.Ifthetargetdatabasealreadycontainsacollectionwiththesamename,itwillbedroppedfirstandthenre-createdwiththepropertiesfoundintheinputdirectory.Settofalsetokeepexistingcollectionsinthetargetdatabase.Ifsettofalseandarangorestoreencountersacollectionthatispresentintheinputdirectorybutnotinthetargetdatabase,itwillabort.Thedefaultvalueistrue.--import-data:settotruetoloaddocumentdataintothecollectionsinthetargetdatabase.Settofalsetonotloadanydocumentdata.Thedefaultvalueistrue.--include-system-collections:whetherornottoincludesystemcollectionswhenre-creatingcollectionsorreloadingdata.Thedefaultvalueisfalse.

Forexample,to(re-)createallnon-systemcollectionsandloaddocumentdataintothem,use:

unix>arangorestore--create-collectiontrue--import-datatrue--input-directory"dump"

Thiswilldroppotentiallyexistingcollectionsinthetargetdatabasethatarealsopresentintheinputdirectory.

Toincludesystemcollectionstoo,use--include-system-collectionstrue:

unix>arangorestore--create-collectiontrue--import-datatrue--include-system-collectionstrue--input-directory"dump"

To(re-)createallnon-systemcollectionswithoutloadingdocumentdata,use:

unix>arangorestore--create-collectiontrue--import-datafalse--input-directory"dump"

Thiswillalsodropexistingcollectionsinthetargetdatabasethatarealsopresentintheinputdirectory.

Tojustloaddocumentdataintoallnon-systemcollections,use:

unix>arangorestore--create-collectionfalse--import-datatrue--input-directory"dump"

Torestrictreloadingtojustspecificcollections,thereisisthe--collectionoption.Itcanbespecifiedmultipletimesifrequired:

unix>arangorestore--collectionmyusers--collectionmyvalues--input-directory"dump"

Collectionswillbeprocessedbyinalphabeticalorderbyarangorestore,withalldocumentcollectionsbeingprocessedbeforealledgecollections.Thisistoensurethatreloadingdataintoedgecollectionswillhavethedocumentcollectionslinkedinedges(_fromand_toattributes)loaded.

Encryption

Seearangodumpfordetails.

RestoringRevisionIdsandCollectionIds

arangorestorewillreloaddocumentandedgesdatawiththeexactsame_key,_fromand_tovaluesfoundintheinputdirectory.However,whenloadingdocumentdata,itwillassignitsownvaluesforthe_revattributeofthereloadeddocuments.Thoughthisdifferenceisintentional(normally,everyservershouldcreateitsown_revvalues)theremightbesituationswhenitisrequiredtore-usetheexactsame_revvaluesforthereloadeddata.Thiscanbeachievedbysettingthe--recycle-idsparametertotrue:

unix>arangorestore--collectionmyusers--collectionmyvalues--input-directory"dump"

Notethatsetting--recycle-idstotruewillalsocausecollectionstobe(re-)createdinthetargetdatabasewiththeexactsamecollectionidasintheinputdirectory.Anypotentiallyexistingcollectioninthetargetdatabasewiththesamecollectionidwillthenbedropped.

ReloadingDataintoadifferentCollection

Arangorestore

382

Withsomecreativityyoucanusearangodumpandarangorestoretotransferdatafromonecollectionintoanother(eitheronthesameserverornot).Forexample,tocopydatafromacollectionmyvaluesindatabasemydbintoacollectionmycopyvaluesindatabasemycopy,youcanstartwiththefollowingcommand:

unix>arangodump--collectionmyvalues--server.databasemydb--output-directory"dump"

Thiswillcreatetwofiles,myvalues.structure.jsonandmyvalues.data.json,intheoutputdirectory.Toloaddatafromthedatafileintoanexistingcollectionmycopyvaluesindatabasemycopy,renamethefilestomycopyvalues.structure.jsonandmycopyvalues.data.json.Afterthat,runthefollowingcommand:

unix>arangorestore--collectionmycopyvalues--server.databasemycopy--input-directory"dump"

Usingarangorestorewithsharding

AsofVersion2.1thearangorestoretoolsupportssharding.Simplypointittooneofthecoordinatorsinyourclusteranditwillworkasusualbutonshardedcollectionsinthecluster.

Ifarangorestoreisaskedtodropandre-createacollection,itwillusethesamenumberofshardsandthesameshardkeysaswhenthecollectionwasdumped.Thedistributionoftheshardstotheserverswillalsobethesameasatthetimeofthedump.ThismeansinparticularthatDBserverswiththesameIDsasbeforemustbepresentintheclusterattimeoftherestore.

Ifacollectionwasdumpedfromasingleinstance,onecanmanuallyaddthestructuraldescriptionfortheshardkeysandthenumberanddistributionoftheshardsandthentherestoreintoaclusterwillwork.

IfyourestoreacollectionthatwasdumpedfromaclusterintoasingleArangoDBinstance,thenumberofshardsandtheshardkeyswillsilentlybeignored.

Notethatinacluster,everynewlycreatedcollectionwillhaveanewID,itisnotpossibletoreusetheIDfromtheoriginallydumpedcollection.ThisisforsafetyreasonstoensureconsistencyofIDs.

Restoringcollectionswithshardingprototypes

arangorestorewillyieldanerror,whiletryingtorestoreacollection,whosesharddistributionfollowsacollection,whichdoesnotexistintheclusterandwhichwasnotdumpedalong:

unix>arangorestore--collectionclonedCollection--server.databasemydb--input-directory"dump"

ERRORgoterrorfromserver:HTTP500(InternalServerError):ArangoError1486:mustnothaveadistributeShardsLikeattribute

pointingtoanunknowncollection

Processed0collection(s),read0byte(s)fromdatafiles,sent0batch(es)

Thecollectioncanberestoredbyoverridingtheerrormessageasfollows:

unix>arangorestore--collectionclonedCollection--server.databasemydb--input-directory"dump"--ignore-distribute-shards-li

ke-errors

RestoreintoanauthenticationenabledArangoDB

OfcourseyoucanrestoredataintoapasswordprotectedArangoDBaswell.Howeverthisrequirescertainuserrightsfortheuserusedintherestoreprocess.TherightsaredescribedindetailintheManagingUserschapter.Forrestorethisshortoverviewissufficient:

Whenimportingintoanexistingdatabase,thegivenuserneedsAdministrateaccessonthisdatabase.WhencreatinganewDatabaseduringrestore,thegivenuserneedsAdministrateaccesson_system.TheuserwillbepromotedwithAdministrateaccessonthenewlycreateddatabase.

Arangorestore

383

ExportingDatafromanArangoDBdatabaseToexportdatafromanArangoDBserverinstance,youwillneedtoinvokearangoexport.arangoexportcanbeinvokedbyexecutingthefollowingcommand:

unix>arangoexport--collectiontest--output-directory"dump"

Thisexportsthecollectionstestintothedirectorydumpasonebigjsonarray.Everyentryinthisarrayisonedocumentfromthecollectionwithoutaspecificorder.Toexportmorethanonecollectionatatimespecifymultiple--collectionoptions.

Thedefaultoutputdirectoryisexport.

arangoexportwillbydefaultconnecttothe_systemdatabaseusingthedefaultendpoint.Ifyouwanttoconnecttoadifferentdatabaseoradifferentendpoint,oruseauthentication,youcanusethefollowingcommand-lineoptions:

--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:username--server.password:passwordtouse(omitthisandyou'llbepromptedforthepassword)--server.authentication:whetherornottouseauthentication

Here'sanexampleofexportingdatafromanon-standardendpoint,usingadedicateddatabasename:

unix>arangoexport--server.endpointtcp://192.168.173.13:8531--server.usernamebackup--server.databasemydb--collectiontes

t--output-directory"my-export"

Whenfinished,arangoexportwillprintoutasummarylinewithsomeaggregatestatisticsaboutwhatitdid,e.g.:

Processed2collection(s),wrote9031763Byte(s),78HTTPrequest(s)

ExportJSON

unix>arangoexport--typejson--collectiontest

Thisexportsthecollectiontestintotheoutputdirectoryexportasonejsonarray.Everyarrayentryisonedocumentfromthecollectiontest

ExportJSONL

unix>arangoexport--typejsonl--collectiontest

Thisexportsthecollectiontestintotheoutputdirectoryexportasjsonl.Everylineintheexportisonedocumentfromthecollectiontestasjson.

ExportCSV

unix>arangoexport--typecsv--collectiontest--fields_key,_id,_rev

ThisexportsthecollectiontestintotheoutputdirectoryexportasCSV.Thefirstlinecontainstheheaderwithallfieldnames.EachlineisonedocumentrepresentedasCSVandseparatedwithacomma.ObjectsandArraysarerepresentedasaJSONstring.

Arangoexport

384

ExportXML

unix>arangoexport--typexml--collectiontest

ThisexportsthecollectiontestintotheoutputdirectoryexportasgenericXML.TherootelementofthegeneratedXMLfileisnamedcollection.EachdocumentinthecollectionisexportedinadocXMLattribute.Eachdocumentattributeisexportinagenericattelement,whichhasatypeattributeindicatingtheattributevalue,andavalueattributecontainingtheattribute'svalue.

ExportXGMMLXGMMLisanXMLapplicationbasedonGML.ToviewtheXGMMLfileyoucanuseforexampleCytoscape.

importantnoteIfyouexportallattributes(--xgmml-label-onlyfalse)keepinmindthataatrributenamestypehavetobethesametypeforalldocuments.Itwontworkifyouhaveaattributenamedrankthatisinonedocumentastringandinanotherdocumentainteger.

Bad

//doc1

{

"rank":1

}

//doc2

{

"rank":"2"

}

Good

//doc1

{

"rank":1

}

//doc2

{

"rank":2

}

XGMMLspecificoptions--xgmml-label-attributespecifythenameoftheattributethatwillbecomethelabelinthexgmmlfile.

--xgmml-label-onlysettotruewillonlyexportthelabelwithoutanyattributesinedgesornodes.

exportbasedoncollections

unix>arangoexport--typexgmml--graph-namemygraph--collectionvertex--collectionedge

Thisexportstheaunnamedgraphwithvertexcollectionvertexandedgecollectionedgeintothexgmmlfilemygraph.xgmml.

exportbasedonanamedgraph

unix>arangoexport--typexgmml--graph-namemygraph

Arangoexport

385

Thisexportsthenamedgraphmygraphintothexgmmlfilemygraph.xgmml.

exportXGMMLwithoutattributes

unix>arangoexport--typexgmml--graph-namemygraph--xgmml-label-onlytrue

Thisexportsthenamedgraphmygraphintothexgmmlfilemygraph.xgmmlwithoutthe<att>taginnodesandedges.

exportXGMMLwithaspecificlabel

unix>arangoexport--typexgmml--graph-namemygraph--xgmml-label-attributename

Thisexportsthenamedgraphmygraphintothexgmmlfilemygraph.xgmmlwithalabelfromdocumentsattributenameinsteadofthedefaultattributelabel.

ExportviaAQLquery

unix>arangoexport--typejsonl--query"forbookinbooksfilterbook.sells>100returnbook"

Exportviaanaqlqueryallowsyoutoexportthereturneddataasthetypespecifiedwith--type.Theexampleexportsallbooksasjsonlthataresoldmorethan100times.

Arangoexport

386

ManagingUsersTheusermanagementinArangoDB3issimilartotheonesfoundinMySQL,PostgreSQL,orotherdatabasesystems.

Usermanagementispossibleinthewebinterfaceandinarangoshwhileloggedontothe_systemdatabase.

Notethattheonlyusernamesmustnotstartwith:role:.

ActionsandAccessLevelsAnArangoDBservercontainsalistofusers.Italsodefinesvariousaccesslevelsthatcanbeassignedtoauser(fordetails,seebelow)andthatareneededtoperformcertainactions.Theseactionscanbegroupedintothreecategories:

serveractionsdatabaseactionscollectionactions

Theserveractionsare

createuser:allowstocreateanewuser.

updateuser:allowstochangetheaccesslevelsanddetailsofanexistinguser.

dropuser:allowstodeleteanexistinguser.

createdatabase:allowstocreateanewdatabase.

dropdatabase:allowstodeleteanexistingdatabase.

shutdownserver:removeserverfromclusterandshutdown

Thedatabaseactionsaretiedtoagivendatabase,andaccesslevelsmustbesetforeachdatabaseindividually.Foragivendatabasetheactionsare

createcollection:allowstocreateanewcollectioninthegivendatabase.

updatecollection:allowstoupdatepropertiesofanexistingcollection.

dropcollection:allowstodeleteanexistingcollection.

createindex:allowstocreateanindexforanexistingcollectioninthegivendatabase.

dropindex:allowstodeleteanindexofanexistingcollectioninthegivendatabase.

Thecollectionactionsaretiedtoagivencollectionofagivendatabase,andaccesslevelsmustbesetforeachcollectionindividually.Foragivencollectiontheactionsare

readdocument:readadocumentofthegivencollection.

createdocument:createsanewdocumentinthegivencollection.

modifydocument:modifiesanexistingdocumentofthegivencollection,thiscanbeanupdateorreplaceoperation.

dropdocument:deletesanexistingdocumentofthegivencollection.

truncatecollection:deletesalldocumentsofagivencollection.

Toperformactionsontheserverleveltheuserneedsatleastthefollowingaccesslevels.TheaccesslevelsareAdministrateandNoaccess:

serveraction serverlevel

createadatabase Administrate

dropadatabase Administrate

createauser Administrate

ManagingUsers

387

updateauser Administrate

updateuseraccesslevel Administrate

dropauser Administrate

shutdownserver Administrate

Toperformactionsinaspecificdatabase(likecreatingordroppingcollections),auserneedsatleastthefollowingaccesslevel.ThepossibleaccesslevelsfordatabasesareAdministrate,AccessandNoaccess.TheaccesslevelsforcollectionsareRead/Write,ReadOnlyandNoAccess.

databaseaction databaselevel collectionlevel

createcollection Administrate Read/Write

listcollections Access ReadOnly

renamecollection Administrate Read/Write

modifycollectionproperties Administrate Read/Write

readproperties Access ReadOnly

dropcollection Administrate Read/Write

createanindex Administrate Read/Write

dropanindex Administrate Read/Write

seeindexdefinition Access ReadOnly

NotethattheaccesslevelAccessforadatabaseisalwaysrequiredtoperformanyactiononacollectioninthatdatabase.

Forcollectionsauserneedsthefollowingaccesslevelstothegivendatabaseandthegivencollection.TheaccesslevelsforthedatabaseareAdministrate,AccessandNoaccess.TheaccesslevelsforthecollectionareRead/Write,ReadOnlyandNoAccess.

action collectionlevel databaselevel

readadocument Read/WriteorReadOnly AdministrateorAccess

createadocument Read/Write AdministrateorAccess

modifyadocument Read/Write AdministrateorAccess

dropadocument Read/Write AdministrateorAccess

truncateacollection Read/Write AdministrateorAccess

Example

Forexample,given

adatabaseexampleacollectiondatainthedatabaseexampleauserJohnSmith

IftheuserJohnSmithisassignedtheaccesslevelAccessforthedatabaseexampleandthelevelRead/Writeforthecollectiondata,thentheuserisallowedtoread,create,modifyordeletedocumentsinthecollectiondata.Buttheuseris,forexample,notallowedtocreateindexesforthecollectiondatanorcreatenewcollectionsinthedatabaseexample.

GrantingAccessLevels

Accesslevelscanbemanagedviathewebinterfaceorinarangosh.

Inordertograntanaccessleveltoauser,youcanassignoneofthreeaccesslevelsforeachdatabaseandoneofthreelevelsforeachcollectioninadatabase.Theserveraccesslevelfortheuserfollowsfromthedatabaseaccesslevelinthe_systemdatabase,itisAdministrateifandonlyifthedatabaseaccesslevelisAdministrate.NotethatthismeansthatdatabaseaccesslevelAccessdoesnotgrantauserserveraccesslevelAdministrate.

ManagingUsers

388

InitialAccessLevels

WhenausercreatesadatabasetheaccessleveloftheuserforthatdatabaseissettoAdministrate.Thesameistrueforcreatingacollection,inthiscasetheusergetRead/Writeaccesstothecollection.

WildcardDatabaseAccessLevel

Withtheabovedefinition,onemustdefinethedatabaseaccesslevelforalldatabase/userpairsintheserver,whichwouldbeverytedious.Inordertosimplifythisprocess,itispossibletodefine,forauser,awildcarddatabaseaccesslevel.Thiswildcardisusedifthedatabaseaccesslevelisnotexplicitlydefinedforacertaindatabase.EachnewcreateduserhasaninitialdatabasewildcardofNoAccess.

Changingthewildcarddatabaseaccesslevelforauserwillchangetheaccesslevelforalldatabasesthathavenoexplicitlydefinedaccesslevel.Notethatthisincludesdatabaseswhichwillbecreatedinthefutureandforwhichnoexplicitaccesslevelsaresetforthatuser!

Ifyoudeletethewildcard,thedefaultaccesslevelisdefinedasNoAccess.

TherootuserhasaninitialdatabasewildcardofAdministrate.

Example

AssumeuserJohnSmithhasthefollowingdatabaseaccesslevels:

accesslevel

database* Access

databaseshop1 Administrate

databaseshop2 NoAccess

ThiswillgivetheuserJohnSmiththefollowingdatabaselevelaccess:

databaseshop1:Administratedatabaseshop2:NoAccessdatabasesomething:Access

Ifthewildcard*ischangedfromAccesstoNoAccessthenthepermissionswillchangeasfollows:

databaseshop1:Administratedatabaseshop2:NoAccessdatabasesomething:NoAccess

WildcardCollectionAccessLevel

Foreachuseranddatabasethereisawildcardcollectionaccesslevel.Thislevelisusedforallcollectionspairswithoutanexplicitlydefinedcollectionaccesslevel.Notethatthisincludescollectionswhichwillbecreatedinthefutureandforwhichnoexplicitaccesslevelsaresetforathatuser!EachnewcreateduserhasaninitialcollectionwildcardofNoAccess.

Ifyoudeletethewildcard,thesystemdefaultstoNoAccess.

TherootuserhasaninitialcollectionwildcardofRead/Writeineverydatabase.

Whencreatingauserthroughdb._createDatabase(name,options,users)theaccessleveloftheuserforthisdatabasewillbesettoAdministrateandthewildcardforallcollectionswithinthisdatabasewillbesettoRead/Write.

Example

AssumeuserJohnSmithhasthefollowingdatabaseaccesslevels:

accesslevel

database* Access

andcollectionaccesslevels:

accesslevel

ManagingUsers

389

database*,collection* Read/Write

databaseshop1,collectionproducts Read-Only

databaseshop1,collection* NoAccess

databaseshop2,collection* Read-Only

Thentheuserdoewillgetthefollowingcollectionaccesslevels:

databaseshop1,collectionproducts:Read-Onlydatabaseshop1,collectioncustomers:NoAccessdatabaseshop2,collectionreviews:Read-Onlydatabasesomething,collectionelse:Read/Write

Explanation:

Databaseshop1,collectionproductsdirectlymatchesadefinedaccesslevel.ThislevelisdefinedasRead-Only.

Databaseshop1,collectioncustomersdoesnotmatchadefinedaccesslevel.However,databaseshop1matchesandthewildcardinthisdatabaseforcollectionlevelisNoAccess.

Databaseshop2,collectionreviewsdoesnotmatchadefinedaccesslevel.However,databaseshop2matchesandthewildcardinthisdatabaseforcollectionlevelisRead-Only.

Databasesomehing,collectionelsedoesnotmatchadefinedaccesslevel.Thedatabasesomethingalsodoeshaveadirectmatches.Thereforethewildcardisselected.ThelevelisRead/Write.

SystemCollections

Theaccesslevelforsystemcollectionscannotbechanged.Theyfollowdifferentrulesthanuserdefinedcollectionsandmaychangewithoutfurthernotice.Currentlythesystemcollectionsfollowtheserules:

collection accesslevel

_users(in_system) NoAccess

_queues Read-Only

_frontend Read/Write

* sameasdb

AllothersystemcollectionshaveaccesslevelRead/WriteiftheuserhasAdministrateaccesstothedatabase.TheyhaveaccesslevelRead/OnlyiftheuserhasAccesstothedatabase.

TomodifythesesystemcollectionsyoushouldalwaysusethespecializedAPIsprovidedbyArangoDB.Forexamplenouserhasaccesstothe_userscollectioninthe_systemdatabase.Allchangestotheaccesslevelsmustbedoneusingthe@arangodb/usersmodule,the/_users/APIorthewebinterface.

ManagingUsers

390

ManagingUsersintheArangoDBShellPleasenote,thatforbackwardcompatibilitytheserveraccesslevelsfollowfromthedatabaseaccesslevelonthedatabase_system.

Alsonotethattheserveranddatabaseaccesslevelsarerepresentedas

rw:forAdministratero:forAccessnone:forNoaccess

Thisisagainforbackwardcompatibility.

Example

Fireuparangoshandrequiretheusersmodule.Useittocreateanewuser:

arangosh>varusers=require('@arangodb/users');

arangosh>users.save('JohnSmith','mypassword');

CreatesausercalledJohnSmith.Thisuserwillhavenoaccessatall.

arangosh>users.grantDatabase('JohnSmith','testdb','rw');

ThisgrantstheuserAdministrateaccesstothedatabasetestdb.revokeDatabasewillrevokethisaccesslevelsetting.

Note:Beawarethatfrom3.2onwardsthegrantDatabasewillnotautomaticallygrantuserstheaccessleveltowriteorreadcollectionsinadatabase.IfyougrantaccesstoadatabasetestdbyouwilladditionallyneedtoexplicitlygrantaccesslevelstoindividualcollectionsviagrantCollection.

Theupgradeprocedurefrom3.1to3.2setsthewildcarddatabaseaccesslevelforalluserstoAdministrateandsetsthewildcardcollectionaccesslevelforalluser/databasepairstoRead/Write.

arangosh>users.grantCollection('JohnSmith','testdb','testcoll','rw');

Saveusers.save(user,passwd,active,extra)

ThiswillcreateanewArangoDBuser.Theusernamemustbespecifiedinuserandmustnotbeempty.

Thepasswordmustbegivenasastring,too,butcanbeleftemptyifrequired.IfyoupassthespecialvalueARANGODB_DEFAULT_ROOT_PASSWORD,thepasswordwillbesetthevaluestoredintheenvironmentvariableARANGODB_DEFAULT_ROOT_PASSWORD.ThiscanbeusedtopassaninstancevariableintoArangoDB.Forexample,theinstanceidentifierfromAmazon.

Iftheactiveattributeisnotspecified,itdefaultstotrue.Theextraattributecanbeusedtosavecustomdatawiththeuser.

Thismethodwillfailifeithertheusernameorthepasswordsarenotspecifiedorgiveninawrongformat,ortherealreadyexistsauserwiththespecifiedname.

Note:Theuserwillnothavepermissiontoaccessanydatabase.YouneedtogranttheaccessrightsforoneormoredatabasesusinggrantDatabase.

Examples

arangosh>require('@arangodb/users').save('my-user','my-secret-password');

showexecutionresults

InArangosh

391

GrantDatabaseusers.grantDatabase(user,database,type)

Thisgrantstype('rw','ro'or'none')accesstothedatabasefortheuser.Ifdatabaseis"*",thissetsthewildcarddatabaseaccesslevelfortheuseruser.

Theserveraccesslevelfollowsfromtheaccesslevelforthedatabase_system.

RevokeDatabaseusers.revokeDatabase(user,database)

Thisclearstheaccesslevelsettingtothedatabasefortheuserandthewildcarddatabaseaccesssettingforthisuserkicksin.IncasenowildcardaccesswasdefinedthedefaultisNoAccess.Thiswillalsocleartheaccesslevelsforallthecollectionsinthisdatabase.

GrantCollectionusers.grantCollection(user,database,collection,type)

Thisgrantstype('rw','ro'or'none')accessleveltothecollectionindatabasefortheuser.Ifcollectionis"*"thissetsthewildcardcollectionaccesslevelfortheuseruserindatabasedatabase.

RevokeCollectionusers.revokeCollection(user,database)

Thisclearstheaccesslevelsettingtothecollectioncollectionfortheuseruser.ThesystemwilleitherfallbacktothewildcardcollectionaccesslevelordefaulttoNoAccess

Replaceusers.replace(user,passwd,active,extra)

ThiswilllookupanexistingArangoDBuserandreplaceitsuserdata.

Theusernamemustbespecifiedinuser,andauserwiththespecifiednamemustalreadyexistinthedatabase.

Thepasswordmustbegivenasastring,too,butcanbeleftemptyifrequired.

Iftheactiveattributeisnotspecified,itdefaultstotrue.Theextraattributecanbeusedtosavecustomdatawiththeuser.

Thismethodwillfailifeithertheusernameorthepasswordsarenotspecifiedorgiveninawrongformat,orifthespecifiedusercannotbefoundinthedatabase.

Note:thisfunctionwillnotworkfromwithinthewebinterface

Examples

arangosh>require("@arangodb/users").replace("my-user","my-changed-password");

showexecutionresults

Updateusers.update(user,passwd,active,extra)

ThiswillupdateanexistingArangoDBuserwithanewpasswordandotherdata.

Theusernamemustbespecifiedinuserandtheusermustalreadyexistinthedatabase.

InArangosh

392

Thepasswordmustbegivenasastring,too,butcanbeleftemptyifrequired.

Iftheactiveattributeisnotspecified,thecurrentvaluesavedfortheuserwillnotbechanged.Thesameistruefortheextraattribute.

Thismethodwillfailifeithertheusernameorthepasswordsarenotspecifiedorgiveninawrongformat,orifthespecifiedusercannotbefoundinthedatabase.

Examples

arangosh>require("@arangodb/users").update("my-user","my-secret-password");

showexecutionresults

isValidusers.isValid(user,password)

Checkswhetherthegivencombinationofusernameandpasswordisvalid.Thefunctionwillreturnabooleanvalueifthecombinationofusernameandpasswordisvalid.

Eachcalltothisfunctionispenalizedbytheserversleepingarandomamountoftime.

Examples

arangosh>require("@arangodb/users").isValid("my-user","my-secret-password");

true

Removeusers.remove(user)

RemovesanexistingArangoDBuserfromthedatabase.

TheusernamemustbespecifiedinUserandthespecifiedusermustexistinthedatabase.

Thismethodwillfailiftheusercannotbefoundinthedatabase.

Examples

arangosh>require("@arangodb/users").remove("my-user");

Documentusers.document(user)

FetchesanexistingArangoDBuserfromthedatabase.

Theusernamemustbespecifiedinuser.

Thismethodwillfailiftheusercannotbefoundinthedatabase.

Examples

arangosh>require("@arangodb/users").document("my-user");

showexecutionresults

All

InArangosh

393

users.all()

FetchesallexistingArangoDBusersfromthedatabase.

Examples

arangosh>require("@arangodb/users").all();

showexecutionresults

Reloadusers.reload()

Reloadstheuserauthenticationdataontheserver

Alluserauthenticationdataisloadedbytheserveronceonstartuponlyandiscachedafterthat.Whenusersgetaddedordeleted,acacheflushisdoneautomatically,andthiscanbeperformedbyacalltothismethod.

Examples

arangosh>require("@arangodb/users").reload();

Permissionusers.permission(user,database[,collection])

Fetchestheaccessleveltothedatabaseoracollection.

Theuseranddatabasenamemustbespecified,optionallyyoucanspecifythecollectionname.

Thismethodwillfailiftheusercannotbefoundinthedatabase.

Examples

arangosh>require("@arangodb/users").permission("my-user","testdb");

rw

InArangosh

394

Command-lineoptions

GeneralOptions

Generalhelp--help

-h

Printsalistofthemostcommonoptionsavailableandthenexits.Inordertoseealloptionsuse--help-all.

Version--version

-v

Printstheversionoftheserverandexits.

ConfigurationFiles

Optionscanbespecifiedonthecommandlineorinconfigurationfiles.IfastringVariableoccursinthevalue,itisreplacedbythecorrespondingenvironmentvariable.

--configurationfilename

-cfilename

Specifiesthenameoftheconfigurationfiletouse.

Ifthiscommandisnotpassedtotheserver,thenbydefault,theserverwillattempttofirstlocateafilenamed~/.arango/arangod.confintheuser'shomedirectory.

Ifnosuchfileisfound,theserverwillproceedtolookforafilearangod.confinthesystemconfigurationdirectory.Thesystemconfigurationdirectoryisplatform-specific,andmaybechangedwhencompilingArangoDByourself.Itmaydefaultto/etc/arangodbor/usr/local/etc/arangodb.Thisfileisinstalledwhenusingapackagemanagerlikerpmordpkg.IfyoumodifythisfileandlaterupgradetoanewversionofArangoDB,thenthepackagemanagernormallywarnsyouabouttheconflict.Inordertoavoidthesewarningforsmalladjustments,youcanputlocaloverridesintoafilearangod.conf.local.

Onlycommandlineoptionswithavalueshouldbesetwithintheconfigurationfile.Commandlineoptionswhichactasflagsshouldbeenteredonthecommandlinewhenstartingtheserver.

Eachoptionisspecifiedonaseparatelineintheform:

key=value

Alternatively,aheadersectioncanbespecifiedandoptionspertainingtothatsectioncanbespecifiedinashorterform

[log]

level=trace

ratherthanspecifying

log.level=trace

Soyouseeingeneral--section.paramvaluetranslatesto

[section]

param=value

ServerConfiguration

395

Whitespacearound=isignoredintheconfigurationfile.Donotputspacesaroundadditional=intheparametervaluehowever.Thefollowingexampleshowsthecorrectwaytospecifyalogleveloftraceforthetopicstartup:

log.level=startup=trace

Notethatthereisnowhitespacebetweenstartupand=,andalsonot=andtrace.

Whereonesectionmayoccurmultipletimes,andthelastoccuranceofparamwillbecomethefinalvalue.Incaseofparametersbeingvectors,multipleoccuranceaddsanotheritemtothevector.Vectorscanbeidentifiedbythe...inthe--helpoutputofthebinaries.

Commentscanbeplacedintheconfigurationfile,onlyifthelinebeginswithoneormorehashsymbols(#).

Theremaybeoccasionswhereaconfigurationfileexistsandtheuserwishestooverrideconfigurationsettingsstoredinaconfigurationfile.Anysettingsspecifiedonthecommandlinewilloverwritethesamesettingwhenitappearsinaconfigurationfile.Iftheuserwishestocompletelyignoreconfigurationfileswithoutnecessarilydeletingthefile(orfiles),thenaddthecommandlineoption

-cnone

or

--configurationnone

whenstartinguptheserver.Notethat,thewordnoneiscase-insensitive.

ServerConfiguration

396

OperatingSystemConfiguration

FileSystems

(LINUX)

WerecommendtonotuseBTRFSonlinux,it'sknowntonotworkwellinconjunctionwithArangoDB.WeexperiencedthatarangodbfacinglatencyissuesonaccessingitsdatabasefilesonBTRFSpartitions.InconjunctionwithBTRFSandAUFSwealsosawdatalossonrestart.

VirtualMemoryPageSizes(LINUX)

Bydefault,ArangoDBusesJemallocasthememoryallocator.Jemallocdoesagoodjobofreducingvirtualmemoryfragmentation,especiallyforlong-runningprocesses.Unfortunately,someOSconfigurationscaninterferewithJemalloc'sabilitytofunctionproperly.Specifically,Linux's"transparenthugepages",Windows'"largepages"andothersimilarfeaturessometimespreventJemallocfromreturningunusedmemorytotheoperatingsystemandresultinunnecessarilyhighmemoryuse.Therefore,werecommenddisablingthesefeatureswhenusingJemallocwithArangoDB.Pleaseconsultyouroperatingsystem'sdocumentationforhowtodothis.

Execute

sudobash-c"echomadvise>/sys/kernel/mm/transparent_hugepage/enabled"

sudobash-c"echomadvise>/sys/kernel/mm/transparent_hugepage/defrag"

beforeexecutingarangod.

SwapSpace

(LINUX)

Itisrecommendedtoassignswapspaceforaserverthatisrunningarangod.Configuringswapspacecanpreventtheoperatingsystem'sOOMkillerfromkillingArangoDBtooeagerlyonLinux.

Over-CommitMemory

FortheMMFilesstorageengine,execute

sudobash-c"echo0>/proc/sys/vm/overcommit_memory"

beforeexecutingarangod.

FortheRocksDBstorageengine,execute

sudobash-c"echo2>/proc/sys/vm/overcommit_memory"

beforestarting.

Fromwww.kernel.org:

Whenthisflagis0,thekernelattemptstoestimatetheamountoffreememoryleftwhenuserspacerequestsmorememory.

Whenthisflagis1,thekernelpretendsthereisalwaysenoughmemoryuntilitactuallyrunsout.

Whenthisflagis2,thekernelusesa"neverovercommit"policythatattemptstopreventanyovercommitofmemory.

OperatingSystemConfiguration

397

Notethatthenusinganovercommit_memorysettingof2,thiswillbydefaultallowprocessestouseallswapspacebutonlyhalfoftheavailableRAM.Thiscanbechangedbyadjustingthevalueofovercommit_ratioaswell.

Fromwww.kernel.org:

Whenovercommit_memoryissetto2,thecommittedaddressspaceisnotpermittedtoexceedswapplusthispercentageofphysicalRAM.

ZoneReclaim

Execute

sudobash-c"echo0>/proc/sys/vm/zone_reclaim_mode"

beforeexecutingarangod.

Fromwww.kernel.org:

ThisisvalueORedtogetherof

1=Zonereclaimon2=Zonereclaimwritesdirtypagesout4=Zonereclaimswapspages

NUMAMulti-processorsystemsoftenhavenon-uniformAccessMemory(NUMA).ArangoDBshouldbestartedwithinterleaveonsuchsystem.Thiscanbeachievedusing

numactl--interleave=allarangod...

MaxMemoryMappings(LINUX)

Linuxkernelsbydefaultrestrictthemaximumnumberofmemorymappingsofasingleprocesstoabout64Kmappings.Whilethisvalueissufficientformostworkloads,itmaybetoolowforaprocessthathaslotsofparallelthreadsthatallrequiretheirownmemorymappings.Inthiscaseallthethreads'memorymappingswillbeaccountedtothesinglearangodprocess,andthemaximumnumberof64Kmappingsmaybereached.Whenthemaximumnumberofmappingsisreached,callstommapwillfail,sotheprocesswillthinknomorememoryisavailablealthoughtheremaybeplentyofRAMleft.

Toavoidthisscenario,itisrecommendedtoraisethedefaultvalueforthemaximumnumberofmemorymappingstoasufficientlyhighvalue.Asaruleofthumb,onecoulduse8timesthenumberofavailablecorestimes8,000.

Fora32coreserver,agoodrule-of-thumbvaluethuswouldbe2,048,000(3288000).Tosetthevalueonce,usethefollowingcommandbeforestartingarangod:

sudobash-c"sysctl-w'vm.max_map_count=2048000'"

Tomakethesettingsdurable,itwillbenecessarytostoretheadjustedsettingsin/etc/sysctl.conforotherplacesthattheoperatingsystemislookingat.

EnvironmentVariables(LINUX)

ItisrecommendedtosettheenvironmentvariableGLIBCXX_FORCE_NEWto1onsystemsthatuseglibc++inordertodisablethememorypoolingbuiltintoglibc++.ThatmemorypoolingisunnecessarybecauseJemallocwillalreadydomemorypooling.

OperatingSystemConfiguration

398

Execute

exportGLIBCXX_FORCE_NEW=1

beforestartingarangod.

32bitWhileitispossibletocompileArangoDBon32bitsystem,thisisnotarecommendedenvironment.64bitsystemscanaddressasignificantlybiggermemoryregion.

OperatingSystemConfiguration

399

ManagingEndpointsTheArangoDBservercanlistenforincomingrequestsonmultipleendpoints.

TheendpointsarenormallyspecifiedeitherinArangoDB'sconfigurationfileoronthecommand-line,usingthe--server.endpoint.ArangoDBsupportsdifferenttypesofendpoints:

tcp://ipv4-address:port-TCP/IPendpoint,usingIPv4tcp://[ipv6-address]:port-TCP/IPendpoint,usingIPv6ssl://ipv4-address:port-TCP/IPendpoint,usingIPv4,SSLencryptionssl://[ipv6-address]:port-TCP/IPendpoint,usingIPv6,SSLencryptionunix:///path/to/socket-Unixdomainsocketendpoint

IfaTCP/IPendpointisspecifiedwithoutaportnumber,thenthedefaultport(8529)willbeused.Ifmultipleendpointsneedtobeused,theoptioncanberepeatedmultipletimes.

ThedefaultendpointforArangoDBistcp://127.0.0.1:8529ortcp://localhost:8529.

EXAMPLES

unix>./arangod--server.endpointtcp://127.0.0.1:8529

--server.endpointssl://127.0.0.1:8530

--ssl.keyfileserver.pem/tmp/vocbase

2012-07-26T07:07:47Z[8161]INFOusingSSLprotocolversion'TLSv1'

2012-07-26T07:07:48Z[8161]INFOusingendpoint'ssl://127.0.0.1:8530'forhttpsslrequests

2012-07-26T07:07:48Z[8161]INFOusingendpoint'tcp://127.0.0.1:8529'forhttptcprequests

2012-07-26T07:07:49Z[8161]INFOArangoDB(version1.1.alpha)isreadyforbusiness

2012-07-26T07:07:49Z[8161]INFOHaveFun!

TCPEndpoints

Givenahostname:

--server.endpointtcp://hostname:port

GivenanIPv4address:

--server.endpointtcp://ipv4-address:port

GivenanIPv6address:

--server.endpointtcp://[ipv6-address]:port

Ononespecificethernetinterfaceeachportcanonlybeboundexactlyonce.YoucanlookupyouravailableinterfacesusingtheifconfigcommandonLinux/MacOSX-theWindowsequivalentisipconfig(SeeWikipediaformoredetails).ThegeneralnamesoftheinterfacesdifferonOS'sandhardwarestheyrunon.However,typicallyeveryhosthasasocalledloopbackinterface,whichisavirtualinterface.Byconventionitalwayshastheaddress127.0.0.1or::1(ipv6),andcanonlybereachedfromexactlytheverysamehost.Ethernetinterfacesusuallyhavenameslikeeth0,wlan0,eth1:17,le0oraplaintextnameinWindows.

Tofindoutwhichservicesalreadyuseports(soArangoDBcan'tbindthemanymore),youcanusethenetstatcommand(itbehavesalittledifferentoneachplatform,runitwith-lnptonLinux,-ptcponMacOSXorwith-anonwindowsforvaluableinformation).

ArangoDBcanalsodoasocalledbroadcastbindusingtcp://0.0.0.0:8529.Thiswayitwillbereachableonallinterfacesofthehost.Thismaybeusefulondevelopmentsystemsthatfrequentlychangetheirnetworksetuplikelaptops.

SpecialnoteonIPv6link-localaddresses

ArangoDBcanalsolistentoIPv6link-localaddressesviaaddingthezoneIDtotheIPv6addressintheform[ipv6-link-local-address%zone-id].However,whatyouprobablyinsteadwantistobindtoalocalIPv6address.LocalIPv6addressesstartwithfd.Ifyouonlyseeafe80:IPv6addressinyourinterfaceconfigurationbutnoIPv6addressstartingwithfdyourinterfacehasnolocalIPv6addressassigned.YoucanreadmoreaboutIPv6link-localaddresseshere.

Example

ManagingEndpoints

400

Bindtoalink-localandlocalIPv6address.

unix>ifconfig

Thiscommandlistsallinterfacesandassignedipaddresses.Thelink-localaddressmaybefe80::6257:18ff:fe82:3ec6%eth0(IPv6addressplusinterfacename).AlocalIPv6addressmaybefd12:3456::789a.TobindArangoDBtoitstartarangodwith--server.endpointtcp://[fe80::6257:18ff:fe82:3ec6%eth0]:8529.Usetelnettotesttheconnection.

unix>telnetfe80::6257:18ff:fe82:3ec6%eth08529

Tryingfe80::6257:18ff:fe82:3ec6...

Connectedtomy-machine.

Escapecharacteris'^]'.

GET/HTTP/1.1

HTTP/1.1301MovedPermanently

Location:/_db/_system/_admin/aardvark/index.html

Content-Type:text/html

Server:ArangoDB

Connection:Keep-Alive

Content-Length:197

<html><head><title>Moved</title></head><body><h1>Moved</h1><p>Thispagehasmovedto<ahref="/_db/_system/_admin/aardvark/inde

x.html">/_db/_system/_admin/aardvark/index.html</a>.</p></body></html>

Reuseaddress--tcp.reuse-address

IfthisbooleanoptionissettotruethenthesocketoptionSO_REUSEADDRissetonallserverendpoints,whichisthedefault.Ifthisoptionissettofalseitispossiblethatittakesuptoaminuteafteraserverhasterminateduntilitispossibleforanewservertousethesameendpointagain.Thisiswhythisisactivatedbydefault.

Pleasenotehoweverthatundersomeoperatingsystemsthiscanbeasecurityriskbecauseitmightbepossibleforanotherprocesstobindtothesameaddressandport,possiblyhijackingnetworktraffic.UnderWindows,ArangoDBadditionallysetstheflagSO_EXCLUSIVEADDRUSEasameasuretoalleviatethisproblem.

Backlogsize--tcp.backlog-size

AllowstospecifythesizeofthebacklogforthelistensystemcallThedefaultvalueis10.Themaximumvalueisplatform-dependent.Specifyingahighervaluethandefinedinthesystemheader'sSOMAXCONNmayresultinawarningonserverstart.Theactualvalueusedbylistenmayalsobesilentlytruncatedonsomeplatforms(thishappensinsidethelistensystemcall).

ManagingEndpoints

401

SSLConfiguration

SSLEndpoints

Givenahostname:

--server.endpointtcp://hostname:port

GivenanIPv4address:

--server.endpointtcp://ipv4-address:port

GivenanIPv6address:

--server.endpointtcp://[ipv6-address]:port

Note:IfyouareusingSSL-encryptedendpoints,youmustalsosupplythepathtoaservercertificateusingthe--ssl.keyfileoption.

Keyfile--ssl.keyfilefilename

IfSSLencryptionisused,thisoptionmustbeusedtospecifythefilenameoftheserverprivatekey.ThefilemustbePEMformattedandcontainboththecertificateandtheserver'sprivatekey.

Thefilespecifiedbyfilenamecanbegeneratedusingopenssl:

#createprivatekeyinfile"server.key"

opensslgenrsa-des3-outserver.key1024

#createcertificatesigningrequest(csr)infile"server.csr"

opensslreq-new-keyserver.key-outserver.csr

#copyawayoriginalprivatekeyto"server.key.org"

cpserver.keyserver.key.org

#removepassphrasefromtheprivatekey

opensslrsa-inserver.key.org-outserver.key

#signthecsrwiththekey,createscertificatePEMfile"server.crt"

opensslx509-req-days365-inserver.csr-signkeyserver.key-outserver.crt

#combinecertificateandkeyintosinglePEMfile"server.pem"

catserver.crtserver.key>server.pem

YoumayusecertificatesissuedbyaCertificateAuthorityorself-signedcertificates.Self-signedcertificatescanbecreatedbyatoolofyourchoice.WhenusingOpenSSLforcreatingtheself-signedcertificate,thefollowingcommandsshouldcreateavalidkeyfile:

-----BEGINCERTIFICATE-----

(base64encodedcertificate)

-----ENDCERTIFICATE-----

-----BEGINRSAPRIVATEKEY-----

(base64encodedprivatekey)

-----ENDRSAPRIVATEKEY-----

Forfurtherinformationpleasecheckthemanualsofthetoolsyouusetocreatethecertificate.

CAFile--ssl.cafilefilename

SSLConfiguration

402

ThisoptioncanbeusedtospecifyafilewithCAcertificatesthataresenttotheclientwhenevertheserverrequestsaclientcertificate.Ifthefileisspecified,TheserverwillonlyacceptclientrequestswithcertificatesissuedbytheseCAs.Donotspecifythisoptionifyouwantclientstobeabletoconnectwithoutspecificcertificates.

ThecertificatesinfilenamemustbePEMformatted.

SSLprotocol

--ssl.protocolvalue

Usethisoptiontospecifythedefaultencryptionprotocoltobeused.Thefollowingvariantsareavailable:

1:SSLv22:SSLv2orSSLv3(negotiated)3:SSLv34:TLSv15:TLSv1.2

Thedefaultvalueis5(TLSv1.2).

SSLcache--ssl.session-cachevalue

SettotrueifSSLsessioncachingshouldbeused.

valuehasadefaultvalueoffalse(i.e.nocaching).

SSLpeercertificate

ThisfeatureisavailableintheEnterpriseEdition.

--ssl.require-peer-certificate

Requireapeercertificatefromtheclientbeforeconnecting.

SSLoptions

--ssl.optionsvalue

ThisoptioncanbeusedtosetvariousSSL-relatedoptions.IndividualoptionvaluesmustbecombinedusingbitwiseOR.

WhichoptionsareavailableonyourplatformisdeterminedbytheOpenSSLversionyouuse.Thelistofoptionsavailableonyourplatformmightberetrievedbythefollowingshellcommand:

>grep"#defineSSL_OP_.*"/usr/include/openssl/ssl.h

#defineSSL_OP_MICROSOFT_SESS_ID_BUG0x00000001L

#defineSSL_OP_NETSCAPE_CHALLENGE_BUG0x00000002L

#defineSSL_OP_LEGACY_SERVER_CONNECT0x00000004L

#defineSSL_OP_NETSCAPE_REUSE_CIPHER_CHANGE_BUG0x00000008L

#defineSSL_OP_SSLREF2_REUSE_CERT_TYPE_BUG0x00000010L

#defineSSL_OP_MICROSOFT_BIG_SSLV3_BUFFER0x00000020L

...

AdescriptionoftheoptionscanbefoundonlineintheOpenSSLdocumentation

SSLcipher--ssl.cipher-listcipher-list

ThisoptioncanbeusedtorestricttheservertocertainSSLciphersonly,andtodefinetherelativeusagepreferenceofSSLciphers.

Theformatofcipher-listisdocumentedintheOpenSSLdocumentation.

Tocheckwhichciphersareavailableonyourplatform,youmayusethefollowingshellcommand:

SSLConfiguration

403

>opensslciphers-v

ECDHE-RSA-AES256-SHASSLv3Kx=ECDHAu=RSAEnc=AES(256)Mac=SHA1

ECDHE-ECDSA-AES256-SHASSLv3Kx=ECDHAu=ECDSAEnc=AES(256)Mac=SHA1

DHE-RSA-AES256-SHASSLv3Kx=DHAu=RSAEnc=AES(256)Mac=SHA1

DHE-DSS-AES256-SHASSLv3Kx=DHAu=DSSEnc=AES(256)Mac=SHA1

DHE-RSA-CAMELLIA256-SHASSLv3Kx=DHAu=RSAEnc=Camellia(256)

Mac=SHA1

...

Thedefaultvalueforcipher-listis"ALL".

SSLConfiguration

404

LDAPThisfeatureisavailableintheEnterpriseEdition.

BasicsConcepts

Therearetwomodesofoperation:simpleauthandbind+search.

ThebasicoptionsforspecifyinghowtoaccesstheLDAPserverare--ldap.enabled,--ldap.tls,--ldap.port,--ldap.server.--ldap.serverand--ldap.portcanbereplacedby--ldap.url.Thedefaultfor--ldap.portis389.

Simpleauth

ArangoDBconnectstotheldapserverandauthenticateswiththeusernameandpasswordprovidedbytheAPIauthenticationrequest.IftheLDAPserversuccessfullyverifiesthepasswordthentheuserisauthenticated.

If--ldap.prefixand/or--ldap.suffixisprovided,thenthesimplemodeisselected.

Inordertoauthorizetheuserforoneormoredatabasestherearetwomodesofoperation:databaseattributeorroles.

Databaseattribute

Inthismode,anLDAPattributeoftheuserisusedtospecifytheaccesslevelswithinLDAP.Thedatabase/collectionaccesslevelsinArangoDBarenotused.

--ldap.permissions-attribute-namehastheformatdatabase-name=(*|rw|none)[,database-name=(*|rw|none)].

Example:

--ldap.enabledtrue--ldap.serverldap.company.com\

--ldap.permissions-attribute-namearangodbPermissions\

--ldap.prefixuid=\

--ldap.suffix,dc=company,dc=com

--ldap.prefixand--ldap.suffixbuildthedistinguishedname(DN).ArangoDBtriestoauthenticatewithprefix+ArangoDBusername+suffixagainsttheLDAPserverandsearchesforthedatabasepermissions.

dn:uid=fermi,dc=example,dc=com

arangodbPermissions:foo=none,bar=rw

ThiswillgiveAdministrateaccesstobarandNoAcesstofoo.Notethatthismethodsonlyallowstospecifydatabaseaccesslevels,notcollectionaccesslevels.

Roles

Inthismode,anLDAPattributeoftheuserisusedtospecifyoneormorerolesforthatusers.Thedatabase/collectionaccesslevelsfortheserolesdefinedinArangoDBarethenused.

Example:

--ldap.enabledtrue--ldap.serverldap.company.com\

--ldap.roles-attribute-namegroupMembership\

--ldap.prefixuid=\

--ldap.suffix,dc=company,dc=com

--ldap.prefixand--ldap.suffixbuildthedistinguishedname(DN).ArangoDBtrystoauthenticatewithprefix+ArangoDBusername+suffixagainsttheldapserverandsearchesfortherolesintheattributegroupMembership.

dn:uid=fermi,dc=example,dc=com

LDAPOptions

405

groupMembership:project-a

groupMembership:project-b

Thiswillgivethecombinedpermissionsoftherolesproject-aandproject-btotheuser.

Rolestransformationsandfilters

--ldap.roles-includecanbeusedtospecifyaregularexpressionthatisusedtofilterroles.Onlyrolesthatmatchtheregularexpressionareused.

--ldap.roles-excludecanbeusedtospecifyaregularexpressionthatisusedtofilterroles.Onlyrolesthatdonotmatchtheregularexpressionareused.

--ldap.roles-transformationcanbeusedtosepcifyaregularexpressionandreplacementtextas/re/text/.Thisregularexpressionisappliedtotherolenamefound.

--ldap.superuser-rolecanbeusedtospecifytheroleassociatedwiththesuperuser.Anyuserbelongingtothisrolegainssuperuserstatus.Thisroleischeckedbeforeapplyinganyregularexpression.

Example:

--ldap.enabledtrue--ldap.serverldap.company.com\

--ldap.roles-attribute-namegroupMembership\

--ldap.prefixuid=\

--ldap.suffix,dc=company,dc=com

--ldap.roles-include"^arangodb"

willonlyconsiderrolesthatstartwitharangodb.

--ldap.enabledtrue--ldap.serverldap.company.com\

--ldap.roles-attribute-namegroupMembership\

--ldap.prefixuid=\

--ldap.suffix,dc=company,dc=com

--ldap.roles-exclude"disabled"

willonlyconsiderrolesthatdocontaintheworddisabled.

--ldap.enabledtrue--ldap.serverldap.company.com\

--ldap.roles-attribute-namegroupMembership\

--ldap.prefixuid=\

--ldap.suffix,dc=company,dc=com

--ldap.superuser-role"arangodb-admin"

anyonebelongingtothegroup"arangodb-admin"willbecomeasuperuser.

bind+search

Examplewithanonymousauth:

--ldap.enabledtrue--ldap.serverldap.company.com\

--ldap.basedndc=company,dc=com\

--ldap.permissions-attribute-namearangodbPermissions

WiththisconfigurationArangoDBbindsanonymouslytotheLDAPserverandsearchesfortheuser.IftheuserisfoundaauthenticationisdonewiththeusersDNandpasswordandthendatabasepermissionsarefetched.

ExamplewithDNandpassword:

--ldap.enabledtrue--ldap.serverldap.company.com\

--ldap.basedndc=company,dc=com\

--ldap.binddncn=admin,dc=company,dc=com\

--ldap.bindpasswdadmin\

--ldap.permissions-attribute-namearangodbPermissions

LDAPOptions

406

WiththisconfigurationArangoDBbindswith--ldap.bindnand--ldap.bindpasswdtotheLDAPserverandsearchesfortheuser.IftheuserisfoundaauthenticationisdonewiththeusersDNandpasswordandthendatabasepermissionsarefetched.

Rolessearch

--ldap.roles-searchsearch-expression

Insteadofspecifyingarolesattributeitispossibletouseasearchwhenusingbind+search.Inthiscasethesearch-expressionmustbeanldapsearchstring.Any{USER}isreplacedbythednoftheuser.

Example:

--ldap.enabledtrue--ldap.serverldap.company.com\

--ldap.basedndc=company,dc=com\

--ldap.binddncn=admin,dc=company,dc=com\

--ldap.bindpasswdadmin\

--ldap.roles-search'(&(objectClass=groupOfUniqueNames)(uniqueMember={USER}))'

Additionaloptions

--ldap.search-filter"objectClass=*"

Restrictthesearchtospecificobjectclasses.ThedefaultisobjectClass=*.

--ldap.search-attribute"uid"

--ldap.search-attributespecifieswhichattributetocomparewiththeusername.Thedefaultisuid.

--ldap.search-scopesub

`--ldap.search-scopespecifiesinwhichscopetosearchforauser.Validareoneofbase,oneorsub.Thedefaultissub.

ldapurl

--ldap.urlldap://ldap.server.com:1234/dc=example,dc=com?uid?sub

Theldapurlconsistsoftheldapserverandport,abasedn,asearchattributeandascopewhichcanbeoneofbase,oneorsub.

TLSoptions

Aencryptedconnectioncanbeestablishedwith--ldap.tlstrueunderUNIXandGNU/Linuxplatforms.

AllfollowingoptionsarenotavailableunderWindows.

--ldap.tls

Thedefaultisfalse.Withtrueatlsconnectionisestablished.

--ldap.tls-version

Thedefaultis1.2.Availableversionsare1.0,1.1and1.2.

--ldap.tls-cert-check-strategy

Thedefaultishard.Availablestrategiesarenever,hard,demand,allowandtry.

--ldap.tls-cacert-file

LDAPOptions

407

Afilepathtooneormore(concatenated)certificateauthoritycertificatesinpemformat.Asdefaultnofilepathisconfigured.

Followingoptionhasnoeffect/doesnotworkundermacOS.

--ldap.tls-cacert-dir

Adirectorypathtocertificateauthoritycertificatesinc_rehashformat.Asdefaultnodirectorypathisconfigured.

LDAPOptions

408

Command-LineOptionsforLogging

Loglevelsandtopics

ArangoDB'slogoutputisgroupedintotopics.--log.levelcanbespecifiedmultipletimesatstartup,forasmanytopicsasneeded.Thelogverbosityandoutputfilescanbeadjustedperlogtopic.Forexample

--log.levelstartup=trace--log.levelqueries=trace--log.levelinfo

willlogmessagesconcerningstartupattracelevel,AQLqueriesattracelevelandeverythingelseatinfolevel.

Inaconfigurationfile,itiswrittenlikethis:

[log]

level=startup=trace

level=queries=trace

level=info

Notethattheremustnotbeanywhitespacearoundthesecond=.

Theavailableloglevelsare:

fatal:onlylogsfatalerrorserror:onlylogserrorswarning:onlylogswarningsanderrorsinfo:logsinformationmessages,warningsanderrorsdebug:logsdebugandinformationmessages,warningsanderrorstrace:logstrace,debugandinformationmessages,warningsanderrors

Notethatlevelsdebugandtracewillbeveryverbose.

SomerelevantlogtopicsavailableinArangoDB3are:

agency:informationabouttheagencycollector:informationabouttheWALcollector'sstatecompactor:informationaboutthecollectiondatafilecompactordatafiles:datafile-relatedoperationsmmap:informationaboutmemory-mappingoperations(includingmsync)performance:performance-releatedmessagesqueries:executedAQLqueries,slowqueriesreplication:replication-relatedinforequests:HTTPrequestsstartup:informationaboutserverstartupandshutdownthreads:informationaboutthreads

Logoutputs

Thelogoption--log.output<definition>allowsdirectingtheglobalorper-topiclogoutputtodifferentoutputs.Theoutputdefinition<definition>canbeoneof

-forstdin+forstderrsyslog://<syslog-facility>

syslog://<syslog-facility>/<application-name>

file://<relative-path>

Theoptioncanbespecifiedmultipletimesinordertoconfiguretheoutputfordifferentlogtopics.Tosetupaper-topicoutputconfiguration,use--log.output<topic>=<definition>,e.g.

LoggingOptions

409

queries=file://queries.txt

logsallqueriestothefile"queries.txt".

Theoldoption--log.fileisstillavailablein3.0forconveniencereasons.In3.0itisashortcutforthemoregeneraloption--log.outputfile://filename.

Theoldoption--log.requests-fileisstillavailablein3.0.Itisnowashortcutforthemoregeneraloption--log.outputrequests=file://....

Using--log.outputalsoallowsdirectinglogoutputtodifferentfilesbasedontopics.Forexample,tologallAQLqueriestoafile"queries.log"onecanusetheoptions:

--log.levelqueries=trace--log.outputqueries=file:///path/to/queries.log

ToadditionallylogHTTPrequesttoafilenamed"requests.log"addtheoptions:

--log.levelrequests=info--log.outputrequests=file:///path/to/requests.log

Forcingdirectoutput

Theoption--log.force-directcanbeusedtodisablelogginginanextraloggingthread.Ifsettotrue,anylogmessagesareimmediatelyprintedinthethreadthattriggeredthelogmessage.Thisisnon-optimalforperformancebutcanaiddebugging.Ifsettofalse,logmessagesarehandedofftoanextraloggingthread,whichasynchronouslywritesthelogmessages.

Localtime

Logdatesandtimesinlocaltimezone:--log.use-local-time

Ifspecified,alldatesandtimesinlogmessageswillusetheserver'slocaltime-zone.Ifnotspecified,alldatesandtimesinlogmessageswillbeprintedinUTC/Zulutime.ThedateandtimeformatusedinlogsisalwaysYYYY-MM-DDHH:MM:SS,regardlessofthissetting.IfUTCtimeisused,aZwillbeappendedtoindicateZulutime.

Colorlogging--log.colorvalue

Loggingtoterminaloutputisbydefaultcolored.Colorfulloggingcanbeturnedoffbysettingthevaluetofalse.

SourcefileandLinenumber

Loglinenumber:--log.line-number

Normally,ifanhumanreadablefatal,error,warningorinfomessageislogged,noinformationaboutthefileandlinenumberisprovided.Thefileandlinenumberisonlyloggedfordebugandtracemessage.Thisoptioncanbeusetoalwayslogthesepiecesofinformation.

Prefix

Logprefix:--log.prefixprefix

Thisoptionisusedspecifyanprefixtologgedtext.

Threads

Logthreadidentifier:--log.threadtrue

Wheneverlogoutputisgenerated,theprocessIDiswrittenaspartoftheloginformation.Settingthisoptionappendsthethreadidofthecallingthreadtotheprocessid.Forexample,

2010-09-20T13:04:01Z[19355]INFOreadyforbusiness

LoggingOptions

410

whennothreadisloggedand

2010-09-20T13:04:17Z[19371-18446744072487317056]readyforbusiness

whenthiscommandlineoptionisset.

Toalsologthreadnames,itispossibletosetthe--log.thread-nameoption.Bydefault--log.thread-nameissettofalse.

Role

Logrole:--log.roletrue

Whensettotrue,thisoptionwillmaketheArangoDBloggerprintasinglecharacterwiththeserver'sroleintoeachloggedmessage.Therolesare:

U:undefined/unclear(usedatstartup)S:singleserverC:coordinatorP:primaryA:agent

Thedefaultvalueforthisoptionisfalse,sonoroleswillbelogged.

LoggingOptions

411

GeneralOptions

DatabaseUpgrade

--database.auto-upgrade

Specifyingthisoptionwillmaketheserverperformadatabaseupgradeatstart.AdatabaseupgradewillfirstcomparetheversionnumberstoredinthefileVERSIONinthedatabasedirectorywiththecurrentserverversion.

Ifthetwoversionnumbersmatch,theserverwillstartnormally.

Iftheversionnumberfoundinthedatabasedirectoryishigherthantheversionnumbertheserverisrunning,theserverexpectsthisisanunintentionaldowngradeandwillwarnaboutthis.Itwillhoweverstartnormally.Usingtheserverintheseconditionsishowevernotrecommendednorsupported.

Iftheversionnumberfoundinthedatabasedirectoryislowerthantheversionnumbertheserverisrunning,theserverwillcheckwhetherthereareanyupgradetaskstoperform.Itwillthenexecuteallrequiredupgradetasksandprinttheirstatuses.Ifoneoftheupgradetasksfails,theserverwillexitandrefusetostart.Re-startingtheserverwiththeupgradeoptionwillthenagaintriggertheupgradecheckandexecutionuntiltheproblemisfixed.Ifalltasksarefinished,theserverwillstartnormally.

Whetherornotthisoptionisspecified,theserverwillalwaysperformaversioncheckonstartup.Runningtheserverwithanon-matchingversionnumberintheVERSIONfilewillmaketheserverrefusetostart.

StorageEngine

AsofArangoDB3.2twostorageenginesaresupported.The"traditional"engineiscalledMMFiles,whichisalsothedefaultstorageengine.

AnalternativeenginebasedonRocksDBisalsoprovidedandcanbeturnedonmanually.

Onestorageenginetypeissupportedperserverperinstallation.Liveswitchingofstorageenginesonalreadyinstalledsystemsisn'tsupported.Configuringthewrongengine(notmatchingthepreviouslyusedone)willresultintheserverrefusingtostart.YoumayhoweveruseautotoletArangoDBchoosethepreviouslyusedone.

--server.storage-engine[auto|mmfiles|rocksdb]

Daemon

--daemon

Runstheserverasadaemon(asabackgroundprocess).Thisparametercanonlybesetifthepid(processid)fileisspecified.Thatis,unlessavaluetotheparameterpid-fileisgiven,thentheserverwillreportanerrorandexit.

DefaultLanguage

--default-languagedefault-language

Thedefaultlanguageistusedforsortingandcomparingstrings.Thelanguagevalueisatwo-letterlanguagecode(ISO-639)oritiscomposedbyatwo-letterlanguagecodewithandatwolettercountrycode(ISO-3166).Validlanguagesare"de","en","en_US"or"en_UK".

Thedefaultdefault-languageissettobethesystemlocaleonthatplatform.

Supervisor--supervisor

Executestheserverinsupervisormode.Intheeventthattheserverunexpectedlyterminatesduetoaninternalerror,thesupervisorwillautomaticallyrestarttheserver.Settingthisflagautomaticallyimpliesthattheserverwillrunasadaemon.Notethat,aswiththedaemonflag,thisflagrequiresthatthepid-fileparameterwillset.

unix>./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

2012-06-27T15:58:28Z[10133]INFOstartingupinsupervisormode

GeneralOptions

412

Ascanbeseen(e.g.byexecutingthepscommand),thiswillstartasupervisorprocessandtheactualdatabaseprocess:

unix>psfax|greparangod

10137?Ssl0:00./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

10142?Sl0:00\_./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

Whenthedatabaseprocessterminatesunexpectedly,thesupervisorprocesswillstartupanewdatabaseprocess:

>kill-SIGSEGV10142

>psfax|greparangod

10137?Ssl0:00./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

10168?Sl0:00\_./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

Useridentity

--uiduid

Thename(identity)oftheusertheserverwillrunas.Ifthisparameterisnotspecified,theserverwillnotattempttochangeitsUID,sothattheUIDusedbytheserverwillbethesameastheUIDoftheuserwhostartedtheserver.Ifthisparameterisspecified,thentheserverwillchangeitsUIDafteropeningportsandreadingconfigurationfiles,butbeforeacceptingconnectionsoropeningotherfiles(suchasrecoveryfiles).Thisisusefulwhentheservermustbestartedwithraisedprivileges(incertainenvironments)butsecurityconsiderationsrequirethattheseprivilegesbedroppedoncetheserverhasstartedwork.

Observethatthisparametercannotbeusedtobypassoperatingsystemsecurity.Ingeneral,thisparameter(anditscorrespondingrelativegid)canlowerprivilegesbutnotraisethem.

Groupidentity

--gidgid

Thename(identity)ofthegrouptheserverwillrunas.Ifthisparameterisnotspecified,thentheserverwillnotattempttochangeitsGID,sothattheGIDtheserverrunsaswillbetheprimarygroupoftheuserwhostartedtheserver.Ifthisparameterisspecified,thentheserverwillchangeitsGIDafteropeningportsandreadingconfigurationfiles,butbeforeacceptingconnectionsoropeningotherfiles(suchasrecoveryfiles).

Thisparameterisrelatedtotheparameteruid.

Processidentity

--pid-filefilename

ThenameoftheprocessIDfiletousewhenrunningtheserverasadaemon.Thisparametermustbespecifiedifeithertheflagdaemonorsupervisorisset.

Checkmaxmemorymappings

--server.check-max-memory-mappingscanbeusedonLinuxtomakearangodcheckthenumberofmemorymappingscurrentlyusedbytheprocess(asreportedin/proc//maps)andcompareitwiththemaximumnumberofallowedmappingsasdeterminedby/proc/sys/vm/max_map_count.Ifthecurrentnumberofmemorymappingsgetsnearthemaximumallowedvalue,arangodwilllogawarninganddisallowthecreationoffurtherV8contextstemporarilyuntilthecurrentnumberofmappingsgoesdownagain.

Iftheoptionissettofalse,nosuchcheckswillbeperformed.Allnon-Linuxoperatingsystemsdonotprovidethisoptionandwillignoreit.

Console

--console

Runstheserverinanexclusiveemergencyconsolemode.Whenstartingtheserverwiththisoption,theserverisstartedwithaninteractiveJavaScriptemergencyconsole,withallnetworkingandHTTPinterfacesoftheserverdisabled.

GeneralOptions

413

Norequestscanbemadetotheserverinthismode,andtheonlywaytoworkwiththeserverinthismodeisbyusingtheemergencyconsole.Notethattheservercannotbestartedinthismodeifitisalreadyrunninginthisoranothermode.

RandomGenerator

--random.generatorarg

Theargumentisaninteger(1,2,3or4)whichsetsthemannerinwhichrandomnumbersaregenerated.Thedefaultmethod(3)istousetheanon-blockingrandom(orpseudorandom)numbergeneratorsuppliedbytheoperatingsystem.

Specifyinganargumentof2,usesablockingrandom(orpseudorandom)numbergenerator.Specifyinganargument1setsapseudorandomnumbergeneratorusinganimplicationoftheMersenneTwisterMT19937algorithm.Algorithm4isacombinationoftheblockingrandomnumbergeneratorandtheMersenneTwister.

Enable/disableauthentication

--server.authenticationSettingthisoptiontofalsewillturnoffauthenticationontheserversidesoallclientscanexecuteanyactionwithoutauthorizationandprivilegechecks.Thedefaultvalueistrue.

JWTSecret--server.jwt-secretsecret

ArangoDBwilluseJWTstoauthenticaterequests.UsingthisoptionletsyouspecifyaJWT.

InsingleserversetupsandwhennotspecifyingthissecretArangoDBwillgenerateasecret.

Inclusterdeploymentswhichhaveauthenticationenabledasecretmustbesetconsistentlyacrossallclustertaskssotheycantalktoeachother.

Enable/disableauthenticationforUNIXdomainsockets

--server.authentication-unix-socketsvalue

SettingvaluetotruewillturnoffauthenticationontheserversideforrequestscominginviaUNIXdomainsockets.Withthisflagenabled,clientslocatedonthesamehostastheArangoDBservercanuseUNIXdomainsocketstoconnecttotheserverwithoutauthentication.Requestscominginbyothermeans(e.g.TCP/IP)arenotaffectedbythisoption.

Thedefaultvalueisfalse.

Note:thisoptionisonlyavailableonplatformsthatsupportUNIXdomainsockets.

Enable/disableauthenticationforsystemAPIrequestsonly

--server.authentication-system-onlybooleanControlswhetherincomingrequestsneedauthenticationonlyiftheyaredirectedtotheArangoDB'sinternalAPIsandfeatures,locatedat/_api/,/_admin/etc.Iftheflagissettotrue,thenHTTPauthenticationisonlyrequiredforrequestsgoingtoURLsstartingwith/_,butnotforotherURLs.Theflagcanthusbeusedtoexposeauser-madeAPIwithoutHTTPauthenticationtotheoutsideworld,buttopreventtheoutsideworldfromusingtheArangoDBAPIandtheadmininterfacewithoutauthentication.NotethatcheckingtheURLisperformedafteranydatabasenameprefixhasbeenremoved.ThatmeanswhentheactualURLcalledis/_db/_system/myapp/myaction,theURL/myapp/myactionwillbeusedforauthentication-system-onlycheck.Thedefaultistrue.NotethatauthenticationstillneedstobeenabledfortheserverregularlyinorderforHTTPauthenticationtobeforcedfortheArangoDBAPIandthewebinterface.Settingonlythisflagisnotenough.YoucancontrolArangoDB'sgeneralauthenticationfeaturewiththe--server.authenticationflag.

Enableauthenticationcachetimeout

--server.authentication-timeoutvalue

Setsthecachetimeouttovalue(inseconds).ThisisonlynecessaryifyouuseanexternalauthenticationsystemlikeLDAP.

Enablelocalauthentication--server.local-authenticationvalue

GeneralOptions

414

Ifsettofalseonlyusetheexternalauthenticationsystem.Iftruealsousethelocal_userscollections.

Thedefaultvalueistrue.

Enable/disablereplicationapplier

--database.replication-applierflag

Iffalsetheserverwillstartwithreplicationappliersturnedoff,evenifthereplicationappliersareconfiguredwiththeautoStartoption.Usingthecommand-lineoptionwillnotchangethevalueoftheautoStartoptionintheapplierconfiguration,butwillsuppressauto-startingthereplicationapplierjustonce.

Iftheoptionisnotused,ArangoDBwillreadtheapplierconfigurationfromthefileREPLICATION-APPLIER-CONFIGonstartup,andusethevalueoftheautoStartattributefromthisfile.

Thedefaultistrue.

Keep-alivetimeout--http.keep-alive-timeout

AllowstospecifythetimeoutforHTTPkeep-aliveconnections.Thetimeoutvaluemustbespecifiedinseconds.Idlekeep-aliveconnectionswillbeclosedbytheserverautomaticallywhenthetimeoutisreached.Akeep-alive-timeoutvalue0willdisablethekeepalivefeatureentirely.

HideProductheader

--http.hide-product-header

Iftrue,theserverwillexcludetheHTTPheader"Server:ArangoDB"inHTTPresponses.Ifsettofalse,theserverwillsendtheheaderinresponses.

Thedefaultisfalse.

Allowmethodoverride--http.allow-method-override

Whenthisoptionissettotrue,theHTTPrequestmethodwilloptionallybefetchedfromoneofthefollowingHTTPrequestheadersifpresentintherequest:

x-http-methodx-http-method-overridex-method-override

Iftheoptionissettotrueandanyoftheseheadersisset,therequestmethodwillbeoverriddenbythevalueoftheheader.Forexample,thisallowsissuinganHTTPDELETErequestwhichtotheoutsideworldwilllooklikeanHTTPGETrequest.Thisallowsbypassingproxiesandtoolsthatwillonlyletcertainrequesttypespass.

Settingthisoptiontotruemayimposeasecurityrisksoitshouldonlybeusedincontrolledenvironments.

Thedefaultvalueforthisoptionisfalse.

Serverthreads

--server.threadsnumber

Specifiesthenumberofthreadsthatarespawnedtohandlerequests.

Togglingserverstatistics

--server.statisticsvalue

Ifthisoptionisvalueisfalse,thenArangoDB'sstatisticsgatheringisturnedoff.StatisticsgatheringcausesregularCPUactivitysousingthisoptiontoturnitoffmightrelieveheavy-loadedinstancesabit.

GeneralOptions

415

Sessiontimeout

timetoliveforserversessions--server.session-timeoutvalue

Thetimeoutforwebinterfacesessions,usingforauthenticatingrequeststothewebinterface(/_admin/aardvark)andrelatedareas.

Sessionsareonlyusedwhenauthenticationisturnedon.

Foxxqueues

enableordisabletheFoxxqueuesfeature--foxx.queuesflagIftrue,theFoxxqueueswillbeavailableandjobsinthequeueswillbeexecutedasynchronously.Thedefaultistrue.Whensettofalsethequeuemanagerwillbedisabledandanyjobsarepreventedfrombeingprocessed,whichmayreduceCPUloadabit.

Foxxqueuespollinterval

pollintervalforFoxxqueues--foxx.queues-poll-intervalvalueThepollintervalfortheFoxxqueuesmanager.Thevalueisspecifiedinseconds.LowervalueswillmeanmoreimmediateandmorefrequentFoxxqueuejobexecution,butwillmakethequeuethreadwakeupandquerythequeuesmoreoften.Whensettoalowvalue,thequeuethreadmightcauseCPUload.Thedefaultis1second.IfFoxxqueuesarenotusedmuch,thenthisvaluemaybeincreasedtomakethequeuesthreadwakeupless.

Directory

--database.directorydirectory

Thedirectorycontainingthecollectionsanddatafiles.Defaultsto/var/lib/arango.Whenspecifyingthedatabasedirectory,pleasemakesurethedirectoryisactuallywritablebythearangodprocess.

YoushouldfurthernotuseadatabasedirectorywhichisprovidedbyanetworkfilesystemsuchasNFS.Thereasonisthatnetworkedfilesystemsmightcauseinconsistencieswhentherearemultipleparallelreadersorwritersortheylackfeaturesrequiredbyarangod(e.g.flock()).

directory

Whenusingthecommandlineversion,youcansimplysupplythedatabasedirectoryasargument.

Examples

>./arangod--server.endpointtcp://127.0.0.1:8529--database.directory

/tmp/vocbase

Journalsize

--database.maximal-journal-sizesizeMaximalsizeofjournalinbytes.Canbeoverwrittenwhencreatinganewcollection.Notethatthisalsolimitsthemaximalsizeofasingledocument.Thedefaultis32MB.

Waitforsync

defaultwaitforsyncbehavior--database.wait-for-syncbooleanDefaultwait-for-syncvalue.Canbeoverwrittenwhencreatinganewcollection.Thedefaultisfalse.

Forcesyncingofproperties

forcesyncingofcollectionpropertiestodisk--database.force-sync-propertiesbooleanForcesyncingofcollectionpropertiestodiskaftercreatingacollectionorupdatingitsproperties.Ifturnedoff,nofsyncwillhappenforthecollectionanddatabasepropertiesstoredinparameter.jsonfilesinthefilesystem.Turningoffthisoptionwillspeedupworkloadsthatcreateanddropalotofcollections(e.g.testsuites).Thedefaultistrue.

LimitingmemoryforAQLqueries

--query.memory-limitvalue

GeneralOptions

416

Thedefaultmaximumamountofmemory(inbytes)thatasingleAQLquerycanuse.WhenasingleAQLqueryreachesthespecifiedlimitvalue,thequerywillbeabortedwitharesourcelimitexceededexception.Inacluster,thememoryaccountingisdonepershard,sothelimitvalueiseffectivelyamemorylimitperquerypershard.

ThegloballimitvaluecanbeoverridenperquerybysettingthememoryLimitoptionvalueforindividualquerieswhenrunninganAQLquery.

Thedefaultvalueis0,meaningthatthereisnomemorylimit.

TurningAQLwarningsintoerrors--query.fail-on-warningvalue

Whensettotrue,AQLqueriesthatproducewarningswillinstantlyabortandthrowanexception.ThisoptioncanbesettocatchobviousissueswithAQLqueriesearly.Whensettofalse,AQLqueriesthatproducewarningswillnotabortandreturnthewarningsalongwiththequeryresults.TheoptioncanalsobeoverriddenforeachindividualAQLquery.

Enable/disableAQLquerytracking

--query.trackingflag

Iftrue,theserver'sAQLslowquerytrackingfeaturewillbeenabledbydefault.Trackingofqueriescanbedisabledbysettingtheoptiontofalse.

Thedefaultistrue.

Enable/disabletrackingofbindvariablesinAQLqueries

--query.tracking-with-bindvarsflag

Iftrue,thenthebindvariableswillbetrackedforallrunningandslowAQLqueries.Thisoptiononlyhasaneffectif--query.trackingwassettotrue.Trackingofbindvariablescanbedisabledbysettingtheoptiontofalse.

Thedefaultistrue.

ThresholdforslowAQLqueries--query.slow-thresholdvalue

BysettingvalueitcanbecontrolledafterwhatexecutiontimeanAQLqueryisconsidered"slow".Anyslowqueriesthatexceedtheexecutiontimespecifiedinvaluewillbeloggedwhentheyarefinished.Thethresholdvalueisspecifiedinseconds.Trackingofslowqueriescanbeturnedoffentirelybysettingtheoption--query.trackingtofalse.

Thedefaultvalueis10.0.

Throwcollectionnotloadederror--database.throw-collection-not-loaded-errorflag

Accessinganot-yetloadedcollectionwillautomaticallyloadacollectiononfirstaccess.Thisflagcontrolswhathappensincaseanoperationwouldneedtowaitforanotherthreadtofinalizeloadingacollection.Ifsettotrue,thenthefirstoperationthataccessesanunloadedcollectionwillloadit.Furtherthreadsthattrytoaccessthesamecollectionwhileitisstillloadingwillgetanerror(1238,collectionnotloaded).Whentheinitialoperationhascompletedloadingthecollection,alloperationsonthecollectioncanbecarriedoutnormally,anderror1238willnotbethrown.

Ifsettofalse,thefirstthreadthataccessesanot-yetloadedcollectionwillstillloadit.Otherthreadsthattrytoaccessthecollectionwhileloadingwillnotfailwitherror1238butinsteadblockuntilthecollectionisfullyloaded.Thisconfigurationmightleadtoallserverthreadsbeingblockedbecausetheyareallwaitingforthesamecollectiontocompleteloading.Settingtheoptiontotruewillpreventthisfromhappening,butrequiresclientstocatcherror1238andreactonit(maybebyschedulingaretryforlater).

Thedefaultvalueisfalse.

AQLQuerycachingmode

GeneralOptions

417

--query.cache-mode

TogglestheAQLquerycachebehavior.Possiblevaluesare:

off:donotusequerycacheon:alwaysusequerycache,exceptforqueriesthathavetheircacheattributesettofalsedemand:usequerycacheonlyforqueriesthathavetheircacheattributesettotrue

AQLQuerycachesize

--query.cache-entries

Maximumnumberofqueryresultsthatcanbestoredperdatabase-specificquerycache.Ifaqueryiseligibleforcachingandthenumberofitemsinthedatabase'squerycacheisequaltothisthresholdvalue,anothercachedqueryresultwillberemovedfromthecache.

Thisoptiononlyhasaneffectifthequerycachemodeissettoeitheronordemand.

JavaScriptcodeexecution--javascript.allow-admin-execute

Thisoptioncanbeusedtocontrolwhetheruser-definedJavaScriptcodeisallowedtobeexecutedonserverbysendingviaHTTPtotheAPIendpoint/_admin/executewithanauthenticateduseraccount.Thedefaultvalueisfalse,whichdisablestheexecutionofuser-definedcode.Thisisalsotherecommendedsettingforproduction.Intestenvironments,itmaybeconvenienttoturntheoptiononinordertosendarbitrarysetuporteardowncommandsforexecutionontheserver.

V8contexts

--javascript.v8-contextsnumber

SpecifiesthemaximumnumberofV8contextsthatarecreatedforexecutingJavaScriptcode.MorecontextsallowexecutingmoreJavaScriptactionsinparallel,providedthattherearealsoenoughthreadsavailable.PleasenotethateachV8contextwilluseasubstantialamountofmemoryandrequiresperiodicCPUprocessingtimeforgarbagecollection.

NotethatthisvalueconfiguresthemaximumnumberofV8contextsthatcanbeusedinparallel.UponserverstartonlyasmanyV8contextswillbecreatedasareconfiguredinoption--javascript.v8-contexts-minimum.TheactualnumberofavailableV8contextsmayfloatatruntimebetween--javascript.v8-contexts-minimumand--javascript.v8-contexts.WhenthereareunusedV8contextsthatlingeraround,theserver'sgarbagecollectorthreadwillautomaticallydeletethem.

--javascript.v8-contexts-minimumnumber

SpecifiestheminimumnumberofV8contextsthatwillbepresentatanytimetheserverisrunning.TheactualnumberofV8contextswillneverdropbelowthisvalue,butitmaygoupashighasspecifiedviatheoption--javascript.v8-contexts.

WhenthereareunusedV8contextsthatlingeraroundandthenumberofV8contextsisgreaterthan--javascript.v8-contexts-minimumtheserver'sgarbagecollectorthreadwillautomaticallydeletethem.

--javascript.v8-contexts-max-invocations

SpecifiesthemaximumnumberofinvocationsafterwhichausedV8contextisdisposed.Thedefaultvalueof--javascript.v8-contexts-max-invocationsis0,meaningthatthemaximumnumberofinvocationspercontextisunlimited.

--javascript.v8-contexts-max-age

Specifiesthetimeduration(inseconds)afterwhichtimeaV8contextisdisposedautomaticallyafteritscreation.Ifthetimeiselapsed,thecontextwillbedisposed.Thedefaultvaluefor--javascript.v8-contexts-max-ageis60seconds.

Ifboth--javascript.v8-contexts-max-invocationsand--javascript.v8-contexts-max-ageareset,thenthecontextwillbedestroyedwheneitherofthespecifiedthresholdvaluesisreached.

Garbagecollectionfrequency(time-based)

--javascript.gc-frequencyfrequency

Specifiesthefrequency(inseconds)fortheautomaticgarbagecollectionofJavaScriptobjects.Thissettingisusefultohavethegarbagecollectionstillworkinperiodswithnoorlittlenumbersofrequests.

GeneralOptions

418

Garbagecollectioninterval(request-based)

--javascript.gc-intervalinterval

Specifiestheinterval(approximatelyinnumberofrequests)thatthegarbagecollectionforJavaScriptobjectswillberunineachthread.

V8options--javascript.v8-optionsoptions

OptionalargumentstopasstotheV8Javascriptengine.TheV8enginewillrunwithdefaultsettingsunlessexplicitoptionsarespecifiedusingthisoption.TheoptionspassedwillbeforwardedtotheV8enginewhichwillparsethemonitsown.Passinginvalidoptionsmayresultinanerrorbeingprintedonstderrandtheoptionbeingignored.

Optionsneedtobepassedinonestring,withV8optionnamesbeingprefixedwithdoubledashes.Multipleoptionsneedtobeseparatedbywhitespace.TogetalistofallavailableV8options,youcanusethevalue"--help" asfollows:

--javascript.v8-options="--help"

AnotherexampleofspecificV8optionsbeingsetatstartup:

--javascript.v8-options="--log"

NamesandfeaturesorusableoptionsdependontheversionofV8beingused,andmightchangeinthefutureifadifferentversionofV8isbeingusedinArangoDB.NotalloptionsofferedbyV8mightbesensibletouseinthecontextofArangoDB.Usethespecificoptionsonlyifyouaresurethattheyarenotharmfulfortheregulardatabaseoperation.

GeneralOptions

419

MMFilesWrite-aheadlogoptionsSinceArangoDB2.2,theMMFilesstorageenginewillwritealldata-modificationoperationsintoitswrite-aheadlog.

WithArangoDB3.2anotherStorageengineoptionbecomesavailable-RocksDB.IncaseofusingRocksDBmostofthesubsequentoptionsdon'thaveausefulmeaning.

Thewrite-aheadlogisasequenceoflogfilesthatarewritteninanappend-onlyfashion.Fulllogfileswilleventuallybegarbage-collected,andtherelevantdatamightbetransferredintocollectionjournalsanddatafiles.Unneededandalreadygarbage-collectedlogfileswilleitherbedeletedorkeptforthepurposeofkeepingareplicationbacklog.

Directory

TheWALlogfilesdirectory:--wal.directory

Specifiesthedirectoryinwhichthewrite-aheadlogfilesshouldbestored.Ifthisoptionisnotspecified,itdefaultstothesubdirectoryjournalsintheserver'sglobaldatabasedirectory.Ifthedirectoryisnotpresent,itwillbecreated.

Logfilesize

thesizeofeachWALlogfile--wal.logfile-sizeSpecifiesthefilesize(inbytes)foreachwrite-aheadlogfile.Thelogfilesizeshouldbechosensothateachlogfilecanstoreaconsiderableamountofdocuments.Thebiggerthelogfilesizeischosen,thelongeritwilltaketofillupasinglelogfile,whichalsoinfluencesthedelayuntilthedatainalogfilewillbegarbage-collectedandwrittentocollectionjournalsanddatafiles.Italsoaffectshowlonglogfilerecoverywilltakeatserverstart.

Allowoversizeentries

whetherornotoversizeentriesareallowed--wal.allow-oversize-entriesWhetherornotitisallowedtostoreindividualdocumentsthatarebiggerthanwouldfitintoasinglelogfile.Settingtheoptiontofalsewillmakesuchoperationsfailwithanerror.Settingtheoptiontotruewillmakesuchoperationssucceed,butwithahighpotentialperformanceimpact.Thereasonisthatforeachoversizeoperation,anindividualoversizelogfileneedstobecreatedwhichmayalsoblockotheroperations.Theoptionshouldbesettofalseifitiscertainthatdocumentswillalwayshaveasizesmallerthanasinglelogfile.

Numberofreservelogfiles

maximumnumberofreservelogfiles--wal.reserve-logfilesThemaximumnumberofreservelogfilesthatArangoDBwillcreateinabackgroundprocess.Reservelogfilesareusefulinthesituationwhenanoperationneedstobewrittentoalogfilebutthereservespaceinthelogfileistoolowforstoringtheoperation.Inthiscase,anewlogfileneedstobecreatedtostoretheoperation.Creatingnewlogfilesisnormallyslow,soArangoDBwilltrytopre-createlogfilesinabackgroundprocesssotherearealwaysreservelogfileswhentheactivelogfilegetsfull.ThenumberofreservelogfilesthatArangoDBkeepsinthebackgroundisconfigurablewiththisoption.

Numberofhistoriclogfiles

maximumnumberofhistoriclogfiles--wal.historic-logfilesThemaximumnumberofhistoriclogfilesthatArangoDBwillkeepaftertheyhavebeengarbage-collected.Ifnoreplicationisused,thereisnoneedtokeephistoriclogfilesexceptforhavingalocalchangelog.Inareplicationsetup,thenumberofhistoriclogfilesaffectstheamountofdataaslavecanfetchfromthemaster'slogs.Themorehistoriclogfiles,themorehistoricdataisavailableforaslave,whichisusefuliftheconnectionbetweenmasterandslaveisunstableorslow.Nothavingenoughhistoriclogfilesavailablemightleadtologfiledatabeingdeletedonthemasteralreadybeforeaslavehasfetchedit.

Syncinterval

intervalforautomatic,non-requesteddisksyncs--wal.sync-intervalTheinterval(inmilliseconds)thatArangoDBwillusetoautomaticallysynchronizedatainitswrite-aheadlogstodisk.Automaticsyncswillonlybeperformedfornot-yetsynchronizeddata,andonlyforoperationsthathavebeenexecutedwithoutthewaitForSyncattribute.

Flushtimeout

Write-AheadLogOptions

420

WALflushtimeout`--wal.flush-timeoutThetimeout(inmilliseconds)thatArangoDBwillatmostwaitwhenflushingafullWALlogfiletodisk.Whenthetimeoutisreachedandtheflushisnotcompleted,theoperationthatrequestedtheflushwillfailwithalocktimeouterror.

Throttling

ThrottlewritestoWALwhenatleastsuchmanyoperationsarewaitingforgarbagecollection:--wal.throttle-when-pending

Themaximumvalueforthenumberofwrite-aheadloggarbage-collectionqueueelements.Ifsetto0,thequeuesizeisunbounded,andnowrite-throttlingwilloccur.Ifsettoanon-zerovalue,write-throttlingwillautomaticallykickinwhenthegarbage-collectionqueuecontainsatleastasmanyelementsasspecifiedbythisoption.Whilewrite-throttlingisactive,data-modificationoperationswillintentionallybedelayedbyaconfigurableamountoftime.Thisistoensurethewrite-aheadloggarbagecollectorcancatchupwiththeoperationsexecuted.Write-throttlingwillstayactiveuntilthegarbage-collectionqueuesizegoesdownbelowthespecifiedvalue.Write-throttlingisturnedoffbydefault.

--wal.throttle-wait

Thisoptiondeterminesthemaximumwaittime(inmilliseconds)foroperationsthatarewrite-throttled.Ifwrite-throttlingisactiveandanewwriteoperationistobeexecuted,itwillwaitforatmostthespecifiedamountoftimeforthewrite-aheadloggarbage-collectionqueuesizetofallbelowthethrottlingthreshold.Ifthequeuesizedecreasesbeforethemaximumwaittimeisover,theoperationwillbeexecutednormally.Ifthequeuesizedoesnotdecreasebeforethewaittimeisover,theoperationwillbeabortedwithanerror.Thisoptiononlyhasaneffectif--wal.throttle-when-pendinghasanon-zerovalue,whichisnotthedefault.

Numberofslots

Maximumnumberofslotstobeusedinparallel:--wal.slots

Configurestheamountofwriteslotsthewrite-aheadlogcangivetowriteoperationsinparallel.Anywriteoperationwillleaseaslotandreturnittothewrite-aheadlogwhenitisfinishedwritingthedata.Aslotwillremainblockeduntilthedatainitwassynchronizedtodisk.Afterthat,aslotbecomesreusablebyfollowingoperations.Therequirednumberofslotsisthusdeterminedbytheparallelityofwriteoperationsandthedisksynchronizationspeed.Slowdisksprobablyneedhighervalues,andfastdisksmayonlyrequireavaluelowerthanthedefault.

Ignorelogfileerrors

Ignorelogfileerrorswhenopeninglogfiles:--wal.ignore-logfile-errors

Ignoresanyrecoveryerrorscausedbycorruptedlogfilesonstartup.Whensettofalse,therecoveryprocedureonstartupwillfailwithanerrorwheneveritencountersacorrupted(thatincludesonlyhalf-written)logfile.Thisisasecurityprecautiontopreventdatalossincaseofdiskerrorsetc.Whentherecoveryprocedureabortsbecauseofcorruption,anycorruptedfilescanbeinspectedandfixed(orremoved)manuallyandtheservercanberestartedafterwards.

Settingtheoptiontotruewillmaketheservercontinuewiththerecoveryprocedureevenincaseitdetectscorruptlogfileentries.Inthiscaseitwillstopatthefirstcorruptedlogfileentryandignoreallothers,whichmightcausedataloss.

Ignorerecoveryerrors

Ignorerecoveryerrors:--wal.ignore-recovery-errors

Ignoresanyrecoveryerrorsnotcausedbycorruptedlogfilesbutbylogicalerrors.Logicalerrorscanoccuriflogfilesoranyotherserverdatafileshavebeenmanuallyeditedortheserverissomehowmisconfigured.

Ignore(non-WAL)datafileerrors

Ignoredatafileerrorswhenloadingcollections:--database.ignore-datafile-errorsboolean

Ifsettofalse,CRCmismatchandothererrorsincollectiondatafileswillleadtoacollectionnotbeingloadedatall.Thecollectioninthiscasebecomesunavailable.IfsuchcollectionneedstobeloadedduringWALrecovery,theWALrecoverywillalsoabort(ifnotforcedwithoption--wal.ignore-recovery-errorstrue).

Write-AheadLogOptions

421

Settingthisflagtofalseprotectsusersfromunintentionallyusingacollectionwithcorrupteddatafiles,fromwhichonlyasubsetoftheoriginaldatacanberecovered.Workingwithsuchcollectioncouldleadtodatalossandfollowuperrors.Inordertoaccesssuchcollection,itisrequiredtoinspectandrepairthecollectiondatafilewiththedatafiledebugger(arango-dfdb).

Ifsettotrue,CRCmismatchandothererrorsduringtheloadingofacollectionwillleadtothedatafilebeingpartiallyloaded,uptothepositionofthefirsterror.Alldatauptountiltheinvalidpositionwillbeloaded.Thiswillenableuserstocontinuewithcollectiondatafileseveniftheyarecorrupted,butthiswillresultinonlyapartialloadoftheoriginaldataandpotentialfollowuperrors.TheWALrecoverywillstillabortwhenencounteringacollectionwithacorrupteddatafile,atleastif--wal.ignore-recovery-errorsisnotsettotrue.

Thedefaultvalueisfalse,socollectionswithcorrupteddatafileswillnotbeloadedatall,preventingpartialloadsandfollowuperrors.However,ifsuchcollectionisrequiredatserverstartup,duringWALrecovery,theserverwillaborttherecoveryandrefusetostart.

Write-AheadLogOptions

422

MMFilesCompactionoptionsTheArangoDBMMFilesstorageenginewillrunacompactionoverdatafiles.

ArangoDBwritesDocumentsintheWALfile.Oncetheyhavebeensealedinthewalfile,thecollectormaycopythemintoapercollectionjournalfile.

Oncejournalfilesfillup,they'resealedtobecomedatafiles.

=>onecollectionmayhavedocumentsintheWALlogs,itsjournalfile,andanarbitrarynumberofdatafiles.

Ifacollectionisloaded,eachofthesefilesareopened(thususeafilehandle)andaremmap'ed.Sincefilehandlesandmemorymappedfilesarealsoasparseresource,thatnumbershouldbekeptlow.

Onceyouupdateorremovedocumentsfromdatafiles(oralreadydidwhileitwasthejournalfile)thesedocumentsaremarkedas'dead'withadeletionmarker.

Overtimethenumberofdeaddocumentsmayrise,andwedon'twanttousethepreviouslymentionedresources,plusthediskspaceshouldbegivenbacktothesystem.Thusseveraljournalfilescanbecombinedtoone,ommittingthedeaddocuments.

Combiningseveralofthesedatafilesintooneiscalledcompaction.Thecompactionprocessreadsthealivedocumentsfromtheoriginaldatafiles,andwritesthemintonewdatafile.

Oncethatisdone,thememorymappingstotheolddatafilesisreleased,andthefilesareerased.

Sincethecompactionlocksthecollection,andalsousesI/Oresources,itscarefullyconfigurableunderwhichconditionsthesystemshouldperformwhichamountofthesecompactionjobs:

ArangoDBspawnsonecompactorthreadperdatabase.Thesettingsbelowvaryinscope.

ActivitycontrolTheactivitycontrolparametersalterthebehaviourintermsofscan/executionfrequencyofthecompaction.

Sleepintervalbetweentwocompactionruns(inseconds):--compaction.db-sleep-timeThenumberofsecondsthecollectorthreadwillwaitbetweentwoattemptstosearchforcompactabledatafilesofcollectionsinoneDatabase.Ifthecompactorhasactuallyexecutedwork,asubsequentlookupisdone.Scope:Database.

Minimumsleeptimebetweentwocompactionruns(inseconds):--compaction.min-intervalWhenanactualcompactionwasexecutedforonecollection,wewaitforthistimebeforeweexecutethecompactiononthiscollectionagain.Thisisheretoleteventuallypiledupuserloadbeworkedout.Scope:collection.

SourcedatafilesTheseparameterscontrolwhichdatafilesaretakenintoaccountforacompactionrun.Youcanspecifyseveralcriteriawhicheachoffmaybesufficcientalone.

Thescanoverthedatafilesbelongingtoonecollectionisexecutedfromoldestdatafiletonewest;iffilesqualifyforacompactiontheymaybemergedwithnewerfiles(containingyoungerdocuments)

Scope:Collectionlevel,someareinfluencedbycollectionsettings.

minimalfilesizethresholdoriginaldatafileshavetobebelowforacompaction:--compaction.min-small-data-file-sizeThisisthethresholdwhichcontrolsbelowwhichminimumtotalsizeadatafilewillalwaysbetakenintoaccountforthecompaction.

Minimumunusedcountofdocumentsinadatafile:--compaction.dead-documents-thresholdDatafileswilloftencontaindeaddocuments.Thisparameterspecifiestheirtopmostaccetpeablecountuntilthedatafilequalifiesforcompaction.

Howmanybytesofthesourcedatafileareallowedtobeunusedatmost:--compaction.dead-size-thresholdThedeaddatasizevariesalongwiththesizeofyourdocuments.Ifyouhavemanybigdocuments,thisthresholdmayhitbeforethedocumentcountthreshold.

CompactionOptions

423

Howmanypercentofthesourcedatafileshouldbeunusedatleast:--compaction.dead-size-percent-thresholdsincethesizeofthedocumentsmayvarythisthresholdworksonthepercentageofthedeaddocumentssize.Thus,ifyouhavemanyhugedeaddocuments,thisthresholdkicksinearlier.

Tonameanexamplewithnumbers,ifthedatafilecontains800kbytesofaliveand400kbytesofdeaddocuments,theshareofthedeaddocumentsis:

400/(400+800)=33%.

Ifthisvalueifhigherthanthespecifiedthreshold,thedatafilewillbecompacted.

CompactedtargetfilesOncedatafilesofacollectionarequalifiedforacompactionrun,theseparameterscontrolhowmanydatafilesaremergedintoone,(orevenonesourcedatafilemaybecompactedintoonesmallertargetdatafile)

Scope:Collectionlevel,someareinfluencedbycollectionsettings.

Maximumnumberoffilestomergetoonefile:--compaction.dest-max-filesHowmanydatafiles(atmost)wemaymergeintooneresultingdatafileduringonecompactionrun.

Howlargetheresultingfilemaybeincomparisontothecollectionsdatabase.maximal-journal-sizesetting:--compaction.dest-max-file-size-factorInArangoDByoucanconfigureadefaultjournalfilesizegloballyandoverrideitonapercollectionlevel.Thisvaluecontrolsthesizeofcollecteddatafilesrelativetotheconfiguredjournalfilesizeofthecollectioninquestion.

Afactorof3meansthatthemaximumfilesizeofthecompactedfileis3timesthesizeofthemaximumcollectionjournalfilesize.

howlargemaythecompactionresultfilebecome:--compaction.dest-max-result-file-sizenexttothefactorabove,atotallymaximumallowedfilesizeinbytesmaybespecified.Thiswilloverruleallpreviousparameters.

CompactionOptions

424

ClustersOptions

Agencyendpoint

Listofagencyendpoints:--cluster.agency-endpointendpoint

Anagencyendpointtheservercanconnectto.Theoptioncanbespecifiedmultipletimes,sotheservercanuseaclusterofagencyservers.Endpointshavethefollowingpattern:

tcp://ipv4-address:port-TCP/IPendpoint,usingIPv4tcp://[ipv6-address]:port-TCP/IPendpoint,usingIPv6ssl://ipv4-address:port-TCP/IPendpoint,usingIPv4,SSLencryptionssl://[ipv6-address]:port-TCP/IPendpoint,usingIPv6,SSLencryption

AtleastoneendpointmustbespecifiedorArangoDBwillrefusetostart.ItisrecommendedtospecifyatleasttwoendpointssoArangoDBhasanalternativeendpointifoneofthembecomesunavailable.

Examples

--cluster.agency-endpointtcp://192.168.1.1:4001--cluster.agency-endpointtcp://192.168.1.2:4002...

Myaddress

Thisserver'saddress/endpoint:--cluster.my-addressendpoint

Theserver'sendpointforcluster-internalcommunication.Ifspecified,itmusthavethefollowingpattern:

tcp://ipv4-address:port-TCP/IPendpoint,usingIPv4tcp://[ipv6-address]:port-TCP/IPendpoint,usingIPv6ssl://ipv4-address:port-TCP/IPendpoint,usingIPv4,SSLencryptionssl://[ipv6-address]:port-TCP/IPendpoint,usingIPv6,SSLencryption

Ifnoendpointisspecified,theserverwilllookupitsinternalendpointaddressintheagency.Ifnoendpointcanbefoundintheagencyfortheserver'sid,ArangoDBwillrefusetostart.

Examples

Listenonlyoninterfacewithaddress192.168.1.1

--cluster.my-addresstcp://192.168.1.1:8530

Listenonallipv4andipv6addresses,whichareconfiguredonport8530

--cluster.my-addressssl://[::]:8530

Myrole

Thisserver'srole:--cluster.my-role[dbserver|coordinator]

Theserver'srole.Isthisinstanceadbserver(backenddataserver)oracoordinator(frontendserverforexternalandapplicationaccess)

NodeID(deprecated)

Thisserver'sid:--cluster.my-local-infoinfo

Somelocalinformationabouttheserverinthecluster,thiscanforexamplebeanIPaddresswithaprocessIDoranystringuniquetotheserver.Specifyinginfoismandatoryonstartupiftheserverid(seebelow)isnotspecified.Eachserveroftheclustermusthaveauniquelocalinfo.Thisisignoredifmy-idbelowisspecified.

ClusterOptions

425

Thisoptionisdeprecatedandwillberemovedinafuturerelease.TheclusternodeidshavebeendroppedinfavourofoncegeneratedUUIDs.

Moreadvancedoptions(shouldgenerallyremainuntouched)

Synchroneousreplicationtiming:--cluster.synchronous-replication-timeout-factordouble

Strechorclinchtimeoutsforinternalsynchroneousreplicationmechanismbetweendbservers.Allsuchtimeoutsareaffectedbythischange.Pleasechangeonlywithintentandgreatcare.Defaultat1.0.

Systemreplicationfactor:--cluster.system-replication-factorinteger

Changedefaultreplicationfactorforsystemcollections.Defaultat2.

ClusterOptions

426

RocksDBengineoptionsRocksDBisahighlyconfigurablekey-valuestoreusedtopowerourRocksDBstorageengine.Mostoftheoptionsonthispagearepass-throughoptionstotheunderlyingRocksDBinstance,andwechangeveryfewoftheirdefaultsettings.

Dependingonthestorageengineyouhavechosentheavailabilityandthescopeoftheseoptionschanges.

Incaseyouhavechosenmmfilessomeofthefollowingoptionsapplytopersistentindexes.Incaseofrocksdbitwillapplytoalldatastoredaswellasindexes.

Pass-throughoptions--rocksdb.wal-directory

AbsolutepathfortheRocksDBWALfiles.Ifleftempty,thiswilluseasubdirectoryjournalsinsidethedatadirectory.

Writebuffers

--rocksdb.write-buffer-size

Theamountofdatatobuildupineachin-memorybuffer(backedbyalogfile)beforeclosingthebufferandqueuingittobeflushedintostandardstorage.Default:64MiB.Largervaluesmayimproveperformance,especiallyforbulkloads.

--rocksdb.max-write-buffer-number

Themaximumnumberofwritebuffersthatbuiltupinmemory.Ifthisnumberisreachedbeforethebufferscanbeflushed,writeswillbeslowedorstalled.Default:2.

--rocksdb.min-write-buffer-number-to-merge

Minimumnumberofwritebuffersthatwillbemergedtogetherwhenflushingtonormalstorage.Default:1.

--rocksdb.max-total-wal-size

MaximumtotalsizeofWALfilesthat,whenreached,willforceaflushofallcolumnfamilieswhosedataisbackedbytheoldestWALfiles.Settingthistoalowvaluewilltriggerregularflushingofcolumnfamilydatafrommemtables,sothatWALfilescanbemovedtothearchive.SettingthistoahighvaluewillavoidregularflushingbutmaypreventWALfilesfrombeingmovedtothearchiveandbeingremoved.

--rocksdb.delayed-write-rate(Hidden)

LimitedwriteratetoDB(inbytespersecond)ifwearewritingtothelastin-memorybufferallowedandweallowmorethan3buffers.Default:16MiB/s.

LSMtreestructure

--rocksdb.num-levels

ThenumberoflevelsforthedatabaseintheLSMtree.Default:7.

--rocksdb.num-uncompressed-levels

Thenumberoflevelsthatdonotusecompression.Thedefaultvalueis2.LevelsabovethisnumberwilluseSnappycompressiontoreducethediskspacerequirementsforstoringdataintheselevels.

--rocksdb.dynamic-level-bytes

Iftrue,theamountofdataineachleveloftheLSMtreeisdetermineddynamicallysoastominimizethespaceamplification;otherwise,thelevelsizesarefixed.ThedynamicsizingallowsRocksDBtomaintainawell-structuredLSMtreeregardlessoftotaldatasize.Default:true.

--rocksdb.max-bytes-for-level-base

Themaximumtotaldatasizeinbytesinlevel-1oftheLSMtree.Onlyeffectiveif--rocksdb.dynamic-level-bytesisfalse.Default:256MiB.

RocksDBEngineOptions

427

--rocksdb.max-bytes-for-level-multiplier

ThemaximumtotaldatasizeinbytesforlevelLoftheLSMtreecanbecalculatedasmax-bytes-for-level-base*(max-bytes-for-level-multiplier^(L-1)).Onlyeffectiveif--rocksdb.dynamic-level-bytesisfalse.Default:10.

--rocksdb.level0-compaction-trigger

Compactionoflevel-0tolevel-1istriggeredwhenthismanyfilesexistinlevel-0.Settingthistoahighernumbermayhelpbulkwritesattheexpenseofslowingdownreads.Default:2.

--rocksdb.level0-slowdown-trigger

Whenthismanyfilesaccumulateinlevel-0,writeswillbesloweddownto--rocksdb.delayed-write-ratetoallowcompactiontocatchup.Default:20.

--rocksdb.level0-stop-trigger

Whenthismanyfilesaccumulateinlevel-0,writeswillbestoppedtoallowcompactiontocatchup.Default:36.

FileI/O

--rocksdb.compaction-read-ahead-size

Ifnon-zero,weperformbiggerreadswhendoingcompaction.Ifyou'rerunningRocksDBonspinningdisks,youshouldsetthistoatleast2MiB.ThatwayRocksDB'scompactionisdoingsequentialinsteadofrandomreads.Default:0.

--rocksdb.use-direct-reads(Hidden)

OnlymeaningfulonLinux.Ifset,useO_DIRECTforreadingfiles.Default:false.

--rocksdb.use-direct-io-for-flush-and-compaction(Hidden)

OnlymeaningfulonLinux.Ifset,useO_DIRECTforwritingfiles.Default:false.

--rocksdb.use-fsync(Hidden)

Ifset,issueanfsynccallwhenwritingtodisk(settofalsetoissuefdatasynconly.Default:false.

Backgroundtasks

--rocksdb.max-background-jobs

Maximumnumberofconcurrentbackgroundcompactionjobs,submittedtothelowprioritythreadpool.Default:numberofprocessors.

--rocksdb.num-threads-priority-high

Numberofthreadsforhighpriorityoperations(e.g.flush).Werecommendsettingthisequaltomax-background-flushes.Default:numberofprocessors/2.

--rocksdb.num-threads-priority-low

Numberofthreadsforlowpriorityoperations(e.g.compaction).Default:numberofprocessors/2.

Caching

--rocksdb.block-cache-size

Thisisthesizeoftheblockcacheinbytes.Increasingthismayimproveperformance.Ifthereislessthan4GiBofRAMonthesystem,thedefaultvalueis256MiB.Ifthereismore,thedefaultis(systemRAMsize-2GiB)*0.3.

--rocksdb.block-cache-shard-bits

Thenumberofbitsusedtoshardtheblockcachetoallowconcurrentoperations.Tokeepindividualshardsatareasonablesize(i.e.atleast512KB),keepthisvaluetoatmostblock-cache-shard-bits/512KB.Default:block-cache-size/2^19.

--rocksdb.table-block-size

Approximatesizeofuserdata(inbytes)packedperblockforuncompresseddata.

--rocksdb.recycle-log-file-num(Hidden)

Numberoflogfilestokeeparoundforrecycling.Default:0.

RocksDBEngineOptions

428

Miscellaneous

--rocksdb.optimize-filters-for-hits(Hidden)

Thisflagspecifiesthattheimplementationshouldoptimizethefiltersmainlyforcaseswherekeysarefoundratherthanalsooptimizeforthecasewherekeysarenot.Thiswouldbeusedincaseswheretheapplicationknowsthatthereareveryfewmissesortheperformanceinthecaseofmissesisnotasimportant.Default:false.

--rocksdb.wal-recovery-skip-corrupted(Hidden)

Iftrue,skipcorruptedrecordsinWALrecovery.Default:false.

Non-Pass-ThroughOptions--rocksdb.wal-file-timeout(Hidden)

TimeoutafterwhichunusedWALfilesaredeleted(inseconds).Default:10.0s.

DataofongoingtransactionsisstoredinRAM.Transactionsthatgettoobig(intermsofnumberofoperationsinvolvedorthetotalsizeofdatacreatedormodifiedbythetransaction)willbecommittedautomatically.EffectivelythismeansthatbigusertransactionsaresplitintomultiplesmallerRocksDBtransactionsthatarecommittedindividually.TheentireusertransactionwillnotnecessarilyhaveACIDpropertiesinthiscase.

ThefollowingoptionscanbeusedtocontroltheRAMusageandautomaticintermediatecommitsfortheRocksDBengine:

--rocksdb.max-transaction-size

Transactionsizelimit(inbytes).TransactionsstoreallkeysandvaluesinRAM,solargetransactionsruntheriskofcausingout-of-memorysitations.Thissettingallowsyoutoensurethatdoesnothappenbylimitingthesizeofanyindividualtransaction.TransactionswhoseoperationswouldconsumemoreRAMthanthisthresholdvaluewillabortautomaticallywitherror32("resourcelimitexceeded").

--rocksdb.intermediate-commit-size

Ifthesizeofalloperationsinatransactionreachesthisthreshold,thetransactioniscommittedautomaticallyandanewtransactionisstarted.Thevalueisspecifiedinbytes.

--rocksdb.intermediate-commit-count

Ifthenumberofoperationsinatransactionreachesthisvalue,thetransactioniscommittedautomaticallyandanewtransactionisstarted.

--rocksdb.throttle

Ifenabled,throttlestheingestrateofwritesifnecessarytoreducechancesofcompactionsgettingtoofarbehindandblockingincomingwrites.Thisoptionistruebydefault.

RocksDBEngineOptions

429

HashcacheoptionsSinceArangoDB3.2,theseveralcorecomponentsoftheserveruseacachesystemwhichpoolsmemoryacrossmanydifferentcachetables.Inordertoprovideintelligentinternalmemorymanagement,thesystemperiodicallyreclaimsmemoryfromcacheswhichareusedlessoftenandreallocatesittocacheswhichgetmoreactivity.

Cachesize

Globalsizelimitforallhashcaches:--cache.size

Theglobalcachingsystem,allcaches,andallthedatacontainedthereinwillfitinsidethislimit.Thesizeisspecifiedinbytes.Ifthereislessthan4GiBofRAMonthesystem,thedefaultvalueis256MiB.Ifthereismore,thedefaultis(systemRAMsize-2GiB)*0.3.

Rebalancinginterval

Timebetweencacherebalancingattempts:--cache.rebalancing-interval

Thevalueisspecifiedinmicrosecondswithadefaultof2secondsandaminimumof500milliseconds.

HashCacheOptions

430

AsynchronousTasks

maximalqueuesize

Maximumsizeofthequeueforrequests:--server.maximal-queue-sizesize

Specifiesthemaximumsizeofthequeueforasynchronoustaskexecution.Ifthequeuealreadycontainssizetasks,newtaskswillberejecteduntilothertasksarepoppedfromthequeue.Settingthisvaluemayhelppreventingfromrunningoutofmemoryifthequeueisfilledupfasterthantheservercanprocessrequests.

AsynchronousTasks

431

DurabilityConfiguration

GlobalConfiguration

Thereareglobalconfigurationvaluesfordurability,whichcanbeadjustedbyspecifyingthefollowingconfigurationoptions:

defaultwaitforsyncbehavior--database.wait-for-syncbooleanDefaultwait-for-syncvalue.Canbeoverwrittenwhencreatinganewcollection.Thedefaultisfalse.

forcesyncingofcollectionpropertiestodisk--database.force-sync-propertiesbooleanForcesyncingofcollectionpropertiestodiskaftercreatingacollectionorupdatingitsproperties.Ifturnedoff,nofsyncwillhappenforthecollectionanddatabasepropertiesstoredinparameter.jsonfilesinthefilesystem.Turningoffthisoptionwillspeedupworkloadsthatcreateanddropalotofcollections(e.g.testsuites).Thedefaultistrue.

intervalforautomatic,non-requesteddisksyncs--wal.sync-intervalTheinterval(inmilliseconds)thatArangoDBwillusetoautomaticallysynchronizedatainitswrite-aheadlogstodisk.Automaticsyncswillonlybeperformedfornot-yetsynchronizeddata,andonlyforoperationsthathavebeenexecutedwithoutthewaitForSyncattribute.

Per-collectionconfiguration

Youcanalsoconfigurethedurabilitybehavioronaper-collectionbasis.UsetheArangoDBshelltochangetheseproperties.

getsorsetsthepropertiesofacollectioncollection.properties()Returnsanobjectcontainingallcollectionproperties.

waitForSync:Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.journalSize:Thesizeofthejournalinbytes.ThisoptionismeaningfulfortheMMFilesstorageengineonly.isVolatile:IftruethenthecollectiondatawillbekeptinmemoryonlyandArangoDBwillnotwriteorsyncthedatatodisk.ThisoptionismeaningfulfortheMMFilesstorageengineonly.keyOptions(optional)additionaloptionsforkeygeneration.ThisisaJSONarraycontainingthefollowingattributes(note:someoftheattributesareoptional):

type:thetypeofthekeygeneratorusedforthecollection.allowUserKeys:ifsettotrue,thenitisallowedtosupplyownkeyvaluesinthe_keyattributeofadocument.Ifsettofalse,thenthekeygeneratorwillsolelyberesponsibleforgeneratingkeysandsupplyingownkeyvaluesinthe_keyattributeofdocumentsisconsideredanerror.increment:incrementvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.offset:initialoffsetvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.

indexBuckets:numberofbucketsintowhichindexesusingahashtablearesplit.Thedefaultis16andthisnumberhastobeapowerof2andlessthanorequalto1024.ThisoptionismeaningfulfortheMMFilesstorageengineonly.Forverylargecollectionsoneshouldincreasethistoavoidlongpauseswhenthehashtablehastobeinitiallybuiltorresized,sincebucketsareresizedindividuallyandcanbeinitiallybuiltinparallel.Forexample,64mightbeasensiblevalueforacollectionwith100000000documents.Currently,onlytheedgeindexrespectsthisvalue,butotherindextypesmightfollowinfutureArangoDBversions.Changes(seebelow)areappliedwhenthecollectionisloadedthenexttime.Inaclustersetup,theresultwillalsocontainthefollowingattributes:numberOfShards:thenumberofshardsofthecollection.shardKeys:containsthenamesofdocumentattributesthatareusedtodeterminethetargetshardfordocuments.replicationFactor:determineshowmanycopiesofeachshardarekeptondifferentDBServers.collection.properties(properties)Changesthecollectionproperties.propertiesmustbeanobjectwithoneormoreofthefollowingattribute(s):waitForSync:Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.journalSize:Thesizeofthejournalinbytes.ThisoptionismeaningfulfortheMMFilesstorageengineonly.indexBuckets:Seeabove,changesareonlyappliedwhenthecollectionisloadedthenexttime.ThisoptionismeaningfulfortheMMFilesstorageengineonly.replicationFactor:ChangethenumberofshardcopieskeptondifferentDBServers,validvaluesareintegernumbersintherangeof1-10(Clusteronly)Note:itisnotpossibletochangethejournalsizeafterthejournalordatafilehasbeencreated.Changingthisparameterwillonlyeffectnewlycreatedjournals.Alsonotethatyoucannotlowerthejournalsizetolessthensizeofthelargestdocumentalreadystoredinthecollection.Note:someothercollectionproperties,suchastype,isVolatile,orkeyOptionscannotbe

Durability

432

changedoncethecollectioniscreated.

Examples

Readallproperties

arangosh>db.example.properties();

showexecutionresultsChangeaproperty

arangosh>db.example.properties({waitForSync:true});

showexecutionresults

Per-operationconfiguration

Manydata-modificationoperationsandalsoArangoDB'stransactionsallowtospecifyawaitForSyncattribute,whichwhensetensurestheoperationdatahasbeensynchronizedtodiskwhentheoperationreturns.

Disk-UsageConfiguration

TheamountofdiskspaceusedbyArangoDBisdeterminedbyafewconfigurationoptions.

GlobalConfiguration

ThetotalamountofdiskstoragerequiredbyArangoDBisdeterminedbythesizeofthewrite-aheadlogfilesplusthesizesofthecollectionjournalsanddatafiles.

Therearethefollowingoptionsforconfiguringthenumberandsizesofthewrite-aheadlogfiles:

maximumnumberofreservelogfiles--wal.reserve-logfilesThemaximumnumberofreservelogfilesthatArangoDBwillcreateinabackgroundprocess.Reservelogfilesareusefulinthesituationwhenanoperationneedstobewrittentoalogfilebutthereservespaceinthelogfileistoolowforstoringtheoperation.Inthiscase,anewlogfileneedstobecreatedtostoretheoperation.Creatingnewlogfilesisnormallyslow,soArangoDBwilltrytopre-createlogfilesinabackgroundprocesssotherearealwaysreservelogfileswhentheactivelogfilegetsfull.ThenumberofreservelogfilesthatArangoDBkeepsinthebackgroundisconfigurablewiththisoption.

maximumnumberofhistoriclogfiles--wal.historic-logfilesThemaximumnumberofhistoriclogfilesthatArangoDBwillkeepaftertheyhavebeengarbage-collected.Ifnoreplicationisused,thereisnoneedtokeephistoriclogfilesexceptforhavingalocalchangelog.Inareplicationsetup,thenumberofhistoriclogfilesaffectstheamountofdataaslavecanfetchfromthemaster'slogs.Themorehistoriclogfiles,themorehistoricdataisavailableforaslave,whichisusefuliftheconnectionbetweenmasterandslaveisunstableorslow.Nothavingenoughhistoriclogfilesavailablemightleadtologfiledatabeingdeletedonthemasteralreadybeforeaslavehasfetchedit.

thesizeofeachWALlogfile--wal.logfile-sizeSpecifiesthefilesize(inbytes)foreachwrite-aheadlogfile.Thelogfilesizeshouldbechosensothateachlogfilecanstoreaconsiderableamountofdocuments.Thebiggerthelogfilesizeischosen,thelongeritwilltaketofillupasinglelogfile,whichalsoinfluencesthedelayuntilthedatainalogfilewillbegarbage-collectedandwrittentocollectionjournalsanddatafiles.Italsoaffectshowlonglogfilerecoverywilltakeatserverstart.

whetherornotoversizeentriesareallowed--wal.allow-oversize-entriesWhetherornotitisallowedtostoreindividualdocumentsthatarebiggerthanwouldfitintoasinglelogfile.Settingtheoptiontofalsewillmakesuchoperationsfailwithanerror.Settingtheoptiontotruewillmakesuchoperationssucceed,butwithahighpotentialperformanceimpact.Thereasonisthatforeachoversizeoperation,anindividualoversizelogfileneedstobecreatedwhichmayalsoblockotheroperations.Theoptionshouldbesettofalseifitiscertainthatdocumentswillalwayshaveasizesmallerthanasinglelogfile.Whendatagetscopiedfromthewrite-aheadlogfilesintothejournalsordatafilesofcollections,fileswillbecreatedonthecollectionlevel.Howbigthesefilesareisdeterminedbythefollowingglobalconfigurationvalue:

Durability

433

--database.maximal-journal-sizesizeMaximalsizeofjournalinbytes.Canbeoverwrittenwhencreatinganewcollection.Notethatthisalsolimitsthemaximalsizeofasingledocument.Thedefaultis32MB.

Per-collectionconfiguration

Thejournalsizecanalsobeadjustedonaper-collectionlevelusingthecollection'spropertiesmethod.

Durability

434

EncryptionThisfeatureisonlyavailableintheEnterpriseEdition.

WhenyoustoresensitivedatainyourArangoDBdatabase,youwanttoprotectthatdataunderallcircumstances.AtruntimeyouwillprotectitwithSSLtransportencryptionandstrongauthentication,butwhenthedataisalreadyondisk,youalsoneedprotection.ThatiswheretheEncryptionfeaturecomesin.

TheEncryptionfeatureofArangoDBwillencryptalldatathatArangoDBisstoringinyourdatabasebeforeitiswrittentodisk.

ThedataisencryptedwithAES-256-CTR,whichisastrongencryptionalgorithm,thatisverysuitableformulti-processorenvironments.Thismeansthatyourdataissafe,butyourdatabaseisstillfast,evenunderload.

MostmodernCPU'shavebuiltinsupportforhardwareAESencryption,whichmakesitevenfaster.

Note:TheEncryptionfeaturerequirestheRocksDBstorageengine.

Encryptionkeys

TheEncryptionfeatureofArangoDBrequiresasingle32-bytekeyperserver.Itisrecommendedtouseadifferentkeyforeachserver(whenoperatinginaclusterconfiguration).Makesuretoprotectthesekeys!

Thatmeans:

Donotwritethemtopersistentdisksoryourserver(s),alwaysstorethemonanin-memory(tmpfs)filesystem.Transportyourkeyssafelytoyourserver(s).Therearevarioustoolsformanagingsecretslikethis(e.g.vaultproject.io).Storeacopyofyourkeyofflineinasafeplace.Ifyouloseyourkey,thereisNOwaytogetyourdataback.

ConfigurationToactivateencryptionofyourdatabase,youneedtosupplyanencryptionkeytotheserver.

Makesuretopassthisoptiontheveryfirsttimeyoustartyourdatabase.Youcannotencryptadatabasethatalreadyexists.

Note:Youalsohavetoactivatetherocksdbstorageengine.

Encryptionkeystoredinfile

Passthefollowingoptiontoarangod:

$arangod\

--rocksdb.encryption-keyfile=/mytmpfs/mySecretKey\

--server.storage-engine=rocksdb

Thefile/mytmpfs/mySecretKeymustcontaintheencryptionkey.Thisfilemustbesecured,sothatonlyarangodcanaccessit.Youshouldalsoensurethatincasesome-onestealsthehardware,hewillnotbeabletoreadthefile.Forexample,byencryption/mytmpfsorcreatingain-memoryfile-systemunder/mytmpfs.

Encryptionkeygeneratedbyaprogram

Passthefollowingoptiontoarangod:

$arangod\

--rocksdb.encryption-key-generator=path-to-my-generator\

--server.storage-engine=rocksdb

Theprogrampath-to-my-generatoroutputtheencryptiononstandardoutputandexit.

Encryption

435

Creatingkeys

Theencryptionkeyfilemustcontain32bytesofrandomdata.

Youcancreateitwithacommandlinethis.

ddif=/dev/randombs=1count=32of=yourSecretKeyFile

Forsecurity,itisbesttocreatethesekeysoffline(awayfromyourdatabaseservers)anddirectlystoretheminyousecretmanagementtool.

Encryption

436

AuditingThisfeatureisavailableintheEnterpriseEdition.

Auditingallowsyoutomonitoraccesstothedatabaseindetail.Ingeneralauditlogsareoftheform

2016-01-0112:00:00|server|username|database|client-ip|authentication|text1|text2|...

Thetime-stampisinGMT.Thisallowstoeasilymatchlogentriesfromserversindifferenttimezones.

Thenameoftheserver.Youcanspecifyacustomnameonstartup.Otherwisethedefaulthostnameisused.

Theusernameisthe(authenticatedorunauthenticated)namesuppliedbytheclient.Adash-isprintedifnonamewasgivenbytheclient.

Thedatabasedescribesthedatabasethatwasaccessed.Pleasenotethattherearenodatabasecrossingqueries.Eachaccessisrestrictedtoonedatabase.

Theclient-ipdescribesthesourceoftherequest.

Theauthenticationdetailsthemethodsusedtoauthenticatetheuser.

Detailsabouttherequestsfollowintheadditionalfields.

Auditing

437

AuditConfigurationThisfeatureisavailableintheEnterpriseEdition.

Output--audit.outputoutput

Specifiesthetargetoftheauditlog.Possiblevaluesare

file://filenamewherefilenamecanberelativeorabsolute.

syslog://facilityorsyslog://facility/application-nametologintoasyslogserver.

Theoptioncanbespecifiedmultipletimesinordertoconfiguretheoutputformultipletargets.

Hostname--audit.hostnamename

Thenameoftheserverusedinauditlogmessages.Bydefaultthesystemhostnameisused.

Configuration

438

AuditEventsThisfeatureisavailableintheEnterpriseEdition.

Authentication

Unknownauthenticationmethods

2016-10-0315:44:23|server1|-|database1|127.0.0.1:61525|-|unknownauthenticationmethod|/_api/version

Missingcredentials

2016-10-0315:39:49|server1|-|database1|127.0.0.1:61498|-|credentialsmissing|/_api/version

Wrongcredentials

2016-10-0315:47:26|server1|user1|database1|127.0.0.1:61528|httpbasic|credentialswrong|/_api/version

Passwordchangerequired

2016-10-0316:18:53|server1|user1|database1|127.0.0.1:62257|-|passwordchangerequired|/_api/version

JWTloginsucceeded

2016-10-0317:21:22|server1|-|database1|127.0.0.1:64214|httpjwt|user'root'authenticated|/_open/auth

Pleasenote,thattheusergivenasthirdpartistheuserthatrequestedthelogin.Ingeneral,itwillbeempty.

JWTloginfailed

2016-10-0317:21:22|server1|-|database1|127.0.0.1:64214|httpjwt|user'root'wrongcredentials|/_open/auth

Pleasenote,thattheusergivenasthirdpartistheuserthatrequestedthelogin.Ingeneral,itwillbeempty.

Authorization

Usernotauthorizedtoaccessdatabase

2016-10-0316:20:52|server1|user1|database2|127.0.0.1:62262|httpbasic|notauthorized|/_api/version

Databases

Createadatabase

2016-10-0415:33:25|server1|user1|database1|127.0.0.1:56920|httpbasic|createdatabase'database1'|ok|/_api/dat

abase

Events

439

Dropadatabase

2016-10-0415:33:25|server1|user1|database1|127.0.0.1:56920|httpbasic|deletedatabase'database1'|ok|/_api/dat

abase

Collections

Createacollection

2016-10-0517:35:57|server1|user1|database1|127.0.0.1:51294|httpbasic|createcollection'collection1'|ok|/_api

/collection

Truncateacollection

2016-10-0517:36:08|server1|user1|database1|127.0.0.1:51294|httpbasic|truncatecollection'collection1'|ok|/_a

pi/collection/collection1/truncate

Dropacollection

2016-10-0517:36:30|server1|user1|database1|127.0.0.1:51294|httpbasic|deletecollection'collection1'|ok|/_api

/collection/collection1

Indexes

Createaindex

2016-10-0518:19:40|server1|user1|database1|127.0.0.1:52467|httpbasic|createindexin'collection1'|ok|{"field

s":["a"],"sparse":false,"type":"skiplist","unique":false}|/_api/index?collection=collection1

Dropaindex

2016-10-0518:18:28|server1|user1|database1|127.0.0.1:52464|httpbasic|dropindex':44051'|ok|/_api/index/colle

ction1/44051

Documents

Readingasingledocument

2016-10-0412:27:55|server1|user1|database1|127.0.0.1:53699|httpbasic|createdocumentok|/_api/document/collecti

on1

Replacingasingledocument

2016-10-0412:28:08|server1|user1|database1|127.0.0.1:53699|httpbasic|replacedocumentok|/_api/document/collect

ion1/21456?ignoreRevs=false

Modifyingasingledocument

2016-10-0412:28:15|server1|user1|database1|127.0.0.1:53699|httpbasic|modifydocumentok|/_api/document/collecti

on1/21456?keepNull=true&ignoreRevs=false

Events

440

Deletingasingledocument

2016-10-0412:28:23|server1|user1|database1|127.0.0.1:53699|httpbasic|deletedocumentok|/_api/document/collecti

on1/21456?ignoreRevs=false

Forexample,ifsomeonestriestodeleteanon-existingdocument,itwillbeloggedas

2016-10-0412:28:26|server1|user1|database1|127.0.0.1:53699|httpbasic|deletedocumentfailed|/_api/document/coll

ection1/21456?ignoreRevs=false

Queries

2016-10-0612:12:10|server1|user1|database1|127.0.0.1:54232|httpbasic|querydocument|ok|foriincollection1r

eturni|/_api/cursor

Events

441

IntroductiontoReplicationReplicationallowsyoutoreplicatedataontoanothermachine.ItformsthebaseofalldisasterrecoveryandfailoverfeaturesArangoDBoffers.

ArangoDBoffersasynchronousandsynchronousreplicationwhichbothhavetheirprosandcons.Bothmodesmayandshouldbecombinedinarealworldscenarioandbeappliedintheusecasewheretheyexcelmost.

Wewilldescribeprosandconsofeachoftheminthefollowingsections.

Synchronousreplication

Synchronousreplicationonlyworkswithinaclusterandistypicallyusedformissioncriticaldatawhichmustbeaccessibleatalltimes.Synchronousreplicationgenerallystoresacopyofashard'sdataonanotherdbserverandkeepsitinsync.Essentially,whenstoringdataafterenablingsynchronousreplicationtheclusterwillwaitforallreplicastowriteallthedatabeforegreenlightingthewriteoperationtotheclient.Thiswillnaturallyincreasethelatencyabit,sinceonemorenetworkhopisneededforeachwrite.However,itwillenabletheclustertoimmediatelyfailovertoareplicawheneveranoutagehasbeendetected,withoutlosinganycommitteddata,andmostlywithoutevensignalinganerrorconditiontotheclient.

Synchronousreplicationisorganizedsuchthateveryshardhasaleaderandr-1followers,whererdenotedthereplicationfactor.ThenumberoffollowerscanbecontrolledusingthereplicationFactorparameterwheneveryoucreateacollection,thereplicationFactorparameteristhetotalnumberofcopiesbeingkept,thatis,itisoneplusthenumberoffollowers.

Satellitecollections

SatellitecollectionsaresynchronouslyreplicatedcollectionshavingadynamicreplicationFactor.Theywillreplicatealldatatoalldatabaseserversallowingthedatabaseserverstojoindatalocallyinsteadofdoingheavynetworkoperations.

Satellitecollectionsareanenterpriseonlyfeature.

Asynchronousreplication

InArangoDBanywriteoperationwillbeloggedtothewrite-aheadlog.WhenusingAsynchronousreplicationslaveswillconnecttoamasterandapplyalltheeventsfromtheloginthesameorderlocally.Afterthat,theywillhavethesamestateofdataasthemasterdatabase.

Replication

442

AsynchronousreplicationAsynchronousreplicationworksbyloggingeverydatamodificationonamasterandreplayingtheseeventsonanumberofslaves.

Transactionsarehonoredinreplication,i.e.transactionalwriteoperationswillbecomevisibleonslavesatomically.

Asallwriteoperationswillbeloggedtoamasterdatabase'swrite-aheadlog,thereplicationinArangoDBcurrentlycannotbeusedforwrite-scaling.ThemainpurposesofthereplicationincurrentArangoDBaretoprovideread-scalabilityand"hotbackups"ofspecificdatabases.

Itispossibletoconnectmultipleslavedatabasestothesamemasterdatabase.Slavedatabasesshouldbeusedasread-onlyinstances,andnouser-initiatedwriteoperationsshouldbecarriedoutonthem.Otherwisedataconflictsmayoccurthatcannotbesolvedautomatically,andthatwillmakethereplicationstop.

Inanasynchronousreplicationscenarioslaveswillpullchangesfromthemasterdatabase.Slavesneedtoknowtowhichmasterdatabasetheyshouldconnectto,butamasterdatabaseisnotawareoftheslavesthatreplicatefromit.Whenthenetworkconnectionbetweenthemasterdatabaseandaslavegoesdown,writeoperationsonthemastercancontinuenormally.Whenthenetworkisupagain,slavescanreconnecttothemasterdatabaseandtransfertheremainingchanges.Thiswillhappenautomaticallyprovidedslavesareconfiguredappropriately.

Replicationlag

Inthissetup,writeoperationsareappliedfirstinthemasterdatabase,andappliedintheslavedatabase(s)afterwards.

Forexample,let'sassumeawriteoperationisexecutedinthemasterdatabaseatpointintimet0.Tomakeaslavedatabaseapplythesameoperation,itmustfirstfetchthewriteoperation'sdatafrommasterdatabase'swrite-aheadlog,thenparseitandapplyitlocally.Thiswillhappenatsomepointintimeaftert0,let'ssayt1.

Thedifferencebetweent1andt0iscalledthereplicationlag,anditisunavoidableinasynchronousreplication.Theamountofreplicationlagdependsonmanyfactors,afewofwhichare:

thenetworkcapacitybetweentheslavesandthemastertheloadofthemasterandtheslavesthefrequencyinwhichslavespollthemasterforupdates

Betweent0andt1,thestateofdataonthemasterisnewerthanthestateofdataontheslave(s).Atpointintimet1,thestateofdataonthemasterandslave(s)isconsistentagain(providednonewdatamodificationshappenedonthemasterinbetween).Thus,thereplicationwillleadtoaneventuallyconsistentstateofdata.

Replicationconfiguration

Thereplicationisturnedoffbydefault.Inordertocreateamaster-slavesetup,theso-calledreplicationapplierneedstobeenabledontheslavedatabases.

Replicationisconfiguredonaper-databaselevel.Ifmultipledatabasearetobereplicated,thereplicationmustbesetupindividuallyperdatabase.

Thereplicationapplierontheslavecanbeusedtoperformaone-timesynchronizationwiththemaster(andthenstop),ortoperformanongoingreplicationofchanges.Toresumereplicationonslaverestart,theautoStartattributeofthereplicationappliermustbesettotrue.

Replicationoverhead

Asthemasterserversarelogginganywriteoperationinthewrite-ahead-loganywayreplicationdoesn'tcauseanyextraoverheadonthemaster.Howeveritwillofcoursecausesomeoverheadforthemastertoserveincomingreadrequestsoftheslaves.Returningtherequesteddataishoweveratrivialtaskforthemasterandshouldnotresultinanotableperformancedegrationinproduction.

AsynchronousReplication

443

Components

ReplicationLogger

Purpose

Thereplicationloggerwillwritealldata-modificationoperationsintothewrite-aheadlog.Thislogmaythenbereadbyclientstoreplayanydatamodificationonadifferentserver.

Checkingthestate

Toquerythecurrentstateofthelogger,usethestatecommand:

require("@arangodb/replication").logger.state();

Theresultmightlooklikethis:

{

"state":{

"running":true,

"lastLogTick":"133322013",

"totalEvents":16,

"time":"2014-07-06T12:58:11Z"

},

"server":{

"version":"2.2.0-devel",

"serverId":"40897075811372"

},

"clients":{

}

}

Therunningattributewillalwaysbetrue.InearlierversionsofArangoDBthereplicationwasoptionalandthiscouldhavebeenfalse.

ThetotalEventsattributeindicateshowmanylogeventshavebeenloggedsincethestartoftheArangoDBserver.Finally,thelastLogTickvalueindicatestheidofthelastoperationthatwaswrittentotheserver'swrite-aheadlog.Itcanbeusedtodeterminewhethernewoperationswerelogged,andisalsousedbythereplicationapplierforincrementalfetchingofdata.

Note:ThereplicationloggerstatecanalsobequeriedviatheHTTPAPI.

Toquerywhichdatarangesarestillavailableforreplicationclientstofetch,theloggerprovidesthefirstTickandtickRangesfunctions:

require("@arangodb/replication").logger.firstTick();

ThiswillreturntheminimumtickvaluethattheservercanprovidetoreplicationclientsviaitsreplicationAPIs.ThetickRangesfunctionreturnstheminimumandmaximumtickvaluesperlogfile:

require("@arangodb/replication").logger.tickRanges();

ReplicationApplier

Purpose

Thepurposeofthereplicationapplieristoreaddatafromamasterdatabase'seventlog,andapplythemlocally.Theapplierwillcheckthemasterdatabasefornewoperationsperiodically.Itwillperformanincrementalsynchronization,i.e.onlyaskingthemasterforoperationsthatoccurredafterthelastsynchronization.

AsynchronousReplication

444

Thereplicationapplierdoesnotgetnotifiedbythemasterdatabasewhenthereare"new"operationsavailable,butinsteadusesthepullprinciple.Itmightthustakesometime(theso-calledreplicationlag)beforeanoperationfromthemasterdatabasegetsshippedtoandappliedinaslavedatabase.

Thereplicationapplierofadatabaseisruninaseparatethread.Itmayencounterproblemswhenanoperationfromthemastercannotbeappliedsafely,orwhentheconnectiontothemasterdatabasegoesdown(networkoutage,masterdatabaseisdownorunavailableetc.).Inthiscase,thedatabase'sreplicationapplierthreadmightterminateitself.Itisthenuptotheadministratortofixtheproblemandrestartthedatabase'sreplicationapplier.

Ifthereplicationappliercannotconnecttothemasterdatabase,orthecommunicationfailsatsomepointduringthesynchronization,thereplicationapplierwilltrytoreconnecttothemasterdatabase.Itwillgiveupreconnectingonlyafteraconfigurableamountofconnectionattempts.

Thereplicationapplierstateisqueryableatanytimebyusingthestatecommandoftheapplier.Thiswillreturnthestateoftheapplierofthecurrentdatabase:

require("@arangodb/replication").applier.state();

Theresultmightlooklikethis:

{

"state":{

"running":true,

"lastAppliedContinuousTick":"152786205",

"lastProcessedContinuousTick":"152786205",

"lastAvailableContinuousTick":"152786205",

"progress":{

"time":"2014-07-06T13:04:57Z",

"message":"fetchingmasterlogfromoffset152786205",

"failedConnects":0

},

"totalRequests":38,

"totalFailedConnects":0,

"totalEvents":1,

"lastError":{

"errorNum":0

},

"time":"2014-07-06T13:04:57Z"

},

"server":{

"version":"2.2.0-devel",

"serverId":"210189384542896"

},

"endpoint":"tcp://master.example.org:8529",

"database":"_system"

}

Therunningattributeindicateswhetherthereplicationapplierofthecurrentdatabaseiscurrentlyrunningandpollingtheserveratendpointfornewevents.

Theprogress.failedConnectsattributeshowshowmanyfailedconnectionattemptsthereplicationappliercurrentlyhasencounteredinarow.Incontrast,thetotalFailedConnectsattributeindicateshowmanyfailedconnectionattemptstheapplierhasmadeintotal.ThetotalRequestsattributeshowshowmanyrequeststheapplierhassenttothemasterdatabaseintotal.ThetotalEventsattributeshowshowmanylogeventstheapplierhasreadfromthemaster.

Theprogress.messagesub-attributeprovidesabriefhintofwhattheappliercurrentlydoes(ifitisrunning).ThelastErrorattributealsohasanoptionalerrorMessagesub-attribute,showingthelatesterrormessage.TheerrorNumsub-attributeofthelastErrorattributecanbeusedbyclientstoprogrammaticallycheckforerrors.Itshouldbe0ifthereisnoerror,anditshouldbenon-zeroiftheapplierterminateditselfduetoaproblem.

Hereisanexampleofthestateafterthereplicationapplierterminateditselfdueto(repeated)connectionproblems:

{

"state":{

"running":false,

"progress":{

"time":"2014-07-06T13:14:37Z",

AsynchronousReplication

445

"message":"applierstopped",

"failedConnects":6

},

"totalRequests":79,

"totalFailedConnects":11,

"totalEvents":0,

"lastError":{

"time":"2014-07-06T13:09:41Z",

"errorMessage":"couldnotconnecttomasterattcp://master.example.org:8529:Couldnotconnectto'tcp:/...",

"errorNum":1400

},

...

}

}

Note:thestateofadatabase'sreplicationapplierisqueryableviatheHTTPAPI,too.PleaserefertoHTTPInterfaceforReplicationformoredetails.

All-in-onesetup

Tocopytheinitialdatafromtheslavetothemasterandstartthecontinuousreplication,thereisanall-in-onecommandsetupReplication:

require("@arangodb/replication").setupReplication(configuration);

Thefollowingexampledemonstrateshowtousethecommandforsettingupreplicationforthe_systemdatabase.Notethatitshouldberunontheslaveandnotthemaster:

db._useDatabase("_system");

require("@arangodb/replication").setupReplication({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

verbose:false,

includeSystem:false,

incremental:true,

autoResync:true

});

Thecommandwillreturnwhentheinitialsynchronizationisfinishedandthecontinuousreplicationisstarted,orincasetheinitialsynchronizationhasfailed.

Iftheinitialsynchronizationissuccessful,thecommandwillstorethegivenconfigurationontheslave.Italsoconfiguresthecontinuousreplicationtostartautomaticallyiftheslaveisrestarted,i.e.autoStartissettotrue.

Ifthecommandisrunwhiletheslave'sreplicationapplierisalreadyrunning,itwillfirststoptherunningapplier,dropitsconfigurationanddoaresynchronizationofdatawiththemaster.Itwillthenusetheprovidedconfigration,overwritinganypreviouslyexistingreplicationconfigurationontheslave.

StartingandStopping

Tomanuallystartandstoptheapplierinthecurrentdatabase,thestartandstopcommandscanbeusedlikethis:

require("@arangodb/replication").applier.start(<tick>);

require("@arangodb/replication").applier.stop();

Note:Startingareplicationapplierwithoutsettingupaninitialconfigurationwillfail.ThereplicationapplierwilllookforitsconfigurationinafilenamedREPLICATION-APPLIER-CONFIGinthecurrentdatabase'sdirectory.Ifthefileisnotpresent,ArangoDBwillusesomedefaultconfiguration,butitcannotguesstheendpoint(theaddressofthemasterdatabase)theappliershouldconnectto.Thusstartingtheapplierwithoutconfigurationwillfail.

Notethatatthefirsttimeyoustarttheapplier,youshouldpassthevaluereturnedinthelastLogTickattributeoftheinitialsyncoperation.

AsynchronousReplication

446

Note:Startingadatabase'sreplicationapplierviathestartcommandwillnotnecessarilystarttheapplieronthenextandfollowingArangoDBserverrestarts.Additionally,stoppingadatabase'sreplicationappliermanuallywillnotnecessarilypreventtheapplierfrombeingstartedagainonthenextserverstart.Allofthisisconfigurableseparately(hangonreading).

Note:whenstoppingandrestartingthereplicationapplierofdatabase,itwillresumewhereitlaststopped.Thisissensiblebecausereplicationlogeventsshouldbeappliedincrementally.Ifthereplicationapplierofadatabasehasneverbeenstartedbefore,itneedssometickvaluefromthemaster'slogfromwhichtostartfetchingevents.

Thereisonecaveattoconsiderwhenstoppingareplicationontheslave:iftherearestillongoingreplicatedtransactionsthatareneithercommittedoraborted,stoppingthereplicationapplierwillcausetheseoperationstobelostfortheslave.Ifthesetransactionscommitonthemasterlaterandthereplicationisresumed,theslavewillnotbeabletocommitthesetransactions,too.Thusstoppingthereplicationapplierontheslavemanuallyshouldonlybedoneifthereiscertaintythattherearenoongoingtransactionsonthemaster.

Configuration

Toconfigurethereplicationapplierofaspecificdatabase,usethepropertiescommand.Usingitwithoutanyargumentswillreturntheapplier'scurrentconfiguration:

require("@arangodb/replication").applier.properties();

Theresultmightlooklikethis:

{

"requestTimeout":600,

"connectTimeout":10,

"ignoreErrors":0,

"maxConnectRetries":10,

"chunkSize":0,

"autoStart":false,

"adaptivePolling":true,

"includeSystem":true,

"requireFromPresent":false,

"autoResync":false,

"autoResyncRetries":2,

"verbose":false

}

Note:Thereisnoendpointattributeconfiguredyet.Theendpointattributeisrequiredforthereplicationappliertobestartable.Youmayalsowanttoconfigureausernameandpasswordfortheconnectionviatheusernameandpasswordattributes.

require("@arangodb/replication").applier.properties({

endpoint:"tcp://master.domain.org:8529",

username:"root",

password:"secret",

verbose:false

});

Thiswillre-configurethereplicationapplierforthecurrentdatabase.Theconfigurationwillbeusedfromthenextstartofthereplicationapplier.Thereplicationappliercannotbere-configuredwhileitisrunning.Itmustbestoppedfirsttobere-configured.

TomakethereplicationapplierofthecurrentdatabasestartautomaticallywhentheArangoDBserverstarts,usetheautoStartattribute.

SettingtheadaptivePollingattributetotruewillmakethereplicationapplierpollthemasterdatabaseforchangeswithavariablefrequency.Thereplicationapplierwillthenlowerthefrequencywhenthemasterisidle,andincreaseitwhenthemastercanprovidenewevents).Otherwisethereplicationapplierwillpollthemasterdatabaseforchangeswithaconstantfrequency.

TheidleMinWaitTimeattributecontrolstheminimumwaittime(inseconds)thatthereplicationapplierwillintentionallyidlebeforefetchingmorelogdatafromthemasterincasethemasterhasalreadysentallitslogdata.ThiswaittimecanbeusedtocontrolthefrequencywithwhichthereplicationappliersendsHTTPlogfetchrequeststothemasterincasethereisnowriteactivityonthemaster.

TheidleMaxWaitTimeattributecontrolsthemaximumwaittime(inseconds)thatthereplicationapplierwillintentionallyidlebeforefetchingmorelogdatafromthemasterincasethemasterhasalreadysentallitslogdataandtherehavebeenpreviouslogfetchattemptsthatresultedinnomorelogdata.Thiswaittimecanbeusedtocontrolthemaximumfrequencywithwhichthereplicationappliersends

AsynchronousReplication

447

HTTPlogfetchrequeststothemasterincasethereisnowriteactivityonthemasterforlongerperiods.NotethatthisconfigurationvaluewillonlybeusediftheoptionadaptivePollingissettotrue.

Tosetatimeoutforconnectionandfollowingrequestattempts,usetheconnectTimeoutandrequestTimeoutvalues.ThemaxConnectRetriesattributeconfiguresafterhowmanyfailedconnectionattemptsinarowthereplicationapplierwillgiveupandturnitselfoff.Youmaywanttosetthistoahighvaluesothattemporarynetworkoutagesdonotleadtothereplicationapplierstoppingitself.TheconnectRetryWaitTimeattributeconfigureshowlongthereplicationapplierwillwaitbeforeretryingtheconnectiontothemasterincaseofconnectionproblems.

ThechunkSizeattributecanbeusedtocontroltheapproximatemaximumsizeofamaster'sresponse(inbytes).Settingittoalowvaluemaymakethemasterrespondfaster(lessdataisassembledbeforethemastersendstheresponse),butmayrequiremorerequest-responseroundtrips.Setitto0touseArangoDB'sbuilt-indefaultvalue.

TheincludeSystemattributecontrolswhetherchangestosystemcollections(suchas_graphsor_users)shouldbeapplied.Ifsettotrue,changesinthesecollectionswillbereplicated,otherwise,theywillnotbereplicated.Itisoftennotnecessarytoreplicatedatafromsystemcollections,especiallybecauseitmayleadtoconfusionontheslavebecausetheslaveneedstohaveitsownsystemcollectionsinordertostartandkeepoperational.

TherequireFromPresentattributecontrolswhethertheapplierwillstartsynchronizingincaseitdetectsthatthemastercannotprovidedatafortheinitialtickvalueprovidedbytheslave.ThismaybethecaseifthemasterdoesnothaveabigenoughbacklogofhistoricWALlogfiles,andwhenthereplicationisre-startedafteralongerpause.WhenrequireFromPresentissettotrue,thenthereplicationapplierwillcheckatstartwhetherthestarttickfromwhichitstartsorresumesreplicationisstillpresentonthemaster.Ifnot,thentherewouldbedataloss.IfrequireFromPresentistrue,thereplicationapplierwillabortwithanappropriateerrormessage.Ifsettofalse,thenthereplicationapplierwillstillstart,andignorethedataloss.

TheautoResyncoptioncanbeusedinconjunctionwiththerequireFromPresentoptionasfollows:whenbothrequireFromPresentandautoResyncaresettotrueandthemastercannotprovidethelogdatatheslaverequests,thereplicationapplierwillstopasusual.ButduetothefactthatautoResyncissettotrue,theslavewillautomaticallytriggerafullresyncofalldatawiththemaster.Afterthat,thereplicationapplierwillgointocontinuousreplicationmodeagain.Additionally,settingautoResynctotruewilltriggerafullre-synchronizationofdatawhenthecontinuousreplicationisstartedanddetectsthatthereisnostarttickvalue.

Automaticre-synchronizationmaytransferalotofdatafromthemastertotheslaveandcanbeexpensive.Itisthereforeturnedoffbydefault.Whenturnedoff,theslavewillneverperformanautomaticre-synchronizationwiththemaster.

TheautoResyncRetriesoptioncanbeusedtocontrolthenumberofresynchronizationretriesthatwillbeperformedinarowwhenautomaticresynchronizationisenabledandkicksin.Settingthisto0willeffectivelydisableautoResync.Settingittosomeothervaluewilllimitthenumberofretriesthatareperformed.Thishelpspreventingendlessretriesincaseresynchronizationsalwaysfail.

Theverboseattributecontrolstheverbosityofthereplicationlogger.Settingittotruewillmakethereplicationapplierwritealinetothelogforeveryoperationitperforms.Thisshouldonlybeusedfordiagnosingreplicationproblems.

Thefollowingexamplewillsetmostofthediscussedpropertiesforthecurrentdatabase'sapplier:

require("@arangodb/replication").applier.properties({

endpoint:"tcp://master.domain.org:8529",

username:"root",

password:"secret",

adaptivePolling:true,

connectTimeout:15,

maxConnectRetries:100,

chunkSize:262144,

autoStart:true,

includeSystem:true,

autoResync:true,

autoResyncRetries:2,

});

Aftertheapplierisnowfullyconfigured,itcouldtheoreticallybestarted.However,wemayfirstneedaninitialsynchronizationofallcollectionsandtheirdatafromthemasterbeforewestartthereplicationapplier.

Theonlysafemethodfordoingafullsynchronization(orre-synchronization)isthusto

stopthereplicationapplierontheslave(ifcurrentlyrunning)performaninitialfullsyncwiththemasterdatabasenotethemasterdatabase'slastLogTickvalueand

AsynchronousReplication

448

startthecontinuousreplicationapplierontheslaveusingthistickvalue.

Theinitialsynchronizationforthecurrentdatabaseisexecutedwiththesynccommand:

require("@arangodb/replication").sync({

endpoint:"tcp://master.domain.org:8529",

username:"root",

password:"secret,

includeSystem:true

});

TheincludeSystemoptioncontrolswhetherdatafromsystemcollections(suchas_graphsand_users)shallbesynchronized.

TheinitialsynchronizationcanoptionallybeconfiguredtoincludeorexcludespecificcollectionsusingtherestrictTypeandrestrictCollectionparameters.

Thefollowingcommandonlysynchronizescollectionfooandbar:

require("@arangodb/replication").sync({

endpoint:"tcp://master.domain.org:8529",

username:"root",

password:"secret,

restrictType:"include",

restrictCollections:["foo","bar"]

});

UsingarestrictTypeofexclude,allcollectionsbutthespecifiedwillbesynchronized.

Warning:syncwilldoafullsynchronizationofthecollectionsinthecurrentdatabasewithcollectionspresentinthemasterdatabase.Anylocalinstancesofthecollectionsandalltheirdataareremoved!Onlyexecutethiscommandifyouaresureyouwanttoremovethelocaldata!

Assyncdoesafullsynchronization,itmighttakeawhiletoexecute.Whensynccompletessuccessfully,itreturnsanarrayofcollectionsithassynchronizedinitscollectionsattribute.Itwillalsoreturnthemasterdatabase'slastlogtickvalueatthetimethesyncwasstartedonthemaster.ThetickvalueiscontainedinthelastLogTickattributeofthesynccommand:

{

"lastLogTick":"231848833079705",

"collections":[...]

}

Nowyoucanstartthecontinuoussynchronizationforthecurrentdatabaseontheslavewiththecommand

require("@arangodb/replication").applier.start("231848833079705");

Note:Thetickvaluesshouldbetreatedasstrings.Usingnumericdatatypesfortickvaluesisunsafebecausetheymightexceedthe32bitvalueandtheIEEE754doubleaccuracyranges.

AsynchronousReplication

449

Per-DatabaseSetupThispagedescribesthereplicationprocessbasedonaspecificdatabasewithinanArangoDBinstance.Thatmeansthatonlythespecifieddatabasewillbereplicated.

Settingupaworkingmaster-slavereplicationrequirestwoArangoDBinstances:

master:thisistheinstancethatalldata-modificationoperationsshouldbedirectedtoslave:onthisinstance,we'llstartareplicationapplier,andthiswillfetchdatafromthemasterdatabase'swrite-aheadlogandapplyitsoperationslocally

Forthefollowingexamplesetup,we'llusetheinstancetcp://master.domain.org:8529asthemaster,andtheinstancetcp://slave.domain.org:8530asaslave.

Thegoalistohavealldatafromthedatabase_systemonmastertcp://master.domain.org:8529bereplicatedtothedatabase_systemontheslavetcp://slave.domain.org:8530.

Onthemaster,nothingspecialneedstobedone,asallwriteoperationswillautomaticallybeloggedinthemaster'swrite-aheadlog(WAL).

All-in-onesetup

Tomakethereplicationcopytheinitialdatafromthemastertotheslaveandstartthecontinuousreplicationontheslave,thereisanall-in-onecommand:

require("@arangodb/replication").setupReplication(configuration);

Thefollowingexampledemonstrateshowtousethecommandforsettingupreplicationforthe_systemdatabase.Notethatitshouldberunontheslaveandnotthemaster:

db._useDatabase("_system");

require("@arangodb/replication").setupReplication({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

verbose:false,

includeSystem:false,

incremental:true,

autoResync:true

});

Thecommandwillreturnwhentheinitialsynchronizationisfinishedandthecontinuousreplicationhasbeenstarted,orincasetheinitialsynchronizationhasfailed.

Iftheinitialsynchronizationissuccessful,thecommandwillstorethegivenconfigurationontheslave.Italsoconfiguresthecontinuousreplicationtostartautomaticallyiftheslaveisrestarted,i.e.autoStartissettotrue.

Ifthecommandisrunwhiletheslave'sreplicationapplierisalreadyrunning,itwillfirststoptherunningapplier,dropitsconfigurationanddoaresynchronizationofdatawiththemaster.Itwillthenusetheprovidedconfigration,overwritinganypreviouslyexistingreplicationconfigurationontheslave.

Initialsynchronization

Theinitialsynchronizationandcontinuousreplicationappliercanalsobestartedseparately.Tostartreplicationontheslave,makesuretherecurrentlyisnoreplicationapplierrunning.

Thefollowingcommandsstoparunningapplierintheslave's_systemdatabase:

db._useDatabase("_system");

AsynchronousReplication

450

require("@arangodb/replication").applier.stop();

Thestopoperationwillterminateanyreplicationactivityinthe_systemdatabaseontheslave.

Afterthat,theinitialsynchronizationcanberun.Itwillcopythecollectionsfromthemastertotheslave,overwritingexistingdata.Toruntheinitialsynchronization,executethefollowingcommandsontheslave:

db._useDatabase("_system");

require("@arangodb/replication").sync({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

verbose:false

});

Usernameandpasswordonlyneedtobespecifiedwhenthemasterrequiresauthentication.Tocheckwhatthesynchronizationiscurrentlydoing,supplysettheverboseoptiontotrue.Ifset,thesynchronizationwillcreatelogmessageswiththecurrentsynchronizationstatus.

Warning:Thesynccommandwillreplacedataintheslavedatabasewithdatafromthemasterdatabase!Onlyexecutethesecommandsifyouhaveverifiedyouareonthecorrectserver,inthecorrectdatabase!

ThesyncoperationwillreturnanattributenamedlastLogTickwhichwe'llneedtonote.Thelastlogtickwillbeusedasthestartingpointforsubsequentreplicationactivity.Let'sassumewegotthefollowinglastlogtick:

{

"lastLogTick":"40694126",

...

}

InitialsynchronizationfromtheArangoShell

Theinitialsynchronizationviathesynccommandmaytakealongtimetocomplete.Theshellwillblockuntiltheslavehascompletedtheinitialsynchronizationoruntilanerroroccurs.Bydefault,thesynccommandintheArangoShellwillpolltheslaveforastatusupdateevery10seconds.

Optionallythesynccommandcanbemadenon-blockingbysettingitsasyncoptiontotrue.Inthiscase,thesynccommandwillreturninstantlywithanidstring,andtheinitialsynchronizationwillrundetachedonthemaster.TofetchthecurrentstatusofthesyncprogressfromtheArangoShell,thegetSyncResultfunctioncanbeusedasfollows:

db._useDatabase("_system");

varreplication=require("@arangodb/replication");

/*runcommandinasyncmode*/

varid=replication.sync({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

async:true

});

/*nowquerythestatusofouroperation*/

print(replication.getSyncResult(id));

getSyncResultwillreturnfalseaslongasthesynchronizationisnotcomplete,andreturnthesynchronizationresultotherwise.

Continuoussynchronization

Whentheinitialsynchronizationisfinished,thecontinuousreplicationappliercanbestartedusingthelastlogtickprovidedbythesynccommand.Beforestartingit,thereisatleastoneconfigurationoptiontoconsider:replicationontheslavewillberunninguntiltheslavegetsshutdown.Whentheslaveservergetsrestarted,replicationwillbeturnedoffagain.Tochangethis,wefirstneedtoconfiguretheslave'sreplicationapplierandsetitsautoStartattribute.

AsynchronousReplication

451

Here'sthecommandtoconfigurethereplicationapplierwithseveraloptions,includingtheautoStartattribute:

db._useDatabase("_system");

require("@arangodb/replication").applier.properties({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

autoStart:true,

autoResync:true,

autoResyncRetries:2,

adaptivePolling:true,

includeSystem:false,

requireFromPresent:false,

idleMinWaitTime:0.5,

idleMaxWaitTime:1.5,

verbose:false

});

Animportantconsiderationforreplicationiswhetherdatafromsystemcollections(suchas_graphsor_users)shouldbeapplied.TheincludeSystemoptioncontrolsthat.Ifsettotrue,changesinsystemcollectionswillbereplicated.Otherwise,theywillnotbereplicated.Itisoftennotnecessarytoreplicatedatafromsystemcollections,especiallybecauseitmayleadtoconfusionontheslavebecausetheslaveneedstohaveitsownsystemcollectionsinordertostartandkeepoperational.

TherequireFromPresentattributecontrolswhethertheapplierwillstartsynchronizingincaseitdetectsthatthemastercannotprovidedatafortheinitialtickvalueprovidedbytheslave.ThismaybethecaseifthemasterdoesnothaveabigenoughbacklogofhistoricWALlogfiles,andwhenthereplicationisre-startedafteralongerpause.WhenrequireFromPresentissettotrue,thenthereplicationapplierwillcheckatstartwhetherthestarttickfromwhichitstartsorresumesreplicationisstillpresentonthemaster.Ifnot,thentherewouldbedataloss.IfrequireFromPresentistrue,thereplicationapplierwillabortwithanappropriateerrormessage.Ifsettofalse,thenthereplicationapplierwillstillstart,andignorethedataloss.

TheautoResyncoptioncanbeusedinconjunctionwiththerequireFromPresentoptionasfollows:whenbothrequireFromPresentandautoResyncaresettotrueandthemastercannotprovidethelogdatatheslavehadrequested,thereplicationapplierwillstopasusual.ButduetothefactthatautoResyncissettotrue,theslavewillautomaticallytriggerafullresyncofalldatawiththemaster.Afterthat,thereplicationapplierwillgointocontinuousreplicationmodeagain.Additionally,settingautoResynctotruewilltriggerafullre-synchronizationofdatawhenthecontinuousreplicationisstartedanddetectsthatthereisnostarttickvalue.

Notethatautomaticre-synchronization(autoResyncoptionsettotrue)maytransferalotofdatafromthemastertotheslaveandcanthereforebeexpensive.Stillit'sturnedonheresothere'slessneedformanualintervention.

TheautoResyncRetriesoptioncanbeusedtocontrolthenumberofresynchronizationretriesthatwillbeperformedinarowwhenautomaticresynchronizationisenabledandkicksin.Settingthisto0willeffectivelydisableautoResync.Settingittosomeothervaluewilllimitthenumberofretriesthatareperformed.Thishelpspreventingendlessretriesincaseresynchronizationsalwaysfail.

Nowit'stimetostartthereplicationapplierontheslaveusingthelastlogtickwegotbefore:

db._useDatabase("_system");

require("@arangodb/replication").applier.start("40694126");

Thiswillreplicatealloperationshappeninginthemaster'ssystemdatabaseandapplythemontheslave,too.

Afterthat,youshouldbeabletomonitorthestateandprogressofthereplicationapplierbyexecutingthestatecommandontheslaveserver:

db._useDatabase("_system");

require("@arangodb/replication").applier.state();

Pleasenotethatstoppingthereplicationapplierontheslaveusingthestopcommandshouldbeavoided.Thereasonisthatcurrentlyongoingtransactions(thathavepartlybeenreplicatedtotheslave)willbeneedtoberestartedafterarestartofthereplicationapplier.Stoppingandrestartingthereplicationapplierontheslaveshouldthusonlybeperformedifthereiscertaintythatthemasteriscurrentlyfullyidleandalltransactionshavebeenreplicatedfully.

Notethatwhileaslavehasonlypartlyexecutedatransactionfromthemaster,itmightkeepawritelockonthecollectionsinvolvedinthetransaction.

AsynchronousReplication

452

YoumayalsowanttocheckthemasterandslavestatesviatheHTTPAPIs(seeHTTPInterfaceforReplication).

AsynchronousReplication

453

Server-levelSetupThispagedescribesthereplicationprocessbasedonacompleteArangoDBinstance.Thatmeansthatallincludeddatabaseswillbereplicated.

Settingupaworkingmaster-slavereplicationrequirestwoArangoDBinstances:

master:thisistheinstancethatalldata-modificationoperationsshouldbedirectedtoslave:onthisinstance,we'llstartareplicationapplier,andthiswillfetchdatafromthemasterdatabase'swrite-aheadlogandapplyitsoperationslocally

Forthefollowingexamplesetup,we'llusetheinstancetcp://master.domain.org:8529asthemaster,andtheinstancetcp://slave.domain.org:8530asaslave.

Thegoalistohavealldataofalldatabasesonmastertcp://master.domain.org:8529bereplicatedtotheslaveinstancetcp://slave.domain.org:8530.

Onthemaster,nothingspecialneedstobedone,asallwriteoperationswillautomaticallybeloggedinthemaster'swrite-aheadlog(WAL).

All-in-onesetup

Tomakethereplicationcopytheinitialdatafromthemastertotheslaveandstartthecontinuousreplicationontheslave,thereisanall-in-onecommand:

require("@arangodb/replication").setupReplicationGlobal(configuration);

ThefollowingexampledemonstrateshowtousethecommandforsettingupreplicationforthecompleteArangoDBinstance.Notethatitshouldberunontheslaveandnotthemaster:

db._useDatabase("_system");

require("@arangodb/replication").setupReplicationGlobal({

endpoint:"tcp://127.0.0.1:8529",

username:"root",

password:"",

autoStart:true

});

Thecommandwillreturnwhentheinitialsynchronizationisfinishedandthecontinuousreplicationhasbeenstarted,orincasetheinitialsynchronizationhasfailed.

Iftheinitialsynchronizationissuccessful,thecommandwillstorethegivenconfigurationontheslave.Italsoconfiguresthecontinuousreplicationtostartautomaticallyiftheslaveisrestarted,i.e.autoStartissettotrue.

Ifthecommandisrunwhiletheslave'sreplicationapplierisalreadyrunning,itwillfirststoptherunningapplier,dropitsconfigurationanddoaresynchronizationofdatawiththemaster.Itwillthenusetheprovidedconfigration,overwritinganypreviouslyexistingreplicationconfigurationontheslave.

Stoppingsynchronization

Theinitialsynchronizationandcontinuousreplicationappliercanalsobestartedseparately.Tostartreplicationontheslave,makesuretherecurrentlyisnoreplicationapplierrunning.

Thefollowingcommandsstoparunningapplierintheslave'sinstance:

db._useDatabase("_system");

require("@arangodb/replication").globalApplier.stop();

AsynchronousReplication

454

ThestopoperationwillterminateanyreplicationactivityintheArangoDBinstanceontheslave.

Afterthat,theinitialsynchronizationcanberun.Itwillcopythecollectionsfromthemastertotheslave,overwritingexistingdata.Toruntheinitialsynchronization,executethefollowingcommandsontheslave:

db._useDatabase("_system");

require("@arangodb/replication").syncGlobal({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

verbose:false

});

Usernameandpasswordonlyneedtobespecifiedwhenthemasterrequiresauthentication.Tocheckwhatthesynchronizationiscurrentlydoing,supplysettheverboseoptiontotrue.Ifset,thesynchronizationwillcreatelogmessageswiththecurrentsynchronizationstatus.

Warning:ThesyncGlobalcommandwillreplacedataintheslavedatabasewithdatafromthemasterdatabase!Onlyexecutethesecommandsifyouhaveverifiedyouareonthecorrectserver,inthecorrectdatabase!

ThesyncoperationwillreturnanattributenamedlastLogTickwhichwe'llneedtonote.Thelastlogtickwillbeusedasthestartingpointforsubsequentreplicationactivity.Let'sassumewegotthefollowinglastlogtick:

{

"lastLogTick":"40694126",

...

}

InitialsynchronizationfromtheArangoShellTheinitialsynchronizationviathesyncGlobalcommandmaytakealongtimetocomplete.Theshellwillblockuntiltheslavehascompletedtheinitialsynchronizationoruntilanerroroccurs.Bydefault,thesyncGlobalcommandintheArangoShellwillpolltheslaveforastatusupdateevery10seconds.

OptionallythesyncGlobalcommandcanbemadenon-blockingbysettingitsasyncoptiontotrue.Inthiscase,thesyncGlobalcommandwillreturninstantlywithanidstring,andtheinitialsynchronizationwillrundetachedonthemaster.TofetchthecurrentstatusofthesyncGlobalprogressfromtheArangoShell,thegetSyncResultfunctioncanbeusedasfollows:

db._useDatabase("_system");

varreplication=require("@arangodb/replication");

/*runcommandinasyncmode*/

varid=replication.syncGlobal({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

async:true

});

/*nowquerythestatusofouroperation*/

print(replication.getSyncResult(id));

getSyncResultwillreturnfalseaslongasthesynchronizationisnotcomplete,andreturnthesynchronizationresultotherwise.

Continuoussynchronization

Whentheinitialsynchronizationisfinished,thecontinuousreplicationappliercanbestartedusingthelastlogtickprovidedbythesyncGlobalcommand.Beforestartingit,thereisatleastoneconfigurationoptiontoconsider:replicationontheslavewillberunninguntiltheslavegetsshutdown.Whentheslaveservergetsrestarted,replicationwillbeturnedoffagain.Tochangethis,wefirstneedtoconfiguretheslave'sreplicationapplierandsetitsautoStartattribute.

Here'sthecommandtoconfigurethereplicationapplierwithseveraloptions,includingtheautoStartattribute:

AsynchronousReplication

455

db._useDatabase("_system");

require("@arangodb/replication").globalApplier.properties({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

autoStart:true,

autoResync:true,

autoResyncRetries:2,

adaptivePolling:true,

includeSystem:false,

requireFromPresent:false,

idleMinWaitTime:0.5,

idleMaxWaitTime:1.5,

verbose:false

});

Animportantconsiderationforreplicationiswhetherdatafromsystemcollections(suchas_graphsor_users)shouldbeapplied.TheincludeSystemoptioncontrolsthat.Ifsettotrue,changesinsystemcollectionswillbereplicated.Otherwise,theywillnotbereplicated.Itisoftennotnecessarytoreplicatedatafromsystemcollections,especiallybecauseitmayleadtoconfusionontheslavebecausetheslaveneedstohaveitsownsystemcollectionsinordertostartandkeepoperational.

TherequireFromPresentattributecontrolswhethertheapplierwillstartsynchronizingincaseitdetectsthatthemastercannotprovidedatafortheinitialtickvalueprovidedbytheslave.ThismaybethecaseifthemasterdoesnothaveabigenoughbacklogofhistoricWALlogfiles,andwhenthereplicationisre-startedafteralongerpause.WhenrequireFromPresentissettotrue,thenthereplicationapplierwillcheckatstartwhetherthestarttickfromwhichitstartsorresumesreplicationisstillpresentonthemaster.Ifnot,thentherewouldbedataloss.IfrequireFromPresentistrue,thereplicationapplierwillabortwithanappropriateerrormessage.Ifsettofalse,thenthereplicationapplierwillstillstart,andignorethedataloss.

TheautoResyncoptioncanbeusedinconjunctionwiththerequireFromPresentoptionasfollows:whenbothrequireFromPresentandautoResyncaresettotrueandthemastercannotprovidethelogdatatheslavehadrequested,thereplicationapplierwillstopasusual.ButduetothefactthatautoResyncissettotrue,theslavewillautomaticallytriggerafullresyncofalldatawiththemaster.Afterthat,thereplicationapplierwillgointocontinuousreplicationmodeagain.Additionally,settingautoResynctotruewilltriggerafullre-synchronizationofdatawhenthecontinuousreplicationisstartedanddetectsthatthereisnostarttickvalue.

Notethatautomaticre-synchronization(autoResyncoptionsettotrue)maytransferalotofdatafromthemastertotheslaveandcanthereforebeexpensive.Stillit'sturnedonheresothere'slessneedformanualintervention.

TheautoResyncRetriesoptioncanbeusedtocontrolthenumberofresynchronizationretriesthatwillbeperformedinarowwhenautomaticresynchronizationisenabledandkicksin.Settingthisto0willeffectivelydisableautoResync.Settingittosomeothervaluewilllimitthenumberofretriesthatareperformed.Thishelpspreventingendlessretriesincaseresynchronizationsalwaysfail.

Nowit'stimetostartthereplicationapplierontheslaveusingthelastlogtickwegotbefore:

db._useDatabase("_system");

require("@arangodb/replication").globalApplier.start("40694126");

Thiswillreplicatealloperationshappeninginthemaster'ssystemdatabaseandapplythemontheslave,too.

Afterthat,youshouldbeabletomonitorthestateandprogressofthereplicationapplierbyexecutingthestatecommandontheslaveserver:

db._useDatabase("_system");

require("@arangodb/replication").globalApplier.state();

Pleasenotethatstoppingthereplicationapplierontheslaveusingthestopcommandshouldbeavoided.Thereasonisthatcurrentlyongoingtransactions(thathavepartlybeenreplicatedtotheslave)willbeneedtoberestartedafterarestartofthereplicationapplier.Stoppingandrestartingthereplicationapplierontheslaveshouldthusonlybeperformedifthereiscertaintythatthemasteriscurrentlyfullyidleandalltransactionshavebeenreplicatedfully.

Notethatwhileaslavehasonlypartlyexecutedatransactionfromthemaster,itmightkeepawritelockonthecollectionsinvolvedinthetransaction.

YoumayalsowanttocheckthemasterandslavestatesviatheHTTPAPIs(seeHTTPInterfaceforReplication).

AsynchronousReplication

456

AsynchronousReplication

457

SyncingCollectionsInordertosynchronizedataforasinglecollectionfromamastertoaslaveinstance,thereisthesyncCollectionfunction:

Itwillfetchalldocumentsofthespecifiedcollectionfromthemasterdatabaseandstoretheminthelocalinstance.Afterthesynchronization,thecollectiondataontheslavewillbeidenticaltothedataonthemaster,providednofurtherdatachangeshappenonthemaster.AnydatachangesthatareperformedonthemasterafterthesynchronizationwasstartedwillnotbecapturedbysyncCollection,butneedtobereplicatedusingtheregularreplicationappliermechanism.

Forthefollowingexamplesetup,we'llusetheinstancetcp://master.domain.org:8529asthemaster,andtheinstancetcp://slave.domain.org:8530asaslave.

Thegoalistohavealldatafromthecollectiontestindatabase_systemonmastertcp://master.domain.org:8529bereplicatedtothecollectiontestindatabase_systemontheslavetcp://slave.domain.org:8530.

Onthemaster,thecollectiontestneedstobepresentinthe_systemdatabase,withanydatainit.

Totransferthiscollectiontotheslave,issuethefollowingcommandsthere:

db._useDatabase("_system");

require("@arangodb/replication").syncCollection("test",{

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd"

});

Warning:ThesyncCollectioncommandwillreplacethecollection'sdataintheslavedatabasewithdatafromthemasterdatabase!Onlyexecutethesecommandsifyouhaveverifiedyouareonthecorrectserver,inthecorrectdatabase!

SettingtheoptionalincrementalattributeinthecalltosyncCollectionwillstartanincrementaltransferofdata.Thismaybeusefulincasewhentheslavealreadyhaspartsoralmostallofthedatainthecollectionandonlythedifferencesneedtobesynchronized.Notethattocomputethedifferencestheincrementaltransferwillbuildasortedlistofalldocumentkeysinthecollectiononboththeslaveandthemaster,whichmaystillbeexpensiveforhugecollectionsintermsofmemoryusageandruntime.Duringbuildingthelistofkeysthecollectionwillberead-lockedonthemaster.

TheinitialSyncMaxWaitTimeattributeinthecalltosyncCollectioncontrolshowlongtheslavewillwaitforamaster'sresponse.Thiswaittimecanbeusedtocontrolafterwhattimethesynchronizationwillgiveupandfail.

ThesyncCollectioncommandmaytakealongtimetocompleteifthecollectionisbig.Theshellwillblockuntiltheslavehassynchronizedtheentirecollectionfromthemasteroruntilanerroroccurs.Bydefault,thesyncCollectioncommandintheArangoShellwillpollforastatusupdateevery10seconds.

WhensyncCollectioniscalledfromtheArangoShell,theoptionalasyncattributecanbeusedtostartthesynchronizationasabackgroundprocessontheslave.Ifasyncissettotrue,thecalltosyncCollectionwillreturnalmostinstantlywithanidstring.Usingthisidstring,thestatusofthesyncjobontheslavecanbequeriedusingthegetSyncResultfunctionasfollows:

db._useDatabase("_system");

varreplication=require("@arangodb/replication");

/*runcommandinasyncmode*/

varid=replication.syncCollection("test",{

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

async:true

});

/*nowquerythestatusofouroperation*/

print(replication.getSyncResult(id));

getSyncResultwillreturnfalseaslongasthesynchronizationisnotcomplete,andreturnthesynchronizationresultotherwise.

AsynchronousReplication

458

AsynchronousReplication

459

ReplicationLimitationsThereplicationinArangoDBhasafewlimitations.SomeoftheselimitationsmayberemovedinlaterversionsofArangoDB:

thereisnofeedbackfromtheslavestothemaster.Ifaslavecannotapplyaneventitgotfromthemaster,themasterwillhaveadifferentstateofdata.Inthiscase,thereplicationapplierontheslavewillstopandreportanerror.Administratorscantheneither"fix"theproblemorre-syncthedatafromthemastertotheslaveandstarttheapplieragain.atthemomentitisassumedthatonlythereplicationapplierexecuteswriteoperationsonaslave.ArangoDBcurrentlydoesnotpreventusersfromcarryingouttheirownwriteoperationsonslaves,thoughthismightleadtoundefinedbehaviorandthereplicationapplierstopping.whenareplicationslaveasksamasterforlogevents,thereplicationmasterwillreturnallwriteoperationsforuser-definedcollections,butitwillexcludewriteoperationsforcertainsystemcollections.Thefollowingcollectionsareexcludedintentionallyfromreplication:_apps,_trx,_replication,_configuration,_jobs,_queues,_sessions,_foxxlogandallstatisticscollections.Writeoperationsforthefollowingsystemcollectionscanbequeriedfromamaster:_aqlfunctions,_graphs,_users.Foxxapplicationsconsistofdatabaseentriesandapplicationscriptsinthefilesystem.ThefilesystempartsofFoxxapplicationsarenottrackedanywhereandthusnotreplicatedincurrentversionsofArangoDB.ToreplicateaFoxxapplication,itisrequiredtocopytheapplicationtotheremoteserverandinstallitthereusingthefoxx-managerutility.masterserversdonotknowwhichslavesareorwillbeconnectedtothem.Allserversinareplicationsetuparecurrentlyonlylooselycoupled.Therecurrentlyisnowayforaclienttoquerywhichserversarepresentinareplication.whennotusingourmesosintegrationfailovermustbehandledbyclientsorclientAPIs.therecurrentlyisonereplicationapplierperArangoDBdatabase.Itisthusnotpossibletohaveaslaveapplyoperationsfrommultiplemastersintothesametargetdatabase.replicationissetuponaper-databaselevel.WhenusingArangoDBwithmultipledatabases,replicationmustbeconfiguredindividuallyforeachdatabase.thereplicationapplierissingle-threaded,butwriteoperationsonthemastermaybeexecutedinparalleliftheyaffectdifferentcollections.Thusthereplicationappliermightnotbeabletocatchupwithaverypowerfulandloadedmaster.replicationisonlysupportedbetweenthetwoArangoDBserversrunningthesameArangoDBversion.ItiscurrentlynotpossibletoreplicatebetweendifferentArangoDBversions.areplicationappliercannotapplydatafromitself.

AsynchronousReplication

460

SynchronousReplicationAtitscoresynchronousreplicationwillreplicatewriteoperationstomultiplehosts.ThisfeatureisonlyavailablewhenoperatingArangoDBinacluster.Wheneveracoordinatorexecutesasychronouslyreplicatedwriteoperationitwillonlybereportedtobesuccessfulifitwascarriedoutonallreplicas.IncontrasttomultimasterreplicationsetupsknownfromothersystemsArangoDB'ssynchronousoperationguaranteesaconsistentstateacrossthecluster.

SynchronousReplication

461

Implementation

Architectureinsidethecluster

SynchronousreplicationcanbeconfiguredpercollectionviathepropertyreplicationFactor.Synchronousreplicationrequiresaclustertooperate.

WheneveryouspecifyareplicationFactorgreaterthan1whencreatingacollection,synchronousreplicationwillbeactivatedforthiscollection.Theclusterwilldeterminesuitableleadersandfollowersforeveryrequestedshard(numberOfShards)withinthecluster.Whenrequestingdataofashardonlythecurrentleaderwillbeaskedwhereasfollowerswillonlykeeptheircopyinsync.Thisisduetothecurrentimplementationoftransactions.

Usingsynchronousreplicationalonewillguaranteeconsistencyandhighavailabiltyatthecostofreducedperformance:Writerequestswillhaveahigherlatency(duetoeverywrite-requesthavingtobeexecutedonthefollowers)andreadrequestswon'tscaleoutasonlytheleaderisbeingasked.

Inaclustersynchronousreplicationwillbemanagedbythecoordinatorsfortheclient.Thedatawillalwaysbestoredonprimaries.

ThefollowingexamplewillgiveyouanideaofhowsynchronousoperationhasbeenimplementedinArangoDB.

1. Connecttoacoordinatorviaarangosh2. Createacollection

127.0.0.1:8530@_system>db._create("test",{"replicationFactor":2})

3. thecoordinatorwillfigureoutaleaderand1followerandcreate1shard(asthisisthedefault)

4. Insertdata

127.0.0.1:8530@_system>db.test.insert({"replication":"ᚇ"})

5. Thecoordinatorwillwritethedatatotheleader,whichinturnwillreplicateittothefollower.

6. Onlywhenbothweresuccessfultheresultisreportedtobesuccessful

{

"_id":"test/7987",

"_key":"7987",

"_rev":"7987"

}

Whenafollowerfails,theleaderwillgiveuponitafter3secondsandproceedwiththeoperation.Assoonasthefollower(orthenetworkconnectiontotheleader)isbackup,thetwowillresynchronizeandsynchronousreplicationisresumed.Thishappensalltransparentlytotheclient.

ThecurrentimplementationofArangoDBdoesnotallowchangingthereplicationFactorlater.Thisissubjecttochange.Inthemeantimetheonlywayistodumpandrestorethecollection.Seethecookbookrecipeaboutmigrating.

Automaticfailover

Whenevertheleaderofashardisfailingandthereisaquerytryingtoaccessdataofthatshardthecoordinatorwillcontinuetryingtocontacttheleaderuntilittimeouts.Theinternalclustersupervisionrunningontheagencywillcheckclusterhealtheveryfewsecondsandwilltakeactionifthereisnoheartbeatfromaserverfor15seconds.Iftheleaderdoesn'tcomebackintimethesupervisionwillreorganizetheclusterbypromotingforeachshardafollowerthatisinsyncwithitsleadertobethenewleader.Fromthenonthecoordinatorswillcontactthenewleader.

Theprocessisbestoutlinedusinganexample:

1. Theleaderofashard(letsnameitDBServer001)isgoingdown.2. Acoordinatorisaskedtoreturnadocument:

SynchronousReplication

462

127.0.0.1:8530@_system>db.test.document("100069")

3. ThecoordinatordetermineswhichserverisresponsibleforthisdocumentandfindsDBServer001

4. ThecoordinatortriestocontactDBServer001andtimeoutsbecauseitisnotreachable.5. Afterashortwhilethesupervision(runninginparallelontheagency)willseethatheartbeatsfromDBServer001arenotcomingin6. Thesupervisionpromotesoneofthefollowers(sayDBServer002)thatisinsynctobeleaderandmakesDBServer001afollower.7. AsthecoordinatorcontinuestryingtofetchthedocumentitwillseethattheleaderchangedtoDBServer0028. Thecoordinatortriestocontactthenewleader(DBServer002)andreturnstheresult:

{

"_key":"100069",

"_id":"test/100069",

"_rev":"513",

"replication":"ᚇ"

}

9. AfterawhilethesupervisiondeclaresDBServer001tobecompletelydead.10. AnewfollowerisdeterminedfromthepoolofDBservers.11. Thenewfollowersyncsitsdatafromtheleaderandorderisrestored.

Pleasenotethattheremaystillbetimeouts.Dependingonwhenexactlytherequesthasbeendone(inregardtothesupervision)anddependingonthetimeneededtoreconfiguretheclusterthecoordinatormightfailwithatimeouterror!

SynchronousReplication

463

Configuration

Requirements

SynchronousreplicationrequiresanoperationalArangoDBcluster.

Enablingsynchronousreplication

Synchronousreplicationcanbeenabledpercollection.WhencreatingacollectionyoumayspecifythenumberofreplicasusingthereplicationFactorparameter.Thedefaultvalueissetto1whicheffectivelydisablessynchronousreplication.

Example:

127.0.0.1:8530@_system>db._create("test",{"replicationFactor":3})

Intheabovecase,anywriteoperationwillrequire2replicastoreportsuccessfromnowon.

Preparinggrowth

Youmaycreateacollectionwithhigherreplicationfactorthanavailable.Whenadditionaldbserversbecomeavailabletheshardsareautomaticallyreplicatedtothenewlyavailablemachines.

Multiplereplicasofthesameshardcannevercoexistonthesamedbserverinstance.

SynchronousReplication

464

SatelliteCollectionsSatelliteCollectionsareanEnterpriseonlyfeature.WhendoingJoinsinanArangoDBclusterdatahastoexchangedbetweendifferentservers.

Joinswillbeexecutedonacoordinator.Itwillprepareanexecutionplanandexecuteit.Whenexecutingthecoordinatorwillcontactallshardsofthestartingpointofthejoinandaskfortheirdata.Thedatabaseserverscarryingoutthisoperationwillloadalltheirlocaldataandthenasktheclusterfortheotherpartofthejoin.Thisagainwillbedistributedtoallinvolvedshardsofthisjoinpart.

Insumthisresultsinmuchnetworktrafficandslowresultsdependingoftheamountofdatathathastobesentthroughoutthecluster.

Satellitecollectionsarecollectionsthatareintendedtoaddressthisissue.

Theywillfacilitatethesynchronousreplicationandreplicateallitsdatatoalldatabaseserversthatarepartofthecluster.

ThisenablesthedatabaseserverstoexecutethatpartofanyJoinlocally.

Thisgreatlyimprovesperformanceforsuchjoinsatthecostsofincreasedstoragerequirementsandpoorerwriteperformanceonthisdata.

TocreateasatellitecollectionsetthereplicationFactorofthiscollectionto"satellite".

Usingarangosh:

arangosh>db._create("satellite",{"replicationFactor":"satellite"});

Afullexample

arangosh>varexplain=require("@arangodb/aql/explainer").explain

arangosh>db._create("satellite",{"replicationFactor":"satellite"})

arangosh>db._create("nonsatellite",{numberOfShards:8})

arangosh>db._create("nonsatellite2",{numberOfShards:8})

Let'sanalyseanormaljoinnotinvolvingsatellitecollections:

arangosh>explain("FORdocinnonsatelliteFORdoc2innonsatellite2RETURN1")

Querystring:

FORdocinnonsatelliteFORdoc2innonsatellite2RETURN1

Executionplan:

IdNodeTypeSiteEst.Comment

1SingletonNodeDBS1*ROOT

4CalculationNodeDBS1-LET#2=1/*jsonexpression*//*constassignment*/

2EnumerateCollectionNodeDBS0-FORdocINnonsatellite/*fullcollectionscan*/

12RemoteNodeCOOR0-REMOTE

13GatherNodeCOOR0-GATHER

6ScatterNodeCOOR0-SCATTER

7RemoteNodeDBS0-REMOTE

3EnumerateCollectionNodeDBS0-FORdoc2INnonsatellite2/*fullcollectionscan*/

8RemoteNodeCOOR0-REMOTE

9GatherNodeCOOR0-GATHER

5ReturnNodeCOOR0-RETURN#2

Indexesused:

none

Optimizationrulesapplied:

IdRuleName

1move-calculations-up

2scatter-in-cluster

3remove-unnecessary-remote-scatter

SatelliteCollections

465

Allshardsinvolvedqueryingthenonsatellitecollectionwillfanoutviathecoordinatortotheshardsofnonsatellite.Insum8shardswillopen8connectionstothecoordinatoraskingfortheresultsofthenonsatellite2join.Thecoordinatorwillfanouttothe8shardsofnonsatellite2.Sotherewillbequitesomenetworktraffic.

Let'snowhavealookatthesameusingsatellitecollections:

arangosh>db._query("FORdocinnonsatelliteFORdoc2insatelliteRETURN1")

Querystring:

FORdocinnonsatelliteFORdoc2insatelliteRETURN1

Executionplan:

IdNodeTypeSiteEst.Comment

1SingletonNodeDBS1*ROOT

4CalculationNodeDBS1-LET#2=1/*jsonexpression*//*constassignment*/

2EnumerateCollectionNodeDBS0-FORdocINnonsatellite/*fullcollectionscan*/

3EnumerateCollectionNodeDBS0-FORdoc2INsatellite/*fullcollectionscan,satellite*/

8RemoteNodeCOOR0-REMOTE

9GatherNodeCOOR0-GATHER

5ReturnNodeCOOR0-RETURN#2

Indexesused:

none

Optimizationrulesapplied:

IdRuleName

1move-calculations-up

2scatter-in-cluster

3remove-unnecessary-remote-scatter

4remove-satellite-joins

Inthisscenarioallshardsofnonsatellitewillbecontacted.Howeverasthejoinisasatellitejoinallshardscandothejoinlocallyasthedataisreplicatedtoallserversreducingthenetworkoverheaddramatically.

Caveats

Theclusterwillautomaticallykeepallsatellitecollectionsonallserversinsyncbyfacilitatingthesynchronousreplication.Thismeansthatwritewillbeexecutedontheleaderonlyandthisserverwillcoordinatereplicationtothefollowers.Ifafollowerdoesn'tanswerintime(duetonetworkproblems,temporaryshutdownetc.)itmayberemovedasafollower.ThisisbeingreportedtotheAgency.

Thefollower(oncebackinbusiness)willthenperiodicallychecktheAgencyandknowthatitisoutofsync.Itwillthenautomaticallycatchup.Thismaytakeawhiledependingonhowmuchdatahastobesynced.WhendoingajoininvolvingthesatelliteyoucanspecifyhowlongtheDBServerisallowedtowaitforsyncuntilthequeryisbeingaborted.

CheckAccessingCursorsfordetails.

DuringnetworkfailurethereisalsoaminimalchancethataquerywasproperlydistributedtotheDBServersbutthataprevioussatellitewritecouldnotbereplicatedtoafollowerandtheleaderdroppedthefollower.Thefollowerhoweveronlycheckseveryfewsecondsifitisreallyinsyncsoitmightindeeddeliverstaleresults.

SatelliteCollections

466

DatacentertodatacenterreplicationadministrationThisSectionincludesinformationrelatedtotheadministrationofthedatacentertodatacenterreplication.

Forageneralintroductiontothedatacentertodatacenterreplication,pleaserefertotheDatacentertodatacenterreplicationchapther.

Startingsynchronization

OnceallcomponentsoftheArangoSyncsolutionhavebeendeployedandarerunningproperly,ArangoSyncwillnotautomaticallyreplicatedatabasestructureandcontent.Forthat,itisisneededtoconfiguresynchronization.

Toconfiguresynchronization,youneedthefollowing:

Theendpointofthesyncmasterinthetargetdatacenter.Theendpointofthesyncmasterinthesourcedatacenter.Acertificate(inkeyfileformat)usedforclientauthenticationofthesyncmaster(withthesyncmasterinthesourcedatacenter).ACAcertificate(publickeyonly)forverifyingtheintegrityofthesyncmasters.Ausername+passwordpair(orclientcertificate)forauthenticatingtheconfigurerequirewiththesyncmaster(inthetargetdatacenter)

Withthatinformation,run:

arangosyncconfiguresync\

--master.endpoint=<endpointsofsyncmastersintargetdatacenter>\

--master.keyfile=<keyfileofofsyncmastersintargetdatacenter>\

--source.endpoint=<endpointsofsyncmastersinsourcedatacenter>\

--source.cacert=<publickeyofCAcertificateusedtoverifysyncmasterinsourcedatacenter>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>

Thecommandwillfinishquickly.Afterwardsitwilltakesometimeuntiltheclustersinbothdatacentersareinsync.

Usethefollowingcommandtoinspectthestatusofthesynchronizationofadatacenter:

arangosyncgetstatus\

--master.endpoint=<endpointsofsyncmastersindatacenterofinterest>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>\

-v

Note:Invokingthiscommandonthetargetdatacenterwillreturndifferentresultsfrominvokingitonthesourcedatacenter.Youneedinsightinbothresultstogeta"completepicture".

Wherethegetstatuscommandgivesinsightinthestatusofsynchronization,therearemoredetailedcommandstogiveinsightintasks&registeredworkers.

Usethefollowingcommandtogetalistofallsynchronizationtasksinadatacenter:

arangosyncgettasks\

--master.endpoint=<endpointsofsyncmastersindatacenterofinterest>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>\

-v

Usethefollowingcommandtogetalistofallmastersinadatacenterandknowwhichmasteristhecurrentleader:

arangosyncgetmasters\

--master.endpoint=<endpointsofsyncmastersindatacenterofinterest>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>\

-v

Datacentertodatacenterreplication

467

Usethefollowingcommandtogetalistofallworkersinadatacenter:

arangosyncgetworkers\

--master.endpoint=<endpointsofsyncmastersindatacenterofinterest>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>\

-v

Stopingsynchronization

Ifyounolongerwanttosynchronizedatafromasourcetoatargetdatacenteryoumuststopit.Todoso,runthefollowingcommand:

arangosyncstopsync\

--master.endpoint=<endpointsofsyncmastersintargetdatacenter>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>

Thecommandwillwaituntilsynchronizationhascompletelystoppedbeforereturning.Ifthesynchronizationisnotcompletelystoppedwithinareasonableperiod(2minutesbydefault)thecommandwillfail.

Ifthesourcedatacenterisnolongeravailableitisnotpossibletostopsynchronizationinagracefulmanner.Ifthathappensabortthesynchronizationwiththefollowingcommand:

arangosyncabortsync\

--master.endpoint=<endpointsofsyncmastersintargetdatacenter>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>

Ifthesourcedatacenterrecoversafteranabortsynchasbeenexecuted,itisneededto"cleanup"ArangoSyncinthesourcedatacenter.Todoso,executethefollowingcommand:

arangosyncabortoutgoingsync\

--master.endpoint=<endpointsofsyncmastersinsourcedatacenter>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>

ReversingsynchronizationdirectionIfyouwanttoreversethedirectionofsynchronization(e.g.afterafailureindatacenterAandyouswitchedtothedatacenterBforfallback),youmustfirststop(orabort)theoriginalsynchronization.

Oncethatisfinished(andcleanuphasbeenappliedincaseofabort),youmustnowconfigurethesynchronizationagain,butwithswappedsource&targetsettings.

Datacentertodatacenterreplication

468

ShardingArangoDBisorganizingitscollectiondatainshards.ShardingallowstousemultiplemachinestorunaclusterofArangoDBinstancesthattogetherconstituteasingledatabase.Thisenablesyoutostoremuchmoredata,sinceArangoDBdistributesthedataautomaticallytothedifferentservers.Inmanysituationsonecanalsoreapabenefitindatathroughput,againbecausetheloadcanbedistributedtomultiplemachines.

Shardsareconfiguredpercollectionsomultipleshardsofdataformthecollectionasawhole.TodetermineinwhichshardthedataistobestoredArangoDBperformsahashacrossthevalues.Bydefaultthishashisbeingcreatedfrom_key.

Toconfigurethenumberofshards:

127.0.0.1:8529@_system>db._create("sharded_collection",{"numberOfShards":4});

Toconfigurethehashingforanotherattribute:

127.0.0.1:8529@_system>db._create("sharded_collection",{"numberOfShards":4,"shardKeys":["country"]});

Thiswouldbeusefultokeepdataofeverycountryinoneshardwhichwouldresultinbetterperformanceforqueriesworkingonapercountrybase.YoucanalsospecifymultipleshardKeys.Notehoweverthatifyouchangetheshardkeysfromtheirdefault["_key"],thenfindingadocumentinthecollectionbyitsprimarykeyinvolvesarequesttoeverysingleshard.Furthermore,inthiscaseonecannolongerprescribetheprimarykeyvalueofanewdocumentbutmustusetheautomaticallygeneratedone.Thislatterrestrictioncomesfromthefactthatensuringuniquenessoftheprimarykeywouldbeveryinefficientiftheusercouldspecifytheprimarykey.

Onwhichnodeinaclusteraparticularshardiskeptisundefined.Thereisnooptiontoconfigureanaffinitybasedoncertainshardkeys.

Uniqueindexes(hash,skiplist,persistent)onshardedcollectionsareonlyallowedifthefieldsusedtodeterminetheshardkeyarealsoincludedinthelistofattributepathsfortheindex:

shardKeys indexKeys

a a ok

a b notok

a a,b ok

a,b a notok

a,b b notok

a,b a,b ok

a,b a,b,c ok

a,b,c a,b notok

a,b,c a,b,c ok

Sharding

469

GeneralUpgradeInformation

Recommendedmajorupgradeprocedure

ToupgradeanexistingArangoDB2.xto3.0pleaseusetheproceduredescribedhere.

Recommendedminorupgradeprocedure

ToupgradeanexistingArangoDBdatabasetoanewerversionofArangoDB(e.g.3.0to3.1,or3.1to3.2),thefollowingmethodisrecommended:

ChecktheCHANGELOGandthelistofincompatiblechangesforAPIorotherchangesinthenewversionofArangoDBandmakesureyourapplicationscandealwiththemStopthe"old"arangodserviceorbinaryCopytheentire"old"datadirectorytoasafeplace(thatis,abackup)InstallthenewversionofArangoDBandstarttheserverwiththe--database.auto-upgradeoptiononce.ThismightwritetothelogfileofArangoDB,soyoumaywanttocheckthelogsforanyissuesbeforegoingon.Startthe"new"arangodserviceorbinaryregularlyandcheckthelogsforanyissues.Whenyou'reconfidenteverythingwentwell,youmaywanttocheckthedatabasedirectoryforanyfileswiththeending.old.ThesefilesarecreatedbyArangoDBduringupgradesandcanbesafelyremovedmanuallylater.

Ifanythinggoeswrongduringorshortlyaftertheupgrade:

Stopthe"new"arangodserviceorbinaryReverttothe"old"arangodbinaryandrestorethe"old"datadirectoryStartthe"old"versionagain

ItisnotsupportedtousedatafilescreatedormodifiedbyanewerversionofArangoDBwithanolderArangoDBversion.Forexample,itisunsupportedandislikelytocauseproblemswhenusing3.2datafileswithanArangoDB3.0instance.

Switchingthestorageengine

Inordertouseadifferentstorageenginewithanexistingdatadirectory,itisrequiredtofirstcreatealogicalbackupofthedatausingarangodump.

Afterthat,thearangodservershouldberestartedwiththedesiredstorageengineselected(thiscanbedonebysettingtheoption--server.storage-engine)andusinganon-existingdatadirectory.

Whentheserverisupandrunningwiththedesiredstorageengine,thedatacanbere-importedusingarangorestore.

Upgrading

470

UpgradingtoArangoDB3.3PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB3.3.Pleasebesurethatyouhavecheckedthelistofchangesin3.3beforeupgrading.

Upgradingto3.3

471

UpgradingtoArangoDB3.2PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB3.2.Pleasebesurethatyouhavecheckedthelistofchangesin3.2beforeupgrading.

Switchingthestorageengine

Inordertouseadifferentstorageenginewithanexistingdatadirectory,itisrequiredtofirstcreatealogicalbackupofthedatausingarangodump.Thatbackupshouldbecreatedbeforetheupgradeto3.2.

Afterthat,theArangoDBinstallationcanbeupgradedandstopped.Theservershouldthenberestartedwiththedesiredstorageengineselected(thiscanbedonebysettingtheoption--server.storage-engine)andusinganon-existingdatadirectory.Thiswillstarttheserverwiththeselectedstorageenginebutwithnodata.

Whentheserverisupandrunning,thedatafromthelogicalbackupcanbere-importedusingarangorestore.

Upgradingto3.2

472

UpgradingtoArangoDB3.1PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB3.1.Pleasebesurethatyouhavecheckedthelistofchangesin3.1beforeupgrading.

Upgradingto3.1

473

UpgradingtoArangoDB3.0PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB3.0.Pleasebesurethatyouhavecheckedthelistofchangesin3.0beforeupgrading.

MigratingdatabasesandcollectionsfromArangoDB2.8to3.0

ArangoDB3.0doesnotprovideanautomaticupdatemechanismfordatabasedirectoriescreatedwiththe2.xbranchesofArangoDB.

InordertomigratedatafromArangoDB2.8(oranolder2.xversion)intoArangoDB3.0,itisnecessarytoexportthedatafrom2.8usingarangodump,andthenimportthedumpintoafreshArangoDB3.0witharangorestore.

Todothis,firstrunthe2.8versionofarangodumptoexportthedatabasedataintoadirectory.arangodumpwilldumpthe_systemdatabasebydefault.Inordertomakeitdumpmultipledatabases,itneedstobeinvokedoncepersourcedatabase,e.g.

#in2.8

arangodump--server.database_system--output-directorydump-system

arangodump--server.databasemydb--output-directorydump-mydb

...

Thatwillproduceadumpdirectoryforeachdatabasethatarangodumpiscalledfor.Iftheserverhasauthenticationturnedon,itmaybenecessarytoprovidetherequiredcredentialswheninvokingarangodump,e.g.

arangodump--server.database_system--server.usernamemyuser--server.passwordmypasswd--output-directorydump-system

ThedumpsproducedbyarangodumpcannowbeimportedintoArangoDB3.0usingthe3.0versionofarangodump:

#in3.0

arangorestore--server.database_system--input-directorydump-system

arangorestore--server.databasemydb--input-directorydump-mydb

...

arangorestorewillbydefaultfailifthetargetdatabasedoesnotexist.Itcanbetoldtocreateitautomaticallyusingtheoption--create-databasetrue:

arangorestore--server.databasemydb--create-databasetrue--input-directorydump-mydb

Andagainitmayberequiredtoprovideaccesscredentialswheninvokingarangorestore:

arangorestore--server.databasemydb--create-databasetrue--server.usernamemyuser--server.passwordmypasswd--input-directo

rydump-system

Pleasenotethattheversionofdump/restoreshouldmatchtheserverversion,i.e.itisrequiredtodumptheoriginaldatawiththe2.8versionofarangodumpandrestoreitwiththe3.0versionofarangorestore.

Afterthatthe3.0instanceofArangoDBwillcontainthedatabasesandcollectionsthatwerepresentinthe2.8instance.

Adjustingauthenticationinfo

AuthenticationinformationwasstoredperdatabaseinArangoDB2.8,meaningtherecouldbedifferentusersandaccesscredentialsperdatabase.In3.0,theusersarestoredinacentrallocationinthe_systemdatabase.Tousethesameusersetupasin2.8,itmayberequiredtocreateextrausersand/oradjusttheirpermissions.

Inordertodothat,pleaseconnecttothe3.0instancewithanArangoShell(thiswillconnecttothe_systemdatabasebydefault):

arangosh--server.usernamemyuser--server.passwordmypasswd

Upgradingto3.0

474

Usethefollowingcommandstocreateanewuserwithsomepasswordandgrantthemaccesstoaspecificdatabase

require("@arangodb/users").save(username,password,true);

require("@arangodb/users").grantDatabase(username,databaseName,"rw");

Forexample,tocreateausermyuserwithpasswordmypasswdandgivethemaccesstodatabasesmydb1andmydb2,thecommandswouldlookasfollows:

require("@arangodb/users").save("myuser","mypasswd",true);

require("@arangodb/users").grantDatabase("myuser","mydb1","rw");

require("@arangodb/users").grantDatabase("myuser","mydb2","rw");

Existinguserscanalsobeupdated,removedorlistedusingthefollowingcommands:

/*updateusermyuserwithpasswordmypasswd*/

require("@arangodb/users").update("myuser","mypasswd",true);

/*removeusermyuser*/

require("@arangodb/users").remove("myuser");

/*listallusers*/

require("@arangodb/users").all();

Foxxapplications

Thedump/restoreproceduredescribedabovewillnotexportandre-importFoxxapplications.Inordertomovethesefrom2.8to3.0,Foxxapplicationsshouldbeexportedaszipfilesviathe2.8webinterface.

Thezipfilescanthenbeuploadedinthe"Services"sectionintheArangoDB3.0webinterface.Applicationsmayneedtobeadjustedmanuallytorunin3.0.PleaseconsultthemigrationguideforFoxxapps.

AnalternativewayofmovingFoxxappsinto3.0istocopythesourcedirectoryofa2.8Foxxapplicationmanuallyintothe3.0Foxxappsdirectoryforthetargetdatabase(whichisnormally/var/lib/arangodb3-apps/_db/<dbname>/buttheexactlocationisplatform-specific).

Upgradingto3.0

475

UpgradingtoArangoDB2.8PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.8.Pleasebesurethatyouhavecheckedthelistofchangesin2.8beforeupgrading.

PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.8cannotbeusedwithearlierversions(e.g.ArangoDB2.7)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.

DatabaseDirectoryVersionCheckandUpgrade

ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.8encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.

Theoutputwillthenlooklikethis:

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':Databasedirectoryversion(20702)islowerthancurrentversion(208

00).

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':---------------------------------------------------------------------

-

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':--upgrade

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':/etc/init.d/arangodbstop

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':/etc/init.d/arangodbstart

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':---------------------------------------------------------------------

-

2015-12-04T17:11:17Z[31432]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption

TomakeArangoDB2.8startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption.

Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).

Forexample,ifyouregularlystartyourArangoDBserverwith

unix>arangodmydatabasefolder

thenrunning

unix>arangodmydatabasefolder--upgrade

willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.

Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.

Thelastlineoftheoutputshouldlooklikethis:

2015-12-04T17:12:15Z[31558]INFOdatabaseupgradepassed

Pleasecheckthefulloutputthe--upgraderun.Upgradingmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolvedmanually,youcanstartArangoDB2.8regularly.

Upgradingaclusterplannedinthewebinterface

Upgradingto2.8

476

AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.

Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:

1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.

2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.

3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".

4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.

ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute

arangosh>require("org/arangodb/cluster").Upgrade("root","");

Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.

UpgradingFoxxappsgeneratedbyArangoDB2.7andearlier

TheimplementationoftherequirefunctionusedtoimportmodulesinArangoDBandFoxxhaschangedinordertoimprovecompatibilitywithNode.jsmodules.

Givenanapp/servicewiththefollowinglayout:

manifest.jsoncontrollers/

todos.jsmodels/

todo.jsrepositories/

todos.jsnode_modules/

models/todo.js

Thefilecontrollers/todos.jswouldpreviouslycontainthefollowingrequirecalls:

var_=require('underscore');

varjoi=require('joi');

varFoxx=require('org/arangodb/foxx');

varArangoError=require('org/arangodb').ArangoError;

varTodos=require('repositories/todos');//<--!

varTodo=require('models/todo');//<--!

Therequirepathsrepositories/todosandmodels/todowerepreviouslyresolvedlocallyasrelativetotheapproot.

Startingwith2.8thesepathswouldinsteadberesolvedasrelativetothenode_modulesfolderortheglobalArangoDBmodulepathsbeforebeingresolvedlocallyasafallback.

Inthegivenexamplelayouttheappwouldbreakin2.8becausethemodulenamemodels/todowouldalwaysresolvetonode_modules/models/todo.js(whichpreviouslywouldhavebeenignored)insteadofthelocalmodels/todo.js.

Inordertomakesuretheappstillworksin2.8,therequirecallsincontrollers/todos.jswouldneedtobeadjustedtolooklikethis:

Upgradingto2.8

477

var_=require('underscore');

varjoi=require('joi');

varFoxx=require('org/arangodb/foxx');

varArangoError=require('org/arangodb').ArangoError;

varTodos=require('../repositories/todos');//<--!

varTodo=require('../models/todo');//<--!

Notethattheold"global"stylerequirecallsmaystillworkin2.8butmaybreakunexpectedlyifmoduleswithmatchingnamesareinstalledglobally.

Upgradingto2.8

478

UpgradingtoArangoDB2.6PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.6.Pleasebesurethatyouhavecheckedthelistofchangesin2.6beforeupgrading.

PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.6cannotbeusedwithearlierversions(e.g.ArangoDB2.5)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.

DatabaseDirectoryVersionCheckandUpgrade

ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.6encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.

Theoutputwillthenlooklikethis:

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Databasedirectoryversion(20501)islowerthancurrentversion(2060

0).

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':----------------------------------------------------------------------

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':--upgrade

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbstop

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbstart

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':----------------------------------------------------------------------

2015-02-17T09:43:11Z[8302]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption

TomakeArangoDB2.6startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption.

Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).

Forexample,ifyouregularlystartyourArangoDBserverwith

unix>arangodmydatabasefolder

thenrunning

unix>arangodmydatabasefolder--upgrade

willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.

Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.

Thelastlineoftheoutputshouldlooklikethis:

2014-12-22T12:03:31Z[12026]INFOdatabaseupgradepassed

Pleasecheckthefulloutputthe--upgraderun.Upgradingmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolvedmanually,youcanstartArangoDB2.6regularly.

Upgradingaclusterplannedinthewebinterface

Upgradingto2.6

479

AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.

Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:

1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.

2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.

3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".

4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.

ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute

arangosh>require("org/arangodb/cluster").Upgrade("root","");

Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.

Upgradingto2.6

480

UpgradingtoArangoDB2.5PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.5.Pleasebesurethatyouhavecheckedthelistofchangesin2.5beforeupgrading.

PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.5cannotbeusedwithearlierversions(e.g.ArangoDB2.4)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.

In2.5wehavealsochangedthepathsforFoxxapplications.PleasealsomakesurethatyouhaveabackupofallFoxxappsinyourjavascript.app-pathandjavascript.dev-app-path.ItissufficienttohavethesourcefilesforFoxxsomewhereelsesoyoucanreinstallthemonerror.Tocheckthateverythinghasworkedduringupgradeyoucouldusetheweb-interfaceApplicationstabor

unix>foxx-managerlist

forallyourdatabases.Thelistedappsshouldbeidenticalbeforeandaftertheupgrade.

DatabaseDirectoryVersionCheckandUpgrade

ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.5encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.

Theoutputwillthenlooklikethis:

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Databasedirectoryversion(20401)islowerthancurrentversion(2050

0).

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':----------------------------------------------------------------------

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':--upgrade

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbstop

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbstart

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':----------------------------------------------------------------------

2015-02-17T09:43:11Z[8302]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption

TomakeArangoDB2.5startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption.Note:WehavechangedFoxxfolderstructureandimplementedanupgradetasktomoveyourapplicationstothenewstructure.InordertotellthisupgradetasktoalsomoveyourdevelopmentFoxxappspleasemakesureyougivethedev-app-pathaswell.IfyouhavenotuseddevelopmentmodeforFoxxappsyoucandropthe--javascript.dev-app-path.Itisonlypossibletoupgradeonedev-app-pathtogetherwithonedatafolder.

unix>arangoddata--upgrade--javascript.dev-app-pathdevapps

wheredataisArangoDB'smaindatadirectoryanddevappsisthedirectorywhereyoudevelopFoxxapps.

Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).

Forexample,ifyouregularlystartyourArangoDBserverwith

unix>arangodmydatabasefolder

thenrunning

unix>arangodmydatabasefolder--upgrade

Upgradingto2.5

481

willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.

Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.

Thelastlineoftheoutputshouldlooklikethis:

2014-12-22T12:03:31Z[12026]INFOdatabaseupgradepassed

Pleasecheckthefulloutputthe--upgraderun.Upgradingmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolvedmanually,youcanstartArangoDB2.5regularly.

UpgradingaclusterplannedinthewebinterfaceAclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.

Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:

1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.

2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.

3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".

4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.

ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute

arangosh>require("org/arangodb/cluster").Upgrade("root","");

Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.

Upgradingto2.5

482

UpgradingtoArangoDB2.4PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.4.Pleasebesurethatyouhavecheckedthelistofchangesin2.4beforeupgrading.

PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.4cannotbeusedwithearlierversions(e.g.ArangoDB2.3)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.

DatabaseDirectoryVersionCheckandUpgrade

ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.4encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.

Theoutputwillthenlooklikethis:

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':Databasedirectoryversion(20302)islowerthancurrentversion(204

00).

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':---------------------------------------------------------------------

-

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':--upgrade

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':/etc/init.d/arangodbstop

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':/etc/init.d/arangodbstart

2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':---------------------------------------------------------------------

-

2014-12-22T12:02:28Z[12001]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption

TomakeArangoDB2.4startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption:

unix>arangoddata--upgrade

wheredataisArangoDB'smaindatadirectory.

Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).

Forexample,ifyouregularlystartyourArangoDBserverwith

unix>arangodmydatabasefolder

thenrunning

unix>arangodmydatabasefolder--upgrade

willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.

Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.

Thelastlineoftheoutputshouldlooklikethis:

2014-12-22T12:03:31Z[12026]INFOdatabaseupgradepassed

Upgradingto2.4

483

Pleasecheckthefulloutputthe--upgraderun.Upgradingmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolvedmanually,youcanstartArangoDB2.4regularly.

Upgradingaclusterplannedinthewebinterface

AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.

Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:

1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.

2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.

3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".

4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.

ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute

arangosh>require("org/arangodb/cluster").Upgrade("root","");

Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.

Upgradingto2.4

484

UpgradingtoArangoDB2.3PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.3.Pleasebesurethatyouhavecheckedthelistofchangesin2.3beforeupgrading.

PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.3cannotbeusedwithearlierversions(e.g.ArangoDB2.2)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.

DatabaseDirectoryVersionCheckandUpgrade

ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.3encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.

Theoutputwillthenlooklikethis:

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':Databasedirectoryversion(2.2)islowerthancurrentversion(20300)

.

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':----------------------------------------------------------------------

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':--upgrade

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':/etc/init.d/arangodbstop

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':/etc/init.d/arangodbstart

2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':----------------------------------------------------------------------

2014-11-03T15:48:06Z[2694]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption

TomakeArangoDB2.3startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption:

unix>arangoddata--upgrade

wheredataisArangoDB'smaindatadirectory.

Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).

Forexample,ifyouregularlystartyourArangoDBserverwith

unix>arangodmydatabasefolder

thenrunning

unix>arangodmydatabasefolder--upgrade

willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.

Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.

Theoutputshouldlooklikethis:

2014-11-03T15:48:47Z[2708]INFOIndatabase'_system':Found24definedtask(s),5task(s)torun

2014-11-03T15:48:47Z[2708]INFOIndatabase'_system':stateprod/standalone/upgrade,tasksupdateUserModel,createStatistics,

upgradeClusterPlan,setupQueues,setupJobs

2014-11-03T15:48:48Z[2708]INFOIndatabase'_system':upgradesuccessfullyfinished

2014-11-03T15:48:48Z[2708]INFOdatabaseupgradepassed

Upgradingto2.3

485

Pleasechecktheoutputthe--upgraderun.Itmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolved,youcanstartArangoDB2.3regularly.

Upgradingaclusterplannedinthewebinterface

AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.

Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:

1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.

2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.

3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".

4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.

ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute

arangosh>require("org/arangodb/cluster").Upgrade("root","");

Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.

Upgradingto2.3

486

UpgradingtoArangoDB2.2PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.2.

PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.2cannotbeusedwithearlierversions(e.g.ArangoDB2.1)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.

DatabaseDirectoryVersionCheckandUpgrade

ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.2encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.

Theoutputwillthenlooklikethis:

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':Databasedirectoryversion(2.1)islowerthanserverversion(2.2).

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':---------------------------------------------------------------------

-

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':--upgrade

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':/etc/init.d/arangodbstop

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':/etc/init.d/arangodbstart

2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':---------------------------------------------------------------------

-

2014-07-07T22:04:53Z[18675]FATALDatabaseversioncheckfailedfor'_system'.Pleasestarttheserverwiththe--upgradeopti

on

TomakeArangoDB2.2startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption:

unix>arangoddata--upgrade

wheredataisArangoDB'smaindatadirectory.

Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).

Forexample,ifyouregularlystartyourArangoDBserverwith

unix>arangodmydatabasefolder

thenrunning

unix>arangodmydatabasefolder--upgrade

willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.

Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.

Theoutputshouldlooklikethis:

2014-07-07T22:11:30Z[18867]INFOIndatabase'_system':startingupgradefromversion2.1to2.2.0

2014-07-07T22:11:30Z[18867]INFOIndatabase'_system':Found19definedtask(s),2task(s)torun

2014-07-07T22:11:30Z[18867]INFOIndatabase'_system':upgradesuccessfullyfinished

2014-07-07T22:11:30Z[18867]INFOdatabaseupgradepassed

Upgradingto2.2

487

Pleasechecktheoutputthe--upgraderun.Itmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolved,youcanstartArangoDB2.2regularly.

Upgradingaclusterplannedinthewebinterface

AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.

Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:

1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.

2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.

3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".

4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.

ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute

arangosh>require("org/arangodb/cluster").Upgrade("root","");

Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.

Upgradingto2.2

488

Troubleshooting

Troubleshooting

489

ArangodIftheArangoDBserverdoesnotstartorifyoucannotconnecttoitusingarangoshorotherclients,youcantrytofindtheproblemcausebyexecutingthefollowingsteps.Iftheserverstartsupwithoutproblemsyoucanskipthissection.

Checktheserverlogfile:Iftheserverhaswrittenalogfileyoushouldcheckitbecauseitmightcontainrelevanterrorcontextinformation.

Checktheconfiguration:Theserverlooksforaconfigurationfilenamedarangod.confonstartup.Thecontentsofthisfilewillbeusedasabaseconfigurationthatcanoptionallybeoverriddenwithcommand-lineconfigurationparameters.Youshouldchecktheconfigfileforthemostrelevantparameterssuchas:

server.endpoint:WhatIPaddressandporttobindtologparameters:Ifandwheretologdatabase.directory:Paththedatabasefilesarestoredin

Iftheconfigurationrevealsthatsomethingisnotconfiguredrighttheconfigfileshouldbeadjustedandtheserverberestarted.

Starttheservermanuallyandcheckitsoutput:Startingtheservermightfailevenbeforeloggingisactivatedsotheserverwillnotproducelogoutput.Thiscanhappeniftheserverisconfiguredtowritethelogstoafilethattheserverhasnopermissionson.Inthiscasetheservercannotloganerrortothespecifiedlogfilebutwillwriteastartuperroronstderrinstead.Startingtheservermanuallywillalsoallowyoutooverridespecificconfigurationoptions,e.g.toturnon/offfileorscreenloggingetc.

ChecktheTCPport:Iftheserverstartsupbutdoesnotacceptanyincomingconnectionsthismightbeduetofirewallconfigurationbetweentheserverandanyclient(s).TheserverbydefaultwilllistenonTCPport8529.PleasemakesurethisportisactuallyaccessiblebyotherclientsifyouplantouseArangoDBinanetworksetup.

Whenusinghostnamesintheconfigurationorwhenconnecting,pleasemakesurethehostnameisactuallyresolvable.ResolvinghostnamesmightinvokeDNS,whichcanbeasourceoferrorsonitsown.

ItisgenerallygoodadvicetonotuseDNSwhenspecifyingtheendpointsandconnectionaddresses.UsingIPaddressesinsteadwillruleoutDNSasasourceoferrors.Anotheralternativeistouseahostnamespecifiedinthelocal/etc/hostsfile,whichwillthenbypassDNS.

Testifcurlcanconnect:Oncetheserverisstarted,youcanquicklyverifyifitrespondstorequestsatall.Thischeckallowsyoutodeterminewhetherconnectionerrorsareclient-specificornot.Ifatleastoneclientcanconnect,itislikelythatconnectionproblemsofotherclientsarenotduetoArangoDB'sconfigurationbutduetoclientorin-betweennetworkconfigurations.

Youcantestconnectivityusingasimplecommandsuchas:

curl--dump--XGEThttp://127.0.0.1:8529/_api/version&&echo

ThisshouldreturnaresponsewithanHTTP200statuscodewhentheserverisrunning.Ifitdoesitalsomeanstheserverisgenerallyacceptingconnections.Alternativetoolstocheckconnectivityarelynxorab.

arangod

490

EmergencyConsoleTheArangoDBdatabaseserverhastwomodesofoperation:Asaserver,whereitwillanswertoclientrequestsandasanemergencyconsole,inwhichyoucanaccessthedatabasedirectly.Thelatter-asthenamesuggests-shouldonlybeusedincaseofanemergency,forexample,acorruptedcollection.Usingtheemergencyconsoleallowsyoutoissueallcommandsnormallyavailableinactionsandtransactions.Whenstartingtheserverinemergencyconsolemode,theservercannothandleanyclientrequests.

Youshouldneverstartmorethanoneserverusingthesamedatabasedirectory,independentofthemodeofoperation.Normally,ArangoDBwillpreventyoufromdoingthisbyplacingalockfileinthedatabasedirectoryandnotallowingasecondArangoDBinstancetousethesamedatabasedirectoryifalockfileisalreadypresent.

InCaseOfDisaster

Thefollowingcommandstartsanemergencyconsole.

Note:Neverstarttheemergencyconsoleforadatabasewhichalsohasaserverattachedtoit.Ingeneral,theArangoDBshelliswhatyouwant.

>./arangod--console--logerror/tmp/vocbase

ArangoDBshell[V8version5.0.71.39,DBversion3.x.x]

arango>1+2;

3

arango>vardb=require("@arangodb").db;db.geo.count();

703

TheemergencyconsoleprovidesaJavaScriptconsoledirectlyrunninginthearangodserverprocess.ThisallowstodebugandexaminecollectionsanddocumentsaswiththenormalArangoDBshell,butwithoutclient/servercommunication.

However,itisverylikelythatyouwillneverneedtheemergencyconsoleunlessyouareanArangoDBdeveloper.

EmergencyConsole

491

DatafileDebugger

InCaseOfDisaster

AranagoDBusesappend-onlyjournals.Datacorruptionshouldonlyoccurwhenthedatabaseserveriskilled.Inthiscase,thecorruptionshouldonlyoccurinthelastobject(s)thathavebeingwrittentothejournal.

Ifacorruptionoccurswithinanormaldatafile,thenthiscanonlyhappenifahardwarefaultoccurred.

Ifajournalordatafileiscorrupt,shutdownthedatabaseserverandstarttheprogram

unix>arango-dfdb

inordertochecktheconsistencyofthedatafilesandjournals.Thisbringsup

_________________

/\___||____/_(_)|___/\/__\/_\

//\/_`|__/_`||_|||/_\//\/__\////_\/

//_//(_||||(_||_|||__///_//\/\//_\\

/___,'\__,_|\__\__,_|_||_|_|\___|/___,'\_____/\____/

Availablecollections:

0:_structures

1:_users

2:_routing

3:_modules

4:_graphs

5:products

6:prices

*:all

Collectiontocheck:

Youcannowselectwhichdatabaseandcollectionyouwanttocheck.Afteryouselectedoneorallofthecollections,aconsistencycheckwillbeperformed.

Checkingcollection#1:_users

Database

path:/usr/local/var/lib/arangodb

Collection

name:_users

identifier:82343

Datafiles

#ofjournals:1

#ofcompactors:1

#ofdatafiles:0

Datafile

path:/usr/local/var/lib/arangodb/collection-82343/journal-1065383.db

type:journal

currentsize:33554432

maximalsize:33554432

totalused:256

#ofentries:3

status:OK

Ifthereisaproblemwithoneofthedatafiles,thenthedatabasedebuggerwillprintitandpromptforwhethertoattempttofixit.

WARNING:Thejournalwasnotclosedproperly,thelastentriesarecorrupted.

ThismighthappenArangoDBwaskilledandthelastentrieswerenot

fullywrittentodisk.

DatafileDebugger

492

Wipethelastentries(Y/N)?

IfyouanswerY,thecorruptedentrywillberemoved.

Ifyouseeacorruptioninadatafile(andnotajournal),thensomethingisterriblywrong.ThesefilesareimmutableandneverchangedbyArangoDB.Acorruptioninsuchfileisanindicationofahard-diskfailure.

DatafileDebugger

493

ArangobenchArangobenchisArangoDB'sbenchmarkandtesttool.Itcanbeusedtoissuetestrequeststothedatabaseforperformanceandserverfunctiontesting.Itsupportsparallelqueryingandbatchrequests.

Relatedblogposts:

MeasuringArangoDBinsertperformanceGainfactorof5usingbatchrequests

Startupoptions--async:Sendasynchronousrequests.Thedefaultvalueisfalse.

--batch-size:Numberofoperationstosendperbatch.Use0todisablebatching(thisisthedefault).

--collection:Nameofcollectiontouseintest(onlyrelevantforteststhatinvokecollections).

--replication-factor:Incaseofacluster,thereplicationfactorofthecreatedcollections.

--number-of-shards:Incaseofacluster,thenumberofshardsofthecreatedcollections.

--wait-for-sync:ThevalueofwaitForSyncforcreatedcollections.

--complexity:Complexityvaluefortestcase(default:1).Meaningdependsontestcase.

--concurrency:Numberofparallelthreadsthatwillissuerequests(default:1).

--configuration:Readconfigurationfromfile.

--delay:Useastartupdelay.Thisisonlynecessarywhenruninseries.Thedefaultvalueisfalse.

--keep-alive:UseHTTPkeep-alive(default:true).

--progress:Showprogressofbenchmark,onevery20threquest.Settofalsetodisableintermediatelogging.Thedefaultvalueistrue.

--requests:Totalnumberofrequeststoperform(default:1000).

--server.endpoint:Serverendpointtoconnectto,consistingofprotocol,IPaddressandport.Defaultstotcp://localhost:8529.

--server.database:Databasenametousewhenconnecting(default:"_system").

--server.username:Usernametousewhenconnecting(default:"root").

--server.password:Passwordtousewhenconnecting.Don'tspecifythisoptiontogetapasswordprompt.

--server.authentication:Wetherornottoshowthepasswordpromptanduseauthenticationwhenconnectingtotheserver(default:true).

--test-case:Nameoftestcasetoperform(default:"version").Possiblevalues:

version:requests/_api/versiondocument:createsdocumentscollection:createscollectionsimport-document:createsdocumentsviatheimportAPIhash:Create/Read/Update/Readdocumentsindexedbyahashindexskiplist:Create/Read/Update/Readdocumentsindexedbyaskiplistedge:Create/Read/Updateedgedocumentsshapes:Create&Deletedocumentswithheterogeneousattributenamesshapes-append:Createdocumentswithheterogeneousattributenamesrandom-shapes:Create/Read/Deleteheterogeneousdocumentswithrandomvaluescrud:Create/Read/Update/Deletecrud-append:Create/Read/Update/Readagain

Arangobench

494

crud-write-read:Create/ReadDocumentsaqltrx:AQLTransactionswithdeepnestedAQLFOR-loopscounttrx:usesJStransactionstocountthedocumentsandinserttheresultagainmultitrx:multipletransactionscombiningreads&writesfromjsmulti-collection:multipletransactionscombiningreads&writesfromjsonmultiplecollectionsaqlinsert:insertdocumentsviaAQLaqlv8:executeAQLwithV8functionstoinsertrandomdocuments

--verbose:PrintoutrepliesiftheHTTPheaderindicatesDBerrors.(default:false).

Examples

arangobench

StartsArangobenchwiththedefaultuserandserverendpoint.

--test-caseversion--requests1000--concurrency1

Runsthe'version'testcasewith1000requests,withoutconcurrency.

--test-casedocument--requests1000--concurrency2

Runsthe'document'testcasewith2000requests,withtwoconcurrentthreads.

--test-casedocument--requests1000--concurrency2--asynctrue

Runsthe'document'testcasewith2000requests,withconcurrency2,withasyncrequests.

--test-casedocument--requests1000--concurrency2--batch-size10

Runsthe'document'testcasewith2000requests,withconcurrency2,usingbatchrequests.

Arangobench

495

TroubleshootingdatacentertodatacenterreplicationThedatacentertodatacenterreplicationisadistributedsystemwithalotdifferentcomponents.Aswithanysuchsystem,itrequiressome,butnotalot,ofoperationalsupport.

Thissectionincludesinformationonhowtotroubleshootthedatacentertodatacenterreplication.

Forageneralintroductiontothedatacentertodatacenterreplication,pleaserefertotheDatacentertodatacenterreplicationchapter.

WhatmeansareavailabletomonitorstatusAllofthecomponentsofArangoSyncprovidemeanstomonitortheirstatus.Belowyou'llfindanoverviewpercomponent.

Syncmaster&workers:Thearangosyncserversrunningaseithermasterorworker,provide:

AstatusAPI,seearangosyncgetstatus.Makesurethatallstatusesreportrunning.Forevenmoredetailthefollowingcommandsarealsoavailable:arangosyncgettasks,arangosyncgetmasters&arangosyncgetworkers.Alogonthestandardoutput.Loglevelscanbeconfiguredusing--log.levelsettings.AmetricsAPIGET/metrics.ThisAPIiscompatiblewithPrometheus.SampleGrafanadashboardsforinspectingthesemetricsareavailable.

ArangoDBcluster:ThearangodserversthatmakeuptheArangoDBclusterprovide:

Alogfile.Thisisconfigurablewithsettingswithalog.prefix.E.g.--log.output=file://myLogFileor--log.level=info.AstatisticsAPIGET/_admin/statistics

Kafkacluster:Thekafkabrokersprovide:

Alogfile,seesettingswithlog.prefixinitsserver.propertiesconfigurationfile.Zookeeper:Thezookeeperagentsprovide:

Alogonstandardoutput.

WhattolookforwhilemonitoringstatusTheveryfirstthingtodowhenmonitoringthestatusofArangoSyncistolookintothestatusprovidedbyarangosyncgetstatus...-v.Whennoteverythingisintherunningstate(onbothdatacenters),thisisanindicationthatsomethingmaybewrong.Incasethathappens,giveitsometime(incrementalsynchronizationmaytakequitesometimeforlargecollections)andlookatthestatusagain.Ifthestatusesdonotchange(orchange,butnotreachrunning)itistimetoinspectsthemetrics&logfiles.Whenthemetricsorlogsseemtoindicateaprobleminasyncmasterorworker,itissafetorestartit,aslongasonly1instanceisrestartedatatime.Giverestartedinstancessometimeto"catchup".

WhattodowhenproblemsremainWhenaproblemremainsandrestartingmasters/workersdoesnotsolvetheproblem,contactsupport.Makesuretoincludeprovidesupportwiththefollowinginformation:

Outputofarangosyncgetversion...onbothdatacenters.Outputofarangosyncgetstatus...-vonbothdatacenters.Outputofarangosyncgettasks...-vonbothdatacenters.Outputofarangosyncgetmasters...-vonbothdatacenters.Outputofarangosyncgetworkers...-vonbothdatacenters.LogfilesofallcomponentsAcompletedescriptionoftheproblemyouobservedandwhatyoudidtoresolveit.

HowtomonitorstatusofArangoSync

Howtokeepitalive

Datacentertodatacenterreplication

496

Whattodoincaseoffailuresorbugs

Whattodowhenasourcedatacenterisdown

WhenyouuseArangoSyncforbackupofyourclusterfromonedatacentertoanotherandthesourcedatacenterhasacompleteoutage,youmayconsiderswitchingyourapplicationstothetarget(backup)datacenter.

Thisiswhatyoumustdointhatcase:

1. Stopsynchronizationusing:

arangosyncstopsync...

Whenthesourcedatacenteriscompletelyunresponsivethiswillnotsucceed.Inthatcaseuse:

arangosyncabortsync...

SeeStopingsynchronizationforhowtocleanupthesourcedatacenterwhenitbecomesavailableagain.

2. Verifythatconfigurationhascompletelystoppedusing:

arangosyncgetstatus...-v

3. Reconfigureyourapplicationstousethetarget(backup)datacenter.

Whentheoriginalsourcedatacenterisrestored,youmayswitchrolesandmakeitthetargetdatacenter.Todoso,usearangosyncconfiguresync...asdescribedinReversingsynchronizationdirection.

Whattodoincaseofaplannednetworkoutage

AllArangoSynctaskssendoutheartbeatmessagesouttotheotherdatacentertoindicate"itisstillalive".Theotherdatacenterassumestheconnectionis"outofsync"whenitdoesnotreceiveanymessagesforacertainperiodoftime.

Ifyou'replanningsomesortofmaintenancewhereyouknowtheconnectivitywillbelostforsometime(e.g.3hours),youcanprepareArangoSyncforthatsuchthatitwillholdofre-synchronizationforagivenperiodoftime.

Todoso,onbothdatacenters,run:

arangosyncsetmessagetimeout\

--master.endpoint=<endpointsofsyncmastersinthedatacenter>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>\

3h

ThelastargumentistheperiodthatArangoSyncshouldhold-ofresynchronizationfor.Thiscanbeminutes(e.g.15m)orhours(e.g.3h).

Ifmaintenanceistakinglongerthanexpected,youcanusethesamecommandtheextendtheholdofperiod(e.g.to4h).

Afterthemaintenance,usethesamecommandrestoretheholdofperiodtoitsdefaultof1h.

Whattodoincaseofadocumentthatexceedsthemessagequeuelimits

Ifyouinsert/updateadocumentinacollectionandthesizeofthatdocumentislargerthanthemaximummessagesizeofyourmessagequeue,thecollectionwillnolongerbeabletosynchronize.Itwillgointoafailedstate.

Torecoverfromthat,firstremovethedocumentfromtheArangoDBclusterinthesourcedatacenter.Afterthat,foreachfailedshard,run:

Datacentertodatacenterreplication

497

arangosyncresetfailedshard\

--master.endpoint=<endpointsofsyncmastersinthedatacenter>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>\

--database=<nameofthedatabase>\

--collection=<nameofthecollection>\

--shard=<indexoftheshard(startingat0)>

Afterthiscommand,anewsetoftaskswillbestartedtosynchronizetheshard.Itcantakesometimefortheshardtoreachrunningstate.

Datacentertodatacenterreplication

498

Monitoring

Monitoring

499

MonitoringdatacentertodatacenterreplicationThissectionincludesinformationrelatedtothemonitoringofthedatacentertodatacenterreplication.

Forageneralintroductiontothedatacentertodatacenterreplication,pleaserefertotheDatacentertodatacenterreplicationchapter.

MetricsArangoSync(master&worker)providemetricsthatcanbeusedformonitoringthedatacentertodatacenterrepliationsolution.ThesemetricsareavailableusingthefollowingHTTPSendpoints:

GET/metrics:ProvidesmetricsinaformatsupportedbyPrometheus.GET/metrics.json:ProvidesthesamemetricsinJSONformat.

Bothendpointsincludehelpinformationpermetrics.

Note:Bothendpointsrequireauthentication.Besidestheusualauthenticationmethodstheseendpointsarealsoaccessibleusingaspecialbearertokenspecifiedusingthe--monitoring.tokencommandlineoption.

ThePrometheusoutput(/metrics)lookslikethis:

...

#HELParangosync_master_worker_registrationsTotalnumberofregistrations

#TYPEarangosync_master_worker_registrationscounter

arangosync_master_worker_registrations2

#HELParangosync_master_worker_storageNumberoftimesworkerinfoisstored,loaded

#TYPEarangosync_master_worker_storagecounter

arangosync_master_worker_storage{kind="",op="save",result="success"}20

arangosync_master_worker_storage{kind="empty",op="load",result="success"}1

...

TheJSONoutput(/metrics.json)lookslikethis:

{

...

"arangosync_master_worker_registrations":{

"help":"Totalnumberofregistrations",

"type":"counter",

"samples":[

{

"value":2

}

]

},

"arangosync_master_worker_storage":{

"help":"Numberoftimesworkerinfoisstored,loaded",

"type":"counter",

"samples":[

{

"value":8,

"labels":{

"kind":"",

"op":"save",

"result":"success"

}

},

{

"value":1,

"labels":{

"kind":"empty",

"op":"load",

"result":"success"

}

}

]

}

Datacentertodatacenterreplication

500

...

}

Hint:Togetalistofametricsandtheirhelpinformation,run:

aliasjq='dockerrun--rm-irealguess/jqjq'

curl-sk-u"<user>:<password>"https://<syncmaster-IP>:8629/metrics.json|\

jq'with_entries({key:.key,value:.value.help})'

Datacentertodatacenterreplication

501

Security

Security

502

DatacentertodatacenterSecurityThissectionincludesinformationrelatedtothedatacentertodatacenterreplicationsecurity.

Forageneralintroductiontothedatacentertodatacenterreplication,pleaserefertotheDatacentertodatacenterreplicationchapter.

Firewallsettings

ThecomponentsofArangoSyncuse(TCP)networkconnectionstocommunicatewitheachother.Belowyou'llfindanoverviewoftheseconnectionsandtheTCPportsthatshouldbeaccessible.

1. Thesyncmastersmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:

ArangoDBagentsandcoordinators(defaultports:8531and8529)Kafkabrokers(defaultport9092)Syncworkers(defaultport8729)

Additionallythesyncmastersmustbeallowedtoconnecttothesyncmastersintheotherdatacenter.

Bydefaultthesyncmasterswilloperateonport8629.

2. Thesyncworkersmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:

ArangoDBcoordinators(defaultport8529)Kafkabrokers(defaultport9092)Syncmasters(defaultport8629)

Bydefaultthesyncworkerswilloperateonport8729.

AdditionallythesyncworkersmustbeallowedtoconnecttotheKafkabrokersintheotherdatacenter.

3. Kafka

Thekafkabrokersmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:

Otherkafkabrokers(defaultport9092)Zookeeper(defaultports2181,2888and3888)

Thedefaultportforkafkais9092.Thedefaultkafkainstallationwillalsoexposesomeprometheusmetricsonport7071.Togainmoreinsightintokafkaopenthisportforyourprometheusinstallation.

4. Zookeeper

Thezookeeperagentsmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:

OtherzookeeperagentsThesetuphereisabitspecialaszookeeperuses3portsfordifferentoperations.Allagentsneedtobeabletoconnecttoalloftheseports.

BydefaultZookeeperuses:

port2181forclientcommunicationport2888forfollowercommunicationport3888forleaderelections

Certificates

DigitalcertificatesareusedinmanyplacesinArangoSyncforbothencryptionandauthentication.

InArangoSyncallnetworkconnectionsareusingTransportLayerSecurity(TLS),asetofprotocolsthatensurethatallnetworktrafficisencrypted.ForthisTLScertificatesareused.TheserversideofthenetworkconnectionoffersaTLScertificate.Thiscertificateis(often)verifiedbytheclientsideofthenetworkconnection,toensurethatthecertificateissignedbyatrustedCertificateAuthority(CA).Thisensurestheintegrityoftheserver.

Datacentertodatacenterreplication

503

Inseveralplacesadditionalcertificatesareusedforauthentication.Inthosecasestheclientsideoftheconnectionoffersaclientcertificate

(ontopofanexistingTLSconnection).Theserversideoftheconnectionusestheclientcertificatetoauthenticatetheclientand(optionally)decideswhichrightsshouldbeassignedtotheclient.

Note:ArangoSyncdoesallowtheuseofcertificatessignedbyawellknowCA(eg.verisign)howeveritismoreconvenient(andcommon)touseyourownCA.

Formats

Allcertificatesarex509certificateswithapublickey,aprivatekeyandanoptionalchainofcertificatesusedtosignthecertificate(thischainistypicallyprovidedbytheCertificateAuthority(CA)).Dependingontheiruse,certificatesstoredinadifferentformat.

Thefollowingformatsareused:

Publickeyonly(.crt):Afilethatcontainsonlythepublickeyofacertificatewithanoptionalchainofparentcertificates(publickeysofcertificatesusedtosignedthecertificate).Sincethisformatcontainsonlypublickeys,itisnotaproblemifitscontentsareexposed.Itmuststillbestoreitinasafeplacetoavoidlosingit.Privatekeyonly(.key):Afilethatcontainsonlytheprivatekeyofacertificate.Itisvitaltoprotectthesefilesandstoretheminasafeplace.Keyfilewithpublic&privatekey(.keyfile):Afilethatcontainsthepublickeyofacertificate,anoptionalchainofparentcertificatesandaprivatekey.Sincethisformatalsocontainsaprivatekey,itisvitaltoprotectthesefilesandstoretheminasafeplace.Javakeystore(.jks):Afilecontainingasetofpublicandprivatekeys.Itispossibletoprotectaccesstothecontentofthisfileusingakeystorepassword.Sincethisformatcancontainprivatekeys,itisvitaltoprotectthesefilesandstoretheminasafeplace(evenwhenitscontentisprotectedwithakeystorepassword).

Creatingcertificates

ArangoSyncprovidescommandstocreateallcertificatesneeded.

TLSservercertificates

TocreateacertificateusedforTLSserversinthekeyfileformat,youneedthepublickeyoftheCA(--cacert),theprivatekeyoftheCA(--cakey)andoneormorehostnames(orIPaddresses).Thenrun:

arangosynccreatetlskeyfile\

--cacert=my-tls-ca.crt--cakey=my-tls-ca.key\

--host=<hostname>\

--keyfile=my-tls-cert.keyfile

Makesuretostorethegeneratedkeyfile(my-tls-cert.keyfile)inasafeplace.

TocreateacertificateusedforTLSserversinthecrt&keyformat,youneedthepublickeyoftheCA(--cacert),theprivatekeyoftheCA(--cakey)andoneormorehostnames(orIPaddresses).Thenrun:

arangosynccreatetlscertificate\

--cacert=my-tls-ca.crt--cakey=my-tls-ca.key\

--host=<hostname>\

--cert=my-tls-cert.crt\

--key=my-tls-cert.key\

Makesuretoprotectandstorethegeneratedfiles(my-tls-cert.crt&my-tls-cert.key)inasafeplace.

Clientauthenticationcertificates

Tocreateacertificateusedforclientauthenticationinthekeyfileformat,youneedthepublickeyoftheCA(--cacert),theprivatekeyoftheCA(--cakey)andoneormorehostnames(orIPaddresses)oremailaddresses.Thenrun:

Datacentertodatacenterreplication

504

arangosynccreateclient-authkeyfile\

--cacert=my-client-auth-ca.crt--cakey=my-client-auth-ca.key\

[--host=<hostname>|--email=<emailaddress>]\

--keyfile=my-client-auth-cert.keyfile

Makesuretoprotectandstorethegeneratedkeyfile(my-client-auth-cert.keyfile)inasafeplace.

CAcertificates

TocreateaCAcertificateusedtosignTLScertificates,run:

arangosynccreatetlsca\

--cert=my-tls-ca.crt--key=my-tls-ca.key

Makesuretoprotectandstorebothgeneratedfiles(my-tls-ca.crt&my-tls-ca.key)inasafeplace.Note:CAcertificateshaveamuchlongerlifetimethannormalcertificates.Thereforeevenmorecareisneededtostorethemsafely.

TocreateaCAcertificateusedtosignclientauthenticationcertificates,run:

arangosynccreateclient-authca\

--cert=my-client-auth-ca.crt--key=my-client-auth-ca.key

Makesuretoprotectandstorebothgeneratedfiles(my-client-auth-ca.crt&my-client-auth-ca.key)inasafeplace.Note:CAcertificateshaveamuchlongerlifetimethannormalcertificates.Thereforeevenmorecareisneededtostorethemsafely.

Renewingcertificates

Allcertificateshavemetainformationinthemthelimittheiruseinfunction,target&lifetime.Acertificatecreatedforclientauthentication(function)cannotbeusedasaTLSservercertificate(sameistrueforthereverse).Acertificateforhostmyserver(target)cannotbeusedforhostanotherserver.AcertficiatethatisvaliduntilOctober2017(limetime)cannotbeusedafterOctober2017.

Ifanythingchangesinfunction,targetorlifetimeyouneedanewcertificate.

Theprocedureforcreatingarenewedcertificateisthesameasforcreatinga"first"certificate.Aftercreatingtherenewedcertificatetheprocess(es)usingthemhavetobeupdated.Thismeanrestartingthem.AllArangoSynccomponentsaredesignedtosupportstoppingandstartingsingleinstances,butdonotrestartmorethan1instanceatthesametime.Assoonas1instancehasbeenrestarted,giveitsometimeto"catchup"beforerestartingthenextinstance.

Datacentertodatacenterreplication

505

Architecture

AppendOnly/MVCC

Insteadofoverwritingexistingdocuments,ArangoDBwillcreateanewversionofmodifieddocuments.Thisiseventhecasewhenadocumentgetsdeleted.Thetwobenefitsare:

Objectscanbestoredcoherentlyandcompactlyinthemainmemory.Objectsarepreserved,isolatedwritingandreadingtransactionsallowaccessingtheseobjectsforparalleloperations.

Thesystemcollectsobsoleteversionsasgarbage,recognizingthemasforsaken.Garbagecollectionisasynchronousandrunsparalleltootherprocesses.

MostlyMemory/Durability

Databasedocumentsarestoredinmemory-mappedfiles.Perdefault,thesememory-mappedfilesaresyncedregularlybutnotinstantly.Thisisoftenagoodtradeoffbetweenstorageperformanceanddurability.Ifthislevelofdurabilityistoolowforanapplication,theservercanalsosyncallmodificationstodiskinstantly.ThiswillgivefulldurabilitybutwillcomewithaperformancepenaltyaseachdatamodificationwilltriggerasyncI/Ooperation.

Architecture

506

Write-aheadlogTheWrite-aheadlogispartoftheMMFilesstorageengine;Thisdoesn'tapplytoyourArangoDBifyouarerunningwiththeRocksDBstorageengine.

Startingwithversion2.2ArangoDBstoresalldata-modificationoperationinitswrite-aheadlog.Thewrite-aheadlogissequenceofappend-onlyfilescontainingallthewriteoperationsthatwereexecutedontheserver.

Itisusedtorundatarecoveryafteraservercrash,andcanalsobeusedinareplicationsetupwhenslavesneedtoreplaythesamesequenceofoperationsasonthemaster.

Bydefault,eachwrite-aheadlogfileis32MiBinsize.Thissizeisconfigurableviatheoption--wal.logfile-size.

Whenawrite-aheadlogfileisfull,itissettoread-only,andfollowingoperationswillbewrittenintothenextwrite-aheadlogfile.Bydefault,ArangoDBwillreservesomesparelogfilesinthebackgroundsoswitchinglogfilesshouldbefast.HowmanyreservelogfilesArangoDBwilltrytokeepavailableinthebackgroundcanbecontrolledbytheconfigurationoption--wal.reserve-logfiles.

Datacontainedinfullwrite-aheadfileswilleventuallybetransferredintothejournalsordatafilesofcollections.Onlythe"surviving"documentswillbecopiedover.Whenallremainingoperationsfromawrite-aheadlogfilehavebeencopiedoverintothejournalsordatafilesofthecollections,thewrite-aheadlogfilecansafelyberemovedifitisnotusedforreplication.

Long-runningtransactionspreventwrite-aheadlogfilesfrombeingfullygarbage-collectedbecauseitisunclearwhetheratransactionwillcommitorabort.Long-runningtransactionscanthusblockthegarbage-collectionprogressandshouldthereforebeavoidedatallcosts.

Onasystemthatactsasareplicationmaster,itisusefultokeepafewofthealreadycollectedwrite-aheadlogfilessoreplicationslavesstillcanfetchdatafromthemifrequired.Howmanycollectedlogfileswillbekeptbeforetheygetdeletedisconfigurableviatheoption--wal.historic-logfiles.

Forallwrite-aheadlogconfigurationoptions,pleaserefertothepageWrite-aheadlogoptions.

Write-aheadlog

507

StorageEnginesAttheverybottomoftheArangoDBdatabaseliesthestorageengine.Thestorageengineisresponsibleforpersistingthedocumentsondisk,holdingcopiesinmemory,providingindexesandcachestospeedupqueries.

Uptoversion3.1ArangoDBonlysupportedmemorymappedfiles(MMFiles)assolestorageengine.Beginningwith3.2ArangoDBhassupportforpluggablestorageengines.ThesecondsupportedengineisRocksDBfromFacebook.

MMFiles RocksDB

default optional

datasetneedstofitintomemory workwithasmuchdataasfitsondisk

indexesinmemory hotsetinmemory,dataandindexesondisk

slowrestartduetoindexrebuilding faststartup(norebuildingofindexes)

volatilecollections(onlyinmemory,optional) collectiondataalwayspersisted

collectionlevellocking(writesblockreads) concurrentreadsandwrites

Blogarticle:ComparingnewRocksDBandMMFilesstorageengines

RocksDBisanembeddablepersistentkey-valuestore.Itisalogstructuredatabaseandisoptimizedforfaststorage.

TheMMFilesengineisoptimizedfortheuse-casewherethedatafitsintothemainmemory.Itallowsforveryfastconcurrentreads.However,writesblockreadsandlockingisoncollectionlevel.Indexesarealwaysinmemoryandarerebuiltonstartup.Thisgivesbetterperformancebutimposesalongerstartuptime.

TheRocksDBengineisoptimizedforlargedata-setsandallowsforasteadyinsertperformanceevenifthedata-setismuchlargerthanthemainmemory.Indexesarealwaysstoredondiskbutcachesareusedtospeedupperformance.RocksDBusesdocument-levellocksallowingforconcurrentwrites.Writesdonotblockreads.Readsdonotblockwrites.

Theenginemustbeselectedforthewholeserver/cluster.Itisnotpossibletomixengines.Thetransactionhandlingandwrite-ahead-logformatintheindividualenginesisverydifferentandthereforecannotbemixed.

RocksDB

Advantages

RocksDBisaveryflexibleenginethatcanbeconfiguredforvarioususecases.

ThemainadvantagesofRocksDBare

document-levellockssupportforlargedata-setspersistentindexes

Caveats

RocksDBallowsconcurrentwrites.However,whentouchingthesamedocumentawriteconflictisraised.ThiscannothappenwiththeMMFilesengine,thereforeapplicationsthatswitchtoRocksDBneedtobepreparedthatsuchexceptioncanarise.ItispossibletoexclusivelylockcollectionswhenexecutingAQL.Thiswillavoidwriteconflictsbutalsoinhibitsconcurrentwrites.

Currently,anotherrestrictionisduetothetransactionhandlinginRocksDB.Transactionsarelimitedintotalsize.Ifyouhaveastatementmodifyingalotofdocumentsitisnecessarytocommitdatainbetween.ThiswillbedoneautomaticallyforAQLbydefault.

Performance

RocksDBisabasedonlog-structuredmergetree.Agoodintroductioncanbefoundin:

StorageEngines

508

http://www.benstopford.com/2015/02/14/log-structured-merge-trees/https://blog.acolyer.org/2014/11/26/the-log-structured-merge-tree-lsm-tree/

Thebasicideaisthatdataisorganizedinlevelswereeachlevelisafactorlargerthantheprevious.Newdatawillresideinsmallerlevelswhileolddataismoveddowntothelargerlevels.Thisallowstosupporthighrateofinsertsoveranextendedperiod.Inprincipleitispossiblethatthedifferentlevelsresideondifferentstoragemedia.ThesmalleronesonfastSSD,thelargeronesonbiggerspinningdisks.

RocksDBitselfprovidesalotofdifferentknobstofinetunethestorageengineaccordingtoyouruse-case.ArangoDBsupportsthemostcommononesusingtheoptionsbelow.

Performancereportsforthestorageenginecanbefoundhere:

https://github.com/facebook/rocksdb/wiki/performance-benchmarkshttps://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide

ArangoDBoptions

ArangoDBhasacacheforthepersistentindexesinRocksDB.Thetotalsizeofthiscacheiscontrolledbytheoption

--cache.size

RocksDBalsohasacachefortheblocksstoredondisk.Thesizeofthiscacheiscontrolledbytheoption

--rocksdb.block-cache-size

ArangoDBdistributestheavailablememoryequallybetweenthetwocachesbydefault.

ArangoDBchoosesasizeforthevariouslevelsinRocksDBthatissuitableforgeneralpurposeapplications.

RocksDBlogstrutureddatalevelshaveincreasingsize

MEM:--

L0:--

L1:----

L2:--------

...

NeworupdatedDocumentsarefirststoredinmemory.Ifthismemtablereachesthelimitgivenby

--rocksdb.write-buffer-size

itwillconvertedtoanSSTfileandinsertedatlevel0.

Thefollowingoptioncontrolsthesizeofeachlevelandthedepth.

--rocksdb.num-levelsN

LimitsthenumberoflevelstoN.Bydefaultitis7andthereisseldomareasontochangethis.Anewlevelisonlyopenedifthereistoomuchdatainthepreviousone.

--rocksdb.max-bytes-for-level-baseB

L0willholdatmostBbytes.

--rocksdb.max-bytes-for-level-multiplierM

EachlevelisatmostMtimesasmuchbytesasthepreviousone.ThereforethemaximumnumberofbytesforlevelLcanbecalculatedas

max-bytes-for-level-base*(max-bytes-for-level-multiplier^(L-1))

StorageEngines

509

Future

RocksDBimposesalimitonthetransactionsize.Itisoptimizedtohandlesmalltransactionsveryefficiently,butiseffectivelylimitingthetotalsizeoftransactions.

ArangoDBcurrentlyusesRocksDB'stransactionstoimplementtheArangoDBtransactionhandling.ThereforethesamerestrictionsapplyforArangoDBtransactionswhenusingtheRocksDBengine.

WewillimprovethisbyintroducingdistributedtransactionsinafutureversionofArangoDB.ThiswillallowhandlinglargetransactionsasaseriesofsmallRocksDBtransactionsandhenceremovingthesizerestriction.

StorageEngines

510

ReleaseNotes

WhatsNew

WhatsNewin3.3WhatsNewin3.2WhatsNewin3.1WhatsNewin3.0WhatsNewin2.8WhatsNewin2.7WhatsNewin2.6WhatsNewin2.5WhatsNewin2.4WhatsNewin2.3WhatsNewin2.2WhatsNewin2.1

KnownIssues

KnownIssuesin3.2

Incompatiblechanges

AlsoseeUpgradingintheAdministrationchapter.

Incompatiblechangesin3.3Incompatiblechangesin3.2Incompatiblechangesin3.1Incompatiblechangesin3.0Incompatiblechangesin2.8Incompatiblechangesin2.7Incompatiblechangesin2.6Incompatiblechangesin2.5Incompatiblechangesin2.4Incompatiblechangesin2.3

Releasenotes

511

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB3.3.ArangoDB3.3alsocontainsseveralbugfixesthatarenotlistedhere.

Datacenter-to-datacenterreplication(DC2DC)

Everycompanyneedsadisasterrecoveryplanforallimportantsystems.Thisistruefromsmallunitslikesingleprocessesrunninginsomecontainertothelargestdistributedarchitectures.Fordatabasesinparticularthisusuallyinvolvesamixtureoffault-tolerance,redundancy,regularbackupsandemergencyplans.Thelargeradatastore,themoredifficultitistocomeupwithagoodstrategy.

Therefore,itisdesirabletobeabletorunadistributeddatabaseinonedata-centerandreplicatealltransactionstoanotherdata-centerinsomeway.Often,transactionlogsareshippedoverthenetworktoreplicateeverythinginanother,identicalsystemintheotherdata-center.Somedistributeddatastoreshavebuilt-insupportformultipledata-centerawarenessandcanreplicatebetweendata-centersinafullyautomaticfashion.

ArangoDB3.3takesanevolutionarystepforwardbyintroducingmulti-data-centersupport,whichisasynchronousdata-centertodata-centerreplication.Oursolutionisasynchronousandscalestoarbitraryclustersizes,providedyournetworklinkbetweenthedata-centershasenoughbandwidth.Itisfault-tolerantwithoutasinglepointoffailureandincludesalotofmetricsformonitoringinaproductionscenario.

DC2DCisavailableintheEnterpriseedition.

Encryptedbackups

ArangodumpcannowcreateencryptedbackupsusingAES256forencryption.Theencryptionkeycanbereadfromafileorfromageneratorprogram.Itworksinsingleserverandclustermode.

Examplefornon-encryptedbackup(everyonewithaccesstothebackupwillbeabletoreadit):

arangodump--collection"secret"dump

Inordertocreateanencryptedbackup,addthe--encryption.keyfileoptionwheninvokingarangodump:

arangodump--collection"secret"dump--encryption.keyfile~/SECRET-KEY

Thekeymustbeexactly32byteslong(requiredbytheAESblockcipher).

Notethatarangodumpwillnotstorethekeyanywhere.Itistheresponsibilityoftheusertofindasafeplaceforthekey.However,arangodumpwillstoretheusedencryptionmethodinafilenamedENCRYPTIONinthedumpdirectory.Thatwayarangorestorecanlaterfindoutwhetheritisdealingwithanencrypteddumpornot.

Tryingtorestoretheencrypteddumpwithoutspecifyingthekeywillfail:

arangorestore--collection"secret-collection"dump--create-collectiontrue

arangorestorewillcomplainwith:

thedumpdataseemstobeencryptedwithaes-256-ctr,butnokeyinformationwasspecifiedtodecryptthedumpitisrecommendedtospecifyeither--encryption.key-fileor--encryption.key-generatorwheninvokingarangorestorewithanencrypteddump

Itisrequiredtousetheexactsamekeywhenrestoringthedata.Againthisisdonebyprovidingthe--encryption.keyfileparameter:

arangorestore--collection"secret-collection"dump--create-collectiontrue--encryption.keyfile~/SECRET-KEY

WhatsNewin3.3

512

Usingadifferentkeywillleadtothebackupbeingnon-recoverable.

NotethatencryptedbackupscanbeusedtogetherwiththealreadyexistingRocksDBencryption-at-restfeature,buttheycanalsobeusedfortheMMFilesengine,whichdoesnothaveencryption-at-rest.

EncryptedbackupsareavailableintheEnterpriseedition.

Server-levelreplicationArangoDBsupportsasynchronousreplicationfunctionalitysinceversion1.4,butreplicatingfromamasterserverwithmultipledatabasesrequiredmanualsetupontheslaveforeachindividualdatabasetoreplicate.Whenanewdatabasewascreatedonthemaster,oneneededtotakeactionontheslavetoensurethatdataforthatdatabasegotactuallyreplicated.Replicationontheslavealsowasnotawareofwhenadatabasewasdroppedonthemaster.

3.3addsserver-levelreplication,whichwillreplicatethecurrentandfuturedatabasesfromthemastertotheslaveautomaticallyaftertheinitialsetup.

Inordertosetupglobalreplicationona3.3slaveforalldatabasesofagiven3.3master,thereisnowtheso-calledglobalApplier.Ithasthesameinterfaceastheexistingapplier,butitwillreplicatefromalldatabasesofthemasterandnotjustasingleone.

Inordertostartthereplicationontheslaveandmakeitreplicatealldatabasesfromagivenmaster,usethesecommandsontheslave:

varreplication=require("@arangodb/replication");

replication.setupReplicationGlobal({

endpoint:"tcp://127.0.0.1:8529",

username:"root",

password:"",

autoStart:true

});

Tocheckiftheapplierisrunning,alsousetheglobalApplierobject:

replication.globalApplier.state().state

Theserver-levelreplicationrequiresboththemasterandslaveserverstoArangoDBversion3.3orhigher.

Asynchronousfailover

Aresilientsetupcannoweasilybeachievedbyrunningapairofconnectedservers,ofwhichoneinstancebecomesthemasterandtheotheranasynchronouslyreplicatingslave,withautomaticfailoverbetweenthem.

Twoserversareconnectedviaasynchronousreplication.Oneoftheserversiselectedleader,andtheotheroneismadeafollowerautomatically.Atstartup,thetwoserversfightforleadership.Thefollowerwillautomaticallystartreplicationfromthemasterforallavailabledatabases,usingtheserver-levelreplicationintroducedin3.3.

Whenthemastergoesdown,thisisautomaticallydetectedbyanagencyinstance,whichisalsostartedinthismode.Thisinstancewillmakethepreviousfollowerstopitsreplicationandmakeitthenewleader.

Thefollowerwillautomaticallydenyallreadandwriterequestsfromclientapplications.Onlythereplicationitselfisallowedtoaccessthefollower'sdatauntilthefollowerbecomesanewleader.

Whensendingarequesttoreadorwritedataonafollower,thefollowerwillalwaysrespondwithHTTP503(Serviceunavailable)andprovidetheaddressofthecurrentleader.Clientapplicationsanddriverscanusethisinformationtothenmakeafollow-uprequesttotheproperleader:

HTTP/1.1503ServiceUnavailable

X-Arango-Endpoint:http://[::1]:8531

....

WhatsNewin3.3

513

Clientapplicationscanalsodetectwhothecurrentleaderandthefollowersarebycallingthe/_api/cluster/endpointsRESTAPI.ThisAPIisaccessibleonleadersandfollowersalike.

TheArangoDBstartersupportsstartingtwoserverswithasynchronousreplicationandfailoveroutofthebox.

ThearangojsdriverforJavaScript,theGodriverandtheJavadriverforArangoDBsupportautomaticfailoverincasethecurrentlyaccessedserverendpointrespondswithHTTP503.

Blogarticle:IntroducingthenewArangoDBJavadriverwithloadbalancingandadvancedfallback

RocksDBthrottling

ArangoDB3.3allowswriteoperationstotheRocksDBenginebethrottled,inordertopreventlongerwritestalls.Thethrottlingisadaptive,meaningthatitautomaticallyadaptstotheactualwriterate.Thisresultsinmuchmorestableresponsetimes,whichisbetterforclientapplicationsandclusterhealthtests,becausetimeoutscausedbywritestallsarelesslikelytooccurandtheserverthusnotmistakenlyassumedtobedown.

Blogarticle:RocksDBsmoothingforArangoDBcustomers

FastershardcreationinclusterWhenusingacluster,onenormallywantsresilience,soreplicationFactorissettoatleast2.Thenumberofshardsisoftensettoratherhighvalueswhencreatingcollections.

Creatingacollectionintheclusterwillmakethecoordinatorstorethesetupmetadataofthenewcollectionintheagencyfirst.Subsequentiallyalldatabaseserversoftheclusterwilldetectthatthereisworktodoandwillbegincreatingtheshards.Thiswillfirsthappenfortheshardleaders.Foreachshardleaderthatfinisheswiththesetup,thesynchronousreplicationwithitsfollowersisthenestablished.Thatwillmakesurethateveryfuturedatamodificationwillnotbecomeeffectiveontheleaderonly,butalsoonallthefollowers.

In3.3thissetupprotocolhasgotsomeshortcutsfortheinitialshardcreation,whichspeedsupcollectioncreationbyroughly50to60percent.

LDAPauthentication

TheLDAPauthenticationmoduleintheEnterpriseeditionhasbeenenhanced.Thefollowingoptionshavebeenaddedtoit:

theoption--server.local-authenticationcontrolswhetherthelocal_userscollectionisalsousedforlookingupusers.Thisisalsothedefaultbehavior.IftheauthenticationshallberestrictedtojusttheLDAPdirectory,theoptioncanbesettotrue,andarangodwillthennotmakeanyqueriestoits_userscollectionwhenlookingupusers.

theoption--server.authentication-timeoutcontrolstheexpirationtimeforcachedLDAPuserinformationentriesinarangod.

basicrolesupporthasbeenaddedfortheLDAPmoduleintheEnterpriseedition.NewconfigurationoptionsforLDAPin3.3are:

--ldap.roles-attribute-name

--ldap.roles-transformation

--ldap.roles-search

--ldap.roles-include

--ldap.roles-exclude

--ldap.superuser-role

PleaserefertoLDAPforadetailedexplanation.

Miscellaneousfeatures

whencreatingacollectioninthecluster,thereisnowanoptionalparameterenforceReplicationFactor:whenset,thisparameterenforcesthatthecollectionwillonlybecreatediftherearenotenoughdatabaseserversavailableforthedesiredreplicationFactor.

AQLDISTINCTisnotchangingtheorderofprevious(sorted)results

WhatsNewin3.3

514

PreviouslytheimplementationofAQLdistinctstoredallencounteredvaluesinahashtableinternally.Whendone,thefinalresultswerereturnedintheorderdictatedbythehashtablethatwasusedtostorethekeys.Thisorderwasmoreorlessunpredictable.Thoughthiswasdocumentedbehavior,itwasinconvenientforendusers.

3.3nowdoesnotchangethesortorderoftheresultanymorewhenDISTINCTisused.

SeveralAQLfunctionshavebeenimplementedinC++,whichcanhelpsavememoryandCPUtimeforconvertingthefunctionargumentsandresults.Thefollowingfunctionshavebeenported:

LEFTRIGHTSUBSTRINGTRIMMATCHES

TheArangoShellpromptsubstitutioncharactershavebeenextended.Nowthefollowingextrasubstitutionscanbeusedforthearangoshprompt:

'%t':currenttimeastimestamp'%a':elpasedtimesinceArangoShellstartinseconds'%p':durationoflastcommandinseconds

Forexample,toshowtheexecutiontimeofthelastcommandexecutedinarangoshintheshell'sprompt,startarangoshusing:

arangosh--console.prompt"%E@%d%p>"

Therearenewstartupoptionsfortheloggingtoaiddebugginganderrorreporting:

--log.role:willshowone-lettercodeofserverrole(A=agent,C=coordinator,...)Thisisespeciallyusefulwhenaggregatinglogs.

Theexistingrolesusedinlogsare:

U:undefined/unclear(usedatstartup)S:singleserverC:coordinatorP:primaryA:agent

--log.line-numbertrue:thisoptionwillnowadditionallyshowthenameoftheC++functionthattriggeredthelogmessage(filenameandlinenumberwerealreadyloggedinpreviousversions)

--log.thread-nametrue:thisnewoptionwilllogthenameoftheArangoDBthreadthattriggeredthelogmessage.WillhavemeaningfuloutputonLinuxonly

maketheArangoShell(arangosh)refillitscollectioncachewhenayet-unknowncollectionisfirstaccessed.Thisfixesthefollowingproblemwhenworkingwiththeshellwhileinanothershellorbyanotherprocessanewcollectionisadded:

arangosh1>db._collections();//shell1listsallcollections

arangosh2>db._create("test");//shell2nowcreatesanewcollection'test'

arangosh1>db.test.insert({});//shell1isnotawareofthecollectioncreated

//inshell2,sotheinsertwillfail

WhatsNewin3.3

515

IncompatiblechangesinArangoDB3.3ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB3.3,andadjustanyclientprogramsifnecessary.

ThefollowingincompatiblechangeshavebeenmadeinArangoDB3.3:

AQL:duringatraversalifavertexisnotfound,arangodwillnotloganerrorandcontinuewithaNULLvalue,butwillinsteadregisterawarningatthequeryandcontinuewithaNULLvalue.

Ifanon-existingvertexisreferencedfromatraversal,itisnotdesirabletologerrorsasArangoDBcanstoreedgespointingtonon-existingvertices(whichisperfectlyvalidifthelow-levelinsertAPIsareused).Aslinkingtonon-existingverticesmayindicateanissuein/withthedatamodelortheclientapplication,thewarningisregisteredinthequerysoclientapplicationshaveaccesstoit.

ArangoDBusernamesmustnotstartwiththestring:role:.

Thestartupconfigurationparameter--cluster.my-iddoesnothaveanyeffectin3.3.Forcompatibilityreasons,ArangoDB3.3willnotfailonstartupiftheoptionisstillusedintheconfiguration,butitwillsilentlyignorethisoption.

Thestartupconfigurationparameter--cluster.my-local-infoisdeprecatednow.Usingitwillmakearangodlogawarningonstartup.

Serverstartup:therecommendedvaluefortheLinuxkernelsettingin/proc/sys/vm/max_map_countwasincreasedtoavalueeighttimesashighasin3.2.arangodcomparesatstartupiftheeffectivevalueofthissettingispresumablytoolow,anditwillissueawarninginthiscase,recommendingtoincreasethevalue.

Thisisnowmorelikelytohappenthaninpreviousversions,astherecommendedvalueisnoweighttimeshigherthanin3.2.Thestartupwarningswilllooklikethis(withactualnumbersvarying):

WARNING{memory}maximumnumberofmemorymappingsperprocessis65530,whichseemstoolow.itisrecommendedtosetit

toatleast512000

PleaserefertotheLinuxkerneldocumentationformoreinformationonthissetting.ThischangeonlyaffectstheLinuxversionofArangoDB.

Clienttools

Theoption--recycle-idshasbeenremovedfromthearangorestorecommand.Usingthisoptioncouldhaveledtoproblemsontherestore,withpotentialidconflictsbetweentheoriginatingserver(thesourcedumpserver)andthetargetserver(therestoreserver).

Theoption--compathasbeenremovedfromthearangodumpcommandandthe/_api/replication/dumpRESTAPIendpoint.InordertocreateadumpfromanArangoDB2.8instance,pleaseuseanolderversionoftheclienttools.OlderArangoDBversionsarenolongerbesupportedbythearangodumpandarangorestorebinariesshippedwith3.3.

Miscellaneous

TheminimumsupportedcompilerforcompilingArangoDBfromsourceisnowg++5.4(bumpedupfromg++4.9).ThischangeonlyaffectsusersthatcompileArangoDBontheirown.

Incompatiblechangesin3.3

516

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB3.2.ArangoDB3.2alsocontainsseveralbugfixesthatarenotlistedhere.

Storageengines

ArangoDB3.2offerstwostorageengines:

thealways-existingmemory-mappedfilesstorageengineanewstorageenginebasedonRocksDB

Memory-mappedfilesstorageengine(MMFiles)

Theformerstorageengine(namedMMFilesenginehenceforth)persistsdatainmemory-mappedfiles.

Anydatachangesaredonefirstintheengine'swrite-aheadlog(WAL).TheWALisreplayedafteracrashsotheengineoffersdurabilityandcrash-safety.DatafromtheWALiseventuallymovedtocollection-specificdatafiles.Thefilesarealwayswritteninanappend-onlyfashion,sodatainfilesisneveroverwritten.Obsoletedatainfileswilleventuallybepurgedbybackgroundcompactionthreads.

Mostofthisengine'sindexesarebuiltinRAM.Whenacollectionisloaded,thisrequiresrebuildingtheindexesinRAMfromthedatastoredondisk.TheMMFilesenginehascollection-levellocking.

Thisstorageengineisagoodchoicewhendata(includingtheindexes)canfitintheserver'savailableRAM.Ifthesizeofdataplusthein-memoryindexesexceedsthesizeoftheavailableRAM,thenthisenginemaytrytoallocatemorememorythanavailable.Thiswilleithermaketheoperatingsystemswapoutpartsofthedata(andcausediskI/O)or,whennoswapspaceisconfigured,invoketheoperatingsystem'sout-of-memoryprocesskiller.

Thelockingstrategyallowsparallelreadsandisoftengoodenoughinread-mostlyworkloads.Writesneedexclusivelocksonthecollections,sotheycanblockotheroperationsinthesamecollection.Thelockingstrategyalsoprovidestransactionalconsistencyandisolation.

RocksDBstorageengine

TheRocksDBstorageengineisnewinArangoDB3.2.Itisdesignedtostoredatasetsthatarebiggerthantheserver'savailableRAM.Itpersistsalldata(includingtheindexes)inaRocksDBinstance.

ThatmeansanydocumentreadorwriteoperationswillbeansweredbyRocksDBunderthehood.RocksDBwillservethedatafromitsownin-RAMcachesorfromdisk.TheRocksDBenginehasawrite-aheadlog(WAL)andusesbackgroundthreadsforcompaction.Itsupportsdatacompression.

TheRocksDBstorageenginehasdocument-levellocking.Readoperationsdonotblockandareneverblockedbyotheroperations.Writeoperationsonlyblockwritesonthesamedocuments/indexvalues.Becausemultiplewriterscanoperateinparallelonthesamecollection,thereisthepossibilityofwrite-writeconflicts.Ifsuchwriteconflictisdetected,oneofthewriteoperationsisabortedwitherror1200("conflict").Clientapplicationscantheneitheraborttheoperationorretry,basedontherequiredconsistencysemantics.

Storageengineselection

ThestorageenginetouseinanArangoDBclusterorasingle-serverinstancemustbeselectedinitially.ThedefaultstorageengineinArangoDB3.2istheMMFilesengineifnostorageengineisselectedexplicitly.Thisensuresallusersupgradingfromearlierversionscancontinuewiththewell-knownMMFilesengine.

Toselectthestorage-engine,thereistheconfigurationoption--server.storage-engine.Itcanbesettoeithermmfiles,rocksdborauto.Whilethefirsttwovalueswillexplicitlyselectastorageengine,theautooptionwillautomaticallychoosethestorageenginebasedonwhichstorageenginewaspreviouslyselected.Ifnoenginewasselectedpreviously,autowillselecttheMMFilesengine.Ifanenginewaspreviouslyselected,theselectionwillbewrittentoafileENGINEintheserver'sdatabasedirectoryandwillbereadfromthereatanysubsequentserverstarts.

WhatsNewin3.2

517

Oncethestorageenginewasselected,theselectioncannotbechangedbyadjusting--server.storage-engine.Inordertoswitchtoanotherstorageengine,itisrequiredtore-starttheserverwithanother(empty)databasedirectory.Inordertousedatacreatedwiththeotherstorageengine,itisrequiredtodumpthedatafirstwiththeoldengineandrestoreitusingthenewstorageengine.Thiscanbeachievedviainvokingarangodumpandarangorestore.

UnlikeinMySQL,thestorageengineselectioninArangoDBisforanentireclusteroranentiresingle-serverinstance.Alldatabasesandcollectionswillusethesamestorageengine.

RocksDBstorageengine:supportedindextypes

TheexistingindexesintheRocksDBengineareallpersistent.Thefollowingindexesaresupportedthere:

primary:thistypeofindexisautomaticallycreated.Itindexes_id/_key

edge:thisindexisautomaticallycreatedforedgecollections.Itindexes_fromand_to

hash,skiplist,persistent:theseareuser-definedindexes,Despitetheirnames,theyareneitherhashnorskiplistindexes.TheseindextypesmaptothesameRocksDB-basedsortedindeximplementation.Thesameistrueforthe"persistent"index.Thenames"hash","skiplist"and"persistent"areonlyusedforcompatibilitywiththeMMFilesenginewheretheseindexesexistedinpreviousandthecurrentversionofArangoDB.

geo:user-definedindexforproximitysearches

fulltext:user-definedsortedrevertedindexonwordsoccurringindocuments

SatelliteCollections

WithSatelliteCollections,youcandefinecollectionstoshardtoaclusterandcollectionstoreplicatetoeachmachine.TheArangoDBqueryoptimizerknowswhereeachshardislocatedandsendstherequeststotheDBServersinvolved,whichthenexecutesthequery,locally.Withthisapproach,networkhopsduringjoinoperationsonshardedcollectionscanbeavoidedandresponsetimescanbeclosetothatofasingleinstance.

SatellitecollectionsareavailableintheEnterpriseedition.

MemorymanagementmakearangodstartwithlessV8JavaScriptcontexts

Thisspeedsuptheserverstartandmakesarangoduselessmemoryatstart.WheneveraV8contextisneededbyaFoxxactionorsomeotherJavaScriptoperationandthereisnousableV8context,anewcontextwillbecreateddynamicallynow.

Upto--javascript.v8-contextsV8contextswillbecreated,sothisoptionwillchangeitsmeaning.PreviouslyasmanyV8contextsasspecifiedbythisoptionwerecreatedatserverstart,andthenumberofV8contextsdidnotchangeatruntime.NowuptothisnumberofV8contextswillbeinuseatthesametime,buttheactualnumberofV8contextsisdynamic.

ThegarbagecollectorthreadwillautomaticallydeleteunusedV8contextsafterawhile.Thenumberofsparecontextswillgodowntoasfewasconfiguredinthenewoption--javascript.v8-contexts-minimum.ActuallythatmanyV8contextsarealsocreatedatserverstart.

ThefirstfewrequestsinnewV8contextsmaytakelongerthanincontextsthathavebeentherealready.PerformancemaythereforesufferabitfortheinitialrequestssenttoArangoDBorwhenthereareonlyfewbutperformance-criticalsituationsinwhichnewV8contextsneedtobecreated.Ifthisisaconcern,itcaneasilybefixedbysetting--javascipt.v8-contexts-minimumand--javascript.v8-contextstoarelativelyhighvalue,whichwillguaranteethatmanynumberofV8contextstobecreatedatstartupandkeptaroundevenwhenunused.

WaitingforanunusedV8contextwillnowalsoabortandwritealogmessageincasenoV8contextcanbeacquired/createdafter60seconds.

thenumberofpendingoperationsinarangodcannowbelimitedtoaconfigurablenumber.Ifthisnumberisexceeded,theserverwillnowrespondwithHTTP503(serviceunavailable).Themaximumsizeofpendingoperationsiscontrolledviathestartupoption--server.maximal-queue-size.Settingitto0means"nolimit".

WhatsNewin3.2

518

thein-memorydocumentrevisionscachewasremovedentirelybecauseitdidnotprovidetheexpectedbenefits.The3.1implementationshadoweddocumentdatainRAM,whichincreasedtheserver'sRAMusagebutdidnotspeedupdocumentlookupstoomuch.

Thisalsoobsoletesthestartupoptions--database.revision-cache-chunk-sizeand--database.revision-cache-target-size.

TheMMFilesenginenowdoesnotuseadocumentrevisionscachebuthasin-memoryindexesandmapsdocumentstoRAMautomaticallyviammapwhendocumentsareaccessed.TheRocksDBenginehasitsownmechanismforcachingaccesseddocuments.

CommunicationLayerHTTPresponsesreturnedbyarangodwillnowincludetheextraHTTPheaderx-content-type-options:nosnifftoworkaroundacross-sitescriptingbuginMSIE

thedefaultvaluefor--ssl.protocolwaschangedfromTLSv1toTLSv1.2.Whennotexplicitlyset,arangodandallclienttoolswillnowuseTLSv1.2.

theJSONdatainallincomingHTTPrequestsinnowvalidatedforduplicateattributenames.

IncomingJSONdatawithduplicateattributenameswillnowberejectedasinvalid.PreviousversionsofArangoDBonlyvalidatedtheuniquenessofattributenamesinsideincomingJSONforsomeAPIendpoints,butnotconsistentlyforallAPIs.

InternalJavaScriptRESTactionswillnowhidetheirstacktracestotheclientunlessinHTTPresponses.Insteadtheywillalwayslogtothelogfile.

JavaScript

updatedV8versionto5.7.0.0

changeundocumentedbehaviourincaseofinvalidrevisionidsinIf-MatchandIf-None-Matchheadersfrom400(BAD)to412(PRECONDITIONFAILED).

changedefaultstringtruncationlengthfrom80charactersto256charactersforprint/printShellfunctionsinArangoShellandarangod.Thiswillemitlongerprefixesofstringvaluesbeforetruncatingthemwith...,whichishelpfulfordebugging.ThischangeismostlyusefulwhenusingtheArangoShell(arangosh).

the@arangodbmodulenowprovidesatimefunctionwhichreturnsthecurrenttimeinsecondsasafloatingpointvaluewithmicrosecondprecision.

FoxxThereisnowanofficialHTTPAPIformanagingservices,allowingservicestobeinstalled,modified,uninstalledandreconfiguredwithouttheadministrativewebinterface.

ItisnowpossibletouploadasingleJavaScriptfileinsteadofaziparchiveifyourservicerequiresnoconfiguration,additionalfilesorsetup.Aminimalmanifestwillbegeneratedautomaticallyuponinstallationandtheuploadedfilewillbeusedastheservice'smainentrypoint.

DistributedGraphProcessing

WeaddedsupportforexecutingdistributedgraphalgorithmsakaPregel.Userscanrunarbitraryalgorithmsonanentiregraph,includinginclustermode.Weimplementedanumberofalgorithmsforvariouswell-knowngraphmeasures:

ConnectedComponentsPageRankShortestPathsCentralityMeasures(CentralityandBetweeness)

WhatsNewin3.2

519

CommunityDetection(viaLabelPropagation,Speakers-ListenersLabelPropagationorDMID)Userscancontributetheirownalgorithms

AQL

Optimizerimprovements

GeoindexesarenowimplicitlyandautomaticallyusedwhenusingappropriateSORT/FILTERstatementsinAQL,withouttheneedtousethesomewhatlimitedspecial-purposegeoAQLfunctionsNEARorWITHIN.

ComparedtousingthespecialpurposeAQLfunctionsthisapproachhastheadvantagethatitismorecomposable,andwillalsohonoranyLIMITvaluesusedintheAQLquery.

ThespecialpurposeNEARAQLfunctioncannowbesubstitutedwiththefollowingAQL(providedthereisageoindexpresentonthedoc.latitudeanddoc.longitudeattributes):

FORdocingeoSort

SORTDISTANCE(doc.latitude,doc.longitude,0,0)

LIMIT5

RETURNdoc

WITHINcanbesubstitutedwiththefollowingAQL:

FORdocingeoFilter

FILTERDISTANCE(doc.latitude,doc.longitude,0,0)<2000

RETURNdoc

Miscellaneousimprovements

addedREGEX_REPLACEAQLfunction

REGEX_REPLACE(text,search,replacement,caseInsensitive)→string

Replacethepatternsearchwiththestringreplacementinthestringtext,usingregularexpressionmatching.

text(string):thestringtosearchinsearch(string):aregularexpressionsearchpatternreplacement(string):thestringtoreplacethesearchpatternwithreturnsstring(string):thestringtextwiththesearchregexpatternreplacedwiththereplacementstringwhereverthepatternexistsintext

addednewstartupoption--query.fail-on-warningtomakeAQLqueriesabortinsteadofcontinuingwithwarnings.

Whensettotrue,thiswillmakeanAQLquerythrowanexceptionandabortincaseawarningoccurs.Thisoptionshouldbeusedindevelopmenttocatcherrorsearly.Ifsettofalse,warningswillnotbepropagatedtoexceptionsandwillbereturnedwiththequeryresults.Thestartupoptioncanalsobeoverridenonaperquery-level.

theslowquerylistnowcontainsthevaluesofbindvariablesusedintheslowqueries.Bindvariablesarealsoprovidedforthecurrentlyrunningqueries.Thishelpsdebuggingsloworblockingqueriesthatusedynamiccollectionnamesviabindparameters.

AQLbreakingchangeincluster:TheSHORTEST_PATHstatementusingedgecollectionnamesinsteadofagraphnamesnowrequirestoexplicitlynamethevertexcollectionnameswithintheAQLqueryinthecluster.ItcanbedonebyaddingWITH<name>atthebeginningofthequery.

Example:

FORv,eINOUTBOUNDSHORTEST_PATH@startTO@targetedges[...]

Nowhastobe:

WITHvertices

FORv,eINOUTBOUNDSHORTEST_PATH@startTO@targetedges[...]

WhatsNewin3.2

520

Thischangeisduetoavoiddeadlocksitationsinclusteredcase.Anerrorstatingtheaboveisincluded.

Clienttools

addeddataexporttool,arangoexport.

arangoexportcanbeusedtoexportcollectionstojson,jsonlorxmlandexportagraphorcollectionstoxgmml.

added"jsonl"asinputfiletypeforarangoimp

added--translateoptionforarangoimptotranslateattributenamesfromtheinputfilestoattriubtenamesexpectedbyArangoDB

The--translateoptioncanbespecifiedmultipletimes(oncepertranslationtobeexecuted).Thefollowingexamplerenamesthe"id"columnfromtheinputfileto"_key",andthe"from"columnto"_from",andthe"to"columnto"_to":

arangoimp--typecsv--filedata.csv--translate"id=_key"--translate"from=_from"--translate"to=_to"

--translateworksforCSVandTSVinputsonly.

added--threadsoptiontoarangoimptospecifythenumberofparallelimportthreads

changeddefaultvalueforclienttoolsoption--server.max-packet-sizefrom128MBto256MB.thisallowstransferringbiggerresultsetsfromtheserverwithouttheclienttoolsrejectingthemasinvalid.

Authentication

addedLDAPauthentication(Enterpriseonly)

Authorization

addedreadonlymodeforuserscollectionlevelauthorizationrights

Readmoreintheoverview.

Foxxthecookiesessiontransportnowsupportsalloptionssupportedbythecookiemethodoftheresponseobject.

it'snowpossibletoprovideyourownversionofthegraphql-syncmodulewhenusingtheGraphQLextensionsforFoxxbypassingacopyofthemoduleusingthenewgraphqloption.

customAPIendpointscannowbetaggedusingthetagmethodtogenerateacleanerSwaggerdocumentation.

MiscellaneousChanges

arangodnowvalidatesseveralOS/environmentsettingsonstartupandwarnsifthesettingsarenon-ideal.Itadditionallywillprintoutwaystoremedytheoptions.

MostofthechecksareexecutedonLinuxsystemsonly.

added"deduplicate"attributeforarrayindexes,whichcontrolswhetherinsertingduplicateindexvaluesfromthesamedocumentintoauniquearrayindexwillleadtoanerrorornot:

//withdeduplicate=true,whichisthedefaultvalue:

db._create("test");

db.test.ensureIndex({type:"hash",fields:["tags[*]"],deduplicate:true});

db.test.insert({tags:["a","b"]});

WhatsNewin3.2

521

db.test.insert({tags:["c","d","c"]});//willwork,becausededuplicate=true

db.test.insert({tags:["a"]});//willfail

//withdeduplicate=false

db._create("test");

db.test.ensureIndex({type:"hash",fields:["tags[*]"],deduplicate:false});

db.test.insert({tags:["a","b"]});

db.test.insert({tags:["c","d","c"]});//willnotwork,becausededuplicate=false

db.test.insert({tags:["a"]});//willfail

WhatsNewin3.2

522

KnownIssuesThefollowingknownissuesarepresentinthisversionofArangoDBandwillbefixedinfollow-upreleases:

RocksDBstorageengine

TheRocksDBstorageengineisintentionallymissingthefollowingfeaturesthatarepresentintheMMFilesengine:

thedatafiledebugger(arango-dfdb)cannotbeusedwiththisstorageengine

RocksDBhasitsowncrashrecoverysousingthedfdbwillnotmakeanysensehere.

APIsthatreturncollectionpropertiesorfigureswillreturnslightlydifferentattributesfortheRocksDBenginethanfortheMMFilesengine.Forexample,theattributesjournalSize,doCompact,indexBucketsandisVolatilearepresentintheMMFilesenginebutnotintheRocksDBengine.ThememoryusagefiguresreportedforcollectionsintheRocksDBengineareestimatevalues,whereastheyareexactfortheMMFilesengine.

theRocksDBenginedoesnotsupportsomeoperationswhichonlymakesenseinthecontextoftheMMFilesengine.Theseare:

therotatemethodoncollectionstheflushmethodforWALfiles

theRocksDBstorageenginedoesnotsupportvolatilecollections

transactionsarelimitedinsize.Transactionsthatgettoobig(intermsofnumberofoperationsinvolvedorthetotalsizeofdatamodifiedbythetransaction)willbecommittedautomatically.EffectivelythismeansthatbigusertransactionsaresplitintomultiplesmallerRocksDBtransactionsthatarecommittedindividually.TheentireusertransactionwillnotnecessarilyhaveACIDpropertiesinthiscase.

Thethresholdvaluesfortransactionsizescanbeconfiguredgloballyusingthestartupoptions

--rocksdb.intermediate-commit-size:ifthesizeofalloperationsinatransactionreachesthisthreshold,thetransactioniscommittedautomaticallyandanewtransactionisstarted.Thevalueisspecifiedinbytes.

--rocksdb.intermediate-commit-count:ifthenumberofoperationsinatransactionreachesthisvalue,thetransactioniscommittedautomaticallyandanewtransactionisstarted.

--rocksdb.max-transaction-size:thisisanupperlimitforthetotalnumberofbytesofalloperationsinatransaction.Iftheoperationsinatransactionconsumemorethanthisthresholdvalue,thetransactionwillautomaticallyabortwitherror32("resourcelimitexceeded").

Itisalsopossibletooverridethesethresholdspertransaction.

Thefollowingknownissueswillberesolvedinfuturereleases:

theRocksDBengineisnotyetperformance-optimizedandpotentiallynotwellconfigured

collectionsforwhichageoindexispresentwillusecollection-levelwritelocksevenwiththeRocksDBengine.Readsfromthesecollectionscanstillbedoneinparallelbutnowrites

modifyingdocumentsinacollectionwithageoindexwillcausemultipleadditionalwritestoRocksDBformaintainingtheindexstructures

thenumberofdocumentsreportedforcollections(db.<collection>.count())maybeslightlywrongduringtransactionsifthereareparalleltransactionsongoingforthesamecollectionthatalsomodifythenumberofdocuments

theanyoperationtoprovidearandomdocumentfromacollectionissupportedbytheRocksDBenginebuttheoperationhasmuchhigheralgorithmiccomplexitythanintheMMFilesengine.Itisthereforediscouragedtocallitforcasesotherthanmanualinspectionofafewdocumentsinacollection

AQLqueriesintheclusterstillissueanextralockingHTTPrequestpershardthoughthiswouldnotbenecessaryfortheRocksDBengineinmostcases

Installer

KnownIssuesin3.2

523

Upgradingfrom3.1to3.2onWindowsrequirestheusertomanuallycopythedatabasedirectorytothenewlocationandrunanupgradeonthedatabase.PleaseconsulttheDocumentationfordetailedinstructions.

SystemIntegration

OnsomeLinuxsystemssystemdandsystemvmightreportthatthearangodbserviceisingoodconditionwhenitcouldnotbestarted.Inthiscasetheuserneedstocheck/var/log/arangodb3forfurtherinformationaboutthefailedstartup.

MacOSX

Storageengineisnotchangeableonanexistingdatabase.Currentlyonlytheinitialselectionofthestorageengineissupported.Inordertouseanotherstorageengine,youhavetodeleteyourArangoDBapplication(MacApplicationFolder)and/Users/<your_user_name>/Library/ArangoDBfolder.

OpenSSL1.1

ArangoDBhasbeentestedwithOpenSSL1.0onlyandwon'tbuildagainst1.1whencompilingonyourown.SeehereforhowtocompileonsystemsthatshipOpenSSL1.1bydefault.

KnownIssuesin3.2

524

IncompatiblechangesinArangoDB3.2ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB3.2,andadjustanyclientprogramsifnecessary.

AQL

AQLbreakingchangeincluster:TheSHORTEST_PATHstatementusingedge-collectionnamesinsteadofagraphnamenowrequirestoexplicitlynamethevertex-collectionnameswithintheAQLqueryinthecluster.ItcanbedonebyaddingWITH<name>atthebeginningofthequery.

Example:

FORv,eINOUTBOUNDSHORTEST_PATH@startTO@targetedges[...]

Nowhastobe:

WITHvertices

FORv,eINOUTBOUNDSHORTEST_PATH@startTO@targetedges[...]

Thischangeisduetoavoiddead-locksitationsinclusteredcase.Anerrorstatingtheaboveisincluded.

RESTAPIRemovedundocumentedinternalHTTPAPI:

PUT/_api/edgesThedocumentedGET/_api/edgesandtheundocumentedPOST/_api/edgesremainsunmodified.

changeundocumentedbehaviourincaseofinvalidrevisionidsinIf-MatchandIf-None-MatchheadersfromreturningHTTPstatuscode400(badrequest)toreturningHTTPstatuscode412(preconditionfailed).

theRESTAPIforfetchingthelistofcurrentlyrunningAQLqueriesandtheRESTAPIforfetchingthelistofslowAQLqueriesnowreturnanextrabindVarsattributewhichcontainsthebindparametersusedbythequeries.

ThisaffectsthereturnvaluesofthefollowingAPIendpoints:

GET/_api/query/currentGET/_api/query/slow

TheRESTAPIforretrievingindexes(GET/_api/index)nowreturnsthededuplicateattributeforeachindex

TheRESTAPIforcreatingindexes(POST/_api/index)nowacceptstheoptionaldeduplicateattribute

TheRESTAPIforexecutingaserver-sidetransaction(POST/_api/transaction)nowacceptstheoptionalattributes:maxTransactionSize,intermediateCommitCount,intermediateCommitSize

TheRESTAPIforcreatingacursor(POST/_api/cursor)nowacceptstheoptionalattributes:failOnWarning,maxTransactionSize,maxWarningCount,intermediateCommitCount,satelliteSyncWait,intermediateCommitSize.skipInaccessibleCollections

JavaScriptAPI

changeundocumentedbehaviourincaseofinvalidrevisionidsinJavaScriptdocumentoperationsfromreturningerrorcode1239("illegaldocumentrevision")toreturningerrorcode1200("conflict").

thecollection.getIndexes()functionnowreturnsthededuplicateattributeforeachindex

thecollection.ensureIndex()functionnowacceptstheoptionaldeduplicateattribute

Incompatiblechangesin3.2

525

Foxx

JWTtokensissuedbythebuilt-inJWTsessionstoragenowcorrectlyspecifytheiatandexpvaluesinsecondsratherthanmillisecondsasspecifiedintheJSONWebTokenstandard.

Thismayresultinpreviouslyexpiredtokensusingmillisecondsbeingincorrectlyaccepted.ForthisreasonitisrecommendedtoreplacethesigningsecretorsetthenewmaxExpoptiontoareasonablevaluethatissmallerthantheoldestissuedexpirationtimestamp.

ForexamplesettingmaxExpto10**12wouldinvalidateallincorrectlyissuedtokensbefore9September2001withoutimpairingnewtokensuntiltheyear33658(atwhichpointthesetokensarehopefullynolongerrelevant).

ArangoDBrunninginstandalonemodewillcommitallservicesinthejavascript.app-pathtothedatabaseonstartup.ThismayresultinuninstalledservicesshowingupinArangoDBiftheywerenotproperlyremovedfromthefilesystem.

ArangoDBcoordinatorsinaclusternowperformaself-healingstepduringstartuptoensureinstalledservicesareconsistentaccrossallcoordinators.WerecommendbackingupyourservicesandconfigurationbeforeupgradingtoArangoDB3.2,especiallyifyouhavemadeuseofthedevelopmentmode.

Servicesinstalledbeforeupgradingto3.2(includingservicesinstalledonalphareleasesofArangoDB3.2)areNOTpickedupbythecoordinatorself-healingwatchdog.Thiscanbesolvedbyeitherupgrading/replacingtheseservicesorbyusingthe"commit"routeoftheFoxxservicemanagementHTTPAPI,whichcommitstheexactservicesinstalledonagivencoordinatortothecluster.Newserviceswillbepickedupautomatically.

TheformatusedbyFoxxtostoreinternalservicemetadatainthedatabasehasbeensimplifiedandexistingdocumentswillbeupdatedtothenewformat.Ifyouhavemadeanychangestothedatastoredinthe_appssystemcollection,youmaywishtoexportthesechangesastheywillbeoverwritten.

ThereisnowanofficialHTTPAPIformanagingservices.IfyouwerepreviouslyusinganyoftheundocumentedAPIsortheroutesusedbytheadministrativewebinterfacewehighlyrecommendmigratingtothenewAPI.TheoldundocumentedHTTPAPIformananagingservicesisdeprecatedandwillberemovedinafutureversionofArangoDB.

Althoughchangestothefilesystemoutsideofdevelopmentmodewerealreadystronglydiscouraged,thisisareminderthattheyarenolongersupported.Allfilesgeneratedbyservices(whetherbyasetupscriptorduringnormaloperationsuchasuploads)shouldeitherbestoredoutsidetheservicedirectoryorbeconsideredextremelyvolatile.

IntroduceddistinctionbetweenarangoUserandauthorizedinFoxxrequests.ClusterinternalrequestswillneverhaveanarangoUserbutareauthorized.InearlierversionsofArangoDBpartsofthestatisticswerenotaccessiblebythecoordinatorsbecausetheunderlyingFoxxservicecouldn'tauthorizetherequests.Itnowcorrectlychecksthenewreq.authorizedproperty.req.arangoUserstillworksasbefore.Endusersmayusethisnewpropertyaswelltoeasilycheckifarequestisauthorizedornotregardlessofaspecificuser.

Command-lineoptionschanged

--server.maximal-queue-sizeisnowanabsolutemaximum.Ifthequeueisfull,then503isreturned.Settingitto0means"nolimit".Thedefaultvalueforthisoptionisnow0.

thedefaultvaluefor--ssl.protocolhasbeenchangedfrom4(TLSv1)to5(TLSv1.2).

thestartupoptions--database.revision-cache-chunk-sizeand--database.revision-cache-target-sizearenowobsoleteanddonothing

thestartupoption--database.index-threadsoptionisnowobsolete

theoption--javascript.v8-contextsisnowanabsolutemaximum.TheservermaystartlessV8contextsforJavaScriptexecutionatstartup.IfatsomepointtheserverneedsmoreV8contextsitmaystartthemdynamically,untilthenumberofV8contextsreachesthevalueof--javascript.v8-contexts.

theminimumnumberofV8contextstocreateatstartupcanbeconfiguredviathenewstartupoption--javascript.v8-contexts-minimum.

addedcommand-lineoption--javascript.allow-admin-execute

Incompatiblechangesin3.2

526

Thisoptioncanbeusedtocontrolwhetheruser-definedJavaScriptcodeisallowedtobeexecutedonserverbysendingviaHTTPtotheAPIendpoint/_admin/executewithanauthenticateduseraccount.Thedefaultvalueisfalse,whichdisablestheexecutionofuser-definedcode.Thisisalsotherecommendedsettingforproduction.Intestenvironments,itmaybeconvenienttoturntheoptiononinordertosendarbitrarysetuporteardowncommandsforexecutionontheserver.

TheintroductionofthisoptionchangesthedefaultbehaviorofArangoDB3.2:3.2nowbydefaultdisablestheexecutionofJavaScriptcodeviathisAPI,whereasearlierversionsallowedit.Torestoretheoldbehavior,itisnecessarytosettheoptiontotrue.

UsersManagement

Itisnolongersupportedtoaccessthe_userscollecctioninanywaydirectly,exceptthroughtheofficial@arangodb/usersmoduleorthe_apit/usersRESTAPI.

Theaccesstothe_userscollectionfromoutsideofthearangodserverprocessisnowforbidden(Throughdrivers,arangoshortheRESTAPI).Foxxservicesarestillbeabletoaccessthe_userscollectionfornow,butthismightchangeinfutureminorreleases.

Theinternalformatofthedocumentsinthe_userscollectionhaschangedfrompreviousversions

The_queuescollectiononlyallowsread-onlyaccessfromoutsideofthearangodserverprocess.

Accessing_queuesisonlysupportedthroughtheofficial@arangodb/queuesmoduleforFoxxapps.

Incompatiblechangesin3.2

527

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB3.1.ArangoDB3.1alsocontainsseveralbugfixesthatarenotlistedhere.

SmartGraphs

ArangoDB3.1addsafirstmajorenterpriseonlyfeaturecalledSmartGraphs.SmartGraphsformanadditiontothealreadyexistinggraphfeaturesandallowtoscalegraphsbeyondasinglemachinewhilekeepingalmostthesamequeryperformance.TheSmartGraphfeatureissuggestedforallgraphdatabaseusecasesthatrequireaclusterofdatabaseserversforwhateverreason.Youcaneitherhaveagraphthatistoolargetobestoredonasinglemachineonly.Oryoucanhaveasmallgraph,butatthesametimeneedadditionaldatawithhastobeshardedandyouwanttokeepalloftheminthesameenvirenment.Oryousimplyusetheclusterforhigh-availability.InalltheabovecasesSmartGraphswillsignificantlyincreasetheperformanceofgraphoperations.Formoredetailedinformationreadthismanualsection.

Dataformat

Theformatoftherevisionvaluesstoredinthe_revattributeofdocumentshasbeenchangedin3.1.Upto3.0theywerestringscontaininglargishdecimalnumbers.With3.1,revisionvaluesarestillstrings,butareactuallyencodedtimestampsofthecreationdateoftherevisionofthedocument.Thetimestampsareacquiredusingahybridlogicalclock(HLC)ontheDBserverthatholdstherevision(fortheconceptofahybridlogicalclockseethispaper).Seethismanualsectionfordetails.

ArangoDB>=3.1canArangoDB3.0databasedirectoriesandwillsimplycontinuetousetheold_revattributevalues.Newrevisionswillbewrittenwiththenewtimestamps.

ItishighlyrecommendedtobackupallyourdatabeforeloadingadatabasedirectorythatwaswrittenbyArangoDB<=3.0intoanArangoDB>=3.1.

CommunicationLayerArangoDBupto3.0usedlibevforthecommunicationlayer.ArangoDBstartingfrom3.1usesBoostASIO.

StartingwithArangoDB3.1webegintoprovidetheVelocyStreamProtocol(vst)asaadditiontotheestablishedhttpprotocol.

Afewoptionshavechangedconcerningcommunication,pleasecheckoutIncompatiblechangesin3.1.

ClusterForitsinternalclustercommunicationa(bundledversion)ofcurlisnowbeingused.Thisenablesasynchronousoperationthroughouttheclusterandshouldimprovegeneralperformanceslightly.

Authenticationisnowsupportedwithinthecluster.

Documentrevisionscache

TheArangoDBservernowprovidesanin-memorycacheforfrequentlyaccesseddocumentrevisions.Documentsthatareaccessedduringread/writeoperationsareloadedintotherevisionscacheautomatically,andsubsequentlyservedfromthere.

Thecachehasatotaltargetsize,whichcanbecontrolledwiththestartupoption--database.revision-cache-target-size.Oncethecachereachesthetargetsize,olderentriesmaybeevictedfromthecachetofreememory.Notethatthetargetsizecurrentlyisahighwatermarkthatwilltriggercachememorygarbagecollectionifexceeded.However,ifallcachechunksarestillinusewhenthehighwatermarkisreached,thecachemaystillgrowandallocatemorechunksuntilcacheentriesbecomeunusedandareallowedtobegarbage-collected.

WhatsNewin3.1

528

Thecacheismaintainedonaper-collectionbasis,thatis,memoryforthecacheisallocatedonaper-collectionbasisinchunks.Thesizeforthecachememorychunkscanbecontrolledviathestartupoption--database.revision-cache-chunk-size.Thedefaultvalueis4MBperchunk.Biggerchunksizesallowsavingmoredocumentsperchunk,whichcanleadtomoreefficientchunkallocationandlookups,butwillalsoleadtomemorywasteifmanychunksareallocatedandnotfullyused.Thelatterwillbethecaseifthereexistmanysmallcollectionswhichallallocatetheirownchunksbutnotfullyutilizethembecauseofthelownumberofdocuments.

AQL

Functionsadded

ThefollowingAQLfunctionshavebeenaddedin3.1:

OUTERSECTION(array1,array2,...,arrayn):returnsthevaluesthatoccuronlyonceacrossallarraysspecified.

DISTANCE(lat1,lon1,lat2,lon2):returnsthedistancebetweenthetwocoordinatesspecifiedby(lat1,lon1)and(lat2,lon2).Thedistanceiscalculatedusingthehaversineformula.

JSON_STRINGIFY(value):returnsaJSONstringrepresentationofthevalue.

JSON_PARSE(value):convertsaJSON-encodedstringintoaregularobject

Indexusageintraversals

3.1allowsAQLtraversalstouseotherindexesthanjusttheedgeindex.Traversalswithfiltersonedgescannowmakeuseofmorespecificindexes.Forexample,thequery

FORv,e,pIN2OUTBOUND@start@@edge

FILTERp.edges[0].foo=="bar"

RETURN[v,e,p]

mayuseahashindexon["_from","foo"]insteadoftheedgeindexonjust["_from"].

Optimizerimprovements

MaketheAQLqueryoptimizerinjectfilterconditionexpressionsreferredtobyvariablesduringfilterconditionaggregation.Forexample,inthefollowingquery

FORdocINcollection

LETcond1=(doc.value==1)

LETcond2=(doc.value==2)

FILTERcond1||cond2

RETURN{doc,cond1,cond2}

theoptimizerwillnowinjecttheconditionsforcond1andcond2intothefilterconditioncond1||cond2,expandingitto(doc.value==1)||(doc.value==2)andmakingtheseconditionsavailableforindexsearching.

Notethattheoptimizerpreviouslyalreadyinjectedsomeconditionsintootherconditions,butonlyifthevariablethatdefinedtheconditionwasnotusedelsewhere.Forexample,thefilterconditioninthequery

FORdocINcollection

LETcond=(doc.value==1)

FILTERcond

RETURN{doc}

alreadygotoptimizedbeforebecausecondwasonlyusedonceinthequeryandtheoptimizerdecidedtoinjectitintotheplacewhereitwasused.

Thisonlyworkedforvariablesthatwerereferredtoonceinthequery.Whenavariablewasusedmultipletimes,theconditionwasnotinjectedasinthefollowingquery

FORdocINcollection

WhatsNewin3.1

529

LETcond=(doc.value==1)

FILTERcond

RETURN{doc,cond}

3.1allowsusingthisconditionsothatthequerycanuseanindexondoc.value(ifsuchindexexists).

Miscellaneousimprovements

Theperformanceofthe[*]operatorwasimprovedforcasesinwhichthisoperatordidnotuseanyfilters,projectionsand/oroffset/limits.

TheAQLqueryexecutorcannowreportthetimerequiredforloadingandlockingthecollectionsusedinanAQLquery.Whenprofilingisenabled,itwillreportthetotalloadingandlockingtimeforthequeryintheloadingcollectionssub-attributeoftheextra.profilevalueoftheresult.TheloadingandlockingtimecanalsobeviewintheAQLqueryeditorinthewebinterface.

AuditLogAuditlogginghasbeenadded,seeAuditing.

ClienttoolsAddedoption--skip-linesforarangoimpThisallowsskippingthefirstfewlinesfromtheimportfileincasetheCSVorTSVimportareusedandsomeinitiallinesshouldbeskippedfromtheinput.

WebAdminInterfaceTheusabilityoftheAQLeditorsignificantlyimproved.InadditiontothestandardJSONoutput,theAQLEditorisnowabletorenderqueryresultsasagraphprevieworatable.FurthermoretheAQLeditordisplaysqueryprofilinginformation.

AddedanewGraphViewerinordertoexchangethetechnicallyobsoleteversion.ThenewGraphViewerisbasedonCanvasbutdoesalsoincludeafirstWebGLimplementation(limitedfunctionality-willchangeinthefuture).ThenewGraphVieweroffersasmoothwaytodiscoverandvisualizeyourgraphs.

Theshardviewinclustermodenowdisplaysaprogressindicatorwhilemovingshards.

AuthenticationUptoArangoDB3.0authenticationofclientrequestswasonlypossiblewithHTTPbasicauthentication.

Startingwith3.1itisnowpossibletoalsouseaJSONWebTokens(JWT)forauthenticatingincomingrequests.

FordetailschecktheHTTPauthenticationchapter.Bothauthenticationmethodsarevalidandwillbesupportedinthenearfuture.Usewhateversuitsyoubest.

Foxx

GraphQL

ItisnoweasytogetstartedwithprovidingGraphQLAPIsinFoxx,seeFoxxGraphQL.

OAuth2

FoxxnowofficiallyprovidesamoduleforimplementingOAuth2clients,seeFoxxOAuth2.

Per-routemiddleware

WhatsNewin3.1

530

It'snowpossibletospecifymiddlewarefunctionsforaroutewhendefiningaroutehandler.Thesemiddlewarefunctionsonlyapplytothesinglerouteandsharetheroute'sparameterdefinitions.CheckouttheFoxxRouterdocumentationformoreinformation.

WhatsNewin3.1

531

IncompatiblechangesinArangoDB3.1ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB3.1,andadjustanyclientprogramsifnecessary.

CommunicationLayer

TheinternalcommicationlayerisnowbasedonBoostASIO.Afewoptionsregardingthreadsandcommunicationhavebeenchanged.

Therearenolongertwodifferentthreadspools(--scheduler.threadsand--server.threads).Theoption--scheduler.threadshasbeenremoved.Thenumberofthreadsisnowcontrolledbytheoption--server.threadsonly.Bydefault--server.threadsissettothenumberofhyper-cores.

Asaconsequenceofthechange,thefollowing(hidden)startupoptionshavebeenremoved:

--server.extra-threads

--server.aql-threads

--server.backend

--server.show-backends

--server.thread-affinity

AQL

ThebehavioroftheAQLarraycomparisonoperatorshaschangedforemptyarrays:

ALLandANYnowalwaysreturnfalsewhentheleft-handoperandisanemptyarray.Thebehaviorfornon-emptyarraysdoesnotchange:

[]ALL==1willreturnfalse[1]ALL==1willreturntrue[1,2]ALL==1willreturnfalse[2,2]ALL==1willreturnfalse[]ANY==1willreturnfalse[1]ANY==1willreturntrue[1,2]ANY==1willreturntrue[2,2]ANY==1willreturnfalse

NONEnowalwaysreturnstruewhentheleft-handoperandisanemptyarray.Thebehaviorfornon-emptyarraysdoesnotchange:

[]NONE==1willreturntrue[1]NONE==1willreturnfalse[1,2]NONE==1willreturnfalse[2,2]NONE==1willreturntrue

WITHinclustertraversalsisnowmandatoryinordertoavoiddeadlocks.

Dataformatchanges

TheattributemaximalSizehasbeenrenamedtojournalSizeincollectionmeta-datafiles("parameter.json").FilescontainingthemaximalSizeattributewillstillbepickedupcorrectlyfornot-yetadjustedcollections.

Theformatoftherevisionvaluesstoredinthe_revattributeofdocumentshasbeenchangedin3.1.Upto3.0theywerestringscontaininglargishdecimalnumbers.With3.1,revisionvaluesarestillstrings,butareactuallyencodedtimestampsofthecreationdateoftherevisionofthedocument.Thetimestampsareacquiredusingahybridlogicalclock(HLC)ontheDBserverthatholdstherevision(fortheconceptofahybridlogicalclockseethispaper).Seethismanualsectionfordetails.

Incompatiblechangesin3.1

532

ArangoDB>=3.1canArangoDB3.0databasedirectoriesandwillsimplycontinuetousetheold_revattributevalues.Newrevisionswillbewrittenwiththenewtimestamps.

ItishighlyrecommendedtobackupallyourdatabeforeloadingadatabasedirectorythatwaswrittenbyArangoDB<=3.0intoanArangoDB>=3.1.

Tochangeallyourold_revattributesintonewstyletimestampsyouhavetousearangodumptodumpalldataout(usingArangoDB3.0),andusearangorestoreintothenewArangoDB3.1,whichisthesafestwaytoupgrade.

Thechangealsoaffectsthereturnformatof_revvaluesandotherrevisionvaluesinHTTPAPIs(seebelow).

HTTPAPIchanges

APIsadded

ThefollowingHTTPRESTAPIshavebeenaddedforonlinelogleveladjustmentoftheserver:

GET/_admin/log/levelreturnsthecurrentloglevelsettingsPUT/_admin/log/levelmodifiesthecurrentloglevelsettings

APIschanged

thefollowingRESTAPIsthatreturnrevisionidsnowmakeuseofthenewrevisionidformatintroducedin3.1.Allrevisionidsreturnedwillbestringsasin3.0,buthaveadifferentinternalformat.

ThefollowingAPIsareaffected:

GET/_api/collection/{collection}/checksum:revisionattributeGET/_api/collection/{collection}/revision:revisionattributeallotherAPIsthatreturndocuments,whichmayincludethedocuments'_revattribute

Clientapplicationsshouldnottrytointerprettheinternalsofrevisionvalues,butonlyuserevisionvaluesforcheckingwhethertworevisionstringsareidentical.

thereplicationRESTAPIswillnowusetheattributenamejournalSizeinsteadofmaximalSizewhenreturninginformationaboutcollections.

thedefaultvalueforkeepNullhasbeenchangedfromfalsetotrueforthefollowingpartialupdateoperationsforverticesandedgesin/_api/gharial:

PATCH/_api/gharial/{graph}/vertex/{collection}/{key}PATCH/_api/gharial/{graph}/edge/{collection}/{key}

ThevalueforkeepNullcanstillbesetexplicitlytofalsebysettingtheURLparameterkeepNulltoavalueoffalse.

theRESTAPIfordroppingcollections(DELETE/_api/collection)nowacceptsanoptionalquerystringparameterisSystem,whichcansettotrueinordertodropsystemcollections.Iftheparameterisnotsetornotsettotrue,theRESTAPIwillrefusetodropsystemcollections.InpreviousversionsofArangoDB,theisSystemparameterdidnotexist,andtherewasnodistinctionbetweensystemandnon-systemcollectionswhendroppingcollections.

theRESTAPIforretrievingAQLqueryresults(POST/_api/cursor)willnowreturnanadditionalsub-attributeloadingcollectionsthatwillcontainthetotaltimerequiredforloadingandlockingcollectionsduringtheAQLquerywhenprofilingisenabled.Theattributecanbefoundintheextraresultattributeinsub-attributeloadingcollections.Theattributewillonlybesetifprofilingwasenabledforthequery.

theRESTAPIforretrievingAQLqueryresults(POST/_api/cursor)willnowaccepttheoptionalattributememoryLimit.

FoxxTestingTheQUnitinterfacetoMochahasbeenremoved.Thisaffectsthebehaviourofthesuite,test,before,after,beforeEachandafterEachfunctionsinFoxxtestsuites.ThesuiteandtestfunctionsarenowprovidedbytheTDDinterface.Thebefore,after,beforeEachandafterEachfunctionsarenowprovidedbytheBDDinterface.

Incompatiblechangesin3.1

533

Thisshouldnotcauseanyproblemswithexistingtestsbutmayresultinfailuresintestcasesthatpreviouslypassedforthewrongreasons.Specificallytheexecutionorderofthebefore,after,etcfunctionsnowfollowstheintendedorderandisnolongerarbitrary.

FordetailsontheexpectedbehaviourofthesefunctionsseethetestingchapterintheFoxxdocumentation.

Incompatiblechangesin3.1

534

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB3.0.ArangoDB3.0alsocontainsseveralbugfixesthatarenotlistedhere.

Internaldataformatchanges

ArangoDBnowusesVelocyPackforstoringdocuments,queryresultsandtemporarilycomputedvalues.Usingasingledataformatremovedtheneedforsomedataconversionsinthecorethatslowedoperationsdownpreviously.

TheVelocyPackformatisalsoquitecompact,andreducesstoragespacerequirementsfor"small"valuessuchasboolean,integers,shortstrings.ThiscanspeedupseveraloperationsinsideAQLqueries.

VelocyPackdocumententriesstoredondiskarealsoself-contained,inthesensethateachstoreddocumentwillcontainallofitsdatatypeandattributenamedescriptions.Whilethismayrequireabitmorespaceforstoringthedocuments,itremovestheoverheadoffetchingattributenamesanddocumentlayoutfromsharedstructuresasinpreviousversionsofArangoDB.Italsosimplifiesthecodepathsforstoringandreadingdocuments.

AQLimprovements

Syntaximprovements

LIKEstring-comparisonoperator

AQLnowprovidesaLIKEoperatorandcanbeusedtocomparestringslikethis,forexampleinsidefilterconditions:

valueLIKEsearch

ThischangemakesLIKEanAQLkeyword.UsingLIKEasanattributeorcollectionnameinAQLthusrequiresquotingthenamefromnowon.

TheLIKEoperatoriscurrentlyimplementedbycallingthealreadyexistingAQLfunctionLIKE,whichalsoremainsoperationalin3.0.UsetheLIKEfunctionincaseyouwanttosearchcase-insensitive(optionalparameter),astheLIKEoperatoralwayscomparescase-sensitive.

AQLarraycomparisonoperators

AllAQLcomparisonoperatorsnowalsoexistinanarrayvariant.Inthearrayvariant,theoperatorisprecededwithoneofthekeywordsALL,ANYorNONE.Usingoneofthesekeywordschangestheoperatorbehaviortoexecutethecomparisonoperationforall,any,ornoneofitslefthandargumentvalues.Itisthereforeexpectedthatthelefthandargumentofanarrayoperatorisanarray.

Examples:

[1,2,3]ALLIN[2,3,4]//false

[1,2,3]ALLIN[1,2,3]//true

[1,2,3]NONEIN[3]//false

[1,2,3]NONEIN[23,42]//true

[1,2,3]ANYIN[4,5,6]//false

[1,2,3]ANYIN[1,42]//true

[1,2,3]ANY==2//true

[1,2,3]ANY==4//false

[1,2,3]ANY>0//true

[1,2,3]ANY<=1//true

[1,2,3]NONE<99//false

[1,2,3]NONE>10//true

[1,2,3]ALL>2//false

[1,2,3]ALL>0//true

[1,2,3]ALL>=3//false

["foo","bar"]ALL!="moo"//true

WhatsNewin3.0

535

["foo","bar"]NONE=="bar"//false

["foo","bar"]ANY=="foo"//true

Regularexpressionstring-comparisonoperators

AQLnowsupportstheoperators=~and!~fortestingstringsagainstregularexpressions.=~testsifastringvaluematchesaregularexpression,and!~testsifastringvaluedoesnotmatcharegularexpression.

Thetwooperatorsexpecttheirleft-handoperandstobestrings,andtheirright-handoperandstobestringscontainingvalidregularexpressionsasspecifiedbelow.

Theregularexpressionsmayconsistofliteralcharactersandthefollowingcharactersandsequences:

.–thedotmatchesanysinglecharacterexceptlineterminators.Toincludelineterminators,use[\s\S]insteadtosimulate.withDOTALLflag.\d–matchesasingledigit,equivalentto[0-9]\s–matchesasinglewhitespacecharacter\S–matchesasinglenon-whitespacecharacter\t–matchesatabcharacter\r–matchesacarriagereturn\n–matchesaline-feedcharacter[xyz]–setofcharacters.matchesanyoftheenclosedcharacters(i.e.x,yorzinthiscase[^xyz]–negatedsetofcharacters.matchesanyothercharacterthantheenclosedones(i.e.anythingbutx,yorzinthiscase)[x-z]–rangeofcharacters.Matchesanyofthecharactersinthespecifiedrange,e.g.[0-9A-F]tomatchanycharacterin0123456789ABCDEF[^x-z]–negatedrangeofcharacters.Matchesanyothercharacterthantheonesspecifiedintherange(xyz)–definesandmatchesapatterngroup(x|y)–matcheseitherxory –matchesthebeginningofthestring(e.g. xyz)$–matchestheendofthestring(e.g.xyz$)

Notethatthecharacters.,*,?,[,],(,),{,}, ,and$haveaspecialmeaninginregularexpressionsandmayneedtobeescapedusingabackslash(\\).Aliteralbackslashshouldalsobeescapedusinganotherbackslash,i.e.\\\\.

Charactersandsequencesmayoptionallyberepeatedusingthefollowingquantifiers:

x*–matcheszeroormoreoccurrencesofxx+–matchesoneormoreoccurrencesofxx?–matchesoneorzerooccurrencesofxx{y}–matchesexactlyyoccurrencesofxx{y,z}–matchesbetweenyandzoccurrencesofxx{y,}–matchesatleastyoccurencesofx

Enclosingidentifiersinforwardticks

AQLidentifierscannowoptionallybeenclosedinforwardticksinadditiontousingbackwardticks.ThisallowsconvenientwritingofAQLqueriesinJavaScripttemplatestrings(whicharedelimitedwithbackticksthemselves),e.g.

varq=`FORdocIN´collection´RETURNdoc.´name´`;

Functionsadded

ThefollowingAQLfunctionshavebeenaddedin3.0:

REGEX_TEST(value,regex):testswhetherthestringvaluematchestheregularexpressionspecifiedinregex.Returnstrueifitmatches,andfalseotherwise.

Thesyntaxforregularexpressionsisthesameasfortheregularexpressionoperators=~and!~.

WhatsNewin3.0

536

HASH(value):Calculatesahashvalueforvalue.valueisnotrequiredtobeastring,butcanhaveanydatatype.Thecalculatedhashvaluewilltakethedatatypeofvalueintoaccount,soforexamplethenumber1andthestring"1" willhavedifferenthashvalues.Forarraysthehashvalueswillbecrearedifthearrayscontainexactlythesamevalues(includingvaluetypes)inthesameorder.Forobjectsthesamehashvalueswillbecreatediftheobjectshaveexactlythesameattributenamesandvalues(includingvaluetypes).Theorderinwhichattributesappearinsideobjectsisnotimportantforhashing.Thehashvaluereturnedbythisfunctionisanumber.ThehashalgorithmisnotguaranteedtoremainthesameinfutureversionsofArangoDB.Thehashvaluesshouldthereforebeusedonlyfortemporarycalculations,e.g.tocompareiftwodocumentsarethesame,orforgroupingvaluesinqueries.

TYPENAME(value):Returnsthedatatypenameofvalue.Thedatatypenamecanbeeithernull,bool,number,string,arrayorobject.

LOG(value):Returnsthenaturallogarithmofvalue.ThebaseisEuler'sconstant(2.71828...).

LOG2(value):Returnsthebase2logarithmofvalue.

LOG10(value):Returnsthebase10logarithmofvalue.

EXP(value):ReturnsEuler'sconstant(2.71828...)raisedtothepowerofvalue.

EXP2(value):Returns2raisedtothepowerofvalue.

SIN(value):Returnsthesineofvalue.

COS(value):Returnsthecosineofvalue.

TAN(value):Returnsthetangentofvalue.

ASIN(value):Returnsthearcsineofvalue.

ACOS(value):Returnsthearccosineofvalue.

ATAN(value):Returnsthearctangentofvalue.

ATAN2(y,x):Returnsthearctangentofthequotientofyandx.

RADIANS(value):Returnstheangleconvertedfromdegreestoradians.

DEGREES(value):Returnstheangleconvertedfromradianstodegrees.

Optimizerimprovements

"inline-subqueries"rule

TheAQLoptimizerrule"inline-subqueries"hasbeenadded.ThisrulecanpulloutcertainsubqueriesthatareusedasanoperandtoaFORlooponelevelhigher,eliminatingthesubquerycompletely.Thisreducescomplexityofthequery'sexecutionplanandwilllikelyenablefurtheroptimizations.Forexample,thequery

FORiIN(

FORjIN[1,2,3]

RETURNj

)

RETURNi

willbetransformedbytheruleto:

FORiIN[1,2,3]

RETURNi

Thequery

FORnameIN(

FORdocIN_users

FILTERdoc.status==1

RETURNdoc.name

)

LIMIT2

WhatsNewin3.0

537

RETURNname

willbetransformedinto

FORtmpIN_users

FILTERtmp.status==1

LIMIT2

RETURNtmp.name

TherulewillonlyfirewhenthesubqueryisusedasanoperandtoaFORloop,andifthesubquerydoesnotcontainaCOLLECTwithanINTOvariable.

"remove-unnecessary-calculations"rule

TheAQLoptimizerrule"remove-unnecessary-calculations"nowfiresinmorecasesthaninpreviousversions.Thisruleremovescalculationsfromexecutionplans,andbyhavinglesscalculationsdone,aquerymayexecutefasterorrequireslessmemory.

Therulewillnowremovecalculationsthatareusedexactlyonceinotherexpressions(e.g.LETa=docRETURNa.value)andcalculations,orcalculationsthatarejustreferencestoothervariables(e.g.LETa=b).

"optimize-traversals"rule

TheAQLoptimizerrule"merge-traversal-filter"wasrenamedto"optimize-traversals".TherulewillremoveunusededgeandpathresultvariablesfromthetraversalincasetheyarespecifiedintheFORsectionofthetraversal,butnotreferencedlaterinthequery.Thissavesconstructingedgesandpathsresultsthatarenotusedlater.

AQLnowusesVelocyPackinternallyforstoringintermediatevalues.Formanyvaluetypesitcannowgetawaywithoutextramemoryallocationsandlessinternalconversions.ValuescanbepassedintointernalAQLfunctionswithoutcopyingthem.ThiscanleadtoreducedqueryexecutiontimesforqueriesthatuseC++-basedAQLfunctions.

"replace-or-with-in"and"use-index-for-sort"rules

Theserulesnowfireinsomeadditionalcases,whichallowssimplifyingindexlookupconditionsandremovingSortNodesfromexecutionplans.

Clusterstatemanagement

Thecluster'sinternalstateinformationisnowalsomanagedbyArangoDBinstances.Earlierversionsreliedonthirdpartysoftwarebeinginstalledforthestoringtheclusterstate.ThestateismanagedbydedicatedArangoDBinstances,whichcanbestartedinaspecialagencymode.Theseinstancescanoperateinadistributedfashion.Theywillautomaticallyelectoneofthemtobecometheirleader,beingresponsibileforstoringthestatechangessentfromserversinthecluster.Theotherinstanceswillautomaticallyfollowtheleaderandwilltransparentlystandinshoulditbecomeunavailable.Theagencyinstancesarealsoself-organizing:theywillcontinuouslyprobeeachotherandre-electleaders.Thecommunicationbetweentheagencyinstancesusetheconsensus-basedRAFTprotocol.

TheoperationsforstoringandretrievingclusterstateinformationarenowmuchlessexpensivefromanArangoDBclusternodeperspective,whichinturnallowsforfasterclusteroperationsthatneedtofetchorupdatetheoverallclusterstate.

_fromand _toattributesofedgesareupdatableandusableinindexes

InArangoDBpriorto3.0theattributes_fromand_toofedgesweretreatedspeciallywhenloadingorstoringedges.Thatspecialhandlingledtotheseattributesbeingnotasflexibleasregulardocumentattributes.Forexample,the_fromand_toattributevaluesofanexistingedgecouldnotbeupdatedoncetheedgewascreated.Nowthisispossibleviathesingle-documentAPIsandviaAQL.

Additionally,the_fromand_toattributescouldnotbeindexedinuser-definedindexes,e.g.tomakeeachcombinationof_fromand_tounique.Finally,as_fromand_toreferencedthelinkedcollectionsbycollectionidandnotbycollectionname,theirmeaningbecameunclearonceareferencedcollectionwasdropped.Thecollectionidstoredinedgesthenbecameunusable,andwhenaccessing

WhatsNewin3.0

538

suchedgethecollectionnamepartofitwasalwaystranslatedto_undefined.

InArangoDB3.0,the_fromand_tovaluesofedgesaresavedasregularstrings.Thisallowsusing_fromand_toinuser-definedindexes.Additionally,thisallowstoupdatethe_fromand_tovaluesofexistingedges.Furthermore,collectionsreferencedby_fromand_tovaluesmaybedroppedandre-createdlater.Any_fromand_tovaluesofedgespointingtosuchdroppedcollectionareunaffectedbythedropoperationnow.

UnifiedAPIsforCRUDoperations

TheCRUDAPIsfordocumentsandedgehavebeenunified.EdgescannowbeinsertedandmodifiedviathesameAPIsasdocuments._fromand_toattributevaluescanbepassedasregulardocumentattributesnow:

db.myedges.insert({_from:"myvertices/some",_to:"myvertices/other",...});

Passing_fromand_toseparatelyasitwasrequiredinearlierversionsisnotnecessaryanymorebutwillstillwork:

db.myedges.insert("myvertices/some","myvertices/other",{...});

TheCRUDoperationsnowalsosupportbatchvariantsthatworksonarraysofdocuments/edges,e.g.

db.myedges.insert([

{_from:"myvertices/some",_to:"myvertices/other",...},

{_from:"myvertices/who",_to:"myvertices/friend",...},

{_from:"myvertices/one",_to:"myvertices/two",...},

]);

ThebatchvariantsarealsoavailableinArangoDB'sHTTPAPI.Theycanbeusedtomoreefficientlycarryoutoperationswithmultipledocumentsthantheirsingle-documentequivalents,whichrequiredoneHTTPrequestperoperation.Withthebatchoperations,theHTTPrequest/responseoverheadcanbeamortizedacrossmultipleoperations.

Persistentindexes

ArangoDB3.0providesanexperimentalpersistentindexfeature.Persistentindexesstoretheindexvaluesondiskinsteadofin-memoryonly.Thismeanstheindexesdonotneedtoberebuiltin-memorywhenacollectionisloadedorreloaded,whichshouldimprovecollectionloadingtimes.

ThepersistentindexesinArangoDBarebasedontheRocksDBengine.Tocreateapersistentindexforacollection,createanindexoftype"rocksdb"asfollows:

db.mycollection.ensureIndex({type:"rocksdb",fields:["fieldname"]});

Thepersistentindexesaresorted,sotheyallowequalitylookupsandrangequeries.Notethatthefeatureisstillhighlyexperimentalandhassomeknowndeficiencies.Itwillbefinalizeduntilthereleaseofthe3.0stableversion.

UpgradedV8version

TheV8enginethatisusedinsideArangoDBtoexecuteJavaScriptcodehasbeenupgradedfromversion4.3.61to5.0.71.39.ThenewversionmakesseveralmoreES6featuresavailablebydefault,including

arrowfunctionscomputedpropertynamesrestparametersarraydestructuringnumericandobjectliterals

WhatsNewin3.0

539

WebAdminInterface

TheArangoDB3.0webinterfaceissignificantlyimproved.Itnowcomeswithamoreresponsivedesign,makingiteasiertouseondifferentdevices.Navigationandmenushavebeensimplified,andrelateditemshavebeenregroupedtostayclosertogetherandallowtighterworkflows.

TheAQLqueryeditorisnowmucheasiertouse.Multiplequeriescanbestartedandtrackedinparallel,whileresultsofearlierqueriesarestillpreserved.Queriesstillrunningcanbecanceleddirectlyfromtheeditor.TheAQLqueryeditornowallowstheusageofbindparameterstoo,andprovidesahelperforfindingcollectionnames,AQLfunctionnamesandkeywordsquickly.

Thewebinterfacenowkeepstrackofwhethertheserverisofflineandofwhichserver-sideoperationshavebeenstartedandarestillrunning.Itnowremainsusablewhilesuchlonger-runningoperationsareongoing.Italsokeepsmorestateaboutuser'schoices(e.g.windowssizes,whetherthetreeorthecodeviewwaslastusedinthedocumenteditor).

Clusterstatisticsarenowintegratedintothewebinterfaceaswell.Additionally,amenuitem"Helpus"hasbeenaddedtoeasilyprovidetheArangoDBteamfeedbackabouttheproduct.

Thefrontendmaynowbemountedbehindareverseproxyonadifferentpath.ForthistoworktheproxyshouldsendaX-Script-Nameheadercontainingthepath.

Abackendconfigurationforhaproxymightlooklikethis:

reqaddX-Script-Name:\/arangodb

Thefrontendwillrecognizethesubpathandproduceappropriatelinks.ArangoDBwillonlyacceptpathsfromtrustedfrontendproxies.Trustedproxiesmaybeaddedonstartup:

--frontend.proxy-request-checktrue--frontend.trusted-proxy192.168.1.117

--frontend.trusted-proxymaybeanyaddressornetmask.

Todisablethecheckandblindlyacceptanyx-script-nameset--frontend.proxy-request-checktofalse.

Foxximprovements

TheFoxxframeworkhasbeencompletelyrewrittenfor3.0withanew,simplerandmorefamiliarAPI.Themostnotablechangesare:

Legacymodefor2.8services

Stuckwitholdcode?Youcancontinueusingyour2.8-compatibleFoxxserviceswith3.0byadding"engines":{"arangodb":"^2.8.0"}(orsimilarversionrangesthatexclude3.0andup)totheservicemanifest.

Nomoreglobalvariablesandmagicalcomments

TheapplicationContextisnowmodule.context.Insteadofmagicalcommentsjustusethesummaryanddescriptionmethodstodocumentyourroutes.

RepositoryandModelhavebeenremoved

InsteadofrepositoriesjustuseArangoDBcollectionsdirectly.Forvalidationsimplyusethejoischemas(butwrappedinjoi.object())thatpreviouslylivedinsidethemodel.CollectionsandqueriesreturnplainJavaScriptobjects.

Controllershavebeenreplacedwithnestablerouters

Createrouterswithrequire('@arangodb/foxx/router')(),attachthemtoyourservicewithmodule.context.use(router).Becauseroutersarenolongermountedautomagically,youcanexportandimportthemlikeanyotherobject.Userouter.use('/path',subRouter)tonestroutersasdeeplyasyouwant.

Routescanbenamedandreversed

NomorememorizingURLs:addanametoyourroutelikerouter.get('/hello/:name',function(){...},'hello')andredirecttothefullURLwithres.redirect(req.resolve('hello',{name:'world'})).

WhatsNewin3.0

540

Simplerexpress-likemiddleware

Ifyoualreadyknowexpress,thisshouldbefamiliar.Here'sarequestloggerinthreelinesofcode:

router.use(function(req,res,next){

varstart=Date.now();

try{next();}

finally{console.log(`${req.method}${req.url}${res.statusCode}${Date.now()-start}ms`);}

});

Sessionsandauthwithoutdependencies

Tomakeiteasiertogetstarted,thefunctionalitypreviouslyprovidedbythesimple-auth,oauth2,sessions-localandsessions-jwtserviceshavebeenmovedintoFoxxasthe@arangodb/foxx/auth,@arangodb/foxx/oauth2and@arangodb/foxx/sessionsmodules.

Logging

ArangoDB'sloggingisnowgroupedintotopics.Thelogverbosityandoutputfilescanbeadjustedperlogtopic.Forexample

--log.levelstartup=trace--log.levelqueries=trace--log.levelinfo

willlogmessagesconcerningstartupattracelevel,AQLqueriesattracelevelandeverythingelseatinfolevel.--log.levelcanbespecifiedmultipletimesatstartup,forasmanytopicsasneeded.

Somerelevantlogtopicsavailablein3.0are:

collector:informationabouttheWALcollector'sstatecompactor:informationaboutthecollectiondatafilecompactordatafiles:datafile-relatedoperationsmmap:informationaboutmemory-mappingoperations(includingmsync)queries:executedAQLqueries,slowqueriesreplication:replication-relatedinforequests:HTTPrequestsstartup:informationaboutserverstartupandshutdownthreads:informationaboutthreads

Thisalsoallowsdirectinglogoutputtodifferentfilesbasedontopics.Forexample,tologallAQLqueriestoafile"queries.log"onecanusetheoptions:

--log.levelqueries=trace--log.outputqueries=file:///path/to/queries.log

ToadditionallylogHTTPrequesttoafilenamed"requests.log"addtheoptions:

--log.levelrequests=info--log.outputrequests=file:///path/to/requests.log

Buildsystem

ArangoDBnowusesthecross-platformbuildsystemCMakeforallitsbuilds.Previousversionsusedtwodifferentbuildsystems,makingdevelopmentandcontributionsharderthannecessary.Nowthebuildsystemisunified,andalltargets(Linux,Windows,MacOS)arebuiltfromthesamesetofbuildinstructions.

Documentation

Thedocumentationhasbeenenhancedandre-organizedtobemoreintuitive.

WhatsNewin3.0

541

AnewintroductionforbeginnersshouldbringyouuptospeedwithArangoDBinlessthananhour.Additionaltopicshavebeenintroducedandwillbeextendedwithupcomingreleases.

ThetopicsAQLandHTTPAPIarenowseparatedfromthemanualforbettersearchabilityandlessconfusion.Aversionswitchermakesiteasiertojumptotheversionofthedocsyouareinterestedin.

WhatsNewin3.0

542

IncompatiblechangesinArangoDB3.0ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB3.0,andadjustanyclientprogramsifnecessary.

Buildsystem

BuildingArangoDB3.0fromsourcenowrequiresCMake.

Thepre-3.0buildsystemusedaconfigure-basedapproach.ThestepstobuildArangoDB2.8fromsourcecodewere:

makesetup

./configure<options>

make

Thesestepswillnotworkanymore,asArangoDB3.0doesnotcomewithaconfigurescript.

Tobuild3.0onLinux,createaseparatebuilddirectoryfirst:

mkdir-pbuild

andthencreatetheinitialbuildscriptsonceusingCMake:

(cdbuild&&cmake<options>..)

Theabovecommandwillconfigurethebuildandcheckfortherequireddependencies.Ifeverythingworkswelltheactualbuildcanbestartedwith

(cdbuild&&make)

ThebinariesfortheArangoDBserverandallclienttoolswillthenbecreatedinsidethebuilddirectory.TostartArangoDBlocallyfromthebuilddirectory,use

build/bin/arangod<options>

Datafilesanddatafilenames

ArangoDB3.0usesanewVelocyPack-basedformatforstoringdatainWALlogfilesandcollectiondatafiles.ThefileformatisnotcompatiblewiththefilesusedinpriorversionsofArangoDB.ThatmeansdatafileswrittenbyArangoDB3.0cannotbeusedinearlierversionsandviceversa.

Thepatternforcollectiondirectorynameswaschangedin3.0toincludearandomidcomponentattheend.Thenewpatterniscollection-<id>-<random>,where<id>isthecollectionidand<random>isarandomnumber.PreviousversionsofArangoDBusedapatterncollection-<id>withouttherandomnumber.

UserManagementUnlikeArangoDB2.x,ArangoDB3.0usersarenowseparatedfromdatabases,andyoucangrantoneormoredatabasepermissionstoauser.

IfyouwanttomimicthebehaviorofArangoDB,youshouldnameyouruserslikeusername@dbname.

Usersthatcanaccessthe_systemdatabaseareallowedtomanageusersandpermissionsforalldatabases.

Incompatiblechangesin3.0

543

Edgesandedgesattributes

InArangoDBpriorto3.0theattributes_fromand_toofedgesweretreatedspeciallywhenloadingorstoringedges.Thatspecialhandlingledtotheseattributesbeingnotasflexibleasregulardocumentattributes.Forexample,the_fromand_toattributevaluesofanexistingedgecouldnotbeupdatedoncetheedgewascreated.Additionally,the_fromand_toattributescouldnotbeindexedinuser-definedindexes,e.g.tomakeeachcombinationof_fromand_tounique.Finally,as_fromand_toreferencedthelinkedcollectionsbycollectionidandnotbycollectionname,theirmeaningbecameunclearonceareferencedcollectionwasdropped.Thecollectionidstoredinedgesthenbecameunusable,andwhenaccessingsuchedgethecollectionnamepartofitwasalwaystranslatedto_undefined.

InArangoDB3.0,the_fromand_tovaluesofedgesaresavedasregularstrings.Thisallowsusing_fromand_toinuser-definedindexes.Additionallythisallowsupdatingthe_fromand_tovaluesofexistingedges.Furthermore,collectionsreferencedby_fromand_tovaluesmaybedroppedandre-createdlater.Any_fromand_tovaluesofedgespointingtosuchdroppedcollectionareunaffectedbythedropoperationnow.Alsonotethatrenamingthecollectionreferencedin_fromand_toinArangoDB2.8alsorelinkedtheedges.In3.0theedgesareNOTautomaticallyrelinkedtothenewcollectionanymore.

DocumentsDocuments(incontrasttoedges)cannotcontaintheattributes_fromor_toonthemainlevelinArangoDB3.0.Theseattributeswillbeautomaticallyremovedwhensavingdocuments(i.e.non-edges)._fromand_tocanbestillusedinsub-objectsinsidedocuments.

The_fromand_toattributeswillofcoursebepreservedandarestillrequiredwhensavingedges.

AQL

Edgeshandling

WhenupdatingorreplacingedgesviaAQL,anymodificationstothe_fromand_toattributesofedgeswereignoredbypreviousversionsofArangoDB,withoutsignalinganyerrors.Thiswasduetothe_fromand_toattributesbeingimmutableinearlierversionsofArangoDB.

From3.0on,the_fromand_toattributesofedgesaremutable,soanyAQLqueriesthatmodifythe_fromor_toattributevaluesofedgeswillattempttoactuallychangetheseattributes.Clientsshouldbeawareofthischangeandshouldreviewtheirqueriesthatmodifyedgestoruleoutunintendedside-effects.

Additionally,whencompletelyreplacingthedataofexistingedgesviatheAQLREPLACEoperation,itisnowrequiredtospecifyvaluesforthe_fromand_toattributes,asREPLACErequirestheentirenewdocumenttobespecified.Ifeither_fromor_toaremissingfromthereplacementdocument,anREPLACEoperationwillfail.

Graphfunctions

Inversion3.0allformergraphrelatedfunctionshavebeenremovedfromAQLtobereplacedbynativeAQLconstructs.Theseconstructsallowformorefine-grainedfilteringonseveralgraphlevels.AlsothisallowstheAQLoptimizertoautomaticallyimprovethesequeriesbyenhancingthemwithappropriateindexes.Wehavecreatedrecipestoupgradefrom2.8to3.0whenusingthesefunctions.

Thefunctions:

GRAPH_COMMON_NEIGHBORSGRAPH_COMMON_PROPERTIESGRAPH_DISTANCE_TOGRAPH_EDGESGRAPH_NEIGHBORSGRAPH_TRAVERSALGRAPH_TRAVERSAL_TREEGRAPH_SHORTEST_PATHGRAPH_PATHSGRAPH_VERTICES

Incompatiblechangesin3.0

544

arecoveredinMigratingGRAPH_*Functionsfrom2.8orearlierto3.0

GRAPH_ABSOLUTE_BETWEENNESSGRAPH_ABSOLUTE_CLOSENESSGRAPH_ABSOLUTE_ECCENTRICITYGRAPH_BETWEENNESSGRAPH_CLOSENESSGRAPH_DIAMETERGRAPH_ECCENTRICITYGRAPH_RADIUS

arecoveredinMigratingGRAPH_*Measurementsfrom2.8orearlierto3.0

EDGESNEIGHBORSPATHSTRAVERSALTRAVERSAL_TREE

arecoveredinMigratinganonymousgraphfunctionsfrom2.8orearlierto3.0

Typecastingfunctions

ThetypecastingappliedbytheTO_NUMBER()AQLfunctionhaschangedasfollows:

stringvaluesthatdonotcontainavalidnumericvaluearenowconvertedtothenumber0.InpreviousversionsofArangoDBsuchstringvalueswereconvertedtothevaluenull.arrayvalueswithmorethan1memberarenowconvertedtothenumber0.InpreviousversionsofArangoDBsucharrayswereconvertedtothevaluenull.objects/documentsarenowconvertedtothenumber0.InpreviousversionsofArangoDBobjects/documentswereconvertedtothevaluenull.

Additionally,theTO_STRING()AQLfunctionnowconvertsnullvaluesintoanemptystring("")insteadofthestring"null",whichismoreinlinewithLENGTH(null)returning0andnot4sincev2.6.

TheoutputofTO_STRING()hasalsochangedforarraysandobjectsasfollows:

arraysarenowconvertedintotheirJSON-stringifyequivalents,e.g.

[]isnowconvertedto[][1,2,3]isnowconvertedto[1,2,3]["test",1,2]isnowconvertedto["test",1,2]`

PreviousversionsofArangoDBconvertedarrayswithnomembersintotheemptystring,andnon-emptyarraysintoacomma-separatedlistofmembervalues,withoutthesurroundingangularbrackets.Additionally,stringarraymemberswerenotenclosedinquotesintheresultstring:

[]wasconvertedto``[1,2,3]wasconvertedto1,2,3["test",1,2]wasconvertedtotest,1,2`

objectsarenowconvertedtotheirJSON-stringifyequivalents,e.g.

{}isconvertedto{}{a:1,b:2}isconvertedto{"a":1,"b":2}{"test":"foobar"}isconvertedto{"test":"foobar"}

PreviousversionsofArangoDBalwaysconvertedobjectsintothestring[objectObject]

ThischangealsoaffectsotherpartsinAQLthatusedTO_STRING()toimplicitlycastoperandstostrings.ItalsoaffectstheAQLfunctionsCONCAT()andCONCAT_SEPARATOR()whichtreatedarrayvaluesdifferently.PreviousversionsofArangoDBautomaticallyflattenedarrayvaluesinthefirstlevelofthearray,e.g.CONCAT([1,2,3,[4,5,6]])produced1,2,3,4,5,6.Nowthiswillproduce[1,2,3,[4,5,6]].Toflattenarraymembersonthetoplevel,youcannowusethemoreexplicitCONCAT(FLATTEN([1,2,3,[4,5,6]],1)).

Incompatiblechangesin3.0

545

Arithmeticoperators

AsthearithmeticoperationsinAQLimplicitlyconverttheiroperandstonumericvaluesusingTO_NUMBER(),theircastingbehaviorhasalsochangedasdescribedabove.

Someexamplesofthechangedbehavior:

"foo"+1produces1now.Inpreviousversionsthisproducednull.[1,2]+1produces1.Inpreviousversionsthisproducednull.1+"foo"+1´produces2now.Inpreviousversionthisproduced1`.

Attributenamesandparameters

PreviousversionsofArangoDBhadsometroublewithattributenamesthatcontainedthedotsymbol(.).SomecodepartsinAQLusedthedotsymboltosplitanattributenameintosub-components,soanattributenameda.bwasnotcompletelydistinguishablefromanattributeawithasub-attributeb.Thisinconsistentbehaviorsometimesallowed"hacks"toworksuchaspassingsub-attributesinabindparameterasfollows:

FORdocINcollection

FILTERdoc.@name==1

RETURNdoc

Ifthebindparameter@namecontainedthedotsymbol(e.g.@bind=a.b,itwasunclearwhetherthisshouldtriggersub-attributeaccess(i.e.doc.a.b)oraaccesstoanattributewithexactlythespecifiedname(i.e.doc["a.b"]).

ArangoDB3.0nowhandlesattributenamescontainingthedotsymbolproperly,andsendingabindparameter@name=a.bwillnowalwaystriggeranaccesstotheattributedoc["a.b"],notthesub-attributebofaindoc.

Forusersthatusedthe"hack"ofpassingbindparameterscontainingdotsymboltoaccesssub-attributes,ArangoDB3.0allowsspecifyingtheattributenamepartsasanarrayofstrings,e.g.@name=["a","b"],whichwillberesolvedtothesub-attributeaccessdoc.a.bwhenthequeryisexecuted.

Keywords

LIKEisnowakeywordinAQL.UsingLIKEineithercaseasanattributeorcollectionnameinAQLqueriesnowrequiresquoting.

SHORTEST_PATHisnowakeywordinAQL.UsingSHORTEST_PATHineithercaseasanattributeorcollectionnameinAQLqueriesnowrequiresquoting.

Subqueries

Queriesthatcontainsubqueriesthatcontaindata-modificationoperationssuchasINSERT,UPDATE,REPLACE,UPSERTorREMOVEwillnowrefusetoexecuteifthecollectionaffectedbythesubquery'sdata-modificationoperationisread-accessedinanouterscopeofthequery.

Forexample,thefollowingquerywillrefusetoexecuteasthecollectionmyCollectionismodifiedinthesubquerybutalsoread-accessedintheouterscope:

FORdocINmyCollection

LETchanges=(

FORwhatINmyCollection

FILTERwhat.value==1

REMOVEwhatINmyCollection

)

RETURNdoc

Itisstillpossibletowritetocollectionsfromwhichdataisreadinthesamequery,e.g.

FORdocINmyCollection

FILTERdoc.value==1

REMOVEdocINmyCollection

Incompatiblechangesin3.0

546

andtomodifydataindifferentcollectionviasubqueries.

Otherchanges

TheAQLoptimizerrule"merge-traversal-filter"thatalreadyexistedin3.0wasrenamedto"optimize-traversals".ThisshouldbeofnorelevancetoclientapplicationsexceptiftheyprogramaticallylookforappliedoptimizerrulesintheexplainoutofAQLqueries.

TheorderofresultscreatedbytheAQLfunctionsVALUES()andATTRIBUTES()wasneverguaranteedanditonlyhadthe"correct"orderingbyaccidentwheniteratingoverobjectsthatwerenotloadedfromthedatabase.Assomeofthefunctioninternalshavechanged,the"correct"orderingwillnotappearanymore,andstillnoresultorderisguaranteedbythesefunctionsunlessthesortparameterisspecified(fortheATTRIBUTES()function).

UpgradedV8version

TheV8enginethatisusedinsideArangoDBtoexecuteJavaScriptcodehasbeenupgradedfromversion4.3.61to5.0.71.39.Thenewversionshouldbemostlycompatibletotheoldversion,buttheremaybesubtledifferences,includingchangesoferrormessagetextsthrownbytheengine.Furthermore,someV8startupparametershavechangedtheirmeaningorhavebeenremovedinthenewversion.ThisisonlyrelevantwhenArangoDBorArangoShellarestartedwithacustomvalueforthe--javascript.v8-optionsstartupoption.

Amongothers,thefollowingV8optionschangeinthenewversionofArangoDB:

--es_staging:in2.8ithadthemeaningenableallcompletedharmonyfeatures,in3.0theoptionmeansenabletest-worthyharmonyfeatures(forinternaluseonly)

--strong_this:thisoptionwasn'tpresentin2.8.In3.0itmeansdon'tallow'this'toescapefromconstructorsanddefaultstotrue.

--harmony_regexps:thisoptionsmeansenable"harmonyregularexpressionextensions"andchangesitsdefaultvaluefromfalsetotrue

--harmony_proxies:thisoptionsmeansenable"harmonyproxies"andchangesitsdefaultvaluefromfalsetotrue

--harmony_reflect:thisoptionsmeansenable"harmonyReflectAPI"andchangesitsdefaultvaluefromfalsetotrue

--harmony_sloppy:thisoptionsmeansenable"harmonyfeaturesinsloppymode"andchangesitsdefaultvaluefromfalsetotrue

--harmony_tostring:thisoptionsmeansenable"harmonytoString"andchangesitsdefaultvaluefromfalsetotrue

--harmony_unicode_regexps:thisoptionsmeansenable"harmonyunicoderegexps"andchangesitsdefaultvaluefromfalsetotrue

--harmony_arrays,--harmony_array_includes,--harmony_computed_property_names,--harmony_arrow_functions,--harmony_rest_parameters,--harmony_classes,--harmony_object_literals,--harmony_numeric_literals,--harmony_unicode:theseoptionhavebeenremovedinV85.

AsaconsequenceoftheupgradetoV8version5,theimplementationoftheJavaScriptBufferobjecthadtobechanged.JavaScriptBufferobjectsinArangoDBnowalwaysstoretheirdataontheheap.ThereisnosharedpoolforsmallBuffervalues,andnopointingintoexistingBufferdatawhenextractingslices.ThischangemayincreasethecostofcreatingBufferswithshortcontentsorwhenpeekingintoexistingBuffers,butwasrequiredforsafermemorymanagementandtopreventleaks.

JavaScriptAPIchanges

ThefollowingincompatiblechangeshavebeenmadetotheJavaScriptAPIinArangoDB3.0:

Foxx

TheFoxxframeworkhasbeencompletelyrewrittenfor3.0withanew,simplerandmorefamiliarAPI.TomakeFoxxservicesdevelopedfor2.8orearlierArangoDBversionsrunin3.0,theservice'smanifestfileneedstobeedited.

ToenablethelegacymodeforaFoxxservice,add"engines":{"arangodb":"^2.8.0"}(orsimilarversionrangesthatexclude3.0andup)totheservicemanifestfile(named"manifest.json",locatedintheservice'sbasedirectory).

Incompatiblechangesin3.0

547

Require

ModulesshippedwithArangoDBcannowberequiredusingthepattern@arangodb/<module>insteadoforg/arangodb/<module>,e.g.

varcluster=require("@arangodb/cluster");

Theoldformatcanstillbeusedforcompatibility:

varcluster=require("org/arangodb/cluster");

ArangoDBpriortoversion3.0allowedatransparentuseofCoffeeScriptsourcefileswiththerequire()function.FileswithafilenameextensionofcoffeewereautomaticallysentthroughaCoffeeScriptparserandtranspiledintoJavaScripton-the-fly.ThissupportisgonewithArangoDB3.0.TorunanyCoffeeScriptsourcefiles,theymustbeconvertedtoJavaScriptbytheclientapplication.

Responseobject

The@arangodb/requestresponseobjectnowstorestheparsedJSONresponsebodyinapropertyjsoninsteadofbodywhentherequestwasmadeusingthejsonoption.Thebodyinsteadcontainstheresponsebodyasastring.

EdgesAPI

Whencompletelyreplacinganedgeviaacollection'sreplace()functionthereplacingedgedatanowneedstocontainthe_fromand_toattributesforthenewedge.PreviousversionsofArangoDBdidnotrequiretheedgedatatocontain_fromand_toattributeswhenreplacinganedge,since_fromand_tovalueswereimmutableforexistingedges.

Forexample,thefollowingcallworkedinArangoDB2.8butwillfailin3.0:

db.edgeCollection.replace("myKey",{value:"test"});

TomakethisworkinArangoDB3.0,_fromand_toneedtobeaddedtothereplacementdata:

db.edgeCollection.replace("myKey",{_from:"myVertexCollection/1",_to:"myVertexCollection/2",value:"test"});

Notethatthisonlyaffectsthereplace()functionbutnotupdate(),whichwillonlyupdatethespecifiedattributesoftheedgeandleaveallothersintact.

Additionally,thefunctionsedges(),outEdges()andinEdges()withanarrayofedgeidswillnowmaketheedgeidsuniquebeforereturningtheconnectededges.Thisisprobablydesiredanyway,asresultswillbereturnedonlyonceperdistinctinputedgeid.However,itmaybreakclientapplicationsthatrelyontheoldbehavior.

DatabasesAPI

The_listDatabases()functionofthedbobjecthasbeenrenamedto_databases(),makingitconsistentwiththe_collections()function.Alsothe_listEndpoints()functionhasbeenrenamedto_endpoints().

CollectionAPI

Examplematching

ThecollectionfunctionbyExampleHash()andbyExampleSkiplist()havebeenremovedin3.0.Theirfunctionalityisprovidedbycollection'sbyExample()function,whichwillautomaticallyuseasuitableindexifpresent.

ThecollectionfunctionbyConditionSkiplist()hasbeenremovedin3.0.ThesamefunctionalitycanbeachievedbyissuinganAQLquerywiththetargetcondition,whichwillautomaticallyuseasuitableindexifpresent.

Revisionidhandling

Incompatiblechangesin3.0

548

Theexists()methodofacollectionnowthrowsanexceptionwhenthespecifieddocumentexistsbutitsrevisioniddoesnotmatchtherevisionidspecified.PreviousversionsofArangoDBsimplyreturnedfalseifeithernodocumentexistedwiththespecifiedkeyorwhentherevisioniddidnotmatch.Itwasthereforeimpossibletodistinguishthesetwocasesfromthereturnvaluealone.3.0correctsthis.Additionally,exists()inpreviousversionsalwaysreturnedabooleanifonlythedocumentkeywasgiven.3.0nowreturnsthedocument'smeta-data,whichincludesthedocument'scurrentrevisionid.

GiventhereisadocumentwithkeytestincollectionmyCollection,thenthebehaviorof3.0isasfollows:

/*testifdocumentexists.thisreturnedtruein2.8*/

db.myCollection.exists("test");

{

"_key":"test",

"_id":"myCollection/test",

"_rev":"9758059"

}

/*testifdocumentexists.thisreturnedtruein2.8*/

db.myCollection.exists({_key:"test"});

{

"_key":"test",

"_id":"myCollection/test",

"_rev":"9758059"

}

/*testifdocumentexists.thisalsoreturnedfalsein2.8*/

db.myCollection.exists("foo");

false

/*testifdocumentwithagivenrevisionidexists.thisreturnedtruein2.8*/

db.myCollection.exists({_key:"test",_rev:"9758059"});

{

"_key":"test",

"_id":"myCollection/test",

"_rev":"9758059"

}

/*testifdocumentwithagivenrevisionidexists.thisreturnedfalsein2.8*/

db.myCollection.exists({_key:"test",_rev:"1234"});

JavaScriptexception:ArangoError1200:conflict

Capconstraints

Thecapconstraintsfeaturehasbeenremoved.Thischangehasledtotheremovalofthecollectionoperationsfirst()andlast(),whichwereinternallybasedondatafromcapconstraints.

AscapconstraintshavebeenremovedinArangoDB3.0itisnotpossibletocreateanindexoftype"cap"withacollection'sensureIndex()function.ThededicatedfunctionensureCapConstraint()hasalsobeenremovedfromthecollectionAPI.

GraphBlueprintsJSModule

Thedeprecatedmodulegraph-blueprintshasbeendeleted.Allit'sfeaturesarecoveredbythegeneral-graphmodule.

GeneralGraphFluentAQLinterface

ThefluentinterfacehasbeenremovedfromArangoDB.It'sfeatureswerecompletelyoverlappingwith"aqb"whichcomespreinstalledaswell.PleaseswitchtoAQBinstead.

UndocumentedAPIs

TheundocumentedfunctionsBY_EXAMPLE_HASH()andBY_EXAMPLE_SKIPLIST(),BY_CONDITION_SKIPLIST,CPP_NEIGHBORSandCPP_SHORTEST_PATHhavebeenremoved.ThesefunctionswerealwayshiddenandnotintendedtobepartofthepublicJavaScriptAPIforcollections.

HTTPAPIchanges

Incompatiblechangesin3.0

549

CRUDoperations

ThefollowingincompatiblechangeshavebeenmadetotheHTTPAPIinArangoDB3.0:

General

TheHTTPinsertoperationsforsingledocumentsandedges(POST/_api/document)donotsupporttheURLparameter"createCollection"anymore.InpreviousversionsofArangoDBthisparametercouldbeusedtoautomaticallycreateacollectionuponinsertionofthefirstdocument.ItisnowrequiredthatthetargetcollectionalreadyexistswhenusingthisAPI,otherwiseitwillreturnanHTTP404error.ThesameistruefortheimportAPIatPOST/_api/import.

CollectionscanstillbecreatedeasilyviaaseparatecalltoPOST/_api/collectionasbefore.

The"location"HTTPheaderreturnedbyArangoDBwheninsertinganewdocumentoredgenowalwayscontainsthedatabasename.ThiswasalsothedefaultbehaviorinpreviousversionsofArangoDB,butitcouldbeoverriddenbyclientssendingtheHTTPheaderx-arango-version:1.4intherequest.ClientscancontinuetosendthisheadertoArangoDB3.0,buttheheaderwillnotinfluencethelocationresponseheadersproducedbyArangoDB3.0anymore.

AdditionallytheCRUDoperationsAPIsdonotreturnanattribute"error"intheresponsebodywithanattributevalueof"false"incaseanoperationsucceeded.

Revisionidhandling

Theoperationsforupdating,replacingandremovingdocumentscanoptionallychecktherevisionnumberofthedocumenttobeupdated,replacedorremovedsothecallercanensuretheoperationworksonaspecificversionofthedocumentandtherearenolostupdates.

PreviousversionsofArangoDBallowedpassingtherevisionidofthepreviousdocumenteitherintheHTTPheaderIf-MatchorintheURLparameterrev.Forexample,removingadocumentwithaspecificrevisionidcouldbeachievedasfollows:

curl-XDELETE\

"http://127.0.0.1:8529/_api/document/myCollection/myKey?rev=123"

ArangoDB3.0doesnotsupportpassingtherevisionidviathe"rev"URLparameteranymore.InsteadthepreviousrevisionidmustbepassedintheHTTPheaderIf-Match,e.g.

curl-XDELETE\

--header"If-Match:'123'"\

"http://127.0.0.1:8529/_api/document/myCollection/myKey"

TheURLparameter"policy"wasalsousableinpreviousversionsofArangoDBtocontrolrevisionhandling.Usingitwasredundanttospecifyingtheexpectedrevisionidviathe"rev"parameteror"If-Match"HTTPheaderandthereforesupportforthe"policy"parameterwasremovedin3.0.

Inordertocheckforapreviousrevisionidwhenupdating,replacingorremovingdocumentspleaseusetheIf-MatchHTTPheaderasdescribedabove.WhennorevisioncheckifrequiredtheHTTPheadercanbeomitted,andtheoperationswillworkonthecurrentrevisionofthedocument,regardlessofitsrevisionid.

AlldocumentsAPI

TheHTTPAPIforretrievingtheids,keysorURLsofalldocumentsfromacollectionwaspreviouslylocatedatGET/_api/document?collection=....ThisAPIwasmovedtoPUT/_api/simple/all-keysandisnowexecutedasanAQLquery.ThenameofthecollectionmustnowbepassedintheHTTPrequestbodyinsteadofintherequestURL.Thesameistrueforthe"type"parameter,whichcontrolsthetypeoftheresulttobecreated.

CallstothepreviousAPIcanbetranslatedasfollows:

old:GET/_api/document?collection=<collection>&type=<type>withoutHTTPrequestbody3.0:PUT/_api/simple/all-keyswithHTTPrequestbody{"collection":"<collection>","type":"id"}

TheresultformatofthisAPIhasalsochangedslightly.InpreviousversionscallstotheAPIreturnedaJSONobjectwithadocumentsattribute.AsthefunctionalityisbasedonAQLinternallyin3.0,theAPInowreturnsaJSONobjectwitharesultattribute.

Incompatiblechangesin3.0

550

EdgesAPI

CRUDoperations

TheAPIfordocumentsandedgeshavebeenunifiedinArangoDB3.0.TheCRUDoperationsfordocumentsandedgesarenowhandledbythesameendpointat/_api/document.ForCRUDoperationsthereisnodistinctionanymorebetweendocumentsandedgesAPI-wise.

ThatmeansCRUDoperationsconcerningedgesneedtobesenttotheHTTPendpoint/_api/documentinsteadof/_api/edge.Sendingrequeststo/_api/edgewillresultinanHTTP404errorin3.0.Thefollowingmethodsareavailableat/_api/documentfordocumentsandedge:

HTTPPOST:insertnewdocumentoredgeHTTPGET:fetchanexistingdocumentoredgeHTTPPUT:replaceanexistingdocumentoredgeHTTPPATCH:partiallyupdateanexistingdocumentoredgeHTTPDELETE:removeanexistingdocumentoredge

WhencompletelyreplacinganedgeviaHTTPPUTpleasenotethatthereplacingedgedatanowneedstocontainthe_fromand_toattributesfortheedge.PreviousversionsofArangoDBdidnotrequiresending_fromand_towhenreplacingedges,as_fromand_tovalueswereimmutableforexistingedges.

The_fromand_toattributesofedgesnowalsoneedtobepresentinsidetheedgesobjectssenttotheserver:

curl-XPOST\

--data'{"value":1,"_from":"myVertexCollection/1","_to":"myVertexCollection/2"}'\

"http://127.0.0.1:8529/_api/document?collection=myEdgeCollection"

PreviousversionsofArangoDBrequiredthe_fromand_toattributesofedgesbesentseparatelyinURLparameterfromandto:

curl-XPOST\

--data'{"value":1}'\

"http://127.0.0.1:8529/_api/edge?collection=e&from=myVertexCollection/1&to=myVertexCollection/2"

Queryingconnectededges

TheRESTAPIforqueryingconnectededgesatGET/_api/edges/<collection>willnowmaketheedgeidsuniquebeforereturningtheconnectededges.Thisisprobablydesiredanywayasresultswillnowbereturnedonlyonceperdistinctinputedgeid.However,itmaybreakclientapplicationsthatrelyontheoldbehavior.

GraphAPI

Somedata-modificationoperationsinthenamedgraphsAPIat/_api/gharialnowreturneitherHTTP202(Accepted)orHTTP201(Created)iftheoperationsucceeds.WhichstatuscodeisreturneddependsonthewaitForSyncattributeoftheaffectedcollection.InpreviousversionssomeoftheseoperationsreturnHTTP200regardlessofthewaitForSyncvalue.

ThedeprecatedgraphAPI/_api/graphhasbeenremoved.Allit'sfeaturescanbereplacedusing/_api/gharialandAQLinstead.

SimplequeriesAPI

TheRESTroutesPUT/_api/simple/firstand/_api/simple/lasthavebeenremovedentirely.TheseAPIswereresponsibleforreturningthefirst-insertedandlast-inserteddocumentsinacollection.Thisfeaturewasbuiltoncapconstraintsinternally,whichhavebeenremovedin3.0.

Callingoneoftheseendpointsin3.0willresultinanHTTP404error.

IndexesAPI

Incompatiblechangesin3.0

551

Itisnotsupportedin3.0tocreateanindexwithtypecap(capconstraint)in3.0asthecapconstraintsfeaturehasbeeremoved.CallingtheindexcreationendpointHTTPAPIPOST/_api/index?collection=...withanindextypecapwillthereforeresultinanHTTP400error.

LogentriesAPI

TheRESTrouteHTTPGET/_admin/logisnowaccessiblefromwithinalldatabases.InpreviousversionsofArangoDB,thisroutewasaccessiblefromwithinthe_systemdatabaseonly,andanHTTP403(Forbidden)wasthrownbytheserverforanyaccessfromwithinanotherdatabase.

FiguresAPI

TheRESTrouteHTTPGET/_api/collection/<collection>/figureswillnotreturnthefollowingresultattributesastheybecamemeaninglessin3.0:

shapefiles.countshapes.fileSizeshapes.countshapes.sizeattributes.countattributes.size

DatabasesandCollectionsAPIs

WhencreatingadatabaseviatheAPIPOST/_api/database,ArangoDBwillnowalwaysreturntheHTTPstatuscode202(created)iftheoperationsucceeds.PreviousversionsofArangoDBreturnedHTTP202aswell,butthisbehaviorwaschangablebysendinganHTTPheaderx-arango-version:1.4.Whensendingthisheader,previousversionsofArangoDBreturnedanHTTPstatuscode200(ok).ClientscanstillsendthisheadertoArangoDB3.0butthiswillnotinfluencetheHTTPstatuscodeproducedbyArangoDB.

The"location"headerproducedbyArangoDB3.0willnowalwayscontainthedatabasename.ThiswasalsothedefaultinpreviousversionsofArangoDB,butthebehaviorcouldbeoverriddenbysendingtheHTTPheaderx-arango-version:1.4.Clientscanstillsendtheheader,butthiswillnotmakethedatabasenameinthe"location"responseheaderdisappear.

TheresultformatforqueryingallcollectionsviatheAPIGET/_api/collectionhasbeenchanged.

PreviousversionsofArangoDBreturnedanobjectwithanattributenamedcollectionsandanattributenamednames.Bothcontainedallavailablecollections,butcollectionscontainedthecollectionsasanarray,andnamescontainedthecollectionsagain,containedinanobjectinwhichtheattributenameswerethecollectionnames,e.g.

{

"collections":[

{"id":"5874437","name":"test","isSystem":false,"status":3,"type":2},

{"id":"17343237","name":"something","isSystem":false,"status":3,"type":2},

...

],

"names":{

"test":{"id":"5874437","name":"test","isSystem":false,"status":3,"type":2},

"something":{"id":"17343237","name":"something","isSystem":false,"status":3,"type":2},

...

}

}

Thisresultstructurewasredundant,andthereforehasbeensimplifiedtojust

{

"result":[

{"id":"5874437","name":"test","isSystem":false,"status":3,"type":2},

{"id":"17343237","name":"something","isSystem":false,"status":3,"type":2},

...

]

}

inArangoDB3.0.

Incompatiblechangesin3.0

552

ReplicationAPIs

TheURLparameter"failOnUnknown"wasremovedfromtheRESTAPIGET/_api/replication/dump.Thisparametercontrolledwhetherdumpingorreplicatingedgesshouldfailifoneofthevertexcollectionslinkedintheedge's_fromor_toattributeswasnotpresentanymore.Inthiscasethe_fromand_tovaluescouldnotbetranslatedintomeaningfulidsanymore.

Thereweretwowaysforhandlingthis:

settingfailOnUnknowntotruecausedtheHTTPrequesttofail,leavingerrorhandlingtotheusersettingfailOnUnknowntofalsecausedtheHTTPrequesttocontinue,translatingthecollectionnamepartinthe_fromor_tovalueto_unknown.

InArangoDB3.0thisparameterisobsolete,as_fromand_toarestoredasself-containedstringvaluesallthetime,sotheycannotgetinvalidwhenreferencedcollectionsaredropped.

TheresultformatoftheAPIGET/_api/replication/logger-followhaschangedslightlyinthefollowingaspects:

documentsandedgesarereportedinthesameway.Thetypefordocumentinsertions/updatesandedgeinsertions/updatesisnowalways2300.PreviousversionsofArangoDBreturnedatypevalueof2300fordocumentsand2301foredges.recordsaboutinsertions,updatesorremovalsofdocumentsandedgesdonothavethekeyandrevattributesonthetop-levelanymore.Instead,keyandrevcanbeaccessedbypeekingintothe_keyand_revattributesofthedatasub-attributesofthechangerecord.

Thesameistrueforthecollection-specificchangesAPIGET/_api/replication/dump.

UsermanagementAPIs

TheRESTAPIendpointPOST/_api/userforaddingnewusersnowrequirestherequesttocontainaJSONobjectwithanattributenameduser,containingthenameoftheusertobecreated.PreviousversionsofArangoDBalsocheckedthisattribute,butadditionallylookedforanattributeusernameiftheuserattributedidnotexist.

UndocumentedAPIs

ThefollowingundocumentedHTTPRESTendpointshavebeenremovedfromArangoDB'sRESTAPI:

/_open/cerberusand/_system/cerberus:theseendpointswereintendedforsomeArangoDB-internalapplicationsonlyPUT/_api/simple/by-example-hash,PUT/_api/simple/by-example-skiplistandPUT/_api/simple/by-condition-skiplist:thesemethodsweredocumentedinearlyversionsofArangoDBbuthavebeenmarkedasnotintendedtobecalledbyenduserssinceArangoDBversion2.3.ThesemethodsshouldnothavebeenpartofanyArangoDBmanualsinceversion2.4./_api/structure:anolderunfinishedandunpromotedAPIfordataformatandtypechecks,supersededbyFoxxapplications.

AdministrationAPIs

/_admin/shutdownnowneedstobecalledwiththeHTTPDELETEmethod

HandlingofCORSrequests

ItcannowbecontrolledindetailforwhichoriginhostsCORS(Cross-originresourcesharing)requestswithcredentialswillbeallowed.ArangoDB3.0providesthestartupoption--http.trusted-originthatcanbeusedtospecifyoneormanyoriginsfromwhichCORSrequestsaretreatedas"trustworthy".

Theoptioncanbespecifiedmultipletimes,oncepertrustedorigin,e.g.

--http.trusted-originhttp://127.0.0.1:8529--http.trusted-originhttps://127.0.0.1:8599

ThiswillmaketheArangoDBserverrespondtoCORSrequestsfromtheseoriginswithanAccess-Control-Allow-CredentialsHTTPheaderwithavalueoftrue.WebbrowserscaninspectthisheaderandcanallowpassingArangoDBwebinterfacecredentials(ifstoredinthebrowser)totherequestingsite.ArangoDBwillnotforwardorprovideanycredentials.

Incompatiblechangesin3.0

553

SettingthisoptionisonlyrequiredifapplicationsonotherhostsneedtoaccesstheArangoDBwebinterfaceorotherHTTPRESTAPIsfromawebbrowserwiththesamecredentialsthattheuserhasenteredwhenloggingintothewebinterface.WhenawebbrowserfindstheAccess-Control-Allow-CredentialsHTTPresponseheader,itmayforwardthecredentialsenteredintothebrowserfortheArangoDBwebinterfacelogintotheothersite.

Thisisapotentialsecurityissue,sotherearenotrustedoriginsbydefault.Itmayberequiredtosetsometrustedoriginsifyou'replanningtoissueAJAXrequeststoArangoDBfromothersitesfromthebrowser,withthecredentialsenteredduringtheArangoDBinterfacelogin(i.e.singlesign-on).Ifsuchfunctionalityisnotused,theoptionshouldnotbeset.

Tospecifyatrustedorigin,specifytheoptiononcepertrustedoriginasshownabove.NotethatthetrustedoriginvaluesspecifiedinthisoptionwillbecomparedbytewisewiththeOriginHTTPheadervaluesentbyclients,andonlyexactmatcheswillpass.

ThereisalsothewildcardallforenablingCORSaccessfromalloriginsinatestordevelopmentsetup:

--http.trusted-originall

SettingthisoptionwillleadtotheArangoDBserverrespondingwithanAccess-Control-Allow-Credentials:trueHTTPheadertoallincomingCORSrequests.

Command-lineoptionsQuiteafewstartupoptionsinArangoDB2weredoublenegations(like--server.disable-authenticationfalse).InArangoDB3thesearenowexpressedaspositives(e.g.--server.authentication).AlsotheoptionsbetweentheArangoDBserveranditsclienttoolshavebeingunified.Forexample,theloggeroptionsarenowthesamefortheserverandtheclienttools.Additionallymanyoptionshavebeenmovedintomoreappropriatetopicsections.

Renamedoptions

Thefollowingoptionshavebeenavailablebefore3.0andhavechangedtheirnamein3.0:

--server.disable-authenticationwasrenamedto--server.authentication.Notethatthemeaningoftheoption--server.authenticationistheoppositeoftheprevious--server.disable-authentication.--server.disable-authentication-unix-socketswasrenamedto--server.authentication-unix-sockets.Notethatthemeaningoftheoption--server.authentication-unix-socketsistheoppositeoftheprevious--server.disable-authentication-unix-sockets.--server.authenticate-system-onlywasrenamedto--server.authentication-system-only.Themeaningoftheoptioninunchanged.--server.disable-statisticswasrenamedto--server.statistics.Notethatthemeaningoftheoption--server.statisticsistheoppositeoftheprevious--server.disable-statistics.--server.cafilewasrenamedto--ssl.cafile.Themeaningoftheoptionisunchanged.--server.keyfilewasrenamedto--ssl.keyfile.Themeaningoftheoptionisunchanged.--server.ssl-cachewasrenamedto--ssl.session-cache.Themeaningoftheoptionisunchanged.--server.ssl-cipher-listwasrenamedto--ssl.cipher-list.Themeaningoftheoptionisunchanged.--server.ssl-optionswasrenamedto--ssl.options.Themeaningoftheoptionisunchanged.--server.ssl-protocolwasrenamedto--ssl.protocol.Themeaningoftheoptionisunchanged.--server.backlog-sizewasrenamedto--tcp.backlog-size.Themeaningoftheoptionisunchanged.--server.reuse-addresswasrenamedto--tcp.reuse-address.Themeaningoftheoptionisunchanged.--server.disable-replication-applierwasrenamedto--database.replication-applier.Themeaningoftheoption--database.replication-applieristheoppositeoftheprevious--server.disable-replication-applier.--server.allow-method-overridewasrenamedto--http.allow-method-override.Themeaningoftheoptionisunchanged.--server.hide-product-headerwasrenamedto--http.hide-product-header.Themeaningoftheoptionisunchanged.--server.keep-alive-timeoutwasrenamedto--http.keep-alive-timeout.Themeaningoftheoptionisunchanged.--server.foxx-queueswasrenamedto--foxx.queues.Themeaningoftheoptionisunchanged.--server.foxx-queues-poll-intervalwasrenamedto--foxx.queues-poll-interval.Themeaningoftheoptionisunchanged.--no-serverwasrenamedto--server.rest-server.Notethatthemeaningoftheoption--server.rest-serveristheoppositeoftheprevious--no-server.--database.query-cache-modewasrenamedto--query.cache-mode.Themeaningoftheoptionisunchanged.--database.query-cache-max-resultswasrenamedto--query.cache-entries.Themeaningoftheoptionisunchanged.

Incompatiblechangesin3.0

554

--database.disable-query-trackingwasrenamedto--query.tracking.Themeaningoftheoption--query.trackingistheoppositeoftheprevious--database.disable-query-tracking.--log.ttywasrenamedto--log.foreground-tty.Themeaningoftheoptionisunchanged.--upgradehasbeenrenamedto--database.auto-upgrade.Incontrastto2.8thisoptionnowrequiresabooleanparameter.Toactuallyperformanautomaticdatabaseupgradeatstartupuse--database.auto-upgradetrue.Tonotperformit,use--database.auto-upgradefalse.--check-versionhasbeenrenamedto--database.check-version.--temp-pathhasbeenrenamedto--temp.path.

Logverbosity,topicsandoutputfiles

Loggingnowsupportslogtopics.Youcancontrolthesebyspecifyingalogtopicinfrontofalogleveloranoutput.Forexample

--log.levelstartup=trace--log.levelinfo

willlogmessagesconcerningstartupattracelevel,everythingelseatinfolevel.--log.levelcanbespecifiedmultipletimesatstartup,forasmanytopicsasneeded.

Somerelevantlogtopicsavailablein3.0are:

collector:informationabouttheWALcollector'sstatecompactor:informationaboutthecollectiondatafilecompactordatafiles:datafile-relatedoperationsmmap:informationaboutmemory-mappingoperationsperformance:someperformance-relatedinformationqueries:executedAQLqueriesreplication:replication-relatedinforequests:HTTPrequestsstartup:informationaboutserverstartupandshutdownthreads:informationaboutthreads

Thenewlogoption--log.output<definition>allowsdirectingtheglobalorper-topiclogoutputtodifferentoutputs.Theoutputdefinition""canbeoneof

"-"forstdin"+"forstderr"syslog://""syslog:///""file://"

Theoptioncanbespecifiedmultipletimesinordertoconfiguretheoutputfordifferentlogtopics.Tosetupaper-topicoutputconfiguration,use--log.output<topic>=<definition>,e.g.

queries=file://queries.txt

logsallqueriestothefile"queries.txt".

Theoldoption--log.fileisstillavailablein3.0forconveniencereasons.In3.0itisashortcutforthemoregeneraloption--log.outputfile://filename.

Theoldoption--log.requests-fileisstillavailablein3.0.Itisnowashortcutforthemoregeneraloption--log.outputrequests=file://....

Theoldoption--log.performanceisstillavailablein3.0.Itisnowashortcutforthemoregeneraloption--log.levelperformance=trace.

Removedoptionsforlogging

Theoptions--log.content-filterand--log.source-filterhavebeenremoved.TheyhavemostbeenusedduringArangoDB'sinternaldevelopment.

Incompatiblechangesin3.0

555

Thesyslog-relatedoptions--log.applicationand--log.facilityhavebeenremoved.Theyaresupersededbythemoregeneral--log.outputoptionwhichcanalsohandlesyslogtargets.

Removedotheroptions

Theoption--server.default-api-compatibilitywaspresentinearlierversionofArangoDBtocontrolvariousaspectsoftheserverbehavior,e.g.HTTPreturncodesortheformatofHTTP"location"headers.ClientapplicationscouldsendanHTTPheader"x-arango-version"withaversionnumbertorequesttheserverbehaviorofacertainArangoDBversion.

Thisoptionwasonlyhonoredinahandfulofcases(describedabove)andwasremovedin3.0becausethechangesinserverbehaviorcontrolledbythisoptionwerechangedevenbeforeArangoDB2.0.Thisshouldhaveleftenoughtimeforclientapplicationstoadapttothenewbehavior,makingtheoptionsuperfluousin3.0.

Threadoptions

Theoptions--server.threadsand--scheduler.threadsnowhaveadefaultvalueof0.When--server.threadsissetto0onstartup,thesuitablenumberofthreadswillbedeterminedbyArangoDBbyaskingtheOSforthenumberofavailableCPUsandusingthatasabaseline.IfthenumberofCPUsislowerthan4,ArangoDBwillstillstart4dispatcherthreads.When--scheduler.threadsissetto0,thenArangoDBwillautomaticallydeterminethenumberofschedulerthreadstostart.Thiswillnormallycreate2schedulerthreads.

Iftheexactnumberofthreadsneedstobesetbytheadmin,thenitisstillpossibletoset--server.threadsand--scheduler.threadstonon-zerovalues.ArangoDBwillusethesevaluesandstartthatmanythreads(notethatsomethreadsmaybecreatedlazilysotheymaynotbepresentdirectlyafterstartup).

ThenumberofV8JavaScriptcontextstobecreated(--javascript.v8-contexts)nowhasadefaultvalueof0too,meaningthatArangoDBwillcreateasmanyV8contextsastherewillbedispatcherthreads(controlledbythe--server.threadsoption).Settingthisoptiontoanon-zerovaluewillcreateexactlyasmanyV8contextsasspecified.

Settingtheseoptionsexplicitlytonon-zerovaluesmaybebeneficialinenvironmentsthathavefewresources(processingtime,maximumthreadcount,availablememory).

AuthenticationThedefaultvaluefor--server.authenticationisnowtrueintheconfigurationfilesshippedwithArangoDB.Thismeanstheserverwillbestartedwithauthenticationenabledbydefault,requiringallclientconnectionstoprovideauthenticationdatawhenconnectingtoArangoDBAPIs.PreviousArangoDBversionsusedthesetting--server.disable-authenticationtrue,effectivelydisablingauthenticationbydefault.

Thedefaultvaluefor--server.authentication-system-onlyisnowtrueinArangoDB.ThatmeansthatFoxxapplicationsrunninginArangoDBwillbepublicaccessible(atleasttheywillnotuseArangoDB'sbuiltinauthenticationmechanism).OnlyrequeststoArangoDBAPIsatURLpathprefixes/_api/and/_adminwillrequireauthentication.Tochangethat,andusethebuiltinauthenticationmechanismforFoxxapplicationstoo,set--server.authentication-system-onlytofalse,andmakesuretohavetheoption--server.authenticationsettotrueaswell.

Thoughenablingtheauthenticationisrecommendedforproductionsetups,itmaybeoverkillinadevelopmentenvironment.Toturnoffauthentication,theoption--server.authenticationcanbesettofalseinArangoDB'sconfigurationfileoronthecommand-line.

WebAdminInterface

TheJavaScriptshellhasbeenremovedfromArangoDB'swebinterface.ThefunctionalitytheshellprovidedisstillfullyavailableintheArangoShell(arangosh)binaryshippedwithArangoDB.

ArangoShellandclienttools

Incompatiblechangesin3.0

556

TheArangoShell(arangosh)andtheotherclienttoolsbundledwithArangoDBcanonlyconnecttoanArangoDBserverofversion3.0orhigher.TheywillnotconnecttoanArangoDB2.8.ThisisbecausetheserverHTTPAPIshavechangedbetween2.8and3.0,andallclienttoolsusestheseAPIs.

InordertoconnecttoearlierversionsofArangoDBwiththeclienttools,anolderversionoftheclienttoolsneedstobekeptinstalled.

ThepreferrednameforthetemplatestringgeneratorfunctionaqlQueryisnowaqlandisautomaticallyavailableinarangosh.Elsewhere,itcanbeloadedlikeconstaql=require('@arangodb').aql.

Command-lineoptionsadded

Allclienttoolsin3.0provideanoption--server.max-packet-sizeforcontrollingthemaximumsizeofHTTPpacketstobehandledbytheclienttools.Thedefaultvalueis128MB,asinpreviousversionsofArangoDB.Incontrasttopreviousversionsinwhichthevaluewashard-coded,theoptionisnowconfigurable.ItcanbeincreasedtomaketheclienttoolshandleverylargeHTTPresultmessagessentbytheserver.

Command-lineoptionschanged

Forallclienttools,theoption--server.disable-authenticationwasrenamedto--server.authentication.Notethatthemeaningoftheoption--server.authenticationistheoppositeoftheprevious--server.disable-authentication.

Theoption--server.ssl-protocolwasrenamedto--ssl.protocol.Themeaningoftheoptionisunchanged.

Thecommand-lineoption--quietwasremovedfromallclienttoolsexceptarangoshbecauseithadnoeffectinthem.

Arangobench

Inordertomakeitspurposemoreapparenttheformerarangobclienttoolhasbeenrenamedtoarangobenchin3.0.

MiscellaneouschangesThechecksumcalculationalgorithmforthecollection.checksum()methodanditscorrespondingRESTAPIGET/_api/collection/<collection</checksumhaschangedin3.0.Checksumscalculatedin3.0willdifferfromchecksumscalculatedwith2.8orbefore.

TheArangoDBserverin3.0doesnotreadafileENDPOINTScontainingalistofadditionalendpointsonstartup.In2.8thisfilewasautomaticallyreadifpresentinthedatabasedirectory.

Thenamesofthesub-threadsstartedbyArangoDBhavechangedin3.0.ThisisrelevantonLinuxonly,wherethreadscanbenamedandthreadnamesmaybevisibletosystemtoolssuchastopormonitoringsolutions.

Incompatiblechangesin3.0

557

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.8.ArangoDB2.8alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.

AQLimprovements

AQLGraphTraversals/PatternMatching

AQLoffersanewfeaturetotraverseoveragraphwithoutwritingJavaScriptfunctionsbutwithalltheotherfeaturesyouknowfromAQL.Forthispurpose,aspecialversionofFORvariableNameINexpressionhasbeenintroduced.

Thisspecialversionhasthefollowingformat:FORvertex-variable,edge-variable,path-variableINtraversal-expression,wheretraversal-expressionhasthefollowingformat:[depth]directionstart-vertexgraph-definitionwiththefollowinginputparameters:

depth(optional):defineshowmanystepsareexecuted.Thevaluecaneitherbeanintegervalue(e.g.3)orarangeofintegervalues(e.g.1..5).Thedefaultis1.direction:defineswhichedgedirectionsarefollowed.CanbeeitherOUTBOUND,INBOUNDorANY.start-vertex:defineswherethetraversalisstarted.Mustbean_idvalueoradocument.graph-definition:defineswhichedgecollectionsareusedforthetraversal.MustbeeitherGRAPHgraph-nameforgraphscreatedwiththegraph-module,oralistofedgecollectionsedge-col1,edge-col2,..edge-colN.

Thethreeoutputvariableshavethefollowingsemantics:

vertex-variable:Thelastvisitedvertex.edge-variable:Thelastvisitededge(optional).path-variable:Thecompletepathfromstart-vertextovertex-variable(optional).

ThetraversalstatementcanbeusedinthesamewayastheoriginalFORvariableNameINexpression,andcanbecombinedwithfiltersandotherAQLconstructs.

AsanexampleonecannowfindthefriendsofafriendforacertainuserwiththisAQLstatement:

FORfoaf,e,pathIN2ANY@startUserGRAPH"relations"

FILTERpath.edges[0].type=="friend"

FILTERpath.edges[1].type=="friend"

FILTERfoaf._id!=@startUser

RETURNDISTINCTfoaf

Optimizerruleshavebeenimplementedtogainperformanceofthetraversalstatement.Theserulesmovefilterstatementsintothetraversalstatements.t.pathswhichcanneverpassthefilterarenotemittedtothevariables.

Asanexampletakethequeryaboveandassumethereareedgesthatdonothavetype=="friend".Ifinthefirstedgestepthereissuchanon-friendedgethesecondstepswillneverbecomputedfortheseedgesastheycannotfulfillthefiltercondition.

ArrayIndexes

Hashindexesandskiplistindexescannowoptionallybedefinedforarrayvaluessothattheyindexindividualarraymembersinsteadoftheentirearrayvalue.

Todefineanindexforarrayvalues,theattributenameisextendedwiththeexpansionoperator[*]intheindexdefinition.

Example:

db._create("posts");

db.posts.ensureHashIndex("tags[*]");

Whengiventhefollowingdocument

WhatsNewin2.8

558

{

"tags":[

"AQL",

"ArangoDB",

"Index"

]

}

thisindexwillnowcontaintheindividualvalues"AQL","ArangoDB"and"Index".

Nowtheindexcanbeusedforfindingalldocumentshaving"ArangoDB"somewhereintheirtagsarrayusingthefollowingAQLquery:

FORdocINposts

FILTER"ArangoDB"INdoc.tags[*]

RETURNdoc

Itisalsopossibletocreateanindexonsub-attributesofarrayvalues.Thismakessensewhentheindexattributeisanarrayofobjects,e.g.

db._drop("posts");

db._create("posts");

db.posts.ensureIndex({type:"hash",fields:["tags[*].name"]});

db.posts.insert({tags:[{name:"AQL"},{name:"ArangoDB"},{name:"Index"}]});

db.posts.insert({tags:[{name:"AQL"},{name:"2.8"}]});

Thefollowingquerywillthenusethearrayindex:

FORdocINposts

FILTER'AQL'INdoc.tags[*].name

RETURNdoc

Arrayvalueswillautomaticallybede-duplicatedbeforebeinginsertedintoanarrayindex.

PleasenotethatfilteringusingarrayindexesonlyworksfromwithinAQLqueriesandonlyifthequeryfiltersontheindexedattributeusingtheINoperator.Theothercomparisonoperators(==,!=,>,>=,<,<=)currentlydonotusearrayindexes.

Optimizerimprovements

TheAQLqueryoptimizercannowuseindexesifmultiplefilterconditionsonattributesofthesamecollectionarecombinedwithlogicalORs,andiftheusageofindexeswouldcompletelycovertheseconditions.

Forexample,thefollowingqueriescannowusetwoindependentindexesonvalue1andvalue2(thelatterqueryrequiresthattheindexesareskiplistindexesduetousageofthe<and>comparisonoperators):

FORdocINcollectionFILTERdoc.value1==42||doc.value2==23RETURNdoc

FORdocINcollectionFILTERdoc.value1<42||doc.value2>23RETURNdoc

Thenewoptimizerrule"sort-in-values"cannowpre-sorttheright-handsideoperandofINandNOTINoperatorssotheoperationcanuseabinarysearchwithlogarithmiccomplexityinsteadofalinearsearch.Therulewillbeappliedwhentheright-handsideoperandofanINorNOTINoperatorinafilterconditionisavariablethatisdefinedinadifferentloop/scopethantheoperatoritself.Additionally,thefilterconditionmustconsistofsolelytheINorNOTINoperationinordertoavoidanyside-effects.

Therulewillkickinforaqueriessuchasthefollowing:

LETvalues=/*someruntimeexpressionhere*/

FORdocINcollection

FILTERdoc.valueINvalues

RETURNdoc

Itwillnotbeappliedforthefollowigqueries,becausetheright-handsideoperandoftheINiseithernotavariable,orbecausetheFILTERconditionmayhavesideeffects:

WhatsNewin2.8

559

FORdocINcollection

FILTERdoc.valueIN/*someruntimeexpressionhere*/

RETURNdoc

LETvalues=/*someruntimeexpressionhere*/

FORdocINcollection

FILTERFUNCTION(doc.values)==23&&doc.valueINvalues

RETURNdoc

AQLfunctionsadded

ThefollowingAQLfunctionshavebeenaddedin2.8:

POW(base,exponent):returnsthebasetotheexponentexp

UNSET_RECURSIVE(document,attributename,...):recursivelyremovestheattributesattributename(canbeoneormany)fromdocumentanditssub-documents.Allotherattributeswillbepreserved.Multipleattributenamescanbespecifiedbyeitherpassingmultipleindividualstringargumentnames,orbypassinganarrayofattributenames:

UNSET_RECURSIVE(doc,'_id','_key','foo','bar')

UNSET_RECURSIVE(doc,['_id','_key','foo','bar'])

IS_DATESTRING(value):returnstrueifvalueisastringthatcanbeusedinadatefunction.Thisincludespartialdatessuchas2015or2015-10andstringscontaininginvaliddatessuchas2015-02-31.Thefunctionwillreturnfalseforallnon-stringvalues,evenifsomeofthemmaybeusableindatefunctions.

Miscellaneousimprovements

theArangoShellnowprovidestheconveniencefunctiondb._explain(query)forretrievingahuman-readableexplanationofAQLqueries.Thisfunctionisashorthandforrequire("org/arangodb/aql/explainer").explain(query).

theAQLqueryoptimizernowautomaticallyconvertsLENGTH(collection-name)toanoptimizedexpressionthatreturnsthenumberofdocumentsinacollection.PreviousversionsofArangoDBreturnedawarningwhenusingthisexpressionandalsoenumeratedalldocumentsinthecollection,whichwasinefficient.

improvedperformanceofskippingovermanydocumentsinanAQLquerywhennoindexesandnofiltersareused,e.g.

FORdocINcollection

LIMIT1000000,10

RETURNdoc

addedclusterexecutionsiteinfoinexecutionplanexplainoutputforAQLqueries

for30+AQLfunctionsthereisnowanadditionalimplementationinC++thatremovestheneedforinternaldataconversionwhenthefunctioniscalled

theAQLeditorinthewebinterfacenowsupportsusingbindparameters

Deadlockdetection

ArangoDB2.8nowhasanautomaticdeadlockdetectionfortransactions.

Adeadlockisasituationinwhichtwoormoreconcurrentoperations(usertransactionsorAQLqueries)trytoaccessthesameresources(collections,documents)andneedtowaitfortheotherstofinish,butnoneofthemcanmakeanyprogress.

Incaseofsuchadeadlock,therewouldbenoprogressforanyoftheinvolvedtransactions,andnoneoftheinvolvedtransactionscouldevercomplete.Thisiscompletelyundesirable,sothenewautomaticdeadlockdetectionmechanisminArangoDBwillautomaticallykickinandabortoneofthetransactionsinvolvedinsuchadeadlock.Abortingmeansthatallchangesdonebythetransactionwillberolledbackanderror29(deadlockdetected)willbethrown.

WhatsNewin2.8

560

Clientcode(AQLqueries,usertransactions)thataccessesmorethanonecollectionshouldbeawareofthepotentialofdeadlocksandshouldhandletheerror29(deadlockdetected)properly,eitherbypassingtheexceptiontothecallerorretryingtheoperation.

Replication

Thefollowingimprovementsforreplicationhavebeenmadein2.8(note:mostofthemhavebeenbackportedtoArangoDB2.7aswell):

addedautoResyncconfigurationparameterforcontinuousreplication.

Whensettotrue,areplicationslavewillautomaticallytriggerafulldatare-synchronizationwiththemasterwhenthemastercannotprovidethelogdatatheslavehadaskedfor.NotethatautoResyncwillonlyworkwhentheoptionrequireFromPresentisalsosettotrueforthecontinuousreplication,orwhenthecontinuoussyncerisstartedanddetectsthatnostarttickispresent.

Automaticre-synchronizationmaytransferalotofdatafromthemastertotheslaveandmaybeexpensive.Itisthereforeturnedoffbydefault.Whenturnedoff,theslavewillneverperformanautomaticre-synchronizationwiththemaster.

addedidleMinWaitTimeandidleMaxWaitTimeconfigurationparametersforcontinuousreplication.

Theseparameterscanbeusedtocontroltheminimumandmaximumwaittimetheslavewill(intentionally)idleandnotpollformasterlogchangesincasethemasterhadsentthefulllogsalready.TheidleMaxWaitTimevaluewillonlybeusedwhenadapativePollingissettotrue.WhenadaptivePollingisdisabled,onlyidleMinWaitTimewillbeusedasaconstanttimespaninwhichtheslavewillnotpollthemasterforfurtherchanges.Thedefaultvaluesare0.5secondsforidleMinWaitTimeand2.5secondsforidleMaxWaitTime,whichcorrespondtothehard-codedvaluesusedinpreviousversionsofArangoDB.

addedinitialSyncMaxWaitTimeconfigurationparameterforinitialandcontinuousreplication

Thisoptioncontrolsthemaximumwaittime(inseconds)thattheinitialsynchronizationwillwaitforaresponsefromthemasterwhenfetchinginitialcollectiondata.Ifnoresponseisreceivedwithinthistimeperiod,theinitialsynchronizationwillgiveupandfail.ThisoptionisalsorelevantforcontinuousreplicationincaseautoResyncissettotrue,asthenthecontinuousreplicationmaytriggerafulldatare-synchronizationincasethemastercannotthelogdatatheslavehadaskedfor.

HTTPrequestssentfromtheslavetothemasterduringinitialsynchronizationwillnowberetriediftheyfailwithconnectionproblems.

theinitialsynchronizationnowlogsitsprogresssoitcanbequeriedusingtheregularreplicationstatuscheckAPIs.

addedasyncattributeforsyncandsyncCollectionoperationscalledfromtheArangoShell.Setthingthisattributetotruewillmakethesynchronizationjobontheservergointothebackground,sothattheshelldoesnotblock.ThestatusofthestartedasynchronoussynchronizationjobcanbequeriedfromtheArangoShelllikethis:

/*startsinitialsynchronization*/

varreplication=require("org/arangodb/replication");

varid=replication.sync({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

async:true

});

/*nowquerytheidofthereturnedasyncjobandprintthestatus*/

print(replication.getSyncResult(id));

TheresultofgetSyncResult()willbefalsewhiletheserver-sidejobhasnotcompleted,anddifferenttofalseifithascompleted.Whenithascompleted,alljobresultdetailswillbereturnedbythecalltogetSyncResult().

thewebadmininterfacedashboardnowshowsaserver'sreplicationstatusatthebottomofthepage

WebAdminInterface

Thefollowingimprovementshavebeenmadeforthewebadmininterface:

theAQLeditornowhassupportforbindparameters.Thebindparametervaluescanbeeditedinthewebinterfaceandsavedwithaqueryforfutureuse.

WhatsNewin2.8

561

theAQLeditornowallowscancelingrunningqueries.Thiscanbeusedtocancellong-runningquerieswithoutswitchingtothequerymanagementsection.

thedashboardnowprovidesinformationabouttheserver'sreplicationstatusatthebottomofthepage.Thiscanbeusedtotrackeitherthestatusofaone-timesynchronizationorthecontinuousreplication.

thecompactionstatusandsomestatusinternalsaboutcollectionsarenowdisplayedinthedetailviewforacollectioninthewebinterface.Thesedatacanbeusedfordebuggingcompactionissues.

unloadingacollectionviathewebinterfacewillnowtriggergarbagecollectioninallv8contextsandforceaWALflush.Thisincreasesthechancesofperfomingtheunloadfaster.

thestatusterminologyforcollectionsforwhichanunloadrequesthasbeenissuedviathewebinterfacewaschangedfromintheprocessofbeingunloadedtowillbeunloaded.Thisismoreaccurateastheactualunloadmaybepostponeduntillateriftherearestillreferencespointingtodatainthecollection.

Foxximprovementsthemoduleresolutionusedbyrequirenowbehavesmorelikeinnode.js

theorg/arangodb/requestmodulenowreturnsresponsebodiesforerrorresponsesbydefault.Theoldbehaviorofnotreturningbodiesforerrorresponsescanbere-enabledbyexplicitlysettingtheoptionreturnBodyOnErrortofalse

Miscellaneouschanges

Thestartupoption--server.hide-product-headercanbeusedtomaketheservernotsendtheHTTPresponseheader"Server:ArangoDB"initsHTTPresponses.ThiscanbeusedtoconcealtheservermakefromHTTPclients.Bydefault,theoptionisturnedoffsotheheaderisstillsentasusual.

arangodumpandarangorestorenowhavebettererrorreporting.Additionally,arangodumpwillnowfailbydefaultwhentryingtodumpedgesthatrefertoalreadydroppedcollections.Thiscanbecircumventedbyspecifyingtheoption--forcetruewheninvokingarangodump.

arangoimpnowprovidesanoption--create-collection-typetospecifythetypeofthecollectiontobecreatedwhen--create-collectionissettotrue.Previously--create-collectionalwayscreateddocumentcollectionsandthecreationofedgecollectionswasnotpossible.

WhatsNewin2.8

562

IncompatiblechangesinArangoDB2.8ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.8,andadjustanyclientprogramsifnecessary.

AQL

Keywordsadded

ThefollowingAQLkeywordswereaddedinArangoDB2.8:

GRAPH

OUTBOUND

INBOUND

ANY

ALL

NONE

AGGREGATE

Usageofthesekeywordsforcollectionnames,variablenamesorattributenamesinAQLquerieswillnotbepossiblewithoutquoting.Forexample,thefollowingAQLquerywillstillworkasitusesaquotedcollectionnameandaquotedattributename:

FORdocIN`OUTBOUND`

RETURNdoc.`any`

Changedbehavior

TheAQLfunctionsNEARandWITHINnowhavestrictervalidationsfortheirinputparameterslimit,radiusanddistance.Theymaynowthrowexceptionswheninvalidparametersarepassedthatmayhavenotledtoexceptionsinpreviousversions.

Additionally,theexpansion([*])operatorinAQLhaschangeditsbehaviorwhenhandlingnon-arrayvalues:

InArangoDB2.8,callingtheexpansionoperatoronanon-arrayvaluewillalwaysreturnanemptyarray.PreviousversionsofArangoDBexpandednon-arrayvaluesbycallingtheTO_ARRAY()functionforthevalue,whichforexamplereturnedanarraywithasinglevalueforboolean,numericandstringinputvalues,andanarraywiththeobject'svaluesforanobjectinputvalue.Thisbehaviorwasinconsistentwithhowtheexpansionoperatorworksforthearrayindexesin2.8,sothebehaviorisnowunified:

iftheleft-handsideoperandof[*]isanarray,thearraywillbereturnedasiswhencalling[*]onitiftheleft-handsideoperandof[*]isnotanarray,anemptyarraywillbereturnedby[*]

AQLqueriesthatrelyontheoldbehaviorcanbechangedbyeithercallingTO_ARRAYexplicitlyorbyusingthe[*]atthecorrectposition.

Thefollowingexamplequerywillchangeitsresultin2.8comparedto2.7:

LETvalues="foo"RETURNvalues[*]

In2.7thequeryhasreturnedthearray["foo"],butin2.8itwillreturnanemptyarray[].Tomakeitreturnthearray["foo"]again,anexplicitTO_ARRAYfunctioncallisneededin2.8(whichinthiscaseallowstheremovalofthe[*]operatoraltogether).Thisalsoworksin2.7:

LETvalues="foo"RETURNTO_ARRAY(values)

Anotherexample:

LETvalues=[{name:"foo"},{name:"bar"}]

RETURNvalues[*].name[*]

Incompatiblechangesin2.8

563

Theabovereturned[["foo"],["bar"]]in2.7.In2.8itwillreturn[[],[]],becausethevalueofname`isnotanarray.Tochangetheresultstothe2.7style,thequerycanbechangedto

LETvalues=[{name:"foo"},{name:"bar"}]

RETURNvalues[*RETURNTO_ARRAY(CURRENT.name)]

Theabovealsoworksin2.7.Thefollowingtypesofquerieswon'tchange:

LETvalues=[1,2,3]RETURNvalues[*]

LETvalues=[{name:"foo"},{name:"bar"}]RETURNvalues[*].name

LETvalues=[{names:["foo","bar"]},{names:["baz"]}]RETURNvalues[*].names[*]

LETvalues=[{names:["foo","bar"]},{names:["baz"]}]RETURNvalues[*].names[**]

Deadlockhandling

Clientapplicationsshouldbepreparedtohandleerror29(deadlockdetected)thatArangoDBmaynowthrowwhenitdetectsadeadlockacrossmultipletransactions.Whenaclientapplicationreceiveserror29,itshouldretrytheoperationthatfailed.

TheerrorcanonlyoccurforAQLqueriesorusertransactionsthatinvolvemorethanasinglecollection.

Optimizer

TheAQLexecutionnodetypeIndexRangeNodewasreplacedwithanewmorecapableexecutionnodetypeIndexNode.ThatmeansinexecutionplanexplainoutputtherewillbenomoreIndexRangeNodesbutonlyIndexNode.Thisaffectsexplainoutputthatcanberetrievedviarequire("org/arangodb/aql/explainer").explain(query),db._explain(query),andtheHTTPqueryexplainAPI.

TheoptimizerrulethatmakesAQLqueriesactuallyuseindexeswasalsorenamedfromuse-index-rangetouse-indexes.Againthisaffectsexplainoutputthatcanberetrievedviarequire("org/arangodb/aql/explainer").explain(query),db._explain(query),andtheHTTPqueryexplainAPI.

Thequeryoptimizerruleremove-collect-intowasrenamedtoremove-collect-variables.Thisaffectsexplainoutputthatcanberetrievedviarequire("org/arangodb/aql/explainer").explain(query),db._explain(query),andtheHTTPqueryexplainAPI.

HTTPAPI

Whenaserver-sideoperationgotcanceledduetoanexplicitclientcancelrequestviaHTTPDELETE/_api/job,previousversionsofArangoDBreturnedanHTTPstatuscodeof408(requesttimeout)fortheresponseofthecanceledoperation.

TheHTTPreturncode408hascausedproblemswithsomeclientapplications.Somebrowsers(e.g.Chrome)handleda408responsebyresendingtheoriginalrequest,whichistheoppositeofwhatisdesiredwhenajobshouldbecanceled.

ThereforeArangoDBwillreturnHTTPstatuscode410(gone)forcanceledoperationsfromversion2.8on.

Foxx

ModelandRepository

DuetocompatibilityissuestheModelandRepositorytypesarenolongerimplementedasES2015classes.

Thepre-2.7"extend"stylesubclassingissupportedagainandwillnotemitanydeprecationwarnings.

varFoxx=require('org/arangodb/foxx');

varMyModel=Foxx.Model.extend({

//...

schema:{/*...*/}

});

Moduleresolution

Incompatiblechangesin2.8

564

ThebehavioroftheJavaScriptmoduleresolutionusedbytherequirefunctionhasbeenmodifiedtoimprovecompatibilitywithmoduleswrittenforNode.js.

Specifically

absolutepaths(e.g./some/absolute/path)arenowalwaysinterpretedasabsolutefilesystempaths,relativetothefilesystemroot

globalnames(e.g.global/name)arenowfirstintepretedasreferencestomodulesresidinginarelevantnode_modulesfolder,abuilt-inmoduleoramatchingdocumentintheinternal_modulescollection,andonlyresolvedtolocalfilepathsifnoothermatchisfound

Previouslythetwoformatsweretreatedinterchangeablyandwouldberesolvedtolocalfilepathsfirst,leadingtoproblemswhenlocalfilesusedthesamenamesasothermodules(e.g.alocalfilechai.jswouldcauseproblemswhentryingtoloadthechaimoduleinstalledinnode_modules).

Formoreinformationseetheblogannouncementofthischangeandtheupgradeguide.

Moduleorg/arangodb/request

Themodulenowalwaysreturnsresponsebodies,evenforerrorresponses.Inversionspriorto2.8themodulewouldsilentlydropresponsebodiesiftheresponseheaderindicatedanerror.

TheoldbehaviorofnotreturningbodiesforerrorresponsescanberestoredbyexplicitlysettingtheoptionreturnBodyOnErrortofalse:

letresponse=request({

//...

returnBodyOnError:false

});

Garbagecollection

TheV8garbagecollectionstrategywasslightlyadjustedsothatiteventuallyhappensinallV8contextsthatholdV8externalobjects(referencestoArangoDBdocumentsandcollections).Thisenablesabettercleanupoftheseresourcesandpreventsotherprocessessuchascompactionbeingstalledwhilewaitingfortheseresourcestobereleased.

InthiscontextthedefaultvaluefortheJavaScriptgarbagecollectionfrequency(--javascript.gc-frequency)wasalsoincreasedfrom10secondsto15seconds,aslessinternaloperationsinArangoDBarecarriedoutinJavaScript.

Clienttools

arangodumpwillnowfailbydefaultwhentryingtodumpedgesthatrefertoalreadydroppedcollections.Thiscanbecircumventedbyspecifyingtheoption--forcetruewheninvokingarangodump

Incompatiblechangesin2.8

565

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.7.ArangoDB2.7alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.

Performanceimprovements

Indexbuckets

Theprimaryindexesandhashindexesofcollectionscannowbesplitintomultipleindexbuckets.ThisoptionwasavailableforedgeindexesonlyinArangoDB2.6.

Abucketcanbeconsideredacontainerforaspecificrangeofindexvalues.Forprimary,hashandedgeindexes,determiningtheresponsiblebucketforanindexvalueisdonebyhashingtheactualindexvalueandapplyingasimplearithmeticoperationonthehash.

Becauseanindexvaluewillbepresentinatmostonebucketandbucketsareindependent,usingmultiplebucketsprovidesthefollowingbenefits:

initiallybuildingthein-memoryindexdatacanbeparallelizedevenforasingleindex,withonethreadperbucket(orwiththreadsbeingresponsibleformorethanonebucketatatime).Thiscanhelpreducingtheloadingtimeforcollections.

resizinganindexwhenitisabouttorunoutofreservespaceisperformedperbucket.Aseachbucketonlycontainsafractionoftheentireindex,resizingandrehashingabucketismuchfasterandlessintrusivethanresizingandrehashingtheentireindex.

Whencreatingnewcollections,thedefaultnumberofindexbucketsis8sinceArangoDB2.7.Inpreviousversions,thedefaultvaluewas1.Thenumberofbucketscanalsobeadjustedforexistingcollectionssotheycanbenefitfromtheoptimizations.Thenumberofindexbucketscanbesetforacollectionatanytimebyusingacollection'spropertiesfunction:

db.collection.properties({indexBuckets:16});

Thenumberofindexbucketsmustbeapowerof2.

Pleasenotethatforbuildingtheindexdataformultiplebucketsinparallelitisrequiredthatacollectioncontainsasignificantamountofdocumentsbecauseforalownumberofdocumentstheoverheadofparallelizationwilloutweighitsbenefits.Thecurrentthresholdvalueis256kdocuments,butthisvaluemaychangeinfutureversionsofArangoDB.Additionally,theconfigurationoption--database.index-threadswilldeterminehowmanyparallelthreadsmaybeusedforbuildingtheindexdata.

Fasterupdateandremoveoperationsinnon-uniquehashindexes

TheuniquehashindexesinArangoDBprovidedanamortizedO(1)lookup,insert,updateandremoveperformance.Non-uniquehashindexesprovidedamortizedO(1)insertperformance,buthadworseperformanceforupdateandremoveoperationsfornon-uniquevalues.Fordocumentswiththesameindexvalue,theymaintainedalistofcollisions.Whenadocumentwasupdatedorremoved,thatexactdocumenthadtobefoundinthecollisionslistfortheindexvalue.WhilegettingtothestartofthecollisionslistwasO(1),scanningthelisthadO(n)performanceintheworstcase(withnbeingthenumberofdocumentswiththesameindexvalue).Overall,thismadeupdateandremoveoperationsinnon-uniquehashindexesslowiftheindexcontainedmanyduplicatevalues.

ThishasbeenchangedinArangoDB2.7sothatnon-uniquehashindexesnowalsoprovideupdateandremoveoperationswithanamortizedcomplexityofO(1),eveniftherearemanyduplicates.

Resizingnon-uniquehashindexesnowalsodoesn'trequirelookingintothedocumentdata(whichmayinvolveadiskaccess)becausetheindexmaintainssomeinternalcachevalueperdocument.Whenresizingandrehashingtheindex(oranindexbucket),theindexwillfirstcompareonlythecachevaluesbeforepeekingintotheactualdocuments.Thischangecanalsoleadtoreducedindexresizingtimes.

Throughputenhancements

TheArangoDB-internalimplementationsfordispatchingrequests,keepingstatisticsandassigningV8contextstothreadshavebeenimprovedinordertouselesslocks.Thesechangesallowhigherconcurrencyandthroughputinthesecomponents,whichcanalsomaketheserverhandlemorerequestsinagivenperiodoftime.

WhatsNewin2.7

566

Whatgainscanbeexpecteddependsonwhichoperationsareexecuted,buttherearereal-worldcasesinwhichthroughputincreasedbybetween25%and70%whencomparedto2.6.

Madvisehints

TheLinuxvariantforArangoDBprovidestheOSwithmadvisehintsaboutindexmemoryanddatafilememory.Thesehintscanspeedupthingswhenmemoryistight,inparticularatcollectionloadtimebutalsoforrandomaccesseslater.ThereisnoformalguaranteethattheOSactuallyusesthemadvisehintsprovidedbyArangoDB,butactualmeasurementshaveshownimprovementsforloadingbiggercollections.

AQLimprovements

Additionaldatefunctions

ArangoDB2.7providesseveralextraAQLfunctionsfordateandtimecalculationandmanipulation.ThesefunctionswerecontributedbyGitHubusers@CoDEmanXand@friday.Abigthanksfortheirwork!

Thefollowingextradatefunctionsareavailablefrom2.7on:

DATE_DAYOFYEAR(date):Returnsthedayofyearnumberofdate.Thereturnvaluesrangefrom1to365,or366inaleapyearrespectively.

DATE_ISOWEEK(date):ReturnstheISOweekdateofdate.Thereturnvaluesrangefrom1to53.Mondayisconsideredthefirstdayoftheweek.Therearenofractionalweeks,thusthelastdaysinDecembermaybelongtothefirstweekofthenextyear,andthefirstdaysinJanuarymaybepartofthepreviousyear'slastweek.

DATE_LEAPYEAR(date):Returnswhethertheyearofdateisaleapyear.

DATE_QUARTER(date):Returnsthequarterofthegivendate(1-based):

1:January,February,March2:April,May,June3:July,August,September4:October,November,December

DATE_DAYS_IN_MONTH(date):Returnsthenumberofdaysindate'smonth(28..31).

DATE_ADD(date,amount,unit):Addsamountgiveninunittodateandreturnsthecalculateddate.

unitcanbeeitherofthefollowingtospecifythetimeunittoaddorsubtract(case-insensitive):

y,year,yearsm,month,monthsw,week,weeksd,day,daysh,hour,hoursi,minute,minutess,second,secondsf,millisecond,milliseconds

amountisthenumberofunitstoadd(positivevalue)orsubtract(negativevalue).

DATE_SUBTRACT(date,amount,unit):Subtractsamountgiveninunitfromdateandreturnsthecalculateddate.

ItworksthesameasDATE_ADD(),exceptthatitsubtracts.ItisequivalenttocallingDATE_ADD()withanegativeamount,exceptthatDATE_SUBTRACT()canalsosubtractISOdurations.NotethatnegativeISOdurationsarenotsupported(i.e.startingwith-P,like-P1Y).

DATE_DIFF(date1,date2,unit,asFloat):Calculatethedifferencebetweentwodatesingiventimeunit,optionallywithdecimalplaces.Returnsanegativevalueifdate1isgreaterthandate2.

DATE_COMPARE(date1,date2,unitRangeStart,unitRangeEnd):Comparetwopartialdatesandreturntrueiftheymatch,falseotherwise.Thepartstocomparearedefinedbyarangeoftimeunits.

WhatsNewin2.7

567

Thefullrangeis:years,months,days,hours,minutes,seconds,milliseconds.PasstheunittostartfromasunitRangeStart,andtheunittoendwithasunitRangeEnd.Allunitsinbetweenwillbecompared.LeaveoutunitRangeEndtoonlycompareunitRangeStart.

DATE_FORMAT(date,format):Formatadateaccordingtothegivenformatstring.Itsupportsthefollowingplaceholders(case-insensitive):

%t:timestamp,inmillisecondssincemidnight1970-01-01%z:ISOdate(0000-00-00T00:00:00.000Z)%w:dayofweek(0..6)%y:year(0..9999)%yy:year(00..99),abbreviated(lasttwodigits)%yyyy:year(0000..9999),paddedtolengthof4%yyyyyy:year(-009999..+009999),withsignprefixandpaddedtolengthof6%m:month(1..12)%mm:month(01..12),paddedtolengthof2%d:day(1..31)%dd:day(01..31),paddedtolengthof2%h:hour(0..23)%hh:hour(00..23),paddedtolengthof2%i:minute(0..59)%ii:minute(00..59),paddedtolengthof2%s:second(0..59)%ss:second(00..59),paddedtolengthof2%f:millisecond(0..999)%fff:millisecond(000..999),paddedtolengthof3%x:dayofyear(1..366)%xxx:dayofyear(001..366),paddedtolengthof3%k:ISOweekdate(1..53)%kk:ISOweekdate(01..53),paddedtolengthof2%l:leapyear(0or1)%q:quarter(1..4)%a:daysinmonth(28..31)%mmm:abbreviatedEnglishnameofmonth(Jan..Dec)%mmmm:Englishnameofmonth(January..December)%www:abbreviatedEnglishnameofweekday(Sun..Sat)%wwww:Englishnameofweekday(Sunday..Saturday)%&:specialescapesequenceforrareoccasions%%:literal%%:ignored

RETURNDISTINCT

Toreturnuniquevaluesfromaquery,AQLnowprovidestheDISTINCTkeyword.ItcanbeusedasamodifierforRETURNstatements,asashorteralternativetothealreadyexistingCOLLECTstatement.

Forexample,thefollowingqueryonlyreturnsdistinct(unique)statusattributevaluesfromthecollection:

FORdocINcollection

RETURNDISTINCTdoc.status

RETURNDISTINCTisnotallowedonthetop-levelofaqueryifthereisnoFORloopinfrontofit.RETURNDISTINCTisallowedinsubqueries.

RETURNDISTINCTensuresthatthevaluesreturnedaredistinct(unique),butdoesnotguaranteeanyorderofresults.Inordertohavecertainresultorder,anadditionalSORTstatementmustbeaddedtoaquery.

Shorthandobjectnotation

WhatsNewin2.7

568

AQLnowprovidesashorthandnotationforobjectliteralsinthestyleofES6objectliterals:

LETname="Peter"

LETage=42

RETURN{name,age}

Thisisequivalenttothepreviouslyavailablecanonicalform,whichisstillavailableandsupported:

LETname="Peter"

LETage=42

RETURN{name:name,age:age}

Arrayexpansionimprovements

Thealreadyexisting[*]operatorhasbeenimprovedwithoptionalfilteringandprojectionandlimitcapabilities.

Forexample,considerthefollowingexamplequerythatfiltersvaluesfromanarrayattribute:

FORuINusers

RETURN{

name:u.name,

friends:(

FORfINu.friends

FILTERf.age>u.age

RETURNf.name

)

}

Withthe[*]operator,thisquerycanbesimplifiedto

FORuINusers

RETURN{name:u.name,friends:u.friends[*FILTERCURRENT.age>u.age].name}

Thepseudo-variableCURRENTcanbeusedtoaccessthecurrentarrayelement.TheFILTERconditioncanrefertoCURRENToranyvariablesvalidintheouterscope.

Toreturnaprojectionofthecurrentelement,therecannowbeaninlineRETURN:

FORuINusers

RETURNu.friends[*RETURNCONCAT(CURRENT.name,"isafriendof",u.name)]

whichisthesimplifiedvariantfor:

FORuINusers

RETURN(

FORfriendINu.friends

RETURNCONCAT(friend.name,"isafriendof",u.name)

)

Arraycontraction

Inordertocollapse(orflatten)resultsinnestedarrays,AQLnowprovidesthe[**]operator.Itworkssimilartothe[*]operator,butadditionallycollapsesnestedarrays.Howmanylevelsarecollapsedisdeterminedbytheamountof*charactersused.

Forexample,considerthefollowingquerythatproducesanestedresult:

FORuINusers

RETURNu.friends[*].name

The[**]operatorcannowbeappliedtogetridofthenestedarrayandturnitintoaflatarray.Wesimplyapplythe[**]onthepreviousqueryresult:

WhatsNewin2.7

569

RETURN(

FORuINusersRETURNu.friends[*].name

)[**]

Templatequerystrings

AssemblingquerystringsinJavaScripthasbeenerror-pronewhenusingsimplestringconcatenation,especiallybecauseplainJavaScriptstringsdonothavemultiline-support,andbecauseofpotentialparameterinjectionissues.WhilemultilinequerystringscanbeassembledwithES6templatestringssinceArangoDB2.5,andquerybindparametersaretheresinceArangoDB1.0topreventparameterinjection,therewasnoJavaScript-ysolutiontocombinethese.

ArangoDB2.7nowprovidesanES6templatestringgeneratorfunctionthatcanbeusedtoeasilyandsafelyassembleAQLqueriesfromJavaScript.JavaScriptvariablesandexpressionscanbeusedeasilyusingregularES6templatestringsubstitutions:

letname='test';

letattributeName='_key';

letquery=aqlQuery`FORuINusers

FILTERu.name==${name}

RETURNu.${attributeName}`;

db._query(query);

ThisismorelegiblethanwhenusingaplainJavaScriptstringandalsodoesnotrequiredefiningthebindparametervaluesseparately:

letname='test';

letattributeName='_key';

letquery="FORuINusers"+

"FILTERu.name==@name"+

"RETURNu.@attributeName";

db._query(query,{

name,

attributeName

});

TheaqlQuerytemplatestringgeneratorwillalsohandlecollectionobjectsautomatically:

db._query(aqlQuery`FORuIN${db.users}RETURNu.name`);

NotethatwhiletemplatestringsareavailableintheJavaScriptfunctionsprovidedtobuildqueries,theyaren'tafeatureofAQLitself.AQLcouldalwayshandlemultilinequerystringsandprovidedbindparameters(@...)forseparatingthequerystringandtheparametervalues.TheaqlQuerytemplatestringgeneratorfunctionwilltakecareofthisseparation,too,butwilldoitbehindthescenes.

AQLqueryresultcache

TheAQLqueryresultcachecanoptionallycachethecompleteresultsofallorjustselectedAQLqueries.Itcanbeoperatedinthefollowingmodes:

off:thecacheisdisabled.Noqueryresultswillbestoredon:thecachewillstoretheresultsofallAQLqueriesunlesstheircacheattributeflagissettofalsedemand:thecachewillstoretheresultsofAQLqueriesthathavetheircacheattributesettotrue,butwillignoreallothers

Themodecanbesetatserverstartupusingthe--database.query-cache-modeconfigurationoptionandlaterchangedatruntime.Thedefaultvalueisoff,meaningthatthequeryresultcacheisdisabled.Thisisbecausethecachemayconsumeadditionalmemorytokeepqueryresults,andalsobecauseitmustbeinvalidatedwhenchangeshappenincollectionsforwhichresultshavebeencached.

Thequeryresultcachemaythereforehavepositiveornegativeeffectsonqueryexecutiontimes,dependingontheworkload:itwillnotmakemuchsenseturningonthecacheinwrite-onlyorwrite-mostlyscenarios,butthecachemaybeverybeneficialincaseworkloadsareread-onlyorread-mostly,andqueryarecomplex.

Ifthequerycacheisoperatedindemandmode,itcanbecontrolledperqueryifthecacheshouldbecheckedforaresult.

WhatsNewin2.7

570

Miscellaneouschanges

Optimizer

TheAQLoptimizerrulepatch-update-statementshasbeenadded.ThisrulecanoptimizecertainAQLUPDATEqueriesthatupdatedocumentsintheacollectionthattheyalsoiterateover.

Forexample,thefollowingqueryreadsdocumentsfromacollectioninordertoupdatethem:

FORdocINcollection

UPDATEdocWITH{newValue:doc.oldValue+1}INcollection

Inthiscase,onlyasinglecollectionisaffectedbythequery,andthereisnoindexlookupinvolvedtofindtheto-be-updateddocuments.Inthiscase,theUPDATEquerydoesnotrequiretakingafull,memory-intensivesnapshotofthecollection,butitcanbeperformedinsmallchunks.Thiscanleadtomemorysavingswhenexecutingsuchqueries.

Functioncallargumentsoptimization

ThisoptimizationwillleadtoargumentsinfunctioncallsinsideAQLqueriesnotbeingcopiedbutbeingpassedbyreference.ThismayspeedupcallstofunctionswithbiggerargumentvaluesorqueriesthatcallAQLfunctionsalotoftimes.

WebAdminInterface

Thewebinterfacenowhasanewdesign.

The"Applications"tabinthewebinterfaceshasbeenrenamedto"Services".

TheArangoDBAPIdocumentationhasbeenmovedfromthe"Tools"menutothe"Links"menu.ThenewdocumentationisbasedonSwagger2.0andopensinaseparatewebpage.

Foxximprovements

ES2015Classes

AllFoxxconstructorshavebeenreplacedwithES2015classesandcanbeextendedusingtheclasssyntax.TheextendmethodisstillsupportedatthemomentbutwillbecomedeprecatedinArangoDB2.8andremovedinArangoDB2.9.

Before:

varFoxx=require('org/arangodb/foxx');

varMyModel=Foxx.Model.extend({

//...

schema:{/*...*/}

});

After:

varFoxx=require('org/arangodb/foxx');

classMyModelextendsFoxx.Model{

//...

}

MyModel.prototype.schema={/*...*/};

Confidentialconfiguration

Itisnowpossibletospecifyconfigurationoptionswiththetypepassword.ThepasswordtypeisequivalenttothetexttypebutwillbemaskedinthewebfrontendtopreventaccidentalexposureofconfidentialoptionslikeAPIkeysandpasswordswhenconfiguringyourFoxxapplication.

WhatsNewin2.7

571

Dependencies

Thesyntaxforspecifyingdependenciesinmanifestshasbeenextendedtoallowspecifyingoptionaldependencies.Unmetoptionaldependencieswillnotpreventanappfrombeingmounted.Thetraditionalshorthandsyntaxforspecifyingnon-optionaldependencieswillstillbesupportedintheupcomingversionsofArangoDB.

Before:

{

...

"dependencies":{

"notReallyNeeded":"users:^1.0.0",

"totallyNecessary":"sessions:^1.0.0"

}

}

After:

{

"dependencies":{

"notReallyNeeded":{

"name":"users",

"version":"^1.0.0",

"required":false

},

"totallyNecessary":{

"name":"sessions",

"version":"^1.0.0"

}

}

}

Replication

TheexistingreplicationHTTPAPIhasbeenextendedwithmethodsthatreplicationclientscanusetodeterminewhetheragivendate,identifiedbyatickvalue,isstillpresentonamasterforreplication.BycallingtheseAPIs,clientscanmakeaninformeddecisionaboutwhetherthemastercanstillprovideallmissingdatastartingfromthepointuptowhichtheclienthadalreadysynchronized.Thiscanbehelpfulincaseareplicationclientisre-startedafterapause.

Masterserversnowalsotrackupthepointuptowhichtheyhavesentchangestoclientsforreplication.Thisinformationcanbeusedtodeterminethepointofdatathatreplicationclientshavereceivedfromthemaster,andifandhowfarapproximatelytheylagbehind.

Finally,restartingthereplicationapplieronaslaveserverhasbeenmademorerobustincasetheapplierwasstoppedwhiletherewerependingtransactionsonthemasterserver,andre-startingthereplicationapplierneedstorestorethestateofthesetransactions.

Clienttools

Thefilenamesindumpscreatedbyarangodumpnowcontainnotonlythenameofthedumpedcollection,butalsoanadditional32-digithashvalue.Thisisdonetopreventoverwritingdumpfilesincase-insensitivefilesystemswhenthereexistmultiplecollectionswiththesamename(butwithdifferentcases).

Forexample,ifadatabasehadtwocollectionstestandTest,previousversionsofarangodumpcreatedthefollowingfiles:

test.structure.jsonandtest.data.jsonforcollectiontestTest.structure.jsonandTest.data.jsonforcollectionTest

Thisdidnotworkincase-insensitivefilesystems,becausethefilesforthesecondcollectionwouldhaveoverwrittenthefilesofthefirst.arangodumpin2.7willcreatetheuniquefilesinthiscase,byappendingthe32-digithashvaluetothecollectionnameinallcase.Thesefilenameswillbeunambiguousevenincase-insensitivefilesystems.

Miscellaneouschanges

WhatsNewin2.7

572

Bettercontrol-Csupportinarangosh

WhenCTRL-Cispressedinarangosh,itwillnowabortthelocallyrunningcommand(ifany).Ifnocommandwasrunning,pressingCTRL-Cwillprinta Cfirst.PressingCTRL-Cagainwillthenquitarangosh.

CTRL-Ccanalsobeusedtoresetthecurrentpromptwhileenteringcomplexnestedobjectswhichspanmultipleinputlines.

CTRL-CsupporthasbeenaddedtotheArangoShellversionsbuiltwithReadline-support(LinuxandMacOSonly).TheWindowsversionofArangoDBusesadifferentlibraryforhandlinginput,andsupportforCTRL-Chasnotbeenaddedthereyet.

Start/stop

LinuxstartupscriptsandsystemdconfigurationforarangodnowtrytoadjusttheNOFILE(numberofopenfiles)limitsfortheprocess.Thelimitvalueissetto131072(128k)whenArangoDBisstartedviastart/stopcommands.

ThiswillpreventarangodrunningoutofavailablefiledescriptorsincaseofmanyparallelHTTPconnectionsorlargecollectionswithmanydatafiles.

Additionally,whenArangoDBisstarted/stoppedmanuallyviathestart/stopcommands,themainprocesswillwaitforupto10secondsafteritforksthesupervisorandarangodchildprocesses.Ifthestartupfailswithinthatperiod,thestart/stopscriptwillfailwithanon-zeroexitcode,allowinganyinvokingscriptstohandlethiserror.Previousversionsalwaysreturnedanexitcodeof0,evenwhenarangodcouldn'tbestarted.

Ifthestartupofthesupervisororarangodisstillongoingafter10seconds,themainprogramwillstillreturnwithexitcode0inordertonotblockanyscripts.Thelimitof10secondsisarbitrarybecausethetimerequiredforanarangodstartupisnotknowninadvance.

Non-sparselogfiles

WALlogfilesanddatafilescreatedbyarangodarenownon-sparse.ThispreventsSIGBUSsignalsbeingraisedwhenamemory-mappedregionbackedbyasparsedatafilewasaccessedandthememoryregionwasnotactuallybackedbydisk,forexamplebecausethediskranoutofspace.

arangodnowalwaysfullyallocatesthediskspacerequiredforalogfileordatafilewhenitcreatesone,sothememoryregioncanalwaysbebackedbydisk,andmemorycanbeaccessedwithoutSIGBUSbeingraised.

WhatsNewin2.7

573

IncompatiblechangesinArangoDB2.7ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.7,andadjustanyclientprogramsifnecessary.

AQLchanges

DISTINCTisnowakeywordinAQL.

AQLqueriesthatuseDISTINCT(inlower,upperormixedcase)asanidentifier(i.e.asavariable,acollectionnameorafunctionname)willstopworking.Tomakesuchqueriesworkingagain,eachoccurrenceofDISTINCTinanAQLqueryshouldbeenclosedinbackticks.ThiswillturnDISTINCTfromakeywordintoanidentifieragain.

TheAQLfunctionSKIPLIST()hasbeenremovedinArangoDB2.7.ThisfunctionwasdeprecatedinArangoDB2.6.Itwasaleft-overfromtimeswhenthequeryoptimizerwasn'tabletouseskiplistindexestogetherwithfilters,skipandlimitvalues.Sincethisissuebeenfixedsinceversion2.3,thereisnoAQLreplacementfunctionforSKIPLIST.QueriesthatusetheSKIPLISTfunctioncanbefixedbyusingtheusualcombinationofFOR,FILTERandLIMIT,e.g.

FORdocIN@@collection

FILTERdoc.value>=@value

SORTdoc.valueDESC

LIMIT1

RETURNdoc

Foxxchanges

Bundlingandcompilation

TheassetspropertyisnolongersupportedinFoxxmanifestsandisscheduledtoberemovedinafutureversionofArangoDB.Thefilespropertycanstillbeusedtoservestaticassetsbutitisrecommendedtouseseparatetoolingtocompileandbundleyourassets.

Manifestscripts

Thepropertiessetupandteardownhavebeenmovedintothescriptspropertymap:

Before:

{

...

"setup":"scripts/setup.js",

"teardown":"scripts/teardown.js"

}

After:

{

...

"scripts":{

"setup":"scripts/setup.js",

"teardown":"scripts/teardown.js"

}

}

FoxxQueues

Function-basedFoxxQueuejobtypesarenolongersupported.Tolearnabouthowyoucanusethenewscript-basedjobtypesfollowtheupdatedrecipeinthecookbook.

Incompatiblechangesin2.7

574

FoxxSessions

ThejwtandtypeoptionshavebeenremovedfromtheactivateSessionsAPI.

IfyouwanttoreplicatethebehaviorofthejwtoptionyoucanusetheJWTfunctionsinthecryptomodule.AJWT-basedsessionstoragethatdoesn'twritesessionstothedatabaseisavailableasthesessions-jwtappintheFoxxappstore.

Thesessiontypeisnowinferredfromthepresenceofthecookieorheaderoptions(allowingyoutoenablesupportforboth).Ifyouwanttousethedefaultsettingsforcookieorheaderyoucanpassthevaluetrueinstead.

ThesessionStorageAppoptionhasbeenremovedinfavourofthesessionStorageoption.

Before:

varFoxx=require('org/arangodb/foxx');

varctrl=newFoxx.Controller(applicationContext);

ctrl.activateSessions({

sessionStorageApp:'some-sessions-app',

type:'cookie'

});

After:

ctrl.activateSessions({

sessionStorage:applicationContext.dependencies.sessions.sessionStorage,

cookie:true

});

Requestmodule

Themoduleorg/arangodb/requestusesaninternallibraryfunctionforsendingHTTPrequests.ThislibraryfunctionallyunconditionallysetanHTTPheaderAccept-Encoding:gzipinalloutgoingHTTPrequests,withoutclientcodehavingtosetthisheaderexplicitly.

Thishasbeenfixedin2.7,soAccept-Encoding:gzipisnotsetautomaticallyanymore.AdditionallytheheaderUser-Agent:ArangoDBisnotsetautomaticallyeither.Ifclientapplicationsrelyontheseheadersbeingsent,theyarefreetoadditwhenconstructingrequestsusingtherequestmodule.

Theinternal.download()functionisalsoaffectedbythischange.Again,theheadercanbeaddedhereifrequiredbypassingitviaaheaderssub-attributeinthethirdparameter(options)tothisfunction.

arangodump/backupsThefilenamesindumpscreatedbyarangodumpnowcontainnotonlythenameofthedumpedcollection,butalsoanadditional32-digithashvalue.Thisisdonetopreventoverwritingdumpfilesincase-insensitivefilesystemswhenthereexistmultiplecollectionswiththesamename(butwithdifferentcases).

Thischangeleadstochangedfilenamesindumpscreatedbyarangodump.Ifanyclientscriptsdependonthefilenamesinthedumpoutputdirectorybeingequaltothecollectionnameplusoneofthesuffixes.structure.jsonand.data.json,theyneedtobeadjusted.

StartingwithArangoDB2.7,thefilenameswillcontainanunderscoreplusthe32-digitMD5value(representedinhexadecimalnotation)ofthecollectionname.

Forexample,whenarangodumpdumpsdataoftwocollectionstestandTest,thefilenamesinpreviousversionsofArangoDBwere:

test.structure.json(definitionsforcollectiontest)test.data.json(dataforcollectiontest)Test.structure.json(definitionsforcollectionTest)Test.data.json(dataforcollectionTest)

In2.7,thefilenameswillbe:

test_098f6bcd4621d373cade4e832627b4f6.structure.json(definitionsforcollectiontest)

Incompatiblechangesin2.7

575

test_098f6bcd4621d373cade4e832627b4f6.data.json(dataforcollectiontest)Test_0cbc6611f5540bd0809a388dc95a615b.structure.json(definitionsforcollectionTest)Test_0cbc6611f5540bd0809a388dc95a615b.data.json(dataforcollectionTest)

Starting/stopping

Whenstartingarangod,theserverwillnowdroptheprocessprivilegestothespecifiedvaluesinoptions--server.uidand--server.gidinstantlyafterparsingthestartupoptions.

Thatmeanswheneither--server.uidor--server.gidareset,theprivilegechangewillhappenearlier.Thismaypreventbindingtheservertoanendpointwithaportnumberlowerthan1024ifthearangodbuserhasnoprivilegesforthat.PreviousversionsofArangoDBchangedtheprivilegeslater,sosomestartupactionswerestillcarriedoutundertheinvokinguser(i.e.likelyrootwhenstartedviainit.dorsystemscripts)andespeciallybindingtolowportnumberswasstillpossiblethere.

Thedefaultprivilegesforuserarangodbwillnotbesufficientforbindingtoportnumberslowerthan1024.TohaveanArangoDB2.7bindtoaportnumberlowerthan1024,itneedstobestartedwitheitheradifferentprivilegeduser,ortheprivilegesofthearangodbuserhavetoraisedmanuallybeforehand.

Additionally,LinuxstartupscriptsandsystemdconfigurationforarangodnowwilladjusttheNOFILE(numberofopenfiles)limitsfortheprocess.Thelimitvalueissetto131072(128k)whenArangoDBisstartedviastart/stopcommands.Thegoalofthischangeistopreventarangodfromrunningoutofavailablefiledescriptorsforsocketconnectionsanddatafiles.

Connectionhandling

arangodwillnowactuallycloselingeringclientconnectionswhenidleforatleastthedurationspecifiedinthe--server.keep-alive-timeoutstartupoption.

InpreviousversionsofArangoDB,idleconnectionswerenotclosedbytheserverwhenthetimeoutwasreachedandtheclientwasstillconnected.Nowtheconnectionisproperlyclosedbytheserverincaseoftimeout.Clientapplicationsrelyingontheoldbehaviormaynowneedtoreconnecttotheserverwhentheiridleconnectionstimeoutandgetclosed(note:connectionsbeingidleforalongtimemaybeclosedbytheOSorfirewallsanyway-clientapplicationsshouldbeawareofthatandtrytoreconnect).

Optionchanges

Configureoptionsremoved

Thefollowingoptionsforconfigurehavebeenremovedbecausetheywereunusedorexotic:

--enable-timings

--enable-figures

Startupoptionsadded

Thefollowingconfigurationoptionshavebeenaddedin2.7:

--database.query-cache-max-results:setsthemaximumnumberofresultsinAQLqueryresultcacheperdatabase--database.query-cache-mode:setsthemodefortheAQLqueryresultscache.Possiblevaluesareon,offanddemand.Thedefaultvalueisoff

Miscellaneouschanges

Simplequeries

Manysimplequeriesprovideaskip()functionthatcanbeusedtoskipoveracertainnumberofdocumentsintheresult.ThisfunctionallowedspecifyingnegativeoffsetsinpreviousversionsofArangoDB.Specifyinganegativeoffsetledtothequeryresultbeingiteratedinreverseorder,soskippingwasperformedfromthebackoftheresult.Asmostsimplequeriesdonotprovideaguaranteedresultorder,

Incompatiblechangesin2.7

576

skippingfromthebackofaresultwithunspecificorderseemsaratherexoticusecaseandwasremovedtoincreaseconsistencywith

AQL,whichalsodoesnotprovidenegativeskipvalues.

NegativeskipvaluesweredeprecatedinArangoDB2.6.

TasksAPI

TheundocumentedfunctionaddJob()hasbeenremovedfromtheorg/arangodb/tasksmoduleinArangoDB2.7.

RuntimeendpointsmanipulationAPI

ThefollowingHTTPRESTAPImethodsforruntimemanipulationofserverendpointshavebeenremovedinArangoDB2.7:

POST/_api/endpoint:todynamicallyaddanendpointwhiletheserverwasrunningDELETE/_api/endpoint:todynamicallyremoveanendpointwhiletheserverwasrunning

ThischangealsoaffectstheequivalentJavaScriptendpointmanipulationmethodsavailableinFoxx.ThefollowingfunctionshavebeenremovedinArangoDB2.7:

db._configureEndpoint()

db._removeEndpoint()

Incompatiblechangesin2.7

577

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.6.ArangoDB2.6alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.

APIsadded

Batchdocumentremovalandlookupcommands

Thefollowingcommandshavebeenaddedforcollectionobjects:

collection.lookupByKeys(keys)collection.removeByKeys(keys)

Thesecommandscanbeusedtoperformmulti-documentlookupandremovaloperationsefficientlyfromtheArangoShell.Theargumenttotheseoperationsisanarrayofdocumentkeys.

ThesecommandscanalsobeusedviatheHTTPRESTAPI.Theirendpointsare:

PUT/_api/simple/lookup-by-keysPUT/_api/simple/remove-by-keys

CollectionexportHTTPRESTAPI

ArangoDBnowprovidesadedicatedcollectionexportAPI,whichcantakesnapshotsofentirecollectionsmoreefficientlythanthegeneral-purposecursorAPI.TheexportAPIisusefultotransferthecontentsofanentirecollectiontoaclientapplication.Itprovidesoptionalfilteringonspecificattributes.

TheexportAPIisavailableatendpointPOST/_api/export?collection=....TheAPIhasthesamereturnvaluestructureasthealreadyestablishedcursorAPI(POST/_api/cursor).

AnintroductiontotheexportAPIisgiveninthisblogpost:http://jsteemann.github.io/blog/2015/04/04/more-efficient-data-exports/

AQLimprovements

EDGESAQLFunction

TheAQLfunctionEDGESgotanewfifthoptionalparameter,whichmustbeanobjectifspecified.Rightnowonlyoneoptionisavailableforit:

includeVerticesthisisabooleanparameterthatallowstomodifytheresultofEDGES().ThedefaultvalueforincludeVerticesisfalse,whichdoesnothaveanyeffect.Settingittotruewillmodifytheresult,suchthatalsotheconnectedverticesarereturnedalongwiththeedges:

{vertex:<vertexDocument>,edge:<edgeDocument>}

SubqueryoptimizationsforAQLqueries

Thisoptimizationavoidscopyingintermediateresultsintosubqueriesthatarenotrequiredbythesubquery.

Abriefdescriptioncanbefoundhere:http://jsteemann.github.io/blog/2015/05/04/subquery-optimizations/

ReturnvalueoptimizationforAQLqueries

Thisoptimizationavoidscopyingthefinalqueryresultinsidethequery'smainReturnNode.

Abriefdescriptioncanbefoundhere:http://jsteemann.github.io/blog/2015/05/04/return-value-optimization-for-aql/

WhatsNewin2.6

578

SpeedupAQLqueriescontainingbigINlistsforindexlookups

INlistsusedforindexlookupshadperformanceissuesinpreviousversionsofArangoDB.Theseissueshavebeenaddressedin2.6sousingbiggerINlistsforfilteringismuchfaster.

Abriefdescriptioncanbefoundhere:http://jsteemann.github.io/blog/2015/05/07/in-list-improvements/

AddedalternativeimplementationforAQLCOLLECT

Thealternativemethodusesahashtableforgroupinganddoesnotrequireitsinputelementstobesorted.ItwillbetakenintoaccountbytheoptimizerforCOLLECTstatementsthatdonotuseanINTOclause.

IncaseaCOLLECTstatementcanusethehashtablevariant,theoptimizerwillcreateanextraplanforitatthebeginningoftheplanningphase.Inthisplan,noextraSORTnodewillbeaddedinfrontoftheCOLLECTbecausethehashtablevariantofCOLLECTdoesnotrequiresortedinput.Instead,aSORTnodewillbeaddedafterittosortitsoutput.ThisSORTnodemaybeoptimizedawayagaininlaterstages.Ifthesortorderoftheresultisirrelevanttotheuser,addinganextraSORTnullafterahashCOLLECToperationwillallowtheoptimizertoremovethesortsaltogether.

InadditiontothehashtablevariantofCOLLECT,theoptimizerwillmodifytheoriginalplantousetheregularCOLLECTimplementation.Asthisimplementationrequiressortedinput,theoptimizerwillinsertaSORTnodeinfrontoftheCOLLECT.ThisSORTnodemaybeoptimizedawayinlaterstages.

Thecreatedplanswillthenbeshippedthroughtheregularoptimizationpipeline.Intheend,theoptimizerwillpicktheplanwiththelowestestimatedtotalcostasusual.Thehashtablevariantdoesnotrequireanup-frontsortoftheinput,andwillthusbepreferredovertheregularCOLLECTiftheoptimizerestimatesmanyinputelementsfortheCOLLECTnodeandcannotuseanindextosortthem.

TheoptimizercanbeexplicitlytoldtousetheregularsortedvariantofCOLLECTbysuffixingaCOLLECTstatementwithOPTIONS{"method":"sorted"}.ThiswilloverridetheoptimizerguessworkandonlyproducethesortedvariantofCOLLECT.

AblogpostonthenewCOLLECTimplementationcanbefoundhere:http://jsteemann.github.io/blog/2015/04/22/collecting-with-a-hash-table/

Simplifiedreturnvaluesyntaxfordata-modificationAQLqueries

ArangoDB2.4sinceversionallowstoreturnresultsfromdata-modificationAQLqueries.Thesyntaxforthiswasquitelimitedandverbose:

FORiIN1..10

INSERT{value:i}INtest

LETinserted=NEW

RETURNinserted

TheLETinserted=NEWRETURNinsertedwasrequiredliterallytoreturntheinserteddocuments.Nocalculationscouldbemadeusingtheinserteddocuments.

Thisisnowmoreflexible.Afteradata-modificationclause(e.g.INSERT,UPDATE,REPLACE,REMOVE,UPSERT)therecanfollowanynumberofLETcalculations.Thesecalculationscanrefertothepseudo-valuesOLDandNEWthatarecreatedbythedata-modificationstatements.

Thisallowsreturningprojectionsofinsertedorupdateddocuments,e.g.:

FORiIN1..10

INSERT{value:i}INtest

RETURN{_key:NEW._key,value:i}

Stillnoteveryconstructisallowedafteradata-modificationclause.Forexample,nofunctionscanbecalledthatmayaccessdocuments.

Moreinformationcanbefoundhere:http://jsteemann.github.io/blog/2015/03/27/improvements-for-data-modification-queries/

AddedAQLUPSERTstatement

WhatsNewin2.6

579

ThisaddsanUPSERTstatementtoAQLthatisacombinationofbothINSERTandUPDATE/REPLACE.TheUPSERTwillsearchforamatchingdocumentusingauser-providedexample.Ifnodocumentmatchestheexample,theinsertpartoftheUPSERTstatementwillbeexecuted.Ifthereisamatch,theupdate/replacepartwillbecarriedout:

UPSERT{page:'index.html'}/*searchexample*/

INSERT{page:'index.html',pageViews:1}/*insertpart*/

UPDATE{pageViews:OLD.pageViews+1}/*updatepart*/

INpageViews

UPSERTcanbeusedwithanUPDATEorREPLACEclause.TheUPDATEclausewillperformapartialupdateofthefounddocument,whereastheREPLACEclausewillreplacethefounddocumententirely.TheUPDATEorREPLACEpartscanrefertothepseudo-valueOLD,whichcontainsallattributesofthefounddocument.

UPSERTstatementscanoptionallyreturnvalues.Inthefollowingquery,thereturnattributefoundwillreturnthefounddocumentbeforetheUPDATEwasapplied.Ifnodocumentwasfound,foundwillcontainavalueofnull.Theupdatedresultattributewillcontaintheinserted/updateddocument:

UPSERT{page:'index.html'}/*searchexample*/

INSERT{page:'index.html',pageViews:1}/*insertpart*/

UPDATE{pageViews:OLD.pageViews+1}/*updatepart*/

INpageViews

RETURN{found:OLD,updated:NEW}

AmoredetaileddescriptionofUPSERTcanbefoundhere:http://jsteemann.github.io/blog/2015/03/27/preview-of-the-upsert-command/

Miscellaneouschanges

WhenerrorsoccurinsideAQLuserfunctions,theerrormessagewillnowcontainastacktrace,indicatingthelineofcodeinwhichtheerroroccurred.ThisshouldmakedebuggingAQLuserfunctionseasier.

WebAdminInterfaceArangoDB'sbuilt-inwebinterfacenowusessessions.Sessioninformationisstoredincookies,soclientsusingthewebinterfacemustacceptcookiesinordertouseit.

Thenewstartupoption--server.session-timeoutcanbeusedforadjustingthesessionlifetime.

TheAQLeditorinthewebinterfacenowprovidesanexplainfunctionality,whichcanbeusedforinspectingandperformance-tuningAQLqueries.ThequeryexecutiontimeisnowalsodisplayedintheAQLeditor.

Foxxappsthatrequireconfigurationoraremissingdependenciesarenowindicatedintheappoverviewanddetails.

Foxximprovements

ConfigurationandDependencies

Foxxappmanifestscannowdefineconfigurationoptions,aswellasdependenciesonotherFoxxapps.

AnintroductiontoFoxxconfigurationscanbefoundintheblog:https://www.arangodb.com/2015/05/reusable-foxx-apps-with-configurations/

AndtheblogpostonFoxxdependenciescanbefoundhere:https://www.arangodb.com/2015/05/foxx-dependencies-for-more-composable-foxx-apps/

MochaTests

YoucannowwritetestsforyourFoxxappsusingtheMochatestingframework:https://www.arangodb.com/2015/04/testing-foxx-mocha/

ArecipeforwritingtestsforyourFoxxappscanbefoundinthecookbook:https://docs.arangodb.com/2.8/cookbook/FoxxTesting.html

WhatsNewin2.6

580

APIDocumentation

TheAPIdocumentationhasbeenupdatedtoSwagger2.YoucannowalsomountAPIdocumentationinyourownFoxxapps.

Alsoseetheblogpostintroducingthisfeature:https://www.arangodb.com/2015/05/document-your-foxx-apps-with-swagger-2/

CustomScriptsandFoxxQueue

InadditiontotheexistingsetupandteardownscriptsyoucannowdefinecustomscriptsinyourFoxxmanifestandinvoketheseusingthewebadmininterfaceortheFoxxmanagerCLI.Thesescriptscannowalsotakepositionalargumentsandexportreturnvalues.

JobtypesfortheFoxxQueuecannowbedefinedasascriptnameandappmountpathallowingtheuseofFoxxscriptsasjobtypes.Thepre-2.6jobtypesareknowntocauseissueswhenrestartingtheserverandareerror-prone;westronglyrecommendedconvertinganyexistingjobtypestothenewformat.

Clienttools

Thedefaultconfigurationvaluefortheoption--server.request-timeoutwasincreasedfrom300to1200secondsforallclienttools(arangosh,arangoimp,arangodump,arangorestore).

Thedefaultconfigurationvaluefortheoption--server.connect-timeoutwasincreasedfrom3to5secondsforclienttools(arangosh,arangoimp,arangodump,arangorestore).

Arangorestore

Theoption--create-databasewasaddedforarangorestore.

Settingthisoptiontotruewillnowcreatethetargetdatabaseifitdoesnotexist.Whencreatingthetargetdatabase,theusernameandpasswordspassedtoarangorestorewillbeusedtocreateaninitialuserforthenewdatabase.

Thedefaultvalueforthisoptionisfalse.

Arangoimp

Arangoimpcannowoptionallyupdateorreplaceexistingdocuments,providedtheimportdatacontainsdocumentswith_keyattributes.

Previously,theimportcouldbeusedforinsertingnewdocumentsonly,andre-insertingadocumentwithanexistingkeywouldhavefailedwithauniquekeyconstraintviolatederror.

Thebehaviorofarangoimp(insert,update,replaceonduplicatekey)cannowbecontrolledwiththeoption--on-duplicate.Theoptioncanhaveoneofthefollowingvalues:

error:whenauniquekeyconstrainterroroccurs,donotimportorupdatethedocumentbutreportanerror.Thisisthedefault.

update:whenauniquekeyconstrainterroroccurs,tryto(partially)updatetheexistingdocumentwiththedataspecifiedintheimport.Thismaystillfailifthedocumentwouldviolatesecondaryuniqueindexes.Onlytheattributespresentintheimportdatawillbeupdatedandotherattributesalreadypresentwillbepreserved.ThenumberofupdateddocumentswillbereportedintheupdatedattributeoftheHTTPAPIresult.

replace:whenauniquekeyconstrainterroroccurs,trytofullyreplacetheexistingdocumentwiththedataspecifiedintheimport.Thismaystillfailifthedocumentwouldviolatesecondaryuniqueindexes.ThenumberofreplaceddocumentswillbereportedintheupdatedattributeoftheHTTPAPIresult.

ignore:whenauniquekeyconstrainterroroccurs,ignorethiserror.Therewillbenoinsert,updateorreplacefortheparticulardocument.IgnoreddocumentswillbereportedseparatelyintheignoredattributeoftheHTTPAPIresult.

Thedefaultvalueiserror.

Afewexamplesforusingarangoimpwiththe--on-duplicateoptioncanbefoundhere:http://jsteemann.github.io/blog/2015/04/14/updating-documents-with-arangoimp/

WhatsNewin2.6

581

Miscellaneouschanges

SomeLinux-basedArangoDBpackagesarenowusingtcmallocformemoryallocator.

UpgradedICUlibrarytoversion54.Thisincreasesperformanceinmanyplaces.

Allowtosplitanedgeindexintobucketswhichareresizedindividually.Thedefaultvalueis1,resemblingthepre-2.6behavior.Usingmultiplebucketswillleadtotheindexentriesbeingdistributedtotheindividualbuckets,witheachbucketbeingresponsibleonlyforafractionofthetotalindexentries.Usingmultiplebucketsmayleadtomorefrequentbutmuchfasterindexbucketresizes,andisrecommendedforbiggeredgecollections.

Defaultconfigurationvalueforoption--server.backlog-sizewaschangedfrom10to64.

Defaultconfigurationvalueforoption--database.ignore-datafile-errorswaschangedfromtruetofalse

Documentkeyscannowcontain@and.characters

Fulltextindexcannowindextextvaluescontainedindirectsub-objectsoftheindexedattribute.

PreviousversionsofArangoDBonlyindexedtheattributevalueifitwasastring.Sub-attributesoftheindexattributewereignoredwhenfulltextindexing.

Now,iftheindexattributevalueisanobject,theobject'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.Iftheindexattributevalueisanarray,thearray'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.

Forexample,withafulltextindexpresentonthetranslationsattribute,thefollowingtextvalueswillnowbeindexed:

varc=db._create("example");

c.ensureFulltextIndex("translations");

c.insert({translations:{en:"fox",de:"Fuchs",fr:"renard",ru:"лиса"}});

c.insert({translations:"FoxistheEnglishtranslationoftheGermanwordFuchs"});

c.insert({translations:["ArangoDB","document","database","Foxx"]});

c.fulltext("translations","лиса").toArray();//returnsonlyfirstdocument

c.fulltext("translations","Fox").toArray();//returnsfirstandseconddocuments

c.fulltext("translations","prefix:Fox").toArray();//returnsallthreedocuments

Addedconfigurationoption--server.foxx-queues-poll-interval

ThisstartupoptioncontrolsthefrequencywithwhichtheFoxxqueuesmanagerischeckingthequeue(orqueues)forjobstobeexecuted.

Thedefaultvalueis1second.Loweringthisvaluewillresultinthequeuemanagerwakingupandcheckingthequeuesmorefrequently,whichmayincreaseCPUusageoftheserver.WhennotusingFoxxqueues,thisvaluecanberaisedtosavesomeCPUtime.

Addedconfigurationoption--server.foxx-queues

ThisstartupoptioncontrolswhethertheFoxxqueuemanagerwillcheckqueueandjobentriesinthe_systemdatabaseonly.RestrictingtheFoxxqueuemanagertothe_systemdatabasewillleadtothequeuemanagerhavingtocheckonlythequeuescollectionofasingledatabase,whereasmakingitcheckthequeuesofalldatabasesmightresultinmoreworktobedoneandmoreCPUtimetobeusedbythequeuemanager.

WhatsNewin2.6

582

IncompatiblechangesinArangoDB2.6ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.6,andadjustanyclientprogramsifnecessary.

Requirements

ArangoDB'sbuilt-inwebinterfacenowusescookiesforsessionmanagement.Sessioninformationidsarestoredincookies,soclientsusingthewebinterfacemustacceptcookiesinordertologinanduseit.

Foxxchanges

FoxxQueues

FoxxQueuejobtypedefinitionswerepreviouslybasedonfunctionsandhadtoberegisteredbeforeuse.Duetochangesin2.5thisresultedinproblemswhenrestartingtheserverordefiningjobtypesincorrectly.

Function-basedjobtypeshavebeendeprecatedin2.6andwillberemovedentirelyin2.7.

Inordertoconvertexistingfunction-basedjobtypestothenewscript-basedjobtypes,createcustomscriptsinyourFoxxappandreferencethembytheirnameandthemountpointoftheapptheyaredefinedin.OfficialjobtypesfromtheFoxxappstorecanbeupgradedbyupgradingfromthe1.xversiontothe2.xversionofthesameapp.

Inordertoupgradequeuedjobstothenewjobtypes,youneedtoupdatethetypepropertyoftheaffectedjobsinthedatabase's_jobssystemcollection.Inordertoseethecollectioninthewebinterfaceyouneedtoenablethecollectiontype"System"inthecollectionlistoptions.

Example:

Before:"type":"mailer.postmark"

After:"type":{"name":"mailer","mount":"/my-postmark-mailer"}

FoxxSessions

Theoptionsjwtandtypeofthecontrollermethodcontroller.activateSessionshavebeendeprecatedin2.6andwillberemovedentirelyin2.7.

IfyouwanttousepureJWTsessions,youcanusethesessions-jwtFoxxappfromtheFoxxappstore.

IfyouwanttouseyourownJWT-basedsessions,youcanusetheJWTfunctionsinthecryptomoduledirectly.

Insteadofusingthetypeoptionyoucanjustusethecookieandheaderoptionsontheirown,whichbothnowacceptthevaluetruetoenablethemwiththeirdefaultconfigurations.

TheoptionsessionStorageApphasbeenrenamedtosessionStorageandnowalsoacceptssessionstoragesdirectly.TheoldoptionsessionStorageAppwillberemovedentirelyin2.7.

Libraries

ThebundledversionofthejoilibraryusedinFoxxwasupgradedtoversion6.0.8.ThismayaffectFoxxapplicationsthatdependonthelibrary.

AQLchanges

AQLLENGTHfunction

Incompatiblechangesin2.6

583

ThereturnvalueoftheAQLLENGTHfunctionwaschangedifLENGTHisappliedonnullorabooleanvalue:

LENGTH(null)nowreturns0.InpreviousversionsofArangoDB,thisreturned4.

LENGTH(false)nowreturns0.InpreviousversionsofArangoDB,thereturnvaluewas5.

LENGTH(true)nowreturns1.InpreviousversionsofArangoDB,thereturnvaluewas4.

AQLgraphfunctions

In2.6thegraphfunctionsdidundergoaperformancelifting.Duringthisprocesswehadtoadopttheresultformatandtheoptionsforsomeofthem.ManygraphfunctionsnowhaveanoptionincludeDatawhichallowstotriggeriftheresultofthisfunctionshouldcontainfullyextracteddocumentsincludeData:trueoronlythe_idvaluesincludeData:false.Inmostusecasesthe_idissufficienttocontinueandtheextractionofdataisanunnecessaryoperation.TheAQLfunctionssupportingthisadditionaloptionare:

SHORTEST_PATHNEIGHBORSGRAPH_SHORTEST_PATHGRAPH_NEIGHBORSGRAPH_EDGES

FurthermoretheresultSHORTEST_PATHhaschanged.Theoldformatreturnedalistofallverticesonthepath.Optionallyitcouldincludeeachsub-pathforthesevertices.Allofthedocumentswerefullyextracted.Example:

[

{

vertex:{

_id:"vertex/1",

_key:"1",

_rev:"1234"

name:"Alice"

},

path:{

vertices:[

{

_id:"vertex/1",

_key:"1",

_rev:"1234"

name:"Alice"

}

],

edges:[]

}

},

{

vertex:{

_id:"vertex/2",

_key:"2",

_rev:"5678"

name:"Bob"

},

path:{

vertices:[

{

_id:"vertex/1",

_key:"1",

_rev:"1234"

name:"Alice"

},{

_id:"vertex/2",

_key:"2",

_rev:"5678"

name:"Bob"

}

],

edges:[

{

_id:"edge/1",

_key:"1",

_rev:"9876",

type:"loves"

Incompatiblechangesin2.6

584

}

]

}

}

]

Thenewversionismorecompact.EachSHORTEST_PATHwillonlyreturnonedocumenthavingtheattributesvertices,edges,distance.Thedistanceiscomputedtakingintoaccountthegivenweight.OptionallythedocumentscanbeextractedwithincludeData:trueExample:

{

vertices:[

"vertex/1",

"vertex/2"

],

edges:[

"edge/1"

],

distance:1

}

ThenextfunctionthatreturnsadifferentformatisNEIGHBORS.Since2.5itreturnedanobjectwithedgeandvertexforeachconnectededge.Example:

[

{

vertex:{

_id:"vertex/2",

_key:"2",

_rev:"5678"

name:"Bob"

},

edge:{

_id:"edge/1",

_key:"1",

_rev:"9876",

type:"loves"

}

}

]

With2.6itwillonlyreturnthevertexdirectly,againusingincludeData:true.Bydefaultitwillreturnadistinctsetofneighbors,usingtheoptiondistinct:falsewillincludethesamevertexforeachedgepointingtoit.

Example:

[

"vertex/2"

]

FunctionandAPIchanges

Graphmeasurementsfunctions

AllgraphmeasurementsfunctionsinJavaScriptmodulegeneral-graphthatcalculatedasinglefigurepreviouslyreturnedanarraycontainingjustthefigure.Nowthesefunctionswillreturnthefiguredirectlyandnotputitinsideanarray.

Theaffectedfunctionsare:

graph._absoluteEccentricity

graph._eccentricity

graph._absoluteCloseness

graph._closeness

graph._absoluteBetweenness

Incompatiblechangesin2.6

585

graph._betweenness

graph._radius

graph._diameter

Clientprogramscallingthesefunctionsshouldbeadjustedsotheyprocessthescalarvaluereturnedbythefunctioninsteadofthepreviousarrayvalue.

CursorAPI

AbatchSizevalue0isnowdisallowedwhencallingthecursorAPIviaHTTPPOST/_api/cursor.

TheHTTPRESTAPIPOST/_api/cursordoesnotacceptabatchSizeparametervalueof0anylonger.Abatchsizeof0nevermademuchsense,butpreviousversionsofArangoDBdidnotcheckforthisvalue.NowcreatingacursorusingabatchSizevalue0willresultinanHTTP400errorresponse.

DocumentURLsreturned

TheRESTAPImethodGET/_api/document?collection=...(thatmethodwillreturnpartialURLstoalldocumentsinthecollection)willnowproperlyprefixdocumentaddressURLswiththecurrentdatabasename.

PreviousversionsofArangoDBreturnedtheURLsstartingwith/_api/butwithoutthecurrentdatabasename,e.g./_api/document/mycollection/mykey.Startingwith2.6,theresponseURLswillincludethedatabasenameaswell,e.g./_db/_system/_api/document/mycollection/mykey.

Fulltextindexing

Fulltextindexeswillnowalsoindextextvaluescontainedindirectsub-objectsoftheindexedattribute.

PreviousversionsofArangoDBonlyindexedtheattributevalueifitwasastring.Sub-attributesoftheindexattributewereignoredwhenfulltextindexing.

Now,iftheindexattributevalueisanobject,theobject'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.Iftheindexattributevalueisanarray,thearray'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.

Deprecatedserverfunctionality

Simplequeries

Thefollowingsimplequeryfunctionsarenowdeprecated:

collection.nearcollection.withincollection.geocollection.fulltextcollection.rangecollection.closedRange

ThisalsoleadtothefollowingRESTAPImethodsbeingdeprecatedfromnowon:

PUT/_api/simple/nearPUT/_api/simple/withinPUT/_api/simple/fulltextPUT/_api/simple/range

ItisrecommendedtoreplacecallstothesefunctionsorAPIswithequivalentAQLqueries,whicharemoreflexiblebecausetheycanbecombinedwithotheroperations:

FORdocINNEAR(@@collection,@latitude,@longitude,@limit)

RETURNdoc

FORdocINWITHIN(@@collection,@latitude,@longitude,@radius,@distanceAttributeName)

RETURNdoc

Incompatiblechangesin2.6

586

FORdocINFULLTEXT(@@collection,@attributeName,@queryString,@limit)

RETURNdoc

FORdocIN@@collection

FILTERdoc.value>=@left&&doc.value<@right

LIMIT@skip,@limit

RETURNdoc`

TheabovesimplequeryfunctionsandRESTAPImethodsmayberemovedinfutureversionsofArangoDB.

UsingnegativevaluesforSimpleQuery.skip()isalsodeprecated.ThisfunctionalitywillberemovedinfutureversionsofArangoDB.

AQLfunctions

TheAQLSKIPLISTfunctionhasbeendeprecatedbecauseitisobsolete.

ThefunctionwasintroducedinolderversionsofArangoDBwithalesspowerfulqueryoptimizertoretrievedatafromaskiplistindexusingaLIMITclause.

Since2.3thesamegoalcanbeachievedbyusingregularAQLconstructs,e.g.

FORdocIN@@collection

FILTERdoc.value>=@value

SORTdoc.value

LIMIT1

RETURNdoc

Startupoptionchanges

Optionsadded

Thefollowingconfigurationoptionshavebeenaddedin2.6:

--server.session-timeout:allowscontrollingthetimeoutofusersessionsinthewebinterface.Thevalueisspecifiedinseconds.

--server.foxx-queues:controlswhethertheFoxxqueuemanagerwillcheckqueueandjobentries.DisablingthisoptioncanreduceserverloadbutwillpreventjobsaddedtoFoxxqueuesfrombeingprocessedatall.

Thedefaultvalueistrue,enablingtheFoxxqueuesfeature.

--server.foxx-queues-poll-interval:allowsadjustingthefrequencywithwhichtheFoxxqueuesmanagerischeckingthequeue(orqueues)forjobstobeexecuted.

Thedefaultvalueis1second.Loweringthisvaluewillresultinthequeuemanagerwakingupandcheckingthequeuesmorefrequently,whichmayincreaseCPUusageoftheserver.

Note:thisoptiononlyhasaneffectwhen--server.foxx-queuesisnotsettofalse.

Optionsremoved

Thefollowingconfigurationoptionshavebeenremovedin2.6.:

--log.severity:thedocsfor--log.severitymentionedlotsofseverities(e.g.exception,technical,functional,development)butonlyafewseverities(e.g.all,human)wereactuallyused,withhumanbeingthedefaultandallenablingtheadditionalloggingofincomingrequests.

Theoptionpretendedtocontrolalotofthingswhichitactuallydidn't.Additionally,theoption--log.requests-filewasaroundforalongtimealready,alsocontrollingrequestlogging.

Becausethe--log.severityoptioneffectivelydidnotcontrolthatmuch,itwasremoved.Asideeffectofremovingtheoptionisthat2.5installationsstartedwithoption--log.severityallwillnotlogrequestsaftertheupgradeto2.6.Thiscanbeadjustedbysettingthe--log.requests-fileoptioninstead.

Incompatiblechangesin2.6

587

Defaultvalueschanged

Thedefaultvaluesforthefollowingoptionshavechangedin2.6:

--database.ignore-datafile-errors:thedefaultvalueforthisoptionwaschangedfromtruetofalse.

Ifthenewdefaultvalueoffalseisused,thenarangodwillrefuseloadingcollectionsthatcontaindatafileswithCRCmismatchesorothererrors.Acollectionwithdatafileerrorswillthenbecomeunavailable.Thispreventsfollowuperrorsfromhappening.

Theonlywaytoaccesssuchcollectionistousethedatafiledebugger(arango-dfdb)andtrytorepairortruncatethedatafilewithit.

--server.request-timeout:thedefaultvaluewasincreasedfrom300to1200secondsforallclienttools(arangosh,arangoimp,arangodump,arangorestore).

--server.connect-timeout:thedefaultvaluewasincreasedfrom3to5secondsforallclienttools(arangosh,arangoimp,arangodump,arangorestore).

Incompatiblechangesin2.6

588

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.5.ArangoDB2.5alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.

V8versionupgrade

Thebuilt-inversionofV8hasbeenupgradedfrom3.29.54to3.31.74.1.ThisallowsactivatingadditionalES6(alsodubbedHarmonyorES.next)featuresinArangoDB,bothintheArangoShellandtheArangoDBserver.Theycanbeusedforscriptingandinserver-sideactionssuchasFoxxroutes,traversalsetc.

ThefollowingadditionalES6featuresbecomeavailableinArangoDB2.5bydefault:

iteratorsandgeneratorstemplatestringsenhancedobjectliteralsenhancednumericliteralsblockscopingwithletandconstantvariablesusingconst(note:constantvariablesrequireusingstrictmode,too)additionalstringmethods(suchasstartsWith,repeatetc.)

Indeximprovements

Sparsehashandskiplistindexes

Hashandskiplistindexescanoptionallybemadesparse.Sparseindexesexcludedocumentsinwhichatleastoneoftheindexattributesiseithernotsetorhasavalueofnull.

Assuchdocumentsareexcludedfromsparseindexes,theymaycontainfewerdocumentsthantheirnon-sparsecounterparts.Thisenablesfasterindexingandcanleadtoreducedmemoryusageincasetheindexedattributedoesoccuronlyinsome,butnotalldocumentsofthecollection.Sparseindexeswillalsoreducethenumberofcollisionsinnon-uniquehashindexesincasenon-existingoroptionalattributesareindexed.

Inordertocreateasparseindex,anobjectwiththeattributesparsecanbeaddedtotheindexcreationcommands:

db.collection.ensureHashIndex(attributeName,{sparse:true});

db.collection.ensureHashIndex(attributeName1,attributeName2,{sparse:true});

db.collection.ensureUniqueConstraint(attributeName,{sparse:true});

db.collection.ensureUniqueConstraint(attributeName1,attributeName2,{sparse:true});

db.collection.ensureSkiplist(attributeName,{sparse:true});

db.collection.ensureSkiplist(attributeName1,attributeName2,{sparse:true});

db.collection.ensureUniqueSkiplist(attributeName,{sparse:true});

db.collection.ensureUniqueSkiplist(attributeName1,attributeName2,{sparse:true});

Notethatinplaceoftheabovespecializedindexcreationcommands,itisrecommendedtousethemoregeneralindexcreationcommandensureIndex:

db.collection.ensureIndex({type:"hash",sparse:true,unique:true,fields:[attributeName]});

db.collection.ensureIndex({type:"skiplist",sparse:false,unique:false,fields:["a","b"]});

Whennotexplicitlyset,thesparseattributedefaultstofalsefornewhashorskiplistindexes.

Thiscausesachangeinbehaviorwhencreatingauniquehashindexwithoutspecifyingthesparseflag:in2.4,uniquehashindexeswereimplicitlysparse,alwaysexcludingnullvalues.Therewasnooptiontocontrolthisbehavior,andsparsitywasneithersupportedfornon-uniquehashindexesnorskiplistsin2.4.Thisimplicitsparsityofuniquehashindexeswasconsideredaninconsistency,andthereforethebehaviorwascleanedupin2.5.Asof2.5,indexeswillonlybecreatedsparseifsparsityisexplicitlyrequested.Existinguniquehashindexesfrom2.4orbeforewillautomaticallybemigratedsotheyarestillsparseaftertheupgradeto2.5.

WhatsNewin2.5

589

Geoindexesareimplicitlysparse,meaningdocumentswithouttheindexedlocationattributeorcontaininginvalidlocationcoordinatevalueswillbeexcludedfromtheindexautomatically.Thisisalsoachangewhencomparedtopre-2.5behavior,whendocumentswithmissingorinvalidcoordinatevaluesmayhavecausederrorsoninsertionwhenthegeoindex'uniqueflagwassetanditsignoreNullflagwasnot.Thiswasconfusingandhasbeenrectifiedin2.5.ThemethodensureGeoConstraint()nowdoesthesameasensureGeoIndex().Furthermore,theattributesconstraint,unique,ignoreNullandsparseflagsarenowcompletelyignoredwhencreatinggeoindexes.

Thesameistrueforfulltextindexes.Thereisnoneedtospecifynon-uniquenessorsparsityforgeoorfulltextindexes.

Assparseindexesmayexcludesomedocuments,theycannotbeusedforeverytypeofquery.Sparsehashindexescannotbeusedtofinddocumentsforwhichatleastoneoftheindexedattributeshasavalueofnull.Forexample,thefollowingAQLquerycannotuseasparseindex,evenifonewascreatedonattributeattr:

FORdocIncollection

FILTERdoc.attr==null

RETURNdoc

Ifthelookupvalueisnon-constant,asparseindexmayormaynotbeused,dependingontheothertypesofconditionsinthequery.Iftheoptimizercansafelydeterminethatthelookupvaluecannotbenull,asparseindexmaybeused.Whenuncertain,theoptimizerwillnotmakeuseofasparseindexinaqueryinordertoproducecorrectresults.

Forexample,thefollowingqueriescannotuseasparseindexonattrbecausetheoptimizerwillnotknowbeforehandwhetherthecomparisonvaluesfordoc.attrwillincludenull:

FORdocIncollection

FILTERdoc.attr==SOME_FUNCTION(...)

RETURNdoc

FORotherINotherCollection

FORdocIncollection

FILTERdoc.attr==other.attr

RETURNdoc

Sparseskiplistindexescanbeusedforsortingiftheoptimizercansafelydetectthattheindexrangedoesnotincludenullforanyoftheindexattributes.

Selectivityestimates

Indexesoftypeprimary,edgeandhashnowprovideselectivityestimates.ThesewillbeusedbytheAQLqueryoptimizerwhendecidingaboutindexusage.Usingselectivityestimatescanleadtofasterqueryexecutionwhenmoreselectiveindexesareused.

TheselectivityestimatesarealsoreturnedbytheGET/_api/indexRESTAPImethodinasub-attributeselectivityEstimateforeachindexthatsupportsit.Thisattributewillbeomittedforindexesthatdonotprovideselectivityestimates.Ifprovided,theselectivityestimatewillbeanumericvaluebetween0and1.

Selectivityestimateswillalsobereportedintheresultofcollection.getIndexes()forallindexesthatsupportthis.Ifnoselectivityestimatecanbedeterminedforanindex,theattributeselectivityEstimatewillbeomittedhere,too.

Thewebinterfacealsoshowsselectivityestimatesforeachindexthatsupportsthis.

Currentlythefollowingindextypescanprovideselectivityestimates:

primaryindexedgeindexhashindex(uniqueandnon-unique)

Noselectivityestimateswillbeprovidedforindexeswhenrunninginclustermode.

AQLOptimizerimprovements

Sortremoval

WhatsNewin2.5

590

TheAQLoptimizerrule"use-index-for-sort"willnowremovesortsalsoincaseanon-sortedindex(e.g.ahashindex)isusedforonlyequalitylookupsandallsortattributesarecoveredbytheequalitylookupconditions.

Forexample,inthefollowingquerytheextrasortondoc.valuewillbeoptimizedawayprovidedthereisanindexondoc.value):

FORdocINcollection

FILTERdoc.value==1

SORTdoc.value

RETURNdoc

TheAQLoptimizerrule"use-index-for-sort"nowalsoremovessortincasethesortcriteriaexcludestheleft-mostindexattributes,buttheleft-mostindexattributesareusedbytheindexforequality-onlylookups.

Forexample,inthefollowingquerywithaskiplistindexonvalue1,value2,thesortcanbeoptimizedaway:

FORdocINcollection

FILTERdoc.value1==1

SORTdoc.value2

RETURNdoc

Constantattributepropagation

ThenewAQLoptimizerrulepropagate-constant-attributeswilllookforattributesthatareequality-comparedtoaconstantvalue,andwillpropagatethecomparisonvalueintootherequalitylookups.ThisrulewillonlylookinsideFILTERconditions,andinsertconstantvaluesfoundinFILTERs,too.

Forexample,therulewillinsert42insteadofi.valueinthesecondFILTERofthefollowingquery:

FORiINc1

FORjINc2

FILTERi.value==42

FILTERj.value==i.value

RETURN1

Interleavedprocessing

TheoptimizerwillnowinspectAQLdata-modificationqueriesanddetectifthequery'sdata-modificationpartcanruninlockstepwiththedataretrievalpartofthequery,orifthedataretrievalpartmustbeexecutedandcompletedfirstbeforethedata-modificationcanstart.

Executingbothdataretrievalanddata-modificationinlockstepallowsusingmuchsmallerbuffersforintermediateresults,reducingthememoryusageofqueries.Notallqueriesareeligibleforthisoptimization,andtheoptimizerwillonlyapplytheoptimizationwhenitcansafelydetectthatthedata-modificationpartofthequerywillnotmodifydatatobefoundbytheretrievalpart.

Queryexecutionstatistics

ThefilteredattributewasaddedtoAQLqueryexecutionstatistics.ThevalueofthisattributeindicateshowmanydocumentswerefilteredbyFilterNodesintheAQLquery.NotethatIndexRangeNodescanalsofilterdocumentsbyselectingonlytherequiredrangesfromtheindex.ThefilteredvaluewillnotincludetheworkdonebyIndexRangeNodes,butonlytheworkperformedbyFilterNodes.

Languageimprovements

DynamicattributenamesinAQLobjectliterals

ThischangeallowsusingarbitraryexpressionstoconstructattributenamesinobjectliteralsspecifiedinAQLqueries.Todisambiguateexpressionsandotherunquotedattributenames,dynamicattributenamesneedtobeenclosedinbrackets([and]).

Example:

WhatsNewin2.5

591

FORiIN1..100

RETURN{[CONCAT('value-of-',i)]:i}

AQLfunctions

ThefollowingAQLfunctionswereaddedin2.5:

MD5(value):generatesanMD5hashofvalueSHA1(value):generatesanSHA1hashofvalueRANDOM_TOKEN(length):generatesarandomstringvalueofthespecifiedlength

SimplifyFoxxusage

ThankstoouruserfeedbackwelearnedthatFoxxisapowerful,yetrathercomplicatedconcept.With2.5wemadeitlesscomplicatedwhilekeepingallitsstrength.Thatincludesarewriteofthedocumentationaswellassomecodechangesasfollows:

MovedFoxxapplicationstoadifferentfolder.

Until2.4foxxappswerestoredinthefollowingfolderstructure:<app-path>/databases/<dbname>/<appname>:<appversion>.Thiscausedsometroubleasappswherecachedbasedonnameandversionandupdatesdidnotapply.Alsothepathonfilesystemandtheapp'saccessURLhadnorelationtooneanother.NowthepathonfilesystemisidenticaltotheURL(excepttheappendedAPP):<app-path>/_db/<dbname>/<mointpoint>/APP

RewriteofFoxxrouting

TheroutingofFoxxhasbeenexposedtomajorinternalchangesweadjustedbecauseofuserfeedback.Thisallowsustosetthedevelopmentmodepermountpointwithouthavingtochangepathsandholdappsatseparatelocations.

FoxxDevelopmentmode

Thedevelopmentmodeuseduntil2.4isgone.Ithasbeenreplacedbyamuchmorematureversion.Thisincludesthedeprecationofthejavascript.dev-app-pathparameter,whichisuselesssince2.5.Insteadofhavingtwoseparateappdirectoriesforproductionanddevelopment,appsnowresideinoneplace,whichisusedforproductionaswellasfordevelopment.Appscanstillbeputintodevelopmentmode,changingtheirbehaviorcomparedtoproductionmode.Developmentmodeappsarestillrereadfromdiskateveryrequest,andstilltheyshipmoredebugoutput.

Thischangehasalsomadethestartupoptions--javascript.frontend-development-modeand--javascript.dev-app-pathobsolete.Theformeroptionwillnothaveanyeffectwhenset,andthelatteroptionisonlyreadandusedduringtheupgradeto2.5anddoesnothaveanyeffectslater.

Foxxinstallprocess

InstallingFoxxappshasbeenatwostepprocess:importthemintoArangoDBandmountthemataspecificmountpoint.Theseoperationshavebeenjoinedtogether.Youcaninstallanappatonemountpoint,that'sit.Nofetch,mount,unmount,purgecycleanymore.Thecommandshavebeensimplifiedtojust:

install:getyourFoxxappupandrunninguninstall:shutitdownanderaseitfromdisk

Foxxerroroutput

Until2.4theerrorsproducedbyFoxxwerenotoptimal.Often,theerrormessagewasjustunabletoparsemanifestandcontainedonlyaninternalstacktrace.In2.5wemademajorimprovementsthere,includingamuchmorefinegrainederroroutputthathelpsyoudebugyourFoxxapps.Theerrormessageprintedisnowmuchclosertoitssourceandshouldhelpyoutrackitdown.

AlsoweaddedthedefaulthandlersforunhandlederrorsinFoxxapps:

YouwillgetaniceinternalerrorpagewheneveryourFoxxappiscalledbutwasnotinstalledduetoanyerror

WhatsNewin2.5

592

Youwillgetapropererrormessagewhenhavinganuncaughterrorappearsinanyapproute

InproductionmodethemessagesabovewillNOTcontainanyinformationaboutyourFoxxinternalsandaresafetobeexposedtothirdpartyusers.Indevelopmentmodethemessagesabovewillcontainthestacktrace(ifavailable),makingiteasierforyourin-housedevstotrackdownerrorsintheapplication.

Foxxconsole

WeaddedaconsoleobjecttoFoxxapps.AllFoxxappsnowhaveaconsoleobjectimplementingthefamiliarConsoleAPIintheirglobalscope,whichcanbeusedtologdiagnosticmessagestothedatabase.Thisconsolealsoallowstoreadtheerroroutputofonespecificfoxx.

Foxxrequests

Weaddedorg/arangodb/requestmodule,whichprovidesasimpleAPIformakingHTTPrequeststoexternalservices.ThisisenablesFoxxtobedirectlypartofamicroservicearchitecture.

WhatsNewin2.5

593

IncompatiblechangesinArangoDB2.5ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.5,andadjustanyclientprogramsifnecessary.

Changedbehavior

V8

TheV8versionshippedwithArangoDBwasupgradedfrom3.29.59to3.31.74.1.ThisleadstoadditionalECMAScript6(ES6or"harmony")featuresbeingenabledbydefaultinArangoDB'sscriptingenvironment.

Apartfromthat,achangeintheinterpretationofcommand-lineoptionsbyV8mayaffectusers.ArangoDBpassesthevalueofthecommand-lineoption--javascript.v8-optionstoV8andleavesinterpretationofthecontentstoV8.Forexample,theArangoDBoption--javascript.v8-options="--harmony"couldbeusedtotellV8toenableitsharmonyfeatures.

InArangoDB2.4,thefollowingharmonyoptionsweremadeavailablebyV8:

--harmony_scoping(enableharmonyblockscoping)--harmony_modules(enableharmonymodules(impliesblockscoping))--harmony_proxies(enableharmonyproxies)--harmony_generators(enableharmonygenerators)--harmony_numeric_literals(enableharmonynumericliterals(0o77,0b11))--harmony_strings(enableharmonystring)--harmony_arrays(enableharmonyarrays)--harmony_arrow_functions(enableharmonyarrowfunctions)--harmony_classes(enableharmonyclasses)--harmony_object_literals(enableharmonyobjectliteralextensions)--harmony(enableallharmonyfeatures(exceptproxies))

Therewastheoption--harmony,whichturnedonalmostallharmonyfeatures.

InArangoDB2.5,V8providesthefollowingharmony-relatedoptions:

--harmony(enableallcompletedharmonyfeatures)--harmony_shipping(enableallshippedharmonyfeatures)--harmony_modules(enable"harmonymodules(impliesblockscoping)"(inprogress))--harmony_arrays(enable"harmonyarraymethods"(inprogress))--harmony_array_includes(enable"harmonyArray.prototype.includes"(inprogress))--harmony_regexps(enable"harmonyregularexpressionextensions"(inprogress))--harmony_arrow_functions(enable"harmonyarrowfunctions"(inprogress))--harmony_proxies(enable"harmonyproxies"(inprogress))--harmony_sloppy(enable"harmonyfeaturesinsloppymode"(inprogress))--harmony_unicode(enable"harmonyunicodeescapes"(inprogress))--harmony_tostring(enable"harmonytoString")--harmony_numeric_literals(enable"harmonynumericliterals")--harmony_strings(enable"harmonystringmethods")--harmony_scoping(enable"harmonyblockscoping")--harmony_classes(enable"harmonyclasses(impliesblockscoping&objectliteralextension)")--harmony_object_literals(enable"harmonyobjectliteralextensions")--harmony_templates(enable"harmonytemplateliterals")

Notethatthereareextraoptionsforbettercontrollingthededicatedfeatures,andespeciallythatthemeaningofthe--harmonyoptionhaschangedfromenablingallharmonyfeaturestoallcompletedharmonyfeatures!

Usersshouldadjustthevalueof--javascript.v8-optionsaccordingly.

PleasenotethatincompleteharmonyfeaturesaresubjecttochangeinfutureV8releases.

Incompatiblechangesin2.5

594

Sparseindexes

Hashindexesandskiplistindexescannowbecreatedinasparsevariant.Whennotexplicitlyset,thesparseattributedefaultstofalsefornewindexes.

Thiscausesachangeinbehaviorwhencreatingauniquehashindexwithoutspecifyingthesparseflag.Theuniquehashindexwillbecreatedinanon-sparsevariantinArangoDB2.5.

In2.4andbefore,uniquehashindexeswereimplicitlysparse,alwaysexcludingnullvaluesfromtheindex.Therewasnooptiontocontrolthisbehavior,andsparsitywasneithersupportedfornon-uniquehashindexesnorskiplistsin2.4.Thisimplicitsparsityofjustuniquehashindexeswasconsideredaninconsistency,andthereforethebehaviorwascleanedupin2.5.

Asof2.5,hashandskiplistindexeswillonlybecreatedsparseifsparsityisexplicitlyrequested.Thismayrequireachangeinindex-creatingclientcode,butonlyiftheclientcodecreatesuniquehashindexesandiftheyarestillintendedtobesparse.Inthiscase,theclientcodeshouldexplicitlysetthesparseflagtotruewhencreatingauniquehashindex.

Existinguniquehashindexesfrom2.4orbeforewillautomaticallybemigratedsotheyarestillsparseaftertheupgradeto2.5.Fortheseindexes,thesparseattributewillbepopulatedautomaticallywithavalueoftrue.

Geoindexesareimplicitlysparse,meaningdocumentswithouttheindexedlocationattributeorcontaininginvalidlocationcoordinatevalueswillbeexcludedfromtheindexautomatically.Thisisalsoachangewhencomparedtopre-2.5behavior,whendocumentswithmissingorinvalidcoordinatevaluesmayhavecausederrorsoninsertionwhenthegeoindex'uniqueflagwassetanditsignoreNullflagwasnot.

Thiswasconfusingandhasbeenrectifiedin2.5.ThemethodensureGeoConstraint()nowdoesthesameasensureGeoIndex().Furthermore,theattributesconstraint,unique,ignoreNullandsparseflagsarenowcompletelyignoredwhencreatinggeoindexes.ClientindexcreationcodethereforedoesnotneedtosettheignoreNullorconstraintattributeswhencreatingageoindex.

Thesameistrueforfulltextindexes.Thereisnoneedtospecifynon-uniquenessorsparsityforgeoorfulltextindexes.Theywillalwaysbenon-uniqueandsparse.

MovedFoxxapplicationstoadifferentfolder.

Until2.4foxxappswerestoredinthefollowingfolderstructure:<app-path>/databases/<dbname>/<appname>:<appversion>.Thiscausedsometroubleasappswherecachedbasedonnameandversionandupdatesdidnotapply.Alsothepathonfilesystemandtheapp'saccessURLhadnorelationtooneanother.NowthepathonfilesystemisidenticaltotheURL(excepttheappendedAPP):<app-path>/_db/<dbname>/<mointpoint>/APP

FoxxDevelopmentmode

Thedevelopmentmodeuseduntil2.4isgone.Ithasbeenreplacedbyamuchmorematureversion.Thisincludesthedeprecationofthejavascript.dev-app-pathparameter,whichisuselesssince2.5.Insteadofhavingtwoseparateappdirectoriesforproductionanddevelopment,appsnowresideinoneplace,whichisusedforproductionaswellasfordevelopment.Appscanstillbeputintodevelopmentmode,changingtheirbehaviorcomparedtoproductionmode.Developmentmodeappsarestillrereadfromdiskateveryrequest,andstilltheyshipmoredebugoutput.

Thischangehasalsomadethestartupoptions--javascript.frontend-development-modeand--javascript.dev-app-pathobsolete.Theformeroptionwillnothaveanyeffectwhenset,andthelatteroptionisonlyreadandusedduringtheupgradeto2.5anddoesnothaveanyeffectslater.

Foxxinstallprocess

InstallingFoxxappshasbeenatwostepprocess:importthemintoArangoDBandmountthemataspecificmountpoint.Theseoperationshavebeenjoinedtogether.Youcaninstallanappatonemountpoint,that'sit.Nofetch,mount,unmount,purgecycleanymore.Thecommandshavebeensimplifiedtojust:

install:getyourFoxxappupandrunninguninstall:shutitdownanderaseitfromdiskDeprecatedfeatures

Foxx:methodModel#toJSONSchema(id)isdeprecated,itwillraiseawarningifyouuseit.PleaseuseFoxx.toJSONSchema(id,model)

Incompatiblechangesin2.5

595

instead.

Removedfeatures

Startupswitch--javascript.frontend-development-mode:Itsmajorpurposewasinternaldevelopmentanyway.Nowthewebfrontendcanbesettodevelopmentmodesimilartoanyotherfoxxapp.Startupswitch--javascript.dev-app-path:WasusedforthedevelopmentmodeofFoxx.Thisisintegratedwiththenormalapp-pathnowandcanbetriggeredonapplevel.Thesecondapp-pathissuperfluous.Foxx:controller.collection:PleaseuseappContext.collectioninstead.Foxx:FoxxRepository.modelPrototype:PleaseuseFoxxRepository.modelinstead.Foxx:Model.extend({},{attributes:{}}):PleaseuseModel.extend({schema:{}})instead.Foxx:requestContext.bodyParam(paramName,description,Model):PleaseuserequestContext.bodyParam(paramName,options)instead.Foxx:requestContext.queryParam({type:string}):PleaseuserequestContext.queryParam({type:joi})instead.Foxx:requestContext.pathParam({type:string}):PleaseuserequestContext.pathParam({type:joi})instead.Graph:Themodulesorg/arangodb/graphandorg/arangodb/graph-blueprint:Pleaseusemoduleorg/arangodb/general-graphinstead.NOTE:Thisdoesnotmeanwedonotsupportblueprintsanymore.Generalgraphcoverseverythingthegraph--blueprintdid,plusmanymorefeatures.General-Graph:Inthemoduleorg/arangodb/general-graphthefunctions_undirectedRelationand_directedRelationarenolongeravailable.Bothfunctionshavebeenunifiedto_relation.

Incompatiblechangesin2.5

596

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.4.ArangoDB2.4alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.

V8versionupgrade

Thebuilt-inversionofV8hasbeenupgradedfrom3.16.14to3.29.59.ThisactivatesseveralES6(alsodubbedHarmonyorES.next)featuresinArangoDB,bothintheArangoShellandtheArangoDBserver.Theycanbeusedforscriptingandinserver-sideactionssuchasFoxxroutes,traversalsetc.

ThefollowingES6featuresareavailableinArangoDB2.4bydefault:

iteratorstheofoperatorsymbolspredefinedcollectionstypes(Map,Setetc.)typedarrays

ManyotherES6featuresaredisabledbydefault,butcanbemadeavailablebystartingarangodorarangoshwiththeappropriateoptions:

arrowfunctionsproxiesgeneratorsString,Array,andNumberenhancementsconstantsenhancedobjectandnumericliterals

ToactivatealltheseES6featuresinarangodorarangosh,startitwiththefollowingoptions:

arangosh--javascript.v8-options="--harmony--harmony_generators"

MoredetailsontheavailableES6featurescanbefoundinthisblog.

FoxxGeneratorArangoDB2.4isshippedwithFoxxGenerator,aframeworkforbuildingstandardizedHypermediaAPIseasily.ThegeneratedAPIscanbeconsumedwithclienttoolsthatunderstandSiren.

HypermediaisthesimpleideathatourHTTPAPIsshouldhavelinksbetweentheirendpointsinthesamewaythatourwebsiteshavelinksbetweenthem.FoxxGeneratorisbasedontheideathatyoucanrepresentanAPIasastatechart:Everyendpointisastateandthelinksarethetransitionsbetweenthem.Usingyourdescriptionofstatesandtransitions,itcanthencreateanAPIforyou.

TheFoxxGeneratorcancreateAPIsbasedonasemanticdescriptionofentitiesandtransitions.AblogseriesontheusecasesandhowtousetheFoxxgeneratorishere:

part1part2part3

AcookbookrecipeforgettingstartedwithFoxxGeneratorishere.

AQLimprovements

Optimizerimprovements

WhatsNewin2.4

597

TheAQLoptimizerhasbeenenhancedtouseofindexesinqueriesinseveraladditionalcases.FilterscontainingtheINoperatorcannowmakeuseofindexes,andmultipleOR-orAND-combinedfilterconditionscannowalsouseindexesifthefilterconditionsrefertothesameindexedattribute.

Hereareafewexamplesofqueriesthatcannowuseindexesbutcouldn'tbefore:

FORdocINcollection

FILTERdoc.indexedAttribute==1||doc.indexedAttribute>99

RETURNdoc

FORdocINcollection

FILTERdoc.indexedAttributeIN[3,42]||doc.indexedAttribute>99

RETURNdoc

FORdocINcollection

FILTER(doc.indexedAttribute>2&&doc.indexedAttribute<10)||

(doc.indexedAttribute>23&&doc.indexedAttribute<42)

RETURNdoc

Additionally,theoptimizerruleremove-filter-covered-by-indexhasbeenadded.ThisruleremovesFilterNodesandCalculationNodesfromanexecutionplanifthefilterconditionisalreadycoveredbyapreviousIndexRangeNode.Removingthefilter'sCalculationNodeandtheFilterNodeitselfwillspeedupqueryexecutionbecausethequeryrequireslesscomputation.

Furthermore,thenewoptimizerruleremove-sort-randwillremoveaSORTRAND()statementandmovetherandomiterationintotheappropriateEnumerateCollectionNode.Thisisusuallymoreefficientthanindividuallyenumeratingandsorting.

Data-modificationqueriesreturningdocuments

INSERT,REMOVE,UPDATEorREPLACEqueriesnowcanoptionallyreturnthedocumentsinserted,removed,updated,orreplaced.Thisishelpfulfortrackingtheauto-generatedattributes(e.g._key,_rev)createdbyanINSERTandinalotofothersituations.

Inordertoreturndocumentsfromadata-modificationquery,thestatementmustimmediatelybeimmediatelyfollowedbyaLETstatementthatassignseitherthepseudo-valueNEWorOLDtoavariable.ThisLETstatementmustbefollowedbyaRETURNstatementthatreturnsthevariableintroducedbyLET:

FORiIN1..100

INSERT{value:i}INtestLETinserted=NEWRETURNinserted

FORuINusers

FILTERu.status=='deleted'

REMOVEuINusersLETremoved=OLDRETURNremoved

FORuINusers

FILTERu.status=='notactive'

UPDATEuWITH{status:'inactive'}INusersLETupdated=NEWRETURNupdated

NEWreferstotheinsertedormodifieddocumentrevision,andOLDreferstothedocumentrevisionbeforeupdateorremoval.INSERTstatementscanonlyrefertotheNEWpseudo-value,andREMOVEoperationsonlytoOLD.UPDATEandREPLACEcanrefertoeither.

Inallcasesthefulldocumentswillbereturnedwithalltheirattributes,includingthepotentiallyauto-generatedattributessuchas_id,_key,or_revandtheattributesnotspecifiedintheupdateexpressionofapartialupdate.

Languageimprovements

COUNTclause

AnoptionalCOUNTclausewasaddedtotheCOLLECTstatement.TheCOUNTclauseallowsformoreefficientcountingofvalues.

InpreviousversionsofArangoDBonehadtowritethefollowingtocountdocuments:

RETURNLENGTH(

FORdocINcollection

FILTER...somecondition...

RETURNdoc

)

WhatsNewin2.4

598

WiththeCOUNTclause,thequerycanbemodifiedto

FORdocINcollection

FILTER...somecondition...

COLLECTWITHCOUNTINTOlength

RETURNlength

ThelatterquerywillbemuchmoreefficientbecauseitwillnotproduceanyintermediateresultswithneedtobeshippedfromasubqueryintotheLENGTHfunction.

TheCOUNTclausecanalsobeusedtocountthenumberofitemsineachgroup:

FORdocINcollection

FILTER...somecondition...

COLLECTgroup=doc.groupWITHCOUNTINTOlength

return{group:group,length:length}

COLLECTmodifications

InArangoDB2.4,COLLECToperationscanbemademoreefficientifonlyasmallfragmentofthegroupvaluesisneededlater.Forthesecases,COLLECTprovidesanoptionalconversionexpressionfortheINTOclause.Thisexpressioncontrolsthevaluethatisinsertedintothearrayofgroupvalues.Itcanbeusedforprojections.

ThefollowingqueryonlycopiesthedateRegisteredattributeofeachdocumentintothegroups,potentiallysavingalotofmemoryandcomputationtimecomparedtocopyingdoccompletely:

FORdocINcollection

FILTER...somecondition...

COLLECTgroup=doc.groupINTOdates=doc.dateRegistered

return{group:group,maxDate:MAX(dates)}

Comparethistothefollowingvariantofthequery,whichwastheonlywaytoachievethesameresultinpreviousversionsofArangoDB:

FORdocINcollection

FILTER...somecondition...

COLLECTgroup=doc.groupINTOdates

return{group:group,maxDate:MAX(dates[*].doc.dateRegistered)}

Theabovequerywillneedtocopythefulldocattributeintothelengthsvariable,whereasthenewvariantwillonlycopythedateRegisteredattributeofeachdoc.

Subquerysyntax

InpreviousversionsofArangoDB,subqueriesrequiredextraparenthesesaroundthem,andthiscausedconfusionwhensubquerieswereusedasfunctionparameters.Forexample,thefollowingquerydidnotwork:

LETvalues=LENGTH(

FORdocINcollectionRETURNdoc

)

buthadtobewrittenasfollows:

LETvalues=LENGTH((

FORdocINcollectionRETURNdoc

))

Thiswasunintuitiveandisfixedinversion2.4sothatbothvariantsofthequeryareacceptedandproducethesameresult.

Webinterface

WhatsNewin2.4

599

TheApplicationstabforFoxxapplicationsinthewebinterfacehasgotacompleteredesign.

ItwillnowonlyshowapplicationsthatarecurrentlyrunninginArangoDB.Foraselectedapplication,anewdetailedviewhasbeencreated.Thisviewprovidesabetteroverviewoftheapp,e.g.:

authorlicenseversioncontributorsdownloadlinksAPIdocumentation

InstallinganewFoxxapplicationontheserverismadeeasyusingthenewAddapplicationbutton.TheAddapplicationdialogprovidesallthefeaturesalreadyavailableinthefoxx-managerconsoleapplicationplussomemore:

installaFoxxapplicationfromGithubinstallaFoxxapplicationfromazipfileinstallaFoxxapplicationfromArangoDB'sapplicationstorecreateanewFoxxapplicationfromscratch:thisfeatureusesageneratortocreateaFoxxapplicationwithpre-definedCRUDmethodsforagivenlistofcollections.ThegeneratedFoxxappcaneitherbedownloadedasazipfileorbeinstalledontheserver.StartingwithanewFoxxapphasneverbeeneasier.

Miscellaneousimprovements

Defaultendpointis127.0.0.1

ThedefaultendpointfortheArangoDBserverhasbeenchangedfrom0.0.0.0to127.0.0.1.ThiswillmakenewArangoDBinstallationsunaccessiblefromclientsotherthanlocalhostunlesstheconfigurationischanged.Thisisasecurityprecautionmeasurethathasbeenrequestedasafeaturealotoftimes.

Ifyouarethedevelopmentoption--enable-relative,theendpointwillstillbe0.0.0.0.

Systemcollectionsinreplication

Bydefault,systemcollectionsarenowincludedinreplicationandallreplicationAPIreturnvalues.Thiswillleadtouseraccountsandcredentialsdatabeingreplicatedfrommastertoslaveservers.Thismayoverwriteslave-specificdatabaseusers.

Ifthisisundesired,the_userscollectioncanbeexcludedfromreplicationeasilybysettingtheincludeSystemattributetofalseinthefollowingcommands:

replication.sync({includeSystem:false});replication.applier.properties({includeSystem:false});

Thiswillexcludeallsystemcollections(including_aqlfunctions,_graphsetc.)fromtheinitialsynchronizationandthecontinuousreplication.

Ifthisisalsoundesired,itisalsopossibletospecifyalistofcollectionstoexcludefromtheinitialsynchronizationandthecontinuousreplicationusingtherestrictCollectionsattribute,e.g.:

require("org/arangodb/replication").applier.properties({

includeSystem:true,

restrictType:"exclude",

restrictCollections:["_users","_graphs","foo"]

});

WhatsNewin2.4

600

IncompatiblechangesinArangoDB2.4ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.4,andadjustanyclientprogramsifnecessary.

Changedbehavior

V8upgrade

ThebundledV8versionhasbeenupgradedfrom3.16.14to3.29.59.

Thenewversionprovidesbettererrorchecking,whichcanleadtosubtlechangesintheexecutionofJavaScriptcode.

Thefollowingcode,thoughnonsense,runswithouterrorin2.3and2.4whenstrictmodeisnotenabled:

(function(){

a=true;

a.foo=1;

})();

Whenenablingstrictmode,thefunctionwillthrowanerrorin2.4butnotin2.3:

(function(){

"usestrict";

a=true;

a.foo=1;

})();

TypeError:Cannotassigntoreadonlyproperty'foo'oftrue

Thoughthisisachangeinbehavioritcanbeconsideredanimprovement.Thenewversionactuallyuncoversanerrorthatwentundetectedintheoldversion.

Errormessageshavealsochangedslightlyinthenewversion.ApplicationsthatrelyontheexacterrormessagesoftheJavaScriptenginemayneedtobeadjustedsotheylookfortheupdatederrormessages.

Defaultendpoint

Thedefaultendpointforarangodisnow127.0.0.1.

ThischangewillmodifytheIPaddressArangoDBlistensonto127.0.0.1bydefault.ThiswillmakenewArangoDBinstallationsunaccessiblefromclientsotherthanlocalhostunlesstheconfigurationischanged.Thisisasecurityfeature.

TomakeArangoDBaccessiblefromanyclient,changetheserver'sconfiguration(--server.endpoint)toeithertcp://0.0.0.0:8529ortheserver'spubliclyvisibleIPaddress.

Replication

SystemcollectionsarenowincludedinthereplicationandallreplicationAPIreturnvaluesbydefault.

Thiswillleadtouseraccountsandcredentialsdatabeingreplicatedfrommastertoslaveservers.Thismayoverwriteslave-specificdatabaseusers.

Thismaybeconsideredafeatureorananti-feature,soitisconfigurable.

Ifreplicationofsystemcollectionsisundesired,theycanbeexcludedfromreplicationbysettingtheincludeSystemattributetofalseinthefollowingcommands:

initialsynchronization:replication.sync({includeSystem:false})continuousreplication:replication.applier.properties({includeSystem:false})

Incompatiblechangesin2.4

601

Thiswillexcludeallsystemcollections(including_aqlfunctions,_graphsetc.)fromtheinitialsynchronizationandthecontinuousreplication.

Ifthisisalsoundesired,itisalsopossibletospecifyalistofcollectionstoexcludefromtheinitialsynchronizationandthecontinuousreplicationusingtherestrictCollectionsattribute,e.g.:

require("org/arangodb/replication").applier.properties({

includeSystem:true,

restrictType:"exclude",

restrictCollections:["_users","_graphs","foo"]

});

Theaboveexamplewillingeneralincludesystemcollections,butwillexcludethespecifiedthreecollectionsfromcontinuousreplication.

TheHTTPRESTAPImethodsforfetchingthereplicationinventoryandfordumpingcollectionsalsosupporttheincludeSystemcontrolflagviaaURLparameterofthesamename.

Buildprocesschanges

Severaloptionsfortheconfigurecommandhavebeenremovedin2.4.Theoptions

--enable-all-in-one-v8

--enable-all-in-one-icu

--enable-all-in-one-libev

--with-libev=DIR

--with-libev-lib=DIR

--with-v8=DIR

--with-v8-lib=DIR

--with-icu-config=FILE

arenotavailableanymorebecausethebuildprocesswillalwaysusethebundledversionsofthelibraries.

WhenbuildingArangoDBfromsourceinadirectorythatalreadycontainedapre-2.4version,itwillbenecessarytorunamakesupercleancommandonceandafullrebuildafterwards:

gitpull

makesuperclean

makesetup

./configure<optionsgohere>

make

MiscellaneouschangesAsaconsequenceofglobalrenaminginthecodebase,theoptionmergeArrayshasbeenrenamedtomergeObjects.ThisoptioncontrolswhetherJSONobjectswillbemergedonanupdateoperationoroverwritten.Thedefaulthasbeen,andstillis,tomerge.Notspecifyingtheparameterwillleadtoamerge,asithasbeenthebehaviorinArangoDBeversince.

ThisaffectstheHTTPRESTAPImethodPATCH/_api/document/collection/key.ItsoptionalURLparametermergeArraysfortheoptionhasbeenrenamedtomergeObjects.

TheAQLUPDATEstatementisalsoaffected,asitsoptionmergeArrayshasalsobeenrenamedtomergeObjects.The2.3query

UPDATEdocINcollectionWITH{...}INcollectionOPTIONS{mergeArrays:false}

shouldthusberewrittentothefollowingin2.4:

UPDATEdocINcollectionWITH{...}INcollectionOPTIONS{mergeObjects:false}

Deprecatedfeatures

Incompatiblechangesin2.4

602

ForFoxxControllerobjects,themethodcollection()isdeprecatedandwillberemovedinfutureversionofArangoDB.Usingthismethodwillissueawarning.PleaseuseapplicationContext.collection()instead.

ForFoxxRepositoryobjects,thepropertymodelPrototypeisnowdeprecated.Usingitwillissueawarning.PleaseuseFoxxRepository.modelinstead.

InFoxxController/RequestContext,callingmethodbodyParam()withthreeargumentsisdeprecated.Pleaseuse.bodyParam(paramName,options)instead.

InFoxxController/RequestContextcallingmethodqueryParam({type:string})isdeprecated.PleaseuserequestContext.queryParam({type:joi})instead.

InFoxxController/RequestContextcallingmethodpathParam({type:string})isdeprecated.PleaseuserequestContext.pathParam({type:joi})instead.

ForFoxxModel,callingModel.extend({},{attributes:{}})isdeprecated.PleaseuseModel.extend({schema:{}})instead.

Inmoduleorg/arangodb/general-graph,thefunctions_undirectedRelation()and_directedRelation()aredeprecatedandwillberemovedinafutureversionofArangoDB.Bothfunctionshavebeenunifiedto_relation().

Themodulesorg/arangodb/graphandorg/arangodb/graph-blueprintaredeprecated.Pleaseusemoduleorg/arangodb/general-graphinstead.

TheHTTPRESTAPI_api/graphandallitsmethodsaredeprecated.PleaseusethegeneralgraphAPI_api/gharialinstead.

Removedfeatures

Thefollowingreplication-relatedJavaScriptmethodsbecameobsoleteinArangoDB2.2andhavebeenremovedinArangoDB2.4:

require("org/arangodb/replication").logger.start()

require("org/arangodb/replication").logger.stop()

require("org/arangodb/replication").logger.properties()

TheRESTAPImethodsforthesefunctionshavealsobeenremovedinArangoDB2.4:

HTTPPUT/_api/replication/logger-startHTTPPUT/_api/replication/logger-stopHTTPGET/_api/replication/logger-configHTTPPUT/_api/replication/logger-config

Clientapplicationsthatcalloneofthesemethodsshouldbeadjustedbyremovingthecallstothesemethods.Thisshouldn'tbeproblematicasthesemethodshavebeenno-opssinceArangoDB2.2anyway.

Incompatiblechangesin2.4

603

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.3.ArangoDB2.3alsocontainsseveralbugfixesthatarenotlistedhere.

AQLimprovements

Frameworkimprovements

AQLqueriesarenowsentthroughaqueryoptimizerframeworkbeforeexecution.Thequeryoptimizerframeworkwillfirstconverttheinternalrepresentationofthequery,theabstractsyntaxtree,intoaninitialexecutionplan.

Theexecutionplanisthensendthroughoptimizerrulesthatmaydirectlymodifytheplaninplaceorcreateanewvariantoftheplan.Newplansmightagainbeoptimized,allowingtheoptimizertocarryoutseveraloptimizations.

Aftercreatingplans,theoptimizerwillestimatethecostsforeachplanandpicktheplanwiththelowestcost(termedtheoptimalplan)fortheactualqueryexecution.

Withtheexplain()methodofArangoStatementuserscancheckwhichexecutionplantheoptimizerpickorretrievealistofotherplansthatoptimizerdidnotchoose.Theplanwillrevealmanydetailsaboutwhichindexesareusedetc.explain()willalsoreturntheofoptimizerrulesappliedsouserscanvalidatewhetherornotaqueryallowsusingaspecificoptimization.

ExecutionofAQLquerieshasbeenrewritteninC++,allowingmanyqueriestoavoidtheconversionofdocumentsbetweenArangoDB'sinternallow-leveldatastructureandtheV8objectrepresentationformat.

Theframeworkforoptimizerrulesisnowalsogenerallycluster-aware,allowingspecificoptimizationsforqueriesthatruninacluster.Additionally,theoptimizerwasdesignedtobeextensibleinordertoaddmoreoptimizationsinthefuture.

Languageimprovements

Alternativeoperatorsyntax

ArangoDB2.3allowstousethefollowingalternativeformsforthelogicaloperators:

AND:logicalandOR:logicalorNOT:negation

Thisnewsyntaxisjustanalternativetotheoldsyntax,allowingeasiermigrationfromSQL.Theoldsyntaxisstillfullysupportedandwillbe:

&&:logicaland||:logicalor!:negation

NOTINoperator

AQLnowhasadedicatedNOTINoperator.

Previously,aNOTINwasonlyachievablebywritinganegatedINcondition:

FORiIN...FILTER!(iIN[23,42])...

InArangoDB2.3,thesameresultcannowalternativelybeachievedbywritingthemoreintuitivevariant:

FORiIN...FILTERiNOTIN[23,42]...

Improvementsofbuilt-infunctions

WhatsNewin2.3

604

ThefollowingAQLstringfunctionshavebeenadded:

LTRIM(value,characters):left-trimsastringvalueRTRIM(value,characters):right-trimsastringvalueFIND_FIRST(value,search,start,end):findsthefirstoccurrenceofasearchstringFIND_LAST(value,search,start,end):findsthelastoccurrenceofasearchstringSPLIT(value,separator,limit):splitsastringintoanarray,usingaseparatorSUBSTITUTE(value,search,replace,limit):replacescharactersorstringsinsideanother

ThefollowingotherAQLfunctionshavebeenadded:

VALUES(document):returnsthevaluesofanobjectasanarray(thisisthecounterparttothealreadyexistingATTRIBUTESfunction)ZIP(attributes,values):returnsanobjectconstructedfromattributesandvaluespassedinseparateparametersPERCENTILE(values,n,method):returnsthenthspercentileofthevaluesprovided,usingrankorinterpolationmethod

ThealreadyexistingfunctionsCONCATandCONCAT_SEPARATORnowsupportarrayarguments,e.g.:

/*"foobarbaz"*/

CONCAT(['foo','bar','baz'])

/*"foo,bar,baz"*/

CONCAT_SEPARATOR(",",['foo','bar','baz'])

AQLqueriesthrowlessexceptions

InpreviousversionsofArangoDB,AQLqueriesabortedwithanexceptioninmanysituationsandthrewaruntimeexception.Forexample,exceptionswerethrownwhentryingtofindavalueusingtheINoperatorinanon-arrayelement,whentryingtousenon-booleanvalueswiththelogicaloperands&&or||or!,whenusingnon-numericvaluesinarithmeticoperations,whenpassingwrongparametersintofunctionsetc.

ThefactthatmanyAQLoperatorscouldthrowexceptionsledtoalotofquestionsfromusers,andalotofmore-verbose-than-necessaryqueries.Forexample,thefollowingqueryfailedwhenthereweredocumentsthatdidnothaveatopicsattributeatall:

FORdocINmycollection

FILTERdoc.topicsIN["something","whatever"]

RETURNdoc

Thisforceduserstorewritethequeryasfollows:

FORdocINmycollection

FILTERIS_LIST(doc.topics)&&doc.topicsIN["something","whatever"]

RETURNdoc

InArangoDB2.3thishasbeenchangedtomakeAQLeasiertouse.Thechangeprovidesanextrabenefit,andthatisthatnon-throwingoperatorsallowthequeryoptimizertoperformmuchmoretransformationsinthequerywithoutchangingitsoverallresult.

Hereisasummaryofchanges:

whenanon-arrayvalueisusedontheright-handsideoftheINoperator,theresultwillbefalseinArangoDB2.3,andnoexceptionwillbethrown.thebooleanoperators&&and||donotthrowinArangoDB2.3ifanyoftheoperandsisnotabooleanvalue.Instead,theywillperformanimplicitcastofthevaluestobooleans.Theirresultwillbeasfollows:

lhs&&rhswillreturnlhsifitisfalseorwouldbefalsewhenconvertedintoaboolean.Iflhsistrueorwouldbetruewhenconvertedtoaboolean,rhswillbereturned.lhs||rhswillreturnlhsifitistrueorwouldbetruewhenconvertedintoaboolean.Iflhsisfalseorwouldbefalsewhenconvertedtoaboolean,rhswillbereturned.!valuewillreturnthenegatedvalueofvalueconvertedintoaboolean

thearithmeticoperators(+,-,*,/,%)canbeappliedtoanyvalueandwillnotthrowexceptionswhenappliedtonon-numericvalues.Instead,anyvalueusedintheseoperatorswillbecastedtoanumericvalueimplicitly.Ifnonumericresultcanbeproducedbyanarithmeticoperator,itwillreturnnullinArangoDB2.3.Thisisalsotruefordivisionbyzero.passingargumentsofinvalidtypesintoAQLfunctionsdoesnotthrowaruntimeexceptioninmostcases,butmayproduceruntime

WhatsNewin2.3

605

warnings.Built-inAQLfunctionsthatreceiveinvalidargumentswillthenreturnnull.

Performanceimprovements

Non-uniquehashindexes

Theperformanceofinsertionintonon-uniquehashindexeshasbeenimprovedsignificantly.Thisfixesperformanceproblemsincaseattributeswereindexesthatcontainedonlyveryfewdistinctvalues,orwhenmostofthedocumentsdidnotevencontaintheindexedattribute.Thisalsofixesproblemswhenloadingcollectionswithsuchindexes.

Theinsertiontimenowscaleslinearlywiththenumberofdocumentsregardlessofthecardinalityoftheindexedattribute.

Reverseiterationoverskiplistindexes

AQLqueriescannowuseasortedskiplistindexforreverseiteration.ThisallowsseveralqueriestorunfasterthaninpreviousversionsofArangoDB.

Forexample,thefollowingAQLquerycannowusetheindexondoc.value:

FORdocINmycollection

FILTERdoc.value>23

SORTdoc.valuesDESC

RETURNdoc

PreviousversionsofArangoDBdidnotusetheindexbecauseofthedescending(DESC)sort.

Additionally,thenewAQLoptimizercanuseanindexforsortingnoweveniftheAQLquerydoesnotcontainaFILTERstatement.ThisoptimizationwasnotavailableinpreviousversionsofArangoDB.

AddedbasicsupportforhandlingbinarydatainFoxx

BufferobjectscannowbeusedwhensettingtheresponsebodyofanyFoxxaction.ThisallowsFoxxactionstoreturnbinarydata.

RequestswithbinarypayloadcanbeprocessedinFoxxapplicationsbyusingthenewmethodres.rawBodyBuffer().ThiswillreturntheunparsedrequestbodyasaBufferobject.

Thereisnowalsothemethodreq.requestParts()availableinFoxxtoretrievetheindividualcomponentsofamultipartHTTPrequest.Thatcanbeusedforexampletoprocessfileuploads.

Additionally,theres.send()methodhasbeenaddedasaconveniencemethodforreturningstrings,JSONobjectsorBuffersfromaFoxxaction.Itprovidessomeauto-detectionbasedonitsparametervalue:

res.send("<p>someHTML</p>");//returnsanHTMLstring

res.send({success:true});//returnsaJSONobject

res.send(newBuffer("somebinarydata"));//returnsbinarydata

Theconveniencemethodres.sendFile()cannowbeusedtoreturnthecontentsofafilefromaFoxxaction.Theyfilemaycontainbinarydata:

res.sendFile(applicationContext.foxxFilename("image.png"));

Thefilesystemmethodsfs.write()andfs.readBuffer()canbeusedtoworkwithbinarydata,too:

fs.write()willperformanauto-detectionofitssecondparameter'svaluesoitworkswithBufferobjects:

fs.write(filename,"somedata");//savesastringvalueinfile

fs.write(filename,newBuffer("somebinarydata"));//saves(binary)contentsofabuffer

fs.readBuffer()hasbeenaddedasamethodtoreadthecontentsofanarbitraryfileintoaBufferobject.

WhatsNewin2.3

606

Webinterface

Batchdocumentremovalandmovefunctionalityhasbeenaddedtothewebinterface,makingiteasiertoworkwithmultipledocumentsatonce.Additionally,basicJSONimportandexporttoolshavebeenadded.

Command-lineoptionsadded

Thecommand-lineoption--javascript.v8-contextswasaddedtoarangodtoprovidebettercontroloverthenumberofV8contextscreatedinarangod.

Previously,thenumberofV8contextsarangodcreatedatstartupwasequaltothenumberofserverthreads(asspecifiedbyoption--server.threads).

InsomesituationsitmaybemoresensibletocreatedifferentamountsofthreadsandV8contexts.ThisisbecauseeachV8contextscreatedwillconsumememoryandrequiresCPUresourcesforperiodicgarbagecollection.Contrary,serverthreadsdonothavesuchhighmemoryorCPUfootprint.

Iftheoption--javascript.v8-contextsisnotspecified,thenumberofV8contextscreatedatstartupwillremainequaltothenumberofserverthreads.ThusnochangeinconfigurationisrequiredtokeepthesamebehaviorasinpreviousArangoDBversions.

Thecommand-lineoption--log.use-local-timewasaddedtoprintdatesandtimesinArangoDB'slogintheserver-localtimezoneinsteadofUTC.Ifitisnotset,thetimezonewilldefaulttoUTC.

Theoption--backslash-escapehasbeenaddedtoarangoimp.SpecifyingthisoptionwillusethebackslashastheescapecharacterforliteralquoteswhenparsingCSVfiles.Theescapecharacterforliteralquotesisstillthedoublequotecharacter.

Miscellaneousimprovements

ArangoDB'sbuilt-inHTTPservernowsupportsHTTPpipelining.

TheArangoShelltutorialfromthearangodb.comwebsiteisnowintegratedintotheArangoDBshell.

PowerfulFoxxEnhancementsWiththenewjobqueuefeatureyoucanrunasyncjobstocommunicatewithexternalservices,FoxxqueriesmakewritingcomplexAQLqueriesmucheasierandFoxxsessionswillhandletheauthenticationandsessionhassleforyou.

FoxxQueries

WritinglongAQLqueriesinJavaScriptcanquicklybecomeunwieldy.Asof2.3ArangoDBbundlestheArangoDBQueryBuildermodulethatprovidesaJavaScriptAPIforwritingcomplexAQLquerieswithoutstringconcatenation.Allbuilt-infunctionsthatacceptAQLstringsnowsupportquerybuilderinstancesdirectly.AdditionallyFoxxprovidesamethodFoxx.createQueryforcreatingparametrizedqueriesthatcanreturnFoxxmodelsorapplyarbitrarytransformationstothequeryresults.

FoxxSessions

ThesessionfunctionalityinFoxxhasbeencompletelyrewritten.TheoldactivateAuthenticationAPIisstillsupportedbutmaybedeprecatedinthefuture.ThenewactivateSessionsAPIsupportscookiesorconfigurableheaders,providesoptionalJSONWebTokenandcryptographicsigningsupportandusesthenewsessionsFoxxapp.

ArangoDB2.3providesFoxxappsforusermanagementandsaltedhash-basedauthenticationwhichcanbereplacedwithorsupplementedbyalternativeimplementations.Foranexampleappusingboththebuilt-inauthenticationandOAuth2seetheFoxxSessionsExampleapp.

FoxxQueues

FoxxnowprovidesasyncworkersviatheFoxxQueuesAPI.Jobsenqueuedinajobqueuewillbeexecutedasynchronouslyoutsideoftherequest/responsecycleofFoxxcontrollersandcanbeusedtocommunicatewithexternalservicesorperformtasksthattakealongtimetocompleteormayrequiremultipleattempts.

WhatsNewin2.3

607

Jobscanbescheduledinadvanceorsettobeexecutedimmediately,thenumberofretryattempts,theretrydelayaswellassuccessandfailurehandlerscanbedefinedforeachjobindividually.Jobtypesthatintegratevariousexternalservicesfortransactionale-mails,loggingandusertrackingcanbefoundintheFoxxappregistry.

Misc

TherequestandresponseobjectsinFoxxcontrollersnowprovidemethodsforreadingandwritingrawcookiesandsignedcookies.

MountedFoxxappswillnowbeloadedwhenarangodstartsratherthanatthefirstdatabaserequest.Thismayresultinslightlyslowerstartuptimes(butafasterresponseforthefirstrequest).

WhatsNewin2.3

608

IncompatiblechangesinArangoDB2.3ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.3,andadjustanyclientprogramsifnecessary.

Configurationfilechanges

Threadsandcontexts

Thenumberofserverthreadsspecifiedisnowtheminimumofthreadsstarted.Therearesituationinwhichthreadsarewaitingforresultsofdistributeddatabaseservers.Inthiscasethenumberofthreadsisdynamicallyincreased.

WithArangoDB2.3,thenumberofserverthreadscanbeconfiguredindependentlyofthenumberofV8contexts.Theconfigurationoption--javascript.v8-contextswasaddedtoarangodtoprovidebettercontroloverthenumberofV8contextscreatedinarangod.

Previously,thenumberofV8contextsarangodcreatedatstartupwasequaltothenumberofserverthreads(asspecifiedbyoption--server.threads).

InsomesituationsitmaybemoresensibletocreatedifferentamountsofthreadsandV8contexts.ThisisbecauseeachV8contextscreatedwillconsumememoryandrequiresCPUresourcesforperiodicgarbagecollection.Contrary,serverthreadsdonothavesuchhighmemoryorCPUfootprint.

Iftheoption--javascript.v8-contextsisnotspecified,thenumberofV8contextscreatedatstartupwillremainequaltothenumberofserverthreads.ThusnochangeinconfigurationisrequiredtokeepthesamebehaviorasinpreviousArangoDBversions.

Ifyouareusingthedefaultconfigfilesormergethemwithyourlocalconfigfiles,pleasereviewifthedefaultnumberofserverthreadsisokayinyourenvironment.AdditionallyyoushouldverifythatthenumberofV8contextscreated(asspecifiedinoption--javascript.v8-contexts)isokay.

Syslog

Thecommand-lineoption--log.syslogwasusedinpreviousversionsofArangoDBtoturnloggingtosyslogonoroff:whensettingtoanon-emptystring,syslogloggingwasturnedon,otherwiseturnedoff.Whensyslogloggingwasturnedon,loggingwasdonewiththeapplicationnamespecifiedin--log.application,whichdefaultedtotriagens.Therewasalsoacommand-lineoption--log.hostnamewhichcouldbesetbutdidnothaveanyeffect.

Thisbehaviorturnedouttobeunintuitiveandwaschangedin2.3asfollows:

thecommand-lineoption--log.syslogisdeprecatedanddoesnothaveanyeffectwhenstartingArangoDB.toturnonsyslogloggingin2.3,theoption--log.facilityhastobesettoanon-emptystring.ThevalueforfacilityisOS-dependent(possiblevaluescanbefoundin/usr/include/syslog.horthelike-usershouldbeavailableonmanysystems).thedefaultvaluefor--log.applicationhasbeenchangedfromtriagenstoarangod.thecommand-lineoption--log.hostnameisdeprecatedanddoesnothaveanyeffectwhenstartingArangoDB.Instead,thehostnamewillbesetbysyslogautomatically.whenloggingtosyslog,ArangoDBnowomitsthedatetimeprefixandtheprocessid,becausethey'llbeaddedbysyslogautomatically.

AQL

AQLqueriesthrowlessexceptions

ArangoDB2.3containsacompletelyrewrittenAQLqueryoptimizerandexecutionengine.ThismeansthatAQLquerieswillbeexecutedwithadifferentenginethaninArangoDB2.2andearlier.PartsofAQLqueriesmightbeexecutedindifferentorderthanbeforebecausetheAQLoptimizerhasmorefreedomtomovethingsaroundinaquery.

Incompatiblechangesin2.3

609

InpreviousversionsofArangoDB,AQLqueriesabortedwithanexceptioninmanysituationsandthrewaruntimeexception.ExceptionswerethrownwhentryingtofindavalueusingtheINoperatorinanon-arrayelement,whentryingtousenon-booleanvalueswiththelogicaloperands&&or||or!,whenusingnon-numericvaluesinarithmeticoperations,whenpassingwrongparametersintofunctionsetc.

InArangoDB2.3thishasbeenchangedinmanycasestomakeAQLmoreuser-friendlyandtoallowtheoptimizationtoperformmuchmorequeryoptimizations.

Hereisasummaryofchanges:

whenanon-arrayvalueisusedontheright-handsideoftheINoperator,theresultwillbefalseinArangoDB2.3,andnoexceptionwillbethrown.thebooleanoperators&&and||donotthrowinArangoDB2.3ifanyoftheoperandsisnotabooleanvalue.Instead,theywillperformanimplicitcastofthevaluestobooleans.Theirresultwillbeasfollows:

lhs&&rhswillreturnlhsifitisfalseorwouldbefalsewhenconvertedintoaboolean.Iflhsistrueorwouldbetruewhenconvertedtoaboolean,rhswillbereturned.lhs||rhswillreturnlhsifitistrueorwouldbetruewhenconvertedintoaboolean.Iflhsisfalseorwouldbefalsewhenconvertedtoaboolean,rhswillbereturned.!valuewillreturnthenegatedvalueofvalueconvertedintoaboolean

thearithmeticoperators(+,-,*,/,%)canbeappliedtoanyvalueandwillnotthrowexceptionswhenappliedtonon-numericvalues.Instead,anyvalueusedintheseoperatorswillbecastedtoanumericvalueimplicitly.Ifnonumericresultcanbeproducedbyanarithmeticoperator,itwillreturnnullinArangoDB2.3.Thisisalsotruefordivisionbyzero.passingargumentsofinvalidtypesintoAQLfunctionsdoesnotthrowaruntimeexceptioninmostcases,butmayproduceruntimewarnings.Built-inAQLfunctionsthatreceiveinvalidargumentswillthenreturnnull.

NestedFORloopexecutionorder

Thequeryoptimizerin2.3maypermutetheorderofnestedFORloopsinAQLqueries,providedthatexchangingtheloopswillnotalteraqueryresult.However,achangeintheorderofreturnedvaluesisallowedbecausenosortorderisguaranteedbyAQL(andwasnever)unlessanexplicitSORTstatementisusedinaquery.

ChangedreturnvaluesofArangoQueryCursor.getExtra()

ThereturnvalueofArangoQueryCursor.getExtra()hasbeenchangedinArangoDB2.3.Itnowcontainsastatsattributewithstatisticsaboutthequerypreviouslyexecuted.Italsocontainsawarningsattributewithwarningsthathappenedduringqueryexecution.Thereturnvaluestructurehasbeenunifiedin2.3forbothread-onlyanddata-modificationqueries.

Thereturnvaluelookslikethisforaread-onlyquery:

arangosh>stmt=db._createStatement("FORiINmycollectionRETURNi");stmt.execute().getExtra()

{

"stats":{

"writesExecuted":0,

"writesIgnored":0,

"scannedFull":2600,

"scannedIndex":0

},

"warnings":[]

}

Fordata-modificationqueries,ArangoDB2.3returnsaresultwiththesamestructure:

arangosh>stmt=db._createStatement("FORiINxxREMOVEiINxx");stmt.execute().getExtra()

{

"stats":{

"writesExecuted":2600,

"writesIgnored":0,

"scannedFull":2600,

"scannedIndex":0

},

"warnings":[]

}

Incompatiblechangesin2.3

610

InArangoDB2.2,thereturnvalueofArangoQueryCursor.getExtra()wasemptyforread-onlyqueriesandcontainedanattributeoperationswithtwosub-attributesfordata-modificationqueries:

arangosh>stmt=db._createStatement("FORiINmycollectionRETURNi");stmt.execute().getExtra()

{

}

arangosh>stmt=db._createStatement("FORiINmycollectionREMOVEiINmycollection");stmt.execute().getExtra()

{

"operations":{

"executed":2600,

"ignored":0

}

}

ChangedreturnvaluesinHTTPmethodPOST/_api/cursor

ThepreviouslymentionedchangealsoleadstothestatisticsbeingreturnedintheHTTPRESTAPImethodPOST/_api/cursor.Previously,thereturnvaluecontainedanoptionalextraattributethatwasfilledonlyfordata-modificationqueriesandinsomeothercasesasfollows:

{

"result":[],

"hasMore":false,

"extra":{

"operations":{

"executed":2600,

"ignored":0

}

}

}

WiththechangedresultstructureinArangoDB2.3,theextraattributeintheresultwilllooklikethis:

{

"result":[],

"hasMore":false,

"extra":{

"stats":{

"writesExecuted":2600,

"writesIgnored":0,

"scannedFull":0,

"scannedIndex":0

},

"warnings":[]

}

}

IfthequeryoptionfullCountisrequested,thefullCountresultvaluewillalsobereturnedinsidethestatsattributeoftheextraattribute,andnotdirectlyasanattributeinsidetheextraattributeasin2.2.NotethatafullCountwillonlybepresentinextra.statsifitwasrequestedasanoptionforthequery.

TheresultinArangoDB2.3willalsocontainawarningsattributewiththearrayofwarningsthathappenedduringqueryexecution.

ChangedreturnvaluesinArangoStatement.explain()

ThereturnvalueofArangoStatement.explain()haschangedsignificantlyinArangoDB2.3.Thenewreturnvaluestructureisnotcompatiblewiththestructurereturnedby2.2.

InArangoDB2.3,thefullexecutionplanforanAQLqueryisreturnedalongsideallappliedoptimizerrules,optimizationwarningsetc.Itisalsopossibletohavetheoptimizerreturnallexecutionplans.Thisrequiredanewdatastructure.

ClientprogramsthatuseArangoStatement.explain()ortheHTTPRESTAPImethodPOST/_api/explainmayneedtobeadjustedtousethenewreturnformat.

Incompatiblechangesin2.3

611

ThereturnvalueofArangoStatement.parse()hasbeenextendedinArangoDB2.3.Inadditiontotheexistingattributes,ArangoDB2.3willalsoreturnanastattributecontainingtheabstractsyntaxtreeofthestatement.Thisextraattributecansafelybeignoredbyclientprograms.

Variablesnotupdatableinqueries

PreviousversionsofArangoDBallowedthemodificationofvariablesinsideAQLqueries,e.g.

LETcounter=0

FORiIN1..10

LETcounter=counter+1

RETURNcounter

Whilethisisadmittedlyaconvenientfeature,thenewqueryoptimizerdesigndidnotallowtokeepit.Additionally,updatingvariablesinsideaquerywouldpreventalotofoptimizationstoqueriesthatwewouldliketheoptimizertomake.Additionally,updatingvariablesinqueriesthatrunondifferentnodesinaclusterwouldlikecausenon-deterministicbehaviorbecausequeriesarenotexecutedlinearly.

ChangedreturnvalueofTO_BOOL

TheAQLfunctionTO_BOOLnowalwaysreturnstrueifitsargumentisanarrayoranobject.InpreviousversionsofArangoDB,thefunctionreturnedfalseforemptyarraysorforobjectswithoutattributes.

ChangedreturnvalueofTO_NUMBER

TheAQLfunctionTO_NUMBERnowreturnsnullifitsargumentisanobjectoranarraywithmorethanonemember.InpreviousversionofArangoDB,thereturnvalueinthesecaseswas0.TO_NUMBERwillreturn0foremptyarray,andthenumericequivalentofthearraymember'svalueforarrayswithasinglemember.

NewAQLkeywords

ThefollowingkeywordshavebeenaddedtoAQLinArangoDB2.3:

NOTANDOR

UnquotedusageofthesekeywordsforattributenamesinAQLquerieswilllikelyfailinArangoDB2.3.Ifanysuchattributenameneedstobeusedinaquery,itshouldbeenclosedinbacktickstoindicatetheusageofaliteralattributename.

Removedfeatures

Bitarrayindexes

Bitarrayindexeswereonlyhalf-waydocumentedandintegratedinpreviousversionsofArangoDBsotheirbenefitwaslimited.ThesupportforbitarrayindexeshasthusbeenremovedinArangoDB2.3.Itisnotpossibletocreateindexesoftype"bitarray"withArangoDB2.3.

WhenacollectionisopenedthatcontainsabitarrayindexdefinitioncreatedwithapreviousversionofArangoDB,ArangoDBwillignoreitandlogthefollowingwarning:

indextype'bitarray'isnotsupportedinthisversionofArangoDBandisignored

FutureversionsofArangoDBmayautomaticallyremovesuchindexdefinitionssothewarningswilleventuallydisappear.

Otherremovedfeatures

TheHTTPRESTAPImethodatPOST/_admin/modules/flushhasbeenremoved.

Incompatiblechangesin2.3

612

Knownissues

InArangoDB2.3.0,AQLqueriescontainingfilterconditionswithanINexpressionwillnotyetuseanindex:

FORdocINcollectionFILTERdoc.indexedAttributeIN[...]RETURNdoc

FORdocINcollection

FILTERdoc.indexedAttributeIN[...]

RETURNdoc

We’recurrentlyworkingongettingtheINoptimizationsdone,andwillshipthemina2.3maintenancereleasesoon(e.g.2.3.1or2.3.2).

Incompatiblechangesin2.3

613

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.2.ArangoDB2.2alsocontainsseveralbugfixesthatarenotlistedhere.

AQLimprovements

DatamodificationAQLqueries

Uptoincludingversion2.1,AQLsupporteddataretrievaloperationsonly.StartingwithArangoDBversion2.2,AQLalsosupportsthefollowingdatamodificationoperations:

INSERT:insertnewdocumentsintoacollectionUPDATE:partiallyupdateexistingdocumentsinacollectionREPLACE:completelyreplaceexistingdocumentsinacollectionREMOVE:removeexistingdocumentsfromacollection

Data-modificationoperationsarenormallycombinedwithotherAQLstatementssuchasFORloopsandFILTERconditionstodeterminethesetofdocumentstooperateon.Forexample,thefollowingquerywillfindalldocumentsincollectionusersthatmatchaspecificconditionandsettheirstatusvariabletoinactive:

FORuINusers

FILTERu.status=='notactive'

UPDATEuWITH{status:'inactive'}INusers

Thefollowingquerycopiesalldocumentsfromcollectionusersintocollectionbackup:

FORuINusers

INSERTuINbackup

Andthisqueryremovesdocumentsfromcollectionbackup:

FORdocINbackup

FILTERdoc.lastModified<DATE_NOW()-3600

REMOVEdocINbackup

Formoreinformationondata-modificationqueries,pleaserefertoDatamodificationqueries.

Updatablevariables

Previously,thevalueofavariableassignedinanAQLquerywiththeLETkeywordwasnotupdatableinanAQLquery.Thispreventedstatementslikethefollowingfrombeingexecutable:

LETsum=0

FORvINvalues

SORTv.year

LETsum=sum+v.value

RETURN{year:v.year,value:v.value,sum:sum}

OtherAQLimprovements

addedAQLTRANSLATEfunction

Thisfunctioncanbeusedtoperformlookupsfromstaticobjects,e.g.

LETcountryNames={US:"UnitedStates",UK:"UnitedKingdom",FR:"France"}

RETURNTRANSLATE("FR",countryNames)

WhatsNewin2.2

614

LETlookup={foo:"foo-replacement",bar:"bar-replacement",baz:"baz-replacement"}

RETURNTRANSLATE("foobar",lookup,"notcontained!")

Write-aheadlogAllwriteoperationsinanArangoDBserverwillnowbeautomaticallyloggedintheserver'swrite-aheadlog.Thewrite-aheadlogisasetofappend-onlylogfiles,anditisusedincaseofacrashrecoveryandforreplication.

Datafromthewrite-aheadlogwilleventuallybemovedintothejournalsordatafilesofcollections,allowingtheservertoremoveolderwrite-aheadlogfiles.

Cross-collectiontransactionsinArangoDBshouldbenefitconsiderablybythischange,aslesswritesthaninpreviousversionsarerequiredtoensurethedataofmultiplecollectionsareatomicallyanddurablycommitted.Alldata-modifyingoperationsinsidetransactions(insert,update,remove)willwritetheiroperationsintothewrite-aheadlogdirectlynow.Inpreviousversions,suchoperationswerebuffereduntilthecommitorrollbackoccurred.TransactionswithmultipleoperationsshouldthereforerequirelessphysicalmemorythaninpreviousversionsofArangoDB.

Thedatainthewrite-aheadlogcanalsobeusedinthereplicationcontext.InpreviousversionsofArangoDB,replicatingfromamasterrequiredturningonaspecialreplicationloggeronthemaster.Thereplicationloggercausedanextrawriteoperationintothe_replicationsystemcollectionforeachactualwriteoperation.Thisextrawriteisnowsuperfluous.Instead,slavescanreaddirectlyfromthemaster'swrite-aheadlogtogetinformedaboutmostrecentdatachanges.Thisremovestheneedtostoredata-modificationoperationsinthe_replicationcollectionaltogether.

Fortheconfigurationofthewrite-aheadlog,pleaserefertoWrite-aheadlogoptions.

Theintroductionofthewrite-aheadlogalsoremovestheneedtoconfigureandstartthereplicationloggeronamaster.ThoughthereplicationloggerobjectisstillavailableinArangoDB2.2toensureAPIcompatibility,starting,stopping,orconfiguringitwillhavenoeffect.

PerformanceimprovementsRemovedsortingofattributenameswhenincollectionshaper

InpreviousversionsofArangoDB,addingadocumentwithpreviouslynot-usedattributenamescausedafullsortofallattributenamesusedinthecollection.Thesortingwasdonetoensurefastcomparisonsofattributenamesinsomerareedgecases,butitconsiderablysloweddowninsertsintocollectionswithmanydifferentorevenuniqueattributenames.

Specializedprimaryindeximplementationtoallowfasterhashtablerebuildingandreducelookupsindatafilesfortheactualvalueof_key.Thisalsoreducestheamountofrandommemoryaccessesforprimaryindexinserts.

Reclamationofindexmemorywhendeletinglastdocumentincollection

Deletingdocumentsfromacollectiondidnotleadtoindexsizesbeingreduced.Instead,theindexmemorywaskeptallocatedandre-usedlaterwhenacollectionwasrefilledwithnewdocuments.Now,indexmemoryofprimaryindexesandhashindexesisreclaimedinstantlywhenthelastdocumentinacollectionisremoved.

Preventbufferingoflongprintresultsinarangosh'sandarangod'sprintcommand

Thischangewillemitbufferedintermediateprintresultsanddiscardtheoutputbuffertoquicklydeliverprintresultstotheuser,andtopreventconstructingverylargebuffersforlargeresults.

Miscellaneousimprovements

Addedinsertmethodasanaliasforsave.Documentscannowbeinsertedintoacollectionusingeithermethod:

db.test.save({foo:"bar"});

db.test.insert({foo:"bar"});

Cleanupofoptionsfordata-modificationoperations

WhatsNewin2.2

615

Manyofthedata-modificationoperationshadsignatureswithmanyoptionalboolparameters,e.g.:

db.test.update("foo",{bar:"baz"},true,true,true)

db.test.replace("foo",{bar:"baz"},true,true)

db.test.remove("foo",true,true)

db.test.save({bar:"baz"},true)

Suchlongparameterlistswereunintuitiveandhardtousewhenonlyoneoftheoptionalparametersshouldhavebeenset.

TomaketheAPIsmoreusable,theoperationsnowunderstandthefollowingalternativesignature:

collection.update(key,update-document,options)

collection.replace(key,replacement-document,options)

collection.remove(key,options)

collection.save(document,options)

Examples:

db.test.update("foo",{bar:"baz"},{overwrite:true,keepNull:true,waitForSync:true})

db.test.replace("foo",{bar:"baz"},{overwrite:true,waitForSync:true})

db.test.remove("foo",{overwrite:true,waitForSync:true})

db.test.save({bar:"baz"},{waitForSync:true})

Added--overwriteoptiontoarangoimp

Thisallowsremovingalldocumentsinacollectionbeforeimportingintoitusingarangoimp.

Honorstartupoption--server.disable-statisticswhendecidingwhetherornottostartperiodicstatisticscollectionjobs

Previously,thestatisticscollectionjobswerestartedeveniftheserverwasstartedwiththe--server.disable-statisticsflagbeingsettotrue.Nowiftheoptionissettotrue,nostatisticswillbecollectedontheserver.

DisallowstoringofJavaScriptobjectsthatcontainJavaScriptnativeobjectsoftypeDate,Function,RegExporExternal,e.g.

db.test.save({foo:/bar/});

db.test.save({foo:newDate()});

Thiswillnowprint

Error:<data>cannotbeconvertedintoJSONshape:couldnotshapedocument

Previously,objectsofthesetypesweresilentlyconvertedintoanemptyobject(i.e.{})andnowarningwasissued.

Tostoresuchobjectsinacollection,explicitlyconvertthemintostringslikethis:

db.test.save({foo:String(/bar/)});

db.test.save({foo:String(newDate())});

Removedfeatures

MRubyintegrationforarangod

ArangoDBhadanexperimentalMRubyintegrationinsomeofthepublishbuilds.Thiswasn'tcontinuouslydeveloped,andsoithasbeenremovedinArangoDB2.2.

Thischangehasledtothefollowingstartupoptionsbeingsuperfluous:

--ruby.gc-interval

--ruby.action-directory

--ruby.modules-path

--ruby.startup-directory

WhatsNewin2.2

616

SpecifyingthesestartupoptionswilldonothinginArangoDB2.2,sousingtheseoptionsshouldbeavoidedfromnowonastheymightberemovedinafutureversionofArangoDB.

Removedstartupoptions

ThefollowingstartupoptionshavebeenremovedinArangoDB2.2.Specifyingthemintheserver'sconfigurationfilewillnotproduceanerrortomakemigrationeasier.Still,usageoftheseoptionsshouldbeavoidedastheywillnothaveanyeffectandmightfullyberemovedinafutureversionofArangoDB:

--database.remove-on-drop

--database.force-sync-properties

--random.no-seed

--ruby.gc-interval

--ruby.action-directory

--ruby.modules-path

--ruby.startup-directory

--server.disable-replication-logger

MultiCollectionGraphs

ArangoDBisamultimodeldatabasewithnativegraphsupport.Inversion2.2thefeaturesforgraphshavebeenimprovedbyintegrationofanewgraphmodule.Allgraphscreatedwiththeoldmoduleareautomaticallymigratedintothenewmodulebutcanstillbeusedbytheoldmodule.

Newgraphmodule

Uptoincludingversion2.1,ArangoDBofferedamoduleforgraphsandgraphoperations.Thismoduleallowedyoutouseexactlyoneedgecollectiontogetherwithonevertexcollectioninagraph.WithArangoDBversion2.2thisgraphmoduleisdeprecatedandanewgraphmoduleisoffered.Thisnewmoduleallowstocombineanarbitrarynumberofvertexcollectionsandedgecollectionsinthesamegraph.Foreachedgecollectionalistofcollectionscontainingsourceverticesandalistofcollectionscontainingtargetverticescanbedefined.IfanedgeisstoredArangoDBchecksifthisedgeisvalidinthiscollection.Furthermoreifavertexisremovedfromoneofthecollectionsallconnectededgeswillberemovedaswell,givingtheguaranteeofnolooseendsinthegraphs.Thelayoutofthegraphcanbemodifiedatruntimebyaddingorremovingcollectionsandchangingthedefinitionsforedgecollections.Alloperationsonthegraphlevelaretransactionalbydefault.

GraphsinAQL

MulticollectiongraphshavebeenaddedtoAQLaswell.Basicfunctionality(gettingvertices,edges,neighbors)canbeexecutedusingtheentiregraph.Alsomoreadvancedfeatureslikeshortestpathcalculations,characteristicfactorsofthegraphortraversalshavebeenintegratedintoAQL.Forthesefunctionsallgraphscreatedwiththegraphmodulecanbeused.

WhatsNewin2.2

617

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.1.ArangoDB2.1alsocontainsseveralbugfixesthatarenotlistedhere.

NewEdgesIndex

Theedgesindex(usedtostoreconnectionsbetweennodesinagraph)internallyusesanewdatastructure.Thisdatastructureimprovestheperformancewhenpopulatingtheedgeindex(i.e.whenloadinganedgecollection).Forlargegraphsloadingcanbe20timesfasterthanwithArangoDB2.0.

Additionally,thenewindexfixesperformanceproblemsthatoccurredwhenmanyduplicate_fromor_tovalueswerecontainedintheindex.Furthermore,thenewindexsupportsfasterremovalofedges.

Finally,whenloadinganexistingcollectionandbuildingtheedgesindexforthecollection,lessmemoryre-allocationswillbeperformed.

Overall,thisshouldconsiderablyspeeduploadingedgecollections.

Thenewindextypereplacestheoldedgesindextypeautomatically,withoutanychangesbeingrequiredbytheenduser.

TheAPIofthenewindexiscompatiblewiththeAPIoftheoldindex.Stillitispossiblethatthenewindexreturnsedgesinadifferentorderthantheoldindex.Thisisstillconsideredtobecompatiblebecausetheoldindexhadneverguaranteedanyresultordereither.

AQLImprovements

AQLoffersfunctionalitytoworkwithdates.DatesarenodatatypesoftheirowninAQL(neithertheyareinJSON,whichisoftenusedasaformattoshipdataintoandoutofArangoDB).Instead,datesinAQLareinternallyrepresentedbyeithernumbers(timestamps)orstrings.ThedatefunctionsinAQLprovidemechanismstoconvertfromanumerictimestamptoastringrepresentationandviceversa.

TherearetwodatefunctionsinAQLtocreatedatesforfurtheruse:

DATE_TIMESTAMP(date)CreatesaUTCtimestampvaluefromdate

DATE_TIMESTAMP(year,month,day,hour,minute,second,millisecond):Sameasbefore,butallowsspecifyingtheindividualdatecomponentsseparately.Allparametersafterdayareoptional.

DATE_ISO8601(date):ReturnsanISO8601datetimestringfromdate.ThedatetimestringwillalwaysuseUTCtime,indicatedbytheZatitsend.

DATE_ISO8601(year,month,day,hour,minute,second,millisecond):sameasbefore,butallowsspecifyingtheindividualdatecomponentsseparately.Allparametersafterdayareoptional.

Thesetwoabovedatefunctionsacceptthefollowinginputvalues:

numerictimestamps,indicatingthenumberofmillisecondselapsedsincetheUNIXepoch(i.e.January1st197000:00:00UTC).Anexampletimestampvalueis1399472349522,whichtranslatesto2014-05-07T14:19:09.522Z.

datetimestringsinformatsYYYY-MM-DDTHH:MM:SS.MMM,YYYY-MM-DDHH:MM:SS.MMM,orYYYY-MM-DD.Millisecondsarealwaysoptional.

Atimezonedifferencemayoptionallybeaddedattheendofthestring,withthehoursandminutesthatneedtobeaddedorsubtractedtothedatetimevalue.Forexample,2014-05-07T14:19:09+01:00canbeusedtospecifyaonehouroffset,and2014-05-07T14:19:09+07:30canbespecifiedforsevenandhalfhoursoffset.Negativeoffsetsarealsopossible.Alternativelytoanoffset,aZcanbeusedtoindicateUTC/Zulutime.

Anexamplevalueis2014-05-07T14:19:09.522ZmeaningMay7th2014,14:19:09and522milliseconds,UTC/Zulutime.Anotherexamplevaluewithouttimecomponentis2014-05-07Z.

Pleasenotethatifnotimezoneoffsetisspecifiedinadatestring,ArangoDBwillassumeUTCtimeautomatically.Thisisdonetoensureportabilityofqueriesacrossserverswithdifferenttimezonesettings,andbecausetimestampswillalwaysbeUTC-based.

WhatsNewin2.1

618

individualdatecomponentsasseparatefunctionarguments,inthefollowingorder:

yearmonthdayhourminutesecondmillisecond

Allcomponentsfollowingdayareoptionalandcanbeomitted.Notethatnotimezoneoffsetscanbespecifiedwhenusingseparatedatecomponents,andUTC/Zulutimewillbeused.

ThefollowingcallstoDATE_TIMESTAMPareequivalentandwillallreturn1399472349522:

DATE_TIMESTAMP("2014-05-07T14:19:09.522")

DATE_TIMESTAMP("2014-05-07T14:19:09.522Z")

DATE_TIMESTAMP("2014-05-0714:19:09.522")

DATE_TIMESTAMP("2014-05-0714:19:09.522Z")

DATE_TIMESTAMP(2014,5,7,14,19,9,522)

DATE_TIMESTAMP(1399472349522)

ThesameistrueforcallstoDATE_ISO8601thatalsoacceptsvariableinputformats:

DATE_ISO8601("2014-05-07T14:19:09.522Z")

DATE_ISO8601("2014-05-0714:19:09.522Z")

DATE_ISO8601(2014,5,7,14,19,9,522)

DATE_ISO8601(1399472349522)

Theabovefunctionsareallequivalentandwillreturn"2014-05-07T14:19:09.522Z".

ThefollowingdatefunctionscanbeusedwithdatescreatedbyDATE_TIMESTAMPandDATE_ISO8601:

DATE_DAYOFWEEK(date):Returnstheweekdaynumberofdate.Thereturnvalueshavethefollowingmeanings:

0:Sunday1:Monday2:Tuesday3:Wednesday4:Thursday5:Friday6:Saturday

DATE_YEAR(date):Returnstheyearpartofdateasanumber.

DATE_MONTH(date):Returnsthemonthpartofdateasanumber.

DATE_DAY(date):Returnsthedaypartofdateasanumber.

DATE_HOUR(date):Returnsthehourpartofdateasanumber.

DATE_MINUTE(date):Returnstheminutepartofdateasanumber.

DATE_SECOND(date):Returnsthesecondspartofdateasanumber.

DATE_MILLISECOND(date):Returnsthemillisecondspartofdateasanumber.

Thefollowingotherdatefunctionsarealsoavailable:

DATE_NOW():Returnsthecurrenttimeasatimestamp.

Notethatthisfunctionisevaluatedoneveryinvocationandmayreturndifferentvalueswheninvokedmultipletimesinthesamequery.

ThefollowingotherAQLfunctionshavebeenaddedinArangoDB2.1:

FLATTEN:thisfunctioncanturnanarrayofsub-arraysintoasingleflatarray.Allarrayelementsintheoriginalarraywillbeexpandedrecursivelyuptoaconfigurabledepth.Theexpandedvalueswillbeaddedtothesingleresultarray.

WhatsNewin2.1

619

Example:

FLATTEN([1,2,[3,4],5,[6,7],[8,[9,10]])

willexpandthesub-arraysonthefirstlevelandproduce:

[1,2,3,4,5,6,7,8,[9,10]]

Tofullyflattenthearray,themaximumdepthcanbespecified(e.g.withavalueof2):

FLATTEN([1,2,[3,4],5,[6,7],[8,[9,10]],2)

Thiswillfullyexpandthesub-arraysandproduce:

[1,2,3,4,5,6,7,8,9,10]

CURRENT_DATABASE:thisfunctionwillreturnthenameofthedatabasethecurrentqueryisexecutedin.

CURRENT_USER:thisfunctionreturnsthenameofthecurrentuserthatisexecutingthequery.Ifauthorizationisturnedofforthequeryisexecutedoutsideofarequestcontext,nouserispresentandthefunctionwillreturnnull.

ClusterDumpandRestore

Thedumpandrestoretools,arangodumpandarangorestore,cannowbeusedtodumpandrestorecollectionsinacluster.Additionally,acollectiondumpfromastandaloneArangoDBservercanbeimportedintoacluster,andviceversa.

WebInterfaceImprovements

Thewebinterfaceinversion2.1hasamorecompactdashboard.Itprovideschartswithtime-seriesforincomingrequests,HTTPtransfervolumeandsomeserverresourceusagefigures.

Additionallyitprovidestrendindicators(e.g.15minaverages)anddistributioncharts(akahistogram)forsomefigures.

FoxxImprovements

ToeasilyaccessafileinsidethedirectoryofaFoxxapplicationfromwithinFoxx,Foxx'sapplicationContextnowprovidesthefoxxFilename()function.Itcanbeusedtoassemblethefullfilenameofafileinsidetheapplication'sdirectory.TheapplicationContextcanbeaccessedasglobalvariablefromanymodulewithinaFoxxapplication.

ThefilenamecanbeusedinsideFoxxactionsorsetup/teardownscripts,e.g.topopulateaFoxxapplication'scollectionwithdata.

TherequirefunctionnowalsopreferslocalmoduleswhenusedfrominsideaFoxxapplication.ThisallowsputtingmodulesinsidetheFoxxapplicationdirectoryandrequiringthemeasily.Italsoallowsusingapplication-specificversionsoflibrariesthatarebundledwithArangoDB(suchasunderscore.js).

WindowsInstaller

TheWindowsinstallershippedwithArangoDBnowsupportsinstallationofArangoDBforthecurrentuserorallusers,withtherequiredprivileges.ItalsosupportstheinstallationofArangoDBasaservice.

Fixesfor32bitsystems

SeveralissueshavebeenfixedthatoccurredonlywhenusingArangoDBona32bitsoperatingsystem,specifically:

acrashinathirdpartycomponentusedtomanageclusterdata

WhatsNewin2.1

620

athirdpartylibrarythatfailedtoinitializeon32bitWindows,makingarangodandarangoshcrashimmediately.

overflowsofvaluesusedfornanosecond-precisiontimeouts:theseoverflowshaveledtoinvalidvaluesbeingpassedtosocketoperations,makingthemfailandre-trytoooften

Updateddrivers

SeveraldriversforArangoDBhavebeencheckedforcompatibilitywith2.1.Thecurrentlistofdriverswithcompatibilitynotescanbefoundonlinehere.

C++11usage

WehavemovedseveralfilesfromCtoC++,allowingmorecodereuseandreducingtheneedforshippingdatabetweenthetwo.WehavealsodecidedtorequireC++11supportforArangoDB,whichallowsustousesomeofthesimplifications,featuresandguaranteesthatthisstandardhasinstock.

ThatalsomeansacompilerwithC++11supportisrequiredtobuildArangoDBfromsource.ForinstanceGNUCCofatleastversion4.8.

MiscellaneousImprovementsCancelableasynchronousjobs:severalpotentiallylong-runningjobscannowbecanceledviaanexplicitcanceloperation.Thisallowsstoppinglong-runningqueries,traversalsorscriptswithoutshuttingdownthecompleteArangoDBprocess.Jobcancelationisprovidedforasynchronouslyexecutedjobsasisdescribedin@refHttpJobCancel.

Server-sideperiodictaskmanagement:anArangoDBservernowprovidesfunctionalitytoregisterandunregisterperiodictasks.Tasksareuser-definedJavaScriptactionsthatcanberunperiodicallyandautomatically,independentofanyHTTPrequests.

Thefollowingtaskmanagementfunctionsareprovided:

require("org/arangodb/tasks").register():registersaperiodictaskrequire("org/arangodb/tasks").unregister():unregistersandremovesaperiodictaskrequire("org/arangodb/tasks").get():retrievesaspecifictasksorallexistingtasks

Anexampletask(tobeexecutedevery15seconds)canberegisteredlikethis:

vartasks=require("org/arangodb/tasks");

tasks.register({

name:"thisisanexampletaskwithparameters",

period:15,

command:function(params){

vargreeting=params.greeting;

vardata=JSON.stringify(params.data);

require('console').log('%sfromparametertask:%s',greeting,data);

},

params:{greeting:"hi",data:"howareyou?"}

});

Pleaserefertothesection@refTasksformoredetails.

Thefiguresmethodofacollectionnowreturnsdataaboutthecollection'sindexmemoryconsumption.Thereturnedvalueindexes.sizewillcontainthetotalamountofmemoryacquiredbyallindexesofthecollection.Thisfigurecanbeusedtoassessthememoryimpactofindexes.

CapitalizedHTTPresponseheaders:fromversion2.1,ArangoDBwillreturncapitalizedHTTPheadersbydefault,e.g.Content-Lengthinsteadofcontent-length.ThoughtheHTTPspecificationstatesthatheadersfieldnamearecase-insensitive,severalolderclienttoolsrelyonaspecificcaseinHTTPresponseheaders.ThischangesmakeArangoDBabitmorecompatiblewiththose.

Simplifiedusageofdb._createStatement():toeasilyrunanAQLquery,themethoddb._createStatementnowallowspassingtheAQLqueryasastring.Previouslyitrequiredtheusertopassanobjectwithaqueryattribute(whichthencontainedthequerystring).

WhatsNewin2.1

621

ArangoDBnowsupportsbothversions:

db._createStatement(queryString);

db._createStatement({query:queryString});

WhatsNewin2.1

622

AppendixReferences:Briefoverviewsoverinterfacesandobjects

db:thedbobjectcollection:thecollectionobjectcursor:thecursorobject

JavaScriptModules:Listofbuilt-inandsupportedJSmodulesDeprecated:FeaturesthatareconsideredobsoleteandmaygetremovedeventuallyErrorcodesandmeanings:ListofallpossibleerrorsthatcanbeencounteredGlossary:Disambiguationpage

Appendix

623

ReferencesThissectioncontainsthedocumentationforsomeoftheAPI'swhicharecommononthearangoshell,arangodserverandforFoxxApps.

References

624

The"db"ObjectThedbobjectisavailableinarangoshbydefault,andcanalsobeimportedandusedinFoxxservices.

db.namereturnsacollectionobjectforthecollectionname.

Thefollowingmethodsexistsonthe_dbobject:

Database

db._createDatabase(name,options,users)db._databases()db._dropDatabase(name,options,users)db._useDatabase(name)

Indexes

db._index(index)db._dropIndex(index)

Properties

db._id()db._isSystem()db._name()db._path()db._version()

Collection

db._collection(name)db._create(name)db._drop(name)db._truncate(name)

AQL

db._createStatement(query)db._query(query)db._explain(query)

Document

db._document(object)db._exists(object)db._remove(selector)db._replace(selector,data)db._update(selector,data)

db

625

The"collection"ObjectThefollowingmethodsexistonthecollectionobject(returnedbydb.name):

Collection

collection.checksum()collection.count()collection.drop()collection.figures()collection.load()collection.properties()collection.revision()collection.rotate()collection.toArray()collection.truncate()collection.type()collection.unload()

Indexes

collection.dropIndex(index)collection.ensureIndex(description)collection.getIndexes(name)collection.index(index)

Document

collection.all()collection.any()collection.closedRange(attribute,left,right)collection.document(object)collection.documents(keys)collection.edges(vertex-id)collection.exists(object)collection.firstExample(example)collection.inEdges(vertex-id)collection.insert(data)collection.edges(vertices)collection.iterate(iterator,options)collection.outEdges(vertex-id)collection.queryByExample(example)collection.range(attribute,left,right)collection.remove(selector)collection.removeByKeys(keys)collection.rename()collection.replace(selector,data)collection.replaceByExample(example,data)collection.save(data)collection.update(selector,data)collection.updateByExample(example,data)

collection

626

SequentialAccessandCursorsIfaqueryreturnsacursor(forexamplebycallingdb._query(...)),thenyoucanusehasNextandnexttoiterateovertheresultsetortoArraytoconvertittoanarray.

Ifthenumberofqueryresultsisexpectedtobebig,itispossibletolimittheamountofdocumentstransferredbetweentheserverandtheclienttoaspecificvalue.ThisvalueiscalledbatchSize.ThebatchSizecanoptionallybesetbeforeorwhenasimplequeryisexecuted.Iftheserverhasmoredocumentsthanshouldbereturnedinasinglebatch,theserverwillsetthehasMoreattributeintheresult.Itwillalsoreturntheidoftheserver-sidecursorintheidattributeintheresult.ThisidcanbeusedwiththecursorAPItofetchanyoutstandingresultsfromtheserveranddisposetheserver-sidecursorafterwards.

TheinitialbatchSizevaluecanbesetusingthesetBatchSizemethodthatisavailableforeachtypeofsimplequery,orwhenthesimplequeryisexecutedusingitsexecutemethod.IfnobatchSizevalueisspecified,theserverwillpickareasonabledefaultvalue.

HasNext

checksifthecursorisexhaustedcursor.hasNext()

ThehasNextoperatorreturnstrue,thenthecursorstillhasdocuments.Inthiscasethenextdocumentcanbeaccessedusingthenextoperator,whichwilladvancethecursor.

Examples

arangosh>vara=db._query("FORxINfiveRETURNx");

arangosh>while(a.hasNext())print(a.next());

showexecutionresults

Next

returnsthenextresultdocumentcursor.next()

IfthehasNextoperatorreturnstrue,thentheunderlyingcursorofthesimplequerystillhasdocuments.Inthiscasethenextdocumentcanbeaccessedusingthenextoperator,whichwilladvancetheunderlyingcursor.Ifyouusenextonanexhaustedcursor,thenundefinedisreturned.

Examples

arangosh>db._query("FORxINfiveRETURNx").next();

showexecutionresults

SetBatchsize

setsthebatchsizeforanyfollowingrequestscursor.setBatchSize(number)

Setsthebatchsizeforqueries.Thebatchsizedetermineshowmanyresultsareatmosttransferredfromtheservertotheclientinonechunk.

GetBatchsize

returnsthebatchsizecursor.getBatchSize()

Returnsthebatchsizeforqueries.Ifthereturnedvalueisundefined,theserverwilldetermineasensiblebatchsizeforanyfollowingrequests.

ExecuteQuery

cursor

627

executesaqueryquery.execute(batchSize)

Executesasimplequery.IftheoptionalbatchSizevalueisspecified,theserverwillreturnatmostbatchSizevaluesinoneroundtrip.ThebatchSizecannotbeadjustedafterthequeryisfirstexecuted.

Note:Thereisnoneedtoexplicitlycalltheexecutemethodifanothermeansoffetchingthequeryresultsischosen.Thefollowingtwoapproachesleadtothesameresult:

arangosh>result=db.users.all().toArray();

arangosh>varq=db._query("FORxINusersRETURNx");

........>result=[];

........>while(q.hasNext()){

........>result.push(q.next());

........>}

showexecutionresultsThefollowingtwoalternativesbothuseabatchSizeandreturnthesameresult:

arangosh>q=db.users.all();q.setBatchSize(20);q.execute();while(q.hasNext()){

print(q.next());}

arangosh>q=db.users.all();q.execute(20);while(q.hasNext()){print(q.next());}

showexecutionresults

Dispose

disposestheresultcursor.dispose()

Ifyouarenolongerinterestedinanyfurtherresults,youshouldcalldisposeinordertofreeanyresourcesassociatedwiththecursor.Aftercallingdisposeyoucannolongeraccessthecursor.

Count

countsthenumberofdocumentscursor.count()

Thecountoperatorcountsthenumberofdocumentintheresultsetandreturnsthatnumber.Thecountoperatorignoresanylimitsandreturnsthetotalnumberofdocumentsfound.

Note:Notallsimplequeriessupportcounting.Inthiscasenullisreturned(Simplequeriesaredeprecated).

cursor.count(true)

Iftheresultsetwaslimitedbythelimitoperatorordocumentswereskipedusingtheskipoperator,thecountoperatorwithargumenttruewillusethenumberofelementsinthefinalresultset-afterapplyinglimitandskip.

Note:Notallsimplequeriessupportcounting.Inthiscasenullisreturned(Simplequeriesaredeprecated)..

cursor

628

JavaScriptModulesArangoDBusesaNode.jscompatiblemodulesystem.Youcanusethefunctionrequireinordertoloadamoduleorlibrary.Itreturnstheexportedvariablesandfunctionsofthemodule.

Theglobalvariablesglobal,process,console,Buffer,__filenameand__dirnameareavailablethroughoutArangoDBandFoxx.

Nodecompatibilitymodules

ArangoDBsupportsanumberofmodulesforcompatibilitywithNode.js,including:

assertimplementsbasicassertionandtestingfunctions.

bufferimplementsabinarydatatypeforJavaScript.

consoleisawellknownloggingfacilitytoalltheJavaScriptdevelopers.ArangoDBimplementsmostoftheConsoleAPI,withtheexceptionsofprofileandcount.

eventsimplementsaneventemitter.

fsprovidesafilesystemAPIforthemanipulationofpaths,directories,files,links,andtheconstructionoffilestreams.ArangoDBimplementsmostFilesystem/Afunctions.

moduleprovidesdirectaccesstothemodulesystem.

pathimplementsfunctionsdealingwithfilenamesandpaths.

punycodeimplementsconversionfunctionsforpunycodeencoding.

querystringprovidesutilitiesfordealingwithquerystrings.

streamprovidesastreaminginterface.

string_decoderimplementslogicfordecodingbuffersintostrings.

urlprovidesutilitiesforURLresolutionandparsing.

utilprovidesgeneralutilityfunctionslikeformatandinspect.

AdditionallyArangoDBprovidespartialimplementationsforthefollowingmodules:

net:onlyisIP,isIPv4andisIPv6.

process:onlyenvandcwd;stubsforargv,stdout.isTTY,stdout.write,nextTick.

timers:stubsforsetImmediate,setTimeout,setInterval,clearImmediate,clearTimeout,clearIntervalandref.

tty:onlyisatty(alwaysreturnsfalse).

vm:onlyrunInThisContext.

ThefollowingNode.jsmodulesarenotavailableatall:child_process,cluster,constants,crypto(butsee@arangodb/cryptobelow),dgram,dns,domain,http(butsee@arangodb/requestbelow),https,os,sys,tls,v8,zlib.

ArangoDBSpecificModulesTherearealargenumberofArangoDB-specificmodulesusingthe@arangodbnamespace,mostlyforinternalusebyArangoDBitself.Thefollowinghoweverarenoteworthy:

@arangodbprovidesdirectaccesstothedatabaseanditscollections.

@arangodb/cryptoprovidesvariouscryptographyfunctionsincludinghashingalgorithms.

JavaScriptModules

629

@arangodb/requestprovidesthefunctionalityformakingsynchronousHTTP/HTTPSrequests.

@arangodb/foxxisthenamespaceprovidingthevariousbuildingblocksoftheFoxxmicroserviceframework.

BundledNPMModules

ThefollowingNPMmodulesarepreinstalled:

aqbistheArangoDBQueryBuilderandcanbeusedtoconstructAQLquerieswithachainingJavaScriptAPI.

chaiisafull-featuredassertionlibraryforwritingJavaScripttests.

dedentisasimpleutilityfunctionforformattingmulti-linestrings.

error-stack-parserparsesstacktracesintoamoreusefulformat.

graphql-syncisanArangoDB-compatibleGraphQLserver/schemaimplementation.

highlight.jsisanHTMLsyntaxhighlighter.

i(inflect)isautilitylibraryforinflecting(e.g.pluralizing)words.

iconv-liteisautilitylibraryforconvertingbetweencharacterencodings

joiisavalidationlibrarythatissupportedthroughouttheFoxxframework.

js-yamlisaJavaScriptimplementationoftheYAMLdataformat(apartialsupersetofJSON).

lodashisautilitybeltforJavaScriptprovidingvarioususefulhelperfunctions.

minimatchisaglobmatcherformatchingwildcardsinfilepaths.

qsprovidesutilitiesfordealingwithquerystringsusingadifferentformatthanthequerystringmodule.

semverisautilitylibraryforhandlingsemverversionnumbers.

sinonisamockinglibraryforwritingteststubs,mocksandspies.

timezoneisalibraryforconvertingdatetimevaluesbetweenformatsandtimezones.

JavaScriptModules

630

ArangoDBModuleconstarangodb=require('@arangodb')

Note:ThismoduleshouldnotbeconfusedwiththearangojsJavaScriptdriverwhichcanbeusedtoaccessArangoDBfromoutsidethedatabase.AlthoughtheAPIssharesimilaritiesandthefunctionalityoverlaps,thetwoarenotcompatiblewitheachotherandcannotbeusedinterchangeably.

The dbobjectarangodb.db

Thedbobjectrepresentsthecurrentdatabaseandletsyouaccesscollectionsandrunqueries.Formoreinformationseethedbobjectreference.

Examples

constdb=require('@arangodb').db;

constthirteen=db._query('RETURN5+8').next();

The aqltemplatestringhandlerarangodb.aql

TheaqlfunctionisaJavaScripttemplatestringhandler.ItcanbeusedtowritecomplexAQLqueriesasmulti-linestringswithouthavingtoworryaboutbindVarsandthedistinctionbetweencollectionsandregularparameters.

TouseitjustprefixaJavaScripttemplatestring(theoneswithbackticksinsteadofquotes)withitsimportname(e.g.aql)andpassinvariableslikeyouwouldwitharegulartemplatestring.ThestringwillautomaticallybeconvertedintoanobjectwithqueryandbindVarsattributeswhichyoucanpassdirectlytodb._querytoexecute.Ifyoupassinacollectionitwillbeautomaticallyrecognizedasacollectionreferenceandhandledaccordingly.

TofindoutmoreaboutAQLseetheAQLdocumentation.

Examples

constaql=require('@arangodb').aql;

constfilterValue=23;

constmydata=db._collection('mydata');

constresult=db._query(aql`

FORdIN${mydata}

FILTERd.num>${filterValue}

RETURNd

`).toArray();

The errorsobjectarangodb.errors

ThisobjectprovidesusefulobjectsforeacherrorcodeArangoDBmightuseinArangoErrorerrors.ThisishelpfulwhentryingtocatchspecificerrorsraisedbyArangoDB,e.g.whentryingtoaccessadocumentthatdoesnotexist.EachobjecthasacodepropertycorrespondingtotheerrorNumfoundonArangoErrorerrors.

Foracompletelistoftheerrornamesandcodesyoumayencounterseetheappendixonerrorcodes.

Examples

consterrors=require('@arangodb').errors;

@arangodb

631

try{

someCollection.document('does-not-exist');

}catch(e){

if(e.isArangoError&&e.errorNum===errors.ERROR_ARANGO_DOCUMENT_NOT_FOUND.code){

thrownewError('Documentdoesnotexist');

}

thrownewError('Somethingwentwrong');

}

The timefunctionarangodb.time

Thisfunctionprovidesthecurrenttimeinsecondsasafloatingpointvaluewithmicrosecondprecisison.

ThisfunctioncanbeusedinsteadofDate.now()whenadditionalprecisionisneeded.

Examples

consttime=require('@arangodb').time;

conststart=time();

db._query(someVerySlowQuery);

console.log(`Elapsedtime:${time()-start}secs`);

@arangodb

632

ConsoleModuleglobal.console===require('console')

Note:Youdon'tneedtoloadthismoduledirectly.TheconsoleobjectisgloballydefinedthroughoutArangoDBandprovidesaccesstoallfunctionsinthismodule.

console.assert

console.assert(expression,format,argument1,...)

Teststhatanexpressionistrue.Ifnot,logsamessageandthrowsanexception.

Examples

console.assert(value==="abc","expected:value===abc,actual:",value);

console.debug

console.debug(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultasdebugmessage.Notethatdebugmessageswillonlybeloggediftheserverisstartedwithloglevelsdebugortrace.

Stringsubstitutionpatterns,whichcanbeusedinformat.

%%sstring%%d,%%iinteger%%ffloatingpointnumber%%oobjecthyperlink

Examples

console.debug("%s","thisisatest");

console.dir

console.dir(object)

Logsalistingofallpropertiesoftheobject.

Exampleusage:

console.dir(myObject);

console.errorconsole.error(format,argument1,...)

Formatstheargumentsaccordingto@FA{format}andlogstheresultaserrormessage.

Stringsubstitutionpatterns,whichcanbeusedinformat.

%%sstring%%d,%%iinteger%%ffloatingpointnumber%%oobjecthyperlink

Exampleusage:

console.error("error'%s':%s",type,message);

console

633

console.getline

console.getline()

Readsinalinefromtheconsoleandreturnsitasstring.

console.groupconsole.group(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultaslogmessage.Opensanestedblocktoindentallfuturemessagessent.CallgroupEndtoclosetheblock.Representationofblockisuptotheplatform,itcanbeaninteractiveblockorjustasetofindentedsubmessages.

Exampleusage:

console.group("userattributes");

console.log("name",user.name);

console.log("id",user.id);

console.groupEnd();

console.groupCollapsedconsole.groupCollapsed(format,argument1,...)

Sameasconsole.group.

console.groupEnd

console.groupEnd()

Closesthemostrecentlyopenedblockcreatedbyacalltogroup.

console.infoconsole.info(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultasinfomessage.

Stringsubstitutionpatterns,whichcanbeusedinformat.

%%sstring%%d,%%iinteger%%ffloatingpointnumber%%oobjecthyperlink

Exampleusage:

console.info("The%sjumpedover%dfences",animal,count);

console.log

console.log(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultaslogmessage.Thisisanaliasforconsole.info.

console.timeconsole.time(name)

Createsanewtimerunderthegivenname.CalltimeEndwiththesamenametostopthetimerandlogthetimeelapsed.

Exampleusage:

console.time("mytimer");

...

console

634

console.timeEnd("mytimer");//thiswillprinttheelapsedtime

console.timeEndconsole.timeEnd(name)

Stopsatimercreatedbyacalltotimeandlogsthetimeelapsed.

console.timeEnd

console.trace()

LogsastacktraceofJavaScriptexecutionatthepointwhereitiscalled.

console.warnconsole.warn(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultaswarnmessage.

Stringsubstitutionpatterns,whichcanbeusedinformat.

%%sstring%%d,%%iinteger%%ffloatingpointnumber%%oobjecthyperlink

console

635

CryptoModuleconstcrypto=require('@arangodb/crypto')

Thecryptomoduleprovidesimplementationsofvarioushashingalgorithmsaswellascryptographyrelatedfunctions.

Nonces

Thesefunctionsdealwithcryptographicnonces.

createNonce

crypto.createNonce():string

Createsacryptographicnonce.

Returnsthecreatednonce.

checkAndMarkNonce

crypto.checkAndMarkNonce(nonce):void

Checksandmarksanonce.

Arguments

nonce:string

Thenoncetocheckandmark.

Returnsnothing.

Randomvalues

Thefollowingfunctionsdealwithgeneratingrandomvalues.

rand

crypto.rand():number

Generatesarandomintegerthatmaybepositive,negativeorevenzero.

Returnsthegeneratednumber.

genRandomAlphaNumbers

crypto.genRandomAlphaNumbers(length):string

Generatesastringofrandomalpabeticalcharactersanddigits.

Arguments

length:number

Thelengthofthestringtogenerate.

Returnsthegeneratedstring.

genRandomNumbers

crypto.genRandomNumbers(length):string

Generatesastringofrandomdigits.

crypto

636

Arguments

length:number

Thelengthofthestringtogenerate.

Returnsthegeneratedstring.

genRandomSalt

crypto.genRandomSalt(length):string

Generatesastringofrandom(printable)ASCIIcharacters.

Arguments

length:number

Thelengthofthestringtogenerate.

Returnsthegeneratedstring.

JSONWebTokens(JWT)

ThesemethodsimplementtheJSONWebTokenstandard.

jwtEncode

crypto.jwtEncode(key,message,algorithm):string

GeneratesaJSONWebTokenforthegivenmessage.

Arguments

key:string|null

Thesecretcryptographickeytobeusedtosignthemessageusingthegivenalgorithm.Notethatthisfunctionwillraiseanerrorifthekeyisomittedbutthealgorithmexpectsakey,andalsoifthealgorithmdoesnotexpectakeybutakeyisprovided(e.g.whenusing"none").

message:string

MessagetobeencodedasJWT.Notethatthemessagewillonlybebase64-encodedandsigned,notencrypted.Donotstoresensitiveinformationintokensunlesstheywillonlybehandledbytrustedparties.

algorithm:string

Nameofthealgorithmtouseforsigningthemessage,e.g."HS512".

ReturnstheJSONWebToken.

jwtDecode

crypto.jwtDecode(key,token,noVerify):string|null

Arguments

key:string|null

Thesecretcryptographickeythatwasusedtosignthemessageusingthealgorithmindicatedbythetoken.Notethatthisfunctionwillraiseanerrorifthekeyisomittedbutthealgorithmexpectsakey.

Ifthealgorithmdoesnotexpectakeybutakeyisprovided,thetokenwillfailtoverify.

token:string

Thetokentodecode.

crypto

637

Notethatthefunctionwillraiseanerrorifthetokenismalformed(e.g.doesnothaveexactlythreesegments).

noVerify:boolean(Default:false)

Whetherverificationshouldbeskipped.Ifthisissettotruethesignatureofthetokenwillnotbeverified.Otherwisethefunctionwillraiseanerrorifthesignaturecannotbeverifiedusingthegivenkey.

ReturnsthedecodedJSONmessageornullifnotokenisprovided.

jwtAlgorithms

AhelperobjectcontainingthesupportedJWTalgorithms.EachattributenamecorrespondstoaJWTalgandthevalueisanobjectwithsignandverifymethods.

jwtCanonicalAlgorithmName

crypto.jwtCanonicalAlgorithmName(name):string

AhelperfunctionthattranslatesaJWTalgvaluefoundinaJWTheaderintothecanonicalnameofthealgorithminjwtAlgorithms.Raisesanerrorifnoalgorithmwithamatchingnameisfound.

Arguments

name:string

Algorithmnametolookup.

Returnsthecanonicalnameforthealgorithm.

Hashingalgorithms

md5crypto.md5(message):string

HashesthegivenmessageusingtheMD5algorithm.

Arguments

message:string

Themessagetohash.

Returnsthecryptographichash.

sha1

crypto.sha1(message):string

HashesthegivenmessageusingtheSHA-1algorithm.

Arguments

message:string

Themessagetohash.

Returnsthecryptographichash.

sha224

crypto.sha224(message):string

HashesthegivenmessageusingtheSHA-224algorithm.

Arguments

crypto

638

message:string

Themessagetohash.

Returnsthecryptographichash.

sha256crypto.sha256(message):string

HashesthegivenmessageusingtheSHA-256algorithm.

Arguments

message:string

Themessagetohash.

Returnsthecryptographichash.

sha384crypto.sha384(message):string

HashesthegivenmessageusingtheSHA-384algorithm.

Arguments

message:string

Themessagetohash.

Returnsthecryptographichash.

sha512

crypto.sha512(message):string

HashesthegivenmessageusingtheSHA-512algorithm.

Arguments

message:string

Themessagetohash.

Returnsthecryptographichash.

Miscellaneous

constantEqualscrypto.constantEquals(str1,str2):boolean

Comparestwostrings.Thisfunctioniteratesovertheentirelengthofbothstringsandcanhelpmakingcertaintimingattacksharder.

Arguments

str1:string

Thefirststringtocompare.

str2:string

Thesecondstringtocompare.

Returnstrueifthestringsareequal,falseotherwise.

pbkdf2

crypto

639

crypto.pbkdf2(salt,password,iterations,keyLength):string

GeneratesaPBKDF2-HMAC-SHA1hashofthegivenpassword.

Arguments

salt:string

Thecryptographicsalttohashthepasswordwith.

password:string

Themessageorpasswordtohash.

iterations:number

Thenumberofiterations.Thisshouldbeaveryhighnumber.OWASPrecommended64000iterationsin2012andrecommendsdoublingthatnumbereverytwoyears.

WhenusingPBKDF2forpasswordhashesitisalsorecommendedtoaddarandomvalue(typicallybetween0and32000)tothatnumberthatisdifferentforeachuser.

keyLength:number

Thekeylength.

Returnsthecryptographichash.

hmac

crypto.hmac(key,message,algorithm):string

GeneratesanHMAChashofthegivenmessage.

Arguments

key:string

Thecryptographickeytousetohashthemessage.

message:string

Themessagetohash.

algorithm:string

Thenameofthealgorithmtouse.

Returnsthecryptographichash.

crypto

640

FilesystemModuleconstfs=require('fs')

TheimplementationtriestofollowtheCommonJSFilesystem/A/0specificationwherepossible.

WorkingDirectory

Thedirectoryfunctionsbelowshouldn'tusethecurrentworkingdirectoryoftheserverlike.or./test.Youwillnotbeabletotellwhethertheenvironmenttheserverisrunninginwillpermitdirectorylisting,readingorwritingoffiles.

YoushouldeitherbaseyourdirectorieswithgetTempPath(),orasafoxxserviceusethemodule.context.basePath.

SingleFileDirectoryManipulation

exists

checksifafileofanytypeordirectoryexistsfs.exists(path)

Returnstrueifafile(ofanytype)oradirectoryexistsatagivenpath.Ifthefileisabrokensymboliclink,returnsfalse.

isFile

testsifpathisafilefs.isFile(path)

Returnstrueifthepathpointstoafile.

isDirectory

testsifpathisadirectoryfs.isDirectory(path)

Returnstrueifthepathpointstoadirectory.

size

getsthesizeofafilefs.size(path)

Returnsthesizeofthefilespecifiedbypath.

mtime

getsthelastmodificationtimeofafilefs.mtime(filename)

Returnsthelastmodificationdateofthespecifiedfile.ThedateisreturnedasaUnixtimestamp(numberofsecondselapsedsinceJanuary11970).

pathSeparatorfs.pathSeparator

Ifyouwanttocombinetwopathsyoucanusefs.pathSeparatorinsteadof/or\.

join

fs.join(path,filename)

Thefunctionreturnsthecombinationofthepathandfilename,e.g.fs.join(Hello/World,foo.bar)wouldreturnHello/World/foo.bar.

getTempFile

fs

641

returnsthenamefora(new)temporaryfilefs.getTempFile(directory,createFile)

Returnsthenameforanewtemporaryfileindirectorydirectory.IfcreateFileistrue,anemptyfilewillbecreatedsonootherprocesscancreateafileofthesamename.

Note:Thedirectorydirectorymustexist.

getTempPath

returnsthetemporarydirectoryfs.getTempPath()

Returnstheabsolutepathofthetemporarydirectory

makeAbsolute

makesagivenpathabsolutefs.makeAbsolute(path)

Returnsthegivenstringifitisanabsolutepath,otherwiseanabsolutepathtothesamelocationisreturned.

chmod

setsfilepermissionsofspecifiedfiles(nonwindowsonly)fs.exists(path)

Returnstrueonsuccess.

list

returnsthedirectorylistingfs.list(path)

Thefunctionsreturnsthenamesofallthefilesinadirectory,inlexicallysortedorder.Throwsanexceptionifthedirectorycannotbetraversed(orpathisnotadirectory).

Note:thismeansthatlist("x")ofadirectorycontaining"a"and"b"wouldreturn["a","b"],not["x/a","x/b"].

listTree

returnsthedirectorytreefs.listTree(path)

Thefunctionreturnsanarraythatstartswiththegivenpath,andallofthepathsrelativetothegivenpath,discoveredbyadepthfirsttraversalofeverydirectoryinanyvisiteddirectory,reportingbutnottraversingsymboliclinkstodirectories.Thefirstpathisalways"" ,thepathrelativetoitself.

makeDirectory

createsadirectoryfs.makeDirectory(path)

Createsthedirectoryspecifiedbypath.

makeDirectoryRecursive

createsadirectoryfs.makeDirectoryRecursive(path)

Createsthedirectoryhierarchyspecifiedbypath.

remove

removesafilefs.remove(filename)

Removesthefilefilenameatthegivenpath.Throwsanexceptionifthepathcorrespondstoanythingthatisnotafileorasymboliclink.If"path"referstoasymboliclink,removesthesymboliclink.

removeDirectory

fs

642

removesanemptydirectoryfs.removeDirectory(path)

Removesadirectoryifitisempty.Throwsanexceptionifthepathisnotanemptydirectory.

removeDirectoryRecursive

removesadirectoryfs.removeDirectoryRecursive(path)

Removesadirectorywithallsubelements.Throwsanexceptionifthepathisnotadirectory.

FileIO

read

readsinafilefs.read(filename)

Readsinafileandreturnsthecontentasstring.PleasenotethatthefilecontentmustbeencodedinUTF-8.

read64

readsinafileasbase64fs.read64(filename)

Readsinafileandreturnsthecontentasstring.ThefilecontentisBase64encoded.

readBuffer

readsinafilefs.readBuffer(filename)

ReadsinafileandreturnsitscontentinaBufferobject.

readFileSyncfs.readFileSync(filename,encoding)

Readsthecontentsofthefilespecifiedinfilename.Ifencodingisspecified,thefilecontentswillbereturnedasastring.Supportedencodingsare:

utf8orutf-8ascii

base64

ucs2orucs-2utf16leorutf16behex

Ifnoencodingisspecified,thefilecontentswillbereturnedinaBufferobject.

write

fs.write(filename,content)

Writesthecontentintoafile.ContentcanbeastringoraBufferobject.Ifthefilealreadyexists,itistruncated.

writeFileSync

fs.writeFileSync(filename,content)

Thisisanaliasforfs.write(filename,content).

appendfs.append(filename,content)

Writesthecontentintoafile.ContentcanbeastringoraBufferobject.Ifthefilealreadyexists,thecontentisappendedattheend.

fs

643

RecursiveManipulation

copyRecursive

copiesadirectorystructurefs.copyRecursive(source,destination)

Copiessourcetodestination.Exceptionswillbethrownon:

Failuretocopythefilespecifyingadirectoryfordestinationwhensourceisafilespecifyingadirectoryassourceanddestination

CopyFile

copiesafileintoatargetfilefs.copyFile(source,destination)

Copiessourcetodestination.IfDestinationisadirectory,afileofthesamenamewillbecreatedinthatdirectory,elsethecopywillgetthespecifiedfilename.

move

renamesafilefs.move(source,destination)

Movessourcetodestination.Failuretomovethefile,orspecifyingadirectoryfordestinationwhensourceisafilewillthrowanexception.Likewise,specifyingadirectoryassourceanddestinationwillfail.

ZIP

unzipFile

unzipsafilefs.unzipFile(filename,outpath,skipPaths,overwrite,password)

Unzipsthezipfilespecifiedbyfilenameintothepathspecifiedbyoutpath.Overwritesanyexistingtargetfilesifoverwriteissettotrue.

Returnstrueifthefilewasunzippedsuccessfully.

zipFile

zipsafilefs.zipFile(filename,chdir,files,password)

Storesthefilesspecifiedbyfilesinthezipfilefilename.Ifthefilefilenamealreadyexists,anerroristhrown.Thelistofinputfilesfilesmustbegivenasalistofabsolutefilenames.Ifchdirisnotempty,thechdirprefixwillbestrippedfromthefilenameinthezipfile,sowhenitisunzippedfilenameswillberelative.Specifyingapasswordisoptional.

Returnstrueifthefilewaszippedsuccessfully.

fs

644

RequestModuleconstrequest=require('@arangodb/request')

TherequestmoduleprovidesthefunctionalityformakingHTTPrequests.

MakingHTTPrequests

HTTPmethodhelpers

InadditiontotherequestfunctionconvenienceshorthandsareavailableforeachHTTPmethodintheformof,i.e.:

request.head(url,options)

request.get(url,options)

request.post(url,options)

request.put(url,options)

request.delete(url,options)

request.patch(url,options)

Theseareequivalenttousingtherequestfunctiondirectly,i.e.:

request[method](url,options)

//isequivalentto

request({method,url,...options});

Forexample:

constrequest=require('@arangodb/request');

request.get('http://localhost',{headers:{'x-session-id':'keyboardcat'}});

//isequivalentto

request({

method:'get',

url:'http://localhost',

headers:{'x-session-id':'keyboardcat'}

});

Therequestfunction

TherequestfunctioncanbeusedtomakeHTTPrequests.

request(options)

PerformsanHTTPrequestandreturnsaResponseobject.

Parameter

Therequestfunctiontakesthefollowingoptions:

urloruri:thefully-qualifiedURLoraparsedURLfromurl.parse.qs(optional):objectcontainingquerystringvaluestobeappendedtotheURL.useQuerystring:iftrue,usequerystringmoduletohandlequerystrings,otherwiseuseqsmodule.Default:false.method(optional):HTTPmethod(case-insensitive).Default:"GET".headers(optional):HTTPheaders(case-insensitive).Default:{}.body(optional):requestbody.MustbeastringorBuffer,oraJSONserializablevalueifjsonistrue.json:iftrue,bodywillbeserializedtoaJSONstringandtheContent-Typeheaderwillbesetto"application/json".AdditionallytheresponsebodywillalsobeparsedasJSON(unlessencodingissettonull).Default:false.form(optional):whensettoastringorobjectandnobodyhasbeenset,bodywillbesettoaquerystringrepresentationofthatvalueandtheContent-Typeheaderwillbesetto"application/x-www-form-urlencoded".AlsoseeuseQuerystring.auth(optional):anobjectwiththepropertiesusernameandpasswordforHTTPBasicauthenticationorthepropertybearerfor

request

645

HTTPBearertokenauthentication.

sslProtocol(optional):whichtlsversionshouldbeusedtoconnecttotheurl.Thedefaultis4whichisTLS1.0.Seesslprotocolformoreopitions.followRedirect:whetherHTTP3xxredirectsshouldbefollowed.Default:true.maxRedirects:themaximumnumberofredirectstofollow.Default:10.encoding:encodingtobeusedfortheresponsebody.Ifsettonull,theresponsebodywillbereturnedasaBuffer.Default:"utf-8".timeout:numberofsecondstowaitforaresponsebeforeabortingtherequest.returnBodyOnError:whethertheresponsebodyshouldbereturnedevenwhentheserverresponseindicatesanerror.Default:true.

ThefunctionreturnsaResponseobjectwiththefollowingproperties:

rawBody:therawresponsebodyasaBuffer.body:theparsedresponsebody.Ifencodingwasnotsettonull,thisisastring.Ifadditionallyjsonwassettotrueandtheresponsebodyiswell-formedJSON,thisistheparsedJSONdata.headers:anobjectcontainingtheresponseheaders.OtherwisethisisidenticaltorawBody.statusCodeandstatus:theHTTPstatuscodeoftheresponse,e.g.404.message:theHTTPstatusmessageoftheresponse,e.g.NotFound.

Forms

Therequestmodulesupportsapplication/x-www-form-urlencoded(URLencoded)formuploads:

constrequest=require('@arangodb/request');

varres=request.post('http://service.example/upload',{form:{key:'value'}});

//or

varres=request.post({url:'http://service.example/upload',form:{key:'value'}});

//or

varres=request({

method:'post',

url:'http://service.example/upload',

form:{key:'value'}

});

Formdatawillbeencodedusingtheqsmodulebydefault.

Ifyouwanttousethequerystringmoduleinstead,simplyusetheuseQuerystringoption.

JSON

IfyouwanttosubmitJSON-serializablevaluesasrequestbodies,justsetthejsonoption:

constrequest=require('@arangodb/request');

varres=request.post('http://service.example/notify',{body:{key:'value'},json:true});

//or

varres=request.post({url:'http://service.example/notify',body:{key:'value'},json:true});

//or

varres=request({

method:'post',

url:'http://service.example/notify',

body:{key:'value'},

json:true

});

HTTPauthentication

TherequestmodulesupportsbothHTTPBasicauthentication.Justpassthecredentialsviatheauthoption:

constrequest=require('@arangodb/request');

varres=request.get(

request

646

'http://service.example/secret',

{auth:{username:'jcd',password:'bionicman'}}

);

//or

varres=request.get({

url:'http://service.example/secret',

auth:{username:'jcd',password:'bionicman'}

});

//or

varres=request({

method:'get',

url:'http://service.example/secret',

auth:{username:'jcd',password:'bionicman'}

});

AlternativelyyoucansupplythecredentialsviatheURL:

constrequest=require('@arangodb/request');

varusername='jcd';

varpassword='bionicman';

varres=request.get(

'http://'+

encodeURIComponent(username)+

':'+

encodeURIComponent(password)+

'@service.example/secret'

);

YoucanalsouseBearertokenauthentication:

constrequest=require('@arangodb/request');

varres=request.get(

'http://service.example/secret',

{auth:{bearer:'keyboardcat'}}

);

//or

varres=request.get({

url:'http://service.example/secret',

auth:{bearer:'keyboardcat'}

});

//or

varres=request({

method:'get',

url:'http://service.example/secret',

auth:{bearer:'keyboardcat'}

});

request

647

Module"actions"constactions=require('@arangodb/actions')

Theactionmoduleprovidestheinfrastructurefordefininglow-levelHTTPactions.

IfyouwanttodefineHTTPendpointsinArangoDByoushouldprobablyusetheFoxxmicroserviceframeworkinstead.

Basics

Errormessage

actions.getErrorMessage(code)

Returnstheerrormessageforanerrorcode.

StandardHTTPResultGeneratorsactions.defineHttp(options)

Definesanewaction.Theoptionsareasfollows:

options.url

TheURL,whichcanbeusedtoaccesstheaction.Thispathmightcontainslashes.Notethatthisactionwillalsobecalled,ifaurlisgivensuchthatoptions.urlisaprefixofthegivenurlandnolongerdefinitionmatches.

options.prefix

Iffalse,thenonlyusetheactionforexactmatches.Thedefaultistrue.

options.callback(request,response)

Therequestargumentcontainsadescriptionoftherequest.Arequestparameterfooisaccessibleasrequest.parametrs.foo.Arequestheaderbarisaccessibleasrequest.headers.bar.Assumethattheactionisdefinedfortheurl/foo/barandtherequesturlis/foo/bar/hugo/egon.Thenthesuffixparts["hugo","egon"]areavailibleinrequest.suffix.

Thecallbackmustdefinefilltheresponse.

response.responseCode:theresponsecoderesponse.contentType:thecontenttypeoftheresponseresponse.body:thebodyoftheresponse

YoucanusethefunctionsResultOkandResultErrortoeasilygeneratearesponse.

Resultok

actions.resultOk(req,res,code,result,headers)

Thefunctiondefinesaresponse.codeisthestatuscodetoreturn.resultistheresultobject,whichwillbereturnedasJSONobjectinthebody.headersisanarrayofheaderstoreturned.Thefunctionaddstheattributeerrorwithvaluefalseandcodewithvaluecodetotheresult.

Resultbad

actions.resultBad(req,res,error-code,msg,headers)

Thefunctiongeneratesanerrorresponse.

Resultnotfoundactions.resultNotFound(req,res,code,msg,headers)

actions

648

Thefunctiongeneratesanerrorresponse.

Resultunsupported

actions.resultUnsupported(req,res,headers)

Thefunctiongeneratesanerrorresponse.

Resulterror

actions.resultError(req,res,code,errorNum,errorMessage,headers,keyvals)*

Thefunctiongeneratesanerrorresponse.TheresponsebodyisanarraywithanattributeerrorMessagecontainingtheerrormessageerrorMessage,errorcontainingtrue,codecontainingcode,errorNumcontainingerrorNum,anderrorMessagecontainingtheerrormessageerrorMessage.keyvalsaremixedintotheresult.

ResultnotImplementedactions.resultNotImplemented(req,res,msg,headers)

Thefunctiongeneratesanerrorresponse.

Resultpermanentredirect

actions.resultPermanentRedirect(req,res,options,headers)

Thefunctiongeneratesaredirectresponse.

Resulttemporaryredirectactions.resultTemporaryRedirect(req,res,options,headers)

Thefunctiongeneratesaredirectresponse.

ArangoDBResultGenerators

Collectionnotfoundactions.collectionNotFound(req,res,collection,headers)

Thefunctiongeneratesanerrorresponse.

Indexnotfound

actions.indexNotFound(req,res,collection,index,headers)

Thefunctiongeneratesanerrorresponse.

Resultexceptionactions.resultException(req,res,err,headers,verbose)

Thefunctiongeneratesanerrorresponse.If@FA{verbose}issettotrueornotspecified(thedefault),thentheerrorstacktracewillbeincludedintheerrormessageifavailable.If@FA{verbose}isastringitwillbeprependedbeforetheerrormessageandthestacktracewillalsobeincluded.

actions

649

QueriesModuleconstqueries=require('@arangodb/aql/queries')

ThequerymoduleprovidestheinfrastructureforworkingwithcurrentlyrunningAQLqueriesviaarangosh.

Properties

queries.properties()Returnstheserverscurrentquerytrackingconfiguration;wechangetheslowquerythresholdtogetbetterresults:

arangosh>varqueries=require("@arangodb/aql/queries");

arangosh>queries.properties();

arangosh>queries.properties({slowQueryThreshold:1});

showexecutionresults

Currentlyrunningqueries

Wecreateataskthatspawnsqueries,sowehaveniceoutput.Sincethistaskusesresources,youmaywanttoincreaseperiod(andnotforgettoremoveit...afterwards):

arangosh>vartheQuery='FORsleepLoooongIN1..5LETsleepLoooonger=SLEEP(1000)RETURN

sleepLoooong';

arangosh>vartasks=require("@arangodb/tasks");

arangosh>tasks.register({

........>id:"mytask-1",

........>name:"thisisasampletasktospawnaslowaqlquery",

........>command:"require('@arangodb').db._query('"+theQuery+"');"

........>});

arangosh>queries.current();

showexecutionresultsThefunctionreturnsthecurrentlyrunningAQLqueriesasanarray.

SlowqueriesThefunctionreturnsthelastAQLqueriesthatexceededtheslowquerythresholdasanarray:

arangosh>queries.slow();

[]

ClearslowqueriesClearthelistofslowAQLqueries:

arangosh>queries.clearSlow();

{

"code":200

}

arangosh>queries.slow();

[]

queries

650

KillKillarunningAQLquery:

arangosh>varrunningQueries=queries.current().filter(function(query){

........>returnquery.query===theQuery;

........>});

arangosh>queries.kill(runningQueries[0].id);

{

"code":200

}

queries

651

Write-aheadlogconstwal=require('internal').wal

Thismoduleprovidesfunctionalityforadministeringthewrite-aheadlogs.Mostofthesefunctionsonlyreturnsensiblevalueswheninvokedwiththemmfilesenginebeingactive.

Configuration

retrievestheconfigurationofthewrite-aheadloginternal.wal.properties()

Retrievestheconfigurationofthewrite-aheadlog.TheresultisaJSONarraywiththefollowingattributes:

allowOversizeEntries:whetherornotoperationsthatarebiggerthanasinglelogfilecanbeexecutedandstoredlogfileSize:thesizeofeachwrite-aheadlogfilehistoricLogfiles:themaximumnumberofhistoriclogfilestokeepreserveLogfiles:themaximumnumberofreservelogfilesthatArangoDBallocatesinthebackgroundsyncInterval:theintervalforautomaticsynchronizationofnot-yetsynchronizedwrite-aheadlogdata(inmilliseconds)throttleWait:themaximumwaittimethatoperationswillwaitbeforetheygetabortedifcaseofwrite-throttling(inmilliseconds)throttleWhenPending:thenumberofunprocessedgarbage-collectionoperationsthat,whenreached,willactivatewrite-throttling.Avalueof0meansthatwrite-throttlingwillnotbetriggered.

Examples

arangosh>require("internal").wal.properties();

showexecutionresultsconfiguresthewrite-aheadloginternal.wal.properties(properties)

Configuresthebehaviorofthewrite-aheadlog.propertiesmustbeaJSONJSONobjectwiththefollowingattributes:

allowOversizeEntries:whetherornotoperationsthatarebiggerthanasinglelogfilecanbeexecutedandstoredlogfileSize:thesizeofeachwrite-aheadlogfilehistoricLogfiles:themaximumnumberofhistoriclogfilestokeepreserveLogfiles:themaximumnumberofreservelogfilesthatArangoDBallocatesinthebackgroundthrottleWait:themaximumwaittimethatoperationswillwaitbeforetheygetabortedifcaseofwrite-throttling(inmilliseconds)throttleWhenPending:thenumberofunprocessedgarbage-collectionoperationsthat,whenreached,willactivatewrite-throttling.Avalueof0meansthatwrite-throttlingwillnotbetriggered.

Specifyinganyoftheaboveattributesisoptional.Notspecifiedattributeswillbeignoredandtheconfigurationforthemwillnotbemodified.

Examples

arangosh>require("internal").wal.properties({

........>allowOverSizeEntries:true,

........>logfileSize:32*1024*1024});

showexecutionresults

Flushing

flushesthecurrentlyopenWALlogfileinternal.wal.flush(waitForSync,waitForCollector)

Flushesthewrite-aheadlog.Byflushingthecurrentlyactivewrite-aheadlogfile,thedatainitcanbetransferredtocollectionjournalsanddatafiles.Thisisusefultoensurethatalldataforacollectionispresentinthecollectionjournalsanddatafiles,forexample,whendumpingthedataofacollection.

Write-aheadlog

652

ThewaitForSyncoptiondetermineswhetherornottheoperationshouldblockuntilthenot-yetsynchronizeddatainthewrite-aheadlogwassynchronizedtodisk.

ThewaitForCollectoroperationcanbeusedtospecifythattheoperationshouldblockuntilthedataintheflushedloghasbeencollectedbythewrite-aheadloggarbagecollector.Notethatsettingthisoptiontotruemightblockforalongtimeiftherearelong-runningtransactionsandthewrite-aheadloggarbagecollectorcannotfinishgarbagecollection.

Examples

arangosh>require("internal").wal.flush();

Write-aheadlog

653

TaskManagementconsttasks=require('@arangodb/tasks')

Note:IfyouaretryingtoscheduletasksinFoxxyoushouldconsiderusingtheFoxxqueuesmoduleinstead,whichprovidesamorehigh-levelAPIthatalsopersiststasksacrossreboots.

IntroductiontoTaskManagementinArangoDB

ArangoDBcanexecuteuser-definedJavaScriptfunctionsasone-shotorperiodictasks.Thisfunctionalitycanbeusedtoimplementtimedorrecurringjobsinthedatabase.

TasksinArangoDBconsistofaJavaScriptsnippetorfunctionthatisexecutedwhenthetaskisscheduled.Ataskcanbeaone-shottask(meaningitisrunonceandnotrepeated)oraperiodictask(meaningthatitisre-scheduledaftereachexecution).Taskscanhaveoptionalparameters,whicharedefinedattasksetuptime.Theparametersspecifiedattasksetuptimewillbepassedasargumentstothetaskwheneveritgetsexecuted.PeriodicTaskshaveanexecutionfrequencythatneedstobespecifiedwhenthetaskissetup.One-shottaskshaveaconfigurabledelayafterwhichthey'llgetexecuted.

Taskswillbeexecutedontheservertheyhavebeensetupon.Taskswillnotbeshippedaroundinacluster.Ataskwillbeexecutedinthecontextofthedatabaseitwascreatedin.However,whendroppingadatabase,anytasksthatwerecreatedinthecontextofthisdatabasewillremainactive.Itisthereforesensibletofirstunregisterallactivetasksforadatabasebeforedroppingthedatabase.

TasksregisteredinArangoDBwillbeexecuteduntiltheservergetsshutdownorrestarted.Afterarestartoftheserver,anyuser-definedone-shotorperiodictaskswillbelost.

CommandsforWorkingwithTasksArangoDBprovidesthefollowingcommandsforworkingwithtasks.Allcommandscanbeaccessedviathetasksmodule,whichcanbeloadedlikethis:

require("@arangodb/tasks")

PleasenotethatthetasksmoduleisavailableinsidetheArangoDBserveronly.ItcannotbeusedfromtheArangoShellorArangoDB'swebinterface.

Registeratask

Toregisteratask,theJavaScriptsnippetorfunctionneedstobespecifiedinadditiontotheexecutionfrequency.Optionally,ataskcanhaveanidandaname.Ifnoidisspecified,itwillbeauto-assignedforanewtask.Thetaskidisalsothemeanstoaccessorunregisteratasklater.Tasknamesareinformationalonly.Theycanbeusedtomakeataskdistinguishablefromothertasksalsorunningontheserver.

Thefollowingserver-sidecommandsregisteratask.ThecommandtobeexecutedisaJavaScriptstringsnippetwhichprintsamessagetotheserver'slogfile:

consttasks=require("@arangodb/tasks");

tasks.register({

id:"mytask-1",

name:"thisisasnippettask",

period:15,

command:"require('console').log('hellofromsnippettask');"

});

Theabovehasregisterataskwithidmytask-1,whichwillbeexecutedevery15secondsontheserver.Thetaskwillwritealogmessagewheneveritisinvoked.

TaskscanalsobesetupusingaJavaScriptcallbackfunctionlikethis:

TaskManagement

654

consttasks=require("@arangodb/tasks");

tasks.register({

id:"mytask-2",

name:"thisisafunctiontask",

period:15,

command:function(){

require('console').log('hellofromfunctiontask');

}

});

Itisimportanttonotethatthecallbackfunctionislateboundandwillbeexecutedinadifferentcontextthaninthecreationcontext.Thecallbackfunctionmustthereforenotaccessanyvariablesdefinedoutsideofitsownscope.Thecallbackfunctioncanstilldefineanduseitsownvariables.

Topassparameterstoatask,theparamsattributecanbesetwhenregisteringatask.NotethattheparametersarelimitedtodatatypesusableinJSON(meaningnocallbackfunctionscanbepassedasparametersintoatask):

consttasks=require("@arangodb/tasks");

tasks.register({

id:"mytask-3",

name:"thisisaparametertask",

period:15,

command:function(params){

vargreeting=params.greeting;

vardata=JSON.stringify(params.data);

require('console').log('%sfromparametertask:%s',greeting,data);

},

params:{greeting:"hi",data:"howareyou?"}

});

Registeringaone-shottaskworksthesameway,exceptthattheperiodattributemustbeomitted.Ifperiodisomitted,thenthetaskwillbeexecutedjustonce.Thetaskinvocationdelaycanoptionallybespecifiedwiththeoffsetattribute:

consttasks=require("@arangodb/tasks");

tasks.register({

id:"mytask-once",

name:"thisisaone-shottask",

offset:10,

command:function(params){

require('console').log('youwillseemejustonce!');

}

});

Note:Whenspecifyinganoffsetvalueof0,ArangoDBwillinternallyaddaverysmallvaluetotheoffsetsowillbeslightlygreaterthanzero.

Unregisteratask

Afterataskhasbeenregistered,itcanbeunregisteredusingitsid:

consttasks=require("@arangodb/tasks");

tasks.unregister("mytask-1");

Notethatunregisteringanon-existingtaskwillthrowanexception.

ListalltasksTogetanoverviewofwhichtasksareregistered,thereisthegetmethod.Ifthegetmethodiscalledwithoutanyarguments,itwillreturnanarrayofalltasks:

TaskManagement

655

consttasks=require("@arangodb/tasks");

tasks.get();

Ifgetiscalledwithataskidargument,itwillreturninformationaboutthisparticulartask:

consttasks=require("@arangodb/tasks");

tasks.get("mytask-3");

Thecreatedattributeofataskrevealswhenataskwascreated.ItisreturnedasaUnixtimestamp.

TaskManagement

656

DeprecatedFeatureslistedinthissectionshouldnolongerbeused,becausetheyareconsideredobsoleteandmaygetremovedinafuturerelease.Theyarecurrentlykeptforbackwardcompatibility.Thereareusuallybetteralternativestoreplacetheoldfeatureswith:

SimpleQueries:Ideomaticinterfaceinarangoshtoperformtrivialqueries.TheyaresupersededbyAQLqueries,whichcanalsoberuninarangosh.AQLisalanguageonitsownandwaymorepowerfulthanSimpleQueriescouldeverbe.Infact,the(stillsupported)SimpleQueriesaretranslatedinternallytoAQL,thentheAQLqueryisoptimizedandrunagainstthedatabaseinrecentversions,becauseofbetterperformanceandreducedmaintenancecomplexity.

Actions:SnippetsofJavaScriptcodeontheserver-sideforminimalcustomendpoints.SincetheFoxxrevampin3.0,itbecamereallyeasytowriteFoxxMicroservices,whichallowyoutodefinecustomendpointsevenwithcomplexbusinesslogic.

Deprecated

657

SimpleQueries

ItisrecommendedtouseAQLinstead.

Simplequeriescanbeusedifthequeryconditionisstraightforward,i.e.,adocumentreference,alldocuments,aquery-by-example,orasimplegeoquery.Inasimplequeryyoucanspecifyexactlyonecollectionandonequerycriteria.InthefollowingsectionswedescribetheJavaScriptshellinterfaceforsimplequeries,whichyoucanusewithintheArangoDBshellandwithinactionsandtransactions.ForotherlanguagesseethecorrespondinglanguageAPIdocumentation.

YoucanfindalistofqueriesatCollectionMethods.

SimpleQueries

658

Pagination

ItisrecommendedtouseAQLinstead,seetheLIMIToperation.

If,forexample,youdisplaytheresultofausersearch,thenyouareingeneralnotinterestedinthecompletedresultset,butonlythefirst10orsodocuments.Ormaybethenext10documentsforthesecondpage.Inthiscase,youcantheskipandlimitoperators.TheseoperatorsworklikeLIMITinMySQL.

skipusedtogetherwithlimitcanbeusedtoimplementpagination.Theskipoperatorskipsoverthefirstndocuments.So,inordertocreateresultpageswith10resultdocumentsperpage,youcanuseskip(n*10).limit(10)toaccessthe10documentsonthenthpage.Thisresultshouldbesorted,sothatthepaginationworksinapredicableway.

Limit

limitquery.limit(number)

Limitsaresulttothefirstnumberdocuments.Specifyingalimitof0willreturnnodocumentsatall.Ifyoudonotneedalimit,justdonotaddthelimitoperator.Thelimitmustbenon-negative.

Ingeneraltheinputtolimitshouldbesorted.Otherwiseitwillbeunclearwhichdocumentswillbeincludedintheresultset.

Examples

arangosh>db.five.all().toArray();

arangosh>db.five.all().limit(2).toArray();

showexecutionresults

Skip

skipquery.skip(number)

Skipsthefirstnumberdocuments.Ifnumberispositive,thenthisnumberofdocumentsareskippedbeforereturningthequeryresults.

Ingeneraltheinputtoskipshouldbesorted.Otherwiseitwillbeunclearwhichdocumentswillbeincludedintheresultset.

Note:usingnegativeskipvaluesisdeprecatedasofArangoDB2.6andwillnotbesupportedinfutureversionsofArangoDB.

Examples

arangosh>db.five.all().toArray();

arangosh>db.five.all().skip(3).toArray();

showexecutionresultsIgnoreanylimitwithcount:

arangosh>db.five.all().limit(2).count();

null

Countinganylimitorskip:

arangosh>db.five.all().limit(2).count(true);

2

SimpleQueries

659

SimpleQueries

660

ModificationQueries

ItisrecommendedtouseAQLinstead,seeDataModificationQueries.

ArangoDBalsoallowsremoving,replacing,andupdatingdocumentsbasedonanexampledocument.Everydocumentinthecollectionwillbecomparedagainstthespecifiedexampledocumentandbedeleted/replaced/updatedifallattributesmatch.

Thesemethodshouldbeusedwithcautionastheyareintendedtoremoveormodifylotsofdocumentsinacollection.

Allmethodscanoptionallyberestrictedtoaspecificnumberofoperations.However,ifalimitisspecificbutislessthanthenumberofmatches,itwillbeundefinedwhichofthematchingdocumentswillgetremoved/modified.RemovebyExample,ReplacebyExampleandUpdatebyExamplearedescribedwithexamplesinthesubchapterCollectionMethods.

SimpleQueries

661

GeoQueries

ItisrecommendedtouseAQLinstead,seeGeofunctions.

TheArangoDBallowstoselectdocumentsbasedongeographiccoordinates.Inorderforthistowork,ageo-spatialindexmustbedefined.ThisindexwilluseaveryelaboratealgorithmtolookupneighborsthatisamagnitudefasterthanasimpleR*index.

Ingeneralageocoordinateisapairoflatitudeandlongitude,whichmustbothbespecifiedasnumbers.Ageoindexcanbecreatedoncoordinatesthatarestoredinasinglelistattributewithtwoelementslike[-10,+30](latitudefirst,followedbylongitude)oroncoordinatesstoredintwoseparateattributes.

Forexample,toindexthefollowingdocuments,anindexcanbecreatedonthepositionattributeofthedocuments:

db.test.save({position:[-10,30]});

db.test.save({position:[10,45.5]});

db.test.ensureIndex({type:"geo",fields:["position"]});

Ifcoordinatesarestoredintwodistinctattributes,theindexmustbecreatedonthetwoattributes:

db.test.save({latitude:-10,longitude:30});

db.test.save({latitude:10,longitude:45.5});

db.test.ensureIndex({type:"geo",fields:["latitude","longitude"]});

Inordertofindalldocumentswithinagivenradiusaroundacoordinateusethewithinoperator.Inordertofindalldocumentsnearagivendocumentusethenearoperator.

Itispossibletodefinemorethanonegeo-spatialindexpercollection.Inthiscaseyoumustgiveahintusingthegeooperatorwhichofindexesshouldbeusedinaquery.

Near

constructsanearqueryforacollectioncollection.near(latitude,longitude)Thereturnedlistissortedaccordingtothedistance,withthenearestdocumenttothecoordinate(latitude,longitude)comingfirst.Ifthereareneardocumentsofequaldistance,documentsarechosenrandomlyfromthissetuntilthelimitisreached.Itispossibletochangethelimitusingthelimitoperator.Inordertousethenearoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.Note:neardoesnotsupportnegativeskips.//However,youcanstilluselimitfollowedtoskip.collection.near(latitude,longitude).limit(limit)Limitstheresulttolimitdocumentsinsteadofthedefault100.Note:Unlikewithmultipleexplicitlimits,limitwillraisetheimplicitdefaultlimitimposedbywithin.collection.near(latitude,longitude).distance()Thiswilladdanattributedistancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.collection.near(latitude,longitude).distance(name)Thiswilladdanattributenametoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.Note:thenearsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthenearoperatoristousetheAQLNEARfunctioninanAQLqueryasfollows:

FORdocINNEAR(@@collection,@latitude,@longitude,@limit)

RETURNdoc

Examples

Togetthenearesttwolocations:

arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});

SimpleQueries

662

arangosh>for(vari=-90;i<=90;i+=10){

........>for(varj=-180;j<=180;j+=10){

........>db.geo.save({

........>name:"Name/"+i+"/"+j,

........>loc:[i,j]});

........>}}

arangosh>db.geo.near(0,0).limit(2).toArray();

showexecutionresultsIfyouneedthedistanceaswell,thenyoucanusethedistanceoperator:

arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});

arangosh>for(vari=-90;i<=90;i+=10){

........>for(varj=-180;j<=180;j+=10){

........>db.geo.save({

........>name:"Name/"+i+"/"+j,

........>loc:[i,j]});

........>}}

arangosh>db.geo.near(0,0).distance().limit(2).toArray();

showexecutionresults

Within

constructsawithinqueryforacollectioncollection.within(latitude,longitude,radius)Thiswillfindalldocumentswithinagivenradiusaroundthecoordinate(latitude,longitude).Thereturnedarrayissortedbydistance,beginningwiththenearestdocument.Inordertousethewithinoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.collection.within(latitude,longitude,radius).distance()Thiswilladdanattribute_distancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.collection.within(latitude,longitude,radius).distance(name)Thiswilladdanattributenametoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.Note:thewithinsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthewithinoperatoristousetheAQLWITHINfunctioninanAQLqueryasfollows:

FORdocINWITHIN(@@collection,@latitude,@longitude,@radius,@distanceAttributeName)

RETURNdoc

Examples

Tofindalldocumentswithinaradiusof2000kmuse:

arangosh>for(vari=-90;i<=90;i+=10){

........>for(varj=-180;j<=180;j+=10){

........>db.geo.save({name:"Name/"+i+"/"+j,loc:[i,j]});}}

arangosh>db.geo.within(0,0,2000*1000).distance().toArray();

showexecutionresults

Geo

constructsageoindexselectioncollection.geo(location-attribute)Looksupageoindexdefinedonattributelocation_attribute.Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.Thisisusefulforcollectionswithmultipledefinedgeoindexes.collection.geo(location_attribute,true)Looksupageoindexonacompoundattributelocation_attribute.Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.collection.geo(latitude_attribute,longitude_attribute)

SimpleQueries

663

Looksupageoindexdefinedonthetwoattributeslatitude_attributeandlongitude-attribute.Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.Note:thegeosimplequeryhelperfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforrunninggeoqueriesistousetheirAQLequivalents.

Examples

Assumeyouhavealocationstoredaslistintheattributehomeandadestinationstoredintheattributework.Thenyoucanusethegeooperatortoselectwhichgeo-spatialattributes(andthuswhichindex)touseinanearquery.

arangosh>for(i=-90;i<=90;i+=10){

........>for(j=-180;j<=180;j+=10){

........>db.complex.save({name:"Name/"+i+"/"+j,

........>home:[i,j],

........>work:[-i,-j]});

........>}

........>}

........>

arangosh>db.complex.near(0,170).limit(5);

arangosh>db.complex.ensureIndex({type:"geo",fields:["home"]});

arangosh>db.complex.near(0,170).limit(5).toArray();

arangosh>db.complex.geo("work").near(0,170).limit(5);

arangosh>db.complex.ensureIndex({type:"geo",fields:["work"]});

arangosh>db.complex.geo("work").near(0,170).limit(5).toArray();

showexecutionresults

Relatedtopics

OtherArangoDBgeographicfeaturesaredescribedin:

AQLGeofunctionsGeoindexes

SimpleQueries

664

Fulltextqueries

ItisrecommendedtouseAQLinstead,seeFulltextfunctions.

ArangoDBallowstorunqueriesontextcontainedindocumentattributes.Tousethis,afulltextindexmustbedefinedfortheattributeofthecollectionthatcontainsthetext.Creatingtheindexwillparsethetextinthespecifiedattributeforalldocumentsofthecollection.Onlydocumentswillbeindexedthatcontainatextualvalueintheindexedattribute.Forsuchdocuments,thetextvaluewillbeparsed,andtheindividualwordswillbeinsertedintothefulltextindex.

Whenafulltextindexexists,itcanbequeriedusingafulltextquery.

Fulltext

queriesthefulltextindexcollection.fulltext(attribute,query)

Thefulltextsimplequeryfunctionsperformsafulltextsearchonthespecifiedattributeandthespecifiedquery.

Detailsaboutthefulltextquerysyntaxcanbefoundbelow.

Note:thefulltextsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforexecutingfulltextqueriesistouseanAQLqueryusingtheFULLTEXTAQLfunctionasfollows:

FORdocINFULLTEXT(@@collection,@attributeName,@queryString,@limit)

RETURNdoc

Examples

arangosh>db.emails.ensureFulltextIndex("content");

arangosh>db.emails.save({content:

........>"HelloAlice,howareyoudoing?Regards,Bob"});

arangosh>db.emails.save({content:

........>"HelloCharlie,doAliceandBobknowaboutit?"});

arangosh>db.emails.save({content:"Ithinktheydon'tknow.Regards,Eve"});

arangosh>db.emails.fulltext("content","charlie,|eve").toArray();

showexecutionresults

FulltextSyntax:

Inthesimplestform,afulltextquerycontainsjustthesoughtword.Ifmultiplesearchwordsaregiveninaquery,theyshouldbeseparatedbycommas.AllsearchwordswillbecombinedwithalogicalANDbydefault,andonlysuchdocumentswillbereturnedthatcontainallsearchwords.Thisdefaultbehaviorcanbechangedbyprovidingtheextracontrolcharactersinthefulltextquery,whichare:

+:logicalAND(intersection)|:logicalOR(union)-:negation(exclusion)

Examples:

"banana":searchesfordocumentscontaining"banana""banana,apple":searchesfordocumentscontainingboth"banana"AND"apple""banana,|orange":searchesfordocumentscontainingeither"banana"OR"orange"ORboth"banana,-apple":searchesfordocumentsthatcontains"banana"butNOT"apple".

Logicaloperatorsareevaluatedfromlefttoright.

SimpleQueries

665

Eachsearchwordcanoptionallybeprefixedwithcomplete:orprefix:,withcomplete:beingthedefault.Thisallowssearchingforcompletewordsorforwordprefixes.Suffixsearchesoranyotherformsarepartial-wordmatchingarecurrentlynotsupported.

Examples:

"complete:banana":searchesfordocumentscontainingtheexactword"banana""prefix:head":searchesfordocumentswithwordsthatstartwithprefix"head""prefix:head,banana":searchesfordocumentscontainwordsstartingwithprefix"head"andthatalsocontaintheexactword"banana".

Completematchandprefixsearchoptionscanbecombinedwiththelogicaloperators.

Pleasenotethatonlywordswithaminimumlengthwillgetindexed.Thisminimumlengthcanbedefinedwhencreatingthefulltextindex.Forwordstokenization,thelibicutextboundaryanalysisisused,whichtakesintoaccountthedefaultasdefinedatserverstartup(--server.default-languagestartupoption).Generally,thewordboundaryanalysiswillfilteroutpunctuationbutwillnotdomuchmore.

Especiallynowordnormalization,stemming,orsimilarityanalysiswillbeperformedwhenindexingorsearching.Ifanyofthesefeaturesisrequired,itissuggestedthattheuserdoesthetextnormalizationontheclientside,andprovidesforeachdocumentanextraattributecontainingjustacomma-separatedlistofnormalizedwords.Thisattributecanthenbeindexedwithafulltextindex,andtheusercansendfulltextqueriesforthisindex,withthefulltextqueriesalsocontainingthestemmedornormalizedversionsofwordsasrequiredbytheuser.

SimpleQueries

666

ArangoDB'sActions

ItisrecommendedtouseFoxxinstead.

IntroductiontoUserActions

InsomewaysthecommunicationlayeroftheArangoDBserverbehaveslikeaWebserver.UnlikeaWebserver,itnormallyrespondstoHTTPrequestsbydeliveringJSONobjects.Remember,documentsinthedatabasearejustJSONobjects.So,mostofthetimetheHTTPresponsewillcontainaJSONdocumentfromthedatabaseasbody.YoucanextractthedocumentsstoredinthedatabaseusingHTTPGET.YoucanstoredocumentsusingHTTPPOST.

However,thereissomethingmore.Youcanwritesmallsnippets-socalledactions-toextendthedatabase.TheideaofactionsisthatsometimesitisbettertostorepartsofthebusinesslogicwithinArangoDB.

Thesimplestexampleistheageofaperson.Assumeyoustoreinformationaboutpeopleinyourdatabase.Itisananti-patterntostoretheage,becauseitchangeseverynowandthen.Therefore,younormallystorethebirthdayandlettheclientdecidewhattodowithit.However,ifyouhavemanydifferentclients,itmightbeeasiertoenrichthepersondocumentwiththeageusingactionsonceontheserverside.

Or,forinstance,ifyouwanttoapplysomestatisticstolargedata-setsandyoucannoteasilyexpressthisasquery.Youcandefineaactioninsteadoftransferringthewholedatatotheclientanddothecomputationontheclient.

Actionsarealsousefulifyouwanttorestrictandfilterdataaccordingtosomecomplexpermissionsystem.

TheArangoDBservercandeliverallkindsofinformation,JSONbeingonlyonepossibleformat.YoucanalsogenerateHTMLorimages.However,aWebserverisnormallybettersuitedforthetaskasitalsoimplementsvariouscachingstrategies,languageselection,compressionandsoon.Havingsaidthat,therearestillsituationswhereitmightbesuitabletousetheArangoDBtodeliverHTMLpages-staticordynamic.Asimpleexampleisthebuilt-inadministrationinterface.YoucanaccessitusinganymodernbrowserandthereisnoneedforaseparateApacheorIIS.

IngeneralyouwilluseFoxxtoeasilyextendthedatabasewithbusinesslogic.Foxxprovidesansimpletouseinterfacetoactions.

Thefollowingsectionswillexplainthelow-levelactionswithinArangoDBonwhichFoxxisbuiltandshowhowtodefinethem.TheexamplesstartwithdeliveringstaticHTMLpages-evenifthisisnottheprimaryuse-caseforactions.ThelatersectionswillthenshowyouhowtocodesomepiecesofyourbusinesslogicandreturnJSONobjects.

TheinterfaceislooselymodeledaftertheJavaScriptclassesforHTTPrequestandresponsesfoundinnode.jsandthemiddleware/routingaspectsofconnect.jsandexpress.js.

Notethatunlikenode.js,ArangoDBismulti-threadedandthereisnoeasywaytosharestatebetweenqueriesinsidetheJavaScriptengine.Ifsuchstateinformationisrequired,youneedtousethedatabaseitself.

Actions

667

AHelloWorldExample

TheclientAPIorbrowsersendsaHTTPrequesttotheArangoDBserverandtheserverreturnsaHTTPresponsetotheclient.AHTTPrequestconsistsofamethod,normallyGETorPOSTwhenusingabrowser,andarequestpathlike/hello/world.ForarealWebserverthereareazillionofotherthingtoconsider,wewillignorethisforthemoment.TheHTTPresponsecontainsacontenttype,describinghowtointerpretthereturneddata,andthedataitself.

Inthefollowingexample,wewanttodefineanactioninArangoDB,sothattheserverreturnstheHTMLdocument

<html>

<body>

HelloWorld

</body>

</html>

ifaskedGET/hello/world.

Theserverneedstoknowwhatfunctiontocallorwhatdocumenttodeliverifitreceivesarequest.Thisiscalledrouting.AlltheroutinginformationofArangoDBisstoredinacollection_routing.Eachentryinthiscollectionsdescribeshowtodealwithaparticularrequestpath.

Fortheaboveexample,addthefollowingdocumenttothe_routingcollection:

arangosh>db._routing.save({

........>url:{

........>match:"/hello/world"

........>},

........>content:{

........>contentType:"text/html",

........>body:"<html><body>HelloWorld</body></html>"

........>}

........>});

showexecutionresultsInordertoactivatethenewrouting,youmusteitherrestarttheserverorcalltheinternalreloadfunction.

arangosh>require("internal").reloadRouting()

Nowusethebrowserandaccesshttp://localhost:8529/hello/world

YoushouldseetheHelloWorldinourbrowser:

shell>curl--dump-http://localhost:8529/hello/world

HTTP/1.1200OK

content-type:text/html

x-content-type-options:nosniff

"HelloWorld"

MatchingaURLTherearealotofoptionsfortheurlattribute.Ifyoudefinedifferentroutingforthesamepath,thenthefollowingsimpleruleisappliedinordertodeterminewhichmatchwins:Iftherearetwomatches,thenthemorespecificwins.I.e,ifthereisawildcardmatchandanexactmatch,theexactmatchispreferred.Ifthereisashortandalongmatch,thelongermatchwins.

Actions

668

ExactMatch

Ifthedefinitionis

{

url:{

match:"/hello/world"

}

}

thenthematchmustbeexact.Onlytherequestfor/hello/worldwillmatch,everythingelse,e.g./hello/world/myor/hello/world2,willnotmatch.

Thefollowingdefinitionisashort-cutforanexactmatch.

{

url:"/hello/world"

}

Note:WhilethetwodefinitionswillresultinthesameURLmatching,thereisasubtledifferencebetweenthem:

Theformerdefinition(definingurlasanobjectwithamatchattribute)willresultintheURLbeingaccessibleviaallsupportedHTTPmethods(e.g.GET,POST,PUT,DELETE,...),whereasthelatterdefinition(providingastringurlattribute)willresultintheURLbeingaccessibleviaHTTPGETandHTTPHEADonly,withallotherHTTPmethodsbeingdisabled.CallingaURLwithanunsupportedordisabledHTTPmethodwillresultinanHTTP501(notimplemented)error.

PrefixMatch

Ifthedefinitionis

{

url:{

match:"/hello/world/*"

}

}

thenthematchcanbeaprefixmatch.Therequestsfor/hello/world,/hello/world/my,and/hello/world/how/are/youwillallmatch.However/hello/world2doesnotmatch.PrefixmatcheswithinaURLpart,i.e./hello/world*,arenotallowed.Thewildcardmustoccurattheend,i.e.

/hello/*/world

isalsodisallowed.

Ifyoudefinetworoutes

{url:{match:"/hello/world/*"}}

{url:{match:"/hello/world/emil"}}

thenthesecondroutewillbeusedfor/hello/world/emilbecauseitismorespecific.

ParameterizedMatch

Aparameterizedmatchissimilartoaprefixmatch,buttheparametersarealsoallowedinsidetheURLpath.

Ifthedefinitionis

{

url:{

match:"/hello/:name/world"

}

}

Actions

669

thentheURLmusthavethreeparts,thefirstpartbeinghelloandthethirdpartworld.Forexample,/hello/emil/worldwillmatch,while/hello/emil/meyer/worldwillnot.

ConstraintMatch

Aconstraintmatchissimilartoaparameterizedmatch,buttheparameterscancarryconstraints.

Ifthedefinitionis

{

url:{

match:"/hello/:name/world",

constraint:{

name:"/[a-z]+/"

}

}

}

thentheURLmusthavethreeparts,thefirstpartbeinghelloandthethirdpartworld.Thesecondpartmustbealllowercase.

ItispossibletousemorethenoneconstraintforthesameURLpart.

{

url:{

match:"/hello/:name|:id/world",

constraint:{

name:"/[a-z]+/",id:"/[0-9]+/"

}

}

}

OptionalMatch

Anoptionalmatchissimilartoaparameterizedmatch,butthelastparameterisoptional.

Ifthedefinitionis

{

url:{

match:"/hello/:name?",

constraint:{

name:"/[a-z]+/"

}

}

}

thentheURL/helloand/hello/emilwillmatch.

Ifthedefinitionsare

{url:{match:"/hello/world"}}

{url:{match:"/hello/:name",constraint:{name:"/[a-z]+/"}}}

{url:{match:"/hello/*"}}

thentheURL/hello/worldwillbematchedbythefirstroute,becauseitisthemostspecific.TheURL/hello/youwillbematchedbythesecondroute,becauseitismorespecificthantheprefixmatch.

MethodRestriction

YoucanrestrictthematchtospecificHTTPmethods.

Ifthedefinitionis

{

url:{

Actions

670

match:"/hello/world",

methods:["post","put"]

}

}

thenonlyHTTPPOSTandPUTrequestswillmatch.CallingwithadifferentHTTPmethodwillresultinanHTTP501error.

Pleasenotethatifurlisdefinedasasimplestring,thenonlytheHTTPmethodsGETandHEADwillbeallowed,anallothermethodswillbedisabled:

{

url:"/hello/world"

}

MoreonMatching

Rememberthatthemorespecificmatchwins.

Amatchwithoutparameterorwildcardismorespecificthanamatchwithparametersorwildcard.Amatchwithparameterismorespecificthanamatchwithawildcard.Ifthereismorethanoneparameter,specificityisappliedfromlefttoright.

Considerthefollowingdefinitions

arangosh>db._routing.save({

........>url:{match:"/hello/world"},

........>content:{contentType:"text/plain",body:"MatchNo1"}});

arangosh>db._routing.save({

........>url:{match:"/hello/:name",constraint:{name:"/[a-z]+/"}},

........>content:{contentType:"text/plain",body:"MatchNo2"}});

arangosh>db._routing.save({

........>url:{match:"/:something/world"},

........>content:{contentType:"text/plain",body:"MatchNo3"}});

arangosh>db._routing.save({

........>url:{match:"/hi/*"},

........>content:{contentType:"text/plain",body:"MatchNo4"}});

arangosh>require("internal").reloadRouting()

showexecutionresultsThen

shell>curl--dump-http://localhost:8529/hello/world

HTTP/1.1200OK

content-type:text/plain

x-content-type-options:nosniff

"MatchNo1"

shell>curl--dump-http://localhost:8529/hello/emil

HTTP/1.1200OK

content-type:text/plain

x-content-type-options:nosniff

"MatchNo2"

shell>curl--dump-http://localhost:8529/your/world

HTTP/1.1200OK

Actions

671

content-type:text/plain

x-content-type-options:nosniff

"MatchNo3"

shell>curl--dump-http://localhost:8529/hi/you

HTTP/1.1200OK

content-type:text/plain

x-content-type-options:nosniff

"MatchNo4"

Youcanwritethefollowingdocumentintothe_routingcollectiontotesttheaboveexamples.

{

routes:[

{url:{match:"/hello/world"},content:"route1"},

{url:{match:"/hello/:name|:id",constraint:{name:"/[a-z]+/",id:"/[0-9]+/"}},content:"route2"},

{url:{match:"/:something/world"},content:"route3"},

{url:{match:"/hello/*"},content:"route4"},

]

}

Actions

672

AHelloWorldExampleforJSON

Ifyouchangetheexampleslightly,thenaJSONobjectwillbedelivered.

arangosh>db._routing.save({

........>url:"/hello/json",

........>content:{

........>contentType:"application/json",

........>body:'{"hello":"world"}'

........>}

........>});

arangosh>require("internal").reloadRouting()

showexecutionresultsAgaincheckwithyourbrowserorcURLhttp://localhost:8529/hello/json

Dependingonyourbrowserandinstalledadd-onsyouwilleitherseetheJSONobjectoradownloaddialog.IfyourbrowserwantstoopenanexternalapplicationtodisplaytheJSONobject,youcanchangethecontentTypeto"text/plain" fortheexample.Thismakesiteasiertochecktheexampleusingabrowser.Orusecurltoaccesstheserver.

shell>curl--dump-http://localhost:8529/hello/json

HTTP/1.1200OK

content-type:application/json;charset=utf-8

x-content-type-options:nosniff

{

"hello":"world"

}

DeliveringContent

Therearealotofdifferentwaysonhowtodelivercontent.Wehavealreadyseenthesimplestone,wherestaticcontentisdelivered.Thefun,however,startswhendeliveringdynamiccontent.

StaticContent

Youcanspecifyabodyandacontent-type.

arangosh>db._routing.save({

........>url:"/hello/contentType",

........>content:{

........>contentType:"text/html",

........>body:"<html><body>HelloWorld</body></html>"

........>}

........>});

arangosh>require("internal").reloadRouting()

showexecutionresults

shell>curl--dump-http://localhost:8529/hello/contentType

Actions

673

HTTP/1.1200OK

content-type:text/html

x-content-type-options:nosniff

"HelloWorld"

Ifthecontenttypeistext/plainthenyoucanusetheshort-cut

{

content:"HelloWorld"

}

ASimpleAction

Thesimplestdynamicactionis:

{

action:{

do:"@arangodb/actions/echoRequest"

}

}

Itisnotadvisabletostorefunctionsdirectlyintheroutingtable.Itisbettertocallfunctionsdefinedinmodules.IntheaboveexamplethefunctioncanbeaccessedfromJavaScriptas:

require("@arangodb/actions").echoRequest

ThefunctionechoRequestispre-defined.Ittakestherequestobjectsandechositintheresponse.

Thesignatureofsuchafunctionmustbe

function(req,res,options,next)

Examples

arangosh>db._routing.save({

........>url:"/hello/echo",

........>action:{

........>do:"@arangodb/actions/echoRequest"

........>}

........>});

showexecutionresultsReloadtheroutingandcheckhttp://127.0.0.1:8529/hello/echo

Youshouldseesomethinglike

arangosh>arango.GET("/hello/echo")

showexecutionresults

Therequestmightcontainpath,prefix,suffix,andurlParametersattributes.pathisthecompletepathassuppliedbytheuserandalwaysavailable.Ifaprefixwasmatched,thenthisprefixisstoredintheattributeprefixandtheremainingURLpartsarestoredasanarrayinsuffix.Ifoneormoreparameterswerematched,thentheparametervaluesarestoredinurlParameters.

Actions

674

Forexample,iftheurldescriptionis

{

url:{

match:"/hello/:name/:action"

}

}

andyourequestthepath/hello/emil/jump,thentherequestobjectwillcontainthefollowingattribute

urlParameters:{

name:"emil",

action:"jump"

}

ActionController

Asanalternativetothesimpleaction,youcanusecontrollers.Acontrollerisamodule,definesthefunctionget,put,post,delete,head,patch.Ifarequestofthecorrespondingtypeismatched,thefunctionwillbecalled.

Examples

arangosh>db._routing.save({

........>url:"/hello/echo",

........>action:{

........>controller:"@arangodb/actions/echoController"

........>}

........>});

showexecutionresultsReloadtheroutingandcheckhttp://127.0.0.1:8529/hello/echo:

arangosh>arango.GET("/hello/echo")

showexecutionresults

PrefixActionController

Thecontrollerisselectedwhenthedefinitionisread.Thereisamoreflexible,butslowerandmaybeinsecurevariant,theprefixcontroller.

Assumethattheurlisaprefixmatch

{

url:{

match:/hello/*"

}

}

Youcanuse

{

action:{

prefixController:"@arangodb/actions"

}

}

todefineaprefixcontroller.IftheURL/hello/echoControllerisgiven,thenthemodule@arangodb/actions/echoControllerisused.

Ifyouuseaprefixcontroller,youshouldmakecertainthatnounwantedactionsareavailableundertheprefix.

Actions

675

Thedefinition

{

action:"@arangodb/actions"

}

isashort-cutforaprefixcontrollerdefinition.

FunctionAction

Youcanalsostoreafunctiondirectlyintheroutingtable.

Examples

arangosh>db._routing.save({

........>url:"/hello/echo",

........>action:{

........>callback:"function(req,res){res.statusCode=200;res.body='Hello'}"

........>}

........>});

showexecutionresults

arangosh>arango.GET("hello/echo")

arangosh>db._query("FORrouteIN_routingFILTERroute.url=='/hello/echo'REMOVEroute

in_routing")

[objectArangoQueryCursor,count:0,cached:false,hasMore:false]

arangosh>require("internal").reloadRouting()

RequestsandResponses

Thecontrollermustdefinehandlerfunctionswhichtakearequestobjectandfilltheresponseobject.

AverysimpleexampleisthefunctionechoRequestdefinedinthemodule@arangodb/actions.

function(req,res,options,next){

varresult;

result={request:req,options:options};

res.responseCode=exports.HTTP_OK;

res.contentType="application/json";

res.body=JSON.stringify(result);

}

Installitvia:

arangosh>db._routing.save({

........>url:"/echo",

........>action:{

........>do:"@arangodb/actions/echoRequest"

........>}

........>})

showexecutionresultsReloadtheroutingandcheckhttp://127.0.0.1:8529/hello/echo

Youshouldseesomethinglike

Actions

676

arangosh>arango.GET("/hello/echo")

arangosh>db._query("FORrouteIN_routingFILTERroute.url=='/hello/echo'REMOVEroute

in_routing")

arangosh>require("internal").reloadRouting()

showexecutionresultsYoumayalsopassoptionstothecalledfunction:

arangosh>db._routing.save({

........>url:"/echo",

........>action:{

........>do:"@arangodb/actions/echoRequest",

........>options:{

........>"Hello":"World"

........>}

........>}

........>});

showexecutionresultsYounowseetheoptionsintheresult:

arangosh>arango.GET("/echo")

arangosh>db._query("FORrouteIN_routingFILTERroute.url=='/echo'REMOVEroutein

_routing")

arangosh>require("internal").reloadRouting()

showexecutionresults

Actions

677

ModifyingRequestandResponse

Aswe'veseeninthepreviousexamples,actionsgetcalledwiththerequestandresponseobjects(namedreqandresintheexamples)passedasparameterstotheirhandlerfunctions.

ThereqobjectcontainstheincomingHTTPrequest,whichmightormightnothavebeenmodifiedbyapreviousaction(ifactionswerechained).

Ahandlercanmodifytherequestobjectinplaceifdesired.Thismightbeusefulwhenwritingmiddleware(seebelow)thatisusedtointerceptincomingrequests,modifythemandpassthemtotheactualhandlers.

Whilemodifyingtherequestobjectmightnotbethatrelevantfornon-middlewareactions,modifyingtheresponseobjectdefinitelyis.Modifyingtheresponseobjectisanaction'sonlywaytoreturndatatothecalleroftheaction.

We'vealreadyseenhowtosettheHTTPstatuscode,thecontenttype,andtheresultbody.Theresobjecthasthefollowingpropertiesforthese:

contentType:MIMEtypeofthebodyasdefinedintheHTTPstandard(e.g.text/html,text/plain,application/json,...)responsecode:theHTTPstatuscodeoftheresponseasdefinedintheHTTPstandard.Commonvaluesforactionsthatsucceedare200or201.PleaserefertotheHTTPstandardformoreinformation.body:theactualresponsedata

Tosetormodifyarbitraryheadersoftheresponseobject,theheaderspropertycanbeused.Forexample,toaddauser-definedheadertotheresponse,thefollowingcodewilldo:

res.headers=res.headers||{};//headersmightormightnotbepresent

res.headers['X-Test']='someValue';//setheaderX-Testto"someValue"

ThiswillsettheadditionalHTTPheaderX-TesttovaluesomeValue.Otherheaderscanbesetaswell.NotethatArangoDBmightchangethecaseoftheheadernamestolowercasewhenassemblingtheoverallresponsethatissenttothecaller.

ItisnotnecessarytoexplicitlysetaContent-LengthheaderfortheresponseasArangoDBwillcalculatethecontentlengthautomaticallyandaddthisheaderitself.ArangoDBmightalsoaddaConnectionheaderitselftohandleHTTPkeep-alive.

ArangoDBalsosupportsautomatictransformationofthebodydatatoanotherformat.Currently,theonlysupportedtransformationsarebase64-encodingandbase64-decoding.Usingthetransformations,anactioncancreateabase64encodedbodyandstillletArangoDBsendthenon-encodedversion,forexample:

res.body='VGhpcyBpcyBhIHRlc3Q=';

res.transformations=res.transformations||[];//initialize

res.transformations.push('base64decode');//willbase64decodetheresponsebody

WhenArangoDBprocessestheresponse,itwillbase64-decodewhat'sinres.bodyandsettheHTTPheaderContent-Encoding:binary.Theoppositecanbeachievedwiththebase64encodetransformation:ArangoDBwillthenautomaticallybase64-encodethebodyandsetaContent-Encoding:base64HTTPheader.

WritingdynamicactionhandlersTowriteyourowndynamicactionhandlers,youmustputthemintomodules.

Modulesareameansoforganizingactionhandlersandmakingthemloadableunderspecificnames.

Tostart,we'lldefineasimpleactionhandlerinamodule/ownTest:

arangosh>db._modules.save({

........>path:"/db:/ownTest",

........>content:

........>"exports.do=function(req,res,options,next){"+

........>"res.body='test';"+

Actions

678

........>"res.responseCode=200;"+

........>"res.contentType='text/plain';"+

........>"};"

........>});

showexecutionresultsThisdoesnothingbutregisteradoactionhandlerinamodule/ownTest.Theactionhandlerisnotyetcallable,butmustbemappedtoaroutefirst.Tomaptheactiontotheroute/ourtest,executethefollowingcommand:

arangosh>db._routing.save({

........>url:"/ourtest",

........>action:{

........>controller:"db://ownTest"

........>}

........>});

arangosh>require("internal").reloadRouting()

showexecutionresultsNowusethebrowserorcURLandaccesshttp://localhost:8529/ourtest:

shell>curl--dump-http://localhost:8529/ourtest

HTTP/1.1200OK

content-type:text/plain

x-content-type-options:nosniff

"test"

Youwillseethatthemodule'sdofunctionhasbeenexecuted.

AWordaboutCachingSometimesitmightseemthatyourchangedonottakeeffect.Inthiscasetheculpritcouldbetheroutingcaches:

Theroutingcachestorestheroutinginformationcomputedfromthe_routingcollection.Wheneveryouchangethiscollectionmanually,youneedtocall

arangosh>require("internal").reloadRouting()

inordertorebuildthecache.

AdvancedUsagesFordetailedinformationseethereferencemanual.

Redirects

Usethefollowingforapermanentredirect:

arangosh>db._routing.save({

........>url:"/redirectMe",

........>action:{

........>do:"@arangodb/actions/redirectRequest",

........>options:{

Actions

679

........>permanently:true,

........>destination:"/somewhere.else/"

........>}

........>}

........>});

arangosh>require("internal").reloadRouting()

showexecutionresults

shell>curl--dump-http://localhost:8529/redirectMe

HTTP/1.1301MovedPermanently

x-content-type-options:nosniff

content-type:text/html

location:/somewhere.else/

"<html><head><title>Moved</title></head><body><h1>Moved</h1><p>Thispagehasmovedto<a

href=\"/somewhere.else/\">/somewhere.else/</a>.</p></body></html>"

RoutingBundles

Insteadofaddingallroutesforpackageseparately,youcanspecifyabundle:

arangosh>db._routing.save({

........>routes:[

........>{

........>url:"/url1",

........>content:"route1"

........>},

........>{

........>url:"/url2",

........>content:"route2"

........>},

........>{

........>url:"/url3",

........>content:"route3"

........>}

........>]

........>});

arangosh>require("internal").reloadRouting()

showexecutionresults

shell>curl--dump-http://localhost:8529/url2

HTTP/1.1200OK

content-type:text/plain

x-content-type-options:nosniff

"route2"

shell>curl--dump-http://localhost:8529/url3

HTTP/1.1200OK

content-type:text/plain

x-content-type-options:nosniff

Actions

680

"route3"

Theadvantageis,thatyoucanputallyourroutesintoonedocumentanduseacommonprefix.

arangosh>db._routing.save({

........>urlPrefix:"/test",

........>routes:[

........>{

........>url:"/url1",

........>content:"route1"

........>},

........>{

........>url:"/url2",

........>content:"route2"

........>},

........>{

........>url:"/url3",

........>content:"route3"

........>}

........>]

........>});

arangosh>require("internal").reloadRouting()

showexecutionresultswilldefinetheURL/test/url1,/test/url2,and/test/url3:

shell>curl--dump-http://localhost:8529/test/url1

HTTP/1.1200OK

content-type:text/plain

x-content-type-options:nosniff

"route1"

shell>curl--dump-http://localhost:8529/test/url2

HTTP/1.1200OK

content-type:text/plain

x-content-type-options:nosniff

"route2"

shell>curl--dump-http://localhost:8529/test/url3

HTTP/1.1200OK

content-type:text/plain

x-content-type-options:nosniff

"route3"

WritingMiddleware

Assume,youwanttologeveryrequestinyournamespacetotheconsole.(ifArangoDBisrunningasadaemon,thiswillendupinthelogfile).InthiscaseyoucaneasilydefineanactionfortheURL/subdirectory.Thisactionsimplylogstherequests,callsthenextinline,andlogstheresponse:

arangosh>db._modules.save({

Actions

681

........>path:"/db:/OwnMiddlewareTest",

........>content:

........>"exports.logRequest=function(req,res,options,next){"+

........>"console=require('console');"+

........>"console.log('receivedrequest:%s',JSON.stringify(req));"+

........>"next();"+

........>"console.log('producedresponse:%s',JSON.stringify(res));"+

........>"};"

........>});

showexecutionresultsThisfunctionwillnowbeavailableasdb://OwnMiddlewareTest/logRequest.YouneedtotellArangoDBthatitisshoulduseaprefixmatchandthattheshortestmatchshouldwininthiscase:

arangosh>db._routing.save({

........>middleware:[

........>{

........>url:{

........>match:"/subdirectory/*"

........>},

........>action:{

........>do:"db://OwnMiddlewareTest/logRequest"

........>}

........>}

........>]

........>});

showexecutionresultsWhenyoucallnext()inthataction,thenextspecificroutingwillbeusedfortheoriginalURL.EvenifyoumodifytheURLintherequestobjectreq,thiswillnotcausethenext()tojumptotheroutingdefinedforthisnextURL.IfproceedsoccurringtheoriginURL.However,ifyouusenext(true),theroutingwillstopandrequesthandlingisstartedwiththenewURL.Youmustensurethatnext(true)isnevercalledwithoutmodifyingtheURLintherequestobjectreq.Otherwiseanendlessloopwilloccur.

Nowweaddsomeothersimpleroutingstotestallthis:

arangosh>db._routing.save({

........>url:"/subdirectory/ourtest/1",

........>action:{

........>do:"@arangodb/actions/echoRequest"

........>}

........>});

arangosh>db._routing.save({

........>url:"/subdirectory/ourtest/2",

........>action:{

........>do:"@arangodb/actions/echoRequest"

........>}

........>});

arangosh>db._routing.save({

........>url:"/subdirectory/ourtest/3",

........>action:{

........>do:"@arangodb/actions/echoRequest"

........>}

........>});

arangosh>require("internal").reloadRouting()

showexecutionresults

Actions

682

Thenwesendsomecurlrequeststothesesampleroutes:

shell>curl--dump-http://localhost:8529/subdirectory/ourtest/1

HTTP/1.1200OK

content-type:application/json;charset=utf-8

x-content-type-options:nosniff

showresponsebodyandtheconsole(and/orthelogfile)willshowrequestsandreplies.Notethatloggingdoesn'twarrantthesequenceinwhichtheselineswillappear.

ApplicationDeploymentUsingsingleroutesorbundlescanbebecomeabitmessyinlargeapplications.Kaerushaswrittenadeploymenttoolinnode.js.

NotethatthereisalsoFoxxforbuildingapplicationswithArangoDB.

CommonPitfallswhenusingActions

Caching

IfyoumadeanychangestotheroutingbutthechangesdoesnothaveanyeffectwhencallingthemodifiedactionsURL,youmighthavebeenhitbysomecachingissues.

Afteranymodificationtotheroutingoractions,itisthusrecommendedtomakethechanges"live"bycallingthefollowingfunctionsfromwithinarangosh:

Youmightalsobeaffectedbyclient-sidecaching.BrowserstendtocachecontentandalsoredirectionURLs.Youmightneedtoclearordisablethebrowsercacheinsomecasestoseeyourchangesineffect.

Datatypes

Whenprocessingtherequestdatainanaction,pleasebeawarethatthedatatypeofallqueryparametersisstring.ThisisbecausethewholeURLisastringandwhentheindividualpartsareextracted,theywillalsobestrings.

Forexample,whencallingtheURLhttp://localhost:8529/hello/world?value=5

theparametervaluewillhaveavalueof(string)5,not(number)5.ThismightbetroublesomeifyouuseJavaScript's===operatorwhencheckingrequestparametervalues.

ThesameproblemoccurswithincomingHTTPheaders.WhensendingthefollowingheaderfromaclienttoArangoDB

X-My-Value:5

thentheheaderX-My-Valuewillhaveavalueof(string)5andnot(number)5.

404NotFound

IfyoudefinedaURLintheroutingandtheURLisaccessiblefineviaHTTPGETbutreturnsanHTTP501(notimplemented)forotherHTTPmethodssuchasPOST,PUTorDELETE,thenyoumighthavebeenhitbysomedefaults.

Bydefault,URLsdefinedlikethis(simplestringurlattribute)areaccessibleviaHTTPGETandHEADonly.TomakesuchURLsaccessibleviaotherHTTPmethods,extendtheURLdefinitionwiththemethodsattribute.

Forexample,thisdefinitiononlyallowsaccessviaGETandHEAD:

Actions

683

{

url:"/hello/world"

}

whereasthisdefinitionallowsHTTPGET,POST,andPUT:

arangosh>db._routing.save({

........>url:{

........>match:"/hello/world",

........>methods:["get","post","put"]

........>},

........>action:{

........>do:"@arangodb/actions/echoRequest"

........>}

........>});

arangosh>require("internal").reloadRouting()

showexecutionresults

shell>curl--dump-http://localhost:8529/hello/world

HTTP/1.1200OK

content-type:application/json;charset=utf-8

x-content-type-options:nosniff

showresponsebodyTheformerdefinition(definingurlasanobjectwithamatchattribute)willresultintheURLbeingaccessibleviaallsupportedHTTPmethods(e.g.GET,POST,PUT,DELETE,...),whereasthelatterdefinition(providingastringurlattribute)willresultintheURLbeingaccessibleviaHTTPGETandHTTPHEADonly,withallotherHTTPmethodsbeingdisabled.CallingaURLwithanunsupportedordisabledHTTPmethodwillresultinanHTTP404error.

Actions

684

Errorcodesandmeanings

Generalerrors

0-ERROR_NO_ERRORNoerrorhasoccurred.

1-ERROR_FAILEDWillberaisedwhenageneralerroroccurred.

2-ERROR_SYS_ERRORWillberaisedwhenoperatingsystemerroroccurred.

3-ERROR_OUT_OF_MEMORYWillberaisedwhenthereisamemoryshortage.

4-ERROR_INTERNALWillberaisedwhenaninternalerroroccurred.

5-ERROR_ILLEGAL_NUMBERWillberaisedwhenanillegalrepresentationofanumberwasgiven.

6-ERROR_NUMERIC_OVERFLOWWillberaisedwhenanumericoverflowoccurred.

7-ERROR_ILLEGAL_OPTIONWillberaisedwhenanunknownoptionwassuppliedbytheuser.

8-ERROR_DEAD_PIDWillberaisedwhenaPIDwithoutalivingprocesswasfound.

9-ERROR_NOT_IMPLEMENTEDWillberaisedwhenhittinganunimplementedfeature.

10-ERROR_BAD_PARAMETERWillberaisedwhentheparameterdoesnotfulfilltherequirements.

11-ERROR_FORBIDDENWillberaisedwhenyouaremissingpermissionfortheoperation.

12-ERROR_OUT_OF_MEMORY_MMAPWillberaisedwhenthereisamemoryshortage.

13-ERROR_CORRUPTED_CSVWillberaisedwhenencounteringacorruptcsvline.

14-ERROR_FILE_NOT_FOUNDWillberaisedwhenafileisnotfound.

15-ERROR_CANNOT_WRITE_FILEWillberaisedwhenafilecannotbewritten.

16-ERROR_CANNOT_OVERWRITE_FILEWillberaisedwhenanattemptismadetooverwriteanexistingfile.

17-ERROR_TYPE_ERRORWillberaisedwhenatypeerrorisunencountered.

18-ERROR_LOCK_TIMEOUTWillberaisedwhenthere'satimeoutwaitingforalock.

19-ERROR_CANNOT_CREATE_DIRECTORYWillberaisedwhenanattempttocreateadirectoryfails.

Errorcodesandmeanings

685

20-ERROR_CANNOT_CREATE_TEMP_FILEWillberaisedwhenanattempttocreateatemporaryfilefails.

21-ERROR_REQUEST_CANCELEDWillberaisedwhenarequestiscanceledbytheuser.

22-ERROR_DEBUGWillberaisedintentionallyduringdebugging.

25-ERROR_IP_ADDRESS_INVALIDWillberaisedwhenthestructureofanIPaddressisinvalid.

27-ERROR_FILE_EXISTSWillberaisedwhenafilealreadyexists.

28-ERROR_LOCKEDWillberaisedwhenaresourceoranoperationislocked.

29-ERROR_DEADLOCKWillberaisedwhenadeadlockisdetectedwhenaccessingcollections.

30-ERROR_SHUTTING_DOWNWillberaisedwhenacallcannotsucceedbecauseaservershutdownisalreadyinprogress.

31-ERROR_ONLY_ENTERPRISEWillberaisedwhenanenterprise-featureisrequestedfromthecommunityedition.

32-ERROR_RESOURCE_LIMITWillberaisedwhentheresourcesusedbyanoperationexceedtheconfiguredmaximumvalue.

HTTPerrorstatuscodes400-ERROR_HTTP_BAD_PARAMETERWillberaisedwhentheHTTPrequestdoesnotfulfilltherequirements.

401-ERROR_HTTP_UNAUTHORIZEDWillberaisedwhenauthorizationisrequiredbuttheuserisnotauthorized.

403-ERROR_HTTP_FORBIDDENWillberaisedwhentheoperationisforbidden.

404-ERROR_HTTP_NOT_FOUNDWillberaisedwhenanURIisunknown.

405-ERROR_HTTP_METHOD_NOT_ALLOWEDWillberaisedwhenanunsupportedHTTPmethodisusedforanoperation.

406-ERROR_HTTP_NOT_ACCEPTABLEWillberaisedwhenanunsupportedHTTPcontenttypeisusedforanoperation,orifarequestisnotacceptableforaleaderorfollower.

412-ERROR_HTTP_PRECONDITION_FAILEDWillberaisedwhenapreconditionforanHTTPrequestisnotmet.

500-ERROR_HTTP_SERVER_ERRORWillberaisedwhenaninternalserverisencountered.

503-ERROR_HTTP_SERVICE_UNAVAILABLEWillberaisedwhenaserviceistemporarilyunavailable.

HTTPprocessingerrors

Errorcodesandmeanings

686

600-ERROR_HTTP_CORRUPTED_JSONWillberaisedwhenastringrepresentationofaJSONobjectiscorrupt.

601-ERROR_HTTP_SUPERFLUOUS_SUFFICESWillberaisedwhentheURLcontainssuperfluoussuffices.

InternalArangoDBstorageerrors

Forerrorsthatoccurbecauseofaprogrammingerror.

1000-ERROR_ARANGO_ILLEGAL_STATEInternalerrorthatwillberaisedwhenthedatafileisnotintherequiredstate.

1002-ERROR_ARANGO_DATAFILE_SEALEDInternalerrorthatwillberaisedwhentryingtowritetoadatafile.

1004-ERROR_ARANGO_READ_ONLYInternalerrorthatwillberaisedwhentryingtowritetoaread-onlydatafileorcollection.

1005-ERROR_ARANGO_DUPLICATE_IDENTIFIERInternalerrorthatwillberaisedwhenaidentifierduplicateisdetected.

1006-ERROR_ARANGO_DATAFILE_UNREADABLEInternalerrorthatwillberaisedwhenadatafileisunreadable.

1007-ERROR_ARANGO_DATAFILE_EMPTYInternalerrorthatwillberaisedwhenadatafileisempty.

1008-ERROR_ARANGO_RECOVERYWillberaisedwhenanerroroccurredduringWALlogfilerecovery.

1009-ERROR_ARANGO_DATAFILE_STATISTICS_NOT_FOUNDWillberaisedwhenarequireddatafilestatisticsobjectwasnotfound.

ExternalArangoDBstorageerrorsForerrorsthatoccurbecauseofanoutsideevent.

1100-ERROR_ARANGO_CORRUPTED_DATAFILEWillberaisedwhenacorruptionisdetectedinadatafile.

1101-ERROR_ARANGO_ILLEGAL_PARAMETER_FILEWillberaisedifaparameterfileiscorruptedorcannotberead.

1102-ERROR_ARANGO_CORRUPTED_COLLECTIONWillberaisedwhenacollectioncontainsoneormorecorrupteddatafiles.

1103-ERROR_ARANGO_MMAP_FAILEDWillberaisedwhenthesystemcallmmapfailed.

1104-ERROR_ARANGO_FILESYSTEM_FULLWillberaisedwhenthefilesystemisfull.

1105-ERROR_ARANGO_NO_JOURNALWillberaisedwhenajournalcannotbecreated.

1106-ERROR_ARANGO_DATAFILE_ALREADY_EXISTSWillberaisedwhenthedatafilecannotbecreatedorrenamedbecauseafileofthesamenamealreadyexists.

1107-ERROR_ARANGO_DATADIR_LOCKEDWillberaisedwhenthedatabasedirectoryislockedbyadifferentprocess.

1108-ERROR_ARANGO_COLLECTION_DIRECTORY_ALREADY_EXISTSWillberaisedwhenthecollectioncannotbecreatedbecauseadirectoryofthesamenamealreadyexists.

Errorcodesandmeanings

687

1109-ERROR_ARANGO_MSYNC_FAILEDWillberaisedwhenthesystemcallmsyncfailed.

1110-ERROR_ARANGO_DATADIR_UNLOCKABLEWillberaisedwhentheservercannotlockthedatabasedirectoryonstartup.

1111-ERROR_ARANGO_SYNC_TIMEOUTWillberaisedwhentheserverwaitedtoolongforadatafiletobesyncedtodisk.

GeneralArangoDBstorageerrorsForerrorsthatoccurwhenfulfillingauserrequest.

1200-ERROR_ARANGO_CONFLICTWillberaisedwhenupdatingordeletingadocumentandaconflicthasbeendetected.

1201-ERROR_ARANGO_DATADIR_INVALIDWillberaisedwhenanon-existingdatabasedirectorywasspecifiedwhenstartingthedatabase.

1202-ERROR_ARANGO_DOCUMENT_NOT_FOUNDWillberaisedwhenadocumentwithagivenidentifierorhandleisunknown.

1203-ERROR_ARANGO_COLLECTION_NOT_FOUNDWillberaisedwhenacollectionwiththegivenidentifierornameisunknown.

1204-ERROR_ARANGO_COLLECTION_PARAMETER_MISSINGWillberaisedwhenthecollectionparameterismissing.

1205-ERROR_ARANGO_DOCUMENT_HANDLE_BADWillberaisedwhenadocumenthandleiscorrupt.

1206-ERROR_ARANGO_MAXIMAL_SIZE_TOO_SMALLWillberaisedwhenthemaximalsizeofthejournalistoosmall.

1207-ERROR_ARANGO_DUPLICATE_NAMEWillberaisedwhenanameduplicateisdetected.

1208-ERROR_ARANGO_ILLEGAL_NAMEWillberaisedwhenanillegalnameisdetected.

1209-ERROR_ARANGO_NO_INDEXWillberaisedwhennosuitableindexforthequeryisknown.

1210-ERROR_ARANGO_UNIQUE_CONSTRAINT_VIOLATEDWillberaisedwhenthereisauniqueconstraintviolation.

1211-ERROR_ARANGO_VIEW_NOT_FOUNDWillberaisedwhenaviewwiththegivenidentifierornameisunknown.

1212-ERROR_ARANGO_INDEX_NOT_FOUNDWillberaisedwhenanindexwithagivenidentifierisunknown.

1213-ERROR_ARANGO_CROSS_COLLECTION_REQUESTWillberaisedwhenacross-collectionisrequested.

1214-ERROR_ARANGO_INDEX_HANDLE_BADWillberaisedwhenaindexhandleiscorrupt.

1216-ERROR_ARANGO_DOCUMENT_TOO_LARGEWillberaisedwhenthedocumentcannotfitintoanydatafilebecauseofitistoolarge.

1217-ERROR_ARANGO_COLLECTION_NOT_UNLOADEDWillberaisedwhenacollectionshouldbeunloaded,buthasadifferentstatus.

Errorcodesandmeanings

688

1218-ERROR_ARANGO_COLLECTION_TYPE_INVALIDWillberaisedwhenaninvalidcollectiontypeisusedinarequest.

1219-ERROR_ARANGO_VALIDATION_FAILEDWillberaisedwhenthevalidationofanattributeofastructurefailed.

1220-ERROR_ARANGO_ATTRIBUTE_PARSER_FAILEDWillberaisedwhenparsinganattributenamedefinitionfailed.

1221-ERROR_ARANGO_DOCUMENT_KEY_BADWillberaisedwhenadocumentkeyiscorrupt.

1222-ERROR_ARANGO_DOCUMENT_KEY_UNEXPECTEDWillberaisedwhenauser-defineddocumentkeyissuppliedforcollectionswithautokeygeneration.

1224-ERROR_ARANGO_DATADIR_NOT_WRITABLEWillberaisedwhentheserver'sdatabasedirectoryisnotwritableforthecurrentuser.

1225-ERROR_ARANGO_OUT_OF_KEYSWillberaisedwhenakeygeneratorrunsoutofkeys.

1226-ERROR_ARANGO_DOCUMENT_KEY_MISSINGWillberaisedwhenadocumentkeyismissing.

1227-ERROR_ARANGO_DOCUMENT_TYPE_INVALIDWillberaisedwhenthereisanattempttocreateadocumentwithaninvalidtype.

1228-ERROR_ARANGO_DATABASE_NOT_FOUNDWillberaisedwhenanon-existingdatabaseisaccessed.

1229-ERROR_ARANGO_DATABASE_NAME_INVALIDWillberaisedwhenaninvaliddatabasenameisused.

1230-ERROR_ARANGO_USE_SYSTEM_DATABASEWillberaisedwhenanoperationisrequestedinadatabaseotherthanthesystemdatabase.

1231-ERROR_ARANGO_ENDPOINT_NOT_FOUNDWillberaisedwhenthereisanattempttodeleteanon-existingendpoint.

1232-ERROR_ARANGO_INVALID_KEY_GENERATORWillberaisedwhenaninvalidkeygeneratordescriptionisused.

1233-ERROR_ARANGO_INVALID_EDGE_ATTRIBUTEwillberaisedwhenthe_fromor_tovaluesofanedgeareundefinedorcontainaninvalidvalue.

1234-ERROR_ARANGO_INDEX_DOCUMENT_ATTRIBUTE_MISSINGWillberaisedwhenanattempttoinsertadocumentintoanindexiscausedbyinthedocumentnothavingoneormoreattributeswhichtheindexisbuilton.

1235-ERROR_ARANGO_INDEX_CREATION_FAILEDWillberaisedwhenanattempttocreateanindexhasfailed.

1236-ERROR_ARANGO_WRITE_THROTTLE_TIMEOUTWillberaisedwhentheserveriswrite-throttledandawriteoperationhaswaitedtoolongfortheservertoprocessqueuedoperations.

1237-ERROR_ARANGO_COLLECTION_TYPE_MISMATCHWillberaisedwhenacollectionhasadifferenttypefromwhathasbeenexpected.

1238-ERROR_ARANGO_COLLECTION_NOT_LOADEDWillberaisedwhenacollectionisaccessedthatisnotyetloaded.

1239-ERROR_ARANGO_DOCUMENT_REV_BADWillberaisedwhenadocumentrevisioniscorruptorismissingwhereneeded.

Errorcodesandmeanings

689

CheckedArangoDBstorageerrors

Forerrorsthatoccurbutareanticipated.

1300-ERROR_ARANGO_DATAFILE_FULLWillberaisedwhenthedatafilereachesitslimit.

1301-ERROR_ARANGO_EMPTY_DATADIRWillberaisedwhenencounteringanemptyserverdatabasedirectory.

1302-ERROR_ARANGO_TRY_AGAINWillberaisedwhenanoperationshouldberetried.

1303-ERROR_ARANGO_BUSYWillberaisedwhenstorageengineisbusy.

1304-ERROR_ARANGO_MERGE_IN_PROGRESSWillberaisedwhenstorageenginehasadatafilemergeinprogressandcannotcompletetheoperation.

1305-ERROR_ARANGO_IO_ERRORWillberaisedwhenstorageengineencountersanI/Oerror.

ArangoDBreplicationerrors1400-ERROR_REPLICATION_NO_RESPONSEWillberaisedwhenthereplicationapplierdoesnotreceiveanyoranincompleteresponsefromthemaster.

1401-ERROR_REPLICATION_INVALID_RESPONSEWillberaisedwhenthereplicationapplierreceivesaninvalidresponsefromthemaster.

1402-ERROR_REPLICATION_MASTER_ERRORWillberaisedwhenthereplicationapplierreceivesaservererrorfromthemaster.

1403-ERROR_REPLICATION_MASTER_INCOMPATIBLEWillberaisedwhenthereplicationapplierconnectstoamasterthathasanincompatibleversion.

1404-ERROR_REPLICATION_MASTER_CHANGEWillberaisedwhenthereplicationapplierconnectstoadifferentmasterthanbefore.

1405-ERROR_REPLICATION_LOOPWillberaisedwhenthereplicationapplierisaskedtoconnecttoitselfforreplication.

1406-ERROR_REPLICATION_UNEXPECTED_MARKERWillberaisedwhenanunexpectedmarkerisfoundinthereplicationlogstream.

1407-ERROR_REPLICATION_INVALID_APPLIER_STATEWillberaisedwhenaninvalidreplicationapplierstatefileisfound.

1408-ERROR_REPLICATION_UNEXPECTED_TRANSACTIONWillberaisedwhenanunexpectedtransactionidisfound.

1410-ERROR_REPLICATION_INVALID_APPLIER_CONFIGURATIONWillberaisedwhentheconfigurationforthereplicationapplierisinvalid.

1411-ERROR_REPLICATION_RUNNINGWillberaisedwhenthereisanattempttoperformanoperationwhilethereplicationapplierisrunning.

1412-ERROR_REPLICATION_APPLIER_STOPPEDSpecialerrorcodeusedtoindicatethereplicationapplierwasstoppedbyauser.

1413-ERROR_REPLICATION_NO_START_TICKWillberaisedwhenthereplicationapplierisstartedwithoutaknownstarttickvalue.

Errorcodesandmeanings

690

1414-ERROR_REPLICATION_START_TICK_NOT_PRESENTWillberaisedwhenthereplicationapplierfetchesdatausingastarttick,butthatstarttickisnotpresentontheloggerserveranymore.

ERROR_REPLICATION_WRONG_CHECKSUM_FORMAT,1415,"thechecksumformatiswrong","Willberaisedwhentheformatofthechecksumiswrong")

1416-ERROR_REPLICATION_WRONG_CHECKSUMWillberaisedwhenanewbornfollowersubmitsawrongchecksum

1417-ERROR_REPLICATION_SHARD_NONEMPTYWillberaisedwhenashardisnotemptyandthefollowertriesashortcut

ArangoDBclustererrors

1450-ERROR_CLUSTER_NO_AGENCYWillberaisedwhennoneoftheagencyserverscanbeconnectedto.

1451-ERROR_CLUSTER_NO_COORDINATOR_HEADERWillberaisedwhenaDBserverinaclusterreceivesaHTTPrequestwithoutacoordinatorheader.

1452-ERROR_CLUSTER_COULD_NOT_LOCK_PLANWillberaisedwhenacoordinatorinaclustercannotlockthePlanhierarchyintheagency.

1453-ERROR_CLUSTER_COLLECTION_ID_EXISTSWillberaisedwhenacoordinatorinaclustertriestocreateacollectionandthecollectionIDalreadyexists.

1454-ERROR_CLUSTER_COULD_NOT_CREATE_COLLECTION_IN_PLANWillberaisedwhenacoordinatorinaclustercannotcreateanentryforanewcollectioninthePlanhierarchyintheagency.

1455-ERROR_CLUSTER_COULD_NOT_READ_CURRENT_VERSIONWillberaisedwhenacoordinatorinaclustercannotreadtheVersionentryintheCurrenthierarchyintheagency.

1456-ERROR_CLUSTER_COULD_NOT_CREATE_COLLECTIONWillberaisedwhenacoordinatorinaclusternoticesthatsomeDBServersreportproblemswhencreatingshardsforanewcollection.

1457-ERROR_CLUSTER_TIMEOUTWillberaisedwhenacoordinatorinaclusterrunsintoatimeoutforsomeclusterwideoperation.

1458-ERROR_CLUSTER_COULD_NOT_REMOVE_COLLECTION_IN_PLANWillberaisedwhenacoordinatorinaclustercannotremoveanentryforacollectioninthePlanhierarchyintheagency.

1459-ERROR_CLUSTER_COULD_NOT_REMOVE_COLLECTION_IN_CURRENTWillberaisedwhenacoordinatorinaclustercannotremoveanentryforacollectionintheCurrenthierarchyintheagency.

1460-ERROR_CLUSTER_COULD_NOT_CREATE_DATABASE_IN_PLANWillberaisedwhenacoordinatorinaclustercannotcreateanentryforanewdatabaseinthePlanhierarchyintheagency.

1461-ERROR_CLUSTER_COULD_NOT_CREATE_DATABASEWillberaisedwhenacoordinatorinaclusternoticesthatsomeDBServersreportproblemswhencreatingdatabasesforanewclusterwidedatabase.

1462-ERROR_CLUSTER_COULD_NOT_REMOVE_DATABASE_IN_PLANWillberaisedwhenacoordinatorinaclustercannotremoveanentryforadatabaseinthePlanhierarchyintheagency.

1463-ERROR_CLUSTER_COULD_NOT_REMOVE_DATABASE_IN_CURRENTWillberaisedwhenacoordinatorinaclustercannotremoveanentryforadatabaseintheCurrenthierarchyintheagency.

1464-ERROR_CLUSTER_SHARD_GONEWillberaisedwhenacoordinatorinaclustercannotdeterminetheshardthatisresponsibleforagivendocument.

1465-ERROR_CLUSTER_CONNECTION_LOSTWillberaisedwhenacoordinatorinaclusterlosesanHTTPconnectiontoaDBserverintheclusterwhilsttransferringdata.

Errorcodesandmeanings

691

1466-ERROR_CLUSTER_MUST_NOT_SPECIFY_KEYWillberaisedwhenacoordinatorinaclusterfindsthatthe_keyattributewasspecifiedinashardedcollectiontheusesnotonly_keyasshardingattribute.

1467-ERROR_CLUSTER_GOT_CONTRADICTING_ANSWERSWillberaisedifacoordinatorinaclustergetsconflictingresultsfromdifferentshards,whichshouldneverhappen.

1468-ERROR_CLUSTER_NOT_ALL_SHARDING_ATTRIBUTES_GIVENWillberaisedifacoordinatortriestofindoutwhichshardisresponsibleforapartialdocument,butcannotdothisbecausenotallshardingattributesarespecified.

1469-ERROR_CLUSTER_MUST_NOT_CHANGE_SHARDING_ATTRIBUTESWillberaisedifthereisanattempttoupdatethevalueofashardattribute.

1470-ERROR_CLUSTER_UNSUPPORTEDWillberaisedwhenthereisanattempttocarryoutanoperationthatisnotsupportedinthecontextofashardedcollection.

1471-ERROR_CLUSTER_ONLY_ON_COORDINATORWillberaisedifthereisanattempttorunacoordinator-onlyoperationonadifferenttypeofnode.

1472-ERROR_CLUSTER_READING_PLAN_AGENCYWillberaisedifacoordinatororDBservercannotreadthePlanintheagency.

1473-ERROR_CLUSTER_COULD_NOT_TRUNCATE_COLLECTIONWillberaisedifacoordinatorcannottruncateallshardsofaclustercollection.

1474-ERROR_CLUSTER_AQL_COMMUNICATIONWillberaisediftheinternalcommunicationoftheclusterforAQLproducesanerror.

1475-ERROR_ARANGO_DOCUMENT_NOT_FOUND_OR_SHARDING_ATTRIBUTES_CHANGEDWillberaisedwhenadocumentwithagivenidentifierorhandleisunknown,oriftheshardingattributeshavebeenchangedinaREPLACEoperationinthecluster.

1476-ERROR_CLUSTER_COULD_NOT_DETERMINE_IDWillberaisedifaclusterserveratstartupcouldnotdetermineitsownIDfromthelocalinfoprovided.

1477-ERROR_CLUSTER_ONLY_ON_DBSERVERWillberaisedifthereisanattempttorunaDBserver-onlyoperationonadifferenttypeofnode.

1478-ERROR_CLUSTER_BACKEND_UNAVAILABLEWillberaisedifarequireddbservercan'tbereached.

1479-ERROR_CLUSTER_UNKNOWN_CALLBACK_ENDPOINTAnendpointcouldn'tbefound

1480-ERROR_CLUSTER_AGENCY_STRUCTURE_INVALIDThestructureintheagencyisinvalid

1481-ERROR_CLUSTER_AQL_COLLECTION_OUT_OF_SYNCWillberaisedifacollectionneededduringqueryexecutionisoutofsync.Thiscurrentlycanonlyhappenwhenusingsatellitecollections

1482-ERROR_CLUSTER_COULD_NOT_CREATE_INDEX_IN_PLANWillberaisedwhenacoordinatorinaclustercannotcreateanentryforanewindexinthePlanhierarchyintheagency.

1483-ERROR_CLUSTER_COULD_NOT_DROP_INDEX_IN_PLANWillberaisedwhenacoordinatorinaclustercannotremoveanindexfromthePlanhierarchyintheagency.

1484-ERROR_CLUSTER_CHAIN_OF_DISTRIBUTESHARDSLIKEWillberaisedifonetriestocreateacollectionwithadistributeShardsLikeattributewhichpointstoanothercollectionthatalsohasone.

1485-ERROR_CLUSTER_MUST_NOT_DROP_COLL_OTHER_DISTRIBUTESHARDSLIKEWillberaisedifonetriestodropacollectiontowhichanothercollectionpointswithitsdistributeShardsLikeattribute.

Errorcodesandmeanings

692

1486-ERROR_CLUSTER_UNKNOWN_DISTRIBUTESHARDSLIKEWillberaisedifonetriestocreateacollectionwhichpointstoanunknowncollectioninitsdistributeShardsLikeattribute.

1487-ERROR_CLUSTER_INSUFFICIENT_DBSERVERSWillberaisedifonetriestocreateacollectionwithareplicationFactorgreaterthantheavailablenumberofDBServers.

1488-ERROR_CLUSTER_COULD_NOT_DROP_FOLLOWERWillberaisedifafollowerthatoughttobedroppedcouldnotbedroppedintheagency(underCurrent).

1489-ERROR_CLUSTER_SHARD_LEADER_REFUSES_REPLICATIONWillberaisedifareplicationoperationisrefusedbyashardleader.

1490-ERROR_CLUSTER_SHARD_FOLLOWER_REFUSES_OPERATIONWillberaisedifanon-replicationoperationisrefusedbyashardfollower.

1491-ERROR_CLUSTER_SHARD_LEADER_RESIGNEDWillberaisedifanon-replicationoperationisrefusedbyaformershardleaderthathasfoundoutthatitisnolongertheleader.

1492-ERROR_CLUSTER_AGENCY_COMMUNICATION_FAILEDWillberaisedifaftervariousretriesanagencyoperationcouldnotbeperformedsuccessfully.

1493-ERROR_CLUSTER_DISTRIBUTE_SHARDS_LIKE_REPLICATION_FACTORWillberaisedifintendedreplicationfactordoesnotmatchthatoftheprototypeshardgiveninditributeShardsLikeparameter.

1494-ERROR_CLUSTER_DISTRIBUTE_SHARDS_LIKE_NUMBER_OF_SHARDSWillberaisedifintendednumberofshardsdoesnotmatchthatoftheprototypeshardgiveninditributeShardsLikeparameter.

1495-ERROR_CLUSTER_LEADERSHIP_CHALLENGE_ONGOINGWillberaisedwhenserversarecurrentlycompetingforleadership,andtheresultisstillunknown.

1496-ERROR_CLUSTER_NOT_LEADERWillberaisedwhenanoperationissenttoanon-leadingserver.

ArangoDBqueryerrors

1500-ERROR_QUERY_KILLEDWillberaisedwhenarunningqueryiskilledbyanexplicitadmincommand.

1501-ERROR_QUERY_PARSEWillberaisedwhenqueryisparsedandisfoundtobesyntacticallyinvalid.

1502-ERROR_QUERY_EMPTYWillberaisedwhenanemptyqueryisspecified.

1503-ERROR_QUERY_SCRIPTWillberaisedwhenaruntimeerroriscausedbythequery.

1504-ERROR_QUERY_NUMBER_OUT_OF_RANGEWillberaisedwhenanumberisoutsidetheexpectedrange.

1510-ERROR_QUERY_VARIABLE_NAME_INVALIDWillberaisedwhenaninvalidvariablenameisused.

1511-ERROR_QUERY_VARIABLE_REDECLAREDWillberaisedwhenavariablegetsre-assignedinaquery.

1512-ERROR_QUERY_VARIABLE_NAME_UNKNOWNWillberaisedwhenanunknownvariableisusedorthevariableisundefinedthecontextitisused.

1521-ERROR_QUERY_COLLECTION_LOCK_FAILEDWillberaisedwhenareadlockonthecollectioncannotbeacquired.

1522-ERROR_QUERY_TOO_MANY_COLLECTIONSWillberaisedwhenthenumberofcollectionsinaqueryisbeyondtheallowedvalue.

Errorcodesandmeanings

693

1530-ERROR_QUERY_DOCUMENT_ATTRIBUTE_REDECLAREDWillberaisedwhenadocumentattributeisre-assigned.

1540-ERROR_QUERY_FUNCTION_NAME_UNKNOWNWillberaisedwhenanundefinedfunctioniscalled.

1541-ERROR_QUERY_FUNCTION_ARGUMENT_NUMBER_MISMATCHWillberaisedwhenthenumberofargumentsusedinafunctioncalldoesnotmatchtheexpectednumberofargumentsforthefunction.

1542-ERROR_QUERY_FUNCTION_ARGUMENT_TYPE_MISMATCHWillberaisedwhenthetypeofanargumentusedinafunctioncalldoesnotmatchtheexpectedargumenttype.

1543-ERROR_QUERY_INVALID_REGEXWillberaisedwhenaninvalidregexargumentvalueisusedinacalltoafunctionthatexpectsaregex.

1550-ERROR_QUERY_BIND_PARAMETERS_INVALIDWillberaisedwhenthestructureofbindparameterspassedhasanunexpectedformat.

1551-ERROR_QUERY_BIND_PARAMETER_MISSINGWillberaisedwhenabindparameterwasdeclaredinthequerybutthequeryisbeingexecutedwithnovalueforthatparameter.

1552-ERROR_QUERY_BIND_PARAMETER_UNDECLAREDWillberaisedwhenavaluegetsspecifiedforanundeclaredbindparameter.

1553-ERROR_QUERY_BIND_PARAMETER_TYPEWillberaisedwhenabindparameterhasaninvalidvalueortype.

1560-ERROR_QUERY_INVALID_LOGICAL_VALUEWillberaisedwhenanon-booleanvalueisusedinalogicaloperation.

1561-ERROR_QUERY_INVALID_ARITHMETIC_VALUEWillberaisedwhenanon-numericvalueisusedinanarithmeticoperation.

1562-ERROR_QUERY_DIVISION_BY_ZEROWillberaisedwhenthereisanattempttodividebyzero.

1563-ERROR_QUERY_ARRAY_EXPECTEDWillberaisedwhenanon-arrayoperandisusedforanoperationthatexpectsanarrayargumentoperand.

1569-ERROR_QUERY_FAIL_CALLEDWillberaisedwhenthefunctionFAIL()iscalledfrominsideaquery.

1570-ERROR_QUERY_GEO_INDEX_MISSINGWillberaisedwhenageorestrictionwasspecifiedbutnosuitablegeoindexisfoundtoresolveit.

1571-ERROR_QUERY_FULLTEXT_INDEX_MISSINGWillberaisedwhenafulltextqueryisperformedonacollectionwithoutasuitablefulltextindex.

1572-ERROR_QUERY_INVALID_DATE_VALUEWillberaisedwhenavaluecannotbeconvertedtoadate.

1573-ERROR_QUERY_MULTI_MODIFYWillberaisedwhenanAQLquerycontainsmorethanonedata-modifyingoperation.

1574-ERROR_QUERY_INVALID_AGGREGATE_EXPRESSIONWillberaisedwhenanAQLquerycontainsaninvalidaggregateexpression.

1575-ERROR_QUERY_COMPILE_TIME_OPTIONSWillberaisedwhenanAQLdata-modificationquerycontainsoptionsthatcannotbefiguredoutatquerycompiletime.

1576-ERROR_QUERY_EXCEPTION_OPTIONSWillberaisedwhenanAQLdata-modificationquerycontainsaninvalidoptionsspecification.

1577-ERROR_QUERY_COLLECTION_USED_IN_EXPRESSIONWillberaisedwhenacollectionisusedasanoperandinanAQLexpression.

Errorcodesandmeanings

694

1578-ERROR_QUERY_DISALLOWED_DYNAMIC_CALLWillberaisedwhenadynamicfunctioncallismadetoafunctionthatcannotbecalleddynamically.

1579-ERROR_QUERY_ACCESS_AFTER_MODIFICATIONWillberaisedwhencollectiondataareaccessedafteradata-modificationoperation.

AQLuserfunctionerrors

1580-ERROR_QUERY_FUNCTION_INVALID_NAMEWillberaisedwhenauserfunctionwithaninvalidnameisregistered.

1581-ERROR_QUERY_FUNCTION_INVALID_CODEWillberaisedwhenauserfunctionisregisteredwithinvalidcode.

1582-ERROR_QUERY_FUNCTION_NOT_FOUNDWillberaisedwhenauserfunctionisaccessedbutnotfound.

1583-ERROR_QUERY_FUNCTION_RUNTIME_ERRORWillberaisedwhenauserfunctionthrowsaruntimeexception.

AQLqueryregistryerrors1590-ERROR_QUERY_BAD_JSON_PLANWillberaisedwhenanHTTPAPIforaquerygotaninvalidJSONobject.

1591-ERROR_QUERY_NOT_FOUNDWillberaisedwhenanIdofaqueryisnotfoundbytheHTTPAPI.

1592-ERROR_QUERY_IN_USEWillberaisedwhenanIdofaqueryisfoundbytheHTTPAPIbutthequeryisinuse.

ArangoDBcursorerrors

1600-ERROR_CURSOR_NOT_FOUNDWillberaisedwhenacursorisrequestedviaitsidbutacursorwiththatidcannotbefound.

1601-ERROR_CURSOR_BUSYWillberaisedwhenacursorisrequestedviaitsidbutaconcurrentrequestisstillusingthecursor.

ArangoDBtransactionerrors1650-ERROR_TRANSACTION_INTERNALWillberaisedwhenawrongusageoftransactionsisdetected.thisisaninternalerrorandindicatesabuginArangoDB.

1651-ERROR_TRANSACTION_NESTEDWillberaisedwhentransactionsarenested.

1652-ERROR_TRANSACTION_UNREGISTERED_COLLECTIONWillberaisedwhenacollectionisusedinthemiddleofatransactionbutwasnotregisteredattransactionstart.

1653-ERROR_TRANSACTION_DISALLOWED_OPERATIONWillberaisedwhenadisallowedoperationiscarriedoutinatransaction.

1654-ERROR_TRANSACTION_ABORTEDWillberaisedwhenatransactionwasaborted.

Usermanagementerrors

Errorcodesandmeanings

695

1700-ERROR_USER_INVALID_NAMEWillberaisedwhenaninvalidusernameisused.

1701-ERROR_USER_INVALID_PASSWORDWillberaisedwhenaninvalidpasswordisused.

1702-ERROR_USER_DUPLICATEWillberaisedwhenausernamealreadyexists.

1703-ERROR_USER_NOT_FOUNDWillberaisedwhenausernameisupdatedthatdoesnotexist.

1704-ERROR_USER_CHANGE_PASSWORDWillberaisedwhentheusermustchangehispassword.

1705-ERROR_USER_EXTERNALWillberaisedwhentheuserisauthenicatedbyanexternalserver.

Servicemanagementerrors(legacy)

ThesehavebeensupercededbytheFoxxmanagementerrorsinpublicAPIs.

1750-ERROR_SERVICE_INVALID_NAMEWillberaisedwhenaninvalidservicenameisspecified.

1751-ERROR_SERVICE_INVALID_MOUNTWillberaisedwhenaninvalidmountisspecified.

1752-ERROR_SERVICE_DOWNLOAD_FAILEDWillberaisedwhenaservicedownloadfromthecentralrepositoryfailed.

1753-ERROR_SERVICE_UPLOAD_FAILEDWillberaisedwhenaserviceuploadfromtheclienttotheArangoDBserverfailed.

LDAPerrors1800-ERROR_LDAP_CANNOT_INITcannotinitaLDAPconnection

1801-ERROR_LDAP_CANNOT_SET_OPTIONcannotsetaLDAPoption

1802-ERROR_LDAP_CANNOT_BINDcannotbindtoaLDAPserver

1803-ERROR_LDAP_CANNOT_UNBINDcannotunbindfromaLDAPserver

1804-ERROR_LDAP_CANNOT_SEARCHcannotsearchtheLDAPserver

1805-ERROR_LDAP_CANNOT_START_TLScannotstaraTLSLDAPsession

1806-ERROR_LDAP_FOUND_NO_OBJECTSLDAPdidn'tfoundanyobjectswiththespecifiedsearchquery

1807-ERROR_LDAP_NOT_ONE_USER_FOUNDLDAPfoundzerooremorethanoneuser

1808-ERROR_LDAP_USER_NOT_IDENTIFIEDLDAPfoundauser,butitsnotthedesiredone

Errorcodesandmeanings

696

1820-ERROR_LDAP_INVALID_MODEcantdistinguishavalidmodeforprovidedldapconfiguration

Taskerrors

1850-ERROR_TASK_INVALID_IDWillberaisedwhenataskiscreatedwithaninvalidid.

1851-ERROR_TASK_DUPLICATE_IDWillberaisedwhenataskidiscreatedwithaduplicateid.

1852-ERROR_TASK_NOT_FOUNDWillberaisedwhenataskwiththespecifiedidcouldnotbefound.

Graph/traversalerrors1901-ERROR_GRAPH_INVALID_GRAPHWillberaisedwhenaninvalidnameispassedtotheserver.

1902-ERROR_GRAPH_COULD_NOT_CREATE_GRAPHWillberaisedwhenaninvalidname,verticesoredgesispassedtotheserver.

1903-ERROR_GRAPH_INVALID_VERTEXWillberaisedwhenaninvalidvertexidispassedtotheserver.

1904-ERROR_GRAPH_COULD_NOT_CREATE_VERTEXWillberaisedwhenthevertexcouldnotbecreated.

1905-ERROR_GRAPH_COULD_NOT_CHANGE_VERTEXWillberaisedwhenthevertexcouldnotbechanged.

1906-ERROR_GRAPH_INVALID_EDGEWillberaisedwhenaninvalidedgeidispassedtotheserver.

1907-ERROR_GRAPH_COULD_NOT_CREATE_EDGEWillberaisedwhentheedgecouldnotbecreated.

1908-ERROR_GRAPH_COULD_NOT_CHANGE_EDGEWillberaisedwhentheedgecouldnotbechanged.

1909-ERROR_GRAPH_TOO_MANY_ITERATIONSWillberaisedwhentoomanyiterationsaredoneinagraphtraversal.

1910-ERROR_GRAPH_INVALID_FILTER_RESULTWillberaisedwhenaninvalidfilterresultisreturnedinagraphtraversal.

1920-ERROR_GRAPH_COLLECTION_MULTI_USEanedgecollectionmayonlybeusedonceinoneedgedefinitionofagraph.,

1921-ERROR_GRAPH_COLLECTION_USE_IN_MULTI_GRAPHSisalreadyusedbyanothergraphinadifferentedgedefinition.,

1922-ERROR_GRAPH_CREATE_MISSING_NAMEagraphnameisrequiredtocreateagraph.,

1923-ERROR_GRAPH_CREATE_MALFORMED_EDGE_DEFINITIONtheedgedefinitionismalformed.Ithastobeanarrayofobjects.,

1924-ERROR_GRAPH_NOT_FOUNDagraphwiththisnamecouldnotbefound.,

1925-ERROR_GRAPH_DUPLICATEagraphwiththisnamealreadyexists.,

Errorcodesandmeanings

697

1926-ERROR_GRAPH_VERTEX_COL_DOES_NOT_EXISTthespecifiedvertexcollectiondoesnotexistorisnotpartofthegraph.,

1927-ERROR_GRAPH_WRONG_COLLECTION_TYPE_VERTEXthecollectionisnotavertexcollection.,

1928-ERROR_GRAPH_NOT_IN_ORPHAN_COLLECTIONVertexcollectionnotinorphancollectionofthegraph.,

1929-ERROR_GRAPH_COLLECTION_USED_IN_EDGE_DEFThecollectionisalreadyusedinanedgedefinitionofthegraph.,

1930-ERROR_GRAPH_EDGE_COLLECTION_NOT_USEDTheedgecollectionisnotusedinanyedgedefinitionofthegraph.,

1931-ERROR_GRAPH_NOT_AN_ARANGO_COLLECTIONThecollectionisnotanArangoCollection.,

1932-ERROR_GRAPH_NO_GRAPH_COLLECTIONcollection_graphsdoesnotexist.,

1933-ERROR_GRAPH_INVALID_EXAMPLE_ARRAY_OBJECT_STRINGInvalidexampletype.HastobeString,ArrayorObject.,

1934-ERROR_GRAPH_INVALID_EXAMPLE_ARRAY_OBJECTInvalidexampletype.HastobeArrayorObject.,

1935-ERROR_GRAPH_INVALID_NUMBER_OF_ARGUMENTSInvalidnumberofarguments.Expected:,

1936-ERROR_GRAPH_INVALID_PARAMETERInvalidparametertype.,

1937-ERROR_GRAPH_INVALID_IDInvalidid,

1938-ERROR_GRAPH_COLLECTION_USED_IN_ORPHANSThecollectionisalreadyusedintheorphansofthegraph.,

1939-ERROR_GRAPH_EDGE_COL_DOES_NOT_EXISTthespecifiededgecollectiondoesnotexistorisnotpartofthegraph.,

1940-ERROR_GRAPH_EMPTYTherequestedgraphhasnoedgecollections.

Sessionerrors1950-ERROR_SESSION_UNKNOWNWillberaisedwhenaninvalid/unknownsessionidispassedtotheserver.

1951-ERROR_SESSION_EXPIREDWillberaisedwhenasessionisexpired.

SimpleClienterrors

2000-SIMPLE_CLIENT_UNKNOWN_ERRORThiserrorshouldnothappen.

2001-SIMPLE_CLIENT_COULD_NOT_CONNECTWillberaisedwhentheclientcouldnotconnecttotheserver.

2002-SIMPLE_CLIENT_COULD_NOT_WRITEWillberaisedwhentheclientcouldnotwritedata.

Errorcodesandmeanings

698

2003-SIMPLE_CLIENT_COULD_NOT_READWillberaisedwhentheclientcouldnotreaddata.

Communicatorerrors

2100-COMMUNICATOR_REQUEST_ABORTEDRequestwasaborted.

2101-COMMUNICATOR_DISABLEDCommunicationwasdisabled.

Foxxmanagementerrors

3000-ERROR_MALFORMED_MANIFEST_FILETheservicemanifestfileisnotwell-formedJSON.

3001-ERROR_INVALID_SERVICE_MANIFESTTheservicemanifestcontainsinvalidvalues.

3002-ERROR_SERVICE_FILES_MISSINGTheservicefolderorbundledoesnotexistonthisserver.

3003-ERROR_SERVICE_FILES_OUTDATEDThelocalservicebundledoesnotmatchthechecksuminthedatabase.

3004-ERROR_INVALID_FOXX_OPTIONSTheserviceoptionscontaininvalidvalues.

3007-ERROR_INVALID_MOUNTPOINTTheservicemountpathcontainsinvalidcharacters.

3009-ERROR_SERVICE_NOT_FOUNDNoservicefoundatthegivenmountpath.

3010-ERROR_SERVICE_NEEDS_CONFIGURATIONTheserviceismissingconfigurationordependencies.

3011-ERROR_SERVICE_MOUNTPOINT_CONFLICTAservicealreadyexistsatthegivenmountpath.

3012-ERROR_SERVICE_MANIFEST_NOT_FOUNDTheservicedirectorydoesnotcontainamanifestfile.

3013-ERROR_SERVICE_OPTIONS_MALFORMEDTheserviceoptionsarenotwell-formedJSON.

3014-ERROR_SERVICE_SOURCE_NOT_FOUNDThesourcepathdoesnotmatchafileordirectory.

3015-ERROR_SERVICE_SOURCE_ERRORThesourcepathcouldnotberesolved.

3016-ERROR_SERVICE_UNKNOWN_SCRIPTTheservicedoesnothaveascriptwiththisname.

JavaScriptmoduleloadererrors3100-ERROR_MODULE_NOT_FOUNDThemodulepathcouldnotberesolved.

Errorcodesandmeanings

699

3101-ERROR_MODULE_SYNTAX_ERRORThemodulecouldnotbeparsedbecauseofasyntaxerror.

3103-ERROR_MODULE_FAILUREFailedtoinvokethemoduleinitscontext.

Enterpriseerrors

4000-ERROR_NO_SMART_COLLECTIONTherequestedcollectionneedstobesmart,butitain't

4001-ERROR_NO_SMART_GRAPH_ATTRIBUTEThegivendocumentdoesnothavethesmartgraphattributeset.

4002-ERROR_CANNOT_DROP_SMART_COLLECTIONThissmartcollectioncannotbedropped,itdictatesshardinginthegraph.

4003-ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTEInasmartvertexcollection_keymustbeprefixedwiththevalueofthesmartgraphattribute.

4004-ERROR_ILLEGAL_SMART_GRAPH_ATTRIBUTEThegivensmartGraphattributeisillegalandconnotbeusedforsharding.Allsystemattributesareforbidden.

Agencyerrors

20001-ERROR_AGENCY_INQUIRY_SYNTAXInquiryhandlesalistofstringclientIds:[,...].

20011-ERROR_AGENCY_INFORM_MUST_BE_OBJECTTheinformmessageintheagencymustbeanobject.

20012-ERROR_AGENCY_INFORM_MUST_CONTAIN_TERMTheinformmessageintheagencymustcontainauintparameter'term'.

20013-ERROR_AGENCY_INFORM_MUST_CONTAIN_IDTheinformmessageintheagencymustcontainastringparameter'id'.

20014-ERROR_AGENCY_INFORM_MUST_CONTAIN_ACTIVETheinformmessageintheagencymustcontainanarray'active'.

20015-ERROR_AGENCY_INFORM_MUST_CONTAIN_POOLTheinformmessageintheagencymustcontainanobject'pool'.

20016-ERROR_AGENCY_INFORM_MUST_CONTAIN_MIN_PINGTheinformmessageintheagencymustcontainanobject'minping'.

20017-ERROR_AGENCY_INFORM_MUST_CONTAIN_MAX_PINGTheinformmessageintheagencymustcontainanobject'maxping'.

20018-ERROR_AGENCY_INFORM_MUST_CONTAIN_TIMEOUT_MULTTheinformmessageintheagencymustcontainanobject'timeoutMult'.

20020-ERROR_AGENCY_INQUIRE_CLIENT_ID_MUST_BE_STRINGInquirybyclientIdfailed

20021-ERROR_AGENCY_CANNOT_REBUILD_DBSWillberaisedifthereadDBorthespearHeadcannotberebuiltfromthereplicatedlog.

Supervisionerrors

20501-ERROR_SUPERVISION_GENERAL_FAILURE

Errorcodesandmeanings

700

Generalsupervisionfailure.

Dispatchererrors

21001-ERROR_DISPATCHER_IS_STOPPINGWillbereturnedifashutdownisinprogress.

21002-ERROR_QUEUE_UNKNOWNWillbereturnedifaqueuewiththisnamedoesnotexist.

21003-ERROR_QUEUE_FULLWillbereturnedifaqueuewiththisnameisfull.

Errorcodesandmeanings

701

Glossary

Collection

Acollectionconsistsofdocuments.Itisuniquelyidentifiedbyitscollectionidentifier.Italsohasauniquenamethatclientsshouldusetoidentifyandaccessit.Collectionscanberenamed.Itwillchangethecollectionname,butnotthecollectionidentifier.Collectionscontaindocumentsofaspecifictype.Therearecurrentlytwotypes:document(default)andedge.Thetypeisspecifiedbytheuserwhenthecollectioniscreated,andcannotbechangedlater.

CollectionIdentifier

Acollectionidentifieridentifiesacollectioninadatabase.Itisastringvalueandisuniquewithinthedatabase.UptoincludingArangoDB1.1,thecollectionidentifierhasbeenaclient'sprimarymeanstoaccesscollections.StartingwithArangoDB1.2,clientsshouldinsteaduseacollection'suniquenametoaccessacollectioninsteadofitsidentifier.

ArangoDBcurrentlyuses64bitunsignedintegervaluestomaintaincollectionidsinternally.Whenreturningcollectionidstoclients,ArangoDBwillputthemintoastringtoensurethecollectionidisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreatthecollectionidsreturnedbyArangoDBasopaquestringswhentheystoreoruseitlocally.

CollectionName

Acollectionnameidentifiesacollectioninadatabase.Itisastringandisuniquewithinthedatabase.Unlikethecollectionidentifieritissuppliedbythecreatorofthecollection.Thecollectionnamemustconsistofletters,digits,andthe_(underscore)and-(dash)charactersonly.PleaserefertoNamingConventionsformoreinformationonvalidcollectionnames.

Database

ArangoDBcanhandlemultipledatabasesinthesameserverinstance.Databasescanbeusedtologicallygroupandseparatedata.AnArangoDBdatabaseconsistsofcollectionsanddedicateddatabase-specificworkerprocesses.

Adatabasecontainsitsowncollections(whichcannotbeaccessedfromotherdatabases),Foxxapplications,andreplicationloggersandappliers.EachArangoDBdatabasecontainsitsownsystemcollections(e.g._users,_replication,...).

TherewillalwaysbeatleastonedatabaseinArangoDB.Thisisthedefaultdatabase,named_system.Thisdatabasecannotbedropped,andprovidesspecialoperationsforcreating,dropping,andenumeratingdatabases.Userscancreateadditionaldatabasesandgivethemuniquenamestoaccessthemlater.Databasemanagementoperationscannotbeinitiatedfromoutofuser-defineddatabases.

WhenArangoDBisaccessedviaitsHTTPRESTAPI,thedatabasenameisreadfromthefirstpartoftherequestURIpath(e.g./_db/_system/...).IftherequestURIdoesnotcontainadatabasename,thedatabasenameisautomaticallyderivedfromtheendpoint.PleaserefertoDatabaseEndpointformoreinformation.

DatabaseName

AsingleArangoDBinstancecanhandlemultipledatabasesinparallel.Whenmultipledatabasesareused,eachdatabasemustbegivenauniquename.Thisnameisusedtouniquelyidentifyadatabase.ThedefaultdatabaseinArangoDBisnamed_system.

Thedatabasenameisastringconsistingofonlyletters,digitsandthe_(underscore)and-(dash)characters.User-defineddatabasenamesmustalwaysstartwithaletter.Databasenamesiscase-sensitive.

DatabaseOrganization

AsingleArangoDBinstancecanhandlemultipledatabasesinparallel.Bydefault,therewillbeatleastonedatabase,whichisnamed_system.

Databasesarephysicallystoredinseparatesub-directoriesunderneaththedatabasedirectory,whichitselfresidesintheinstance'sdatadirectory.

Glossary

702

Eachdatabasehasitsownsub-directory,nameddatabase-.Thedatabasedirectorycontainssub-directoriesforthecollectionsofthedatabase,andafilenamedparameter.json.Thisfilecontainsthedatabaseidandname.

InanexampleArangoDBinstancewhichhastwodatabases,thefilesystemlayoutcouldlooklikethis:

data/#theinstance'sdatadirectory

databases/#sub-directorycontainingalldatabases'data

database-<id>/#sub-directoryforasingledatabase

parameter.json#filecontainingdatabaseidandname

collection-<id>/#directorycontainingdataaboutacollection

database-<id>/#sub-directoryforanotherdatabase

parameter.json#filecontainingdatabaseidandname

collection-<id>/#directorycontainingdataaboutacollection

collection-<id>/#directorycontainingdataaboutacollection

Foxxapplicationsarealsoorganizedindatabase-specificdirectoriesinsidetheapplicationpath.Thefilesystemlayoutcouldlooklikethis:

apps/#theinstance'sapplicationdirectory

system/#systemapplications(canbeignored)

_db/#sub-directorycontainingdatabase-specificapplications

<database-name>/#sub-directoryforasingledatabase

<mountpoint>/APP#sub-directoryforasingleapplication

<mountpoint>/APP#sub-directoryforasingleapplication

<database-name>/#sub-directoryforanotherdatabase

<mountpoint>/APP#sub-directoryforasingleapplication

Document

DocumentsinArangoDBareJSONobjects.Theseobjectscanbenested(toanydepth)andmaycontainarrays.Eachdocumentisuniquelyidentifiedbyitsdocumenthandle.

DocumentEtag

Thedocumentrevision(_revvalue)enclosedindoublequotes.TherevisionisreturnedbyseveralHTTPAPImethodsintheEtagHTTPheader.

DocumentHandle

Adocumenthandleuniquelyidentifiesadocumentinthedatabase.Itisastringandconsistsofthecollection'snameandthedocumentkey(_keyattribute)separatedby/.Thedocumenthandleisstoredinadocument's_idattribute.

DocumentKey

Adocumentkeyisastringthatuniquelyidentifiesadocumentinagivencollection.Itcanandshouldbeusedbyclientswhenspecificdocumentsaresearched.Documentkeysarestoredinthe_keyattributeofdocuments.ThekeyvaluesareautomaticallyindexedbyArangoDBinacollection'sprimaryindex.Thuslookingupadocumentbyitskeyisregularlyafastoperation.The_keyvalueofadocumentisimmutableoncethedocumenthasbeencreated.

Bydefault,ArangoDBwillauto-generateadocumentkeyifno_keyattributeisspecified,andusetheuser-specified_keyvalueotherwise.

Thisbehaviorcanbechangedonaper-collectionlevelbycreatingcollectionswiththekeyOptionsattribute.

UsingkeyOptionsitispossibletodisallowuser-specifiedkeyscompletely,ortoforceaspecificregimeforauto-generatingthe_keyvalues.

Therearesomerestrictionsforuser-definedkeys(seeNamingConventionsfordocumentkeys).

DocumentRevision

Glossary

703

AsArangoDBsupportsMVCC,documentscanexistinmorethanonerevision.ThedocumentrevisionistheMVCCtokenusedtoidentifyaparticularrevisionofadocument.Itisastringvaluecurrentlycontaininganintegernumberandisuniquewithinthelistofdocumentrevisionsforasingledocument.Documentrevisionscanbeusedtoconditionallyupdate,replaceordeletedocumentsinthedatabase.Inordertofindaparticularrevisionofadocument,youneedthedocumenthandleandthedocumentrevision.

Thedocumentrevisionisstoredinthe_revattributeofadocument,andissetandupdatedbyArangoDBautomatically.The_revvaluecannotbesetfromtheoutside.

ArangoDBcurrentlyuses64bitunsignedintegervaluestomaintaindocumentrevisionsinternally.Whenreturningdocumentrevisionstoclients,ArangoDBwillputthemintoastringtoensuretherevisionidisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreattherevisionidreturnedbyArangoDBasanopaquestringwhentheystoreoruseitlocally.ThiswillallowArangoDBtochangetheformatofrevisionidslaterifthisshouldberequired.Clientscanuserevisionsidstoperformsimpleequality/non-equalitycomparisons(e.g.tocheckwhetheradocumenthaschangedornot),buttheyshouldnotuserevisionidstoperformgreater/lessthancomparisonswiththemtocheckifadocumentrevisionisolderthanoneanother,evenifthismightworkforsomecases.

Edge

Edgesarespecialdocumentsusedforconnectingotherdocumentsintoagraph.Anedgedescribestheconnectionbetweentwodocumentsusingtheinternalattributes:_fromand_to.Thesecontaindocumenthandles,namelythestart-pointandtheend-pointoftheedge.

EdgeCollection

Edgecollectionsarecollectionsthatstoreedges.

EdgeDefinition

Edgedefinitionsarepartsofthedefinitionofnamedgraphs.Theydescribewhichedgecollectionsconnectwhichvertexcollections.

GeneralGraph

Modulemaintaininggraphsetupinthe_graphscollection-akanamedgraphs.Configureswhichedgecollectionsrelatetowhichvertexcollections.Ensuresgraphconsistencyinmodificationqueries.

NamedGraphs

Namedgraphsenforceconsistencybetweenedgecollectionsandvertexcollections,soifyouremoveavertex,edgespointingtoitwillberemovedtoo.

Index

Indexesareusedtoallowfastaccesstodocumentsinacollection.Allcollectionshaveaprimaryindex,whichisthedocument's_keyattribute.Thisindexcannotbedroppedorchanged.

Edgecollectionswillalsohaveanautomaticallycreatededgesindex,whichcannotbemodified.Thisindexprovidesquickaccesstodocumentsviathe_fromand_toattributes.

Mostuser-landindexescanbecreatedbydefiningthenamesoftheattributeswhichshouldbeindexed.Someindextypesallowindexingjustoneattribute(e.g.fulltextindex)whereasotherindextypesallowindexingmultipleattributes.

Indexingthesystemattribute_idinuser-definedindexesisnotsupportedbyanyindextype.

EdgesIndex

Anedgesindexisautomaticallycreatedforedgecollections.Itcontainsconnectionsbetweenvertexdocumentsandisinvokedwhentheconnectingedgesofavertexarequeried.Thereisnowaytoexplicitlycreateordeleteedgesindexes.

FulltextIndex

Glossary

704

Afulltextindexcanbeusedtofindwords,orprefixesofwordsinsidedocuments.Afulltextindexcanbedefinedononeattributeonly,andwillincludeallwordscontainedindocumentsthathaveatextualvalueintheindexattribute.SinceArangoDB2.6theindexwillalsoincludewordsfromtheindexattributeiftheindexattributeisanarrayofstrings,oranobjectwithstringvaluemembers.

Forexample,givenafulltextindexonthetranslationsattributeandthefollowingdocuments,thensearchingforлисаusingthefulltextindexwouldreturnonlythefirstdocument.SearchingfortheindexfortheexactstringFoxwouldreturnthefirsttwodocuments,andsearchingforprefix:Foxwouldreturnallthreedocuments:

{translations:{en:"fox",de:"Fuchs",fr:"renard",ru:"лиса"}}

{translations:"FoxistheEnglishtranslationoftheGermanwordFuchs"}

{translations:["ArangoDB","document","database","Foxx"]}

Iftheindexattributeisneitherastring,anobjectoranarray,itscontentswillnotbeindexed.Whenindexingthecontentsofanarrayattribute,anarraymemberwillonlybeincludedintheindexifitisastring.Whenindexingthecontentsofanobjectattribute,anobjectmembervaluewillonlybeincludedintheindexifitisastring.Otherdatatypesareignoredandnotindexed.

Onlywordswitha(specifiable)minimumlengthareindexed.Wordtokenizationisdoneusingthewordboundaryanalysisprovidedbylibicu,whichistakingintoaccounttheselectedlanguageprovidedatserverstart.Wordsareindexedintheirlower-casedform.Theindexsupportscompletematchqueries(fullwords)andprefixqueries.

GeoIndex

Ageoindexisusedtofindplacesonthesurfaceoftheearthfast.

IndexHandle

Anindexhandleuniquelyidentifiesanindexinthedatabase.Itisastringandconsistsofacollectionnameandanindexidentifierseparatedby/.

HashIndex

Ahashindexisusedtofinddocumentsbasedonexamples.Ahashindexcanbecreatedforoneormultipledocumentattributes.

Ahashindexwillonlybeusedbyqueriesifallindexedattributesarepresentintheexampleorsearchquery,andifallattributesarecomparedusingtheequality(==operator).Thatmeansthehashindexdoesnotsupportrangequeries.

AuniquehashindexhasanamortizedcomplexityofO(1)forlookup,insert,update,andremoveoperations.Thenon-uniquehashindexissimilar,butamortizedlookupperformanceisO(n),withnbeingthenumberofindexentrieswiththesamelookupvalue.

SkiplistIndex

Askiplistisasortedindextypethatcanbeusedtofindrangesofdocuments.

AnonymousGraphs

Youmayuseedgecollectionswithvertexcollectionswithoutthegraphmanagementfacilities.However,graphconsistencyisnotenforcedbythese.Ifyouremovevertices,youhavetoensurebyyourselvesedgespointingtothisvertexareremoved.Anonymousgraphsmaynotbebrowsedusinggraphviewerinthewebinterface.Thismaybefasterinsomescenarios.

View

Aviewisconceptiallyatransformationfunctionoverdocumentsfromzeroormorecollections.Itisuniquelyidentifiedbyitsviewidentifier.Italsohasauniquenamethatclientsshouldusetoidentifyandaccessit.Viewscanberenamed.Renamingaviewwillchangetheviewname,butnottheviewidentifier.Theconceptualtransformationfunctionemployedbyaviewtypeisimplementationspecific.Thetypeisspecifiedbytheuserwhentheviewiscreated,andcannotbechangedlater.Thefollowingviewtypesarecurrentlysupported:

arangosearch

ViewIdentifier

Glossary

705

Aviewidentifieridentifiesaviewinadatabase.Itisastringvalueandisuniquewithinthedatabase.Clientsshoulduseaview'suniquenametoaccessaviewinsteadofitsidentifier.

ArangoDBcurrentlyuses64bitunsignedintegervaluestomaintainviewidsinternally.Whenreturningviewidstoclients,ArangoDBwillputthemintoastringtoensuretheviewidisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreattheviewidsreturnedbyArangoDBasopaquestringswhentheystoreorusethemlocally.

ViewName

Aviewnameidentifiesaviewinadatabase.Itisastringandisuniquewithinthedatabase.Unliketheviewidentifieritissuppliedbythecreatoroftheview.Theviewnamemustconsistofletters,digits,andthe_(underscore)and-(dash)charactersonly.PleaserefertoNamingConventionsformoreinformationonvalidviewnames,whichfollowthesameguidelinesascollectionnames.

Glossary

706