Embed Size (px)
Citation preview
openDIEL: A Parallel Workflow Engine and DataAnalytics Framework
Frank BetancourtThe University of Tennessee
Kwai WongThe University of Tennessee
Efosa AsemotaMorehouse College
Quindell MarshallMorehouse College
Daniel NicholsThe University of Tennessee
Stanimire TomovThe University of Tennessee
ABSTRACTopenDIEL is a work�ow engine that aims to give researchersand users of HPC an e�cient way to coordinate, organize,and interconnect many disparate modules of computation inorder to e�ectively utilize and allocate HPC resources [13].A GUI has been developed to aid in creating work�ows, andallows for the speci�cation of data science jobs, includingspeci�cation neural network architectures, data processing,and hyperparameter tuning. Existing machine learning toolscan be readily used in the openDIEL, allowing for easy ex-perimentation with various models and approaches.
CCS CONCEPTS• Software and its engineering → Software creationandmanagement;Designing software; Software designengineering;
KEYWORDSwork�ow engine, neural network, data science, graphicaluser interface
ACM Reference Format:Frank Betancourt, Kwai Wong, Efosa Asemota, Quindell Marshall,Daniel Nichols, and Stanimire Tomov. 2019. openDIEL: A ParallelWork�ow Engine and Data Analytics Framework. In Practice andExperience in Advanced Research Computing (PEARC ’19), July 28-August 1, 2019, Chicago, IL, USA. ACM, New York, NY, USA, 7 pages.https://doi.org/10.1145/3332186.3333051
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for pro�t or commercial advantage and thatcopies bear this notice and the full citation on the �rst page. Copyrightsfor components of this work owned by others than the author(s) mustbe honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior speci�cpermission and/or a fee. Request permissions from [email protected] ’19, July 28-August 1, 2019, Chicago, IL, USA© 2019 Copyright held by the owner/author(s). Publication rights licensedto ACM.ACM ISBN 978-1-4503-7227-5/19/07. . . $15.00https://doi.org/10.1145/3332186.3333051
1 INTRODUCTIONWhen conducting complicated simulations, it is often re-quired to utilize a large variety of applications to answera research question. Utilizing disparate modules of compu-tation, requires the careful coordination of dependencies,communication, and synchronization between them, andthere is not always a clear path on how to do these kindsof tasks. One approach to this is to utilize a script to runall of the necessary modules. However, this falls short, asthis loses much detail with regard to complex interactionbetween modules, such as inter-moduler communicationand non-linear dependencies, limiting the ability to take fulladvantage of HPC systems to run modules in parallel.This problem is solved by work�ow engines, allowing
researchers to de�ne complex dependencies between mod-ules, and schedule communication between them. openDIELspeci�cally focuses on unifying modules into a single ex-ecutable that uses MPI for communication. Additionally,methods are provided for specifying dependencies betweenmodules, with the framework resolving dependencies andrunning modules in parallel.
openDIEL is designed not only to support the creation ofgeneric work�ows, but facilities are also provided to createdata science work�ows, supporting simple data cleaning,creation of neural network architectures, and subsequentlysearching for optimal parameters with a grid engine.
The major work that has been done to extend the originalopenDIEL framework is to provide a GUI with the goal ofincreasing usability. The standard manner in which work-�ows are con�gured is to edit con�guration �les that de�nethe way the engine will work. The major drawback to thisapproach is that it is not always clear how to set up a �le oreven what options are available. The GUI overcomes this byproviding a clear list of what can be speci�ed, which reducesthe load on the end user to think of everything ahead of time,but instead recognize what needs to be done.
2 RELATEDWORKExisting work�ow engines such as Galaxy give researchersthe means to utilize tools on cloud computing resources.Galaxy is designed to operate with and couple togetherdomain-speci�c tools, without requiring the end user todo large amounts of work to integrate tools together [1].Apache Airavata is another work�ow engine that is designedto manage complex computational jobs and work�ows. Itis designed to work on a variety of di�erent computing re-sources, including local clusters and the cloud [8].openDIEL shares many of the same goals core goals as
these frameworks, but di�ers in several ways. The �rst is thatthe openDIEL framework uses MPI as a core component forcommunication between modules and for managing runningwork�ows. The other way in which openDIEL di�ers is thatit is designed speci�cally for tightly coupled HPC systems.Another one of the goals of the openDIEL GUI is to pro-
vide a general framework for performing hyperparametersearch and model selection tasks. A number of tools alreadyexist to assist in doing this. Tools such as HyperBand, Optu-nity, and BayesOpt provide methods to perform optimizationwith a variety of di�erent algorithms, but do have supportfor training in a distributed manner out-of-the-box [3, 6, 9].Many existing systems also lack support for implementingalgorithms beyond what is provided in the tool kit. The TuneFramework is a system that supports custom algorithm im-plementation, and is designed to be used in a distributedmanner [7].
openDIEL Framework Design OverviewAt a high level, the openDIEL system consists of a C librarythat contains all of the function needed to manage modules.Typically, a main driver �le is created which contains all ofthe needed function calls to set up the main MPI commu-nicator that the IEL library uses, set up necessary modulesfor tuple space communication, and calling the user de�nedmodules.
Themainway that users interact with the system is througha con�guration �le. There are two major components of theopenDIEL system: the executive library, and the communi-cation library.
Configuration File. Information about how modules are com-municate and rely on one another is contained in a con�gu-ration �le. The con�guration �le de�nes what resources themodules requires, such as the number of cores, and numberof GPUs required by the module. After de�ning the modulesthemselves, a section of the �le subsequently de�nes themanner in which groups of modules depend on one another,how many iterations need to run, amongst other character-istics.
Figure 1: Overview of architecture of openDIEL, top to bot-tom: (A) The GUI launcher creates a con�guration �le forthe work�ow, and the executive will read this �le to set upwork�ows. (B) After initial con�guration, the executive willstart all of the modules. (C) The modules have access to thecommunication library, and directly communicate or utilizetuple space communication.
Communication Library. The communication library is es-sentially a wrapper around various openMPI function calls,and is responsible for managing both tuple space and directcommunication between modules. This is done by creatinga main MPI_COMM_WORLD communicator in which all ofthe modules run, and then subdividing this main commu-nicator at the level of single modules. If there are multipleconcurrent copies of the same module running at the sametime, then the module sub-communicator is further subdi-vided between the copies [13].
Two di�erent methods of communication are provided byan API: tuple space communication, and direct module-to-module communication. With tuple space communication, atuple server module is utilized that allows modules to concur-rently send data to and receive data from a shared associativearray. Modules can use this form of communication with pro-vided library functions IEL_tput and IEL_tget to send andreceive data from the tuple space respectively. Each modulethat puts data into the tuple space can issue a non-blockingIEL_tput function call, and provide a tag for the data placed in
2
the tuple space. The receiving module can utilize a blockingIEL_tget function call to retrieve the data with the speci�edtag.
The IEL_tput function is a wrapper around the MPI_Sendfunction, and the tuple server runs as an MPI process inthe MPI_COMM_WORLD communicator. The tuple serverreceives the data from a module, and then places it into ared-black tree keyed on the tag, and values is a queue intowhich the data is placed. The IEL_tget function utilizes theMPI_Send function to notify the tuple server that it wantsto retrieve a tag, and whether or not it wants to remove thedata from the tuple server after retrieving it. MPI_Recv isthen called, and the data is retrieved.
Simpli�ed versions of IEL_tput and IEL_tget are also pro-vided: IEL_tput_simpli�ed and IEL_tget_simpli�ed. The al-low users to specify the name of the calling module andtag for the data as a string, and does not require the userto specify the rank upon which the tuple server is running.This function is just a wrapper around the normal IEL_tputfunction call, but the string provided is hashed and used asthe tag in the call.
Figure 2: Layout of the distributed tuple servermodel, clock-wise from the bottom right: (A) The client data is brokenup and distributed across an array of tuple servers, and (B)the metadata for the distributed data is stored in a metadataserver. (C)When data is to be retrieved, themetadata for therequested data is retrieved, and (D) the data is retrieved fromthe distributed array of tuple servers based on the metadata.The received data is then reconstructed and returned to therequesting client.
The tuple space can also be distributed across a numberof di�erent modules, essentially providing a way to storeand retrieve data in a distributed manner. The functionsIEL_dist_tget and IEL_dist_tput utilize this multiple tuple
server model. The IEL_dist_tget function will take a pointerto data and a string to tag the data, and distribute it amongstan array of tuple servers. Information about the distributionof the data is stored on a meta-data server. The IEL_dist_tgetfunction will retrieve the data by querying the meta dataserver, which returns the locations of the servers holdingthe data, the data servers are queried, and the stored data isreconstructed.
Executive Library. The executive library is the other majorpart of the library responsible for starting job and managingdependencies. When a job starts, the executive will read in awork�ow con�guration �le, and then based on this �le, theexecutive will create a dependency graph of the speci�edwork�ow, and then start modules based on the graph. Typi-cally, a module is included in openDIEL by linking a libraryagainst a driver �le, and function pointers are provided tothe executive so that they can be called with the appropriatearguments [13]. Executables can also be run by calling fork()and exec() in an MPI process, but limits the ability of themodule to utilize the inter-modular communication providedby openDIEL.
Typical Usage. Typically, all of the needed functions arecalled in a main driver �le. This driver �le will call MPI_init,and then it will call openDIEL member function IELAddMod-ule, which will take a pointer to the function in the linkedlibrary for the module. This will be used later to start themodule in the work�ow. For modules that are executables, amodel that calls fork() and exec() on the proper argumentsis started for each serial module. After this setup, the mainIELExecutive() member function is called. This function willsplit up the MPI_COMM_WORLD communicator into theappropriate subcommunicators, resolve dependencies fromthe con�guration �le, and then start modules.
Graphical User InterfaceThe GUI provides methods to easily specify modules, theresources required to run them, and the libraries needed tolink against in order to run them. For work�ow speci�cation,methods to visually display the resulting work�ow as a de-pendency graph are provided to aid in the ease of creating awork�ow. The other enhancement is automatically creatinga driver �le to run parallel codes. A number of precon�guredmodules can also be provided, allowing for a reduced amountof set up on the part of the end user of the system.
While manual con�guration can still be accomplished, thegraphical user interface aims to reduce the amount of workrequired to set up a work�ow, and improve the overall useability of the openDIEL framework. Most operations the GUIare normally accomplished with editing con�guration �les,writing code in a driver �le, linking libraries, and writing
3
modules. The GUI itself is divided into several di�erent tabs:
Figure 3: The module speci�cation tab, clockwise from topleft: (A) This section is where new modules are created, andwhere createdmodules are edited. (B)A list of the savedmod-ules. (C) A listing of the precon�gured modules which canbe loaded, along with the option to load modules previouslycreated by the user
Module Specification. The module speci�cation tab allowsusers to specify information about each individual module.Resource requirements such as the number of processesneeded (size), the number of copies of the module to run,and the number of GPUs needed can be speci�ed, amongothers. Additionally, a number of precon�gured modulescan be loaded. These precon�gured modules consist of allthe basic information needed to include it in the openDIELsystem, such as library locations, function calls, and basicresource requirements.
Workflow. After con�guring modules in the work�ow tab,these modules can then be added to work�ows. Modulesare organized into groups, and dependencies are de�ned be-tween groups of modules. Within groups, a list of modules isspeci�ed that is run linearly. If no other groups are speci�edas dependencies, the groups will run as resources becomeavailable, making parallelism the default. Work�ows can alsobe loaded from �les, which allows users to use work�owsthat have already been created, and save work�ows for lateruse.
Save and Launch. The Save and Launch tabs are used forsaving the current state and launching the job respectively.The save tab will create a con�guration �le that can be eitherused by the GUI later on to load in the same work�ow, or itcan be used by the openDIEL back end to run a work�owjob. The launch tab allows a user to launch the job as it waspreviously speci�ed, and directs the output to a speci�ed
Figure 4: The work�ow speci�cation tab, left to right: (A)Listing of themodules de�ned in the previous section of theGUI, and a list of the already created work�ow groups. Thisis used to addmodules to work�ow groups, and to add an ex-isting work�ow group as a dependency to the current group.(B) Entry �elds for the group currently being de�ned.
Figure 5: The Save Tab, left to right: (A) The contents of thework�ow con�guration �le that was created and saved. Thisis what the driver will read when it starts in order to con�g-ure modules. (B) A dependency graph of the groups to aidin visualizing the dependencies between con�gured work-�ow groups. The vertices represent work�ow groups, anddirected edges terminate at the group the originating vertexdepends upon.
directory. Example work�ows are also provided for the pur-poses of tutorials, which will load a pre-existing work�owcontaining the requisite information to run.
Machine Learning. The Machine Learning tab implementsa number of functions. Users can specify a CSV data �le,and get a basic idea of the contents of the �le, such as thenumber of rows, columns, and the names of the columnsin the �le. The option to perform common pre-processingtasks on columns in the data set is provided, such as data
4
Figure 6: Machine Learning Tab, left to right: (A) Data tab,CSV �les can be loaded, and a summary of the columns isprovided. Shown are a number of transformations that canbe applied to the columns of the CSV �le that was read in.(B) Network architecture can be de�ned here, along with theparameters for each layer. (C) Hyperparameters grid searchrange parameters can be de�ned here.
centering, normalizing, missing value imputation, and encod-ing. The interface also provides a method to specify a DNNarchitecture for classi�cation tasks, with support for fullyconnected, �attening and activation layers. After specify themodel, hyperparameter search can be performed. This isdone by providing a starting, and ending value for the hyper-parameter, and a step size. The grid search is then performedon the Cartesian product of all of the parameters.
Grid EngineOne of the goals of the framework is to not only providefacilities for hyperparameter optimization and search, butallow for users to readily use existing libraries to performthese tasks. One such implementation is a grid search en-gine that uses the openDIEL tuple space communication todistribute parameters to worker processes in an exhaustivesearch of a speci�ed parameter space, collect the results, andreport the best parameters found.The module consists of a master process that chooses
hyperparameters, and a set of processes that receive thehyperparameters. The master process �rst selects a set ofparameters, distributes them to the workers via the tuplespace, and waits for the group to �nish. The group of workerprocesses receive the parameters, train, and report their re-sults to the trainer via tuple space communication. Theseresults are then gathered by the master process, and thenthe next group of processes is started on the next batch ofparameters.openDIEL uses magmaDNN to de�ne neural networks.
magmaDNN is a deep neural network library that makes use
Figure 7: Distributing Parameters to Workers With TupleSpace
of highly optimized Magma BLAS to achieve speeds beyondother frameworks [10]. The grid engine module works bycreating a con�guration �le that contains a speci�cationof a parameter space, and a network architecture. Whenthe openDIEL framework starts, this �le is read in, and theexhaustive search over the speci�ed hyperparameter spacebegins. Workers use the identical network architecture, withvaried parameters for each run.
Usability ImprovementsA large part of the usability of the openDIEL system is ham-pered by the fact that editing and creating con�guration �les,creating modules, linking the appropriate libraries to satisfydependencies, and compiling everything together is often atime consuming and error prone task. One of the goals ofthe interface is to shift the work�ow creation process froma recall task into a task where one only has to recognizeoptions when they are seen.To highlight the di�erences between the new interface
and the traditional way, an example of setting up a work�owis provided here to compare and contrast between the two.Suppose a user wants to create and run a work�ow. For
simplicity, the user wants to run two modules: Module 1uses MPI and produces a set of �les as output, and anothermodule, Module 2 that waits for it to �nish and then collectsthe results from the �les that were produced. For Module 1,the user would need to create a static library containing thefunction that they want to run. For the Module 2, we canassume that it is simply a shell script that gets run after thejob is completed.After both of the modules have been created, a main dri-
ver �le needs to be created to start the openDIEL system,add function pointers for managed modules, include header�les, and appropriately start automatic modules. This ofteninvolves using numerous openDIEL member functions, andrequires at least a base knowledge of the system in order tocorrectly create a driver �le. After that, the user will thenneed to compile and link the driver with the static librariesthat represent modules, and then it will be in a runnable state.
5
This process di�ers somewhat when using the GUI, wherethe user will provide path to the module library and it’s cor-responding header �les, and then the name of the functionis also provided. The GUI then automatically generates thedriver �le, and compiles it with the provided modules.With driver �le done, the user will then need to create
the work�ow itself, and de�ne dependencies between mod-ules. With the original method, this would require the userto create a new con�guration �le, and add the appropriateoptions. In the work�ow �le, the user would need to addthe appropriate keywords for the number of processes eachmodule requires, along with the function name, amongstother attributes. This requires the user to have knowledgeof what these keywords are, and adds additional overheadin terms of what the user needs to know. The GUI easesthis burden by automatically generating work�ow �les withthe appropriate formatting and keywords based on the userprovides in the GUI. At this point, with the driver �le cre-ated, and the work�ow con�gured, the work�ow can thenbe started.
3 CONCLUSION AND FUTUREWORKThe openDIEL GUI enhances usability of the openDIEL sys-tem and allows for the creation of complex, useful work�owsin an expedient and e�cient manner. The task of writingdrivers, creating con�guration �les, and knowing the systemin general is greatly simpli�ed. Testing the overall usabilityof the framework as a whole is a major area for future study.A user study would need to be conducted to determine howeasily a user could accomplish common work�ow tasks, andhow much work it takes to learn the system as a whole.Increasing the ability of the openDIEL framework to do
full pipeline optimization of machine learning systems isenvisioned. Currently, the hyperparameter search is con-strained to simple grid search through a speci�ed searchspace, leading to an exponential increase in search time asmore parameters are added, limiting the feasibility of doingthis kind of analysis. Support of more “smartly” choosingparameters is envisioned, by allowing the master processto perform a speci�ed optimization procedure between re-ceiving parameters from workers and distributing new ones.Common approaches such as Bayesian Optimization [12]and learning curve prediction [5] will be supported [2]. Par-allel to improving the hyperparameter grid search, supportfor searching for an entire model is envisioned. Similar toproviding the framework with a generic search space forselecting the best hyperparameters, the framework couldprovide the capability a search for a model.
Currently, magmaDNN is the only way to create machinelearning models. In the future, support will be added to useframeworks such as TensorFlow [4], Sklearn [11], and otherother machine learning frameworks.
The openDIEL framework provides a powerful set of toolsto create work�ows with a GUI, and run many di�erent mod-ules on HPC resources. A set of communication primitivesis provided to e�ciently and easily connect together parallelMPI codes. Machine learning model creation and hyperpa-rameter search is done easily without the need for heavycoding to be done.
4 ACKNOWLEDGEMENTSThis work was conducted at the Joint Institute for Com-putational Sciences (JICS), sponsored by the National Sci-ence Foundation (NSF), through NSF REU Award #1659502,with additional Support from the University of Tennessee,Knoxville (UTK), and the National Institute for Computa-tional Sciences (NICS). This work used the Extreme Scienceand Engineering Discovery Environment (XSEDE), whichis supported by National Science Foundation grant numberACI-1548562. Computational Resources are available througha XSEDE education allocation award TG-ASC170031. In ad-dition, the computing work was also performed on technicalworkstations donated by the BP High Performance Comput-ing Team.
REFERENCES[1] Enis Afgan, Jeremy Goecks, Dannon Baker, Nate Coraor, Anton
Nekrutenko, James Taylor, Galaxy Team, et al. 2011. Galaxy: A gatewayto tools in e-science. In Guide to e-Science. Springer, 145–177.
[2] Tal Ben-Nun and Torsten Hoe�er. 2018. Demystifying parallel anddistributed deep learning: An in-depth concurrency analysis. arXivpreprint arXiv:1802.09941 (2018).
[3] Marc Claesen, Jaak Simm, Dusan Popovic, Yves Moreau, and BartDe Moor. 2014. Easy hyperparameter search using optunity. arXivpreprint arXiv:1412.1114 (2014).
[4] Sanjay Surendranath Girija. 2016. Tensor�ow: Large-scale machinelearning on heterogeneous distributed systems. Software availablefrom tensor�ow. org (2016).
[5] Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hut-ter. 2016. Learning curve prediction with Bayesian neural networks.(2016).
[6] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, andAmeet Talwalkar. 2016. Hyperband: A novel bandit-based approach tohyperparameter optimization. arXiv preprint arXiv:1603.06560 (2016).
[7] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph EGonzalez, and Ion Stoica. 2018. Tune: A research platform for dis-tributed model selection and training. arXiv preprint arXiv:1807.05118(2018).
[8] Suresh Marru, Lahiru Gunathilake, Chathura Herath, PatanachaiTangchaisin, Marlon Pierce, Chris Mattmann, Raminder Singh, ThilinaGunarathne, Eran Chinthaka, Ross Gardler, Aleksander Slominski, AteDouma, Srinath Perera, and Sanjiva Weerawarana. 2011. Apache Aira-vata: A Framework for Distributed Applications and ComputationalWork�ows. In Proceedings of the 2011 ACM Workshop on GatewayComputing Environments (GCE ’11). ACM, New York, NY, USA, 21–28.https://doi.org/10.1145/2110486.2110490
[9] Ruben Martinez-Cantin. 2014. Bayesopt: A bayesian optimizationlibrary for nonlinear optimization, experimental design and bandits.The Journal of Machine Learning Research 15, 1 (2014), 3735–3739.
6
[10] Lucien Ng, KwaiWong, AzzamHaidar, Stanimire Tomov, and Jack Don-garra. 2017. Magmadnn high-performance data analytics for manycoregpus and cpus. In magma-DNN, 2017 Summer Research Experiences forUndergraduate (REU).
[11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O.Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas,A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.2011. Scikit-learn: Machine Learning in Python. Journal of Machine
Learning Research 12 (2011), 2825–2830.[12] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical
bayesian optimization of machine learning algorithms. In Advances inneural information processing systems. 2951–2959.
[13] Kwai Wong, Logan Brown, Jason Coan, and David White. 2014. Dis-tributive Interoperable Executive Library (DIEL) for Systems of Multi-physics Simulation. In 2014 15th International Conference on Paralleland Distributed Computing, Applications and Technologies. IEEE, 49–55.
7
