Value Analytics: A Financial Module for the Open XDMoDProject

Ben FultonPervasive Technology Institute

Indiana UniversityBloomington, Indiana 43017-6221

befulton@iu.edu

Steven GalloCenter for Computational ResearchState University of New York at

Bu�aloBu�alo, New York

smgallo@bu�alo.edu

Ma� LinkPervasive Technology Institute

Indiana UniversityBloomington, Indiana

mrlink@iu.edu

Robert HenschelPervasive Technology Institute

Indiana UniversityBloomington, Indianahenschel@iu.edu

Tom YearkeCenter for Computational ResearchState University of New York at

Bu�aloBu�alo, New York

tyearke@bu�alo.edu

Katy BornerInformation Visualization Laboratory

Indiana UniversityBloomington, Indianakaty@indiana.edu

Robert L. DeLeonCenter for Computational ResearchState University of New York at

Bu�aloBu�alo, New York

rldeleon@bu�alo.edu

�omas FurlaniCenter for Computational ResearchState University of New York at

Bu�aloBu�alo, New York

furlani@ccr.bu�alo.edu

Craig A. StewartPervasive Technology Institute

Indiana UniversityBloomington, Indiana

stewart@iu

ABSTRACTUnderstanding the value of campus-based cyberinfrastructure (CI)to the institutions that invest in such CI is intrinsically di�cult.Given today’s �nancial pressures, administrative support for campus-based CI centers o�ering resources to local campus users is underconstant budgetary pressure. �is is partly due to the di�culty inobtaining quantitative metrics that clearly demonstrate the utilityof investment in campus CI centers in enhancing scienti�c researchand the �nancial aspects of enhanced competitive ability in seekingfunding for research. We propose here the addition of a new realmof metrics to the standard cyberinfrastructure tool Open XDMoD(XD Metrics on Demand) that will allow us to correlate HPC usagewith funding and publications. �e modules to be added will allowCI centers to viewmetrics relevant to both scienti�c output in termsof in publications, and �nancial data in terms of awarded grants.

CCS CONCEPTS•General and reference→Metrics; Evaluation; •Human-centeredcomputing →Visualization; •Social and professional topics→Management of computing and information systems; Fund-ing; •Applied computing→Business intelligence;Decision anal-ysis; •Information systems →Data mining;

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor pro�t or commercial advantage and that copies bear this notice and the full citationon the �rst page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).PEARC17, New Orleans, Louisiana USA© 2017 Copyright held by the owner/author(s). 123-4567-24-567/08/06. . .$15.00DOI: 10.475/123 4

KEYWORDSACM proceedings, LATEX, text tagging

ACM Reference format:Ben Fulton, Steven Gallo, Ma� Link, Robert Henschel, Tom Yearke, KatyBorner, Robert L. DeLeon, �omas Furlani, and Craig A. Stewart. 2017.Value Analytics: A Financial Module for the Open XDMoD Project. InProceedings of Practice & Experience in Advanced Research Computing, NewOrleans, Louisiana USA, July 2017 (PEARC17), 6 pages.DOI: 10.475/123 4

1 INTRODUCTIONMany universities debate the importance of investing in their ownhigh-performance computing systems. �ese systems are a com-plex combination of hardware and so�ware and require skilledpersonnel to maintain and administrate them. All this requiresa signi�cant investment, both in the short term as the hardwareand supporting systems are purchased, and in the long term asmaintenance, support, and so�ware of the systems must be takeninto account. How does a research center justify the cost to the�nancial o�cers of its institution?

Existing studies of return on investment (ROI) in campus cy-berinfrastructure show that a steady and signi�cant investmentin high performance computing is very likely to lead to increasesin publications and grant income [6]. �e e�ect is demonstrated,for example, by Indiana University’s Big Red II supercomputer,purchased in 2013 with a projected annual cost of approximately$3,000,000. Yet, according to an analysis performed at the univer-sity, facilities and administration monies coming to the universityvia grants awarded to users of the system over its �rst few years

PEARC17, July 2017, New Orleans, Louisiana USA T. Furlani et al.

of use, can be estimated at close to double the projected cost [13].�us, one can make a reasoned argument that investing in thissupercomputer was a positive decision for that institution.

However, the calculations that come into play in performing thisanalysis are based on certain assumptions. What was the valueof the supercomputer to the researcher? Could the research havebeen performed without the machine, and if so, at what cost? Werepublications enabled or enhanced by the use of the computer?While there can never be complete answers to these questions,we propose here a suite of metrics to be added to the standardcyberinfrastructure analysis tool, Open XDMoD (XD Metrics onDemand). �e metrics we propose encompass quantitative dataon relevant publications, to be�er understand scienti�c value, andquantitative data on awarded grants, to be�er understand the returnon investment in cyberinfrastructure.

Users of Open XDMoDwill be able to load their �nancial fundingdata into the Open XDMoD database using a standardized inputscheme. �e Value Analytics realm will support fully integratedanalysis of funding and usage. �ere will be options to enter the�nancial data in a customizable fashion that is most useful to theindividual institution depending on what �nancial data they collectand what types of analyses thay want to do. �e individual OpenXDMoD instance with its �nancial data will be accessible only tothe Open XDMoD user institutional users that have been grantedpermission. �e remainder of this paper will provide details onthis process and describe the progress that we have made towardsconstructing a fully functional value analytics module.

2 OPEN XDMODAs described in the introduction, the �nancial analysis will be ac-complished by the addition of a new module to the Open XDMoDHPC monitoring tool. �is is the latest addition in the continuingdevelopment of the Open XDMoD technology. Originally in 2010,the State University of New York at Bu�alo Center for Computa-tional Research (CCR) began developing an open source tool toprovide metrics, basic accounting and visualization of CPU andstorage usage at that institution [11]. Initially titled UBMoD, theproject quickly expanded into XDMoD for XSEDE, performingsimilar functions for the XSEDE cyberinfrastructure, a powerfulcollection of HPC compute, visualization, and storage resources.�e expanded functionality for that project included an improveduser interface, high-level charting, and additional analytical tools.�e XDMoD project in turn led to an open source version of thetool, Open XDMoD [4], a version of XDMoD suitable for installa-tion at an individual data center. Administrators can utilize OpenXDMoD to gather a wide range of metrics on HPC resources (Fig-ure 1), including resource utilization and performance, as well asmonitor the jobs that are running on the system to determine theire�ciency and resource consumption. �is knowledge is bene�cialin planning for future upgrades and acquisitions.

Open XDMoD is a web application suitable for installation ona Linux-based server at a data center. By parsing and processingthe log �les generated by the data center, Open XDMoD providescharts, graphs, and reports of relevant metrics over a customizabletime period. Open XDMoD is able to accept data from multiplesources, including SLURM and Torque log �les and campus LDAP

Figure 1: XDMoD allows administrators at HPC institutionsto view metrics and reports in various modes and formats.

services. �rough the highly customizable web interface, users canview summary charts and create dynamic charts and reports basedon what they �nd pertinent to their particular needs.

3 IUPTI FUNDING ANALYTICSHistorically Indiana University’s Pervasive Technology Institute(IUPTI) has been active in be�er understanding the data and usagefrom the university’s cyberinfrastructure. Initially comprising avariety of university labs focusing on di�erent types of advancedtechnology, IUPTI is currently responsible for various global initia-tives such as Jetstream, the Science Gateway Group, the AdvancedVisualization Lab, the Center for Applied Cybersecurity Research,and the National Center for Genome Analysis Support. �e Re-search Technologies division has responsibility for supporting andmaintaining the high-performance cyberinfrastructure providedto Indiana University researchers. As of 2016, RT supported threeseparate supercomputing clusters: Big Red II, Karst, and Mason.In addition, RT supports a high-speed scratch storage system andlong-term tape storage for researchers at all eight of the university’sstatewide campuses.

�e Research Technology Statistics (RT-STATS) project was be-gun with a similar goal as XDMoD: accounting, statistical andgraphical analysis, and visualization of the usage and users of theuniversity’s supercomputers via a web browser. Given the sizeof the university, however, a plethora of other data sources wereavailable for analysis. For example, when the purchase of Big RedII was proposed, IUPTI self-set a goal of having Big Red II used byadherents of at least 150 disciplines and sub-disciplines practicedat IU [12]. To determine the success or failure of this metric, userswere asked to to self-select up to three disciplines at the time ofaccount creation [13], and this information was incorporated intothe RT-STATS project. From this information, administrator areable to view the relationships of discipline information to CPUusage, job times, and various other statistics. Similarly, utilizationof long-term storage, environment module usage and other datasources were made available to users of the application.

Among the many sources incorporated were grants. IndianaUniversity uses the Kuali Financial Services package [2] to trackgrants awarded to university researchers from the NSF, the NIH

Value Analytics: A Financial Module for the Open XDMoD Project PEARC17, July 2017, New Orleans, Louisiana USA

Figure 2: Example data available through Indiana Univer-sity’s RT-STATS project. Grant income to Principal Investi-gators with accounts on Indiana University’s Big Red II clus-ter. A signi�cant percentage of the grants were awarded tousers in the College of Arts and Sciences (COAS).

and other sources. By incorporating this information into the RT-STATS system, IUPTI was able to demonstrate useful informationrelating grant dollars raised by university researchers to their usageof HPC resources, by developing a set of tools that links �nancialinformation – such as grant awards to IU faculty members – withusage of the supercomputers and HPC systems. Figure 2 showsan example: many users of Big Red II belong to the College ofArts and Sciences, and the �gure shows grant dollars awardedto those users compared to the overall grant dollars awarded tousers. �us, the system facilitates quanti�cation of �nancial incometo the university in the form of grants and contracts, and cross-referencing and comparisons with usage of HPC and other researchcyberinfrastructure systems.

4 XDMOD - VALUE ANALYTICS MODULETo bring the capabilities of the RT-STATS �nancial analysis toolsto the wider community, Indiana University and �e University atBu�alo are collaborating to develop novel modules to be added tothe existing Open XDMoD application. �ese modules are intendedto provide a starting point for assessing the value of investmentin campus-based CI both in scienti�c terms (number of publica-tions) and in �nancial terms (grant income from researchers whouse campus CI). �e Value Analytics Module (XDMoD-VA) [5] willallow cyberinfrastructure centers and IT organizations to begin theprocess of quantifying the scienti�c and �nancial value of invest-ments in HPC systems and supercomputers. By presenting a viewof the institutional �nancial, collaboration, and publication dataalongside the current HPC usage analysis in XDMoD, the moduleprovides users with invaluable insight into the pros and cons ofinvesting in cyberinfrastructure.

XDMoD-VA is being developed so that it can be implementedwith or without direct connections to a university or college’s local

Figure 3: Screenshot of an XDMoD instance with the ValueAnalytics module enabled. �e view shows Indiana Univer-sity funding overlaid with HPC usage pro�les in calendaryear 2016.

�nancial systems, providing �exibility for data acquisition. Where itis permi�ed by policy and practice, XDMoD-VAwill enable analysisof grants and contracts received by a particular institution (Figure3). If it is not possible for a college or university’s IT organizationto have direct read access from the institution’s �nancial systems,XDMoD-VA will provide the capability to download award data di-rectly from NSF and NIH web services. Using this grant data, usersmay perform an analysis of institutional HPC data cross-referencedto the data from those funding agencies. It may be relatively com-mon that institutional policies restrict access to internal �nancialmanagement systems, and this capability will enable institutionsto work with data from these major funding agencies, which arein many cases the most signi�cant sources of grant income for anacademic institution.

Many institutions that provide HPC resources have their usersrequest an allocation of computing time and/or storage space. Inthis case, it becomes easier to associate a particular resource usagewith a researcher’s lab or center. However, in 1998, a strategic planfor information technology developed at Indiana University stated,

Advances in computing and communication havecreated increased demands for data storage andmanagement. And underpinning all of this is theneed to provide researchers with good so�waretools and good support services.

In support of research, UITS should providebroad support for basic collaboration technolo-gies and…should provide advanced data storageand management services to researchers. �e Uni-versity should continue its commitment to highperformance computing and computation [1].

To further this goal, the decision was made to operate under aprincipal of abundance, meaning that resources are available to

PEARC17, July 2017, New Orleans, Louisiana USA T. Furlani et al.

all users who are granted accounts. Actual usage of the systemis granted on a ”fair share” basis. While this scheme is popularwith researchers, it makes it more di�cult to associate a particularproject with a particular usage of resources. In this case, XDMoD-VA’s ability to import group information becomes paramount: ifthe system can associate a user with a particular lab or center, itcan reasonably infer that the resources the user consumed will beassociated with that lab or center. Strategies will be put into placeto address cases where a user may be a member of multiple labs orcenters.

4.1 Ingestion StrategiesUniversity �nancial systems are as varied and unique as the uni-versities themselves. While for the most accurate data it is recom-mended that administrators use data available from institutional�nancial systems, it is possible to work from NSF and NIH grantdata that is publicly available. In either case, the data must betransformed into a format that is usable by XDMoD. Our intent isto provide standardized scripts to directly ingest data from major�nancial systems such as Peopleso�, Ellucian, and Kuali; however,even users of these systems may want to customize the data theywish to make available to Open XDMoD. �ese customizationspotentially make the use of a standard script di�cult.

To facilitate the loading of data into Open XDMoD, we havede�ned schemas for several JSON [3] documents that specify thestructure of the data that XDMoD is designed to handle: one forgrant information and one for person information for people associ-ated with a grant. XDMoD-VA scripts will output data conformingto the schema, and any custom modi�cations administrators makemay be validated against the schema to verify that data still can beingested successfully. For each grant, the schema includes �eldsfor starting and ending dates, dollar amounts, the funding agency,and grant identi�ers both for the organization and the agency. Ad-ditionally for each grant, the PI, Co-PI, and any key personnelare identi�ed based on an ID code and organization. Standardizedscripts will be available to users to translate data from their �nancialsystem into JSON matching the schemas. A forthcoming schemawill be made available for publication data, but our experience isthat there is li�le commonality between institutional systems usedto internally track publication data, and there would be li�le ad-vantage in a�empting to provide a standard script. For this reasonwe anticipate that ingestion from NSF/NIH data sources will besu�cient in the majority of instances.

However, disambiguation of authorship may become an issue.As no authoritative identi�er for researchers exists connectingtheir publications, grants, and other accomplishments, integratingthese data may be di�cult [10]. It is not always straightforward todiscern a single author with potentially multiple a�ribution styleson multiple papers. To partially overcome this issue, we intendto support cross-referencing author information from the ORCIDproject [9], and in the future may support other disambiguationstrategies.

At Indiana University, all faculty, sta�, students and a�liatesare assigned a unique identi�er over all eight campuses which isused as the login name on any HPC systems, ID’s related to anygrants, and other systems. �is uni�ed namespace scheme greatly

simpli�es the task of disambiguation, but it may not be the case atall universities. In many cases, even the internal systems of a singleinstitution may not have the ability to identify an HPC system user,researcher, grant awardee, or professor as a single individual. Inthis case additional work may need to be done to properly correlatean HPC user to a grant recipient. While in the end disambiguationmust be le� to the institution, the XDMoD-VA person informationschema supports an arbitrary number of organizations and ID’sfor each individual in the hopes of simplifying the task of cross-referencing grant information to user information. �e personinformation schema also includes title and department, as well asan association with any number of arbitrary groupings, for examplelabs or centers.

4.2 Extensions to the XDMoD ToolsetWhat a particular university or institution may �nd relevant isdependent on the objectives and goals of that institution. SinceOpen XDMoD supports the creation of arbitrary reports and charts,almost any data that is available in the system may be shown asa correlation. With the base Open XDMoD package, a numberof metrics are available on HPC jobs: core utilization percentage,number of jobs utilizing multiple cores, total jobs, wait time, andothers. �e Value Analytics package makes additional metricsavailable based on �nancial and publication records for a given timeperiod: total grants and total grant dollars, number of publications,etc. (Figure 4).

Items that might be of interest to users of the Value Analyticsmodule include: (1) Grant dollars and users; (2) Sources and totalamount of income; (3) Researchers and/or departments that areparticularly e�ective at securing grants; and (4) grant money thatis currently active and grants that will be soon be ending. Forexample, a user may want to view all grants brought in by usersof a cluster for the previous �nancial year. A new feature will bedeveloped for Open XDMoD that allows this to be expressed as aline graph, showing information for each month, or as a pie chart,spli�ing the users by department. Other options are also available.

One open question for each institution will be: should all grantdata be imported into the XDMoD system? Or should it only consistof users of the HPC resources? �is will have to be decided by eachinstitution dependent upon the relevant policies of the institution’s�nancial o�ce or o�ces. However, at Indiana University we havefound it useful to compare the grant data of users of HPC systemsto non-users - not necessarily in absolute terms, but to show whatpercentage of grant dollars have come from users of the systems.

4.3 PublicationsMany might argue that, rather than an analysis of external fundingto demonstrate ROI, a more appropriate measure might be in termsof publications that researchers are able to create. �us, leaders oflocal CI facilities need to be able to document the scholarly out-puts of users of local CI facilities and the collaborative relationshipsamong faculty users of such facilities, as individuals and as academicunits. With the support of the NSF, Indiana University’s Center forNetwork Science (CNS) has implemented a pair of custom tools:the Network Workbench Tool (NWB) for the study of large scale

Value Analytics: A Financial Module for the Open XDMoD Project PEARC17, July 2017, New Orleans, Louisiana USA

Figure 4: Metrics available to an Open XDMoD instance run-ning with the Value Analytics module enabled.

networks and the Science of Science Tool (Sci2) optimized for sci-ence of science research and practice [8]. �ese tools enable novelalgorithms and data sets to be combined in a plug-and-play fashion.NWB facilitates network analysis of author relationships and col-laborations, and how science accomplishments relate to the fundingand the evolution of scienti�c communities. In a strategic planninge�ort to be�er understand how local facilities enable and facilitatecollaboration among units and their faculty and sta�, NWB wasutilized to create Figure 5, showing money �ows between unitsthat use IU’s advanced CI. Similar examples will be integrated intoXDMoD-VA, utilizing the tools and publication data made availableby the NSF and other agencies, allowing administrators to leverageand cross-reference publication data, grant data, and the alreadyexisting HPC data made available by XDMoD.

4.4 Visualizations�e CNS is a leader in the generation of scholarly visualizationssuch as science maps, which are generated through a scienti�c anal-ysis of large-scale scholarly datasets in an e�ort to extract, connect,and make sense of the bits and pieces of knowledge they contain [7].�e tools that they have made available for network analysis haveproved invaluable in understanding the structure and dynamicsof science, and facilitate network analysis of such topics as what

Figure 5: An IndianaUniversity analysis performedwith theNetwork Workbench Tool demonstrates a layout of collab-orative relationships. �e analysis shows money �ows be-tween units that use IU�s advanced cyberinfrastructure.

in�uence relationships are, who collaborates with whom, and howscience accomplishments relate to the funding and the evolutionof scienti�c communities. Similar issues arise in a�empting todemonstrate the relationships between funding, publications, andcyberinfrastructure, which are complex and do not lend themselveseasily to simple charts. XDMoD-VA will leverage the experienceof the CNS center by providing innovative visualizations demon-strating the relationships between these multivariate dimensions.Figure 6 shows a Sankey diagram relating the usage of IndianaUniversity cyberinfrastructure systems to funding agencies andpublications.

5 CONCLUSIONS�e Open XDMoD - Value Analytics module will allow users toquantify the Return on Investment (ROI) of their investment incyberinfrastructure resources. It will do this by allowing adminis-trators to import �nancial and publication data into the same toolsthat are used to analyze statistical information that can be gleanedfrom the cyberinfrastructure itself. Once the �nancial informa-tion and HPC information are correlated, administrators shouldmore easily be able to create projections, anticipate objections, andhave raw data allowing them to make more informed decisions onthe ROI of their cyberinfrastructure investment. �e modules areanticipated to be available for general usage in October of 2017.

6 ACKNOWLEDGMENTS�is material is based, in part, upon work supported by the NationalScience Foundation under Grant No. 1053575 and 1566393 and also

PEARC17, July 2017, New Orleans, Louisiana USA T. Furlani et al.

Figure 6: Sample visualization to be made available inXDMoD-VA: Display of a multivariate analysis of the rela-tionship between IT resources, funding agencies, and pub-lications. �e width of each line represents grant dollarsawarded to researchers.

by the Indiana University Pervasive Technology Institute (foundedwith funding from Indiana University and the Lilly Endowment,Inc.). Any opinions, �ndings, and conclusions or recommendationsexpressed in this material are those of the authors and do notnecessarily re�ect the views of the National Science Foundation,Indiana University, or the Lilly Endowment. We would like to thankour vendor partners Cray and DDN for their collaboration in theimplementation of Big Red II.

REFERENCES[1] 1998. Indiana University Information Technology Strategic Plan: Architecture

for the 21st Century. h�p://hdl.handle.net/2022/471. (May 1998). Accessed:2017-03-08.

[2] 2005. Kuali Financial Services. h�p://www.kuali.org/. (2005). Accessed: 2017-03-02.

[3] 2009. JSON Schema: A Media Type for Describing JSON Documents. h�p://json-schema.org/latest/json-schema-core.html. (2009). Accessed: 2017-02-24.

[4] 2010. XDMoD (XDMetrics on Demand). h�ps://github.com/ubccr/xdmod. (2010).Accessed: 2017-03-02.

[5] 2016. XDMoD (XD Metrics on Demand) Value Analytics Module. h�p://indiana.edu/∼xdmodva/. (2016). Accessed: 2017-03-02.

[6] A. W. Apon, L. B. Ngo, M. E. Payne, and P. W. Wilson. 2015. Assessing thee�ect of high performance computing capabilities on academic research output.Empirical Economics 48, 1 (2015), 283–312.

[7] Katy Borner, Richard Klavans, Michael Patek, Angela M. Zoss, Joseph R. Bibers-tine, Robert P. Light, Vincent Lariviere, and Kevin W. Boyack. 2012. Design andUpdate of a Classi�cation System. PLoS ONE 7 (2012).

[8] K. Borner, N. Ma, J.R. Biberstine, R.M. Wagner, R. Berhane, and H. Jiang. 2010.Introducing the science of science (Sci2) tool to the reporting branch, o�ceof extramural research/o�ce of the director, national institutes of health.. InWorkshop on the Science of Science Measurement.

[9] Editorial. 2009. Credit where credit is due. Nature 462 (2009).[10] Stephen B. Johnson, Michael E. Bales, Daniel Dine, Suzanne Bakken, Paul J.

Albert, and Chunhua Weng. 2014. Automatic generation of investigator bibli-ographies for institutional research networking systems. Journal of BiomedicalInformatics 51 (October 2014), 8–14.

[11] JT Palmer, SM Gallo, TR Furlani, MD Jones, DeLeon RL, JP White, N Simakov,AK Patra, J Sperhac, T Yearke, R Rathsam, M Innus, CD Cornelius, JC Browne,WL Barth, and RT Evans. 2015. Open XDMoD: A Tool for the ComprehensiveManagement of High-Performance Computing Resources. Computing in Science& Engineering 17 (July/August 2015). DOI:h�p://dx.doi.org/10.1109/MCSE.2015.68

[12] Craig A. Stewart, Beth Plale, Von Welch, Ma�hew R. Link, �erese Miller, Eric A.Wernert, Michael J. Boyles, Ben Fulton, David Y. Hancock, Robert Henschel,Sco� A. Michael, Marlon Pierce, Robert J. Ping, Tassie Gniady, Geo�rey C.Fox, and Gary Miksik. 2015. Pervasive Technology Institute Annual Report:Research Innovations and Advanced Cyberinfrastructure Services in Support ofIU Strategic Goals During FY 2015. (2015). h�p://hdl.handle.net/2022/20566

[13] Abhinav�ota, Ben Fulton, Le Mai Weakley, Robert Henschel, David Y. Hancock,Ma�hew Allen, Jene� Tillotson, Ma�hew Link, and Craig A. Stewart. 2016. APetaFLOPS Supercomputer as a Campus Resource: Innovation, Impact, andModels for Locally-Owned High Performance Computing at Research Collegesand Universities. In SIGUCCS 16,. ACM.