IEEE Transactions on Nuclear Science, Vol. NS-32, No. 4, August 1985

A FLEXIBLE HIGH SPEED DATA ACQUISITION SYSTEMFOR REAL-TIME MEASUREMENTS AND DATA ANALYSIS

Rodney L. Taylor

Interfield Research Associates607 Georgina Ave.

Santa Monica, California 90402

AbstractInterfield Research Associates (IRA), in

cooperation with the California Institute ofTechnology (Caltech) Aeronautics Department,,and the University of California, Los Angeles(UCLA) Plasma Physics Department hasdeveloped a flexible, high throughput dataacquisition system for use with basicscientific experiments. The data acquisitionsystem utilizes a high speed computernetwork, developed at Caltech, and aninteractive data acquisition software system.developed by Interfield Research Associates.The system is powerful enouigh to handle thecomplex measurements required by resear-chprojects at Caltech and UCLA; at the sametime it is simple enough to enable a personwith little computer experience to performsophisticated experiments.

Design Consideration for Computer System

Caltech and UCLA had the same basic dataacquisition and analysis needs although thedesign considerations for each systemdiffered widely. The AeronauticalEngineering Department at Caltech needed acomputer system that could respond quickly toa large number of interrupt-driven events andconcurrently display and analyze thecollected data. The UCLA Plasma PhysicsDepartment, on the other hand, did not needthe capability to respond to frequentinterrupt events but needed the ability tomove, analyze, and store large data buffersquickly at less frequtent rates.

In addition, the nuimber of users on thecomputer systems at UCLA and Caltech wascompletely different. UCLA had a VAX-ll/750with a high speed array processor that wasfully dedicated to their plasma physicsexperiment so that the number of users wasminimal. Caltech's situation was theopposite. The experiment had to share aVAX-11/750 with a large group of users doinga variety of computing tasks at the same timethe experiment was being conducted.

The common needs of both research groupswere for a data acquisition system capable ofhandling high data throughput while beingsimple enough for a person with limitedcomputer experience to use. The maindifference between the two laboratories was

the amouint of load that could be placed on

the VAX. Both laboratories' needs were

completely satisfied by the same networkeddata acquisition system.

The network uses Interlan iEthernethardware, which can support over 100computers per 500 meter cable segment, andmultiple cable segments can be linkedtogether if needed. rhe principle computerin the Caltech and UCLA systems is a

VAX-1l/750. UCLA has a CSPI MAP 200 arrayprocessor on the VAX which can be used forreal-time data analysis. UJCLA also has anLSI-1l/23 on their network which controls twoCAMAC crates with memory-mapped controllers.Although Caltech has many computers on theirnetwork an LSI-1l/2 with a programmablereal--time counter/timer was used for thesystem described in this paper.

The network's software structure isideally sLlited to distribute computing tasksthat need to share a common data base, so asneeds grow, data acquisition computers can beadded to perform more sophisticatedmeasurements or share in the real-time dataanalysis tasks.

The data acquisition software, which wasdeveloped by Interfield Research Associates,is a highly interactive menu-driven system.Flexibility and speed are maximized bydistributing the process control, datacollections and data analysis tasks among thesatellite computers and several processesrunning concurrently on the VAX.Modularization of the software insures thatif a measurement requires a specializedcomputer code, which is not in the menusystem, the software can be written rapidlyand easily integrated.

Also incorporated in the software is adata structure which supports currentarchiving needs and anticipates future ones.The data structure allows all relevantinformation associated with a data runsregardless of size or format, to be recordedon a single sequential device (e.g. magnetictape). An interactive program was developedthat allows the user to easily access thedata structure. The user can directory thesequential data structures select single ormultiple data records for plotting, or builddata records into a data structure that canbe mapped into a shared memory region on theVAX for further analysis. Also, an extensivelibrary of mathematical and graphicalroutines was obtained from Caltech which aidthe experimenter in the analysis of data.

The software system4 described in moredetail belows was developed and fullyimplemented at UCLA in a period of two monthsand at Caltech in a period of two weeks bythe author and Dr. Jonathan Melvin fromCal tech and IRA. The structure of thenetwork hardware and software used, as wellas the versatility of the VAX hardware andVMS operating systems helped make completionof these complex tasks possible in such a

short time frame.

Brief Description of the Caltech Network

A unique feature of the network softwareis that it sutpports disk emuilation on thesatellite computers C1,2,53. The satellites,or smaller computers, can run the standardDEC RT-i1 disk operating system but do not

0018-9499/85/0008-1436$01.00 0 1985 IEEE

1436

1437

require disks of their own. This reducesappreciably the cost of satellite computersand consolidates all the storage of data andprograms on the central VAX computer disks.

Files and data are transferred by thesatellite to regions of shared memory, calledglobal sections, on the VAX. The data in aglobal section can be mapped to commonstorage areas, such as Fortran common blocks,in multiple programs running on the VAX.This important capability means that the dataacquired with one processor is concurrentlyavailable for analysis, display, andarchiving by another processor. The networksoftware is designed so that the RT-11operating system on the smaller computersappears as though it has several disks or"pseudodisks" attached to it. When a file ordata is written to one of the pseudodisks itappears in the global section associated withthat particular pseudodisk. The networksoftware allows the dynamic mapping andunmapping of pseudodisks to various globalsections.

The actual rate of transfer through thenetwork (disk emulation throughput) rangesfrom 125 to 170 Kbytes per second, dependingon the amount of disk activity on the VAXassociated with the transfer (2,33. The dataacquisition software is written to optimizethe transfer rate of the network bylocalizing the data buffer in a continuousregion of one pseudodisk, thereby minimizingdisk activity on the VAX.

All standard RT-11 software will rununmodified on the diskless satellitecomputers because of the network emulation ofdisk input/output. Most of the routines onthe data collection satellites are written inFortran, and standard Fortran or RT-11 systemlibrary input/output can be used. Inaddition, a program on one of the satellitescan be edited from the terminal connected tothe satellite or from any of the terminalsconnected to the VAX.

Maximtum flexibility is achieved bydistributing the process control and datacollection tasks over the LSI and severalprograms running concurrently on the VAX.

Description of the Interactive DataCol lection and Archiving

Software Systems

The overall structure of the softwaresystem used at both Caltech and UCL-A is thesame. However, to accommodate the differentsystem configurations in the two laboratoriesthe detailed methods used to collect,analyze, display, and archiive the data are

different at each site.

The Caltech Software

Overview. At Caltech the VAX could notbe dedicated to the experiment; the generaluser population would not tolerate a large,high priority job rurnning on the VAX for longperiods of time. However, the computingneeds for the Caltech experiments were very

difficult to meet, with the LSI-11/2 which was

available for data collection. In order tomeet the compu,ting needs but keep the load on

the VAX to a minimum, the data collec tion anddisplay were handled by the LSI-1l; only thecompute intensive mathematical analyses were

performed on the VAX. The network providedsufficiently fast data transfer between thetwo computers that the users were unawarethey were using two machines.

The experiments being conducted atCaltech involve the measurement of thevelocity of suspended particles in a fluidflow. The rate at which a particle passesthroLugh the fringes created by twointersecting laser beams is measured and theparticles velocity is calculated. As aparticle passes through the fringe area itsinterference frequency is counted for a fixedtime interval using a programmablecounter/timer on the LSI-11. Theexperimenter can define the the countinginterval and the number of counting intervalsthat will reside in each data set for a givenposition in the fluid stream. An entire dataset consists of many spacial positions thatmap the fluid velocity in two or threedimensional space.

Technical description. A menu-drivensoftware system allows the experimenter toset several user-defined variables whichaffect the operation of the programmablecounter/timer on the LSI. The variables arethen passed to a Macro coded interruptservice routine (ISR) through a Fortrancommon block. Interrupts are enabled whenthe user begins data acquisition. Each timethe specified time interval elapses, aninterrupt is generated by the counter/timerboard. The interrupt service routine readsout the number of counts from thecounter/timer board and stores the result ina ring buffer, which is also located in theFortran common block. The timer is thenreset and counting begins again until thespecified number of counting intervals iscompleted. The main Fortran program watchesthe ring btuffer, and each time a new value isadded it is dynamically displayed on theuser's graphics terminal.

The experimenter at any time can exitthe data collection and display mode andenter the data analysis mode while datacollection continUes. In the data analysismode commands are passed to the VAX through apseudodisk write [2,3]. When the VAXreceives a command, it executes the codeassociated with the command and on completionsignals the l SI-11 through a message c:hannelimplemented in the network software. Withthe software strtuctutred in this way the VAXis executing high priority code for only ashort time on an occasional basis, therebyminimizinig the disturbance to other VAXusers.

The UCLA Software

Overview. The central computer at UCLAis a VAX-11/750 which has a CSPI MAP-200high-speed array processor on its Unibus.Since the Vax is dedicated to the plasmaphysics experiment there are very few userson the system. UCL-A has an LSI-11/23 withtwn memory mapped CAMAC crates attached and avariety of medium-resolution graphicsdevices. Data is primarily collected byseveral high speed CAMAC transient recordersand passed as large data btiffers by theLSI-11 to the VAX over the networ-k. TwentyMHz and 100 Mhz transient recorders are ulsedto collect signals from the diagnostic probes

1438

in a plasma for several milliseconds during aplasma pulse. The plasma is pulsed every 1.5seconds, and a typical data collection runcan last for several days. The size of asingle data set can be from 40 to 100 Mbytes.

In contrast to the Caltech system theUCLA VAX is used to control the entire datacollection, analysis, and archiving process.A supervising program runs on the VAX thatcoordinates the data collection and archivingroutines and directly controls the satellite.The supervising task is a menu-driven systemused to set up the experiment and selectwhich CAMAC devices are active for aparticular measurement. The code on theLSI-11 is not interrupt-driven and is writtenentirely in Fortran. This software structureputs a heavy load on the VAX and very littleload on the LSI-11; however, it allo'ws for amuch more sophisticated menu system thancould be fit on the small LSI-11 in theCaltech system.

Techn ical descri.pti on. The experimentis supervised by a program called CONFIG.CONFI3 includes a menu-driven software systemwhich allows all of the parameters associatedwith a particular measurement to be specifiedby a user on the VAX. The VAX code builds anarray in shared memory which is mapped to anLSI pseudodisk, and stores in it thespecified parameters. The LSI uses thisarray (called the Device Control Block orDCB) to determine which CAMAC devices are tobe used for data acquisition during aparticular plasma pulse (also called a shot).This method allows the user to dynamicallyalter the experiment and to display theresults each time there is a plasma shot.CONFIG communicates with two other tasksrunning concurrently on the VAX via asystem-wide mailbox. Each time the plasma ispulsed, the data collection task is awakenedby a mailbox message from CONFIG and proceedsto read the DCB to determine which CAMACdevices are active in order to retrieves thenew data. The data collection task thencalls a user-supplied data analysis task thatperforms the actual data analysis. The dataanalysis may consist of anything from asimple averaging over many plasma shots to a

smoothing of the data, the finding of itsderivative and the fitting of the derivativeto a known function. When the data analysisis completed, the new data is returned to a

different section of the VAX data buffer sothat the raw data will be preserved.

The data is archived by a separate VAXprocess. After the data has been analyzed,CONFIG signals the data archiving task, whichwrites the analyzed data -- and the raw dataif so directed -- to the VAX disk. Aspecialized data structure was developed tohandle the large data sets associated witheach measurement. The data structureincludes a small direct access file (indexfile) containing pointers into a largesequential file containing the data (datafile). The experimenter can use the smallindex file to locate data records of interestand extract them for later analysis.

All of the VAX code is written entirelyin Fortran and can be easily modified ifnecessary. Generally,, the only part of theVAX code that is commonly modified is theuser supplied data analysis code. The dataanalysis code is written as a Fortran

subroutine and compiled and linked to a fixeddata collection program before the system isrun. This software structure allows forsimple maintenance of a large, powerfulsoftware system that is flexible enough toaccommodate a wide variety of measurementsand yet is simple enough for anyone to use.

The LSI-1 software is also entirelywritten in Fortran and is notinterrupt-driven. The LSI-11 simply waitsfor an instruction from the VAX, reads theDCB, sets up the selected CAMAC devices, andloops on the LAM present bit in the CAMACcrate controller. Since all of the dataassociated with a plasma shot is availablewhen any LAM is set, and since shots occurapproximately every second, this method iscompletely suitable. Once a LAM is present,the LSI-11 moves the data from the transientrecorder buffers into a pseudodisk file,moves the probe-drive stepping motor, andsignals the VAX that it has completed itsinstructions.

Conclusion

The di-fferent needs of laboratories atCaltech and UCLA were completely met with asingle software structure based on theCaltech Physics/Engineering network and IRAexperiment control software. At Caltech thedata acquisition, analysis, and display weremostly carried out by the satellite computer,and a networked VAX was used only forcompute-intensive data transformations. AtUCLA, on the other hand, a dedicated VAX was

used for experiment set up, control, dataanalysis, and data archiving. A networkedLSI-11 was used to control CAMAC dataacquisition interfaces and to move data intothe VAX.

Both data acquisition systems are basedon a networked modular hardwareconfiguration; both systems have a modular,easy-to-maintain software structure; bothare flexible and can be expanded to meetfuture needs.

References

1. J.D. Melvin, "The CaltechPhysics/Engineering Network," IEEE Trans.on Nucl. Sci., in same issue as thisarticle. (1985)

2. J.D. Melvin, M.H. Mendenhall, T.A.Tombrello, and D.L. Clark, "NetworkedVAX/LSI/CAMAC Data Acquisition SystemDevelopment,." IEEE Trans. on Nucl.Sci.,L NS-30 (1983) 3816.

3. J.D. Melvin,, M.H. Mendenhall, R.McKeown, and T.A. Tombrello,"VAX/LSI- 1/CAMAC Nuclear DataAcquisition System Under Development atthe W.K. Kellogg Radiation Laboratory,Caltech," IEEE Trans. on Nucl. Sci4NS-28 (1981) 3739.

4. R.L. Taylor, "A Flexible High Speed DataAcquisition System for PlasmaMeasurements," presented at theTwenty-Sixth Annual Meeting of The APS,Division of Plasma Physics, Boston, MA.October 29 to November 2, 1984