Upload
bethany-perry
View
216
Download
1
Tags:
Embed Size (px)
Citation preview
Fermilab Control System(“ACNET”)
J. PatrickFebruary 17, 2005
SMTF Review 2/17/05 J. Patrick 2
Fermilab Control System
Overview Applications Central Services Front-ends/Instrumentation Timing Security and Remote Access Summary
SMTF Review 2/17/05 J. Patrick 3
Fermilab Control System aka “ACNET” Unified control system for the entire complex
400 MeV Linac 8 GeV Booster Synchrotron 120 GeV Main Injector Synchrotron 1 TeV Tevatron Synchrotron antiproton source – target/debuncher/accumulator antiproton “Recycler” storage ring fixed target lines
Simultaneous operation of Tevatron proton-antiproton collider (storage ring) Antiproton production and storage (~0.5 Hz) 120 GeV fixed target to Meson lab (~0.1 Hz) 120 GeV fixed target to NUMI/MINOS (~0.5 Hz) 8 GeV fixed target to MiniBoone (~5 Hz)
SMTF Review 2/17/05 J. Patrick 4
Fermilab Accelerator Complex
SMTF Review 2/17/05 J. Patrick 5
Control System Overview
ConsoleApplications
CentralServices
IRMFront-Ends
JavaApplications
WebApplications
MOOCFront-Ends
LabviewFront-Ends
Open AccessClients
Field Hardware
ethernet
Database
CAMAC, VME, PMC, IP, Multibus, CIA, GPIB, …
field bus: VME, SLD, Arcnet, ethernet, …
Servlets Consolidators
Application
Central
Front-End
SMTF Review 2/17/05 J. Patrick 6
History Originally developed for Tevatron in early 80’s Substantial evolution over the years CAMAC field hardware -> very broad diversity PDP-11/Mac-16 front-ends
-> x86 Multibus-> VME based processors + Labview
DEC PCL network-> IEEE 802.5 token ring-> ethernet
VAX database -> Sybase VAX central services -> Java PDP-11 applications -> VAXstations
-> Java application infrastructure-> Port of VAX code to Linux in progress
SMTF Review 2/17/05 J. Patrick 7
System Scale
200,000 devices/350,000 properties 475 front-ends 90 Sun Netras + 12 Linux PCs that run Java based data
loggers, servlets, open access clients + VAXes to run VAX central services
70 VAXes that run high console level applications + 70 Linux PCs to to parallel VAXes until VAX retirement. X-displayed out to desktop PCs
SMTF Review 2/17/05 J. Patrick 8
Device Model <= 8 character devices names
e.g. T:BEAM Informal conventions provide some rationality but no enforcement More verbose hierarchy layer created, but not utilized/maintained
Each device has one or more properties: Reading, setting, analog & digital alarms, digital status & control
Device descriptions stored in Sybase database Scaling transformations defined in device database Add new devices with “DABBEL” command language on VAX
Templates available for various device types View/modify device definitions with GUI Monotype data, arrays are transparently supported Structured data is often done, though support is not transparent
Application (library) layer must properly interpret the data from the front-end and perform required data translations
SMTF Review 2/17/05 J. Patrick 9
Device Database GUI
SMTF Review 2/17/05 J. Patrick 10
Communication
Applications/Central Services/Front-ends communicate via “ACNET” protocol
Custom protocol now exclusively based on UDP Extensive network/switches etc… Asynchronous request/reply; request multiple reply Timeout/retry mitigates UDP unreliability. “RETDAT/SETDAT” protocol for readings/settings
Can have multiple devices/packet
Fast Time Plot (FTP) returns blocks of readings taken at up to 1440 Hz every 0.5 seconds
Snapshot protocol returns blocks of 2048 points taken at an arbitrarily high rate
SMTF Review 2/17/05 J. Patrick 11
Communication Application communication with front-ends passes
through central layer Datapool Manager on VMS via shared memory Data Acquisition/Server Engine (DAE) in Java (via RMI) Consolidation of requests to avoid overloading front-ends Centralizes security, monitoring, error logging
Issues include: Limited packet size and internal device address space. Timestamps not part of protocol packet, applied on data
receiptGPS sync clock event sent to all front-ends, can be included in
structured devices New get/set32 protocol addresses the above, only on centrals
now Lack of transparent support for structured data.
Bridging to network devices that don’t directly support ACNET is via front-ends or Open Access Clients
SMTF Review 2/17/05 J. Patrick 12
Database
Central Sybase database Three parts:
Device and node database Application database
Used by applications as they wish Save/Restore/Shot Data database
MySQL used for data loggers, and for some other specialized purposes. Not for general application use
SMTF Review 2/17/05 J. Patrick 13
Application Environment Two major high level application environments Console environment
VAX based C language (+ legacy FORTRAN) Port to Linux in progress
Expect some operational utilization in the fall
Java environment Platform independent Windows/Sun/Linux/Mac all supported and utilized Full access to entire control system
Both environments have a standard application framework that provides a common look and feel
All applications launched from “Index Pages” Currently separate pages for Console, Java environments
Also some browser applications of varying styles
SMTF Review 2/17/05 J. Patrick 14
Index Page (Console)
SMTF Review 2/17/05 J. Patrick 15
Application Index (Java)
Web Browser
Standalone Application
SMTF Review 2/17/05 J. Patrick 16
Application Overview Utility applications
Parameter Page Fast Time Plot Data Logger viewer Database Viewer Synoptic displays Diagnostics Save/Restore Alarms …
Machine Specific Applications ~600 Primary + 125 Secondary applications in console
environment >100 applications in Java environment
Mostly utility, shot data analysis
SMTF Review 2/17/05 J. Patrick 17
Parameter Page (Console)
SMTF Review 2/17/05 J. Patrick 18
Parameter Page (Java)
SMTF Review 2/17/05 J. Patrick 19
Fast Time Plot (Java)
SMTF Review 2/17/05 J. Patrick 20
Synoptic Display (Console) Graphical Builder
“Lex Draw”
SMTF Review 2/17/05 J. Patrick 21
Synoptic Builder (Java)
SMTF Review 2/17/05 J. Patrick 22
Synoptic Display (Web)
SMTF Review 2/17/05 J. Patrick 23
Sequencer (Console) Automaticall
y Sequence Collider Shots, other activities
Relies on ACL scripting language
SMTF Review 2/17/05 J. Patrick 24
Finite State Machine (Java) Graphical Builder Primary current usage is
Pelletron control Keeps track of HV sparks and
beam related trips
SMTF Review 2/17/05 J. Patrick 25
Sequenced Data Acquisition Automated acquisition of data for collider shots Configure readings for various stages of collider
shot Snapshots/FTP as well as single scalar readings SDAEdit tool makes it easy to add new things
Data acquired automatically and stored in Sybase database
Summary html reports generated when store goes in Step efficiencies, error reports, luminosity
“Supertable” captures ~200 most popular items Html, Excel, Java Analysis Studio formats
automatically generated Suite of tools to view detailed data Data can be directly read by Excel, Java
Analysis Studio, potentially other external tools Simple Java API for user written code
SMTF Review 2/17/05 J. Patrick 26
Alarms Display
SMTF Review 2/17/05 J. Patrick 27
Scripting
Accelerator Control Language (ACL) Console environment Access to much of the functionality in the control system VMS DCL style syntax Extensively used in Sequencer
Jython Java API in principle accessible to Jython Some but limited testing/use to date
SMTF Review 2/17/05 J. Patrick 28
External Tools
Excel Excel based parameter page Easy cut/paste of Java tables into Excel Excel data can be imported into a variety of tools
MathCAD, OptimA used locally
MATLAB MATLAB supposedly has a standard Java interface, in theory it
should be straightforward to integrate with the control system However we haven’t tried it An attempt to do this with Mathematica > 1 year ago was not
successful, it does not use a standard Java distribution
Java Analysis Studio Currently data logger and shot data plug-ins
SMTF Review 2/17/05 J. Patrick 29
Excel Parameter Page
SMTF Review 2/17/05 J. Patrick 30
Java Analysis Studio
SMTF Review 2/17/05 J. Patrick 31
Save/Restore
Devices may be assigned to save/restore lists
Console applications to save/restore to/from database on user request
Scheduled saves of the entire control system 4 times/day
SMTF Review 2/17/05 J. Patrick 32
Electronic Log Based on Oak Ridge
product Web browser client
Perl/cgi back-end Shift log + machine /
subsystem logs Used by A0 API to add entries
programmatically, including images Both Console and Java
environments Java application
framework includes standard menu item to make posts.
SMTF Review 2/17/05 J. Patrick 33
Central Layer
Open Access Clients (“virtual front-ends”) Java Servlets
Bridge requests from outside firewall
Request Consolidation Alarms server Front-end download Other things …
SMTF Review 2/17/05 J. Patrick 34
Open Access Clients Processes with no user interface, always running
Obey same communication protocol as front-ends Compare to EPICS “Soft IOC”
Several classes: Utility – Data loggers, scheduled data acquisition, virtual devices Calculations – Both database driven and custom Process control – Autotune of fixed target lines Bridges –to ethernet connected scopes, instrumentation, control
systems… Easy access to devices on multiple front-ends Friendlier programming environment than VxWorks front-ends
Framework transparently handles ACNET communication Access to database, other high level operating system features Do not provide hard real-time response
Clock events via ethernet multicast > 100; Almost all run in Java framework; VAX framework deprecated
SMTF Review 2/17/05 J. Patrick 35
Data Loggers ~70 instances running in parallel on central nodes
~80 GB compressed storage per node in MySQL database Circular buffer; wraparound time depends on device count/rates
Data at maximum 1 Hz rate extracted to “Backup” logger every 24 hours Backup logger is available online => data accessible forever
Generally allocated to various departments to configure as they wish No central management of archiving lists, so inevitable duplication
Tasks follow the Open Access Client architecture Simple GUI to specify logging lists
Single readings or snapshots; on clock events or at fixed rates to 15 Hz
Programmatic logging interface available for custom loggers Variety of retrieval tools available
Standard console applications, Console and Java environments Web applications Java Analysis Studio programmatic APIs
SMTF Review 2/17/05 J. Patrick 36
Front-Ends
Interface field hardware to the high level control system
Several architectures MOOC – Minimally Object-Oriented Communication
Very wide diversity of functionality Processors all VME (VXI) based
Very wide diversity of field hardware supportedLinks to many other form factors such as CAMAC, etc.MVME 162 (68k) and 2400 (Power PC)VxWorks Operating System
~275 in the system
SMTF Review 2/17/05 J. Patrick 37
Front-Ends - IRM IRM – Internet Rack Monitor MVME 162; PSOS operating
system Chassis includes 64 channel x
1 KHz 16 bit continuous digitizer 8 x 10-20 MHz option available
8 bytes external digital I/O 8 x 12 bit DAC outputs Expandable via IndustryPak I/O ~125 in the system Eight used at A0 photo-injector
Variety of functions A few at TTF for modulators
SMTF Review 2/17/05 J. Patrick 38
Front-Ends
Linac front-ends VME based Power PC processors Non-MOOC infrastructure; tied to 15 Hz Linac cycle rate ~50 in the system
HRM – Hotlink Rack Monitor Evolution of IRM architecture I/O chassis separate from processor chassis 64 x 10 KHz ADC; up to 2 per chassis 10-20 MHz snapshot board Timestamps included with ADC data 8 byte digital I/O per chassis Multiple chassis’ per PPC processor VxWorks (not MOOC) Initial build of 15 units in progress
SMTF Review 2/17/05 J. Patrick 39
Labview Front-Ends General control/acquistion
from high level controlsystem
Interact via ACNETprotocol written inLabview VI language No FTP/Snap
Non-local consoleaccess viaPCAnywhere,Timbuktu etc.
~25 in system Instrumentation:
Flying Wires Sample Bunch Display …
Both PCs & Macs
SMTF Review 2/17/05 J. Patrick 40
Field Hardware
Wide variety of hardware supported CAMAC, VME, VXI, Multibus, CIA, … Too extensive to describe here…
Support for wide variety of commercial instrumentation Scopes, spectrum analyzers, signal generators, DVMs, … Ethernet->GPIB; direct ethernet Through front-ends; through OACs; direct in some cases Mostly in console environment, ports to Java as needed
Siemens-Moore APACS PLC supervisor Used for cryogenic control Interface through Open Access Client using library from
company Other PLCs also supported through front-ends
SMTF Review 2/17/05 J. Patrick 41
Bridge to External Systems
Accelerator NETwork control system (AccelNET) is a software package designed for control of electrostatic particle accelerator systems.
Runs on PC hardware under the Linux operating system
Modular Linux programs that communicate to each other through a central database server using TCP/IP.
CAMAC- GROUP3 I/O ACNETAccelNET Bridge
SMTF Review 2/17/05 J. Patrick 42
Timing and Control Networks TCLK (Clock Events)
10 MHz Manchester encoded signal with 8 + 2 bits/event 256 possible events defined for the complex Decoding hardware/signal generators for variety of buses Standalone version of this system used at A0 photo-injector
MDAT (Machine Data) 10 Mbit/sec serial link; 8 bit type + 16 bits data per frame Originally developed to rapidly distribute ramp currents around the
main ring and Tevatron; usage has broadened Beam Sync – revolution markers derived from RF States
State devices defined Any node may issue transition request, gets multicast everywhere Can be distributed via MDAT
Beam permit/abort CAMAC interfaces….
SMTF Review 2/17/05 J. Patrick 43
Security
Control system network inside a firewall Restricted access from/to outside the accelerator division
network
Privilege classes assigned based on account, service, (node) Control from where applications may be run Control which programs/users/nodes may do settings
Emphasis on accountability as well as security Setting log Application log
SMTF Review 2/17/05 J. Patrick 44
Remote Access
Console environment via remote X display Administrative controls on usage, particularly settings etc.
Java applications can be launched from anywhere from a browser via Web Start Access to inner parts of control system only via VPN Applications to be run remotely w/o VPN must be designed for
thatApplications that work via servlets;“Secure Controls Framework” – Subset of data access API
implemented in this way, better performance than servlets/xml
Access from outside the standard application environments xml-rpc is preferred method Readings of any device; setting of “virtual” devices No setting of hardware associated devices Used by all current experiments
SMTF Review 2/17/05 J. Patrick 45
ACNET Summary
Should meet requirements for SMTF Has been/is being continually modernized
Extensive experience at FNAL. Less time to get a system running A broad range of people could contribute quickly
Limited experience outside of Fermilab Remotely accessible but full system not portable Documentation…