1

CCL: A Test Language for Automating Spacecraft Checkout Operations

U.N.Vasantha Kumari Head-SRCD/SCG

ISRO Satellite Centre Bangalore, India – 560017

unv@isac.gov.in or anaghaun@yahoo.com

Abstract—Every Complex Spacecraft undergoes rigorous testing of 3 to 6 months before being declared ready for launch. 12 The satellite integration and testing activities are an important milestone in the realization of any spacecraft. The time and effort required to completely test any spacecraft is enormous and any means of reducing the time and effort and still delivering a fully tested fault free spacecraft is a very challenging job. The only way to achieve this is by improving automation efforts in this phase of spacecraft building. The in-house designed and developed Checkout Command Language (CCL) helps in preparing the test schedules and test procedure files well in advance, reduces the manpower requirements and minimizes testing time. CCL is a powerful procedure language for complete automation of test environment. CCL is Simple, English like, easy to learn and provides all the necessary instruction required for testing a spacecraft. The 50 odd instructions provide means to send telecommands to spacecraft, check the telemetry, instructions for test equipment monitoring and Control. CCL also provides means to call another test procedure file. With the help of CCL procedures once written can be executed many times, thus ensuring repetivity of tests. These procedures written can be executed either in Automode or single step mode or interactive mode depending on the operator/test need. CCL compiler will check for the syntax of the test procedure before execution. An interpreter for CCL is developed which is used for executing the test procedures written using CCL. Interpreter has additional features like linking the files, proving statistical information about the tests which help in improving automation. This paper will present in detail the features of CCL and the feedback from the users and the improvements planned in future.

TABLE OF CONTENTS

1. INTRODUCTION ................................................................. 1 2. AUTOMATIC CHECKOUT SOFTWARE SYSTEM ................ 2 3. SPACECRAFT CHECKOUT SYSTEM .................................. 3 4. CHECKOUT COMMAND LANGUAGE ................................. 3 5. CLASSIFICATION OF INSTRUCTIONS IN CCL .................. 4 6. TEST SEQUENCE FILE ORGANIZATION ........................... 5 1 978-1-4244-7351-9/11/$26.00 ©2011 IEEE 2 IEEEAC paper#1335, Version 3, Updated 2011:01:01

7. EXECUTION OF TEST PROCEDURE .................................. 5 8. A TYPICAL TEST PROCEDURE WRITTEN USING CCL ... 6 9. MANAGEMENT OF TEST PROCEDURES ........................... 6 10. ADVANTAGES OF CCL .................................................. 7 11. LIMITATIONS OF CCL .................................................. 7 12. LINKAGE OF CCL WITH ACSS .................................... 7 13. FUTURE DIRECTIONS ..................................................... 7 14. CONCLUSION .................................................................. 8 ACKNOWLEDGEMENTS ........................................................ 8 REFERENCES ........................................................................ 8 BIOGRAPHY ......................................................................... 8

1. INTRODUCTION Complexity of the satellites is increasing day by day. Automation reduces time required for system level testing of complex spacecrafts, Reduces operator workload. Potential for errors during test and mission operations decreases with automation. Automation ensures repetivity of tests. All the above mentioned points have an effect on the overall spacecraft cost. At ISRO Satellite Centre, Spacecraft Checkout Group, there has been a continuous effort to improve automation in spacecraft testing to achieve the goals mentioned above. Automatic Checkout Software System (ACSS), designed and developed in house will cater to all the requirements of testing a complex spacecraft. This is a generalized software, which can be configured to test any class of spacecraft with minimum efforts.

Automation of ground systems operations is crucial to efficient and cost-effective spacecraft testing. Automation is achieved when scripted procedures are used to conduct normal operations, such as configuration of ground equipment for a satellite test, commanding the spacecraft or payload to a new configuration and monitoring of spacecraft data. Experience has shown that these procedures substantially reduce both operator workload and the potential for errors during testing. These scripted procedures involve commands to the spacecraft, commands to ground equipment, references to ground equipment status, references to current telemetry values.

2

2. AUTOMATIC CHECKOUT SOFTWARE SYSTEM Automatic Checkout Software System (ACSS) is a set of software products which is developed for the automation of spacecraft checkout operations. It is organized into various real-time, offline, support and special purpose software packages. The following is the list of major packages under the ACSS.

• Test Sequence Interpreter (TSI) for automatic sequencing of Test Procedures, running in the foreground which also controls/monitors other controllers WHICH ARE PART OF spacecraft Checkout System(SCS).

• An in-house designed and developed test language called Checkout Command Language (CCL) to prepare the test procedures well in advance as sequence of test steps

• Test Report for detailed summary of each step of Test Procedure execution running in the foreground

• Simultaneous Acquisition of multiple chains of Telemetry data, Telemetry data Archival, Telemetry data Processing, Telemetry (TM) data Analysis running in the background

• Display and PLOT of Raw, Processed TM data in various formats

• Continuous monitoring of Critical TM Parameters • Control and monitoring of all ground support

equipments.

• Powerful analysis tools and support software packages

Analysis of Telemetry data is carried out with respect to the test step being executed. Analysis of the data involves, for every command/stimuli set to the spacecraft verifying the parameters that are supposed to be affected for the expected values and checking all other parameters that they are not affected in any way. The confirmation of the command execution is carried out automatically by verifying the command counter and auxiliary register parameters. There are many support packages available in ACSS for TM page creation, sorting of databases, modules to view the task status, shared memory. Special purpose packages include tasks to process special sensors data, special purpose payloads data. ACSS tasks are depicted in figure-1. All interfaces are standardised on Ethernet and SCS has full redundancy, which ensures seamless testing even in case of any ground system failure.

Figure-1: ACSS TASKS

Off-Line

Analysis

packages

Retrieval ProcessingData

Presentation

Reports

GenerationAnalysis

HK data

processing

Dwell data

processing

Support

Software

packages

CCL Syntax

Verifier

Generation of

DatabasesUtilities

Document

Generation

Par FoundUNIX editor

For generation

Database

Sort

Interactive

Messaging

(XTalk)

CCL

Syntax

Verifier

Real-Time

packages

Data

Acquisition

Data

Processing

Data

Presentation

ArchivalAnalysisTest

Sequence

Interpreter

3

P/L COMPUTERS ETHERNET BUS

MAIN CHECKOUT SERVER

REDT SERVER

Documentation

EA186C00

DAS MAIN/REDT

ENCODERMAIN/REDT

RF,BAS,SAS,STIMULLI & UMBLICAL

P/L CHECKOUT SERVER

S/C

CRP OFF LINE

ANALYSIS/ BPMS

PRINTER

B/W LASER PRINTER THERMAL-PC

CLIENT

Figure.2 Typical Spacecraft Checkout System

ETHERNET I/F

OBIS PC To Mission Client

SCOS CNTRL

3. SPACECRAFT CHECKOUT SYSTEM

A typical spacecraft checkout system consists of Spacecraft Checkout Computer (SCC) connected on network to all the required SCS equipments (both main and redundant) and peripherals. Typical SCS equipments include Data Acquisition System to acquire telemetry data from the spacecraft, Telecommand encoders to send commands to spacecraft, battery and solar array simulators to feed power to satellite. Many measurement equipments and peripherals like printer will be on the network.

All SCS equipments are on secured private network CHECNET which not only interconnects all SCS equipments but also provides authenticated access to spacecraft data for authorized external users. Typical CHECNET is shown in figure-2.

4. CHECKOUT COMMAND LANGUAGE This is the in-house designed and developed test language which is used for the checkout purposes. Checkout Command Language (CCL) is the command language with simple English like instructions like SEND, CHECK, etc. developed for writing test procedures to be executed by ACSS for performing checkout operation on the spacecraft. CCL is designed in such a way that the test files created using CCL can serve both as an input file to the computer as well as a document for the test procedure. CCL enables the checkout engineer to prepare the test files well in advance

which should be interpreted and executed by the Test Sequencer.

CCL provides all possible instructions required for the checkout of a satellite like:

• Commanding the Spacecraft. • Control and monitoring of Ground support

equipments. • Analysis of data obtained from different sources

and presentation of processed data. Apart from the above instructions, CCL also provides instructions for linking files, interactive mode control and other general instructions.

CCL has been developed keeping in mind all the requirements for spacecraft checkout and provision has been provided to define new instructions as and when the need arises for future Spacecraft Checkouts. The instructions available in CCL are listed below. These instructions can be divided in two types as executable instructions and others which directly do not do any operations or control but provide the user with the necessary help and information.

4

Some of the instructions available in CCL are:

SEND BREAKPOINT CALL CHECK DEFLIMIT DISPLAY END EXECUTE GO HELP IGNORE INCLUDE INIT INHIBIT MACRO PERMIT PLOT PRINT READ REMARK SELECT SENDCODE SET TABULATE TESTNAME WAIT SINGLESTEP TOLERANCE WAIT-TILL SWITCH !...! - Comment Statement Test procedures are written well in advance and validated using the CCL compiler before being used for execution. Separate directory of test procedure files is maintained by ACSS. CCL also provides instructions that controls and monitors slave controllers like payload checkout computer where the high bit rate payload data for remote sensing payloads or RF measurement data of communication payloads are acquired and processed. SCOS controller, which controls all the simulators, connected to the spacecraft. TTC-RF controller, which control the uplink, downlink equipments of checkout. In addition transfer of selected pre-defined processed data to different agencies will be enabled depending on the tests in progress. All these interfaces are on standard Ethernet TCP/IP protocol. All softwares residing on these slave controllers is also automated to a great extent and does the job of equipment control based on the instructions from the main spacecraft checkout computer and passes on the relevant data required back to the main checkout computer.

5. CLASSIFICATION OF INSTRUCTIONS IN CCL The instruction set available in CCL can be broadly classified as

(a) Instructions for transmitting telecommand to the satellite and verifying telemetry received from the satellite, instructions for data analysis (SEND, SENDCODE, CHECK, EXPECTED) – these instructions when executed will interface with

ground equipments like telecommand encoder, data acquisition system as the case may be.

(b) Instructions to linking test files, related to test file itself (CALL, TESTNAME, END, REMARK, !..!)

(c) Instructions for control/monitoring of slave controllers (SET, READ, INIT, CHECKSCOE) – These instructions at the time of execution will interface with slave controllers which controls the actual ground simulators like Battery Simulator, Solar Array Simulator or Stimuli generator.

(d) Instructions to control the execution of test procedures (GO, SINGLESTEP, BREAKPOINT, SELECT, WAIT, WAIT-TILL)

(e) Instructions for Visual displays (DISPLAY-PAGE, PLOT, PRINT-PAGE) – These instructions interact with the corresponding systems/peripheral connected on the CHECNET for the execution

(f) General Instructions (REFRESH, HELP) Instructions for visual display and hard copy options have many sub-instructions like DISPLY-PAGE, DISPLAY-RAW, PRINT-PAGE, PRINT-PLOT. PLOT instruction has got multiple sub-instructions like PLOT-GRID, PLOT-TITLE, PLOT-SUBTITLE. All telemetry, telcommand mnemonics are defined in the ACSS database, which are used along with the CCL instruction while writing test procedures. Instructions used for telemetry data analysis like CHECK, EXPECTED will have a syntax where what is the expected value or against what the parameter has to be checked needs to be provided. For ex: CHECK BUS-A-VOL = 42V means check the telemetry parameter Bus-A-Voltage is reading 42 volts. This will be reported to the user with whether check failed or ok and with the current value of the BusA Voltage. If user wishes to bypass some telemetry parameters from being checked, he can do so by using the IGNORE instruction. INCLUDE instruction can be used to put them back to the CHECK.

Similarly a SEND instruction can have only one telecommand to be sent or a series of commands to be sent to the spacecraft. The result of the SEND instruction is reported to the user after verifying whether the telecommand reached the spacecraft, the required parameters like command counter updated. User will get the report of every instruction on his console.

A list of commands to do a specific job can be defined as MACRO with a new mnemonic and can be sent using that mnemonic. For ex: WHEELS-OFF is a MACRO which sends 4 telecommands WH1-OFF, WH2-OFF, WH3-OFF, WH4-OFF. MACRO’s can be defined dynamically using

5

the MACRO instruction. Using PRINT MACRO instruction user can view and print the list of MACROs defined.

User can avoid sending telecommnads by defining them under INHIBIT instruction. Such commands will not be sent to spacecraft even if they are in the procedure and test engineer gets an alert. Using PERMIT instruction test engineer can enable the telecommand to be sent if required. CALL instruction helps the test engineer to link the test procedure files. Syntax like file cannot call itself or 2 files cannot be calling each other is verified by the CCL-Compiler before execution. REMARK instruction can be used to remind the operator to ensure some pre-conditions or to do some operations manually before going to the next step. CCL also provides provision for inserting comments in the test procedure file using ‘!’. These lines are skipped by the Test Sequence Interpreter during execution. User can continue the instruction in more than one line using the ‘;’ . This if put at the end of the line indicates that the instruction is continued in the next line also. As CCL interpreter is Line based, this provision is provided to take care of instances where a list of telecommands have to be sent or a big list of telemetry parameters have to checked. Provision to come to interactive mode of operation while executing the procedure in automode is made available to be user with BREAKPOINT instruction. When Test sequencer encounters a BREAKPOINT instruction, it halts the procedure at that step and hands over the control to test engineer. Test engineer can do what he wants (can call any other procedure, can end the file, can type any instruction he wants) or type GO to execute next step.

6. TEST SEQUENCE FILE ORGANIZATION A well organized Test Sequence file would look something like this: SECTION 1

Test filename Brief Description of the test Spacecraft Configuration for the test Constraints if any for conducting the test Files being called by this File.

SECTION 2 TION 2 Definitions of MACROS Defining Dynamic Limits Inhibiting any TCs as required by the test. Defining Parameters to be ignored

SECTION 3 Actual Test Procedure Organised in a structured

manner. Can be divided into blocks if required Use of “CALL” instruction to link the files Make it self explanatory by judicious use of

REMARKS and Comments. SECTION 4

Restore the conditions specifically set for this test at the end of test before the “END” instruction.

This section will contain instructions to put back the INHIBIT telecommands after the test, IGNORE telemetry parameters if required etc.

SECTION 5 The End Instruction.

A typical test file can be as small as 10 lines to as big as 10,000 lines. Test procedures can be written on any PC and later can transferred to the UNIX/LINUX machine where they will be executed

CCL based procedures have to pass through CCL-Compiler before getting executed. CCL-Compiler will check for the syntax of the test procedure, validity of the instructions and their syntax also. It will generate a report of errors encountered which can be corrected by the test engineer before execution.

The test procedure files created using CCL will serve as - Input to the Test Sequencer - Document for Test Procedure

The Main characteristics of CCL are: - Simple, Easy to learn and use - Self Documentation

7. EXECUTION OF TEST PROCEDURE Test procedures written using CCL can be executed either in singlestep mode or Automode or in interactive mode.

Test engineer has got the option to choose one of these at the Test sequencer console.

Test engineer can prepare all the procedures required for spacecraft testing offline and validate them using CCL compiler. The valid test files can be invoked at the test console during spacecraft testing either in Automode or SingleStep Mode by the test engineer. Eventhough all procedures are validated offline, in case of interactive mode of operations, Test file interpreter itself will provide the user with the syntax errors which can be corrected on-line before execution.

6

All the results of the tests conducted on the spacecraft are logged with proper date, time and test conditions for offline analysis.

Repetivity of the tests are ensured if procedures are prepared using CCL.

Figure.3 – Steps involved in satellite testing using CCL

8. A TYPICAL TEST PROCEDURE WRITTEN

USING CCL ! THIS SAMPLE PROCEDURE PUTS ON GYRO! ! BEFORE GYRO ON, ITS TEMP. IS CHECKED! ! IF TEMP < 20’C, GYRO HEATER IS PUTON! ! AFTER ENSURING GYRO TEMP > 20’C ! ! GYRO IS MADE ON ! 10 TESTNAME GYRO-ON 20 REMARK IF GYRO TEMP IS > 20 THEN ONLY PUT ON GYRO ELSE PUT ON GYRO HEATER 30 CHECK GYRO1-TEMP 40 SEND GYRO-HTR-ON 50 SEND GYRO1ON 60 CHECK GYROSTS=ON, GYROVOL,GYROCUR 70 WAIT-TILL GYROSPEED=6000 80 SEND GYROLOOPCLOSE 90 REMARK NOW GYRO1 IS READY FOR USE 100 CALL GYRO2-ON 110 CALL GYRO-3-ON 120 REMARK ALL 3 GYROS ARE ON AND READY FOR USE 130 REMARK REQUIRED STIMULI TO BE PROVIDED

140 SET SCO Gy1-Stimuli = xxx 150 END As mentioned earlier this is a sample file and actual test procedure file may run into thousands of lines.

9. MANAGEMENT OF TEST PROCEDURES

All the test procedure files that are CCL Compiled and ready to use are stored in an identified directory on the Checkout computer from where test files are executed. This computer will have interface with all the required spacecraft checkout systems like telecommand encoder, data acquisition systems, other slave controllers, printer, other PCs which are part of CHECNET. User can use the support software package called Test file Editor to update/modify the test procedure with proper authentication, and the updates done by him are also logged with date and time.

CCL can be compared with other satellite test languages like STOL (Satellite Test and Operations Language) developed

7

0123456789

1 0

1 9 9 0 1 9 9 5 2 0 0 0 2 0 0 5 2 0 1 0

I n c r e a s e i n A u t o ma t i o n

0

2

4

6

1990 1995 2000 2005 2010

Increase in Automation

Tim

e for

tes

ting

(m

ont

hs)

02468

101214161820

1990 1995 2000 2005 2010

Increase in Automation

Man

pow

er for te

stin

g

0

50

100

150

200

1990 1995 2000 2005 2010

Increase in Automation

Software

Module

s

by NASA, PLUTO (A procedural Language for users in Test and Operations) by ESA.

CCL has been developed using C language on UNIX/LINUX platform with a compiler to support the same. But test procedures can be written using MSWORD, ‘VI’ editor or any other text based editors and later can be compiled by CCL compiler before being using for satellite testing.

10. ADVANTAGES OF CCL

Simple, Easy to Learn and Use Procedures from earlier spacecraft testing can be

imported and modified for next satellite Can be used effectively to train new engineers

inducted to satellite testing Repetivity of the tests is ensured Self documenting

11. LIMITATIONS OF CCL

At present CCL has got very few control instructions/statements, needs many more decision making instructions to make it fully automatic. CCL is under constant improvement and new instruction sets are being added based on test requirement and users feedback. In near future CCL is planned to be augmented like any other procedural language with provision for defining a set of heterogeneous instructions to do a particular function.

12. LINKAGE OF CCL WITH ACSS ACSS maintains a set of libraries for interpreting and executing CCL instructions. These are invoked by the Test Sequence Interpreter of ACSS. It also provides interface with the ground support equipments through appropriate device drivers available in ACSS. User gets success/failure notification with proper error messages from ACSS during the execution of test procedures. Databases maintained by ACSS are used by the CCL for the TM, TC mnemonics and the expected values are also against the processed buffer outputs of ACSS. CCL Compiler and executor make use of all the real-time, support packages of ACSS. Test engineer makes used of support packages available in ACSS for test file generation/editing/modification as well as for data analysis.

13. FUTURE DIRECTIONS The instruction set of Checkout Command Language is continuously evolving with many additional features, which improve automation. The aim of these improvements is to provide test engineers with simple, easy to use instructions for testing of future complex spacecrafts. The advantages we have achieved over the years from the improvements in automation are shown in the figure-4

Figure-4: Advantages of increased Automation

8

Designers have worked out the future updates in CCL in the following areas:

1. Additional instructions will be added to the existing instruction set to improve automation, conditional execution.

2. User Friendly GUI’s for ease of entry/edit and execution of test procedures.

3. Graphical representation of the test file being executed to enable user to visualize the current step, activity in real-time.

Future developments in other related parts of ACSS include Multi-sequencing and execution of Test Procedures in parallel, which have been thought of as a viable solution in reducing testing time. The challenges in realizing a parallel Test scheduler lies in the definition of mutually exclusive procedures that can be executed in parallel, decision making, scheduling based on a scheduling policy, assigning priorities to the tests running in parallel and providing access to shared resources.

14. CONCLUSION

Checkout procedures for different class of satellites have been written using CCL and executed at ISRO for more than a decade now. Test engineers are able to import test procedures from previous satellite checkouts and modify/update if required for the new satellite checkout with very little effort. This has been a great advantage in terms of time, effort and understanding the sub-systems being tested. Procedures from the previous checkouts are provided to new test engineers, as part of their training and to make them understand satellite testing faster. CCL has been evolving with additional features and instructions and ACSS is also getting updated with new libraries in this process. CCL is found to be very simple, easy to learn, use from test engineers point of view.

ACKNOWLEDGEMENTS The author expresses thanks to the entire software development team at checkout group of ISRO SATELLITE CENTRE. Author also expresses gratitude to Mr. K.B.A.R.Sarma, Group Director, and Mr. K.V.Govinda, Deputy Director, Mr.S.K.Shivakumar, Associate Director for their valuable guidance and useful suggestions. I express my heartfelt thanks to our centre director, Dr.T.K.Alex for his support and providing necessary permissions for generating and submitting this paper for IEEE Aerospace Conference. My sincere thanks to Chairman, ISRO Dr.K.Radhakrishnan for approving my deputation to attend and present this paper at the conference. .

REFERENCES [1] Features of Automatic Checkout Software system.

Doc no.INTS-TR-AIT-03-90-66(0), ISRO Internal Document [2] GuideLines for writing Test Sequence files using Checkout Command Language. Doc no. SCD-CSS-01-1-93, ISRO internal document [3] Papers published by the author in many national/international conferences. ACRONYMS ACSS – Automatic Checkout Software System ATS – Automatic Test Scheduler CCL – Checkout Command Language GUI – Graphical User Interface HK – House Keeping IST – Integrated Satellite Test P/L – Payload SCS – Spacecraft Checkout System SCOS – Special Checkout System TC – Telecommand TM – Telemetry TSI – Test Sequence Interpreter

BIOGRAPHY

U.N.VasanthaKumari is currently Division Head for Scientific

and Remote Sensing Satellite Checkout Division at ISRO Satellite Centre, Bangalore, India. She has 27 years of experience in software development and satellite testing. She was heading a team of software designers/ developers and test

engineers. She was the key person in the development of Checkout Software for all Satellites built by ISRO. She has guided the teams in bringing about Automation concepts and improving the same over the years. She has worked as Project Manager, Deputy Project Director Checkout for many Remote Sensing Satellites. Currently she is also holding the additional responsibility as Associate Study Director for Cartosat-3 series of sarellites. She has many papers published in both National & International Seminars/Conferences. Her research interests include software engineering, automation, expert systems, and data mining.