Using Built-In-Test to

Reduce TPS Run Times

Improve TPS Reliability and

Hairy Rogin, M&T Corporation

ABSTRACT

This paper addresses using information derived from Built-in-Test (BIT) to fault diagnose Units Under Test (UUTs), wherever possible. This philosophic approach to diagnostic testing is not new. It has been studied over the past 20 years under the visor of “Integrated Diagnostics,” but it has yet to he truly implemented in a “real life” military diagnostic test environment. The mindset of Test Program Set design engineering, along with customer and contractor management alike, remains “complete diagnostic testing based upon single catastrophic component failure modes.” If we are to generate cost efficient Test Program Sets (TPSs) under reduced military budget constraints, this will have to change! The test engineer must be encouraged to use methodologies to speed up development time and decrease TPS run times. Using present technology, this is possible now, and as the technology matures, will become a truly viable approach in the future.

For the purpose of this paper, the author relies heavily on his extensive US Navy Automatic Test Equipment (ATE) and Test Program Set (TPS) experience, as well as on previous studies performed on using BIT to fault diagnose Unit Under Test failures on US Naval Air weapon systems.

Author’s Current Address: Lead Engineer, M&T Corporation, Building 463, Naval Aviation Depot, NAS, Nonh Island, Coronado, CA92122. USA.

Based on a presentation at AUTOTESTCON ‘99.

0885/8985/99/ $1O.W 0 1999 IBEE

INTRODUCTION

One of the major frustrations of test engineers has been the non-testability of military electronics. This problem has taken the form of insufficient and/or inappropriate test points, inability to meet ambiguity group size requirements (the number of components being replaced at any one time), insufficient Built-in-Test (BIT),

Automated Test Equipment (ATE). For the purpose of this report, the concept of using BIT to fault diagnose a UUT is addressed. This is covered from the standpoint that, if extensively implemented, BIT has the ability to reduce the overall conventional testing of a UUT by eliminating those functions that are adequatcly tested by BIT, and can be used to steer diagnostics testing based upon the BIT results.

BACKGROUND

and Unit Under Test (UUT) incompatibility with

The advent of fully automated testing of military avionics began in the late 1960s and early 1970’s with VAST (Versatile Avionics Shop Test). This concept was so new at the time, that there were no military specifications on how to perform this task. The guidelines were pre-specifications AR-8 and AR-9. These provided the guidelines for performing UUT/ATE compatibility studies and set programming standards for the Test Program Sets (ambiguity group size definitions and standard displays). In addition, when TPS were first being developed on VAST, there were no guidelines for design engineers providing methodologies for implementing BIT in the avionics, much less methodologies for TPS developers on how to test BIT

IEEE AES Systems Magarine, December 1999 9

that did exist in TPSs. On early 1960’s aircraft, such as the E-2C, F-14, and S-3A, BIT was developed as an Operational Level (OLEVEL) diagnostic tool. It provided information to the air crew as to what avionics box and/or what aircraft function failed. Hence, it was used by maintenance personnel to determine which box(es) to pull off the aircraft for checkout. It was so early in the technological scheme of aircraft BIT, that it was not uncommon for numerous good boxes to be pulled off the aircraft due to insufficient resolution. This resulted in TPS “retest OK’ displays (if everything else was working OK), or incorrect diagnostic callouts if the test specifications (usually based upon design parameters) did not simulate the BIT conditions that caused the box to be pulled from the airplane in the first place. This could be caused by UUT interface parameters being slightly out of design specification, while the UUT would function properly in the aircraft, or by the TPS not properly simulating aircraft operating conditions. The aircraft systems and black box specifications at the time required the avionics to be “VAST Compatible.” The problems with this are many, including the fact that nobody had done this before and hence, nobody knew how to make avionics compatible with VAST. “Design for Testability” was a phrase that had yet to be invented. Moreover, few, if any, of the avionics design engineers knew what VAST was, much less how to make their electronics compatible. The result - each box had a “Test Point Connector” added. This was sufficient to meet this ambiguous requirement. The test points might have been useful, or they might not.

Built-In-Test was to aid in diagnosing the aircraft, not the electronics. Hence, BIT was considered by the TPS engineer to be an avionics function to be tested, not necessarily to be used to aid in diagnosis of the failure. Nobody had any confidence in the ability of BIT to aid in the diagnosis of electronics to the next lower assembly. BIT was tested to make sure it reported a good UUT (if in fact there was no failure detectable by BIT) and that it was able to report a failure, if indeed one could be simulated on the Automatic Test Equipment.

The test philosophy of performing a full functional test on avionics boxes, with BIT as just one of those functions, remained in effect until the sale of the F/A18 aircraft to the Australians. The Australian Air Force was not overly thrilled with the concept of diagnosing avionics in the shop, away from the aircraft. They did not like the cost of buying and maintaining the Automated Test Equipment, the excess cost and schedule delays that were occurring in TPS development, the incompatibility between aircraft BIT and TPS functional tests, the training of technicians in ATE operation and maintenance, and maintaining ATE spare parts. They believed that there must be a simple way to perform the avionics diagnosis function, and they came up with the idea of an “Avionics Fault Tree Analyzer” (AFTA). This essentially was a portable computer that plugged into the main aircraft data

As mentioned previously, the purpose of

bus and executed aircraft BIT, while the aircraft was on the ground. It predicted which Weapon Replaceable Assembly and related subassembly was at fault based on BIT results and predicted (calculated) reliability rates. No additional functional tests or diagnostic tests were performed. This had some basic advantages and disadvantages over conventional testing techniques. First, as mentioned above, it was cheaper and faster to develop. It was also much faster to execute. There was no incompatibility between aircraft BIT and diagnostic testing techniques. However, its flaws were substantial - it was only as good as Aircraft BIT, and BIT on the F/A-18 aircraft, while an improvement over earlier aircraft systems, is inconsistent. Some systems are excellent and some are lacking, with others somewhere in between. Also, in a US Navy environment, running diagnostics on a carrier flight deck during operations is not considered safe! The U S Navy attempted to improve on these concepts with I-AFTA, later renamed IATS test system. This consisted of an aircraft simulator built around the M A . Some supplemental testing to attempt to functionally test the UUT circuitry missed by BIT. This resulted in test programs which executed substantially faster than conventional ATE TPSs, but were considerably slower than the original AFTA tests. The IATS test programs were written in a modified C, which programmed the Aircraft Simulator, a register based hardware set. This made the programs difficult to generate and maintain without extensive circuitry descriptions (schematics or cheat sheets) for the Aircraft Simulator section of the IATS. The aircraft simulator section of IATS was also based exclusively on the F/A-18, and would most definitely require modification to test avionics from other aircraft.

HISTORIC IMPROVEMENTS

As billions of dollars were sunk into early ATE and Test Program Sets, it became obvious that something had to be done to improve Design for Testability. Early (pre-microprocessor controlled) UUTs centered around defining test points requirements and breaking feedback loops at the PC board interface. This yielded some simplifications in test programs, but was quickly overtaken by increases in UUT circuit complexity and capability, enabled by advancing electronic technology. Large Scaled Integrated (LSI) and Very Large Scaled Integrated (VLSI) circuitry posed new test problems that would have to be addressed if we were to bring this monster called ATE/TPS under control.

comes via Built-In-Test (BIT), incorporated into the avionics box. This has come about over the years as microprocessors and microcontrollers were integrated into the avionics. However, BIT has not solved, and probably cannot solve, all testability problems. Let’s consider the example whereby BIT checks the circuit of an aircraft weapon decoder (which fires a missile, drops a bomb or

One answer to this Design for Testability problem

IO IEEE AES SysIem Magazine, December 1999

fire!% a gun). It cannot check the components right at the interface. in an operational environment without actually firing the weapon or providing a simulated realistic load on the aircraft. The decoder must be electrically detached from the weapon first and the component that detaches it will not be tested by BIT. Being at the interface, this component may even be a stressed (high voltage and/or high current) component with a high failure rate. This component, as well as other interface components that are not adequately tested by BIT, will still require conventional functional testing techniques.

BUILT-IN-TEST (BIT) - THE SOLUTION! (OR IS IT?)

For years, those of us in Avionics Test Program Set Development have been hearing that, as ”SMART BOXES” that can truly test themselves are developed, we will be put out of work. We may be getting closer to this ideal, but it probably will not happen in the near future. While BIT has helped the diagnostic process, problems remain:

What tests BIT? * How thorough is BIT? 0 What are BIT’S diagnostic capabilities?

These questions can only be answered on a case-by-case hasis. However, lets look at the implications of each question.

Whiat Tests BIT? While Built-In-Test checks the functionality of the

circuitry around it, we usually have no guarantees that BIT itself is fully operational and is properly reporting failure modes. Two conditions can exist. First, the BIT circuitry may be reporting a failure that does not exist. This is the easiest case to handle. This can he detected by functionally checking UUT outputs relating specifically to the reported failure. If the UUT function is operational, the failure is in BIT. The second case is much more difficult to handle. If BIT is incapable of reporting a circuitry failure that it is supposed to detect, we may not be aware that the BIT function, per se, is faulty - possibly until it is too late. BIT miist include the ability to test itself by simulating failures and making sure that they are detectable. Hence, the answer to this question becomes obvious, but not always easy or cost effective to implement - BIT tests BIT. The problem here is real estate (size and weight constraints). Additional circuitly and firmware (which may require additional memory and additional computing power), are required.

How Thorough is BIT? The ideal situation would he that BIT is contained

within the firmware of the UUT and tests the UUT 100%. But, we’re first making an assumption that the UUT is a

“Smart Box.” That is, that it contains a microprocessor and firmware. Here is an important limitation - not all UUTs are “Smart” (although technology is going in that direction). For non-microprocessor UUTs, BIT analysis becomes a moot point.

The test engineer developing a conventional military TPS is usually required to generate a “Fault Accountability List” as part of the Test Strategy Report (TSR). In the past (and still today), this task is performed during the TPS development stage, well after the tinal UUT design has been completed. This is totally unacceptable if we are to optimize BIT testing of UUTs. The function of defining what can (and is) tested by BIT is performed by the UUT design engineer. This information then needs to be provided to the TPS development engineer, on a component hasis if possible, or at a minimum, on a functional basis, with the translation to a component basis performed at a later date. An ideal situation would be to generate a set of BIT standards and/or get the test engineer involved early in the design process.

What are BITS Diagnostic Capabilities?

But most past implementations of using BIT for diagnostic purposes has resulted in calling out only the failed function or EVERY component or subassembly relating to the functional UUT failure. This is definitely not an efficient way to repair electronics. This requires refinement in the future, and a restructuring of BIT tests to provide additional insight into failed component(s).

BIT and TESTABILITY

It’s great for BIT to isolate to a functional failure.

The questions pertaining to BIT and Testability have been discussed, but methodologies for implementing a solution on a case-by-case basis needs to he addressed. The requirement is to use BIT to test as much of the UUT circuitry as possible (feasible), and to determine how to report the effectiveness of BIT hack to the design engineers. At present, there are no real solutions to automating this process.

of simulators and software (firmware) emulators. Hardware simulators, such as LASAR (Logic Automated Stimulus and Response), have been used by design engineers and test engineers to check the validity of the design, and to develop test diagnostic vectors, respectively. What can also be accomplished using LASAR is the simulation of the BIT firmwarehardware combination. This is relatively simple to accomplish once the UUT LASAR model is developed (something that is required for the traditional LASAR uses). The UUT can then he commanded to run BIT (only), just like it would be on ATE during TPS execution. The UUT can he simulated (SIMUL execution), followed hy Diagnostic Generation simulation (DYSOGEN). The results of this run would define the BIT detected failure mode environment hy

One such solution for digital UUTs might be the use

IEEE AES System Magazine, December 1999

listing UUT component failure modes that have not yet been detected. This information is then provided back to the design team of engineers, who can decide if BIT enhancement is warranted or required.

procedure can remain the same, but the actual implementation is much more difficult to implement. This is because analog and hybrid simulators, such as ORCAD, SPICE, and MicroCAP, are not yet as sophisticated as the purely digital simulators. Analog models are not as straightfonvard as digital ones, inasmuch as most of them perform successive approximations to reach a solution, and if the component models are not “just right,” the algorithm may not converge properly. This becomes critical if we bring this to a logical conclusion of emulating component failure modes - the procedure that LASAR uses to determine fault diagnosis effectiveness.

However, with present technology, we are not quite to a point of completely automating this process. Until that point arrives, we can use the analog simulators to provide information on BIT effectivity, albeit, semiautomatically.

CONCEFTS FOR IMPLEMENTING BIT IN TPSs

For hybrid (part analog’part digital) UUTs, the

Now that we’ve discussed some of the problems of implementing Built-In-Test in avionics, we need to discuss how we are to use this BIT information in TPS development. The main topic of this paper is to use BIT to reduce TPS development and execution time. This requires the TPS development engineer to utilize BIT information effectively! With tight budget and schedule constraints that we experience today, we no longer have the luxury of duplicate testing. As discussed before, we need to get information on Built-In-Test hardware and firmware implementation to the engineer developing the test philosophy. If BIT is implemented so as to allow for diagnosis to specific subassemblies, no additional analysis along these lines is necessary. If BIT resolution does not reach this optimum solution, additional analysis is required to relate the BIT matrix information to faulty component(s). In a worst case situation, this analysis would be performed by reverse engineering the BIT related firmware, and cross-referencing it to UUT hardware failure modes. By process of elimination, a critical result of this analysis would be a list of components that are not fully tested by BIT. Once this information is obtained, we can reduce duplicate testing and (here’s the critical part of the concept):

IT IS ONLY NECESSARY TO FUNCTIONALLY TEST FOR COMPONENTISUBASSEMBLY FAILURE MODES NOT DETECTED BY BIT!

I believe that we all see that a TPS implemented so in this manner would be considerably shorter, with resultant reduced development times, reduced integration time, reduced run times and reduced maintenance.

METHODOLOGIES FOR IMPLEMENTING BIT IN TPSs

There exist two distinct approaches toward implementing Built-In-Test information into Test Program Sets:

1 . The conventional approach -Exercise BIT on the Automatic Test Equipment. This is normally done on TPSs as a diagnostic prescreening, prior to functional testing the UUT. UUT inputs that are critical to the execution of BIT on the UUT are simulated by the ATE and BIT is executed. The unconventional approach - Just read the UUT BIT matrix without executing BIT. This has the potential of obtaining information about the UUT failure as it was experienced on the aircraft itself.

Neither is “Right” or “Wrong.” There are strengths

2.

and weaknesses to each of the two approaches. For approach Number 2, there are certain assumptions that have to be made. These include that the UUT has a non-volatile memory to store the Aircraft BIT data, that the aircraft BIT data is indeed intact (this will usually be partially true, as it is typical for some BIT routines to be re-executed every time the UUT is powered up), and that BIT was indeed executed on the aircraft and was the primary reason for isolating the aircraft failure to this specific piece of avionics. Assuming all these to be true, a wealth of information can be obtained using this methodology. The aircraft failure data is directly related to the maintenance action, and is obtained under actual operating conditions, thereby increasing the likelihood that the maintenance action will be effective. And we don’t have to simulate signals on the ATE, thereby keeping our Interface Device design and TPS software simple and straightfonuard. The down side is that we now have no ideal way to re-verify the UUT without putting it back into the aircraft (not a good solution for obvious reasons).

Approach Number 1 has the down side of complicating the TPS hardware and sofhvare designs and simulating the aircraft conditions, thereby decreasing test reliability. It does provide a means to test those functions that are implemented, either completely or partially, in BIT. It also provides a methodology for retesting the UUT.

What is obvious for this discussion on BIT implementation is that neither approach, in and of itself, is a complete solution. However, used together, they can complement each other’s weaknesses and provide an optimum solution to increasing the effectivity of BIT as a diagnostic tool.

A methodology of using both BIT implementations in TPS development may be:

1 . After powering up the UUT, read the UUT (aircraft) BIT matrix. Fault diagnose the UUT if a failure is indicated. Run functional tests on

12 IEEE AES SySrems Ma8azine. December 1994’

the failed function if necessary to decrease the ambiguity group size. If no failures are stored in the UUT BIT matrix, re-execute BIT. If a failure is now indicated, perform fault diagnosis using the same methodology as above. Perform supplemental functional tests on only those components that arc not thoroughly tested by BIT. Ensure that parameters (such as loading) reflect those experienced during aircraft operations.

2.

3.

Using this methodology, we can decrease the functional testing requirements and hence the GO Path execution time, while providing traceable diagnostics to actual aircraft failures where possible. Ambiguity Group size requirements arc maintained through conventional testing methods, sometimes by re-hosting GO-Path functional tests into TPS diagnostic paths.

Devices might he able to he simplified, TPS software can be simplified, all with a resultant decrease in development and maintenance costs. And what docs all this cost us - just some additional engineering effort in the front end of the TPS development effort to analyze the BIT functions contained within the UUT hardware and firmware.

Keep in mind that using BIT to diagnose and isolate failures is not a new concept. It has been done before on the IAXS tester. It just has never heen implemented on general purpose Automated Test Equipment, such as CASS or IFIE, that the US military bas now directed to use.

As a result, TPS integrity can be improved, Interface

WHLY WAS THIS NEVER IMPLEMENTED BEFORE?

There are many reasons that this approach has never been implemented on general purpose ATE. Studies performed by the author on I-AFTMATS testing effectivity on F/A-18 weapon systems, along with various IATS TF’S engineering investigations, have shown some weaknesses in IATS testing techniques.

Let us first address the limitations of TPS implementation on the IATS tester. It is important to remember the original concepts behind AFTA (the predecessor to IATS) to realize why the implementation fell short. This original testing methodology, used by the Australian Air Force, was meant to do nothing more than execute aircraft BIT, which it docs very effectively. The concept of diagnostic testing was completely eliminated. Faulty subassembly callouts included everything relating to the failed BIT function. According to the rules of Fault Tree Analysis, these subassemblies were ranked based upon predicted reliability rates. In real life, the implementation of this concept resulted in huge ambiguity groups (at times approaching all of the subassemblies within the UUT), all with very similar failure rate percentages. Attempts have been made on improving this

data with actual failure rates, hut for various reasons, this has never heen accomplished. In an operational test environment, this has resulted in the 80 year old troubleshooting technique that the old-timers call “tube swapping.” Supplemental testing on IATS was the afterthought. In order to implement this supplemental testing, an aircraft simulator was developed around the interface requirements of the F/A-18 avionics boxes that were to be tested on IATS. The major problem with the design of this simulator was the interface circuitry. For reasons that arc not obvious to the author, the implementation of the interface circuitry contained within this aircraft simulator was not representative of aircraft loading. For example, insufficient loading of high current circuitry contained within the F/A-18 Weapons Decoders caused the output drivers (many of which supply in excess of 30A in real life conditions) to he tested at very low currents (milliAmps). In a conventional Test Program Set, this load would either he part of the ATE assets, or he contained within the Interface Device. However, it is not in the IATS and there are no Interface Devices for IATS TPSs, just cables and holding fixtures. Deficiencies of this type have caused degraded testing of the UUTs. Also, insufficient UUT supplemental tests (used to test circuitry that is not checked by BIT) have caused problems. Improvements have occurred throughout the years on IATS TPSs, hut some of these problems are inherent in the system itself and still remain. This is not to imply that the concept of AFTMATS is wrong. Far from it! The concept is an excellent one! BIT can he used to fault diagnose and isolate UUT problems effectively, as long as we do not swing from one extreme (conventional diagnostic testing) to the other (BIT testing with no diagnostics). The advantages of both methodologies need to be integrated in a way that allows for the goals of faster TPS development and execution times, while minimizing the engineering trade-offs.

Another impediment to the approach of removingldecreasing UUT functional tests that overlap with BIT tested functions, has been military specifications and contractual requirements. Military Specifications such as MIL-STD-1345, MILSTD-1519, MIL-STD-2076 and MIL-STD-2077 as well as the CASS Red Team Specification have required complete and thorough testing of UUTs based upon component failure modes. Most TPS development contracts in the past have required the generation of Fault Accountability Lists, detailing all UUT components along with all of the catastrophic failure modes for each of these components. In addition, requirements have been imposed that ensure that a “Ready For Issue” (RFI) UUT will operate in the next higher assembly (i.e., aircraft). This has been interpreted in the past by the military and contractors alike (and still is interpreted this way) as requiring full complete testing of all UUT functions and interface parameters. It is generally felt that this methodology is the only one that will insure that the final TPS he accepted by the customer. It is

IEEE AES Systems Magazine, December 1999 13

generally accepted today that military specifications are going away, (many are already obsolete) and the CASS Red Team Specification is in danger of doing the same thing. This allows for a smooth transition to this modified methodology of TPS implementation.

HOW DO WE GET THERE?

Well, now that we have made a case for restructuring TPSs so as to eliminate duplicate testing of component failure modes by both BIT and conventional functional testing techniques, we need to address what it takes to actually implement this change in new, modified and offloaded TPSs.

1 . First, we need to change TPS development contracts. Remove references to obsolete military specifications and the CASS Red Team specification, Change contractual requirements from “Find every catastrophic failure mode for each and every component or subassembly.” to “Ensure that the UUT will operate properly in the next higher assembly.” Institute the requirement that BIT testing be used to diagnose and fault isolate the UUT wherever possible. Encourage the usage of diagnostic tests, steered by BIT for the purpose of reducing ambiguity group size. Require that components/subassemhlies and functions that are inadequately tested by BIT be tested functionally under proper loading conditions. For new avionics under development, create a standard for BIT implementation. Create standardized BIT matrix and BIT documentation definitions. The purpose of this is to be able to provide BIT implementation information (such as which BIT firmware routine tests what functions, modes, components andlor subassemblies) to the TPS engineer. Put the requirement on the TPS engineer to implement this BIT oriented test philosophy. In order to verify that the UUT BIT functions were adequately analyzed and incorporated into the TPS, require the TPS development engineer to fill

2.

3.

-

.

.

.

.

in a modified “Fault Accountability List.” The information on this list should include: Firmware BIT function, UUT function tested, UUT components tested fully, partially or not at all, and supplemental functional test number for components/subassemblies and associated failure modes that are not fully tested by Built-In-Test under proper operating conditions. This would require the TPS engineer to:

Invest an engineering analysis at the front end of his TPS development task to analyze UUT BIT firmware and the related UUT circuitry; Determine what UUT circuitry is fully or partially tested by BIT, and what functional testing can be eliminated because it provides no additional information to the diagnosis process; Determine the level of isolation (ambiguity group size) for BIT related functions; Determine the UUT circuitry that requires additional supplemental functional and diagnostic testing; and Develop the supplemental tests as required.

CONCLUSION

Through the careful implementation of utilizing UUT BIT data derived from aircraft operations, along with the re-execution of the UUT BIT functions on the Automatic Test Equipment, it is possible to decrease the functional testing of UUTs. By applying an additional effort early in the test design stage, BIT routines can be analyzed and utilized for diagnostic purposes, paying dividends of shorter TPSs which are simpler to develop, integrate and maintain. This has an additional potential of simplifying ID complexity.

The bottom line benefits of this approach are: Reduced TPS development time; Reduced TPS run time; Simplified Interface Devices; Reduced TPS maintenance; Lower TPS Development Costs: and Lower Life Cycle Costs for Avionics Systems.

Systems Magazine Situations Wanted Column As part of the Board of Governors ongoing effort to assist A E S S members who are unem loyed, or soon to be unemployed, we

offer the opportuni to run a single paragraph, “Situation WantePad in Systems Magazine. !he ads should be sent to: IEEEAESS Systems Magazine, 4300 North Park Avenue, Chevy Chase, Maryland 20815: A T I N D. Dobson, or fax (301) 657-0209. Ads will run for two months at no charge.

Below is a sample format. Ads are strictly limited to a 60 word maximum. Names are optional: include a telephone andlor fax number where you can be reached hy interested parties. If you have questions, contact Tim Grayson at the address on Cover 2.

SAMPLE FORMAR

Electrical Engineer seeking Systems Engineering position, BSEE, graduate work in computer science. 14 years experience in the development of both military and commercial avionics. Extensive experience in the negotiation, development and allocation of system requirements. Familiar with both Mibtaty and FAA specifications. Proficient with PCs. Good verbal and written communication skills. Will relocate. Call (101) 234-5678.

I4 IEEEAES Systems Magazine, December 1999