System Vulnerability Review Report

Hope Creek Feedwater

December 2011

Station: Hope Creek

System Engineer: Walter Bischoff / Pete Pino

Date Completed: 12/09/2011

Station Challenge: TBD

CAP Tracking #: 80105125

1

Table of Contents1 Executive Summary………………………………………………………………

2 Purpose…………………………………………………………………………… 4

3 Scope……………………………………………………………………………....4

4 Team Members…………………………………………………………………....5

5 Methodology..……………………………………………………………………..5

6.0 Fault Tree Analysis/Single Point Vulnerability Findings…………………………6

6.1 Feedpump suction side vulnerabilities and enhancements………………………..7

6.2 Feedpump and discharge vulnerabilities and enhancements…...…………………8

6.3 Feed pump turbine vulnerabilities ………………………………………………14

6.4 Feed pump and turbine control and bearing oil package………………………...18

6.5 Number 6 feedwater heaters……………………………………………………..25

6.6 Operations Feedback…………………………………………………………….32

6.7 Maintenance Feedback…………………………………………………………..36

6.8 External operating experience

6.9 Simulator results…………………………………………………………………47

6.10 Corrective Action Program Items………………………………………………. 49

6.11 System Walkdown…………………………………………………………….…53

7 Scheduling Priority………………………………………………………………53

8 Vulnerability Elimination or Mitigating Strategies…………………………….. 54

9 Review of SPV Initiatives from Other Sites……………………………………..63

10 References………………………………………………………………………..65

11 Attachments

2

1.0 Executive Summary

The feedwater system vulnerability review team identified several latent vulnerabilities based on

deep and broader reviews of system design drawings, industry operating experience (OPEX),

EPRI topical reports, and initiatives, both previous and current, related to equipment reliability

(2005 scram derate, critical parts). The major system components included in this review were

the Reactor Feed Pumps (RFP), Reactor Feed Pump Turbines (RFPT), RFPT steam control

valves, Control and Lubricating oil auxiliaries, and the high pressure feedwater heaters.

Significant findings by the team consisted of the single point vulnerabilities associated with the

feedpump suction valve position trip, feedpump turbine moisture drain valve position trip, 2-out-

of-3 sensing trip logic that uses a single sensing header, the feedpump turbine thrust bearing wear

trip, the loss of feedpump power supply trip solenoid, Control and Lube Oil vulnerability for oil

pump auto swap failures, and oil pump discharge check valves.

The team recommendation is to eliminate most of the significant equipment vulnerabilities by use

of plant design change modifications. Examples of major modifications is to install additional

sensing lines for the 2-out-of-3 trip switches connected to a single sensing header line, remove

the position trip logics for the feedpump suction valve and feedpump turbine moisture drain

valve, install larger accumulators and faster redundant oil pump start modifications, and changing

the failsafe auto trip logic on a loss of power trip.

The identified deadlines have been evaluated per ER-AA-2004 and implementation should be

commensurate with the assigned risk ranking. . Also, strong consideration for augmented quality

assurance or additional supervisor oversight for SPV work during outages would be highly

beneficial for achieving breaker to breaker runs.

The team also identified other vulnerabilities in the BWROG report for Scram Frequency

Reductions. Such improvements will not eliminate a trip but will provide added margin and

defense in depth. The RFP suction trips are 2/3 logic and a have time delays, however the trip set

points for all three trains (A,B, and C) are all the same. Staggering the trip set points will prevent

a trip of all 3 pumps. If a low suction pressure transient develops, tripping one pump sooner will

prevent all 3 from tripping.

3

Some of the single point vulnerabilities (SPVs) identified by the team will not be eliminated and

will remain with new mitigating strategies (preventive maintenance) proposed to ensure high

reliability and risk management to plant operations. The cost benefit and feasibility of

elimination was balanced by the team in its final recommendations.

The team’s recommendation is for eight (8) new modifications, one (1) CM work order, thirty

(30) new PCR tasks, and one (1) procedure change activity to be implemented based on the

schedule risk category suggested by the team. The final recommendations will then be

incorporated into the CAP process and LTA Manager for management oversight.

2.0 Purpose

Hope Creek has experienced a declining trend in unplanned major power losses and will continue

to strive for excellence and intolerance for unexpected equipment failures. Industry experience

has shown that some scrams are initiators to more significant events and challenge both

equipment and operators.

Consistent with the Principles for a Strong Nuclear Safety Culture, A Questioning Attitude is

Cultivated, this action supports the attribute that anomalies are recognized, thoroughly

investigated, promptly mitigated, and periodically analyzed in the aggregate.

3.0 Scope

The scope for this vulnerability review included all major components and instrumentation

associated with the feedwater portion of the main condensate system.

The boundary included all components from the suction valves of the main feedwater pumps up

to and including the feedwater dual isolation check valves. The listing of major components is as

follows:

Reactor Feed Pump (RFP)

Reactor Feed Pump Turbine (RFPT)

RFP and RFPT lube oil system

RFP Recirc valves

RFPT control steam control valves

RFP, RFPT, and lube oil trip logic

4

RFP and RFPT vibration monitoring system

#6 Feedwater heaters (FWH) and associated level controllers and control valves

#6 FWH bypass Start up level control valve (SULCV)

Applicable portions of support systems:

Gland sealing steam

Instrument air

Fire protection deluge

Major components or systems excluded from this review were: digital feedwater control (DFCS)

logic and associated electronics (for example Field bus modules and computer processors). DFCS

is a double redundant system and there are no known fault modes that could provide a single

point of vulnerability. Condensate pumps and 1-5 FWHs were excluded from this review. FWHs

1-5 will be reviewed in main condensate system vulnerability review team. The other systems

were not considered based on not meeting the threshold of problematic systems during the

selection of systems for the ER HIT Team charter.

5

4.0 Team Members

Team Leader: Richard Cummins

System Engineer: Walter Bischoff / Pete Pino

ER HIT Team Lead Rudy Chan

Senior Reactor Operator: Michael Cline, Mariaz Davis

Operations Simulator Instructor: Gary Schmelz

Maintenance: Richard Chuck

5.0 Methodology

The approach used by the team involved several steps listed as follows to probe design,

operations, training, maintenance, and parts for potential vulnerabilities that could be latent to the

organization:

Identify single point vulnerabilities (SPV) components by reviewing system P&IDs,

instrumentation and controls schematics and logic drawings.

Review of simulator scenarios to understand the type of equipment faults used to

simulate plant transients and trips was performed to determine other single point

vulnerabilities not readily apparent looking at drawings. In addition, use of the simulator

to inject other component failures was used to verify or refute potential impacts to the

plant.

Reviewed operations procedures such as abnormal procedures, system operating

procedures, overhead alarm response, and local alarm response type procedures. The

review focused on identifying actions that the operating crew will take based on a single

indication that would result in a plant trip, derate, or shutdown. These single indications

represent single point vulnerabilities when operator action is taken instead of an

equipment type failure. The second focus of the procedure reviews is to identify

potential human induced errors when operating the equipment with lack of or deficiencies

in the level of detail with guidance or instructions.

6

Reviewed sample of maintenance work orders and procedures to determine the level of

detail contained in these documents that could affect the ability of the worker to properly

complete the tasks without relying on knowledge skills

Reviewed internal and external operating experience (OPEX), for at least the past five

years, to learn from the industry and past experiences onsite to determine if appropriate

actions have been taken to eliminate or mitigate the threat to plant operations.

Reviewed previous scram derate initiative reports to determine if previous actions or

strategies to eliminate or mitigate have been taken and to identify if the components we

properly identified as SPV.

A team field walkdown was conducted. The purpose of this field walkdown was to

visually look at the operating and environmental conditions of the operating system for

other potential vulnerabilities not previously identified or known to station personnel.

For example, the team will look for vibrations on equipment, water, oil, or steam leaks on

components, physical position vulnerabilities, and visual materiel degradation.

At the completion of the vulnerability reviews, the team will determine the best strategy

to eliminate the threat or mitigate by use of maintenance strategies (for example,

preventive maintenance), performance monitoring (for example, PdM), or procedure

changes. The philosophy for the team is to eliminate the threat to the greatest extent

possible and then mitigate as appropriate.

The team also referred to the guidance from the system vulnerability review process in

accordance with ER-AA-2004. In addition, the team also developed enhanced guidelines to

supplement this procedure.

6.0 Fault Tree Analysis/Single Point Vulnerability FindingsThe team focused on logic and components that posed a vulnerable threat to the station and

generation. A vulnerability was identified as a single failure that would result in a derate of

>20%, scram, or plant trip. The review only focused on normal 100% power operations. Pump

shutdowns and start ups were not included for this review. OE was used to help confirm a

7

vulnerability and to provide supporting documentation for the corrective action. The review

considered vulnerabilities for elimination or mitigating strategies. Elimination was the preferred

method but some components are required to protect the equipment and have no alternate

methods for maintaining the protective functions. Vulnerabilities of this nature can be mitigated

with PMs or additional margin in procedures or improved guidance. Some items already have

PMs and can not be eliminated and are mitigated to the full extent possible. These items will be

mentioned but may not include additional corrective actions.

6.1 RFP suction side vulnerabilities and enhancements

The investigation for the suction side of the RFP identified several vulnerabilities and

enhancements that can be implemented to improve reliability. Areas for improvement include the

RFP suction valve trip function and the RFP low suction pressure trip. These improvements

included recommendations from the SCRAM frequency reduction report in addition to polling

the industry for feedback.

6.1.1 RFP suction valve position 1/1 logic (vulnerability)

Each RFP is equipped with a suction valve H1AD –AD-HV-1781A (B,C). The valves are

operated from the control room with a push button. These valves are located in the turbine

building mezzanine on 153ft elevation upstream of the RFPs. These valves function to isolate the

RFPs during shutdown. The valve is a 20 inch motor operated gate valve and manufactured by

Pacific Valve Manufacturing. The valve is normally operated in the 100% open position. When

the valve is not 100% open, logic interlocks will trip the RFPT.

The team reviewed the valve components and determined the SPV threat was related to failures

of the switch contact, breaker control, power fuse, bailey module failure, or the breaker opening.

The switch logic is only a single 1/1 permissive with no time delay function. Therefore this trip

function will be considered a single point of vulnerability and evaluated for elimination.

The team considered the possibility of the valve drifting and actually changing position.

However, the valve is a motor operated valve and the team has found nothing to support failures

of this nature.

The team determined this trip function was intended to protect the pump from a low suction

pressure condition and pump cavitations. If the suction valve was closed via a pushbutton, there

8

would be a low suction pressure condition and it would damage the pump internals. However,

this valve is a motor operated gate valve and incidental closure was determined to be unlikely.

Each RFP is already equipped with a low suction pressure trip. The team determined the actual

low suction pressure trip provides more equipment protection than the suction valve not 100%

open trip. The team determined that the RFP suction valve not 100% open trip can be removed

and, replaced by an alarm.

6.1.2 A,B,C RFP low suction pressure trip (enhancement)

The team considered input from the BWROG SCRAM Frequency Reduction committee to

stagger the low suction pressure trips. The SFRC recommendation #16 is to stagger the low RFP

suction pressure trips. The action can be to either stagger the suction pressure set points or to

stagger the time delays until the trip actuates. Hope Creek has time delays of 10 seconds for each

RFP suction pressure trip; however the setpoints for all three pumps are set at 230PSIG. The

recommendation directs installation of time delays and either stagger the low suction pressure trip

set points or stagger the time delays.

However, the intent of this recommendation “is to avoid simultaneous trip of all feed pumps

when a single pump trip, or no trip at all, would suffice.”

With the pump tripping logic scheme described in Hope Creek’s response and review of Lesson

Plan NOH01MNCONDC-05 ,Condensate System, the Team has determined that the intent of this

recommendation is met because the designed intent of this logic is meant to keep as many RFPs

running as the condensate system can support.

Staggering these trips to prevent all 3 RFPs from tripping simultaneously should be implemented

at Hope Creek. The recommendation to change the time delay should not be implemented. In the

event a low suction pressure condition actually exists, the RFP could experience this damaging

condition for an extended period of time. The protective function of this trip is to stop the RFP in

the event a low suction pressure condition exists. This function can not be performed as it was

intended if the time delay is increased too long. If the trip set points are changed to a lesser time

delay, the margin of the time delay is reduced and more likely to trip under a false pretense.

Staggering the trip setpoints would be more effective. The current set point is 230 psig and the

feed pumps normally operate 350 psig. The suction pressures can be raised by 10 and 20 psig and

9

leaving the third RFP suction pressure as-is. This would satisfy the SFRC recommendation to

stagger the suction pressure trips to prevent a simultaneous trip of all 3 RFPs. Margin would only

be slightly reduced and the protective feature would not be challenged or lost.

6.2 RFP and discharge vulnerabilities and enhancements

This portion of the vulnerability review documented vulnerabilities found related to the pump.

The vulnerabilities for this section involved the vibration monitoring equipment

6.2.1 Vibration monitoring equipment (Vulnerability)

The RFP vibration monitoring equipment used to monitor RFP vibrations is a General Electric

Bentley Nevada 3500 model. On the pump side of the feedwater system, the vibration data

provides alarms and indication only. There are no vulnerabilities due to logic trip outputs from

the monitoring equipment.

Vulnerability exists due to the potential for Ops alarm response to cause an inadvertent reduction

in RFP speed to reduce vibrations. Ops procedures HC.OP-AR.ZZ-0028 and HC.OP-AR.ZZ-

0022 include guidance to response to axial vibration experience on the RFP and journal bearings.

10

11

The procedures instruct operators to reduce RFP speeds to maintain vibrations below the danger

levels. This practice is supported to protect RFP internal components. If the vibration data is false

indication, then operators will reduce RFP speed inadvertently. In several occurrences a probe has

failed resulting in a spike in vibration indication. The chart below shows the data that operators

observed shortly after a probe failed.

12

The team determined that an enhancement is required for operations to verify vibrations before

taking action to remove the RFP. The attached graph above was one example of how a vibration

probe can suddenly fail and show a sudden increase in vibrations. The actions should be a quick

verification check that can be done immediately. A lengthy evaluation can result in damage to the

pump internals if there is a real vibration. If the vibrations are due to false indication, operations

could derate the plant when it is not necessary.

Bearing temperature is a close indication of bearing vibrations. Sudden changes and rises in

bearing temperature are likely due to increased vibrations. Bearing temperature can be verified

quickly in the control room using CRIDS. If there is a sudden change in bearing temperature or a

rising trend in bearing temperature, the operators should remove/reduce the respective RFP to

prevent excessive damage. However if the bearing temperatures appear unchanged and unaffected

by the vibrations, then the RFP should stay in service at the respective speed.

6.2.2 RFP discharge Check valves (No vulnerability)

The team considered the discharge check valves for potential vulnerability. However the only

vulnerability that these valves present is during shut down and start up. A review of maintenance

procedures indicates an adequate preventative maintenance plan is in place. No vulnerability

exists for these valves within the scope of this investigation.

13

6.2.3 RFP minflow recirc line to the condenser (No vulnerability)

The minflow recirc valve was originally considered to be vulnerability. The minflow valve fails

open on a loss of air and a single failed air line would result in the minflow valve failing open.

The theory was if the valve failed open it would cause the full 5500 gpm flow to suddenly flow to

the condenser from the reactor. The sudden impact to operations would cause a low level in the

reactor forcing a derate to restore level. If level could not be maintained by a reactor derate, the

feed pumps would speed up and eventually trip on overspeed forcing a derate greater than 20%.

This failure was long thought to be a vulnerability and would cause a plant trip similar to INPO

document IER 11-10.IER 11-10, which referenced a minflow valve failing open at Palo Verde

plant while at 100% power. Palo Verde and Hope Creek differ since Palo Verde is a two

feedpump plant while Hope Creek has three feedpumps. In the IER, SONGS minflow valve failed

open diverting flow from the reactor. Inventory dropped in the reactor causing the speed

controller for the two pumps to speed up. As flow speeds up the pump with the failed minflow

valve tripped on low suction pressure, and the remaining pump could not maintain level.

The simulator refuted this theory. In the simulator, the minflow valve was failed open while at

100% reactor power. Level drops and the two other pumps speed up to compensate for the lost

flow and falling level. During this test, reactor level dropped to level 4 with a minimum of 25

inches with no scram, power reduction, or runbacks. Manual Operator action to SCRAM would

not be taken since a reactor level of 25 inches is above the procedural direction to SCRAM at 15

inches. The final outcome is all pump speeds increased to 5450 rpm (14.9 Kgpm) to compensate

for loss of flow to minflow line. The affected pump then only supplies 8.4 Kgpm.

6.2.4 RFP high-high discharge pressures (No vulnerability)

It was brought up during the study that there might be a vulnerability on the high-high discharge

pressure sensors, due to the fact that they are all connected to a single header, which could cause

all 3 to get a false reading from their common header, but this was dismissed due to the fact that a

shearing or disturbance of the header would cause a lower pressure reading than true, and would

not cause a trip.

14

The purpose for this trip was to prevent over pressurization in the event the Start Up Level

Control Valve (SULCV) fails closed during unit start up. All three of the pumps receive a trip

since any pump of the three may be in service during SULCV operation.

6.2.5 Feed pump discharge piping and bypass lines (Enhancement / No

vulnerability)

The team reviewed discharge piping of the RFPs up to the #6 Feedwater heaters. Specific

components and vulnerabilities reviewed in this section include the RFP minflow recirc line that

redirects water to the main condensers the start up level control valve (SULCV), and the RFP

high-high discharge pressure system. The team was able to utilize the field walkdown results for

further review of physical plant conditions, and the simulator to verify plant response to

transients.

The SULCV fails closed on a loss of air so in the event an air line fails the valve will remain

closed and have no impact on 100% power operations. Therefore the SULCV will not be

considered a SPV. During past outages the SULCV challenged plant shutdown and start up when

the air line failed requiring emergent work. The valve should be enhanced to prevent future

complications.

6.2.5.1 Start up level control valve (Reliability Enhancement)

The SULCV is a 12 inch air operated drag valve manufactured by “Control Components Inc”.

The valve uses a series of steel airlines and boosters to move the valve as required. The valve

fails closed on a loss of air. A demand signal is generated via a digital positioner manufactured by

Fisher. The valve is rated for up to 14000 gpm of water at 370 degrees F, and 1180 psig. The

valve is also equipped with an isolation valve that allows for maintenance and performing a

function check activity.

The valve is normally only used during shut down and start up. The SULCV function is to bypass

feedwater around the #6 FWH and control level in the reactor. The valve’s demand signal is

based solely on level upsets from the level set point. The valve normally controls level in

automatic control during power start up from depressurized up to approximately 15% reactor

power. During shut down the SULCV is called to service during plant cooldown. A valve failure

15

would result in challenges to an outage or planned shutdown, but not to 100% normal power

operations.

The team identified a reliability issue during the system walkdown. The air lines are steel tubing

and were vibrating due to vibrations resonating from nearby structures. A full operating cycle

with the connections and tubing vibrating puts stress on the tubing and fittings which could result

in an air line break. The air line break would have no impact on generation during normal modes

of operation. However during shut down, the valve would be unavailable for plant shutdown. On

start up, the valve would not function and would delay the start up.

Internal Operating experience has shown metal tubing is subject to failure if proper measures are

not taken to address the vibrations. In some instances, maintenance practices are to form a spiral

pattern with the tubing so the vibrating tubing won’t put stress on the connections and fittings.

Another approach has been to install flex fit tubing. Both approaches have been successfully

implemented throughout the feedwater system for the minflow lines and the seal water injection

lines.

16

The steel tubing should be replaced with flex fit tubing. This item was written up as part of the

walkdown but will not be considered an SPV. The SULCV will fail closed on a loss of air and

has no impact to full power operations. Since this valve is not in service during normal power

operations it was not considered a SCRAM or derate single point of vulnerability.

6.3 Feed pump turbine vulnerabilities:

The team reviewed the RFP Turbines (RFPTs) for any potential vulnerability. The team

determined this portion of the Feedwater system would benefit from additional PMs and design

changes. Some of the components do not have PMs and are subject to age related failures. The

vibration monitoring equipment would benefit from a design change to address a single point of

vulnerability due to a single sensor failure leading to a trip.

17

6.3.1 Thrust bearing probe and wear detector (vulnerability):

The team reviewed the vibration monitoring installation. It is a GE Bentley Nevada 3500

installation and was installed during RF15 in spring of 2009. The thrust bearing probes monitor

the turbines’ axial position. The shaft’s position is fed into the ‘thrust bearing wear’ detector

which alarms when the thrust exceeds the set point. If the shaft thrusts much further it will reach

the danger set point. At this time the RFPT is automatically tripped.

The thrust bearing installation is configured such that the thrust collar will thrust into the thrust

shoes and achieve a medium and will maintain a thrust -5 to -10 mils from the 0 mils position.

The trip is set for -30 mils. After a refuel the turbine is thrust against the shoes and is considered

‘zero’ed’.

The purpose for this logic is to provide protection to the RFPT rotating to station subcomponents.

Vendor information has determined that the thrust required for rotating to station subcomponent

interaction is far less for turbine components than pump side components. This is why the RFP

does not have a thrust bearing wear trip. The turbine rotating to stationary parts is approximately

65 mils to 75 mils. For the pump thrust, the clearance is 250 to 350 mils. The turbine rotating to

18

stationary parts interaction will occur in the turbine before it does on the pump. This is why the

thrust trip is on the turbine thrust and only alarms for the pump.

Vulnerability exists for this component since there is a single thrust probe installed for this trip

logic. A probe failure or shift will actuate a trip of the RFPT on ‘thrust bearing wear detector

Danger limits’. Internal operating experience has shown the vibration probes have failed before.

The probes that have failed before had no dependent trip logic but were for alarm and indications

only. The RFPT will trip if the thrust bearing probe were to fail.

The best strategy to address this vulnerability is a design change for elimination. PMs are

performed to mitigate the vulnerability; however the risk is still present and should be addressed

with a design change. The change should install a second thrust probe and modify the trip logic.

The trip should be converted to a 2/2 logic so both probes would have to fail to actuate the trip.

This will eliminate the 1/1 only logic.

Reference (498-950124-1) for a single thrust probe failure that resulted in a unit trip. The design

was similar to Hope Creek since there was a single probe providing indication. The false

indication drove Operators to reduce feedpump speed and trip the pump.

6.3.2 Feed turbine trip on a loss of trips system power (Vulnerability)

The team review of the RFPT trip logic determined that the RFPT will trip on a loss of power to

the trip system. The trip is integrated in the oil delivery system. On a loss of power, the 2

solenoids will open and redirect control oil from the control valve to the reservoir, closing the

RFPT control valves and tripping the turbine.

These solenoids are closed while energized, with 2 fuses supplying power. If either of these

fuses, or the power source is lost, the RFPT trip solenoid valves will fail open and dump oil back

to the reservoir tripping the RFPT. A single solenoid valve failure will also result in a RFPT trip.

The vulnerability exists on a loss of power or a fuse failure. Without power to the trip system the

pump will automatically trip and may not be the desirable outcome. Also if the power to the trip

solenoids is interrupted momentarily, they will lift resulting in a RFPT trip on low control oil

pressure. There are two solenoid valves and both should be removed while installing an alarm

that will alert the control room if power is lost instead of automatically tripping the turbine.

19

Manual trip capability is still available locally, giving operations the ability to trip the turbine

when necessary

The Following Drawing shows the normal flow path for hydraulic control oil and the loss of

power trip solenoid valves.

20

6.3.3 Include the rupture disc on the RFPT overhaul

The team reviewed other RFPT components and indentified the RFPT steam rupture disc as a

single point of vulnerability. Exhaust steam from each RFPT is directed to the main condenser

via a 60-inch header. These headers are each equipped with a breakable diaphragm which

ruptures at 5 psig pressure. These discs were supplied by the Delaval manufacturer. The purpose

for these diaphragms is to protect the expansion joints from over pressurizing.

21

The plant is vulnerable to a contaminated steam release into the turbine building if these rupture

disks fail. Failure modes of these rupture diaphragms include, but are not limited to; Cycle

fatigue, excessive pressure application of sealing steam to the RFPT, and filling of the RFPT

exhaust line isolation valve water seal. These discs can not be eliminated from the plant because

they provide a protective function to the RFPT expansion joints.

A review of the RFPT overhaul activity determined that there are no actions to inspect or to

replace the rupture disc. There is a note in the procedure that directs maintenance technicians to

inspect the rupture disc for any signs of wear or fatigue. An inspection of the surface will not

detect failure or wear from the inside of the steam header. Since elimination can not be performed

via a design change, the best strategy is to mitigate the vulnerability and replace it as part of the

RFPT overhaul. Replacing this component will prevent age related failures from occurring.

Reference (390-960428-1): An automatic reactor scram occurred at Watts Bar following loss of

both main feedwater pumps (MFP). A leaking valve on the B MFP train caused a MFP turbine

condenser rupture disk actuation. The actuation caused the loss of vacuum which tripped the A

pump and resulted in the SCRAM. The turbine automatically tripped, followed by a reactor trip.

The OE did not reference if the steam leak was directly responsible for the rupture disc actuation.

At Hope Creek there is no action to replace these discs. After reviewing this OE, it is apparent

that a rupture disc failure will cause a loss of condenser vacuum and will force a derate. These

components should be added to the Hope Creek RFPT overhaul.

6.3.4 Feed pump turbine steam exhaust bellow (vulnerability)

The team determined the expansion joints on the RFPT exhaust were a vulnerability. Each main

feedpump turbine is equipped with exhaust bellows to the main condenser.

Industry OPEX suggests that bellows type arrangements in steam exhaust or bleed systems have

failed prematurely. For example reference OE27323: During power operation rising dissolved

oxygen levels and declining main condenser vacuum were detected. A Steam Generator Feed

Pump Turbine (SGFPT) exhaust bellows developed leakage that required a downpower to repair.

The cause was determined to be age, vibration and high cycle fatigue related.

The team determined PMs were required to prevent a sudden failure such as the one referenced in

the OE. The PM strategy will mitigate the vulnerability but can not eliminate the vulnerability.

22

6.3.5 Feed pump turbine first stage moisture removal drain valve (vulnerability)

The team reviewed the RFPT first stage moisture removal drain valve, equipped on each RFPT.

Each valve is a Velan manufactured 6-inch motor-operated gate valve that drains to the main

condenser. The valves are operated from Control Room panel 10C651A, using the

OPEN/CLOSE, momentary contact push-button provided for each valve. These provide isolation

for the RFPT moisture drain line.

These valves are vulnerable due to trip logic when not fully open. The trip is a 1/1 contact switch

engaged when the valve is not 100% open. The team reviewed the valve components and

determined the SPV threat was related to failures of the switch contact, bailey module, power

fuse, or the breaker control failure. The logic is only a single 1/1 permissive with no time delay

function. Therefore this trip function will be considered a single point of vulnerability and

evaluated for elimination.

The basis for this trip is to prevent the drain valve from closing and allowing moisture to fill the

drain line and damage the RFPT blades. According to Salem station and other industry peers, this

trip is nonexistent and not required. Indication for valve position is maintained to ensure it does

not go closed, isolating the drain line. The team polled the industry peers and determined this trip

is not required. None of the responding plants with turbine driven feedpumps have this trip.

Without a strong basis for this trip’s function and purpose, the team determined the trip should be

eliminated. The trip is also a 1/1 logic and makes the station vulnerable. The team determined the

most favorable strategy is to eliminate the trip logic entirely. If the logic provided a more

reasonable protective function, the team would have considered a different strategy. Since there is

minimal basis and industry disposition for this trip to remain, a design change will be presented to

remove the trip.

6.4 Feed pump and turbine control and bearing oil package

The Lube oil system and delivery was reviewed by the team for vulnerabilities. The system is

comprised of 3 independent trains responsible for delivering lubricating oil to the pump and

turbine bearings. The oil is also used as hydraulic control oil for the steam inlet valves. Each train

is comprised of 2 AC driven oil pumps and an emergency DC driven oil pump. The emergency

oil pump provides oil directly to the bearings and only the bearings.

23

6.4.1 SSPV- Oil system design and electrical lineup (vulnerability)

The station’s oil system has a legacy design issue is due to a combination oil pump power supply

lineup and the design of the oil pump delivery system. Hope Creek has been subject to 2

SCRAMs in 2003 and 2007 due to the RFPT design issue that occurs when automatically

swapping oil pumps and the electrical configuration with 2 of the oil pumps on the same 480V

power supply. The cause of the RFPT trips is that the oil system design is not adequate to assure

that the standby lube oil pump will start and achieve operating pressure on loss of the operating

oil pump. This is an original equipment manufacturer design deficiency related to system margin.

Other contributing factors are due to the oil system being placed on and elevation 17 ft below the

turbine further reducing pressure margin.

6.4.1.1 SSPV oil pump electrical line up (vulnerability)

During the 2 SCRAMs an unexpected slow transfer of a 4 kV Class 1-E bus from the normal to

alternate source occurred during monthly relay testing. The slow transfer and subsequent loss of a

non-safety related motor control center resulted in the loss of an MCC set with 2 I/S oil pump

power supplies. The power supplies are configured as follows:

Pump 480V bus 4160V Bus

A1 10-B-323 “B” channel 4160 1E

A2 10-B-272 “C” Channel 4160 1E

B1 10-B-323 “B” channel 4160 1E

B2 10-B-313 “A” Channel 4160 1E

C1 10-B-272 “C” Channel 4160 1E

C2 10-B-323 “B” channel 4160 1E

The following line up must be used:

A1P124, B2P124 and C1P124

A2P124, B2P124 and C2P124

Procedure HC.OP-SO.AE-0001 contains the following operator work around to prevent making

the plant vulnerable to a Single SCRAM Point of Vulnerability (SSPV):

24

NOTE:To prevent a single 4KV bus failure causing more than one RFPT to trip due to loss of an oil pump, the preferred lineup for the pumps should

be:A1P124, B2P124 and C1P124 in service

ORA2P124, B2P124 and C2P124 in service

If this lined up is not used, the station is at risk of a single 4KV bus failure resulting in a

SCRAM. Both of the 2003 and 2007 SCRAMs occurred while the oil pumps were not in the

listed lineup. A slow power transfer occurred resulting in a 4KV bus trip. Two oil pumps tripped

due to the loss of power. After the 2007 SCRAM, the station implemented the corrective action to

operate with the listed pump lineup. This eliminated the SSPV for a SCRAM but did not

eliminate the SPV.

6.4.1.2 Design SPV for the Oil system swaps:

The station is still vulnerable to a design issue for the oil system. The oil system is designed to

allow, in the event of an oil pump trip, a transfer of pumps without a drop in oil pressure through

the implementation of an accumulator. After the 2007 SCRAM the oil accumulator was found

dry of oil after an inability to maintain oil pressure. Therefore, when an oil pump tripped, the

back up oil pump could not start fast enough to maintain header pressure. When pressure dropped

below the trip set point, the RFPT tripped on low control oil pressure. The loss of a feed pump

resulted in a unit derate. The vulnerability was determined to be due to undersized oil

accumulators and the auto start logic being pressure based and not instantaneous.

RFPT Control Oil Accumulators

The existing control oil accumulators are 10 gallon bladder accumulators that provide a reserve

supply of control oil to mitigate the effects of any hydraulic transients such as a sudden increase

in control oil pressure or a sudden decrease in control oil pressure. The accumulator is designed

only for a momentary change in pressure. During the past plant events when an oil pump trips,

the accumulator does not have a large enough reserve to compensate for the momentary loss of

oil pressure. After both events in 2003 and 2007, pressure dropped below the RFPT low control

25

oil pressure trip setpoint. The accumulators were found empty after the event. In RF17 a DCP

will be implanted to remove the 10 gallon accumulators and install two 40 gallon accumulators.

This will allow for maintenance and provide additional margin before the accumulator is fully

drained.

Oil pump auto start vulnerability:

The existing oil pump logic is configured such that the back up oil pump will only start

automatically when a low oil pressure condition is sensed. In the event that the operating pump

trips, header pressure will decay to a point where the standby oil pump will start on low oil

pressure. This auto start feature is not effective for addressing a sudden oil pump trip. A review of

header pressure data

26

- Item 1 is the control oil pressure header designed to start the back up oil pump at 100 psig and

trip the RFPT when pressure falls below 60 psig.

- Item 2 is bearing oil pressure and is designed to start the back up oil pump at 8 psig and trip the

RFPT when pressure falls below 5 psig. Both trips are 2/3 logics and the bearing oil trip has a 3

second time delay.

- Item 3 is the Main oil pump breaker going in the open position

- Item 4 is the back up oil pump breaker going in the closed position.

This data confirms that when the oil pump tripped there was a period that both oil pump breakers

are open allowing header pressure to further decay. The delay was not due to the breaker but

because oil header pressure decay is not linear and remains above the set point not clearing the

27

permissive for the standby oil pump to start. When pressure decayed to the start set point, oil

pressure decay rate had significantly increased, and the back up pump was not able to make up

for the lost oil pressure before reaching the RFPT trip set point.

6.4.1.3 DCP (80102874) to eliminate SSPV and oil pump swap SPV

A DCP has been issued for installation refueling outage RF17 in spring of 2012. Installation of

this DCP will eliminate the vulnerability caused by undersized oil accumulators and auto start

logic:

RFPT Oil Pump Auto Start

The Hope Creek Reactor Feed Pump Turbines (RFPTs) each have two oil pumps, one normally

operating (main) and one on standby (auxiliary), that provide high pressure oil to the control oil

system and low pressure oil to the lube oil system. On low lube oil or low control oil pressure,

the standby oil pump will start if in the automatic mode. This auto start will not prevent an RFPT

trip if the lube oil pressure falls below 5 psig. This (DCP) modifies the auto start feature of the

RFPT main and auxiliary oil pumps so that the standby pump auto starts on a trip of the operating

pump or on low lube/control oil pressure.

RFPT Control Oil Accumulators

The existing 10 gallon RFPT control oil accumulators do not provide a large enough reserve

volume of pressurized control oil to provide adequate protection in the event of low control oil

conditions. This DCP replaces each existing 10 gallon RFPT control oil accumulator with two 40

gallon accumulators to provide a larger reserve of control oil which increases the margin for

recovering from hydraulic transients with no adverse impact to the control oil system.

6.4.2 Lube oil pumps

The RFPT main and auxiliary oil pumps are DeLaval, 25 hp, positive displacement, submerged

suction pumps that provide high pressure oil to the RFPT hydraulic controls and low pressure oil

to the RFP bearings and the RFPT bearings. Both of these pumps are rotary vertical pump. The

flow path of the oil starts at the reservoir and is discharged through a discharge check valve,

where it combines with the oil discharged from the other pump (main or aux). The oil then goes

through a duplex filter and dual oil coolers used to regulate oil temperatures.

28

The normal control oil pressure supplied by these pumps is 125 psig. The normal bearing

pressure discharge is between 15-20 psig. Normally, one pump is in service (main) with the

second pump in stand by (Aux). The pumps are equipped with an auto start feature that auto starts

the Aux oil pump if the control oil header pressure drops below 100psig or the bearing pressure

drops below 8 psig. The emergency oil pump will start when pressure drops below 7 psig. The

purpose of the emergency oil pump is to protect the bearings from a loss of oil.

During the review, the team identified that the oil pumps had no preventative maintenance

activities scheduled. The pumps are classified as non-critical and should have a preventative

maintenance strategy. Vibrations are taken on these oil pumps quarterly and have shown a

degrading trend. Recently, the B oil pump was taken to the ‘restricted use’ category due to

elevated vibration trends. The aux oil pump was placed in service until the following refuel

outage when the pump is scheduled to be rebuilt. With the one pump in a degraded condition, the

RFPT is at risk. If the running oil pump stops, the station will be forced to operate with a

degraded pump.

A preventative maintenance strategy will be implemented to rebuild these oil pumps. The pumps

can be rebuilt while online using a spare pump and ordering new parts as required. This activity

can be performed while online. Then the pumps can be changed during the next outage.

6.4.2.1 Oil pump discharge relief valves:

The team reviewed the oil pump discharge relief valves. These valves are installed on both the

main and the aux oil pumps. The relief valves are 4 inch Fulflo pilot valves, designed to lift when

pressure reaches 170 psig. Their function is to prevent over pressurizing the oil pump discharge

line and dead heading the oil pump.

During the OE review, the team identified instances where relief valves lifted and resulted in a

sudden drop in header pressure. Relief valves’ internal components such as the spring can

degrade to the point that the relief valve will lift at a lower pressure than expected. If the valve

goes without inspection and verification that the valve will lift at the required pressure, the

internals could be degrading without any indication until it actually starts to prematurely lift.

The team determined that these relief valves require a preventative maintenance strategy to

mitigate the vulnerability. Elimination of these relief valves is not a favorable strategy since they

29

provide a protective feature for the oil pumps. A mitigation strategy is preferred to verify these

relief valves and internals will function as they are expected to. The springs must be verified that

they will lift at the required set point and not at a lower set point.

6.4.2.2 Oil pump discharge check valves (Vulnerability):

The team reviewed the drawings of the oil system and determined there were check valves on

both of the oil pumps discharge piping. These check valves were installed to prevent back flow of

the oil through the out of service oil pump, which would result in a loss of oil pressure and a

RFPT trip.

Industry OE has identified several failure modes for check valves. Check valves stick open and

the spring fails to the close the valve resulting in oil flowing back to the reservoir and tripping on

low oil pressure. Another failure mode is if the valve internals connection fails and the valve disc

breaks away. A return line would open and direct oil back to the reservoir.

Reference OE (341-020519-1 and SER 27-87) for events that caused a loss of oil pressure due to

oil check valves failing to close when the pumps were swapped. This caused a sudden drop in oil

header pressure resulting in the feedpump tripping on low oil pressure.

During the review, the team determined that the check valves for the oil pumps do not have

functional locations or preventative maintenance strategies. Immediate action was taken and a

notification was generated to create functional locations for the listed check valves. Once these

valves have functional locations they will be classified as ‘critical mild environment and mild

duty’. Actions from this review will also include implementing a preventative maintenance

strategy to perform check valve rebuilds and inspections.

6.4.2.3 Oil reservoir vapor extractor (Vulnerability)

The oil reservoirs have been equipped with a vapor extractor. This device functions to create a

vacuum within the reservoir cavity by applying suction to the reservoir. Oil fumes are extracted

through this device and out of the turbine building through a vent. This is a safety function that

must be maintained to sustain a habitable reservoir room. The extractor also provides the driving

force for draining oil back to the reservoir by applying a vacuum to the reservoir. This vacuum

force is intended to keep oil from leaking from the joints and seals.

30

Internal experience with the vapor extractor has proven it to be vulnerable to failure. In May of

2009, the vapor extractor structurally collapsed and failed. Shortly after the extractor failed,, the

harmful vapors accumulated in the oil room, creating a thin could of oil vapor in the air that

presented a fire hazard. Oil leaks from the feed pump and turbine became additional effects of the

extractor failure. Other OE expressed how oil leaks, if left not mitigated can accumulate. If the oil

accumulation is near any fire source, the oil can ignite.

To mitigate the May 2009 vapor extractor failure, maintenance technicians had to configure

blower and hose to exhaust harmful fumes to the turbine building truck bay. The extractor was

replaced but needs scheduled action to prevent the extractor from failing again. The risk was

rather low since there was no challenge to generation. Therefore a preventative maintenance

strategy would be the best strategy.

Reference OE 31000 and the Salem SPV review for how a vapor extractor can reduce oil leaks

and how oil leaks can result in a fire. OE 31000 confirms the risk of fire for leaking oil and is

important for the vapor extractor to perform without allowing degraded vacuum. In OE 31000 a

fire started due to oil leak ignition. The station allowed oil to drip and accumulate to the point

where it contacted a hot surface. The oil atomized and ignited burning the surrounded spilt oil.

The fire caused a SCRAM. The vapor extractor is designed to maintain vacuum on the reservoir

and be the driving force to direct oil back to the reservoir. If the vapor extractor fails or can not

perform its function the oil will be more likely to leak. Salem recently worked on their vapor

extraction line and found the exhaust pipe plugged with debris from the environment. Hope Creek

will add a similar action for a vapor extractor and exhaust line.

6.4.3 Low oil pressure trips with 2/3 logic with 1 sensing line (Vulnerability)

The low oil pressure trips for the oil system are for low bearing oil pressure and low control oil

pressure. Both trips are 2/3 logic and the low bearing oil pressure trip has a time delay to provide

additional margin. The logic and switches are not vulnerabilities, however all three switches use a

single sensing line. A loss of a fitting or connection for these switches could cause a trip. If a

connection or tubing fails, the oil pressure will drop and exceed the low pressure set point.

Changing these pressure trips will be difficult because the cabinet these switches are in is small

and there is not enough room to provide the switches with individual sensing lines.

31

Reference OE24195. A feedwater Pump tripped on low suction flow. The low suction flow trip

signal was caused by clogging of the common impulse line for three transmitters.

At Hope Creek the oil in this system is filtered and reduces the possibility plugging the oil line

with a piece of FME. However if the line is subject to sludge accumulation and build up similar

to OE33423, the station could be subject to a trip on low oil pressure. OE33423 was for oil sludge

build up in the oil line that resulted in elevated bearing temperatures because line blockage was

preventing oil from reaching the bearing. The blockage was determined to be oil sludge build up

from years without flushing.

The station will evaluate if the switches can support a different configuration and will implement

a PM to flush the system out to prevent sludge build up.

6.4.4 Oil delivery lines and sludge accumulation

The team reviewed the oil delivery system and piping and determined a vulnerability is possible

due to sludge accumulation. Oil sludge build up is an aging issue and can not be precluded with

FME controls. If oil sludge accumulation occurs in a smaller section of piping it can cause a

derate or degradation of a component that a derate is required to address the issue.

Reference OE33423 was for oil sludge build up in the oil line that resulted in elevated bearing

temperatures because line blockage was preventing oil from reaching the bearing. The blockage

was determined to be oil sludge build up from years without flushing. During the OE, the oil

accumulated in an oil line that fed a pump bearing. As oil sludge accumulated over the years the

bearing temperature slowly rose. However it was not an immediate concern because all

parameters were well within the required limits and the trend was very subtle. The bearing

temperature was addressed during a planned outage for an unrelated issue. During the bearing

inspection blockage was found in the oil line. This blockage was determined to be sludge and

residue accumulation. There was no FME in the reservoir or amongst the residue. Flushing the oil

system is required to maintain proper component lubrication so oil can flow freely through the

system.

6.5 Number 6 feedwater heaters

The 6A, 6B, and 6C feedwater heaters are two zone, horizontal, shell, U-tube heat exchangers.

These high pressure feedwater heaters provide the final stage of feedwater heating before

32

injection into the reactor vessel. Extraction steam from the 4th stage of the high-pressure turbine

is admitted to the shell side of the heaters where it is condensed as it supplies the heat for the

feedwater. The #6 heaters have shell side relief valves for overpressure protection, and internal

drain coolers. These drain coolers can be bypassed to dump directly into the condenser to

minimize the possibility of heater flooding.

Wide Range Level Transmitters

The wide range level transmitters provide FW Heater level (shell side) signals to electronic bi-

stable alarm cards and wide range indication function to the main control room, local panels

10A/B/C-C102 and CRIDS.

Narrow Range Level Transmitters

Feedwater heaters 6 (A, B and C) each have two narrow range (10”) level transmitters. One of

these transmitters provides input to the normal drain valve level controller and the other provides

input to the emergency/dump level controller for the heater.

6.5.1 Feedwater heater level control (Vulnerability)

The team investigated the 6 FWHs and their ability to control level within the heater shells. Each

Heater is equipped with four level transmitters which perform the following functions:

One level transmitter/level indicating controller is used to position the normal heater

drain valve.

A second level transmitter/level indicating controller positions the alternate drain valve

to the main condenser.

The third and fourth level transmitters provide local (panel 1A/B/C C102) and control

room (10C650A) indications, control room alarms, and heater trip functions. The local

heater level indication can be selected to either of these two transmitters.

Reference the following diagram for the level control span of the 6 FWHTR:

33

Level is normally controlled using the drain valves up to 8 inches.

Above 8 inches, the dump valve will being to lift to permit additional drainage directly to the

condenser. The dump valve will continue to lift as level rises until it reaches full open.

High level at 22.5 inches will cause the dump valve to fail fully open allowing maximum

drainage to the condenser.

High High level trip is at 29 inches. At this level the dump valve is fully open and the heater

will trip. There is a 10 second time delay for this heater that will prevent a momentary false trip

actuation signal from isolating the heater string.

For level control, level is maintained at a set point for optimal performance. If shell level is

higher than required the excess water will reduce the tube surface area exposed to the steam,

34

which will reduce heat transfer and result in cooler feedwater. Potential water intrusion into the

Main Turbine if Hi Hi level isolations do not occur automatically

Insufficient heater level could result in inadequate subcooling of the condensate which would

lead to flashing as it enters the lower pressure heater shell. This results in erosion of tubes and

other components in the heater and /or the drain lines if the normal level or dump valves are

malfunctioning.

If shell levels start to increase to abnormal levels, the associated alternate drain (dump) valve will

start to open to restore appropriate levels. The dump valve is modulated as a function of the high

level in its respective heater. Dump valve drainage is directed to the main condenser shell instead

of the 5th Feedwater heaters. If heater level reaches the "Hi" setpoint, the alternate drain valve

will open. The level control band for the dump valve is set higher than the level control band for

the normal drain valve. The normal control and high control bands do not overlap. The Hi level

setpoint is above (or near) the top of the high control band.

If the "Hi-Hi" setpoint is reached, the respective train 3-6 heaters will experience the following:

Extraction steam to that heater is isolated.

Cascading drain flow from the upstream heater is isolated (level control valve fails

closed

For the 1 and 2 feedwater heaters there is a 10 second time delay before the isolation to

occur.

Vulnerability exists for the level controllers and transmitters due to location. They are locked in a

high rad area and can only be accessed while the respective feedwater heater is tagged out of

service and cooled. This requires a derate of approximately 20% when planned. If a heater trips

unexpectedly the derate will be more than 20%. The transmitters will be placed outside of the

locked high-radiation area during RF17

The team also determined vulnerability exists for the level control transmitters that provide a

signal to modulate the drain valves. Internal OE of the normal level controller failing is

referenced in the timeline as follows:

35

4/19/05- The normal drain valve for 6A FWH AF-LV-1506A fails to control level. 6A FWH

level is being controlled by dump valve 1505A. Positioner replaced with same model. Failed

Positioner sent to Exelon Power Labs for failure analysis.

5/2/505- Exelon Power Labs concludes the 4/05 Positioner failed from vibration and side loading

of the valve spool by wear on the aluminum bell crank arm.

7/17/05- 1AFLV-1506A fails to control 6A FWH level and dump valve slowly starts to open.

LV-1506A slowly fails closed until fully closed. 6A FWH level maintained by dump valve (LV-

1505A). LV-1506A Positioner failed due to high vibration per System Engineering inspection

and comparison to 4/05 failure and Exelon Power Labs report. Positioner replaced on 7/23/05

with vibration resistant model.

8/16/05 - Control air tubing to LV-1506A Positioner pulls out of Positioner causing LV-1506A

to fail close. 6A FWH level controlled by dump valve (1AFLV-10505A).

8/21/05 - 6A FWH cascading drain line to 5A FWH observed to be vibrating at the piping run in

the 3/4/5A FWH room on remote video cameras 10. Piping vibration visually measured and

engineering calculation determined displacement is within fatigue limits.

8/22/05 - The normal drain lines for the 6B to 5B FWH & 6C to 5C FWH walked downs and

verified not to be vibrating. Walkdown performed in the 3/4/5-B/C FWH rooms.

8/24/05 - Troubleshooting performed to determine if cycling of LV-1506A is a cause of

vibration. LIC-1506A placed in manual for approx. 5 mins. Piping vibrations did not change.

8/25/05 - 6A FWH level increased by 1.2" to determine if increased sub cooling affects piping

vibration. Increased level maintained for approx 16 hours. Piping vibrations did not change.

8/28/05 Forced Outage - DCP added additional pipe support (hanger H06) to the 6A to 5A FWH

drain line to reduce piping vibration observed on 8/21/05.

9/6/05 - Engineering verified 6A FWH drain line is not vibrating as part of the retest for the DCP

that added hanger H06.

36

9/25/05 - Greater than expected noise was heard from the 3/4/5A FWH room. The remote video

camera showed greater than expected movement of the 6A to 5A FWH normal drain line, and

debris and water on the floor in the location of the 6A to 5A FWH drain line. An entry into the

room confirmed the noise was from hanger H05 on the 6A to 5A FWH drain line, the drain line

had excessive motion, the debris was pipe insulation from the same drain line, and the water was

due to leaks at the MSDT level control valves at the 5A FWH. NOTE The MSDT valves are not

connected to the 6A to 5A FWH drain line.

9/26/05 - System Engineering started the Complex Trouble Shooting process to identify all

possible failure modes and causes for excessive drain piping vibration.

10/4/05 - While at 100% reactor thermal power, the control room crew observed the 6A to 5A

Feedwater Heater (FWH) drain line MOV isolation valve (1AFHV-1508A) OPEN indication

signals were failing. A walk down was performed by operations and maintenance personnel in the

6A FWH room to measure drain piping movement. The piping movement measurements obtained

was approximately 1/8" steady state horizontal and 3/8" occasional horizontal peaks.

10/5/05 - An operational decision was made to remove the 6A FWH from service. After the 6A

FWH was removed from service, an inspection revealed the 6A FWH drain line MOV isolation

valve (1AFHV-1508A) operator hand wheel was found on floor. This MOV internal limit

switches were very loose and rotors damaged. The valve MOV hand wheel and internals were

repaired and returned to service. Vibration instrumentation (accelerometers) was installed on the

piping and valve in 6A FWH room.

10/7/05 - 6A FWH was placed into service using an operational evolution plan to raise overall

heater levels to approximately 17" for piping vibration reduction. The visual indications of

vibration were observed to decrease slightly. The FWH level was returned too normal operating

level band. Prior to placing the 6A FWH in service Adverse Condition Monitor (ACM) criteria

with established for piping displacement limits developed by PSEG and an independently

consulting firm on piping integrity analysis. The potential for two-phase flow to cause internal

piping erosion was evaluated. An assessment of NDE results during refueling outage 12 (RF12)

indicates no internal piping damage has occurred. The 6A FWH remained in service, piping

vibration were acceptable IAW ACM.

37

10/9/05 - The 6A FWH was removed from service due to visual observation of piping vibration

and conservative management decision-making.

10/13/05 - While at 98.6% reactor thermal power an Infrequently Performed Test & Evolution

(IPTE) for placing the 6A FWH in service. The IPTE was performed successfully to raise FWH

levels by 5" increments up to 25" dependent upon overall piping vibration reduction. This test had

vibration-monitoring instrumentation (accelerometers, acoustics, etc.) installed on the piping to

determine the source of the vibration (i.e. FWH, flow in drain, degradation in the FWH drain

cooler, etc.). A 6A FWH extended service decision flowchart was developed to determine the

next actions based on the results of the testing.

10/15/05 - Design Change Package (DCP) was developed and issued to increase the normal water

level into the existing dump valve level region (15" versus normal level of 5") by using the dump

level transmitter LT-1505A to provide the level input signal to both the normal LIC-1506A and

dump LIC-1505A level indicating controllers. The dump level controller setpoint was also

adjusted to actuate slightly higher than the new normal control level. The existing high-level trip

at 22.5" for the solenoid trip of the dump valve and the high FWH isolation at 29" were not

changed. The increased operating levels were also supported by written concurrence by the

vendor (YUBA). Prior to placing the heating into service an Operational and Technical Decision

Making (OTDM) and Adverse Condition Monitoring (ACM) plan were completed with

established FWH removal criteria. The DCP process was used for the level control modification

instead of the temporary modification process to provide higher level of process rigor.

10/18/05 - The 6A FWH was placed back to service at 98.6% reactor thermal power with higher

operating heater levels IAW the DCP. Visual observations of piping IAW ACM vibration levels

were acceptable at this power level. When reactor power was returned to 100%, the visual piping

vibration levels in the 6A FWH room were less than the ACM limits. Visual observations

indicated increased vibration in the 3/4/5 "A" room consequent with the power increase.

10/19/05 - The raw accelerometer piping data was analyzed for actual piping displacement. The

"Y" axis data peaked periodically exceeding the current ACM displacement criteria (0.038"

actual vs. ACM limit 0.036") and the "Z" axis data was not present on the tape.

38

10/20/05 - Data was collected and evaluated. The "Y" axis exceeded the acceptable ACM limit

of 0.036" (actual 0.038" - 0.042") and verified the "Z" axis accelerometer had failed. The 6A

FWH was removed from service IAW approved ACM displacement criteria and failure of a

monitoring method.

While 6A Feedwater heater was removed from service reaching 100% rated power was limited by

current operating procedures.

11/24/05 - Subsequent analysis performed by Westinghouse to allow application of Crossflow

Correction Factor with the 6A FWH OOS (GE Sub case 20051101-0308-1) with the following

stipulations:

• Crossflow can only be “Applied” in approved configurations of the extraction steam,

heater vents and drains system IAW HC.RE-RA.ZZ-0011.

• When the configuration of the extraction steam, heater vents and drains system changes,

the Crossflow Correction Factor should be closely monitored and HC.RE-RA.ZZ-0011

referenced to ensure that the configuration is approved.

• Reactor Engineering should be notified of any Crossflow alarms or significant changes in

the Crossflow Correction Factor (> 0.0050).

Reference OE 395-010501-1 and several internal events. A failed transmitter has resulted in the

level in the FWH rising to the point where the dump valve opens. In the OE referenced, a

transmitter failed and caused a slug of cool water being injected to the core and caused an

unexpected rise in power above 100%. Due to several internal events with failed transmitters

forcing down powers to repair, Hope Creek has set actions in place to upgrade the transmitters

and their configuration. This installation is to be worked under DCP 80103819

Past Hope Creek DCPs installed in RF 13 spring 2006

A past DCP restored the normal level transmitter LT-1506A input signal to normal level

controller LIC-1506A, returning the 6A FWH normal level control back to the original band of 0-

10. This DCP also removed accelerometers and cables from the 6A FWH piping on pipe

installed previously..

As a result of the troubleshooting associated with these issues the station has decided to replace

the level control valves in the 6A, 6B and 6C heater drain lines.

39

• Flashing in the valve internals of the 6A FWH LCV (1AFLV-1506A) is believed to be

the cause of the unacceptable 6A piping vibrations.

• The valves will be replaced with a CCI DRAG valves. CCI DRAG valves are designed to

solve this type of problem.

• The past plan was to replace H1AF –1AFLV-1506A and to remove hanger 1-P-AF-075-

H006 in the future refueling outage RF13. Valves 1 H1AF –1AFLV-1506B & C were replaced

during the following refueling outage RF14.

In 2011, Engineering evaluated a method for placing the level controllers outside of the FWH

rooms so if they began to degrade, they could be repaired without having to derate the plant and

removing the heaters from service to gain access to the heater room. DCP 80103819 has already

been issued and is scheduled for the R17 refueling outage in spring 2012. The team determined

the only required actions for this DCP is to verify it is installed as scheduled.

6.5.1.1 DCP 80103819 to reconfigure the feedwater heater level transmitters

The team reviewed DCP 80103819. This DCP replaces the current Feedwater Heater Masoneilan

Torque Tube Level Transmitters with Rosemount Differential Pressure Level Transmitters for all

four of the 6 FWH level transmitters. The replacement of the Masoneilan transmitters with

Rosemount differential pressure transmitters improves the reliability of the system and has been

implemented previously on the #2 FWH wide range level transmitters. To implement this change,

remote mounted diaphragm seals for the Rosemount Transmitters are bolted to flanges on the

sensing lines. The #6 FWH Rosemount Transmitters will be mounted on the lower process piping

outside of the FWH rooms at Elevation 137’ of the Turbine Building. The piping for the #6 FWH

process lines are routed from inside the rooms through a series of new core bores on the east wall

of the FWH rooms for mounting of the diaphragm seals. The level transmitters are mounted on

the exterior of the east wall of the FWH rooms to flanges attached to the process piping. New

valves are included for flushing of the lower sensing lines and venting the process flanges. This

provides access to the #6 FWH Level Transmitters without entry into high radiation areas if

needed. The basic functions of the feedwater heaters and the level transmitters are not changed by

this modification. This DCP only replaces the level transmitters, elements, other associated

components, and their location.

40

The following table provides the performance for the Masoneilon transmitters:

Table 4.1.5-1 Existing Masoneilon 12127AB Transmitter

Accuracy 0.6%

Response

TimeNot Listed

Radiation Not Listed

The following table provides the performance for the Rosemount transmitters per

VTD 135818:

Table 4.1.5-2 Replacement Transmitter Characteristics

Rosemount

3051CD2

(#6 FWH Narrow

and Wide Range)

Accuracy 0.1%

Response

Time100 ms**

Radiation None

The team has determined that this DCP will improve reliability of the level control function as

well as allow access to the level transmitters while online. This DCP will eliminate the

vulnerability that requires the plant be derated to gain access to address a failing controller. The

dump valve is still available to provide protection high-high level trips. Therefore the single point

of vulnerability for the 6 FWH will be eliminated.

6.5.1.2 LTAM item H-11-0057 Upgrade FWH WR Level Trip and Indication Circuits

The team reviewed additional initiatives to improve FWH level control. This initiative is

currently in the LTAM database and does not have a DCP. The initiative has been approved by

41

PHC to perform a conceptual study and is targeted to start in refueling outage RF18. The design

change will replace the Westinghouse 7500 electronic signal condition cards and alarm cards for

the Feedwater heater control Panels 1A-C-102, 1B-C-102 and 1B-C-102. The replacement will

consist of 72 cards, 24 cards per panel. It was proposed to utilize OTEK Model HQ-114 Digital

Programmable Intelligent Controllers (IC) or Foxboro 762 digital controllers. The intent is to

have each IC provide power, trip, alarm and indication for each wide range level transmitters.

The conceptual design for this change is currently in progress.

Reference OE26586, False Control Room Annunciation from OTEK HI-Q2000 Instruments. St.

Lucie Unit 1 and 2 utilizes over 300 OTEK indicators for measurement and display of plant

parameters. Several months after installation false alarms were occurring on the plant

Annunciator system. The Annunciator system uses 125vdc that interfaces with the OTEK output

alarm relay contact. Late in 2007, 3 more failures occurred with the OTEK meters output relays,

associated with circuits to small Agastat relays. The failure was increased circuit open contact

resistance with the OTEK alarm relays made by HANDOUK. The station corrective action

required them to replace the indicator output relays with a more suitable relay. The proposed

design uses Otek modules with dry relay contacts rated for 10A at 30VDC/240VAC. Otek rating

meets and exceeds the 2A rating for current 7500 card relays. Suitability with in panel repeater

relays will be checked in the DCP process.

Hope Creek’s design change proposal will address aging and obsolescence and mitigate the

vulnerability. This item still requires the concept study and PRC approval and is intended to be

implemented in RF18. The team determined the only action is to ensure the issue is approved by

PRC and to track it to implementation.

6.6 Operations feedback

The team considered the input from SROs for vulnerabilities they encounter while operating the

plant. No additional items have been added from Operation input. These items are required to be

reviewed quarterly for system health report inclusion. All of these issues are already being track

and have action for mitigation and elimination via the normal procedure. Reference procedure

ER-AA-2002 for system health reporting. The following excerpt was provided by operations for

this evaluation:

42

Components Off-Normal Report (includes interfacing systems)

o Reviewed off-normal and off-normal tagged reports (run date 09/16/11) with the

following components off-normal or tagged:

Off-normal not tagged NONE

Off-normal tagged The following components are tagged per HC.OP-

IO.ZZ-0003

H1AE -52-252073 MOV AE-HVF011A FDWTR INL S/O

H1AE -52-264062 MOV AE-HVF011B FDWTR INL S/O

H1AE -HS-AE-F011A FW SPLY LN A HV-F011A C/SW

H1AE -HS-AE-F011B FW SPLY LN A HV-F011B C/SW

Operator Burdens Assessment Quarterly Report

For operator burdens, the issue is tracked in the system health report until completion. The

issue can not be removed until Operator screening clears it from the burden report. None of

the issues on the operator burden assessment are a threat to the feedwater system and

continued 100% reactor power operations.

o Reviewed quarterly operator burden assessment report for 2nd quarter 2011:

The AE and AD systems were both categorized as LOW and showed no

change from the previous quarter.

Items identified as a burden that have an impact on the AE system were:

20483568 SULCV PDS Communication Failures (When in

MANUAL, a SULCV PDS communication failure will result in

closure of the SULCV (ODTM HC-2010-0011). MANUAL

operation of the SULCV will be minimized and training has been

provided to the operators. This condition will be corrected in

RF17).

a) Relation to AE system – SULCV is the primary

flowpath of feedwater to the vessel when power is <20%

20468349 Low Condenser Vacuum margin during hot weather

(During hot weather, increased monitoring of condenser

backpressure, CDI temperature, and SJAE performance may

result in the need to de-rate to ensure the heat input to the

43

condenser does not exceed the capacity of the cooling tower.

Operating procedures contain enhanced monitoring guidance.

The Condenser Backpressure alarm has been re-evaluated and

raised to 5.7” HgA which has greatly reduced the required power

maneuvers. This item is on the Engineering margin management

list.)

a) Relation to AE - RFP’s trip at 10”HgA.

20510973 Service Air compressor discharge Check valves

(Service Air Compressor discharge check valve failures have

caused Air Header transients and required entry into Abnormal

Procedure HC.OP-AB.COMP-0001. EQACE 70124136 was

completed 06/16/11 and was presented to CARB 07/11/2011.

Field order 600973714 is scheduled 11/14/2012

a) Relation to AE – RFP min flow valves fail open on loss

of air. SULCV fails closed on loss of air.

20461800 Excess CRIDS alarms (Continuously flashing CRIDS

alarms have potential to make identification of a real alarm more

difficult. This will be corrected by the CRIDS upgrade project –

due to complete 10/31/11

a) Relation to AE – As states above, continuous alarm

mask real issues.

TEMP LOGS – Reviewed the current list of temp logs. The following temp logs have

an impact on the AE system:

o Temp log 11-069 1BD483 / 1DD483 Inverters

CRIDS pt D4980 locked in alarm due to 1BD483 Load on Alt Source

from 1B1D473 on equalize

Loss of BD483 will result in a trip of the B RFP, as well as a loss of all

RFP PDS’s.

Aux Bldg operator is verifying no new alarms are present on BD483

2x/shift.

BD483 is currently on it’s alternate source 20525290, it appears it shifted

to the alternate after 1B1D473 battery was placed on equalize.

44

CRAB (Control Room Alarm Bypass) – The following alarm bypass will have an

impact on the AE system:

o 10-007 A Reactor Feed Pump Eccentricity 60093247

H1FW -1FWVY-3769A RFPT A SHAFT VIBR CONVERTER is

providing indications of vibrations when the pump is not inservice. This

vibration proximity probe was replaced in RF15.

Relation to AE – Currently there is no guidance in HC.OP-AR.ZZ-0020

for CRIDS points A2334 or A2331 which would be driven from FWVY-

3801A as fed from FWVY-3769A (reference M-31-1 G6). This could

lead to the incorrect removal of a RFP due to an erratic vibration point

ACM (Adverse Condition Monitoring)

o HC10-011 #4 TCV anti-rotation bearing 20483804

Ability of the #4 Turbine Control Valve (TCV) actuator spring can anti-

rotation bearing to maintain radial alignment of the Upper Guide (Item

#24 of GE Drawing 823E884 OR GEK figure 8-13). This Guide provides

axial alignment of the turbine control valve Linear Variable Differential

Transformers (LVDT). There are 3 LVDTs on the #4 TCV providing a

valve position signal to the Mark VI controller. At least two-out-of-three

LVDTs must provide a valid position signal to the Mark VI control

system. Failure of two-out-of-three LVDTs will result automatic slow

closure of the #4 TCV.

During implementation of WO 60087893 for repairing the anti-rotation

slot on the #1 TCV, the slots on the remaining 3 TCVs were examined

and the slot on the #4 TCV was noted as having similar wear to the #1

TCV 20483804. Engineering performed a follow up inspection and

concluded the following:

The slot for the #4 TCV has been previously weld repaired and is

moderately worn.

The anti rotation bearing is heavily worn and cannot be turned by

hand.

There is a large gap on one side of the bearing which is indicative of

the opposing side worn slot and worn/ flat area of the bearing.

45

o A Scope Change Request Form (SCRF) was processed to

perform the repairs in R16. The Outage Scoping Panel rejected

the SCRF on 11/2/10 due to the amount of hours/resources

involved to perform the repair

Relation to AE – The slow closure of #4 TCV could cause a Rx scram

due to high APRM flux thus challenging the operator to maintain level in

band

o HC11-014 C CW discharge valve drifting 20487634

The “C” Circ Water Pump Discharge Valve H1DA –DA-HV-2152C,

periodically drifts from the OPEN FULL toward the Open / Closed Mid

position resulting in unexpected Open/ Closed Mid position indication.

The alarm condition is occurring with about ¼ inch of valve drift in the

closed direction. Two conditions are believed to be contributing to the

issue: 1) The HV-2152C positioner actuator is leaking internally

allowing HV-2152C to drift toward Open/ Closed Mid position. 2) The

Open/ Closed Mid alarm / CRIDS indication is occurring early, most

likely due to degradation of the HV-2152C ZS-1 (Not 100% Open) snap

lock limit switch. The “C” Circ Water Pump Discharge Valve H1DA –

DA-HV-2152C drifting to Open / Closed Mid position could adversely

impacting Main Condenser Vacuum as a result of lowering total cooling

water flow. The intent of this ACM is to minimize challenges to the

Hydraulic Control Unit (HCU) for HV-2152C by minimizing the number

of times a OPEN FULL signal is given.

Relation to AE – Full closure of the HV-2152C from 100% power could

degrade vacuum to point where the 6.5” HgA Retainment Override of

HC.OP-AB.BOP-0006 Main Condenser Vacuum would require a Rx

Scram if Immediate Operator actions to reduce power are not taken

promptly enough.

o HC11-012 A Moisture Separator LIC-1039A output drift 20515490

The A moisture separator dump level controller, controls LV-1039A in

the event the normal range controller is unable to maintain level or stops

functioning. Level controller LIC-1039A is currently indicating 8.6% of

46

the dump range. This would mean that the level in the tank is above the

normal range transmitter and can be monitored in CRIDS with point

HC.A2622. However, the normal range is set to control LV-1364A (B,

C) at 30% of the normal drain range, and is fluctuating between 20 –

40% with occasional spikes; this can be followed with CRIDS point

A2624.

Relation to AE – High level in the Moisture separators is a trip signal

(2/3 input) to the Main TB, thus a Rx Scram. From 100% power the

scram recovery and subsequent Level 2 (-38”), HPCI/RCIC response and

potential overfeed to +54” (RFP trip) is a challenge to the RFP’s to

respond.

OTDM

o HC10-011- DFCS Startup Level Control PDS 20483568-

When in MANUAL, a SULCV PDS communication failure will result in

closure of the SULCV (ODTM HC-2010-0011). MANUAL operation of

the SULCV will be minimized and training has been provided to the

operators. This condition will be corrected in RF17).

Relation to AE system – SULCV is the primary flowpath of feedwater to

the vessel when power is <20%

The decision made was to continue to operate the DFCS with the SULC

PDS in automatic during plant startup and shutdown. A DCP is required

to replace the termination assembly and upgrade the FBM224 firmware

at the next system outage in RF17

The contingencies required were When the SULC PDS is in manual

during plant startup and shutdown an operator should keep the PDS

under observation and respond promptly to DFCS system alarms and

adjust valve position as required if the demand goes to zero. Specific

Operator Training was provided.

o HC11-008 10-A-104 Bus bus under voltage relays B-C tripped 20515932

This OTDM will document the risks associated with continued operation

of Hope Creek Unit 1 with tripped 4.16 KV AC Non-1E (NB) \ H1NB -

10-A-104 Bus bus under voltage (UV) relays until permanent repairs can

be implemented.

47

Relation to AE system – 10A104 bus undervoltage condition could

potential trip the C SCP thus challenging the abiltity of the RFP’s to

respond.

To minimize the risk of equipment damage or an unnecessary plant

transient the following actions were implemented:

A Temporary Configuration Change Package (TCCP) to defeat

the degraded bus UV trip logic for the C SCP.

Schedule fuse inspection and replacement, and or

troubleshooting of the UV Relay scheme for B-C Phases during

a bus outage in R17.

Continue to protect (via the clearance and tagging process) the

bus voltmeter selector switch as it is a potential cause of PT fuse

blowing (if the switch fails during use).

OPEVAL – No current OPEVAL’s have any affect on the AE system.

POD –

o II D. OPERATIONS CONCERNS – 10A104

Attempted to swap 10A104 in-feeds during last down power.

When depressed 10A104-AUTO CLOSE BLOCK would not extinguish.

Relation to AE – C SCP is powered from 10A104 bus, loss of C

SCP would challenge the RFP to respond.

o II G. NUISANCE (N) OR ILLUMINATED (L) ALARMS

120 VAC UPS TROUBLE – BD483 on Backup power

o II I. OTDM’s / ACM’s / Op Determinations

HC-2010-11 DFCS PDS loss of communications

HC-2011-08 10A-104 Bus Undervoltage relay de-energized (1 of 2)

H10-11 #4 TCV anti-rotation bearing

H11-012 A Moisture Separator Dump VLV

H11-014 C CW disch vlv drift

TCCP’s

o 4HT-10-046 Splice 1AP102 PCP “C” phase power supply cable to PCP motor

lead

48

A PCP pump tripped on start. Upon investigation it was discovered

that the ‘C’ phase power supply cable to the PCP motor lead was

damaged. "C" Phase Motor lead was terminated with a bolted

connection.

Relation to AE – Loss of A PCP will challenge the ability of the RFP’s

to respond

o 4HT-11-013 Defeat “C” SCP 1CP137 Under Voltage Trip

A Temporary Configuration Change Package (TCCP) to defeat the

degraded bus UV trip logic for the C SCP.

Scheduled fuse inspection and replacement, and or troubleshooting of

the UV Relay scheme for B-C Phases during a bus outage in R17.

Relation to AE – Loss of A SCP will challenge the ability of the RFP’s

to respond

6.7 Maintenance feedback

Maintenance feedback from turbine services was provided as follows:

Salem found their oil system vapor extractor lines were fouled (rust and beetles), which

impacted the operation of the system, and led to oil leaks. They created a PM for periodic

testing and cleaning. HCGS doesn’t have this.

One of the actions the team is going to implement is to perform a PM for the vapor extractors.

The PM will address this issue since it will inspect and repair the extractor as needed. At Hope

Creek there are no PM’s scheduled to perform this task and it will be added to the new lube oil

system PM.

Taking oil pressures at the turbine with it running can help determine how well the extractors

are functioning.

This task can be performed by installing a temporary gage on the RFP casing. The gage will

measure vacuum from the oil reservoir and perform this task. However this is not an SPV and

will not be included in the corrective actions.

49

HCGS has no spare rotating elements for the feed pump

HCGS has spare parts inventory issues with the turbine stop valves

Thermocouple reliability is questionable. There are indications that operations will remove a

pump if the limit is exceeded. A false indication could result in removal of a pump under false

pretenses.

This action was reviewed with maintenance and determined that maintenance practices may not

be thorough enough. When machine overhauls are performed that remove thermocouples, they

are not replaced and the seals are not replaced. Maintenance stated that the oil leaks in the turbine

room are probable causes of thermocouples and vibration probes that do not have proper seals

installed. Oil then drips out of the thermocouple port and accumulates over time.

The elements are placed back in the pump or turbine without being replaced. Several events have

resulted in bad indications from these points. Certain points can result in operations taking action

to mitigate the condition, including reducing RFP speeds.

Overhaul activities need to include actions to replace aged sensing elements and seals.

Maintenance activities will be changed to include new probes and seals to prevent probe failures

and other mitigate small oil leaks.

There are also actions from this report to include a procedure change that will prevent operators

from removing the pump from service. Normally when a thermocouple fails, the temperature

drops and would not require Operators to remove the pump from service. In some instances a

thermocouple has failed high where there is a momentary spike in temperature to the maximum

possible indicated value. The team determined that enhancements during maintenance overhauls

are required to ensure degraded equipment is not overlooked. Outage Services is currently

reviewing Hope Creek feedpump maintenance procedures for enhancements based on lessons

learned from Salem. Feedback has been provided to the Outage Services group for improved

maintenance practices.

6.8 OPEX review

50

The team reviewed the original OE review from the 2005 SPV review. The 2005 review has been

attached. The action items were captured under WO 80081141. The majority of the items were

alignment with the Exelon PCM templates. The team reviewed order 80081141 to verify that the

actions had been completed or were in progress. The report from 2005 has been included in this

report since. The original OE list was reviewed again to verify that no entries were overlooked

and that the actions generated from the review will mitigate or eliminate those vulnerabilities.

The team also reviewed OE dating after 2005 up to the most recent entries. In addition to the OE

reviewed before 2005, the team created a new OE review matrix that will also be included with

this review.

6.8.1 INPO - Industry Operating Experience (OPEX)

The following INPO documents were reviewed (SOER, SER, SEN, and topical reports) for

identifying vulnerabilities not found from review of design drawings and field walkdowns.

There were over 400 industry type OPEX reviewed by the team. The team reviewed the previous

OPEX list from the 2005 scram/derate initiative and included any new OPEX up until November

2010. For the most part, the OPEX reviewed provided validation of suspected SPV components

or vulnerabilities that the team identified during Phases 1 and 2 of the project and captured in the

vulnerability matrix included in this report. There were no noteworthy OPEX identifying new

SPVs. The team captured their review in an OE matrix included as part of this report. Each

OPEX found applicable was cross referenced to the most applicable component found in the

vulnerability matrix. Each of the vulnerabilities identified below have been captured in the

vulnerability matrix with appropriate elimination/mitigation strategies.

The team reviewed OE documents for applicability and then reviewed programs and procedures

implemented at the station to determine if a vulnerability existed. The following is a list of INPO

SEN’s

DocID Title / Issue HC

Applicable

HC Comments

SEN 271 Failures in redundant auctioneered No The backup power

51

power supplies for digital FW

control (DFWC) system resulted in

an invalid high reactor water level

trip signal. This caused main turbine

and FW turbines to trip and an auto

scram

supply failed without

indications to the

operating crew. Of

interest is that the

redundant power supply

degraded without the

plant being aware of it.

Hope Creek has

indication if the back up

and main power supply

fails.

No Vulnerability

SEN 248 On January 23, 2004, Calvert Cliffs

Unit 2 automatically scrammed from

100 percent power on low steam

generator level when one of two

turbine-driven main feedwater

pumps inadvertently tripped.

yes This SEN identifies

several equipment

issues. The cause was

the result of a spurious

electronic overspeed

signal because of a fuse

holder developed high

resistance due to local

corrosion. Hope Creek

requires two signals for

trip actuation.

No Vulnerability

SEN 199 On May 14, 1999, with Point Beach

Unit 1 operating at 100 percent

reactor power, the shell of feedwater

heater 4B ruptured, blowing

insulation and debris into the turbine

building. Control room operators,

alerted by the noise and a 5-

megawatt decrease in

yes Of interest in this SEN

is the shell rupture to

feedwater heater due to

wall thinning. The sites

FAC program precludes

this issue.

No Vulnerability

52

SEN 174 On September 6, 1997, both

McGuire units automatically

scrammed from 100 percent power

when the alternate supply breaker to

nonsafety-related 120-volt AC

instrument and control power bus

KXA opened, stripping control

power to several important plant

components

Yes In this SEN a backup

power supply breaker

failed that resulted in

the main feedpump to

trip due to loss of

power. This may be

similar vulnerability

with the power to the

trip system.

Vulnerability exists in

the ‘loss of power trip to

the FW trip system’

Reference 80105125

0200 for vulnerability

elimination

SEN 155 On January 26, 1997, Indian Point 2

was being shut down because of

level control problems on one steam

generator. During the power

reduction, the main feedwater

regulating valve for the steam

generator appeared to bind.,

No This SEN is not

applicable because

Hope Creek does not

have FRVs.

No Vulnerability

SEN 19 On September 6, 1987, with Davis-

Besse operating at full power, a

feedwater flow transmitter failed

causing the integrated control system

to increase feedwater flow to the

steam generators. This resulted in

cooldown of the reactor coolant

system.

No HC digital feed will fail

to the back up signal.

There are 2 transmitters

that feed a flow signal.

Also Hope Creek

DFWC will adjust RFP

speed with respect to Rx

level and there are 3

level transmitters.

No Vulnerability

53

SEN 4 On December 9, 1986, Surry Unit 2

was operating at 97% power when

the main steam isolation valve to the

"C" steam generator closed for

unknown reasons.

No Pipe failure due to wall

thinning should be

precluded from FAC

program

implementation.

No Vulnerability

The team reviewed OE documents for applicability and then reviewed programs and

procedures implemented at the station to determine if a vulnerability existed. The

following is a list of INPO SER’s

DocID Title / Issue HC

Applicable

HC Comments

SER 5-06 All reactor types have experienced

FAC events in piping systems. Even

with mature FAC programs, events

continue to occur. In addition, as

plants age, and many increase power

output, secondary plant conditions

and piping wear rates may change.

Yes This SER focus on

industry failures with

FW and MSR heater

shells and piping due to

wall thinning. The site

FAC program should

govern and provide the

necessary oversight to

prevent premature

failure.

No Vulnerability

SER 6-91 Several recent plant transients have

been caused by instrument air

system fitting failures. The following

contributed to these events:

Yes This SER focuses on

instrumentation failures

of fittiings and resultant

plant transients.

Problems related to

design deficiencies and

inadequate PMT

followiing maintenance.

54

Perform Walkdown For

vulnerability:

The AOVs have been

walked down to identify

any air supplies that

were not fitted with

Flex Fit tubing or the

‘spiral’ tubing

configuration. The

SULCV does not have

any of these

configurations.

SER 24-

88

This event is significant because it

demonstrates that during loss of

offsite power, the failure of either

stop valve on the main feedwater

pump turbine to fully close can

result in the destruction of this

turbine.

No The spring valves were

returned to service with

excessive spring force

and could not close and

isolate the turbine. Also

the steam valves failed

as is. HCs steam control

valves would fail closed

on a loss of oil. Oil

controls our valve

positions. This SER

focuses on damage to

the main feedpump

turbine when

stop/governor valves

fail to close following

loss of offsite power.

The SER stresses the

need to test turbine

valves.

55

No Vulnerability

SER 4-88 Following a transient initiated by a

feedwater flow transmitter failure

the reactor tripped due to a control

rod misoperation. Subsequent

recovery of the plant was

complicated by failure of a steam

safety valve to reseal, failure of a

13.8 kV supply break

No This failure was the

result of a single FW

flow transmitter failure

and caused the event. A

single transmitter failure

results in DFCS failing

to the remaining

transmitter for

indication and control.

There were several

issues identified in this

SER but the FW flow is

not a trip initiator.

No Vulnerability since

the DFCS adjusts RFP

speed with respect to

the Rx level before the

Feed flow.

SER 27-

87

A reactor trip occurred on low steam

generator level due to failure of the

train “A” main feedwater pump

discharge check valve with the “B”

main feedwater pump running. The

“B” feedwater pump discharge

pressure decreased to about 700 psig

due to back flow

Yes This failure was due to

maintenance. The valve

was reassembled with

the wrong clearences

causing FW to back

flow through a tripped

feed pump. HC has

similar valves, Though

there are also discharge

isolatoion valves for

feedwater pump

discharge, the check

valves must maintained.

56

Also this failure was

stated a lack of vendor

oversight resulted in this

failure.

Vulnerability exists in

the maintenance

procedures and

maintenance practices.

SER 1-87 Following a reactor trip from full

power, an elbow in the 18- inch

suction pipe to the "A" main

feedwater pump ruptured, injuring 8

workers. Four of the workers

subsequently died, and two were

hospitalized. Inspection of the

ruptured elbow revealed severe

No FAC issue and there are

programs in place for

this issue. In 1986,

Slurry did not have a

FAC that monitored

water but only steam

and 2 phase. Their

design did not allow

turbulence to disperse

before changing

direction. This SER

focuses on the trend of

high pressure pipe

failures on the suction

of main feedpumps due

to wall thining from

erosion/corrosion

issues. The siets FAC

program should provide

the oversight and

program strategy.

No Vulnerability

SER 2-86 During a power reduction from 40

percent power to perform a main

Yes Various equipment

failures resulted in this

57

turbine overspeed trip test,

secondary system instabilities and

multiple secondary system

equipment failures resulted in a

steam leak, loss of feedwater and

reactor trip.

SCRAM. Most were

PWR related and

beyond the scope of this

review (one event lead

to several subsequent

events) However a

steam relief valve on the

FWH could pose

vulnerability.

No Vulnerability since

Hope Creek has a PM

for this valve and

checks the

The team reviewed OE documents for applicability and then reviewed programs and procedures

implemented at the station to determine if a vulnerability existed. The following is a list of INPO

SOER’s

DocID Title / Issue HC

Applicable

HC Comments

SOER

88-1

Instrument air systems are typically

classified as non-safety-related

systems. However, both safety-

related and non-safety-related

systems use instrument air and have

been adversely affected by air

system failures.

Yes OE does not apply to

the extent this review is

investigating and should

be assigned to the

station air system

manager. This SOER

focuses on the

importance of air

qualitiy and reliability

of the instrument air

58

system and the affects

to plant transients and

trips.

Perform Walkdown For

vulnerability:

The AOVs have been

walked down to identify

any air supplies that

were not fitted with

Flex Fit tubing or the

‘spiral’ tubing

configuration. The

SULCV does not have

any of these

configurations.

SOER

84-4

Reactor trips and plant transients

initiated by main feedwater system

control problems are common

events. Reducing the frequency and

severity of these trips would improve

plant availability and reduce the

number of challenges to plant

protection systems

No This SOER focuses on

plant trips related to

control system problems

associated with FW

pumps and regualting

valves.

No Vulnerability. Hope

Creek does not have

FRVs

The team reviewed OE documents for applicability and then reviewed programs and procedures

implemented at the station to determine if a vulnerability existed. The following is a list of INPO

Topical Reports.

59

TR6-55 Review of Large-Pump-Related

Events Resulting in Scrams, S/D and

Outage Ext. Review of Large Pump

Related Events Resulting in Scrams,

Shutdowns, and Outage Extensions,

November 2006”

 No  No Vulnerability –

issues are captured in

this report.

TR4-41 INPO TOPICAL REPORT TR4-41

Review of Main Feedwater System

Related Events, November 2004”

Yes About 36 percent of the

events were attributed to

problems associated

with preventive

maintenance (PM). The

most common problems

noted included

inappropriate PM

frequencies, insufficient

actions identified in the

PM, inadequate

implementation of a PM,

or a PM activity did not

exist. Operation 0210

will be to verify the

results of the RF17

RFP/T overhauls to

determine the impacts

EPU had on the B RFP

and C RFPT including

wear and aging. Using

the as found conditions,

determine if the

overhaul frequency

60

should be extended or

shortened.

TR4-34 INPO TOPICAL REPORT TR4-34

Review of Feedwater System

Ultrasonic Flowmeter Problems,

March 2004”

No There are no ultrasonic

flow meters at Hope

Creek. However, Hope

Creek. continuously

monitors the correction

factor and provides an

alarm if a deviation is

noted. The station's

operators are provided

with Crossflow alarm

and procedures to

respond

once a Crossflow alarm

is received.

6.8.2 NRC IN reviews

The team reviewed OE documents for applicability and then reviewed programs and procedures

implemented at the station to determine if a vulnerability existed. The following is a list of NRC

Information Notices (IN):

DocID Title / Issue HC

Applicable

HC Comments

IN 2008-

09

Turbine-driven Auxiliary Feedwater

Pump Bearing Issues

No These issues were for

Aux feedwater pumps

and not the main Feed

pumps.

No vulnerability. Hope

Creek does not use Aux

feedwater

IN 2008- Main Feedwater System Issues and Yes Loss of power to the

61

13 Related 2007 Reactor Trip Data trip system will cause

trip solenoid valves to

open tripping the RFPT.

Vulnerability will be

addressed under

80105125 0200

IN 2010-

20

The lessons learned from these

events may apply to turbine driven

pumps in other systems such as

reactor core isolation cooling and

high-pressure coolant

injection systems.

Yes Plants identified in this

IN did not properly test

the control and trip

functions of the system

during plant start up

such as over speed and

control valve stroking.

No Vulnerability

because Hope Creek

strokes the control

valves on start up to

verify full valve travel.

Hope Creek also

performs electric

overspeed tests on start

up.

6.8.3 BWROG SCRAM frequency reduction report:

The team reviewed the SCRAM frequency reduction report and determined that the only

applicable item was recommendation 16 ‘install a time delay on the RFP low suction pressure trip

and stagger the RFP low suction pressure trip set points OR the respective time delays’. The basis

for this recommendation is that the staggered trip actuation will prevent all three RFPs from

tripping on low suction pressure. In the event that a low suction pressure condition actually

occurs, the staggered trips will shed a single feedpump at a time instead of all 3 at once.

62

Hope Creek already has a 2/3 trip with a 10 second time delay. However all three switches for all

three RFPs will actuate at 230psig. The suction pressure trips should be staggered to prevent a

sudden simultaneous loss of all three RFPs. Increasing the time delay could cause damage to the

RFP if the low suction pressure exists. The pump would be subject to pump cavitations. Reducing

the time delay would significantly reduce margin for the RFP trip time delay. Increasing the

suction pressure trip pressure set point would also reduce margin. Reducing the suction pressure

trip set point would allow the respective pump to operate at a lower suction pressure and increase

margin but make it more vulnerable to cavitations. The best mitigating strategy will be to stagger

the pressure set points. Increase one to 250 psig, increase the second to 240 psig, and leave the

third one as is. This was chosen because the team knew that damage was possible if the suction

pressure set points were lowered or the time delay was increased, and there was little margin to

decrease the time delays. Increasing pressure trip setpoints did not increase damage risk, yet left

enough operating margin for uninterrupted operation. This action will be captured under

80105125 operation 0150.

6.8.4 EPRI documents review:

The team reviewed EPRI documents for applicability and then reviewed programs and procedures

implemented at the station to determine if a vulnerability existed. The following is a list of EPRI

documents and dispositions.

EPRI

1884-10

Document provides a guide for

proper pump warm up before being

placed in service. The document

included a section for reducing

thermal shock and water hammer.

Yes Hope Creek has had

complications in the past

while starting up and

seizing a RFP internals.

The breakdown bushing

would seize to the shaft.

The vendor documents for

the HC RFPs gives a limit

of 40 degrees F for delta

temperature across the top

and bottom of the case.

The report referenced a

failure to properly warm

the RFP resulting in

63

thermal shocks.

No vulnerability. Hope

Creek has implemented

procedure changes to

prevent inappropriate

warm up of a cold

feedpump.

EPRI

1003094

Mechanical hydraulic controls

(MHC) used on feedwater pump

turbines (FWPTs) as installed in

nuclear power plants are a high

contributor to plant capacity derates.

Specifically, the controls and

associated lubrication systems have

produced chronic problems in the

FWPT applications. The majority of

FWPT trips have resulted in tripping

the plant off-line. The mechanical

hydraulic controls are also widely

used in large capacity fossil power

plants as the control for the prime

mover on the boiler feedwater

pumps.

Yes Hope Creek has

determined that the

Lubricating oil system is

in need of additional PMs

for components that are

not normally service. For

instance the team has

already identified check

valves that do not receive

preventative maintenance.

Vulnerabilities were

captured under the

80105125 order has

operations to improve the

lube oil preventative

maintenance program for

this system.

EPRI

1021066

Mechanical hydraulic controls

(MHCs) in nuclear plant feedwater

pump turbines (FPTs) are a high

contributor to plant capacity derates.

The purpose of this study was to

evaluate MHCs’ degradation over

time and life-limiting properties in

order to provide guidance about their

Yes Review determine that the

majority of applicable

failures were attributed to

the following Age

—failures attributed to

normal wear, insufficient

PM, the unsuitability of a

component

64

expected life span and the point at

which major refurbishment or

replacement should be considered.

Insights from this study should

provide readers with life-cycle

guidance that helps in the

management of their facilities’ long-

term operating strategies.

for its application, valve

leak by, and other

mechanical issues

• Sticking/binding—

failures related to the

sticking and/or binding of

linkages and valves due

to issues like

misalignment and wear

• Oil quality—failures

attributed to

contamination in the

hydraulic/lubricating oil

due to dirt

and contamination and

buildup/deposits/corrosio

n on components

Vulnerabilities were

captured under the lube

oil PM program to be

implemented under

80105125

EPRI

105933

This guide provides basic

information on the design ,

construction and maintenance of the

main feedwater pump equipment

supplied by the six pump

manufacturers to the domestic

United States nuclear power

generation industry. This guide is

intended to provide useful

information to all disciplines and

skills associated with the

Yes This EPRI document

provided general

information for feed

pumps and oil delivery

systems throughout the

industry. This review

included material for all

models and styles of

feedpumps including

motor and turbine driven

feed pumps. Most of the

65

maintenance of main feedwater

pump equipment, the planning of its

maintenance, and the monitoring and

evaluation of its performance.

information in this review

was applicable to Hope

Creek but had already

been captured under other

SPV entries.

No vulnerability

6.8.5 Industry Operating Experience

There were over 400 industry type OPEX reviewed by the team. The OEs reviewed dated as far

back as January 1986 up to November 2011. During the OE review, the team determined that

OEs containing Feedwater Regulating Valves (FRV), Auxiliary Feedwater, and motor driven

pumps were not considered applicable to Hope Creek. The team found 74 OEs applicable to Hope

Creek. The failures were categorized by failure to include electrical, I&C, mechanical, and others

related to human error. Failures are as follows:

- There were 5 failures that were related to electrical issues

1 connection failure

4 fuse failures

- There were 23 failures related to I&C issues

2 circuit card failures

11 controller failures

4 power supply failures

5 switch failures

1 transducer failure

- There were 32 failures related to mechanical issues

5 corrosion and erosion failures

4 oil leak failures

1 steam leak large enough to force a derate

1 water leak large enough to force a derate

2 linkage failures

1 impeller failure

1 shaft failure

11 valve failures (includes AOV, MOV, isolation, and control valves)

66

- There were 14 failures categorized as ‘other’ for failures not directly related to component

failures

3 design failures

4 FME induced failures

5 human Error failures

1 maintenance induced failure

1 procedure induced failure

For the most part, the OPEX reviewed provided validation of suspected SPV components or

vulnerabilities that the team identified during Phases 1 and 2 of the project and captured in the

vulnerability matrix included in this report.

During this review, the team noticed a subtle trend in OE failures and their complexity. The OEs

reviewed included failures dating back to as early as the late 1980’s. The OEs that caused

SCRAMs early on were related to single failures such as a false switch actuation, fuse failure, or

connection failure usually caused by lack of PMs and procedural guidance. The more recent

failures are the result of a chain of events that force the derate. For instance OE33423 was the

result of oil sludge accumulation in the supply piping. Sludge build up caused bearing

temperatures to rise and forced the feed pump to be removed for repairs. Or in some instances the

failure was an overlooked subcomponent that was overcome by aging. A PM was being

performed on the component, but there might have been a subcomponent that was not replaced or

configured incorrectly. The subcomponent fails, the component fails, a transient is experienced in

the system flow/level control and results in a derate.

6.9 Feedwater Simulator Results :

The simulator was used by the team as additional input in determining whether the component

should be considered a single point vulnerability. It is understood by the team that limitations in

simulator modeling in the secondary systems may not provide sufficient response to clearly

discern an SPV.

All simulator faulted scenarios associated with feedwater induced transients or plant trips were

reviewed. The conclusion from this review did not identify any new or different equipment

failures not already considered on the vulnerability matrix. However the simulator actually

67

eliminated the failure of the minflow recirc valve. This valve was originally thought to be an SPV

but is not after verifying simulator data.

6.9.1 Trip of 6A FWH from 100%

The team simulated a loss of the 6A FWH at 100% Rx power.

20 degree reduction in feedwater temperature. Requires Ops to reduce power by 20% per

HC.OP-AB.BOP-0001. Reactor power rise without operator action was 3% to 3967 MWth.

6.9.2 Trip of in-service RFP lube oil pump

Swap occurs to standby pump per design. Station vulnerability is not modeled. The station

vulnerability would cause a trip of the RFP since the back up oil pump can not restore oil

pressure before the trip set point is breached.

6.9.3 Loss of speed input to Woodward governor (SE-3749)

Sensed as control signal failure. RFP control auto-transfer to manual. No impact on operation.

Several plants have had experienced a loss of speed input to the controller which has resulted in

the loss of a feedpump, however this is not a vulnerability at Hope Creek.

6.9.4 Min flow valve fails open

No scram. No power reduction. Level lowers to level 4 at 25 inches which is above the 15 inch

Operator procedure limit to manually SCRAM. There were no runbacks since this is not a trip.

All 3 pumps feed pumps speed up to 5450 RPM to compensate. Those pump flows go to 14.9

Kgpm. Affected pump flow lowers to 8.4 Kgpm. Simulator models flow being restricted to min-

flow line capacity of 5000 gpm.

6.9.5 Fail open start-up level control valve

Power rises to 100.6%. No level transient.

6.9.6 RFPT Trip

Level 4 – recirc runback. Power lowers to 71%. Lowest level seen was 27”. Operators manually

trip the unit if level drops to 15 inches.

6.9.7 Transmitter failures – could not simulate

68

Drawing review with instructor.

FT-N011 – only gives flow indication at DFCS. No impact to system

FT-1755A – indication to CRIDS only. No impact

FT1800A – input for min flow controller, also low flow trip vs. speed. Min flow valve would

open on low failure.

FT1770A – min flow controller setpoint input to controller. No impact.

6.9.8 Electrical – loss of 10-B-313 or 10-B-323

In the accepted proceduralized lineup- RFP lube oil pumps swap as designed. This lineup

maintains availability of power to standby pumps.

Power supply to feed pump lube oil pumps:

1A1P124 – 10-B-323 (“B” 4160 1E source)

1A2P124 - 10-B-272 (“C” 4160 1E source)

1B1P124 - 10-B-323 (“B” 4160 1E source)

1B2P124 – 10-B-313 (“A” 4160 1E source)

1C1P124 – 10-B-272 (“C” 4160 1E source)

1C2P124 – 10-B-323 (“B” 4160 1E source)

Loss of 10-B-323 will trip three pumps, if running.

Loss of 10-B-272 will trip two pumps, if running.

Allowable lineups prevent loss of one 4KV bus from tripping two pumps (one pump from each

1E bus).

A1P124, B2P124 and C1P124 in service

Or

A2P124, B2P124 and C2P124 in service

Note that B2 pump always runs. PM is being generated to eliminate the vulnerability of having

the B2 pump constantly operating. Order 70123865 has action to create a PM that will swap the

RFP/T lube oil pumps quarterly to prevent excessive pump use.

69

6.9.9 Loss of 1B-D-483 inverter:

“B” RFP Trip

Loss of power to control PDS’s. and control

Manual control available (no procedure) at 10-C-612 Woodward station.

Level dropped to 15” before recovering. Operators would have manually scrammed. Recirc

runback due to pump trip and level 4. Possible single element control due to loss of steam flow

signal – which gave slower response for two remaining feed pumps.

6.9.10 Loss of 1A / B-D-318 125VDC Busses:

Respective RFP trip on loss of trip circuit power. Single Recirc runback. Final power 82%.

Loss of main power transformer cooling for loss of 1A-D-318 will likely result in taking unit off

line if alternate power is not selected.

Simulator cannot run loss of 1C-D-318. This would cause turbine trip, due to loss of DEHC.

6.10 Review of past internal CAP items

The team performed a review of past corrective actions, corrective maintenance activities that

have not been performed, and open CAP items. The purpose for this review was to verify there

were no items that were overlooked that could challenge that station if not mitigated. A review of

current open evaluation activities was performed to verify all tasks have been assigned and the

evaluations are the appropriate level and are not aged beyond 100 days. Past evaluations were

also reviewed for accuracy and timely implementation of corrective actions.

No additional actions were generated from this review. All maintenance items are being properly

scheduled and planned for implementation for earliest opportunity to repair. All the evaluation

activities’ age are under 100 days which is the requirement for excellence. The past evaluations

were closed out and the mitigating actions are being performed to mitigate degraded conditions.

6.10.1 Review of open Maintenance activities

The team reviewed open Maintenance activities. The majority of the feedwater corrective

maintenance actions have been scheduled for the next possible outage. Scheduling considers the

‘risk’ that could be sustained by the plant. Risk is considered ‘impact’ and ‘possibility’. If an

activity is too high of risk to the plant it will be scheduled for the refuel outage.

70

Most of the scheduled maintenance activities must be performed during R17 due to high radiation

or the requirement to tag a power supply or a pump out of service to gain access. Most of these

items were generated from failing indication trends or equipment that was found to start showing

signs of degradation. There are no major equipment failures on this list that caused a derate or

challenge to the station. The team was unable to produce any additional items for the system.

This list was generated from SAP:

Order Functional location Description

60060327 H1AE CONTCY-1A/B-S-105- JUMPER ZERO SPEED SW

60077333 H1CJ -1CJTE-

3181C1 C RFPT THRUST BRG TEMP IND FAILED

60081975 H1AE -AE-HV-

F032A (CTGY) AE-HV-F032A: CLEAN/INSP VLV(MOV)

60090512 H1AE -1AELV-1785 R16 80101747 INSTALL LOW DRAG PLUG SEAL

60091396 H1AE -AE-HV-

F074A F074A - CONTINGENCY VALVE REPAIR.

60095662 H1AE -1AEFI-

R603A-C32 RPLC -H1AE -1AEFI-R603A-C32 PER 80098173

60095663 H1AE -1AEFI-

R603C-C32 RPLC -H1AE -1AEFI-R603C-C32 PER 80098173

60093692 H1AE -1AEVT-

7908B6 REPLACE H1AE -1AEVT/VE-7908B6

60099210 H1FW -1FWYIS-

BC653081005 DIGITAL POINT D2005 COMING IN/OUT dn

60099310 H1AE 80100455 / 1SBPISH-N652D REVISE SETPOINT

60097044 H1AE -1C-E-106 H1AE -1C-E-106 REPLACE FAC PIPING

60094237 H1AE -1A-P-101 INSULATION FALLING OFF FEEDPUMP PIPING

60099159 H1AE -1AELV-1785 RPLC- 1AELV-1785 SULC LOWER BOOSTER

60096939 H1AE -1AETV-

1796A A-RFP SEAL WATER VALVE DEGRADED

60096941 H1AE -1AETV- B-RFP SEAL WATER INJ VALVE DEGRADED

71

1796B

60089122 H1CJ -1CJ-

HS3178C2 TS&R H1CJ -1CJ-HS3178C2

60094805 H1AE -1B-P-101 TROUBLESHOOT B RFP OSCILLATIONS

60092954 H1AE -1AETIC-

1796C 1AETIC-1796C/UNABLE TO BRING INTO CAL

60096240 H1FW -1-FW-V063 1FW-V063 LEAKING BY SEAT

60097899 H1CJ -1CJTI-3160C REPLACE H1CJ -1CJTI-3160C

60091631 H1AE -1AEZT-

1783A A RFT RECIRC VLV POS IND ON COMP POINT

60097807 H1CJ -1CJTI-3157B RPLC- 1CJTI-3157B TEMPERATURE IND

60097808 H1CJ -1CJTE-

3180A1 A-RFPT/CRIDS A2340: RPLC 1CJTE-3180A1/A2

60097898 H1CJ -1CJTI-3153B REPLACE H1CJ -1CJTI-3153B

60093691 H1AE -1AEVT-

7908A6 1A-P-101 REPLACE H1AE -1AEVT/VE-7908A6

60095660 H1AE DFCS FBM UPGRADE DCP 80103280

60095910 H1CJ 'A' RFPT CNTRL OIL ACCUM 80102874

60097743 H1CJ 'B' RFPT CNTRL OIL ACCUM 80102874

60097744 H1CJ 'C' RFPT CNTRL OIL ACCUM 80102874

60099307 H1AE 80100455 / 1SBPISH-N652A REVISE SETPOINT

60085831 H1FW -1FWZTS-

1794B H1FW -1FWZTS-1794B FAILING INDICATION

60093247 H1FW -1FWVY-

3769A TS&R 1FWVY-3769A AND 1FWVE-3769A

60097912 H1AE -10-C-612 RP- 10-C-612 CARD RACK RAILS

60099155 H1FW -1FWZY-

1794C C RFPT CV POS IND DEGRADING A2315 dc

60083138 H1AE -52-113023 H1AE -52-113023/REPAIR STAB BLOCK

60095698 H1CJ -1CJHS-

3177B2 RP-1CJHS-3177B2-"B" RFP OIL PMP CNTRL SW

60097734 H1FW -FW-HV-

1760C HV-1760C DRAIN VALVE LEAK BY

60097820 H1AE -1AETIC- TIC-1796B/TIC-1780B PERFORM TUNING

72

1796B

60099127 H1CJ -1B2P-124 RADIAL VIBRATION INCREASING TREND

60097706 H1AE -AE-HV-

F032A RF17 - BORESCOPE MOTOR OF 1AEHV-F032A

60097902 H1FW R17 RPLC BAILEY MODULES FOR RFPT A

60097903 H1FW R17 RPLC BAILEY MODULES FOR RFPT B

60097904 H1FW R17 RPLC BAILEY MODULES FOR RFPT C

60097043 H1FW -1C-S-105 Replace C RFPT Drain Lines H1R17

60097864 H1AE -1-AE-V092 1-AE-V092 LEAKING FROM PIPE CAP

60097888 H1AE -1AEXI-

651LCD DFCS Operator Monitor Burn In

60097889 H1AE -1AEXY-

651WP1 Operator WS Keyboard Difficult to Access

60077090 H1AE -52-242115 DCP 80095620 RPLC MCC 52-242115 MOV F039

60077091 H1AE -52-242161 DCP 80095620 RPLC MCC 52-242161 HV-F039

60086973 H1AE -52-222102 DCP 80098425 RPLC MCC 52-222102 MOV F032

60096000 H1AE 80103166: On-Line Noble Chem FW Tie-ins

60098460 H1AE -1A-E-106 DCP 80103819/FWH 6A XMTRS UPGRADE

60098482 H1AE -1B-E-106 DCP 80103819/FWH 6B XMTRS UPGRADE

60098483 H1AE -1C-E-106 DCP 80103819/FWH 6C XMTRS UPGRADE

60099309 H1AE 80100455 / 1SBPISH-N652C REVISE SETPOINT

60099308 H1AE 80100455 / 1SBPISH-N652B REVISE SETPOINT

60096958 H1AE -1-AE-V189 H1AE -1-AE-V189 LEAKBY WITH RWCU I/S

60086917 H1AE -52-212102 DCP 80098304-Replace MCC H1AE -52-212102

60087204 H1AE -52-232054 DCP 80098424 Replace MCC H1AE -52-232054

60087205 H1AE -52-232171 DCP 80098424 Replace MCC H1AE -52-232171

60079815

H1AE -10-Z-370A

EXTEND REAR DOOR XFLOW COMP PANEL-

CAP

6.10.2 Review of Engineering items

The team reviewed open Engineering activities. The majority of the feedwater corrective

Engineering actions are for work scheduled for the next possible outage. The evaluations

73

under 7000 series wor orders have been performed. The engineering goal is to evaluate

and resolve issues within 100 days of creation.

Order DescriptionFunctional location

30142986

PM DEFERRAL (PMDR) EVALUATION

GUIDELINES H1AE -1C-P-101

30142986 NUPM DEFERRAL SUPERVISOR REVIEW H1AE -1C-P-101

30159696

ENGINEERING SUPPORT EDDY CURRENT

TESTING H1AE -1C-E-106

30159696 1C-E-106: PERFORM INTERNAL INSPECTION H1AE -1C-E-106

30159696 1C-E-106: SYS ENGR. CLOSEOUT INSPECTION H1AE -1C-E-106

30206023 First Call Risk Review H1AE -1C-E-106

30190564 DETERMINE HOLE LOCATIONS / SIZE H1AE -1B-E-106

30190564 DATA EVALUATION H1AE -1B-E-106

30201131

ENGINEERING SUPPORT EDDY CURRENT

TESTING H1AE -1B-E-106

30201131 1B-E-106: SYS ENGR. CLOSEOUT INSPECTION H1AE -1B-E-106

30180550

ENGINEERING SUPPORT EDDY CURRENT

TESTING H1AE -1A-E-106

70117252 Validate procedure and order changes H1AE

70124662 RFP FLOW AND SPEED DIFFERENCES H1AE

6.10.3 Review of past Engineering evaluations

The team reviewed past engineering evaluations for the feedwater system. These issues have been

evaluated and presented to MRC. Corrective actions came from these evaluations. Evaluations are

reviewed by a peer, then the supervisor and director if the evaluation is an apparent cause

evaluation or root cause evaluation. Evaluations are presented at MRC where they must have a

corrective action to mitigate or eliminate the degraded condition identified. About three quarter of

the evaluations were performed by the current system manager but the earlier quarter were

performed by past system managers.

Order Functional loc. Description

74

70046614 H1AE -1C-P-101

REACTOR FEEDPUMP VIBRATION ALARM-

A2010

70057113 H1CJ -1CJPI-3150A A" RFPT CONTROL OIL PRESS HOOS

70057305 H1FW -1B-S-105 During O/S testing W/U did not work.

70057360 H1AE Startup Level Control Oscillations

70057405 H1AF BLEEDER CHECK VALVES ISSUES

70057567 H1FW reactor feed pump turning gear speed SW

70075109 H1AE -1AELV-1785 R14 BOTH STARTUP LCVS ARE STUCK OPEN

70075773 H1AF -1AFLV-1506C R14 Blocking point missed LSLD

70076610 H1AE R14 DFCS ISSUES DURING RF14 STARTUP

70077303 H1AE -1A-P-101 'A' Rx Feed Pump Seized

70077472 H1AE -10-Z-370C XFLOW CPU HARD DRIVE FAILED

70077812 H1AF -1AFLV-1506B

6B FEEDHTR NORM DRN VLV FAILED X OP

ON

70091658 H1AF -1AFLV-1532B POSITIONER BELL CRANK DEGRADED

70094704 H1AE -10-Z-370C CROSS FLOW CABLING DISCREPANCY

70097600 H1AF -1AFLIC-1040B

'B' MOISTURE SEP DRAIN TNK LVL

CHALLENGE

70097677 H1AE -1AELV-1785 Incorrect S/U Lvl software config loaded

70097936 H1AF -HS-1506B 5B AND 5C FWH TRIP ON SCRAM

70097937 H1AE RFP / RFPT AXIAL POSITION CONTROL

70099018 H1AE -1AEFV-1783B B RFP MIN FLOW IND FLUCTUATING

70099542 H1AE -1AELV-1785

SULCV POSTIONS NOT AT EXPECTED FOR 3

PO

70099906

H1FW -1FWVY-

3769A CRI A-RFP OUTBD BRNG VIBE FAILED

70101641 H1AF -1AFLV-1363A 'B' MOIS SEP LOW LEVEL, DRAIN VALVE

70103635 H1FW CRIDS POINT A2352 IS FAILING

70105948 H1AF -1AFLV-1363A 'B' MOIS SEP LOW LEVEL, DRAIN VALVE

70106135 H1AF -1AFLT-1559B FWH 5B Trip

70106403 H1AE -1AELV-1785 SULCV did not control Rx Level during 1/

70106475 H1AF -1AFLT-1559B PROBLEMS RESTORING LT-1559B FOR FWH 5B

70106478 H1FW -1C-S-105 7366475Inboard Bearing Oil Leak

70107495 H1AE -1C-P-101 Delays in warming C RFPT

75

70111059 H1AE -1B-P-101 FEEDWATER FLOW OSCILLATIONS

70111105 H1AF -1AFLV-1464A EXCESSIVE USE OF NEOLUBE PIPE SEALANT

70112065 H1AE -AE-HV-F032B CLOCK RESET FOR RF16 MOV DCP

70113195 H1AE Let him fix the RFP Minflow oscillations

70114760 H1AE 5 R16 PMS NOT REZEROED AFTER R15

70115348 H1AE DFCS System Alarm

70115467 H1AE -AE-HV-F032B DCP steps not incorporated in work order

Order Functional loc. Description

70117252 H1AE Heat Exch. workflow process gap - RF16

70118359 H1AF -1AFLV-1464A

F.O./2A FWH DRN VLV LIC OUTPUT NEAR

100%

70128893 H1AE -10-D-497 Water dripping in lower relay room.

6.11 System Walkdown

Part of the Hope Creek Single Point of Vulnerability investigation is to perform a walkdown of

the system. The SPV activity required the system be investigated to find any field vulnerabilities

that could challenge this system and are not identified on drawings or procedures. This walkdown

does not include the Feed Pump Turbine rooms due to high radiation conditions. This walkdown

was also performed with the Conduct of Plant Engineering template for walkdowns as found in

ER-AA-2030 Attachment 4. The attachment is a great tool for tracking items that should be

monitored while on a walkdown. Other general inspections checked for leaks or oil from

equipment, coatings and insulation, area lighting, scaffolding installed per procedure and no

seismic ll/l issues., unusual noises, smells, leaking fluids and general housekeeping.

All the vibration trends and local panels were checked. The RFPs’ thrust has trended in the

positive direction and is not a concern at this time. The B RFP IB bearing vibration peaked in the

3rd qtr at 3.0 mils and has since trended down to 2.6 mils this quarter. No actions are required at

this time. The B RFP IB and OB bearing temperatures have been diverging indicating a possible

undistributed load or misalignment. This can only be addressed in the R17 outage when the B

RFP is overhauled.

An airline for the SULCV was found vibrating. The line was a copper tube and linear with a 90

degree elbow. The better practice is to install these lines with a spiral to remove any uneven load

76

and stress from the joints and fittings. The other alternative is to install a flex fit tube which

moves and absorbs the stress instead of the joints and fittings. A notification will be generated to

install a flex fit copper tube for these lines.

A small oil leak was noticed on the H1AE -AE-HV-1753A 6A FWH outlet valve. This leak was

too small to quantify but was documented with a notification.

The 2/3 trip logic with a single sensing line is located in the high radiation feedpump turbine

rooms and could not be checked at this time. During the refueling outage, engineering will

investigate this panel for a possible elimination to the single sensing line vulnerability.

Will continue to monitor vibration trends and initiate notifications if the trends regress in a

degraded pattern. Notification 20529257 has been generated to address the bearing metal trend

divergence. This notification has been rolled to the B RFP overhaul in R17 in WO 30142985.

Notifications 20530352 and 20530357 have been generated to address the shaking SULCV

airline and the 6A FWH outlet valve oil leak.

7.0 Scheduling PriorityIt’s been deemed that any SPV represents a high consequence to a plant trip or derate and as such

warranted additional guidance to help the senior leadership team make informed decisions as the

scheduling priority of proposed SPV strategies.

Discussed here was the team proposed approach to providing input to scheduling priority based

on a low, medium, and high likelihood of occurrence. Each component in the vulnerability

matrix will be assigned a scheduling priority risk based on the below table.

Consequence

Probability

HIGH> 100,000 MW-hrs lost

MEDIUM10,000 - 100,000 MW-hrs

lost

LOW< 10,000 MW-hrs lost

77

HIGHA degraded condition or negative

trend exists

- OR -

Component has failed previously

at PSEG

HIGHImplement within

2 years or next fuel cycle

Fast Track

HIGHImplement within

2 years or next fuel cycle

Fast Track

MEDIUMImplement within 2-5

years or 2-3 fuel cycles

Normal implementation

MEDIUMA neutral trend exists that if not

improved could lead to a

degrading trend or failure

- OR -

Component has never failed at

PSEG, but industry OE exists

regarding failures

HIGHImplement within

2 years or next fuel cycle

Fast Track

MEDIUMImplement within 2-5

years or 2-3 fuel cycles

Normal implementation

LOWImplement within > 5

years or >3 fuel cycles

Long Range Plan

LOWNo degraded condition or trend

exists

- AND -

No PSEG failures

- AND -

No industry OE failures

MEDIUMImplement within 2-5

years or 2-3 fuel cycles

Normal implementation

LOWImplement within >5 years

or >3 fuel cycles

Long Range Plan

LOWImplement within >5

years or >3 fuel cycles

Long Range Plan

Some of the vulnerabilities identified were procedure related resulting in Ops actions in response

to alarms. Other vulnerabilities were a lack of preventative maintenance strategies. These

vulnerabilities can be implemented sooner than the design changes strategies.

The following table lists vulnerabilities and their associated risk levels and their current mitigat

bring immediate management attention. The following SPV threats were assessed high risk and

recommended for priority at the next available refueling or forced outage opportunity. The bases

for the high risk is that these components have known failure or degradation based on historical

search of SAP and/or industry OPEX that could result in the high consequence of plan transient

or trip:

8.0 Vulnerability Elimination or Mitigating StrategiesThe team evaluated different mitigation and elimination strategies for the identified

vulnerabilities. Elimination strategies consisted mostly of design changes to augment the margin

before the station would succumb to the vulnerability. Several elimination strategies have already

78

been approved by PHC and PRC and will be implemented in the following refueling outage in

Spring 2012. Other design changes will be scheduled for R18 in fall of 2013 or R19 in spring of

2015.

Some vulnerabilities were components that impact generation if failed but can not be removed

since they provide a protective function or are required for continued operation. Therefore

mitigation was the proposed solution. Mitigation strategies involved implementing new PM

strategies. Most of the new PM strategies were proposed to compensate for a lack of PM strategy

for the components that have a potential to impact generation.

Vulnerability: “RFP suction valve not 100% open” Trip (80105125 0020)

Recommendation: Modify Bailey logic to eliminate the RFPT trip actuation, but keep the

indication, for when the RFP suction valve is not 100% open. Install an alarm to preclude an

inadvertent valve closing

Basis: The RFP suction valve not 100% open trip is not necessary and can be removed. The

plant is also vulnerable to the open limit switch failure, control power fuse failure, breaker failing

open, or a bailey logic failure. The trip function is to protect the RFP in the event that the RFP

suction valve inadvertently closes resulting in low suction pressure. However, the RFPs are

already equipped with a low suction pressure trip to protect the RFPs from damage caused by

pump cavitations. The pumps are protected from a low suction pressure condition by a different

trip. In the event that the suction valve did inadvertently close the low suction pressure trip would

protect the pump. Install an alarm to preclude an inadvertent valve closing

Review of Salem station report and other industry reports have determined that all single switch

trips should be eliminated from the plant. The other alternative is to change the trip logic to a 2/2

or 2/3 logic. This issue was asked amongst industry peers and none of the other stations have this

trip function. The low suction pressure trip sufficiently protects the feedpumps from low suction

pressure without a suction valve close trip.

The contacts and other components should remain since they illuminate or extinguish the valve’s

position indication light. The drawing below expresses where the logic should be broken to

preserve indication while removing the unnecessary trip function. The only PMs for this

vulnerability are a 12Y limitorque PM. Mitigating actions are to process DCP presentations to

PHC.

79

Vulnerability: Lube oil vapor extractor failure PCR and enhancements

Recommendation: Generate a reliability PM to inspect and repair the lube oil components.

(80105125 0030, 0040, 0050, 0130, and 0140)

Basis: The action is to mitigate the vulnerability. After performing a search for existing PMs on

the lube oil reservoir, the search determined there were no PMs in place for the oil pump

discharge relief valves, the pump discharge check valves, the vapor extractor or to rebuild the oil

pumps. This PCR will address this gap and prevent age related failures for these components. The

PM will align the components and stagger the implementation to prevent multiple trains being

worked in one outage. There are 6 oil pump discharge relief valves, 6 oil pump discharge check

valves with 6 oil pumps, 3 flow orifices, and 6 pressure control valves. The PCM templates

recommend performing pump refurbishments ‘as required’. The check valves are 8 years, the

relief valves are required to be tested every 10 years, and the vapor extractor is a RTF component.

The recommendation is to perform work on each train once every 12 years.

The vapor extractor function is to remove toxic fumes from the lube oil rooms and to provide

drainage for the return portion of the lube oil system. A design change will not eliminate the

vulnerability and the vapor extractor can not be removed. The vulnerability will not trip the unit

or result in a derate. The only suitable strategy is to mitigate the vulnerability. There are no PMs

80

for these components. The action is to process a PCR and generate a PM that will inspect and

repair the oil vapor extractor. This action should be performed once every 12 years. Mitigating

actions at this time will be to process the PCR and to verify spares are onsite to implement the

PM. This action should include checking the vapor extractor discharge line and clearing it of

debris and residue. If the exhaust line clogs it will prevent the extractor from maintaining a

negative pressure on the reservoir. Reference (OE 31000) for a fire starting due to oil leak

ignition. The station allowed oil to drip and accumulate to the point where it contacted a hot

surface, atomized, and ignited burning the surrounded spilt oil.

The oil check valves’ function is to prevent oil from flowing backwards to the reservoir. This

could cause a trip of the RFPT on low bearing and control oil pressure. The vulnerability would

exist with any check valve and is not warranted. There are no other means to eliminate these

check valves. Therefore SPV elimination is not possible but it can be mitigated by generating

new PMs. These PMs can be performed in conjunction with the respective proposed oil pump

refurbishment PM. The PM scope should disassemble the check valve and replace the internal

subcomponents as required. Post maintenance testing should be performed to verify the check

valves will close as required. Mitigating actions at this time will be to process the PCR and to

verify spares are onsite to implement the PM. Additional actions are to verify the FLOCs get

created for these check valves. Reference OE (341-020519-1 and SER 27-87) for events that

caused a loss of oil pressure due to oil check valves failing to close when the pumps were

swapped.

The oil pump function is to provide adequate control oil to the RFPT steam control valves and

allow for steam valve actuator to respond to a demand change as required. Another oil pump

function is to provide lubricating oil to the bearings. This ensures that the bearings and the shaft

will continue to operate smoothly without excessive heat and vibrations. There is an Aux oil

pump and the function is to be able to auto start as required without perturbation to bearing or

control oil header pressures. The vulnerability can not be eliminated since these pumps are

required to be in service to maintain RFPT operation. A pump upgrade would not eliminate the

vulnerability or guarantee internal degradation would not cause the pump to fail. By performing a

pump overhaul, the station can identify degraded parts and replace as needed and prevent an

equipment failure from occurring. The recommendation from the PCM template is to rebuild as

required. The installation will have to be performed when the RFPT is offline in the refueling

outage. However, these pumps can be rebuilt while online and staged for installation once the

81

refueling outage starts. This will help alleviate outage resources demand. The rebuild frequency

should be once every 12 years, per pump. Mitigating actions at this time will be to process the

PCR and to verify spares are onsite to implement the PM.

The oil pumps’ discharge piping is equipped with a pressure relief valve on the pump discharge

piping. The relief valves prevent over pressurizing the oil pump discharge line and deadheading

the pump. The oil pumps are not equipped with a high discharge pressure trip function and the

pumps would dead head if a block developed. The vulnerability is if the valve falsely lifts, it

would cause a pressure transient that would trip a RFPT on low control oil pressure or bearing

oil. The relief valves can not be removed since there would be no means to protect the pump from

an over pressurized condition. No PMs were identified while performing this evaluation.

Mitigation is the best strategy for addressing this vulnerability. A PM to inspect the relieve valve

internals such as the spring, fittings, valve body, and that the valve will lift at the required set

points would be sufficient. Verifying the valve internals are satisfactory and the valve will lift

when required will mitigate the vulnerability. These actions should align with the respective oil

pump for optimization. Immediate mitigation actions are to process the PCR and verify spares are

onsite to implement the PM. Reference (OE Event 341-020519-1 and OE15380) for details on

how a discharge relief valve failed open and resulted in a sudden loss of header pressures. The

loss of header pressures caused the feed pump to trip on oil pressures causing a SCRAM.

The oil pressure control valves’ function is to control oil pressure to the bearing header and the

control oil header. The pressure control valves maintain pressure across the headers. The control

oil pressure valve ports excess oil to the bearing oil line. The bearing oil header control valve

ports excess oil to the reservoir. The vulnerability for this component is if the control oil header

valve failed, it would fail open and port excessive amounts of oil to the bearing header line. The

bearing header would respond and port the excess oil to the reservoir. This could result in a RFPT

trip on low header pressure. The bearing oil header valve is designed to fail closed and would

preserve oil flow to the bearings. Removing these pressure control valves would require a major

design change to the oil system and is not recommended. The most effective strategy is to

mitigate the SPV with a new PM. These valves currently do not have PMs and need to be

addressed. The two valves should be worked during the same outage so they may be calibrated

together. Calibrating the two valves together would verify the proper pressures are being

maintained for the headers. The control header pressures valve impacts the bearing oil pressure

control valve. If the control valve is porting excess oil to the bearing oil line, the bearing oil

82

pressure valve will attempt to maintain oil as close to the set point as possible. Immediate

mitigation actions are to process the PCR and verify spares are onsite to implement the PM.

A PM was recently approved to flush the lube oil system. The flush will address the vulnerability

for the oil flow orifices becoming fouled due to sludge and oil breakdown. The PM will be

scheduled on a 6Y frequency and can line up with each oil system PM. No actions are required at

this time.

Vulnerability: RFPT oil pressure SPV (80105125 0060)

Recommendation: Implement DCP for oil SPV

Basis: The action is to eliminate the lube oil low pressure SPV. Hope Creek approved a design

change to install 2 larger accumulators on each RFPT oil train and modify the bailey logic to auto

start the backup oil pump instantaneously in the event running oil pump trips. The original low oil

pressure auto start feature will remain and continue to start an oil pump if header pressure drops

to the start set point. The previous auto start logic was based only on pressure and would start the

back up oil pump when a low oil pressure condition was sensed. The back up oil pump breaker

remained open while pressure decayed, costing valuable time to restore oil pressure. The

approved DCP will decrease the time between the main oil pump trip and the aux oil pump

start. . The new logic will allow the breaker to close immediately following the main oil pump

trip, instead of after the pressure decays to the lower auto start set point. The remaining

mitigating actions for this SPV are to verify the new, larger accumulators are installed and the

bailey logic is modified to the required specifications in the next refueling outage.

A considered mitigation strategy was to change the pump auto start set points. The mitigation

would have raised RFPT oil pump auto start pressure so the pump could start sooner. This was

refuted because there was too little margin in between operating pressure and the auto start set

point. The back up oil pump would auto start when not required resulting in 2 oil pumps in

service. The back up oil pump would need to be secured which resulted in pressure perturbations

to maintain oil pressure on the header. This strategy was refuted since it created more risk than

benefit while addressing the vulnerability.

Vulnerability: RFPT moisture drain valve not 100% trip (80105125 0070)

83

Recommendation: Perform DCP to eliminate this trip function. Install an alarm to preclude an

inadvertent valve closing

Basis: The action is to eliminate the trip function on the moisture drain valve. If the drain valve

is not 100% the respective RFPT will trip. This elimination strategy is similar to the one listed in

section 8.1. The trip function has no margin to prevent an instantaneous trip actuation since there

is a single contact switch with no time delay buffer. The station is vulnerable to the open limit

switch failure, control power fuse failure, breaker failing open, or a bailey logic failure. The

station should not attempt to harden the trip since it is not necessary to remove moisture from the

turbine case. Industry peers have confirmed that this trip function is not required and is not

implemented at other stations. There is indication of the valve’s position on the process

monitoring computer. Operators can identify if the valve is or is not full open.

The trip function should be removed from the drain valve while keeping all the other indication

and feed to the process computer. An alarm should be installed to preclude an inadvertent valve

closing. A DCP needs to be processed to perform the elimination strategy. Refer to the attached

logic drawing below for the scope of the proposed change. The next mitigating actions are to

process the DCP request presentations and present to PHC subcommittee and PHC. This issue

was polled to the industry and asked which stations have a similar trip configuration. None of the

stations polled indicated they have this trip.

Vulnerability: SPV - RFPT thrust bearing wear detector trip (80105125 0090)

84

Recommendation:

1. Install second redundant thrust probe. The thrust probe will be of the same make and

model as the current thrust probe.

2. Modify the bailey logic to actuate the RFPT trip when both high thrust signals are

received. An ‘And’ module will have to be added to the bailey card for both thrust signals to

output a single trip signal to the trip circuits.

Basis: The actions are to harden the thrust bearing wear detector trip. The current logic has a

single probe with no time delay. There is no margin for this trip function and could

instantaneously trip the respective RFPT. The trip provides a protective function to the RFPT. If

the turbine is able to thrust excessively the possibility of stationary to rotating component contact

is possible. Review of the RFPT clearance data confirms that the thrust bearing clearances have

the least margin of all the ‘rotating to stationary’ parts tolerances. This trip function can not be

eliminated since there is no redundancy to protect the equipment from excessive thrust and

‘stationary to rotating parts’ vulnerability. Additional margin can be installed to make the trip

function less susceptible to a false actuation. Adding the second probe and modifying the bailey

logic will eliminate the single point of vulnerability that will cause a pump trip. The thrust

bearing wear detector will still provide its intended protective feature. Mitigating actions include

a PM to perform calibrations every 36 months. Additional mitigating actions are to present to

PHC subcommittee and PHC to process a DCP. Installation should occur during the refueling

outage in Fall of 2013. Reference (498-950124-1) for a single thrust probe failure that resulted in

a unit trip. The design was similar to Hope Creek since there was a single probe providing

indication. The false indication drove Operators to reduce feedpump speed and trip the pump.

Vulnerability: SULCV instrument air tubing enhancements. (80105125 0110)

Recommendation: Install flex fit bronze tubing on components exposed to vibration.

Considerations should be given to plant equipment lines whose failure could result in plant

derates. The SULCV tubing would benefit most from flex tubing.

Basis: The action is to implement the enhancement to improve the equipment air lines. Currently

the steel and copper tubing installed is subject to light vibrations. The system walkdown observed

the equipment lines vibrating. A line break in the air supply would result in the SULCV failing

closed on a loss of air. This would not cause a derate during full power operations but would

challenge the plant on shutdown and start up. The steel and copper tubing ends should be fitted

with flex fit tubing. The tubing will not change plant configuration or functionality of the

SULCV. Current mitigating actions are to isolate the SULCV bypass line and stroke the SULCV

85

one month before an outage. This verifies if the SULCV is capable of performing its function and

will detect degraded components. The next mitigating action is to add flex fit tubing to the lines.

Risk exists while isolating the SULCV line while at 100% power, if the SULCV is suddenly

needed to stroke and the respective bypass line is unavailable. The tubing should be installed in

October 2013 outage.

Vulnerability: Revise Ops procedure for vibration response (80105125 0120)

Recommendation: Change Ops procedure HC.OP-AR.ZZ-0028 and HC.OP-AR.ZZ-0022 to

include guidance for response to axial vibration experience on the RFP and RFPT journal

bearings. Vibrations should be confirmed before reducing pump speeds. Bearing temperatures

should be monitored for an increase and/or unexpected fluctuations to verify high vibrations. If

bearing temperature rises or shows an unexpected or erratic change then reduce RFP speeds.

Basis: The action is to eliminate the vulnerability. Operations concern was that the referenced

procedures direct operations to reduce RFP speed to maintain vibrations below the danger set

point, which could be done unnecessarily without ruling out a false signal. This is the appropriate

action to address actual vibration increases and should remain in the procedure. However, if the

vibrations are the result of false indication, operations would inadvertently reduce RFP speed due

to false indications. Bearing temperatures are expected to rise if the bearings are experiencing

excessive vibrations. Before reducing RFP speeds, operations must verify the bearing

temperatures have not shown a change that would result from vibrations. A false vibration due to

a degraded instrument would show no temperature indication. Vulnerability mitigating PMs exist

to minimize the possibility of false indication. The vibration monitoring equipment is calibrated

every 36 months. To eliminate the vulnerability Ops procedure should be changed to prevent Ops

from reducing RFP speed without verifying the respective bearing temperature has not changed.

Vulnerability: Stagger RFP suction pressure trip set points (80105125 0150)

Recommendation: Modify the trip set points for the RFP suction pressure to trip at staggered

pressures. Currently all 3 RFP’s have 2/3 logic and a 5 second time delay that actuates when RFP

suction pressure falls to 230 psig. For 1 RFP change the suction pressure trip set point to 240

psig. For a different RFP change the suction pressure trip set point to 250 psig. Leave the third

RFP suction set point trip as is.

Basis: This action was suggested by the SFRC. The suggestion was to stagger 2/3 trip pressure

set point to prevent a simultaneous trip of all 3 RFPs. A low pressure condition may exist that

would drop low enough to actuate the trips for all 3 RFPs and would take the unit offline. If a

86

pressure transient does occur, tripping one pump before the others would restore pressure and

prevent the other 2 from tripping. The change in set point is for a higher pressure and a more

conservative change.

The SFRC other method to address this vulnerability was to stagger the time delays. Staggering

the time delays would reduce margin and trip the RFP sooner. A change in the opposite direction

would make the pumps more vulnerable to damage in the event an actual low suction pressure

existed. The pump would be subject to cavitations for a longer period of time. The decision to

stagger the pump time delay trips was refuted for this review.

Mitigating actions already calibrate the sensing instruments and switches every 18 months. The

next action is to process the set point change request and implement the change. The change

should be implemented during the R18 outage in October 2013. This is a set point change and

does not require a DCP but will require an update to the ICD cards.

Vulnerability: Create PM to replace the RFPT expansion joints (80105125 0160)

Recommendation: Create a PM to replace the expansion joints on the RFPT. The expansion

joint replacement should be performed every 18 years and align with the respective RFPT turbine

overhaul.

Basis: These expansion joints do not have PMs for replacement. Aging will eventually result in

failure forcing a loss in generation. The extent of the loss depends on the remaining time until the

following refueling outage. If a failure results early in the operating cycle the loss the station will

be forced to operate at a derated level longer than if the failure occurs closer to the end of the

operating cycle. If the losses endured while operating in a derated state will exceed the cost of a

maintenance outage, the station would have to plan a maintenance outage to repair the leak.

An 18 year PM is the most effective duration to replace the expansion joints. To gain access to

the expansion joints, the respective RFPT turbine must be disassembled. It would be effective to

schedule a replacement while the LP turbine is disassembled. Each respective RFPT is

overhauled every 6 years. OE states that these expansion joints begin to fail after 20 years. An 18

year PM will meet the life expected limits and not require addition work be performed in the

refueling outage. The mitigating action is to process the PCR to implement the suggested PM.

87

Reference (OE17885 and 19832) For the impacts an expansion bellow failure can have on plant

operations.

Vulnerability: Upgrade the A,B,C 6FWH level transmitters (80105125 0170 & 80103819)

Recommendation: Ensure the DCP is installed in R17. The DCP has already been issued and

is currently scheduled for R17.

Basis: The strategy to eliminate the vulnerability has already been evaluated and is scheduled for

the next refueling outage. The current configuration for the narrow and wide range transmitters

has them all within the #6 FWH rooms. These rooms are inaccessible with the heaters online due

to heat and radiation levels. This DCP will install the level control loops outside of the FWH

rooms so they may be calibrated and repaired while online. Several events have occurred at Hope

Creek that involved the level control function of the heaters to fail or degrade. Failure of level

control function has resulted in level rising or falling causing the dump valve to lift. In some

instances the heater tripped resulting in a forced derate. Mitigating actions have already been

planned for the next refueling outage in Spring of 2012. The only remaining actions are to verify

the design change is installed and the new level control loop functions as expected. Reference

(EPIX 77 and 395-010501-1) for instances FWH dump and drain valves can fail to open and will

result in the trip of a FWH.

Vulnerability: Upgrade the A,B,C 6FWH Upgrade FWH wide range level trip and indicating

Circuits (80105125 0180 & LTAM H-11-0057)

Recommendation: Ensure the recommendation is approved by PRC and is installed in R18.

The recommendation has been approved by PHC for a conceptual design study.

Basis: The FWH level Control Panel Wide Range level circuits uses Westinghouse 7500

electronic Signal Conditioning Cards (SCC) & Alarms Cards (AC). The SCCs and ACs are

obsolete and require periodic refurbishment of electronic components to maintain basic

reliability. These controllers have been subject to age related failures since 2002. Actions have

been presented to PHC and approved. The team determined these controllers should be upgraded

no later than the refueling outage in Fall 2013. The proposed work consists of the replacement of

72 cards, 24 cards per panel. It is proposed to utilize OTEK Model HQ-114 Digital

Programmable Intelligent Controllers (IC) or Foxboro 762 digital controllers. The intent is to

have each IC provide power, trip, alarm and indication for each wide range level transmitters.

The proposed solution have 12 new controllers/panel to replace the 24 signal conditioners. This

88

action item will be used for tracking purposes and to verify the corrective action is performed on

time.

Vulnerability: RFPT rupture disc failure (80105125 0190)

Recommendation: RFPT steam rupture discs require a replacement PM. Add the rupture disc

replacement to the RFPT overhaul.

Basis: The action is to implement a PM change and add the rupture disc to the RFPT overhaul

PM. The overhaul procedure has a note to inspect for signs of steam leaks however this will not

prevent a sudden failure of the disc during an operating cycle. The integrity of the disc can not be

evaluated with a visual inspection and will age until failure occurs. The rupture disc is designed

to protect the exhaust line expansion joints in the event that the line over pressurizes. The disc

can not be removed, eliminating the vulnerability. Mitigation is the preferred method for

addressing the vulnerability. Including action to replace the disc during the RFPT overhaul is the

most effective strategy to address this vulnerability. The next mitigating action is to process a

PCR to add the rupture disc to the overhaul PM.

Vulnerability: RFPT loss of power trip solenoid (80105125 0200)

Recommendation: Eliminate the trip solenoid and trip that occurs due to a loss of power to the

trip system. Install an alarm that will annunciate due a loss of power to trip system so the pumps

will not trip on a loss of power to the trip system.

Basis: The strategy is to eliminate the vulnerability with a design change to the trip system. The

current purpose of this trip is to stop the turbine if there is a loss of power to the trip system.

These solenoid valves will dump oil back to the reservoir in the event that the trip system loses

power. If they are not energized they will lift and trip the RFPT. Therefore a momentary loss of

power to the trip system would trip the RFPT. These solenoid valves should be removed and an

alarm should be installed that will annunciate in the event trip power is lost. Removing these

solenoids will keep the RFPT online and eliminate the possibility of a momentary loss of power

that would result in a trip.

In the event that power to 1A(B,C)-D-318 panel is lost, or failure of either normally charged

solenoid coils will result in a feedpump trip. The trip solenoids are subject to a fuse failure as well

which also results in a RFPT trip. In the event that power is lost to the trip system, the manual

89

push buttons would still be available and will shut down the RFPT is Operations determines it is

necessary to shut them down.

Currently there are PM tasks that verify the function of the solenoid valves to open when not

energized. DCP should be implemented in R19. This DCP will require more engineering

compared to the other design changes of this review. The following actions for this item are to

process the PHC sub committee and PHC presentations.

9.0 Review of System Vulnerability Initiatives from Other Sites

The team reviewed other SPV reports from other stations in the fleet to compare corrective

actions with. This review will confirm the proper actions are being implemented as well as other

issues that may have been overlooked. Hope Creek is implanting actions as part of the SPV

review and actions not considered SPVs but are enhancements for improving equipment

reliability. Some of the major trends noted across the industry are as follows:

Common themes for the industry is to upgrade of single 1/1 trip logic to 2/3 trip logic. Hope

Creek verified that all of the 1/1 logic was removed or upgraded to remove the SPV. The single

contacts for the pump suction valve and Turbine drain valves not being 100% open will be

removed. The excessive thrust bearing wear detector will be upgraded so a single switch is not

responsible for actuating a trip.

Expansion bellows are being overlooked as “passive” components and do not get replaced.

Several sites identified the expansion bellows as an SPV that is subject to aging that requires a

PM to mitigate. The team determined that Hope Creek replaced their expansion bellows after

20-23 years of service and has action to implement a PM program.

There were some differences between the stations when performing the report and evaluation.

Some stations defined a Single Point of Vulnerability as a derate greater than 5%. Some

stations used the 20% or greater criteria to identify an SPV. Some Exelon and Entergy plants

were more involved as a fleet than as individuals when performing the report. It seemed like

when these reviews were done, it was more beneficial to share ideas directly while writing the

report.

9.1 Grand Gulf station SPV report review

90

Company – Entergy

Station – Grand Gulf

Findings - The team reviewed the 2010 reactor feed system vulnerability report from Grand

Gulf Station. The review was focused on the reactor feedpump and condensate system with

several of the single point vulnerabilities similar to the ones identified in the Hope Creek

vulnerability report. The following are noteworthy observations:

Feedpump trip logic uses 2/2 or 2/3 logic to eliminate SPV threat. For example, the low

suction and high discharge trips are 2/3 coincidence. In contrast, Hope Creek has

vulnerability with 1/1 logic for the RFP suction valve position not 100% open, RFPT

drain valve position not 100% open, and the thrust bearing wear detector. Other trips

were verified to be 2/3 or 2/2.

Many of the instrumentation switches are off a common instrument line. Hope Creek

shares the same vulnerability, the low bearing and low control oil pressure trip switches

2/3 logic but are all off a common sensing line.

Grand Gulf identified the expansion bellows as a passive component that does not have a

PM but is subject to age related failures. Failure of the expansion bellows results in a loss

of condenser vacuum and requires a unit outage to repair.

9.2 Salem Station SPV report review

Company – PSEG

Station – Salem

Findings - The team reviewed the 2010 reactor feed system vulnerability report from Salem

Station. The review was focused on the reactor feedpump and condensate system with several of

the single point vulnerabilities similar to the ones identified in the Hope Creek vulnerability

report. Salem and Hope Creek have similar feedpumps, turbines, and oil delivery systems. The

following are noteworthy observations are as follows:

Salem noted their turbine loss of power trip solenoids as an SPV as well as Hope Creek.

In contrast, Salem recommended the trip solenoid valves design should be changed

from de-energize to actuate (trip valve closed) to energize actuate. This would

eliminate the possibility of failure on a momentary loss of power or a fuse failure. At

91

Hope Creek, the stations is going to change the design and remove the loss of power

trip solenoids from the plant.

The rigid stainless steel (SS) instrument air supply line from the instrument tray to the

air supply regulator should be changed to high pressure SS flex hose. Also, other rigid

SS tubing connecting the accessories on the actuator should be given consideration to

the use of flex hose to the extent possible. Hope Creek has performed a similar change

for the feedwater minflow valves and will implement the change as an enhancement.

This change applies to copper tubing in addition to SS.

Several changes are being made to the trip system. At Salem there are several trips that

use 1/1 logic without time delay. Their actions from the SPV review are to change the

1/1 logic to 2/2 or 2/3 logic to prevent a momentary false switch actuation forcing a

feedpump to trip.

There are several action for improving the lube oil delivery system. Most of these

actions are similar to the ones Hope Creek identified including orifice inspection,

check valves, and relief valves. In contrast, there are no actions to flush the oil system

as suggested in the Hope Creek review.

Company – Entergy

Station – Indian Point

Findings - The team reviewed the 2010 reactor feed system vulnerability report from Indian

Point Station. The review was focused on the reactor feedpump and condensate system with

several of the single point vulnerabilities similar to the ones identified in the Hope Creek

vulnerability report. The following are noteworthy observations:

Indian Point identified the expansion bellows as a passive component that does not have

a PM but is subject to age related failures. Failure of the expansion bellows results in a

loss of condenser vacuum and requires a unit outage to repair. There were no

documented failures at Indian point but they did have the original bellows still installed

and were perusing action to replace them.

1/1 logic was being replaced at Indian Point with 2/2 or 2/3 logic. This action is being

implemented at Hope Creek as well.

Lube Oil Leaks/System Issues (AC motor swap over without trip). Chronic leaks at

Flanged and threaded joints. Need consistent design/test approach to ensure reliable

AC motor swap over on loss of running pump and proper Lube oil cooler equalizing

92

and vent lines for cooler swaps. Hope Creek has a similar issue but Indian Point stated

an action was to improve check valve location and allow for a bumpless swap of the oil

coolers. This is not the same issue that Hope Creek has.

Attachment 1 – Vulnerability Matrix

Attachment 2 – OE Vulnerability Matrix

Attachment 3 – Cooper Reactor Feed System Vulnerability Report 2007

Attachment 4 – Exelon OPCC list for Feedwater System 2010

Attachment 5 – ANO Feed and Condensate System Vulnerability Report 2010

Attachment 6 – Indian Point Feedwater System Vulnerability Report 2010

Attachment 7 – Salem Feedwater System Vulnerability Report 2010

93