THEMIS/GBO Review 1 Banff, March 29, 2006 Ground Based Observatories (GBO) System Design Overview Operational Status Lessons Learned S. E. Harris University

THEMIS/GBO Review 1 Banff, March 29, 2006

Ground Based Observatories (GBO) System Design Overview

Operational StatusLessons Learned

S. E. Harris

University of California - Berkeley


GBO Components

AC Power

Ground Magnetometer (GMAG)

Telesat Dish

Iridium

GPS

All Sky Imager (ASI)

Observatory Support Equipment (OSE)

inside the “Hut” enclosure

Internet

Variations possible at some sites:

Existing magnetometer

“Hut” enclosure not needed

Existing Internet connection


Complete Installation at McGrath


All Sky Imager (ASI)

Main Components• Camera, Starlight Xpress, MX716

– Designed for amateur astronomy– CCD Format: Interline transfer, 376 x 290

pixels (binned), 8mm diag.– Cooled with TEC for reduced dark current

• All Sky Lens– Telecentric design by Keo Consultants– Full hemisphere viewing, full resolution

obtained 256 x 256 image• Environmental Enclosure with Sun Shade


ASI Unit Test Summary

Acceptance Tests:• FOV (>170°)

• Exposure Time (duration at least 1 sec)

• Spectral Response – Produce detectable response to source radiance < 10kR

– Goal < 1kR– Passband 400 – 750nm– 1 sec exposure duration

• Spatial Resolution (>250 pixel image diameter)– Verify point source response

• Cadence (better than 10s)– Goal < 3s

• Record dark and bias images (room temperature)

Pre-Shipment:• Focus adjustment and verification

• Alignment verification (top of image relative to alignment datum)

• Heater control functional test

• Sun Shade solenoid functional test (#iterations: 360)

• Internal mechanical inspection checklist (thm_gbo_116)

• Final functional test and System Burn-in (duration: several days)


ASI Responsivity Testing

Data report: thm_gbo_118

Recorded on all Imagers:

• On Axis spectral response

• Off Axis spectral response

• Dark vs exposure time

• Resolution

0 500 1000 15002500

2510

2520

2530

2540

2550

2560

Desired Exposure Time (msec)

Dar

k In

tens

ity

Cam 7 Dark Intensity vs. Desired Exposure Time

400 450 500 550 600 650 700 750 8000

0.01

0.02

0.03

0.04

0.05

0.06

Wavelength (nm)

Res

pon

se (

Co

unts

/Ra

yle

igh

/sec

)

Cam 7 On-axis Response vs. Wavelength 450-700 nm

-80 -70 -60 -50 -40 -30 -20 -10 00.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

0.055

Angle in Degrees

Res

pon

se (

Co

unts

/Ra

yle

igh

/sec

)

Cam 7 Response vs. Angle for 500 nm, 600 nm Wavelengths

500 nm

600 nm


ASI Resolution Testing

On-Axis 40º Off-Axis 80º Off-Axis


ASI Enclosure Design

Design: Allison Park Group, Inc.

Heaters

S.S. Enclosure

Acrylic Dome


Sun Shade

ASI must survive non-operating exposure to daytime sun• CCD is Sony ICX249AL

– Plastic “microlens” on each pixel, transmission reduction w/ exposure to UV

• All Sky Lens uses Peleng Fisheye f/3.5– A/R coating exhibited discoloration after one summer in ATHA

Solenoid drive actuates to close shade• Falls open when solenoid power is removed


ASI Enclosure Performance

ASI installed at Ft. Yukon, AK

Dome:• Generally stays clear of snow without

intervention• Some ice buildup on flange

Heating:• Maintains 60° T using about 50W avg.

heat power• Total heating power available is 240W

(60W in dome)

18:00 00:00 06:00 12:00 18:00 00:00 06:00-50

-40

-30

-20

-10

0

10

20Tue Jan 24 16:00:13 PST 2006

Time of day

ext tempasi tempasi duty cycle


Ground Magnetometer

GPS Receiver (included at all GBO sites) Antenna and Electronics Integrated into one package NTP compatible (1msec time accuracy)

Fluxgate Magnetometer (sensor shown below) ±72KnT dynamic range @ 0.01nT Resolution (~23 bits) Offset DAC system for 256 possible ranges per axis 2 Vectors per second data rate Low Power < 4W USB interface for data retrieval and firmware upload


GMAG Installation

Fluxgate Magnetometer Installed at Athabasca


OSE Layout (1)

Rack Mount Shipping Case (12U)

System Computer

GMAG Interface Electronics

APS Serial Switch

Swappable Hard Drive (2 ea)

Iridium modem

UPS

28”


OSE Layout (2)

Power Control Unit

CR10X Datalogger

UPS

ASI Power Supply

CR10X Battery


Power Control Unit (PCU)

PCU Features• CR10X Datalogger is the environment controller for GBO

– Extended temperature range (-55º to +85ºC)– Always operating and remotely accessible (via Iridium) – Low power consumption (battery can operate it for months)– Simple programming and data logging capability

• Provides room temperature environment inside OSE and ASI – Enables use of standard commercial computers, hard drives, etc.– Implements graceful shutdown of system in either event of:

– Loss of Power– Loss of Temperature (either too high, or too low)

• CR10X is the analog & digital I/O peripheral for System Computer• Provides “watchdog” function for System Computer (PC)


Power Control Unit Interfaces

SSRJ7

OSE Heater

CR10X

5A

SSRJ8

UPS

6.3A

SSRJ13

OSE Cooler

6.3A

SSRJ14

System Pwr

SSRJ15

Iridium/APS

SSRJ12

ASI Power

1A, SB

From AC Main

SSRJ3

ASI Heater

5A

Ext_temperature

OSE_temperature

ASI_temperature

Int_temperature

Vline (AC Main)

Sys_imon (AC line current)

Asi_vmon

Stl_vmon

Sol_imon (Sun Shade)

OSE_humidity

UPS Battery Charge Time

CB1

CR10X Power Supplyw/ Battery Backup

Charger

SystemComputer

UPS

APS SerialSwitch

IridiumModem

PowerControl

Unit

Serial


OSE Heating / Cooling Devices

Small Space Heaters 175W, 120VAC, 2 ea

Solid State Air Conditioner

163W Capacity

120VAC Power


“Hut” Enclosure

“A Box within a Box”

Internal Rack Mount Case for Equipment

(Doubles as Shipping Case)

External Insulated Environmental Enclosure


Hut Performance

No heating in Hut required down to –40ºC

GBO#10, Ft. Smith

Cooling required above –5ºC

GBO#10, Ft. Smith

00:00 06:00 12:00 18:00 00:00 06:00 12:00-10

-5

0

5

10

15

20

25

30

35Wed Mar 22 18:00:10 PST 2006

Time of day

External temp

OSE temp

A/C duty cycle

00:00 06:00 12:00 18:00 00:00 06:00 12:00-50

-40

-30

-20

-10

0

10

20

30

Time of day

Tue Feb 14 18:00:10 PST 2006

External temp

OSE temp

Heater duty cycle


Instrument Data Flow

System Computer

ASI

GMAG Telesat Modem

Hot Swap Hard

Drive(s)

USB

Serial I/O

10/100 Base T

GPS

Internet


Status of Operations

Number of GBOs Deployed: 11• Installation and logistics can be difficult• Monitoring and maintaining is harder

Some operational hazards that we’ve survived so far:• Cold

– e.g. door to the shack stuck open• Hot

– e.g. stuck thermostat in the shack• Power

– generally it’s been reliable, but outages are a fact of life• Inaccessibility

– Ekati and Inuvik• RF Interference (More on this later)• Internet connection outages

– data back up is working fine• Software issues

– system processes can hang up (USB interfaces, NTP daemon)– periodically we need to reboot the system

• Non-responsive custodian


System Uptime

Initial analysis, 8 of 10 stations analyzed• Based on availability of CR10X records logged on site• At each station, number of records is averaging 97.5%

– Of 288 records maximum per day

• Indicates the systems are up and running normally most of the time


Learning and Improving

System issues are addressed in Problem Reports• PFRs describe problem

and corrective action• We track PFRs for each

station – Leaves a paper trail of

changes for each site– A copy also is attached to

site log on ‘wiki’


Lessons Learned - Prototype

Ice buildup on ASI dome• Added dome heaters

Insulation outgassing in ASI• Changed material to foil-

faced air pillow


Lessons Learned (1)

Issue PFR Description / Resolution

ASI cable length A deployment issue first increased to 60’ settled on 100’

-001-003-006-008-021

Changed ASI supply to regulated 12V (60’)Increased sun shade solenoid drive to 24VACChanged ASI 12V supply to meet in-rush currentModify ASI power interface wiring (100’)Increased sun shade solenoid drive to 28VAC

ASI Power Diagram

Condor 12V,3.4A Linear

Starlight PowerBrick

J3 P3 P1 J1

Starlight MX716

400mACAM_P12V

TEC_CCS

CAM_RTN

100 ft cable, 9 pr

All Sky Imager

12.5 V

asi_vmon stl_vmon

P12Switched

Outlet


Lessons Learned (2)


ASI USB InterfaceSusceptible to EMI, causes lock upDesign is “always on”

-005-018

Add’l power line filtering on Starlight Xpress boardDesign remote-control USB Disconnect deviceEnables an easy power reset optionInuvik is skipping exposures to elim. RF interference

Modified USB Board

USB Disconnect Device Prototype


Lessons Learned (3)


System monitoring “more is better”

-002-010

Add AC line current monitorAdd monitor for PC +5VAdd monitor on ASI power supplies

ASI Power Supply Overheating

-015 Modify power brick, add heat sink and fan

Overheated PC Boardcourtesy of Mikko Syrjasuo

Modified Starlight Brick


Lessons Learned (4)


Component failures -019-020

Ekati – USB hard drive failed, PC would not bootAtha – PC power supply failed, blew internal fuse

Obsolescence -022 USB Extender device now unavailable, changing all to make them interchangeable

PC USB DisconnectUSB Extender

Base Unit

Control fromPCU

J4 P4 P2 J2

Starlight MX716

100 ft cable, CAT5

All Sky Imager

USB ExtenderRemote

Starlight USB I/F

USB

ASI Data Interface Diagram


Open PFRs


Iridium modem configLearning curve

-016 APS serial switch auto-configures modem on resetPower reset modem and APS every dayLearn all failure modes Recently discovered a situation where modem and APS switch can get into hung state – investigation is ongoing

GMAG restart problem

-007 Problem associated with GMAG USB interface, does not restart correctly if system reboots when it’s coldVery infrequent occurrence, can be handled by operator intervention

Documents

THEMIS/GBO Review 1 Banff, March 29, 2006 Ground Based Observatories (GBO) System Design Overview Operational Status Lessons Learned S. E. Harris University