Upload
camilla-powell
View
216
Download
1
Tags:
Embed Size (px)
Citation preview
March 29, 2010 RFI Mitigation Workshop, Groningen The Netherlands
1
Statistics of the Statistics of the Spectral Kurtosis Spectral Kurtosis
EstimatorEstimatorGelu M. Nita and Dale E. Gelu M. Nita and Dale E.
GaryGary
New Jersey Institute of New Jersey Institute of TechnologyTechnology
March 29, 2010 2RFI Mitigation Workshop, Groningen The Netherlands
Population Spectral Population Spectral KurtosisKurtosis
The PSD estimate obtained by an N-points FFT
of a Gaussian time domain sig
1
nal h
as an expo
nential dist
; 1 1
ributio
2
n
kP
k
Np P e k
2
2
Population Spectral Kurtosis: statistical RFI discrim
in tor
1
a
k
k
SK
March 29, 2010 3RFI Mitigation Workshop, Groningen The Netherlands
Spectral Kurtosis Spectral Kurtosis EstimatorEstimator
22
2 21
Spectral Kurtosis biased estimat
1
r
1
o
k
k
MSMSK
M S
2
1 21 1
Power and squared power sums
M M
k ki i
S P S P
22 2 1
1
1
Mean and variance unbiased estimat
(
rs
1
o
)k k
MS SS
M M M
March 29, 2010 4RFI Mitigation Workshop, Groningen The Netherlands
The SK SpectrometerThe SK Spectrometer
Key features:
Conceptual simplicity
Frequency channel independence
Straightforward FPGA implementation
March 29, 2010 5RFI Mitigation Workshop, Groningen The Netherlands
Statistical thresholds for Statistical thresholds for the rejection of RFI outliers the rejection of RFI outliers
(M>>1)(M>>1)(Nita et. al, 2007 PASP, 119, 805:827)(Nita et. al, 2007 PASP, 119, 805:827)
Assumed negligible bias
E 1SK
Variance (theoretically proven)
4 Var SK
M
RFI symmetrical thresholds (1 3 )
(assumed SK normalit
6
y)
1M
Hardware implementation of the SK excision algorithm and initial tests revealed that, although it performs generally well,
•the lower threshold level is set too low, failing to reject some RFI contaminated channels
•the upper threshold level is set too low, rejecting more RFI free channels than statistically expected
CONCLUSION: More accurate RFI rejection levels are needed for improved and reliable performance.
March 29, 2010 6RFI Mitigation Workshop, Groningen The Netherlands
Characteristics of the SK Characteristics of the SK EstimatorEstimator
(Monte Carlo Simulations)(Monte Carlo Simulations)
1Skewness: 0
1
M
Bias: 1
1E SKM
2Kurtosis Excess: 50
2
M
The probability distribution of the SK estimator remains asymmetric even for a fairly large accumulation number M, approaching normality at a slower pace than needed for practical applications.
Main goals of this study:
Redefine the SK estimator to remove bias
Find an analytical expression for probability distribution of the SK estimator to allow accurate calculation of its tail probabilities
March 29, 2010 7RFI Mitigation Workshop, Groningen The Netherlands
Starting PointStarting Point
22
1
Spectral Kurtosis estimator
11
MSMSK
M S
2 2
22 22
1 1
SK statistics is determined by the statistics of the
mean of squares
to square of mean ra
tio
k
k
SPMS M
S S PM
March 29, 2010 8RFI Mitigation Workshop, Groningen The Netherlands
Joint Distribution (Monte Joint Distribution (Monte Carlo): Carlo):
Mean of Squares-Square of Mean of Squares-Square of MeanMean SS22 and and SS1122 are are
strongly correlated strongly correlated random variablesrandom variables
Their (unknown) joint Their (unknown) joint distribution would be distribution would be needed to derive the needed to derive the distribution of their distribution of their ratioratio
Is there any work-Is there any work-around approach?around approach?
March 29, 2010 9RFI Mitigation Workshop, Groningen The Netherlands
Joint Distribution (Monte Carlo): Joint Distribution (Monte Carlo): Mean of Squares to Square of Mean Ratio - Mean of Squares to Square of Mean Ratio -
Square of MeanSquare of Mean
Monte Carlo simulations suggest:Monte Carlo simulations suggest:
SS22 / /SS112 2 and and SS112 2 are are uncorrelated random variablesuncorrelated random variables
Their individual distributions are Their individual distributions are independentindependent
This is the fundamental property that This is the fundamental property that makes the whole SK concept work makes the whole SK concept work by allowing SK to have a unity by allowing SK to have a unity value independently of the power value independently of the power levellevel
This property was analytically This property was analytically proven for the exponential proven for the exponential distribution based on first distribution based on first principles (Nita & Gary, PASP, principles (Nita & Gary, PASP, in press)in press)
March 29, 2010 10RFI Mitigation Workshop, Groningen The Netherlands
Raw statistical moments of the Raw statistical moments of the Mean of Squares to Square of Mean of Squares to Square of
Mean RatioMean Ratio
2 2
2 1 2 2 12 2
1 1
, 0MS S S MS S
Cov E E ES M M S M
,Cov x y E xy E x E y
2
22 21 1
SE
MS ME
S SE
M
2 1 1, ,n n n nCov f g n f g Cov f g
12 2( 1) 222 1 2 1 2 1
2 2 21 1 1
, , 0n nn n
MS S MS S MS SCov n Cov
S M S M S M
2
1
2
221
n
SM
nSEn MMS
ES
E
March 29, 2010 11RFI Mitigation Workshop, Groningen The Netherlands
Analytical Results Analytical Results (Nita & Gary, PASP, in press)(Nita & Gary, PASP, in press)
2 2
1
Raw moments of the square of mea
1 2 !
1 !
nn nM nS
M M ME
2
2
00
Raw moments of the mean of square
2
s
!
!
Mnn n nr
nr
t
rSt
M t rME
0
0
Raw moments of the mean of squares to square of mean
1 ! 2 !
1 2 !
ratio
!2
21
Mn n nr
nr
t
M M rt
M n t r
nMS
ES
March 29, 2010 12RFI Mitigation Workshop, Groningen The Netherlands
Redefinition of the SK Redefinition of the SK Estimator Estimator
2 22 2
1 1
Bias of the originally propose
11 1
d SK e
11 1 1
stimator
MS MSM M ME
MS
M SK
S M ME E
22
1
The unbiased SK esti
11
1
m
1
ator
SKMSM
M
E K
S
S
March 29, 2010 13RFI Mitigation Workshop, Groningen The Netherlands
/1
2
2 2
223
1 2 232
3 2
42 22
2
1 1 3/2
2 2 2
1
4 4 1
1 2 3
4 2 3 5 7
1 4 5
3 2 3 98 185 78
1 4 5 ( 6)( 7)
10 1
256 13
E SK
MO
M M M M M
M M M
M M M
M M M M M
M M M M M
OMM
OM M
Standard Moments of the Standard Moments of the SK EstimatorSK Estimator
March 29, 2010 14RFI Mitigation Workshop, Groningen The Netherlands
Moment based Moment based approximation of the SK approximation of the SK estimator distribution estimator distribution
using Pearson Probability using Pearson Probability CurvesCurves
2
1 2
2 1 2 1
3
4 4 3 2 3 6
March 29, 2010 15RFI Mitigation Workshop, Groningen The Netherlands
Pearson Type IV PDFPearson Type IV PDF
21 2 2
( )
Pearso
11
n 1895; Nagahara 1999
2
m i m ix x
p x Exp ArcTana aa m m
1
21
22 12 1
2 1
/1 2 1
( 2)
16( 1) ( 2)
6( 1) 16 1 2
2 3 6 4
2 1 ( 2)
H
einrich 20
2 4
04r r
r r
r a r r
rm r
March 29, 2010 16RFI Mitigation Workshop, Groningen The Netherlands
Pearson IV CF and CCFPearson IV CF and CCF
( ) ( ) ; 1 ( ) ( )x
x
P x p x dx P x p x dx
To compute the tail probabilities of the SK estimator, one To compute the tail probabilities of the SK estimator, one needs to evaluate the cumulative function (CF) and needs to evaluate the cumulative function (CF) and complementary cumulative function (CCF) of the Pearson complementary cumulative function (CCF) of the Pearson IV probability curveIV probability curve
Knowing the analytical expression for the Pearson IV Knowing the analytical expression for the Pearson IV PDF, the CF and CCF can be computed analytically, by PDF, the CF and CCF can be computed analytically, by using the closed form expressions involving using the closed form expressions involving Hypergeometric series provided Heinrich(2004) or Hypergeometric series provided Heinrich(2004) or Willink(2008). Alternatively, CF and CCF can be Willink(2008). Alternatively, CF and CCF can be computed by a simple numerical integration.computed by a simple numerical integration.
The asymmetrical RFI thresholds are then chosen so as The asymmetrical RFI thresholds are then chosen so as to provide symmetric tails probabilities of rejecting true to provide symmetric tails probabilities of rejecting true Gaussian signals of 0.13499%, which are equivalent to Gaussian signals of 0.13499%, which are equivalent to the ±3the ±3 thresholds of a normal distribution. thresholds of a normal distribution.
March 29, 2010 17RFI Mitigation Workshop, Groningen The Netherlands
RFI Threshold Computation RFI Threshold Computation ExampleExample
March 29, 2010 18RFI Mitigation Workshop, Groningen The Netherlands
Pearson IV PDF vs. Monte Carlo Pearson IV PDF vs. Monte Carlo SimulationsSimulations
March 29, 2010 19RFI Mitigation Workshop, Groningen The Netherlands
Time Domain Kurtosis Time Domain Kurtosis EstimatorEstimator
The DC (k=0) and Nyquist (k=N/2) frequency channels of a
FFT Spectrometer obey a different statistics.
The same statistics is obeyed by the power corresponding
to a narrow band time domain signal (FIR
1/21
filter s
pectrometer)
kP
kkp P P e
22
1
Following similar steps, an unbiased Time Domain Kurtosis (TDK) estimator
may be defined in
2
this case to detect RFI
conta
1
mina .
1
tion
MSMK
M S
March 29, 2010 20RFI Mitigation Workshop, Groningen The Netherlands
Standard Moments of the Standard Moments of the time domain Kurtosis time domain Kurtosis
EstimatorEstimator
/1
2
2 2
223
1 2 232
3 2
42 2
2
1 1 3/2
2 2 2
2
24 24 1
1 4 6
216 4 6 2
1 8 10
3 4 6 213 474 368
1 8 10 ( 12)( 14)
6 6 1
540 13
E K
MO
M M M M M
M M M
M M M
M M M M M
M M M M M
OMM
OM M
March 29, 2010 21RFI Mitigation Workshop, Groningen The Netherlands
Kurtosis estimator Pearson Kurtosis estimator Pearson IV PDF IV PDF
and RFI thresholds (M>45)and RFI thresholds (M>45)
March 29, 2010 22RFI Mitigation Workshop, Groningen The Netherlands
SummarySummary
The Spectral Kurtosis RFI excision algorithm recommends itself by
Conceptual simplicity
Frequency channel independence
Straightforward FPGA implementation (See Dale Gary’s following presentation)
Theoretically determined RFI thresholds for arbitrary integration time
March 29, 2010 23RFI Mitigation Workshop, Groningen The Netherlands
Main ReferencesMain References Gelu M. Nita and dale E. Gary, “Statistics of The Spectral
Kurtosis Estimator”, 2010, PASP, in press Gelu M. Nita, Dale E. Gary, Zhiwei Liu, Gordon J. Hurford,
& Stephen M. White, 2007, "Radio Frequency Interference Excision Using Spectral Domain Statistics," Publications Of The Astronomical Society Of The Pacific, 119, 805.
Yuichi Nagahara, 1999, "The PDF and CF of Pearson type IV distributions and the ML estimation of the parameters," Statistics & Probability Letters, 43, (1999), page 251.
Joel Heinrich, 2004, "A Guide to the Pearson Type IV Distribution," Collider Detector at Fermilab internal note 6820, 2004, http://www-cdf.fnal.gov/publications/cdf6820 pearson4.pdf
Willink, R., 2008, "A Closed-Form Expression For The Pearson Type IV Distribution Function,“ Aust. N. Z. J. Stat. 50(2), 199, 205