York H. Dobyns- Selection Versus Influence Revisited: New Method and Conclusions

8/3/2019 York H. Dobyns- Selection Versus Influence Revisited: New Method and Conclusions

1/15

Journal of Scientific Exploration, Vol. 10, No. 2, pp. 253-267, 1996 0892-3310/96 1996 Society for Scient i f ic Explorat ion

Selection Versus Influence Revisited:New M ethod and Conclusions

Y O R K H . D O B Y N SPrinceton Engineering Anomalies Research, C-131 Engineering Quadrangle, Princeton University,Princeton, N J 08544-5263

Abstract A previous paper by the author (Dobyns , 1993) on this topicmistakenly fails to account for nonindependence in certain test measures,leading to exaggerated conclusions. An analysis that avoids this problemproduces weaker stat ist ical ev idence , al though the qu ali ta t ive conclus ions ofth e earlier analys is are susta ined.1

I. BackgroundA pr iva t e com m un ica t ion f rom Je s s ica Ut t s ca ll ed m y a t t en t ion to a di f f icul tyin m y pre v iou s a r ti c le on the s e l ec t ion mode l for r e m o t e RE G e x pe r im e n t s( D o b y n s , 1993). The prob lem appears on p . 265, im m ed ia te ly following for-m u l a (6): "The agg rega te like l ihoo d of the hyp othe s i s over a l l th ree in ten t ion sm a y b e calcu la ted b y repeat ing th e i nd iv id ua l l ik e l i hood ca lcu l a t ion for eachi n t e n t i on . .. " Un fo r t u n a t e l y , w h i l e th e Bernou l l i fo rmu la u sed in eqs . (5) and(6) cor rec t ly acc oun t s for the cons t r a in t equa t ion s gove rn ing popu la t ion s an dprobab i l i t i e s wi th i n an i n t e n t i on , i t fa i l s to a c c o u n t for the n o n i n d e p e n d e n c ei n d u c e d b y t h e f u r t h e r cons t r a in t cond i t ion s ( s ee Sec t ion III below) opera t ingb e t w e e n i n t e n t i on s . Th e a l t e rna t i ve fo rm u la t ion d i s cu s sed la t e r i n t he s a m epa r a g r a ph fa i l s for the s ame rea son ; wh i l e th e ro l e s have been sw i t ched , th ef o rm u l a is s t i l l correct ing for one set of cons t r a in t s an d i g n o r i n g th e othe r .S ince t he componen t l i k e l i hoods do no t de r ive f rom i n d e pe n d e n t e v e n t s , th eagg rega te li k e l i hood form ed b y m u l t ip l y i n g t h e m is in error.Th e c on c l u s i on m igh t b e s a lvaged b y de r iv ing a correc t ion fac tor for the ef -f ec t s of n o n i n d e p e n d e n c e , b u t w i t h the raw da ta r ead i ly ava i l ab l e , it seemsm ore p rod uc t iv e to r e form u la t e t he an a ly s i s i n such a w ay a s to avo id t he n on -independence p rob l em en t i r e ly .

II. A Brief Reprise: Selection and Influence ModelsFor the cur re n t a r t ic le to s tand a lone , a br ie f d i scu ss ion of te rm s and ex per i -men ta l background s eems nece s sa ry . The expe r imen ta l da t aba se cons ide redcomes f rom r e m o t e e x pe r im e n t s u s i n g a Random Event Generator (REG), a1 am i nd eb t ed to Jess ica Ut t s fo r he r detec t ion an d c o m m u n i c a t i o n of the er ror i n the ear l ie r analys i s .T he Eng inee r i ng An om al i e s Resea rch p rog ram i s s u ppo rt ed i n pa r t b y g r an t s f rom t h e Fe t ze r I n s t i t u t e ,

Lau ran ce S . Rocke fe ll e r, and He l ix I nv e s tm en t s .

253


2/15

254 Y. H. D oby nsdev i c e w h i c h records d ig i t ized r a n d o m ou tpu t f rom a noi s e d iode or othe rsource. In t he se remote e x p e r i m e n t s , opera tors d i s t an t2 f rom t he dev ice a t -t e m p t to alter th e m a c h i n e ' s m e a n o u t p u t leve l . These e x p e r i m e n t s a re tripo-lar; each remote session comprises t h r e e consecutive r u n s s t a r ted a t t w e n t y -m i n u t e i n t e rva l s , w i th th e operator a t t e m p t i n g to i nc rea se th e m e a n o u t p u tleve l i n one of the ru ns (H i n t e n t i o n ) , decrease it in a n o t h e r (L i n t e n t i o n ) , a n dleave it una l t e red in a th i rd (B i n t en t i o n ) . Each of t he se run s cons i s ts of 2 x l05b i n a ry r andom s a mp le s collected as 1000 s u m s of 200 b i t s each . B e c a u s e ofth e lack of contac t b e t w e e n opera tor an d dev ice , i t can p l a u s i b l y b e proposedt h a t th e obse rved exp e r im en t a l effect der ives n ot f rom an ac tua l c h a n g e in theo u t p u t , b u t f rom j u d i c i o u s cho ice of i n t en t i o na l label ing to correspond wi thth e r andom o u t p u t s of an u n d i s t u r b e d dev i c e gene ra t i ng t r ipolar se t s . This typeof ef fec t c ou l d c o n s t i t u t e a g e n u i n e a n om a ly , i n w h i c h operators b y some a s -y e t - u n k n o w n a n d p roba b l y u n c o n s c i o u s m e a n s acq u i r e i n fo rmat ion abou t t heru n o u t c o m e s a n d choose t h e i r i n t e n t i o n s to s u i t , or i t cou ld represent a b r e a k -d o w n of the exper im en ta l con t ro l s in w h i c h opera tors someh ow l e a rn ed of theru n ou tcomes before report ing t h e i r i n t e n t i on s . Reg a rd l e s s of the de ta i l s , an ym o d e l i n wh ich t he effect is ach ieved b y selec t ing t he i n t en t ion order ing to fitothe rwise u n m od i f i e d ou t pu t can be cons ide red a selection model. In con t ra s t ,an influence model a s s u m e s t ha t any obse rved effects are due to ac tua l di f fe r -ences i n t h e machine's p e r f o r m a n c e u n d e r d i f f e r e n t i n t e n t i o n a l co nd i t i o n s . I ts h o u ld b e no t ed that t h i s a n a l y s i s does no t , st r ic t ly s p eak i ng , a dd r e s s t h e q u e s -tion of w h e t h e r t he re is in f ac t an effect ; i t a im s a t d i s t i n gu i s h i n g th e or ig in an dn a t u r e of an effect if one is presen t . The real i ty or o the rw i s e of the effect hasbeen d i s cu s s ed e ls ew he re ( D u n n e an d J a h n , 1992).

III. Redefined Rank FrequencyTh e 1993 p a p e r uses th e term rank frequency to refer to the f r e q ue nc y w i t hw h i c h a g iven i n t en t ion posses ses a g i v e n ord ina l r a n k wi t h i n i ts t r ipolar se t .S i n ce each ru n wi th in a t r ipolar se t has a def in i te rank and m u s t also be a s -s i g ned a def in i te i n t en t i o n , t he re a re n ine r a n k f r equenc ie s , w h i c h can m o s t in -

t e l l ig ib ly b e a r rayed in a 3 x 3 m a t r i x of i n t en t i o n s ve r su s ord ina l r a n k s . Th edua l con s t ra in t s of one of each r a n k , and one of each i n t en t i o n , in each t r ipolarset , man i f e s t as a set of f ive i n dep end en t con s t r a in t eq u a t ion s on these n in ema t r i x e l e me n t s . (Ac t u a l l y , t he re are s ix cons t ra in ing cond i t ions , one on eachrow and co lum n o f th e m a t r i x ; however , any one of the s ix eq ua t ion s m a y b eexpressed as a l i nea r com b in a t ion o f th e other f i v e , an d t h u s e l i m i n a t e d . )Th e n o n i n d e p e n d e n c e p rob lem can be avoided b y fo rmula t i ng t he p rob lemin terms of t r ipolar sets, r a t h e r t h a n i n d i v i d u a l r u n s . There a r e j u s t s i x d i s t i n c tw a y s in w h i c h th ree non iden t i ca l run ou t comes c a n b e a s s i g ned to the th ree in -t en t ions u n d e r th e protocol cons t r a in t t ha t each t r ipolar s e t conta i ns exac t ly

2Dis tan t is he r e t aken to m e a n t h a t th e opera tor -device separa t ion is a t least on th e order of a m i l e andf r e q uen t l y r anges u p to h u n d r e d s or t h o u s a n d s of miles . Fur th e r deta i l s of the r emo t e expe r im en t s can bef o und in D u n n e an d J a h n , 1992.


3/15

Select ion Versus I n f l u ence Revis i t ed 255one ins tance of each i n t en t ion . If we cons ide r th e f r equency wi th which tripo-la r sets appear in each of t hese poss ib le con f i g u r a t i on s , w e a re c lea r ly examin-ing a s ing le se t o f s ix exhaus t ive and m u t u a l l y exc lu s i ve a l te rna t i ve s , r a t he rt h a n t h r ee correlated sets of t h ree such a l te rna t ives ; the n o n i n d e p e n d e n c ep rob l e m t h a t c o r r u p t e d t h e p r e v iou s ana ly s i s then becomes i r r e l evan t . The re -m a i n d e r of t h i s ana ly s i s sha l l be ca s t in t e rms of t he se f requencies of the s ixposs ib le t r ipo la r r a n k i n g s . (Note t h a t t hese " rank f r equenc i e s " a re t h u s d e -f ined d i f f e r en t ly t h a n in the 1993 pap er.)W h e r e i n d i v i d u a l ident i f ica t ion i s neces sa ry , r ank f r eq uenc i e s wil l be la-beled by the i n t e n t i on a l subsc r ip t s a s s i gned to the highes t , m i d d l e , an d lowes tru n of the set, respectively. Thus pH B L refers to the f r equency of appearance ofth e "correct" t r ipolar label ing in w h i c h th e h ighes t r un i s a s s i gned to the highin t en t ion and t he lowest to the low i n t en t i on ; pB L H refers to the f r eq uency oftr ipolar sets in w h i c h the high es t run is labeled a base l ine , th e m i d d l e run a low,and t h e lowes t r un a h i g h ; and so fo r th . The k ey to the an a lys i s is t ha t i n f l u encean d se lec t ion m o d e l s predic t d i f f e r en t f u n c t i on a l r e la t ionsh ips b e t w e e n th er a n k f r eq uenc i e s a n d t h e d i s t r i b u t i on s ta t i s t i cs of the observed data.

IV. Observational DatabaseA s noted i n the 1993 paper, the da tabase compr i se s 494 t r ipolar se t s . Four oft hese sets conta i n t ie s be tw een i n t en t i on s , a c o n s e q u e n c e of the discre te na t u r e

of t he expe r im ent bu t no t one t h a t can r ead i ly be dea l t w i t h in this c o n t i n u o u sfo rma l i sm . They m a y be d i s ca rded w i t h o u t apprec i ab ly altering th e stat is t ics .The overa l l b i t - l eve l dev i a t i on s f rom expecta t ion i n t he r ema in ing 490 setsshow th e fol lowing m e a n s an d s t anda rd d e v i a t i o n s :

In ten t ionHBL

TABLE 1Bi t D ev i a t i on s

M e a n32.815.5782.102

Std Dev225.5212.5217.6

The t h eo re t i ca l ly expected d i s t r i b u t i on for these b it d e v i a t i on s i s no r m a l w i th am e a n = 0 and s tandard dev i a t i on s = 223.6.Howeve r , to c o n d u c t an ana lys i s a ga i n s t a selec t ion m o d e l on e m u s t no rma l -ize the da t a , no t by t he i r theore t ical d i s t r i bu t ion , bu t by t he emp i r i c a l m e a nand s t anda rd d e v i a t i on of the pooled data t hemse lve s . Th i s i s nece s sa ry toavo id i m p r o p e r pre jud ice a ga i n s t th e se lec tion m ode l . This m o d e l a s s u m e s th eproces s i s app lied to = 0, s = 1 norm al da ta , and i t s predic t ions neces sa r i lyha v e the proper ty t h a t i f the th ree i n t e n t i on s a re com bined i n to a s ingle collec-t i v e pool, th e agg r ega t e wil l h a v e = 0, s = 1. The i n f l u ence m o d e l , on theother h a n d , is i n d i f f e r e n t to a un i fo rm l i n e a r t r a n s f o r m a t i o n appl i ed to all of


4/15

256 Y. H. D ob yn sth e da ta . A n o t h e r wa y o f u n d e r s t a n d i n g th i s cons ide ra t ion i s to say th at the se -lect ion model , a t leas t as r ega rd s m e a n s an d v a r i a nc e s , predic t s th e relatives t a tu s of t he i n t en t ions w i t h i n th e aggrega te da tabase , r a t he r t h an t he i r ab -solute s t a tu s u n d e r th e machine's t heore t i ca l ou tpu t .So, w h e n the raw data are norma l i zed to h a v e an overa l l mean of 0 and s tan-dard dev i a t ion of 1 across all three i n t e n t i on s , the resu l ts are:

TABLE 2Normalized Sta t i s t i c s

In t en t ionHBL

Me a n0.0883-0.0362-0.0521

Std D ev1.03040.97070.9941

S kew-0.13390.1005-0.1803

Krt.-0.06970.1481-0.4903

Table 2 ha s i nc luded th e h i g h er m o m e n t s t h a t wil l b e u s ed in stat is t ical e v a lu a -t ions . These h a v e th e s a m e va l ue s for t he non -norm a l ized da t a , s ince t h e y a reuna f fec t ed by l i nea r t r ans forma t ions . The obse rved r ank f r eq uenc i e s are:TABLE 3

R a n k Frequenc i e sR a n k Orde r N (ou t of 490) Observed pHBL 92 0.188HLB 88 0.180BHL 80 0 .163BLH 79 0.161LHB 87 0.178LBH 64 0.131

The quoted values for p h a v e a o n e - s stat is t ical unce r t a i n t y of 0.017, du e toth e n u m b e r of obse rva t ions .V. Inferences from ModelsThe i n f l u e nc e model t rea t s the d i s t r i b u t i o n data in Table 2 as primary; the

expected r a n k f r eq uenc i e s c a n b e calculated f rom these d i s t r i bu t ion s t a t is t ic st h rough a s t ra igh t fo rward i f ted iou s process of nu m e r ic a l i n t egra t ion . The se-lect ion m odel , on the o the r h a n d , t rea t s the r ank f r equen c i es of Table 3 as pri-mary, and allows dis t r ibu t ion s ta t i s t ics to be ca lcu la ted from the m . I t i s not ,howeve r , immed ia t e ly obv ious how one may i n t e rp re t t he re su l t s of s uch cal-cu la t ions . In the on e case the pred ic t ion genera tes a set of d i s t r ibu t ion statist icsto be compared w i th t he obse rva t ion ; in the other, a set of ran k f r eq uenc i e s ispredicted. It is not clear how one may c o n s t r u c t a s ing l e goodnes s -o f - f i t para-meter that can be applied in both cases to compare the relative merits of thetw o hypo the s e s . Th e prev ious w o r k avoided this problem b y i nve r t ing th ef u n c t i on a l dependence of the selection model calculations ( t he i n t eg ra l s in-volved in the i n f l uence m o d e l are not read i ly inver t ible) , al lowing a ca lcu la t ion


5/15

Select ion Versus In f luence Revis i ted 257f rom d i s t r i b u t i o n s t a t is t ic s t o r ank f r equ enc i e s fo r both models. This a l lowed adi rec t com par i son o f rank f r equency pred ic t ions b e t w e e n t h e t w o m o d e l s . Un -fo r t una t e ly , th e r ede fined r an k f r equenc ie s in Table 3 are not a m e n a b l e to suchf unc t i ona l i nve r s ion for the select ion mode l ; t he re lev an t equa t ions a re non l in -ear in the r ank f r equenc ie s , and a d m i t of mu l t ip l e so lu t ions for a g iven set ofd i s t r i bu t ion s tat is t ics .In the a b s e nc e of a clear theore t ica l model for a goodness-of- f i t c ompa r i s on ,i t is none th e le s s pos s ib le to d e t e rm i ne th e good ne s s of fit for e a c h mod e l em -pir ica l ly via a Monte Carlo procedu re . This not only establishes each m o d e l ' si d ea l pred ic t ions for the i n pu t da t a , b u t also direct ly de t e rmine s t he d i s t r ibu-t ion of v a r i a t i on s in the p red i c t ions , m a k i n g it p o s s i b l e to tes t each model's f i tto the obse rved da ta w i t h o u t th e need for a f unc t i o na l i nve r s ion to create c o m -pa rab le pred ic t ions . As an added b o n u s , us ing th e ac tua l da ta in the M o n t eCar lo process a s su re s t h a t the real charac te r i s t i cs of the da ta a re be ing accou n t -ed for to exac t ly th e degree tha t they are s ta t i s t ical ly es tabl ished, w i t h o u t an yr isk t h a t s imp l i fy i ng a s s u m p t i o n s in a t heore ti ca l m ode l3 are d is tor t ing the pre-d ic t ions .

VI. Monte Carlo AlgorithmsSelection by Monte Carlo

Th e se l ec t ion mode l a s sum es tha t the da ta a re the resul t of a select ion proce-du re app l ied to the ex tan t t r ipolar sets . A cer ta in propor t ion of t hem are cor-r ec t ly i den t i f i e d as to t h e i r o r d i n a l r ank , wi t h t h e h i ghes t r un labeled H, thelowes t labe led L , and the m idd le run labeled B. Likewise v a ry i n g propor t ionsof th e t r ipolar se t s , as detai led in Table 3, are "mis labe led" to va r iou s degreesof i n accu racy .The ques t ion we ask o f the selec t ion model m ay be exp re ss ed t h u s : Giventhe 490 t r ipolar s e t s made avai lable for the se lec t ion p rocess , and g iven alsoth e r a n k f requenc ie s of Table 3 as the def in i t ion of the ef f ic iency of the selec-t ion proces s , h ow l i k e ly are the observed statist ics of the t h r ee i n t en t iona l dis-t r i b u t i on s ? Th e qu e s t i on , t h u s phras ed , i s in i tself nearly a speci f ica t ion of thedes i red Monte Car lo a lgor i thm. First , w e in te rna l ly sort the tr ipolar sets sot ha t , for each set , we can i den t i fy its high es t , low est , an d m i d m o s t e l e me n t . W et h en r a n d o m l y choose 92 of the 490 to receive the "correct" HBL label ing; w eass ign the HLB label ing ( h ighe s t ru n labeled H and lowest labeled B) to 88 r a n -d o m l y selected sets out of the r ema in i ng 398; and so for th . O n c e th e se t s hav ebeen d i s t r i b u t e d among the six possible i n t e n t i on a l l abe l i ng s according to thepopu la t ion f i gu res in Table 3, we f ind which r u n s h a v e been a s s i gned to eachof th e t h r ee i n t en t ion s and ca lcu la te in ten t iona l d i s t r ibu t ion statist ics accord-ing ly . Final ly , w e repea t th e who l e process m a n y t ime s , and s e e how t he ac t u a lt w e l v e - e l e m e n t mat r ix ( m e a n , s t anda rd dev i a t i on , skew , and kur tos i s for each

3A s, for e x a m p l e , t h e a s s u m pt ion o f no rma l i t y i n an i n f luence mode l i n t e g r a t io n .


6/15

258 Y. H. D oby nsof th ree i n t en t i o n s ) of in ten t ional s ta t i s t ics compares to the d i s t r i bu t ion ofm a n y s u ch m at r ices ca lcu la ted in the M o n t e Carlo process .Influence by Monte Carlo

For the i n f l uence mod e l , the ques t ion i s : How l ikely is the obse rved set ofr ank f r eq uenc i e s , g iven th e obse rved s tat is t ics of h ig h , low , and ba s e l i n e r u n s ?Here, th e procedure i s to pres e rve th e i n t e n t i o n a l i d en t i t y o f each r un and t os c r a mb le th e t r ipolar se t s . A n e w g r o u p of 490 tr ipolar se ts is created b y ran -d o m l y d r a w i n g ( w i t h o u t r e p l a c e m e n t ) on e each f rom th e h i g h , b a s e l i n e , a n dlo w da t a s e t s , un t i l th e da ta a re e x h a u s t e d . The r a n k f r equenc ie s of th i s re-ar ranged d a t a s e t a re t hen calcu la ted an d recorded. This process i s the n i te ra tedto bu i ld up an emp i r i c a l d i s t r i bu t ion of r ank f requen cy pred ic t ions .

VII. Goodness of Fit: The Em pirical Distance ParameterTo c o m p a r e th e s ingle set of obse rv a t iona l va lue s w i t h th e d i s t r i bu t ion s g en -erated by the M o n t e Car lo p rocedure , it is s im p l e s t to rega rd the s et of n u m b e r sas def in ing a s ing le po i n t in a m u l t i d i m e n s i o n a l space . For the se lec t ionm o d e l , w h i c h gene ra t e s t w e l v e s ta t i s t ica l measures , t h e p a r ame t e r space istwe l ve - d imen s iona l ; 4 for the i n f l u e n c e m o d e l , w h i c h produces s ix r ank fre-q u e n c y pred ic t ions , t he space is s i x - d im e n s ion a l . O nc e we s ta r t t h i n k i n g int e rms of a spa t ia l representa t ion of the da ta fo rmat , h o w e v e r m u l t i d i m e n s i o n -

al, it b e c o m e s q u i t e na t u r a l to t h i n k of s u m m a r i z i n g t he many d i f f e rence s be -t w e e n ( s a y ) a n y t w o M o n t e Car lo o u t c o m e s b y t h e distance b e t w e e n tw opo in t s in t h e s e m a n y - d i m e n s i on e d spaces .This p a r a m e t e r d i s t a n c e p r e s en t s th e u l t i m a t e key to q u a n t i f y i n g t h e qu e s -t ion of w h e t h e r th e obse rved va l u e s a re "like" or "u n l i ke " th e pred ic t ion se me rg i ng f rom t h e Mon t e Carlo calcu la t ion. For each mod e l , w e ca l cu l a t e th ecen t e rp o in t of the d i s t r ibu t ion of M o n t e Carlo ou tcomes b y t a k i ng th e m e a nv a l u e of each "coordinate ." We can t h e n ca l cu l a t e th e d i s t r i bu t ion of p a r a m e -te r d is tances from all of the i n d i v i d u a l Monte Ca r lo ou t comes to this cen t e r -

poin t , an d c ompa r e th is d i s t r i bu t ion to the d i s t ance be t we e n th e Mont e Carlocen t e rp o in t and the obse rved da t a . Figure 1 demons t r a t e s the appl ica t ion ofthis c onc e p t in a read i ly v i s u a l i z ab l e pa r a me t e r space of 2 d i m e n s i o n s . Th escat terplot show s 500 po in t s gene ra t ed w i t h a G a u s s i a n r ad i a l den s i t yp(x,y) oc e-r'12 =e-(x2+y2)/2.

(Note t h a t t h i s is e q u i v a l e n t to i n d e p e n d e n t v a r i a t i on s on b o t h th e x an d y4Actua l ly , the two cons t r a in ts on m e a n an d va r i a nce imposed by the se lec t ion m o d e l c o n f i n e th e

p o i n t s t o a t e n - d i m e n s i o n a l h y p e r s u r f ac e i n t h e t w e l v e - d i m e n s i o n a l space. This i s a u t o m a t i c a l l y h a n d l e dcorrect ly by the empi r i ca l t r e a tm en t , s i n c e t he no r m a l ized o bs e r va t io na l d a t a obey t h e s am e co n s t r a in t .A s imi lar d i m e n s i o n a l r ed u c t ion , caus ed by the cons t r a in t E p = 1 , applies to the r ank f r equ ency ca l cu l a -t io n ; aga in , the use o f an empi r i ca l dis t r ibu t ion obeying the s a m e co n s t r a in t i ns t ead of a t heore t i ca l cal-cu la t ion depend en t on t he num be r of d im en s io n s used au toma t i ca l ly compens a t e s fo r th is p roblem w i t h -ou t any need for expl ic i t ly t a k ing i t in to acco un t .


7/15

Selec t ion Versus In f luence Revis i t ed 259

Fig. 1. The d is tance prob lem i n tw o d im en s ion s .axes). The cen t e r of the d i s t r ibu t ion is m a r k e d wi th a filled circle; a few rad ia ll ines from th e cen t e rp o in t to some of the i n d i v i d u a l poin t s a r e s hown . Als os h o w n i s the r ad i a l l ine to an arbi t rar i ly chosen poin t x = 2, y = -0.5m a rke d bya d i a m o n d . If the statist ics of the scat tered po i n t s were no t k n o w n a priori w ecou ld con s t r u c t a stat is t ical test for the l ike l ihood t h a t t he d i am ond i s an ord i -nary m e m b e r of the d i s t r i bu t ion by com par ing i t s r ad ia l dis tance from th e col-l ec t ive cen te rpo in t wi th th e d i s t r i bu t ion of a l l othe r rad ia l d i s tances su ch as thee x a m p le s s h ow n .Axis Normalization

A n ext ra compl ica t ion appears in the M onte Carlo d i s t r ibu t ion of selec t ion-m o d e l stat is t ics, s ince the var ious s ta t i s t ical m easu re s ca lcu la ted a re no t equa l -ly s t ab le . This requ i re s tha t the dis tance m e a s u r e b e norma l i zed , as s hown inF igu re 2 .This f i gu re s hows two d i m e n s i o n s from an ac tua l s equence of 500 M onteCarlo selec t ion m o d e l r u n s , specif ical ly plott ing t he base l ine skew aga ins t th e


8/15

260 Y. H. D o b y n sba s e l i n e s t a nda rd d ev i a t ion . A ga i n , t h e i n d i v i d u a l run s a re plotted by po in t sand the cen t e r of the d is t r ibut ion i s shown by a filled circle. Figure 2a s h o w sth e d i s t r ibu t ion in ab so l u t e units ; i t is e v i d e n t t ha t t he skew i s i n t r i n s i ca l lymore va r i ab l e t h an th e s t anda rd dev ia t ion , a s expected f rom f i r s t pr inc ip le s .Th e d e n s i t y con tou r s are e l l ip t ical ra ther t h a n c i rcu la r in t he s e t w o d i m e n -s ions ; t he do t t ed e l lip se i s an ap p rox im ate con tou r . Th e sol id c i rc le , w i t h tw opoin t s m a r k e d , s h o w s w hy t h e d i s t a n c e calcula t ion ne e d s to normal ized fors uch cases . A l t hou gh al l poin ts on the circle c lea r ly sha re the sam e d i s t ance rfrom th e cen te r of the d i s t r i bu t ion , i t i s obv ious t h a t t he po in t m a r k e d b y af i lled d i a m o n d is f a i r ly typica l of the d i s t r i b u t i o n , w h i l e th e point marked b y acircle an d c ros sha i r s i s ex t remely a typ ica l . Since bo th t yp ica l an d e x t r e mepoin t s can share a c om m o n r a d i u s , t he non-normal ized r a d i u s is clearly not ana d e q u a t e representa t ion of how wel l a g i v e n po in t f i ts th e d i s t r i bu t ion . For il-lus t ra t ion , the open circle connec t ed to the dis t r ibu t ion cen te rpo in t by a r ad i a ll ine s h o w s th e va l ue s of t h e s e tw o p a r a m e t e r s in the obse rved da t a .Figu re 2b s h o w s th e effect of norm al izing , in th i s case by a mp l i f y i n g all lat-era l dis tances b y a su i t ab ly cho sen sca le factor. (The s a m e sca le is u s ed on thex-axis for display pu rpose s , a n d n o longe r reports th e ac tua l s t anda rd dev ia t ionv a l u e of the plotted r u n s . ) We can see tha t the dens i ty c on tou r s a re now ap-prox im a te ly c ir cu l a r and t h a t two poin t s a t t he s a m e dis tance r are in c o m p a r a -b le regions of the d i s t r i bu t i on , r ega rd le s s of t he i r a n g u l a r pos i t i on . The r e sca l -in g h as sh i f ted th e position of the observed da t a p o i n t as w e l l as of thei n d i v i d u a l M o n t e Car lo ou tcomes ; i ts renormal ized r ad iu s i s now su i t ab l e fo rcomp a r i son w i t h th e renormal ized rad i i of the i n d i v i d u a l M o n t e Carlo ou t -comes as d i s cu s sed wi th th e e x a m p l e of Figure 1.

VIII. Conclusions from Monte CarloTo ev a lua te the two mod e l s , 105 i t e r a t ion s of M onte Carlo w ere run for e a c h .Th e dis tances of i n d i v i d u a l Monte Carlo r u n s from th e aggrega t e popu la t ioncen t e r p o i n t s were ca l cu l a ted an d b i n n e d to establish th e p r ob a b i li t y d e n s i t y of

th e distance parameter for each model. Th e r e su l t s a re s h own g r a ph i c a l l y inFigure 3. The raw bin popu l a t i on s a re s h o w n by the scat terp lot of crosses ; asmoothed vers ion i s i l lus t ra ted b y a con t inuous l ine . Th e posi t ion of the ac tua lobserved da ta on the dis tance scale is s h o w n b y a labeled ver t ica l sp ike . It isq u i t e ev i den t t h a t th e ac tua l da ta fall mode ra t e l y far out on the tai l of the se-lection m o d e l ' s d i s t r ibu t ion o f pred ic t ions (top graph in Figure 3) , w h i l e th e yare qu i t e close to the peak of the d is t r ibut ion of p red ic t ions from th e i n f l uencem o d e l ( lower graph in Figu re 3). The tai l-area p-v a lu e can be ca lcu la ted q u i t edi rec t ly s im p l y by coun t i ng t h e numbe r o f r u n s i n the upper tai l for eachmodel, t h a t i s , t he f r a c t ion of the Monte Carlo r u n s t h a t a re m o r e u n l i k e t h e a v -e r age pred ic t ion than the observ a t ion. For the se lec t ion m odel , th i s p-va lue i s0.0296 0.0005 ( the u n c e r t a i n t y qu oted i s the s ta t i s t ical one-s error in estab-l ishing a b i nomi a l probab i l i ty f rom 10 5 obse rva t ion s ) . For the i n f l uencemodel, on the other h a n d , p =0.347 0.002. T h u s , by a s t anda r d p = 0.05 sig-


9/15

Selec t ion Versus In f luence Revis i ted 261

Fig . 2. Wh y norm al iza t ion is need ed .n i f i cance c r i t e r ion , th e pred ic t ions of the se lec t ion m odel c an b e d i s t ingu i shedf rom th e s t ruc tu re of the observed da ta , whe r e a s th e pred ic t ions of the in f lu-ence m o d e l cannot . Or, to expres s the con sequence s of the M onte Car lo ana l y -s is more d i rec t l y : W h e n i n f luence is a s s u m e d , and the exis t ing da ta d i s t r ibu-t ions a re u s e d to cons t ruc t rank f requenc ie s , the result is s tat is t icallyi nd i s t i ngu i shab l e f rom th e ac tua l da t a . In con t ra s t , w hen s e lec t ion is a s s u m e d ,and the exis t ing r a n k f r e qu e n c ie s an d t r ipolar sets a re used to cons t ruc t da t adis t r ibu t ions , the resu l t is s ta t i s t ical ly dis t inc t f rom t he ac tua l da ta s t ruc tu re .


10/15

262 Y . H . D ob y n sIX. The oretical Concerns

The Nelson ProblemR. D. Nelso n , in pr iva te c o m m u n i c a t i o n , out l ined a pos s ib i l i ty for the selec-

tion proces s t ha t w o u l d inc rease th e d i f f i c u l ty of da ta i n t e rp re t a t ion by a t leas tan order o f m a g n i t u d e .T h e cu r r en t a n a l y s i s p r e s u m e s t h a t th e h y po t h e t i c a l selection b y t h e opera-tor is pu re ly qua l i t a t i ve . I t i s a s s umed , fo r t h a t m o d e l , tha t the operator h assome e r r a t i c ab i l i t y to d i s ce rn w h i c h of the three r u n m e a n s i s h i gh e s t ( or low-est , etc.). W h a t i f the eff icacy of s u c h an abi l i ty is cond i t ioned by the dis t inc t -ne s s of t he run s? To d raw a v i s u a l analogy, i t is clear t h a t h u m a n s can dis t in-gu i sh two d i f f e r en t p r im a ry colors u n d e r m u c h more adve r s e cond i t ion s t hantw o l ight ly con t ra s t i ng s h a d e s of be i g e . It does n ot seem u n r e a s on a b l e a priorit ha t if t he hu m an pa r t i c ipan t s have s o m e ab i l i ty to d i s t i ngu i s h an d sort a m o n gt h e e x pe r im e n t a l r u n s , t h e y cou ld m o r e r ead i ly d i s ti ngu i sh a ( norm a l ized ) sp l i tof 3 b e t w e e n t wo i n t e n t i on s t h a n one of 0.001.If t he rank f r equenc ie s a re da ta -depen den t , t he p rob lem of pred ic t ing the ex-pected selec t ion d i s t r i bu t ion s from t h e m be c ome s v e ry m u c h harder . For onet h ing , th e obse rved r ank f r eq uenc i e s a re a lr e ady s om ew ha t unce r t a in , s i mp lydue to the l imi ted n u m b e r of obse rva t ion s ava i l ab l e to cha rac t e r i ze t h e m . Ift he add i t iona l d i m e n s i o n of var ia t ion w i t h regard to run m e a n is a d d e d , w eh a v e n o hope of be ing ab le to ch a rac te r ize t h e i r var ia t ion on the bas is of theda t a , an d w o u l d h a v e to a s s u m e a m o d e l for s u ch v a r i a t i on . Fu r t he rmo re , eveng i v e n a m o d e l , th e c a l c u l a t i o n of expected s t a t i s t i cs f rom s u c h v a r i a b l e r a n kf r e q ue nc i e s becomes q u i t e i n t r a c t a b l e .Fortuna te ly , the propos i t ion of da t a - dep enden t r ank f r eq uenc i e s isa m e n a b l e to a d i rec t test, or ra the r , to seve ra l . It is neces sa ry f i r s t to qu a n t i f yth e degree of accu r acy an operator displays in m a k i n g a pa r t i cu l a r a s s i g n m e n tof i n t e n t i o n s to a t r ipolar set.Clear ly , t here i s s o m e sense in w h i c h th e "HBL"a s s i g n m e n t is "comple te ly r i gh t " and the b a c k w a r d s "LBH" a s s i g n m e n t i s"comple t e ly wrong , " bu t how shou ld i n t e rm e d i a t e a s s i g n m e n t orde rs ber a n k e d ? There a re th ree b ina ry dec i s ions t h a t can be m a d e in eva l u a t i n g th erela t ive r a n k i n g s of a tr ipolar set: I s the H run higher t h a n the B? I s the B runh ighe r t h a n the L? I s the H run h i ghe r t h a n the L? ( I t shou ld be no ted t h a t thesethree dec i s ions are not i n d e p e n d e n t , b u t th is i s i r re levan t to the ana ly s i s . ) I f w edef i ne an accu r acy i n d e x by t h e n u m be r o f t he se c on d i t i on s t h a t are sat is f iedby a g iven set, w e f i nd tha t a se t in the HBL order has an accu racy i ndex of 3 ,w h i l e a se t in the LBH order has an i ndex o f 0 . For t h e o the r f o u r o r de r i ng s ,both HLB and BHL have an i n d e x of 2 , wh i l e bo th BLH and LHB have ani n dex of 1. Since , in each case , the re is no o b v i o u s qua l i t a t i ve way in wh ichone of the two r a n k i n g s w i th an equ a l i ndex is "better" or "more accu r a t e " t h a nth e other , th i s i n d e x s e e m s a sat isfactory qu a n t i t a t i v e measu re o f t h e s ome -w h a t vag ue no t ion of accu racy in ju d g e m e n t .Th e othe r m e a s u r e of in teres t is the span of the t r ipolar set, th e in t e rva l b e-


11/15

Selec t ion Ve r s u s I n f l u e n c e Rev i s i t ed 263

Fig. 3. Fit of theory to o b s e r v a t i o n , both models.


12/15

264 Y. H. D o b y n s

Fig. 4. Accuracy a s a f u n c t i o n of span.t we e n i ts h ighes t an d lowest e l emen t s . The da t a -depend en t s e lec tion hypoth e -s is posi ts that accuracy should b e greates t for t ho s e sets with th e w idest span .F igu re 4 i l lus t ra tes tha t th i s is not the ca s e . Th e icons w i t h error bars show th eave rage accu racy i n d e x an d associa ted s t anda rd error, as a f unc t i o n of the sp anof th e tripolar set. These averages were ca l cu l a t ed by a b i n n i n g process; th ef i r s t point represents th e average acc u ra cy fo r a l l sets w i t h a s pa n les s t h a n 0.5,and so for th . Th e r i gh tmos t b in , nomina l l y [3.5,4.0], i nc lude s a con t r i bu t ionf rom tw o sets w i th s pa n v a l u e s greater t h a n 4 . This i n c l u s i on does n ot appre-ciably change i t s s ta t i s t ics . Tw o regression lines w i t h t h e i r 95% c o n f i d e n c e h y -perbolas are shown: the sol id l ine is a we ig h t ed regress ion to the averagedpoin t s , th e dotted l ine i s the reg ress ion to the ac tua l 490 span an d accu racy va l -ues . This latter may be considered more accurate, s i n ce some i n f o r ma t i o n is in-ev i t ab ly lost t h roug h t h e b i nn i ng proces s ; b i n n i n g wa s c ond u c t e d s imp ly b e-c a u s e th e scat terp lot of the 490 accuracy va l ue s ( no t s hown) is di f f icul t an du n i n f o r m a t i v e to judge b y eye.The conc lus ion of the r egres s ions is clear: wh i l e s o m e slope is v i s i b l e in theregression l ines , the confidence hyperbolas i nc lude l ines of the opposite slope,and in c o n s e q u e n c e th e slope of the regress ion l i ne i s not stat is t ically dis t in-gu i s h a b l e f rom zero. Looking at the b inw i se ave rages w e can no te s o m e mod-es t sugges t ion , not of a t r end , but of s o m e k i n d of d i s t in c t ion: the d a t a appearto cons is t of two g roup s , an ex t r eme g roup of very large an d sma l l spans forwh i c h accuracy is poor, and a r a nge of i n t e rmed ia t e spans for w h i c h accuracyis s o m e w h a t bet ter . Resolv ing the re al i ty of th i s a ppa r e n t s t r u c t u r e m us t aw a i tth e collection of more da ta , preferably in i n d e p e n d e n t repl ica t ions . For thepurposes of the cur rent ana lys i s , i t i s suf f ic ien t to no te t ha t a ny s pa n -d e pe n -dence of the r a n k f r equenc i e s is too w e a k to be de tec tab le in the e x p e r i m e n t a lda t aba se . W e m a y therefore resolve th e Ne l s on p rob l e m b y no t ing t h a t the as -


13/15

Selec t ion Versus In f luence Revis i ted 265s u m p t i on of cons tan t se lec t ion ef f ic iency used in the foregoing ana lys is i s anadeq ua t e app rox im at ion fo r t rea t ing t he g iven d a t a b a s e .Timing Selection

Th e ana ly s i s above has addressed a select ion m o d e l in w h i c h in tent ions a rea s s i g ned to s u i t th e respect ive ou t c ome s of a tr ipolar set, or at leas t th e proba-bi l i ty of a m a t c h is i nc reased by s ome anoma lou s k n o w l e d g e on the par t of theoperator. Th e remote protocol u sed a t P EAR does , however , a l low one ( a ndonly o n e ) other vo l i t iona l choice by the opera tor , n a m e l y th e t i me a t w h i c hda ta collection is to start . This al lows ano the r po ten t i a l ven ue in w hich a selec-t ion process could operate : r a t he r t h a n choos ing th e in ten t iona l order to fi t theou tcom e, an opera to r aw are of the m a c h i n e ' s f u t u r e behav io r cou ld choose tos tar t col lec t ing da t a a t a momen t w h e n its va r i a t ions w ou l d correspond to acho sen i n t en t i o na l order .The opera tor h as only a s ingle choice of t im ing for each t r ipolar set ; th e sec-ond and th i rd r u n s are started a t tw e n t y - m in u t e i n t e rva l s after th e f i rs t . Thism e a n s t h a t , as wi th i n t en t iona l se l ec t ion , t im ing select ion is a process t h a tm u s t b e ana lyzed in t e rms of ent ire t r ipolar sets ra the r than i n d i v i d u a l ru n s .Clear ly , t i m i ng selec t ion i s poten t ia l ly fa r more power fu l t han in ten t iona lse lec t ion. A perfectly eff icient i n t en t iona l select ion process is l imited in i tsabil i t ies . The bes t i t can d o, by label ing each s et opt imal ly , is to put the high in -ten t ion in the d i s t r i bu t ion gene ra t ed by t ak ing th e h ighes t of th ree i nd e pe nd e n ts t anda rd no rma l d ev i a te s , and the low i n t en t ion in the sym m et r i c lowes t -o f-three d i s t r i b u t i o n . In contrast, a pe r fec t ly e f f i c i en t t i m i n g selector is limitedon ly by the numbe r o f poss ible ou tcomes ava i l ab le for choice . Given a suf f i -c i en t ly broad " m e n u " of al te rna t ives , a t iming selector wi th perfect d iscr imi-na t ion cou ld create an y ou tpu t dis t r ibut ion desired for the th ree i n t en t iona l cat-egor ies .Howeve r , w i t h a s ingle cons t ra in t , t im ing select ion can be analyzed wi th th es a m e tools u s e d above i n fact , i t m akes ex ac t ly the s a m e predic t ions as in-t en t iona l selec t ion , and therefore th e s a m e conc l u s ion s already reached wi l lapply . Th e cons t r a in t is s im p l y t ha t an opera tor w ho i s u s ing t im ing select ionto f avo r des i rab le resu l t s wil l tend on ly to choose a m o m e n t t ha t p roduces re-sul t s in l ine wi th t he i n t en t ion s , w i t h o u t f u r t h e r op t imiz ing th e ou tcome . Inothe r w ords , i f one a s sum es t h a t an opera tor sea rch ing (pe rhaps subconsc ious -ly) for an ausp ic ious t im e to beg in th e ser ies is sat isfied by f i nd ing s ome m o-m e n t th a t g ive s m e a n sh i f t s in the decla red d i rec t ions , r a t he r t han sea rch ingam ong a broad range of pos s ibi l i t ies to f ind th e very bes t , t im i n g select ion pre-d ic t s the same re l a t ionsh ip s be t we e n r a n k f requenc ie s an d dis t r ibut ion para -mete r s as i n t en t i o na l se l ec t ion .

Th e reason for th is shou ld b e clear. Cons ide r a cons t ra ined t iming select ionprocess tha t i s a lw ay s s u cce s sf u l at choosing th e HBL order. By the cons t ra in ta s sumpt ion , i t s ou tpu t s a r e unb i a s ed se lec t ions from the d is t r ibut ion of t r ipo-la r se t s tha t happen to be in the correct order. The ou tpu t of the process , over


14/15

266 Y. H. D oby nsm a n y s amp l e s , is e q u i v a l e n t to the a l g o r i t h m : "Genera te a t r ipolar se t w i th a t-t ached i n t en t ion labels . If i t is in the correc t order, ke e p it; if not, discard an dtry aga in . " An in ten t ional se lec t ion process t h a t a l w a y s s ucceed s is e q u i v a l e n tto t h e a l g o r i t hm: "Gene ra t e a tripolar set. Label the lowes t L, the h ighes t H,and t he thi rd B." But both of these p rocedures wil l create exac t ly th e s a m e dis-t r i bu t ions . In each ca se t he p robab i l i t y den s i t y of H r u n s is g i v e n (up to anove ra ll norm a l iza tion ) by the jo in t p rob ab i l i t y t ha t a n o r m a l d e v i a t e wil l t akeon a g i v e n v a l u e w h i l e tw o o the r i n d e p e n d e n t n o r m a l dev ia te s t ake on lowerva l ue s . Th e probab i li t y d i s t r i bu t ion s for L and B run v a l u e s are l i kew i se i den t i -cal in both processes .The c on s t r a i n t on t im in g selec t ion m ay seem arb i t ra ry , bu t in fact it is h igh lyp laus ib le for the da ta se t u n d e r con s i d e r a t i on . It is o b v i o u s t h a t th e opera torsare not f unc t i on i ng a n y w h e r e near the r e g i m e of perfec t eff iciency. Table 3s how s t h a t , i f a selection process i s opera t ing , i t achieves only a mod e s t in -c rease in the probab i l i ty of correct ly labeled se ts . If t h i s is the ou t come of at im ing selec t ion process , i t s e e m s t h a t operators are f r equen t ly w rong in t he i rj u d g e m e n t th a t a g iven i n i t i a t ion t im e wi l l p roduce r e su l t s in the desired direc-t ion. Is i t cred ib le t h a t t h e y a re m a n a g i n g to op t imize th e i r cho ices wi th i n th edis t r ibu t ion of correct ly- labeled se ts , g iven tha t t hey are only marg ina l ly suc-ce s s f u l a t i d en t i f y i ng s u c h sets a t all? Th e solu t ion of the N elson prob lem dis-cu s s ed in the preceding sect ion a lso s uppor t s th e not ion tha t operators do no tseem to be opt im iz ing t h e i r s ucces s fu l cho ices .In short, th e limited e f f i c i ency o f ra n k f r e q u e n c y se l ec t i on seen i n Table 3 ,and the neg a t i v e o u t c o m e of the Nelson tes t , s t rong ly i nd i ca t e t h a t an y t i m i n gselect ion process p r e s e n t m u s t b e operat ing in a r e g im e w h e r e its effects are in -d i s t i ngu i shab l e from i n t en t ion -ba sed selec t ion. If th is is so, then t i m i ng selec-t ion u n d e r t hese c i r c u m s t a n c e s m a k e s th e s a m e pred ic t ions a s an i n t en t ion se -l ec t ion mode l , and the p ~ 0.03 reject ion seen above appl ies to it as wel l .T iming se l ec tion toge the r w i t h i n t en t ion se l ec tion span s the poss ib l e range o fs e lec t ion models for th i s e x p e r i m e n t , so t h e r e s u l t c an w i t h considerable conf i -d e n c e b e appl ied to all selec t ion m o d e l s for th i s par t i cu la r d a t a b a s e . A morege n e r a l i z e d t i m i n g se l ec t i on model i s no t r e f u t ed , b u t m u s t a t a m i n i m u m i n -c l ude s o m e explana t ion for the odd i ty t h a t operators w h o a r e on ly modes t lysucces s fu l a t choos ing s t a rt ing t im es t ha t m a k e the H run high and the L runlo w none the l e s s a r e c hoos i n g t h e i r m om e n t s so clever ly as to s p u r i o u s l ym i m i c th e stat is t ical features of an i n f l uence m o d e l .

X. ConclusionsW e arr ive , f ina l ly , at the co nc lu s i o n t ha t th e select ion hypo thes i s g ives apoor f i t to the da ta s t r uc t u r e , wh i le th e i n f l uence hypo thes i s g ive s abou t a s

good a f it a s can be expected. Th e r e s u l t is s ta t i s t ica l ly w e a k e r t h a n t h a t report-ed in 1993:p ~ 0.03 i n s t ead o f p = 0.0095 a s i n the p r ev iou s a n a l y s i s . Since it isk n o w n t h a t th e co n f o und i n the p rev ious ana ly s i s was an in f la t ion of its s i gn i f -i c ance , i t shou ld not be su rp r is ing tha t t h i s is the ca se .


15/15

Selec t ion Versu s I n f luence Revis i ted 267W e there fore m ay sa fe ly con t inu e to conc lude , a lbe i t w i t h less force, th at theobserved da t a do no t s u ppo r t a run -ba s ed select ion process for the app aren t re-mot e RE G a n om a l y . A s i d e f rom its im p l i c a t i on s for theore t ica l mod e l i ng , th isalso reinforces th e va l i d i t y of the e x p e r i m e n t , s i nce fa i lure of the e xpe r i me n t a lcon t ro l s w o u l d man i f e s t a s j u s t s uch a r un - ba s ed select ion effect .

ReferencesD o b y n s , Y. H. (1993). Selec t ion v e r s u s i n f l u ence in r emote REG a n o m a l i e s . Journal of Scientific

Exploration, 7, 259.D u n n e , B . J ., a n d J a h n , R . G . ( 1 9 9 2 ). E x p e r i m e n t s i n remote h u m a n / m a c h i n e i n t e r a c t i o n . Journalof Scient i f ic Exploration, 6,311.

Documents

York H. Dobyns- Selection Versus Influence Revisited: New Method and Conclusions