Iconicity, Frequency, and Sign length effects in TSL lexical processing

Iconicity, Frequency, and Sign Iconicity, Frequency, and Sign length effects in length effects in

TSL lexical processingTSL lexical processingYi-Hsuan ChiuYi-Hsuan Chiu

Cognitive Neuropsychology Laboratory, National YanCognitive Neuropsychology Laboratory, National Yang-Ming University, Taipei, Taiwang-Ming University, Taipei, Taiwan

Institute of Neuroscience, School of Life Science, NatioInstitute of Neuroscience, School of Life Science, National Yang-Ming University, Taipei, Taiwannal Yang-Ming University, Taipei, Taiwan

An animal or a tool?

DOG KEY

Iconicity of sign languageIconicity of sign languageThe components ofThe components of

LocationLocationHandshapeHandshapeMovementMovement

constrain the meaning of the objects!constrain the meaning of the objects!

Language ---------- Language ---------- ThoughtThought




Language ---------- Language ---------- ThoughtThought

iconiciiconicityty




iconiciiconicityty

Language --- Language --- Thought Thought

The usage of Sign languages in The usage of Sign languages in TaiwanTaiwan

Sign language in TaiwanSign language in TaiwanNature sign language (Taiwan Sign Language)Nature sign language (Taiwan Sign Language)

ArbitraryArbitraryIconicityIconicity

Vision-basedVision-based Manipulation-basedManipulation-based

{the spatial information (location, handshape, mo{the spatial information (location, handshape, movement) is meaningful}vement) is meaningful}

Chinese Character Sign LanguageChinese Character Sign Language

Iconicity in Sign LanguageIconicity in Sign Language

Visuion-basedVisuion-based Manipulation-Manipulation-basedbased

handshapehandshape All possibleAll possibleshape-for-shape iconicityshape-for-shape iconicityGINGER GINGER Path-for-shape iconicityPath-for-shape iconicityVACEVACE

Ease articulatorEase articulator

shape-for-shape iconicity shape-for-shape iconicity KITCHEN-KNIFE KITCHEN-KNIFE GLASSGLASS

locationlocation body-for-body iconicity body-for-body iconicity ELEPHANTELEPHANTNeutral: VACENeutral: VACE Neutral: KEY, GLASSNeutral: KEY, GLASS

movementmovement atemporal motion atemporal motion

Fictive motionFictive motion

Path-for-path iconicityPath-for-path iconicityFactive motionFactive motion

Sign lexical accessSign lexical access Task: Picture signing, picture delay signing, Task: Picture signing, picture delay signing,

picture naming, picture delay naming picture naming, picture delay naming Subjects: Subjects: deaf signerdeaf signer, , interpreterinterpreter (hear signer), and (hear signer), and

normalnormal (hear non-singer) (hear non-singer) Independent variablesIndependent variables

Iconicity: manipulation, visionIconicity: manipulation, vision Frequency: high, lowFrequency: high, low Sign length: short, longSign length: short, long

Dependent variable: reaction timeDependent variable: reaction time

+ &$Release button to sign/ name out

RT

Picture signing / picture namingPicture signing / picture naming

+ &$Release button to sign/ name out

RT

Picture delay signing / picture delay namingPicture delay signing / picture delay naming

+ &$

Release button to sign/ name out

RT

*

1s1s

Behavior studyBehavior study

Picture naming/ signingPicture naming/ signing Language modalityLanguage modality NamingNaming SigningSigning

Visual processing/Visual processing/ConceptConcept

0-150ms 0-150ms Lexical accessLexical access

150-275ms 150-275ms (lemma) (lemma) 275-400ms 275-400ms (Phonological encoding)(Phonological encoding)

Phonetic/ArticulationPhonetic/Articulation

400-600ms 400-600ms

Picture naming/ signingPicture naming/ signing

NamingNaming SigningSigning

ConceptConcept Iconicity: no difference (all noun)Iconicity: no difference (all noun)

Lexical accessLexical access Sign length: Rt (Short < Long)Sign length: Rt (Short < Long)

Frequency: Rt (High < Low)Frequency: Rt (High < Low)

ArticulationArticulation

RT

Picture delay naming/ signingPicture delay naming/ signing

Delay NamingDelay Naming Delay SigningDelay Signing

ConceptConcept Iconicity: no difference (all noun)Iconicity: no difference (all noun)

Lexical accessLexical access Sign length: Rt (ShortSign length: Rt (Short = = Long)Long)

Frequency: Rt (High Frequency: Rt (High == Low) Low)

ArticulationArticulation

RT

Language modality

SubjectsSubjects

Deaf Deaf InterpreterInterpreter NormalNormal

SigningSigning

NamingNaming ??

Hearing Hearing abilityability

Dif.: Hearing loss Dif.: Language experience

However, what is the interpreter’s mother tongue?

Deaf signing (n=24)Deaf signing (n=24)

All subjects learned TSL before their All subjects learned TSL before their puberty, and took TSL as their daily puberty, and took TSL as their daily language.language.

M=14, F=10M=14, F=10Mean age= 37.04 years-old (19-57ys)Mean age= 37.04 years-old (19-57ys)

Deaf and normal (N=24)Deaf and normal (N=24)Deaf signing and normal naming

300

800

1300

Signing and naming

Rea

ctio

n tim

e

1數列 657.4459492 448.3453684 932.1413069 397.8814922

Signing De. Signing Naming De. Naming

D/SD/S N/NN/N pp **

SubjectSubject(deaf/ normal)(deaf/ normal)

552.90552.90 665.01665.01 0.00000.0000 ****

TaskTask(signing/delay signing)(signing/delay signing)

794.79794.79 423.11423.11 0.00000.0000 ****

InteractionInteraction 0.00000.0000 ****

Deaf and normal (N=24)Deaf and normal (N=24)

pp **

InteractionInteraction 0.00000.0000 ****

sign-de. signsign-de. sign 657.45657.45 448.35448.35 0.00000.0000 ****

namede. namename- de. name 932.14932.14 397.88397.88 0.00000.0000 ****

sign-namesign-name 657.45657.45 932.14932.14 0.00000.0000 ****

de. sign- de. namede. sign- de. name 448.35448.35 397.88397.88 0.11200.1120

Deaf signing and normal naming

300

500

700

900

1100

Signing Naming

Signing or naming

Rea

ctio

n tim

e

Picture

Picture delay**

**

**

IconicityIconicity

Articulation

N=24N=24 DeafDeaf InterpreterInterpreter NormalNormal

Signing/Signing/

GesturingGesturingIconicityIconicity ** ****

FrequencyFrequency **** **** ****

Sign lengthSign length **** **** ****

Delay Delay signingsigning

IconicityIconicity

FrequencyFrequency

Sign lengthSign length

NamingNaming IconicityIconicity ****

FrequencyFrequency **** ****

Sign lengthSign length **** ****

Delay Delay namingnaming

IconicityIconicity

FrequencyFrequency

Sign lengthSign length **

ERP studyERP study

Picture naming/ signingPicture naming/ signing

Iconicity: (verb < noun)~ Iconicity: (verb < noun)~ (manipulation < vision) N400(manipulation < vision) N400

Frequency: 225-250ms (High < Low)Frequency: 225-250ms (High < Low)

350-450ms (High < Low) 350-450ms (High < Low) N400N400

Sign length: 60-120ms (Short < Long) Sign length: 60-120ms (Short < Long)

Time windowTime windowWindow (ms)Window (ms) IconicityIconicity FrequencyFrequency Sign lengthSign length

60-9060-90

90-12090-120

60-12060-120

125-175125-175

125-225125-225

150-200150-200

200-300200-300

250-350250-350

285-335285-335

350-400350-400

450-550450-550

550-600550-600

Location (channel) Location (channel)

Brain areasBrain areas ChannelsChannels Ico.Ico. Fre.Fre. Leng.Leng.

Frontal cortexFrontal cortex F1, FCz, F2F1, FCz, F2 M>VM>V

M<VM<V

H<LH<L

Inferior frontal (44)/ Inferior frontal (44)/ premotor cortex (6)premotor cortex (6)

FC5, FC6FC5, FC6 M>VM>V

Motor cortexMotor cortex C3, C4C3, C4 M>VM>V

Occipital cortexOccipital cortex O1, Oz, O2O1, Oz, O2 M<VM<V S<LS<L

Deaf SigningDeaf Signing (N=8) (N=8) andand Normal Normal naming naming (N=12)(N=12)

O2O1

OZ

T8

FCZ

T7

PO5

POZ

PO6

F6F5 FPZ

Subject: EEG file: name.avg Recorded : 12:55:36 24-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off

NeuroscanSCAN 4.3Printed : 16:47:45 28-May-2004

NounsNouns and and verbsverbs

Federmeier et al. 2000Federmeier et al. 2000

NounVerb

N400: N>V

Deaf signing (Iconicity: Deaf signing (Iconicity: MM//VV)) (N=8) (N=8)

ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0

*Msign_0.avgVsign_0.avg

Electrode: F1

Subject: EEG file: Msign_0.avg Recorded : 15:54:01 30-Apr-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off


ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0


Electrode: F2



ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0


Electrode: FZ



ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0


Electrode: PO3



ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0


Electrode: POZ



ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0


Electrode: PO4



ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0


Electrode: CP6



Frequency (Frequency (HH<<LL)) 225-250225-250: : HH<<LL 350-450350-450: : HH<<LL

Muente et al. Muente et al. 20012001

Deaf signing (Fre.: Deaf signing (Fre.: HH//LL)) (N=8) (N=8)

ms

-100.0 150.0 400.0 650.0 900.0

uV0.0

-1.0

-2.0

-3.0

-4.0

-5.0

1.0

2.0

3.0

4.0

*Hfresign_0.avgLfresign_0.avg

Electrode: F1

Subject: EEG file: Hfresign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off


ms

-100.0 150.0 400.0 650.0 900.0

uV0.0

-1.0

-2.0

-3.0

-4.0

-5.0

1.0

2.0

3.0

4.0


Electrode: F8



ms

-100.0 150.0 400.0 650.0 900.0

uV0.0

-1.0

-2.0

-3.0

-4.0

-5.0

1.0

2.0

3.0

4.0


Electrode: C4



ms

-100.0 150.0 400.0 650.0 900.0

uV0.0

-1.0

-2.0

-3.0

-4.0

-5.0

1.0

2.0

3.0

4.0


Electrode: C6



ms

-100.0 150.0 400.0 650.0 900.0

uV0.0

-1.0

-2.0

-3.0

-4.0

-5.0

1.0

2.0

3.0

4.0


Electrode: F3



225-250ms225-250ms:H<L

Sign length (Sign length (SS< < LL))

60-120ms60-120ms: : SS<<LL

Assadollahi et al. 2003 MEG

Short wordLong word

ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0

*Slensign_0.avgLlensign_0.avg

Electrode: OZ

Subject: EEG file: Slensign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off


ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0


Electrode: O1



ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0


Electrode: PO5



ms

-100.0 150.0 400.0 650.0 900.0uV 0.0

-1.0

-2.0

-3.0

-4.0

1.0

2.0

3.0

4.0


Electrode: PO6



Deaf signing (Leng.: Deaf signing (Leng.: SS//LL)) (N=8) (N=8)60-120ms60-120ms:S<L

ERP data summaryERP data summary

IconicityIconicity FrequencyFrequency Sign lengthSign length

N400N400

N>V N>V classclass

V>M V>M ico.ico.

225-250225-250

H<LH<L

60-12060-120

S<LS<L

Deaf signingDeaf signing ** ** **

Normal namingNormal naming **

Interpreter signingInterpreter signing ** S>L ?S>L ?

Interpreter namingInterpreter naming **

SummarySummaryFrequency, sign length effects in TSL lexiconFrequency, sign length effects in TSL lexicon~~

Chinese lexicon (spoken language lexicon)Chinese lexicon (spoken language lexicon)

Sign language Iconicity ~ Sign language Iconicity ~ semanticsemanticRT: signing< naming, delay signing~ delay namingRT: signing< naming, delay signing~ delay namingERP: 100-200, 200-300ms , signing< namingERP: 100-200, 200-300ms , signing< namingNormal gesture: RT: M<VNormal gesture: RT: M<V

Iconicity: Iconicity: ERP: Manipulation/Vision~ Verbs/NounsERP: Manipulation/Vision~ Verbs/Nouns

Documents

Iconicity, Frequency, and Sign length effects in TSL lexical processing