9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Vision, Spiders & Time - 1 Vision, Spiders & Time David Pierre Leibovitz

April 10, 2023April 10, 2023 David Pierre Leibovitz (Carleton University)David Pierre Leibovitz (Carleton University) Vision, Spiders & Time - Vision, Spiders & Time - 11

Vision, Spiders & TimeVision, Spiders & Time

David Pierre LeibovitzDavid Pierre Leibovitz

April 10, 2023April 10, 2023

April 10, 2023April 10, 2023 David Pierre Leibovitz (Carleton University)David Pierre Leibovitz (Carleton University) Time Matters - Time Matters - 22

SummarySummaryHow is vision perception related to imagination and How is vision perception related to imagination and planningplanning

What is the role of attention (saccades)What is the role of attention (saccades)

Can smart spiders shed light on human cognition?Can smart spiders shed light on human cognition?– They have severe engineering restrictionsThey have severe engineering restrictions– They they take a long time to thinkThey they take a long time to think– How does that affect cognitionHow does that affect cognition

I will relate spider time to practical mattersI will relate spider time to practical matters

Hopefully, you will also come to appreciate spiders Hopefully, you will also come to appreciate spiders as well as well

Human Vision Like a Camera?Human Vision Like a Camera?Static Camera vs. VideoStatic Camera vs. Video– Do we process changes?Do we process changes?

Process entire scene or saccade/attend to Process entire scene or saccade/attend to othersothers

Is it process-interpret-act or a Is it process-interpret-act or a top-down/bottom-up iterative thingtop-down/bottom-up iterative thing

What about imagination?What about imagination?



VisionVisionMovies @ 24hz (42ms/frame)Movies @ 24hz (42ms/frame)– But flicker at higher rates?But flicker at higher rates?

Major saccades @ 5hz (20-200ms/move)Major saccades @ 5hz (20-200ms/move)– Can miss Can miss


SaccadesSaccades


SaccadesSaccadesMajorMajor

(Clinically observable)(Clinically observable)MinorMinor

(Hard to observe)(Hard to observe)

LargeLarge SmallSmall Mini/FlickMini/Flick Micro/Micro/TremorTremor

Angular Angular DistanceDistance

>6.2>6.2°° 6.26.2° - ° - 1.21.2°° 1.21.2°° – 2’ – 2’ 40”40”

OccurrenceOccurrence .05 - .3hz.05 - .3hz Up to 5hzUp to 5hz 40hz - 40hz - - 200hz- 200hz

TimeTime 200ms200ms 20-200ms20-200ms 10-20ms10-20ms <10ms<10ms

ReasonReason Head, Body Head, Body Orientation Orientation ChangesChanges

View other View other features of features of object, next object, next character character groupinggrouping

Refresh Refresh Rod/Cone Rod/Cone SignallingSignalling

Extra Extra Detail?Detail?


Saccadic Eye MotionSaccadic Eye MotionMajor SaccadesMajor Saccades– Large SaccadesLarge Saccades

– Enables the building up of a high resolution mental map from “interesting” parts of a Enables the building up of a high resolution mental map from “interesting” parts of a scenescene

– The fastest part of the human body (1000The fastest part of the human body (1000°°/s)./s).Microsaccades (20 arc seconds @ ~90Hz) serve to refresh image on rod and cone Microsaccades (20 arc seconds @ ~90Hz) serve to refresh image on rod and cone cells which respond to changes in luminance (otherwise a fixed stare would cause cells which respond to changes in luminance (otherwise a fixed stare would cause an image to slowly disappear)an image to slowly disappear)– It is possible that this also improves resolutionIt is possible that this also improves resolution– Jitter cameraJitter camera– 1 pixel camera1 pixel camera

During saccade, details are masked (mirror experiment). Motion of eye, blur of During saccade, details are masked (mirror experiment). Motion of eye, blur of image nor time gap are perceived.image nor time gap are perceived.– Can’t see flash of light during saccadeCan’t see flash of light during saccade– Can’t see object moved during saccadeCan’t see object moved during saccade– However, masking stops if blur stops, i.e., when saccade follows a moving object. During However, masking stops if blur stops, i.e., when saccade follows a moving object. During

a 1ms a “snapshot” can be taken (+250ms for later processing).a 1ms a “snapshot” can be taken (+250ms for later processing).Suppressing (magnocellular) M-pathway responsible for motion responding to transient, high-Suppressing (magnocellular) M-pathway responsible for motion responding to transient, high-velocity stimuli of low spatial frequencyvelocity stimuli of low spatial frequencyEnhancing (parvocellular) P-pathway responsible for colour informationEnhancing (parvocellular) P-pathway responsible for colour information tradeoff to differentially process changes over time due to motiontradeoff to differentially process changes over time due to motion

– Optokinetic reflex – ability to follow (slower) motion with eyes fixedOptokinetic reflex – ability to follow (slower) motion with eyes fixed



Jitter CameraJitter CameraSame as interlacingSame as interlacing

Related to micro-saccadesRelated to micro-saccades

Jitter (which doubles the sensing time) Jitter (which doubles the sensing time) doubles perceptual resolutiondoubles perceptual resolution


Visual NeglectVisual NeglectIn Visual Neglect, Patients can only see, or In Visual Neglect, Patients can only see, or imagine half a scene.imagine half a scene.

However, they can mentally rotate 180° and However, they can mentally rotate 180° and then imagine the other half (from the other then imagine the other half (from the other perspective)perspective)

This lends credence to the view that each side This lends credence to the view that each side GENERATEs a view.GENERATEs a view.


Scanning Tunneling MicroscopeScanning Tunneling Microscope

Example of aExample of a1 Pixel Camera1 Pixel CameraNot even optical! Not limited by diffractionNot even optical! Not limited by diffractionHighest resolution microscope (0.2 nm)Highest resolution microscope (0.2 nm)

Gold (Au) showing individual atoms


Jumping Spider - Jumping Spider - Portia labiataPortia labiata““1” pixel animal1” pixel animal

Cognitive AbilitiesCognitive Abilities– Scene buildingScene building– Route planning/plotting (15-Route planning/plotting (15-

60 minutes)60 minutes)– Plan execution (several Plan execution (several

hours)hours)Goal may be invisibleGoal may be invisible

– Lure via adaptive mimicryLure via adaptive mimicryTickle web to mimic other Tickle web to mimic other animals (or try different animals (or try different patterns)patterns)

– Tactical Attack decisionsTactical Attack decisionsIf prey carrying egg-sac, If prey carrying egg-sac, mount frontal assault (don’t mount frontal assault (don’t worry about spitting spider)worry about spitting spider)Else backwards assaultElse backwards assaultTop assault?Top assault?

BackgroundBackground– Jumping (venomous) spiderJumping (venomous) spider– Eats other spidersEats other spiders

E.g., E.g., Scytodes pallidaScytodes pallida, a , a spitting (poison coated silk) spitting (poison coated silk) spider that also eats other spider that also eats other spidersspiders

– 600,000 neurons600,000 neurons– Better vision than most Better vision than most

mammals (but narrow)mammals (but narrow)


Brain SizeBrain SizeNeurons Neurons (not connectivity)(not connectivity)

Humans:Humans: 100 100 billionbillion

Mouse:Mouse: 70 million70 million

Honeybee:Honeybee: 1 million1 million

Portia:Portia: ½ million½ million

Housefly:Housefly: ¼ million¼ million

In Portia, limited size will force a lengthy and complex In Portia, limited size will force a lengthy and complex (algorithmic) visual scan to convert scene into a more (algorithmic) visual scan to convert scene into a more compact (plan oriented?) representation.compact (plan oriented?) representation.There just isn’t enough neurons to rebuild a visual sceneThere just isn’t enough neurons to rebuild a visual scene


Spider EyesSpider Eyes Most spiders have 8 eyes and Most spiders have 8 eyes and poor vision. poor vision. PortiaPortia sees very sees very well – it jumps far and makes well – it jumps far and makes long circuitous routes to distant long circuitous routes to distant preyprey

Postero-Lateral Eyes (PLE)Postero-Lateral Eyes (PLE)– Wide AngleWide Angle– Motion Sensing (Far)Motion Sensing (Far)– Blurry VisionBlurry Vision

Postero Median Eyes (PME)Postero Median Eyes (PME)– Often tiny, VestigialOften tiny, Vestigial

Antero-Lateral Eyes (ALE)Antero-Lateral Eyes (ALE)– Judging DistanceJudging Distance– Blurry VisionBlurry Vision

Antero-Median Eyes (AME)Antero-Median Eyes (AME)– Narrow AngleNarrow Angle– Excellent VisionExcellent Vision– ColourColour– SwivelSwivel


Spider Eyes – Fields of ViewSpider Eyes – Fields of ViewALE overlap ALE overlap enables enables accurate accurate judgement judgement of distance of distance (binocular (binocular vision)vision)AME do not AME do not overlapoverlap– Narrow Narrow

telephoto telephoto systemssystems


AcuityAcuityPortiaPortia visual visual acuity is better acuity is better than cats and than cats and pigeonspigeons– Measured in Measured in

angular angular resolution, not resolution, not via light via light sensitivitysensitivity

But slow…But slow…

Best Insect


Spider – Field of ViewSpider – Field of ViewRetina: several thousand photoreceptorsRetina: several thousand photoreceptors

Layered for different colours (green, blue, ultraviolet)Layered for different colours (green, blue, ultraviolet)Layer 1 (green) most acute, stacked for various focus Layer 1 (green) most acute, stacked for various focus distances. Swinging tube also helpsdistances. Swinging tube also helpsFovea: several hundred photoreceptors (“1” pixel)Fovea: several hundred photoreceptors (“1” pixel)(human eye has 150 million photoreceptors; fovea less (human eye has 150 million photoreceptors; fovea less than 1% of retina but uses 50% visual cortex)than 1% of retina but uses 50% visual cortex)


Spider – Route Scanning & PlanningSpider – Route Scanning & PlanningCan see everything from the top of the Can see everything from the top of the doweldowel– 15-60 minutes to scan and plan 15-60 minutes to scan and plan

(anticipatory maze learning)(anticipatory maze learning)Circuitous route (detour)Circuitous route (detour)Simulate? its future movements or place Simulate? its future movements or place memory?memory?

– Adaptive (goal directed) scanning vs. one-Adaptive (goal directed) scanning vs. one-pixel camera (non-adaptive, entire scene)pixel camera (non-adaptive, entire scene)

Attend more to the correct routeAttend more to the correct routeVisually following tracks backwards from Visually following tracks backwards from goalgoalSpecific scanning patterns may be used for Specific scanning patterns may be used for finding/resolving specific objectsfinding/resolving specific objects

– Can find previously eaten prey fasterCan find previously eaten prey faster

ExecutionExecution– Up to several hours to execute (in wild)Up to several hours to execute (in wild)– Lure could not be seen once downLure could not be seen once down– Correct route chosen 75%Correct route chosen 75%

Even when further awayEven when further awayEven if longerEven if longerEven if both routes had lures, but one had a Even if both routes had lures, but one had a gapgap

– Wrong route abandoned more oftenWrong route abandoned more often– On par with in complexity with vertebraOn par with in complexity with vertebra– Select secondary objectives if needed?Select secondary objectives if needed?


PrinciplesPrinciplesTime can be used to accumulate smaller bits of information into larger bits Time can be used to accumulate smaller bits of information into larger bits (scanning)(scanning)– Scanning may be algorithmicScanning may be algorithmic

However, to reduce scanning time (as compared to an actual 1 pixel However, to reduce scanning time (as compared to an actual 1 pixel camera) it must be adaptive, i.e. selective attention based on goals camera) it must be adaptive, i.e. selective attention based on goals affected by memoryaffected by memoryMoreover, to reduce representation size, scanning must be done in Moreover, to reduce representation size, scanning must be done in conjunction with planningconjunction with planning– It is possible that the “plan” is the representation of perceptionIt is possible that the “plan” is the representation of perception– We “see” and attend to affordancesWe “see” and attend to affordances

Perception and decision making (goal system) are part of a dynamic Perception and decision making (goal system) are part of a dynamic system and cannot be divorcedsystem and cannot be divorced– It is only when the timescales of both are similar that this is made apparentIt is only when the timescales of both are similar that this is made apparent– Seeing the “big picture” is a more human example. Information needs to be Seeing the “big picture” is a more human example. Information needs to be

synthesized in order to make good planssynthesized in order to make good plansEven in humans, visual neglect shows that perception is goal oriented – a Even in humans, visual neglect shows that perception is goal oriented – a visuomotor systemvisuomotor system


Spider GallerySpider Gallery - - portia africana, 10mm @ Kenyaportia africana, 10mm @ Kenya


Spider GallerySpider Gallery - - epeus flavobilineatu, 7mm @ Malaysiaepeus flavobilineatu, 7mm @ Malaysia


Spider GallerySpider Gallery - - unknown species, 7mm @ Thailandunknown species, 7mm @ Thailand


Spider GallerySpider Gallery - - myrmarachne plataleoides, 8mm @ Thailandmyrmarachne plataleoides, 8mm @ Thailand

Ant or Spider?


Spider GallerySpider Gallery - - chrysilla lauta, 6mm @ Malaysiachrysilla lauta, 6mm @ Malaysia


Spider GallerySpider Gallery - - viciria praemandibularis, 10mm @ Malaysiaviciria praemandibularis, 10mm @ Malaysia


Spider GallerySpider Gallery - - thiania bhamoensis, 7mm @ Malaysiathiania bhamoensis, 7mm @ Malaysia


Spider GallerySpider Gallery - - Lysomannes salticidae @ ChinaLysomannes salticidae @ China


One Pixel CameraOne Pixel CameraCompressive Imaging/SensingCompressive Imaging/Sensing

– Rather than take mega-pixel images and then compress (which is extremely resource intensive), build up your image Rather than take mega-pixel images and then compress (which is extremely resource intensive), build up your image smartlysmartly

– Like our saccadic system?Like our saccadic system?– Compressive classificationCompressive classification

What we know (possible classifications) can influence what we are looking for, i.e., look quicklyfor distinctive salient features, look What we know (possible classifications) can influence what we are looking for, i.e., look quicklyfor distinctive salient features, look more at discriminantsmore at discriminantsScanning isScanning isGoal directed perceptionGoal directed perception

– Single pixel can be extremely specialized (sensitive, terahertz)Single pixel can be extremely specialized (sensitive, terahertz)

Initial designs are not adaptive (no memory).Initial designs are not adaptive (no memory).– Quick scans of slowly (temporal & spatial) changing areas, more detailed elsewhere Quick scans of slowly (temporal & spatial) changing areas, more detailed elsewhere


Compression vs. IntegrationCompression vs. IntegrationTop RowTop Row– 64x64 bit image with 64x64 bit image with

various wavlet various wavlet compressions compressions (throwing data out)(throwing data out)

Bottom RowBottom Row– 320x240 DMD with 320x240 DMD with

various random various random samplingssamplings

Existing system quite dumb. Previous sample (pixel) does not Existing system quite dumb. Previous sample (pixel) does not direct next one.direct next one.

Leads to cognitive/dynamic systemsLeads to cognitive/dynamic systems

Does it take longer to reduce a mega pixel image to an Does it take longer to reduce a mega pixel image to an internal representation than to build one up pixel by pixel?internal representation than to build one up pixel by pixel?


TerahertzTerahertzTravels in Travels in line of sightline of sightNon-Non-ionizing ionizing (safe unlike (safe unlike x-rays)x-rays)Penetrates Penetrates non-non-conductorsconductors


One-Pixel One-Pixel CRT CRT

Although One-Pixel Cameras (perception) is Although One-Pixel Cameras (perception) is not common (what about saccades?), what not common (what about saccades?), what about generation (imagination)about generation (imagination)


Jitter Jitter Interlacing Interlacing

Jitter & Interlacing can reduce bandwidth, butJitter & Interlacing can reduce bandwidth, but

They increase motion artifactsThey increase motion artifacts

How do Artists Draw?How do Artists Draw?Is it similar to how they eye-track?Is it similar to how they eye-track?


Documents

9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Vision, Spiders & Time - 1 Vision, Spiders & Time David Pierre Leibovitz