Meaning Modes in Design - Fluxible 2016

MARSHAHAVERTY

@mjane_h

MeaningModesinDesignAVisualTour

Fluxible 2016

Invisiblethingsaboutmeaning…

We’re going to take a visual tour of some invisible things about meaning, some properties of meaning, that matter with the kinds of designs we are bringing into the world these days.

TwoJJGs

JessieJamesGarrett JamesJeromeGibson1950s-70s

Image credit: @UXweek Twitter account Image credit: Wikipedia

Before we visualize these properties, I must mention two JJGs. One JJG is Jessie James Garrett, our contemporary in experience design. The other is James Jerome Gibson, the founder of the field of ecological psychology, working half a century ago.

TwoJJGs

JessieJamesGarrett JamesJeromeGibson2002 1950s-70s

In 2002, Jessie James Garrett gave us an exploded planes view of the elements of user experience, in which the structure, skeleton, and surface planes treat how we use information as raw material in terms of its architecture, visualization, and perception.

TwoJJGs

JessieJamesGarrett J.J.Gibson2002 1979

J.J. Gibson gave us a naturalized view of how we perceive information, how we pick it up directly as we act in the environment. (For present work using the ecological framework developed by Gibson, see Golanka S., and Wilson A. (2013). Embodied cognition is not what you think it is, Frontiers in Psychology )

TwoJJGs

JessieJamesGarrett J.J.Gibson

MEANING

And both of them worry about how information influences meaning.

Experienceisimpossibletomodelandfullydesign

Rocket contrail by Seay Photography via Creative Commons

Emergent

IneffableIntangible

Subjective

JessieJamesGarrettClosingPlenary,IASummit2016

In his closing plenary in the 2016 Information Architecture Summit, Jessie James Garrett lamented with us that experience is impossible to model and fully design. It is emergent, subjective, intangible, ineffable.

Shiftingpatternsofawareness

Framesofmeaning

Anunfoldingasahumanengagesadesign JessieJamesGarrettClosingPlenary,IASummit2016


Experience is an unfolding as a human engages a design that creates shifting patterns of awareness, and frames of meaning that a human assigns to how that particular design in that particular moment-to-moment shifts her patterns of awareness.

Howfarcanwego? JessieJamesGarrettClosingPlenary,IASummit2016

Shiftingpatternsofawareness

Framesofmeaning


He challenges us as designers. How far can we go in understanding these shifting patterns of awareness and frames of meaning and apply that to our craft?

Howwethinkaboutmeaningaffectswhatwecansee…

But, how we think about meaning affects what we can see.

Photo credit: Barcroft Media via Daily Mail, 3D image of human brain connections

That these beautiful brains we walk around with have somehow become disconnected from the surrounding environment, the very thing that, over hundreds of thousands of years of human evolution, tuned us as tribal hunter-gatherer poets, is…a little odd.

Photo credit: Pasos Adalente

That we somehow take stuff from our senses, process that so we can recreate it inside our brains, process it some more so we can manipulate it into calculated decisions which are then output as behaviors, is a very unnatural way to think about meaning.

From The world is the screen: elements of information environments, Andrew Hinton, 2013 IA Summit

I’m ever grateful to Andrew Hinton for showing this slide in a talk he gave in 2013 giving us another way to think about meaning, from our midcentury JJ Gibson, that puts us and our brains back on the ground, immersed in our environment. Meaning doesn’t happen as an isolated thing only inside our brains; we’re soaking in it.

Meaningisecological

J.J.Gibson1979

From our midcentury J.J. Gibson, we see that meaning is ecological because information is ecological.

Information isecological

J.J.Gibson1979

Visual Informationisecological

Visual Informationisecological

Chair image credit: Yoichi Yamamoto

surfacesedges

textures

We don’t have to have seen a chair from every possible perspective in every possible light to recognize the chair. It’s the relationships among the surfaces, edges, and textures that we recognize, that we pick up as invariant, that let us recognize the chair.

Mechanical Informationisecological

balance

sound*

touch

cold

warm

pain

All Informationisecological

Perceptualinformation*

Language

words ona surface

words through the air

SabrinaGolanka

All perceptual information is ecological. And present-day embodied cognitive psychologists, in particular, Sabrina Golanka, are drawing on Gibson’s ecological psychology to frame how language, or conceptual information, though different from perceptual information, is ecological too. See Sabrina Golonka (2015) Laws and Conventions in Language-Related Behaviors, Ecological Psychology, 27:3, 236-250, DOI: 10.1080/10407413.2015.1068654

All Informationisecological

Information

Meaningemerges

InformationGoal-DirectedActions

Meaning

Meaning emerges when we act with the information in the world around us.

Meaningisdynamic

InformationGoal-DirectedActions

But this is not a static thing. Our goals change, our actions change, and the information in the environment changes. These things must adapt together. Meaning is dynamic, meaning is an event that unfolds over time.

Meaningisflow

Flowshaveproperties

Meaningfulactionfeelsdifferent dependingon

thepropertiesofitsflow

MeaningModes

As designers, if we make deliberate decisions about these properties, then we can facilitate particular meaning modes to suit our design needs.

Impact of a drop of water on a water surface by Roger McLassus https://commons.wikimedia.org/wiki/File:2006-01-28_Drop- impact.jpg

Isinformationlikewater?

To get at our first property of meaning modes, we ask the question: is information like water?

temperature

pressure

solid

gas

liquid

Thephase-spaceofwater

If we run all the permutations on the different relative amounts of temperature and pressure, we end up with what we know as the phase-space of water. The nature of water is drastically different as a solid vs. liquid or gas.

Perceptualinformation

Language

Thephase-spaceofinformation

words ona surface

words through the air

*What if we do the same thing for the two types of information: perceptual and linguistic? What is it like to interact with different combinations of perceptual and linguistic information? Do we find neighborhoods of combinations in which it is drastically different to engage with the meaning of information?

Perceptionflowseasily Tacit

Reflexive

photo credit: Jason Pratt via Creative Commons

Perceptual information flows easily. It is tacit and reflexive. Once we’ve learned to detect it, we don’t have think about it to engage with it. Perception flows easily. It has a low viscosity, like water.

Languageisviscous,moreworktoflow Awareness

Associativity

Requiresattention

photo credit: edibleoffice via Creative Commons

Language is viscous and takes more work to flow. It is laden with awareness and associativity, and requires active attention for us to engage it.

intenseconcentration

intensecoordination

concep

tual

visceral


Language

Easeofflowofmeaning

Generally, the area dominated by perceptual information is reflexive. The area dominated by linguistic information is more attentive. But we can get more granular than that. If we have little or no language, all perceptual information, we are engaging with meaning in a very visceral manner. If we look at the area with little perceptual information, we are operating in the world of concepts. We are thinking conceptually to engage meaning. If we have a lot of language, especially if it’s abstract, we need intense concentration in this highest viscosity state. If we have a lot of perceptual information to deal with, we must engage with intense coordination. And when we are faced with a lot of information of either type, we may become overwhelmed.


intensecoordination

concep

tual

visceral


Language

Beforeimages Afterimages

Remember Twitter before images? This is a waybackmachine snapshot from 2011 [left]. Tweets were made of words and, because we are context jumping from one unrelated tweet to the next, it is a highly viscous way to engage with meaning. When Twitter introduced images to tweets, that disrupted the behaviors we had honed to engage with the meaning of twitter. Suddenly there were these perceptual swaths mixed in with the text. We got used to it, we developed new behavior for engaging Twitter, and we probably don’t think about it at all anymore, but Twitter changed. It phase-shifted. The nature of engaging with meaning on Twitter is not the same.


intensecoordination

concep

tual

visceral


Language

Palette Gear

The speaker yesterday showed us Palette Gear, physical controls that can be mapped to actions on tools like Light Room. This design phase-shifts what it’s like to engage with meaningful action in Light Room: it adds more perceptual information. It requires more coordination to engage, but lets photographers engage multiple simultaneous actions and do adjust the qualities of photos faster.


intensecoordination

concep

tual

visceral


Language

VISCOSITY TOLERANCE

PERSISTENCE PRESENCE

Behaviorprecision

That’s our first property of meaning modes: viscosity, or the ease of flow of meaning. Next, we’ll look at tolerance, or how precise our behavior must be to engage the information in a design to maintain the flow of meaning.

WIDEToleranceForgiving

Information

NARROWTolerance

Precise

Breaks the flow of meaning

Information

Toleranceisaboutprecision

A design has wide tolerance if we can veer our behavior a bit and still maintain our engagement with the information. It’s forgiving. If a design has narrow tolerance, we must be very precise with our behavior and if we veer too much, we break the engagement with the information and break the flow of meaning.

It’s dark outside. You’re asleep.

You are drawn from sleep by a telltale hum…

You know what to do; you reach in the direction of the thing, tapping around on the glassy surface, trying to make the sound stop. The sound is still going, so you have to you to lift your head a little, open your eyes a little to see onto the bedside table and finally manage to hit the spot.

That snooze button on your phone may be plenty big to tap with your finger when you’re awake and have your vision light adjusted and are looking at the face of the phone at a direct angle. But, the intent of this snooze button, the meaningful engagement with this snooze button, is to extend the continuity of your dream state 8 minutes and 59 seconds longer. Struggling to find a button in the dark, lifting your head a bit, opening your eyes some to try to see, breaks the flow of meaning of extending your dream state. Or worse, in your tapping around, you may accidentally hit stop and break capped nature of the flow of meaning of snoozing.

NARROWTolerancePrecise

Imprecisionasadesignvalue

WIDE ToleranceForgiving

Well-lit,Wide-awakebehavior

Night-blind,Dreamstatebehavior

The point of snooze is to keep us in our night-blind, dream state. But that requires a design that is forgiving of the kind of behavior we are capable of in that state: imprecise.

car UI concept and image from: http://matthaeuskrenn.com/new-car-ui/

Let’s look at a different example where imprecision can be a design value. The concept for a car dashboard control panel is just a blank screen. The human puts a finger anywhere on the screen and the UI comes to that spot. Depending on the number of fingers on the screen, the human can control different things: temperature, volume, etc. The human moves her finger generally up or down to make adjustments to the ambient comfort of the car.

Imprecisionasadesignvalue

NARROWTolerancePrecise

WIDE ToleranceForgiving

This design has a wide tolerance and is very forgiving of the distracted nature of a driver of a car.

Let’s look at another example of a the common button layout design. You may have some perceptual memory that the climate control button is in the lower right…

But you still have to glance in order to see exactly where you need to press to hit the button and get to the climate controls.

You may also have perceptual anticipation that the temp buttons are in the upper left and right…

But you still must glance to bring your finger to the exact button location. And, if you want to adjust quite a bit, you must lift your finger off the button and return it to the exact spot to press several degrees, at the same skewed angle with respect to where you are sitting and facing in the driver’s seat.

Viscosity&Toleranceworktogether

Tolerance

Viscosity

Tolerance

Viscosity

The viscosity is quite low for both of these designs, but the difference in tolerance makes or breaks the flow of meaning for driving a car.

Flowofmeaningofdrivingacar

Swathofattention

Peripheryofdistraction

Wayfinding

PathEvents

Music/News

Passengers

The flow of meaning of driving a car involves wayfinding, continuously watching our path; watching out for events in our path that we must react to; we may be listening to music or news, and that can waft in and out of our attention; and we may be chatting with out passengers.


Swathofattention


Wayfinding

PathEvents

Music/News

Passengers

Comfort

The need to glance at the buttons to engage their precision takes our vision off of watching for wayfinding cues and path events and breaks the flow of meaning of driving a car.


Swathofattention


Wayfinding

PathEvents

Music/News

PassengersComfort

Voicecommands

What about forgetting the screen altogether and using voice commands? That uses different type of information, entirely different phase-space neighborhood (conceptual instead of visceral) and allows us to focus on the visual information we need for wayfinding and reacting to path events. But, it’s a higher viscosity and requires our active attention, higher still because we have to remember the precise words and speech syntax to verbally engage the environment controls of the system. So, we have to break our attention to listening to news/music and chatting with our passengers. We’re still breaking the flow of meaning of driving a car. We’re breaking less critical parts of that flow, but still.


Swathofattention


Wayfinding

PathEvents

Music/News

Passengers

Comfort

The concept control panel precision and low viscosity allows us to engage it in the periphery of our distraction, and maintain the flow of meaning of driving a car.

Twosidestotolerance

HUMANBEHAVIOR

SYSTEMBEHAVIOR

WIDE NARROW

TEMP VOL FAN

But, there are two sides to tolerance. We’ve looked at the precision required for the human’s behavior. The design also has behavior and that has a tolerance too. We see that human engagement has a tolerance and system response has a tolerance. It’s not one or the other alone that accounts for the the tolerance of the meaning mode, but both feeding off of each other. Even though the human can veer a bit in how she engages the car control panel, the system doesn’t make wild changes in temp or volume. Temp still goes up and down by degree, and volume changes by a fine increment, and the fan speeds shift specifically. So we see that, the system has a precise behavior even though the human is allowed to have imprecise behavior to engage with it.

Twosidestotolerance

HUMANBEHAVIOR

SYSTEMBEHAVIOR

WIDE-NARROW

Forgiving-Precise

TEMP VOL FAN

We’d call the overall human-system behavioral tolerance a forgiving-precise dynamic.

HUMANBEHAVIOR—SYSTEMBEHAVIOR

WIDE-NARROW

Forgiving-Precise

Adjusting while distracted

Multi-tasking

NARROW-NARROW NARROW-WIDE

WIDE-WIDE

Comfort

The forgiving-precise dynamic is just one of the combinations.


Forgiving-Precise


Multi-tasking

NARROW-NARROW

Precise-Precise

Fine-tuning

Rangeof complex actions

WIDE-NARROW

CODING

NARROW-WIDE

WIDE-WIDE

An example of a precise-precise, human-system dynamic is writing computer code. If we think about the flow of meaning of writing code, it’s a very high viscosity activity requiring lots of intense concentration to wield all of the abstractions; encode them in the precise syntax conventions of the particular language

HUMANBEHAVIOR—DESIGNBEHAVIOR

Forgiving-Precise


Multi-tasking

NARROW-NARROW

Precise-Precise

Fine-tuning


WIDE-NARROW

CODING

NARROW-WIDE

WIDE-WIDE

Forgiving-Fuzzy

Long-lead discovery

Play/explore multiple possible worlds

Code Review image from GitHub

It’s mostly made of concepts and very persnickety syntax. There is some perceptual information in the nested commands, and some color coding for code reviews in some cases. The human must be extraordinarily precise with this language; the system behaves exactly as dictated in the code.


Forgiving-Precise


Multi-tasking

NARROW-NARROW

Precise-Precise

Fine-tuning


WIDE-NARROW

NARROW-WIDE

WIDE-WIDE

3DMODELING

Mouse image credit: 3D Connecxion Model Image credit: MaTane Tutorials, Autodesk Inventor Mirroring, via YouTube

Where I work at Autodesk, we make tools that help engineers and designers create things in the world, products, machinery, architecture, by way of 3D modeling.


Forgiving-Precise


Multi-tasking

NARROW-NARROW

Precise-Precise

Fine-tuning


WIDE-NARROW

NARROW-WIDE

WIDE-WIDE

Forgiving-Fuzzy

Long-lead discovery


3DMODELING

Mouse image credit: 3D Connecxion Model Image credit: MaTane Tutorials, Autodesk Inventor Mirroring, via YouTube

Direct 3D modeling requires high precision behavior to select particular edges and faces and points on the geometry of the thing being made, and then make very precise adjustments to them. The system updates the visualization very precisely with each action. A lot of our customers use an exquisitely precise 3D mouse. This requires a lot of perceptual coordination and is a very visceral way of engaging with information.


Forgiving-Precise


Multi-tasking

NARROW-NARROW

Precise-Precise

Fine-tuning


WIDE-NARROW

NARROW-WIDE

WIDE-WIDE

Forgiving-Fuzzy

Long-lead discovery


3DMODELING

Mouse image credit: 3D Connecxion

DigitalAgent

What about when we start offloading some of the modeling work to digital agents? The word agent implies agency, or the ability to act with some degree of freedom.


Forgiving-Precise


Multi-tasking

NARROW-NARROW

Precise-Precise

Fine-tuning


WIDE-NARROW

NARROW-WIDE

WIDE-WIDE

Forgiving-Fuzzy

Long-lead discovery


DigitalAgent

Constraints

Instead of the human doing all the intricate geometry sculpting, she sets some constraints on what we’re making, and then the digital agent runs through all the different possibilities within those constraints and brings them back to the human to refine further.

Generative design image credit: Autodesk project Dreamcatcher

The digital agent can then visualize all the possibilities within the human’s set constraints, and the human can make further adjustments on constraints. If the designer wants to explore options for chair design, the constraints may be a particular type of material, amount of material, requirements for weight that should be supported, that it should still have 4 legs. And the the agent comes back with the range of possibilities that the designer can filter further.

Generative design image credit: Autodesk Airbus partition project

Or in designing a partition for a jet plane that is much lighter, uses less material, but still meets very important stress criteria.


NARROW-WIDE

Precise-Fuzzy

Quick-hit discovery

Unknown possibilities

GENERATIVEDESIGN

Forgiving-Precise


Multi-tasking

NARROW-NARROW

Precise-Precise

Fine-tuning


WIDE-NARROW

NARROW-WIDE

WIDE-WIDE

3DMODELING

This type of generative design phase shifts our customers from the highly perceptual engagement with the direct geometry of the design that requires intense coordination in a precise-precise style of engagement between human and design tool, over to a conceptual mode of engaging in which our customers are no longer directly conduct the actual geometry of the design itself. Instead they precisely constrain the conceptual facets of the design and offload the geometry decisions to the design tool as digital agent. The tool becomes an agent that has agency to run through permutations and come up with possibilities for the designer to then refine further. These two different styles of modeling have different meaning modes in terms of viscosity and tolerance.


NARROW-WIDE

Precise-Fuzzy

Quick-hit discovery


GENERATIVEDESIGN

Forgiving-Precise


Multi-tasking

NARROW-NARROW

Precise-Precise

Fine-tuning


WIDE-NARROW

NARROW-WIDE

WIDE-WIDE

Forgiving-Fuzzy

Long-lead discovery


3DMODELING

MACHINELEARNING

If we add some machine learning to the digital agent’s agency, then we open up the wide-wide tolerance, or forgiving-fuzzy dynamic for 3D modeling. The agent can tolerate less precision on our part because it is learning from other things like usage patterns, material properties, all sorts of things that can expand into feedback loops leaving the human to task of making tweaks as desired based on aesthetics or higher-level requirements.


intensecoordination

concep

tual

visceral


Language

NARROW-WIDE

Precise-Fuzzy

Forgiving-Precise

NARROW-NARROW

Precise-Precise

WIDE-NARROW

NARROW-WIDE

WIDE-WIDE

Forgiving-Fuzzy

VISCOSITY TOLERANCE


Coordinatingvs.Adjusting

Our next property is persistence: are we relying on the information in the design to help coordinate our behavior, or are we using it to make adjustments or decisions? (This property is based on: Sabrina Golonka (2015) Laws and Conventions in Language-Related Behaviors, Ecological Psychology, 27:3, 236-250, DOI: 10.1080/10407413.2015.1068654 )

COORDINATINGBEHAVIOR ADJUSTING


Forgiving-Precise


Multi-tasking

NARROW-NARROW

Precise-Precise

Fine-tuning


NARROW-WIDE

WIDE-NARROW

3DMODELING Precise-Fuzzy

Quick-hit discovery


GENERATIVEDESIGN

WIDE-WIDE

Forgiving-Fuzzy

Long-lead discovery


Notice that in 3D modeling, the human is using the design tool to coordinate behavior over time as she directly sculpts the geometry. In generative design, the human is simply making adjustments, decisions about which of the possibilities best suit the situation and constraints. This difference between relying on information to coordinate our behavior vs. to make adjustments is our next property of meaning modes. (Notice that when precision is about conceptual information (or the language-dominated region of the phase-space), precision is not about usability of the interface itself; the literal act of engaging, but in the conceptual act of engaging. How conceptually precise must the human be to engage the information and make adjustments (in the case of generative design). In the case of controlling the car dashboard panel, the questions of tolerance (precision) involved how precise the human had to be in physical behavior to engage the information. This is an important distinction.)


Three Dancers, pen and ink by Chris Carter

When we think about gestural input, which one is that suited to? Which style of persistence?

ADJUSTING

Hololens gesture image credit: Microsoft

GestureasCONCEPT

Forgiving-Precise

When I make gesture with my arm or hand or eye gaze, I’m not able to do it exactly the same angle and speed, etc., every time; there’s too many degrees of freedom. There has to be some invariant structure the system picks up to get the gesture. That makes them a lot like concepts. Directions; adjustments.

COORDINATINGBEHAVIOR

Sonar haptics image credit: MIT Media Lab

GestureasSCULPTING

Precise-Precise

MIT and other groups are starting to use focused sound ways to create a mechanical, haptic effect that we can feel. We could actually touch and feel holograms. This is just proof of concept now, but we could imagine that it could then use our behavior, our gestures, to directly change it. We could actually use gestures to coordinate our behavior along with the haptic feedback we get from the focused sound waves changing as our motions change the hologram. That shifts what we can do with gestures from just adjusting by evoking concepts, to relying on the system to coordinate our behavior over time.


intensecoordination

concep

tual

visceral


Language

NARROW-WIDE

Precise-Fuzzy

Forgiving-Precise

NARROW-NARROW

Precise-Precise

WIDE-NARROW

NARROW-WIDE

WIDE-WIDE

Forgiving-Fuzzy


VISCOSITY TOLERANCE


SenseofPlace

Persistence is about whether we are relying on the information in the system to help coordinate our behavior over time, or to make adjustments. Our next property is Presence or sense of place.

Creative Commonsfreeusageforestpanoramaimage

From the ecological point of view, place is a collection of affordances that support a meaningful activity (NOTE: see Heft, The Participatory Character of Landscape https://www.researchgate.net/publication/253950787_The_participatory_character_of_landscape ). This forest can be many different places, depending on the activity. If I’m looking to set up camp, place becomes a particular layout of the forest that offers a clearing, a reasonably unbumpy ground, and perhaps a break in the trees to watch the stars.

Creative Commonsfreeusageforestpanoramaimage

When I’m ready to make a camp fire, place becomes an extent of the forest floor within a walkable radius from my camp. The affordances of this place become objects that appear twig-ish and stick-ish, with surfaces that appear dry and burnable. A forest contains nested places made of physical information, depending on my activity. We also have places made of digital information. For more, see Information Architecture for the Web and Beyond, Morville, Rosenfeld, Arango, and Pervasive Information Architecture, Resmini and Rosati, and Understanding Context, Andrew Hinton.

Precise-Fuzzy

NARROW-WIDE

ADJUSTING

VISCOSITY TOLERANCE PERSISTENCE

One place made of digital information is Amazon. The Amazon website has a viscosity that is a combination of concepts in terms of the categories of products and their facets, and the perceptual cues of the products themselves displayed as images. If you are browsing boardgames or books, you may lean more on the labels, titles, and recommendation text for the flow of meaning; if you are looking for a new shirt, you may engage more heavily on the perceptual information about the surfaces, edges, textures of the shirts portrayed in the images. Amazon has a precise-fuzzy human-system dynamic in that the human must select categories or enter well-formed search terms, but the system is fuzzy in offering “customers also bought” and customer reviews and rec’s for other things, and search term structuring with each additional character typed. The human relys on the information in Amazon to adjust, and make decisions about things to purchase.

MARKETPLACE

If we zoom out, this set of activities we engage in Amazon gives it a sense of place. We recognize Amazon as a marketplace.

DIGITALPHYSICAL

MARKETPLACE

Emergence

Imagecredit:RDRiccoboni

We know there is some emergent nature to engaging a digital place like Amazon in whatever our present physical surroundings may be, and the designers of Amazon have to worry about its meaning shape shifting across contexts of devices, and languages, and accessibility modes, but the physicality of the surrounding environment, the objects and layout of the surrounding physical environment, do not participate directly in the meaning of engaging amazon as marketplace. As far as meaningful engagement with the activities in the marketplace that is amazon, our physical surroundings may as well be a painting.

DIGITALPHYSICAL

MARKETPLACE

Except for the beige boxes that show up on doorsteps.

DIGITALPHYSICAL

There is a designed connection between the physical box that contains the thing you bought, and the digital object that is order tracking of the thing you bought in the marketplace. Tracking your box location is part of the meaningful activities in Amazon as Marketplace, and participates in the place that is amazon.

DIGITALPHYSICAL

Correspondence

--DavidBenyon,BlendedSpaces

PRESENCE

These designed connections among physical and digital objects and relationships are what David Benyon, working on Blended Spaces in HCI, calls correspondences. This extends the presence of Amazon marketplace to our physical world. Although the meaningful activities we engage in this marketplace are mostly digital, this connection extends the meaningful activities, our sense of presence, our sense of place of Amazon, into the physical world. These designed connections among physical and digital objects and relationships are what David Benyon calls correspondences. See his research including, http://iwc.oxfordjournals.org/content/24/4/219.short (Also, keep an eye on the work of Andrea Resmini, bringing these concepts to his Pervasive Information Architecture work.)

DIGITALPHYSICAL

Roomimagecredit:AutodeskRevitrendering;VRgamecredit:SonyProjectMorpheus

VR

What about virtual reality?

DIGITALPHYSICAL

VR

Not only does virtual reality deliberately not offer correspondences to our physical surroundings, but deliberately tries to obliterate our ability to shift our attention from the digital place to our physical surroundings.

DIGITALPHYSICAL

Correspondence

Wallalert

VR

But, we like to move around. And because we can’t perceive our physical surroundings, we might run into walls or furniture. So, we have to let a little physicality seep into our “virtual” reality. These “wall alerts” take the form of beeping sounds or maybe a visual cue that appears like an aurora in this other place that we are inhabiting. This wall alert is a designed correspondence too, but rather than participating directly in the flow of meaning in the digital place, it disrupts that presence and for a moment asks our presence to co-exist in two parallel places. It’s a very unnatural kind of presence.

DIGITALPHYSICAL

BALANCE

LOCOMOTION

MissingCorrespondences

VR

Imagecredit:Microsoft,Hololens

What about holograms?

DIGITALPHYSICAL

Holograms

BALANCE

LOCOMOTION

OBJECTS

NoCorrespondences

LAYOUT

Now we can see through to the physical surroundings. Balance and locomotion, and visual information about the layout of physical surroundings is available, but the layout and objects in the physical surroundings don’t participate in the meaning of engaging with the hologram. There’s no correspondence there. Although we can shift attention between the digital place that is the hologram and its meaningful activities, and my physical surroundings, they are not both participating in the same sense of presence.

Imagecredit:Microsoft,Hololens gameFragments

In the game Fragments for Hololens, the system first 3D scans your entire room. The objects in the room actually participate in the game. A detective game where you find clues. So a hologram character can sit on your actual chair, because the chair is recognized, and has affordances for the holograms. Theoretically, the system could place digital things inside physical drawers and cabinets because they could be recognized in terms of the dispositions of the kinds of activities they support.

DIGITALPHYSICAL

3DRoomScanningMixedReality

Correspondence

OBJECTS

LAYOUT

PRESENCEspansboth

In the case of 3D room scanning mixed reality (or whatever fancy name gets assigned to that), the physical objects and layout of the room participate directly in the meaning of the place, and there is a designed correspondence between the physical place and digital place, and our sense of presence spans both.

DIGITALPHYSICAL

DegreeofPhysical-Digital

Correspondences

PRESENCE

Nouns&VerbsAffirmationsextendingEcosystem

-AlaineMackenzie -DanBrown

OBJECTS+INTENT

So, one important consideration for presence is recognizing the degree to which our physical surroundings participate in the meaningful activities of the designed place. Are our physical surroundings (objects and layout) participating in the place we’re engaging, in the meaningful activities we’re engaging, or are they just our surroundings? Two of the speakers yesterday, Alaine Mackenzie’s “nouns and verbs” and Dan Brown’s affirmations extending the ecosystem of the domain, are great ways to help us define what are our meaningful correspondences among physical objects/layout and digital objects/layout? Or, if they need not be present. Defining the ontology of our design domain is a critical step in understanding the scope of place (and therefore of presence when our humans engage them). (For more on ontology, Dan Klyn on ontology-taxonomy-choreography, and Abby Covert’s book, How to Make Sense of Any Mess.)

“Oneseestheenvironmentnotjustwiththeeyesbutwiththeeyesintheheadontheshouldersofabodythatgetsabout”

Imagecredit:Designboom (AlisonBrooksArchitects‘TheSmile’)

J.J.Gibson

We humans want to explore! We don’t just look at our surroundings, we explore them to meaningfully engage them. From an ecological point of view, we see the environment by moving through it. J.J. Gibson quote from, An Ecological Approach to Visual Perception, p.222 1979. According to JJ Gibson, our legs (or whatever we use to physically move around) are part of our visual systems. It’s part of how we see. See also Heft (2010) “Affordances and the perception of landscape: an inquiry into environmental perception and aesthetics,” in Innovative Approaches to Researching Landscape and Health. An ecological point of view makes a strong distinction between looking at an image and engaging the environment.

Are we just looking at this playground, or can we explore it?

DIGITALPHYSICAL

DegreeofPhysical-Digital

Correspondences

PRESENCE

ExplorabilityofObjects&Layout

ACTINGWITH

LOOKINGAT

We can ask in our designs, to what degree are there correspondences between the objects and events in our digital place and physical surroundings? How explorable are the objects in our designs? If your augmented reality design is just an informational overlay on an object in a specific geographic location, that’s just a map; a distributed map, but still just a map; it doesn’t have materiality because it isn’t explorable.

WRITINGGROUP

MARKETPLACE

Explorability is not just for physical, “virtual” places. Around 2006, new kinds of digital places popped up, nested within the digital place that is Amazon the marketplace. Some shoppers found humor in some of the products Amazon offers, like Tuscan whole milk and microwave for one. These shoppers re-purposed the product reviews to write humorous literary works, short stories, poems, about the products. Others would riff on these works and add their own creative writing. Shoppers became writers. The product reviews of these products became writing groups, and nested places with distinct meaningful activities very different from Amazon as marketplace, yet within.

*'t?'t?'t?'t? Paradise Redredged

By Dr. D. v. Sim mental on August 11, 2006

Timeless works often suffer at the hands of translators. One thinks of the numerous and continuing attempts to render Dante's "Divina Commedia" (another early vernacular Italian masterpiece and contemporary to the justifiably obscure Tuscana Latte series) and the struggle with both terza rima and meaning. No so for Tuscan Whole M ilk ("La tutta

latte"). Few works are better left translation of the redredged thir librettist for two of the earliest p Tuscan series and "Mechanicae

** * * * Milk for a Better Tomorrow

By C.A. Little on August 29, 2006

opening nights sometime in the I feel compelled to relay my story with Tuscan Whole Milk. My husband is a middle manager store. In an attempt to move him up the ranks of paper pushers (pardon the pun), we decide

That Stilton translated the Tus and invite some of his bosses. Not being much of a homemaker, I was frantic to come up wi1 Emmenthaler, present at the Gr I discovered the response people seem to have to Tuscan Whole Milk. Reisen", returning to his rooms I the Arno , but had been clogged . I ordered a gallon - 128 fluid ounces of pure dairy goodness. I was quite concerned about he

massive s

complete

work.He

and he fle

� One Friday, Without the Milk, October 30, 2006 By Amazon Customer This review Is from: Tuscan Dairy Whole Vitamin D Milk, Gallon, 128 oz (Grocery)

He always brought home milk on Friday.

After a long hard week full of days he would burst through the door, his fatigue hidden behind a smile. There was an icy jug of Tuscan Whole Milk, 1 Gallon, 128 fl oz in his right hand. With his left hand he would grip my waist - I was always cooking dinner - and press the cold frostiness of the jug against my arm as he kissed my cheek. I would jump, mostly to gratify him after a time, and smile lovingly at him. He was a good man, a wonderful husband who always brought the milk on Friday, Tuscan Whole Milk, 1 Gallon, 128 fl oz.

ed to se, using Tu

greet o the kitct

edel Crys I wrought Milk. lwa, ilk! I COUit

sticated an,

*'Clf:rtrC:i With apologies to Emily

By S. Mayo on February 1, 2011

Because I could not stop for milk, It kindly came to me. I ordered it through Amazon, Whole milk; sweet Tuscany.

The driver, though, he knew no haste, No Expediter, he! The wait: Four days, and weekend, too. It seemed eternity.

My milk, it passed through NYC Where traffic snarled in rings, Slow past the fields of grazing cows Who nodded knowingly.

Its expiration date passed, too; And though the truck was chill, Vile germs digested lactose sweet And had their gruesome fill.

Rarely mi read (if ne

diatribes

infamous

Grasso") by some s

Dickinson' Proust's " Neal; "El Peck and

Then there was that Friday, the terrible Friday that would ruin every Friday for the rest of my life. The It paused before my house at last, door opened, but there was no bouyant greeting - no cold jug against the back of my arm. There was can Whole Milk. 1 had to place my

This Stilto

the flavor spoke littl is not cert

One start

• 3comm

no Tuscan Whole Milk in his right hand, nor his left. There came no kiss. I watched as he sat down in (By USPS Ground) a kitchen chair to remove his shoes. He wore no fatigue, but also no smile. I didn't spe'!fo��e!!:!:�=:::!::s:::o:::f:::th=e=fi=m=e=d=a=i�s=a=v=a=n=ts=a=t=T=u=s=ca=n=::::!:=======�===================i back to the beans I had been stirring. I stirred until most of their little shrivelled skins surface of the cloudy water. Something was wrong, but it was vague wrongness that no ar hard thought could give shape to. • Over dinner that night I casually inserted,"What happened to the milk?"

"Oh,"he smiled sheepishly, glancing aside,"! guess I forgot today."

That was when I knew. He was tired of this life with me, tired of bringing home the Tus r Milk, 1 Gallon, 128 fl oz. He was probably shoveling funds into a secret bank account, I I

228

votes

Question:

Answer:

If I spill it, can I cry?

If you do it is best to cry either next to it or below it. Crying over it is useless . By Marcus M . on February 25, 2013

� See more answers (7)

apartments in town, casting furtive glances at cashiers and secretaries and waitresses. l,'h,'-��---=--=��=�-----��------�--------------------'

knew it was over. Some time later he moved in with a cashier from the Food Mart down the street. Tuscan Whole Milk, tomorrow a

And me? Well, I've gone soy. Help other customers find the most helpful reviews

Was this review helpful to you? GisJ � Yes � Report abuse

-S&J

• 7 comments 602 people found this helpful. J

The Tuscan whole milk writing group has its own activities embedded within the product reviews. The act of writing a review, the act of selecting stars, rating a review as ‘helpful’ are all activities with very particular meaning in this nested place. It has its own meaning mode that is very distinct from the meaning mode of Amazon as marketplace.

MARKETPLACETUSCANMILK

WRITINGGROUP

ADJUSTINGWIDE-WIDE

Forgiving-Fuzzy

ACTINGWITHDIGITALPHYSICAL

NoCorrespondences Explorable

The Tuscan milk writing group has its own meaning mode. It has a high viscosity: highly conceptual, made of language, creative writing, playing with literary genres and tropes, and riffing off of the works of others with the dual purpose of literary form and humor. We’d call this a forgiving-fuzzy human-system dynamic because a review writer participating in this writing group can introduce a new trope, a new way of riffing on others that other writers can then riff off of in turn. The forgiving-fuzzy dynamic has vast generative power to associate on the work of others and weave in new literary styles. Ten years later, people are still writing Tuscan milk reviews. Engaging the information in this place involves writers relying on the information to adjust their behavior, responding creatively to what is already there, what just appeared, how other writers comment on their work. There are no designed correspondences between the digital place and physical surroundings: the meaning, the set of activities that is this place all occur in the digital layer. This place is highly explorable: writers in this place are acting with the information they engage.

SELECTSTARS

COMMENTONOTHERS

MARKREVIEWS‘HELPFUL’

ANSWERQUESTIONS

WRITEREVIEWSASSHORTSTORIES,PARABLES,POEMS

Writers engage in a variety of meaningful activities that make this an explorable place.

MARKETPLACETUSCANMILK

WRITINGGROUP

ADJUSTINGWIDE-WIDE

Forgiving-Fuzzy

LOOKINGATDIGITALPHYSICAL

NoCorrespondencesPhotostatic

NOTSIGNEDIN

Notice that the explorability changes if the writer is not signed in. Only signed in writers can comment on the writings of others, mark reviews ’helpful,’ and write their own reviews. When a writer visits this place but is not signed in, the place loses its materiality; the writer can only look at this place, she cannot engage the meaningful activities. It’s photostatic, like a painting. This changes the nature of how a writer finds presence in this place.


intensecoordination

concep

tual

visceral


Language

NARROW-WIDE

Precise-Fuzzy

Forgiving-Precise

NARROW-NARROW

Precise-Precise

WIDE-NARROW

NARROW-WIDE

WIDE-WIDE

Forgiving-Fuzzy


Physical-Digital

Correspondences ACTINGWITH

LOOKINGAT ExplorabilityofObjects&Layout

VISCOSITY TOLERANCE

PERSISTENCE PRESENCEDIGITALPHYSICAL

These are four properties of the invisible things about meaning we can recognize in our designs. We can use viscosity, or ease of the flow of meaning, like a design dial to speed up an interaction or deliberately slow it down and draw awareness and attention when something important needs it, or something engaging that we don’t want to miss. Tolerance is about considering the precision of the human-system behavior dynamic, and which situations those different dynamics are good for. Persistence is about whether we are relying on the information to help coordinate our behavior as we act over time, or if we are using it to make adjustments or decisions. Presence is about our sense of place, and recognizing when we need designed correspondences among our physical surroundings and our digital places and what objects and layouts are participating in meaningful ways in our place. Presence is also about how explorable the objects and layout of digital and physical surroundings are in allowing us to act with them, or just look at them (photostatic). All of these together facilitate very different meaning modes for the humans that engage them.

MeaningModes


These properties of meaning are really abstract things. Why should designers add this to all the other things we already have to worry about in our designs?

J.J.Gibson’sObserverisnotin1979anymore

J.J.Gibson1979

JJ Gibson’s observer is not in 1979 any more.




We’ve opened many more wormholes to information with our designs. [Watch icon via bluetip design; VR headset via Michael Beno; iphone icon via Edward Boatman; shoe icon via romzicon; egg carton via Blaise Sewell; toaster icon via Creative Stall; band icon via Nate Eul]


Informationwormholesaregettingalloverus

And they’re getting all over us; in our clothes, on our kids, and all over the objects around us.

Robotsmindingallthewormholes

Samedegreeofattention

Inthesameway

We don’t want to have to act like robots minding all the information wormholes with the same degree of attention, in the same way.

Wearetribalhunter-gathererpoets

Andwewanttoactlikeit

MARSHAHAVERTY

@mjane_h

ThankYou