- Welcome, everybody, to Edgewood College.
I'm very happy to introduce Dr. Ferrine Spector,
assistant professor in psychology
here at Edgewood College.
I'm Joan Schilling, chair of the psychology department,
and I'm delighted to have Dr. Spector for my colleague.
Tonight she is going to give our talk,
Multisensory Perception: The Brain's Original Hack.
Dr. Spector earned her Bachelor of Science degree
in psychology at Trinity College in Hartford, Connecticut,
and her doctorate in psychology at McMaster University
in Hamilton, Ontario, with a focus on multisensory
perception in development.
Here at Edgewood, Dr. Spector is busy exciting
students about research in neuropsychology.
Her students love her for her enthusiasm for her subject
and her passion in teaching it to them.
Please welcome Dr. Ferrine Spector.
(audience applauds)
- Golly, hi everybody.
So everybody was asking me if I was nervous today,
but I see a lot of really friendly faces
and I want to thank the honors
students of Edgewood College for being here,
and of course all of my friends and neighbors
who are showing up for support.
Thank you, Joan, for that introduction,
and Wisconsin Public Television for coming
to Edgewood College to film us.
So hi, everybody.
Joan actually asked what I meant by hack,
and I thought that was interesting.
We use that term so casually and I attempted
to describe it appropriately by saying,
"I think it's when you take something
for which there is already an infrastructure,
something that already exists,
and you exploit it for your purposes.
I think that's about right.
And what I wanna start out, I want you to start out
by thinking, and you can go ahead and draw,
of course, I will give you more clear instructions
as we go on, but you've got paper and crayons, draw away.
But I wanna start out by thinking about something.
I want you to think about the idea
that every single piece of knowledge
that you have ever had
and you will ever have
comes in through your sensory systems.
Every single one.
Think about that.
And you're like, "Oh no,
"there's stuff I knew when I was born."
That's not knowledge.
One might argue that's instinct.
What is knowledge?
Well, conveniently, definitions are easy to find.
"Facts, informations, and skills acquired
"by a person through experience ,"
emphasis mine , "or education;
"awareness or familiarity gained by experience."
Where do you get that experience?
We take our senses for granted,
but some people argue that,
and I might sometimes argue,
that they're, in fact, more important than almost anything.
And so where does this knowledge come from?
And now I'm gonna show you a picture of a baby in the womb.
This is how they show it to you
when they're showing you pictures of babies,
but actually it looks more like this.
In the beginning it is dark.
(audience laughing)
But actually, (whooshing sounds)
it's dark, but it's not quiet.
And, well, if you live in my house,
probably sounds a little bit more like (people talking)
people talking in the background.
Oh yes, of course.
And so while you're not seeing anything,
you are immersed, as a fetus, you are immersed
in a rich auditory world.
In addition, there's lots of tactile information
that happens there too.
And so a really important thing,
and something that is a theme throughout the rest
of this talk, is the idea that experience
drives brain development.
We are not passive receivers of information.
We actively move through the world,
and that active movement through the world
actively shapes how our brain develops.
So what am I showing you here?
It's a highly-schematized graph that represents
the difference between auditory development
and visual development in the beginning of life
through the first four years of life.
And what you'll notice, I'm not supposed to move
around very much, that's a little hard for me,
but what you'll notice is that our auditory abilities
are actually quite well developed when we're born,
and our visual abilities are not.
And you could probably start to guess why.
Guess what?
We're not using our eyes in the womb.
I mean, we're opening them and we're closing them
and we're getting random visual activity
and things like that, but we're not seeing anything.
There is no experience for us
to be developing that system.
And then we open our eyes, and all of a sudden,
and that's the blue line, all of a sudden
our visual sense gets really, really good
and surpasses our auditory abilities,
simply because we've seen.
Well, now we're getting the experience,
and that experience is driving our visual development.
Our auditory system is going about its business
in a sort of a standard, linear trajectory.
So what do babies know?
I mean, what is that knowledge we're born with?
Because not all things that we're born with
are necessarily instinctual.
Guess what?
Babies are born with a preference
for their mother's native language.
Babies are born with a preference
for their mother's voice over the voice
of any other woman.
Babies are born
with some kinds of auditory memories.
That's knowledge, they have learned that.
Your mother's native language was not programmed
in your genes at conception.
Okay, so experience drives brain development.
And there are two different ways that we like
to think about this.
We have experience-expectant brain development
and experience-dependent brain development.
Experience-expectant development is the property
of a system whereby,
if given certain experiences
that are typical, the system will develop optimally.
So if you think about experiences that are typical
for humans, you're talking about what?
You're talking about language,
you're talking about lights and movement
and faces, by the way.
Faces are the coolest things to babies, the coolest things.
Ditch the mobiles, right?
Ditch all the black and white things,
just make faces at them, just talk to them.
Have that, it's the most interesting thing they can imagine.
Well, that's experience-expectant development.
You don't have to do anything special
to get the visual system to develop
in an experience-expectant way.
Or the language system, just immerse the child
in an environment that is rich with language.
Experience-expectant.
Experience-dependent development, however,
is the property of the system that there are variations,
and what variations you experience,
how your developmental
outcomes may be more individual.
So while every human can expect to be immersed
in a linguistic environment, what language you learn
is experience-dependent,
how that drives your brain development.
Okay, good, great.
And so what does it look like, in the beginning of life,
in the beginning, I am talking about the beginning,
because that's the root, that's the root of everything.
So what you're looking at here is this is a graph
of the development of the number of synapses,
synapses are the connections between neurons in your brain.
Throughout we've got newborn, two months, six months,
and all the way up to 70 years.
On that vertical axis
is synaptic density.
And I want you to just see the pattern that I am seeing.
What happens is, when a baby is born,
all of a sudden he opens his eyes
and there's all this stuff,
and then there's all these connections, right?
Connections and connections, they proliferate.
And then they start to get pruned away.
We don't need them all.
But we might.
And so, in other words, well, I'll make it,
I'll make a scheme test for you.
In other words, there's baby, right?
Newborn baby, there's its brain,
there's some connections.
And then, very soon after birth, wow,
you get so many.
Golly, I mean, it's a cacophony.
There's so much activity going back and forth
that it's actually really hard to make sense
of the world.
Has anybody ever observed a newborn?
They do a lot of, right?
They do a lot of that.
Well, you can understand why.
Right?
Now, eventually, a little bit later in the first year
of life, as the baby gains experience with the world,
the unnecessary connections get pruned.
They're gone.
You don't need them anymore.
And you can start to see that not only do sensory
areas become differentiated,
but baby's behavior becomes differentiated.
And they can respond very differently.
An eight-month-old baby is very different from a newborn.
But even then, and there, a little bit later in life,
what we have done is we have...
A brain, we didn't do that, actually, excuse me.
What your brain has done is: it has said,
"But I didn't really know what it was gonna be like out there,
"so I'm just gonna throw everything at it.
"But now that I have moved through the world,
"now that I have gained experience with the kinds
"of things that I can expect to see,
"I'm gonna get rid of the connections
that I do not need anymore.
I am going to maintain the connections that I do need
because I want my brain to be an efficient machine.
And this is a really important thing
about brain development.
The process of the developing brain
is not more connections, but fewer.
That's cool, in my opinion.
And so how do we know this?
How do we know that all these connections
in infancy are functional?
Well, lucky for you, there is a whole lot
of evidence that suggests that.
For example, if you tickle a baby's wrist,
and they love that, when you tickle a baby's wrist
you actually see activation in their brain
not only where you would expect it,
and where I would see it in all of your brains
if I tickled your wrist,
but also over the auditory cortex,
which is kinda funny.
And then, if you play a sound at the same time,
do you know what happens?
Their response to that wrist tickle is greater.
It summates, it adds together.
They're working together.
But that's not gonna happen to you.
If I tickle your wrist, and it would be fun,
I mean, you can come to my office,
I'll just give you a little wrist tickle.
If I tickled your wrist and played sound
at the same time, that's not gonna amplify
the signal.
Well, you're just gonna have some activation
in your auditory cortex, and you're gonna have
some activation where that gets processed,
somatosensory cortex, it's called.
It's not the wrist-touch cortex,
that'd be kind of a cool thing.
For babies, human speech activates auditory
and visual areas of the brain.
By the time you're three years old,
it activates only auditory cortices.
What is happening is the brain
is becoming more differentiated,
the senses are becoming more differentiated.
Not only that, throughout development in general,
and what you see, I know, you like to throw up
some brain pictures and that makes everybody go,
"Woo, brain pictures," but what we're actually seeing
in blue is we're seeing the difference,
we're seeing a reduction in brain activity
over a number of years of development.
And so a process of development is the process
of diffuse brain activity, so activity over a large
amount of the cortex, and it moves
from diffuse to focal.
So it starts with lots and lots of things being activated
all the time, and then as you develop,
as you gain active experience with the world,
you start to have visual information stimulates
the visual area, auditory information stimulates
the auditory area, et cetera.
But what that means, and see, here's funny,
'cause I'm sitting here, I'm telling you
about differentiation of sensory areas,
and I'm talking about multisensory perception.
(whimpers) That's kind of a funny.
But you see, those connections,
the unnecessary ones got pruned.
But don't you want some connections to remain?
Don't you want to be able to seamlessly
integrate sensory information?
We've only gotten rid of the unnecessary ones,
not all of them.
And so as it turns out, the brain starts out multisensory.
The brain starts out profoundly multisensory.
In fact, the suggestion is that infants experience
all sensory information
in a completely undifferentiated way.
They are not distinguishing between sights
and sounds and touches, just as they're not really
distinguishing between being hungry
versus being scared versus being wet
in the beginning of their lives.
They're just like, "I am unhappy or I am fine."
By the way.
But if you think about this, the brain's default
is to start out multisensory...
And then gradually differentiate.
Now, that's different than starting out differentiating
and then having to actively integrate.
There's a difference.
And so this is really nice, because guess what?
Not everybody's experience is expected.
Not everybody's brain develops
in a experience-expectant way.
That's just a placeholder for me
to make sure I talked about these things.
For example, if you are born
without one of your senses,
what happens if you are born blind, for example?
All those rich, strong connections connecting your visual
areas to the other part of the brain,
you don't want those to go away.
Your visual cortex is not gonna get any information
from your eyes.
It's not gonna be able to process the information
from your eyes.
And guess what?
People who are born blind or who go blind
very early in their lives, the actual official term
is early blind, what we're seeing here,
what I've circled, up at the top we have blind adults,
on the bottom we have sighted adults.
What I have circled is I have circled
visual cortical activity in response
to braille reading.
Oh.
They're seeing with their hands, and that's no joke.
That is how the visual world is being represented
via touch.
Now, imagine how difficult that would be
if their brains started out unisensory.
Differentiated sensory areas, and then they had
to actively create those connections.
Doesn't it make more sense to have them to begin with?
And that's what the brain's doing.
Again, got all those connections 'cause it's like,
"I don't know, what's it gonna be like out there?
"Am I gonna be raised in a cave?
"Am I going to have hearing?
"Or are my parents going to speak five languages?
"Is my room gonna be blue?"
I mean, that probably doesn't matter.
This activation of the visual cortex,
it correlates with tactile acuity.
Have you ever felt braille?
It's really hard to read it if you are untrained.
And, of course, you could get trained,
but if you already have more of your brain dedicated
to touch, then you're gonna be better at it
to begin with.
So this activation, it's not only functional
and measurable, but it actually is correlated
to better ability,
better tactile ability in general.
Individuals who are born deaf,
what we're seeing here, we have magnitude
of activation in the auditory cortex, okay,
so in the part of your brain that is dedicated to hearing.
Now remember, a person who is born deaf
is not going to have had hearing information
feeding into that.
So in white we have deaf individuals,
in gray we have hearing individuals.
When you present them, when you present deaf individuals
with a, excuse me, if you present them
with a visual stimulus, you see activation
in their auditory cortex.
And notice, over here on the right,
when you present hearing individuals
with an auditory stimulus,
you see activation in the auditory cortex.
That is what you would expect, right?
So you're seeing the same patterns for people
whose brain development is not experience-expectant.
It is, in fact, experience-dependent.
And they are missing some fundamental experiences
and so their brains develop a little bit different.
Yeah, if we didn't start out fundamentally multisensory,
it would be very difficult to navigate through the world...
With a deficit of this variety.
But guess what?
If I took any one of you sighted individuals
and blindfolded you for five days,
anybody want to volunteer?
Blindfolded you for five days, what I would start
to see, and that's what I've circled there,
on day five I would start to see activation
in your visual cortex when asking you
to do something with your fingers,
to feel something, to do a tactile task.
And that activation, although not nearly as strong
as if you had spent your entire life visually deprived,
it does in fact correlate with increased tactile ability.
So if I blindfolded you for five days
and taught you how to read braille,
you would get better at it,
and that would be correlated with the amount
of activity I'm seeing in your visual cortex.
What does that mean about all those connections?
They're not gone.
They're not all gone.
It's just the ones you don't need.
Well, most of the time you don't need them
until you do, and then you want them to be there
so that you can continue to navigate
through the world effectively.
So what that means is experience-expectant
and dependent brain development allows the baby
to navigate in a world in which it can expect
to experience integrated input.
That's the experience-expectant part.
But it also sets the stage for specific multisensory
experience, experience-dependent multisensory experience.
We think about things as being, "Okay,
"well this is vision and this is hearing and this is touch."
That is how we all learned about the senses.
Now, some very extreme multisensory researchers
will say there are no differentiated senses.
They don't exist.
I'm not sure I'm quite that extreme,
but that's an idea that's out there.
Because guess what?
Every experience is inherently multisensory,
and the world teaches us that.
Did you know we all associate size to pitch?
Larger objects make lower-pitched noises.
We also associate size to loudness.
Larger objects are in fact louder.
We have learned that from maneuvering
through our world, active experience.
But it also helps that color indicates edibility in food.
And not only that, but when you're looking
at someone's face, you can understand
what they're saying a little bit better.
You ever noticed that?
Every experience is inherently multisensory.
I'd challenge anybody to think about a single,
isolated sensory experience that doesn't involve
any element of another one.
We'll just see that guy again, he's so cute.
Hello, hello.
And that's why you have to turn your radio down
when you're trying to see while you're driving.
There's interference.
So how does this influence our behavior
and our experience?
And that's what I get really interested in,
and it's all very, very cool.
And how does it influence our behavior?
Well, we all know what a loud tie is.
Right?
And we all know what sharp cheddar is...
Even though it doesn't actually cut us.
We're really good at using multisensory metaphors.
We don't even realize how good we are at using them.
We're like, "Yeah, loud tie? Psht!"
"Yeah, my uncle had one of those.
"It was hideous."
Well, you just used a multisensory metaphor
to describe something that you have probably thought
of as fundamentally visual.
Sharp cheddar, you could probably all explain
why that cheddar is sharp.
I mean, we are from Wisconsin, right?
And you could probably explain why that cheddar is sharp.
"Well, it's something about the..."
That one's a little harder than the loud tie,
(laughing) to be perfectly honest.
But the point is is that we really seamlessly integrate
language and sensory information,
and putting it all together without even realizing it.
And that's what got me started when I started
thinking about multisensory perception.
I was like, "I feel like multisensory perception
"is the default
"and not the, what's the word,
"exception," excuse me.
I don't have a script in front of me.
Okay, okay, fine.
So this is how it influences our experience
as we move through the world and we use language
and we think about everyday things like loud ties
and sharp cheeses.
But what about in the lab?
Okay, fair enough.
In the lab, how does multisensory
perception influence our experience?
I've got some examples.
Let's think about concurrent stimulation.
So concurrent stimulation is when you've got
stimulation of two senses at the same time.
Now, I'm gonna give you a lot of examples
from the audiovisual domain.
This would be the point where I would ask you the question
of why do you think that we have so many examples
from the audiovisual domain?
I'm not gonna ask you that question because we're filming.
But it has to do with the dominance of our senses.
Vision is, for sighted individuals, our dominant sense,
and hearing does not fall too far behind.
But moreover, if you think about how much audiovisual
integration there is right now, you are listening
to me and you are watching me talk.
In fact, if you looked away, you would be less good
at understanding my words, even though you think
about speech as something that is fundamentally auditory.
It's not.
And so the audiovisual interactions are the ones
that we're gonna see most often.
There's this really neat one called
the illusory flash effect.
Someday I'd love to find an effect and give it
a catchy little name like this, the illusory flash effect.
What happens with the illusory flash effect
is when a single flash is presented at the same time
as two short beeps.
What do you think I'm gonna say?
You perceive multiple flashes, even though
when it was presented with one beep
you perceived one flash,
when it was presented with no beeps
you perceive one flash.
When it is presented with two beeps,
you perceive multiple flashes.
Hmm.
When we play noise
and light together, concurrently,
our perception of how intense the sound is,
we actually perceive it as more intense
even though the sound is actually the same intensity
as it was when it was presented alone.
But when we present it with a light,
all of a sudden we perceive the noise
as being more intense.
Another really neat one.
You ever notice sometimes you hear something,
but you're not sure if you hear it?
It's like, "Did I hear that?"
That's usually something that's near threshold.
You may have heard it, you may not have,
sorta depends on all your other factors
and your intentional factors and all of those things.
But if you take a light
that is just around your threshold,
that means just where you are able to perceive it,
and you take that light that you may or may not
be able to perceive, and you pair it with a noise,
then all of a sudden you are going to perceive that light
far more than not.
Usually if it's near threshold it's sort of maybe 50%
of the time when you add the noise,
and all of the sudden it's 75%-90% of the time
that you are actually perceiving that light.
Although, what is really cool too, and I love this.
The illusory flash effect also counts for touches.
So when a single flash is presented along
with tactile stimulation in the form of taps,
again, if you've got a single flash and two taps
you're gonna perceive more than one flash.
And so it's not just audiovisual,
it's just that a lot of our examples are audiovisual.
But this is clearly a vision to touch example.
And there are many, many more, as I said here,
but I wanna get to what I find most interesting,
basic matching.
You see me, I am gleefully rubbing my hands together
because this is where I think it gets really fun.
You know, it's all well and good to bring people in the lab
and show them flashes and lights and tones
and things like this, but I really like
to just kinda ask people what they think,
how they match things, I like them to draw pictures.
I see lots of you drawing pictures, that's wonderful.
And so this is a (laughing)
we looked at sensory interference and sensory facilitation,
but now we're just talking about, really,
what sensory information do you pair together?
Straightforward.
And what I like about this, and why it fills me with glee,
it's because the simplicity of it.
How simple it is to just say, "Which one of these things
"do you think matches?"
Or the simplicity of saying, "Listen to this sound,
"draw what you think it looks like."
That's fun, and also I get to use crayons.
And, it's win-win, right?
So what do we learn from here?
Okay.
Which one of these balls goes pong
and which one goes ping?
You can think about it in your head,
I'm not supposed to ask you questions.
Well, one of them goes pong,
and one of them goes ping.
And I'm sorry, for those of you standing right there
you can't really see.
One of those balls is a dark gray
and one of them is white.
As it turns out, if we show people,
toddlers and adults, who are both people,
just at different times of their lifespan...
Greater than can be expected due to chance,
in fact over and well above what we would see as randomly,
toddlers and adults will say that the gray ball
goes pong, the darker ball says pong.
That's also really fun, I get to say these things
over and over again and it's all for science,
so, I mean, it's really so good.
And that light ball goes ping.
We see this in adults, we see this in toddlers.
Now, here's my question.
Does not a white cat and a black cat meow
at the same pitch?
Yeah, does not a white mouse and a brown mouse
squeak in the same frequency.
We are not learning this the same way
we are learning about our friend the Great Dane
and our little buddy the Chihuahua.
I mean, that's something you can say,
"Well, yeah, you can learn that
"through active experience with the world."
How do you learn this through active experience
in the world?
Lighter-colored objects do not routinely
make higher-pitched sounds.
We didn't learn that.
(moaning in frustration)
It's a bit of a head scratcher,
but that's why I find it really interesting.
Here's another one for you.
"Which one of these is a kiki and which one is a koko?"
(audience laughing)
Mmhmhm, you like that? (laughing)
Well, (laughing) this is an interesting phenomenon.
It's actually an incredibly robust phenomenon
that has been studied for nearly 100 years,
if you can believe it.
And what we know is that words that have O,
an O sound in them, are associated to rounded shapes.
And words that have that E sound are associated
to jagged shapes, and you can see lots of studies
that suggest this.
Now, what effect does the consonant sound on it?
Not much.
The consonant sound can facilitate it a little bit,
but it's really all about the vowels.
And actually, if we start to look at world languages,
then we can start to see patterns
whereby objects like fish
that are rounded are often labeled
with sounds that have vowels,
rounded vowels in them.
And objects that are angular, like lots of birds,
are often labeled with sounds that have that E,
E sound in them.
And it's really neat because it's the idea
that it's possible that some of these multisensory...
Some of this multisensory integration,
not only could it have possibly driven some
of the evolution of language itself,
that's kind of a big one,
but it also helps bootstrap learning.
Bootstrap learning in the individual child
as that child learns language in a place where objects
that have O's in them,
like things that are round,
or balls, okay, it's not all O's, right, rounded vowels,
where those are reliably labeling objects
that look just like they sound,
versus spikes and points
and teeny little things.
And that's really cool.
It's a good system.
So we know information is consistently linked
across the senses.
Some of that information, some of those associations
are learned, right?
That Great Dane is barking louder,
that Great Dane is barking lower
than our little, wee Chihuahua.
We've learned that, we can learn that.
But some of these associations we can't explain by learning.
And so I've coined a term, I don't know if I've coined it,
it's not in the Oxford English Dictionary yet,
but I do use it over and over again.
It's called naturally-biased, naturally-biased.
We can't say that it's innate.
In order to say that something is innate we would have
to see it in newborns, like minutes after they are born.
I mean, I don't know if anybody's ever birthed a newborn,
but you know nobody's walking in there in that moment
and saying, "Oh, can I test your baby?"
And people do that, and you know what?
Bless their hearts, but it's very hard to say something
is innate, because once, as we've said,
as we saw with that visual development,
once you gain experience with the world then all
of a sudden, everything has changed.
Experience drives brain development.
And so we say that which we can't explain by learning
is naturally-biased.
Well, I think it's biased by our neural architecture,
by the way in which our sensory systems developed together.
But it also seems to be more common in audio,
visual, tactile senses.
And if you think about that, I mean, it makes sense.
You can visually see rhythm
the same way that you can bong, bong,
bong, hear it, right?
It's just coming down to frequency.
But what about the other senses?
And that's a place where I got my hands a little dirty
against the better judgment of some
of my academic mentors.
The thing about vision and hearing
and touch is they're frankly easier to measure.
We know more about them.
Do we know about them because they're more sophisticated
systems, or do we know more about them
because it's easier to figure out how to know about them?
Smell's a tough one, it's a really tough one.
But I'm up for the challenge.
So what do we know?
Okay, so we know that there is sensory interference
and facilitation in the audio, visual,
and the tactile domains.
But there's also sensory interference
and facilitation in the smell and taste domains.
Those arrows are anomalous.
Okay, we know that adding color,
that adding color to a smell, right,
just taking an odorant that is clear
and then adding color to it, people will report
that odorant as being more pleasant,
if it's a pleasant smell, of course.
They will rate it as being more intense, right?
Remember it just like that light plus the noise,
more intense when you add multisensory information.
We know that the amount of color increases
the perceived sweetness of a solution,
even if the color doesn't make any sense.
And if you take a lemon solution and you add red to it,
people will still report it as being sweeter.
This one's one of my favorites,
if anybody has ever read a bottle of wine
or heard somebody talk about wine and say things like,
"Oh, this has got oak on the nose,"
or something like that.
If you take white wine and you add red dye to it,
then all of a sudden those wine experts
who definitely have expertise in their area
will actually start using words that they usually use
to describe red wine, even though it's white wine
with red dye to it.
So they will say things like, I don't know,
"rosehips and red floral tones."
I have no idea, I'm really not a wine connoisseur,
but the point is is that it can even fool an expert,
somebody who has spent their entire career
learning about the differences between wines.
You simply manipulate one multisensory cue
and then all of a sudden.
It doesn't make them any less expert.
I'm sorry if there's any wine experts in this room,
but we'd fool you too, and then that's the point.
(laughing)
We also know that texture can influence perceived odor.
So if you have a fabric and you add a pleasant odor,
people will actually report that fabric as being softer.
And if you add an unpleasant odor,
they will report that fabric as being harsher, scratchier.
And it was this series of things where I said,
"I don't think it's all about audition and vision.
Everybody thinks it's all about audition and vision,
but what about smell, what about smell?
Down here, it's smell's time down here, right now.
And so, with glee, of course,
I got a whole bunch of smells,
and I got a whole bunch of people,
not at the same time.
And I had them smell the smells,
and very simply, and this is what I love about my job,
is I very simply just said,
"What color do you think this smell is?
"What texture do you think this smell is?"
And sometimes I got some blank stares.
"What do you mean what texture do I think this smell is?"
"If this smell had a texture.
"Just humor me."
Sometimes I actually had to add that
into my experimental instructions.
"If this smell had a texture, what would it feel like?
"If this smell had a color, what would it be?"
And what's neat, and I'll get to this a little bit later,
but what's neat is that some people were like,
"Oh, yeah totally."
And some people were like, "I don't,
"I have no idea what you're talking about."
And so there's a spectrum of ability
to understand multisensory experience,
which is really cool.
And so what did we find?
Okay, well as you can expect,
odors that were easily identifiable, like cinnamon,
were overwhelmingly reported as being the colors
that you would expect.
Right, cinnamon, red and brown, okay.
Lemon, orange and yellow.
Anise, which is like a licorice smell, black.
Peppermint, yeah, okay.
But those were the only, by the way,
these were the only smells that were identified.
None of the rest could be identified nearly
by anybody, and we tested 80 people
in this measure.
Almond, red, white, and blue.
That's actually purple on my screen.
And dull, of a dull texture, right?
Onion was brown, vanillin, vanilla was white and brown,
and ginger was black.
Well, I didn't learn that.
I'm not putting black ginger on my kid's food.
That's the ginger I throw away, right?
And so what we're seeing, and I'm not gonna go
through these in great detail, but what we're seeing
is that, similar to the patterns that we've seen
with audiovisual associations, some of the associations
are clearly learned.
Some of them are not.
And we can't explain them all through learning.
And if we can't explain them all through learning,
then there's gotta be another explanation.
Our idea is that these learned associations,
they take over.
We might have some naturally-biased associations.
Here is our chemical, by the way,
just in case you were wondering.
Moth balls, although nobody was able to accurately
identify moth balls, everybody said the smell was white.
I know, isn't that interesting?
Maybe somewhere, somebody was like,
"Maybe somewhere they were identifying
"the smell of moth balls."
Fair enough, right?
That's the point.
Skepticism in science is the thing
that makes science go forward.
The environment, notice, environmental smells
have these kinds of earth-toney type colors.
That's pretty cool.
Okay, well, now what?
That's what was really funny about it,
is I finished this study and my supervisors were like,
"Okay, cool, now what?"
I was like, "That's a great question."
Because now what, or so what, is kind of the thing
that drives a scientific discovery.
You're like, "Oh, now what?
"Okay, I think..."
That there is a universal sensory code.
I think that all of our senses can speak the same language.
I don't know what it is.
I can think about what it is,
for those audio, visual, tactile domains.
It's a little bit harder to think about what it is
for the smell and the taste.
But how do we crack this code?
If only, if only there were a group of people
somewhere in the world who had conscious access
to multisensory experiences.
If only.
Do you guys know where I'm going with this?
There are.
There are people who have something called "synesthesia."
Synesthesia officially means "syn," together,
"aesthesis," sensation.
So together sensation, as oopposed to anesthesia,
which is without sensation.
So people who have synesthesia,
for an individual with synesthesia, external,
typical stimulation of one sense such as the sound
of a trumpet making a B flat, reliably elicits
an experience in another sense.
For example, the experience of red.
That means that people who have synesthesia
have a conscious experience of what we believe
to be multisensory perceptual processes
that are common to everybody.
We're just not aware of them.
I think I pressed a button.
And so I use synesthesia as a tool
for understanding typical perception
because all evidence suggests
that what is happening in their brains,
in their perceptual systems, is not actually different
than what's happening in our brains,
in our perceptual systems, either at a different stage
in development or at an earlier stage of processing.
And why do I know that?
Well, there are similar connections
in synesthetes and typical infants.
Right, remember all those connections?
They may not all get pruned.
We know that.
And there is evidence that individuals with synesthesia
have a little more connectivity between their brain areas.
Some people have even gone out and said
that infants, or neonates, newborns,
are actually synesthetic because they experience
all sensory information in an undifferentiated way.
Now, an individual with synesthesia will say,
"That's not the way it is, it's always going
"to be between two senses, very specific."
An individual with synesthesia does not experience
chaotic sensory information like a baby does.
They're not going all the time like babies are,
okay, so just to be clear.
It's typically specifically between a few
sensory areas, but there is more connectivity.
Maybe those connections, those connections
just didn't pruned in these individuals.
Not only that, when an individual with synesthesia
who, for example, sees colored letters,
so letters have color.
When they're viewing a letter that is printed in black
you actually see activation of their visual cortex
which you would expect, but over areas
of the visual cortex that deal with color,
which you do not see in individuals
who do not perceive letters as having color.
But remember the pong, remember that?
Remember how we associate large objects to loud sounds
and teeny little objects to pointy things?
Well, we see the similar associations in synesthetes.
A person with synesthesia who has a visual experience
to a sound, that visual experience is consistent
with what other people are reporting.
Higher pitches, somebody who has synesthesia,
an audiovisual synesthete, when they hear higher pitches
it reliably induces the experience of a lighter color.
A smaller object.
Lower pitches reliably induce the experience
of a darker color, a larger object.
We have every reason to believe that synesthesia
and typical perception
use common mechanisms.
They're not different.
They are at one end of a spectrum
and the rest of us are somewhere in the middle.
Some of us are very, very much at the bottom.
Those are the ones that go,
"Yeah, I really don't understand what you mean
"when you're asking me to associate a color to a smell."
You all have your crayons?
Okay.
So what we like to do
is we like to,
what I really want to do, in order to crack
this multisensory code,
and that is my life's work.
I'm gonna crack that multisensory code,
just like a coconut, just crack it right open.
It's not gonna be that easy.
Actually, cracking open a coconut is quite difficult.
But we wanna crack the multisensory code,
the universal sensory code, then we need to ask
synesthetes and non-synesthetes
what their experiences are.
And one of the big projects that we're doing here
right now with my students here at Edgewood College
and with collaborators in other places,
is we are collecting what I call audiovisualizations.
We're playing sounds, we're having people draw pictures.
And so, ready?
I'm gonna play you a sound.
(high-pitched twinkling music)
I'm not collecting your data, don't worry.
Here it was.
Now, of course, in the actual experiment
you can listen through it headphones
and you can play it again, and draw your experience.
I'm wondering, did anybody's picture look like those?
These are visualizations drawn by non-synesthetes
in response to this sound.
These are visualizations drawn by synesthetes
in response to this sound.
Synesthetes for whom this is their actual,
visual experience.
I'm not asking them to imagine what the sound
looks like, but in fact they are drawing what they see.
They look kinda similar.
Ready for the next one?
Ready with your crayons?
(low-pitched suspenseful orchestral music)
I find that one very pleasing.
I'll do it one more time.
(low-pitched suspenseful orchestral music)
Huh.
Did it look something like that, maybe?
These are the drawings for non-synesthetes.
And these are the drawings for synesthetes.
It's a little light, but it's yellow.
You see some of the same elements,
the jaggedness, the movement.
(high-pitched violin music)
Oh, you're not supposed to see this already.
Oh well.
And you can see that when the sound is represented
in a more sweeping movement,
that that is actually represented through the drawings
of non-synesthetes.
I like this one in particular.
That one's very attractive.
A lot of synesthetes happen to be artists, unrelated.
All right, and the last one, which you'll notice
differs very specifically from the previous one.
(high-pitched staccato music)
The difference is the style of the musical note.
And so what do we see differently?
That's the goal here, is how are these audiovisualizations
represented differently across these different
characteristics of the sound?
Well, the drawings look very staccato,
as do the drawings of non-synesthetes.
And so what we're doing right now
through some tireless efforts of my students here,
is we are actually combing through all of this data,
non-synesthetes and synesthete data,
and we are looking across individuals
within a certain musical clip, and we are trying
to figure out, what are the characteristics of the sound
that reliably induce the visualization?
Is it the pitch or the timbre of the sound?
Is it the rhythm, is it the tone?
One thing we are seeing over and over again
is something called form constants.
And these form constants were proposed a long
time ago when somebody was studying hallucinations.
And that individual said, "For some reason,
"in all simple hallucinations," we often think
about hallucinations as being complex,
like horses running through a field,
that's not usually the way it is.
Most hallucinations are actually quite simple.
They involve very basic sensory input.
And most of the time when you're having
a simple hallucination you are having
a simple hallucination that involves
what are called form constants.
Some variation of this very basic set of shapes.
Did you see those in the drawings I showed you?
Because they were there too.
There's something real special about them.
And just in case you were wondering,
when we show people two options,
a visualization that was drawn in response to the music,
and a visualization that was drawn
in response to different music,
they are able to select...
Reliably which image fits with which picture.
Which image, excuse me, fits with which song.
But why?
(laughing)
Isn't that the question?
But why?
The world is mapped out on your brain.
The world is mapped out on your brain everywhere.
The visual world is mapped out on your retina
and it's mapped out in your visual cortex.
The auditory world is mapped out in your cochlea,
it's mapped out on your auditory cortex.
Why shouldn't all elements of the world
be mapped out on your brain?
We know that there are specific parts of the visual cortex
that specifically process basic shapes
or angular versus curved objects.
There are similar components in auditory neurons.
This is our universal sensory code.
It makes me very excited.
So, we have processes that allow us
to seamlessly integrate information
that we can expect to encounter
so that we can move through the world
without being like, "Wait, there's a loud sound
"and there's a large object.
"What am I supposed to do here?"
We've seamlessly integrated the information
so that we can seamlessly react to it,
just like everything else that our brain does.
As a consequence, we have consistent associations
that we can't explain by learning.
Well, that's what I like to have.
How can I measure those?
Because it's very easy to be like,
"Oh, well you just learned that."
Well, but what about the ones that I didn't learn?
What does that mean?
The ones that I didn't learn, how are elements
reliably associated across sensory areas?
That's the universal sensory code.
And finally, I believe that we all exist
on a multisensory spectrum.
And on one end, what you see here
is that is a visualization that was drawn
in response to one of those sounds.
It's the drawing of a piano.
That is a non-abstract image, totally appropriate.
We never said that the images had to be abstract.
Most people choose abstract images,
most people draw abstract images,
some people drew non-abstract images.
I think that those are the people
that live on this side of the spectrum.
Nothing wrong with that.
Those are the ones that go, "What do you mean,
"what color is this smell?"
They look at me like I've got lobsters
crawling out of my ears.
It's kind of funny.
And then somewhere in the middle is most of us,
who say, "Oh yeah, I drew something
"that looked kind of like that."
And then all the way up here are people
who have synesthesia, who actually have conscious
and reliable and involuntary
multisensory percepts.
And that is how we're gonna start
to crack our universal sensory code.
But of course, I couldn't even dream
of cracking our universal sensory code
without extraordinary help from my wonderful students
here at Edgewood College, and all of my collaborators.
To them, and to you, I say thank you.
I'm done.
(audience applauds)