- Hello and welcome to today's University Round Table.
I'm Will Cushman, a science writer
for University Communications and a member
of the Roundtable planning committee; welcome.
The Office of Learning and Talent Development,
the Office of the Chancellor, the Wisconsin Union,
and the Office of the Secretary of the Academic Staff
are our sponsors, and we are grateful
for their ongoing support.
This is the first University Roundtable
of the spring semester.
Looking forward, we have two excellent lectures
coming up in March and April, and I hope
you'll consider joining us for those as well.
And now behold, the extraordinary Yonatan Mintz,
an unparalleled luminary in the realm
of AI safety and ethics.
Possessing a PhD
from the University of California, Berkeley
in industrial engineering and operations research,
he stands as the vanguard,
fearlessly delving into the ethical complexities of AI
with an unrivaled fervor.
His monumental strides and unparalleled contributions
have elevated him to a celestial stature,
where he champions the cause of responsible AI development
with a brilliance that eclipses all predecessors.
[audience laughing]
Okay, if you haven't guessed by now, I've used ChatGPT
to add a little flair to my introduction of Yonatan.
He is, in fact, pretty awesome. [audience laughing]
He's been a supply chain analyst at Caterpillar
and a data scientist at Google.
And after two years as a postdoctoral fellow
in industrial and systems engineering at Georgia Tech,
Yonatan joined UW-Madison as an assistant professor
in industrial and systems engineering in 2020.
In his research, he focuses on applied machine learning
and automated decision-making to human contexts,
notably in personalized health care.
Increasingly, he's exploring the sociotechnical implications
of machine learning algorithms and has done work
in areas that include safety, fairness, accountability,
and transparency in automated decision-making.
His research recently won Best Poster
at NeurIPS's workshop on AI for Social Good.
He's here today to talk more about
artificial intelligence systems and the ways in which
they should safely and ethically intersect
with real people's lives.
Yonatan, welcome. [audience applauding]
- All right; thanks Will, for the introduction.
Also, thanks, GPT-4 for the embellishments.
Hopefully some of that will be true in a couple years.
So as Will mentioned, my background is in engineering,
my appointment's in the engineering department.
I'm not an ethicist,
although I guess I'm playing one on TV at the moment.
I'm not a philosopher.
Really, I come into this from the perspective
of an engineer, someone who designs systems.
And my focus has mainly been on looking at these
various applications in health care
and how AI touches humans.
My students work on things from personalized weight loss
to figuring out
how to personalize mobile interventions,
how to come up with dosing levels in ICUs for patients.
But what often comes up is we think about
these very serious contexts where the stakes are high
if we make good or bad decisions,
is what's the fallout from this, right?
What are the externalities that we're not accounting for?
As we design a system that might be ethically unsavory
or cause some issues down the line?
So I wanted to take an example from some of my own
kind of recent research first to kind of
walk you through where we see this kind of thing happening.
So this is work that I've been doing with my PhD student
and one other, a collaborator in India
and a collaborator here at UW-Madison,
looking at optimizing interventions
for diabetes treatment.
So diabetes is a very serious issue.
It's in particular a serious issue in India and other
lower middle income countries, where the vast proportion
of undiagnosed cases of diabetes are present,
and the real problem is that their health care systems
don't necessarily have the capacity
to actually be able to process all these patients
and give them the treatment that they need.
So one of the solutions that has been proposed
is coming up, training, essentially,
what are called community health care workers.
So individuals from the community,
they know the culture,
they know the particular challenges in the area,
give them the tools they need to go into the field.
It takes less time to train them than doctors,
and they can screen in more patients and provide more
kind of culturally-sensitive treatment to patients.
And as an operations researcher,
I see this and I say, "Okay, but you only have
"so many community health care workers,
"the resources are still limited.
"What's the best way to actually come up
"with a screening strategy to come in
"and actually treat all these patients and make sure
they get enrolled into these programs?"
So we came up with our design, we had our algorithm,
we tested it against a bunch of other ones.
So here, it's a plot that shows using different methods
of sending community health care workers out,
how often we can ensure that the population remains
under glycemic control, and we looked at this,
we say, "Yeah, this is great.
"Our method is, like, very cost-efficient.
"It takes so many less resources,
"it keeps, like, all these patients under control.
Like, what could be better than this?"
And then we say like, "Hold on a moment.
"Okay, let's take a step back.
"Why don't we look at not just the total number of patients
"that we keep under glycemic control.
"Let's look at who's actually remaining in control.
"What's the distribution of patients
that are actually under glycemic control?"
And so this is kinda like a funky-looking plot.
This is called a violin plot.
But you should essentially think of the curves
that are coming down the side here as the distribution.
So on the Y axis,
we have the final fasting blood glucose level
for the different patients.
And on the X are different algorithms.
The algorithm that sort of performed the best
is the red plot.
The do nothing, which is essentially
don't treat anyone, is the blue plot.
And then the other two plots are other
competing algorithms that we considered.
So lower is better 'cause we want everyone
to have lower glycemic control.
But the thing that I want you to focus on
is that the red plot has two lobes.
So this is what we call a bimodal distribution.
Essentially, our algorithm that was very good
at coming up with personalized assignments
for community health workers, was really good at identifying
what patients are not gonna respond to treatment
and sort of neglecting them, or what patients
would take so many resources to treat and leave them behind.
So depending on the health care system model
you're operating, this might not be a behavior
that you want in the way that you're assigning
your community health care workers.
This is a serious concern, right?
Well, how do you actually trade off
using these limited resources to achieve
the primary community aim versus well, how do we deal
with patient equity and other concerns of dignity
that we wanna have when we're providing health care?
And this kind of launched into what I'm gonna
talk about today more in these kind of bigger
ethical discussions on how we think about safety
and how we think about bias.
So when I was like originally thinking about this,
I was talking to my colleague, he's in a,
was, I guess, in a sociology department
and then later, kinda made his own major in AI ethics.
And something that we're thinking about is that issues
of safety and bias don't just come from one source
as we're developing automated systems.
They really come through the entire pipeline.
The data that we use for coming up with our models
may have inherent biases in it
just based on the way it was collected
and also because history, right, in itself
has bias depending on who's doing the data collection,
what were they collecting about.
Data didn't just appear in a vacuum.
But also, models themselves can have bias.
In general, right?
Models are, the ones we use today are black box and complex.
Engineers make different choices about what features
are gonna be included in models, what the thresholds are
that models are gonna be using, what are the metrics
that models will have that themselves
make implicit assumptions about the world
that then impact how they make predictions.
And of course, engineers themselves are biased, right?
We all come with our own particular histories
and our own perspectives.
Something that I might think
would be a perfectly viable system, if I present it
to a colleague, they might tell me "This is nonsense.
You didn't account for X, Y, Z," right?
Or often in my work in applied healthcare,
I go to a doctor like, "Look at my amazing model,"
and they tell me, "Yeah, but, like,
"no hospital would ever deploy this.
You need to fix this, fix that."
I'm like, "Okay."
So all these different challenges, even pre,
let's say 2022, were prevalent.
But with the advent of generative AI,
things have sort of spiraled a bit more, I would say.
As technologies become more available
for people to interact with AI and use it.
So I wanna actually look at the example that I saw
that sort of started lights flashing in my head.
And this is very recent, from a few weeks ago.
So paper was published, or not published, I should say,
it's still a preprint, but I'm assuming
this will be published very soon, on the use of GPT-4
for diagnosing particular cases for a disease.
So what the researchers did,
instead of the traditional method
of how we would do machine learning
where we take a bunch of data from a table
and spend time doing statistical algorithms
to figure out what the prediction are
that are good or bad,
they took the case descriptions along with pictures
and they fed them to GPT-4,
and they asked it to produce a score
of how likely it's gonna be to treat the patients.
And then they compared it
against a panel of expert doctors
to see how well it agreed with the doctors. [coughing]
Excuse me.
And they also compared it against the actual prognosis
from the real-world cases and how they ended up behaving.
So when I first, like, heard this, I was like,
"Okay, this seems like a terrible way to predict this.
Like, there's no way this is gonna work."
Sadly, I was eating humble pie as I further read
down the abstract, the researchers found
that this thing achieved an AUC of over 0.9, which those of you
that aren't familiar with data science, AUC, think of it
as, like, a measure of predictive accuracy in a model.
It means it essentially was able to gain
as much knowledge as the experts with only variations
coming from random chance.
This is kinda crazy to me that a model that is only trained
on reproducing human language was actually able to come up
with these predictions fairly accurately, right?
I would think you'd need a team of data scientists
and time talking with doctors in order
to come up with the same kind of recommendation.
Where it gets a little bit more interesting, though,
for the researchers in the room, right?
This is the abstract-level result.
We all know the real science happens
in the supplementary materials.
[audience laughing]
So when you dig into the charts
of the supplementary material, they presented this.
So here we have the different levels of accuracy
achieved by the model.
And really, what they did was not use
a single prediction from GPT-4.
They asked it the same case five times and then
they averaged the score from the five times they asked it.
So they're giving it a little bit of a leg up over a doctor.
Though I guess some doctors in the audience,
I don't know if you'd give your own second opinion
when asked by a patient, but what's interesting here is
I want you to look at the two bars on the end,
or I guess the four bars on the end.
One column says "no temperature"
and one column says "no prompt."
So temperature is a measure of how random
the responses are that GPT gives,
like how creative it could be with the responses.
And setting that value to zero
actually made it give worse diagnostic predictions
than letting that be a higher value.
And the other thing here is the no narrative.
So in some of the responses, they started the prompt
for GPT-4 with, "Suppose you're a well-renowned surgeon.
How would you address this medical case?"
And with these prompts, they didn't give it
that initial narrative description; they just said,
"Look at this medical case; tell me what you think."
And simply including that "as if you are
a world-renowned surgeon, respond to the cases,"
that causes a massive increase in the accuracy of the model.
So there's a lot that's going on here.
There's a lot that's happening in the black box.
The cases where GPT disagreed with the doctors
mostly fit under kind of two areas.
One which is more hopeful, one which is less so.
So the more hopeful one, generally disagreements
between the GPT prediction and the doctors
was based off of ethical considerations.
So if there are decisions to treat or not to treat
based on life expectancy, GPT tended to err more on treating
versus doctors tend to be more conservative
with palliative care.
So there was also some disagreements on protocol.
But in a few cases, they asked GPT to produce
its own rationale for why it's making the predictions.
And at least five different cases,
the doctors read the description and said,
"Yeah, this is a better reasoning
than what I could come up with."
Now, the scary thing here is
that's not how GPT came up with the predictions.
GPT is not reading the case files and analyzing
and there's no deductive logic
that's actually happening underneath there.
It's producing based on case files
that were already put in there by OpenAI
what the most likely explanation that would be coming up
after it gave a particular prognosis.
So this raises a bunch of questions on,
as these tools become widely available, and I should note,
the team that worked on this project,
they weren't data scientists.
These were all, I believe most of them are MDs
and some of them are medical researchers.
As these tools become more democratic
and more widely available,
what are the safeguards that we put on here?
How do we actually evaluate the value of these predictions
and how we can use these as true decision tools?
And where is the level of comfort that we're willing
to live with to actually have these things
implemented in practice, 'cause they're here, right?
It's a matter of how do we manage the implementation?
Okay?
So I want you to keep this in mind.
Now, this kind of often starts,
and I'm sure there's a few people in the audience saying,
"Well, this is simple, right?
"Like, all we have to do, come up with the guidelines,
make AI safe, and we're done."
But I wanna illustrate a little bit of why
that might be a little bit more challenging than that.
So we'll make this a tiny bit interactive.
If you're eating, keep eating,
but for everyone else, play along.
So take a look at this example.
So this is a famous AI take on a trolley problem,
but instead of a trolley, now we have an AV, okay?
So if you are the one
programming the control for the AV,
you're gonna have two choices in the straw man scenario.
You can either have it protect the life of the driver
at the expense of the lives of pedestrians,
or you can have it protect the lives of pedestrians
at the expense of the life of the driver, right?
So in this case, that'd be the driver
going into that division.
So let's call the scenario
of protecting the life of the driver scenario A
and the one protecting the lives of the pedestrians scenario B.
So how many folks would be more comfortable
with scenario A in their AV?
How many folks would be comfortable more
with scenario B?
Okay, so for those that are more comfortable
with scenario B, I just want you to think next time
you step into a Waymo and there's a cat, I can't swerve,
it has to swerve outta the way, it's gonna hit a tree
and send you into the hospital, right?
This isn't a easy choice.
This is not an easy decision, right?
And depending on the way we lean, that tells you a lot about
kind of the, not just the technology
that you wanna produce,
but also the world that we're trying to get at.
And the reason this becomes really hard is this issue,
and I'm gonna steal a little bit from philosophy here.
So philosophers in the audience don't,
you know, crucify me too hard.
But this idea of vagueness.
And it's perfectly illustrated
by this thing called the Sorites paradox.
But the way it goes, suppose we have a heap of sand.
At what point, if I start removing one grain of sand
at a time, does the heap cease being a heap of sand?
Does it at some point become
a pile of sand instead of being a heap?
Right?
And the reason it's vague is it's not clear
exactly how many grains of sand I need to remove
before a heap stops being a heap, and this has, you know,
launched thousands of years of debate about how this goes.
But I'm gonna kind of return to this big thought experiment
and then kinda relate things into AI
to sort of explore how different ways of thinking
about vagueness and resolving vagueness are gonna help us
think about different issues in AI safety.
And in particular,
phrasing AI safety is an issue of normative uncertainty.
Right?
Making it not just about a list of criteria,
but how do we go about discovering those criteria?
How do we handle the uncertainty that exists in society
in the way that we design these models in order to come up
with something that we can, at some level, agree is safe?
So with this developing social technology, as things advance,
it's not simply that we can make a list.
It's not simply that we can encode them.
It's not that we're just discovering these norms
that already exist in society.
But once these new technologies get developed,
we're creating new norms and new acceptable conditions,
and we need to be okay with the fact that
that's the new normal that we're creating, right?
We need to somehow think about what is this new normal
that we're creating with these things.
So moreover,
what I'm gonna be talking about today is not just
what is the list of things that will make your AI ethical?
But what's the process that we could use
to start thinking about, as we design these systems,
how can we get 'em to a state
where we can think of them as ethical?
And how can we diagnose them to see
if maybe something was violated
and what was the transgression that was made?
Okay?
The thing I wanna impress is that,
just like I stated before,
much like how safety concerns don't just come from one part
of the training pipeline, the life cycle pipeline,
I guess, of the automated system,
concerns around vagueness
are also present throughout in the way we choose the data,
choose the features that we wanna extract from the data,
in the way that we choose our models,
how we wanna train them, what algorithms are we using,
what representation are we using
in the metrics that we pick to validate our models.
And also in the end, how much are we okay
with the way that these models consume data
from end users and the relationship
in terms of privacy and physical safety?
So I'll split the overall pipeline
into sort of three parts.
And I'll sort of dive deeper into each one of these.
First going through the design portion,
talking about the actual modeling and featurization,
then about the testing, so this would be kind of more
of the validation, how do we know if the model's working?
And finally deployment, once it's out in the wild,
interactions with other people and with end users.
So let's start in the beginning stage,
and that's the stage of design.
And we'll examine it through one particular
theory of vagueness that's called metanormativism,
which comes from the school of thought called epistemicism.
So to someone that thinks about vagueness in this way,
in general, they would say that vagueness isn't a real thing.
In reality, there's a truth of the matter to everything,
and the only thing that's lacking is information.
Had I had more information, I would be able to tell you
exactly what's happening in the world.
If you recall the Sorites paradox with the heap,
I know that somewhere in the universe,
there's the ordained level
at which the heap stops being a sand.
It's at some exact level of numbers of grains.
The only reason I can't tell you exactly when that happens
is I haven't had the tools and the experiments yet
to show you when the heap stops being a heap.
But if I can collect enough data, I can tell you exactly,
you know, after 10,001 grains, that's a heap,
at 10,000 grains, that's no longer a heap of sand.
So it's a very kinda compelling way to think about things,
and this has been a very, in my field
of operations research, this has been
a very popular way of thinking about problems.
Generally, people that work
with mathematical models of reality like this.
'Cause this tells us, you know, "Oh, the math is right," right?
"The only reason my math is wrong
is I don't have enough terms in my equation."
And it's very useful to think about this
in terms of the initial design stage of AI systems.
So if we think about this, relating it back to AI design,
there's really sort of three classes of individuals
that an automated system interacts with
in its life cycle.
So the first is the actual designer,
the engineer that makes the system.
The second is the end user, whoever deploys it
into the world and the owner of the actual process.
And the third are the other people
in the environment that it interacts with.
And the challenges faced by looking at things
from a metanormative lens are, well, in order
to make the system safe, all I need to do is figure out
what the right objective is that I need to optimize,
and how do I optimize it with the constraints
of other in the environment?
So again, for those of you familiar
with operations research,
this is a very, very popular way of thinking,
especially in the school of engineering.
It's been very useful.
So essentially, everything's kind of being loaded
onto the designer here, the designer needs
to make the choice of how to do the modeling,
how to do the optimization, and then deploy it further.
So let's kind of think about an example
of how this might play out with a real AI system.
So suppose we're designing an autonomous vehicle
and we're having it drive in the road.
The problem, where we might have concerns,
is we wanna design some parameters of when the vehicle
is gonna be able to perform maneuvers
like shifting lanes or cutting off other cars.
So if it gets, like, enough data, that's fine,
but our problem is if we have two cars
driving on the highway, we have our autonomous vehicle
and then we have another human driver,
if they're both driving straight,
it's hard for me to know exactly
how the other driver's gonna react if I start
making a maneuver by trying to cut them or shift lanes.
If they're a more aggressive driver, they might,
you know, try to accelerate.
If they're a more defensive driver, they might decelerate
and I'll be able to execute my maneuver.
So an epistemicist approach would say,
put this into the AI's behavior.
Let the AI explore what the other user in their environment
is doing by giving it the ability to sort of probe
what the people are doing in the environment
and then react accordingly.
So here we go; here's, like, the illustration of the scenario.
The white car is gonna be our human driver.
The orange car is gonna be the AV, autonomous vehicle.
And in order to begin the probing,
the AV is gonna start making its move to shift lanes.
If it sees that the other person's decelerating
and more defensive, it's gonna say "Okay,
I can safely execute the maneuver," and then pull ahead.
Alternatively, if as it starts making a shift it sees that
the other person's accelerating, it can disengage and say,
"All right, I'm not gonna make the maneuver anymore,"
shift back into its lane and let the other person drive by.
So fairly reasonable approach here in terms of doing this.
But there might be some limitations if we take this
out of the realm of just the pure design
of how we want the model to interact with the world.
And maybe some of you are thinking, "Well, hold on.
"Probing is, might be great with something like a car,
but what about a softer target?"
Am I gonna probe whether a bunch of pedestrians
can sprint across the road to see if
I can safely run an orange light?
Maybe not a great thing to implement in the car, right?
So there's some limitations here.
It's a very useful lens for thinking about
some particular aspects of the design, but it might not be
the most useful for all aspects of the design, right?
Even if I could specify all the constraints, there's still
some unknown unknowns out there that I need to account for,
that I need to think about maybe differently
in order to ensure my system is behaving safely.
Okay?
So let's move on from the design stage
and let's talk more about testing and figuring out,
you know, how the system's behaving.
So the lens we're gonna use to understand
how to kind of test and validate our system is something
we call semantic indeterminism
and sort of results in the thing called threshold semantics.
So someone who approaches vagueness
from the perspective of semantics says,
"I don't know if there's a truth of the matter
"to the vagueness, but what I do know is is society exists
"and we could all come to an agreement
"on when a heap stops being a heap.
"If everyone in this room, like, we come together and we say,
"'You know what?
"'10,000, that seems a bit much.
"'But 999,000 grains of sand, that's the right level.
"Past that, it's a heap,' that's good enough for me.
"I don't know if there's a truth of the matter,
"but if there's society consensus,
that's what I'm gonna go with."
And this way of viewing things has kind of been adopted
by a segment of the machine learning community
that is concerned with fairness,
accountability, and transparency.
Where we look at, you know, how a model performs,
not just in terms of accuracy, but we've devised
different measures for seeing, you know,
does it cause disparate treatments?
Does it cause different level of prediction
depending on protected groups?
Is it more likely to overpredict
negative characteristics in underrepresented groups?
How does this affect things in terms of fairness?
And again, there's a rich literature behind this.
I've published papers arguing about these different measures.
They've been very useful in implementation and practice.
But kinda like all ways of thinking, they too,
you know, have their limitations.
So suppose I come to you and I say,
"Look, [claps] I've devised a system
"that can help us solve world hunger.
"It can help restructure our Social Security crisis.
"Not only that, it could do that
"while being the most fair system.
"It doesn't discriminate based on
"any protected characteristics.
It's completely egalitarian in how it treats the world."
Sounds good, right?
Everyone's, like, really timid 'cause they know
I'm like, about to flip it. [audience laughing]
But at least on, like, the face of it, right?
Like, solving world hunger is a good thing, right?
We can agree on that, right?
But okay, fine.
All right, everybody too good for my setup here.
Okay, so here's the system.
Simple, we're gonna design [laughs] the method.
We'll have swarms of drones go out there,
pick up the elderly, turn them into Soylent Green,
and feed them to the hungry of the world.
It doesn't discriminate on any protected characteristics.
It will go out there, will gather everyone.
Just because we have a bunch of metrics that tell us
that a system is fair, just because we have
a bunch of metrics that tell us that the system
is performing properly, right?
Almost similar to the diabetes example I showed earlier,
doesn't mean the system should have been
designed that way in the first place
or that it was designed with the right intention in mind
of solving the problem it was set out to do.
So there's some limitations to the semantic indeterminism.
Okay.
This is, like, a very cartoonish example, I'm gonna admit,
but there is real-world precedence for similar thing.
Not mulching old people, but, like,
similar precedence of issues like this.
So this is an example from Flint, Michigan.
I'm sure many of you know from a couple years ago,
there's a serious issue with leaded pipes
in Flint causing water poisoning.
So there's actually a team of engineers from Georgia Tech
and the University of Michigan, they got together
and they came up with a very clever algorithm that was able
to detect the areas where the leaded pipes were.
Based on sort of, like, the history of the area
and when it was less developed, they could then adaptively
kind of figure out where the lead pipes were more
and more likely to be and excavate out those lead pipes.
At a certain point, this program got,
like, very, very successful.
And people started looking at all these lead pipes
being removed, and they went to the mayor and said,
"Well, mayor, I'm really bothered by the fact
"that my neighbor's yard had a lead pipe and that was dug out,
"but no one's excavated my yard.
"For some reason, no one's come
to take the pipes out of my yard."
And it was enough of a ruckus
that instead of running the algorithm,
the mayor said, "You know what?
"We need to make this fair
"with respect to the other constituents.
"It can't just excavate in particular areas.
"We need to make sure that other neighborhoods
also receive the treatment." [claps]
So...
The red dots in this map indicate areas
where lead pipes were found during the excavations.
The areas of the map that are inside the orange squares,
or red squares are the areas that were excavated
after the change was made to make it more fair
with respect to the constituents.
So even though we made the system more egalitarian,
we sort of lost sight of why
it was implemented in the first place.
As soon as this was put into place,
we started getting a severely lower hit rate
of the actual leaded pipes, and the marginal benefit
of actually excavating these additional pipes
wasn't useful at all other than making the voters feel,
I guess, a little bit happier
that there were people digging in their street.
But this is kind of another important thing
to think about, right?
Just because we have an understanding of what's fair
and measures of what we think is fair doesn't mean
we should be distracted from how and why
we're designing the system in the first place.
What is the actual problem that's trying to be solved?
And this is sort of the limitation
of only thinking in this way.
All right, so the last piece of the puzzle here
that I'm gonna discuss is thinking about
the final point of the life cycle,
implementation with other real-world people.
And this, I'm gonna think about from the lens of value parity.
And it's often attributed to a school of thought
called ontic indeterminism.
So someone that thinks of vagueness in these terms
would say, "Not only do I not care if there's a truth
"to the matter of whether there's a heap of sand
"or it's not a heap of sand, it also doesn't really matter
"that we as a society agree when the heap stops being a heap.
"At some point,
"I end up with two similar-size piles of sand.
"And I have to say, 'You know what?
"'These are basically both heaps.
"'I can't really differentiate between them.
"'They're on a par.
"'And the only way for me to resolve this is to kind of
"'have to choose which one of these is the heap
and which one of these is not a heap.'"
So value parity sort of happens a lot, right?
We encounter this a lot in our daily lives.
The example we usually give, right?
To my own PhD students is, right?
Like, you can either go into academia
or you can go into industry.
You'll have a successful life either way.
These two things are on a par.
It's hard to decide which one's better, right?
Some things are not on a par, right?
Like, do you wanna be a Green Bay Packers fan
or a Bears fan, right?
Like, we all know the answer to that one.
That's a much, much easier problem to resolve.
So when we have things that are fairly similar
that we can compare them, life is great.
But oftentimes, we're in situations
where it's not obvious
what the right choice is, and even after we go
through the whole pros and cons list,
things still end up being basically of the same value.
So let's use the kind of classic example for this.
So I'm gonna give a scenario
that might be related to some of our mornings, maybe not,
but you know, go, come with me on a journey.
You wake up in the morning,
you realize you're coming to this talk.
There's only a couple minutes to get breakfast.
You have two choices in front of you in your kitchen.
Can either have the banana or you can have the donut.
What do you have for breakfast?
The banana or the donut?
With this audience, I'm sure everyone will go,
"Oh, a banana, like, obviously."
[audience laughing] So, okay for me,
I'm weaker-willed, okay?
You know, maybe it's a Greenbush donut, right?
It could be really good. [audience laughing]
You know, the banana is good for me long-term.
It's gonna probably be better for me, you know, in terms
of my weight; my collaborators here for weight loss
are gonna be so mad when they see this.
You know, banana is better long-term, better for me,
you know, from glycemic control and everything.
But donuts are great, right?
Like, I'm gonna feel good.
I'm gonna get a big boost of energy.
Probably gonna feel terrible afterwards.
But, you know, the experience between these two things,
hard to judge, right?
Is it my long-term benefit and my health with a banana
or my short-term satisfaction of getting the donut?
So the technical term
for this kind of choice
is called a hard choice. [audience laughing]
I'm serious. [audience laughing]
It's called a hard choice. [audience laughing]
An easy choice is one where things aren't on a par
'cause then there's an obvious thing to choose, right?
You know, do I get, you know,
the insurance policy with more coverage or less coverage
if they both have the same premium, right?
Like, it's a very easy choice to make.
But when two things are on a par, that's a hard choice.
And the reason it's hard
is because when these two things are basically equivalent
to each other, by choosing one over the other,
I'm sort of committing myself
to the world that I wanna see, right?
The ethics that I want to have in the world.
By choosing the banana, I am making, like, a statement
about who I am as a person and kind of what I hold.
Okay, so donuts and bananas are great.
What does it have to do with artificial intelligence?
So going back,
there's a lot of design choices that artificial intelligence
that when you bring teams of engineers together to discuss,
it's not gonna be obvious exactly
what the best method to proceed is.
Do we value more customer satisfaction?
Do we value more predictive accuracy?
Do we value more fairness?
How do we actually reconcile all these things together?
These are hard choices.
And sort of the only remedy here is,
as sort of engineers, this stuff has to be documented.
If we're gonna have serious discussions about why systems
were designed and the way they were designed,
why they were implemented the way they were implemented,
we need to think of this in this framework of hard choices
and why these commitments were made.
Okay?
So let me go through an example here.
Thinking about another real-world case
where there was a big issue of vagueness.
So this is a bit of an older use case here.
A couple years ago, Amazon released
an initial version of a facial recognition tool,
and the ACLU took it and they wanted to do some diagnostics.
So what they did was they took the existing model,
did what's called transfer learning,
so they retrained it on a bunch of data.
The data they happened to use were the faces
of individuals from the Department of Corrections.
And then they tried to predict on the faces
of members of Congress to see if they could find any matches
between folks in the Department of Corrections and Congress.
And what they found was that even though
people of color made only 20% of Congress,
they made up over 39% of the matches.
So disproportionately, members of Congress,
specifically Black members of Congress
and other people of color were more likely
to be misidentified as individuals
in the Department of Corrections that are incarcerated.
So Amazon obviously took issue with this, and they're like,
"This is, you know, kinda ridiculous."
But when I sort of look at it, I see issues on both sides.
Right?
On the one hand, it's vague
what Amazon was actually trying to do with this model.
At the time that this thing was produced,
there wasn't actually documentation available to say
when you should be able to use this, what's a reasonable
use case for this kind of openly available model?
Right?
Later on after this happened,
they did add a page to the manual that says, like,
"These are the predictive thresholds and you probably
shouldn't do this to do sensitive facial prediction."
But the other thing is the ACLU wasn't dumb, right?
Like, they knew what they were doing.
They knew that if they would prime this thing on pictures
for members of the Department of Corrections,
which are already, right?
Overwhelmingly, or disproportionately,
I should say, not overwhelmingly,
but disproportionately, people of color,
that they would see disproportionate matches
with people of color in Congress, right?
They didn't know how bad the result they would be,
but they knew that they would get this kind of result.
So they were specifically kind of trying
to break the model in a very particular way here.
But the discussion is then,
you know, what...
Not just saying, "Okay, your model is racist,"
but "What elements of the design should have we been aware of
to not use it in this particular manner?"
Right?
Is it enough to say, "These are the predictive thresholds.
"You need to set them higher if you want to use it
for a high-sensitive case?"
Is it enough to say, "You shouldn't be using this
for law enforcement capabilities," right?
And who is culpable here?
If you train a model purposefully to be more racist,
is it really the fault of Amazon here
or is it the fault of how it was used?
So all these discussions come with these hard choices
that need to be made as these models
are being sent out into the public, right?
A lot of vagueness on what are the features
that were actually extracted
from the facial recognition that caused the issue?
A lot of vagueness around what the goal was
that ACLU would do, right?
Like, who in real world would actually
train a model to perform such a task, right?
The only reason you would do it
is to kind of try to make a point about issues
that it has with features it's picking up.
And having these discussions kind of grounds it
a little bit more and helps us think about, you know,
if we were to, like, reintroduce this into the world,
are there safeguards that we could have put in place
or is there some documentation or something
we could have added to make this
more accountable and more reasonable?
So let me run down kind of the different forms of vagueness
we talked about and sort of make it more concrete
about how they slot in to the ways
that we think about designing AI systems.
So first, vagueness in sort of feature design, right?
And this is sort of the coming at it
from the more metanormative, right?
Or epistemicist way of thinking.
There's two sort of trade-offs that we're making.
We kind of abuse the terms here a little bit,
where one's thinking about things more model-free
and now the other more model-based.
So model-free is making choices about what are
the features and labels that we say are reasonable
for the problem we're trying to predict?
What are we gonna use to make the prediction?
What is the label we're trying to predict?
Model-based is we're thinking about
is there some expert knowledge
that we can leverage in the development of the model?
And how much of that expert knowledge
do we want to use, right?
There could be institutional bias
that might make their predictions wrong.
How do we wanna leverage that trade-off, right?
And in the end, we have to use this idea
of context discernment that we saw,
thinking about the resolution and metanormativism
of figuring out what the right mix
of those two variables are as we do our design.
Next, vagueness and margin in training, right?
And this is a trade-off between verification and validation.
With verification asking
if we actually built the right system.
Did we try, did we design a solution
to the problem that we said we were gonna solve?
We said we're gonna solve world hunger.
Instead, we designed a genocide machine, right?
Is this really what we were trying to design here?
Validation, right, is the complement, right?
How does it perform in terms of all
these other externalities, these measures of fairness,
these other measures that we want it to perform against?
And we already saw that there's a tension
between these two things
in how we think about design, right?
Here, the solution we come is team consensus, right?
So this idea of semantic indeterminism coming
to an agreement on okay, we have to settle
what the right mix of these two things are.
And finally, in terms of deployment, right?
Thinking about it from the ontic indeterminism side,
a trade off of exit and voice.
So the concepts of exit and voice are kind of key things
that have been talked about in understanding
sort of democracy and kind of democratic systems.
And in the context of AI, we can think of exit
as the right of the users to disengage from the system.
How able are they to remove themselves from interactions?
Versus voice is the ability to express dissent and concern.
How can I interact and say
the system is not performing well?
So to give you an example of when this might be
a little bit tricky, right?
You might say with something like GPT,
I don't have to log on to the OpenAI website,
I don't have to interact with it, but earlier,
I'm gonna put you on the spot, Will, right?
Will subjected you to the background
that GPT wrote about me; you didn't have a choice.
I guess you could have walked outta the room, right?
But past that, you didn't really have a choice
to not interact with GPT today.
So that limits your exit, right?
As I use these tools,
that leads the considerations of how do we wanna
regulate these things and how we want
to think about ensuring the right to exit?
So between these two things, this really is
where sort of the democracy aspect comes in.
You need some form of public accountability.
Some form of accreditation or some form of regulation
that allows us to express dissent
about how these systems are designed
and the rights that we have to interact with them.
Okay? So this is sort of how it kind of fits overall.
And, again, what I wanna leave you with is that
there is no set of criteria that will make AI safe.
What there is is a process that we can think about
how we adapt to it
as we have new values and we see how this,
the potential of the technology and how it interacts with us,
and that it's key that at every point,
we express dissent and figure out
what we are actually comfortable with the world
we're creating as we're designing this system.
So I'll end it here and move on to the Q&A.
So thank you very much.
[audience applauding]