I think we're ready to get going again, we're going to have a slight deviation in the
agenda because the clickers are acting up a little bit.
So we have a clicker session next on
but we're going to jump to the next thing on the agenda
which is a talk on Weight of evidence a Dutch experience by Charles Berger,
so thanks Charles.
Thank you, and I'd like to thank the organization for inviting me.
So, in this talk I'll be talking about very basic stuff, very simple stuff,
and I do realize for most of you here it's way too simple.
My hope is that there's enough people online for whom it could be useful
and what I'm showing you is actually the material that I would usually use
to inform legal practitioners.
I'm not very good with disclaimers, I got a bit scared by seeing a lot of other disclaimers.
But this is where I work most of the time,
I'm principal scientist at the Netherlands Forensic Institute,
which is a big forensic institute, part of the Dutch Ministry of Security and Justice.
And these are our core tasks, our main job is to do examinations in case work.
15% of our time and money we can devote to research and development that's not related to cases,
so for publications,
and 15% of our work
relates to being a center of knowledge and expertise
and so there's a lot of teaching involved there as well.
And I get to spend at least one day a week being a
professor by special appointment in criminalistics at the University of Leiden.
So, the conclusions at the Netherlands Forensic Institute
are based on the likelihood ratio or Bayes Factor if you want to be part of the incrowd,
and it has been the case since 2010 at the NFI when the direction decided
to implement logically correct concluding as we called it then,
in all areas of expertise. Of course in a number of areas
we already did so, and DNA of course would be a prime example of that.
In 2015 the European Network of Forensic Science Institutes' guideline
on evaluative forensic reporting also suggested the likelihood ratio
to express expert opinion, and a few more things as well,
and it may be important to note that the ENFSI board pledged to implement this
in all ENFSI member labs, and there's about 67 of them.
So, in my opinion use of weight of evidence follows from a division of roles.
I think at the core is just this basic idea that
there's different tasks involved for different people.
And here it would be the expert that assigns a value to the evidence
within his area of expertise only,
so that's very much limited, it's a specialist
with very specific expertise,
and he's trying to help the court with all the information.
So, then there's the trier of fact, the judge or jury,
that combines all the evidence and information in the case, including that bit from the expert
and decides on the issues at stake.
Now my bit is about criminalistics, and I see criminalistics as
reasoning backwards, that would be my very short definition of it.
With that I mean that we observe traces in the present,
we can see it with our own eyes or some equipment, and we are sure of that.
But about what caused the traces we can only hypothesize
and we try to reason backwards from those consequences
back to the cause: who left a trace,
through which activity was the trace left, questions like that.
Now reasoning backwards will be important at every stage of the legal process,
also during the investigation,
and so it's important also for the various players in the justice chain:
police, legal practitioners, and the forensic scientists.
So basically we are trying to go back from result to cause.
And there's a few principles we need when we try to reason backwards.
I've listed three of them:
First of all there are several hypotheses in competition,
we've talked about that before this morning.
When there's really only hypothesis you probably don't need to deal with uncertainty
and at least there is no way you can expect to get any other answer than the answer you imposed.
So to reasonably deal with uncertainty you would at least consider one alternative.
Also, the very definition of evidential value
counts on you having more than one proposition to explain the evidence you found.
Another principle is that experts do not have all the information in the case,
they just have the information that they need
to do their part, which is to assign a weight of the evidence
to what they observed.
So they don't have all the evidence and information that would impact the probability
of a certain proposition being true.
And also of course they have to remain within their area of expertise.
A third principle I hope is very obvious: you want to reason rationally and optimally.
So, a few considerations concerning hypotheses or propositions:
We use more than one we already said, we use them in pairs.
They have to be mutually exclusive,
there's not much use for your interpretation if they overlap,
and they do not necessarily have to be exhaustive, although actually that's usually the case if
you have a prosecution that says 'this happened and nothing else',
and a defense that says 'no, this happened and nothing else'
then you will be exhaustive anyway.
Hypotheses of course also must be relevant for the case
for your analysis to have any importance,
and hypotheses must be specific.
An earlier mention was to have the complement of the prosecution's proposition
which would then be a negation of some situation
but that would not always define a situation
in such a way that you can think about the probability of the evidence
given that negatively defined proposition.
So they have to be specific.
Well what do we do with evidence, we generally update our odds,
and I picked this example from a world championship football,
and I call it football because you can only use your feet in this variety.
World Championship 2014, the Netherlands played against
the ruling world champion at that time which was Spain,
and before the tournament the predictions were that the chances were 1 to 26
that the Netherlands would become world champion that time,
which would be new unfortunately because it never happened before.
But the Netherlands started the tournament by beating the Spanish by 5 to 1.
Which is quite impressive...
and impressive also means that it should have some effect
on your odds that the Netherlands would finally become worldchampion.
And so for the bookmakers it did. Their odds went up from 1 to 26,
to 1 to about 11.
So they more than doubled those odds.
And again it seems to make sense if you do such an observation
to change your odds, and in fact this is directly linked
to what the Federal Rules of Evidence here in the United States
refer to when they talk about relevance:
'evidence is relevant if it has any tendency
to make a fact more or less probable than it would be without the evidence'.
So the evidence would change the odds of the proposition
relative to the other proposition.
And what I did here by the way, is something I will do throughout the talk,
I indicated words that are linked to the observation to the evidence in blue,
and words that are related to the propositions in green.
So Thomas Bayes, a tremendous friend of mine, came up with his theorem,
and a theorem of course means it's not a theory, it's not just something he
invented in a sense, it's something he proved
from the basic laws of, in this case, probability theory.
And it basically shows you what we just did with those odds
concerning that football match and championship.
It shows that prior odds are updated by a multiplication
with an evidential value, to end up with posterior odds.
So that evidential value is really the update factor to multiply your prior odds with.
Okay, next question is of course how do you get to that evidential value
that we have been talking about today
and it's the probability of the evidence given those propositions
forming the likelihood ratio by comparing the probability of the observation
under the hypothesis of the prosecution
to the probability of the observation if the alternative proposition is true.
So that's our famous likelihood ratio, and that's what you can use in practice.
And speaking of practice I'd like to start with a non-forensic example.
Alright...
Dutch people, always polite and enthusiastic of course.
Now to analyze this in a forensic way,
we need first of all those propositions.
And I could make you guess, but you probably would guess that I'm going to consider the propositions
about whether she was pregnant or not.
And I'd have to pick an observation, something that we could use as evidence
I would pick the belly in this case,
so here we have, again color-coded, the hypotheses in the case
and the observation that we have, or the one I singled out at least.
So the evidential value would be the probability of observing that potbelly if she's pregnant,
divided by the probability of that potbelly if she is not pregnant.
Alright, and to see whether that makes sense intuitively,
we could think of a situation where maybe women were
so lucky that they would only get a big belly in the last week of the pregnancy.
If that were the case the probability in our numerator would go down,
and the evidential value would go down as well.
Seems to be okay.
And for the denominator, if we would be somewhere
where everybody had a big belly, regardless of whether pregnant or not,
the probability for the denominator would go up,
and again, that would mean our evidential value would go down.
So at least in that way it seems to make sense.
Alright, moving on to a forensic example and pretty much the simplest I could think of,
the source question for a full beautiful quality DNA profile,
we have hypotheses concerning the donor of the trace,
and we have as an observation
that the peaks in the DNA profile are in the same positions in both profiles,
and it fits more easily on the slide if you just call it a match,
even if I don't like that word at all.
Alright, so the evidential value would be given by the probability of such a match
if the suspect is the donor,
and in this case that probablity would be very high.
If you would profile the donor and the trace is also from the donor
you would certainly expect to find the same profile
in terms of where the peaks are, same genotype,
and so that would be at least very close to one,
and the evidential value is determined almost exclusively for this simple case
by the denominator, by the probability of finding a match
if in fact somebody else was the donor of the trace,
and we talk about the random match probability in that case.
So, the important principle was a division of roles
and the expert does not have all information.
He can comment on this evidential value, on the weight of evidence,
but not on the probability of the propositions.
In the early stages, during the investigation the police has all the information that is available,
then sends out the evidential material to for example, our forensic institute,
and it's then given to the various experts who all have their very specific area of expertise.
And they have to come up with information that can be combined again
into something that the courts can use to make a decision.
In the court again all the information comes together,
and so it needs to be possible to integrate all that information,
come up with posterior odds, and then add another ingredient:
the expected utilities of the various possible decisions.
My point here is just that the expert has to limit himself
to the evidential value to the weight of evidence.
If the expert would actually report on the probability of the hypotheses,
he would probably combine his part of the evidence
with prior odds somehow, so with other evidence and information,
or you would have to make unfounded assumptions about the other evidence and information,
and if various experts would do the same,
they would all come up with an opinion on the probability of the hypotheses,
and it would only be possible to have conflicting opinions, I would say.
Another problem with that would be that if that happens, the trier of fact
is insufficiently aware that there is bits of information going into their conclusions,
evidence might very well be counted twice,
so the DNA expert looks at the fingerprint expert,
and the fingerprint expert looks at the DNA expert,
the judge combines both, he might be counting things twice or even more than that.
If the expert would decide on the hypothesis, which is basically what PCAST suggests,
there is all these previous problems and then some more.
In this case the expert has assumed yet another role of the trier of fact, and that's to
assign, what I mentioned before, the expected utilities
of the possible decisions, so that relates to how high do you put the threshold,
when are you sufficiently convinced it's him, to say that it's actually him.
It might depend on many things, also other than evidence.
And if you turn such decisions into likelihood ratios, if you say well...
you know we need likelihood ratios but we want the experts to give categorical decisions
and then we look at their error rates to turn it into a likelihood ratio,
then likelihood ratios can only take on two values.
I think a lot of information would get lost if you
look at the variety and strengths of evidence that you can find in real cases,
and then say well we have only one possible value for any evidence that is incriminating,
it would not be right to do that, in my opinion.
So, for a lot of people it's difficult to see that
it would be rather strange for an expert to comment, or impossible even, to comment
on the probability of the proposition, and intuitively
it seems to be a bit tricky. So let's imagine somebody who enters a bank,
and decides to rob it, and he writes a note on the spot inside the bank
and proceeds to rob that bank and gets away with a good amount of money.
A few days later police have set their eyes on a suspect
and they obtain, among other things, some handwriting material of that suspect.
And there's a handwriting expert that gets the bank robbery note
and the reference material from the suspect.
So why can't the expert, the handwriting expert,
look at both notes and come up with the probability that
both notes were written by the same person?
Based on his examination it would seem that he has all the handwriting that he needs to look at
and still that wouldn't be enough to comment on that probability?
Well no it wouldn't, because the probability that the suspect wrote that note
does depend on things that have nothing to do with handwriting at all.
The probility also depends on whether there were witnesses that saw this guy,
or maybe cameras even, whether the suspect has an alibi or a strong motive,
whether on the note there were finger marks that where detected, or his DNA.
So those are all factors that are clearly not within the area of expertise
of the handwriting expert but they do impact
the probability of these two notes having been written by the same person.
It was the British Appeal Court that said something
very nice on this, they said: 'The scientist should not be asked his opinion
of the likelihood that it was the defendant who left the crime stain',
and I've turned these words green because clearly they relate to a proposition.
The proposition that it was the defendant who left the crime stain,
and it's the court here that says the forensic scientist should not be asked his
opinion on the likelihood of that proposition being true,
'nor when giving evidence should he use terminology which may lead the jury to believe
that he is expressing such an opinion'.
Not everything that the courts have said in the past are things that we like very much,
but this surely was a good statement.
So, what is the contribution of the expert,
if it's not talking about the probability of the propositions?
Christophe refered to this once as the line and it immediately made me think of La Linea,
so there you have it La Linea, and on the one side we have the prosecution,
this is a bit of Powerpoint art, I'm very proud of it.
And the prosecution of course has their story, you might be very convinced
if so you might move in their direction.
Or if the story of the defense is more appealing and
convincing to you, you would move in that direction.
So you put yourself here in the place of the judge or jury, not the forensic scientist.
Alright, so if you're hearing a certain case,
the case that the prosecution has made so far might be quite strong in your opinion,
you would lean towards their story. And now say that writing expert comes in
and gives you a likelihood ratio that points in the direction of the defense's case,
the defense proposition, and does so with a certain strength.
So that would mean that you would move back a bit from their side.
Of course it could also be that before hearing the evidence about the handwriting
you were quite convinced of the story of the defense.
If then the same evidence comes in from the handwriting expert
it would not tell you where to be on the line but how much to shift.
It would be in the same direction, it would be over the same distance
because the evidence hasn't changed in both situations.
So the evidence does not tell you where you are on that line,
or where you should go, just how much you should shift
relative to where you were before.
It's just the change of the prior odds, not the odds themselves.
So, then what is the expert's contribution?
Well generally they cannot give certainty about an hypothesis,
and I say generally because in some cases you might be able to exclude something,
but in a positive sense I don't see much of a possibility there,
but more shocking to people in general is that the experts contribution
is not even to give the probability of the hypotheses.
And so it's quite a limitation it seems
that they can only give the increase in the odds of the hypotheses,
the evidential value, or a weight of the evidence of their observation,
the likelihood ratio or the diagnostic value.
At the NFI we started this in 2010 and again,
of course in many areas of expertise we were already doing that,
it didn't make any sense to do different things in different areas of expertise,
so since 2010 we've been doing the same thing everywhere.
And in the beginning we allowed people to use verbal terms, we still do that,
but it was a few years later that we defined those verbal terms in terms of numbers.
And we were actually surprised we weren't forced to do so earlier, but we weren't, and so
for us it was logical that if you decide to use those verbal terms
you have to be very clear about what they mean.
In the sense of how it works, but also in the literal sense:
'how much more is more probable?'. So if you look at this statement
it talks about the probability of the evidence given both propositions:
how much more probable are the examination results
if hypothesis 1 is true then if hypothesis 2 is true.
You see six categories here, of course
if you just report with numbers you don't need any categories, or you can make as many as you like,
if you use words you're going to be limited to a fairly small number.
Sometimes you have to warn people and tell them how this should be seen, it's not the way to go
from words to numbers, it's not a magical way to get numbers, at all,
and we want to be very clear about that. It's just that if you use words
you should be clear about what you intend with those words.
And of course, if you have a whole institute filled with experts, you want to be sure
that when different experts use same word, they mean the same thing.
And then also, it doesn't matter much which words you actually use,
for me you could call it yellow evidence and green evidence,
as long as you define it in terms of numbers it's finally the numbers that tell you what is meant.
The last one I refered to this morning already,
when I said the whole reason about having a verbal scale,
apart from maybe making the implementation more gradual,
is that we sometimes feel it could be good to indicate
that our numbers do not come from calculations
but from subjective assessment.
Now, I realize it for this audience everything I've said so far is very simple and obvious,
but when I give this talk or something similar to it for legal practitioners,
it causes them a bit of stress and they get nervous and they think it's very complicated,
and it first it certainly is.
And so a question that comes up sometimes is:
well if it's so complicated, do we need it to be logically correct?
Do we need it to adhere to those principles of logic?
And I would say yes.
And of course for legal practitionersit helps if you give them a legal argument,
so I usually do, and I tell them that well if the forensic expert
knows that a conclusion that he would write down and sign for is not correct...
you can't do that.
And of course at the same time, if he writes down a conclusion
that he feels is scientifically correct and can stand all the logical tests
but is not understood by anyone who makes the final decision,
that's of course not a very good situation either,
but it's at least a start and something you can work on.
You've started by doing your own part of the work well,
and then you put in some more energy to make sure the other side,
the next link in the chain of justice can also do their work.
So I would say that a logically incorrect conclusion which is well understood
well understood and well accepted by clients, so to speak,
is still no alternative to a logically correct conclusion which needs an explanation,
and indeed a lot of explaining is required.
So, the Netherlands being a small country, I could do presentations every few weeks
and it's also organized very nicely so that I could have,
in one classroom, 30 prosecutors and judges most of the time,
sometimes a group of defense lawyers, and you do that for a couple of years,
and you've pretty much seen them all,
since there is maybe 600 criminal law judges in the Netherlands, for example.
So that helps, it's still a lot of work.
I wrote about five basic publications in Dutch aimed at legal practitioners.
Very often the legal practitioners are very much linked to their own language,
and so you cannot expect them to read the international literature.
But apart from sending out information, all this one way communication,
it's also important to have two way communication with everybody involved.
This morning the topic of terminology came up
and sometimes it's very easily solved:
if some word already has a reserved meaning in the legal community, just use another word.
So, that can really help. I already mentioned the guideline
for evaluative reporting in forensic science, of course that can also help,
as a way of communicating how things work.
And also the second edition of 'Interpreting Evidence' was already mentioned this morning.
So, next question could be: well when you did this in 2010,
and it was not a small step I can tell you that the direction took the decision,
and honestly they took the decision three times, they were sort of dragging their feet.
They were quite scared that their customers, so to speak,
the requesters of our services, would not understand and give up on our reports.
But the short answer is: no, it was not the solution to all of our problems,
it did also not cause horrible new problems.
There was quite some attention for
the fallacy of the transposed conditional, the prosecutor's fallacy if you will,
and I think that's a good thing. Because honestly, we were making that mistake before,
and now we can make a big deal out of whether
the readers of our reports make that mistake, and of course we should make a big deal out of it,
but it's good that's under discussion now,
because before, we made that mistake already, they didn't even have to make it anymore.
And another important thing is that
it didn't have as an effect that we had to redo all our previous cases from years back.
And that is important,
and I have to be very thankful for the legal community in the Netherlands
because they allowed us to improve.
And it sounds maybe a bit strange to be allowed to improve,
but if you cannot accept that you were worse in the past,
you have no way to improve.
So if it's almost made impossible to admit that something can be done better...
it inhibits you from becoming better. You don't have to be ill to get better we sometimes say.
Alright, I did a recent presentation in London and was responding to dr. Jim Wayman
and he was there at the time so, he's not here unfortunately now,
but I was asked to repeat this bit and so I'll end with that part
where I respond to the issues he brought up.
First, he talks about 'the very mystery that surrounds mathematical arguments'.
So, that makes it difficult to convey and
they might actually give a lot of credence to such arguments,
they might also give less credence to it,
but it might make it difficult to see what kind of weight you should attach to it.
My response would be that the likelihood ratio is exactly about which weight to assign to it,
and that it's true that a layman might not be doing a very good job.
You could of course just trust the expert and just copy that value,
I would not encourage that too much.
The expert should be very transparent about what that conclusion is based on.
So 'the expert opinion and the basis for it should be made as transparent as possible
to avoid the risk mentioned'.
Another one was that 'results are typically misstated'
and in the example given there's talk about 'support for the hypothesis
that the compared speech material does or does not originate from the same speaker
as opposed to the correct statement that the results do or do not show greater support
for the same-speaker hypothesis than for the alternative hypothesis.
And there I would agree, but you have to definitely be clear about
how much greater that support would be as well, so it's not just about which hypothesis supported
but also about the weight of that evidence, the strength of that support.
And evidential value would be meaningless if you would not explicitly
state an alternative hypothesis, if you would focus on just one hypothesis.
And he brings up: 'there is confusion as to whether the relevant population should model the
unknown speaker or the defendant
in terms of factors such as regional background, age, sex, socio-economic background and ethnicity',
and he adds: 'it should model defendant', and there I must say I disagree.
The relevant population is defined by the defense proposition,
and the circumstances of the case,
and it does not take into account
the defendant, in fact the defense is probably saying it was somebody else,
and doesn't even refer to the defendant in that.
Now in some cases, the defense might want to very narrowly define
the relevant population, based on the defendant,
hoping that the ratio will get very small and at the trier of fact
will forget about the prior odds.
For example, if they would say well it's somebody else with the same voice,
and it's not a joke I've seen this happen in a real case
where the defense came up with the alternative proposition that it was somebody else
with exactly the same voice,
and logically that's not a problem
as long as you take into account that somebody needs to look at
the prior odds as well as the likelihood ratio. The likelihood ratio in this case of course
would go down to one, but the prior odds would definitely be affected as well.
'Allocation to the forensic scientist of responsibility for computing the LR,
but not the priors, allows selection of a relevant population
to inflate / deflate the former while avoiding consideration of the inverse effect
on the latter', well this is really what we just talked about.
'If the trier of fact does not understand that deflating the likelihood ratio this way in fact
inflates the prior odds', by the defense's proposition being very improbable,
'mistakes can be made'.
'Doing evidence interpretation the right way requires competence.
The scientist cannot assign the prior odds which involve all the other evidence and information',
something we discussed quite extensively.
Then: 'determination of the likelihood ratio inherently involves subjective,
although not arbitrary, measures of belief in which personal probabilities are assigned
not calculated - a basis easily lost in the precise rhetoric of mathematics'.
I mention there that 'I think that essentially all probability is subjective,
but in many forensic cases very little structured data might be available.
The limited basis should be made very clear'. This is very important,
you cannot just state a conclusion and leave it unknown whether it's based
on enormous amounts of information or no information at all.
So, it should be made very clear what the basis is,
and if the basis is very limited of course that should also have consequences for
the magnitude of the likelihood ratio itself.
And then: 'Successive likelihood ratio estimates', I would never talk about estimates, but okay,
generally have an unknown degree of correlation
thus preventing their rational consideration in combination'.
When we talked about weight of evidence we said that well you can
put them all on the balance and they add up, just like weights add up,
but then you have to be talking about exactly the same hypotheses or propositions
and it needs to be completely independent. So it's quite rare that things are so simple.
So just using likelihood ratios does not necessarily make the combination of evidence easy.
I don't think anything will make the combination of evidence easy.
It would require things like Bayesian Networks that Anjali was talking about before.
Alright, then the last one
'there's no mechanism within the United States legal system for jurors
to establish prior probabilities before hearing the evidence,
nor to instruct them on how to apply likelihood ratios correctly to those priors'.
Well I am not optimisic for any rational method to work very well,
if you cannot properly train people to use such a method.
And as a forenic scientist I can only make sure that I do my part of the work right,
I cannot be held responsible for the other parts in the chain.
I will try to be as helpful as I possibly can,
but if the trier of fact is not able to use a rational method well,
then that's going to be a problem in any case no matter what method you come up with.
So, I think the forensic scientists should justify the methods they use,
and be very transparent about it, and I think triers of fact should also
justify the methods they use.
And I for one am very committed to enabling them to understand the meaning of my evidence,
and one way to do that was to work on this book.
And with that I thank you for your attention.