Error Rates and Random Match Probabilities Based on Research Findings
A comprehensive study conducted over nearly 25 years analyzed error rates and random match probabilities utilizing the RUGER 10-Barrel Test and GLOCK Cartridge Case Tests. The research, involving 626 scientists from 24 countries, established base error rates through advanced Bayesian techniques. The project produced a total of 240 test sets, with participants identifying 9,382 out of 9,390 unknown bullets. Noteworthy findings, methodologies, and conclusions from the project are detailed in this article.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Error Rates and Random Match Probabilities (RMP) Based on the RUGER 10-Barrel Test and the GLOCK Cartridge Case Tests James E. Hamby, Ph.D., David J. Brundage, M.S., Steven A. Norris, B.S., Nicholas D.K. Petraco, Ph.D., and James W. Thorpe, Ph.D.
Conclusions on Research Conducted over nearly 25 years involving the RUGER 10-Barrel Test, GLOCK Cartridge Case Tests Base error rates were established for both studies using advanced Bayesian techniques
The 10-Barrel Test The Identification of Bullets Fired From 10 Consecutively Rifled RUGER 9mm Pistol Barrels A Research Project Involving 626 Scientists From Labs In 24 Countries
James E. Hamby, Ph.D., International Forensic Science Laboratory & Training Centre, Indianapolis, Indiana David J. Brundage, M.S., Forensic Science Consultant, Nashville, Tennessee James W. Thorpe, Ph.D., University of Strathclyde, Glasgow, Scotland (Retired)
A total of 240 test sets were produced by Dave Brundage, Mickey French and Jim Hamby all then employed at the Indianapolis-Marion County Forensic Laboratory, Indianapolis, IN Recently 20 additional test sets were produced by Hamby and 10 sets sent to Europe and other countries for examination. The various test sets have used 4 different types of 9mm ammunition. Of the 626 participants in this research project, nine have used some type of ballistics imaging equipment.
Of the 626 participants in this worldwide project,9,382 of the possible 9,390 unknown bullets were identified. (15 unknowns x 626 participants = 9,390) Three bullets were mutilated during test firing (tank rash) and five unknown bullets could not be associated to the known bullets by two young examiners-in-training - listed as inconclusive.
This worldwide research project and numerous other research projects reported in the scientific literature the AFTE Journal, the Journal of Forensic Sciences, and others - since the early 1930 s should provide data that will obviate some of the concerns within the legal community - especially in the United States.
Request for Assistance We request that those members who haven t participated take the test in their laboratory & submit their answers! We also request that you return the set if you are no longer using it as there are labs that have requested test sets. Some labs may have two sets in their possession.
GLOCK Cartridge Case Test The Examination, Evaluation and Identification of 9mm Caliber Cartridge Cases Fired from 1,632 GLOCK Semiautomatic Pistols Manufactured Over a 23 Year Period - Using Optical Microscopy, Confocal Microscopy and Computational Pattern Recognition
This paper was presented at the 2013 AAFS Meeting by Petraco & Hamby with Norris as a co-author. The 1,632 fired cases were examined and the original 617 fired cases evaluated using the heritage IBIS.
This GLOCK research involved both IBIS and examiner based results. (over 1,300,000 exams) Nick Petraco will discuss the Error Rates/RMP for both research projects in a few minutes.
Modified SWGGUN Power Point Using the excellent SWGGUN power point, I have made a few changes that work well when testifying especially in Frye or Daubert Hearings. The following are a few of the changes. Our beloved AV man Mike has the power point if you are interested.
Definition: Subjective Examination The results of a subjective examination are based on an individual s opinion based on his/her training and experience. This does not mean that this type of examination is unreliable or unscientific. There is subjectivity in every science and in every test, whether it is a doctor diagnosing a head cold, a physician examining an x-ray or a cytotechnologist examining slides for cancer cells (pap smears).
Grant or Accepted Government Teaching Programs for Firearms Identification National Institute of Justice, Washington, D.C. NIJ - in conjunction with the National Forensic Science Technology Center (NFSTC) funded a distance learning program titled Firearms Examiner Training Program to augment training for new examiners, Federal Bureau of Investigation (FBI) Since 1986, the FBI has offered a one week course titled Specialized Techniques in Firearms Identification . The course has been attended by several hundred firearms examiners. Bureau of Alcohol, Tobacco and Firearms (ATF) Since 1999, the ATF has offered a one-year long course titled the National Firearms Examiner Academy. To date, 140+ firearms examiners have attended the NFEA. (Funded by ATF and NIJ) California Criminalistics Institute (CCI), CA DOJ For several years, CCI has offered a variety of specialized courses in firearms and toolmark related fields.
Grant or Accepted Government Teaching Programs for Firearms Identification United Nations Office on Drugs and Crime (UNODC) In 2013, the UNODC with funding from the Canadian Government is developing a limited service forensic laboratory for the Palestinian Civil Police (PCP) Temporary Forensic Science Laboratory (TFSL) in Ramallah, Palestine.. The TFSL will have the following forensic specialties: Forensic Chemistry Drug and Arson Analysis Questioned Documents Analysis Firearms and Tool Mark Identification United States Department of State In 2014, the US Department of State through the Bureau of International Narcotics and Law Enforcement Affairs, US Embassy, Belize, funded a program to strengthen the Belize National Forensic Science Laboratory by providing technical assistance related to the handling, processing and analysis of ballistics (firearms / tool marks) related evidence. Other programs such as ICITAP teach firearms and tool mark Identification around the world.
Summary Firearm & Toolmark Identification meets the reliability standard put forth by both Daubert or Frye requirements. The science has been accepted in every state and Federal court that uses Daubert or Frye. Examples: A recent (9-11) Indiana Supreme Court Decision (Frye) upheld the reliability of Firearms and Tool Mark Identification; and a recent (12-11) Kansas District Court Decision (Frye) upheld general acceptance in the field. A recent (7-13) Florida Court Issued a Daubert decision upholding the science. Cases are available for review on the SWGGUN website (under ARK) at www.swggun.org.
Data Analysis Average Examiner Error Rate and Random Match Probability The Good News: o 10-Barrel Test: # examiner errors committed = 0 o GLOCK Cartridge Case Test: # of falsely matched cartridge cases = 0 THIS SHOULD BE GREAT! We WANT error rates of 0% for good classification systems!!
Data Analysis Average Examiner Error Rate and Random Match Probability The Bad News: o You have to deal with the U.S. court system o0% error rates present an opportunity to muddy waters with the unrealistic study criticisms oLittle court interest in understanding the technicalities of estimating small error rates.
Data Analysis Average Examiner Error Rate and Random Match Probability So what can we do? 0% error rate is the frequentist estimate o Bayesian statistics provide complementary methods oCan work much better in estimating small probabilities oWe looked to sports statistics for low scoring games
Data Analysis Average Examiner Error Rate and Random Match Probability For 10-Barrel we need to estimate a small error rate For GLOCK we need to estimate a small random match probability (RMP) Use Bayesian Beta-binomial method when no failures are observed (Schuckers)
Data Analysis Bayesian Methodology Basic idea of the reverend Bayes: Prior Knowledge Data = Updated Knowledge Error Rate/RMP = + U( , ) Beta-Binomial(data | , ) Posterior( , | data) Get updated estimates of Error rate/RMP
Data Analysis Bayesian Methodology So given the observed data and assuming prior ignorance o Posterior error rate/RMP distributions: Posterior Distribution of Average Examiner Error Rate for 626 Participants Posterior Dist. 10-Barrel Posterior Distribution of RMP, 1632 Cartridge Cases Posterior Dist. GLOCK 400 RMP Average Examiner Error Rate 0.011% [0.00023%, 0.040%] 300 300 0.000086% Frequency Frequency [0.0000020%, 0.00031%] 200 100 100 0 0 0e+00 0.00 0.02 0.04 0.06 0.08 0.10 2e 04 4e 04 6e 04 8e 04 RMP (%) Error Rate (%)
Data Analysis Extrapolations Estimated Average Examiner Error Rate (%) 10-Barrel GLOCK Estimated Examiner Error Rate (%) Examiner Error Rate (%) 0.020 4e 04 RMP (%) Actual: Actual: 626 participants 1632 cartridge cases 2e 04 0.010 0e+00 300 400 500 600 700 800 900 1000 # Cartridge Cases Examined With No Errors in I.D. 2000 3000 4000 5000 6000 # of Participants # of Participants # Cartrige Cases Examined with No Errors in Identification
