Determination of Evaluation Criteria and Performance Assessment Procedures
Evaluation criteria and performance assessment procedures for participants are crucial in determining proficiency levels for different measurements. This includes calculating standard deviations, setting maximum permissible errors, and making decisions based on expert perceptions. Methods such as using previous data, general models, and expert judgments play a role in defining acceptable performance levels.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Determination of EVALUATION CRITERIA & different procedures for performance evaluation of participants S. Subramanian
EVALUATION CRITERIA Standard Assessment dispersion used in the evaluation of results of Proficiency Testing, based on available information Deviation SDPA for Proficiency Measure ( pt)- of Maximum (Limit of Error) Delta E ( E)- Extreme value of measurement error, with respect to a known reference quantity value, permitted by specifications or regulations for a given measurement, measuring measuring system permissible measurement error instrument, or 2 03-03-2025 S.SUBRAMANIAN
Calculation of SDPA 1. By perception of experts (8.2) 2. By experience from previous rounds of a PT Scheme (8.3) 3. By use of a General model (8.4) 4. Using the repeatability and reproducibility standard deviations from a previous collaborative study of precision of a measurement method (8.5) 5. From data obtained in the same round of a PT Scheme (8.6) 3 03-03-2025 S.SUBRAMANIAN
By perception of experts (8.2) MPE or SDPA may be set at a value that corresponds to the level of performance that a regulatory body, AB or technical experts of the PT provider believe is reasonable for participants. E.g Analytical method for aflatoxin in nuts not to have reproducibility CV of more than 50% . Fix 50% of assigned value as SDPA 4 03-03-2025 S.SUBRAMANIAN
By perception of experts (8.2) If a regulatory requirement or fitness for purpose goal is given as a SD, it may be directly used as SDPA If fitness for purpose goal is given as maximum permissible measurement error, it may be directly used as Delta E ( E) Delta E ( E) may be transformed to SDPA, by dividing this limit by the number of multiples of SDPA that are used to define an action signal (or unacceptable result) and vice versa 5 03-03-2025 S.SUBRAMANIAN
By perception of experts (8.2) Choice according to a fitness-for- purpose - wish for the laboratory Fitness of purpose decided by Regulatory authority, Accreditation Body or Technical experts of PT Provider This value can be unrealistic to the reproducibility of the method 6 03-03-2025 S.SUBRAMANIAN
By experience from previous rounds of a PT Scheme (8.3) Consider the assigned value and SDPA fixed in a no. of accredited PT Schemes conducted by different PT Providers for same matrix, measurand and concentration range Calculate SDPA % for each of the PT rounds Fix the average or median of SDPA % for the current round. 8 03-03-2025 S.SUBRAMANIAN
By experience from previous rounds of a PT Scheme (8.3) PT Code ABC 3 ABC 7 ABC 9 ABC 12 ABC 15 XYZ 2 XYZ 7 XYZ 9 XYA 14 SDPA 1.72 5.42 3.12 7.14 3.21 2.38 3.58 4.56 1.14 SDPA% 11.3 13.3 12.3 11.7 10.4 11.6 11.4 11.2 11.3 Assigned Value 15.2 40.6 25.4 60.8 30.8 20.6 31.5 40.8 10.1 Median= 11.4 9 03-03-2025 S.SUBRAMANIAN
By experience from previous rounds of a PT Scheme (8.3) SDPA 8 y = 0.1204x - 0.1035 R = 0.9764 7 6 5 4 3 2 1 0 0 10 20 30 40 50 60 70 10 03-03-2025 S.SUBRAMANIAN
By use of a General model (8.4) Horwitz formula pt = 0.22 x c when c is less than 1.2 x 10-7 (<120 ppb) pt = 0.02 x c 0.8495 when 1.2 x 10-7< c < 0.138 (Between 120 ppb to 13.8 %) pt = 0.01 x c 0.5 when c is more than 0.138 (>13.8%) where c is the concentration of the analyte, expressed as a mass fraction 11 03-03-2025 S.SUBRAMANIAN
By use of a General model (8.4) Value of the SDPA is derived from a general model for the reproducibility of the measurement method (Horwitz curve) Advantages SDPA can be derived from a general model for the reproduciblity of the measurement method Has advantage of (a) objectivity (b) consistency across measurands & (c) being empirically based Disadvantage: The true reproduciblity can vary from the value of the model Reproducibility is dependent on the parameter, the method, sample and not only on the concentration 12 03-03-2025 S.SUBRAMANIAN
Using the repeatability and reproducibility standard deviations from a previous collaborative study of precision of a measurement method (8.5) When one standardised method is used in the PT Requirement: Information of the repeatability and reproducibility must be available pt = [ R2 - r2 (1- 1/m)] where m is the no. of replicate measurements done on the PT item. 13 03-03-2025 S.SUBRAMANIAN
From data obtained in the same round of a proficiency testing scheme (8.6) Calculated with robust statistic from the results of the participants in PT Robust statistic includes: a) Algorithm A b) nIQR c) Q-Method 14 03-03-2025 S.SUBRAMANIAN
Robust methods used for determination of Assigned Value and/or SDPA perfrom acceptably only when the effective number of participants, p (after removal of blunders) is 12 15 03-03-2025 S.SUBRAMANIAN
From data obtained in the same round of a proficiency testing scheme (8.6) Advantages: This approach has simplicity and conventional acceptance due to successful use Disadvantages: SDPA may vary from round to round and Trend analysis is difficult SDPA can be unreliable when (a) p is small and (b) results from different methods are combined Using dispersion measures from the data will lead to constant proportion of APPARENTLY ACCEPTABLE scores 16 03-03-2025 S.SUBRAMANIAN
Performance evaluation of participants
PERFORMANCE STATISTICS 1. Estimation of Deviation D (9.3) 2. Estimation of Deviation %- D% (9.3) 3. z Score(9.4) 4. z - Score (9.5) 5. Zeta Score ( ) (9.6) 6. En Score (9.7) 7. PA Score (9.8) 18 03-03-2025 S.SUBRAMANIAN
Estimation of Deviation D Deviation, Di = (xi xpt) xi - value reported by participant i xpt - Assigned Value To assess performance evaluation of participant i , compare Di with MPE for the concerned test parameter. Di MPE : Performance is satisfactory Di > MPE : Performance is unsatisfactory 19 03-03-2025 S.SUBRAMANIAN
Estimation of Deviation % D % Deviation, Di %= (xi xpt)/ xpt X 100 xi - value reported by participant i xpt - Assigned Value To assess performance evaluation of participant i , compare Di with MPE % for the concerned test parameter. Di % MPE %: Performance is satisfactory Di % > MPE % : Performance is unsatisfactory 20 03-03-2025 S.SUBRAMANIAN
Estimation of z Score z Z Score, zi = (xi xpt)/ pt xi - value reported by participant i xpt - Assigned Value pt - SDPA Interpretation of |z| score: Between 0 to 2.0 : Satisfactory; 2.0 but < 3.0 : Questionable (Straggler); 3.0 : Unsatisfactory (Outlier). 21 03-03-2025 S.SUBRAMANIAN
Estimation of z prime Score z Z Score, z i = (xi xpt)/ [ pt2 +u(xpt)2] xi - value reported by participant i xpt - Assigned Value pt - SDPA u(xpt) -Std. uncertainty of Assigned Value Interpretation of |z | score: Between 0 to 2.0 : Satisfactory; 2.0 but < 3.0 : Questionable (Straggler); 3.0 : Unsatisfactory (Outlier). 22 03-03-2025 S.SUBRAMANIAN
When to use z prime Score ? Criteria to use z score: When u(xpt) 0.3 pt u(xpt) - Std. uncertainty of Assigned Value pt - SDPA Use z score only if the above criteria is not complied. In z score allowance for uncertainty of assigned value is considered while evaluating performance of participant S.SUBRAMANIAN 03-03-2025 23
Estimation of Zeta Score Zeta Score, i = (xi xpt)/ [u(xi)2 +u(xpt)2] xi - value reported by participant i xpt - Assigned Value u(xi) -Std. uncertainty of participant i u(xpt) -Std. uncertainty of Assigned Value Interpretation of | | score: Between 0 to 2.0 : Satisfactory; 2.0 but < 3.0 : Questionable (Straggler); 3.0 : Unsatisfactory (Outlier). En Score is used for validation of uncertainty claims 24 03-03-2025 S.SUBRAMANIAN
Estimation of En Score En Eni = (xi xpt)/ [U(xi)2 +U(xpt)2] xi - value reported by participant i xpt - Assigned Value u(xi) - Expanded uncertainty of participant i u(xpt) - Expanded uncertainty of Assig. Value Interpretation of |En | score: Between 0 to 1.0 : Satisfactory; 1.0 : Unsatisfactory (Outlier). En Score is used in calibration PT Schemes 25 03-03-2025 S.SUBRAMANIAN
Estimation of PA Score PA PAi = (xi xpt)/ E xi - value reported by participant i xpt - Assigned Value E - MPE fixed for the concerned test Interpretation of | PA | score: Between 0 to 1.0 : Satisfactory; 1.0 : Unsatisfactory (Outlier). 26 03-03-2025 S.SUBRAMANIAN
Relationship between MPE & SDPA MPE = 3 X SDPA SDPA = MPE/3 E = 3 pt pt = E /3 27 03-03-2025 S.SUBRAMANIAN
EVALUATION CRITERIA GENERAL CONSIDERATIONS Statistics used for performance evaluation to be consistent with OBJECTIVES of PT Scheme Performance statistics are meaningful only when participants results are reviewed & are determined to be consistent with the DESIGN of PT Scheme. i.e No evidence of - deterioration of PT items - mixture of populations (groups) of participants - no severe violation of any statistical assumptions Not appropriate to use evaluation methods that intentionally classify a fixed proportion of results as generating an ActionSignal 28 03-03-2025 S.SUBRAMANIAN
IF En SCORE OR ZETA SCORE IS USED, THESE WILL BE MEANINGFUL ONLY IF THE UNCERTAINTY ESTIMATES ARE DETERMINED BY THE PARTICIPANTS IN A CONSISTENT MANNER USING ISO/IEC GUIDE 98 29 03-03-2025 S.SUBRAMANIAN
