ASTM D6299 Standard Practice for Applying Statistical Quality Assurance Techniques to Evaluate Analytical Measurement System Performance
8. Procedure for Pretreatment, Assessment, and Interpretation of Test Results
8.1 Overview - Results accumulated from QC, check standard, and VA sample testing are pretreated and screened. Statistical techniques are applied to the pretreated data to achieve the following objectives:
8.1.1 Identify erroneous data,
8.1.2 Assess initial results,
8.1.3 Deploy, interpret and maintain of control charts, and
8.1.4 Quantify long term measurement precision and bias.
NOTE 13 - Refer to the annex for examples of the application of the techniques that are discussed below and described in Section 9.
8.2 Pretreatment of Test Results - Assessment, control charting, and evaluation are applied only to appropriately pretreated test results. The purpose of pretreatment is to standardize the control chart scales so as to allow for data from multiple check standards to be compared on the same chart.
8.2.1 For QC sample test results, no data pretreatment is typically used since results for different QC samples are generally not plotted on the same chart.
8.2.2 For check standard sample test results, two cases apply, depending on the measurement system precision:
8.2.2.1 Case 1 - If either (1) all of the check standard test results are from one or more lots of check standard material having the same ARV(s), or (2) the precision of the measurement system is constant across levels, then pretreatment consists of calculating the difference between the test result and the ARV:
Pretreated result = test result - ARV(for the sample)
8.2.2.2 Case 2 - Test results are for multiple lots of check standards with different ARVs, and the precision of the measurement system is known to vary with level,
where the standard deviation at the ARV level is the published reproducibility standard deviation. In the event that no published reproducibility exists and the ARV was established through round robin testing, standard deviations determined from round robin testing may be used.
8.2.2.3 If there is no published reproducibility standard deviation and the ARV was not arrived at by round robin testing, a standard deviation should be determined by users in a technically acceptable manner.
NOTE 14 - It is recommended that the method used to determine the standard deviation be developed under the guidance of a statistician.
NOTE 15 - To calculate the reproducibility standard deviations from published reproducibilities, divide the accepted reproducibility value at each level by 2.77.
8.2.3 Pretreatment of results for VA samples is done in the same manner as described in 8.2.1 and 8.2.2.
8.3 Assessment of Initial Results - Assessment techniques are applied to test results collected during the startup phase of or after significant modifications to a measurement system. Perform the following assessment after at least 15 pretreated results have become available. The purpose of this assessment is to ensure that these results are suitable for deployment of control charts (described in A1.4).
NOTE 16 - These techniques can also be applied as diagnostic tools to investigate out-of-control situations.
8.3.1 Screen for Suspicious Results - Pretreated results should first be visually screened for values that are inconsistent with the remainder of the data set, such as those that could have been caused by transcription errors. Those flagged as suspicious should be investigated. Discarding data at this stage must be supported by evidence gathered from the investigation. If, after discarding suspicious pretreated results there are less than 15 values remaining, collect additional data and start over.
8.3.2 Screen for Unusual Patterns - The next step is to examine the pretreated results for non-random patterns such as continuous trending in either direction, unusual clustering, and cycles. One way to do this is to plot the results on a run chart (see A1.3) and examine the plot. If any non-random pattern is detected, investigate for and eliminate the root cause(s). Discard the data set and start the procedure again.
8.3.3 Test "Normality" Assumption - For measurement systems with no prior performance history, or as a diagnostic tool, it is useful to test that the results from the measurement are adequately described by a normal distribution. One way to do this is to use a normal probability plot and the Anderson-Darling Statistic (see A1.4). If the results show obvious deviation from normality, then the statistical control charting techniques described are not directly applicable to the measurement system.
NOTE 17 - Transformations may lead to normally distributed data, but these techniques are outside the scope of this practice.
8.4 Control Charts (1, 2) - Individual (I) and moving range of two (MR) control charts are the recommended tools for (a) routine recording of QC sample and check standard test results, and (b) immediate assessment of the "in statistical control" (3) status of the system that generated the data. Optionally, the exponentially weighted moving average (EWMA) (4, 5) may be overlaid on the I chart to enhance detection power for small level shifts.
NOTE 18 - The control charts and statistical techniques described in this practice are chosen for their simplicity and ease of use. It is not the intent of this practice to preclude use of other statistically equivalent or more advanced techniques, or both.
8.4.1 Construction of Control Charts - If no obvious unusual patterns are detected from the run charts, and no obvious deviation from normality is detected, proceed with construction of the control charts
8.4.1.1 MR Chart - Construct an MR plot and examine it for unusual patterns. If no unusual patterns are found in the MR plot, calculate and overlay the control limits on the MR plot to complete the MR chart.
8.4.1.2 I Chart - Calculate control limits and overlay them on the "run chart" to produce the I chart.
8.4.1.3 EWMA Overlay - Optionally, calculate the EWMA values and plot them on the I chart. Calculate the EWMA control limits and overlay them on the I chart.
8.4.2 Control Chart Deployment - Put these control charts into operation by regularly plotting the pretreated test results on the charts and immediately interpreting the charts.
8.5 Control Chart Interpretation:
8.5.1 Apply control chart rules (see A1.5) to determine if the data supports the hypothesis that the measurement system is under the influence of common causes variation only (in statistical control).
8.5.2 Investigate Out-of-Control Points in Detail - Exclude from further data analysis those associated with assignable causes, provided the assignable causes are deemed not to be part of the normal process.
NOTE 19 - All data, regardless of in-control or out-of-control status, needs to be recorded.
8.6 Scenario 1 for Periodic Updating of Control Charts Parameters:
8.6.1 Scenario 1 covers (a) control charts for a QC material where there had been no change in the system, but more data of the same level has been accrued; or (b) control charts for check standard pretreated results.
8.6.2 When a minimum of 15 new in-control data points becomes available, the precision estimate used to calculate the control limits can be updated to incorporate the information from this new data. Update calculations that involve pooling of old and new data sets shall be preceded by an F-test (see A1.8) of sample variances for the new data set versus the existing in-control data set.
8.6.3 If the outcome of the F-test is not significant, then the precision estimate is updated by statistically pooling both sample variances. A significant F-test should trigger an investigation for assignable causes.
8.7 Scenario 2 for Periodic Updating of Control Charts Parameters:
8.7.1 Scenario 2 covers control chart for QC materials where an assignable cause change in the system had occurred due to a change in the level for the QC material. Minor or major differences may exist between QC material batches. Since control limit calculations for the I chart require a center value established by the measurement system, a special transition procedure is required to ensure that the center value for a new batch of QC material is established using results produced by a measurement system that is in statistical control. This practice presents two procedures to be selected at the users' discretion.
8.7.2 Procedure 1 - Concurrent Testing:
8.7.2.1 Collect and prepare a new batch of QC material when the current QC material supply remaining can support no more than 20 analyses.
8.7.2.2 Concurrently test and record data for the new material each time a current QC sample is tested. The result for the new material is deemed valid if the measurement process in-control status is validated by the current QC material and control chart.
8.7.2.3 Optionally, to provide an early indication of the status of the new batch of QC material, immediately start a run chart and an MR plot for the new material. After five valid results become available for the new material, convert the run chart into an I chart with trial control limits by adding a center line based on the average of the five results and control limits based on the MR from previous control charts for materials at the same nominal level. Set trial control limits for the MR chart based on limits from previous charts for materials at the same nominal level.
8.7.2.4 After a minimum of 15 in-control data points are collected on the new material, perform an F test of sample variances for the new data set versus the historical variance demonstrated at nominal level of the new material. If the outcome of the F test is not significant then the precision estimate is updated by statistically pooling both sample variances. A significant F test should trigger an investigation for root cause(s).
8.7.2.5 Construct new I and MR charts (and optional EWMA overlay) for this new material as per Section 8, using the pooled MR .
8.7.2.6 Switch over to the new I and MR charts upon depletion of current QC material.
8.7.3 Procedure 2 - Q Procedure (see A1.9) (6):
8.7.3.1 This procedure is designed to alleviate the need for concurrent testing of two materials. A priori knowledge of the measurement process historical standard deviation applicable at the new QC material composition and property level is required.
NOTE 20 - It is recommended that this standard deviation estimate be based on at least 50 data points.
8.7.3.2 When the Q procedure is operational (minimum of two data points), it can be used in conjunction with a MR chart constructed using the observations to provide QA of the measurement process.
NOTE 21 - The Q procedure is not suitable for monitoring measurement system bias relative to an external value. It is designed to monitor the stability of the system mean. When used in conjunction with the MR chart, "in statistical control" status of the measurement system can be ascertained.