Kelsea Cox, Lindsay Jones, R. Michael Lawton, Frode Stavehaug

Download full paper

Qualification is required before Boeing accepts materials from a supplier. This qualification ensures that the material meets Boeing requirements in the form of criteria for statistical distributions of material properties. Often, these criteria include requirements for both central tendency and spread of the distribution. However, it is also common for these requirements to take the form of more complex distribution attributes. Specifically, A- and B-basis requirements, as defined in the MMPDS Handbook [1], are often used to ensure that no more than a specified percentage of the distribution will fall below a defined value. The A-basis requirement states that 99% of the distribution falls above a defined value with 95% confidence, and the B-basis requirements states that 90% of the distribution falls above a defined value with 95% confidence. These types of requirements are known as allowables because they allow only a pre-specified percentage of samples to fall below a given value.

To provide context for this effort, when a second source is introduced to produce an existing material, the new material must meet requirements defined by the primary source. Verification that the supplier meets these allowables requirements can be difficult and expensive, as narrow confidence intervals about quantiles require many samples. MMPDS provides guidance on how to qualify a second source; however, limitations of current methodology are well-known in the community.

Once the supplier is qualified, lot release testing is required for acceptance of the supplier materials. This process is very similar to statistical process control (SPC) found in industrial statistics and six sigma practices.

SPC traditionally begins with a requirement on the process capability index (Cpk) which provides a measure of the location and spread of the distribution with respect to specification limits. A high Cpk indicates low fallout rates. Figure 1 provides a detailed flow diagram of the SPC process.

This paper provides an explanation of how merging allowables methodologies with traditional SPC approaches can benefit the qualification process by enabling acceptable sources to qualify without sacrificing the integrity of published allowables. By framing the problem of supplier qualification in the language of SPC, we not only benefit from the sound theory of standard industrial statistics but also bring to bear the accompanying suite of monitoring methodologies. This paper describes the calculation of Cpk to qualify a second source material. Once a second source is qualified, traditional SPC principles can easily be implemented per standard textbook practices.

In order to assess performance of each of two legacy methods and the proposed SPC approach across a broad range of possible baseline and alternate material distributions, we carry out the previously described simulation process for every combination of the test conditions given in Table 1. In total, we evaluate test performance for over 150,000 test cases. In each test case, we simulate 10,000 samples from the alternate supplier and apply each of the three statistical methods, computing error rates for each method. From this collection of observed Type I error rates, we determine the maximum Type I error rate for each method and examine the distribution of Type I error rates across all test cases for each method. The true expected outcome for allowables verification Type I error should be less than 5% because allowables are defined as a 95% mathematical confidence bound about a quantile.

Across all test cases, the maximum Type I error rate for SPC method and the Legacy Approach 2 was never significantly greater than 5%, thus the Type I error rate is controlled in these tests and our published allowable is protected when we accept new suppliers. Legacy Approach 1, on the other hand, has a maximum Type I error rate close to 75%. This method fails to protect the allowable, and suppliers we accept using this method may not truly meet the requirement, which may lead to inappropriate design use and potential escapes. The SPC method and Legacy Approach 2 perform as expected for a statistical test claiming a 95% level of confidence. Legacy Approach 1, in contrast, exhibits many values above the expected Type I error rate, indicating that it violates the purported level of confidence. The fact that the SPC method performs as expected results from its foundation in sound statistical theory. In fact, the Legacy Approach 2 method simplifies to the SPC approach when the information utilized in the approach is inferred from data collected. However, the power of Legacy Approach 2 decreases to zero when information is not inferred from the data. That is, due to the use of lookup tables in the implementation of Legacy Approach 2, under certain conditions, the test will not allow for acceptance of a second source supplier even if the performance of the second source is demonstrated to be far superior to that of the baseline supplier. This can be seen in Figure 2. Note that Type II error can only be fairly evaluated for methods with controlled Type I error. Because the Type I error rate was not controlled for Legacy Method 1, this method’s Type II error rate was not assessed.

Type II Error Example

We provide an example in order to illustrate the mechanisms behind the performance comparison in terms of Type II error. In this example, an alternate second source material is simulated through draws from a normal distribution characterized by a known mean and standard deviation.

In this example, we examine the behavior of each method in a case for which we expect to accept the alternate material. We stipulate an alternate material following a normal distribution with a mean of 72.5 ksi and standard deviation of 0.5 ksi. The 10th percentile of this distribution is 71.9 ksi, and we specify a requirement of 72 ksi with a 1 ksi equivalence margin, meaning that the B-basis must be within 1 ksi of the requirement. Therefore, the alternate supplier meets the requirement, and we expect to accept the supplier given sufficient sample size. For this example, we consider sample sizes of 40 and 80, for which we expect B-basis values of approximately 72 ksi.

Acceptance rates from simulation for the three methods are shown in Table 2. Due to the use of lookup tables, Legacy Approach 2 never accepts the material for either of the examined sample sizes. Legacy Approach 1 only accepts the material 3.3% of the time for a sample size of 40 and 5.0% of the time for a sample size of 80. The SPC approach, on the other hand, accepts the material 100% of the time for sample sizes of both 40 and 80. While Type II error is not as severe as a Type I error in terms of risk avoidance, a high Type II error rate is indicative of increased cost. Although we did not consider overall Type II error rates for Legacy Method 1 in the previous section due to the method’s uncontrolled Type I error rate, this example demonstrates that the method will perform poorly in terms of Type II error in specific test cases as well. Conclusions drawn from this example indicate that both Legacy Method 1 and Legacy Method 2 can inflate Type II error (increasing cost due to unnecessary material rejection).

Now that we understand the behaviors of each statistical method, we can assess the practical use for each approach. While Legacy Approach 2 is statistically sound for our use case (in the sense that it has a controlled Type I error rate), it imposes a more rigorous sampling plan and thus unnecessarily increases cost. The SPC approach, however, maintains an accepted Type I error rate while providing the lowest Type II error theory allows. For these reasons, the SPC approach is the most appropriate method for qualifying second source materials.

Conclusions

In this paper we have introduced and described a method for second source supplier qualification using theoretically sound tools from SPC that are standard to industrial statistics. We conclude that this approach is superior to previously proposed methods for second-source qualification through simulation and analysis. Furthermore, this approach opens the door to the reconciliation of common practices such as lot acceptance sampling and the development of SPC control charts, an avenue to advancing and standardizing supplier quality control. By allowing application of these SPC methods to quality control of raw materials, we can enable process monitoring and early fault detection capabilities already appreciated by other high throughput industrial applications.

References

1. “Metallic Materials Properties Development and Standardization (MMPDS) – 12.” July 2017.
2. Scholz, F., Vangel, M. “Tolerance Bounds and Cpk Confidence Bounds Under Batch Effects.” Advances in Stochastic Models for Reliability, Quality, and Safety, 1998, pp. 361-379. 3. R Core Team (2016). “R: A language and environment for statistical computing. R Foundation for Statistical Computing,” Vienna, Austria. https://www.R-project.org/