Simple technique determines whether or not a set of five data points lies within a specified close tolerance of a linear fit

Bari, F. A. N.

English

NASA Technical Reports Server

July 1970
	 Brief 70-10085 
S	 NASA TECH BRIEF 
NASA Tech Briefs announce new technology derived from the U .S. space program. They are issued to encourage commercial 
application. Tech Briefs are available on a subscription basis from the Clearinghouse for Federal Scientific and Technical 
Information, Springfield, Virginia 22151. Requests for individual copies or questions relating to the Tech Brief program may 
be directed to the Technology Utilization Division, NASA, Code UT, Washington, D.C. 20546. 
A Method for Rapidly Evaluating the Linearity of Calibration Data 
Full Scale Graph of Data Points and Reference Line 	 Difference Plot and Least-Snuaras Straiciht Lina Fit 
Difference Between V. and 
X Yi Corresponding Point on 
Line 
1 6 +1 
2 7
-1 
3 12 +1 
4 12
-2 
5 16
-1
Figure 1. Sample Data Set 
A simple technique is presented for determining 
whether or not a set of five data points lies within a 
specified close tolerance of a linear fit. 
The following theorem, which justifies this tech-
nique, will be proved: Define two arbitrary constants, 
T and C. Select data points (X i , Y,) with i = 1,2,3,4,5 
which have the property
- Fitted 
X Difference Difference Residual By Method of 
Least-Squares 
1 +1 0.6 0.4 
2 -1 0.1
-1.1 
3 +1 -0.4 1.4 
4 -2 -0.9
-1.1 
5 -1 -0.4 0.4
Figure 2. Difference Plot 
Let f(X) = a0
 + a 1 X be the least-squares linear fit 
to these data (regression of Y on X). If g(X) = b 0
 + b 1 X 
is another line which has the property 
IY - g(X) < T/1.6 for all i,	 (2) 
then	 IY - f(X1)I < T for all i. 	 (3) 
Consider the following situation. A calibration check 
X• 1
 - X = C.
	 (I)	 is run on amplifier modules with a linearity tolerance 
(continued overleaf) 
This document was prepared under the sponsorship of the National	 Government assumes any liability resulting from the use of the 
Aeronautics and Space Administration. Neither the United States	 information contained in this document, or warrants that such use 
Government nor any person acting on behalf of the United States
	 will be free frbm privately owned.rights. 
https://ntrs.nasa.gov/search.jsp?R=19700000081 2020-03-17T00:24:02+00:00Z
of 0.25% of the full-scale output. The input signal is 
increased in five equal steps. Full scale output is 
approximately 5 V (tolerance 0.0125 V). The expected 
results follow: 
Coded Nominal 
input output 
0 0.00V 
1.25 V 
2 2.50V 
3 3.75V 
4 5.00V
A logical first step to verify acceptable data linearity 
is to plot the points and see if a line can be drawn to 
bring the points within tolerance. In this case, how-
ever, if a graph is plotted using a scale of I V : 2 in., 
the tolerance is only 0.025 in., less than 1/32 in. Thus, 
plotted on 8-1/2 x II in. graph paper, the tolerance in 
question is barely discernible. To avoid this problem 
of scale, a difference plot is used. 
Figure 1 presents a sample data set (chosen delib-
erately to exaggerate certain aspects of difference 
plots) in tabular and graphic form, together with an 
approximately fit reference line: Y = 2 + 3X. Figure 
2 shows the difference plot, a graph of the vertical 
separation of each point from the reference line. These 
differences may be plotted on a magnified scale, to 
allow easy displayof discrepancies otherwise too small 
to see. The line plotted on Figure 2 is the least-squares 
best fit to the difference data 
h(X) = 1.1 - 0.3X.	 (4) 
In general, the least-squares line h(X) fitted to a 
difference plot is related to the reference line g(X) 
used to determine the differences and to the least-
squares line f(X) fitted to the raw data, by the relation-
ship
f(X) = g(X) + h(X).	 (5) 
Thus, in this example, 1(X) may be computed from the 
sum
g(X) = 2.0 + 3.OX 
h(X) = 1.1	 0.3X 
f(X) = 3.1 + 2.7X 
Worst-case analysis was used to obtain the constant 
1.6 which appears in equation 2. Let the tolerance on 
the y-distance be unity. Then the worst case of a line 
within tolerance exists when each of the five points is 
at the maximum distance of ±1 from the reference 
line. There are 32 such cases, but, because of the 
condition of equation I, imposed on the independent 
variable, certain of the 32 difference patterns possess 
the same maximum absolute distance of a point from 
Brief 70-10085
the least-squares line fit. Identification of identical 
patterns of differences reduces to eleven the number 
of essentially different cases. Each of these cases is 
tabulated below, giving the absolute value of the 
residual for the farthest point from the least-squares 
line.
Maximum
	
Difference	 absolute 
pattern	 residual 
I	 I	 I	 I	 I	 0.0 
I	 I	 I	 I	 -1	 1.4 
I	 I	 I-I	 I	 1.4 
I	 I	 I -1 -1	 0.8 
I I -1 I I	 1.6 greatest maximum residual 
I	 I	 -1	 1 ° -I	 1.2 
I	 I	 -1	 -1	 1	 1.2 
1 -1	 I	 I -1	 1.4 
I	 -1	 I	 -1	 1	 1.2 
I	 -1	 -1	 1 -1	 1.4 
1	 -1	 -1	 -1	 1	 1.2 
Since the greatest (absolute) y-distance of one of these 
points from a least-squares line is 1.6, the theorem. 
stated in equations 2 and 3, is proved. Note, however, 
that the theorem provides a sufficient, but not a neces-
sary condition. 
With this background, the amplifier module cali-
bration data may be evaluated. The tolerance is ad-
justed to 0.0078 V (0.0125/1.6). A standard form is 
used to make difference plots of the calibration data. 
If a line can be drawn on the difference plot which 
passes within 0.0078 V of each data point, the-linearity 
requirement is satisfied. If no such line can be drawn, 
judgement is deferred, and the part is held for further 
investigation. 
Notes: 
I. This technique has been successfully applied to 
reduce delays in a large-cale testing program. 
About 95% of a large group of amplifier modules 
which were evaluated using this technique, were 
accepted without waiting for computer curve fits. 
2. No additional documentation is available. Specific 
questions, however, may be directed to: 
Technology Utilization Officer 
Marshall Space Flight Center 
Huntsville, Alabama 35812 
Reference: B70-10085 
Patent status: 
No patent action is contemplated by NASA. 
Source: Frances A. Norton Bari of 
The Boeing Company 
under contract to 
Marshall Space Flight Center 
(MFS-14834) 
Category 03 


A method for rapidly evaluating the linearity of calibration data

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server