An experiment was designed to test the ability of the Classification and Mensuration Subsystem rework operations to improve wheat proportion estimates for segments that had been processed previously. Sites selected for the experiment included three in Kansas and three in Texas, with the remaining five distributed in Montana and North and South Dakota. The acquisition dates were selected to be representative of imagery available in actual operations. No more than one acquisition per biophase were used, and biophases were determined by actual crop calendars. All sites were worked by each of four Analyst-Interpreter/Data Processing Analyst Teams who reviewed the initial processing of each segment and accepted or reworked it for an estimate of the proportion of small grains in the segment. Classification results, acquisitions and classification errors and performance results between CAMS regular and ITS rework are tabulated

Chhikara, R. S.

Hsu, E. M.

Liszcz, C. J.

English

NASA Technical Reports Server

General Disclaimer 
One or more of the Following Statements may affect this Document 
 
 This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 
 
 This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 
 
 This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 
 
 This document is paginated as submitted by the original source. 
 
 Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 
 
 
 
 
 
 
 
Produced by the NASA Center for Aerospace Information (CASI) 
https://ntrs.nasa.gov/search.jsp?R=19760021544 2020-03-22T14:26:32+00:00Z
7
(NASA-CR-147829) LACIE PHASE 1
	 N76-28632
CLASSIFICATION AND MENSURATION SUBSYSTEM
(CAMS) REWORK EXPERIMENT (Lockheed
Electronics Co.) 13 p HC $3.50	 CSCL 02C	 Unclas
G3/43 48572
l0CKHc10	 E(I^R0NICS	 CDMPA, N y .	 fNC
AEROSPACE SYSTEMS DIVISION
16111	 f 	 C A M I N 0 NEAT	 HOUSTON, TEXAS 110$1	 Y	 TELEPHONE (AREA
	 CODE 7'1])	 /61 0010
Ref: 642-1975
Contract NAS 9-12200
Job Order 73-763
NASA GR-
TECHNICAL MEMORANDUM
LACIE PHASE I CLASSIFICATION AND MENSURATION
SUBSYSTEM (CAMS) REWORK EXPERIMENT
By
R. S. Chhikara, E. M. Hsu, and C. J. Liszcz
Approved By:
M. L. Mannen, Supervisor
Test and Evaluation Section
t
Distribution:
JSC/R. M. Bizzell
L.	 F. Childs
A.	 L. Grandfield
F.	 G. Hall
R.	 P. Heydorn
D.	 E. Pitts
LEC/M. L. Bertrand >. _ 
B.	 L. Carroll
J.	 E. Davis
T.	 C. Minter{ rnl4	 1
C.	 T. Nosworthy
!.
J. A. Quirein
€	 D.	 G. Saile NUT ^.^IM,
P.	 C. Swanz Y
R.	 E. Tokerud
I	 REDAF (3
Technica.l.Library (2)
Job Order File
x June 1976	 LEC-8655
I	 A	 S U	 B S	 I D	 I A R Y	 O F L O C K H E E D	 A I R C R A F T	 C 0 R P 0 R A T 1.0 N
1. INTRODUCTION
An experiment was designed to test the ability of the Classifica-
tion and Mensuration Subsystem (CAMS) rework operations to improve
wheat proportion estimates for segments that had been processed
previously. Eleven segments were selected for the experiment,
including three in Kansas and three in Texas, with the remaining
d
five segments distributed in Montana and North and South Dakota.
The Kansas and Texas sites were selected to provide information
on the southern Great Plains. The remaining sites were selected
to augment the knowledge acquired from the blind site study of
the mixed and spring wheat sites in the northern Great Plains.
The acquisition dates were selected to be representative of
imagery available in actual operations. No more than one acqui-
sition per biophase would be used, and biophases were to be deter-
mined by actual crop calendars. All sites were intensive test 	
3N
sites (ITS's) over which at least two passes had been made, and
each had an acquisition from either biophase 2 or 3.
i
3
All sites were worked by each of four Analyst-Interpreter/Data
Processing Analyst (AI /DPA) Teams randomly selected from teams
which were familiar with CAMS rework methodology. Each AI/DPA 	 3
Team reviewed the initial processing of each segment and accepted
or reworked it for an estimate of the proportion of small grains 3
in the segment. Specific procedures and required rework products
are described in section 2. The classification results are given
in table I; table II lists the acquisitions; and tables III and
 IV describe classification errors and performance results between
CAMS regular and ITS rework.
i
The experimental design provided 44 classification runs with a
factor of 4 (AI/DPA Teams) times 11 (sites) 	 However, two of the
F	
i
isites were declared unworkable by some of the teams and were
subsequently dropped from the experiment.
2. PROCEDURES
a. The AI/DPA Team reviews processed acquisitions for a segment
and determines whether the processed acquisition acreage
estimation is acceptable.
b. The AI/DPA Team determines if rework is necessary.
O. If rework is not necessary, the AI/DPA Team confers on the
results and forwards the data to the Accuracy Assessment
Team.
d. If rework is necessary, the AI/DPA Team confers on the rework
technique.
e. The segment is reworked by an AI/DPA Team, using the follow-
ing rework products:
•	 All classification maps
All evaluation forms
- All ancillary data
Color film for all classified acquisitions
e	 Batch printouts
The results are discussed and forwarded to the Accuracy
Assessment Team.
f. As a ground rule, selected acquisitions are not discu,ssad
between the teams.
g. Team arrangement:
o	 One AI/DPA interpretation is selected randomly from AI/DPA
Teams A, B, C, or D.
r	 Four AI/DPA Teams are selected randomly from all experiencedi
AI/DPA Teams.`
i
`
a
2
t	 1	 ^	 ^	 I
I
3. DATA ANALYSIS
Since each AI/DPA Team was provided with CAMS classification
results from regular processing of each segment, the CAMS rework
proportion estimates are expected to be correlated with the
initial estimates.
To study classification errors of the CAMS rework procedure and
to determine whether or not proportion estimates, are improved, a
linear regression was performed using the CAMS rework estimate as
the dependent variable and the CAMS regular estimate as an inde-
pendent variable. The arc sin Ap-
 transformation, where p denotes
proportion, was employed to stabilize the variance and thus
enable valid statistical inferences to be made. Details on
terminology and results are presented in section 3.2. The
regression is significant, and 44 percent of the total errors
in the CAMS rework estimates are caused by errors in the CAMS
regular estimates.
Elimination of the effect of CAMS regular outputs on the CAMS'
rework results was considered, and residuals were obtained. An
analysis of variance (ANOVA) was performed on the residuals in
order to study whether or not the CAMS rework performance varied
according to AI/DPA Team and/or site. These results are given in
section 3.3. No significant difference is detected between the
four AI/DPA Teams. However, a difference is observed between
sites but is significant only at the 10-percent level. Since
this ANOVA is for a 4-by-9 factorial design with no replications,
the interaction between sites and teams cannot be evaluated and
hence cannot be tested for its statistical significance.
3
I
m3.1 ANALYSIS OF ERROR SOURCES
An analysis of error sources by segment is set out below.
• Segment 1687, Hand County (1), South Dakota — Only biophases 1
and 3 were available for rework. Biophase 3 was acquired in
late July, which, according to the crop calendar, was the time
for harvest and should probably be called biophase 4. Some
grass in this biophase has the signature of ripening wheat.
Therefore, labeling errors occurred, which resulted in
overestimation.
• Segment 1960, Finney County, Kansas — Some of the field numbers
used in the fall-planted wheat inventory report could not be
located on the updated field overlay. This indicates that some
wheatfield boundaries were changed later in the growing season.
Thus, the original small-grain proportions, which were computed
on the basis of the fall inventory report, were inadequate. As
a result, the ground-truth small-grain proportions presented in
table I may be in error. The results for this segment can be
interpreted only with reservations.
Some of the wheatfields in this segment were in very poor con-
dition and did not have a wheat signature. For this reason,
they were not recognized by the interpreters as wheat and were
classified as nonwheat A thres'lold of 8 6 ercent occurredp
in the ground-truth area. According to the ground-truth report,
a major portion of the thresholded area is actually wheat,
which indicates a wheat signature was missed.
• Segment 1962, Saline County, Kansas — The ground -truth small
grain proportions were computed based on the fall-planted wheat
inventory report. No final inventory was available for this
segment.
AI/DPA Teams C and D missed some wheat signatures, which caused
the underestimation. Both AI/DPA Teams A and B had estimates
which were slightly higher than the ground truth; a very thorough
4
1comparison of the classification map with the ground truth
indicates that both are good classifications.
• Segment 1963, Rice County, Kansas - This segment contains
narrow strip fields, which are less than one pixel in width
and for which no training fields were provided. Most of the
pixels in these fields were classified as mixed. It was also
found that most of these mixed pixels were actually wheat but
misclassified as nonwheat, resulting in a reduction of at
least 6 percent in wheat from the estimated small-grain
proportions.
In addition to the problems caused by the narrow strip fields,
labeling errors were made by AI/DFA Teams C and D.
• Segment 1965, Burke County, North Dakota - Every team under-
estimated the small-grain proportions in this segment. Some
wheat signatures were mislabeled as nonwheat. only biophases l
and 2 were available for rework. Use of a later biophase acqui-
sition would made it possible to identify these signatures
correctly.
• Segment 1969, Toole County, Montana - The classification results
were generally good. AI/DPA Team C mislabeled one nonwheat
1
signature, which probably caused overestimation.
• Segment 1970, Liberty County, Montana - As was the case with
s
segment 1963, problems occurred because of narrow strip fields,
resulting in the misclassification of mixed pixels.
l
• Segment 1978, Randall County, Texas - Small-grain proportions
were originally underestimated at least 25 percent by the AI/DPA
w
Teams. The major problem was volunteer wheat which was counted
-3
as small grain. These fields of volunteer wheat were plowed
when the data were acquired. The ground-truth proportion was
3
corrected to exclude these fields from the small grain. The
ground-truth proportion is therefore reduced 13 percent.
	 j
Missing wheat signatures were also an error source. Signatures
	 {
were mislabeled because of the low ground cover (60 to 79 percent)
5
i	 • Segment 1979, Deaf Smith County, Texas — All the volunteer
wheatfields in this segment were plowed when data were acquired.
After the ground-truth proportions were adjusted to exclude
the volunteer wheat, they were almost identical to the classi-
cation result.
• Segment 1980, Oldham County, Texas — All the volunteer wheat-
fields were listed in the crop inventory rej^Drt as summer fallow.
a
Original ground-truth proportions were adequately computed.
• Segment 1986, Hand County (2), South Dakota - Since clouds and
haze were observed in biophases 2 and 3, only biophase 1 was
considered workable. Because of the poor data, this segment
should not have been worked or reworked.
3.2 REGRESSION ANALYSIS
The linear regression equation is
y = 0.02833 + 0.9928x
where
y = arc sin CAMS rework estimated proportion
- arc sin true proportionj
I	
and
x = arc sin CAMS initial est mated proportion
arc sin true proportion
The ANOVA for regression is shown in the following table.
d
3
Mean ..
Degrees of	 Sum of	 square
Source	 freedom	 squares	 error
	 F-ratio
Regression	 1	 0.278	 0.278	 a26.5
Residual	 34	 .357
	
.010
aSignificant at 1-percent level of significance.
6 i
C
Thus, R2 = 0 .^ 278 = 0.44
3.3 ANALYSIS OF VARIANCE
The ANOVA in this section is performed to study the site effect
and the AI/DPA effect on the classification proportion errors.
r
Degrees of	 Sum of	 square
Source	 freedom	 squares	 error	 F-ratio
Site	 8	 0.12972	 0.01622	 al.954
AI/DPA	 3	 .02817	 .00939	 1.131
Residual	 24	 .19908	 .0083
Total	 35	 .35698
aSignificant at 10-percent level of significance.
4.	 CONCLUSION
The following conclusions are made with respect to,the ITS's
selected for the present CAMS rework experiment.
CAMS rework performance is significantly correlated with CAMS
€
initial performance in this test procedure. 	 However, only
44 percent of the error variability in CAMS rework results is
caused by the CAMS initial errors.	 This implies that CAMS
rework performance is as variable as the CAMS initial classi-
fication performance.
•	 No significant difference exists between performances of the
two procedures, which implies that the CAMS rework procedures
T
.: offer no significant improvement.
•	 No significant difference is apparent in terms of different
E AI/DPA Teams, which implies that the CAMS is repeatable for
reviewing and reworking a segment.
7
l• The effect of sites on the CAMS rework performance is statis-
tically significant but only at the 10-percent level.
w ^
i
=1
3
8
1TABLE I.- CAMS REWORKED SMALL-GRAIN PROPORTIONS,
[^ = Average of team proportions minus ground-truth
I
AI/DPA Team GroundSegment Biophases truthA B C D
1687 1,3 35.2 38.8 33.0 (a) 5.9 29.8
1960 1,3 b19.7 b19.7 b19.7 b19.7 -11.7 c31.4
1962 1,2 67.3 70.0 d42.9 427.7 -9.8 c61.8
1963 1,2 31.7 29.5 d28.1 d16.2 -15.0 e41.4
1965 1,2 432.5 d32.5 d30.6 d32.5 -20.3 052.3
1969 1,2,3,4 48.3 45.4 60.3 33.1 5.2 42.8
1970 1,2,4 32.0 58.4 62.7 58.0 11.3 e41.5
1978 1,3 d25.6 d25.6 d25.6 d32.6 -14.6 f41.9
1979 1,3 30.9 30.9 30.9 30.9 -.2 f31.1
1980 1,3 56.9 54.3 58.4 66.5 -1.3 60.3
1986 1,92,93 19.6 24.4 (a) (a) 15.2 6.8
Overall -3.7
aAI/DPA Team declared segment unsuitable for classification.
bMissing signatures.
cCould be in error; no complete ground truth was available for
verification.
dMislabeling.
eContains narrow strip fields.
fWas recomputed to exclude the volunteer wheat.
gExtremely hazy data.
a
C
a(
E
zI'I	
9	 -
TABLE II.— ACQUISITIONS FOR CAMS REWORK EXPERIMENT
x.
Segment Acquisition number for biophase
2 3 4
1687 74133 75205
1960 74291 75150
1962 74324 75131
1963 74289 75131
1965 75155 75191
a1967
1969 75161 75179 75215 75233
1970 75142 75179 75233
1978 74291 75133
1979 74291 75133
1980 74291 75133
b1986
allot suitable for processing because of lack of ground
truth.
bNot suitable for processing because of haze and clouds.
r
I
r
TABLE III.- ERROR SUMMARY FOR ITS REWORK
[p - p = error, where p = estimated percentage of wheat and
p = ground-truth percentage of wheat]
AI/DPA TeamSegment
A B C D Oa
1960 -11.7 -11.7 -11.7 -11.7 -11.7
1962. 5.5 8.2 -19.9 -34.1 -19.9
1963 -9.7 -11.9 -13.3 -25.2 -13.3
1965 -19.8 -19.8 -21.7 -19.8 -19.8
1969 5.5 2.6 17.5 -4.7 5.5
1970 -3.5 16.9 21.2 16.5 -3.5
1978 -16.3 -16.3 -16.3 -9.3 -16.3
1979 -.2 -.2 -.2 -.2 -.2
1980 -3.4 -6.0 -1.9 6.2 -3.4
Team average	 b-6.12 -9.18
Team standard deviation	 b12.99 9.11
aThis column lists the original results from Team O,
which classified the segment during the Four-AI Experi-
ment. The AI was randomly chosen from the four AI's
and turned out to be AI Y. These results were from the
latest results analyzed during the year by AI Y and were
used as the starting point for the CAMS rework analysis.
bFor AI/DPA Teams A through D.
9
1
l
3
i
t'	 ti
TABLE IV. — COMPARISON OF CAMS REGULAR VERSUS ITS REWORK RESULTS
I = Improved result
W = Worse than original
N = Original accepted
U = Segment declared unworkable
segment €	 AI/DPA Team
A B C D
1687 I W I U
1960 N N N N
1962 I I N W
1963 I I N W
1965 N N W N
1969 N I W I
1970 N W W W
1978 N N N I
1979 N N N N
1980 N W I W
1986 I I U U
Totals	 12 I's	 3 U's	 10 W's	 19 N's


Lacie phase 1 Classification and Mensuration Subsystem (CAMS) rework experiment

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server