The meaningful evaluation of the NASA Space Grant Consortium and Fellowship Programs must overcome unusual difficulties: (1) the program, in its infancy, is undergoing dynamic change; (2) the several state consortia and universities have widely divergent parochial goals that defy a uniform evaluative process; and (3) the pilot-sized consortium programs require that the evaluative process be economical in human costs less the process of evaluation comprise the effectiveness of the programs they are meant to assess. This paper represents an attempt to assess the context in which evaluation is to be conducted, the goals and limitations inherent to the evaluation, and to recommend appropriate guidelines for evaluation

Eisenberg, Martin A.

English

NASA Technical Reports Server

N93-23141
Workshop 1
Evaluation of NASA Space Grant Consortia Programs
Martin A. Eisenberg
Program Director, Florida Space Grant Consortium
University of Florida
Gainesville, Florida 32611
First National Space Grant Conference
Columbia, Maryland
January 16-19, 1990
9
https://ntrs.nasa.gov/search.jsp?R=19930013952 2020-03-17T07:01:20+00:00Z
Evaluation of NASA Space Grant Consortia Programs
Abstract
The meaningful evaluation of the NASA Space Grant Consortium
and Fellowship Programs must overcome unusual difficulties:
a) the program, in its infancy, is undergoing dynamic change;
b) the several state consortia and universities have widely
divergent parochial goals that defy a uniform evaluative
process; and c) the pilot-sized consortium programs require
that the evaluative process be economical in human costs lest
the process of evaluation comprise the effectiveness of the
programs they are meant to assess. This paper represents an
attempt to assess the context in which evaluation is to be
conducted, the goals and limitations inherent to the
evaluation, and to recommend appropriate guidelines for
evaluation.
Introduction
The NASA Space Grant Program inaugurated in September 1989
was designed to catalyze the development of ideas, programs,
and a broad-based institutional commitment and infrastructure
that will, in the long run, satisfy the following explicitly
or implicitly stated objectives:
• To arouse the interest of a generation of K-12 students
in mathematics and science, to improve their levels of
competency in such subjects, and to stimulate their
collective interests in, preparation for, and dedication
to careers in diverse technologically-based disciplines.
• To arouse the interest of the general public in
aerospace-related activities of NASA and other
governmental and private agencies, to get John and Mary
Q. Public to stop yawning at the day-to-day successes of
NASA et al., To develop a public appreciation for the
scientific and technological challenges of aerospace
science and technology, to develop and understanding of
the scientific and technological benefits to accrue from
a vigorous program of aerospace-related research and
development, to develop a public understanding of the
economic benefit of such programs to the nation, indeed,
to convince the public that such programs are imperative
to our economic health and national security•
To engender broad-based pubic support and the associated
political constituency necessary for budgetary
commitments essential to realize these objectives•
• To co-opt increasing devotion of resources from State,
Federal, and private agencies toward aerospace-related
research and development and human resource development.
• To assure a stream of well qualified and motivated
technologically educated students being graduated at the
BS, MS, and PhD levels, adequate to meet the needs of
NASA, DOD, and our aerospace industries, and thereby
preserve and enhance our technological competitiveness,
balance of payments, national economy, and national
security.
• Affirmative action goals to enhance the opportunities for
affected minorities and women are an independent
objective and inherent to and a necessary condition to
the meeting---_f the above stated five goals. Given the
demographics of the work force projected for the coming
decades, even the most mean-spirited, socially
retrograde, morally perverse, but intelligent individual,
would adopt as a Machiavellian strategy, a strong
pro-affirmative action bias.
The above stated long-term objectives of the Space Grant
Program define the context in which one can attempt
evaluation of the National program and several State
Consortia. The resources currently allocated to the task are
woefully inadequate to fulfill the above goals but they can
encourage the development of a cadre of committed people and
institutions, and the establishment of effective means of
communications among them.
Goals of the Evaluation Process
All that can be asked of the current programs at the current
levels of funding commitment is the demonstration of
promising approaches, and the identification of pitfalls, and
promising looking but blind alleys, so that, when (not if)
Congress, NASA, the States and private industry develop the
resolve to provide the levels of investment necessary to
attack problems that must be attacked we will do it with
greater wisdom and efficiency•
Thus, the primary purpose of the evaluation process must be
to set the stage for a cost-effective scaleup of the
operations of the Space Grant Program• Since significant
institutional and individual stakes will be riding on these
evaluative assessments they will be necessarily biased.
ii
Limitations to the Evaluation Process
Don Griffin, formerly of Westinghouse's Bettis Atomic Power
Labs articulated what I will call, Griffin's Law:
"Under the best of circumstances, the product of
objectivity and expertise in any one observer is a
constant."
This "law", somewhat reminiscent of Heisenberg's Uncertainty
Principle, articulated in the context of evaluation of high
technology programs in a different discipline is applicable
to NASA's goal of evaluation of the Space Grant Program. We
will have to rely to a considerable extent on people with an
"ax to grind" to prepare the evaluations. The best that we
can do is to require that the bases for the evaluations be
clearly articulated, that the underlying data be public, that
the authors of the evaluations be identified, and that those
responsible for reviewing the evaluations do so with clear
understanding of the inherent biases of the authors. I am
sufficiently sanguine with regard to human nature to trust to
the basic intellectual integrity of the evaluators (ensemble
average) not to fabricate the data. On the other hand, there
will be wishful thinking that manana we will see the light at
the end of the tunnel and our programs will be productive.
NASA has already missed (I believe) the opportunity to
perform the evaluations with scientific rigor. To do so they
should have rank ordered all of the Space Grant Proposals and
funded all of the odd-ranked proposals, denying funding to
the even-ranked proposals. One could then compare the
performances of paired States with universities of inherently
comparable qualities and would-be PI's of comparable
imagination and enthusiasm. Such a controlled experiment
would then allow one to isolate the effect of NASA funding on
the outcomes. NASA was probably wise not to conduct such an
experiment. The basic message is that we shall be hard
pressed to measure the extent to which the NASA funding was
the cause of the measurable advances. Those institutions and
individuals represented at this meeting are aggressive,
capable, and dedicated to the Space Grant goals. They would
have found alternative ways to achieve some of the successes
that we shall report.
It should also be noted that the Space Grant Program is only
one of many factors that will affect the realization of the
above stated goals. The overall state of the national and
world economies, the national perception of the relative
severity and importance of social problems, the worldwide
geopolitical trends and Congress' and State legislative
reactions to them, particularly as they may affect funding
for DOD, NASA, and education, can be expected to have major
impacts on the very variables that one would like to evaluate
to assess the NASA Space Grant Program.
12
Conclusions and Recommendations
The Space Grant Program is in it's infancy• We are just
beginning on the learning curve• The resources allocated to
the problems are at the proof-of-concept level. The several
Consortia are starting from diverse positions, have
established diverse initial strategies, have articulated
diverse short-term and long-term goals, dictated by
conditions parochial to their specific situations.
Accordingly, the following criteria for evaluation are
recommended:
• Quantitative Space Grant-wide objective functions should
not be defined to evaluate individual programs.
• It will probably be useful to gather data on standard
quantitative measure of productivity (enrollments,
degrees granted, papers published, patents awarded...)
to report for the NASA Space Grant Program at large.
The data will be of most interest in terms of
year-to-year changes•
• First year results should not be given heavy weight.
The evaluations of the programs should be made over a
longer haul.
• Significant experimentation with and modification of
programs is anticipated in the early years of the
programs• Evidence of internal evaluation and
responsive adaptation of program strategies is to be
encouraged. Wherever possible such evaluation
processes should be designed into the programs to
assure timely feedback. Such internal use of evaluation
should be the primary purpose of Consortium evaluations.
• The consortia should be encouraged to develop
parochially appropriate (that doesn't mean self-serving)
evaluative criteria.
• The evaluative criteria and means of assessment should
be anticipated to be dynamic in the early years of
the program•
• From annual review of the individual criteria and
evaluative processes will evolve a more systematic and
common basis for evaluation as the programs mature•
• NASA should provide early general guidance for the
manner in which evaluation issues are to be treated
in the September annual report•
13
• It iS our understanding that the first step in the
Consortium internal evaluation process is to review the
NASA RFP goals and to restate them in Consortium-
specific terms.
10. During the formative years the primary thrust of the
evaluation process is to assess overall national program
effectiveness.
ii. Longer-term evaluation of the national program and the
consortia should ask the basic questions: Did we
achieve the development of an effective network? Did we
provide meaningful space-related experiences for
students? Did we achieve leverage from the seed
funding? Did we achieve a genuine commitment from our
universities, industry, NASA and other public agencies?
12. We must avoid the development of an overly formalized
and burdensome evaluation process, disproportionate
to the programmatic size and level of effort•
For all the reasons stated herein, ultimate assessment of
evaluations will remain to some extent subjective, requiring
sagacity and judgment, and an ability to look beyond
statistics to form a valid gestalt assessment of program(s)
effectiveness.
14


Evaluation of NASA space grant consortia programs

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server