In June 1991, the MITRE Corporation submitted a series of recommendations as part of a Marshall Space Flight Center (MSFC) Management Information System Requirements Study, initiated by the Information Systems Office (ISO). A major recommendation of the study was to develop an Executive Information System (EIS) for MSFC executives. ISO was directed, by center management, to proceed with the development of a Center-Wide Executive Information System. Existing EIS prototypes, developed by the Space Shuttle Projects Office and the Payload Projects Office, were reviewed. These existing MSFC prototypes were considered not to encompass the required functionality needed on a center-wide basis. A follow-up study by MITRE provided top-level system requirements. These were later incorporated into a final requirements specification document by Boeing Computer Support Services

Moynihan, Gary P.

English

NASA Technical Reports Server

1992
NASA/ASEE SUMMER FACULTY FELLOWSHIP PROGRAM
MARSHALL SPACE FLIGHT CENTER
THE UNIVERSITY OF ALABAlVlA
DEVELOPMENT OF VENDOR EVALUATION CRITERIA AND
POST-IMPLEMENTATION CONSIDERATIONS FOR
MSFC CENTER-WIDE EXECUTIVE INFORMATION SYSTEM
Prepared By:
Academic Rank:
Institution and Department:
NASA/MSFC:
Office:
Division:
MSFC Colleague:
Gary P. Moynihan, Ph.D.
Assistant Professor
The University of Alabama,
Department of Industrial Engineering
Information Systems
Systems Engineering and Integration
Alan Fomey
XXXVII
https://ntrs.nasa.gov/search.jsp?R=19930008127 2020-03-17T09:10:25+00:00Z

In June 1991, the MITRE Corporation submitted a series of recommendations as part of a Marshall
Space Flight Center (MSFC) Management Information System Requirements Study, initiated by the
Information Systems Office (ISO). A major recommendation of the study was to initiate the development
of an Executive Information System (EIS) for MSFC executives. (2) ISO was directed, by center
management, to proceed with the development of a Center-Wide Executive Information System. Existing
EIS prototypes, developed by the Space Shuttle Projects Office and the Payload Projects Office, were
reviewed. These existing MSFC prototypes were considered not to encompass the required functionality
needed on a center-wide basis. A follow-up study by MITRE provided top-level system requirements. (4)
These were later incorporated into a final requirements specification document by Boeing Computer
Support Services. (1)
Another MITRE study addressed the issue of whether to develop the Center-Wide EIS solely using
in-house personnel and resources, or the purchase a Commercial Off-The-Shelf (COTS) product. (3). This
second alternative was subsequently recommended and accepted by senior center management. A
Request For Information (P,FI), identifying system requirements, was then published in Commerce
Business Daily. Vendor responses to the RFI were then reviewed. It was decided that the expertise of an
"8A" contractor (designating a small, disadvantaged firm) would be utilized to evaluate the COTS
products under consideration. The appropriate COTS product would then be purchased. The evaluation
criteria, developed during this research, are intended to be used as a guide for software selection by the
8A contractor.
An Executive Information System is a computer system designed to support the informational
needs of very senior executives. It is intended to provide timely, pertinent information to aid in decision-
making, thereby eliminating the need to sift through lengthy reports. The concept of EIS is a relatively
recent phenomenon. It was only in 1982 that Rockart and Treacy introduced the term to describe this
emerging category of information systems. (7) An EIS may be defined as having the following
characteristics:
1. An easy to use and maintainable graphical user interface.
2. Integrated capabilities for data access, security and control.
3. On-request "drill-down" capability to lower levels of detail.
4. Depiction of organizational health indicators.
5. Functionality for decision support, ad-hoc queries and what-if analysis.
6. Sophisticated tools for navigation.
7. Data analysis features.
8. Advanced report generation.
9. Statistical analysis.
10. Access to a variety of external data sources.
EIS may be considered to be a subset of a lmaader group known as Management Information
Systems (MIS). A variety of techniques have been developed to support evaluation of MIS proposals.
Relevant published literature on this subject was reviewed, and various methodologies were investigated
for their applicability. A weighted scoring technique was selected for the overall evaluation technique. It
is considered to be a widely accepted technique by procurement agencies for evaluating "multiple
proposals with varying prices and capabilities. (5) The scoring technique focuses on a listing of desired
functional characteristics. Weights are assigned to each characteristic based on its perceived desirability.
A composite score is then generated. The scoring technique was viewed as a means of fairly prioritizing
MSFC requirements and quantifying the resulting vendor responses. It was also considered to be an
effective means of countering any potential vendor protest.
The evaluation process analyzes the vendor product alternatives from three perspectives: technical,
cost and risk. The overall strategy for the evaluation is to eliminate any vendor alternative that cannot
effectively meet the technical requirements. Cost and risk factors are considered only after all alternatives
have been functionally reviewed. The composite scores for the technical, cost and risk analyses will be
XXXVII-1
factored and combined as identified in Equation 1, for any alternative that is deemed functionally
acceptable:
TOTAL SCORE = TECHNICAL EVALUATION (50%) + COST EVALUATION (30%)
+ RISK EVALUATION (20%) [1]
The vendor alternative with the highest resulting score will then be purchased by the 8A contractor.
The initial phase of the evaluation process utilizes three types of factors that are applied to each
functional line item: requirements qualifier, weighting factor and scoring factor. Technical evaluation
criteria were obtained primarily from the requirements specifications documents. (1, 4) Each of the
technical criteria have been qualified as to whether they are considered to be mandatory or highly
desirable. The scoring factors vary with each designation. Each mandatory line item requires a specific
response flom the vendor. If a mandatory requirement is not bid by the vendor, it could be cause for
disqualification. If a highly desirable requirement is not bid, it is not cause for disqualification, however a
penalty is assessed via the scoring system for the absence of this functional capability. The weighting
factor indicates the relative importance among the requirements. The weighting factors range from 1
(low) to 10 (high). The scoring factor is used to rate each vendor's response. The scoring factors are
assigned relative to the following guidelines:
Requirement Oualifier M H
Have Capability Now +1 + 1
No Bid - 3 -1
The product of the scoring factor and the respective weighting factor for each line item is accumulated
into the overall technical score for the vendor's proposal.
A series of forms were created in order to document the acceptability of the vendor product
functionality, and to thus substantiate the overall technical evaluation. These consisted of a System Test
Execution Log, Test Case Form, and a Problem Identification Form. The System Test Execution Log
provides a summary of the individual test cases. The Test Case Form provides a standard format for
documenting the original evaluation. For each line item requirement of the EIS Technical Evaluation
Work, sheet, a separate test case is conducted. Upon rejection of a test, a problem identification form is
completed. The form provides documentation of system inadequacies.
The cost analysis phase of the evaluation considers not only the price of the individual EIS software
package itself, but of additional supporting hardware, annual ficensing fees and operational transaction
costs. These factors are combined in order to create a total systems cost. The Cost Analysis Summary
Worksheet depicts the highest level of detail. Backup should be provided in order to indicate the pricing
components of each major category.
The final component of the evaluation involves assigning a confidence factor to each
vendor/product. A risk assessment worksheet is used to factor each vendor's response to the RFI. The
higher the rating, the higher the perceived credibility. Each vendor's offering is evaluated based on the
following general criteria: functionality, compatibility (with existing hardware, software and
communications), installation, documentation, total systems cost, and vendor service. Similar to the
technical evaluation, the product of the weighting and scoring factors is used to establish a risk
assessment score.
Draft versions of analysis worksheets, test case forms and all criteria developed as part of this
research, were submitted to personnel from the MSFC Information Systems Office and Boeing Computer
XXXVII-2
Support Services, for review and comment. Subsequent recommendations for modification were
incorporated into the fmal report to the client.
One of the major barriers to the success of Executive Information Systems has been a failure to
consider post-implementation factors regarding these systems. For more traditional information systems
applications (e.g. transactions processing, decision support systems), post-implementation issues have
been recognized based on years of experience and much trial and error. Unfortunately this experience
frequently can not be transferred directly to the EIS environment. "The fragmented nature of executive
work, the high degree of environmental uncertainty at this level in the organization, and the political
ramifications of providing top management with more and better information, as well as other factors,
make implementing (EIS) a special challenge." (6) The subsequent analysis focuses on some critical
factors that should be considered following the implementation of the Center-Wide Executive
Information System at MSFC.
A post-implementation evaluation of the system should be conducted. This phase consists of two
major components: the Development Recap and the Post-Implementation Review. The Development
Recap provides an in-depth review of the EIS development activities that have just been completed.
Analyses of cost and schedule variances are conducted as part of this recap. It should also identify
describe and classify programming errors; suggest any needed revisions to the development methodology
used; and provide any other relevant suggestions or insights. The Post-Implementation Review is
performed four to six months after the system has been installed. The purpose of the review is to evaluate
how well the EIS has performed in meeting the original expectations and projections. It is also intended
to identify any further maintenance projects that should be undertaken to enhance or improve the
implemented EIS.
The value of any information system depends on the quality of its data; its timeliness, accessibility,
accuracy and completeness. This is particularly true of Executive Information Systems. By attempting to
provide high-quality data and information, an EIS may often highlight existing data management
problems, while at the same time creating new ones. Analysis of the existing MSFC data environment
identified a series of issues regarding data availability, ownership, integrity, infrastructure, integration
and management. Detailed recommendations were formulated regarding each of these areas.
The primary emphasis of the early post-implementation phase is to isolate and correct any system
errors as soon as possible. A series of recommended procedures were suggested regarding this issue. The
same overall process is used to address enhancements to the system. These enhancements should be
compiled and evaluated after the primary fixes have been made. Because the executive user is
particularly sensitive to changes in response time, the implications of any enhancement on this must be
carefully considered. It is also important to emphasize that any changes involving the data sources must
be communicated to the EIS maintenance group in order to avoid system malfunction. Current
procedures regarding information system change control, configuration control, and version control were
reviewed and found both adequate and applicable to the EIS.
Consideration should be given to planning the post-implementation migration and evolution of the
system. The EIS is likely to spread to additional users. The migration may be both hierarchical and
lateral. With hierarchical migration, use of the EIS spreads from the top down in an organization. In
lateral migration, use of the EIS moves across organizational units. The system is also likely to evolve to
include information that is broader in scope, more detailed and closer to real time. The most effective
method of EIS migration appears to utilize a strategic approach. The system is progressively made
available to those executives where the need and expected return is the greatest.
Construction of the Center-Wide EIS implies the need for a security system that will control access
to sensitive information (e.g. Privacy Act information, long-range plans). In addition to prohibiting
XXXVII-3
access to the system by unauthorized personnel, the security system must be multi-level in nature, in
order to be able to control access within a variety of classifications. According to the litexatm¢, there is a
wide divergence of strategies in addressing this problem. Aerospace companies tend to use a multi-level
security system utiliTing the capabilities of the EIS application development software. An alternate
approach utilizes a multi-level database to support data having different classifications and users having
different clearances. Current plans for the MSFC EIS employ this last approach which is resident in the
ORACLE source database. Art ORACLE table will be used to identify user access privileges. This table
will act in conjunction with queries associated with programmed SELECT statements on an individual
field level. This planned security mechanism appears to satisfy the applicable NASA security
requirements. Executives will want access to the EIS from remote sites. Security on dial-up access lines
should be considered as a priority post-implementation system enhancement. The security system should
also be as transparent as possible to the executive user. Multiple levels of passwords should be avoided.
Most senior managers prefer "one-button" access to their systems. In the EIS this would be analogous to
the WPS (Workstation Presentation Services) password synchronization feature that is currently used.
In order to support the construction of an effective Executive Information System, this research has
provided a quantitative method for assessing commercial off-the-shelf EIS development software. An
analysis was also conducted which identified important post-implementation considerations. The final
judgment concerning the effectiveness of the center-wide EIS will be determined largely by the executive
user's expectations and their perceptions of the system's ability to meet their particular support
requirements. This highlights the need to obtain agreement from the user community on a clear set of
metrics for determining system effectiveness, prior to actual system implementation.
REFERENCES
1. Boeing Computer Support Services, "Final Requirements Specifications for an Enterprise-wide
Executive Information System Application Development Environment and Decision Support Tool
Set" (prepared for Marshall Space Flight Center) March 18, 1992.
2. MITRE Corp. "Marshall Space Flight Center Management Information System Requirements Study",
June 1991, Contract No. NAS9-18057.
3. MITRE Corp. "MSFC Executive Information System Make Versus Buy Analysis Details", March
1992, Contract No. NAS9-18057.
4. MITRE Corp. "Marshall Space Flight Center Executive Information System Requirements
Specification", April 1992, Contract NAS9-18057.
5. Powers, Michael J., Adams, David R. and MiUs, Harian D., Computer Information Systents
Development: Analysis and Design, Cincinnati: South-Western Publishing Co., 1984.
6. Rockart, John F. and BuUen, Christine V., The Rise of Managfrial Computing. Homewood, IL: Dow
Jones - Irwin, 1986.
7. Rockart, J.F. and Treacy, M.E., "The CEO Goes On-Line", Harvard Business Review, Vol. 60,
No. 1, January - February 1982, pp. 84 - 88.
XXXVII-4


Development of vendor evaluation criteria and post-implementation considerations for MSFC center-wide executive information system

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server