Lai, Vincent

Mahapatra, Radha

English

AIS Electronic Library (AISeL)

Association for Information SystemsAIS Electronic Library (AISeL)ICIS 1999 Proceedings International Conference on Information Systems(ICIS)December 1999Evaluation of Intranet-based End-user TrainingRadha MahapatraUniversity of Texas at ArlingtonVincent LaiChinese University of Hong KongFollow this and additional works at: http://aisel.aisnet.org/icis1999This material is brought to you by the International Conference on Information Systems (ICIS) at AIS Electronic Library (AISeL). It has been acceptedfor inclusion in ICIS 1999 Proceedings by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contactelibrary@aisnet.org.Recommended CitationMahapatra, Radha and Lai, Vincent, "Evaluation of Intranet-based End-user Training" (1999). ICIS 1999 Proceedings. 58.http://aisel.aisnet.org/icis1999/58524EVALUATION OF INTRANET-BASEDEND-USER TRAININGRadha K. MahapatraUniversity of Texas at ArlingtonU.S.A.Vincent S. LaiChinese University of Hong KongHong KongAbstractEnd-user training is a critical factor in the success of information systems.  With rising demand for end-usertraining, organizations are seeking new and innovative approaches for providing low-cost and effective training.Recently, intranet technology has received wide acclaim in the business press as a powerful and cost-effectivemethod for providing anytime-anyplace training.  Most of these claims, however, are anecdotal in nature.  Wehave used a case study research method to evaluate intranet-based training in an organization.  This study relieson extensive collection and analysis of both qualitative and quantitative data to study two research  issues:(1) the role of intranet technology in all phases of the training process and (2) effectiveness of intranet-basedtraining.  This research expands our understanding of a new and innovative strategy for training end-users.  Theresults of this study will provide a foundation for conducting further research on the role and effectiveness ofintranets in providing end-user training.  It will also provide guidelines to IS and training managers inevaluating the appropriateness of intranet technology for training end-users.1. INTRODUCTIONSeveral research studies have demonstrated the critical role of end-user training in the success of information systems (IS) (Nelsonand Cheney 1987).  The demand for end-user training is expected to grow rapidly as business organizations increasingly rely oninformation technologies (IT) to compete in an information age.  A recent survey of HRD executives by the American Societyfor Training and Development (ASTD 1998) identified IT training as one of the fastest growing segments of the technical trainingfield.  Organizations spent between 5% and 25% of their total training and development budget on IT training and expected thisshare to increase in future years.  The tremendous pace of change in IT was reported to pose the biggest challenge in providingquality and timely training.  Measuring the effectiveness of IT training was another challenge faced by organizations.  Identifyinglow cost and effective training strategies that can keep up with technological changes is a major concern of training executives.Ease of use, cross platform compatibility, organization-wide access, and the ability to handle multimedia data are some of thefeatures that make the intranet a powerful and popular technology for information dissemination and communication within theorganization (Bernard 1996).  Recently it has caught the attention of innovative corporate trainers as a technology for providingtraining (Violino 1998).  While intranet-based training has received wide acclaim in the business press as a powerful and cost-effective method for providing anytime-anyplace training, most of these reports are anecdotal in nature.  Very little research datais available to systematically evaluate the use of intranet technology in providing effective training.  The goal of this research isto fill this gap in the evaluation of intranet-based training.  This is a critical research issue considering the growing importanceof providing low cost and effective end-user training.  We have specifically addressed the following research issues in this study:Evaluation of Intranet-based End-user Training525• The role of intranets in end-user training.• Effectiveness of intranet-based end-user training.The remainder of this paper is organized into four sections.  Section 2 reviews the research on end-user training to put the currentstudy in perspective and to provide a theoretical foundation for this study.  Section 3 elaborates on the research issues addressedin this research.  The research methodology is described in section 4.  Section 5 concludes by reporting the current status and theplan for completing this research.2. REVIEW OF END-USER TRAINING RESEARCHComparing alternative training strategies to identify an effective training method is a major research theme in the IS discipline.While lecture-based training was found to be superior to self-study in enhancing learning (Davis and Davis 1990), no significantdifference was found between instruction-based and exploration-based training strategies (Davis and Bostrom 1993).  Behaviormodeling, which combined lectures with hands-on experience, resulted in superior knowledge retention and transfer of learningcompared to other training methods (Compeau and Higgins 1995; Simon et al. 1996).  Studies investigating how people learn andits impact on training design found that learning style of a trainee affected his/her response to a training program (Bostrom,Olfman and Sein 1990).  Trainees who developed a conceptual mental model performed better than those who formed a proceduralmental model (Santhanam and Sein 1994).  Learning together using a codiscovery strategy helped trainees form better mentalmodel of the system and resulted in improved understanding (Lim, Ward and Benbasat 1997).  A study comparing theeffectiveness of concept-based training material with that of procedure-based material found no significant difference (Olfmanand Mandviwalla 1994).  These studies demonstrate that there is no one superior training method for all situations.  The trainermust select an appropriate strategy based on the training context (Leidner and Jarvenpaa 1995).Research studies have found a positive relationship between end-user training and information system acceptance.  Trainingenhances the user’s ability to use an IS and leads to higher system acceptance (Nelson and Cheney 1987).  Identification of costeffective training strategies (Fitzgerald and Cater-Steel 1995) and analysis of training needs to enhance training effectiveness(Nelson, Whitener and Philcox 1995) are some of the other important research issues investigated by end-user training researchers.This study complements the end-user training research literature by systematically evaluating the role of an emerging informationtechnology, namely intranets, in training end-users in an organization.  We have taken a comprehensive view of training bystudying the role of intranets in various phases of end-user training including design, implementation, delivery, evaluation, andadministration of training.  We have analyzed the experiences of a wide cross section of end-users, from several functional areasand belonging to several levels in the organizational hierarchy, in learning to use Enterprise Resource Planning (ERP) Systemsoftware.  Thus, this study provides insights into intranet-based end-user training for a complex software system in a real-lifesituation.3. RESEARCH ISSUESThe following research issues have been addressed:• The role of intranets in various phases of training.• Evaluation of the effectiveness of intranet-based training.Training is a complex process involving several tasks that can be grouped into a set of distinct but interdependent phases (Mayoand DuBois 1987; Olfman and Mandviwalla 1994).  These are requirement analysis, design, implementation, delivery, andevaluation (Figure 1).  The trainee population is identified and their training needs are determined during the analysis phase.Training goals are developed in this phase.  The training plan is created in the design phase.  Training strategy and course contentsare determined and test items are developed to meet the training goals.  The implementation phase involves implementing thetraining and test materials using a suitable medium, and training the trainers, wherever appropriate.  Several alternative mediaare available for training implementation.  Traditionally, paper has been used for delivering course and test material.  Lately,electronic media, such as CD-ROM and intranet servers are gaining popularity because of several advantages offered by thesemedia.  The trainees are trained in the delivery phase.  Finally, the effectiveness of the training program is evaluated.  Thefeedback loop (shown using dashed arrows in Figure 1) indicates interchange of information between different phases.Mahapatra and Lai526AnalysisEvaluation Delivery ImplementationDesignFeedbackFigure 1.  Training PhasesIntranet technology has a number of features thatare useful in most phases of end-user training.Requirements analysis, which involves identifyingtraining requirements, is least likely to directlybenefit from the use of intranet technology.  Theversatility of intranets in implementing a widerange of training strategies makes it easy to adaptthe training design to meet the trainee’srequirements.  Based on the locus of control,training strategies may vary from trainer-centric inone extreme to trainee-centric on the other.  Thetrainer controls the pace of learning in a trainer-centric strategy.  Traditional lecture-based trainingis an example of this strategy.  It is useful when thetrainee has very little or no familiarity with thetopic.  Computer aided instruction, an example ofa trainee-centric strategy, offers the trainee totalcontrol over the pace of learning.  An intranet-based training program can be implemented using either of these methods as well as variations of these, such as behavior modeling(Simon et al. 1996).  Selecting an appropriate training strategy for a given task can enhance training effectiveness.The rapid changes in IT make content modification and enhancement a continuous process in the implementation phase to keepthe training material up-to-date.  This task is easily done when the training material is stored in an intranet web-server, ascompared to a paper-based medium or CD-ROM.  An intranet-based training program can also supplement its own trainingmaterial by providing access to relevant information on the WWW, a rich source of information on IT. The global reach of intranets enables the training program to be made available to employees in remote locations of a company.This results in substantial savings related to travel time and cost.  This facility of an intranet can be exploited to create acontinuous on-demand training program in an organization. Evaluation is an important phase in training as it provides feedback with regard to the effectiveness of the program.  Intranet-basedtools are available to evaluate the reaction of the trainee to the program and to test his/her learning. The second research issue deals with measuring the effectiveness of intranet-based training.  We will use the widely acceptedmodel developed by  Kirkpatrick (1998) to evaluate training at four levels.  The first level measures the reaction of the traineeto the training program.  The second level involves evaluating the effectiveness of the program in enhancing the knowledge andskill of the trainee.  This is usually measured by administering a test to the trainee at the end of the program.  A training programshould result in improved performance of the trainee in his/her job.  Measuring the transfer of learning to job constitutes the thirdlevel of evaluation.  Finally, the impact of the training program on achieving organizational goals and objectives is measured inthe fourth level of training evaluation.  In addition to measuring training effectiveness at these four levels, we also plan to assessthe cost of implementing an intranet-based training program.  This will provide useful guidelines in evaluating the cost-effectiveness of such training.4. RESEARCH METHODWe have used a case study research method in this research (Yin 1984). This method allows the researcher to study variousaspects of a phenomenon in its natural setting and enables extensive data collection from different sources.  This was consideredappropriate because of the novelty of intranet-based training and the exploratory nature of our research.  We opted for a single-sitedesign in order to be able to provide a deeper level of analysis (Nelson, Whitener and Philcox 1995).  The data collection planinvolved using multiple methods to collect both qualitative and quantitative data from various sources.Focused interviews were conducted with the IT training group, including training designers, content developers, trainers, thetechnology support team, and the training manager to understand the role of the intranet in IT training.  Reports, memos, and otherEvaluation of Intranet-based End-user Training527documentation about the training development and implementation processes were analyzed to cross-reference the data gatheredfrom these interviews.  Trainees’ reactions to the training program, learning, and transfer of skill were assessed throughquestionnaire surveys.  Selected trainees were interviewed to evaluate the effectiveness of the training program.  Interviews withdepartment managers and supervisors provided further evidence about transfer of skill of the trainees.  The impact of the trainingprogram on organizational goals was assessed through structured interviews with department managers.  This data collection planallowed verification of the findings through triangulation.5. CURRENT STATUS AND PLAN FOR COMPLETIONThe case study site is a leading manufacturer and distributor of telecommunications equipment with worldwide operations.  Itshead office is located in a southern state.  It employed about 1,200 employees and had a sales turnover of approximately $300million in 1998.  This company had decided to replace its legacy information systems by an ERP system from BAAN.  Thistransition required extensive training for almost all employees in the organization on the ERP system.  The transition teamconsidered various alternative training strategies and elected to use in-house intranet-based training because of its flexibility andcost effectiveness.  A cross-functional training team was set up to design and deliver training programs for the end-users.  A totalof 1,000 employees were trained on using BAAN and related software prior to the roll out of the ERP system.  Currently we arein the final phase of data collection.  The project will be concluded before the conference and the results will be presented at themeeting. 6. REFERENCESASTD.  “National HRD Executive Survey – Information Technology Training,” American Society for Training and Development1998 (http://www.astd.org/virtual_community/research/nhrd_executive_survey_98it.html).Bernard, R.  The Corporate Intranet, New York:  John Wiley, 1996.Bostrom, R. P.; Olfman, L.; and Sein, M. K.  “The Importance of Learning Style in End-User Training,” MIS Quarterly, March1990, pp. 101-119.Compeau, D. R., and Higgins, C. A.  “Application of Social Cognitive Theory to Training for Computer Skills,” InformationSystems Research (6:2), 1995, pp. 118-143.Davis, D. L., and Davis, D. F.  “The Effect of Training Techniques and Personal Characteristics on Training End Users ofInformation Systems,” Journal of Management Information Systems (7:2), 1990, pp. 93-110.Davis, S. A., and Bostrom, R. P.  “Training End Users:  An Experimental Investigation of the Roles of the Computer Interfaceand Training Methods,” MIS Quarterly, March 1993, pp. 61-85.Fitzgerald, E. P., and Cater-Steel, A.  “Champagne Training on a Beer Budget,” Communications of the ACM (38:7), 1995, pp.49-60.Kirkpatrick, D. L.  Evaluating Training Programs, San Francisco:  Berrett-Koehler Publishers, 1998.Leidner, D. E., and Jarvenpaa, S. L.  “The Use of Information Technology to Enhance Management School Education: ATheoretical View,” MIS Quarterly, September 1995, pp. 265-291.Lim, K. H.; Ward, L. M.; and Benbasat, I.  “An Empirical Study of Computer system Learning:  Comparison of Codiscovery andSelf-Discovery Methods,” Information Systems Research (8:3), 1997, pp. 254-272.Mayo, G. D., and DuBois, P. H.  The Complete Book of Training, San Diego:  University Associates Inc., 1987.Nelson, R. R., and Cheney, P. H.  “Training End Users:  An Exploratory Study,” MIS Quarterly, December 1987, pp. 547-559.Nelson, R. R.; Whitener, E. M.; and Philcox, H. H.  “The Assessment of End-User Training Needs,” Communications of the ACM(38:7), 1995, pp. 27-39.Olfman, L., and Mandviwalla, M.  “Conceptual Versus Procedural Software Training for Graphical User Interfaces: ALongitudinal Field Experiment,” MIS Quarterly, December 1994, pp. 405-426.Santhanam, R., and Sein, M. K.  “Improving End-user Proficiency: Effects of Conceptual Training and Nature of Interaction,”Information Systems Research (5:4), 1994, pp. 378-399.Simon, S. J.; Grover, V.; Teng, J. T. C.; and Whitcomb, K.  “The Relationship of Information Systems Training Methods andCognitive Ability to End-user Satisfaction, Comprehension, and Skill Transfer:  A Longitudinal Field Study,” InformationSystems Research (7:4), 1996, pp. 466-490.Violino, B.  “Web Training Catches On,” Informationweek, July 13, 1998, pp. 111-112.Yin, R. K.  Case Study Research, Newbury Park, CA:  Sage Publications, 1984.

Evaluation of Intranet-based End-user Training

https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1137&amp;context=icis1999

Evaluation of Intranet-based End-user Training

Abstract

Similar works

Full text

Available Versions

AIS Electronic Library (AISeL)