In this paper a Student Satisfaction survey was conducted. The used questionnaire was proposed by a very large Research Group in 2010. The collected data was elaborated by a full reflective Structural Equation Model using PLS path model estimation. The first results showed that the influence of the Organization and Infrastructure on the Student Satisfaction were not statistically significant. Therefore a more complex model was supposed, the end results showed that the influence of Organization and Infrastructure on the SS was indirect, that is the Organization and the Infrastructures exert an influence upon the SS through the Didactics

D'Ambra L

Sarnacchiaro P

D'AMBRA, LUIGI

Archivio della ricerca - Università degli studi di Napoli Federico II

	  Electronic Journal of Applied Statistical Analysis EJASA (2012), Electron. J. App. Stat. Anal., Vol. 5, Issue 3, 406 – 412 e-ISSN 2070-5948, DOI 10.1285/i20705948v5n3p406 © 2012 Università del Salento –	  http://siba-ese.unile.it/index.php/ejasa/index  406 STUDENTS' EVALUATIONS OF UNIVERSITY TEACHING: A STRUCTURAL EQUATION MODELING ANALYSIS   Pasquale Sarnacchiaro*(1), Luigi D'Ambra(2)  (1)Faculty of Economics , University of Rome Unitelma Sapienza, Italy (2)Department of Biological Sciences, University of Naples Federico II  Received 04 August 2012; Accepted 29 October 2012 Available online 16 November 2012   Abstract: In this paper a study on student satisfaction of university teaching was conducted. The proposed questionnaire was elaborated by a very large Research Group in 2010. The collected data was elaborated by a full reflective Structural Equation Model using PLS path model estimation. The first results showed that the influence of the Organization and Infrastructures on the Student Satisfaction were not statistically significant. Therefore a more complex model was supposed, the final results showed that the influence of Organization and Infrastructures on the SS was indirect, that is the Organization and the Infrastructures exert an influence upon the SS through the Didactics.  Keywords: Structural equation model, reflective measurement model, partial least  squares, tetrad test, student satisfaction   1. Introduction  The practice of student’s evaluations of university teaching, through teaching evaluation questionnaires, is now widespread [12] [1]. In Italy, all the Universities carry out surveys to measure Student Satisfaction (SS). Most of these surveys are conducted through the administration of a evaluation questionnaire. In 2008 the CNVSU (Comitato Nazionale per la Valutazione del Sistema Universitario - National Committee for University System Evaluation) charged to a very large Research Group (RG) to build and validate a questionnaire for the assessment of teaching to be administered through the web [4]. This questionnaire has been used for a SS assessment at the University Federico II of Naples and the aim of this paper is to propose a full reflective Structural Equation Model (SEM) to analyze the collected data. In the second section, you can see the main results of the research, conducted by the RG. In the third 	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  *	  E-mail: pasquale.sarnacchiaro@unitelma.it Sarnacchiaro, P., D’Ambra, L. (2012). Electron. J. App. Stat. Anal., Vol. 5, Issue 3, 406 – 412. 407 section, it is showed both the SS study conducted at the University Federico II of Naples and the SEM used to analyse the SS and to detect the drivers of their behaviour. In the fourth section, the results of the SEM are pointed out, a brief conclusion ends the paragraph.    2. The construction of the Questionnaire  This paragraph summarizes the main results of the project carried out in 2010, supported by CNVSU and developed by RG with the aim to design, build and evaluate a questionnaire for the student evaluation of university teaching in Italy [5]. The RG, after a survey regarding the questionnaires used in Italy and abroad for the SS assessment, has compared four different questionnaires by a survey. The compared questionnaires were: the Standard Questionnaire (SQ) proposed by CNVSU and composed by 15 questions with items rated on a 4-point scale, a revisited version of the SQ composed by 15 questions with items rated on a 10-point scale, the Experimental Questionnaire (EQ) proposed by RG and formed by 9 questions with items rated on two joint 4 and 10-point scales, the EQ revisited with 9 question with items rated on two disjoint 4 and 10-point scales (for information about questionnaires see [4]). The four questionnaires were administered to a sample of about 1500 students of the University of Brescia and of the University of Sannio.  In order to investigate on the importance of the questions and on the more appropriate scale to evaluate the SS, the RG adopted the following strategies. For the first point you performed a factorial analysis, not rotated and rotated with Promax rotation, and a logistic regression analysis. Particularly, to detect the best model, to describe the 4 and 10-point data obtained through the questionnaires, the minimum residual method [6] on the polychoric correlation matrix was used; the software used was Prelis (Version 2.54). The choice of questions to insert in the questionnaire was made considering the results of factor analysis and using the criterion proposed by Tabachnick and Fiddell [14]. In order to verify the stability of parameters or the replicability of pattern/structure coefficients (loadings), a nonparametric bootstrap factor analysis have been performed [5]. Then, to evaluate the importance of questions in the questionnaires in reference to the overall SS, a logistic regression analysis (using the forward selection approach) was used. The Rasch Analysis with the Rating Scale Model [17] was used to assess the properties of all the ordinal 4 and 10-point scales of response used in the questionnaires. The performed scaling analysis has pointed out the greater flexibility offered by the 10-point scale: its votes can be merged in a 4-point scale preserving the comparisons with previous surveys, and more informative statistical analysis could be conducted for the future surveys. For these reasons, the RG proposed a new scale [4]. As a final product of this research, the RG proposed a new version of the questionnaire for the evaluation of university teaching composed by twelve questions (Table 1) with the scale proposed above. This questionnaire has been used for a study conducted on a sample of students of the University of Naples Federico II.     Students' evaluations of university teaching: a structural equation modeling analysis 408 3. Analysis of the student satisfaction by structural equation model   The proposed questionnaire was administrated to a total sample of 511 students in Economics of the University of Naples Federico II, they were selected through simple random sampling. Firstly, in order to identify the main aspects that influence the SS a Factorial Analysis has been elaborated. According to the Kaiser and Guttman rule, four-factor was retained and, considering the association factor-question, they were labelled as follow: Organization, Infrastructures, Didactics and SS. In order to formalize a scheme for the interpretation of the SS and detecting the drivers of their behaviour, a SEM was elaborated, particularly Organization (ξ1), Infrastructures (ξ2) and Didactics (ξ3) were considered exogenous latent variables (LVs), while SS (η1)	  endogenous latent variable (LV). The use of the SEM in SS study is quite widespread [10]. In SEM techniques we distinguish between two families: covariance-based techniques, as represented by linear structural relations (LISREL) [8], and variance-based techniques, of which partial least squares (PLS) path modelling is the most representative [16].  Table 1. The questionnaire - Sections and Items.  In PLS approach, there are less probabilistic hypotheses, data are modelled by a succession of simple or multiple regression and there is no identification problem. In LISREL, the estimation is done by maximum likelihood, based on the hypothesis of multinormality and allows the modelisation of the variance-covariance matrix. However, you can sometimes find identification problems and non-convergence of the algorithm. PLS approach was chosen because it has less stringent assumptions about the distribution of variables and error terms, in addition although PLS algorithm adopts a formative scheme, and whether it may always be used in reflective schemes is now debated, it is currently used with both kind of models. [9] [15]  3.1 Formative versus reflective measurement models In order to decide whether the measurement models were formative or reflective a set of decision rules have been analysed [7]. This decision rules are based on theoretical and empirical considerations [13]. The theoretical considerations are: the nature of the construct, the direction of causality between the indicators and the LV, and the characteristics of the indicators used to measure the LV [11]. In our case regarding the nature of the LVs, as in a reflective model, the four LVS exist independent of the measures (questions), for example the LV Didactics exists  Sections  Organization Infrastructure Didactics Student Satisfaction Questions Clarity of exam modality (X1) Appropriate classroom for exercitations (X4) Supplementary activities are useful (X6) Overall satisfaction (Y1) Teacher available for explanation (X2) Teacher stimulate the interest (X7) Teacher exposes clearly (X8) Teaching timetable respected (X3) Appropriate classroom for lessons (X5) Charged study proportional to the CFU (X9) Interest for the topics (Y2) Appropriate teaching materials (X10) Sarnacchiaro, P., D’Ambra, L. (2012). Electron. J. App. Stat. Anal., Vol. 5, Issue 3, 406 – 412. 409 separately from the linked questions. The second key theoretical consideration is the direction of causality between the LV and the indicators. In our case the causality flows from the LV to the indicators, as in reflective models, in fact a change of the LV causes a change of the indicators, namely if the LV Didactics (that is Professor) changes consequently the teacher capacity to explain clearly changes too. The last one theoretical consideration regards if all the indicators share a common theme and if the inclusion (or exclusion) of one or more indicators from the domain does not materially alter the LV content validity. In our case the indicators are interchangeable, as in reflective measurement model. Simultaneously to the theoretical considerations, there are some empirical considerations, analysed in the next paragraph, that suggest which measurement model might be chosen. An important empirical test is the tetrad test [3]. However the empirical considerations cannot either support or disconfirm theoretical expectations as to the nature of the measurement model, but they are very useful to confirm the goodness of the chosen measurement model. In our case, all the theoretical considerations suggested a full reflective measurement model, that’s why we chose it. SEM model is defined by the following two sets of relationships:  η 	  =	  Βη 	  +	  Γξ 	  +	  ζ ,          (1) Y	  =	  ΛY	  η 	  +	  ε 	  	  	  and	  	  	  X	  =	  ΛX	  ξ 	  +	  δ ,        (2)  called, respectively, the inner and the outer model, where η 	  and	  ξ 	  are two arrays of endogenous (m×1) and exogenous (n×1) LVs, Β 	   and	   Γ 	   are matrices	   (m×m)	   and	   (m×n) of unknown parameters; Y	  (p×1)	  and	  X	  (q×1)	  denote	  the	  vectors	  of	  the	  manifest	  endogenous	  and	  exogenous	  variables; the coefficient matrices ΛY	  (p×m)	  and	  ΛX	  (q×n) measure the relationships between the manifest and latent endogenous and exogenous variables, respectively. The vectors of errors ζ ,	  ε 	  and	  δ 	  are	   the structural and the measurement errors vectors, respectively. In our case we have m=1, n=3, p=2 and q=10. In (2) the relationships among manifest and LVs are formulated according to a so-called reflective measurement model. In model (1) the matrix Β  is assumed to be lower triangular with zero elements on the main diagonal, so the resulting model is the recursive type [2].   4. Results and discussion   The main results of the PLS estimations are in Figure 1. The coefficients for inner and outer models and the t-statistics (in parentheses) are shown. Regarding the inner model only the coefficient Didactics on Student Satisfaction is significant, while in measurement models all the coefficients are statistically significant. In order to improve the model, we tested a new more complex model in which the LV Infrastructures was unified with the LV Organization and the relationship among the LVs were changed (Figure 2). Performing PLS estimation on the new model we pointed out that all the coefficients of the inner and outer models resulted statistically significant (Figure 2). Then, to evaluate the goodness of the SEM a two-step process was performed. Firstly, SEM assessment focused on the measurement models. This evaluation was performed through the empirical consideration introduced above. In a reflective model, the indicators are evoked by the underlying construct and have positive and high intercorrelations. In our case, as in reflective model case, for each measurement model all the manifest variables are Students' evaluations of university teaching: a structural equation modeling analysis 410 strongly correlated. Since reflective indicators have positive intercorrelations, we used to empirically assess the individual and composite reliabilities of the indicators the Cronbach alpha (>0.70), the average variance extracted (>0.45) and internal consistency (0,80). All these measures confirmed the suitableness of the reflective measurement models (Table 2). Moreover to verify the appropriate classification of each variable with the corresponding LV, the cross loadings between variables and LVs have been computed. Also the tetrad test performed for the exogenous LVs confirmed the adequacy of the reflective measurement models. In the second step of the evaluation model, after the check of the goodness of the measurement models, we underlined how the goodness of the fit of the inner model is substantial (R2= 0,50). Considering the path coefficients we observed that the impact of the Didactics on the SS was considerable (0,708). The influence of the Organization on the Didactics was statistically significant (0,504) and, also, the indirect impact of the Organization on the SS is important (0,35). Therefore to improve SS is important to have teachers that expose adequately (0,849), stimulate the interest (0,846) and offer appropriate teaching materials (0,704), but it is, however, strategic to organize better and better the course in order to allow to the teachers to have a good performance. In order to obtain this aim it is very important to plan the teachers’ availability for explanation (0,787) and to announce, at the beginning of the course, the exam modality (0,788).   Figure 1. Structural Equation Model – Coefficients and t-tests (in parentheses).  The paper proposes a structural equation model with a full reflective scheme for student’s evaluations of university teaching. The proposed SEM represents a quick and very powerful multivariate statistical instrument to understand the drivers of SS. This full reflective model, supported by the large sample size of the research and the results of statistical methods, is able to conceptualize SS as an endogenous latent variable.      ! X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 Organization Infrastructure Didactics Student Satisfaction Y1 Y2 0,826 (11,28) 0,830 (14,22) 0,693 (5,146) 0,900 (5,144) 0,920 (6,111) 0,511 (4,103) 0,86 (31,825) 0,858 (23,477) 0,662 (7,716) 0,690 (10,046) 0,104 (1,010) 0,060 (0,846) 0,646 (7,972) 0,932 (13,167) 0,834 (12,433) Sarnacchiaro, P., D’Ambra, L. (2012). Electron. J. App. Stat. Anal., Vol. 5, Issue 3, 406 – 412. 411    Figure 2. Structural Equation Model – Coefficients and t-tests (in parentheses).  Table 2. Content validity and reliability of Measurement model.   AVE Composite Reliability R Square Cronbach’s Alpha Redundancy Didactics 0,530933 0,846284 0,254372 0,770341 0,135397 Organization 0,457496 0,804248   0,722010   Student Satisfaction 0,782052 0,877343 0,501461 0,732763 0,373595   References  [1]. Attanasio, M., Capursi, V. (2011). Statistical Methods for the Evaluation of University Systems. Italy: Physica Verlag Heidelberg. [2]. Boari, G., Cantaluppi, G. (2010). Construction of the balanced scorecard by using structural equation models with latent variables. Electron. J. App. Stat. Anal: Decision Support System and Services Evaluation, 1, 1, 66–78 [3]. Bollen, K.A., Ting, K.F. (2000). A tetrad test for causal indicators. Psychological Methods. 5, 3–22. [4]. D'Ambra, L., Brentari, E., Carpita, M., Scippacercola, S., Amenta, P., Simonetti, B., Lombardo, R., D'Ambra, A., Sarnacchiaro, P. (2010). Progettazione, implementazione e validazione di un questionario per la valutazione della didattica erogata a studenti universitari. URL: http://www.cnvsu.it/_library/downloadfile.asp?id=11775 [5]. D’Ambra, L., Carpita, M. (2012). Design, implementation and validation of a questionnaire for university teaching evaluation. In Studies in Theoretical and Applied Statistics, Eds. Springer. [6]. Harman, H.H. (1960). Modern factor analysis. Chicago: University of Chicago Press.  [7]. Jarvis, C. B., MacKenzie, S. B., Podsakoff, P. M. (2003). A critical review of construct ! X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 Organization Didactics Student Satisfaction Y1 Y2 0,558 (3,56) 0,533 (3,326) 0,786 (12,397) 0,787 (13,915) 0,674 (7,708) 0,530 (4,608) 0,846 (24,284) 0,849 (25,836) 0,664 (7,1) 0,704 (10,74) 0,504 (6,453) 0,708 (10,814) 0,934 (19,884) 0,832 (14,088) Students' evaluations of university teaching: a structural equation modeling analysis 412 indicators and measurement model misspecification in marketing and consumer research. Journal of Consumer Research, 30, 199–218. [8]. Joreskog, K. G. (1970). A general method for analysis of covariance structures. Biometrika, 57, 293-351. [9]. Marcoulides, G. A., Chin, W., Saunders, C. (2009). A Critical Look at Partial Least Squares Modeling. MIS Quarterly, 33, 1, 171-175. [10]. Marsh, H.W., Muthén, B., Asparouhov, T., Lüdtke, O., Robitzsch, A., Morin, A.J.S. Trautwein, U.(2009). Exploratory Structural Equation Modeling, Integrating CFA and EFA: Application to Students' Evaluations of University Teaching. Structural Equation Modeling, 16, 3, 439-476 [11]. Montuori, P. Triassi, M. Sarnacchiaro, P. (2012). The consumption of genetically modified foods in Italian high school students. Food Quality and Preference, 26, 246–251 [12]. Petruzzellis, L., D’Uggento, A., Romanazzi, S. (2006). Student satisfaction and quality of services in Italian universities. Managing Service Quality, 16, 349–364. [13]. Petter, S., Straub, D., Rai, A. (2007). Specifying formative constructs in information systems research. MIS Quartely, 31, 623–656. [14]. Tabachnick, B. G., Fidell, L. S. (2001). Using multivariate statistics (4th Ed.). Needham Heights, MA: Allyn & Bacon.  [15]. Vittadini, G., Minotti, S.C., Fattore, M., Lovaglio, P.G. (2007). On the relationships among latent variables and residuals in PLS path modeling: the formative-reflective scheme. Computational Statistics & Data Analysis, 51, 5828-5846 [16]. Wold, H. (1975). Path Models with Latent Variables: the NIPALS Approach. In Quantitative Sociology: International Perspectives on Mathematical and Statistical Model Building, Eds. H. M. Blalock et al. New York: Academic Press, 307–357. [17]. Wright, B.D., Masters, G.N. (1982). Rating scale analysis. Chicago: MESA.                   This paper is an open access article distributed under the terms and conditions of the Creative Commons Attribuzione - Non commerciale - Non opere derivate 3.0 Italia License. 

STUDENTS' EVALUATIONS OF UNIVERSITY TEACHING: A STRUCTURAL EQUATION MODELING ANALYSIS

Abstract

Similar works

Full text

Available Versions

Archivio della ricerca - Università degli studi di Napoli Federico II