Dill, David

Henkel, Mary

Neave, Guy

Stensaker, Bjorn

Westerheijden, Don F.

Stensaker, Bjørn

Westerheijden, Don

NARCIS 

Third report of the Scientific Council to A3ES: Meeting of 10th of October 2012

University of Twente Research Information

THIRD REPORT OF THE SCIENTIFIC COUNCILTOA3ES.MEETING OF 10TH OF OCTOBER 2012The Scientific Council (SC) hereby submits its third Report to the A3ES – the Agency for the Evaluation and Accreditation of Higher Education in Portugal. As in the past, we appreciate both the invitation to discuss the current  and  future  state  of  affairs  regarding  quality  assurance  in Portugal, as well as the willingness of the Agency and its leadership to engage  frankly  and  openly  in  a  comprehensive  and  wide-ranging exchange  about  how  the  system  in  place  may  be  enhanced  and strengthened. The current Report follows on from its two predecessors.  We start by paying tribute to developments and activities seen as promising and relevant  to  higher  education  in  Portugal.  In  the  second  part  of  the Report  attention  focuses  on  future-oriented  questions.  Here  we  also make some recommendations that we hope may prove beneficial to the Agency in its development of the Nation’s system of higher education. COMMENDATIONSIn  previous  reports,  the  SC  noted  with  great  satisfaction  the commitment of the Agency to designing a national system of quality assurance  that  fulfils  the  expectation  of  both  effectiveness  and efficiency. The Agency’s effectiveness is linked to the double purpose of balancing the control and the enhancement dimensions of its mission, while its efficiency is linked to the ways by which this is accomplished. With a very modest number of administrative staff, the Agency serves the whole higher education system in such a way that accreditation processes are experienced as swift and predictable by the institutions – 1although  both  universities  and  polytechnics  sometimes  may  be disappointed with the outcome. In part, the Agency’s success appears to  rest  on  a  well-functioning  electronic  platform  through  which  the accreditation  processes pass.  However,  an  electronic  platform on its own is not sufficient for handling of key decisions yet to be made.  Here, both the Agency’s staff and leadership should take credit. From what we have gathered during our visits to Portugal, the SC has good cause to note the Agency’s consistency and willingness to abide by the plans and strategies it has set itself. We applaud the capacity of the  Agency  consistently,  steadily  and  incrementally  to  build  up  the national  system  for  quality  assurance  in  Portugal,  while  making allowance both for  experience to be taken into  account  and for  the careful planning of future activities. That the Agency already in 2012 is considering the range of activities it will take up after the completion of the current accreditation cycle in 2016 is impressive, and deserves the appreciation of universities and polytechnics – not least for the degree of predictability it engenders.  Predictability should enable universities and  polytechnics  to  plan  how,  as  responsible  institutions,  they  may shape their response to future initiatives. This, SC sees as a system fed by an on-going dialogue between Agency and institutions, a system to be  grounded  in  and  building  out  from trust  and  mutual  confidence between both parties.A3ES is  one of  the few quality  assurance agencies in Europe,  which deliberately and consistently underpins its activities and procedures by research, analysis and careful experimentation. The latter is evident in the  piloting  of  the  institutional  audits,  carried  out  during  2012.  Our examination of the publications and analyses the Agency undertook or commissioned over the period 2011–2012, left us impressed by both the amount of work in this area, by the Agency’s international presence and outreach through its disseminating these studies to other Agencies and to the international scholarly community. The SC also commends the  development  of  A3ES’s  in-house  research  capacity  next  to  the existing virtually  organic  links  with  other  research institutes  such as Cipes, from the point of view of staff development. In an Agency where 2efficiency  is  at  a  premium,  there  is  always  the  risk  of  overlooking ‘competence building’ for staff as a key feature of future sustainability. That Agency staff, through the option of taking part in the work of the new Office for Studies and Analyses can shift between more analytical and  more  hands-on  work  in  the  processes  of  accreditation  and evaluation.  This,  we feel,  equally is  important for strengthening the Agency’s knowledge capital in the future.We would also like to express our support for the initiatives the Agency has  taken  systematically  to  bring  students  into  quality  assurance procedures.  And  in  particular  A3ES’  practice  of  open  tendering  for student participation. This is highly commendable. In Europe, student participation  is  often  secured  through  student  interest  groups  and organizations. While this approach may secure the student interest in many ways, the potential downside is that student representatives are seen  less  as  independent  “experts  of  learning”,  and  more  as representatives  of  an  interest  group.  The  Agency’s  open  search  for candidates may be a fruitful approach for creating review committees. RECOMMENDATIONSWe have commented upon, and commended, a number of issues that address the Agency’s current operations. In this section, we dwell on some of the issues that the agency is planning for the years ahead. These thoughts and reflections are set out in the hope they may be of service in the deliberations the Agency will initiate. SUPPORT FOR AN AUDIT APPROACHWe have learned that the Agency is seriously considering building up an institutional audit approach as a new evaluation method in the years to come.  The  experiment,  already  underway,  is  intended  to  provide experience and insight about how such a method may be implemented. The  SC  sees  the  audit  method  as  highly  relevant  for  strengthening institutional responsibility for quality assurance in Portugal. We support these initiatives. Yet, we would also wish to underline some issues we 3hold  to  be  of  special  importance  for  developing  a  successful  audit system.Within the audit system currently being piloted, institutions are required to provide a self-evaluation report. The report has a double function: to provide information and input to the process of external audit, and to stimulate reflection inside the institutions under audit. We were given sight of the way one university filled in an on-line form. Self-evaluation, we  feel,  may  easily  become  a  highly  standardized  and  descriptive component  in  the  audit  process.  True,  it  may  meet  the  information needs of external audit committees. But, it also risks failing to stimulate either  institutional  commitment  to,  or  responsibility  for,  the management  of  quality.  Certainly,  there  are  benefits  to  be  had  in standardizing self-evaluations – not least for purposes of comparison. But, there is also the danger that standardization may have a negative effect  on  creativity  and  innovation  within  the  institution.  Besides, completing  an  on-line  form  may  be  perceived  by  higher  education institutions as simply an administrative exercise to inform A3ES, rather than as an authentic and considered self-evaluation report that stands as a summary of an institute-wide reflection on its quality assurance practices.  We  would  therefore  urge  the  Agency  to  reflect  on  what measures might be envisaged to ensure a stronger institutional “buy-in” to  the  phrase  of  self-evaluation  –  especially  since  it  might  serve  to secure  more  institutional  ownership and identity  with the evaluatory process as a whole. As for ways to achieve “buy in” in the UK, the QAA’s “learning  from  audit”  reports  were  held  to  be  a  valuable  way  of spreading knowledge about good practice and clarifying how “quality enhancement”  may  effectively  be  implemented  at  the  institutional level. A similar approach might also be useful for A3ES.The building-up of an audit system, we would suggest, calls for linking the new audit initiative in with existing systems of accrediting study programmes.  This  poses  a  two-fold  issue.  First,  coupling  a  new methodology  to  existing  practices  may  well  be  seen  as  enlarging “bureaucracy”  in  higher  education.  To  counter  this  perception,  the Agency would do well to invest considerable energy in explaining the value added of audit to higher education in general. Second, adding a new method, almost always runs the risk that the logic underlying the 4different methods employed – accreditation and audit – may not align with  one  other.  Nevertheless,  building  consistency  between  the  two methods and approaches is important. Audits and accreditation are two methods that  serve different  purposes.  As  experience across  Europe has shown, methods aimed at control tend to dominate over methods aimed at enhancement, even if the latter formed part of the intent at the  outset.  To  achieve  a  proper  balance  between  the  way  higher education perceives the two methods is no small challenge. We would submit that along with the audit approach, A3ES develop incentives of sufficient attraction that universities and polytechnics cannot fail to see the benefits that come from actively engaging in the process of audit. The ‘lighter touch’ approach is obviously linked to this condition, if it is not dependent on it.SUPPORT FOR A “LIGHTER TOUCH”.One of the approaches A3ES is considering, so the SC has been told, involves  putting in  place  the  technique of  “risk-management”.   This approach has a double purpose: to make evaluation less burdensome for  those  institutions  clearly  committed  to,  and  engaged  in,  the substantive advance of quality assurance; to be more economical with both time and resources at the system level. The SC recognizes these benefits. It recognizes the logic behind the risk-management approach. There are, nevertheless, issues here that merit further consideration. In the first place, the SC takes the view that indiscriminate use of the terminology of “risk-management” may well  undermine the key aims the Agency seeks to advance in the higher education system, not least the development of an audit approach. The Agency’s audit initiative, so it strikes us, has as one of its key objectives to forge both trust in higher education  and  to  underline  the  responsibility  of  universities  and polytechnics  for  actively  upholding  their  own  quality.  “Risk-management”  builds  upon  a  different  logic  –  that  of  distrust  of universities and polytechnics – and of their workings. Whilst, in part, such differences may stem from differences in semantics, it would be wise  not  to  discount  the  possibility  of  the  perverse  effects  such  an approach may have upon the evolving relationship between Agency and institution. 5Central to this latter issue and most particularly so, is which criteria are to be applied to identifying “risk” in higher education. Given the way A3ES  currently  operates  and  the  efficiency  with  which  various  data sources are exploited and used in Portugal’s higher education today, to rely  heavily  on  traditional  performance  indicators  as  potential  “risk signals”  is  not  without  its  temptations  –  or  its  perils.  “Risk management”  is  not  greatly  dissimilar  to  the  approach  used  in accreditation – which, itself is a method clearly used for purposes of control. However, if a policy of “lighter touch” auditing seeks to elevate institutional engagement in quality work as a permanent and on-going responsibility  of  the  individual  university  and  polytechnic,  it  is questionable whether traditional performance indicators exert sufficient leverage  and  persuasion  to bring  about  the  necessary  degree  of reflection, curiosity and creativity to support such a process.  Applying performance  indicators  that  are  more  historically  conceived  or contextually determined runs the risk of setting in place a system of higher  education  where  some  universities  and  polytechnics  would always fall  into  the  category  of  ‘high  risk  institutions’  given  the precariousness of the conditions under which they currently operate. A technique cannot be sundered from the purpose to which it is put.One  possible  approach  alternative  to  establishing  a  “lighter  touch”, might be to rate how far each institution has developed  an  effective educational quality assurance system.  Hong Kong, for example, has developed  ’maturity  ratings’  for  academic  quality  work,  based  upon published academic audit reports.1  The index ranges from  no effort, where the department or institution under scrutiny has no organized educational  quality  processes,  on  to  firefighting,  informal  effort,  organized effort, and finally  mature effort.  Departments, schools, and institutions  with  ’immature’  systems  of  quality  monitoring  may  be encouraged -  or  spurred on -  to do better.   Institutions  with mature systems recognize the planning, tracking, and performance evaluation of quality processes as important elements of peer accountability and collegiality. They have developed appropriate and feasible performance 1 Massy, W. F. (2010) Education Quality Audit as Applied in Hong Kong. In M. Beerkens and D. D. Dill (Eds.) Public Policy for Academic Quality: Analyses of Innovative Policy Instruments, pp. 209-234.  Dordrecht:  Springer.6indicators.   Hence, a ’mature’ system of quality assurance within the individual  university  or  polytechnic  may  be  duly  noted  and  the appropriate institution offered an academic audit regime with a “lighter touch”.  NEW OPPORTUNITY, VISTAS NEW: STRENGTHENING INFORMATION TO THE PUBLIC. Earlier,  we  commended  the  Agency  for  its  impressive  activities  in research and analysis  –  work that  is  disseminated both domestically and internationally to fellow agencies, universities and polytechnics. We would also underline the great opportunity that lies before A3ES to act as  a  valued  provider  of  information  on  higher  education  to  the Portuguese  public  in  general,  not  least  to  prospective  students.  In contrast to the Agency’s 2011 Activity Report, which covered academic publications  and  information  to  participating  institutions  of  higher education, more ‘popularly slanted’ publications may well repay earnest consideration.  Both the database and the Agency’s webpage provide excellent  points  of  departure  for  strengthening  further  information aimed at the general public. While the SC recognizes that current work loads  may  force  the  agency  to  introduce  priorities  for  current evaluations, making the result of such evaluations more easily available to the public- for instance, through a more sophisticated search engine for  tracking  the  results  of  accreditation  -  would  not  only  benefit students. It could also pay dividends with respect to the public trust in, and thus the standing of, the Agency. We remain,Yours very truly,DAVID DILLEmeritus ProfessorUniversity of North Carolina,USAMARY HENKELProfessor AssociateBrunel University, Uxbridge,UK.GUY NEAVE7Professor Emeritus CHEPS, University of Twente;Netherlands.BJØRN STENSAKERProfessorUniversity of Oslo,Norway.DON WESTERHEIJDENSenior Research FellowCHEPS, University of Twente,Netherlands. Consolidated final November 5th 2012 15.11hrs)  8

English

Third report of the Scientific Council to A3ES:Meeting of 10th of October 2012

Third report of the Scientific Council to A3ES meeting of 10th of October 2012

Third report of the Scientific Council to A3ES meeting of 10th of October 2012

Abstract

Similar works

Full text

Available Versions

NARCIS

University of Twente Research Information