Dimitriadis, Yannis

Gasevic, Dragan

Martinez-Maldonado, Roberto

Prieto, Luis P.

Rodríguez-Triana, María Jesús

The adoption of LA proposals in everyday learning and teaching practice is still  slow,andrequireseffectiveidentificationandcommunicationbetweendifferentstakeholder  communities (including researchers, teachers, students and technology developers). To  complement high-level institutional, policy-oriented frameworks to promote LA adoption,  this workshop proposes to look at how LA innovations impact, or are conditioned by, everyday practice at the classroom level (what some authors call “classroom orchestration”).  In this half-day collaborative knowledge building event, participants from these different stakeholder communities bring examples of LA adoption efforts,discuss them through the lens of such classroom orchestration,and further develop frameworks and guidance on what issues should be effectively discussed (and how this communication can be supported)

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Orchestrating learning analyticsCitation for published version:Prieto, LP, Rodríguez-Triana, MJ, Martinez-Maldonado, R, Gasevic, D & Dimitriadis, Y 2018, Orchestratinglearning analytics: Learning analytics adoption at the classroom level. in Proceedings of the 8thInternational Conference on Learning Analytics and Knowledge (LAK'18). Society for Learning AnalyticsResearch (SoLAR).Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK'18)General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 30. Jun. 2018Companion	Proceedings	8th	International	Conference	on	Learning	Analytics	&	Knowledge	(LAK18)	Creative	Commons	License,	Attribution	-	NonCommercial-NoDerivs	3.0	Unported	(CC	BY-NC-ND	3.0)		1 Orchestrating Learning Analytics: Learning Analytics Adoption at the Classroom Level Author(s):	Please	Leave	This	Section	Blank	for	Review	Institution	Email		ABSTRACT:	The	adoption	of	 LA	proposals	 in	everyday	 learning	and	 teaching	practice	 is	 still	slow,	and	requires	effective	identification	and	communication	between	different	stakeholder	communities	 (including	 researchers,	 teachers,	 students	 and	 technology	 developers).	 To	complement	 high-level	 institutional,	 policy-oriented	 frameworks	 to	 promote	 LA	 adoption,	this	 workshop	 proposes	 to	 look	 at	 how	 LA	 innovations	 impact,	 or	 are	 conditioned	 by,	everyday	practice	at	the	classroom	level	(what	some	authors	call	“classroom	orchestration”).	In	 this	 half-day	 collaborative	 knowledge	 building	 event,	 participants	 from	 these	 different	stakeholder	 communities	bring	examples	of	 LA	adoption	efforts,	discuss	 them	 through	 the	lens	of	such	classroom	orchestration,	and	further	develop	frameworks	and	guidance	on	what	issues	should	be	effectively	discussed	(and	how	this	communication	can	be	supported).		Keywords:	Orchestration,	Learning	Analytics	Adoption,	Inter-Stakeholder	Communication.	1 BACKGROUND: ORCHESTRATION AND LEARNING ANALYTICS Despite	 the	 recent	explosion	of	 research	 in	 the	 field	of	 learning	analytics	 (LA),	 the	adoption	of	 its	proposals	 in	 everyday	 classroom	 practice	 is	 still	 quite	 limited,	 and	 progresses	 slowly	 (Ali,	 Asadi,	Gašević,	 Jovanović,	&	Hatala,	 2013).	Multiple	 researchers,	 indeed,	 have	 looked	 at	 the	 problem	of	large-scale	LA	adoption,	especially	considering	how	institutions	can	drive	such	adoption	or	develop	strategies	 to	 favor	 it	 (Ferguson	 et	 al.,	 2014;	Macfadyen,	 Dawson,	 Pardo,	 &	 Gaševic,	 2014).	Many	high-level	LA	adoption	frameworks,	often	aimed	at	higher	education	institutions,	recognize	the	need	for	 stakeholder	 identification	 and	 communication	 with	 such	 stakeholders	 in	 order	 to	 meet	 their	specific	 needs	 (Macfadyen	 et	 al.,	 2014).	 They	 do	 not	 provide,	 however,	 concrete	 guidelines	 or	support	for	such	communication,	or	what	topics	should	most	urgently	be	addressed	by	it.	Yet,	the	problem	of	slow	adoption	is	not	specific	to	learning	analytics,	but	rather	is	a	manifestation	of	the	more	general	gap	between	research	and	practice	that	plagues	different	areas	of	educational	research.	 Research	 on	 systemic	 and	 large-scale	 innovations	 have	 noted	 that	 success	 in	 these	endeavors	entails	a	holistic	approach	that	not	only	considers	strategic	policy,	but	also	the	impact	on	classroom-level	practice	(Looi,	So,	Toh,	&	Chen,	2011).	Especially	crucial	in	this	regard	is	the	role	of	teachers/practitioners	 as	 major	 gatekeepers	 for	 the	 technological	 and	 practice	 innovations	 that	reach	 the	 classrooms.	 The	 need	 for	 proposals	 that	 take	 into	 account	 the	 often-dire	 contextual	constraints	 of	 classroom	practice	 have	 led	 to	 the	 notion	 of	 designing	 for	 classroom	orchestration	(defined	 as	 “the	 process	 of	 productively	 coordinating	 supportive	 interventions	 across	 multiple	learning	activities	occurring	at	multiple	social	levels”	(Dillenbourg,	Järvela	&	Fischer,	2009,	p.	12).	Recent	reviews	of	technology-enhanced	learning	(TEL)	and	LA	literature	[anonymized]	have	mapped	LA-specific	 issues	 and	 frameworks	with	 this	 general	 concern	 about	 the	 impact	 on	 classroom-level	Companion	Proceedings	8th	International	Conference	on	Learning	Analytics	&	Knowledge	(LAK18)	Creative	Commons	License,	Attribution	-	NonCommercial-NoDerivs	3.0	Unported	(CC	BY-NC-ND	3.0)		2 practice,	 focusing	 on	 the	 knowledge	 gaps	 that	 arise	 between	 the	 different	 stakeholders	 (e.g.,	teachers,	students,	researchers	and	technology	providers)	during	the	adoption	of	an	LA	tool	(Figure	1).	 In	 this	 workshop,	 such	 mapping	 serves	 as	 a	 starting	 point	 (or	 a	 ‘boundary	 object’,	 Star	 &	Griesemer,	 1989)	 for	 the	 dialogue	 among	 stakeholders	 about	what	 issues	 need	 to	 be	 shared	 and	understood,	and	how	to	support	more	effective	inter-stakeholder	communication	in	this	process	of	adoption.	 Figure	1:	A	framework	for	LA	adoption	at	the	classroom	level	This	workshop	builds	upon	previous	events	 that	brought	 together	different	stakeholders	groups	 to	discuss	 classroom-level	 factors	 and	 how	 they	 condition	 adoption,	 both	 in	 the	 case	 of	 LA1,	 and	 in	other	fields	of	educational	technology	research2.	As	such,	this	workshop	serves	as	a	complement	to	policy-oriented	 LA	 adoption	 workshops	 held	 in	 previous	 LAK	 conferences	 (Tsai,	 Gaševic,	 Muñoz-Merino,	&	Dawson,	2017).	2 GOALS AND OUTCOMES The	 present	 workshop	 aims	 at	 engaging	 the	 LA	 research	 community	 in	 a	 dialogue	 with	 other	stakeholders	 (practitioners,	 technology	 developers),	 to	 share	 concrete	 adoption	 experiences	 and	discuss	 about	 what	 factors	 influence	 a	 successful	 adoption	 of	 LA	 solutions,	 by	 looking	 at	 the	classroom	level,	rather	than	a	more	institutional	viewpoint.		The	 outcomes	 of	 the	 workshop	 may	 thus	 consist	 of	 an	 enriched	 framework	 and	 instruments	 to	support	 communication	 about	 the	 adoption	 of	 LA	 (from	 the	 perspective	 of	 orchestration).	 These	products,	 together	with	 the	 rest	of	workshop-generated	materials	will	 be	 shared	 in	 the	workshop	website	and	social	media,	and	eventually	provide	the	basis	for	a	SoLAR	sub-community	around	this	topic.	Other	future	steps	towards	the	establishment	of	this	community	will	also	be	discussed	during	the	workshop,	including	a	journal	special	issue,	a	follow-up	workshop	in	the	frame	of	LAK’19	or	other	related	 conferences,	 or	 the	 creation	 of	 a	 virtual	 community	 to	 share	 experiences	 and	refined/contextualized	boundary	objects	around	inter-stakeholder	communication	for	LA	adoption.	As	such,	the	workshop	is	strongly	aligned	with	the	LAK’18	theme,	with	respect	to	LA	innovation	and	                                                            1	In	the	LASI	Spain	2016	event:	http://lprisan.wixsite.com/orla2016		2	In	CSCL2015	(https://sites.google.com/site/occw15/),	or	ICLS2012	(https://www.isls.org/icls/2012/program/#workshops)	Companion	Proceedings	8th	International	Conference	on	Learning	Analytics	&	Knowledge	(LAK18)	Creative	Commons	License,	Attribution	-	NonCommercial-NoDerivs	3.0	Unported	(CC	BY-NC-ND	3.0)		3 adoption	 in	 authentic	 “classroom-level”	 contexts,	 promoting	 stakeholder	 engagement	 and	communication,	accountability,	and	co-design	of	effective	LA	tools.	3 WORKSHOP ORGANIZATION 3.1 General organizational details • Type	of	event	and	proposed	duration:	Half-day	workshop.	• Type	 of	 participation:	 Mixed	 participation.	 Both	 participants	 with	 a	 paper	 submission	(following	an	open	call)	and	other	members	of	the	LAK	community.	• Expected	 participants:	 We	 expect	 15-20	 participants	 from	 the	 main	 abovementioned	stakeholder	 groups	 (including	 5-8	 participants	 presenting	 brief	 papers	 with	 LA	 adoption	cases):	 LA	 researchers,	 practitioners,	 as	 well	 as	 learning	 analytics	 developers/providers.	Special	care	will	be	taken	prior	to	the	workshop	to	recruit	and	 involve	at	 least	these	three	groups,	 so	 that	 their	 perspectives	 could	 be	 directly	 be	 taken	 into	 account	 during	 the	workshop	 activities	 and	 in	 the	 community	 to	 emerge	 from	 the	 workshop.	 Researcher	participants	 will	 be	 recruited	 using	 the	 scientific	 networks	 in	 which	 the	 organizers	 are	directly	involved:	mailing	lists	and	groups	such	as	EATEL,	Kaleidoscope,	AIED,	EDM,	ISLS,	and	of	course	the	SoLAR	mailing	 lists	and	Facebook	groups.	Additional	special	attention	will	be	paid	 to	 the	 social	 networks	 established	 through	 previous	 workshops	 organized	 by	 the	authors	in	related	topics,	in	the	ICLS,	CSCL,	EC-TEL	and	LAK	conferences.	Local	teachers	and	other	educational	practitioners	will	be	addressed	through	associations	such	as	ATEA,	APTA	and	 AEU.	 Additionally,	 local	 educational	 technology	 developers/providers	 and	 other	practitioners	will	 be	 recruited	 from	 Sydney-based	 universities	 and	 enterprises,	with	which	the	organizers	are	in	contact.	• Required	equipment:	 The	workshop	will	 require	 Internet	connectivity,	 a	projector	 (for	 the	presentations	by	organizers,	authors	of	accepted	papers,	and	spokespersons	of	 the	groups	that	 intervene	 in	 the	 workshop),	 a	 whiteboard	 (optional),	 as	 well	 as	 flipchart	 paper	 and	markers	 (for	 the	 collaborative	 activities).	 Flexible	 room	 layout	 (e.g.,	 movable	 chairs)	 and	clean	walls	(to	hang	flipchart	paper	or	put	post-its)	are	also	highly	recommended.		3.2 Specific workshop activities The	half-day	workshop	schedule	aims	to	provide	an	intense,	balanced	and	flexible	context	in	which	participants	will	be	actively	engaged.	Also,	it	aims	to	create	the	best	conditions	for	the	creation	of	a	sustained	community	involving	researchers,	teachers	and	other	stakeholders	regarding	adoption	of	learning	 analytics.	 Thus,	 the	workshop	will	 be	 structured	 in	 three	parts	 (A-C)	 (indicative	durations	are	included	below,	allowing	for	variations	or	delays,	depending	on	the	group	dynamics).	Breaks	will	serve	to	reinforce	the	workshop	group	cohesion,	 to	allow	for	 informal	continuation	of	discussions,	and	 to	 offer	 a	 working	 slot	 for	 the	 organizers	 to	 generate	 a	 refined	 framework	 proposal,	 to	 be	discussed	during	the	last	part	of	the	workshop.			• A.1	Warm	up	(20	minutes):	Sharing	the	workshop	objectives	with	participants,	letting	them	know	 each	 other,	 and	 creating	 the	 initial	 conditions	 for	 an	 engaging	work	 in	 groups	 (also	serving	to	account	for	eventual	initial	delays).		• A.2	Initial	framework	proposals	(20	minutes):	The	organizers	will	present	the	[anonymized]	framework,	as	a	starting	point	for	the	discussion,	together	with	other	alternative	classroom-level	 frameworks/schemes	 for	 LA	 adoption.	 This	 activity	 builds	on	existing	materials	 to	be	Companion	Proceedings	8th	International	Conference	on	Learning	Analytics	&	Knowledge	(LAK18)	Creative	Commons	License,	Attribution	-	NonCommercial-NoDerivs	3.0	Unported	(CC	BY-NC-ND	3.0)		4 shared	(before	the	workshop)	through	the	workshop	website	and	the	call	for	contributions.	These	 frameworks	 aim	 to	 reinforce	 a	 shared	 understanding	 of	 the	 problem,	 as	 well	 as	provide	a	common	structure	to	submitted	proposals,	and	of	the	underlying	issues.	• A.3	Pecha	Kucha-like	presentations	of	accepted	contributions	(40	minutes):	Authors	of	the	5-8	accepted	contributions	will	make	short	presentations.	In	these	presentations,	they	may	focus	on	specific	 features	of	 the	 [anonymized]	conceptual	 frameworks,	map	concrete	case	studies	and	 findings	 to	 them,	show	the	need	and	 feasibility	of	boundary	objects	 that	align	stakeholders,	or	even	propose	alternative	formulations	of	the	framework	or	other	artifacts	to	scaffold	inter-stakeholder	communication.	• B.1	Discussion	in	small	groups	(30	minutes):	The	participants	split	to	small	groups	in	order	to	 discuss	 the	 framework	 proposals,	 assuming	 their	 perspective	 as	 different	 stakeholders	(teacher,	 technology	 designer,	 researcher),	 and	 looking	 for	 the	 most	 suitable	 boundary	objects	that	may	support	their	communication	needs	towards	a	decision	on	LA	adoption.	• B.2	Reporting	back	and	whole	group	discussion	 (30	minutes):	Using	a	 jigsaw	scheme,	 the	expert	groups	that	reflect	the	views	of	the	stakeholders	report	back	to	the	whole	workshop,	followed	by	discussion	in	terms	of	the	main	issues	and	proposals.	• C.1	Synthesis	and	further	steps	(30	minutes):	The	organizers	will	provide	a	synthesis	of	the	output	 of	 the	workshop	 through	 a	 shared	 document,	 allowing	 for	 the	 integration	 of	 new	proposals	 and	 comments	 by	 the	 participants	 in	 real	 time.	 The	 future	 steps	 towards	 the	establishment	 of	 a	 community	will	 also	 be	 discussed,	which	may	 include	 a	 journal	 special	issue,	 a	 follow-up	 workshop	 in	 the	 frame	 of	 LAK19	 or	 other	 related	 conferences,	 or	 the	creation	of	a	virtual	community	to	share	experiences	and	refined	boundary	objects	around	inter-stakeholder	communication	for	LA	adoption.	REFERENCES  Ali,	 L.,	 Asadi,	 M.,	 Gašević,	 D.,	 Jovanović,	 J.,	 &	 Hatala,	 M.	 (2013).	 Factors	 influencing	 beliefs	 for	adoption	of	a	 learning	analytics	tool:	An	empirical	study.	Computers	&	Education,	62,	130–148.	doi:10.1016/j.compedu.2012.10.023	Dillenbourg,	 P.,	 Järvelä	 S.	 &	 Fischer,	 F.	 (2009).	 The	 evolution	 of	 research	 on	 computer-supported	collaborative	learning.	In	Technology	enhanced	learning.	Springer,	3–19.	Ferguson,	 R.,	 Macfadyen,	 L.	 P.,	 Clow,	 D.,	 Tynan,	 B.,	 Alexander,	 S.,	 &	 Dawson,	 S.	 (2014).	 Setting	Learning	Analytics	 in	Context:	Overcoming	 the	Barriers	 to	Large-Scale	Adoption.	 Journal	of	Learning	Analytics,	1(3),	120–144.	doi:10.18608/jla.2014.13.7	Looi,	C.	K.,	So,	H.	J.,	Toh,	Y.,	&	Chen,	W.	(2011).	The	Singapore	experience:	Synergy	of	national	policy,	classroom	 practice	 and	 design	 research.	 International	 Journal	 of	 Computer-Supported	Collaborative	Learning,	6(1),	9-37.	Macfadyen,	 L.	 P.,	 Dawson,	 S.,	 Pardo,	 A.,	 &	 Gaševic,	 D.	 (2014).	 Embracing	 Big	 Data	 in	 Complex	Educational	Systems:	The	Learning	Analytics	 Imperative	and	the	Policy	Challenge.	Research	&	Practice	in	Assessment,	9,	17-28.	Star,	 S	 L,	 &	 Griesemer,	 J.	 R.	 (1989).	 Institutional	 Ecology,	 `Translations’	 and	 Boundary	 Objects:	Amateurs	 and	 Professionals	 in	 Berkeley’s	Museum	 of	 Vertebrate	 Zoology,	 1907-39.	 Social	Studies	of	Science,	19(3),	387–420.	doi:10.1177/030631289019003001	Tsai,	 Y.	 S.,	 Gaševic,	 D.,	 Muñoz-Merino,	 P.	 J.,	 &	 Dawson,	 S.	 (2017).	 LA	 policy:	 developing	 an	institutional	 policy	 for	 learning	 analytics	 using	 the	 RAPID	 outcome	mapping	 approach.	 In	Proceedings	of	the	Seventh	LAK	Conference	(pp.	494-495).	ACM.	

Orchestrating learning analytics:Learning analytics adoption at the classroom level

https://www.pure.ed.ac.uk/ws/files/47401963/LAK_18_paper_33.pdf

Orchestrating learning analytics:Learning analytics adoption at the classroom level

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer