PDF of presentation used by Andrew Ng of Stanford University and Coursera for his 23 July 2012 Invited Talk "Teaching Machine Learning to 100,000 Students" at the 2012 conference of the American Association for the Advancement of Artificial Intelligence - http://www.webcitation.org/69jee6a8D .



Published by kind permission of Andrew Ng.



Note that a June 2012 TED talk by Daphne Koller - http://goo.gl/IRJqr - covers similar but different ground

Koller, Daphne

Ng, Andrew

English

ALT Open Access Repository

The Online Revolution:Education at ScaleDaphne Koller & Andrew NgStanford University & Coursera1South AfricaUniv. of JohannesburgAvailability2All consumer itemsAffordabilityPrice Changes Since 1985GasMedical CareCollege Tuition & FeesSource: Bureau of Labor Statistics559% Increase3In jobs requiringa college degree55.6%In jobs not requiring a college degree22%Not working22.4%OpportunitySource: "A College Degree, but Not a College Job."  (A. Sum, New York Times,19 May 2011)4Big breakthroughs happen when what is suddenly possible meets what is desperately necessary.—Thomas Friedman May 15, 2012 · New York Times Tom Friedman summarized better than anyone what I’m about to tell you: Big breakthroughs are what happens when what is suddenly possible meets with is desperately necessary.  We’ve talked about the desperately necessary.  What about the suddenly possible?5Coursera4001 ,000What is suddenly possible is to use technology to offer education at scale.  This started with 3 courses that Stanford opened to the world in the fall.  For example, the machine learning class was taught by my colleague Andrew Ng.  Andrew’s ML class is one of the larger at Stanford, with 400 students.  Andrew’s public machine learning class had an enrollment of over 100K students.  So for Andrew to reach the same audience by teaching his Stanford class, he would have to teach it for … 250 years.6Coming from a middle class family from a small town in India. Never had the luck and guidance to reach Stanford for education.  Guess what? God has sent the opportunity right across my door step! Heartfelt thanks to the great team and teachers who made this happen!                     (Akash Goswami)7I'm a single mom 39 years old with two college boys... I've been trying to get back to college for  my masters and want to learn more about computers. Looking forward to this class!   (Jenny Ramirez)8High academic standards2.4 million enrollments190 countries111 courses 16 UniversitiesIt turns out that people like to get great content from the best instructors … for free.  Since mid-February, when our website opened, we have accumulated more than 600K students from 190 countries.  We have close to 1.5 million enrollments in 41 courses, across a range of disciplines.  In the 15 courses that have already launched, we have 14 million video views and 6 million quiz submissions.10Education at Scale11Timeline# users on siteReal Course100K50KCourse Beginsv12VIDEO-BASED INSTRUCTIONCoursera13Video-Based InstructionADD GRAPH?14Writing on tablet appears on monitorWebcam to record instructorLecture RecordingModern sequencing methods (11 min)Genomic economics (8 min)Personalized medicine (13 min)History: The Human Genome Project (12 min)Modern GenomicsGenes and alleles (10 min)Basic Genetics Refresher (OPTIONAL)Mendelian inheritance (13 min)What is DNA?  (12 min)Genetic testing in the commercial world (11 min)Protecting privacy (9 minutes)Case study: direct-to-consumer genetics (12 min)Case study: family planning (13 min)Commercial Genomics (OPTIONAL)Personalized LearningLet’s talk about some of the key elements that went into these courses.  First, when we designs content for the online environment, we can remove the constraints of classroom scheduling, and abandon the monolithic 1hr lecture. The content can be divided into short, coherent modules of 8-12 minutes each, which can be traversed in different ways by different students.  Some students might benefit from some additional preparation.  Others might want go into depth in certain topics. So we can finally move away from the one-size-fits-all model of education into a much more personalized experience.16ASSESSMENTSCoursera17Testing Improves Learning"Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping." J. Karpicke, J. Blunt.  Science (2011).Study0.80.70.60.50.40.30.20.10.0Repeated StudyRetrievalPracticeFinalexamscoreConceptMapping18Interactive Video• Multiple-choice (radio button, checkbox)• Numerical answers• Short answer• Math• Structured outputs: Programming and modeling assignments• Peer assessmentCourseraMultiple forms of assessmentCourseraColleen Lee, Ethan Fast, Alex AikenMath assessmentsExpression equivalence (e.g., a^2 – b^2 = (a+b)(a-b)..)Theorem proving21• Programming assignments– Support for any language: R, Python, Matlab, Java, …– Advanced assessments: timing, memory use, test set performance, …– Spell-check, Kinect™ action recognition• Modeling assignments (e.g., electrical circuits)• Excel spreadsheets (data analysis, financial models, …)CourseraGrading structured outputsIn-Depth ExercisesCourseraPersonalization and MasteryOpen-Ended WorkPeer GradeSelf GradeTeacher GradeStudent Grade“The Impact of Self-and Peer-Grading on Student Learning”.P. Sadler, E. Good. Educational Assessment (2006).25CourseraCf. Calibrated Peer Review™(Chapman, 2001)Peer Grading WorkflowCourseraPeer GradingCOMMUNITYCoursera** font size proportional to sqrt(number of participants)28Sociology 101 Student MapGlobal community of learnersStudentsCourseraDiscussionsCommunity TACommunity TAsvvFellow students on these forums really gave the sense that I wasn't just sitting in my office working on it by myself. The spirit of cooperation and information sharing has been far more than any “non-virtual”course I ever took.       (Sanjaya Kumar)GuatemalaMiamiNigeriaAthens KenyaAustin, TexasCzechoslovakiaNepalMinnesotaA multilingual universal study group VietnamArabic speaking students Russian speaking studentsStudent Study GroupsSan Francisco HCI-Class Study GroupSTATISTICS & ANALYTICSFirst lecturespostedCoursera34New Window into Human LearningWrong answers submitted for machine learning class programming assignmentON HOLD35Submission timelineExperimental setup:Final “correct”submission“incorrect”submission“incorrect”submissionForum visitsFor each forum question q, estimate:Increase in probability of transitioning to correct response given that a user viewed question q(focusing only one “cluster” of incorrect responses at a time)[Jon Huang]New Window into Human LearningQuestion 830: 98 views and 1949 not viewed.- Fraction transitioned to correct if viewed: 0.643- Fraction transitioned to correct if not viewed: 0.34- Fraction improvement by viewing: 1.893CourseraWhat helped the most?[Jon Huang]IndividualTutoringLectureMastery LearningAchievement ScoreThe 2 Sigma Problem"The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring."  B. Bloom, Educational Researcher (1984).38College is a place where a professor’s lecture notes go straight to the students’ lecture notes, without passing through the brains of either.—Mark Twain—Plutarchfrom Ian Kidd's translation of EssaysThe mind is not a vessel that needs filling, but wood that needs igniting.39LectureActive LearningIgniting Minds"Improved Learning in a Large-Enrollment Physics Class." L. Deslauriers, E. Schelew, and C. Wieman. Science (2011).Attendance Engagement Learning40• Just-in-time teaching• Small group problem solving• Higher-level discussion of material• Presentation of real-world applications• Optional interactive sessions better attended than lectures in standard televised coursesIn-class activities• Enhanced interaction, with immediate feedback• Individual tailoring of flow and pace through content• Less threatening environment for students• Detailed analytics to improve courses• Time for meaningful engagement between students and faculty, students and peers• Interaction and creative problem solving are the real value of top universitiesCourseraBenefits to on-campus teachingUniversal Access to EducationWhat would we get if we could offer a free high-quality education to everyone?  First, it establishes education as a basic human right, so that anyone with the motivation and the ability would have the opportunity to get the skills that they need to make a better life for themselves, their families and their communities.  Second, it enables lifelong learning.  It’s a shame that for most of us, learning stops when we finish our formal education.  With these amazing courses, we would always have the opportunity to explore new directions, whether to expand our minds or to make a change in our livesk.  Finally, it opens the door to a wave of innovation.  Because talent can be found everywhere.  Maybe the next Albert Einstein or the next Steve Jobs is living in some remote village in Africa.  With access to education, he or she can come up with the next big idea, and help make the world a better place for all of us.43

The Online Revolution: Education at Scale

Name not available

The Online Revolution:Education at ScaleDaphne Koller & Andrew NgStanford University & Coursera1South AfricaUniv. of JohannesburgAvailability2All consumer itemsAffordabilityPrice Changes Since 1985GasMedical CareCollege Tuition & FeesSource: Bureau of Labor Statistics559% Increase3In jobs requiringa college degree55.6%In jobs not requiring a college degree22%Not working22.4%OpportunitySource: "A College Degree, but Not a College Job."  (A. Sum, New York Times,19 May 2011)4Big breakthroughs happen when what is suddenly possible meets what is desperately necessary.—Thomas Friedman May 15, 2012 · New York Times Tom Friedman summarized better than anyone what I’m about to tell you: Big breakthroughs are what happens when what is suddenly possible meets with is desperately necessary.  We’ve talked about the desperately necessary.  What about the suddenly possible?5Coursera4001 ,000What is suddenly possible is to use technology to offer education at scale.  This started with 3 courses that Stanford opened to the world in the fall.  For example, the machine learning class was taught by my colleague Andrew Ng.  Andrew’s ML class is one of the larger at Stanford, with 400 students.  Andrew’s public machine learning class had an enrollment of over 100K students.  So for Andrew to reach the same audience by teaching his Stanford class, he would have to teach it for … 250 years.6Coming from a middle class family from a small town in India. Never had the luck and guidance to reach Stanford for education.  Guess what? God has sent the opportunity right across my door step! Heartfelt thanks to the great team and teachers who made this happen!                     (Akash Goswami)7I'm a single mom 39 years old with two college boys... I've been trying to get back to college for my masters and want to learn more about computers. Looking forward to this class!   (Jenny Ramirez)8High academic standards2.4 million enrollments190 countries111 courses 16 UniversitiesIt turns out that people like to get great content from the best instructors … for free.  Since mid-February, when our website opened, we have accumulated more than 600K students from 190 countries.  We have close to 1.5 million enrollments in 41 courses, across a range of disciplines.  In the 15 courses that have already launched, we have 14 million video views and 6 million quiz submissions.10Education at Scale11Timeline# users on siteReal Course100K50KCourse Beginsv12VIDEO-BASED INSTRUCTIONCoursera13Video-Based InstructionADD GRAPH?14Writing on tablet appears on monitorWebcam to record instructorLecture RecordingModern sequencing methods (11 min)Genomic economics (8 min)Personalized medicine (13 min)History: The Human Genome Project (12 min)Modern GenomicsGenes and alleles (10 min)Basic Genetics Refresher (OPTIONAL)Mendelian inheritance (13 min)What is DNA?  (12 min)Genetic testing in the commercial world (11 min)Protecting privacy (9 minutes)Case study: direct-to-consumer genetics (12 min)Case study: family planning (13 min)Commercial Genomics (OPTIONAL)Personalized LearningLet’s talk about some of the key elements that went into these courses.  First, when we designs content for the online environment, we can remove the constraints of classroom scheduling, and abandon the monolithic 1hr lecture. The content can be divided into short, coherent modules of 8-12 minutes each, which can be traversed in different ways by different students.  Some students might benefit from some additional preparation.  Others might want go into depth in certain topics. So we can finally move away from the one-size-fits-all model of education into a much more personalized experience.16ASSESSMENTSCoursera17Testing Improves Learning"Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping." J. Karpicke, J. Blunt.  Science (2011).Study0.80.70.60.50.40.30.20.10.0Repeated StudyRetrievalPracticeFinalexamscoreConceptMapping18Interactive Video• Multiple-choice (radio button, checkbox)• Numerical answers• Short answer• Math• Structured outputs: Programming and modeling assignments• Peer assessmentCourseraMultiple forms of assessmentCourseraColleen Lee, Ethan Fast, Alex AikenMath assessmentsExpression equivalence (e.g., a^2 – b^2 = (a+b)(a-b)..)Theorem proving21• Programming assignments– Support for any language: R, Python, Matlab, Java, …– Advanced assessments: timing, memory use, test set performance, …– Spell-check, Kinect™ action recognition• Modeling assignments (e.g., electrical circuits)• Excel spreadsheets (data analysis, financial models, …)CourseraGrading structured outputsIn-Depth ExercisesCourseraPersonalization and MasteryOpen-Ended WorkPeer GradeSelf GradeTeacher GradeStudent Grade“The Impact of Self-and Peer-Grading on Student Learning”.P. Sadler, E. Good. Educational Assessment (2006).25CourseraCf. Calibrated Peer Review™(Chapman, 2001)Peer Grading WorkflowCourseraPeer GradingCOMMUNITYCoursera** font size proportional to sqrt(number of participants)28Sociology 101 Student MapGlobal community of learnersStudentsCourseraDiscussionsCommunity TACommunity TAsvvFellow students on these forums really gave the sense that I wasn't just sitting in my office working on it by myself. The spirit of cooperation and information sharing has been far more than any “non-virtual”course I ever took.       (Sanjaya Kumar)GuatemalaMiamiNigeriaAthens KenyaAustin, TexasCzechoslovakiaNepalMinnesotaA multilingual universal study group VietnamArabic speaking students Russian speaking studentsStudent Study GroupsSan Francisco HCI-Class Study GroupSTATISTICS & ANALYTICSFirst lecturespostedCoursera34New Window into Human LearningWrong answers submitted for machine learning class programming assignmentON HOLD35Submission timelineExperimental setup:Final “correct”submission“incorrect”submission“incorrect”submissionForum visitsFor each forum question q, estimate:Increase in probability of transitioning to correct response given that a user viewed question q(focusing only one “cluster” of incorrect responses at a time)[Jon Huang]New Window into Human LearningQuestion 830: 98 views and 1949 not viewed.- Fraction transitioned to correct if viewed: 0.643- Fraction transitioned to correct if not viewed: 0.34- Fraction improvement by viewing: 1.893CourseraWhat helped the most?[Jon Huang]IndividualTutoringLectureMastery LearningAchievement ScoreThe 2 Sigma Problem"The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring."  B. Bloom, Educational Researcher (1984).38College is a place where a professor’s lecture notes go straight to the students’ lecture notes, without passing through the brains of either.—Mark Twain—Plutarchfrom Ian Kidd's translation of EssaysThe mind is not a vessel that needs filling, but wood that needs igniting.39LectureActive LearningIgniting Minds"Improved Learning in a Large-Enrollment Physics Class." L. Deslauriers, E. Schelew, and C. Wieman. Science (2011).Attendance Engagement Learning40• Just-in-time teaching• Small group problem solving• Higher-level discussion of material• Presentation of real-world applications• Optional interactive sessions better attended than lectures in standard televised coursesIn-class activities• Enhanced interaction, with immediate feedback• Individual tailoring of flow and pace through content• Less threatening environment for students• Detailed analytics to improve courses• Time for meaningful engagement between students and faculty, students and peers• Interaction and creative problem solving are the real value of top universitiesCourseraBenefits to on-campus teachingUniversal Access to EducationWhat would we get if we could offer a free high-quality education to everyone?  First, it establishes education as a basic human right, so that anyone with the motivation and the ability would have the opportunity to get the skills that they need to make a better life for themselves, their families and their communities.  Second, it enables lifelong learning.  It’s a shame that for most of us, learning stops when we finish our formal education.  With these amazing courses, we would always have the opportunity to explore new directions, whether to expand our minds or to make a change in our livesk.  Finally, it opens the door to a wave of innovation.  Because talent can be found everywhere.  Maybe the next Albert Einstein or the next Steve Jobs is living in some remote village in Africa.  With access to education, he or she can come up with the next big idea, and help make the world a better place for all of us.43

https://repository.alt.ac.uk/2224/1/From_Andrew_Ng_The_Online_Revolution_Education_at_Scale_20120806.pdf

The Online Revolution: Education at Scale

Abstract

Similar works

Full text

Available Versions

ALT Open Access Repository

Name not available