In most Computer Science programmes, students are required to undertake an individual project under the guidance of a supervisor during their studies. With increasing student numbers, matching students to suitable supervisors is becoming an increasing challenge. This paper presents a software tool which assists Computer Science students in identifying the most suitable supervisor for their final year project. It does this by matching a list of keywords or a project proposal provided by the students to a list of keywords which were automatically extracted from freely available data for each potential supervisor. The tool was evaluated using both manual and user testing, with generally positive results and user feedback. 83% of respondents agree that the current implementation of the tool is accurate, with 67% saying it would be a useful tool to have when looking for a supervisor. The tool is currently being adapted for wider use in the School.Postprin

Terzic, Kasim

Zemaityte, Gintare

In most Computer Science programmes, students are required to undertake an individual project under the guidance of a supervisor during their studies. With increasing student numbers, matching students to suitable supervisors is becoming an increasing challenge. This paper presents a software tool which assists Computer Science students in identifying the most suitable supervisor for their final year project. It does this by matching a list of keywords or a project proposal provided by the students to a list of keywords which were automatically extracted from freely available data for each potential supervisor. The tool was evaluated using both manual and user testing, with generally positive results and user feedback. 83% of respondents agree that the current implementation of the tool is accurate, with 67% saying it would be a useful tool to have when looking for a supervisor. The tool is currently being adapted for wider use in the School

University of St. Andrews - Pure

Supervisor recommendation tool for Computer Science projects

St Andrews Research Repository

Supervisor Recommendation Tool for Computer ScienceProjectsGintare ZemaityteSchool of Computer Science, University of St AndrewsSt Andrews, Scotland, UKgz25@st-andrews.ac.ukKasim Terzić∗School of Computer Science, University of St AndrewsSt Andrews, Scotland, UKkt54@st-andrews.ac.ukABSTRACTIn most Computer Science programmes, students are required toundertake an individual project under the guidance of a supervisorduring their studies. With increasing student numbers, matchingstudents to suitable supervisors is becoming an increasing challenge.This paper presents a software tool which assists Computer Sciencestudents in identifying the most suitable supervisor for their finalyear project. It does this by matching a list of keywords or a projectproposal provided by the students to a list of keywords which wereautomatically extracted from freely available data for each potentialsupervisor. The tool was evaluated using both manual and usertesting, with generally positive results and user feedback. 83% ofrespondents agree that the current implementation of the tool isaccurate, with 67% saying it would be a useful tool to have whenlooking for a supervisor. The tool is currently being adapted forwider use in the School.CCS CONCEPTS• Applied computing → Education; • Information systems→ Information retrieval;ACM Reference Format:Gintare Zemaityte and Kasim Terzić. 2019. Supervisor RecommendationTool for Computer Science Projects. In Computing Education Practice (CEP’19), January 9, 2019, Durham, United Kingdom. ACM, New York, NY, USA,Article 39, 4 pages. https://doi.org/10.1145/3294016.32940301 INTRODUCTIONAll undergraduate and taught postgraduate programmes in theSchool of Computer Science at the University of St Andrews re-quire completion of an individual project under the guidance of amember of staff. At the moment, there are about 40 staff memberswho can supervise projects, and about 100 undergraduate projects,and 100 MSc dissertations completed every year. In order to matchstudents to supervisors, we have a project blog on which supervi-sors advertise available projects, and students are encouraged tocontact supervisors directly, or propose their own projects.∗This is the corresponding authorPermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.CEP ’19, January 9, 2019, Durham, United Kingdom© 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.ACM ISBN 978-1-4503-6631-1/19/01. . . $15.00https://doi.org/10.1145/3294016.3294030Currently there are over 200 project topics advertised on the blog,which means that it can take a long time for students to find suitablesupervisors and interesting projects. Most of our MSc students areonly with us for a little over a semester by the time they startlooking for dissertation supervisors, so they do not have a completeoverview of the research background of every staff member, so theytend to concentrate on the subset of staff who have taught themduring the semester.The tool presented in this paper provides students with a shortlistof suitable supervisors based on a list of keywords.2 RELATEDWORKWe are not aware of existing algorithms for the particular prob-lem of supervisor application though we suspect that various ap-proaches have been implemented in different departments aroundthe country. There is, however, a wealth of of relevant literatureon matching in the context of publish-subscribe systems, expertiseretrieval, and matching conference papers to reviewers.The main principle of publish/subscribe systems is matchingevents to subscriptions. The matches are more relevant when theyare conditioned on particular attributes and value ranges. An ex-ample would be a classified ad subscription which generated alertsfor all new listings for all items of type “car” with a price attributelower than “£10000” [2, 6]. The complexity of matching large num-ber of events to subscriptions is addressed by using trees [2] variousindexing approaches [13] and optimised database queries [3]. Al-though our problem deals with smaller quantities of data, we makeuse of tree-based structures to speed up the search in our work.Matching quality can be improved by introducing semantic sim-ilarity [11], such as incorporating ontologies [1] and qualifiers forexact and inexact predicate matches [12]. For example, Paolucciet al. employ a hierarchical ontology and flexible matches exhibit-ing a degree of similarity [10]. In our work, we apply these ideasby looking for semantically similar keywords using the WordNetontology.While not phrased explicitly as a matching problem, expertisesearch and discovery is relevant because it attempts to identify a listof people who are knowledgeable about a given topic. A study byBalog et al. used a corpus of heterogeneous publicly available datafrom their own academic institution to determine the probabilityof a given academic being an expert in an identified topic area [5].Using keyword frequency for this task is a common approach [8],and although recent work has produced more complex Bayesianmodels [4, 9], the simpler metric term frequency-inverse documentfrequency (TF-IDF) used by Balog et al. have proven useful [5]. Inour work, we use TF-IDF for ranking suggested supervisors.CEP ’19, January 9, 2019, Durham, United Kingdom G. Zemaityte and K. TerzićFigure 1: The keyword extraction process.Finally, we note the increasing use of tools such as the TorontoPaper Matching System [7], but it requires a manually self-assessedlist of topics for each potential reviewers, which increases the workwhen setting up the system for a new conference.3 AUTOMATED MATCHING TOOL3.1 Automated Data Gathering and ProfileGenerationWe begin by scraping publicly available sources and extracting use-ful text and keywords from them. It is accepted that a researcher’sbody of work is a good indicator of their expertise and interestareas [7], so the sources we use include:• existing projects advertised on our project blog,• existing webpages, including the list of potential PhD super-visors, which come with keywords,• our online student handbook which includes descriptions ofour taught modules and the associated teaching staff,• abstracts from staff papers, available from University pages.The scraping process stores the relevant text in a directory structurewhich is then processed by the RAKE algorithm [? ] to automaticallyextract keywords.The entire process takes about 20 minutes and is automated soit can be repeated often to take new publications into account. Theresult is a keyword database which related keywords to academics,and which can be used in the matching process. The current data-base contains about 6000 key terms. Since it has been noted thatmanually curating a list of keywords and keeping it up-to-date isa tedious process [4], the ability to automatically update our data-base (e.g. when new staff members join the school) is an importantfeature.3.2 RankingIn order to rank supervisors, we calculate the TF-IDF (term frequency-inverse document frequency) measure:TF-IDF = TF × IDF,where (1)TF =# occurrences of a term in the document# all words in the document , andIDF = log(# all documents# documents containing the term).Prior to calculating the scores, we normalise the words by split-ting multi-phrase terms into single words and stemming the wordsusing the Porter stemmer available in the NLTK library. The termfrequency for each stemmed word for each researcher is calculatedin relation to the rest of the terms associated with that researcher.For the inverse document frequency, the total list of documentsis equal to the number of entries in the dictionary, and the searchrequires checking which other researchers are associated with thesame word. We store the final TF-IDF score in a dictionary structurefor each term for every researcher in the database. Precalculatingand storing these values means that any subsequent lookup will befast.3.3 Matching algorithmOur matching algorithm accepts a list of keywords and tries tomatch each of them to a list of supervisors in turn. The algorithmweemploy is a simplified version of the semantic matching algorithmproposed by Paolucci et al. [10]. We start with an empty list ofmatches and then, for each keyword in turn, search a hash-baseddictionary comprising 6000 keywords for matches. Each of thesekeywords is associated with a list of supervisors. We then lookup each supervisor in a second hash-based dictionary structurein order to find the TF-IDF measure for this keyword/supervisorcombination. This is then used to rank the proposals.We apply several heuristics to filter out undesired results. Thisincludes matching only whole words for keywords shorter thanthree characters (i.e. a search for “AI” would not return keywords“aim” or “contain”), and matching parts of words only if they start aword (so searching for “data” will return matches for “database”). Ifa keyword is not found, we obtain a list of ynonyms using NLTK’sWordNet corpus, and look for those. If there are still no matches,the algorithm splits multi-word keywords into single words andlooks for those. So if there are no matches for “data intensive com-putation”, the algorithm would look for “data”, “intensive” and“computation”.Supervisor Recommendation Tool for Computer Science Projects CEP ’19, January 9, 2019, Durham, United KingdomFigure 2: The matching process.3.4 Matching based on written proposalsIn addition to searching by a list of keywords, the tool also allowsa student to upload a short proposal and match it to existing su-pervisor. In this case, the RAKE algorithm will first extract a list ofkeywords from the proposal, and then match them to supervisorsas described above.4 EVALUATIONOur prototype was implemented in Python, using NLTK and theexisting implementation of the RAKE algorithm. There is a rudi-mentary GUI for entering keywords and displaying the results inorder to test the tool with actual users.4.1 User studyWe conducted a small study on 12 MSc student volunteers from thedepartment, all of whom were currently writing their dissertation.They were asked to attempt to attempt to find a supervisor for theircurrent MSc dissertation, and to complete a survey afterwards.The feedback was generally positive. Ten out of twelve peoplefound the tool accurate and eight out of nine found it useful (seeFig. 3). With seven out of twelve users, the tool recommended theircurrent supervisor, which is promising, but shows potential forimprovement. However, ten out of twelve people reported that thetool recommended a suitable supervisor they did not originallyconsider (see Fig. 4).The two users who did not find the tool accurate did not engagemuch with the tool and only tested a very small number of key-words. However, this indicates that the tool requires a certain num-ber of descriptive keywords, which could pose a limit to adoption.This can lead to problems with new topics such as “blockchain” and“ethereum”. Since there are no experts on this topic in the school,the tool will fail to recommend a supervisor, although some staffmembers may be able to supervise such a dissertation.4.2 Manual testingThe traditional way to find suitable supervisors is to ask the projectcoordinator (in this case, the second author). In addition to the userstudy, we tested the tool by trying all the keywords found on ourproject blog and school pages and verifying that the recommenda-tions made by the tool are sensible.Figure 3: 10 out of 12 users in our study found the tool accu-rate. 8 out of 9 found it useful.We also tested the tool by uploading a number of project propos-als and looking for potential supervisors. To evaluate this, we down-loaded 32 existing project proposals which were hidden (and thusnot found by the scraping process). These hidden project proposalscorrespond to past projects which were successfully completedand are no longer advertised to students. In each of these cases,we tested whether our tool would recommend the supervisor whooriginally wrote the proposal. Fig. 5 shows the results. Although ourtool extracted the keywords from the uploaded proposal automati-cally, our tool successfully recommended the original author in 50%of the cases. In most cases, at least one of the suggested supervisorswas a suitable choice for supervising the proposed project. Whilethere is obviously still room for improvement, the ability of the toolto process natural text and suggest suitable supervisors is alreadypromising.CEP ’19, January 9, 2019, Durham, United Kingdom G. Zemaityte and K. TerzićFigure 4: 83% of the users in our study found that the tool rec-ommended a suitable supervisor they would not have other-wise considered.Figure 5: We tested our tool by uploading 32 project pro-posals which correspond to old, completed projects fromprevious years and automatically extracting desired key-words from the text of the proposal. In 50% of the cases, thetool suggested the original supervisor. Even when we onlymatched based on the first 10 extracted keywords, it sug-gested the original author in 17% of the cases.5 CONCLUSIONWe have presented a new tool for suggesting project supervisorsin a higher education setting. With the exception of the manualselection of data sources, it is almost completely automatic andeasy to keep updated. Initial evaluation indicates that users findit accurate and useful, although it fails when used with very rarekeywords.In our school, we currently assign two batches of 100 studentsto project supervisors each year. Even small improvements to theprocess could save students and staff a lot of stress and time, andmatching students to suitable supervisors early could help withstudent satisfaction and learning outcomes. The tool we have de-scribed in this paper is not ideal, but initial evaluation suggests thatit could be useful for improving the allocation process.5.1 Further WorkWe are planning to adapt the existing code to provide a RESTfulAPI which can serve a list of suggested supervisors. This can beintegrated with our existing school webpages as an online toolfor searching for supervisors. This would provide links to existingprojects by suggested supervisors so students can focus on a subsetof relevant projects without having to read all project proposals.We believe that this would make it interesting to other departmentsaround the country.REFERENCES[1] W. Abramowicz, E. Bukowska, J. Dzikowski, A. Filipowska, and M. Kaczmarek.2011. Semantically Enabled Experts Finding System âĂŞ Ontologies, ReasoningApproach and Web Interface Design. In East-European Conference on Advances inDatabases and Information Systems. 157–166.[2] M. K. Aguilera, R. E. Strom, D. C. Sturman, M. Astley, and T. D. Chandra. 1999.Matching Events in a Content-based Subscription System. In ACM Symposium onPrinciples of Distributed Computing. 53–61. https://doi.org/10.1145/301308.301326[3] G. Ashayer, H. K. Y. Leung, and H. A. Jacobsen. 2002. Predicate Matching andSubscription Matching in Publish/Subscribe Systems. In International Conferenceon Distributed Computing Systems Workshops. https://doi.org/10.1109/ICDCSW.2002.1030823[4] K. Balog, L. Azzopardi, and M. de Rijke. 2009. A Language Modeling Frameworkfor Expert Finding. Information Processing & Management 45, 1 (2009), 1–19.https://doi.org/10.1016/j.ipm.2008.06.003[5] K. Balog and M. de Rijke. 2007. Finding Similar Experts. In International ACMSIGIR Conference on Research and Development in Information Retrieval. 821–822.https://doi.org/10.1145/1277741.1277926[6] G. Banavar, T. Chandra, B. Mukherjee, J. Nagarajarao, R. E. Strom, and D. C.Sturman. 1999. Efficient Multicast Protocol for Content-Based Publish-SubscribeSystems. In IEEE International Conference on Distributed Computing Systems.262–272. https://doi.org/10.1109/ICDCS.1999.776528[7] L. Charlin and R. S. Zemel. 2013. The Toronto Paper Matching System: An auto-mated paper-reviewer assignment system. In ICML Workshop on Peer Reviewingand Publishing Models.[8] N. Craswell, D. Hawking, A. Vercoustre, and P. Wilkins. 2001. P@NOPTIC Expert:Searching for Experts not just for Documents. In AusWeb 2001. 21–25.[9] H. Fang and C. Zhai. 2007. Probabilistic Models for Expert Finding. In EuropeanConference on Information Retrieval Research. 418–430. https://doi.org/10.1007/978-3-540-71496-5_38[10] M. Paolucci, T. Kawamura, T. R. Payne, and K. Sycara. 2002. Semantic Matchingof Web Services Capabilities. In International Semantic Web Conference (ISWC).333âĂŞ347. https://doi.org/10.1007/3-540-48005-6_26[11] M. Petrovic, I. Burcea, and H. A. Jacobsen. 2003. S-ToPSS: Semantic TorontoPublish/Subscribe System. In International Conference on Very Large Data Bases,Vol. 29. 1101–1104.[12] K. Sycara, M. Klusch, and S. Widoff. 1999. Dynamic Service Matchmaking AmongAgents in Open Information Environments. ACM SIGMOD Record 28, 1 (1999),47–53. https://doi.org/10.1145/309844.30989[13] T. W. Yan and H. Garcia-Molina. 1999. The SIFT Information DisseminationSystem. ACM Transactions on Database Systems 24, 4 (1999), 529–565. https://doi.org/10.1145/331983.331992Received Oct 10 2018; accepted Nov 9 2018

https://research-repository.st-andrews.ac.uk/bitstream/10023/16935/1/paper.pdf

Supervisor recommendation tool for Computer Science projects

Abstract

Similar works

Full text

Available Versions

University of St. Andrews - Pure

University of St. Andrews - Pure

St Andrews Research Repository