Our group has been working on a report for the Spanish Society of Software Engineering and Software Development Technologies (SISTEDES) to provide a general overview of the Spanish scientific production and its contributions worldwide in the field of Software Engineering. Although a Database solution could have been used, we decided to employ Model-Driven Development (MDD) techniques in order to evaluate their applicability, suitability and fitness for these kinds of purposes, and to learn from the experience in this domain, which combines data integration, large scale models, and complex queries.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Burgueño, Loli

Delgado-Moreno, Antonio

Vallecillo-Moreno, Antonio Jesus

English

Burgueño-Caballero, Lola

Vallecillo-Moreno, Antonio Jesús

Repositorio Institucional Universidad de Málaga

Analysis of the Scientific Production of theSpanish Software Engineering CommunityLoli Burguen˜o, Antonio Moreno-Delgado, and Antonio VallecilloUniversidad de Ma´laga, Atenea Research Group, Ma´laga, Spain{loli,amoreno,av}@lcc.uma.esAbstract. Our group has been working on a report for the SpanishSociety of Software Engineering and Software Development Technolo-gies (SISTEDES) to provide a general overview of the Spanish scientificproduction and its contributions worldwide in the field of Software Engi-neering. Although a Database solution could have been used, we decidedto employ Model-Driven Development (MDD) techniques in order toevaluate their applicability, suitability and fitness for these kinds of pur-poses, and to learn from the experience in this domain, which combinesdata integration, large scale models, and complex queries.Keywords: SISTEDES, scientific contribution, Spain, software engi-neering, MDD1 IntroductionThe increase in research in the past years and the fact that in most of the cases itis funded by public institutions or private companies raised the need of indicatorsto measure its quality. Among the most popular are the Journal Citation Report(JCR), which measures the impact factor of a journal, and the h-index, whichis a measure of the number of high impact papers a scientist has published.Given those metrics it is easy to know whether a publication or a researcheris remarkable. Nevertheless, in the case that the interest resides in knowingthe influence of a whole community there is no other option than collectingthe information of each member and apply descriptive statistical techniques.Descriptive statistics aims at analyzing a population in order to describe orsummarize quantitatively its main features.Our group has been working on a report for the Spanish Society of SoftwareEngineering and Software Development Technologies (SISTEDES) to providea general overview of the Spanish scientific production and its contributionsworldwide in the field of Software Engineering, using the DBLP database as thesource of publications.Although a Database solution could have been used, we decided to employMDD techniques to eat our own dog food. We wanted to check whether currentMDD technologies and languages could easily deal with these kinds of applica-tions, and to evaluate their applicability, suitability and fitness for these kindsof purposes. This was a good way to learn from the experience in this domain,which combines data integration, large scale models, and complex queries. Thisproblem also represents several non-trivial challenges. In the first place, the so-lution needs to be formulated in terms of models and model transformations,where the models integrate data from different heterogeneous sources (DBLP,spreadsheets with information about SISTEDES members, JCR journal rank-ings). Second, models are too big to be stored in main memory, and to be handledby existing platforms and transformation languages (e.g., QVT [1] or ATL [2]).Thirdly, we need to cope with non-existing or missing data.In this paper we describe how the above challenges have been addressed, andhow the final report has been generated.The paper is organized as follows. Section 2 presents how the solution to theproblem was formulated using MDD techniques, in terms of models that capturethe data sources, and transformations implementing the data integration and thequeries. Then, Section 3 discusses the queries that were made to the model, andtheir performan¡ce results. Finally, Section 4 presents our conclusions and anoutlook on future work.2 Analysis definition and procedure2.1 Modeling the Information SourcesThe first step was to identify the information that needed to be handled. Figure 1depicts the main sources of information and how it is processed.Fig. 1. Information sources and main process.In the first place, we need the data about the community to be analyzed(CommunityDB). In our case study the community consists of the members ofSISTEDES, although our solution does not depend on any particular scientificcommunity, but on the data that we need from their members. This is capturedby the Community metamodel shown in Figure 2, which represent Members thatbelong to Organizations (universities or companies) and can attend the Conferencesorganized by the community.Fig. 2. Community metamodel.The second source of information is about the publications (Publications DBin Figure 1). In our case it comes from the DBLP database (http://dblp.uni-trier.de/). DBLP was selected because it is commonly used in our field,and contains representative information about the contributions that normallymatter. Other alternative sources of information that we also considered wereGoogle Scholar, Web of Science or SCOPUS. In any case, in the solution we pro-pose we try to be as agnostic as possible from the concrete publication database,as we shall later see. The DBLP metamodel is shown in Figure 3. It correspondsto the XML structures used to store the data in the DBLP system.Fig. 3. DBLP metamodel.Finally, there is the need to incorporate information about the quality ofthe contributions, in terms of how they rank in commonly used internationalreports, such as JCR for journals and CORE for conferences. This informationis captured by the Ranking metamodel shown in Figure 4.Fig. 4. Ranking metamodel.2.2 Extracting the dataThe second step consisted of populating the models with the data from the threedifferent sources.Getting the DBLP information was quite straightforward. An XML file withall the DBLP data can be downloaded from the DBLP website (http://dblp.uni-trier.de/). Once downloaded, it is a matter of parsing it to generate amodel conforming to the DBLP metamodel previously shown in Figure 3. Thereare two different strategies when parsing an XML file: event-based, where thatthe parser launches an event each time a starting or an end tag is found duringthe parsing process; and tree-based, where the parser loads the entire XMLdocument, generates the tree and once it is complete it is returned. Given thesize of the XML file, performance and the memory usage was a critical factortherefore, our choice was an event-based parser – in particular we used SAX,which better fits our needs. Another alternative could have been the use of theAPI that EMF offers to handle XML files. However, the performance of SAXwas a key factor for selecting it to parse the large XML files.The information about the members of the community came from SISTEDESdirectly. We were provided with an Excel file that contained the informationdescribed in the metamodel shown in Figure 2 for 308 members.Finally, the ranking information about the journals and conferences camefrom two different sources. From an Excel file with the 2015 edition of the JCRlistings (the latest available JCR report at the time of the study) we extractedthe ranking information about the journals. Given that our scope was the sci-entific production in the context of SISTEDES, we only considered three JCRcategories: Software Engineering, Information Systems, and Theory and Meth-ods. Although the members of SISTEDES also publish in journals from othercategories (Medical information, Multidisciplinary, etc.), we focused in the jour-nals that fall within the scope of the Society.Conferences were ranked using the information taken from a SCIE1 reporton Computer Science conferences, which used several international rankings, in-cluding CORE, and calculated impact factors for the most relevant conferences.A total of 201 conferences were considered in this study, covering all areas ofpotential interest to SISTEDES (software engineering, databases, logic, program-ming, etc.). Conferences were classified in three groups, A*, A and B, depend-ing on their calculated impact factor according to that report. The spreadsheetwith the ranking of conferences is available from https://www.dropbox.com/s/2a85fuki3zdx9dw/RankingCongresos2015.pdf?dl=0. From that file the modelwith the conferences ranking was populated.2.3 Merging all sources of informationThe information from the three sources needed to be merged into a single, in-tegrated model. For this we designed a Unified metamodel (shown in Figure 5)that contains all the relevant information for our purposes.Fig. 5. Unified metamodel.Then, it is just a matter of developing a model transformation that integratesthe models with the three sources of information into one consolidated modelconforming to the Unified metamodel. One of the problems found here is thatthe size of the DBLP model, and hence of the unified model, was too large to behandled by existing model transformation engines, such as QVT or ATL. Then,we decided to use jLinTra [3,4], a parallel model transformation language andengine that is also able to deal with very large models (no matter if they can fitin memory or need to reside in external storage).1 Sociedad Cient´ıfica Informa´tica de Espan˜a, http://www.scie.esThe size of the DBLP model that we used is 5 millions elements, basically thesame as the size of the resulting unified model. The transformation that buildsthe consolidated models took 132.88 seconds in a machine with Ubuntu 12.04 64-bits operating system, 11.7 Gb of RAM and 2 processors with 4 hyperthreadedcores (8 threads) of 2.67GHz each. The transformation contains 732 lines of Javacode, and was built in 5 hours by an experienced developer.Another interesting problem had to do with establishing the correspondencesbetween the different models. We decided to use the names of the members tomatch the names of the authors in DBLP, and the acronyms of the journals andconferences to match the DBLP and ranking models (i.e., name-based correspon-dences). This implied that we had to used normalized information, making surethat the names of the authors in DBLP and the names of the members referredto the right person. This process took some time, because both the communityand the ranking models had to be manually checked for consistency.3 Queries, Results and DiscussionOnce we have a model with all the information that we need, it is just a mat-ter of performing the queries to generate the figures for the report. Basically, thekind of queries consisted in extracting information about different aspects of in-terest to the proponents of the report. For example, the number of contributionsper year of the community, classified according to the kind of publication (JCRjournal, non-JCR journal, A-conference, B-conference, etc.). We could also askfor that information, but organized according particular subgroups such as indi-vidual universities, or considering only quality publications (JCR journals andA*-conferences). Moreover, we could query the model to identify the journalsand conferences that are normally used by the members of the community todisseminate their results.Queries were modeled by developing a metamodel for each one, and defininga transformation from the unified model into them, which produced the requiredresults. Figure 6 shows two of the metamodels used to generate the results fortwo of the queries.The transformation was also developed in jLinTra. It contains 323 lines ofJava code, and was built in 4 hours by an experienced developer. Starting fromevery member of the community, it navigates the unified model in order toextract the information about his/her publications and the university wherehe/she belongs. Then, it synthesizes the results obtained in order to determinethe number publications of each type.The execution of the queries using jLinTra took 3.644 seconds. Given theparallel nature of LinTra, executing in parallel one model transformation foreach query, or executing a single model transformation that produced the outputmodels of all queries yielded very similar results.Finally, the resulting models were parsed and transformed into csv files forthe corresponding exploitation to produce the report. Nevertheless, in the futurewe plan to do this by means of a model-to-text transformation.(a) Query 1. (b) Query 2.Fig. 6. Metamodels for two of the queries.4 Conclusions and Further WorkIn this paper we have described the “Making of” of a report about the scientificproduction of the SISTEDES community, using DBLP as bibliographic source 2We have shown how the information can be organized, merged and exploited us-ing MDD techniques, and a technical solution is valid and feasible. We have alsorealized that existing MDD languages and platforms still find problems for cop-ing with large models, but recent proposals (such as LinTra) permit supportingthe scalability and performance required to address some of these issues.Using MDD solutions also brings along interesting advantages, and in partic-ular modularity. One important benefit of the architecture of the solution, de-picted in Figure 1, is that it does not necessarily depend on the specific sources.In fact, we have worked in the report for the production of the SISTEDES com-munity using the DBLP information, but reports for other communities, or usingdifferent bibliographic databases, are easy to develop. It is just a matter of defin-ing transformations from the specific sources to the metamodels that are usedas input in our merging transformation, but whole process does not need to bechanged. Similarly, generating new queries becomes a simple matter of devel-oping a model transformation from the unified metamodel to the metamodel ofthe query of interest. Thus, for example it would be interesting to compare theresults obtained using DBLP with the ones that we would obtain if we used theGoogle Scholar database, or SCOPUS.Furthermore, this work is a preliminary step into a more detailed study onthe exploration of communities using MDD techniques. We plan to extend ourstudy according to different research lines. First, the description of the membersof the community is currently achieved in an explicit manner, i.e., giving the listof members. But sometimes we would also like to explore the bibliographic sourceusing wildcards or any implicit description method (e.g., all Spanish authors).This is difficult in general, mainly because the information about the authorsin DBLP is quite limited, which means having to provide the required informa-2 The report will be available from the SISTEDES website (http://www.sistedes.es/informes-recursos/) in September 2015.tion about the authors using alternative mechanisms. A second improvement ofour work would consist of using weaving models between the different sourcesinstead of the very basic name-matching approach used here. Thirdly, we wantto conduct a comparison analysis between our proposal and other approaches tobibliometric analysis that use traditional methods (e.g., databases). Similarly,the use of JLintra allowed us to handle very large models in an efficient way, butthere are other approaches that use solutions such as EMFStore, CDO, Morsa orNeo4EMF and they provide querying mechanisms. A comparison analysis withthese approaches could also help us better understand the advantages and limi-tations of our proposed solution. Fourth, we would like to automate the reportgeneration and come up with a Domain Specific Language so that users coulddefine the reports they want/need, and the whole process could take care of ev-erything, specially the generation of the queries metamodels. Finally, we wouldalso like to explore our points in common with MetaScience [5], a recent tool toanalyze Scientific Conferences. Although the focus of the tool is on conferencesand not on communities, many of the ideas of the two proposals could be shared.Acknowledgments This work is funded by Research Project TIN2014-52034-Rand by the Universidad de Ma´laga (Campus de Excelencia Internacional An-daluc´ıa Tech).References1. OMG: MOF QVT Final Adopted Specification. Object Management Group. (2005)2. Jouault, F., Allilaire, F., Be´zivin, J., Kurtev, I.: ATL: A model transformation tool.Science of Computer Programming 72(1-2) (2008) 31–393. Burguen˜o, L., Troya, J., Wimmer, M., Vallecillo, A.: On the concurrent executionof model transformations with linda. In: Proc. of BigMDE’13, Budapest, Hungary,ACM (2013) 3:1–3:104. Burguen˜o, L.: Concurrent Model Transformations based on Linda. In: DoctoralSymposium @ MODELS. (2013)5. Ca´novas, J., Consentino, V., Cabot, J.: MetaScience – a tool to analyze researchconferences (2015) http://atlanmod.github.io/metaScience/.

Analysis of the scientific production of the Spanish Software Engineering Community

Repositorio Institucional de la Universidad de Málaga

https://riuma.uma.es/xmlui/bitstream/10630/10290/1/JISBD2015.pdf

Analysis of the scientific production of the Spanish Software Engineering Community

Abstract

Similar works

Full text

Available Versions

Repositorio Institucional Universidad de Málaga

Repositorio Institucional de la Universidad de Málaga