Semantic-based padding in convolutional neural networks for improving the performance in natural language processing. A case of study in sentiment analysis

This is the author's version of a work that was accepted for publication in Neurocomputing. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Neurocomputing, Volume 378, 22 February 2020, DOI: 10.1016/j.neucom.2019.08.096[EN] In this work, a methodology for applying semantic-based padding in Convolutional Neural Networks for Natural Language Processing tasks is proposed. Semantic-based padding takes advantage of the unused space required for having a fixed-size input matrix in a Convolutional Network effectively, using words present in the sentence. The methodology proposed has been evaluated intensively in Sentiment Analysis tasks using a variety of word embeddings. In all the experimentation carried out the proposed semantic-based padding improved the results achieved when no padding strategy is applied. Moreover, when the model used a pre-trained word embeddings, the performance of the state of the art has been surpassed.We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. The work of the first author is financed by Grant PAID-01-2461 2015, from the Universitat Politecnica de Valencia. This work is partially supported by and grantnumber. the Grant PROMETEO/2018/002 from GVA.Giménez, M.; Palanca Cámara, J.; Botti Navarro, VJ. (2020). Semantic-based padding in convolutional neural networks for improving the performance in natural language processing. A case of study in sentiment analysis. Neurocomputing. 378:315-323. https://doi.org/10.1016/j.neucom.2019.08.096S315323378Ye, Q., & Doermann, D. (2015). Text Detection and Recognition in Imagery: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(7), 1480-1500. doi:10.1109/tpami.2014.2366765Zhao, W., Chellappa, R., Phillips, P. J., & Rosenfeld, A. (2003). Face recognition. ACM Computing Surveys, 35(4), 399-458. doi:10.1145/954339.954342Li, P., & Mao, K. (2019). Knowledge-oriented convolutional neural network for causal relation extraction from natural language texts. Expert Systems with Applications, 115, 512-523. doi:10.1016/j.eswa.2018.08.009Yoo, S., Song, J., & Jeong, O. (2018). Social media contents based sentiment analysis and prediction system. Expert Systems with Applications, 105, 102-111. doi:10.1016/j.eswa.2018.03.055LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989). Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4), 541-551. doi:10.1162/neco.1989.1.4.541W. Yin, K. Kann, M. Yu, H. Schütze, Comparative study of CNN and RNN for natural language processing, arXiv:1702.01923 (2017).J. Villena Román, S. Lana Serrano, E. Martínez Cámara, J.C. González Cristóbal, Tass-workshop on sentiment analysis at SEPLN (2013).Pang, B., & Lee, L. (2008). Opinion Mining and Sentiment Analysis. Foundations and Trends® in Information Retrieval, 2(1–2), 1-135. doi:10.1561/1500000011Mohammad, S. M., & Turney, P. D. (2012). CROWDSOURCING A WORD-EMOTION ASSOCIATION LEXICON. Computational Intelligence, 29(3), 436-465. doi:10.1111/j.1467-8640.2012.00460.xKiritchenko, S., Zhu, X., & Mohammad, S. M. (2014). Sentiment Analysis of Short Informal Texts. Journal of Artificial Intelligence Research, 50, 723-762. doi:10.1613/jair.4272T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, arXiv:1301.3781 (2013).P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information, arXiv:1607.04606 (2016).Araque, O., Corcuera-Platas, I., Sánchez-Rada, J. F., & Iglesias, C. A. (2017). Enhancing deep learning sentiment analysis with ensemble techniques in social applications. Expert Systems with Applications, 77, 236-246. doi:10.1016/j.eswa.2017.02.002Chen, T., Xu, R., He, Y., & Wang, X. (2017). Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN. Expert Systems with Applications, 72, 221-230. doi:10.1016/j.eswa.2016.10.065Y. Zhang, B. Wallace, A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification, arXiv:1510.03820 (2015).Y. Kim, Convolutional neural networks for sentence classification, arXiv:1408.5882 (2014).Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157-166. doi:10.1109/72.279181Zhang, W., Itoh, K., Tanida, J., & Ichioka, Y. (1990). Parallel distributed processing model with local space-invariant interconnections and its optical architecture. Applied Optics, 29(32), 4790. doi:10.1364/ao.29.004790S.M. Mohammad, S. Kiritchenko, X. Zhu, NRC-Canada: building the state-of-the-art in sentiment analysis of tweets, arXiv:1308.6242 (2013).J. Barnes, R. Klinger, S.S.i. Walde, Assessing state-of-the-art sentiment models on state-of-the-art sentiment datasets, arXiv:1709.04219 (2017).Medhat, W., Hassan, A., & Korashy, H. (2014). Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal, 5(4), 1093-1113. doi:10.1016/j.asej.2014.04.011M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G.S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, TensorFlow: Large-scale machine learning on heterogeneous systems, 2015, Software available from tensorflow.org

A Computational Argumentation Framework for Agent Societies

Starting from the idea that the social context of agents determines the way in which agents can argue and reach agreements, this context should have a decisive influence in the computational representation of arguments. In this report, we advance research in the area of computational frameworks for agent argumentation by proposing a new argumentation framework (AF) for the design of open MAS in which the participating software agents are able to manage and exchange arguments between themselves taking into account the agents¿ social context. In order to do this, we have analysed the necessary requirements for this type of framework 1 and taken into account them in the design of our framework. Also, the knowledge resources that the agents can use to manage arguments in this framework are presented in this work. In addition, if heterogeneous agents can interact in the framework, they need a common language to represent arguments and argumentation processes. To cope with this, we have also designed an argumentation ontology to represent arguments and argumentation concepts in our framework.Heras Barberá, SM.; Botti Navarro, VJ.; Julian Inglada, VJ. (2011). A Computational Argumentation Framework for Agent Societies. http://hdl.handle.net/10251/1103

Case-Based Argumentation Framework. Reasoning Process

The capability of reaching agreements is a necessary feature that large computer systems where agents interoperate must include. In these systems, agents represent self-motivated entities that have a social context, including dependency relations among them, and different preferences and beliefs. Without agreement there is no cooperation and thus, complex tasks which require the interaction of agents with different points of view cannot be performed. In this work, we follow a case-based argumentation approach for the design and implementation of Multi-Agent Systems where agents reach agreements by arguing and improve their argumentation skills from experience. A set of knowledge resources and a reasoning process that agents can use to manage their positions and arguments are presented.Heras Barberá, SM.; Botti Navarro, VJ.; Julian Inglada, VJ. (2011). Case-Based Argumentation Framework. Reasoning Process. http://hdl.handle.net/10251/1109

Case-Based Argumentation Framework. Dialogue Protocol

On top of the simpler ability to interact, open MAS must include mechanisms for their agents to reach agreements by taking into account their social context. Argumentation provides MAS with a framework that assures a rational communication, which allows agents to reach agreements when conflicts of opinion arise. In this report we present the communication protocol that agents of a case-based argumentation framework use to interact when they engage in argumentation dialogues. The syntax and semantics of the framework are formalised and discussed.Heras Barberá, SM.; Botti Navarro, VJ.; Julian Inglada, VJ. (2011). Case-Based Argumentation Framework. Dialogue Protocol. http://hdl.handle.net/10251/1109

Case-Based Argumentation Framework. Strategies

In agent societies, agents perform complex tasks that require different levels of intelligence and give rise to interactions among them. From these interactions, conflicts of opinion can arise, specially when MAS become adaptive and open with heterogeneous agents dynamically entering in or leaving the system. Therefore, software agents willing to participate in this type of systems will require to include extra capabilities to explicitly represent and generate agreements on top of the simpler ability to interact. In addition, agents can take advantage of previous argumentation experiences to follow dialogue strategies and easily persuade other agents to accept their opinions. Our insight is that CBR can be very useful to manage argumentation in open MAS and devise argumentation strategies based on previous argumentation experiences. To demonstrate the foundations of this suggestion, this report presents the work that we have done to develop case-based argumentation strategies in agent societies. Thus, we propose a case-based argumentation framework for agent societies and define heuristic dialogue strategies based on it. The framework has been implemented and evaluated in a real customer support application.Heras Barberá, SM.; Botti Navarro, VJ.; Julian Inglada, VJ. (2011). Case-Based Argumentation Framework. Strategies. http://hdl.handle.net/10251/1109

Applying CBR to manage argumentation in MAS

[EN] The application of argumentation theories and techniques in multi-agent systems has become a prolific area of research. Argumentation allows agents to harmonise two types of disagreement situations: internal, when the acquisition of new information (e.g., about the environment or about other agents) produces incoherences in the agents' mental state; and external, when agents that have different positions about a topic engage in a discussion. The focus of this paper is on the latter type of disagreement situations. In those settings, agents must be able to generate, select and send arguments to other agents that will evaluate them in their turn. An efficient way for agents to manage these argumentation abilities is by using case-based reasoning, which has been successfully applied to argumentation from its earliest beginnings. This reasoning methodology also allows agents to learn from their experiences and therefore, to improve their argumentation skills. This paper analyses the advantages of applying case-based reasoning to manage arguments in multi-agent systems dialogues, identifies open issues and proposes new ideas to tackle them.This work was partially supported by CONSOLIDERINGENIO 2010 under grant CSD2007-00022 and by the Spanish government and FEDER funds under CICYT TIN2005-03395 and TIN2006-14630-C0301 projects.Heras Barberá, SM.; Julian Inglada, VJ.; Botti Navarro, VJ. (2010). Applying CBR to manage argumentation in MAS. International Journal of Reasoning-based Intelligent Systems. 2(2):110-117. https://doi.org/10.1504/IJRIS.2010.034906S1101172

Modelling dialogues in agent societies

Besides the simpler ability to interact, open multi-agent systems must include mechanisms for their agents to reach agreements by taking into account their social context. Argumentation provides multi-agent systems with a framework that assures a rational communication, which allows agents to reach agreements when conflicts of opinion arise. In this paper, we present the dialogue protocol that agents of a case-based argumentation framework can use to interact when they engage in argumentation dialogues. The syntax and semantics of the argumentation protocol are formalised and discussed. To illustrate our proposal, we have applied the protocol in the context of a water market. By using our dialogue protocol, agents represent water users that are able to explore different water allocations and justify their views about what is the best water distribution in a certain environment.This work is supported by the Spanish government Grants CONSOLIDER INGENIO 2010 CSD2007-00022, MINECO/FEDER TIN2012-36586-C03-01, and MICINN TIN2011-27652-C03-01.Heras Barberá, SM.; Botti Navarro, VJ.; Julian Inglada, VJ. (2014). Modelling dialogues in agent societies. Engineering Applications of Artificial Intelligence. 34:208-226. https://doi.org/10.1016/j.engappai.2014.06.003S2082263

Reasoning about norms under uncertainty in dynamic environments

The behaviour of norm-autonomous agents is determined by their goals and the norms that are explicitly represented inside their minds. Thus, they require mechanisms for acquiring and accepting norms, determining when norms are relevant to their case, and making decisions about norm compliance. Up un- til now the existing proposals on norm-autonomous agents assume that agents interact within a deterministic environment that is certainly perceived. In prac- tice, agents interact by means of sensors and actuators under uncertainty with non-deterministic and dynamic environments. Therefore, the existing propos- als are unsuitable or, even, useless to be applied when agents have a physical presence in some real-world environment. In response to this problem we have developed the n-BDI architecture. In this paper, we propose a multi -context graded BDI architecture (called n-BDI) that models norm-autonomous agents able to deal with uncertainty in dynamic environments. The n-BDI architecture has been experimentally evaluated and the results are shown in this paper.This paper was partially funded by the Spanish government under Grant CONSOLIDER-INGENIO 2010 CSD2007-00022 and the Valencian government under Project PROMETEOH/2013/019.Criado Pacheco, N.; Argente, E.; Noriega, P.; Botti Navarro, VJ. (2014). Reasoning about norms under uncertainty in dynamic environments. International Journal of Approximate Reasoning. 55(9):2049-2070. https://doi.org/10.1016/j.ijar.2014.02.004S2049207055

An energy-aware algorithm for electric vehicle infrastructures in smart cities

[EN] The deployment of a charging infrastructure to cover the increasing demand of electric vehicles (EVs) has become a crucial problem in smart cities. Additionally, the penetration of the EV will increase once the users can have enough charging stations. In this work, we tackle the problem of locating a set of charging stations in a smart city considering heterogeneous data sources such as open data city portals, geo-located social network data, and energy transformer substations. We use a multi-objective genetic algorithm to optimize the charging station locations by maximizing the utility and minimizing the cost. Our proposal is validated through a case study and several experimental results.This work was partially supported by MINECO/FEDER, Spain RTI2018-095390-B-C31 project of the Spanish government. Jaume Jordan and Vicent Botti are funded by UPV, Spain PAID-06-18 project. Jaume Jordan is also funded by grant APOSTD/2018/010 of Generalitat Valenciana -Fondo Social Europeo, Spain.Palanca Cámara, J.; Jordán, J.; Bajo, J.; Botti Navarro, VJ. (2020). An energy-aware algorithm for electric vehicle infrastructures in smart cities. Future Generation Computer Systems. 108:454-466. https://doi.org/10.1016/j.future.2020.03.001S454466108Gan, L., Topcu, U., & Low, S. H. (2013). Optimal decentralized protocol for electric vehicle charging. IEEE Transactions on Power Systems, 28(2), 940-951. doi:10.1109/tpwrs.2012.2210288Ma, T., & Mohammed, O. A. (2014). Optimal Charging of Plug-in Electric Vehicles for a Car-Park Infrastructure. IEEE Transactions on Industry Applications, 50(4), 2323-2330. doi:10.1109/tia.2013.2296620Needell, Z. A., McNerney, J., Chang, M. T., & Trancik, J. E. (2016). Potential for widespread electrification of personal vehicle travel in the United States. Nature Energy, 1(9). doi:10.1038/nenergy.2016.112Franke, T., & Krems, J. F. (2013). Understanding charging behaviour of electric vehicle users. Transportation Research Part F: Traffic Psychology and Behaviour, 21, 75-89. doi:10.1016/j.trf.2013.09.002Shukla, A., Pekny, J., & Venkatasubramanian, V. (2011). An optimization framework for cost effective design of refueling station infrastructure for alternative fuel vehicles. Computers & Chemical Engineering, 35(8), 1431-1438. doi:10.1016/j.compchemeng.2011.03.018Nie, Y. (Marco), & Ghamami, M. (2013). A corridor-centric approach to planning electric vehicle charging infrastructure. Transportation Research Part B: Methodological, 57, 172-190. doi:10.1016/j.trb.2013.08.010Tu, W., Li, Q., Fang, Z., Shaw, S., Zhou, B., & Chang, X. (2016). Optimizing the locations of electric taxi charging stations: A spatial–temporal demand coverage approach. Transportation Research Part C: Emerging Technologies, 65, 172-189. doi:10.1016/j.trc.2015.10.004Dong, J., Liu, C., & Lin, Z. (2014). Charging infrastructure planning for promoting battery electric vehicles: An activity-based approach using multiday travel data. Transportation Research Part C: Emerging Technologies, 38, 44-55. doi:10.1016/j.trc.2013.11.001He, J., Yang, H., Tang, T.-Q., & Huang, H.-J. (2018). An optimal charging station location model with the consideration of electric vehicle’s driving range. Transportation Research Part C: Emerging Technologies, 86, 641-654. doi:10.1016/j.trc.2017.11.026Jordán, J., Palanca, J., del Val, E., Julian, V., & Botti, V. (2018). A Multi-Agent System for the Dynamic Emplacement of Electric Vehicle Charging Stations. Applied Sciences, 8(2), 313. doi:10.3390/app8020313Jurdak, R., Zhao, K., Liu, J., AbouJaoude, M., Cameron, M., & Newth, D. (2015). Understanding Human Mobility from Twitter. PLOS ONE, 10(7), e0131469. doi:10.1371/journal.pone.0131469Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002). A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2), 182-197. doi:10.1109/4235.996017Coello Coello, C. A. (2002). Theoretical and numerical constraint-handling techniques used with evolutionary algorithms: a survey of the state of the art. Computer Methods in Applied Mechanics and Engineering, 191(11-12), 1245-1287. doi:10.1016/s0045-7825(01)00323-

IN-RED 2017.Congreso nacional de innovación educativa y de docencia en red 13 y 14 de julio 2017

Con el fin de apoyar las acciones docentes realizadas para la mejora del aprendizaje que conllevan una innovación metodológica así como el desarrollo y/o utilización de tecnologías como recursos didácticos,os presentamos el II Congreso Nacional de Innovación Educativa y Docencia en Red,cuyo objetivo es la difusión de experiencias y el debate e intercambio de ideas respecto a estas cuestiones. Dado el éxito del I Congreso Nacional de Innovación Educativa y Docencia en Red del pasado año,planteamos una segunda edición del mismo,que ayude a la expansión del conocimiento y al intercambio de experiencias dentro de la comunidad educativa a nivel nacionalFernández Prada, MÁ.; Botti Navarro, VJ. (2017). IN-RED 2017.Congreso nacional de innovación educativa y de docencia en red 13 y 14 de julio 2017. Editorial Universitat Politècnica de València. http://hdl.handle.net/10251/86908EDITORIA