AIS: A nonlinear activation function for industrial safety engineering

In the task of Chinese named entity recognition based on deep learning, activation function plays an irreplaceable role, it introduces nonlinear characteristics into neural network, so that the fitted model can be applied to various tasks. However, the information density of industrial safety analysis text is relatively high, and the correlation and similarity between the information are large, which is easy to cause the problem of high deviation and high standard deviation of the model, no specific activation function has been designed in previous studies, and the traditional activation function has the problems of gradient vanishing and negative region, which also lead to the recognition accuracy of the model can not be further improved. To solve these problems, a novel activation function AIS is proposed in this paper. AIS is an activation function applied in industrial safety engineering, which is composed of two piecewise nonlinear functions. In the positive region, the structure combining exponential function and quadratic function is used to alleviate the problem of deviation and standard deviation, and the linear function is added to modify it, which makes the whole activation function smoother and overcomes the problem of gradient vanishing. In the negative region, the cubic function structure is used to solve the negative region problem and accelerate the convergence of the model. Based on the deep learning model of BERT-BiLSTM-CRF, the performance of AIS is evaluated. The results show that, compared with other activation functions, AIS overcomes the problems of gradient vanishing and negative region, reduces the deviation of the model, speeds up the model fitting, and improves the extraction ability of the model for industrial entities

A criação de um modelo de Natural Language Processing para extração de habilidades técnicas na área de Ciência de Dados

Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced Analytics, specialization in Data ScienceO trabalho surgiu da necessidade do homem de suprir suas necessidades básicas. Na antiguidade, época de Gregos e Romanos, era ensinado a prole como cuidar da terra e esse conhecimento era passado por gerações. Com a chegada da primeira revolução industrial e posteriormente com a popularização dos computadores, o trabalho se tornou o que conhecemos hoje. A forma de contratação mudou ao longo dos anos e com a chegada da tecnologia mais do que nunca, o mercado de trabalho está em constante mudança e como consequência disto as habilidades necessárias para estes trabalhos também seguem esta mudança. Grande parte das vagas de trabalho hoje estão anunciadas em sites de buscas de emprego como Indeed, Glassdoor e Linkedin. Um grande desafio atual, é prever as mudanças e tendências do mercado de trabalho em termos de habilidades, e a analise textual pode gerar uma vantagem competitiva neste sentido. A proposta deste trabalho é analisar através de técnicas de Natural Language Processing (NLP) diferentes oportunidades de emprego da área de Data Science a fim de obter um modelo que possa ser utilizado para extrair as habilidades requisitadas para esta área. Para alcançar este objetivo primeiro é feita uma revisão dos conceitos de Aprendizado de Máquina, Natural Language Processing, Transfer Learning e das técnicas de preparação de dados, e em seguida será apresentada a metodologia utilizada. Depois, são destacadas as técnicas que funcionam melhor para extração de habilidades, a escolha e criação do modelo, e por fim a apresentação de resultados.The work originated from man's need to meet his basic needs. In ancient times, the times of the Greeks and Romans, offspring were taught how to take care of the land and this knowledge was passed on for generations. With the arrival of the first industrial revolution and later with the popularization of computers, work became what we know today. The way of hiring has changed over the years and with the arrival of technology more than ever, the job market is constantly changing, and consequently, the skills needed for these jobs also follow this change. A large part of job vacancies today is advertised on job search sites such as Indeed, Neuvoo, and Linkedin. A big challenge today is to predict the changes and trends in the job market in terms of skills and textual analysis can generate a competitive advantage in this sense. The purpose of this work is to analyze through Natural Language Processing (NLP) techniques different job opportunities in the Data Science area to obtain a model that can be used to extract the required skills for this area. To achieve this goal first a review of the concepts of Machine Learning, Natural Language Processing, Transfer Learning, and data preprocessing, then the methodology used is presented. Next, the techniques that work best for skill extraction is highlighted, the choice and creation of the model, and finally the presentation of results

Transfer Learning for Named Entity Recognition in Financial and Biomedical Documents

status: publishe

Transfer Learning for Named Entity Recognition in Financial and Biomedical Documents

Recent deep learning approaches have shown promising results for named entity recognition (NER). A reasonable assumption for training robust deep learning models is that a sufficient amount of high-quality annotated training data is available. However, in many real-world scenarios, labeled training data is scarcely present. In this paper we consider two use cases: generic entity extraction from financial and from biomedical documents. First, we have developed a character based model for NER in financial documents and a word and character based model with attention for NER in biomedical documents. Further, we have analyzed how transfer learning addresses the problem of limited training data in a target domain. We demonstrate through experiments that NER models trained on labeled data from a source domain can be used as base models and then be fine-tuned with few labeled data for recognition of different named entity classes in a target domain. We also witness an interest in language models to improve NER as a way of coping with limited labeled data. The current most successful language model is BERT. Because of its success in state-of-the-art models we integrate representations based on BERT in our biomedical NER model along with word and character information. The results are compared with a state-of-the-art model applied on a benchmarking biomedical corpus