19 research outputs found
EVALITA Evaluation of NLP and Speech Tools for Italian Proceedings of the Final Workshop
Editor of the proceedings of EVALITA 2016
Using Lucene for Developing a Question-Answering Agent in Portuguese
Given the limitations of available platforms for creating conversational agents, and that a question-answering agent suffices in many scenarios, we take advantage of the Information Retrieval library Lucene for developing such an agent for Portuguese. The solution described answers natural language questions based on an indexed list of FAQs. Its adaptation to different domains is a matter of changing the underlying list. Different configurations of this solution, mostly on the language analysis level, resulted in different search strategies, which were tested for answering questions about the economic activity in Portugal. In addition to comparing the different search strategies, we concluded that, towards better answers, it is fruitful to combine the results of different strategies with a voting method
EVALITA Goes Social: Tasks, Data, and Community at the 2016 Edition
EVALITA, the evaluation campaign of Natural Language Processing and Speech Tools for the Italian language, was organised for the fifth time in 2016. Six tasks, covering both re-reruns as well as completely new tasks, and an IBM-sponsored challenge, attracted a total of 34 submissions. An innovative aspect at this edition was the focus on social media data, especially Twitter, and the use of shared data across tasks, yielding a test set with layers of annotation concerning PoS tags, sentiment information, named entities and linking, and factuality information. Differently from the previous edition(s), many systems relied on a neural architecture, and achieved best results when used. From the experience and success of this edition, also in terms of dissemination of information and data, and in terms of collaboration between organisers of different tasks, we collected some reflections and suggestions that prospective EVALITA chairs might be willing to take into account for future editions
OddAssist - An eSports betting recommendation system
It is globally accepted that sports betting has been around for as long as the sport itself. Back in
the 1st century, circuses hosted chariot races and fans would bet on who they thought would
emerge victorious. With the evolution of technology, sports evolved and, mainly, the
bookmakers evolved. Due to the mass digitization, these houses are now available online, from
anywhere, which makes this market inherently more tempting. In fact, this transition has
propelled the sports betting industry into a multi-billion-dollar industry that can rival the sports
industry.
Similarly, younger generations are increasingly attached to the digital world, including
electronic sports – eSports. In fact, young men are more likely to follow eSports than traditional
sports. Counter-Strike: Global Offensive, the videogame on which this dissertation focuses, is
one of the pillars of this industry and during 2022, 15 million dollars were distributed in
tournament prizes and there was a peak of 2 million concurrent viewers. This factor, combined
with the digitization of bookmakers, make the eSports betting market extremely appealing for
exploring machine learning techniques, since young people who follow this type of sports also
find it easy to bet online.
In this dissertation, a betting recommendation system is proposed, implemented, tested, and
validated, which considers the match history of each team, the odds of several bookmakers and
the general feeling of fans in a discussion forum.
The individual machine learning models achieved great results by themselves. More specifically,
the match history model managed an accuracy of 66.66% with an expected calibration error of
2.10% and the bookmaker odds model, with an accuracy of 65.05% and a calibration error of
2.53%.
Combining the models through stacking increased the accuracy to 67.62% but worsened the
expected calibration error to 5.19%. On the other hand, merging the datasets and training a
new, stronger model on that data improved the accuracy to 66.81% and had an expected
calibration error of 2.67%.
The solution is thoroughly tested in a betting simulation encapsulating 2500 matches. The
system’s final odd is compared with the odds of the bookmakers and the expected long-term
return is computed. A bet is made depending on whether it is above a certain threshold. This
strategy called positive expected value betting was used at multiple thresholds and the results
were compared.
While the stacking solution did not perform in a betting environment, the match history model
prevailed with profits form 8% to 90%; the odds model had profits ranging from 13% to 211%;
and the dataset merging solution profited from 11% to 77%, all depending on the minimum
expected value thresholds.
Therefore, from this work resulted several machine learning approaches capable of profiting
from Counter Strike: Global Offensive bets long-term.É globalmente aceite que as apostas desportivas existem há tanto tempo quanto o próprio
desporto. Mesmo no primeiro século, os circos hospedavam corridas de carruagens e os fãs
apostavam em quem achavam que sairia vitorioso, semelhante Ă s corridas de cavalo de agora.
Com a evolução da tecnologia, os desportos foram evoluindo e, principalmente, evoluĂram as
casas de apostas. Devido à onda de digitalização em massa, estas casas passaram a estar
disponĂveis online, a partir de qualquer sĂtio, o que torna este mercado inerentemente mais
tentador. De facto, esta transição propulsionou a indústria das apostas desportivas para uma
indústria multibilionária que agora pode mesmo ser comparada à indústria dos desportos.
De forma semelhante, gerações mais novas estão cada vez mais ligadas ao digital, incluindo
desportos digitais – eSports. Counter-Strike: Global Offensive, o videojogo sobre o qual esta
dissertação incide, é um dos grandes impulsionadores desta indústria e durante 2022, 15
milhões de dĂłlares foram distribuĂdos em prĂ©mios de torneios e houve um pico de espectadores
concorrentes de 2 milhões. Embora esta realidade não seja tão pronunciada em Portugal, em
vários paĂses, jovens adultos do sexo masculino, tĂŞm mais probabilidade de acompanharem
eSports que desportos tradicionais. Este fator, aliado à digitalização das casas de apostas,
tornam o mercado de apostas em eSports muito apelativo para a exploração técnicas de
aprendizagem automática, uma vez que os jovens que acompanham este tipo de desportos têm
facilidade em apostar online.
Nesta dissertação é proposto, implementado, testado e validado um sistema de recomendação
de apostas que considera o histórico de resultados de cada equipa, as cotas de várias casas de
apostas e o sentimento geral dos fãs num fórum de discussão – HLTV. Deste modo, foram
inicialmente desenvolvidos 3 sistemas de aprendizagem automática.
Para avaliar os sistemas criados, foi considerado o perĂodo de outubro de 2020 atĂ© março de
2023, o que corresponde a 2500 partidas. PorĂ©m, sendo o perĂodo de testes tĂŁo extenso, existe
muita variação na competitividade das equipas. Deste modo, para evitar que os modelos
ficassem obsoletos durante este perĂodo de teste, estes foram re-treinados no mĂnimo uma vez
por mĂŞs durante a duração do perĂodo de testes.
O primeiro sistema de aprendizagem automática incide sobre a previsão a partir de resultados
anteriores, ou seja, o histórico de jogos entre as equipas. A melhor solução foi incorporar os
jogadores na previsĂŁo, juntamente com o ranking da equipa e dando mais peso aos jogos mais
recentes. Esta abordagem, utilizando regressĂŁo logĂstica teve uma taxa de acerto de 66.66%
com um erro expectável de calibração de 2.10%.
O segundo sistema compila as cotas das várias casas de apostas e faz previsões com base em
padrões das suas variações. Neste caso, incorporar as casas de aposta tendo atingido uma taxa
de acerto de 65.88% utilizando regressĂŁo logĂstica, porĂ©m, era um modelo pior calibrado que o
modelo que utilizava a média das cotas utilizando gradient boosting machine, que exibiu uma
taxa de acerto de 65.06%, mas melhores métricas de calibração, com um erro expectável de
2.53%.
O terceiro sistema, baseia-se no sentimento dos fĂŁs no fĂłrum HLTV. Primeiramente, Ă© utilizado
o GPT 3.5 para extrair o sentimento de cada comentário, com uma taxa geral de acerto de
84.28%. No entanto, considerando apenas os comentários classificados como conclusivos, a taxa de acerto é de 91.46%. Depois de classificados, os comentários são depois passados a um
modelo support vector machine que incorpora o comentador e a sua taxa de acerto nas partidas
anteriores. Esta solução apenas previu corretamente 59.26% dos casos com um erro esperado
de calibração de 3.22%.
De modo a agregar as previsões destes 3 modelos, foram testadas duas abordagens.
Primeiramente, foi testado treinar um novo modelo a partir das previsões dos restantes
(stacking), obtendo uma taxa de acerto de 67.62%, mas com um erro de calibração esperado
de 5.19%. Na segunda abordagem, por outro lado, sĂŁo agregados os dados utilizados no treino
dos 3 modelos individuais, e Ă© treinado um novo modelo com base nesse conjunto de dados
mais complexo. Esta abordagem, recorrendo a support vector machine, obteve uma taxa de
acerto mais baixa, 66.81% mas um erro esperado de calibração mais baixo, 2.67%.
Por fim, as abordagens são postas à prova através de um simulador de apostas, onde sistema
cada faz uma previsão e a compara com a cota oferecia pelas casas de apostas. A simulação é
feita para vários patamares de retorno mĂnimo esperado, onde os sistemas apenas apostam
caso a taxa esperada de retorno da cota seja superior Ă do patamar.
Esta cota final Ă© depois comparada com as cotas das casas de apostas e, caso exista uma casa
com uma cota superior, uma aposta é feita. Esta estratégia denomina-se de apostas de valor
esperado positivo, ou seja, apostas cuja cota Ă© demasiado elevada face Ă probabilidade de se
concretizar e que geram lucros a longo termo. Nesta simulação, os melhores resultados, para
uma taxa de mĂnima de 5% foram os modelos criados a partir das cotas das casas de apostas,
com lucros entre os 13% e os 211%; o dos dados histĂłricos que lucrou entre 8% e 90%; e por
fim, o modelo composto, com lucros entre os 11% e os 77%.
Assim, deste trabalho resultaram diversos sistemas baseados em machine learning capazes de
obter lucro a longo-termo a apostar em Counter Strike: Global Offensive
Applications of Deep Neural Networks
Deep learning is a group of exciting new technologies for neural networks.
Through a combination of advanced training techniques and neural network
architectural components, it is now possible to create neural networks that can
handle tabular data, images, text, and audio as both input and output. Deep
learning allows a neural network to learn hierarchies of information in a way
that is like the function of the human brain. This course will introduce the
student to classic neural network structures, Convolution Neural Networks
(CNN), Long Short-Term Memory (LSTM), Gated Recurrent Neural Networks (GRU),
General Adversarial Networks (GAN), and reinforcement learning. Application of
these architectures to computer vision, time series, security, natural language
processing (NLP), and data generation will be covered. High-Performance
Computing (HPC) aspects will demonstrate how deep learning can be leveraged
both on graphical processing units (GPUs), as well as grids. Focus is primarily
upon the application of deep learning to problems, with some introduction to
mathematical foundations. Readers will use the Python programming language to
implement deep learning using Google TensorFlow and Keras. It is not necessary
to know Python prior to this book; however, familiarity with at least one
programming language is assumed
CLARIN
The book provides a comprehensive overview of the Common Language Resources and Technology Infrastructure – CLARIN – for the humanities. It covers a broad range of CLARIN language resources and services, its underlying technological infrastructure, the achievements of national consortia, and challenges that CLARIN will tackle in the future. The book is published 10 years after establishing CLARIN as an Europ. Research Infrastructure Consortium
CLARIN. The infrastructure for language resources
CLARIN, the "Common Language Resources and Technology Infrastructure", has established itself as a major player in the field of research infrastructures for the humanities. This volume provides a comprehensive overview of the organization, its members, its goals and its functioning, as well as of the tools and resources hosted by the infrastructure. The many contributors representing various fields, from computer science to law to psychology, analyse a wide range of topics, such as the technology behind the CLARIN infrastructure, the use of CLARIN resources in diverse research projects, the achievements of selected national CLARIN consortia, and the challenges that CLARIN has faced and will face in the future.
The book will be published in 2022, 10 years after the establishment of CLARIN as a European Research Infrastructure Consortium by the European Commission (Decision 2012/136/EU)
CLARIN
The book provides a comprehensive overview of the Common Language Resources and Technology Infrastructure – CLARIN – for the humanities. It covers a broad range of CLARIN language resources and services, its underlying technological infrastructure, the achievements of national consortia, and challenges that CLARIN will tackle in the future. The book is published 10 years after establishing CLARIN as an Europ. Research Infrastructure Consortium
Tweaking Word Embeddings for FAQ Ranking
Presentiamo il sistema sviluppato presso FBK per la risoluzione del task EVALITA 2016 “QA4FAQ - Question Answering for Frequently Asked Questions”. Una caratteristica peculiare di questo task ´e la totale mancanza di dati di training, pertanto abbiamo creato una rappresentazione significativa dei dati utilizzando solamente word embeddings. Presentiamo il sistema assieme ai risultati ottenuti dalle due esecuzioni che abbiamo inviato e un’analisi qualitativa dei risultati stessi.We present the system developed at FBK for the EVALITA 2016 Shared Task “QA4FAQ – Question Answering for Frequently Asked Questions”. A peculiar characteristic of this task is the total absence of training data, so we created a meaningful representation of the data using only word embeddings. We present the system as well as the results of the two submitted runs, and a qualitative analysis of them