258 research outputs found
A Gold Standard for Emotion Annotation in Stack Overflow
Software developers experience and share a wide range of emotions throughout
a rich ecosystem of communication channels. A recent trend that has emerged in
empirical software engineering studies is leveraging sentiment analysis of
developers' communication traces. We release a dataset of 4,800 questions,
answers, and comments from Stack Overflow, manually annotated for emotions. Our
dataset contributes to the building of a shared corpus of annotated resources
to support research on emotion awareness in software development.Comment: To appear in Proceedings of the 15th International Conference on
Mining Software Repositories (MSR '18) Data Showcase Track, 28-29 May,
Gothenburg, Swede
Can We Use SE-specific Sentiment Analysis Tools in a Cross-Platform Setting?
In this paper, we address the problem of using sentiment analysis tools
'off-the-shelf,' that is when a gold standard is not available for retraining.
We evaluate the performance of four SE-specific tools in a cross-platform
setting, i.e., on a test set collected from data sources different from the one
used for training. We find that (i) the lexicon-based tools outperform the
supervised approaches retrained in a cross-platform setting and (ii) retraining
can be beneficial in within-platform settings in the presence of robust gold
standard datasets, even using a minimal training set. Based on our empirical
findings, we derive guidelines for reliable use of sentiment analysis tools in
software engineering.Comment: 12 page
EmoTxt: A Toolkit for Emotion Recognition from Text
We present EmoTxt, a toolkit for emotion recognition from text, trained and
tested on a gold standard of about 9K question, answers, and comments from
online interactions. We provide empirical evidence of the performance of
EmoTxt. To the best of our knowledge, EmoTxt is the first open-source toolkit
supporting both emotion recognition from text and training of custom emotion
classification models.Comment: In Proc. 7th Affective Computing and Intelligent Interaction
(ACII'17), San Antonio, TX, USA, Oct. 23-26, 2017, p. 79-80, ISBN:
978-1-5386-0563-
How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow
Context: The success of Stack Overflow and other community-based
question-and-answer (Q&A) sites depends mainly on the will of their members to
answer others' questions. In fact, when formulating requests on Q&A sites, we
are not simply seeking for information. Instead, we are also asking for other
people's help and feedback. Understanding the dynamics of the participation in
Q&A communities is essential to improve the value of crowdsourced knowledge.
Objective: In this paper, we investigate how information seekers can increase
the chance of eliciting a successful answer to their questions on Stack
Overflow by focusing on the following actionable factors: affect, presentation
quality, and time.
Method: We develop a conceptual framework of factors potentially influencing
the success of questions in Stack Overflow. We quantitatively analyze a set of
over 87K questions from the official Stack Overflow dump to assess the impact
of actionable factors on the success of technical requests. The information
seeker reputation is included as a control factor. Furthermore, to understand
the role played by affective states in the success of questions, we
qualitatively analyze questions containing positive and negative emotions.
Finally, a survey is conducted to understand how Stack Overflow users perceive
the guideline suggestions for writing questions.
Results: We found that regardless of user reputation, successful questions
are short, contain code snippets, and do not abuse with uppercase characters.
As regards affect, successful questions adopt a neutral emotional style.
Conclusion: We provide evidence-based guidelines for writing effective
questions on Stack Overflow that software engineers can follow to increase the
chance of getting technical help. As for the role of affect, we empirically
confirmed community guidelines that suggest avoiding rudeness in question
writing.Comment: Preprint, to appear in Information and Software Technolog
A Benchmark Study on Sentiment Analysis for Software Engineering Research
A recent research trend has emerged to identify developers' emotions, by
applying sentiment analysis to the content of communication traces left in
collaborative development environments. Trying to overcome the limitations
posed by using off-the-shelf sentiment analysis tools, researchers recently
started to develop their own tools for the software engineering domain. In this
paper, we report a benchmark study to assess the performance and reliability of
three sentiment analysis tools specifically customized for software
engineering. Furthermore, we offer a reflection on the open challenges, as they
emerge from a qualitative analysis of misclassified texts.Comment: Proceedings of 15th International Conference on Mining Software
Repositories (MSR 2018
Assessment of Off-the-Shelf SE-specific Sentiment Analysis Tools: An Extended Replication Study
Sentiment analysis methods have become popular for investigating human
communication, including discussions related to software projects. Since
general-purpose sentiment analysis tools do not fit well with the information
exchanged by software developers, new tools, specific for software engineering
(SE), have been developed. We investigate to what extent SE-specific tools for
sentiment analysis mitigate the threats to conclusion validity of empirical
studies in software engineering, highlighted by previous research. First, we
replicate two studies addressing the role of sentiment in security discussions
on GitHub and in question-writing on Stack Overflow. Then, we extend the
previous studies by assessing to what extent the tools agree with each other
and with the manual annotation on a gold standard of 600 documents. We find
that different SE-specific sentiment analysis tools might lead to contradictory
results at a fine-grain level, when used 'off-the-shelf'. Conversely,
platform-specific tuning or retraining might be needed to take into account
differences in platform conventions, jargon, or document lengths.Comment: Accepted for publication in Empirical Software Engineerin
Emotion Classification In Software Engineering Texts: A Comparative Analysis of Pre-trained Transformers Language Models
Emotion recognition in software engineering texts is critical for
understanding developer expressions and improving collaboration. This paper
presents a comparative analysis of state-of-the-art Pre-trained Language Models
(PTMs) for fine-grained emotion classification on two benchmark datasets from
GitHub and Stack Overflow. We evaluate six transformer models - BERT, RoBERTa,
ALBERT, DeBERTa, CodeBERT and GraphCodeBERT against the current best-performing
tool SEntiMoji. Our analysis reveals consistent improvements ranging from 1.17%
to 16.79% in terms of macro-averaged and micro-averaged F1 scores, with general
domain models outperforming specialized ones. To further enhance PTMs, we
incorporate polarity features in attention layer during training, demonstrating
additional average gains of 1.0\% to 10.23\% over baseline PTMs approaches. Our
work provides strong evidence for the advancements afforded by PTMs in
recognizing nuanced emotions like Anger, Love, Fear, Joy, Sadness, and Surprise
in software engineering contexts. Through comprehensive benchmarking and error
analysis, we also outline scope for improvements to address contextual gaps
- …