Breaking down a document or a conversation into multiple contiguous segments
based on its semantic structure is an important and challenging problem in NLP,
which can assist many downstream tasks. However, current works on topic
segmentation often focus on segmentation of structured texts. In this paper, we
comprehensively analyze the generalization capabilities of state-of-the-art
topic segmentation models on unstructured texts. We find that: (a) Current
strategies of pre-training on a large corpus of structured text such as
Wiki-727K do not help in transferability to unstructured conversational data.
(b) Training from scratch with only a relatively small-sized dataset of the
target unstructured domain improves the segmentation results by a significant
margin. We stress-test our proposed Topic Segmentation approach by
experimenting with multiple loss functions, in order to mitigate effects of
imbalance in unstructured conversational datasets. Our empirical evaluation
indicates that Focal Loss function is a robust alternative to Cross-Entropy and
re-weighted Cross-Entropy loss function when segmenting unstructured and
semi-structured chats.Comment: Accepted to IntelliSys 2023. arXiv admin note: substantial text
overlap with arXiv:2211.1495