Berragan, Cillian

Calafiore, Alessia

Morley, Jeremy

Singleton, Alex

English

Social interaction is typically studied from the context of physical movement, where geographic distance and ease of connectivity influence the strength of interaction between regions. From the point of view of social media networks however, these limitations appear to still persist, despite interactions not being reliant on physical movement, suggesting non-physical geographic characteristics influence interaction between social communities. Unlike geotags, which provide explicit geographic information about social media users as coordinates, unstructured text presents an alternative perspective for the study of social interaction between regions, instead allowing for the comparison between the language used when mentioning locations in context. Our paper analyses the corpora associated with major cities across the UK, first vectorising Reddit comments through transformer-based embeddings, which capture semantic information, then using these to establish unsupervised clusters and similarity between them. We find that distinct groups emerge which broadly conform with established regional identities of locations across the UK, but with interesting deviations

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Evaluating the similarity of location-based corpora identified inReddit commentsCitation for published version:Berragan, C, Singleton, A, Calafiore, A & Morley, J 2023, Evaluating the similarity of location-based corporaidentified in Reddit comments. in X Hu, Y Hu, B Resch, J Kersten & K Stock (eds), Proceedings of the FirstWorkshop on Geographic Information Extraction from Texts (GeoExT 2023) co-located with The 45thEuropean Conference on Information Retrieval (ECIR 2023). vol. 3385, CEUR Workshop Proceedings,CEUR Workshop Proceedings, pp. 1-6, 1st Workshop on Geographic Information Extraction from Texts,GeoExT 2023, Dublin, Ireland, 2/04/23. <https://ceur-ws.org/Vol-3385/paper1.pdf>Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher's PDF, also known as Version of recordPublished In:Proceedings of the First Workshop on Geographic Information Extraction from Texts (GeoExT 2023) co-locatedwith The 45th European Conference on Information Retrieval (ECIR 2023)General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 23. Nov. 2023Evaluating the Similarity of Location-based CorporaIdentified in Reddit CommentsCillian Berragan1, Alex Singleton1, Alessia Calafiore2 and Jeremy Morley31University of Liverpool, Liverpool, L69 3BX, United Kingdom2University of Edinburgh, Edinburgh, EH8 9YL, United Kingdom3Ordnance Survey, Southampton, SO16 0AS, United KingdomAbstractSocial interaction is typically studied from the context of physical movement, where geographic dis-tance and ease of connectivity influence the strength of interaction between regions. From the pointof view of social media networks however, these limitations appear to still persist, despite interactionsnot being reliant on physical movement, suggesting non-physical geographic characteristics influenceinteraction between social communities. Unlike geotags, which provide explicit geographic informationabout social media users as coordinates, unstructured text presents an alternative perspective for thestudy of social interaction between regions, instead allowing for the comparison between the languageused whenmentioning locations in context. Our paper analyses the corpora associated with major citiesacross the UK, first vectorising Reddit comments through transformer-based embeddings, which cap-ture semantic information, then using these to establish unsupervised clusters and similarity betweenthem. We find that distinct groups emerge which broadly conform with established regional identitiesof locations across the UK, but with interesting deviations.KeywordsSocial media, Natural Language Processing, Social Interaction1. IntroductionSocial interaction is typically studied in the context of mobility, using data sources like Censusor transport records, where physical movement is restricted by distance and ease of connectiv-ity between two locations [1, 2]. In contrast to this, social interaction has also been studied us-ing phone call data [3], and social media networks [4], where the spatial and temporal boundsof connectivity between two locations does not restrict interactions. Despite this however,many studies have found that geographic identities within communities still persist in thesenetworks, with interaction strength influenced by the geographic distance between them [5, 6].Social media also presents rich semantic information regarding locations through text as-sociated with geotagged social media posts. Comparative analysis of corpora associated withGeoExt 2023: First International Workshop on Geographic Information Extraction from Texts at ECIR 2023, April 2,2023, Dublin, IrelandC.Berragan@liverpool.ac.uk (C. Berragan); ucfnale@liverpool.ac.uk (A. Singleton); calafio@ed.ac.uk(A. Calafiore); jeremy.morley@os.uk (J. Morley)0000-0003-2198-2245 (C. Berragan); 0000-0002-2338-2334 (A. Singleton); 0000-0002-5953-2891 (A. Calafiore);0000-0002-3658-8796 (J. Morley)© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons Licence Attribution 4.0 International (CC BY 4.0).CEURWorkshopProceedingshttp://ceur-ws.orgISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org)geotagged locations similarly exhibit regionality; for example, tweets from the North East ofEngland are statistically different compared with the South [5].Our paper explores the similarity of corpora with respect to locational mentions from datataken directly from text, without relying on geotagged metadata. This approach offers an alter-native perspective for the analysis of social interaction, built directly from the semantic infor-mation associated with locations, rather than the location associated with social media usersthemselves. Collective semantic information from social media embeds the regional identityof locations across a continuous spectrum, allowing for the direct comparison between theseidentities and their relationships.2. MethodologyThe following section gives an overview of our data source and the data processing methodol-ogy used in our paper. All code, analysis and data are available on our DagsHub repository.Reddit is a public discussion, news aggregation social network, among the top 20 most vis-ited websites in the United Kingdom. As of 2020, Reddit had around 430 million active monthlyusers, comparable to the number of Twitter users [7, 8]. Reddit is divided into separate inde-pendent subreddits each with specific topics of discussion, where usersmay submit posts whicheach have dedicated nested conversation threads that users can add comments to. Subredditscover a wide range of topics, and in the interest of geography, they also act as forums for thediscussion of local places. The United Kingdom subreddit acts as a general hub for relatedtopics, notably including a list of smaller and more specific related subreddits. This list pro-vides a ‘Places’ section, a collection of local British subreddits, ranging in scale from countrylevel (/r/England), regional (/r/thenorth, /r/Teeside), to cities (/r/Manchester) and smalltowns (/r/Alnwick). In total there are 213 subreddits that relate to ‘places’ within the UnitedKingdom1. For each subreddit, every single historic comment was retrieved using the PushshiftReddit archive [9]. In total 8,282,331 comments were extracted, submitted by 490,535 uniqueusers, between 2011-01-01 and 2022-04-17.We extracted and geolocated all place names in this collection of comments using a cus-tom built geoparsing pipeline. To identify place names, we used a BERT transformer-basedNER model trained on the WNUT 2017 dataset [10], available on the HuggingFace Model Hub.We then implemented a disambiguation methodology using contextual place names and twogazetteers to geolocate place names; OS Open Names and ‘natural’ location types from theGazetteer of British Place Names. Processed comments consist of a collection of geolocatedplace names, alongside their natural language context sentence.2.1. Similarity of Place CorporaComparing the similarity between two or more distinct texts first relies on an appropriatemethod for processing the text into a numerical format. For each location we obtained a corpusof comments, consisting of sentences where each location is mentioned. These were thenprocessed into a single vector, reflecting the semantic information attributed with locations.1https://www.reddit.com/r/unitedkingdom/wiki/british_subredditsTypically, a TF-IDF approach is used to generate document embeddings [11], howeverwe foundcomparative analysis between embeddings did not always provide insightful information. Eachvector shared similar properties, giving cosine similarities which did not result in any distinctvariation between locations. This is likely a problem with the language between locationssharing similar properties, meaning the more nuanced semantic information is not capturedthrough a TF-IDF method.We therefore extracted embeddings from a deep neural network called a transformer. UnlikeTF-IDF or simpler neural network models, transformers are able to use contextual informationto generate word embeddings, meaning the same word in two different contexts will not sharethe exact same vector, capturing different embedded semantic information [12]. Additionally,transformers are pre-trained on a large corpus of text, meaning general information regardingthe English language is already embedded within the model, allowing for improved under-standing of semantic information. These core features mean that embeddings generated fromtransformers are likely to capture information that allows for more the accurate comparativeanalysis. We generated embeddings using the all-mpnet-base-v2model from the sentence-transformers library in Python [13]. Unlike a standard ‘BERT’-like transformer, this libraryimplements modifications to base models that more appropriately captures semantic informa-tion in their output embeddings.Before calculating embeddings we first masked every mention of a location with a generictoken ‘PLACE’, this ensured that when analysing embeddings, no explicit geographic informa-tionwas captured accidentally. For example, Manchester and Liverpool maymentionmatchinglocations frequently in each of their comments because they are geographically close. To bothremove noise and reduce the computational requirements for this work, only locations withover 10,000 unique mentions were kept, from these a random sample of 1,000 comments wereselected for each. Once embeddings were generated for every comment in each city corpus,the mean for each corpus was generated, giving a vector 768 decimal values for each city.With a single vector for each selected location, we first calculated K-Means clusters to de-termine whether distinct groupings of locations could be identified across the UK. To visualisethese clusters we used a PCA decomposition to reduce the dimensionality from 768 down to2 dimensions. Finally, we calculated the cosine similarity between each and every locationvector.3. Results & DiscussionFigure 1 gives K Means clusters for transformer embeddings decomposed into two dimensionswith 𝑘 = 5. These Clusters show corpora that share similar semantic properties, however, itis worth noting that while points that are closer together likely indicate increased similarity,the position of these points reflect PCA decomposed values, which capture less informationcompared with the clusters calculated on non-decomposed vectors. Notably London appearsas a single value in a cluster, suggesting the corpus associated with the capital of the UK issemantically distinct from the rest of the country. There is also a single cluster associated withthe four Scottish cities considered in our study (Cluster 1), as well as a cluster for Cambridgeand Oxford (Cluster 5). Figure 1 (B) reveals that clusters do broadly appear to reflect distance-NewcastleEdinburghSheffieldBristolNottinghamLiverpoolLeedsLondonDundeeOxfordCambridgeBrightonManchesterBirminghamAberdeenGlasgow(A)1 2 3 4 5(B)Figure 1: Average transformer vector associated with each location corpus coloured by K Means clus-ters where 𝐾 = 5. (A) PCA decomposed into 2 dimensions. (B) Visualised with their easting andnorthing coordinates.restricted geographic properties, while also capturing some divergences from this, with loca-tions like London, Newcastle, Bristol and Brighton geographically distant from locations theyshare clusters with.With our high dimensional transformer embeddings we compare the cosine similarity be-tween them on Figure 2. The highest and lowest similarity score for each location is high-lighted in red and green respectively. As with Figure 1, corpora in Scottish cities appear tolargely share similarities, with Glasgow and Edinburgh sharing their highest similarity values.The city with the lowest similarity to the most other locations is Oxford, which shares low val-ues with cities in Scotland, as well as Liverpool and Manchester. London again stands out, withoverall very low similarities with all other cities, but the highest similarity with Manchester.4. ConclusionOur paper demonstrates the ability to compare Reddit comments relating to cities across theUK, using document embeddings generated from a transformer neural network. Instead offocussing on physical interactions between people or social media interactions, our work iden-tifies relationships between cities through their semantic footprint, and analysing each corpuscomputationally allows for direct comparisons between cities through clustering and cosinesimilarity.Our analysis reveals distinct clusters which largely reflect geographic proximity of locations,however, interesting deviations from proximity do emerge. Oxford and Cambridge are bothclustered and share a high cosine similarity, but generate the lowest similarity with many otherlocations in the UK, including London. London in particular appears distinct from the rest ofAberdeenBirminghamBrightonBristolCambridgeDundeeEdinburghGlasgowLeedsLiverpoolLondonManchesterNewcastleNottinghamOxfordSheffieldAberdeenBirminghamBrightonBristolCambridgeDundeeEdinburghGlasgowLeedsLiverpoolLondonManchesterNewcastleNottinghamOxfordSheffieldFigure 2: Cosine similarity between each and every location related transformer vector embedding.Values scaled between 0 and 1. Green highlights indicate the highest value in each row, while redindicates the lowest value in each row.the UK, while cities that are not geographically close exhibit clustering and high similarity,such as Liverpool and Newcastle.The information generated through our work presents an alternative view of relationshipsbetween cities that are not captured by existing data sources, all of which rely on explicit ge-ographic coordinate information. Instead, we build similarities and clusters directly from thesemantic information that exists within their respective corpora. Unlike traditional data, whichcaptures objective social interactions between regions, the deviations from the restriction ofgeographic distance between several cities in our work appears to reflect the more subjectivelanguage that shapes the cultural and perceived identity of regions, and the relationships be-tween them.While our work enables the direct numerical comparison between city-based corpora, itcannot explain the similarities and dissimilarities between them. Additional work may explorethe use of topic-modelling to identify shared topics between locations, and differences in thesentiment towards these topics may explain dissimilarity.References[1] A. Rae, From spatial interaction data to spatial interaction information? Geovisualisa-tion and spatial structures of migration from the 2001 UK census, Computers, Environ-ment and Urban Systems 33 (2009) 161–178. doi:10.1016/j.compenvurbsys.2009.01.007.[2] H. Titheridge, K. Achuthan, R. L. Mackett, J. Solomon, Assessing the extent of transportsocial exclusion among the elderly, Journal of Transport and Land Use 2 (2009). doi:10.5198/jtlu.v2i2.44.[3] S. Sobolevsky, M. Szell, R. Campari, T. Couronné, Z. Smoreda, C. Ratti, Delineating Geo-graphical Regions with Networks of Human Interactions in an Extensive Set of Countries,PLoS ONE 8 (2013) e81707. doi:10.1371/journal.pone.0081707.[4] B. Lengyel, A. Varga, B. Ságvári, Á. Jakobi, J. Kertész, Geographies of an Online SocialNetwork, PLOS ONE 10 (2015) e0137248. doi:10.1371/journal.pone.0137248.[5] R. Arthur, H. T. P. Williams, The human geography of Twitter: Quantifying regionalidentity and inter-region communication in England and Wales, PLOS ONE 14 (2019)e0214466. doi:10.1371/journal.pone.0214466.[6] C. Ratti, S. Sobolevsky, F. Calabrese, C. Andris, J. Reades, M. Martino, R. Claxton, S. H.Strogatz, Redrawing the Map of Great Britain from a Network of Human Interactions,PLoS ONE 5 (2010) e14248. doi:10.1371/journal.pone.0014248.[7] N. Murphy, Reddit’s 2019 Year in Review - Upvoted,https://www.redditinc.com/blog/reddits-2019-year-in-review/#content, 2019.[8] Statista, Most popular social networksworldwide as of January 2022, ranked by number ofmonthly active users, https://www.statista.com/statistics/272014/global-social-networks-ranked-by-number-of-users/, 2022.[9] J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, J. Blackburn, The Pushshift RedditDataset, 2020. arXiv:arXiv:2001.08435.[10] L. Derczynski, E. Nichols, M. van Erp, N. Limsopatham, Results of theWNUT2017 SharedTask on Novel and Emerging Entity Recognition, in: Proceedings of the 3rd Workshopon Noisy User-generated Text, Association for Computational Linguistics, Copenhagen,Denmark, 2017, pp. 140–147. doi:10.18653/v1/W17-4418.[11] J. Daniel, M. James H, Speech and Language Processing: An Introduction to NaturalLanguage Processing, Computational Linguistics, and Speech Recognition, prentice hall,2007.[12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polo-sukhin, Attention Is All You Need, arXiv:1706.03762 [cs] (2017). arXiv:1706.03762.[13] N. Reimers, I. Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Lan-guage Processing and the 9th International Joint Conference on Natural Language Pro-cessing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China,2019, pp. 3980–3990. doi:10.18653/v1/D19-1410.

Evaluating the similarity of location-based corpora identified in Reddit comments

https://www.research.ed.ac.uk/files/391061256/Calafiore2023GIE23Evaluating.pdf

Evaluating the similarity of location-based corpora identified in Reddit comments

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer