This research focuses on assessing the ability of large language models
(LLMs) in representing geometries and their spatial relations. We utilize LLMs
including GPT-2 and BERT to encode the well-known text (WKT) format of
geometries and then feed their embeddings into classifiers and regressors to
evaluate the effectiveness of the LLMs-generated embeddings for geometric
attributes. The experiments demonstrate that while the LLMs-generated
embeddings can preserve geometry types and capture some spatial relations (up
to 73% accuracy), challenges remain in estimating numeric values and retrieving
spatially related objects. This research highlights the need for improvement in
terms of capturing the nuances and complexities of the underlying geospatial
data and integrating domain knowledge to support various GeoAI applications
using foundation models