Models trained on semantically related datasets and tasks exhibit comparable
inter-sample relations within their latent spaces. We investigate in this study
the aggregation of such latent spaces to create a unified space encompassing
the combined information. To this end, we introduce Relative Latent Space
Aggregation, a two-step approach that first renders the spaces comparable using
relative representations, and then aggregates them via a simple mean. We
carefully divide a classification problem into a series of learning tasks under
three different settings: sharing samples, classes, or neither. We then train a
model on each task and aggregate the resulting latent spaces. We compare the
aggregated space with that derived from an end-to-end model trained over all
tasks and show that the two spaces are similar. We then observe that the
aggregated space is better suited for classification, and empirically
demonstrate that it is due to the unique imprints left by task-specific
embedders within the representations. We finally test our framework in
scenarios where no shared region exists and show that it can still be used to
merge the spaces, albeit with diminished benefits over naive merging.Comment: To appear in the NeurReps workshop @ NeurIPS 202