2 research outputs found
Tokenizer Choice For LLM Training: Negligible or Crucial?
The recent success of LLMs has been predominantly driven by curating the
training dataset composition, scaling of model architectures and dataset sizes
and advancements in pretraining objectives, leaving tokenizer influence as a
blind spot. Shedding light on this underexplored area, we conduct a
comprehensive study on the influence of tokenizer choice on LLM downstream
performance by training 24 mono- and multilingual LLMs at a 2.6B parameter
scale, ablating different tokenizer algorithms and parameterizations. Our
studies highlight that the tokenizer choice can significantly impact the
model's downstream performance, training and inference costs. In particular, we
find that the common tokenizer evaluation metrics fertility and parity are not
always predictive of model downstream performance, rendering these metrics a
questionable proxy for the model's downstream performance. Furthermore, we show
that multilingual tokenizers trained on the five most frequent European
languages require vocabulary size increases of factor three in comparison to
English. While English-only tokenizers have been applied to the training of
multi-lingual LLMs, we find that this approach results in a severe downstream
performance degradation and additional training costs of up to 68%, due to an
inefficient tokenization vocabulary
Dosimetry and optimal scan time of 18FSiTATE-PET/CT in patients with neuroendocrine tumours
PURPOSE Radiolabelled somatostatin analogues targeting somatostatin receptors (SSR) are well established for combined positron emission tomography/computer tomography (PET/CT) imaging of neuroendocrine tumours (NET). 18FSiTATE has recently been introduced showing high image quality, promising clinical performance and improved logistics compared to the clinical reference standard 68Ga-DOTA-TOC. Here we present the first dosimetry and optimal scan time analysis. METHODS Eight NET patients received a 18FSiTATE-PET/CT (250 ± 66~MBq) with repeated emission scans (10, 30, 60, 120, 180~min after injection). Biodistribution in normal organs and SSR-positive tumour uptake were assessed. Dosimetry estimates for risk organs were determined using a combined linear-monoexponential model, and by applying 18F S-values and reference target masses for the ICRP89 adult male or female (OLINDA 2.0). Tumour-to-background ratios were compared quantitatively and visually between different scan times. RESULTS After 1 h, normal organs showed similar tracer uptake with only negligible changes until 3 h post-injection. In contrast, tracer uptake by tumours increased progressively for almost all types of metastases, thus increasing tumour-to-background ratios over time. Dosimetry resulted in a total effective dose of 0.015 ± 0.004~mSv/MBq. Visual evaluation revealed no clinically relevant discrepancies between later scan times, but image quality was rated highest in 60 and 120~min images. CONCLUSION 18FSiTATE-PET/CT in NET shows overall high tumour-to-background ratios from 60 to 180~min after injection and an effective dose comparable to 68Ga-labelled alternatives. For clinical use of 18FSiTATE, the best compromise between image quality and tumour-to-background contrast is reached at 120~min, followed by 60~min after injection