Scaling Law for Recommendation Models: Towards General-purpose User Representations

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.Comment: Accepted at AAAI 2023. This version includes the technical appendi

Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learning

Recent studies have proposed unified user modeling frameworks that leverage user behavior data from various applications. Many of them benefit from utilizing users' behavior sequences as plain texts, representing rich information in any domain or system without losing generality. Hence, a question arises: Can language modeling for user history corpus help improve recommender systems? While its versatile usability has been widely investigated in many domains, its applications to recommender systems still remain underexplored. We show that language modeling applied directly to task-specific user histories achieves excellent results on diverse recommendation tasks. Also, leveraging additional task-agnostic user histories delivers significant performance benefits. We further demonstrate that our approach can provide promising transfer learning capabilities for a broad spectrum of real-world recommender systems, even on unseen domains and services.Comment: 14 pages, 5 figures, 9 table

Miss rate of colorectal neoplastic polyps and risk factors for missed polyps in consecutive colonoscopies

Background/Aims: Colonoscopic polypectomy is the best diagnostic and therapeutic tool to detect and prevent colorectal neoplasms. However, previous studies have reported that 17% to 28% of colorectal polyps are missed during colonoscopy. We investigated the miss rate of neoplastic polyps and the factors associated with missed polyps from quality-adjusted consecutive colonoscopies.Methods: We reviewed the medical records of patients who were found to have colorectal polyps at a medical examination center of the Kangbuk Samsung Hospital between March 2012 and February 2013. Patients who were referred to a single tertiary academic medical center and underwent colonoscopic polypectomy on the same day were enrolled in our study. The odds ratios (ORs) associated with polyp-related and patient-related factors were evaluated using logistic regression analyses.Results: A total of 463 patients and 1,294 neoplastic polyps were analyzed. The miss rates for adenomas, advanced adenomas, and carcinomas were 24.1% (312/1,294), 1.2% (15/1,294), and 0% (0/1,294), respectively. Flat/sessile-shaped adenomas (adjusted OR, 3.62; 95% confidence interval [CI], 2.40–5.46) and smaller adenomas (adjusted OR, 5.63; 95% CI, 2.84– 11.15 for ≤5 mm; adjusted OR, 3.18; 95% CI, 1.60–6.30 for 6–9 mm, respectively) were more frequently missed than pedunculated/sub-pedunculated adenomas and larger adenomas. In patients with 2 or more polyps compared with only one detected (adjusted OR, 2.37; 95% CI, 1.55–3.61 for 2–4 polyps; adjusted OR, 11.52; 95% CI, 4.61–28.79 for ≥5 polyps, respectively) during the first endoscopy, the risk of missing an additional polyp was significantly higher.Conclusions: One-quarter of neoplastic polyps was missed during colonoscopy. We encourage endoscopists to detect smaller and flat or sessile polyps by using the optimal withdrawal technique