Aligning Large Language Models through Synthetic Feedback

Aligning large language models (LLMs) to human values has become increasingly important as it enables sophisticated steering of LLMs, e.g., making them follow given instructions while keeping them less toxic. However, it requires a significant amount of human demonstrations and feedback. Recently, open-sourced models have attempted to replicate the alignment learning process by distilling data from already aligned LLMs like InstructGPT or ChatGPT. While this process reduces human efforts, constructing these datasets has a heavy dependency on the teacher models. In this work, we propose a novel framework for alignment learning with almost no human labor and no dependency on pre-aligned LLMs. First, we perform reward modeling (RM) with synthetic feedback by contrasting responses from vanilla LLMs with various sizes and prompts. Then, we use the RM for simulating high-quality demonstrations to train a supervised policy and for further optimizing the model with reinforcement learning. Our resulting model, Aligned Language Model with Synthetic Training dataset (ALMoST), outperforms open-sourced models, including Alpaca, Dolly, and OpenAssistant, which are trained on the outputs of InstructGPT or human-annotated instructions. Our 7B-sized model outperforms the 12-13B models in the A/B tests using GPT-4 as the judge with about 75% winning rate on average.Comment: Preprint, 9 pages (with 10 pages of supplementary

Exploring the utility of alcohol flushing as an instrumental variable for alcohol intake in Koreans

Abstract Previous studies have indicated an association of higher alcohol intake with cardiovascular disease and related traits, but causation has not been definitively established. In this study, the causal effect of alcohol intake on hypertension in 2,011 men and women from the Ansan-Ansung cohort was estimated using an instrumental variable (IV) approach, with both a phenotypic and genotypic instrument for alcohol intake: alcohol flushing and the rs671 genotype (in the alcohol dehydrogenase 2 [ALDH2] gene), respectively. Both alcohol flushing and the rs671 genotype were associated with alcohol intake (difference in alcohol intake with alcohol flushers vs. non-flushers: −9.07 g/day; 95% confidence interval [CI]: −11.12, −7.02; P-value: 8.3 × 10−18 and with the rs671 GA + AA vs. GG genotype: −7.94 g/day; 95% CI: −10.20, −5.69; P-value: 6.1 × 10−12). An increase in alcohol intake, as predicted by both the absence of alcohol flushing and the presence of the rs671 GG genotype in the IV analyses, was associated with an increase in blood pressure in men from this Korean population. In conclusion, this study supports a causal effect of alcohol intake on hypertension and indicated that alcohol flushing may be a valid proxy for the ALDH2 rs671 polymorphism, which influences alcohol intake in this Korean population

Aberrant Pyramidal Tract in Comparison with Pyramidal Tract on Diffusion Tensor Tractography: A Mini-Review

The pyramidal tract (PT) is a major neural tract that controls voluntary movements in the human brain. The PT has several collateral pathways, including the aberrant pyramidal tract (APT), which passes through the medial lemniscus location at the midbrain and pons. Diffusion tensor tractography (DTT) allows visualization and estimation of the APT in three dimensions. In this mini-review, eight DTT studies on the APT were reviewed. Two studies for normal subjects reported the prevalence (17–18% of hemispheres) and the different characteristics (different cortical origin, less directionality, and fewer neural fibers) of the APT compared with the PT. Six studies reported on the APT in patients with cerebral infarct, traumatic brain injury, and cerebral palsy and suggested that the APT could contribute to motor recovery following brain injury. The research on the APT in patients with brain injury has important implications for neuro-rehabilitation because understanding of the motor recovery mechanism can provide the basis for scientific rehabilitation strategies. Therefore, studies involving various brain pathologies with large numbers of patients on this topic should be encouraged. In addition, further studies are needed on the exact role of the APT in normal subjects

Compression of the Lateral Antebrachial Cutaneous Nerve due to Leakage of Iron after an Intravenous Iron Infusion

Skin staining due to iron leakage into the subcutaneous tissue can sometimes occur during intravenous iron infusion. We describe a case of lateral antebrachial cutaneous nerve (LACN) entrapment due to extravasated iron after an intravenous iron infusion. A 41-year-old woman received an intravenous ferric carboxymaltose infusion for iron deficiency anemia. However, during the infusion, extravasation of iron occurred and brown pigmentation developed on the lateral side of the cubital fossa. Sixteen months later, the patient still had some staining in her anterolateral elbow and proximal forearm. In addition, she complained of tingling pain over her left forearm. Ultrasonography (US) revealed a lateral antebrachial cutaneous nerve (LACN) under the stained area. When we swept the stained area with the US transducer, she reported a tingling pain on her left lateral forearm, the region innervated by the left LACN. Therefore, we considered that the pain resulted from the compression of the left LACN by the leaked iron during the intravenous infusion. Leaked iron can compress the cutaneous nerve and result in neuropathic pain and cosmetic problems. When patients with skin staining after iron infusion have neuropathic pain, clinicians should consider the possibility of entrapment of the cutaneous nerves

Impaired consciousness due to injury of the ascending reticular activating system in a patient with bilateral pontine infarction: A case report

The ascending reticular activating system (ARAS) is known to play an essential role in maintaining arousal and consciousness. In this report, we describe the case of a patient with impaired consciousness due to injury of the ARAS after bilateral pontine infarction. A 73-year-old female patient presented with anterior chest pain to the Emergency Department of our university hospital. She was diagnosed with chronic stable angina pectoris, three-vessel disease, and chronic total occlusion of the left anterior descending artery by coronary angiography and received conservative treatment. After five days, she showed deep drowsy mentality and brain MRI revealed bilateral paramedian pontine infarction. Four weeks after the pontine infarction, she showed severely impaired consciousness, with a Glasgow Coma Scale score of 7 (eye-opening: 2, best verbal response: 2, and best motor response: 3). Coma Recovery Scale-Revised score was 10 (auditory function: 2, visual function: 3, motor function: 2, verbal function: 2, communication: 0, and arousal: 1). Results of diffusion tensor tractography (DTT) for the ARAS showed decreased neural connectivity in the left lower dorsal ARAS, both lower ventral ARAS, and both upper ARAS. To the best of our knowledge, this is the first report of injury to the ARAS in bilateral pontine infarction diagnosed by DTT. We presume that our report would provide clinicians a better understanding of the mechanism of impaired consciousness in patients with pontine infarction

Directional Motion on Water Surface with Keel Extruded Footpads Propelled by Marangoni Effect

Marangoni propulsion is well-known aquatic locomotion exhibited by some aquatic insects. After secreting chemicals to reduce the surface tension nearby, they are rapidly propelled by the force originated from the surface tension gradient. Inspired by their locomotion, a water surface skimming device was implemented in this work. It could obtain thrust by periodically dripping alcohol behind to utilize Marangoni effect. In particular, this work firstly proposed a passively triggerable vessel to explore a new way of initiating Marangoni propulsion, which does not require electric power to store and release alcohol fuel. From the experimental analysis, it was found that using a Teflon tube not only facilitated periodic dripping, but it also caused the dripping period and the number of droplets become less sensible to the tilting angle of the vessel. In addition, a new model to calculate the supporting force of a footpad was proposed, and the modeling error of a circular and an elliptical footpad was 3.54% and 4.36%, respectively. Those footpads were also extruded with keel to demonstrate passive directional motion without an active controller, which exhibited 30.9 mm of lateral deviation per 1 m of forward movement. The entire device weight was 11 g, and the estimated maximum moving distance was 9 m

CNN-Based Acoustic Scene Classification System

Acoustic scene classification (ASC) categorizes an audio file based on the environment in which it has been recorded. This has long been studied in the detection and classification of acoustic scenes and events (DCASE). This presents the solution to Task 1 of the DCASE 2020 challenge submitted by the Chung-Ang University team. Task 1 addressed two challenges that ASC faces in real-world applications. One is that the audio recorded using different recording devices should be classified in general, and the other is that the model used should have low-complexity. We proposed two models to overcome the aforementioned problems. First, a more general classification model was proposed by combining the harmonic-percussive source separation (HPSS) and deltas-deltadeltas features with four different models. Second, using the same feature, depthwise separable convolution was applied to the Convolutional layer to develop a low-complexity model. Moreover, using gradient-weight class activation mapping (Grad-CAM), we investigated what part of the feature our model sees and identifies. Our proposed system ranked 9th and 7th in the competition for these two subtasks, respectively