Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation

In this work, we evaluate 10 open-source instructed LLMs on four representative code comprehension and generation tasks. We have the following main findings. First, for the zero-shot setting, instructed LLMs are very competitive on code comprehension and generation tasks and sometimes even better than small SOTA models specifically fine-tuned on each downstream task. We also find that larger instructed LLMs are not always better on code-related tasks. Second, for the few-shot setting, we find that adding demonstration examples substantially helps instructed LLMs perform better on most code comprehension and generation tasks; however, the examples would sometimes induce unstable or even worse performance. Furthermore, we find widely-used BM25-based shot selection strategy significantly outperforms the basic random selection or fixed selection only on generation problems. Third, for the fine-tuning setting, we find that fine-tuning could further improve the model performance on downstream code comprehension and generation tasks compared to the zero-shot/one-shot performance. In addition, after being fine-tuned on the same downstream task dataset, instructed LLMs outperform both the small SOTA models and similar-scaled LLMs without instruction tuning. Based on our findings, we further present practical implications on model and usage recommendation, performance and cost trade-offs, and future direction

ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation

In this work, we make the first attempt to evaluate LLMs in a more challenging code generation scenario, i.e. class-level code generation. We first manually construct the first class-level code generation benchmark ClassEval of 100 class-level Python code generation tasks with approximately 500 person-hours. Based on it, we then perform the first study of 11 state-of-the-art LLMs on class-level code generation. Based on our results, we have the following main findings. First, we find that all existing LLMs show much worse performance on class-level code generation compared to on standalone method-level code generation benchmarks like HumanEval; and the method-level coding ability cannot equivalently reflect the class-level coding ability among LLMs. Second, we find that GPT-4 and GPT-3.5 still exhibit dominate superior than other LLMs on class-level code generation, and the second-tier models includes Instruct-Starcoder, Instruct-Codegen, and Wizardcoder with very similar performance. Third, we find that generating the entire class all at once (i.e. holistic generation strategy) is the best generation strategy only for GPT-4 and GPT-3.5, while method-by-method generation (i.e. incremental and compositional) is better strategies for the other models with limited ability of understanding long instructions and utilizing the middle information. Lastly, we find the limited model ability of generating method-dependent code and discuss the frequent error types in generated classes. Our benchmark is available at https://github.com/FudanSELab/ClassEval

Determining Absorption, Emissivity Reduction, and Local Suppression Coefficients inside Sunspots

The power of solar acoustic waves is reduced inside sunspots mainly due to absorption, emissivity reduction, and local suppression. The coefficients of these power-reduction mechanisms can be determined by comparing time-distance cross-covariances obtained from sunspots and from the quiet Sun. By analyzing 47 active regions observed by SOHO/MDI without using signal filters, we have determined the coefficients of surface absorption, deep absorption, emissivity reduction, and local suppression. The dissipation in the quiet Sun is derived as well. All of the cross-covariances are width corrected to offset the effect of dispersion. We find that absorption is the dominant mechanism of the power deficit in sunspots for short travel distances, but gradually drops to zero at travel distances longer than about 6 degrees. The absorption in sunspot interiors is also significant. The emissivity-reduction coefficient ranges from about 0.44 to 1.00 within the umbra and 0.29 to 0.72 in the sunspot, and accounts for only about 21.5% of the umbra's and 16.5% of the sunspot's total power reduction. Local suppression is nearly constant as a function of travel distance with values of 0.80 and 0.665 for umbrae and whole sunspots respectively, and is the major cause of the power deficit at large travel distances.Comment: 14 pages, 21 Figure

Reduced Energy Metabolism Impairs T Cell-Dependent B Cell Responses in Patients With Advanced HBV-Related Cirrhosis

Background and AimsPatients with decompensated HBV-related liver cirrhosis (HBV D-LC) showed compromised immune responses, which manifested as a proneness to develop infections and hyporesponsiveness to vaccines, resulting in accelerated disease progression. The alterations in T cell-dependent B cell responses in this pathophysiological process were not well understood. This study aimed to investigate T cell-dependent B cell responses in this process and discuss the mechanism from the perspective of metabolism.MethodsChanges in phenotypes and subsets of peripheral B cells between HBV D-LC patients and healthy controls (HCs) were compared by flow cytometry. Isolated B cells were activated by coculture with circulating T follicular (cTfh) cells. After coculture, the frequencies of plasmablasts and plasma cells and immunoglobin levels were analyzed. Oxidative phosphorylation (OXPHOS) and glycolysis were analyzed by a Seahorse analyzer. Mitochondrial function and the AKT/mTOR pathway were analyzed by flow cytometry.ResultsThe proliferation and differentiation capacities of B cells after T cell stimulation were impaired in D-LC. Furthermore, we found that B cells from D-LC patients showed reductions in OXPHOS and glycolysis after activation, which may result from reduced glucose uptake, mitochondrial dysfunction and attenuated activation of the AKT/mTOR pathway.ConclusionsB cells from HBV D-LC patients showed dysfunctional energy metabolism after T cell-dependent activation. Understanding the regulations of B cell metabolic pathway and their changes may provide a new direction to rescue B cell hyporesponsiveness in patients with HBV D-LC, preventing these patients be infected and improving sensitivity to vaccines

Nano Co3O4/NiO Catalysts Pyrolysis of Cotinus nana Bark for Bio-oil and Biochemicals Raw Material

Cotinus nana W. W. Smith is a valued landscape shrub and a good afforestation species that is also widely used in the pharmaceutical industry. In this study, the use of Cotinus nana’s bark (CNB) as biofuel and a biochemical under the catalysis of nano-Co3O4/NiO was explored by various thermogravimetric methods and Fourier transform infrared (FTIR) analysis. The bark powder was extracted using a methanol/benzene solution, and then analyzed by FTIR and gas chromatography-mass spectrometry (GC-MS). The results showed that the pyrolysis products of CNB are rich in phenols, alcohols, and biofuels. The Co3O4 and NiO act as nanometal catalysts in the release of pyrolysis gases, accelerating the precipitation of gaseous products. Among them, NiO has the most obvious catalytic effect in the pyrolysis process of the material components. At the same time, in the temperature range of 40 to 850 °C, as the pyrolysis rate of the sample increases, the pyrolysis process becomes more intense. In contrast, the contents of the extracts N,N-diethyl-formamide, butyric acid, and oleic acid are not only widely used in industry, but also play a pivotal role in medicine. Therefore, the bark of Cotinus nana is an excellent plant material for biofuels and biochemicals

Stability Monitoring of Batch Processes with Iterative Learning Control

In recent years, the iterative learning control (ILC) is widely used in batch processes to improve the quality of the products. Stability is a preoccupation of batch processes when the ILC is applied. Focusing on the stability monitoring of batch processes with ILC, a method based on innerwise matrix with considering the uncertainty of the model and disturbance was proposed. First, the batch process with ILC was derived as a two-dimensional autoregressive and moving average (2D-ARMA) model. Then two kinds of stability indices are constructed based on the innerwise matrix through the identification of the 2D-ARMA. Finally, the statistical process control (SPC) chart was adopted to monitor those stability indices. Numerical results are presented to demonstrate the effectiveness of the proposed method

Research of Mechanical Fault SVM Intelligent Recognition Based on EEMD Sample Entropy

The extraction of fault information is the key of fault intelligent recognition of support vector machine for rolling bearing. Because of the non-adaptive and mode mixture of wavelet transform and empirical mode decomposition, ensemble empirical mode decomposition (EEMD) and sample entropy have been adopted to extract fault information of rolling bearing. For three kinds of conditions and pitting diameters, the vibration signal of rolling bearing has been acquired by experiment. Then by wavelet transform to reduce noise, the noise reduction signal has been decomposed into several intrinsic mode function components by EEMD, and the complexity of major components has been described by sample entropy. In addition, a SVM rolling bearing fault classification recognizer which EEMD sample entropy has been adopted as training and recognition samples is proposed. The experiment result shows that under small sample, the inner race, outer race and ball fault of bearing can be accurately recognized and the accuracy for reorganization enhance with the number of samples increasing

Molecules and functions of rosewood: Pterocarpus cambodianus

Pterocarpus is a high-end, expensive furniture materials collectively. Pterocarpus products have a certain human health function. In this paper, Pterocarpus cambodianus Pierre as an example, we study its human health components by using PY–GC–MS, TDS–GC–MS and GC–MS. The composition of known human health functions was studied by reviewing the literature. 1-Heptatriacotanol has anti-hypercholesterolemic effects. Cryptomeridiol is a natural product of anti-Alzheimer's disease and antispasmodic nature, and has a significant medicinal value. 7-Methyl-Z-tetradecen-1-ol acetate has the effect of heat and heat cough. .alpha.-Bisabolol can be used to treat leishmaniasis caused by Lactobacillus infants. Keywords: Pterocarpus, Pterocarpus cambodianus Pierre, PY–GC–MS, GC–MS, TDS–GC–MS, Health care ingredient