MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications

Large Language Models (LLMs) have demonstrated remarkable performance across various natural language tasks, marking significant strides towards general artificial intelligence. While general artificial intelligence is leveraged by developing increasingly large-scale models, there could be another branch to develop lightweight custom models that better serve certain domains, taking into account the high cost of training and deploying LLMs and the scarcity of resources. In this paper, we present MindLLM, a novel series of bilingual lightweight large language models, trained from scratch, alleviating such burdens by offering models with 1.3 billion and 3 billion parameters. A thorough account of experiences accrued during large model development is given, covering every step of the process, including data construction, model architecture, evaluation, and applications. Such insights are hopefully valuable for fellow academics and developers. MindLLM consistently matches or surpasses the performance of other open-source larger models on some public benchmarks. We also introduce an innovative instruction tuning framework tailored for smaller models to enhance their capabilities efficiently. Moreover, we explore the application of MindLLM in specific vertical domains such as law and finance, underscoring the agility and adaptability of our lightweight models.Comment: Working in progres

Comprehensive evaluation of the water-fertilizer coupling effects on pumpkin under different irrigation volumes

Compared to conventional irrigation and fertilization, the Water-fertilizer coupling can significantly enhance the efficiency of water and fertilizer utilization, thereby promoting crop growth and increasing yield. Targeting the challenges of poor crop growth, low yield, and inefficient water and fertilizer utilization in the arid region of northwest China under conventional irrigation and fertilization practices. Therefore, a two-year on-farm experiment in 2022 and 2023 was conducted to study the effects of water-fertilizer coupling regulation on pumpkin growth, yield, water consumption (ET), and water and fertilizer use efficiency. Simultaneously the comprehensive evaluation of multiple objectives was carried out using principal component analysis (PCA) methods, so as to propose an suitable water-fertilizer coupling regulation scheme for the region. The experiment was set up as a two-factor trial using water-fertilizer integration technology under three irrigation volume (W1 = 37.5 mm, W2 = 45.5 mm, W3 = 52.5mm) and three organic fertilizer application amounts (F1 = 3900-300 kg ha-1, F2 = 4800-450 kg·ha-1, F3 = 5700-600 kg·ha-1), with the traditional irrigation and fertilization scheme from local farmers as control treatments (CK). The results indicated that irrigation volume and organic fertilizer application significantly affected pumpkin growth, yield, and water and fertilizer use efficiency (P<0.05). Pumpkin yield increased with increasing irrigation volume. Increasing organic fertilizer levels within a certain range benefited pumpkin plant growth, dry matter accumulation, and yield, however, excessive application beyond a certain level had inhibited effects on those. The increased fertilizer application under the same irrigation volume enhanced the efficiency of water and fertilizer utilization. However excessive irrigation only resulted in inefficient water consumption, reducing the water and fertilizer use efficiency. The Comprehensive evaluation by PCA revealed that the F2W3 treatment outperformed all the others, effectively addressing the triple objectives of increasing production, improving efficiency, and promoting green production. Therefore, F2W3 (Irrigation volume: 52.5 mm; Fertilizer application amounts: 4800-450 kg/ha-1) as a water and fertilizer management scheme for efficient pumpkin production in the arid region of northwest China

Immunomodulation-based strategy for improving soft tissue and metal implant integration and its implications in the development of metal soft tissue materials

The difficulties associated with metal implants and soft tissue integration have significantly affected the applications of metal implants in soft-tissue-related areas. Prompted by the close association between soft tissue integration and the immune response, an immunomodulation-based strategy is proposed to manipulate the immune microenvironment and improve metal implant–soft tissue integration. Considering their vital roles in soft tissue responses to metal implants, macrophages are used and the cytokines fingerprints of M1 and M2 macrophage immune microenvironments are evaluated for their potential modulatory effects on metal implant–soft tissue integration. The modulatory effects of different immune microenvironments on model soft tissue cells (human gingival epithelium cells) cultured on model metal implants (titanium alloy disks) are then described, with the underlying possible mechanism FAK-AKT-mTOR signaling unveiled. As further proof of concept, IL-4/PDA (polydopamine)-coated titanium alloy implants, aiming at modulating M2 macrophage polarization, are prepared and found to improve the in vivo metal implant-soft tissue integration. It is the authors' ambition that this immunomodulation-based strategy will change the negative perception and encourage the active development of metal materials with favorable soft tissue integration properties, thus improving the success rates of perforating metal implants and broadening their application in soft-tissue-related areas.</p

Sodium Fluoride under Dose Range of 2.4-24 μm, a Promising Osteoimmunomodulatory Agent for Vascularized Bone Formation

Fluoride has essential effects on bone physiological activity and is widely used in bone biomaterials modification. However, this beneficial effect is highly related to the dose range and improper dosing can lead to pathological conditions such as fluorosis of bone. Therefore, this study first investigated the dose dependent effect of fluoride on bone regeneration. In the range of 0.24–240 μM, in vivo vascularized bone formation can be achieved via fine-tuning the fluoride concentration, and the peak osteogenic effect was found at 2.4–24 μM. The underlying mechanism is related to the modulation of the osteoimmune environment. Fluoride elicited significant osteoimmunomodulatory effect in modulation of the inflammatory cytokines and expression of osteogenic factors (BMP2, OSM, spermine/spermidine) and angiogenic factor (VEGF, IGF-1) during the early response. Fluorine with the doses of 2.4 and 24 μM could increase polyamines and IGF-1 production in macrophages, thus promoting osteogenesis of BMSCs and angiogenesis of HUVECs. These doses could also inhibit the inflammatory response of macrophages. In vitro osteogenesis and angiogenesis were both improved by the fluorine (2.4 and 24 μM)/macrophage conditioned medium, which is consistent with the in vivo results. These results collectively imply that fluoride is an effective osteoimmunomodulatory agent that can regulate both osteogenesis and angiogenesis. “Osteoimmune-smart” bone biomaterials can be developed via incorporating fluorine, and the release concentration should be controlled within the range of 2.4–24 μM for improved bone formation

The osteoimmunomodulatory property of a barrier collagen membrane and its manipulation via coating nanometer-sized bioactive glass to improve guided bone regeneration

A barrier membrane is a major component of guided bone regeneration (GBR), which is traditionally viewed as a physical barrier. Due to its “foreign body” nature, the implantation of a barrier membrane would inevitably modulate immune response and subsequently affect bone dynamics, which has long been neglected. To bridge this knowledge gap, we investigated the osteoimmunomodulatory effects of barrier collagen membranes. It is found that barrier collagen membranes elicit significant effects on modulating the osteoimmune response of macrophages, by upregulating the expression of pro-inflammatory cytokines (TNFα, IL-1β, IL-6, and IL-18) and osteogenic factors (BMP2/6, WNT10b, OSM). The modulated-osteoimmune environment was beneficial for the osteogenic differentiation of BMSCs, due to the activation of BMP, canonical WNT/β-catenin, and OSM signalling pathways. The membrane-mediated osteoimmunomodulation was further modulated to show whether osteogenesis could be enhanced via manipulating the membrane-mediated osteoimmunomodulation. The membrane-mediated osteoimmune response was successfully tuned through coating the collagen membranes with nanometer-sized bioactive glass Ca2ZnSi2O7 by pulsed laser deposition, which is indicated from the change in the expression profile of inflammatory cytokines and the upregulated expression of osteogenic factors. The modulated osteoimmune environment enhanced the osteogenic differentiation of BMSCs, suggesting that collagen membranes with nanometer-sized Ca2ZnSi2O7 coating can be promising for GBR applications. These results collectively imply that barrier membranes are bioactive barriers with an osteoimmunomodulatory effect and not just physical barriers. New generation barrier membranes should be designed with a favourable osteoimmunomodulatory property

Calculating Volume of Pig Point Cloud Based on Improved Poisson Reconstruction

Pig point cloud data can be used to digitally reconstruct surface features, calculate pig body volume and estimate pig body weight. Volume, as a pig novel phenotype feature, has the following functions: (a) It can be used to estimate livestock weight based on its high correlation with body weight. (b) The volume proportion of various body parts (such as head, legs, etc.) can be obtained through point cloud segmentation, and the new phenotype information can be utilized for breeding pigs with smaller head volumes and stouter legs. However, as the pig point cloud has an irregular shape and may be partially missing, it is difficult to form a closed loop surface for volume calculation. Considering the better water tightness of Poisson reconstruction, this article adopts an improved Poisson reconstruction algorithm to reconstruct pig body point clouds, making the reconstruction results smoother, more continuous, and more complete. In the present study, standard shape point clouds, a known-volume Stanford rabbit standard model, a measured volume piglet model, and 479 sets of pig point cloud data with known body weight were adopted to confirm the accuracy and reliability of the improved Poisson reconstruction and volume calculation algorithm. Among them, the relative error was 4% in the piglet model volume result. The average absolute error was 2.664 kg in the weight estimation obtained from pig volume by collecting pig point clouds, and the average relative error was 2.478%. Concurrently, it was determined that the correlation coefficient between pig body volume and pig body weight was 0.95

Plasma polymerized bio-interface directs fibronectin adsorption and functionalization to enhance “epithelial barrier structure” formation via FN-ITG β1-FAK-mTOR signaling cascade

Background: Transepithelial medical devices are increasing utilized in clinical practices. However, the damage of continuous natural epithelial barrier has become a major risk factor for the failure of epithelium-penetrating implants. How to increase the “epithelial barrier structures” (focal adhesions, hemidesmosomes, etc.) becomes one key research aim in overcoming this difficulty. Directly targeting the in situ “epithelial barrier structures” related proteins (such as fibronectin) absorption and functionalization can be a promising way to enhance interface-epithelial integration. Methods: Herein, we fabricated three plasma polymerized bio-interfaces possessing controllable surface chemistry. Their capacity to adsorb and functionalize fibronectin (FN) from serum protein was compared by Liquid Chromatography-Tandem Mass Spectrometry. The underlying mechanisms were revealed by molecular dynamics simulation. The response of gingival epithelial cells regarding the formation of epithelial barrier structures was tested. Results: Plasma polymerized surfaces successfully directed distinguished protein adsorption profiles from serum protein pool, in which plasma polymerized allylamine (ppAA) surface favored adsorbing adhesion related proteins and could promote FN absorption and functionalization via electrostatic interactions and hydrogen bonds, thus subsequently activating the ITG β1-FAK-mTOR signaling and promoting gingival epithelial cells adhesion. Conclusion: This study offers an effective perspective to overcome the current dilemma of the inferior interface-epithelial integration by in situ protein absorption and functionalization, which may advance the development of functional transepithelial biointerfaces. Graphical Abstract: Tuning the surface chemistry by plasma polymerization can control the adsorption of fibronectin and functionalize it by exposing functional protein domains. The functionalized fibronectin can bind to human gingival epithelial cell membrane integrins to activate epithelial barrier structure related signaling pathway, which eventually enhances the formation of epithelial barrier structure.</p

Blood Prefabrication Subcutaneous Small Animal Model for the Evaluation of Bone Substitute Materials

Because of the size of bone substitute material particles, large animal bone defect models are usually required for the assessment of these materials. However, these models have several disadvantages including high cost, complicated operation procedures, ethical issues, and difficulties in sample analysis. In addition, for mimicking the bone environment, conventional subcutaneous models require the addition of osteogenic factors and stem cells, resulting in an expensive model with a complex experimental procedure. To avoid these issues, in this study, we proposed a convenient and effective blood prefabrication subcutaneous small animal model that could be applied to assess bone substitute materials. Our results demonstrated that blood prefabrication could be an economical, convenient, and useful “adhesive” for handling bone substitute particles. This process provided porcine hydroxyapatite (PHA) with a microenvironment enriched with mesenchymal stem cells and growth factors. Using this strategy, a bonelike structure could form in a rat subcutaneous pocket. Furthermore, the optimized subcutaneous model was used to evaluate the PHA’s osteoinductivity, producing results similar to those of the calvarial bone defect in terms of osteogenesis, osteoclastogenesis, and blood vessel formation. These results collectively imply that the blood prefabrication subcutaneous small animal model is convenient and effective for the assessment of osteoinductivity of bone substitute materials