Empowering LLM to use Smartphone for Intelligent Task Automation

Mobile task automation is an attractive technique that aims to enable voice-based hands-free user interaction with smartphones. However, existing approaches suffer from poor scalability due to the limited language understanding ability and the non-trivial manual efforts required from developers or end-users. The recent advance of large language models (LLMs) in language understanding and reasoning inspires us to rethink the problem from a model-centric perspective, where task preparation, comprehension, and execution are handled by a unified language model. In this work, we introduce AutoDroid, a mobile task automation system that can handle arbitrary tasks on any Android application without manual efforts. The key insight is to combine the commonsense knowledge of LLMs and domain-specific knowledge of apps through automated dynamic analysis. The main components include a functionality-aware UI representation method that bridges the UI with the LLM, exploration-based memory injection techniques that augment the app-specific domain knowledge of LLM, and a multi-granularity query optimization module that reduces the cost of model inference. We integrate AutoDroid with off-the-shelf LLMs including online GPT-4/GPT-3.5 and on-device Vicuna, and evaluate its performance on a new benchmark for memory-augmented Android task automation with 158 common tasks. The results demonstrated that AutoDroid is able to precisely generate actions with an accuracy of 90.9%, and complete tasks with a success rate of 71.3%, outperforming the GPT-4-powered baselines by 36.4% and 39.7%. The demo, benchmark suites, and source code of AutoDroid will be released at url{https://autodroid-sys.github.io/}

Facilitating Self-monitored Physical Rehabilitation with Virtual Reality and Haptic feedback

Physical rehabilitation is essential to recovery from joint replacement operations. As a representation, total knee arthroplasty (TKA) requires patients to conduct intensive physical exercises to regain the knee's range of motion and muscle strength. However, current joint replacement physical rehabilitation methods rely highly on therapists for supervision, and existing computer-assisted systems lack consideration for enabling self-monitoring, making at-home physical rehabilitation difficult. In this paper, we investigated design recommendations that would enable self-monitored rehabilitation through clinical observations and focus group interviews with doctors and therapists. With this knowledge, we further explored Virtual Reality(VR)-based visual presentation and supplemental haptic motion guidance features in our implementation VReHab, a self-monitored and multimodal physical rehabilitation system with VR and vibrotactile and pneumatic feedback in a TKA rehabilitation context. We found that the third point of view real-time reconstructed motion on a virtual avatar overlaid with the target pose effectively provides motion awareness and guidance while haptic feedback helps enhance users' motion accuracy and stability. Finally, we implemented \systemname to facilitate self-monitored post-operative exercises and validated its effectiveness through a clinical study with 10 patients

UbiPhysio: Support Daily Functioning, Fitness, and Rehabilitation with Action Understanding and Feedback in Natural Language

We introduce UbiPhysio, a milestone framework that delivers fine-grained action description and feedback in natural language to support people's daily functioning, fitness, and rehabilitation activities. This expert-like capability assists users in properly executing actions and maintaining engagement in remote fitness and rehabilitation programs. Specifically, the proposed UbiPhysio framework comprises a fine-grained action descriptor and a knowledge retrieval-enhanced feedback module. The action descriptor translates action data, represented by a set of biomechanical movement features we designed based on clinical priors, into textual descriptions of action types and potential movement patterns. Building on physiotherapeutic domain knowledge, the feedback module provides clear and engaging expert feedback. We evaluated UbiPhysio's performance through extensive experiments with data from 104 diverse participants, collected in a home-like setting during 25 types of everyday activities and exercises. We assessed the quality of the language output under different tuning strategies using standard benchmarks. We conducted a user study to gather insights from clinical experts and potential users on our framework. Our initial tests show promise for deploying UbiPhysio in real-life settings without specialized devices.Comment: 27 pages, 14 figures, 5 table

MindShift: Leveraging Large Language Models for Mental-States-Based Problematic Smartphone Use Intervention

Problematic smartphone use negatively affects physical and mental health. Despite the wide range of prior research, existing persuasive techniques are not flexible enough to provide dynamic persuasion content based on users' physical contexts and mental states. We first conduct a Wizard-of-Oz study (N=12) and an interview study (N=10) to summarize the mental states behind problematic smartphone use: boredom, stress, and inertia. This informs our design of four persuasion strategies: understanding, comforting, evoking, and scaffolding habits. We leverage large language models (LLMs) to enable the automatic and dynamic generation of effective persuasion content. We develop MindShift, a novel LLM-powered problematic smartphone use intervention technique. MindShift takes users' in-the-moment physical contexts, mental states, app usage behaviors, users' goals & habits as input, and generates high-quality and flexible persuasive content with appropriate persuasion strategies. We conduct a 5-week field experiment (N=25) to compare MindShift with baseline techniques. The results show that MindShift significantly improves intervention acceptance rates by 17.8-22.5% and reduces smartphone use frequency by 12.1-14.4%. Moreover, users have a significant drop in smartphone addiction scale scores and a rise in self-efficacy. Our study sheds light on the potential of leveraging LLMs for context-aware persuasion in other behavior change domains

Meta-analysis of the effects of 1-methylcyclopropene (1-MCP) treatment on climacteric fruit ripening

1-Methylcyclopropene (1-MCP) is an inhibitor of ethylene perception that is widely used to maintain the quality of several climacteric fruits during storage. A large body of literature now exists on the effects of 1-MCP on climacteric fruit ripening for different species and environmental conditions, presenting an opportunity to use meta-analysis to systematically dissect these effects. We classified 44 ripening indicators of climacteric fruits into five categories: physiology and biochemistry, quality, enzyme activity, color, and volatiles. Meta-analysis showed that 1-MCP treatment reduced 20 of the 44 indicators by a minimum of 22% and increased 6 indicators by at least 20%. These effects were associated with positive effects on delaying ripening and maintaining quality. Of the seven moderating variables, species, 1-MCP concentration, storage temperature and time had substantial impacts on the responses of fruit to 1-MCP treatment. Fruits from different species varied in their responses to 1-MCP, with the most pronounced responses observed in rosaceous fruits, especially apple, European pear fruits, and tropical fruits. The effect of gaseous 1-MCP was optimal at 1 μl/l, with a treatment time of 12–24 h, when the storage temperature was 0 °C for temperate fruits or 20 °C for tropical fruits, and when the shelf temperature was 20 °C, reflecting the majority of experimental approaches. These findings will help improve the efficacy of 1-MCP application during the storage of climacteric fruits, reduce fruit quality losses and increase commercial value

Amount and Distribution of Micro-Defects in Solidified 2219 Al Alloy Ingots: a Metallographic Investigation

Micro-defects detection in solidified castings of aluminum alloy has always been a hot topic, and the method employed is mainly depends upon the size and shape of the specimens. In present paper, the amount and distribution characters of micro-defects in a series of 2219 aluminum alloy ingot, with diameters of φ1380 mm, φ1250 mm, φ1000 mm, φ850 mm and φ630 mm, prepared by direct chill casting were investigated by means of metallographic, respectively. Samples were cut along the radius directionfrom slices in the steady casting stage. The result reveals that typical micro-defects are consist of inclusions, porosity and shrinkage under optical microscope, and the total amount of micro-defect per unit area in an ingot slightly decreased with the increase of its diameter. Meanwhile, defects were classified into 2 types according to its size, the results suggesting that defects greater than 40 μm account for the largest proportion among the counted two kinds of defects. Moreover, the distribution of defects greater than 40 μm along the radial direction was detected, its amount increases as its distance from the side, indicating that the micro-defects greater than 40 μm distributed the most in the center zone of ingots and the larger the ingot diameter, the more obvious the tendency was

Contamination, Spatial Distribution and Source Analysis of Heavy Metals in Surface Soil of Anhui Chaohu Economic Development Zone, China

Anthropogenic activities may result in the accumulation of heavy metals in the soil, especially in economic development zones with frequent industrial activities. Therefore, the investigation and assessment of soil heavy metal pollution in economic development zones is one of the important measures for soil environmental management and sustainable development. This study used Nemero evaluation, Kriging interpolation, cluster analysis, and principal component analysis to investigate the contamination degree, spatial distribution, and origin of heavy metal in Anhui Chaohu Economic Development Zone (ACED), Anhui, East China. The result showed that different land use types can cause different levels and types of soil heavy metal pollution. The maximum concentrations of heavy metals in the study area all exceeded their background value but did not exceed the guide values. The highest average concentrations were found in Zn, followed by Cr and Ni. The concentrations of As in soils have the largest coefficient of variation (CV) at 38%. The concentration of heavy metals in different functional areas was varied, the areas with higher Ni, As, Cd, Zn, and Cr concentrations were mainly distributed in Hot Springs Resort (HSR), the relatively higher concentrations of Pb, Hg, and Cu were mainly distributed in Integrated Zone (IZ), while all heavy metal (except for Ni) have relatively higher content in the surface soil of Huashan Industrial Zone (HIZ). Origin analysis showed that soil As, Cd, and Zn in HSR surface soil were predominantly influenced by agricultural activities, while Ni and Cr were mainly controlled by parent material. Pb and Hg in IZ surface soil were predominantly originated from the vehicle and domestic exhaust, and Cu was mainly controlled by industrial pollutants. Industrial activity was the main source of soil heavy metals in HIZ. Although heavy metal in ACED surface soil did not reach pollution levels, the concentration of Cd, Hg, Pb, and Cu was significantly affected by anthropogenic activities, especially in HIZ, which the necessary attention of heavy metals needs to be given