Micro- and Macro-Level Validation in Agent-Based Simulation: Reproduction of Human-Like Behaviors and Thinking in a Sequential Bargaining Game

This paper addresses both micro- and macro-level validation in agent-based simulation (ABS) to explore validated agents that can reproduce not only human-like behaviors externally but also human-like thinking internally. For this purpose, we employ the sequential bargaining game, which can investigate a change in humans' behaviors and thinking longer than the ultimatum game (i.e., one-time bargaining game), and compare simulation results of Q-learning agents employing any type of the three types of action selections (i.e., the ε-greedy, roulette, and Boltzmann distribution selections) in the game. Intensive simulations have revealed the following implications: (1) Q-learning agents with any type of three action selections can reproduce human-like behaviors but not human-like thinking, which means that they are validated from the macro-level viewpoint but not from the micro-level viewpoint; and (2) Q-learning agents employing Boltzmann distribution selection with changing the random parameter can reproduce both human-like behaviors and thinking, which means that they are validated from both micro- and macro-level viewpoints.Micro- and Macro-Level Validation, Agent-Based Simulation, Agent Modeling, Sequential Bargaining Game, Reinforcement Learning

Guiding Robot Exploration in Reinforcement Learning via Automated Planning

Reinforcement learning (RL) enables an agent to learn from trial-and-error experiences toward achieving long-term goals; automated planning aims to compute plans for accomplishing tasks using action knowledge. Despite their shared goal of completing complex tasks, the development of RL and automated planning has been largely isolated due to their different computational modalities. Focusing on improving RL agents' learning efficiency, we develop Guided Dyna-Q (GDQ) to enable RL agents to reason with action knowledge to avoid exploring less-relevant states. The action knowledge is used for generating artificial experiences from an optimistic simulation. GDQ has been evaluated in simulation and using a mobile robot conducting navigation tasks in a multi-room office environment. Compared with competitive baselines, GDQ significantly reduces the effort in exploration while improving the quality of learned policies.Comment: Accepted in International Conference of Planning and Scheduling (ICAPS-21

Iodine-Loaded Calcium Titanate for Bone Repair with Sustainable Antibacterial Activity Prepared by Solution and Heat Treatment

In the orthopedic and dental fields, simultaneously conferring titanium (Ti) and its alloy implants with antibacterial and bone-bonding capabilities is an outstanding challenge. In the present study, we developed a novel combined solution and heat treatment that controllably incorporates 0.7% to 10.5% of iodine into Ti and its alloys by ion exchange with calcium ions in a bioactive calcium titanate. The treated metals formed iodine-containing calcium-deficient calcium titanate with abundant Ti-OH groups on their surfaces. High-resolution XPS analysis revealed that the incorporated iodine ions were mainly positively charged. The surface treatment also induced a shift in the isoelectric point toward a higher pH, which indicated a prevalence of basic surface functionalities. The Ti loaded with 8.6% iodine slowly released 5.6 ppm of iodine over 90 days and exhibited strong antibacterial activity (reduction rate >99%) against methicillin-resistant Staphylococcus aureus (MRSA), S. aureus, Escherichia coli, and S. epidermidis. A long-term stability test of the antibacterial activity on MRSA showed that the treated Ti maintained a >99% reduction until 3 months, and then it gradually decreased after 6 months (to a 97.3% reduction). There was no cytotoxicity in MC3T3-E1 or L929 cells, whereas apatite formed on the treated metal in a simulated body fluid within 3 days. It is expected that the iodine-carrying Ti and its alloys will be particularly useful for orthopedic and dental implants since they reliably bond to bone and prevent infection owing to their apatite formation, cytocompatibility, and sustainable antibacterial activity

Go with the flow: reinforcement learning in turn-based battle video games

Game flow represents a state where the player is neither frustrated nor bored. In turn-based battle video games it can be achieved by Dynamic Difficulty Adjustment (DDA), whose research has begun rising over the last decade. This paper introduces an idea for incorporating DDA through the use of Reinforcement Learning (RL) to agents of turn-based battle video games. We design and implement an RL agent that shows, in a simple environment, the idea of how a game could achieve balance through adequate choices in actions depending on the player's level of skill. For achieving this purpose, we incorporated the design and implementation of state-action-reward-state-action (SARSA) algorithm to the agent of our implemented game. In addition, we added tracking of the on-going games and depending on the frequency of the player's repeated wins or losses, the rewards of the RL agent are modified. This modification of the rewards has an impact on the RL agent's actions, which involves an increase/decrease of the difficulty of the battle game. The evaluation performed shows that the idea of the paper is demonstrated, since players face personalized challenges that we believe are in range of game flow

Nucleation and growth of biomimetic apatite layers on 3D plotted biodegradable polymeric scaffolds : effect of static and dynamic coating conditions

Apatite layers were grown on the surface of newly developed starch/polycaprolactone (SPCL)-based scaffolds by a 3D plotting technology. To produce the biomimetic coatings, a sodium silicate gel was used as nucleating agent, followed by immersion in a simulated body fluid (SBF) solution. After growing a stable apatite layer for 7 days, the scaffolds were placed in SBF under static, agitated (80 strokes min!1) and circulating flow perfusion (Q = 4 ml min!1; tR = 15 s) for up to 14 days. The materials were characterized by scanning electron microscopy/energy dispersive X-ray spectroscopy, Fourier transform infrared spectroscopy and thin-film X-ray diffraction. Cross-sections were obtained and the coating thickness was measured. The elemental composition of solution and coatings was monitored by inductively coupled plasma spectroscopy. After only 6 h of immersion in SBF it was possible to observe the formation of small nuclei of an amorphous calcium phosphate (ACP) layer. After subsequent SBF immersion from 7 to 14 days under static, agitated and circulating flow perfusion conditions, these layers grew into bone-like nanocrystalline carbonated apatites covering each scaffold fiber without compromising its initial morphology. No differences in the apatite composition/chemical structure were detectable between the coating conditions. In case of flow perfusion, the coating thickness was significantly higher. This condition, besides mimicking better the biological milieu, allowed for the coating of complex architectures at higher rates, which can greatly reduce the coating step.The authors acknowledge the Portuguese Foundation for Science and Technology (PhD grant to A.L.O., SFRH/BD/10956/2002 and post-doctoral Grant to R.A.S., SFRH/BPD/17151/2004, under the POCTI Program). This work was partially supported by FCT through POCTI and/or FEDER programmes and also partially supported by the EU Project HIPPOCRATES (NMP3-CT-2003-505758) and EXPERTISSUES (NMP-CT-2004-500283)