Scaling Reinforcement Learning Paradigms for Motor Control

Reinforcement learning offers a general framework to explain reward related learning in artificial and biological motor control. However, current reinforcement learning methods rarely scale to high dimensional movement systems and mainly operate in discrete, low dimensional domains like game-playing, artificial toy problems, etc. This drawback makes them unsuitable for application to human or bio-mimetic motor control. In this poster, we look at promising approaches that can potentially scale and suggest a novel formulation of the actor-critic algorithm which takes steps towards alleviating the current shortcomings. We argue that methods based on greedy policies are not likely to scale into high-dimensional domains as they are problematic when used with function approximation – a must when dealing with continuous domains. We adopt the path of direct policy gradient based policy improvements since they avoid the problems of unstabilizing dynamics encountered in traditional value iteration based updates. While regular policy gradient methods have demonstrated promising results in the domain of humanoid notor control, we demonstrate that these methods can be significantly improved using the natural policy gradient instead of the regular policy gradient. Based on this, it is proved that Kakade’s ‘average natural policy gradient’ is indeed the true natural gradient. A general algorithm for estimating the natural gradient, the Natural Actor-Critic algorithm, is introduced. This algorithm converges with probability one to the nearest local minimum in Riemannian space of the cost function. The algorithm outperforms nonnatural policy gradients by far in a cart-pole balancing evaluation, and offers a promising route for the development of reinforcement learning for truly high-dimensionally continuous state-action systems. Keywords: Reinforcement learning, neurodynamic programming, actorcritic methods, policy gradient methods, natural policy gradien

A Silicon Surface Code Architecture Resilient Against Leakage Errors

Spin qubits in silicon quantum dots are one of the most promising building blocks for large scale quantum computers thanks to their high qubit density and compatibility with the existing semiconductor technologies. High fidelity single-qubit gates exceeding the threshold of error correction codes like the surface code have been demonstrated, while two-qubit gates have reached 98\% fidelity and are improving rapidly. However, there are other types of error --- such as charge leakage and propagation --- that may occur in quantum dot arrays and which cannot be corrected by quantum error correction codes, making them potentially damaging even when their probability is small. We propose a surface code architecture for silicon quantum dot spin qubits that is robust against leakage errors by incorporating multi-electron mediator dots. Charge leakage in the qubit dots is transferred to the mediator dots via charge relaxation processes and then removed using charge reservoirs attached to the mediators. A stabiliser-check cycle, optimised for our hardware, then removes the correlations between the residual physical errors. Through simulations we obtain the surface code threshold for the charge leakage errors and show that in our architecture the damage due to charge leakage errors is reduced to a similar level to that of the usual depolarising gate noise. Spin leakage errors in our architecture are constrained to only ancilla qubits and can be removed during quantum error correction via reinitialisations of ancillae, which ensure the robustness of our architecture against spin leakage as well. Our use of an elongated mediator dots creates spaces throughout the quantum dot array for charge reservoirs, measuring devices and control gates, providing the scalability in the design

Documentation technique : équipement et réparation des livres

For robots of increasing complexity such as humanoid robots, conventional identification of rigid body dynamics models based on CAD data and actuator models becomes difficult and inaccurate due to the large number of additional nonlinear effects in these systems, e.g., stemming from stiff wires, hydraulic hoses, protective shells, skin, etc. Data driven parameter estimation offers an alternative model identification method, but it is often burdened by various other problems, such as significant noise in all measured or inferred variables of the robot. The danger of physically inconsistent results also exists due to unmodeled nonlinearities or insufficiently rich data. In this paper, we address all these problems by developing a Bayesian parameter identification method that can automatically detect noise in both input and output data for the regression algorithm that performs system identification. A post-processing step ensures physically consistent rigid body parameters by nonlinearly projecting the result of the Bayesian estimation onto constraints given by positive definite inertia matrices and the parallel axis theorem. We demonstrate on synthetic and actual robot data that our technique performs parameter identification with 10 to 30% higher accuracy than traditional methods. Due to the resulting physically consistent parameters, our algorithm enables us to apply advanced control methods that algebraically require physical consistency on robotic platforms

Conceptualizing change in teaching and learning through structural equation modeling

Quality mathematics teaching that results in student learning is considered critical to heighten American competitiveness. Evaluation for verification of results of promising approaches in mathematics education is equally important for the achievement of this goal. In this study, data were reanalyzed from a study conducted by George, Hall, and Uchiyama (2000), documenting a highly-successful district-wide change in mathematics teaching and learning in a manner closely aligned with National Council of Teachers of Mathematics standards (1989). Well-specified data were collected using the Concerns-Based Adoption Model (Hall & Hord, 2006). In this correlational, causal-comparative dissertation study, data were re-analyzed using first- and second-generation latent structural equation modeling approaches, providing insight into relationships among student outcomes and instructional quality in grades 2-8 classrooms with respect to levels of implementation behavior and fidelity of implementation of constructivist approaches to teaching mathematics. Second-generation structural equation models provided a lens through which to view dynamics of change. A model associates quality of instruction with student achievement, along with recommendations for future research

The development of anticipation in the fetus: a longitudinal account of human fetal mouth movements in reaction to and anticipation of touch

Background: Research suggests that fetuses open or close their mouth in relation to directed movements (e.g. Myowa-Yamakoshi & Takeshita, 2006) but it is unclear whether mouth opening anticipates the touch or is a reaction to touch, as there has been no analysis so far of 1) the facial area of touch and 2) the sequential ordering of touch and mouth movements. If there is prenatal development of touch we would expect the frequency of fetal mouth opening immediately preceding the arriving hand at the mouth area to increase with fetal age. Participants: Fifteen healthy fetuses, 8 girls and 7 boys, underwent four additional 4-D scans at 24, 28, 32 and 36 weeks gestation. Results: Changes in the frequency of touch for different facial regions indicated a significant decline in touch upper and side part of the face and a significant increase in touching lower and perioral regions of the face with increasing gestational age. Results supporting the hypothesis showed a significant increase in the proportion of anticipatory mouth movements before touching increasing by around 8% with each week of gestational age. Additionally there was a decrease in the proportion of reactive mouth movements decreasing by around 3% for each week of gestational age