The Importance of Clipping in Neurocontrol by Direct Gradient Descent on the Cost-to-Go Function and in Adaptive Dynamic Programming

In adaptive dynamic programming, neurocontrol and reinforcement learning, the objective is for an agent to learn to choose actions so as to minimise a total cost function. In this paper we show that when discretized time is used to model the motion of the agent, it can be very important to do "clipping" on the motion of the agent in the final time step of the trajectory. By clipping we mean that the final time step of the trajectory is to be truncated such that the agent stops exactly at the first terminal state reached, and no distance further. We demonstrate that when clipping is omitted, learning performance can fail to reach the optimum; and when clipping is done properly, learning performance can improve significantly. The clipping problem we describe affects algorithms which use explicit derivatives of the model functions of the environment to calculate a learning gradient. These include Backpropagation Through Time for Control, and methods based on Dual Heuristic Dynamic Programming. However the clipping problem does not significantly affect methods based on Heuristic Dynamic Programming, Temporal Differences or Policy Gradient Learning algorithms. Similarly, the clipping problem does not affect fixed-length finite-horizon problems

The Divergence of Reinforcement Learning Algorithms with Value-Iteration and Function Approximation

This paper gives specific divergence examples of value-iteration for several major Reinforcement Learning and Adaptive Dynamic Programming algorithms, when using a function approximator for the value function. These divergence examples differ from previous divergence examples in the literature, in that they are applicable for a greedy policy, i.e. in a "value iteration" scenario. Perhaps surprisingly, with a greedy policy, it is also possible to get divergence for the algorithms TD(1) and Sarsa(1). In addition to these divergences, we also achieve divergence for the Adaptive Dynamic Programming algorithms HDP, DHP and GDHP.Comment: 8 pages, 4 figures. In Proceedings of the IEEE International Joint Conference on Neural Networks, June 2012, Brisbane (IEEE IJCNN 2012), pp. 3070--307

A search for rapidly modulated emission in bright X-ray sources using the HEAO A-1 data base

A search was performed in the HEAO A-1 Data Base (located at the Naval Research Laboratory in Washington, D.C.) for evidence of rapidly-rotating neutron stars that could be sources of coherent gravitational radiation. A new data analysis algorithm, which was developed, is described. The algorithm was applied to data from observations of Cyg X-2, Cyg X-3, and 1820-30. Upper limits on pulse fraction were derived and reported

The Reliability and Effectiveness of a Radar-Based Animal Detection System

This document contains data on the reliability and effectiveness of an animal detection system along U.S. Hwy 95 near Bonners Ferry, Idaho. The system uses a Doppler radar to detect large mammals (e.g., deer and elk) when they approach the highway. The system met most of the suggested minimum norms for reliability. The total time the warning signs were activated was at most 90 seconds per hour, and likely substantially less. Animal detection systems are designed to detect an approaching animal. After an animal has been detected, warning signs are activated which allow drivers to respond. Results showed that 58.1–67.9% of deer were detected sufficiently early for northbound drivers, and 70.4–85% of deer were detected sufficiently early for southbound drivers. The effect of the activated warning signs on vehicle speed was greatest when road conditions were challenging (e.g., freezing temperatures and snow- and ice-covered road surface) and when visibility was low (night). In summer, there was no measurable benefit of activated warning signs, at least not as far as vehicle speed is concerned. Depending on the conditions in autumn and winter, the activated warning signs resulted in a speed reduction of 0.69 to 4.43 miles per hour. The report includes practical recommendations for operation and maintenance of the system and suggestions for potential future research

China\u27s Future

I am concerned with the historical and social science ap­proach to China and our China problems. And I want to do three things: first, characterize the old Chinese society; second, charac­terize the process of revolution which is now turning that society in­side out; and third, comment, from that point of view, on American relations with China, past, present and future

Velocity measurements by laser resonance fluorescence

The photonburst correlation method was used to detect single atoms in a buffer gas. Real time flow velocity measurements with laser induced resonance fluorescence from single or multiple atoms was demonstrated and this method was investigated as a tool for wind tunnel flow measurement. Investigations show that single atoms and their real time diffusional motion on a buffer gas can be measured by resonance fluorescence. By averaging over many atoms, flow velocities up to 88 m/s were measured in a time of 0.5 sec. It is expected that higher flow speeds can be measured and that the measurement time can be reduced by a factor of 10 or more by careful experimental design. The method is clearly not ready for incorporation in high speed wind tunnels because it is not yet known whether the stray light level will be higher or lower, and it is not known what detection efficiency can be obtained in a wind tunnel situation

Use and Effectiveness of Wildlife Crossing Structures with Short Sections of Wildlife Fencing

Our growing transportation infrastructure in the United States has many direct and indirect impacts to wildlife populations. Humans are also impacted by the interaction of roads and wildlife in terms of wildlife-vehicle collisions (WVC’s); which annually result in: hundreds of human fatalities, tens of thousands of human injuries, and billions of dollars in property damage. In response to concerns for wildlife and human safety, road mitigation measures are becoming an increasingly important tool for transportation agencies to minimize the risks of WVC’s. The construction of multiple wildlife crossing structures in combination with wildlife fencing have been shown to reduce WVC’s by over 80% where extensive continuous wildlife fencing (covering many miles) has been implemented, and by 50-60% in areas where more limited fencing (covering 1-3 miles) has been implemented. In areas where land use is dominated by humans (agriculture, housing, access roads, etc.) such mitigation measures are not always possible or desirable. This results in a push towards more isolated crossing structures with little to no wildlife fencing to provide frequent road access and preserve landscape aesthetics. The effectiveness of isolated crossing structures with short sections of fencing (only a few hundred meters or less) is not well documented in terms of potential WVC reduction or wildlife use of the structures. In this study I investigate: the use of isolated crossing structures and fence ends by target species, the effect of fence length on at-grade crossings, and the ability of short sections of fencing to keep wildlife off the road. Overall, 82% of wildlife used the crossing structures for crossing as opposed to going around fence ends. Over the length of fence lengths sampled (3m-256m) there was no relationship between fence length and the number of crossings at fence ends. Deer were often foraging at fence ends, with nearly half of all foraging events occurring in the right of way (closer to the road than the fence is/would be). Overall, deer generally choose to use the crossing structures to get to the other side of the road, but they will still often be present in the right of way where they are not excluded with fencing. This indicates that while isolated crossing structures with short fencing may provide wildlife safe access to habitat on either side of the road, they may not provide the desired reduction in WVC’s