Predecessor Features

Any reinforcement learning system must be able to identify which past events contributed to observed outcomes, a problem known as credit assignment. A common solution to this problem is to use an eligibility trace to assign credit to recency-weighted set of experienced events. However, in many realistic tasks, the set of recently experienced events are only one of the many possible action events that could have preceded the current outcome. This suggests that reinforcement learning can be made more efficient by allowing credit assignment to any viable preceding state, rather than only those most recently experienced. Accordingly, we examine ``Predecessor Features'', the fully bootstrapped version of van Hasselt's ``Expected Trace'', an algorithm that achieves this richer form of credit assignment. By maintaining a representation that approximates the expected sum of past occupancies, this algorithm allows temporal difference (TD) errors to be propagated accurately to a larger number of predecessor states than conventional methods, greatly improving learning speed. The algorithm can also be naturally extended from tabular state representation to feature representations allowing for increased performance on a wide range of environments. We demonstrate several use cases for Predecessor Features and compare its performance with other approaches.Comment: Accepted to RLDM 202

Effect of Aqueous and Ethanolic Extracts of Tribulus terrestris, Phoenix dactylifera and Nasturtium officinale Mixture on Some Reproductive Parameters in Male Mice

The present investigation was conducted to evaluate the effect of the crude extracts mixture of three plants (Tribulus terrestris, Phoenix dactylifera and Nasturtium officinale) on semen quality,sex hormones and reproductive performance of mature male mice. A group of 25 male mice given 150mg/kg/day of the powder of the plants mixture with the food for four weeks and another three groups of 25 animals each given intraperitoneal injection from each of the aqueous and ethanolic extracts with a doses 75, 150, and 300mg/kg/day for two weeks. A remarkable increase in sperm concentration and motility with a decreased abnormal morphology was obtained in the experimental groups. A significant increase in hormones level were recognized in most groups. The results of mating untreated females with treated males of the four experimental groups revealed a decreased gestation period and an increased litter size. The results showed a dose dependent pattern of activity and the effect of the extracts were enhanced with increasing the dose level. The ethanolic extract being the more effective extract in all parameters

Influence of Peri-duodenal Non-constrictive Cuff on the Body Weight of Rats

Background Weight loss has been found to improve or re- solve cardiovascular comorbidities. There is a significant need for reversible device approaches to weight loss. Methods Non-constrictive cuff (NCC) is made of implantable silicone rubber with an internal diameter greater than the duodenum. Ten or 11 NCC were individually mounted along the duodenum from the pyloric sphincter toward the distal duodenum to cover ~22 mm in the length. Twelve Wistar rats were implanted with NCC, and six served as sham, and both groups were observed over 4 months. Six rats with implant had their NCC removed and were observed for additional 4weeks. Results The food intake decreased from 40.1 to 28.1 g/day after 4 months of NCC implant. The body weight gain decreased from 1.76 to 0.46 g/day after 4 months of NCC implant. The fasting glucose decreased from 87.7 to 75.3 mg/ dl at terminal day. The duodenal muscle layer covered by the NCC increased from 0.133 to 0.334 mm. After 4 weeks of NCC removal, the food intake, body weight gain, and fasting glucose recovered to 36.2, 2.51 g/day, and 83.9 mg/dl. The duodenal muscle layer covered by the NCC decreased to 0.217 mm. Conclusion The NCC implant placed on the proximal duode- num is safe in rats for a 4-month period. The efficacy of the NCC implant is significant for decrease in food intake, body weight gain, and fasting glucose in a normal rat model. The removal of NCC implant confirmed a cause-effect relation with food intake and hence body weight

Emergence and reconfiguration of modular structure for synaptic neural networks during continual familiarity detection

While advances in artificial intelligence and neuroscience have enabled the emergence of neural networks capable of learning a wide variety of tasks, our understanding of the temporal dynamics of these networks remains limited. Here, we study the temporal dynamics during learning of Hebbian Feedforward (HebbFF) neural networks in tasks of continual familiarity detection. Drawing inspiration from the field of network neuroscience, we examine the network's dynamic reconfiguration, focusing on how network modules evolve throughout learning. Through a comprehensive assessment involving metrics like network accuracy, modular flexibility, and distribution entropy across diverse learning modes, our approach reveals various previously unknown patterns of network reconfiguration. In particular, we find that the emergence of network modularity is a salient predictor of performance, and that modularization strengthens with increasing flexibility throughout learning. These insights not only elucidate the nuanced interplay of network modularity, accuracy, and learning dynamics but also bridge our understanding of learning in artificial and biological realms