Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds

We describe a method to use discrete human feedback to enhance the performance of deep learning agents in virtual three-dimensional environments by extending deep-reinforcement learning to model the confidence and consistency of human feedback. This enables deep reinforcement learning algorithms to determine the most appropriate time to listen to the human feedback, exploit the current policy model, or explore the agent's environment. Managing the trade-off between these three strategies allows DRL agents to be robust to inconsistent or intermittent human feedback. Through experimentation using a synthetic oracle, we show that our technique improves the training speed and overall performance of deep reinforcement learning in navigating three-dimensional environments using Minecraft. We further show that our technique is robust to highly innacurate human feedback and can also operate when no human feedback is given

Additional file 1: of Survey of chiropractic clinicians on self-reported knowledge and recognition of concussion injuries

Appendix. Concussion Survey of Primary Contact Chiropractic Practitioners with Data Results. (DOCX 213 kb

Crack propagation and fracture toughness of hind wings.

<p>(a) Stress-strain curve and corresponding crack length from one wing sample with an induced notch. The numbers indicate the K_C indices (see text). With increasing strain the stress on the wing membrane increases until the crack starts growing (0). When reaching cross veins (1–4), the crack propagation temporarily stops and the stress further increases. When the cross veins break, the stress decreases and the crack continues to propagate. (b) Crack length and corresponding fracture toughness K_C. The markers 0–4 correspond to the markers in (a). (c) Fracture toughness of hind wing membrane. Although slightly decreasing towards the anal part of the wing, there was no significant difference in-between the fracture toughness K_C0 of the membrane from the tested three wing zones (F_2,16 = 2.087, p>0.1, ANOVA). (d) The membrane alone had a mean fracture toughness of 1.04±0.25 MPa√m (N = 17). The presence of the first cross-vein (index 1) significantly increased the fracture toughness of the wing structure to 1.57±0.38 MPa√m (t₉ = −3.513, p<0.01, paired t-test, both figures show mean±SD, numbers show sample size).</p

Size and distribution of wing cells in <i>S. gregaria</i> hind wings.

<p>(a) Typical structure of a hind wing, showing the distribution of the wing cells’ major axis length. Cells with smaller major axis lengths are mostly arranged around the perimeter of the wing (CCL: critical crack length). (b) Mean frequency of wing cell sizes from six hind wings. The distribution of cells corresponds very well to a normal distribution around a mean major axis length of 1.103 mm (σ = 544.16, a = 7.33, R = 0.98). The cumulative membrane area formed by cells smaller than the critical crack length is 19.44% of the overall membrane area (mean ± SD, N = 5553 cells from 6 wings). The colour map of the bars corresponds to subfigure A. (c) 2D-Histogram showing the relative frequency of cell size and their distance to the wing edge. There is a significant positive correlation of the major axis length with the distance to the wing edge (ρ = 0.393, R² = 0.154, p<0.001, linear correlation, N = 5553 cells).</p

Microcrack accumulation at different intervals during fatigue testing of compact bone.

Fatigue damage in bone occurs in the form of microcracks. This microdamage contributes to the formation of stress fractures and acts as a stimulus for bone remodelling. A technique has been developed, which allows microcrack growth to be monitored during the course of a fatigue test by the application of a series of fluorescent chelating agents. Specimens were taken from bovine tibiae and fatigue tested in cyclic compression at a stress range of 80MPa. The specimens were stained before testing with alizarin and up to three other chelating agents were applied during testing to label microcracks formed at different times. Microcracks initiated in interstitial bone in the early part of a specimen's life. Further accumulation of microcracks is then suppressed until the period late in the specimen's life. Microcracks were found to be longer in the longitudinal than in the transverse direction. Only a small proportion of cracks are actively propagating; these are longer than non-propagating cracks. These results support the concept of a microstructural barrier effect existing in bone, whereby cracks initiate easily but slow down or stop at barriers such as cement lines

Summary of stiffness, strength and fracture toughness from fresh isolated wing membrane sections and complete wing sections (membrane plus veins) from three different regions of the hind wing (see Figure 1 a).

<p>Significant differences are indicated.</p>*<p>For details see Results.</p

The nature of fatigue damage in bone

Bone is unusual among structural materials as it is alive and capable of self-repair. Fatigue-induced microdamage is repaired by bone remodelling, but if damage accumulates too quickly, or remodelling is deficient, fatigue failure may result. Fatigue is thought to contribute to both stress and fragility fractures which are of major clinical importance. Despite this, we do not fully understand the nature of fatigue damage in bone. Human rib sections, containing microcracks stained with basic fuchsin, were serially sectioned and microcracks identified and reconstructed in three dimensions using computer software. Microcracks were elliptical in shape, 400 μm long and 100 μm wide, typical of a transversely isotropic material. Chelating agents which bind Ca2+ were found to label microcracks in rib, as well as mineralising bone surfaces and resorption sites, suggesting that microcracks are Ca2+ ion-lined discontinuities in the hydroxyapatite matrix. Ca2+ ions were exposed by scratching the surface of bovine bone specimens and labelled with chelating agents in sequence. The optimal four agent sequence was: alizarin, xylenol orange, calcein and calcein blue. Two dye sequences were used to differentiate between pre-existing and test-induced microdamage in bovine samples fatigue tested in compression and longer sequences labelled microcrack growth. Microcrack dimensions can be used to calculate stress intensity values and, together with fatigue test data, can aid theoretical models to predict fatigue failure in bone

Average membrane thickness from three wing zones R, B and C (see Figure 1 a), measured from SEM cross sections.

<p>Sample size shows number of measurements taken from 9 insects.</p

Experimentally measured mechanical properties.

<p>Shown are mean values ± 1 standard deviation.</p

An improved labelling technique for monitoring microcrack growth in compact bone.

Fatigue-induced damage plays an important role in bone remodelling and in the formation of stress and fragility fractures. Recently, a technique has been developed (Lee, T.C. et al., Sequential labelling of microdamage in bone using chelating agents. Journal of Orthopedic Research, 18 (2000) 322-325) which allows microcrack growth in trabecular bone to be monitored by the application of a series of chelating fluorochromes, however, some limitations were identified with the process. The aims of this study were to refine the method of detection using these agents in order to determine the optimal sequence of application and the optimal concentrations which allowed all the agents to fluoresce equally brightly using UV epifluorescence. A chemical analysis process, ion chromatography, followed by validation tests on bone samples showed that the optimal sequence of application and concentration of each agent was alizarin complexone (0.0005 M) followed by xylenol orange (0.0005 M), calcein (0.0005 M) and calcein blue (0.0001 M). A fifth agent, oxytetracycline was excluded from the study after recurring problems were found with its ability to chelate exposed calcium when applied in sequence with the other agents. This work has developed a sequential labelling technique, which allows for microcrack propagation during fatigue testing of bone specimens to be monitored without the problem of chelating agent substitution occurring