Deep learning applied to computational mechanics: A comprehensive review, state of the art, and the classics

Three recent breakthroughs due to AI in arts and science serve as motivation: An award winning digital image, protein folding, fast matrix multiplication. Many recent developments in artificial neural networks, particularly deep learning (DL), applied and relevant to computational mechanics (solid, fluids, finite-element technology) are reviewed in detail. Both hybrid and pure machine learning (ML) methods are discussed. Hybrid methods combine traditional PDE discretizations with ML methods either (1) to help model complex nonlinear constitutive relations, (2) to nonlinearly reduce the model order for efficient simulation (turbulence), or (3) to accelerate the simulation by predicting certain components in the traditional integration methods. Here, methods (1) and (2) relied on Long-Short-Term Memory (LSTM) architecture, with method (3) relying on convolutional neural networks. Pure ML methods to solve (nonlinear) PDEs are represented by Physics-Informed Neural network (PINN) methods, which could be combined with attention mechanism to address discontinuous solutions. Both LSTM and attention architectures, together with modern and generalized classic optimizers to include stochasticity for DL networks, are extensively reviewed. Kernel machines, including Gaussian processes, are provided to sufficient depth for more advanced works such as shallow networks with infinite width. Not only addressing experts, readers are assumed familiar with computational mechanics, but not with DL, whose concepts and applications are built up from the basics, aiming at bringing first-time learners quickly to the forefront of research. History and limitations of AI are recounted and discussed, with particular attention at pointing out misstatements or misconceptions of the classics, even in well-known references. Positioning and pointing control of a large-deformable beam is given as an example.Comment: 275 pages, 158 figures. Appeared online on 2023.03.01 at CMES-Computer Modeling in Engineering & Science

Solving inverse problems for medical applications

It is essential to have an accurate feedback system to improve the navigation of surgical tools. This thesis investigates how to solve inverse problems using the example of two medical prototypes. The first aims to detect the Sentinel Lymph Node (SLN) during the biopsy. This will allow the surgeon to remove the SLN with a small incision, reducing trauma to the patient. The second investigates how to extract depth and tissue characteristic information during bone ablation using the emitted acoustic wave. We solved inverse problems to find our desired solution. For this purpose, we investigated three approaches: In Chapter 3, we had a good simulation of the forward problem; namely, we used a fingerprinting algorithm. Therefore, we compared the measurement with the simulations of the forward problem, and the simulation that was most similar to the measurement was a good approximation. To do so, we used a dictionary of solutions, which has a high computational speed. However, depending on how fine the grid is, it takes a long time to simulate all the solutions of the forward problem. Therefore, a lot of memory is needed to access the dictionary. In Chapter 4, we examined the Adaptive Eigenspace method for solving the Helmholtz equation (Fourier transformed wave equation). Here we used a Conjugate quasi-Newton (CqN) algorithm. We solved the Helmholtz equation and reconstructed the source shape and the medium velocity by using the acoustic wave at the boundary of the area of interest. We accomplished this in a 2D model. We note, that the computation for the 3D model was very long and expensive. In addition, we simplified some conditions and could not confirm the results of our simulations in an ex-vivo experiment. In Chapter 5, we developed a different approach. We conducted multiple experiments and acquired many acoustic measurements during the ablation process. Then we trained a Neural Network (NN) to predict the ablation depth in an end-to-end model. The computational cost of predicting the depth is relatively low once the training is complete. An end-to-end network requires almost no pre-processing. However, there were some drawbacks, e.g., it is cumbersome to obtain the ground truth. This thesis has investigated several approaches to solving inverse problems in medical applications. From Chapter 3 we conclude that if the forward problem is well known, we can drastically improve the speed of the algorithm by using the fingerprinting algorithm. This is ideal for reconstructing a position or using it as a first guess for more complex reconstructions. The conclusion of Chapter 4 is that we can drastically reduce the number of unknown parameters using Adaptive Eigenspace method. In addition, we were able to reconstruct the medium velocity and the acoustic wave generator. However, the model is expensive for 3D simulations. Also, the number of transducers required for the setup was not applicable to our intended setup. In Chapter 5 we found a correlation between the depth of the laser cut and the acoustic wave using only a single air-coupled transducer. This encourages further investigation to characterize the tissue during the ablation process

Control Theory-Inspired Acceleration of the Gradient-Descent Method: Centralized and Distributed

Mathematical optimization problems are prevalent across various disciplines in science and engineering. Particularly in electrical engineering, convex and non-convex optimization problems are well-known in signal processing, estimation, control, and machine learning research. In many of these contemporary applications, the data points are dispersed over several sources. Restrictions such as industrial competition, administrative regulations, and user privacy have motivated significant research on distributed optimization algorithms for solving such data-driven modeling problems. The traditional gradient-descent method can solve optimization problems with differentiable cost functions. However, the speed of convergence of the gradient-descent method and its accelerated variants is highly influenced by the conditioning of the optimization problem being solved. Specifically, when the cost is ill-conditioned, these methods (i) require many iterations to converge and (ii) are highly unstable against process noise. In this dissertation, we propose novel optimization algorithms, inspired by control-theoretic tools, that can significantly attenuate the influence of the problem's conditioning. First, we consider solving the linear regression problem in a distributed server-agent network. We propose the Iteratively Pre-conditioned Gradient-Descent (IPG) algorithm to mitigate the deleterious impact of the data points' conditioning on the convergence rate. We show that the IPG algorithm has an improved rate of convergence in comparison to both the classical and the accelerated gradient-descent methods. We further study the robustness of IPG against system noise and extend the idea of iterative pre-conditioning to stochastic settings, where the server updates the estimate based on a randomly selected data point at every iteration. In the same distributed environment, we present theoretical results on the local convergence of IPG for solving convex optimization problems. Next, we consider solving a system of linear equations in peer-to-peer multi-agent networks and propose a decentralized pre-conditioning technique. The proposed algorithm converges linearly, with an improved convergence rate than the decentralized gradient-descent. Considering the practical scenario where the computations performed by the agents are corrupted, or a communication delay exists between them, we study the robustness guarantee of the proposed algorithm and a variant of it. We apply the proposed algorithm for solving decentralized state estimation problems. Further, we develop a generic framework for adaptive gradient methods that solve non-convex optimization problems. Here, we model the adaptive gradient methods in a state-space framework, which allows us to exploit control-theoretic methodology in analyzing Adam and its prominent variants. We then utilize the classical transfer function paradigm to propose new variants of a few existing adaptive gradient methods. Applications on benchmark machine learning tasks demonstrate our proposed algorithms' efficiency. Our findings suggest further exploration of the existing tools from control theory in complex machine learning problems. The dissertation is concluded by showing that the potential in the previously mentioned idea of IPG goes beyond solving generic optimization problems through the development of a novel distributed beamforming algorithm and a novel observer for nonlinear dynamical systems, where IPG's robustness serves as a foundation in our designs. The proposed IPG for distributed beamforming (IPG-DB) facilitates a rapid establishment of communication links with far-field targets while jamming potential adversaries without assuming any feedback from the receivers, subject to unknown multipath fading in realistic environments. The proposed IPG observer utilizes a non-symmetric pre-conditioner, like IPG, as an approximation of the observability mapping's inverse Jacobian such that it asymptotically replicates the Newton observer with an additional advantage of enhanced robustness against measurement noise. Empirical results are presented, demonstrating both of these methods' efficiency compared to the existing methodologies

Online Optimization-based Gait Adaptation of Quadruped Robot Locomotion

Quadruped robots demonstrated extensive capabilities of traversing complex and unstructured environments. Optimization-based techniques gave a relevant impulse to the research on legged locomotion. Indeed, by designing the cost function and the constraints, we can guarantee the feasibility of a motion and impose high-level locomotion tasks, e.g., tracking of a reference velocity. This allows one to have a generic planning approach without the need to tailor a specific motion for each terrain, as in the heuristic case. In this context, Model Predictive Control (MPC) can compensate for model inaccuracies and external disturbances, thanks to the high-frequency replanning. The main objective of this dissertation is to develop a Nonlinear MPC (NMPC)-based locomotion framework for quadruped robots. The aim is to obtain an algorithm which can be extended to different robots and gaits; in addition, I sought to remove some assumptions generally done in the literature, e.g., heuristic reference generator and user-defined gait sequence. The starting point of my work is the definition of the Optimal Control Problem to generate feasible trajectories for the Center of Mass. It is descriptive enough to capture the linear and angular dynamics of the robot as a whole. A simplified model (Single Rigid Body Dynamics model) is used for the system dynamics, while a novel cost term maximizes leg mobility to improve robustness in the presence of nonflat terrain. In addition, to test the approach on the real robot, I dedicated particular effort to implementing both a heuristic reference generator and an interface for the controller, and integrating them into the controller framework developed previously by other team members. As a second contribution of my work, I extended the locomotion framework to deal with a trot gait. In particular, I generalized the reference generator to be based on optimization. Exploiting the Linear Inverted Pendulum model, this new module can deal with the underactuation of the trot when only two legs are in contact with the ground, endowing the NMPC with physically informed reference trajectories to be tracked. In addition, the reference velocities are used to correct the heuristic footholds, obtaining contact locations coherent with the motion of the base, even though they are not directly optimized. The model used by the NMPC receives as input the gait sequence, thus with the last part of my work I developed an online multi-contact planner and integrated it into the MPC framework. Using a machine learning approach, the planner computes the best feasible option, even in complex environments, in a few milliseconds, by ranking online a set of discrete options for footholds, i.e., which leg to move and where to step. To train the network, I designed a novel function, evaluated offline, which considers the value of the cost of the NMPC and robustness/stability metrics for each option. These methods have been validated with simulations and experiments over the three years. I tested the NMPC on the Hydraulically actuated Quadruped robot (HyQ) of the IIT’s Dynamic Legged Systems lab, performing omni-directional motions on flat terrain and stepping on a pallet (both static and relocated during the motion) with a crawl gait. The trajectory replanning is performed at high-frequency, and visual information of the terrain is included to traverse uneven terrain. A Unitree Aliengo quadruped robot is used to execute experiments with the trot gait. The optimization-based reference generator allows the robot to reach a fixed goal and recover from external pushes without modifying the structure of the NMPC. Finally, simulations with the Solo robot are performed to validate the neural network-based contact planning. The robot successfully traverses complex scenarios, e.g., stepping stones, with both walk and trot gaits, choosing the footholds online. The achieved results improved the robustness and the performance of the quadruped locomotion. High-frequency replanning, dealing with a fixed goal, recovering after a push, and the automatic selection of footholds could help the robots to accomplish important tasks for the humans, for example, providing support in a disaster response scenario or inspecting an unknown environment. In the future, the contact planning will be transferred to the real hardware. Possible developments foresee the optimization of the gait timings, i.e., stance and swing duration, and a framework which allows the automatic transition between gaits

Nonlinear Systems

Open Mathematics is a challenging notion for theoretical modeling, technical analysis, and numerical simulation in physics and mathematics, as well as in many other fields, as highly correlated nonlinear phenomena, evolving over a large range of time scales and length scales, control the underlying systems and processes in their spatiotemporal evolution. Indeed, available data, be they physical, biological, or financial, and technologically complex systems and stochastic systems, such as mechanical or electronic devices, can be managed from the same conceptual approach, both analytically and through computer simulation, using effective nonlinear dynamics methods. The aim of this Special Issue is to highlight papers that show the dynamics, control, optimization and applications of nonlinear systems. This has recently become an increasingly popular subject, with impressive growth concerning applications in engineering, economics, biology, and medicine, and can be considered a veritable contribution to the literature. Original papers relating to the objective presented above are especially welcome subjects. Potential topics include, but are not limited to: Stability analysis of discrete and continuous dynamical systems; Nonlinear dynamics in biological complex systems; Stability and stabilization of stochastic systems; Mathematical models in statistics and probability; Synchronization of oscillators and chaotic systems; Optimization methods of complex systems; Reliability modeling and system optimization; Computation and control over networked systems