Physics-informed neural networks (PINNs) are known to suffer from
optimization difficulty. In this work, we reveal the connection between the
optimization difficulty of PINNs and activation functions. Specifically, we
show that PINNs exhibit high sensitivity to activation functions when solving
PDEs with distinct properties. Existing works usually choose activation
functions by inefficient trial-and-error. To avoid the inefficient manual
selection and to alleviate the optimization difficulty of PINNs, we introduce
adaptive activation functions to search for the optimal function when solving
different problems. We compare different adaptive activation functions and
discuss their limitations in the context of PINNs. Furthermore, we propose to
tailor the idea of learning combinations of candidate activation functions to
the PINNs optimization, which has a higher requirement for the smoothness and
diversity on learned functions. This is achieved by removing activation
functions which cannot provide higher-order derivatives from the candidate set
and incorporating elementary functions with different properties according to
our prior knowledge about the PDE at hand. We further enhance the search space
with adaptive slopes. The proposed adaptive activation function can be used to
solve different PDE systems in an interpretable way. Its effectiveness is
demonstrated on a series of benchmarks. Code is available at
https://github.com/LeapLabTHU/AdaAFforPINNs