This paper introduces a novel targetless method for joint intrinsic and
extrinsic calibration of LiDAR-camera systems using plane-constrained bundle
adjustment (BA). Our method leverages LiDAR point cloud measurements from
planes in the scene, alongside visual points derived from those planes. The
core novelty of our method lies in the integration of visual BA with the
registration between visual points and LiDAR point cloud planes, which is
formulated as a unified optimization problem. This formulation achieves
concurrent intrinsic and extrinsic calibration, while also imparting depth
constraints to the visual points to enhance the accuracy of intrinsic
calibration. Experiments are conducted on both public data sequences and
self-collected dataset. The results showcase that our approach not only
surpasses other state-of-the-art (SOTA) methods but also maintains remarkable
calibration accuracy even within challenging environments. For the benefits of
the robotics community, we have open sourced our codes