This paper proposes a novel adaptive fusion bank (LAFB) for multi-modal salient object detection (MSOD). The LAFB is designed to handle five major challenges in MSOD: center bias, scale variation, image clutter, low illumination, and thermal crossover or depth ambiguity. The LAFB consists of five representative fusion schemes, each specifically designed to address the characteristics of a particular challenge. These fusion schemes are embedded into hierarchical layers for sufficient fusion of different source data. Additionally, an indirect interactive guidance module (IIGM) is designed to accurately detect salient hollow objects via the skip integration of high-level semantic information and low-level spatial details. The LAFB is scalable, and more fusion schemes can be incorporated into the bank to address additional challenges. The proposed method is evaluated on three RGBT datasets and seven RGBD datasets, demonstrating its outstanding performance compared to state-of-the-art methods. The code and results are available at https://github.com/Angknpng/LAFB.This paper proposes a novel adaptive fusion bank (LAFB) for multi-modal salient object detection (MSOD). The LAFB is designed to handle five major challenges in MSOD: center bias, scale variation, image clutter, low illumination, and thermal crossover or depth ambiguity. The LAFB consists of five representative fusion schemes, each specifically designed to address the characteristics of a particular challenge. These fusion schemes are embedded into hierarchical layers for sufficient fusion of different source data. Additionally, an indirect interactive guidance module (IIGM) is designed to accurately detect salient hollow objects via the skip integration of high-level semantic information and low-level spatial details. The LAFB is scalable, and more fusion schemes can be incorporated into the bank to address additional challenges. The proposed method is evaluated on three RGBT datasets and seven RGBD datasets, demonstrating its outstanding performance compared to state-of-the-art methods. The code and results are available at https://github.com/Angknpng/LAFB.