Abstract: Pine wilt disease (PWD) is well known as the cancer of pine trees. However, the false and missed detection of discolored dead trees has often confined the large-scale production in recent years. In this study, an adaptive multi-scale feature fusion network was proposed to optimize the recognition accuracy of the PWD. The backbone layer of the lightweight network (EfficientNetv2) was used to extract the feature. The spatial pyramid pooling fusion (SPPF) module in YOLOv5s was retained to compress the number of parameters, in order to reduce the amount of computation; Secondly, the convolution block attention module (CBAM) was added into the YOLOv5s. The attention was then strengthened on the target disease. The PANet was replaced with the BiFPN, particularly for the high accuracy of detection on the discolored dead trees. The feature weight was also introduced to enhance the fusion of features at different scales. Finally, the SIoU loss function was replaced with the CIoU one, in order to improve the accuracy of the model. Direction matching between the real and predicted frames was considered to enhance the convergence of model. The results showed that the accuracy, recall and mAP0.5 of the improved model in the modeling area reached 92.95%, 94.88% and 94.21%, respectively, which were 4.31, 5.60, and 5.13 percentage point higher than those of the original one. The number of parameters and computation were reduced by 22.93% and 64.19%, which were 77.07% and 35.81% of the original YOLOv5s, respectively. At the same time, the network processing speed of the improved YOLOv5s model increased by 45.66% to 136.05 frames/s, indicating the more real-time model. The ablation test proved that each measure with the YOLOv5s was improved the performance of the original model, where all four measures were necessary. Compared with the SSD, Faster R-CNN and YOLO series models, the improved model of object detection performed better in the recall rate, detection speed, the number of parameters and weights. The improved model was achieved with the 95.25% and 93.17% of the single-class average detection accuracy (AP) for abnormally discolored pine trees and dead pine trees, respectively. In addition, the frame rate and weight file size of the improved model were 136.05 frames/s and 11.90 MB, respectively, and the F1 score was 93.91%. The average detection accuracy of the improved model for two types of targets in the natural forest area was 94.02%, which was 1.07% higher than that in the modeling area. The recall rate of dead pine trees reached 95.88% in the Plot 8, which was 1.00% higher than that in the modeled area. Furthermore, the average F1 score of the model was 93.68% for the two types of targets in the test area, which was roughly equivalent to the modeling area. It was of great practical significance for this rapid and accurate detection of abnormal discolored pine trees in natural scenes. This finding can also provide a strong reference to improve the intelligent prevention and control of pine wood nematode diseases.