Abstract: Grading is one of the most important steps in the tomato hole-pan seedlings. In this study, an improved YOLOv5s target model was proposed to optimize the recognition accuracy of tomato hole-pan diseased seedlings using transfer learning. The backbone part of the lightweight network (EfficientNetv2) was used as the feature extraction. The spatial pyramid pooling fusion (SPPF) module in YOLOv5s was retained to compress the number of model parameters, in order to reduce the amount of computation; The lightweight up-sampling module of CARAFE was used to introduce a small number of parameters in the Neck part of the model; And the PANet was replaced with BiFPN to optimize the accuracy of tomato hole-pan seedling identification using transfer learning. The feature weight information was introduced to enhance the fusion of features at different scales; The efficient multi-scale attention mechanism (EMA) was introduced to increase the attention for the tomato hole-pan seedlings, and reduce the background interference; The CIoU loss function was replaced by the SIoU loss function to improve the model accuracy. Direction matching between the real and predicted frame was considered to improve the convergence of the model. The mean average accuracy of the improved YOLOv5s target model reached 95.6% after training by transfer learning, which was 0.7 percentage points higher than before; The improved YOLOv5s model was 53.1% of the number of parameters in the original model, and the computation was 20.0% of the original with only 3.20G, while the weights were 53.6% of the original and the weight size was only 7.35MB, and the mean average accuracy was improved by 2.6 percentage points, compared with the original; The visualization show that the stronger feature extraction was achieved in the improved model. The feature weights were centrally distributed in the center of the burrow and the edge of the burrow holes. Meanwhile, the heat map drawn with the GradCam showed that the improved YOLOv5s model was focused mainly on the burrow seedling itself and the edge of the burrow hole, which reduced the interference of the substrate; The ablation test verified that the improved YOLOv5s model was performed the best in the detection accuracy and the frame rate, compared with the Faster-RCNN, CenterNet, and YOLO series. The findings can provide a strong reference for the grading detection and subsequent deployment of hole tray seedlings.