Abstract: Rapeseed is one of the most important oil crops in China. Winter rapeseed can account for more than 85% of the total cultivation acreage. However, frost damage has pose a serious threat to both the yield and quality of rapeseed, due to the low temperatures at the seedling stage. It is often required for the cold-resistant varieties using screening. Variety evaluation can be expected to accurately quantify the degree of frost damage. But conventional manual assessment cannot fully meet the large-scale production in recent years, such as the long cycles, low efficiency, and strong subjectivity. An intelligent evaluation framework is greatly hindered to construct after assessment. In this study, an improved model called YOLOv7-DH was proposed to grade the frost damage in the scenario of the rapeseed seedlings. According to the complex agronomic conditions, YOLOv7-DH was incorporated three measures: Firstly, DCNv2 was adopted to replace the standard convolution in the feature extraction backbone. With the help of learnable offset parameters, the receptive field was dynamically adjusted to better capture the irregular features that caused by frost, such as the leaf curling and heterogeneous necrosis, thereby improving the fidelity of the feature extraction. Secondly, the Focal-EIoU loss function was introduced to optimize the bounding box regression. A focal mechanism was utilized to balance the various class. At the same time, the width and height differences between the predicted and the real box were reduced for the convergence speed and the positioning accuracy. Particularly, the adjacent damage levels (such as level 1 and level 2) were distinguished. Thirdly, the lightweight CARAFE upsampling operator was used to replace the conventional interpolation in the neck and head modules. The content-aware feature reconstruction was realized after adaptive kernel adjustment. Subtle features were retained, such as the initial edge discoloration. The receptive field was expanded to effectively overcome the spatial information degradation in conventional upsampling. Empirical results show that the YOLOv7-DH performed excellently in the frost damage classification. On the test set, the precision rate reached 95.4%, the recall rate was 94.5%, and the mean average precision (mAP) was 95.2%. Three indicators increased by 9.1, 9.3, and 9.4 percentage points, respectively, compared with the baseline YOLOv7. Ablation experiments confirmed the effectiveness of each improvement: DCNv2 increased mAP by 6.0%, Focal-EIoU by 3.5%, and CARAFE by 4.5%. Moreover, the combination of these three improvements was achieved the best performance. Compared with the mainstream models, the mAP of YOLOv7-DH was 15.3, 22.4, 29.4, and 21.8 percentage points higher than that of YOLOv5s, YOLOv7-tiny, Faster R-CNN, and SSD, respectively. At the same time, a real-time speed of 60.3 frames per second was met the needs of the field detection. In the verification of the frost damage index, the mean absolute error (MAE) of the model relative to manual measurement was 1.22, and the root mean square error (RMSE) was 1.36, indicating the high quantification accuracy. In general, the YOLOv7-DH was provided for an accurate, efficient, and lightweight solution to the multi-class frost damage detection of the rapeseed seedlings. DCNv2, Focal-EIoU and CARAFE were integrated to capture the complex frost features. This finding can provide the support for the cold-resistant variety screening after agricultural stress assessment using computer vision. An intelligent evaluation was also constructed for the rapeseed in the precision agriculture.