Abstract: Grading has been one of the most important procedures in potato harvesting. However, manual grading has suffered from low efficiency, high subjectivity, high cost, and insufficient accuracy. High precision of detection is then required to fully meet the demands of large-scale production in recent years. Alternatively, several challenges still remain in the detection of potato surface defects. Firstly, simple image processing cannot concurrently recognize the variety of defects, such as disease spots, pest damage, and mechanical injuries, together with their diverse morphologies. Secondly, the high resolution of detection can further be confined to the complex environmental factors in agricultural settings, such as lighting and soil coverage. Moreover, the real-time operation has also failed to be realized on resource-constrained devices using most existing models with large parameter sizes and high computation. In this study, a lightweight algorithm (DATW-YOLOv8) was proposed to detect potato surface defects using an improved YOLOv8n model. The bottleneck module in the C2f was replaced with the Dilation-wise Residual (DWR) module. Dilated Reparam Block (DRB) was then introduced to optimize the extraction of the features for the high accuracy of detection. Additionally, a lightweight adaptive downsampling (ADOWN) convolution module was integrated to reduce the dimensionality for high processing efficiency. The detection head was upgraded to a Task Align Dynamic Detection Head (TADDH). As such, the high accuracy of defection was obtained to predict the boundary. The real-time detection was then realized to focus precisely on the key regions of surface defects. Finally, the Wise-EIoU was adopted as the bounding box regression loss function, in order to increase the attention to the difficult samples, thereby enhancing the boundary regression accuracy and model robustness. Experimental results show that the improved DATW-YOLOv8 model was achieved in the detection accuracy, recall, and mean average precision (mAP) of 95.8%, 88.1%, and 94.3%, respectively. The parameter size and weight size were 1.5 M and 3.6 MB, respectively, which were 50.0% and 42.9% smaller than those of the original YOLOv8n model. Additionally, the accuracy, recall, and mAP were improved by 2.8, 1.6, and 1.4, respectively. The better performance of defect detection was achieved in the ADOWN downsampling, compared with the YOLOv7 E-ELAN, SPDConv, WaveletPool, and Light-weight Context Guided DownSample, with the mAP improvements of 0.8, 1.3, 1.4, and 1.7 percentage points, respectively, while reducing parameter counts by 11.8, 60.5, 6.3, and 48.3 percentage points, respectively. The Wise-EIoU was also achieved in the lowest loss value, the fastest convergence, and the smallest fluctuation among various bounding box loss functions. The DATW-YOLOv8 also outperformed the YOLOv5Lite-g, YOLOv8n, YOLOv10n, and Mobilenetv2-SSD, in terms of the accuracy, recall, and mAP@0.5, particularly with the lower model weight. Specifically, the model weight of DATW-YOLOv8 was only one-third of that of YOLOv7tiny. The superior performance was then confirmed after multiple evaluations. Furthermore, the improved DATW-YOLOv8 model was deployed on an online detection of potato surface defects. A sorting test of online detection was conducted on the different potato varieties and conveyor speeds. The maximum sorting accuracy reached 95.8%, fully meeting the real-time detection requirements of potato surface defects in practical production. Overall, this finding can also provide a valuable technical reference for online detection of potato surface defects and model deployment on mobile devices.