Abstract: Citrus surface defects play a pivotal role in the fruit inspection and grading during agricultural production. Surface imperfections are also much easier to spot than inside ones, leading to accelerate the deterioration. However, conventional detection of citrus surface defects cannot fully meet the overall quality assessment in the large-scale production in recent years, due to the low efficiency and accuracy. In this study, an accurate, rapid and real-time detection was proposed to consider the diverse and complex nature of surface imperfections observed in citrus fruits. This speed and precision of detection were also enhanced for the quality of surface defects. Firstly, the images of citrus fruits were captured by industrial cameras. The generated images were enhanced to make the region of interest more outstanding. Then, the YOLOv7-CACT model was improved for the defect region in the enhanced citrus image. The coordinate attention (CA) module was introduced in the backbone network, in order to increase the attention to the defective part. The contextual transformer (CT) module was introduced in the head of the network to fuse the static and dynamic contextual representation features, thus enhancing the feature expression of the defective part. The superior performance was achieved in the modified YOLOv7-CACT model, compared with the baseline version. Especially, the detection accuracy was improved by 4.1% in the mean average precision (mAP). Consequently, the modified model was fully met the accuracy requirements for the identification of citrus surface defects in an offline setting. TensorRT was also employed around YOLOv7-CACT for the deployment of improved model, in order to real-time detect in practical scenarios. The results show that the improved YOLOv7-CACT-RT model was performed the best to detect the surface defects on the surface of citrus fruits in the grading and sorting production line with a transition rate of 10 fruits per second. The deployed YOLOv7-CACT-RT model was loaded into the grading software programmed by C++ language, in order to validate the performance. An online detection was conducted on 198 mixed normal and defective citrus fruits on the sorting line, achieving a detection accuracy of 94.4%. The improved model can be directly applied to grade and sort fruit in the production line, according to the external qualities. Meanwhile, this model can also be extended to the real-time surface defects detection of other fruits without specialized knowledge. Our future research will focus on the registration and fusion of RGB and NIR image, in order to improve the detection accuracy of fruit defects.