Abstract: To address the challenge of real-time monitoring of grain breakage and impurity rates during rice harvesting, this study proposes an intelligent monitoring system that integrates a stratified sampling device with an optimized deep learning model. The system achieves accurate, non-destructive, and online detection of rice harvest quality by simultaneously enhancing the physical structure for image acquisition and improving the detection algorithm for small, densely distributed targets. The hardware component features a three-stage intermittent sampling mechanism composed of a grooved wheel, porous deflector plates, and adjustable baffles. This design facilitates the layered dispersion of rice grains and suppresses particle overlap. The structure was optimized using Discrete Element Method (DEM) simulations in EDEM software, which revealed that a baffle gap of 7.5 mm, deflector angle of 50°, and conveyor speed of 0.15 m/s resulted in the lowest particle distribution coefficient of variation (CV = 0.152). At the same time, the average occlusion rate remained at a moderate level, indicating that the evaluation of grain stratification should not rely solely on CV but rather on the combined analysis of CV and occlusion rate to balance distribution uniformity and particle overlap. This condition therefore demonstrated superior uniformity and flowability of the grain spread. On the software side, the YOLOv8n network was enhanced for embedded application scenarios. Shuffle Attention (SA) modules were incorporated to strengthen the model’s ability to capture dense, small-object features, while the VoV-GSCSP module reduced parameter count and computational complexity. To address localization errors in overlapping targets, the conventional CIoU loss was replaced by the PIoU2 loss function, which incorporates normalized positional deviation and geometric penalty terms. Ablation studies demonstrated the effectiveness of these components: the improved model achieved a mean average precision (mAP@50) of 95.3%, F₁-score of 92.53%, and maintained a real-time inference speed of 126 FPS with only 2.886 million parameters. Field and bench-scale experiments were conducted using the “Lingliangyou-268” rice variety. In controlled bench trials, the predicted average error rates for grain breakage and impurity were 3.50% and 6.27%, respectively. Field tests using a 4LZ-3.2 combine harvester equipped with a Jetson Orin Nano-based embedded system yielded average relative errors of 5.53% for breakage and 6.74% for impurity under varying operating speeds. Both trials confirmed the system’s robust real-time performance, accuracy, and stability under complex agricultural conditions. This research contributes a cost-effective, lightweight, and highly accurate rice quality monitoring solution, suitable for deployment in intelligent agricultural machinery. The system's dual optimization in hardware and software provides a reliable framework for real-time field deployment, setting a foundation for future improvements in precision agriculture and automated crop harvesting.