Abstract: Black fungus has been ever-increasing in the market at present, due to its high nutritional value and remarkable economic benefits. However, the manual grading of black fungus quality cannot fully meet the large-scale production in recent years. In addition, the mesh machine filter can be only confined to the size of black fungus as the classification feature. A huge challenge has been posed on the classification accuracy of different quality black fungus on the market. In this study, a MICS-CoTNet network model was proposed to realize the quality grading for the various quality dried and fresh fungus using deep learning. The experimental data was collected from the black fungus cultivation base in Dunhua, Jilin Province, China. Firstly, the number of stacks was fine-tuned for the backbone feature layers of the CoTNet model. The activation function was then unified as the Gelu to reduce the computational redundancy in the model. The computational effectiveness of the model was optimized to improve the overall robustness of the model. Secondly, an improved attention module (known as CNAM) was proposed. In particular, the computational load of complex convolution in the coordinate attention (CA) was optimized by the normalized attention module (NAM), in order to avoid the feature loss from dimensional compression operations in the CA attention module. Thirdly, a backbone feature extraction module in the MobileNetV2 model (the Inverted Block) was introduced into the MICS-CoTNet model, in order to improve the recognition of detailed pixel information of black fungus images. Finally, a multi-scale convolutional module (MDSC) was proposed to optimize the local information extraction of CoT block, the core module of the MICS-CoTNet model. Specifically, the grouped convolution in the CoT block module was replaced by the multi-scale convolution module, which significantly improved the efficiency of model feature information transmission and learning capability. Six optimizers were selected to test the accuracy of the model recognition: SGD, Adam, RAdam, Adamw, RMSprop, and Ranger. The experiment demonstrated that the Ranger optimizer was used as the training model, where the convergence speed of the training model was faster and the accuracy of the model was better. The MICS-CoTNet model was verified to compare with 12 models, including CoTNet, EfficientNetV2, BotNet, ResNeSt, DenseNet, ConvNeXt, NfNet, GhostNet, MobileNetV3, ViT, Swin Transformer, and MobileViT. The MICS-CoTNet model was achieved the best performance in four evaluation indexes. The identification accuracy was 98.45%, the precision was 98.30%, the recall was 98.15%, and the F1 accuracy value was 98.20% in the dried black fungus. By contrast, the identification accuracy was 98.89%, the precision was 98.84%, the recall was 98.68%, and the F1 value was 98.75% in the fresh black fungus. In addition, the parameter capacity of the MICS-CoTNet model was reduced by 96.57 M, compared with the CoTNet model. The MICS-CoTNet model was deployed in the removable device Jetson TX2 NX, in order to achieve the real-time grading of various quality black fungus at an inference speed of (18 Frame/s).