Super resolution reconstruction model for the drone photography of ginkgo forest using generative adversarial networks

The growth status of the crops can be effectively monitored using unmanned aerial vehicle (UAV) multispectral images. However, the monitoring accuracy is often confined to the resolution of the remote sensing images. Image resolution can be reduced significantly, as the UAV flight altitude increases for the flight efficiency. In this study, an image super resolution (SR) reconstruction model, named residual transformer generative adversarial network (RTGAN), was designed to effectively balance the flight efficiency and monitoring accuracy. Firstly, a multispectral image dataset of the ginkgo canopies was constructed with the high resolution (HR) and low resolution (LR) remote sensing images. Among them, the HR images were captured by UAV at an altitude of 15 m. The dataset of the LR image consisted of LR30 and LR60, which was captured at the altitudes of 30 and 60 m, respectively. A series of the multispectral image preprocessing was performed on the raw images, including image stitching, radiometric calibration, multi-channel integration, image registration, and image cropping. The number of the preprocessed images reached 10,000 to form the real HR/LR image datasets of the ginkgo canopies. The RTGAN model was used to train the dataset. Next, the network loss function was improved to incorporate the pixel loss, adversarial loss, perceptual loss, and regularization loss. The SR architecture was comprised a generator and a discriminator network. Specifically, the generator network was optimized to introduce the multiple dense residual block (MDRB), in order to extract the global features from remote sensing images. While the discriminator network was integrated with the U-Net and Transformer module. The complex textures were processed to generate the high-quality SR images using RTGAN model. Finally, the accuracy of the ginkgo leaf yield prediction was also evaluated to verify the RTGAN model. Correlation analysis of the vegetation indices was performed for the optimal indices. Multiple linear regression (MLR), partial least squares regression (PLSR), and random forest regression (RFR) models were employed to establish the yield prediction models using HR, LR, and SR images. The results showed that the real HR/LR image dataset shared the detailed textures and structural features over the different resolutions, indicating the better reconstruction and generalization of the SR model. A comparison was also made on the yield prediction of the ginkgo leaf before and after SR. The high resolution was obtained for the texture of the ginkgo canopies after the SR by RTGAN. In the SR images, the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) increased by an average of 67.22% and 74.54%, respectively; Learned perceptual image patch similarity (LPIPS) and Fréchet inception distance (FID) decreased by an average of 84.42% and 90.50%, respectively; And the correlation coefficient (r) for the yield prediction accuracy of the ginkgo leaf was improved by 33.34%, thereby approaching the yield estimation accuracy of the HR images at the lower flight altitude (r = 0.83). Therefore, the RTGAN model with the SR technology was effectively enhanced the accuracy of the ginkgo yield prediction from the LR images, while maintaining the high flight efficiency. In summary, the RTGAN model was enhanced the robustness of the remote sensing images against environmental interference, fully meeting the practical demands of the large-scale monitoring. The finding can hold the significant potential for the application in the smart cultivation of the ginkgo