Abstract: An accurate and rapid identification of heavy metals (HMs) can be required to prevent environmental pollution and health risks near coal mining sites. However, the multiple contaminants have posed a great challenge to monitoring the heavy metals in agricultural soils. In this study, the hyperspectral inversion technique was developed for the soil heavy metals during pollution assessment and prevention. Spatial distribution maps were then constructed to quantitatively evaluate the heavy metal pollution in mountainous agricultural soils. An agricultural area was taken from the mountain farmland that was severely impacted by mining activities near a gold mine in Mojiang County, Yunnan Province, China. A total of 121 soil samples were collected from the study area. The laboratory hyperspectral data was obtained with the contents of heavy metals (arsenic (As), chromium (Cr), copper (Cu), and nickel (Ni)). The Nemerow pollution index and the potential ecological risk index were employed to assess the heavy metal pollution in the study area. Simultaneously, the CARS-IRIV (competitive adaptive reweighted sampling- iteratively retains informative variables) algorithm was applied to select the sensitive bands for the soil heavy metals. The hyperspectral inversion models were then constructed to estimate the contents of the various heavy metals in the soil. Additionally, the spatial distribution of soil heavy metals was obtained in the study area. The results indicated that: 1) The Nemerow pollution index revealed severe pollution in soil, while the potential ecological risk index suggested a moderate level of ecological risk. The first- and second-order derivatives, standard normal variate, and reciprocal transformations were found to significantly enhance the spectral responses. The efficient band selection of the CARS was combined with the variable refinement of the IRIV. The sensitive bands were successfully selected to explore the spectral response of heavy metals. The better performance of CARS-IRIV was achieved in the number of characteristic bands, computational runtime, and inversion accuracy, compared with the CARS and IRIV only. 3) The back-propagation neural network (BPNN) achieved the highest inversion accuracy for As, while the support vector machine (SVM) was identified as the optimal inversion model for Cr, Cu, and Ni. The R² values for the optimal inversion models of As, Cr, Cu, and Ni were 0.90, 0.93, 0.67, and 0.94, respectively. The root mean squared errors of external validation (RMSE) were 87.33, 142.63, 2.63, and 70.31 mg/kg, respectively, and the relative percent differences (RPD) were 3.25, 3.84, 1.74, and 4.17, respectively. 4) The spatial analysis indicated that the high-value areas of heavy metals in the study area were primarily concentrated in the upper and lower areas, while the low-value areas were along the edges. There was an overall trend of gradual decrease from the interior to the exterior. The distribution patterns of heavy metal concentration were attributed to the spatial heterogeneity of pollution under mining activities. The finding can provide a strong reference to monitor the heavy metal pollution in agricultural soils near mining areas. The data support can be offered for the monitoring accuracy of soil pollution for the subsequent assessment and remediation.