Photospectral Data Obtaining with the Unmanned Aerial Spectrometry Vehicle

Study of the Earth’s surface objects reflectance characteristics with unmanned aerial vehicles is one of the most actual remote sensing trends. Aim of this work was to develop a method for obtaining of photospec - tral data using unmanned aerial spectrometry vehicle. An adaptation of the cameras spatial resolution evaluating technique based on a specialized target photographic fixation was proposed. A method for synchronizing of the camera and spectrometer of the videospectral device was also proposed. It was based on an experiment with spectra and screen images recording. Different colors were sequentially displayed on the screen. The percentage contribution of each of colors to the “mixed” spectra was calculated. So the out-of-sync time estimation became possible. In addition the work proposed the method for combining images and spectra with their merging into photo - spectral images. The method allows to consider the aircraft displacement when linking the spectrometer field of view to the RGB image. The way for photospectral images combining based on the images key points detectors was also proposed. Spatial resolutions for 3 aerial vehicle cameras were obtained. The study showed that the spatial resolution decrease of Zenmuse H 20 T caused by the device carrier movement with a speed of up to 5 m/s can be ignored. The videospectral device camera and spectrometer out-of-sync time was evaluated. An automatic merging of a set of images using key points detection was made. The spectrometry areas were linked to the panoramic image. The reflectance coefficients were obtained for each of the areas in the range of 350–900 nm. The areas to image linking accuracy was 84.9 ± 11.6 %. A discrepancy between the angular spatial resolution values got experimentally and theoretically was revealed as a result of the cameras spatial resolution evaluating. This indicates the importance of the imaging equipment spatial resolution experimental evaluation. The videospectral device spectrometer and observation camera out-of-sync time evaluation made it possible to correct the data recording time. This led to the timing error standard deviation reduction from 142 ms to 15 ms. The way for the unmanned aerial spectrometry vehicle data obtaining in a photospectral representation was developed. The proposed methods and techniques can be used in similar unmanned systems.


Introduction
Surface images recorded using unmanned aerial vehicles (UAVs) are widely used in the modern world, along with data obtained using space-based remote sensing equipment [1].
Remote sensing using UAVs has its advantages compared to satellite systems: high spatial resolution, high speed of information acquisition, ease of use, no cloud cover influence, low cost [2].UAV data is currently widely used in agriculture and forestry [3][4][5].As a result, UAV data acquisition and processing techniques are developing rapidly.
The devices with multispectral cameras are the most widely used among UAVs [1].Such systems have a set of undeniable advantages, for example, the possibility of obtaining geographically referenced images of vegetation indices for vast areas.At the same time, multispectral systems allow recording data for subsequent analysis only for a limited number of vegetation indices.It is caused, firstly, by limited number of spectral bands and, secondly, by vast bandwidths.
The development of videospectral systems allowing to record high spectral resolution spectra combined with the underlying surface images is one of the approaches to expand the scope of remote sensing data obtained using UAVs.The unmanned aerial spectrometry vehicle (UASV) became such a system [6].The final stage in the vehicle development was the technical characteristics analysis: the resolution of the spectrometer and observation camera, the system modules angle of view, the system modules level of synchronization, etc.
To date, there are no state standards for evaluating the angular spatial resolution per pixel of aerial photosystems.At the same time, there is a technique describing a way of the spatial resolution evaluating based on a flight experiment results [7].However, this technique does not describe the requirements for the target intended to evaluate the spatial resolution and allows to find only the linear resolution of the system without assessing the uncertainties.As a result, it needs to be supplemented and adapted for use in systems like the UASV.
There are no studies describing modules synchronism level in systems like the UASV.Very little attention is paid to videospectral systems due to the difficulty of obtaining data from such systems in a representation convenient for analysis (for example, a geotagged RGB-image with a well-defined area on it corresponding to a surface reflectance spectrum).To obtain such a representation, it is necessary to apply data preprocessing methods, including modules synchronization, georeferencing, correction of distortion effects in images.In addition, there is a task of images combining with the use of computer vision libraries.This task becomes more difficult when spectrometer and camera exposition times are considerably different.
There are several tasks that this study solved: elaboration and applying of the observation camera spatial resolution evaluating methodology based on the flight experiment results; development of the method for the data of the spectrometry vehicle spectrometer and observation camera synchronizing; development of the method for obtaining the UASV data in a presentation convenient for analysis.Thus, the aim of the study was to develop the method for obtaining a photospectral data using the UASV.

Devices and methods
The UASV in the configuration used is based on the UAV DJI Matrice 300 RTK (Real Time Kinematic).The positioning accuracy of it, when the RTK system is on, is 1 sm ± 1 mm/km in horizontal axes and 1.5 sm ± 1 mm/km in vertical axes (hovering accuracy in P-GPS mode is ± 0.5 m) 1 .The videospectral device (VSD) and the Zenmuse H20T quad-sensor camera on a stabilizing gimbal2 can be installed on the UAV simultaneously or separately (Figure 1).
The VSD (Figure 2) includes the single-board computer (SBC) Raspberry Pi 4 Model B 3 .It controls in automatic mode the observation camera (Raspberry Pi Camera) and the spectrometer (connected through the microcontroller STM32F405RGT64 ) mounted coaxially.The observation camera allows to register RGB-images with resolution up to 2592 × 1944 pixels in photography mode.The spectrometer optical scheme is built according to the Rowland scheme [8]  grating with an additional flat mirror to reduce the dimensions.The spectral range of the spectrometer is 350-900 nm, the spectral resolution is 4 nm.
The spectrometer angular field of view (FOV) is 0.7×1.2degrees, the observation camera FOV is 15 × 27 degrees.The VSD also includes a power supply module with 10 Ah capacity.The body of the VSD is made using three-dimensional printing with three types of filaments for various parts.
The data registered by the VSD are preprocessed and stored on the flash drive connected to the SBC.The UASV output data is a video se-quence of data divided by directories (30 frames per second, 1920 × 1080 pixels) and data registered with the spectrometer (the recording frequency depends on the exposure of the spectrometer and is about 10 spectra/s).Geographic location data during measurements is recorded and stored on the flash drive connected directly to the UAV.

The observation camera spatial resolution evaluation
The information from the article on the aerial photosystems linear resolution per pixel estimating methodology based on the flight tests [7] were taken as a basis to determine the spatial resolution of the UASV cameras.It was complemented with recommendations described in the state standards for the resolution evaluating methods and photogrammetry techniques 5 .
The first step for holding an experiment to evaluate the spatial resolution in flight conditions 5 Photogrammetry.Requirements for creating oriented aerial images for building stereo models of built-up areas (GOST R 58854-2020, in Russian); Method for determining the photographic resolving power (GOST 2819-84, in Russian) is to determine the theoretical linear resolution per pixel using the formula: where L 0 is theoretical linear resolution per pixel; H is shooting height; f is camera lens focal length; δ is pixel size in the camera matrix.
It is necessary to make a target for experimental evaluation of the camera spatial resolution based on the theoretical values calculated with the formula (1) for all cameras needed.The minimum distance between alternating white strokes has to be equal to the calculated value L 0 for a given flight altitude, and the maximum distance between alternating white strokes has to be equal to the expected experimental L 0 value for a given flight altitude.It is important to note that formula (1) is a rough below estimation of the camera spatial resolution.For more accurate calculations, in addition to the focal length of the camera lens, the shooting height and the physical size of the matrix pixel it is needed to consider the area, diameter and transmission of the input lens, the specific detectivity of the photodetector, as well as the speed of the UAV during the experiments [9].
The UASV incorporates several cameras, which resolutions are needed to be evaluated experimentally: the VSD observation camera and the Zenmuse H20T camera including observation, wideangle and radiometric thermal cameras.The Zenmuse H20T observation camera is a zoom camera, so it stands out from the general list.As a result, it was not included it in the current analysis.It is necessary to calculate the L 0 value for heights of 50 and 200 m using the formula (1) for all the cameras (minimum and maximum flight altitude to evaluate the spatial resolution in this study).After that, it is necessary to choose extremum L 0 values.These values will be the basis for the line width border values in the target.
The cameras parameters, as well as the theoretical linear resolutions for various flight altitudes, are shown in Table 1 (L 0 50 is the theoretical linear resolution per pixel for 50 m altitude, L 0 200 is the theoretical linear resolution per pixel for 200 m altitude).
In general, the target should consist of a set of line groups with different frequencies.Each group consists of five light parallel lines on a dark background.The spacing width between the lines must be equal to the line width in the group.The line length to width ratio is constant and equal to 10. Wherein the width of the lines in the target should decrease from the previous group to the next in a geometric progression with a 0.91 denominator.A chart target based on the data from Table 1 was designed.The minimum and maximum distances between white lines was respectively 32 mm and 100 mm.As a result, the target consists of 13 groups of lines.The width of lines depending on the group number is presented in Table 2.The general view of the chart target is shown in Figure 3.

Table 2
The chart target lines width depending on the group number Group number (N) The cameras spatial resolution evaluating experiment have to be carried out in the absence of shading the measurement area surface with any objects [7].Spatial resolution measurement is performed on at least 20 target images for each altitude.Four independent decryption operators are recommended for image analysis.The work of the operators is to recognize groups of lines with all lines differ along their entire length.Next, the experimental angular resolution of the cameras is determined by the formula: where θ is the experimental angular resolution for a group of lines; l is the target line width from the group of lines recognized by the operator; H is the UAV flight altitude.
It is necessary to recognize the tightest groups of lines in each of the images for a given flight altitude, and then calculate the average value and the standard deviation (SD) of the experimental angular resolution per pixel over the entire set of images from a given camera at a given flight altitude.At the same time, since the flight speed of the UAV affects the resolution ( [9]), in this study measurements were carried out both in static mode (when the UAV was at one point during the shooting) and in dynamic mode (when the UAV was moving during the shooting).
This study realizes statistical processing of the results on the basis of state standards in the field of laboratory measurements evaluation 6 .At the first stage, it was estimated that the measurement results fell into the critical range at a 95 % confidence level for the number of independent estimates equal to the number of decryption operators.Then the average or median value (depending on the scope of all operators results) for all processing results for each group was calculated.If several independent groups of measurements were obtained to evaluate the spatial resolution at a given altitude (for example, with different lighting conditions), the Cochran criterion statistics were calculated.As a result, measurement groups with outlier characteristics were excluded.After that, the values of angular resolutions obtained for different flight altitudes were averaged.The number of independent degrees of freedom in the evaluation of extended uncertainties was chosen equal to the total number of decrypted images for all groups of measurements for a particular camera.The coverage ratio was evaluated based on this.

The unmanned aerial spectrometry vehicle modules data synchronization
The task of synchronizing the spectrometer and the VSD observation camera was the next task needed to solve for the UASV thematic data processing possibility.This was necessary for accurate binding of the spectrometry area to the RGB image both recorded by the VSD.The highest level of synchronization could be achieved with hardware methods, but they were not available with current UASV realization.Therefore, a software method of equipment synchronization was chosen in this work.
The VSD registers the observation camera video with a 30 Hz frequency and spectra with a 10 Hz frequency in flight mode.Subsequently, the registered data is processed: the observation camera data is matched to the spectrometer data.Each data set corresponds to the registration time related with the SBC time.However, both data sources have their own delays.The out-of-sync time is determined here as the time difference between the own delays of two data sources.
In addition, it is important to check the constancy of such a time difference over a long period of measurements (≈ 30 min).
The new additional software module designed for various colors photo images broadcasting on a screen was created to solve the problem.In this case it was possible to make a videospectral data registration using the VSD in laboratory conditions.A certain screen changing colors frequency could be set.Then it was possible to evaluate the out-of-sync time of the VSD observation camera and the VSD spectrometer.
Spectra registered during the measurements were "mixed" from initially selected color pairs due to time displacement.The percentage contribution of each of the colors to the "mixed" spectra was calculated with the method for evaluating the composition of a spectrum mixture (active set algorithm, [10]).Thus, the out-of-sync time needed could be estimated.

Images and spectra merging
Two RGB images are allocated for each spectrum: a frame of the beginning and a frame of the end of spectrometry.These are two images shifted relative to each other in time by the spectrometer exposure value (with an error caused by the out-of-sync time).
Merging of two images described above with common panoramic image generation can be made after radial and tangential distortion effects correction (for example, with the use of ChArUco method, [11]).This merging task was performed using the method of forming panoramic images based on key points detection [12].Special algorithms (detectors) were used for this.The numerical characteristics of key points in doing so are determined using other computing algorithmsdescriptors, where a descriptor is a kind of mathematical construct (usually a vector) that describes a key point in some way and allows to compare different points with each other [11].The ORB detector (Oriented FAST and Rotated BRIEF), the FREAK descriptor (Fast Retina Key-point) and the Hamming matching method were used as algorithms for searching and describing key points [11].A panoramic image merged from two frames is the result of this stage.
The part of image matching to the spectrometry area can be defined for any image registered with the VSD since the spectrometer FOV is rigidly connected with the observation camera FOV.In this case, for the panoramic image obtained at the previous stage, two regions can be distinguished.The actual spectrometry region bound to the RGB image becomes known after combining such regions (with the inclusion of the entire area between them).
The steps described above were repeated for the data set registered during the experiment.Comparison of videospectral data and UAV location data by timestamps was the last stage of UASV data preprocessing.Thus, each panoramic image was assigned with the coordinate of the UAV for the further photospectral data geotagging possibility.

Results of experimental studies and their discussion
The flight experiment was conducted on the territory of the educational geographical station of Belarusian State University "Western Berezina" to evaluate the spatial resolution of the cameras from the UASV on November 6-11, 2022.Data recording altitudes varied from 50 to 200 m (different cameras require different shooting altitudes due to different theoretical spatial resolutions).The values calculated with the use of Table 1 and experimentally obtained angular spatial resolution values estimation (θ) and their SD (σ) for various cameras are presented in Table 3.
Table 3 shows that theoretical cameras spatial resolution may differ significantly from the experimental value.Thus, the theoretical resolution differs from the experimental one for the Zenmuse H20T wide-angle camera by 30 %.Meanwhile, the relative extended uncertainty of angular resolution for 95 % confidence probability was 14 %.The theoretical resolution for the Zenmuse H20T thermal imaging camera differs from the experimental one by 20 %.The relative extended uncertainty of angular resolution for 95 % confidence was 7.7 %.Various factors may be the reason for the differences in theoretical and experimentally obtained resolutions: features of the frequency-contrast characteristics, the quality of the lens, the accuracy of its installation, as well as the influence of external factors.
An experiment to evaluate the spatial angular resolution was also conducted while the UAV is moving at a speed of 5 m/s in addition to the static mode experiment.Further sequential measurements were carried out in static and dynamic modes in similar light conditions.The results of the resolution estimates for the Zenmuse H20T cameras (θ) and their SD (σ) are shown in Table 4.
Table 4 shows that angular spatial resolution relative difference for various measurement modes was 6.9 % for a thermal camera and 1.6 % for a wide-angle camera.Meanwhile, standard deviations of the values were comparable or higher than the resolution difference.This suggests that the spatial resolution deterioration caused by the shooting equipment carrier movement at a speed of up to 5 m/s can be ignored in experiments with the Zenmuse H20T camera.An experiment to find out-of-sync time for the VSD observation camera and the VSD spectrometer was made.The measurements of the timevarying test screen were carried out.Spectra of "pure" colors were recorded, as well as "mixed" from pairs of initially selected colors due to time displacement.
The out-of-sync time evaluation is shown by the example of a mixture of two colors pair spectrum: (0, 255, 255) and (0, 255, 0). Figure 4 shows spectrum of screen with colors (0, 255, 255) and (0, 255, 0) in "pure" (Figure 4a) and "mixed" (Figure 4b) forms.Figure 5 shows a time diagram corresponding to the registration of spectrum shown in Figure 4b.Time moments t 1 , t 2 and t 4 shown in this figure are the moments of color change in camera images.Time moments t 3 and t 5 are the moments of the beginning and the end of spectrometry.The percen-tage of the energy component in the spectrum can be used as a percentage of the particular color type re-gistration time for time shifting evaluation, because the color change period T was equal to the spectro-meter exposure time t с in the experiment.Thus, it is possible to evaluate the out-of-sync time for each spectrum in the recorded data series.Data registration have been carrying out for 30 min.After analyzing the entire volume of information, it was found out that the out-of-sync time was on average 128 ms.It changed by an average of 15 ms during the entire time of the experiment.Thus, the FOV displacement caused by a synchronization error (Figure 6) was average from 0.71 pixels at a UAV speed (V) of 1 m/s (with a SD equal to 0.56 pixels) to 3.56 pixels at a UAV speed of 5 m/s (with a SD equal to 2.79 pixels).The FOV displacement ∆y t (with a SD ) in the direction perpendicular to the average velocity vector may be caused by the UAV roll angles changes (the vibration isolation used in the UASV design at the VSD installation site is not enough to compensate for the UAV rolls), as well as wind load.This calculation assumes that the displacement velocity in the direction perpendicular to the average velocity vector is equal to the modulus of the average specified UAV flight speed.In this case, with the flight altitude of 100 m, the UAV speed of 1 m/s and the spectrometer exposure of 100 ms, the FOV position relative accuracy considering synchronization errors was: where S is the area of the spectrometer's FOV (in pixels) after its binding to the RGB image; is the spectrometer FOV area change caused by displacement due to synchronization errors; is the spectrometer FOV area change caused by the uncertainty of the FOV displacement; k is coverage factor for 95 % confidence probability used in the extended measurement uncertainty calculation.Videospectral data registered in flight experiment needed to be preprocessed.The video sequence data were converted first to pairs of images, and then to the form of geographically linked panoramic images.The direction and magnitude of the offset between the two images allow determining the position of the spectrometer FOV in the panoramic image (Figure 7), since the spectrometry area is linked to each of the images with an accuracy determined only by the VSD synchronization.It is also possible to generate a panoramic image with several spectrometry regions attached to it after merging together several images (Figure 8).Ground-based registration of solar radiation reflection spectra from the MS-10 milk glass was carried out using the FSR-02 photospectroradiometer with the "Calibrovka" hardware-software system [13].It was made synchronously with flight measurements in the flight area and was needed to obtain the spectral brightness coefficients based on the spectrometry results.The FSR-02 photospectroradiometer has spectral resolution and spectral range similar to the VSD spectrometer.The data of both spectrometers was recalculated into the energy brightness spectral density values using their radiometric calibrations.After that, the spectral brightness coefficients spectra of the underlying surfaces were calculated.
Methods based on the use of key points detectors and descriptors cope well with the task of merging images into a panoramic image.The average deviation of the offset between key points ∆x m when merging images accounting the shooting equipment distortion correction, is 2.29 pixels (with a SD σ m equal to 1 pixel).It is 1.53 pixels (with a SD of 0.48 pixels) while determining the displacement magnitude in the region close to the spectrometry area.These values include the displacement both along the axis collinear to the average velocity vector and along the axis perpendicular to this vector.Then, by analogy with (3), at the flight altitude of 100 m, the UAV speed of 1 m/s and the spectrometer exposure of 100 ms, the FOV position relative accuracy considering merging errors was: where ∆S m is changes in the spectrometer's FOV area caused by displacement due to merging errors.
The dependence function of the general uncertainty value of the spectrometer FOV binding to the RGB image was taken as the sum of independent variables: a variable considering merging errors and a variable considering synchronization errors.In this case, basing on (3) and ( 4), the total relative accuracy of the spectrometer FOV to the RGB image binding was: ε = 0.849 ± 0.116.

Conclusion
The adaptation of existing techniques and recommendations for the spatial resolution of UAV cameras determining was carried out in the course of flight experiments with the unmanned aerial spectrometry vehicle.It consisted in considering several standards for determining the characteristics of photosystems, in the transition from determining the linear resolution per pixel to determining the angular resolution.It also allowed evaluating the accuracy of measurement results.The discrepancy between the experimentally obtained values and the theoretical values of the angular spatial resolution calculated on the cameras factory characteristics basis was revealed as a result of determining the unmanned aerial spectrometry vehicle cameras spatial resolution.This suggests the need for an experimental assessment of the shooting equipment spatial resolution to identify the real characteristics of the cameras.The proposed adaptation of the spatial resolution estimation technique can be used in systems similar to the unmanned aerial spectrometry vehicle.The spectrometer and the videospectral device observation camera out-of-sync time was determined using the developed method of software synchronization.The out-of-sync time was 128 ms.This made it possible to correct the time of data registration, which led to a decrease in the standard deviation value of the synchronization error from 142 ms to 15 ms.The proposed way of out-of-sync time estimation for various sensors of videospectral systems can be used in systems similar to the unmanned aerial spectrometry vehicle.
A method for obtaining the unmanned aerial spectrometry vehicle data in the photospectral representation was developed in the course of the study.It provides the spectrometer field of view to RGB images binding with a relative accuracy of 84.9 ± 11.6 %.Combining a set of photospectral images into a single cross-linked image allows the formation of sparse hyperspectral images.Thus, the unmanned aerial spectrometry vehicle is a promising system for getting remote sensing data for a wide range of the Earth surface studying and monitoring tasks.

Figure 1 -
Figure 1 -The unmanned aerial spectrometry vehicle general form (a), Zenmuse H20T camera with a stabilizing gimbal (b)

Figure 3 -
Figure 3 -General view of the chart target for the unmanned aerial spectrometry vehicle cameras spatial resolution evaluating

Figure 6 -
Figure 6 -Uncertainty of binding the spectrometer field of view to the RGB image during the UAV movement: 1 -spectrometer field of view in the data registration start moment; 2 -spectrometer field of view in the data registration end moment; w -spectrometer field of view size in the UAV flight direction; ∆w -spectrometer field of view bias caused by UAV movement

Figure 8 -
Figure 8 -Videospectral device data combining: connected image (a) and reflectance spectra corresponding selected areas (b) based on a concave diffraction

Table 3 Comparative table of theoretical and experimental angular spatial resolution values for different cameras in static mode
Devices and Methods of Measurements 2023, vol.14, no. 1, pp. 7-17 A.A. Lamaka et al.