The objective of the proposed software package EasyIDP (easy intermediate data processor) is to address previously identified difficulties and decrease the workload in intermediate data processing for agronomists and breeders.
.
Compared with traditional manual field mensuration, which is time consuming, labor intensive, and subjective, the recently emerged 3D reconstruction technologies provide a non-destructive and high-throughput solution for plant phenotyping.
As one branch of 3D reconstruction, photogrammetry (structure from motion and multi-view stereo (SfM-MVS)) technology, which requires only a standard digital camera, has been widely used on both ground and unmanned aerial vehicle (UAV) platforms in the open field. As the ground platform is often constrained by field conditions and vehicle in-field movement, the UAV platform opens the door for large-scale experimental fields with higher flexibility, efficiency, and throughput
The general workflow of UAV-based plant phenotyping by photogrammetry is demonstrated in Figure 1 with four main parts:
(1) The flight design stage. Setting ground control points (GCPs) appropriately [10] and designing an appropriate flight route with enough overlap between images are critical before flight tasks.
(2) The UAV imaging stage. Fly the UAVs with a predefined plan and collect and organize images to the computer. Most researchers remain manually operating at this stage. However, some fully automated UAV systems have started showing up recently, such as Scout, sunflower, and Airobotics, which can be expected to make this stage fully automated in the near future.
(3) The 3D reconstruction procedure. Commercial software, such as Pix4Dmapper (Pix4D, Lausanne, Switzerland) or Agisoft Metashape (Agisoft LLC, St. Petersburg, Russia), significantly decreases the workload of obtaining whole-field reconstruction in the format of the point cloud (PCD), digital surface model (DSM), and digital orthomosaic (DOM). Most of the steps at this stage can be processed by photogrammetry software once proper parameters are set. Although human intervention is required for those raw images for which GCPs cannot be precisely and automatically detected, some pipelines have been proposed to make this step automated.
(4) The plant phenotyping stage. Intermediate data processing is used to prepare the proper quality and size of plant data for later trait calculation. However, in the intermediate data processing steps, some challenges still exist.
First, the reconstruction quality is fundamental for all follow-up steps. However, the current quality assessment is still operated manually. When it comes to high-throughput phenotyping (HTP) applications in large-scale fields, this tedious manual step remains an ongoing challenge. One possible semi-automated solution is cropping each plot over time and detecting sudden changes.
The detected sudden change points may meet bad reconstruction quality. This idea relies on a handy tool for batch-cropping time series point cloud data, but no such tool has been developed yet.
Second, agronomists and breeders are often only interested in a specific area in the entire field, called the region of interest (ROI). ROIs can be individual plots with different treatments or production organs, such as lettuce leaves, broccoli flowers, sorghum, and rice head.
It is a standard operation to extract corresponding ROIs from the whole field for better data management. Alternatively, the whole field is split into several small parts, facilitating organ detection and segmentation.
Currently, this cropping step is still manually operated by geographic information system (GIS) software, for example, ArcGIS and QGIS .
When it comes to HTP in the large field, although GIS-programmable APIs and the “drone-dataflow” toolbox in MATLAB are available for batch processing, it requires professional programming and GIS operating skills. These tools are still difficult to directly focus on agriculture-related output.
Third, limited by the complex field conditions, the concatenated DOM often has image quality diminution compared to raw images.
A technique called reverse calculation, which links the same place from DOM back to relative raw UAV images, has attracted extensive interest, as it not only improves the ground cover estimation accuracy but also enables small-organ (e.g., sorghum head) detection tasks from UAV images.
However, previous studies have applied Pix4D, while Metashape is also used in many plant phenotyping studies, but these techniques have not been examined and supported yet. Furthermore, there are no handy tools available for agronomists and breeders without professional programming skills.
The objective of the proposed software package (easy intermediate data processor (EasyIDP), MIT license) is to address previously identified difficulties and decrease the workload in intermediate data processing for agronomists and breeders, including
(1) cropping both large PCD and DOM to small parts by given ROIs;
(2) reverse-calculating given ROIs on the corresponding place on raw UAV images for both Pix4D and Metashape projects; and
(3) testing the accuracy and performance of the developed functions using six different plots with various crops.
EasyIDP, a Python package, was proposed to decrease the workload in several manual operation tasks for HTP in large open fields, especially for assessing reconstruction quality and cropping ROIs from the whole field by GIS software. Meanwhile, the reverse calculation used in many studies for both Pix4D and Metashape was also included.
The results showed that this tool works, as expected, on both cropping tasks and reversing tasks on both software programs for a variety of crops. Manually marked references validated the reverse calculation accuracy.
The effects of ROI height selection and reversed ROI position were discussed. This tool also shows great potential for decreasing the data annotation workload in machine learning applications.
Read the study:
Wang, H, Duan Y, Shi Y, Kato Y, Ninomiya S, Guo W. (2021) EasyIDP: A Python package for intermediate data processing in UAV-based plant phenotyping. Remote Sens. 13, 2622.
