Maize (Zea mays L.) seedling detection based on the fusion of a modified deep learning model and a novel Lidar points projecting strategy

Accurate crop detection is the prerequisite for the operation of intelligent agricultural machinery. Image recognition usually lacks accurate orientation information, and Lidar point clouds are not easy to distinguish different objects. Fortunately, the fusion of images and Lidar points can complement each other. This research aimed to detect maize (Zea mays L.) seedlings by fusing Lidar data with images. By applying coordinate transformation and time stamps, the images and Lidar points were realized homogeneous in spatial as well as temporal dimensions. Deep learning was used to develop a maize seedling recognition model, then the model recognized maize seedlings by labeling them with bounding boxes. Meanwhile, Lidar points were mapped to the bounding boxes. Only one-third of points that fell into the right middle of bounding boxes were selected for clustering operation, the calculated center of the cluster provided spatial information for target maize seedlings. This study modified the classical single shot multi-box detector (SSD) by merely linking the last feature map to the final output layer, owing to the higher feature maps having the unique advantages of detecting relatively larger objects. In images, maize seedlings were just the largest objects owing to be shot on purpose. This modification enabled the recognition model to finish recognizing an image by only consuming around 60 ms, which saved about 10 ms/image compared with the classical SSD model. The experiment was conducted in a maize field, and the maize was during the elongation stage. Experimental results demonstrated that the standard deviations for maximum distance error and maximum angle error were 1.4 cm and 1.1°, respectively, which can be tolerated under current technical requirements. Since agricultural fields are subject to staple crop-orientated and changeable ambient environment, the fusion of images and Lidar points can derive more precision information, and make agricultural machinery smarter. This study can act as an upstream technology for other researches on intelligent agricultural machinery.