Fruit detection and picking keypoint localization with state space model and geometric compensation for strawberry harvesting robots

Strawberry harvesting robots are vital for efficient harvesting and labor cost reduction. However, accurate strawberry detection and picking keypoint localization remain challenging due to dense fruit clusters, variable illumination, leaf and stem occlusions, and limited onboard computational resources. This study proposed STRAW-MAMBA, a lightweight state space model (SSM)-based network for accurate strawberry detection and keypoint localization in unconstrained environments. Specifically, the C2f-MAMBA block was developed to enhance global and multi-scale feature extraction for densely clustered or partly obscured strawberries. It redesigned the bottleneck layer of C2f as the Straw-Vim module, which integrates Hidden State Mixer-based State Space Duality (HSM-SSD) for global feature extraction and multi-scale convolutional attention (MSCA) for multi-scale feature extraction. Meanwhile, to preserve more fruit edge feature details, the large-stride convolution was optimized as a Haar wavelet downsampling module, thereby addressing illumination-induced edge blurring of strawberries while improving computational efficiency. In addition, the FasterNet block was superseded to reduce the model size further. Finally, a novel five-point geometric compensation method was proposed to address occlusions, reduce unpickable fruits, and minimize crop waste. Experiments show STRAW-MAMBA achieves 85.5% precision, 84.8% recall, and 89.7% mAP@0.5 for detection, and 88.7% precision, 77.5% recall, and 84.9% mAP@0.5 for keypoint localization, with 93.5 fps and only 1.9 M parameters. Ablation studies confirm each module’s effectiveness, demonstrating suitability for greenhouse strawberry harvesting robots. This study can provide an efficient and robust visual perception framework, significantly advancing the practical deployment of automated harvesting systems in precision agriculture.