Research Projects
Strawberry cultivation is labor-intensive, particularly for tasks requiring dexterous manipulation, such as harvesting occluded strawberries. Imitation learning has emerged as a promising approach to enable robotic arms to perform such functions with human-like dexterity. However, training precise manipulation policies requires large-scale, high-quality demonstration data, which is often challenging to obtain in unstructured field environments. To address this challenge, we introduce the Strawberry Robotic Operation Interface (SROI)—an open-source device designed for collecting dexterous manipulation data in robotic strawberry cultivation. The SROI consists of a handheld unit with a modular end effector and a stereo camera system, facilitating efficient data collection in real-world agricultural settings. Additionally, we develop a data post-processing pipeline to extract spatial trajectories and gripper states from the collected demonstrations. To further support research in dexterous robotic manipulation, we released an open-source dataset of strawberry-picking demonstrations. By enabling systematic data collection, the SROI contributes to advancing the automation of complex strawberry cultivation tasks and reducing reliance on manual labor.
Strawberries naturally grow in clusters, interwoven with leaves, stems, and other fruits, which frequently leads to occlusion. This inherent growth habit presents a significant challenge for robotic picking, as traditional percept-plan-control systems struggle to reach fruits amid the clutter. Effectively picking an occluded strawberry demands dexterous manipulation to carefully bypass or gently move the surrounding soft objects and precisely access the ideal picking point—located at the stem just above the calyx. To address this challenge, we introduce a strawberry-picking robotic system that learns from human demonstrations. Our system features a 4-DoF SCARA arm paired with a human teleportation interface for efficient data collection and leverages an End Pose Assisted Action Chunking Transformer (ACT) to develop a fine-grained visuomotor picking policy. Experiments under various occlusion scenarios demonstrate that our modified approach significantly outperforms the direct implementation of ACT, underscoring its potential for practical application in occluded strawberry picking.
Recent breakthroughs in large foundation models have enabled the possibility of transferring knowledge pre-trained on vast datasets to domains with limited data availability. Agriculture is one of the domains that lacks sufficient data. This study proposes a framework to train effective, domain-specific, small models from foundation models without manual annotation. Our approach begins with SDM (Segmentation-Description-Matching), a stage that leverages two foundation models: SAM2 (Segment Anything in Images and Videos) for segmentation and OpenCLIP (Open Contrastive Language-Image Pretraining) for zero-shot open-vocabulary classification. In the second stage, a novel knowledge distillation mechanism is utilized to distill compact, edge-deployable models from SDM, enhancing both inference speed and perception accuracy. The complete method, termed SDM-D (Segmentation-Description-Matching-Distilling), demonstrates strong performance across various fruit detection tasks (object detection, semantic segmentation, and instance segmentation) without manual annotation. It nearly matches the performance of models trained with abundant labels. Notably, SDM-D outperforms open-set detection methods such as Grounding SAM and YOLO-World on all tested fruit detection datasets. Additionally, we introduce MegaFruits, a comprehensive fruit segmentation dataset encompassing over 25,000 images, and all code and datasets are made publicly available at https://github.com/AgRoboticsResearch/SDM-D.git.
Training real-world neural network models to achieve high performance and generalizability typically requires a substantial amount of labeled data, spanning a broad range of variation. This data-labeling process can be both labor and cost intensive. To achieve desirable predictive performance, a trained model is typically applied into a domain where the data distribution is similar to the training dataset. However, for many agricultural machine learning problems, training datasets are collected at a specific location, during a specific period in the growing season. Since agricultural systems exhibit substantial variability in terms of crop type, cultivar, management, seasonal growth dynamics, lighting condition, sensor type, etc, a model trained from one dataset often does not generalize well across domains. To enable more data efficient and generalizable neural network models in agriculture, we propose a method that generates photorealistic agricultural images from a synthetic 3D crop model domain into real world crop domains. The method uses a semantically constrained GAN (generative adversarial network) to preserve the fruit position and geometry. We observe that a baseline CycleGAN method generates visually realistic target domain images but does not preserve fruit position information while our method maintains fruit positions well. Image generation results in vineyard grape day and night images show the visual outputs of our network are much better compared to a baseline network. Incremental training experiments in vineyard grape detection tasks show that the images generated from our method can significantly speed the domain adaption process, increase performance for a given number of labeled images (i.e. data efficiency), and decrease labeling requirements.All code and datasets are made publicly available at https://github.com/plant-ai-biophysics-lab/CropGAN.
Headland maneuvering is a crucial part of the field operations performed by autonomous agricultural vehicles (AAVs). While motion planning for headland turning in open fields has been extensively studied and integrated into commercial autoguidance systems, the existing methods primarily address scenarios with ample headland space and thus may not work in more constrained headland geometries. Commercial orchards often contain narrow and irregularly shaped headlands, which may include static obstacles, rendering the task of planning a smooth and collision-free turning trajectory difficult. To address this challenge, we propose an optimization-based motion planning algorithm for headland turning under geometrical constraints imposed by headland geometry and obstacles. Our method models the headland and the AAV using convex polytopes as geometric primitives, and calculates optimal and collision-free turning trajectories in two stages. In the first stage, a coarse path is generated using either a classical pattern-based turning method or a directional graph-guided hybrid A* algorithm, depending on the complexity of the headland geometry. The second stage refines this coarse path by feeding it into a numerical optimizer, which considers the vehicle's kinematic, control, and collision-avoidance constraints to produce a feasible and smooth trajectory. We demonstrate the effectiveness of our algorithm by comparing it to the classical pattern-based method in various types of headlands. The results show that our optimization-based planner outperforms the classical planner in generating collision-free turning trajectories inside constrained headland spaces. Additionally, the trajectories generated by our planner respect the kinematic and control limits of the vehicle and, hence, are easier for a path-tracking controller to follow. In conclusion, our proposed approach successfully addresses complex motion planning problems in constrained headlands, making it a valuable contribution to the autonomous operation of AAVs, particularly in real-world orchard environments.
Accurate robot localization relative to orchard row centerlines is essential for autonomous guidance where satellite signals are often obstructed by foliage. Existing sensor-based approaches rely on various features extracted from images and point clouds. However, any selected features are not available consistently, because the visual and geometrical characteristics of orchard rows change drastically when tree types, growth stages, canopy management practices, seasons, and weather conditions change. In this study, we introduce a novel localization method that does not rely on features; instead, it relies on the concept of a row-sensing template, which is the expected observation of a 3D sensor traveling in an orchard row, when the sensor is anywhere on the centerline and perfectly aligned with it. First, the template is built using a few measurements, provided that the sensor's true pose with respect to the centerline is available. Then, during navigation, the best pose estimate (and its confidence) is estimated by maximizing the match between the template and the sensed point cloud using particle-filtering. The method can adapt to various orchards and conditions by rebuilding the template. Experiments were performed in a vineyard, and in an orchard in different seasons. Results showed that the lateral mean absolute error (MAE) was less than 3.6% of the row width, and the heading MAE was less than 1.72°. Localization was robust, as errors did not increase when less than 75% of measurement points were missing. The results indicate that template-based localization can provide a generic approach for accurate and robust localization in real-world orchards.
Orchard harvest-aid platforms are used in high-density orchards with SNAP (Simple, Narrow, Accessible, and Productive) tree architectures. In these orchards, trees are planted and trained into fruiting walls, and workers harvest the fruit while standing on the platform. A worker in the front – the “driver” - is responsible for controlling the platform speed and keeping it centered in the row while they pick fruit. The use of harvest-aid platforms improves the picking efficiency, safety, and ergonomics of the workers compared to ladder-based picking. However, the non-uniform fruit distribution in tree canopies results in an incoming fruit rate (demand for labor) that varies spatially and temporally and is not necessarily matched to the workers' picking positions and harvest speeds (supply of labor). This mismatch lowers the overall harvest efficiency. A previous study improved upon this mismatch by introducing an independent actuated lift for each worker and dynamically adjusting each worker's picking height based on the estimated incoming fruit distribution and the workers’ harvesting rates. The horizontal moving speed of the platform was not controlled. This work presents an integrated system that optimizes the platform's travel speed and lift heights in real-time to increase the platform’s harvest throughput. Simulation experiments using digitized fruit distributions investigated the algorithm's potential gain under different settings. Field experiments were performed in a commercial apple orchard in Lodi, CA, with Fuji apples on V-trellised trees using a “robotized” platform and two workers. Two modes were implemented in the experiments: the “conventional” mode, which represents current practice, where workers' heights are fixed, and the platform speed is adjusted by the front worker, and the “co-robotic” mode, where the optimizing algorithm dynamically adjusts worker's heights and platform speed. A total of 3,227 kg of apples were harvested during the experiment. The overall throughput of the “co-robotic” mode was 261.8 kg/h if apple stems were clipped and 501.1 kg/h if apple stems were not clipped. The corresponding overall throughputs of the “conventional” mode were 235.3 kg/h and 397.7 kg/h. The results showed that the “co-robotic” mode improved the harvesting throughput by 11% (clipping) and 25% (without clipping). The code for the models, optimization system, and simulation has been made available as open-source on https://github.com/AgRoboticsResearch/corobotic-platform.
Estimating and recording a worker’s picking rate during tree fruit harvesting can provide useful information for better workforce management, orchard platform crew management, and generation of yield maps (in combination with position). A commercial picking bag was instrumented to estimate harvested fruit weight in real-time. All electronics were placed inside an enclosure that was placed between the bag and its shoulder straps, without hindering picking motions. The electronics included two load cells to measure the forces exerted on the straps by the bag and fruits, an Arduino microcontroller, signal conditioning circuits, data storage, wireless communication components, and inertial sensors. Software was developed for data acquisition, filtering, transmission, and storage. Two calibration models were developed to estimate fruit weight. One model (model 2) used inertial sensor data to compensate for the picking bag’s angle with respect to gravity direction, while the other model (model 1) did not. Dynamic calibration experiments were performed over the entire weight range of the bag (0 to 20 kg) with reference objects of known weight (baseballs and fresh apples). The weight was divided into three ranges: light load (<8 kg), medium load (8 to 13 kg), and heavy load (>13 kg). Results showed that model 1 performed slightly better in the light load range, but model 2 was superior in the medium and heavy load ranges, presumably due to bag angle compensation. The best root mean squared error over the entire range was achieved by model 2 and was 0.36 kg (1.8% of bag capacity). In an application case study, two bags were used by workers harvesting from a platform in a commercial apple orchard. From the data, the pickers’ harvesting speeds were estimated, and the fruit yield distribution was calculated for one side of a tree row.