Applying intelligent automation to the problem of pruning vines during the dormant season.

by George Kantor, Ph.D.

The CMU team is using robotics and sensing to address many important problems in vineyard production, but the effort I am most excited about right now is our effort to apply intelligent automation to the problem of pruning vines during the dormant season.  This work is being led by postdoctoral project scientist Abhi Silwal, and it involves a top-to-bottom approach that is developing the necessary perception, manipulation, and mobility capabilities in order to automate the entire process. The manipulation will be performed by a commercial robotic arm (Universal Robotics UR5) equipped with an extra degree of freedom and a custom pruning tool. For moving the arm around in the field, we will re-purpose our “Cave Crawler” robot, which is capable of agile motion on rough and muddy terrain. Abhi and his team are working hard in the lab to pull all of these pieces together in time to do an end-to-end demonstration of the concept during the current dormant season, hopefully sometime in February 2018.

One of our recent breakthroughs has been on the perception front. Here, we use the same imaging system that has been developed for crop load measurements to take pictures of dormant vines in the field. We use a deep learning architecture called Generative Adverserial Networks (GANs) to analyze the resulting images to detect the vines and estimate pruning weight. GANs are a recent breakthrough in artificial intelligence for image analysis. In a GAN, two neural networks play a game against each other. The first network (the “generator”) tries to artificially mimic the way a human would label vines and buds in an image by trying to match the distribution of training images that have actually been labeled by a human. The second network (the “discriminator”) tries to guess if a given labeled image is artificial (i.e., is created by the generator network) or a human-labeled image from the training set. During training, both networks evolve to improve their performance, getting better and better the more times they go up against each other. In this way, the generator network can learn to mimic a human with a relatively small human-labeled training set. This is a big advantage because labeling images is a time intensive process, and getting enough labeled images is typically the bottleneck in other deep learning approaches.  This is an important advantage, and we have hopes that the successes developed for vine detection can be transferred to improve our other perception efforts on the Efficient Vineyard project such as detecting and measuring fruit on the vine.

Applying intelligent automation

The image above shows the use of a GAN to automatically detect vines in an image. This particular GAN was trained using 40 manually labeled images, which is an order of magnitude fewer than other approaches.

Cave Crawler Applying intelligent automation

Cave Crawler is a high-mobility autonomous platform designed at CMU for the purpose of exploring abandoned coal mines. We are adapting it to carry the robotic pruning system through a vineyard.

Universal Robotics UR5 Manipulator Applying intelligent automation

A Universal Robotics UR5 manipulator has been mounted to a horizontal slide to test perception and manipulation required for vine pruning in the lab. We plan to integrate this with the Cave Crawler platform for testing in vineyards in February 2018.

This Post Has 3 Comments

  1. When will we see this pruner in the Lake Erie region? What kind of price range are you trying to keep this kind of equipment to?

  2. Thom, the pruner is currently being developed on a research level. We don’t have a specific plan in mind for commercialization. That said, if the technology could be refined for applications in a region with harsh winters, it would certainly address some of the labor issues facing LE growers.

    James: To my knowledge, no one has reached out to Vision Robotics, although I did stumble across that video a while back. Robotics is certainly a growing field of research and development, so it is great to see companies emerging that are addressing vineyard applications.

Leave a Reply

Your email address will not be published. Required fields are marked *