A half-day workshop at International Conference on Robotics and Automation
From 30 May to 5 June 2021
Xi’an – China

Previous workshops from this series:

TIG-II at RSS 2019 ( https://lcas.lincoln.ac.uk/wp/tig-ii/ )

TIG-I at IROS 2018 (https://www.birmingham.ac.uk/iros18)

Title: Empowering Growers to Exploit Adaptive Robots

Abstract: One of the key challenges faced by horticultural producers in the UK is labour shortage, driven by a combination of factors most recently travel restrictions due to the Covid-19 pandemic. Collaborative and learning robots are a potential solution where off-the-shelf automation is unavailable, but this presents the problem as to how to best equip growers to make use of this technology for their production setting. This talk will outline some of the efforts of the Robot Learning Lab at King’s in this area, ranging from training growers to become effective teachers of cobots, instrumentation of clothing to gain motor skill data, to direct automation of challenging fresh herb manipulation problems.

Bio: Dr Matthew Howard is a senior lecturer at the Centre for Robotics Research, Department of Engineering, King’s College London. Prior to joining King’s in 2013, he held a Japan Society for Promotion of Science fellowship at the Department of Mechanoinformatics at the University of Tokyo and was a research fellow at the University of Edinburgh from 2009-2012. He also obtained his PhD in 2009 at Edinburgh with an EPSRC CASE award sponsored by Honda Research. His research interests span the fields of robotics and autonomous systems, statistical machine learning and adaptive control. His current work focuses on teaching and learning of robotic motor skills by demonstration, especially for soft robotic devices, based on human musculoskeletal control. He works with a number of large companies in protected ornamental and food crop production, looking at automation through collaborative robotics in horticulture.

Title: Toward More Effective Reinforcement Learning

Abstract: Grasping is one of the most fundamental robotics problems, and becomes an even bigger challenge in task context. One of the big recent drivers towards better grasp policies has been data-driven approaches, especially reinforcement learning. Interestingly, it’s often only the data that’s grasping-specific, and the reinforcement learning algorithms uses are quite general and applicable across many other tasks. I believe that general advances in reinforcement learning will play a key role in the future of grasping and will present two major recent advances: (i) How to effectively combine representation learning (specifically contrastive learning) with reinforcement learning, resulting in learning almost as efficiently from image input as when given access to the underlying state. We evaluated this in the DMControl Suite as well as on a real robot. (ii) How to incorporate human feedback to avoid issues with reward misspecification. We showcase effective human-in-the-loop reinforcement learning, including teaching a range of behaviors in less than 1 hour of human time.

Bio: Professor Pieter Abbeel is Director of the Berkeley Robot Learning Lab and Co-Director of the Berkeley Artificial Intelligence (BAIR) Lab. Abbeel’s research strives to build ever more intelligent systems, with main emphasis on deep reinforcement learning and deep unsupervised learning. His lab also investigates how AI could advance other science and engineering disciplines. Abbeel has founded several companies, including Gradescope (AI to help instructors with grading homework and exams), Covariant (AI for robotic automation of warehouses and factories). Abbeel has received many awards and honors, including the PECASE, NSF-CAREER, ONR-YIP, Darpa-YFA, TR35. His work is frequently featured in the press, including the New York Times, Wall Street Journal, BBC, Rolling Stone, Wired, and Tech Review. 

Title: Soft robotics tackling object manipulation in kitchens and beyond

Abstract: Soft robotics focuses on the deformable and non-rigid character of our world. Soft robots made of deformable materials surpass the limited degrees of freedom of rigid robots and therefore potentially offer adaptable and inherently safe ways of achieving versatile forms in locomotion and manipulation. Soft robots can homogeneously combine actuation, sensing, and structural complexity within the same component. Two core challenges must be overcome to achieve capable soft robots: First, reproducibly construct their body with fully integrated actuation and sensing. Second, develop robotic “brains” to fully understand and exploit the unique properties of soft robots and the deformable world we life in. In the first part of this talk, we will discuss some of the challenges faced when one tries to make rigid robots handle soft and highly deformable food items for the preparation of meals in industrial kitchens. In the second part of this talk, we will cover some of our lab’s recent developments in soft robotics that overcome the challenges in the creation, modeling, and control of soft robotic systems.

Bio: Robert Katzschmann is an Assistant Professor of Robotics at ETH Zurich. Robert earned his Diplom-Ingenieur in 2013 from the Karlsruhe Institute of Technology (KIT) and his Ph.D. in Mechanical Engineering in 2018 from the Computer Science and Artificial Intelligence Lab (CSAIL) at the Massachusetts Institute of Technology (MIT) in 2018. Robert worked on robotic manipulation technologies as Applied Scientist at Amazon Robotics and as CTO at Dexai Robotics. In July 2020, Robert founded the Soft Robotics Lab at ETH Zurich to push robots’ abilities for real-life applications by being more compliant and better adapt to their environment to solve challenging tasks. His group develops soft robots whose compliant properties resemble living organisms and advances modeling, control, and learning techniques tailored to the needs of soft robots. His work has appeared in leading academic journals, including Science Robotics, and has been featured in major news outlets, including the New York Times. Robert is a member of the ETH AI Center, the Max Planck ETH Center for Learning Systems (CLS), and the ETH Competence Center for Materials and Processes (MaP). Robert is an Area Chair for Robotics Science and Systems (RSS), a Guest Editor for the International Journal on Robotics Research (IJRR), and a reviewer for leading peer-reviewed journals, including Science and Nature.

Title: AI, Machine Vision and Robotics for Fruit Harvesting and Fruit Tree Pruning

Abstract: Decreasing availability and increasing cost of farm labor is a critical challenge faced by agricultural industry around the world. Robotics has played a key role in reducing labor use and increasing productivity in farming. Modular automation and robotics technologies developed in recent years, decreasing cost and increasing capabilities of sensing, control and automation technologies such as UAVs, and increasing emphasis by governments around the world in mechanizing and automating agriculture have created a conductive environment to develop and adopt smart farming technologies for the benefit of agricultural industries around the world including smaller, subsistence farming operations common in developing and underdeveloped countries. In this presentation, the author will first discuss the importance of precision and automated/robotic systems for the future of farming (Smart Farming, Ag 4.0). He will then summarize past efforts and current status of agricultural automation and robotics, particularly for fruit harvesting and fruit tree pruning, followed by an introduction of the novel systems being developed in his program. The technologies to be introduced include robotic fruit harvesting, targeted shake-and-catch harvesting, and fruit tree and berry bush pruning. At the end, major challenges and opportunities in agricultural robotics and related areas, as well as potential future directions in research and development will be discussed

Bio: Dr. Manoj Karkee is an Associate Professor in the Biological Systems Engineering Department at Washington State University (WSU). He received his undergraduate in Computer Engineering from Tribhuvan University, Nepal and his MS in remote sensing and GIS from Asian Institute of Technology, Thailand . His PhD was in Agricultural Engineering and Human Computer Interaction from Iowa State University. Dr. Karkee leads a strong research program in the area of sensing, machine vision and agricultural robotics at the WSU Center for Precision and Automated Agricultural Systems. He has published widely in such journals as ‘Journal of Field Robotics’, ‘Computers and Electronics in Agriculture’, and ‘Transactions of the ASABE’, and has been an invited speaker at numerous national and international conferences and universities. Dr. Karkee is currently serving as an elected chair for International Federation of Automatic Control (IFAC) Technical Committee 8.1, Control in Agriculture, as an associate editor for ‘Transactions of the ASABE’, as a guest editor for ‘Sensors’, and Editorial Committee Member of Information Processing in Agriculture. Dr. Karkee was awarded ‘2020 Railbird Engineering Concept of the Year’ by American Society of Agricultural and Biological Engineers, and was recognized as ‘2019 Pioneer in Artificial Intelligence and IoT’ by Connected World magazine.

Title:  Towards Vineyard Automation: Robot Pruning and Harvesting

Abstract: Grapevine winter pruning is a complex task, that requires skilled workers to execute it correctly. The complexity of this task is also the reason why it is time consuming. Considering that this operation takes about 80-120 hours/ha to be completed, and therefore is even more crucial in large size vineyards, an automated system can help to speed up the process. To this end, this talk presents a novel multidisciplinary approach that tackles this challenging task by performing object segmentation on grapevine images, used to create a representative model of the grapevine plants. Then a set of potential pruning points is generated from this plant representation. The technologies will be applied to various mobile platforms, the wheeled platform as well as the quadruped robot which is able to adapt to various challenging terrains. Similar pipeline is also implemented for the table grape harvesting application.

Bio: Dr. Fei Chen is an assistant professor with T-Stone Robotics Institute (CURI), The Chinese University of Hong Kong (CUHK), as well as the Hong Kong Center for Logistics Robotics (HKCLR). Before joining CUHK in 2020, he has been leading the Active Perception and Robot Interactive Learning (APRIL) laboratory with Department of Advanced Robotics at Italian Institute of Technology (IIT), Italy. He was the IIT PI of several EU and Italian projects. His research interests lie in robot perception, learning, planning and control of mobile manipulation for various application, such as manufacturing, health-care, agri-food. He is the co-chair of IEEE Robotics and Automation Society Technical Committee on Neuro-Robotics Systems, as well as chairs for several international conferences and workshops.

Title: Planning the Harvesting Sequence in Complex Environment for Fruit-picking Robot

Abstract: This talk will introduce a piece of work about planning the sequence of tasks for automated strawberry harvesting. In this research, we created a video game in order to collect data for an application of learning from demonstration during the Covid-19 pandemic. The game simulates the arrangement of strawberries on a farm, and it requires players to make decisions about harvesting. The data we collected from players indicates that there is a pattern to human behaviour when planning the sequence in which strawberries are harvested.

Bio: Zhuoling Huang is a PhD student in Computer Science at King’s College London supervised by Professor Simon Parsons and Professor Elizabeth Sklar. Her research interests mainly focus on the design of automated fruit harvesting robot, especially strawberry-picking robots that can work in a complex environment.

Title: Tactile sensing for robotic manipulation and fruit quality control

Abstract: Most of today’s robots are employed in heavy manufacturing industries (e.g. automotive): however, future robots will be more and more employed in other applications, e.g. light manufacturing industries, logistics, healthcare, agriculture. To achieve this, several improvements in robotic dexterity will be needed. Explicit insights from biology and psychology, well established control and engineering principles, modern AI techniques, novel sensing devices, have to be combined and properly integrated. In the talk I will discuss some of our recent activities in this area, focusing in particular on new soft tactile sensors and their use for robotic object manipulation and fruit quality control.

Bio: Lorenzo Jamone is a Senior Lecturer in Robotics at the Queen Mary University of London (QMUL), where he is leading the CRISP Team. He received his BSc and MSc degrees in computer engineering from the University of Genoa in 2003 and 2006 (with honors), and his PhD in humanoid technologies from the University of Genoa and the Italian Institute of Technology (IIT) in 2010. Before joining QMUL in 2016, he was a Research Fellow at the RBCS Department of the IIT in 2010, Associate Researcher at Takanishi Lab (Waseda University, Tokyo, Japan) from 2010 to 2012, and Associate Researcher at VisLab (the Computer Vision Laboratory of the Instituto de Sistemas e Robotica, Instituto Superior Tecnico, Lisbon, Portugal) from 2013 to 2016.  Lorenzo’s main research interests include: sensorimotor learning and control in humans and robots; robotic reaching, grasping, manipulation and tool use; force and tactile sensing; intelligent systems and cognitive developmental robotics.

Title: Generative Attention Learning – A “GenerAL” Framework for High-Performance Multi-fingered Grasping in Clutter Using Vision and Touch

Abstract: Generative Attention Learning (GenerAL) is a framework for high-DOF multi-fingered grasping that is not only robust to dense clutter and novel objects but also effective with a variety of different parallel-jaw and multi-fingered robot hands. This framework introduces a novel attention mechanism that substantially improves the grasp success rate in clutter. Its generative nature allows the learning of full-DOF grasps with flexible end-effector positions and orientations, as well as all finger joint angles of the hand. Trained purely in simulation, this framework skillfully closes the sim-to-real gap. To close the visual sim-to-real gap, this framework uses a single depth image as input. To close the dynamics sim-to-real gap, this framework circumvents continuous motor control with a direct mapping from pixel to Cartesian space inferred from the same depth image.  This framework also demonstrates inter-robot generality by achieving over 92% real-world grasp success rates in cluttered scenes with novel objects using two multi-fingered robotic hand-arm systems with different degrees of freedom.  We also show how we can close the remaining failure cases with the use of tactile sensing.   When this tactile methodology is used to realize grasps from
coarse initial positions provided by a vision-only planner, the system is made dramatically more robust to calibration errors in the camera-robot transform.

Bio: Peter Allen is Professor of Computer Science at Columbia University, and Director of the Columbia Robotics Lab.  His work includes building the GraspIt! grasping simulator, the IREP surgical robot, and the Visibot disposable laparoscope.  He is the recipient of the CBS Foundation Fellowship, Army Research Office fellowship, the Rubinoff Award for innovative uses of computers, and the NSF PYI award.  His current research interests include robotic grasping, medical robotics and Brain-Computer Interfaces for Human-Robot interaction.

Title: Mobile robotics and HRI in the fruit harvesting chain 

Abstract: As part of the overall harvesting challenge, product flow post-picking, and the in-field logistics pose additional challenges and opportunities for automation agriculture. What happens to a crop once it has been successfully removed from the plant? How can we ensure logistics can keep up with harvesting speed? And how can we facilitate the transition from the current manual harvesting arrangements gradually towards further automation to offer the productivity gains the sector needs. In this talk I’ll reflect on some of the recent R&D developments in the context of strawberry production, ranging from fleet coordination (offering scalable solutions in robot harvesting) to interactions with human workers in the farm environment. 

Bio: Marc Hanheide is a Professor of Intelligent Robotics & Interactive Systems in the School of Computer Science at the University of Lincoln, UK, and the director of the University’s cross-disciplinary research centre in Robotics, the Lincoln Centre for Autonomous Systems (L-CAS).

Title: Compliant end-effectors for food handling: the CLASH hand

Abstract: This talk presents the main features of the CLASH hand, a compliant end effector developed especially for food handling. The main feature of the hand is the variable stiffness, which allows its adaptation to the object weight and properties, while being able to withstand collisions with the environment. Embedded ToF and tactile sensors, among others, provide the information required to quickly recover from potential failures, and to continue operation even in the case of problems in the motors or tendons. The hand can thus pick very delicate and light objects, such as a strawberry, or heavy and rigid objects, such as a melon. The performance of the hand is evaluated using standard robotic manipulation benchmarks.

Bio: Maximo Roa is a Senior Scientific Researcher at the Institute of Robotics and Mechatronics in the German Aerospace Center – DLR. He is Group Leader for Robotic Planning and Manipulation, focused on development and implementation of locomotion and manipulation skills at different levels for industrial, service and humanoid robots. He received his doctoral degree in 2009 from Universitat Politecnica de Catalunya, and also holds the Project Management Professional (PMP) Certification since 2016. Dr. Roa is an IEEE Senior Member, and served as co-chair of the IEEE-RAS Technical Committee on Mobile Manipulation from 2013 until 2019. He is also member of ASME, IAU, and PMI.

Title: Feeding the world with 8-armed outdoor robots

Abstract: Our world’s agricultural system faces an urgent need to automate most aspects of fruit and vegetable production. Automated growing and harvesting requires a robotics platform of unprecedented scope and capability, many aspects of which have yet to be solved. Successful grasping and manipulation requires perceiving a complex scene, accurately determining the position of an object in 3d, placing an end effector at that position using a robot arm mounted on an unstable platform, and planning routes that avoid obstacles and optimize performance. This must all be done quickly in order to achieve important economic requirements. Finally, it must be reliable enough to pick 100,000s of berries per day, 6 days/week, 300+ days/year, for years on end. We rely on recent advances in machine learning and other areas of robotics combined with novel improvements to certain critical areas in order to deploy a robotics platform that solves an urgent need.

Bio: Lewis Anderson is Co-founder and CEO of Traptic, which he started to solve a critical labor shortage in the world’s food production system using giant robots. Currently, $200B worth of produce is picked by hand, but a large percentage is wasted because not enough people are available to pick it. Traptic’s giant robots are saving the world’s food production system by doing the work people don’t want to do. Before starting Traptic, Lewis built software for PowerPoint at Microsoft which has been used by hundreds of millions of users. He started his career building autonomous airplanes and studying Computer Science at UC San Diego.