Dr Paul Robertson – CART: Pointing the Way to Reliable Robotic Assistants
Robots that assist us with both everyday and highly specialised tasks are no new concept, but so far, their actual development has appeared to be far from becoming a reality. Yet through the research of Dr Paul Robertson, Chief Scientist at Dynamic Object Language Labs (DOLL) in Massachusetts, such sophisticated technology is now looking increasingly feasible. By considering how robots can be programmed to recognise tasks and comprehend human emotions, his research could be bringing engineers a step closer towards reliable robotic assistants.
How to be a Good Assistant
The idea of robotic assistants has existed in the public consciousness for quite some time. These smart machines are often viewed as a highly desirable concept, but until now, they have mostly been confined to the realms of science fiction. Recent advances in the technology, including Amazon’s Alexa and Google Home, are perhaps the closest we have come so far. However, if truly reliable robotic assistants are to be fully realised, there are still several fundamental barriers to overcome.
Currently, technologies are limited in their capabilities, since programs lack a basic understanding of how humans think and behave. To offer assistance, they must either only carry out useful tasks when specifically asked, or offer help even when it is not needed, meaning they become invasive to everyday life. In addition, they can only be programmed to carry out very specific tasks; as such, the time required to obtain their assistance can be longer than the time they save. Clearly, the technology needs a new approach.
In his research, Dr Robertson and his team, building on earlier research in story understanding by team member Professor Patrick Winston, have drawn out a set of requirements for building truly useful robotic assistants, which incorporates models of what its users believe and think. If fully realised, the robots would be able to understand what a user is doing and why, recognise tasks they are struggling with, and offer them advice; though only when it is needed. Dr Robertson believes that such ‘Theory of Mind’ capabilities would allow for significant advances towards genuinely useful robotic assistants. The goal is achievable, but incorporating such human concepts of common sense into a machine is no easy task.
Aiding Repair Missions
As an initial demonstration scenario, and to avoid short-term limitations of robot dexterity, Dr Robertson has turned his attention towards robots specifically designed to aid repair missions. Such cases are clear examples of where a detailed knowledge of the tasks involved, and a recognition of human abilities, is particularly critical.
To realise these characteristics, Dr Robertson has developed a program named ‘Context-driven Active-sensing for Repair Tasks’ (CART). ‘The idea is simple,’ he explains. ‘How can a program reflect upon what it is trying to do, and the environment in which it is trying to do it, and to modify itself so as to be more effective? For me, that is the essence of what we call intelligent behaviour.’
Since CART’s initial development, Dr Robertson has made meaningful advancements towards developing robots that provide critical assistance during repair missions. ‘Today I am applying these ideas in the context of social robotics, of which the CART project is an example,’ he continues. ‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’
Therefore, by incorporating the principles of a Theory of Mind, robots programmed with CART can read repair manuals, recognise the different parts they depict, and build a ‘story’ of how they need to be pieced together to achieve the goals of the repair mission.
Achieving these capabilities has required Dr Robertson to consider a variety of key attributes of reliable robotic assistants. These include the ability of the machines to manoeuvre themselves to get a good vantage point of the repair operation, to understand what the repairer is doing, and to offer useful guidance in response to the person’s questions. In addition, the robot must use its own initiative to intervene with guidance if the story wanders off track, or if the conditions of the surrounding environment change. As it monitors all aspects of the mission, the robot presently assists the person both through verbal guidance and using physical gestures, but in the future it will participate physically as an able human assistant would.
One particularly important aspect which governs the effectiveness of these attributes is the ability of the robot to understand human emotions. Dr Robertson is achieving this through a modification to CART, named ‘Continuous Affective Robot Learning’ (CARL). This program integrates a ‘fear and reward’ system into CART, which causes robots to develop a ‘desire’ to see a story unfold in a certain way. CARL then ‘fears’ any mishaps, and losing sight of the objective, especially if such events have happened in the past.
Therefore, the program can learn to follow prior sequences in which its fears were mostly avoided. If this happens, the robot will feel a sense of ‘reward’, and so its desire to follow that path again in the future will increase. By desiring certain outcomes, therefore, CARL will react more effectively when potential setbacks arise. This addition of a simple model of emotions allows the robot to improve over time by collecting data about events whose importance is suggested by their affective import. This data is later used to improve future performance through reinforcement learning techniques.
‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’
In real-world situations, robotic assistants may likely be required to operate in environments presenting many distractions, which could prevent the machines from focusing on a user. Dr Robertson has considered this issue as well: through the integration of a ‘model of attention’ into CART. This model can transform noisy information into stable representations of the story by first acknowledging all objects the robot detects. It can then identify the specific objects that appear in the repair manual.
Thanks to the model, objects are more robustly tracked if they appear in an upcoming part of the story, which elevates the levels of attention they receive. In contrast, objects that have already been used, and do not appear again in the story, can be completely forgotten. Through its knowledge of the story, CART can recall what is supposed to happen in the future, allowing its user to focus their attention only on the tasks immediately required of them. The story-driven model of attention results in what we would call common sense behavior on the part of the robot.
Dr Robertson envisages a variety of situations where the use of CART could be particularly critical, including a navy ship under fire, where equipment such as a radar unit has been damaged. Typically, this would require a person to climb up the ship’s mast and replace the faulty parts while consulting a repair manual. Clearly, this would be a difficult and stressful task when under fire, making errors likely.
Instead, Dr Robertson proposes that a quadcopter, programmed with CART, would assist the person; hovering behind them as they make the necessary repairs. With its knowledge of the full radar repair process, understanding of the person’s capabilities and ability to ignore distractions, the robot would ensure that the person doesn’t need to remember everything or continually consult the manual, significantly hastening the repair.
Demonstrating Helpful Gestures
In addition, Dr Robertson has now tested out CART’s capabilities in a far simpler, more familiar situation: replacing a cell phone battery. To do this, he has exploited our instinctive reactions to faces and hands gesturing towards objects of interest. These instincts are so strong that we can recognise these gestures even in a simple humanoid robot.
‘As humans, as with other mammals, we have become surprisingly accurate at determining what a “face” is looking at,’ Dr Robertson explains. ‘Humans are extraordinarily good at following gaze and pointing, so it is good to take advantage of this capability.’
Since its initial development, Dr Robertson has made significant strides towards developing robots that provide critical assistance during repair missions. ‘CART can point out where the phone is in response to the question “where is the phone”,’ he continues. ‘She always knows where the important pieces are. She can point them out with a combination of head gaze and arm. This combination is very effective. Pointing works very well especially when combined with head gaze.’ With this demonstration, Dr Robertson has clearly shown that robots can indeed be genuinely useful in tasks that arise in everyday situations.
Robot Assistants for Everyday Tasks
In the future, Dr Robertson believes that CART will be able to assist humans with increasingly complex tasks, and in increasingly difficult, wide-ranging environments. He also hopes that improvements in robotics, such as the ability to manoeuvre individual fingers, will allow robots to express far more complex emotions and gestures than are currently possible. Ultimately, these advances will make it increasingly easy for human users to respond to and benefit from robotic advice.
Through continuing improvements to CART, Dr Robertson foresees that robots could one day be able to assist with far more diverse types of task than repair missions alone. For now, it seems that the idea of robotic assistants that help us to conduct tasks in our everyday lives seem less far-fetched than ever before.
Meet the researcher
Dr Paul Robertson
Dynamic Object Language Labs Inc
Dr Paul Robertson holds a DPhil from the University of Oxford, for a research project entitled ‘A self-adaptive architecture for image understanding’. Since then, he has worked at numerous prestigious institutions and companies, including the University of Massachusetts at Boston, the University of Texas at Dallas, Artelligence Inc, Symbolics Inc, Massachusetts Institute of Technology, and BBN Technologies. He has had a lifelong interest in reflection in intelligent systems, and the role that reflection can play in adapting responses to changes in the environment. A key interest has been advanced planning, especially in the context of self-adaptation. Today, Dr Robertson’s research at Dynamic Object Language Labs (DOLL) Inc centres around the role of emotions and self-awareness in robotic systems as a natural extension from his earlier work on reflection and self-adaptation.
E: paulr AT dollabs.com
Professor Patrick Winston, Dr Adam Kraft, Mr Cagri Zaman and Ms Zhutian Yang, MIT
Dr Andreas Hofmann, Dan Cerys and Prakash Manghwani, DOLL Inc
This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. D16PC00144.
Creative Commons Licence
(CC BY 4.0)
This work is licensed under a Creative Commons Attribution 4.0 International License.
What does this mean?
Share: You can copy and redistribute the material in any medium or format
Adapt: You can change, and build upon the material for any purpose, even commercially.
Credit: You must give appropriate credit, provide a link to the license, and indicate if changes were made.
More articles you may like
N-nitrosodimethylamine (NDMA for short) is a worryingly prevalent, potentially potent carcinogen found in food, water, cigarettes and medical drugs. Researchers at the Massachusetts Institute of Technology, USA, are working to better understand what NDMA does and how cells can defend themselves from its effects, with important implications for public health.
Dr Martín Medina-Elizalde | Collapse of the Ancient Maya Civilisation: Aligning History with Geological Analysis
Between 800 and 1000 CE, one of the world’s most advanced ancient civilisations underwent a devastating decline. The collapse of ancient Maya society has widely been attributed to a century-long drought; but so far, there have been few efforts to quantify this event, or to equate scientific findings with historical sources. Through new geological and paleoclimatological analyses, Dr Martín Medina-Elizalde at the University of Massachusetts, Amherst has revealed that the climate changes experienced during the drought followed more complex patterns than previously thought. His team’s discoveries could have important implications for predicting our own society’s future.
Dr Evelyn Cooper | Dr Candice Duncan – Improving Agriculture and Geoscience through Educational Initiatives
Addressing the skills shortage within scientific sectors requires a targeted approach for attracting and retaining students in STEM education. Summer Opportunities in Agricultural Research and the Environment (SOARE), SOARE: Strategic Work in Applied Geosciences (SWAG) and AgDiscovery, three innovative programs at the University of Maryland, provide a gateway for continued education, particularly for students who are traditionally under-represented in scientific fields. Implemented by Dr Evelyn Cooper, the success of the AgDiscovery and SOARE programs at the university has led to the inception of the new SOARE:SWAG program. Co-directed by Dr Candice Duncan, SOARE:SWAG focuses on students within geoscience disciplines.
Founded in 1987, the European Society for Evolutionary Biology (ESEB) is an academic society that brings together over 2000 biologists from Europe and beyond. In this exclusive interview, we speak with Professor Astrid Groot, President of ESEB, who discusses the many ways that the society supports scientists and helps to advance the diverse field of evolutionary biology.