Dr Paul Robertson – CART: Pointing the Way to Reliable Robotic Assistants

Oct 24, 2019 | Engineering and Tech

Robots that assist us with both everyday and highly specialised tasks are no new concept, but so far, their actual development has appeared to be far from becoming a reality. Yet through the research of Dr Paul Robertson, Chief Scientist at Dynamic Object Language Labs (DOLL) in Massachusetts, such sophisticated technology is now looking increasingly feasible. By considering how robots can be programmed to recognise tasks and comprehend human emotions, his research could be bringing engineers a step closer towards reliable robotic assistants.

How to be a Good Assistant

The idea of robotic assistants has existed in the public consciousness for quite some time. These smart machines are often viewed as a highly desirable concept, but until now, they have mostly been confined to the realms of science fiction. Recent advances in the technology, including Amazon’s Alexa and Google Home, are perhaps the closest we have come so far. However, if truly reliable robotic assistants are to be fully realised, there are still several fundamental barriers to overcome.

Currently, technologies are limited in their capabilities, since programs lack a basic understanding of how humans think and behave. To offer assistance, they must either only carry out useful tasks when specifically asked, or offer help even when it is not needed, meaning they become invasive to everyday life. In addition, they can only be programmed to carry out very specific tasks; as such, the time required to obtain their assistance can be longer than the time they save. Clearly, the technology needs a new approach.  

In his research, Dr Robertson and his team, building on earlier research in story understanding by team member Professor Patrick Winston, have drawn out a set of requirements for building truly useful robotic assistants, which incorporates models of what its users believe and think. If fully realised, the robots would be able to understand what a user is doing and why, recognise tasks they are struggling with, and offer them advice; though only when it is needed. Dr Robertson believes that such ‘Theory of Mind’ capabilities would allow for significant advances towards genuinely useful robotic assistants. The goal is achievable, but incorporating such human concepts of common sense into a machine is  no easy task.

Aiding Repair Missions

As an initial demonstration scenario, and to avoid short-term limitations of robot dexterity, Dr Robertson has turned his attention towards robots specifically designed to aid repair missions. Such cases are clear examples of where a detailed knowledge of the tasks involved, and a recognition of human abilities, is particularly critical.

To realise these characteristics, Dr Robertson has developed a program named ‘Context-driven Active-sensing for Repair Tasks’ (CART). ‘The idea is simple,’ he explains. ‘How can a program reflect upon what it is trying to do, and the environment in which it is trying to do it, and to modify itself so as to be more effective? For me, that is the essence of what we call intelligent behaviour.’

Since CART’s initial development, Dr Robertson has made meaningful advancements towards developing robots that provide critical assistance during repair missions. ‘Today I am applying these ideas in the context of social robotics, of which the CART project is an example,’ he continues. ‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’

Therefore, by incorporating the principles of a Theory of Mind, robots programmed with CART can read repair manuals, recognise the different parts they depict, and build a ‘story’ of how they need to be pieced together to achieve the goals of the repair mission.

Embodying Intelligence

Achieving these capabilities has required Dr Robertson to consider a variety of key attributes of reliable robotic assistants. These include the ability of the machines to manoeuvre themselves to get a good vantage point of the repair operation, to understand what the repairer is doing, and to offer useful guidance in response to the person’s questions. In addition, the robot must use its own initiative to intervene with guidance if the story wanders off track, or if the conditions of the surrounding environment change. As it monitors all aspects of the mission, the robot presently assists the person both through verbal guidance and using physical gestures, but in the future it will participate physically as an able human assistant would.

One particularly important aspect which governs the effectiveness of these attributes is the ability of the robot to understand human emotions. Dr Robertson is achieving this through a modification to CART, named ‘Continuous Affective Robot Learning’ (CARL). This program integrates a ‘fear and reward’ system into CART, which causes robots to develop a ‘desire’ to see a story unfold in a certain way. CARL then ‘fears’ any mishaps, and losing sight of the objective, especially if such events have happened in the past.  

Therefore, the program can learn to follow prior sequences in which its fears were mostly avoided. If this happens, the robot will feel a sense of ‘reward’, and so its desire to follow that path again in the future will increase. By desiring certain outcomes, therefore, CARL will react more effectively when potential setbacks arise. This addition of a simple model of emotions allows the robot to improve over time by collecting data about events whose importance is suggested by their affective import. This data is later used to improve future performance through reinforcement learning techniques.

‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’

Handling Distractions

In real-world situations, robotic assistants may likely be required to operate in environments presenting many distractions, which could prevent the machines from focusing on a user. Dr Robertson has considered this issue as well: through the integration of a ‘model of attention’ into CART. This model can transform noisy information into stable representations of the story by first acknowledging all objects the robot detects. It can then identify the specific objects that appear in the repair manual.

Thanks to the model, objects are more robustly tracked if they appear in an upcoming part of the story, which elevates the levels of attention they receive. In contrast, objects that have already been used, and do not appear again in the story, can be completely forgotten. Through its knowledge of the story, CART can recall what is supposed to happen in the future, allowing its user to focus their attention only on the tasks immediately required of them. The story-driven model of attention results in what we would call common sense behavior on the part of the robot.

Dr Robertson envisages a variety of situations where the use of CART could be particularly critical, including a navy ship under fire, where equipment such as a radar unit has been damaged. Typically, this would require a person to climb up the ship’s mast and replace the faulty parts while consulting a repair manual. Clearly, this would be a difficult and stressful task when under fire, making errors likely.

Instead, Dr Robertson proposes that a quadcopter, programmed with CART, would assist the person; hovering behind them as they make the necessary repairs. With its knowledge of the full radar repair process, understanding of the person’s capabilities and ability to ignore distractions, the robot would ensure that the person doesn’t need to remember everything or continually consult the manual, significantly hastening the repair.

Demonstrating Helpful Gestures

In addition, Dr Robertson has now tested out CART’s capabilities in a far simpler, more familiar situation: replacing a cell phone battery. To do this, he has exploited our instinctive reactions to faces and hands gesturing towards objects of interest. These instincts are so strong that we can recognise these gestures even in a simple humanoid robot.

‘As humans, as with other mammals, we have become surprisingly accurate at determining what a “face” is looking at,’ Dr Robertson explains. ‘Humans are extraordinarily good at following gaze and pointing, so it is good to take advantage of this capability.’

Since its initial development, Dr Robertson has made significant strides towards developing robots that provide critical assistance during repair missions. ‘CART can point out where the phone is in response to the question “where is the phone”,’ he continues. ‘She always knows where the important pieces are. She can point them out with a combination of head gaze and arm. This combination is very effective. Pointing works very well especially when combined with head gaze.’ With this demonstration, Dr Robertson has clearly shown that robots can indeed be genuinely useful in tasks that arise in everyday situations.

Robot Assistants for Everyday Tasks

In the future, Dr Robertson believes that CART will be able to assist humans with increasingly complex tasks, and in increasingly difficult, wide-ranging environments. He also hopes that improvements in robotics, such as the ability to manoeuvre individual fingers, will allow robots to express far more complex emotions and gestures than are currently possible. Ultimately, these advances will make it increasingly easy for human users to respond to and benefit from robotic advice.

Through continuing improvements to CART, Dr Robertson foresees that robots could one day be able to assist with far more diverse types of task than repair missions alone. For now, it seems that the idea of robotic assistants that help us to conduct tasks in our everyday lives seem less far-fetched than ever before.

Reference
https://doi.org/10.33548/SCIENTIA432

Meet the researcher


Dr Paul Robertson

Dynamic Object Language Labs Inc
Lexington, MA
USA

Dr Paul Robertson holds a DPhil from the University of Oxford, for a research project entitled ‘A self-adaptive architecture for image understanding’. Since then, he has worked at numerous prestigious institutions and companies, including the University of Massachusetts at Boston, the University of Texas at Dallas, Artelligence Inc, Symbolics Inc, Massachusetts Institute of Technology, and BBN Technologies. He has had a lifelong interest in reflection in intelligent systems, and the role that reflection can play in adapting responses to changes in the environment. A key interest has been advanced planning, especially in the context of self-adaptation. Today, Dr Robertson’s research at Dynamic Object Language Labs (DOLL) Inc centres around the role of emotions and self-awareness in robotic systems as a natural extension from his earlier work on reflection and self-adaptation.

CONTACT

E: paulr AT dollabs.com

W: https://www.dollabs.com/people/dr-paul-robertson-chief-scientist-president

W: http://paulrobertson-anglais.bcdoll.com/

KEY COLLABORATORS

Professor Patrick Winston, Dr Adam Kraft, Mr Cagri Zaman and Ms Zhutian Yang, MIT

Dr Andreas Hofmann, Dan Cerys and Prakash Manghwani, DOLL Inc

FUNDING

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. D16PC00144.

 

Creative Commons Licence
(CC BY 4.0)

This work is licensed under a Creative Commons Attribution 4.0 International License. Creative Commons License

What does this mean?

Share: You can copy and redistribute the material in any medium or format

Adapt: You can change, and build upon the material for any purpose, even commercially.

Credit: You must give appropriate credit, provide a link to the license, and indicate if changes were made.

Subscribe now!


More articles you may like

Dr Frank Wuest – Innovative Strategies for Imaging Cancers

Global Navigation Satellite Systems such as GPS are the backbone of many global communications, but they are not immune to failure. Progeny Systems Corporation is dedicated to mitigating such disasters if and when satellite-based communications fail, by developing Earth-based systems that work in comparable ways to synchronised satellite networks. As an alternative to GPS, the company’s technology could provide communicating parties with a crucial yet inexpensive safeguard against future failures.

Progeny: Developing Safeguards Against GPS Outages

Global Navigation Satellite Systems such as GPS are the backbone of many global communications, but they are not immune to failure. Progeny Systems Corporation is dedicated to mitigating such disasters if and when satellite-based communications fail, by developing Earth-based systems that work in comparable ways to synchronised satellite networks. As an alternative to GPS, the company’s technology could provide communicating parties with a crucial yet inexpensive safeguard against future failures.

FARAD: Reducing Chemical Residues in Animal Products

Residues of drugs, pesticides and other chemical substances can reduce the safety of animal-derived foods, adversely affecting the health and confidence of consumers. To address this pressing issue, scientists at several US universities founded the Food Animal Residue Avoidance Databank (FARAD), a program aimed at providing veterinarians and livestock producers with knowledge and tools that can help in preventing or reducing the presence of chemical residues in food.

Ocean Surface Holds the Key to Understanding Our Climate

Every 10 years, the ocean science community comes together to present a vision for advancing key scientific priorities to maximize societal benefit. In September 2019, leading ocean and climate scientists from around the world presented a plan for improving weather and climate forecasts through better observations of the heat exchange between the ocean and atmosphere. Such improved forecasts will be critical for effectively managing natural resources and mitigating risk to vulnerable populations.