Dr Paul Robertson – CART: Pointing the Way to Reliable Robotic Assistants

Oct 24, 2019 | Engineering and Tech

Robots that assist us with both everyday and highly specialised tasks are no new concept, but so far, their actual development has appeared to be far from becoming a reality. Yet through the research of Dr Paul Robertson, Chief Scientist at Dynamic Object Language Labs (DOLL) in Massachusetts, such sophisticated technology is now looking increasingly feasible. By considering how robots can be programmed to recognise tasks and comprehend human emotions, his research could be bringing engineers a step closer towards reliable robotic assistants.

How to be a Good Assistant

The idea of robotic assistants has existed in the public consciousness for quite some time. These smart machines are often viewed as a highly desirable concept, but until now, they have mostly been confined to the realms of science fiction. Recent advances in the technology, including Amazon’s Alexa and Google Home, are perhaps the closest we have come so far. However, if truly reliable robotic assistants are to be fully realised, there are still several fundamental barriers to overcome.

Currently, technologies are limited in their capabilities, since programs lack a basic understanding of how humans think and behave. To offer assistance, they must either only carry out useful tasks when specifically asked, or offer help even when it is not needed, meaning they become invasive to everyday life. In addition, they can only be programmed to carry out very specific tasks; as such, the time required to obtain their assistance can be longer than the time they save. Clearly, the technology needs a new approach.  

In his research, Dr Robertson and his team, building on earlier research in story understanding by team member Professor Patrick Winston, have drawn out a set of requirements for building truly useful robotic assistants, which incorporates models of what its users believe and think. If fully realised, the robots would be able to understand what a user is doing and why, recognise tasks they are struggling with, and offer them advice; though only when it is needed. Dr Robertson believes that such ‘Theory of Mind’ capabilities would allow for significant advances towards genuinely useful robotic assistants. The goal is achievable, but incorporating such human concepts of common sense into a machine is  no easy task.

Aiding Repair Missions

As an initial demonstration scenario, and to avoid short-term limitations of robot dexterity, Dr Robertson has turned his attention towards robots specifically designed to aid repair missions. Such cases are clear examples of where a detailed knowledge of the tasks involved, and a recognition of human abilities, is particularly critical.

To realise these characteristics, Dr Robertson has developed a program named ‘Context-driven Active-sensing for Repair Tasks’ (CART). ‘The idea is simple,’ he explains. ‘How can a program reflect upon what it is trying to do, and the environment in which it is trying to do it, and to modify itself so as to be more effective? For me, that is the essence of what we call intelligent behaviour.’

Since CART’s initial development, Dr Robertson has made meaningful advancements towards developing robots that provide critical assistance during repair missions. ‘Today I am applying these ideas in the context of social robotics, of which the CART project is an example,’ he continues. ‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’

Therefore, by incorporating the principles of a Theory of Mind, robots programmed with CART can read repair manuals, recognise the different parts they depict, and build a ‘story’ of how they need to be pieced together to achieve the goals of the repair mission.

Embodying Intelligence

Achieving these capabilities has required Dr Robertson to consider a variety of key attributes of reliable robotic assistants. These include the ability of the machines to manoeuvre themselves to get a good vantage point of the repair operation, to understand what the repairer is doing, and to offer useful guidance in response to the person’s questions. In addition, the robot must use its own initiative to intervene with guidance if the story wanders off track, or if the conditions of the surrounding environment change. As it monitors all aspects of the mission, the robot presently assists the person both through verbal guidance and using physical gestures, but in the future it will participate physically as an able human assistant would.

One particularly important aspect which governs the effectiveness of these attributes is the ability of the robot to understand human emotions. Dr Robertson is achieving this through a modification to CART, named ‘Continuous Affective Robot Learning’ (CARL). This program integrates a ‘fear and reward’ system into CART, which causes robots to develop a ‘desire’ to see a story unfold in a certain way. CARL then ‘fears’ any mishaps, and losing sight of the objective, especially if such events have happened in the past.  

Therefore, the program can learn to follow prior sequences in which its fears were mostly avoided. If this happens, the robot will feel a sense of ‘reward’, and so its desire to follow that path again in the future will increase. By desiring certain outcomes, therefore, CARL will react more effectively when potential setbacks arise. This addition of a simple model of emotions allows the robot to improve over time by collecting data about events whose importance is suggested by their affective import. This data is later used to improve future performance through reinforcement learning techniques.

‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’

Handling Distractions

In real-world situations, robotic assistants may likely be required to operate in environments presenting many distractions, which could prevent the machines from focusing on a user. Dr Robertson has considered this issue as well: through the integration of a ‘model of attention’ into CART. This model can transform noisy information into stable representations of the story by first acknowledging all objects the robot detects. It can then identify the specific objects that appear in the repair manual.

Thanks to the model, objects are more robustly tracked if they appear in an upcoming part of the story, which elevates the levels of attention they receive. In contrast, objects that have already been used, and do not appear again in the story, can be completely forgotten. Through its knowledge of the story, CART can recall what is supposed to happen in the future, allowing its user to focus their attention only on the tasks immediately required of them. The story-driven model of attention results in what we would call common sense behavior on the part of the robot.

Dr Robertson envisages a variety of situations where the use of CART could be particularly critical, including a navy ship under fire, where equipment such as a radar unit has been damaged. Typically, this would require a person to climb up the ship’s mast and replace the faulty parts while consulting a repair manual. Clearly, this would be a difficult and stressful task when under fire, making errors likely.

Instead, Dr Robertson proposes that a quadcopter, programmed with CART, would assist the person; hovering behind them as they make the necessary repairs. With its knowledge of the full radar repair process, understanding of the person’s capabilities and ability to ignore distractions, the robot would ensure that the person doesn’t need to remember everything or continually consult the manual, significantly hastening the repair.

Demonstrating Helpful Gestures

In addition, Dr Robertson has now tested out CART’s capabilities in a far simpler, more familiar situation: replacing a cell phone battery. To do this, he has exploited our instinctive reactions to faces and hands gesturing towards objects of interest. These instincts are so strong that we can recognise these gestures even in a simple humanoid robot.

‘As humans, as with other mammals, we have become surprisingly accurate at determining what a “face” is looking at,’ Dr Robertson explains. ‘Humans are extraordinarily good at following gaze and pointing, so it is good to take advantage of this capability.’

Since its initial development, Dr Robertson has made significant strides towards developing robots that provide critical assistance during repair missions. ‘CART can point out where the phone is in response to the question “where is the phone”,’ he continues. ‘She always knows where the important pieces are. She can point them out with a combination of head gaze and arm. This combination is very effective. Pointing works very well especially when combined with head gaze.’ With this demonstration, Dr Robertson has clearly shown that robots can indeed be genuinely useful in tasks that arise in everyday situations.

Robot Assistants for Everyday Tasks

In the future, Dr Robertson believes that CART will be able to assist humans with increasingly complex tasks, and in increasingly difficult, wide-ranging environments. He also hopes that improvements in robotics, such as the ability to manoeuvre individual fingers, will allow robots to express far more complex emotions and gestures than are currently possible. Ultimately, these advances will make it increasingly easy for human users to respond to and benefit from robotic advice.

Through continuing improvements to CART, Dr Robertson foresees that robots could one day be able to assist with far more diverse types of task than repair missions alone. For now, it seems that the idea of robotic assistants that help us to conduct tasks in our everyday lives seem less far-fetched than ever before.


Meet the researcher

Dr Paul Robertson

Dynamic Object Language Labs Inc
Lexington, MA

Dr Paul Robertson holds a DPhil from the University of Oxford, for a research project entitled ‘A self-adaptive architecture for image understanding’. Since then, he has worked at numerous prestigious institutions and companies, including the University of Massachusetts at Boston, the University of Texas at Dallas, Artelligence Inc, Symbolics Inc, Massachusetts Institute of Technology, and BBN Technologies. He has had a lifelong interest in reflection in intelligent systems, and the role that reflection can play in adapting responses to changes in the environment. A key interest has been advanced planning, especially in the context of self-adaptation. Today, Dr Robertson’s research at Dynamic Object Language Labs (DOLL) Inc centres around the role of emotions and self-awareness in robotic systems as a natural extension from his earlier work on reflection and self-adaptation.


E: paulr AT dollabs.com

W: https://www.dollabs.com/people/dr-paul-robertson-chief-scientist-president

W: http://paulrobertson-anglais.bcdoll.com/


Professor Patrick Winston, Dr Adam Kraft, Mr Cagri Zaman and Ms Zhutian Yang, MIT

Dr Andreas Hofmann, Dan Cerys and Prakash Manghwani, DOLL Inc


This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. D16PC00144.


Creative Commons Licence
(CC BY 4.0)

This work is licensed under a Creative Commons Attribution 4.0 International License. Creative Commons License

What does this mean?

Share: You can copy and redistribute the material in any medium or format

Adapt: You can change, and build upon the material for any purpose, even commercially.

Credit: You must give appropriate credit, provide a link to the license, and indicate if changes were made.

More articles you may like

Dr Xu Hannah Zhang | p38γ – It’s More than Just a Kinase

Dr Xu Hannah Zhang | p38γ – It’s More than Just a Kinase

Kinases take energy from adenosine triphosphate molecules to fuel other molecules in performing vital biological processes. Dr Xu Hannah Zhang at City of Hope, Los Angeles, has worked with colleagues to better understand the p38 family of kinases, and in particular, how the p38γ isoform plays a role in cancer. Her work has shown – for the first time – that p38γ is much more than just a kinase, and her recent studies point to new avenues in the search for cutaneous T-cell lymphoma therapeutics.

Professor Alberto Posso – The Neglected Consequences of Child Labour

Professor Alberto Posso – The Neglected Consequences of Child Labour

Child labour is a major social problem that contributes to poor physical health and lower educational achievement. Professor Alberto Posso (Royal Melbourne Institute of Technology) worked with Professor Simon Feeny (Royal Melbourne Institute of Technology), Dr Ahmed Skali (University of Groningen), Professor Amalendu Jyotishi (Azim Premji University), Dr Shyam Nath (Amrita University) and Dr P. K. Viswanathan (Amrita Vishwa Vidyapeetham) to address important gaps in the literature by conducting a large-scale study of children in rural areas of India. This work confirms the hugely negative impact of child labour on psychosocial well-being and opens up important implications for policy, practice and future research.

Dr Daisuke Minakata – Sunshine and Organic Molecules in Water

Dr Daisuke Minakata – Sunshine and Organic Molecules in Water

Organic molecules dissolved in rivers, lakes, seas and oceans are essential to plant and animal life. Some of these molecules are also degraded and enter a complex cycle of carbon, nitrogen and sulphur containing compounds. Surprisingly, scientists currently have a limited understanding of the fate of these molecules. Dr Daisuke Minakata and his colleagues from Michigan Technological University are involved in an ambitious programme to overcome this critical knowledge gap.

Professor Christian Laforsch | Professor Andreas Greiner – Microplastics: Solutions for a Persistent Pollutant

Professor Christian Laforsch | Professor Andreas Greiner – Microplastics: Solutions for a Persistent Pollutant

Plastics have revolutionised human existence. Medicine, technology, agriculture and construction all rely on highly durable plastic materials. However, the enduring legacy of plastics extends far beyond our cities and towns. Everywhere we look, from the deepest parts of the oceans to alpine glaciers, we find tiny fragments called microplastics. Recently, the collaborative research centre, ‘CRC 1357 Microplastic’, at the University of Bayreuth was granted a second funding phase by the German Science Foundation, to continue their intensive research into microplastics. The CRC 1357 team studies the formation and behaviour of microplastics in the environment and their long-term effects on soils, plants, organisms, and ecosystem processes. Through their research, the University of Bayreuth will be able to contribute to ground-breaking recommendations for policy-makers, industry and society.