Dr Paul Robertson – CART: Pointing the Way to Reliable Robotic Assistants

Robots that assist us with both everyday and highly specialised tasks are no new concept, but so far, their actual development has appeared to be far from becoming a reality. Yet through the research of Dr Paul Robertson, Chief Scientist at Dynamic Object Language Labs (DOLL) in Massachusetts, such sophisticated technology is now looking increasingly feasible. By considering how robots can be programmed to recognise tasks and comprehend human emotions, his research could be bringing engineers a step closer towards reliable robotic assistants.

How to be a Good Assistant

The idea of robotic assistants has existed in the public consciousness for quite some time. These smart machines are often viewed as a highly desirable concept, but until now, they have mostly been confined to the realms of science fiction. Recent advances in the technology, including Amazon’s Alexa and Google Home, are perhaps the closest we have come so far. However, if truly reliable robotic assistants are to be fully realised, there are still several fundamental barriers to overcome.

Currently, technologies are limited in their capabilities, since programs lack a basic understanding of how humans think and behave. To offer assistance, they must either only carry out useful tasks when specifically asked, or offer help even when it is not needed, meaning they become invasive to everyday life. In addition, they can only be programmed to carry out very specific tasks; as such, the time required to obtain their assistance can be longer than the time they save. Clearly, the technology needs a new approach.  

In his research, Dr Robertson and his team, building on earlier research in story understanding by team member Professor Patrick Winston, have drawn out a set of requirements for building truly useful robotic assistants, which incorporates models of what its users believe and think. If fully realised, the robots would be able to understand what a user is doing and why, recognise tasks they are struggling with, and offer them advice; though only when it is needed. Dr Robertson believes that such ‘Theory of Mind’ capabilities would allow for significant advances towards genuinely useful robotic assistants. The goal is achievable, but incorporating such human concepts of common sense into a machine is  no easy task.

Aiding Repair Missions

As an initial demonstration scenario, and to avoid short-term limitations of robot dexterity, Dr Robertson has turned his attention towards robots specifically designed to aid repair missions. Such cases are clear examples of where a detailed knowledge of the tasks involved, and a recognition of human abilities, is particularly critical.

To realise these characteristics, Dr Robertson has developed a program named ‘Context-driven Active-sensing for Repair Tasks’ (CART). ‘The idea is simple,’ he explains. ‘How can a program reflect upon what it is trying to do, and the environment in which it is trying to do it, and to modify itself so as to be more effective? For me, that is the essence of what we call intelligent behaviour.’

Since CART’s initial development, Dr Robertson has made meaningful advancements towards developing robots that provide critical assistance during repair missions. ‘Today I am applying these ideas in the context of social robotics, of which the CART project is an example,’ he continues. ‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’

Therefore, by incorporating the principles of a Theory of Mind, robots programmed with CART can read repair manuals, recognise the different parts they depict, and build a ‘story’ of how they need to be pieced together to achieve the goals of the repair mission.

Embodying Intelligence

Achieving these capabilities has required Dr Robertson to consider a variety of key attributes of reliable robotic assistants. These include the ability of the machines to manoeuvre themselves to get a good vantage point of the repair operation, to understand what the repairer is doing, and to offer useful guidance in response to the person’s questions. In addition, the robot must use its own initiative to intervene with guidance if the story wanders off track, or if the conditions of the surrounding environment change. As it monitors all aspects of the mission, the robot presently assists the person both through verbal guidance and using physical gestures, but in the future it will participate physically as an able human assistant would.

One particularly important aspect which governs the effectiveness of these attributes is the ability of the robot to understand human emotions. Dr Robertson is achieving this through a modification to CART, named ‘Continuous Affective Robot Learning’ (CARL). This program integrates a ‘fear and reward’ system into CART, which causes robots to develop a ‘desire’ to see a story unfold in a certain way. CARL then ‘fears’ any mishaps, and losing sight of the objective, especially if such events have happened in the past.  

Therefore, the program can learn to follow prior sequences in which its fears were mostly avoided. If this happens, the robot will feel a sense of ‘reward’, and so its desire to follow that path again in the future will increase. By desiring certain outcomes, therefore, CARL will react more effectively when potential setbacks arise. This addition of a simple model of emotions allows the robot to improve over time by collecting data about events whose importance is suggested by their affective import. This data is later used to improve future performance through reinforcement learning techniques.

‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’

Handling Distractions

In real-world situations, robotic assistants may likely be required to operate in environments presenting many distractions, which could prevent the machines from focusing on a user. Dr Robertson has considered this issue as well: through the integration of a ‘model of attention’ into CART. This model can transform noisy information into stable representations of the story by first acknowledging all objects the robot detects. It can then identify the specific objects that appear in the repair manual.

Thanks to the model, objects are more robustly tracked if they appear in an upcoming part of the story, which elevates the levels of attention they receive. In contrast, objects that have already been used, and do not appear again in the story, can be completely forgotten. Through its knowledge of the story, CART can recall what is supposed to happen in the future, allowing its user to focus their attention only on the tasks immediately required of them. The story-driven model of attention results in what we would call common sense behavior on the part of the robot.

Dr Robertson envisages a variety of situations where the use of CART could be particularly critical, including a navy ship under fire, where equipment such as a radar unit has been damaged. Typically, this would require a person to climb up the ship’s mast and replace the faulty parts while consulting a repair manual. Clearly, this would be a difficult and stressful task when under fire, making errors likely.

Instead, Dr Robertson proposes that a quadcopter, programmed with CART, would assist the person; hovering behind them as they make the necessary repairs. With its knowledge of the full radar repair process, understanding of the person’s capabilities and ability to ignore distractions, the robot would ensure that the person doesn’t need to remember everything or continually consult the manual, significantly hastening the repair.

Demonstrating Helpful Gestures

In addition, Dr Robertson has now tested out CART’s capabilities in a far simpler, more familiar situation: replacing a cell phone battery. To do this, he has exploited our instinctive reactions to faces and hands gesturing towards objects of interest. These instincts are so strong that we can recognise these gestures even in a simple humanoid robot.

‘As humans, as with other mammals, we have become surprisingly accurate at determining what a “face” is looking at,’ Dr Robertson explains. ‘Humans are extraordinarily good at following gaze and pointing, so it is good to take advantage of this capability.’

Since its initial development, Dr Robertson has made significant strides towards developing robots that provide critical assistance during repair missions. ‘CART can point out where the phone is in response to the question “where is the phone”,’ he continues. ‘She always knows where the important pieces are. She can point them out with a combination of head gaze and arm. This combination is very effective. Pointing works very well especially when combined with head gaze.’ With this demonstration, Dr Robertson has clearly shown that robots can indeed be genuinely useful in tasks that arise in everyday situations.

Robot Assistants for Everyday Tasks

In the future, Dr Robertson believes that CART will be able to assist humans with increasingly complex tasks, and in increasingly difficult, wide-ranging environments. He also hopes that improvements in robotics, such as the ability to manoeuvre individual fingers, will allow robots to express far more complex emotions and gestures than are currently possible. Ultimately, these advances will make it increasingly easy for human users to respond to and benefit from robotic advice.

Through continuing improvements to CART, Dr Robertson foresees that robots could one day be able to assist with far more diverse types of task than repair missions alone. For now, it seems that the idea of robotic assistants that help us to conduct tasks in our everyday lives seem less far-fetched than ever before.