Dr Paul Robertson – CART: Pointing the Way to Reliable Robotic Assistants
Robots that assist us with both everyday and highly specialised tasks are no new concept, but so far, their actual development has appeared to be far from becoming a reality. Yet through the research of Dr Paul Robertson, Chief Scientist at Dynamic Object Language Labs (DOLL) in Massachusetts, such sophisticated technology is now looking increasingly feasible. By considering how robots can be programmed to recognise tasks and comprehend human emotions, his research could be bringing engineers a step closer towards reliable robotic assistants.
How to be a Good Assistant
The idea of robotic assistants has existed in the public consciousness for quite some time. These smart machines are often viewed as a highly desirable concept, but until now, they have mostly been confined to the realms of science fiction. Recent advances in the technology, including Amazon’s Alexa and Google Home, are perhaps the closest we have come so far. However, if truly reliable robotic assistants are to be fully realised, there are still several fundamental barriers to overcome.
Currently, technologies are limited in their capabilities, since programs lack a basic understanding of how humans think and behave. To offer assistance, they must either only carry out useful tasks when specifically asked, or offer help even when it is not needed, meaning they become invasive to everyday life. In addition, they can only be programmed to carry out very specific tasks; as such, the time required to obtain their assistance can be longer than the time they save. Clearly, the technology needs a new approach.
In his research, Dr Robertson and his team, building on earlier research in story understanding by team member Professor Patrick Winston, have drawn out a set of requirements for building truly useful robotic assistants, which incorporates models of what its users believe and think. If fully realised, the robots would be able to understand what a user is doing and why, recognise tasks they are struggling with, and offer them advice; though only when it is needed. Dr Robertson believes that such ‘Theory of Mind’ capabilities would allow for significant advances towards genuinely useful robotic assistants. The goal is achievable, but incorporating such human concepts of common sense into a machine is no easy task.
Aiding Repair Missions
As an initial demonstration scenario, and to avoid short-term limitations of robot dexterity, Dr Robertson has turned his attention towards robots specifically designed to aid repair missions. Such cases are clear examples of where a detailed knowledge of the tasks involved, and a recognition of human abilities, is particularly critical.
To realise these characteristics, Dr Robertson has developed a program named ‘Context-driven Active-sensing for Repair Tasks’ (CART). ‘The idea is simple,’ he explains. ‘How can a program reflect upon what it is trying to do, and the environment in which it is trying to do it, and to modify itself so as to be more effective? For me, that is the essence of what we call intelligent behaviour.’
Since CART’s initial development, Dr Robertson has made meaningful advancements towards developing robots that provide critical assistance during repair missions. ‘Today I am applying these ideas in the context of social robotics, of which the CART project is an example,’ he continues. ‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’
Therefore, by incorporating the principles of a Theory of Mind, robots programmed with CART can read repair manuals, recognise the different parts they depict, and build a ‘story’ of how they need to be pieced together to achieve the goals of the repair mission.
Embodying Intelligence
Achieving these capabilities has required Dr Robertson to consider a variety of key attributes of reliable robotic assistants. These include the ability of the machines to manoeuvre themselves to get a good vantage point of the repair operation, to understand what the repairer is doing, and to offer useful guidance in response to the person’s questions. In addition, the robot must use its own initiative to intervene with guidance if the story wanders off track, or if the conditions of the surrounding environment change. As it monitors all aspects of the mission, the robot presently assists the person both through verbal guidance and using physical gestures, but in the future it will participate physically as an able human assistant would.
One particularly important aspect which governs the effectiveness of these attributes is the ability of the robot to understand human emotions. Dr Robertson is achieving this through a modification to CART, named ‘Continuous Affective Robot Learning’ (CARL). This program integrates a ‘fear and reward’ system into CART, which causes robots to develop a ‘desire’ to see a story unfold in a certain way. CARL then ‘fears’ any mishaps, and losing sight of the objective, especially if such events have happened in the past.
Therefore, the program can learn to follow prior sequences in which its fears were mostly avoided. If this happens, the robot will feel a sense of ‘reward’, and so its desire to follow that path again in the future will increase. By desiring certain outcomes, therefore, CARL will react more effectively when potential setbacks arise. This addition of a simple model of emotions allows the robot to improve over time by collecting data about events whose importance is suggested by their affective import. This data is later used to improve future performance through reinforcement learning techniques.
‘Simply put, we want the robot to reflect on the mission at hand and the human that it must work with, in order to adapt to the specific situation. Then, it can be a good team member in bringing whatever competence it may have to bear on the problem at hand.’
Handling Distractions
In real-world situations, robotic assistants may likely be required to operate in environments presenting many distractions, which could prevent the machines from focusing on a user. Dr Robertson has considered this issue as well: through the integration of a ‘model of attention’ into CART. This model can transform noisy information into stable representations of the story by first acknowledging all objects the robot detects. It can then identify the specific objects that appear in the repair manual.
Thanks to the model, objects are more robustly tracked if they appear in an upcoming part of the story, which elevates the levels of attention they receive. In contrast, objects that have already been used, and do not appear again in the story, can be completely forgotten. Through its knowledge of the story, CART can recall what is supposed to happen in the future, allowing its user to focus their attention only on the tasks immediately required of them. The story-driven model of attention results in what we would call common sense behavior on the part of the robot.
Dr Robertson envisages a variety of situations where the use of CART could be particularly critical, including a navy ship under fire, where equipment such as a radar unit has been damaged. Typically, this would require a person to climb up the ship’s mast and replace the faulty parts while consulting a repair manual. Clearly, this would be a difficult and stressful task when under fire, making errors likely.
Instead, Dr Robertson proposes that a quadcopter, programmed with CART, would assist the person; hovering behind them as they make the necessary repairs. With its knowledge of the full radar repair process, understanding of the person’s capabilities and ability to ignore distractions, the robot would ensure that the person doesn’t need to remember everything or continually consult the manual, significantly hastening the repair.
Demonstrating Helpful Gestures
In addition, Dr Robertson has now tested out CART’s capabilities in a far simpler, more familiar situation: replacing a cell phone battery. To do this, he has exploited our instinctive reactions to faces and hands gesturing towards objects of interest. These instincts are so strong that we can recognise these gestures even in a simple humanoid robot.
‘As humans, as with other mammals, we have become surprisingly accurate at determining what a “face” is looking at,’ Dr Robertson explains. ‘Humans are extraordinarily good at following gaze and pointing, so it is good to take advantage of this capability.’
Since its initial development, Dr Robertson has made significant strides towards developing robots that provide critical assistance during repair missions. ‘CART can point out where the phone is in response to the question “where is the phone”,’ he continues. ‘She always knows where the important pieces are. She can point them out with a combination of head gaze and arm. This combination is very effective. Pointing works very well especially when combined with head gaze.’ With this demonstration, Dr Robertson has clearly shown that robots can indeed be genuinely useful in tasks that arise in everyday situations.
Robot Assistants for Everyday Tasks
In the future, Dr Robertson believes that CART will be able to assist humans with increasingly complex tasks, and in increasingly difficult, wide-ranging environments. He also hopes that improvements in robotics, such as the ability to manoeuvre individual fingers, will allow robots to express far more complex emotions and gestures than are currently possible. Ultimately, these advances will make it increasingly easy for human users to respond to and benefit from robotic advice.
Through continuing improvements to CART, Dr Robertson foresees that robots could one day be able to assist with far more diverse types of task than repair missions alone. For now, it seems that the idea of robotic assistants that help us to conduct tasks in our everyday lives seem less far-fetched than ever before.
Reference
https://doi.org/10.33548/SCIENTIA432
Meet the researcher
Dr Paul Robertson
Dynamic Object Language Labs Inc
Lexington, MA
USA
Dr Paul Robertson holds a DPhil from the University of Oxford, for a research project entitled ‘A self-adaptive architecture for image understanding’. Since then, he has worked at numerous prestigious institutions and companies, including the University of Massachusetts at Boston, the University of Texas at Dallas, Artelligence Inc, Symbolics Inc, Massachusetts Institute of Technology, and BBN Technologies. He has had a lifelong interest in reflection in intelligent systems, and the role that reflection can play in adapting responses to changes in the environment. A key interest has been advanced planning, especially in the context of self-adaptation. Today, Dr Robertson’s research at Dynamic Object Language Labs (DOLL) Inc centres around the role of emotions and self-awareness in robotic systems as a natural extension from his earlier work on reflection and self-adaptation.
CONTACT
E: paulr AT dollabs.com
W: https://www.dollabs.com/people/dr-paul-robertson-chief-scientist-president
W: http://paulrobertson-anglais.bcdoll.com/
KEY COLLABORATORS
Professor Patrick Winston, Dr Adam Kraft, Mr Cagri Zaman and Ms Zhutian Yang, MIT
Dr Andreas Hofmann, Dan Cerys and Prakash Manghwani, DOLL Inc
FUNDING
This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. D16PC00144.
Creative Commons Licence
(CC BY 4.0)
This work is licensed under a Creative Commons Attribution 4.0 International License.
What does this mean?
Share: You can copy and redistribute the material in any medium or format
Adapt: You can change, and build upon the material for any purpose, even commercially.
Credit: You must give appropriate credit, provide a link to the license, and indicate if changes were made.
More articles you may like
Dr Peter Gent | Climate ‘Normals’ Are Anything But Normal In Weather Forecasting
Weather forecasters and meteorologists have long used the term ‘climate normal’ to describe average temperatures, but this seemingly innocuous phrase might be causing widespread public misunderstanding. New research suggests that describing temperature data as ‘normal’ leads many people to incorrectly assume these values represent the most common or expected temperatures when, in reality, actual temperatures regularly deviate significantly from these averages.
Revolutionary Battery Technology Promises to Transform Energy Storage
The world of energy storage is on the cusp of a significant breakthrough. As society increasingly shifts towards electrification, from personal devices to transportation and beyond, the limitations of current battery technology have become increasingly apparent. An innovative company in Tallahassee, Florida, is developing a revolutionary new battery that could transform how we store and use energy. The technology, developed by a team of experts at Piersica Inc., promises to deliver an energy density of 630 watt-hours per kilogram (Wh/kg)—approximately two and a half times higher than current lithium-ion batteries. This advancement will extend the range of electric vehicles, enable long-distance electric aircraft, and dramatically increase the battery life of portable electronics.
Dr Maka Tsulukidze – Dr David Reardon | After the Loss: Exploring Cardiovascular Risks Linked to Pregnancy Loss
Carrying a pregnancy is associated with numerous health problems, but little is known about the impact of pregnancy loss on heart health. Dr Maka Tsulukidze from the Florida Gulf Coast University, Dr David Reardon, based at the Elliot Institute, and Dr Christopher Craver at the Charlotte Lozier Institute in Arlington, conducted research into the adverse effects of pregnancy loss on heart function and health. In particular, their research focused on the impact of pregnancy loss on the risk of cardiovascular diseases.
Dr Zhi-yong Huang | The Scientific Hepatectomy: Pioneering Approaches for Tackling Liver Cancer
Cancer research is an ever-evolving field of science fuelled by the pressure of the ongoing need to find new approaches to tackle this killer. Dr Zhi-yong Huang is based at the Tongji Medical College at Huazhong University of Science and Technology in China. He works tirelessly to develop novel treatments for diseases of the liver and biliary system, with a particular focus on liver cancer surgery. He has pioneered methods to optimise the treatment of this type of cancer.