Download this transcript (29KB)

This podcast is presented by The Centers for Disease Control and Prevention. CDC - Safer. Healthier. People.

I did my dissertation work with Ros, so she was referring to a lot of things I'm doing, so the early work I did in her lab, I'll review a little bit of that before talking about some of the new projects that I'm doing. So what I do is the area of research I call Relational Agents, which are as Renata said, computational artifacts designed to build long term social and emotional relationships with users. So why would I want to do this? I'm not just interested in building computer friends or buddies. But I'm particularly interested in looking at professional relationships in cast settings in which the relationship or the quality of relationship is known to have a significant impact on past outcomes. And so one of the areas in which this is really known to be true is in helping professions in general and healthcare in particular, there's a notion of working alliance or therapeutic alliance which is the trust and belief that a patient and provider have in working together to achieve a therapeutic outcome.

So in psychotherapy, in physician/patient interaction, and nurse/patient interaction, this has been shown to lead to greater patient satisfaction of course, but it also leads to greater adherence or compliance to a health prescribed health regimens. And because of that, it leads to greater outcomes. So what I'm interested in doing is building automated health communication systems that can have some of these elements of the verbal and non-verbal behaviors that professionals use to establish a trusting relationship with a patient in order to boost adherence and outcomes to the intervention that's being done. Again, I'm trying to simulate this one on one interaction between an expert health professional and a patient or consumer which is arguably sort of the gold standard of health communication.

In a nutshell my goal is to take the decades of research and health communication between providers and patients and building that into systems for health behavior change and health education applications. So in the area of health, behavior change; I've built several systems now for doing exercise promotion and I'll talk about some of those. I have a system right now that's in clinical trial for doing medication adherence promotion for young adults with schizophrenia. And in health education I'll talk about a system we're doing now for educating patients prior to hospital discharge; about all their self care procedures and medication taking regimens and so on before they get sent home.

If we want to simulate face-to-face interactions between health provider and a patient, and we want to be able to emulate the verbal and none verbal behaviors that go on, such things as facial displays of empathy, hand gestures for communicating information, gaze cues for turn taking, then my stance anyway is that you need to give the computer a body so that it can project these non verbal cues in the same way that people do. So the animated systems that I build have an animated computer character that sort of represents a health professional that you interact with. To the extent that I can I try to sense non verbal behavior on the part of the user as well, but mostly on getting their verbal input into the conversation, so simulating face to face conversation.

The relational aspect I'm drawing from a number of different disciplines, looking at the way that people build long term relationships in professional settings. So from health communication we have things like appropriate – looking for empathic opportunities and displaying appropriate empathy for the patient. From social psychology we have things like using self disclosure to build intimacy over time. For social linguistics we have politeness theory which is a form of request that you might make to somebody based on the nature of the relationship that you have. And also longitudinal things like increasing common ground. So learning things about the patient over time, both for therapeutic affect so you can bring those up in later conversations to say, I know you like movies, so why don't you treat yourself as a self reinforcement because you met your exercise goals for this week. But also to give a sense of continuity to the relationship as well; doing things like social dialogue, appropriate uses of humor, communication and so on. So I'm trying to bring a lot of these into the human computer interface again to simulate this ideal relationship. So this is the first relational agent that I built while I was in Ros's group. This is Laura; she's an exercise advisor that chats with you a few minutes every day about your exercise. She tries to get sedentary adults to do more walking. So I'll show you a brief sample interaction. So this is a typical daily interaction that you might have with Laura. She talks using a synthetic speech and synchronized non verbal behavior, user inputs are limited in this case to multiple choice input, for a number of reasons which I could talk about, but. (video plays)

So she goes on. We can stop her at that point. So she eventually gets around to talking about exercise, but after some amount of social chit chat, that's characteristic of these sorts of ongoing interactions that you might have with an exercise advisor. We did a study at MIT with a hundred, mostly students and showed that those who interacted with an agent versus no agent did do more walking behavior. We also looked at whether she used these social and emotional and relational behaviors made a difference relative to Laura without all these social behaviors. That led to a significant increase in working alliance as measured by a standard instrument, so this is the social bonding with the agent, although that did not translate into increased exercise, partly because it was only a one month daily contact intervention.

When I was at The Boston University School of Medicine, I replicated this study. Partly I extended the length of it so this is a two month daily contact intervention. But I was also interested in looking at the user population that perhaps had low computer literacy. So Boston Medical Center is a safety net hospital, urban population and the geriatrics clinic that I collaborated with had significant doubts as to whether their population would even use the system if I installed it in their homes. So I basically gave these folks in the study a desktop computer system for two months, installed it in their home, it had a touch screen interface dedicated use, so they just had to turn it on once a day. Laura walked on and they had their chat about the exercise and then the thing shut itself off. It was a very small pilot; we had 21 subjects in this study. The age range was between 63 and 85, mostly African-Americans, mostly overweight, mostly low reading literacy. Of those randomized into the computer group, half of them had never used a computer before, and another third had only used one a few times. One of the most important results is just ease of use. So the training session went something like this: I would sit down with them and I'd have Laura come on for initial interaction. She would say hi, and they would inevitably say, hi. And then I'd say, no you just have to touch what you want to say on the screen. And that was the end of the training session. From that point they were off and running on their own pretty much. But they all found it extremely easy to use; there were no problems with the system. At the end of the two months I asked them to characterize their relationship with Laura, and they felt that she was much closer to a close friend than a stranger. High levels reported trust, high levels of liking. And perhaps most importantly, the control condition was a standard of care intervention. So this is an intervention that they were just doing in the geriatrics clinic at that time, which involved pretty traditional giving patients a pedometer, some brochures about the benefits of walking, and some log sheets and saying good luck. And relative to that the patients who used the daily advisor roughly doubled the amount of steps they were walking over time. So that was a good result but most importantly for me was that a population who wasn't used to using computers perhaps found this to be a very natural easy to use system that they could get up and running very quickly.

This was still by health care standards a fairly short study; it was only two months long. One of the interesting research questions that came out of this was what kind of behaviors do you need to put in such a system so that people stay interested and engaged with it over the long term. So everyday she had some variation in the conversations, some new topics that she would bring up in social dialogue. But they still felt that it was a little bit repetitive by the end of the two months. One woman said it would be great if Laura could just change her clothes occasionally. So I have an ongoing project now to sort of look at what it would take to scale this up to say a year or longer of daily contact and interactions and what kinds of things can we bury in the verbal and non verbal content to keep people's interest and engagement in this.

So what I really want to talk about today is a couple of projects that have spun out of this work. The first one is what if we could put a relational agent on a portable device that you could carry with you all the time. So it's available if you need help with your health behavior; if you need information, and if you couple it with a sensor that can determine if you're engaged in a healthy behavior or not, it could be proactive about interrupting you and motivating you to do something. So obviously the wearable form factor or the portable form factor makes it widely available. It can not only initiate interaction, but if you initiate an interaction, it can know something about what you've just been doing. So are you in the middle of a walk, did you just finish a walk, how long was it, for physical activity intervention. There's also untested conjecture that if you have a device that you own that's personal and it's with you all the time, and you have more frequent interaction with the character, perhaps that leads to better working alliance formation as well, it's more a part of your life. There are a lot of commercial products that are coming out on this basis, the one that Astro mentioned, there's also a cell phone now that has integrated accelerometers and a Nike iPod and so on. As Jay mentioned this morning, just providing information to people about their behavior is a necessary but not sufficient condition for behavior change. So it has to be coupled with counseling, problem solving behavior, goal setting, positive reinforcement, all things that a good counselor health provider would do. So this is a scenario that we're building a system for. Again this is an exercise promotion. We're having people schedule a couple of times each day that they're committing to go for a brief walk. And at those times if they're not doing their walk, we can sense that and the system interrupts them and engages them in a problem solving dialogue to say, looks like you're having trouble going for your walk, can I help you with that. What's keeping you from doing it? And seeing if we can help them overcome those obstacles and actually do it. If they do their walk, when they're finished we give them positive reinforcement. In this interface because we expect people to be using this at work, we're not using speech output, so it's using text balloons with the animation for privacy reasons so people can't overhear what you're talking about, and then the multiple choice input again. So we're still preparing a field study that's supposed to start any day now. We're going to give these to a hundred users for a year long study each, and they'll get it for ten weeks to see how well giving them just-in-time counseling does relative to end of the day counseling. But we've done a number of studies in the lab looking at different design variables or usability issues. One that we did is just saying does the animation add anything to this. Why not just give them text messages or a static character image or if we give some limited linguistic audio like disperse markers or back channels, things like “uh-huh, okay” such that it gives you perhaps more of a natural feel of conversation but an overhear can't understand what's being said. Does that add anything to the interaction? So we had people engage different versions of the system in a brief conversation with mostly social dialogue, then with a few health tips at the end, and we found that the animation did in fact score higher, significantly higher for working alliance, which is a social bond measure, perceived caring of the agent, when it used the animation compared to static image or text. This is also true for rate of creditability of the health information that was delivered.

Another thing we've been interested in, and I think it's related to something that Ros is looking at also is people are going to have these things with them at work. It's time for a walk, but they forgot about it or they're in a meeting or they're busy with their email. What's the best way to interrupt them such as to maximize compliance with the health behavior that you're trying to get them to do; to go for a walk, to rest their wrists, to take their medication, whatever that might happen to be? And we're particularly interested not just in immediate compliance, but long term. What is it – you know if somebody is going to be wearing this thing for years, what's the best way to ensure long term adherence? Based on some theories from social linguistics and politeness, our guess was that if you just simply look at the politeness of the manner in which you interrupt someone, that as you get more annoying people will be more likely to stop what they're doing and pay attention to the device. If it makes a louder sound, or buzzes or does something that's more disruptive, it makes sense they're going to stop and pay more attention than if it just gave a light ping, and you could just ignore it and keep going. But there's a cost to that which is as you get more annoying people are more likely to throw it in the trashcan at the end of the day. So there's some balance between short term compliance and long term compliance, and also from social linguistics, the shape of this curve and the sweet spot changes over time as your relationship evolves. That is as you get more familiar with somebody, they can use less politeness in making requests of you. So you could get more and more annoying as people got more familiar with the device.

In the lab studies that we did on this we couldn't have people get up and go for walks during a session, so what we did was we looked at wrist rest. We have a primary task where we're asking people to type a report as quickly as they can and then periodically while they're doing that the PDA that's sitting next to the desktop computer has an audio alarm and then they pick it up and have a conversation with the character, and the character basically tells them that the benefits of resting their wrist periodically to avoid carpal tunnel syndrome and other problems like that, and could they please rest now for as long as they can. So there's sort of this tension set up between what the experiment is asking them to do and what the character is asking them to do. So in the first study we just simply wanted to look at politeness versus annoying. We just simply varied the audio alarm that was used; everything else was exactly the same. That was the most polite, going up to the most annoying. And it's kind of as we predicted. People rated these on the polite versus the annoying scale as we had predicted. But as it became more and more annoying, stated desire to continue using the device which is a proxy for long term adherence, fell off which is what we guessed. And when we looked at how long they rested for, on the initial presentation of each of these stimuli, in fact their rest time did go up as we got more annoying, which is sort of what we predicted. This wasn't statistically significant, but it's a pretty obvious trend. But on the second presentation of the stimuli we were already seeing some of the long term effects which we thought we would. That is for the most annoying sound people just basically ignored the thing totally. So there is this trade off between long term and short term adherence, and picking your interruption method.

We did another study in which we looked at negotiation strategies. So when we interrupt someone they might say, you know I can't stop right now but I can stop in five minutes once I finish this phone call. Or maybe we give them a notice that in two minutes it's going to be time for your walk. So we looked at a couple of these strategies, and we also compared this to and I think Ros talked about which is, when you interrupt you apologize and you offer appropriate empathy, so it's social and relational strategies for negotiating interruption. We found that overall that in fact social and relational strategies worked the best, both in desire to continue use and also in terms of how long they rested. Followed closely by negotiated, which is basically a snooze button; I can't do it right now but give me ten minutes. So in the system that we're building, we're sort of integrating the best of these features. In the field study that we're just starting, we going to be looking at specifically the efficacy of delivering counseling just in time, at the moment of decision making. So you said you were going for a walk at 2:00, it's 2:00 right now, and I see you're not walking, can I help you with that. Or if you just finished your walk, giving positive reinforcement; great job, we're doing good work together, and comparing that with giving exactly the same counseling content, but at the end of the day review session the way that it's normally done.

Another project I want to talk about is what we have called a Virtual Nurse. This is being done with Boston Medical Center. This is a system to educate patients about self care regimens just prior to hospital discharge. So I should be careful about this; we're not talking about replacing nurses. In fact the BMC got calls from the Nurses Union the last time I gave this talk. So how do we best explain health information to individuals particularly with low health literacy in order to maximize adherence to the recommended regimen? And the application that we're working on is hospital discharge, this is one of these points of transitioning care in which there's a significant number of errors that occur. Twenty percent of individuals who are discharged from the hospital have adverse events or are readmitted within 30 days, and a third of these have been shown to be preventable. The team I'm working with at Boston Medical Center is led by Dr. Brian Jack, who's been researching this for the last several years. There are a number of reasons for the errors at this point of transition, but one of the big ones is lack of education for the patient. There are some other ones like medication reconciliation, but the number one seems to be proper education of the patients before they get sent home. They're not clear on what they're supposed to do.

Dr. Jack has developed this manual discharge process, which involves a number of steps starting with patient education, medication reconciliation after the patients are sent home; they have a nurse follow up with a phone call to make sure they're not having any problems, or seeing if their medication need adjustments or so on. And this manual discharge process has just been adopted by the National Quality Forum as the new standard, so I believe about half the hospitals in the US are members of the National Quality Forum, so this is going to be adopted widely in the near future. There's a study underway at BMC right now to look at the efficacy of doing this manual discharge process compared to the standard of care the way they normally do it. The preliminary results are they have about 450 patients that have gone through the study, and they have a target of about 750. But the results so far seem to be that this makes a significant difference for patients with low health literacy, both on process measures so the patients are more satisfied, they feel that they're better prepared to go home, but also on outcome measures, looking at 90 day readmits to the hospital.

So where I come in is we have another grant to look at if we wanted to disseminate this discharge process now to all the hospitals in the US, we know that it takes a nurse almost an hour of face to face time with a patient to go through this process. So hospitals are going to have to hire an army of nurses to provide all of this education to patients. So can we automate this process so that hospitals can adopt this new strategy, but not have to hire quite so many nurses? So that's what we're building one of these animated characters that will be wheeled up bedside prior to discharge, go over all the medications, follow up appointments and other self care issues with a patient before they get discharged. Test comprehension on key issues; if there are further questions that the patient has, relaying those to a human nurse to follow up with them afterwards. And then having an automated phone system that does the follow up phone call with patients to see if they're having problems after discharge.

So this is what the system is going to look like. It's just a rolling kiosk with an articulated arm so while they're sitting in the bed, they'll have the touch screen computer that they pull over in front of them, and we're going to do a randomized trial in which we compare the automated system with the current manual reengineered discharge versus standard of care to see which does best. One of the things that we're particularly interested in looking at here is also looking at how health professional explain health documents to patients in general. They have a discharge book they put together for patients as part of this process, so looking at the verbal and non verbal behaviors that are used in drawing attention to different and specific items in the book to maximize comprehension, especially for individuals with low health literacy. And we're also interested in looking at this for other documents like informed consent documents and so on.

I will attempt to show a quick demo of this. This is a system still in development. We're spending most of our time on medications, sort of describing side affects and warnings that have to do with all of the standard medications that are prescribed at hospital discharge time. We've developed a number of tools for building these systems, partly so that individuals that don't have programming backgrounds that health professionals can actually design the dialogue content. So we have a visual dialogue design tool that lets people design augmented transition networks like this. Basically each one of these boxes represents the state of the dialogue, where the character says something or asks a question and then transition based on what the user says or information from the database or from the electronic medical records. As this is constructed you can incrementally run tests of how well it's working. So this is Louise. So Louise is not cooperating today. She was a minute ago, trust me. So that's it. I have a number of students who are working with me on this at Northeastern but I think that this sort of natural intuitive face to face interaction is particularly good for educating patients, and particularly those with low health literacy or computer literacy. Thanks.

To access the most accurate and relevant health information that affects you, your family and your community, please visit www.cdc.gov.