2018 PCORI Annual Meeting *********************

Breakout Session How Telehealth Can Improve Patient Care

and Outcomes

Presenters: April W. Armstrong, MD, MPH

Dror Ben-Zeev, PhD Moderator:

Penny Mohr, MA Discussants:

Mark Ishaug, MA Jennifer Reck, MA

Sylvia J. Trujillo, MPP, JD

SESSION TRANSCRIPT This is being provided in a rough-draft format. Communication Access Realtime Translation (CART) captioning is provided in order to facilitate communication accessibility and may not

be a totally verbatim record of the proceedings. Names might be misspelled. Opinions, positions, and statements presented are those of the individual participants and

not necessarily those of PCORI, its Board of Governors, or Methodology Committee, unless so indicated.

>> Penny: we're ready to go. Ok. We're a little behind time but we'll make it up and hopefully we'll have plenty of time for discussion afterwards. Welcome very much to this session. I'm very excited about this particular area. I'm a senior advisor within the healthcare delivery and disparities research program at PCORI. I also serve as a role -- providing sort of broad expertise and subject matter expertise in the area of telehealth. So i'm very excited to have the panel that's joining me today. The focus of this annual meeting is on evidence to impact. And we're very excited that two of our presenters today are going to be presenting the results of their study that are pretty much hot off the press. So Dr. April armstrong's study was just published in jama open science about two weeks ago. Is that right? Two weeks ago. And Dr. Dror ben-zeev's study was published in psychiatric services in october. Is that right? >> Dr. Dror ben-zeev: may. >> may. Ok. Almost hot off the press then. [laughter] So two of the panelists that are going to be presenting, Dr. Armstrong, she's associate dean for clinical research at the university of southern california. And then Dr. Dror ben-zeev is the professor of psychiatry and behavioral sciences and also a co-director of the center for behavioral research and technology and engineering at the university of washington. In addition, though to thinking about these great results, both of them have very promising results for patients. But in addition to hearing about that we also want to learn from stakeholders about what is needed to actually get these put into practice. And so how do we get to that impact stage. And to discuss their thoughts on this matter, we have three people representing diverse stakeholder perspectives. Dr. Mark ishaug, who is also a co-investigator on Dr. Ben-zeev's study, He serves as a ceo of thresholds, this is one of the largest and oldest behavioral health system -- providers in the state of illinois and was also the setting for Dr. Ben-zeev's study. In addition to that, we have providing the viewpoint of practicing clinicians, sylvia trujillo, a senior legislative counsel at the american medical association with an expertise in regulatory and legislative issues in telemedicine. And finally, representing the perspective of state health policy makers we have jennifer reck, a project director at the national academy for state health policy where she helps

advance evidence-based policy using patient-centered outcomes research. Perfect. And there's our panel right there. So i'm going to give a brief overview of our telehealth research and investment in PCORI and then i'm going to hand it off to first Dr. Armstrong and then Dr. Ben-zeev supported by mark ishaug to talk a little bit about the results in his setting. And then we're going to turn it over to our stakeholders for reaction. What's needed to actually put these into practice? So, we have a very large portfolio in the area of telehealth. I think there's a lot of good reasons for that. This is really reshaping the healthcare delivery system very rapidly. This is a slide that just gives a basic image of some of the rapid change here. This is in telemental health in the medicare program. But over a 10-year period the number of telehealth visits in medicare grew by 14-fold. I read in 2015, kaiser permanente, about half of their visits were virtual visits. And they were anticipating by 2018, which is this year, that the number of virtual visits would actually surpass the number of inpatient visits. So we have invested to fund over 78 studies in this area. And a lot of that is driven by investigator interest. So a lot of this came from investigators that were submitting applications to what we call our broad portfolio. What I show up here is sort of these concentric circles in telehealth, health and telemedicine. I'm not going to get into the details of the definitions there but basically one thing that I did want to point out is when you think about telehealth, a lot of times people think traditionally back to the consultative model that works with a specialist and a private patient in a clinic through interactive video. And the largest part of our portfolio is actually mhealth, meeting the patient where they need when they need it. So mobile health accounts for about 2/3 of our portfolio. But that said, that's not to lessen the importance of the investment that we've made in the area of telemedicine. Some of our larger and longer studies that cross multiple states are in the area of telemedicine. So we're very excited about those study that we funded there. We think that telehealth, also one of the reasons why we've gotten so much interest from the investigator community in this area is it's a very fruitful area for patient-centered outcomes research. First of all whether we're talking about personal health -- personal care, capturing individuals' preferences, autonomy and needs, the interface through telehealth has the great potential to do that. And potentially through the cultural tailoring of the interface one can better reach low health literacy populations, limited english proficiency. Also, this is an area where there's a lot of need for comparative effectiveness research. There are lots of studies out there. There's lots of systematic reviews.

There's been a lot of research done in this area but not a lot of head-to-head trials that say should we use, for example, this mobile health app versus that mobile health app that can health with both clinician and patient choices. And finally, it's one of the barriers that people have talked about in terms of using telehealth has been that there is like a disconnect between clinicians and -- end user and the patient and the interface. So potentially by engaging patients and other stakeholders in the development of this interface from the beginning and in the implementation, we can better integrate telehealth into the work flow. And so it's more acceptable to patients at the end of the study. Our portfolio spans a broad array of conditions. I think this is also the fact that telehealth is becoming ubiquitous. There's a lot of studies we have concentrated in the area of telemental health as well as in diabetes and management of chronic diseases we also have some areas we're looking at telehealth for screening or looking at telehealth for managing rare diseases. We have classified our telehealth portfolio by purpose of the studies. And similar, as I said before, in terms of the focus on mobile health and self-management of chronic disease a lot of this is really related to promoting self-management and on the educational piece. But one thing that did I want to point out, too, is that another unique aspect of our investment in telehealth is that a large proportion, almost half of our studies, are focused on addressing disparities. So the sort of digital divide is raised a lot. We have a lot of study that are working to try to bridge that. In the area of addressing disparities, we have addressed racial and ethnic minorities, low income, persons with disability, and also as I mentioned, low literacy among these other priority populations for the technology platform, a large number of our studies do use the mobile phone or the tablet but we also have a few study that are talking about storing forward technology. And apple talk about that. We also are looking at outcomes that are important to patients. And it's interesting. When we were look at systematic reviews, this was something that was called out a lot. A lot of studies have looked at acceptability but haven't really looked at the clinical outcomes that are important to patients or those outcomes that are important to their health and well-being. Obviously being PCORI, that's something that's important to all of our studies. When we did look at the systematic reviews in this area, some recent systematic reviews we categorized some of the evidence gaps. And we feel that our portfolio is really addressing a lot of these quite well. So we have some -- as I mentioned, a lot of them are in patient stakeholder engagement. About 60% are head-to-head trials. And while we do have this focus on addressing disparities, a smaller proportion are looking at culturally tailored interventions but there are some important ones there. So in conclusion, I think that our portfolio really has the potential to impact the field,

advanced knowledge in this particular area, because of the unique aspects of what we've invested in. And i'm going to, without further ado, hand this over to Dr. Armstrong. >> Dr. April armstrong: thank you, penny. And thank you for inviting me here to speak. Today i'm going to talk about how we could use telehealth to improve specialty care delivery in chronic skin diseases. I have no relevant conflict of interest specific to this presentation. First i'm going to discuss some evidence gap and then I will provide a study overview of the study that was funded by PCORI and then highlight some of the study findings. And then finally explore what's needed to change practice. First, evidence gap. If I could, by a show of hands, ask, how many of you have found it difficult at times to get to see your dermatologist? Ok. So most of the room have. Yes. In fact, the waiting time for dermatologists is among the highest among the specialty providers. I often have patients who will come to my clinic and say, Dr. Armstrong, I tried to see you but the wait is three months so i'm here right now but my rash is gone that I originally called you for. And I say, oh, well, that makes it an easy visit. But true question is, how can we be responsive to our patients the most that they really needed us? My husband's grandfather, my husband is pushing 50 now, and his grandfather is 107 years old. And I asked him -- I said, "gosh, grandpa, how many doctors do you see? You must see a lot of doctors." He says, "no. I only see three doctors. My primary care doctor, my cardiologist, and surprisingly my dermatologist." And I say, "really? You see" -- "those are one of the three physicians you see?" He said, "yup. And, by the way, you can look at this?" >> [laughter] >> Dr. April armstrong: skin is, as you know, a fascinating organ, actually the largest organ in our body. If you line up all the blood vessels in your skin end-to-end it would be around 11 miles. Many of you see a dermatologist because skin is visible and skin can show different pathology. I'm not here to convince you the importance of dermatology but rather use that as an example of showing how specialty care is really important and how we can improve access to specialty care. One of the key gaps in terms of accessing specialists is that it really depends on where

you live. Studies after studies have shown that if you live in rural areas, your access to specialty care is often limited. Your outcome not only for dermatologist care but even including mortality can be effected by where you live. We know that regular access to dermatology care, if you don't have that, as a rule poor outcomes. These are pictures of patients with psoriasis. I think while the awareness of psoriasis was low many years ago, I think these days it's difficult to actually probably go through your evening program without seeing a commercial for psoriasis medications but the key thing is that a lot of patients are still untreated or undertreated. And part of that is access. So patients not only present with these red plaques but also a third of them develop psoriatic arthritis and patients with severe psoriasis are at high risk for cardiovascular adverse outcomes. So when we think about the traditional in-person model, we can think about, in terms of the patient typically to the primary care doctor. And then depending on that primary care doctor's decision making, the patient can be referred to the dermatologist or the relationship can exist with the pcp. Now, with traditional asynchronous teledermatology -- what I mean is that your storing forward dermatology, that the images are captured but then they're sent to the specialist at a certain time. And the specialist looks at these images at a different time. So in the traditional asynchronous teledermatology, the patient actually has to find a pcp, primary care provider, who is interested in doing telemedicine. And then that pcp would take the photos and send that to a dermatologist. And one would then look at these photos, render their opinion, bring those opinions back to the pcp. Now, there's a lot of problems with this model. Number one is that the patient has to find a pcp who is interested in telemedicine and that's actually more challenging than what is imagined. The primary care physician also has to be flexible in terms of looking at dermatologists' recommendations and knowing how to convey that to the patient. The other part that we found is that patients like to understand the recommendations from the specialists. What if they have questions about the recommendations? Who will then be able to answer their questions? So in this model the pcp often plays a middle man. The third model is the collaborative connected health model, which is an online model we actually tested in our study. This model originated from the idea that if the technology is good enough, can we create an environment where anything that can be done in person could be done online? So with that, we like the patient to have direct contact with a dermatologist. Pcp could have direct contact with the dermatologist.

And the pcp could have direct contact with the patient. And i'll go into detail later on. So here is the gap in terms of thinking about the different models to care for our patients with skin diseases and then finally, something I think penny has alluded to earlier is a lack of validated instruments to really measure our outcomes in telehealth research, and especially in teledermatology. I can probably count on both of my hands the number of study that have used validated instruments, things that instruments that are created on the fly typically has a lot of problems. For example, there are studies that used -- whether you asked patients whether they have improved or whether they have worsened, such dynamic instruments are problematic because it depends on a patient's recall. What if they can't really recall to a precise degree how they did? So i'm going to tell you a little bit about my study. And then first i'm going to tell you about the intervention we tried to test. And then i'm going to go over the study design. So the intervention is as such. Intervention is such that we have a telemedicine provisional platform. If you were a psoriasis patient, you could go to your pcp through this online platform, mean going there virtually. You can log on. You can submit your images. And you can answer a set of structured questions that's intended to elicit how you are doing. And then your pcp can converse with you on this platform, render recommendations, and prescribe medications through that platform. Now, the pcp can also communicate with the dermatologist via that platform, can do it by one of two ways. Number one is that if they want to keep primary responsibility of the patient, they can do it consultatively, much like the arm I talked about earlier in terms of traditional telemedicine. Or, two, they can actually transfer care of the patient to the dermatologist. Once the patient is in the care of the dermatologist, the patient can also visit the dermatologist virtually online just as they would in person and they can ask questions, can submit their images, their history as they would. So here's an example of an online encounter. You can have the pcp or the psoriasis patient who then they have instructions on taking these standardized images. They can be uploaded on to a connected health website. And then the dermatologist can evaluate these images, make recommendations, and e-prescribe for the patients. So our study type a pragmatic randomized controlled equivalency trial. I want to emphasize this is an equivalency trial. It may seem like a minor point but in study design it's what we consider pretty major. Equivalency trial is designed to test intervention a -- if intervention a is equivalent to intervention b.

We intentionally design it this way because we want to understand is it exactly the same. We have three specific aims. The first aim is to determine whether this online model would result in equivalent improvement in psoriasis disease severity using validated outcome instruments. Number two, we wanted to know if this online intervention would result in equivalent improvement in quality of life as compared to in-person care. And finally, we wanted to know whether this intervention would result in better access to specialists. Here's the trial design. We recruited patients with psoriasis and they were randomized into two arms in one arm is the in-person arm, essentially standard care. They can go to their primary care doctors or their dermatologists, however they usually carry out their care and then they are followed over the course of the year. In the other arm, they would have these interactions online in the way I described earlier. So the study was year-long. This is a chronic disease and we wanted to know how patients did over time. The visit frequency, however, is determined by the patient and doctor's interaction specifically. Because what if you have a severe patient that you needed to see more often than the patient with a mild psoriasis that you didn't need to see that often? We want that piece to be pragmatic just as it would in real life. I know, while the visit frequencies are determined individualized to the patient, we collected the outcome as structured intervals this is because we want to make sure that the data collection was uniform, across the two arms. So, here's a baseline patient demographics. I just wanted to point out a few things. About 34% of our patients were hispanic. And this is twice that of the national average. So national is 17% of patients are hispanic or latino in origin. And also that these patients had experienced various type of therapies before. So they span the disease spectrum. Importantly, the sites were northern california, southern california, and colorado. And hopefully with this multi-centered approach, our findings are more generalizable. First i'm going show you our primary outcome, which is a change in what's called a posi score. A posi score is nothing more than a granular way of scoring how the patients are doing. And we take their psoriasis plaques and then patients look at the redness, thickness, and the scaling as well as the body surface area that's involved. That's tracked by both arms over time. So as you can see, so the red bars here represent the online arm and the blue circles represent the in-person arm. What we're looking for is if there's differences over time between the red bars and the blue bars, these vertical bars. And there's a measure in which we do.

So on the lower right-hand corner, you will see that we have for this study defined an equivalence margin. And what we want to see is that if the two interventions were equivalent, would the difference reside within this equivalence margin? In other words, if you had intervention a, online intervention, and the in-person intervention, how do you define what is sameness or equivalence matters? So you define that priority based on clinically significant differences based on both groups. You define that sameness first. And then do your study to see if the two are actually the same. If they weren't the same, they would fall within that equivalence margin. And if any part of that bar goes beyond, that means that they're not the same. So as you can see, in terms of psoriasis severity, there were no differences with regards to the improvement in psoriasis severity in both arms. What about body surface area over time? Here is another measure of psoriasis severity. The same thing happens. There are no differences between the two arms. How about patient global assessment? This is patient's assessment of how they are doing in terms of their psoriasis overall. What you will notice is that, interestingly, while most of the confidence interval is within the pre-specified margin, it is actually outside of the margin and for one area. Actually in the favor of the online group. That means that the patients overall in the online group actually thought their psoriasis did just a little bit better than the in-person group. What about improvements in quality of life? So, we measure that through two validated instruments. One is called skindex, looks at the patient's quality of life with regards to their emotional health as well as their functional health. And what we find is that, as you can see here, again, it's within the equivalence margin; that there were no differences in the quality of life between the two groups. How about dermatology-life quality index? Another outcome measure that looks at the quality of life. Just making sure that using different validated measures that we still find the same thing. As you can see, again, there were no measurable differences between the two groups. In fact, the two groups in terms of improvement in quality of life was the same. How about access to care to specialists? We measured two things. Number one, how far the patients had to travel. Number two, how long they had to wait in the waiting room. So the result here is not earth shattering. On the left-hand side is the in-person group. And the right-hand side is the online group. Because it's a pragmatic model, online patients if they wanted to go have in-person visits, they could.

And we track that. And out of the 149 patients that were randomized to the online group over the course of the year, there were only eight in-person visits. So that is quite telling. I was surprised about that because I thought everyone would just rush to someone in person. But as you can see here, there were a significant difference, as would be expected between the total distance traveled, between the in-person online group and as well as in-office waiting time between the in-person and online group. We also did qualitative assessments. We asked patients regarding their feedback. As you can see, many of them had positive feedback regarding this intervention. I think while some were apprehensive in the beginning, over the course of the year they did get used to it. So considerations for implementation are a few. I'll make my comments brief now because I know later we'll have a wonderful panel who will also focus on this. I think technology is central to the dissemination. Having a scalable and sustainable reimbursement model is key. And finally, we want to consider medical-legal considerations, one of the key considerations for the providers. So we want to be sensitive to that. And number four, workforce considerations. We can't clone dermatologists, at least not yet, but we want to see how technology can make the best use of their time through various means. So in conclusion, the online model produced equivalent outcomes with regards to disease severity as well as quality of life and it provided superior access to care. I think that this is one example of patient-centeredness. Instead of having patient go to the location of care, we are trying to bring care to the patients and then finally, I hope that you will see this model has the potential of being transformative and potentially applicable to other areas in medicine. You can learn all of this at the PCORI website. Our primary outcomes were published in jama network open and then our secondary outcomes on quality of life is accepted by a journal of investigative dermatology, a top-tier impact factor journal in the dermatology world. And our third aim has been published in telemedicine and ehealth and a fourth manuscript is being submitted. So it takes a village. There were three major centers. So i'm indebted to all of these people who helped carry out this project. My many thanks to PCORI and penny for spending the last few years with me. I want to thank you for your attention. [applause] >> penny: thanks very much. I'll turn it over to Dr. Ben-zeev. >> Dr. Dror ben-zeev: hi, everyone.

Let the record show that I had a jacket on. Those spotlights are super hot up here so I took it off. But no sign of disrespect. I wasn't wearing a tie, though. I have an issue with ties let's start with the disclosure, quick on my part, my colleague mark, doesn't have anything to disclose. We'll jump right into it. Before we talk about it briefly, let's put some of the conversation about mobile health into context of the we are all alive to see the fastest adoption of technology in recorded human history. So all of you, i'm sure, remember a time where there were no cell phones and there were no smartphones. The u.n. estimates that back in 2001 there were approximately 1 billion mobile cellar subscriptions worldwide, estimate then of 2017 is that that number exceeds 7 billion. Now, that, of course, doesn't mean every single person on the planet has a mobile phone. Companies have more than one license. Some people have more than one device. But by and large it means the vast majority of the adult population on the planet has access to a mobile device and mobile cellar infrastructure that allows us reach them in realtime and in real place or for them to reach us or resources that are on the device in real time, in real place. So from a healthcare perspective, this is a game-changing opportunity and I can't emphasize that enough. There are more people on the planet with access to a mobile phone than there are to a working toilet or running electricity. And your smartphones, most of you, judging by the demographics of this audience, most of you have smartphones. Your smartphones have greater computational capacities than nasa had when they sent the apollo missions to the moon. And that's sitting in your pocket right now. So hopefully that helps contextualize the territory that we're in. It's a really, really exciting time. Back to serious mental illness. When we talk about serious mental illness, referred to severe -- person is bipolar disorder or major depressive disorder where there severe functional impairment, the core aspect of serious mental illnesses. We're roughly talking about 4% of the world's population that have a disproportion yacht impact on all levels in terms of days lost to disability, in terms of distress, in terms of the er visits. So they are highly represented and untreated -- unmet need, untreated serious mental illness has dire consequences. So if you have an smi and you don't get care, you are dramatically more likely to have a shorter lifespan, to be the victim of abuse, to be homeless, to attempt suicide to succeed with that suicide attempt, and to be incarcerated. So it is a major, major public health concern.

But, while we think of serious mental illness as chronic, they are not necessarily static. That means that there are periods of recovery where people, even in the context of persistent symptoms, do rather well. People move between these periods of psychiatric system exacerbation and perhaps hospitalization and relapse through periods where people are quite high functioning and really, we know now that illness management interventions can really lengthen and increase these recovery periods. That's really the backdrop for the intervention that we tested in the study. So in this, interventions are typically deployed in the context of a clinic setting, medical center, brick-and-mortars clinic, and, of course, there are a lot of challenges in this type of deployments. Distance to the clinic, transportation, hours and schedules of the clinic. If you have a serious mental illness and you may struggle with [indiscernible] sitting in a room with other patients and other people might not be on the top of your to do list that day. The stigma that is associated both with the label of a mental illness or coming to services at a clinic is pretty significant with this population. And, of course, there's an issue of quality of care. The majority of people with smi get services from individuals who are not necessarily doctoral level, trained psychiatrists and psychologists. People who do their best but may have a bachelor's level education or less or perhaps very little experience and background in providing services. And at the community mental health centers where most people get services, the turnover, because they're highly under resourced environments, the turnover is great. So it's difficult to establish a high level of expertise and delivery of illness management strategies. So that's where mobile health, for mental health, may come in. We mentioned that mobile phones are ubiquitous. This is true for people with serious mental illness. Now analysis show, with studies conducted on four continents, show that people with smi can use mobile devices, that the penetration rate are not all that different from the general population. In fact, it's one of the few areas where the gaps between people with smi and the general population are quite narrow because the affordability of the devices and in the u.s. access to the federal lifeline assistance program which provides mobile devices and call in text minutes for people who have a certain level of disability. Now, smartphones with those computational capacities and multi-media resources can really bring mobile health to life. It doesn't have to be a stayed written intervention. We can leverage video and audio and a host of sensors that are embedded in the device to really try to do something that perhaps we might not be able to do with other forms of technology. So there have been several studies, some from our group, some certainly others, from other groups in the u.s. and beyond, showing real promise in terms of mhealth for smi. So acceptability, feasibility, single-arm trials, mhealth has this problem with pilatitis, pilot studies, things published but few rct's and certainly very few comparative effectiveness

trials in the area of mhealth for mental health. So we wanted to see how does mhealth compare to clinic-based care? There's all of this promise of technology being able to transport delivery of services outside of the brick-and-mortar. How does this play out when you put this head-to-head with a valid evidence-based intervention? So i'll describe the mobile health intervention that was deployed in our PCORI-funded study. It's called focus. It's a mobile -- it's a multi-mobile smartphone delivered intervention that my team and I developed over the course of multiple years. The way the focus works, it's installed on to a device and then the participate gets up to three daily prompts that will take over the home screen and engage in an assessment. And i'll demonstrate in a moment. In addition to that, all the self-management resources are installed, downloaded on to the phone and are available on demand 24/7 regardless of whether the individual has connectivity or not. So this can work even in areas that are blackout zones in terms of access to internet or a cell plan. In addition to that the focus intervention package also has a clinician dashboard. So the responses that patients have to those self-assessments are uploaded to a secure remote dashboard and they are available to an authorized individual. In this case, a person we call the mhealth support specialist. A person that would look at the data, help with technical trouble shooting using weekly calls to the phone, and also help bring the intervention to life. Tailor it with the individual specific life circumstances. And so how does this actually look? Focus will take over the home screen and say can you check in with focus right now? And if the individual says yes, it will immediately launch into an assessment of a target area that's one of five core treatment targets. And those core treatment targets are: voices or auditory hallucination, medicine or medication use. This is not necessarily medication adherence. So part of it is remembering to take your medication. Some of it is also how to strike up a conversation with your physician and your prescriber if you're concerned about side effects; social, which really means social skills training or coping with persecutory ideation, which is a common symptom in psychotic disorders; mood, which means anxiety or depression, depending on what you endorse in the assessments it will take you down a choose your own intervention route,: and sleep, which is not a core symptom of schizophrenia but we know from time and time again, from patient reports and family members, sleep is often an early indicator when someone is not doing well and certainly a consequence of exacerbation of psychiatric symptoms. When people are fatigued, everything else becomes worse. So we integrated that into the app that was directly informed by patients and the user center design face.

So focus jumps into an assessment of auditory hallucination -- [no audio] Baseline interview. It will ask, "hello, rachael, have you been bothered by voices today?" With a multiple choice response option screen. If rachel says anything but not at all, focus will follow-up asking about common dysfunctional beliefs. This is the type of thinking we use in cognitive behavioral interventions for psychosis. The target or the beliefs and the stress that the symptoms might have to that individual. The functional impairment that the existence of the symptoms may lead to. How would you describe the voices? They can't be controlled. They know everything. They're powerful. They're unpleasant. Or something else? These are common dysfunctional beliefs in psychosis. So here rachel endorses the voices know everything and focus jumps into action. Again, action means ideally very streamlined, unidirectional, simplified screens. Our user center design showed us that a lot of bells and whistles and multiple directions to navigate are really -- is not really well suited for folks -- for users with smi. So if it anything, we have to whittle down and simplify the development process. Voices may sound like they know everything but they don't. Can you think of any time when the voices made a mistake? How about when the voices were sure something was going to happen but it didn't? If the voices got it wrong once, they probably don't know everything. Right? Think about it. And the idea is to give people these quick and dirty tools to examine their beliefs about their symptoms, to engage in behavioral strategies like relaxation or striking up conversations with others who aren't socially isolated. In addition to the written version of this, like I mentioned, smartphones have these multi-media capabilities so there is this parallel video or audio version of it. And this is especially relevant for people who may have limited literacy and education. So people can select to either read an intervention or see an intervention. If they choose to see an intervention, they might see something like this. >> voices make mistakes all the time. Try this experiment to test your voices. Ask them to make a prediction about the color of the next three cars you see on the street. Write down the colors and the order that the voice is saying. Now test it. Are the next three cars exactly in the same order of colors that they mentioned? If they got it wrong, think about it. What else could the voices be getting wrong? Their predictions are not always right.

>> Dr. Dror ben-zeev: so the idea is to chisel away at beliefs in some indications people have been coping with these for decades, these ideas. By the way, those dishes breaking were not from the video. That would be an evil thing to do, right? To sort of talk about voices and at the same time have something in the back ground like that. So just a quick clarification about that. So the dashboard, again, accessible to the person who is authorized to log on remotely. It summarizes the person's engagement over the last seven days. The treatment areas that were assigned to that -- to that individual. The severity of the responses whether they're severe or non-symptomatic. And an overall log of the responses to the symptom assessments since they were involved in the intervention. So motivated clinician can see progress over time. Now to the study. We were thankfully funded by PCORI to conduct a comparative effectiveness trial to see how mobile health for smi compares to clinic-based care. So the study site was thresholds in chicago, a multi-reciting large agency that provides services to people with smi in the greater chicago area. It was a randomized control trial. 163 individuals with smi. About half of them people with psychotic disorders, randomized into one of two arms. Either the mhealth intervention or wrap, wellness recovery action planning which is a clinic-based group intervention delivered by facilitators with lived experience. Wrap. The modalities are quite different but the overall objective increased awareness to impending symptoms to reach out to others when in crisis. The overall objectives are quite similar. In terms of the study sample, the average age was 49. So not especially young or digital or technologically motivated group. Ethnicity 65% african-american. On average, a 5th grade reading level. And so folks that really represented the type of person with smi that thresholds provides services to. And we conducted assessments at baseline, post treatment, which is three months, and at six-month follow-up to see if the gains -- ideally gains are retained. So, again, a quick breakdowns of the allocation to arms. I will cut to the chase. In terms of the outcomes, there were no significant differences in the clinical outcomes or satisfaction ratings between these conditions. Both produced positive gains and satisfaction ratings were also high but there was the magnitude of effect and the direction of the effect was not statistically different between these arms. Again, dramatically different treatment modalities. One conducted entirely remotely through a smartphone and one in-person visit with that ml support specialist at the beginning of the three months.

But from that point on, everything's done remotely. Versus a clinic-based model weekly two-hour sessions with a group usually comprised of six to 10 other patients, two facilitators. In addition to the clinical outcomes, I want to draw your attention to something that was really quite eye-opening when we saw the outcomes of this. We looked at engagement and treatment commencement. If we're talking about access to care, will people have some sort of differential pattern when they are allocated to mobile health versus clinic-based models? If you look at the left -- if we look at the left side over here, once people are randomized into one of two arms, they get an envelope that tells them you either call the mobile health specialist to get up and running with your smartphone intervention or you contact your facilitator from wrap and you set up a clinic appointment. 90% of the people who are assigned to the mobile health arm went on, followed up, and used the smart phone intervention at least once. So they started the commence treatment. Whereas only 58% of people who were asked to come to the clinic ever made it into a single session. So we lose before people know anything about the intervention itself. They don't know what's going to happen in mobile health. They just know the modality and they don't know much about wrap, just that it's a clinic-based intervention. We lose 42% of our target audience right then and there when we ask them to come to the clinic. And that, in my mind, has implications for things well beyond smi. This speaks to mobile health, the potential for mobile health broadly. In terms of engagement, throughout the study the engagement was superior for mobile health. We lost people in terms of fully engaged participants throughout the study, which is common. Working with patients with smi. But our bar for engagement here was quite high. So what we thought of as engager was either a person who used the mobile app at least once a day for five out of seven days every week. So that's a fully engaged patient. Or person who sat for 70 out of 90 minutes of that group session. So there were certainly people who were using the app less, engaged in sessions for 20 minutes, so on and so forth, before they left. In terms of clinical outcomes, we saw significant but similar reductions in depression as measured by the inventory at three months and at six months. Similar reductions in symptom checklist, a measure of general psychopathology, things like anxiety and obsessive beliefs. Similar improvements in recovery measured by the assessments scale. So this is an indicator of hope, of confidence, in coping with symptoms. And when we look at treatment satisfaction, the outcomes were almost identical. So these are the average ratings on five patients, satisfaction ratings. I'm satisfied with the program.

The treatment helped me feel better. The treatment program was not interactive enough. That's the third one that, of course, is reverse scored. So the lower the score, the better that is. I enjoyed the treatment program. And I would recommend the treatment program to a friend. Again, dramatically different interventions. Almost identical in terms of patient satisfaction. So conclusions. Mhealth demonstrated superior treatment commencement and engagement when compared to clinic-based care, focus and wrap [indiscernible: speaking too fast.] To answer that original question, it seems that mobile health can actualize that potential. And that for some people, the mobile health approach might be more relevant and more engaging regardless of the content than other clinic-based approaches. From our perspective, next step is, how do we actually disseminate and implement this in the real world? How do we take things from research -- even though it's PCORI-funded, real world, pragmatic deployments, this is still artificial research to some extent. How do we integrate this into the work flow at community health centers and other clinics? That needs to be developed. No one's really done that yet. Hasn't really taken things the full length, that full pipeline of development all the way with rigorous research and real world meaningful sustainable implementation of mhealth. It's very interesting to think about how a program like the federal lifeline assistance program that offers devices and call plans might also leverage things like available apps if we know that there are evidence-based tools that are helpful, couldn't we also offer this as part of the basic things that we think are part of your rights as a citizen of our country to benefit from like access to 911? And what about our billing codes and reimbursement models How do we start incentivizing, paying, and reimbursing for these services whether it's to help patients navigate an app, look at patient data online, make meaningful decisions and care based on that data. There's a lot to be done. Right? Ok. Here you can learn more about our study and about PCORI in general. Feel free to e-mail me or to check out our activity -- there's a typo there which I should have corrected. It's mh4mh. If you google me, you'll find us at the university of washington. I'm happy to respond. Thanks for your attention, guys. I look forward for a lively discussion. [applause]

Oh, I should have presented mark because we're doing -- we thought we would do improvisational discussing. The slides are done. Mark, maybe you can comment a little bit on study from your perspective, ceo of thresholds, oversaw the overall activities but is well aware of the landscape of mental health services in chicago and beyond. >> mark ishaug: thank you. I have no slides. I'm going to be pretty brief so we can open it up to the full panel discussion of both presentations I have to say quickly that i've been introduced today as a doctor and a founder. I'm not smart enough to be a doctor. And i'm too young to be the founder of thresholds. But I love both of those introductions just a little bit about thresholds. To give it some context. We are 60 years old. I have 1,400 staff serving 15,000 patients. We have a budget of around -- it's a big, complex organization. One of the reasons i'm so excited about this kind of study at thresholds, and we have eight or nine research studies going on at any time at our center in chicago, is that 80% of our work is delivered in the community in vivo, where people live. So it's a very different model. We're not a traditional community mental health center where people come in for services. We know people with serious mental illness don't come in often. So for 60 years we have been reaching people where they are at, through aggressive, assertive community treatment. So right now, I don't know what time it is in chicago, but I probably have 700 workers on the streets at jails, at emergency rooms, at nursing homes, under the viaduct, at a coffee shop, delivering services where they're at. So what i'm really excited about with something like mhealth and focus and mobile technology is that we're already out in the community and so they have an extender to our staff that people can own themselves to improve their own health. It's really exciting for me. But as ceo, I think about what you all know is the aim. Every day i'm thinking are we meeting the triple aim in thresholds? Are we improving patient experience, improving population health and client outcomes and saving money or at least putting our limited resources in the right place? So what excites me about this is that we found that in terms of client satisfaction, quality -- am I not on? I'm so loud. I just figured I was on. Wow. That focus and wrap, the study that dror compared the mobile technology to, equivalent -- high satisfaction, which is really exciting, Improvements in recovery and health, health outcomes and quality of life indicators,

which is really fantastic. And the big question that I as the ceo think about every day are the costs. And what are the costs of doing this? Pcori, we know, does not focus on cost, cost effectiveness or looks at cost but that's what I have to figure out. How do we take this great idea that's proving in the field to work and how do we pay for it? So very quickly, I think of translating this wonderful research into practice is going to be really, really hard. But we have to figure out ways that we can make it happen. So I use the acronym or the -- win, we have to focus on the workforce investment, especially investment infrastructure. And what are the natural supports, really the unnatural supports, of the systems and the sectors that are going to help us win in this field. We have 15,000 patients, this wonderful study had 80 -- >> Dr. Dror ben-zeev: 160. >> mark ishaug: total, 80 in each arm. So how do I figure out a way if we're going to implement this, to reach 15,000 people and more? And what are the workforce challenges? We provide most evidence-based practices or evidence-informed practices. We know how important the mental health interventionist was to the success -- the results of this work. And how would we incorporate that kind of position into a system now that it doesn't have a code for mobile health interventionists the investments are huge and daunting. And we have to figure out ways to purchase smartphones. Flip phones are available. The technology we need, smartphones are not, at least in illinois. Data plans are expensive. There's no current reimbursement for that. How we figure out how to invest in an ehr system, which we're always investing in, and make sure that the information from the mobile apps is appropriately integrated into the ehr, not to mention into the clinical work flows of my staff. And just how we get public sector and the mco's and managed care companies in the state of I will toil move beyond a fee-for-service system to a value-based or capitated payment that allows us just do what we need to do to achieve the outcomes that we need to achieve. And finally, there's just the natural supports of integrating this into the technology and really coming up with a less complex funding system to support our good work. So I believe we can win this. And I believe we can achieve the triple aim. And very quickly, I do have a disclosure. This fabulous guy was my research director until he moved on to bigger and better stuff. >> Dr. Dror ben-zeev: different not bigger. Different. >> mark ishaug: so i'm very, very grateful that we got to be the place -- not where the

clients were the objects of the study but very much the subjects under his great vision. Thank you. >> penny: we're going to have our first reaction from sylvia trujillo. She's going to tell you a little about what she does regarding telehealth and give a little bit of reaction to the panel. >> sylvia trujillo: I think both presentations could not be more timely. I have a big announcement to make but i'll first tell you -- and it is historic. I think the people on the panel will be super delighted. So I work with a very large team within the ama as well as with external advisors to identify -- actually, can I get the clicker? If possible. To identify pathways to clinical integration of digital medicine, interventions that advance the quadruple aim, that they improve value, they improve outcomes and patient experience. They improve the overall population health disposition of the patient panels and more globally within the health system, and they also help the healthcare team, not just physicians but the whole healthcare team, have both less administrative burdens and more delight and joy in the practice of medicine. And my goal really is to work with this very sprawling, big, external and internal team to help focus on some key areas that are depicted up here. And the reason why it's important for me to emphasize it's a very large team is that if you can imagine, those sort of represent the big pieces. Medicine and they represent what clinicians ask when they're going to integrate a disruptive, potentially disruptive, type of technology or pathway or intervention. They want to know whether or not it's clinical efficacy. And the presentations today underscore I think very compelling examples of both the need and the merit of doing these types of comparative studies so that we are able to assess the sufficiency of the clinical evidence to support these types of interventions. We then have to, once there's demonstration of clinical efficacy, figure out a payment pathway both under fee-for-service and alternative payment models which may include accountable care organizations or capitated payments. We have to think about any liability that clinicians may face or the healthcare team and provider organizations we think a great deal, it turns out, about interoperability, integration with ehr's, and infrastructure capacity. And then, of course, we also focus on the training and workforce needs once you have such technologies in place. So both of the presentations talking both about mental health and determine logical conditions could not be more appropriate for what was just announced maybe about 30 minutes ago. And I was very anxious, because I am the ama's washington counsel and regulatory counsel on the medicare physician fee schedule with regard to any of the digital medicine modalities. And cms just after 4:00 issued the final rule for medicare, for cost year 2019, and announced some pretty historic decisions around digital medicine. So some of the examples, both examples today actually, could very much benefit in terms of a payment pathway under fee-for-service for these types of modalities.

So cms announced that they will be covering an additional set of chronic care management codes that include component for patient education and setup, which certainly I think is something you all looked at in terms of clinical integration. It also includes the technical component around the actual transmission of data and creating a meaningful set of transmissions to the treating clinicians and the last one relates to the actual service involved. They also, for the first time ever, have agreed to cover virtual check-ins which are very brief check-ins, which can be used in substance abuse treatment programs or in mental health programs not needing an immediate appointment or a -- or a full appointment. And then I think for the determine at a logical presentation that we just heard, cms has also for first time ever in this history going to cover evaluation of remote patient-recorded images or video for clinician evaluation and to make determinations of how that image or a video should be staged. Cms is also going to cover for the first time ever interprofessional consultations. So it is a big day. But in saying that, the road to this day is also complemented by historic federal legislation which will expand mental health and substance abuse telehealth, two-way audio-visual real time. Congress mandated coverage in the medicare program without restrictions on the geographic location or originating site, meaning it can happen anywhere at any time. And it is to be effective by july 2019. There are some additional major historic changes that happened this year legislatively but I want to emphasize that the type of research that we've just heard was really the linchpin of the advocacy that has been undertaken to secure this type of coverage, that the head-to-head comparison with in-person versus virtual, evaluating the staffing needs, the technology needs, are very important to demonstrate to commercial [indiscernible] and to clinician that these services are not only clinically valid but they can be integrated into their practice. I think the dermatological example is I think a compelling example where we have a disconnect between the specialties that are needed and where the patient is located and ease of access. So it's not simply geographic proximity but it may be mobility impairment and resources in order to see their clinician. And these challenges are only going to grow. So I think both of the presentations you have seen are just the tip of the iceberg of the type of research we will continue to need for other areas of clinical diagnostic interventions in order expand coverage beyond what we're looking at right now is a historic expansion for mental health services and substance abuse in the medicare program on tap for july 1 and augmented by these new services that cms says they will cover on january 1. And on the dermatological example, I think we're seeing that this is an area where imaging and images increasingly are going to be very important outside of radiology and even dermatology. We're going to start to see that digital virtual services will play a big role but the type of research you heard today which was rigorous, focused not only on the clinical outcomes but key issues like how does this integrate into work flow, how do you scale this up.

Also, for our purposes in making the case to both clinician and to payers, we need to demonstrate value. And we're not talking about cost effectiveness. It often means you can address a condition when it's lower acuity or can you address a condition in a lower cost side of care that is more accessible and affordable to the patient? And I think both of these examples demonstrated that and I think i'm at time. So i'm going to go ahead and hand this over. >> mark ishaug: can I ask quickly -- you said medicare three times. You meant to say and medicaid, right? >> sylvia trujillo: so the medicare physician fee schedule is medicare specific. So the rule that came out today would impact dual eligibles as well. So to the extent that medicare beneficiaries who are also eligible under medicaid have been limited in their access to telehealth and other services, this would lift those restrictions that they would face in those categories that I just identified. >> mark ishaug: that's wonderful. I'm just being facetious. 95% of our clients are medicaid, cut off at the knees. >> sylvia trujillo: but it will positively impact the dual eligibles. >> penny: just wanted to say, whoo-hoo! [applause] That was just fabulous. And you heard it first here. Right? This is really fantastic. Great news. Ok. Go ahead, jennifer. Beat that. [laughter] >> jennifer reck: all right. Thank you, penny for having me here today. I've got the clicker. I'm really happy to be here today as part of this panel to represent the perspective of state health policy makers. I'm a project director at nashp. Our work is guided by an academy of leaders in state health policy working across agencies and branches of state government. We have had the pleasure to work with PCORI on a number of engagement awards. For purposes of time today i'll just mention the work that we're currently doing. We convene a work group of state policy leaders to explore the policy implications of emerging research coming out of PCORI. That's our pcor work group. And the work that we do with that work group is informing the comments and the perspective that i'm sharing today. So first off, why are state policymakers interested in telehealth?

Before answering that, I just want to note that there is an extremely high level of interest in telehealth. It is a very pervasive topic. I feel like a very high percentage of things that pop up in my in-box are telehealth. It is the topic of the day. And I think part of the reason is because it's seen as a strategy with a lot of potential to improve access and to address workforce shortages that states struggle with. In terms of serving underserved populations, especially rural areas it can provide greater access to specialty providers, As we've seen today. And it also comes up a lot as a potential response to the opioid epidemic and in that context. Really quickly, just as an indicator of the level of interest and activity at the state level around telehealth, I just want to give you this quick snapshot of telehealth legislation at the state level. In the last session we saw 38 state laws enacted on a range of topics here from broadband access to cross-state licensing and reimbursement. I'm going to talk briefly about some barriers to the adoption of telehealth interventions related to payment. As one of our pcor work group members very aptly put it, it's a little risky to think of telemedicine as a solution to rural health access if you don't think about payment for it. And I also want to say that these brief remarks obviously are not a complete list of the barriers to the adoption of telehealth but were the things that rose to the top and were the points of discussion shared with our work group. So the first payment-related barrier has to do with lack of alignment across payers in terms of telehealth reimbursement. In her publication, Dr. Armstrong noted that teledermatology has not been adopted as widely as might be expected. The state health policy makers that we work with note the same thing. There's medicaid coverage for telehealth but they just don't see it used as pervasively or widely as they might imagine or hope it might be. And when we ask them why this might be happening, they pointed to the patchwork of varying coverage policies across payers as a factor that could be inhibiting the use of telehealth. The policy response to that would be parody laws requiring coverage of telehealth across payers, state lawmakers in 35 states and washington dc have passed parody laws to require private insurers to cover telehealth. So we're more than half way there but not all the way there yet. The next payment-related barrier i'll discuss is facility fees. Facility fees are fee that hospitals or hospital-owned clinics often charge to cover their overhead costs in addition to charges for the actual services that they are provided, that they provide. Currently only 32 state medicaid programs allow payment of a facility fee or as it might be called a transmission fee whether telehealth is used. So we can imagine how that could be a barrier for the adoption of telehealth if it leads to a decline in revenues for clinics.

What additional information might be needed to change practice? Health policymakers are interested in clinical outcomes related to telehealth but because they are policymakers, they also have to ask questions like are telehealth interventions cost effective. They might look at a study from the angle of wondering if the intervention replaces existing in-person care or creates a new type of care and, therefore, a new cost. And also need to understand how telehealth fits into existing payment models. So, you know, the economic impact of a telehealth intervention might be beyond the scope of a given study. It's basically beyond the scope of what PCORI is authorized to do. But it's very vital and important information for policymakers. So I would just say that even if it's not something that a study could speak to directly, even if there is indirect information, it could still be of use to policymakers. So it doesn't apply to the studies we've heard about today but, for example, if a study shows an increase in patient activation as an outcome, if there's any related research that might show a connection between increased patient activation and better clinical outcomes and/or decreased cost, that type of information would be really important and meaningful context for policymakers. And I also just want to close by saying that I have heard and i'm sympathetic to the perspective of some researchers feeling frustrated that all policymakers care about are costs. I would just like to say on behalf of the folks that I work with in our work group that that's fortunately really not the case but unfortunately it is part of their reality in what they do. States that are responsible in most cases for having balanced budgets and having to make difficult decisions and that's part of the reality of what they do. So i'll stop now and leave time for a discussion. >> penny: I have a lot of questions but I wanted to make sure the audience has questions, too. I just wanted to raise a couple of them. I think one of the things that came up this morning -- I don't know how many of you saw Dr. Dorsey talk about the telemedicine intervention that he had with for parkinson's disease. But he talked about the digital divide. And I know that Dr. Armstrong, your population, you had a large hispanic population. And I think when you were implementing it, it was quite a struggle to actually get acceptance to the technology. Can you talk a little bit more about that and what that means in order for implementation into practice? >> Dr. April armstrong: sure. So, I work at university of southern california which is in east l.a., different from ucla which is in west l.a. and the predominant patient population that we serve in our radius are hispanic. In fact, when I got there, be my research team, we didn't have -- we had bilingual research coordinators but did not have bicultural research coordinators. I essentially made some staff changes because I realized that to be responsive to the needs of the community that we're serving, it's really important to have not only bilingual

but also bicultural staff members who really understand their needs. A few things, some of them, while they had access to internet, it was not at their homes. So some of them had to go to their relative's homes to access. They wanted to be in the study. And I said, you know, you have to have access to internet and imaging device. So they actually got their family involved and had to access the internet that way. We realized that it's very important when we do research to ensure that our staff really understands the needs of the population. So everything that we do for the study translated at the platform itself is entirely translated also into spanish. So that we could do this. And one thing is that our providers may not speak spanish, so for part of the study, then we had coordinators especially making sure that all of those -- essentially making sure all of those were translated. So I think it is to say that it's very important that we really take a patient-centric approach. Even if that involves difficult choices like staff selections I had to make staff changes that needed to really be responsive to our patients. >> penny: Dr. Ben-zeev, did you have some comments about the digital divide? I know it's a very sort of difficult population that you're working with. You did talk about access to cell phones was pretty common. Are there any sort of cultural issues that came up with the different populations? populations you work with? >> Dr. Dror ben-zeev: so, again, if anything, i'm a little bit more optimistic. I'm used to having such disparities between population that i'm interested in on every other level that in the area of the digital divide. I'm really quite optimistic in seeing the patterns i'm seeing. I think the technologies broadly are being accessed. And, of course, there are -- I think there are certain regions we found in the context of the chicago study and program of research involves mobile health interventions deployed now up to 11 states where we've done this. So certainly we see certain pockets where access to federal lifeline assistance program is poor. Where we see that in washington state, for example, it's much easier to get access to a smartphone as part of these programs rather than other areas which, of course, opens up opportunities. But if anything, my message is an optimistic one. I think that's the overall trend that i'm seeing. >> penny: ok. Go ahead. >> mark ishaug: to add, it's the economic divide. It's to the question I asked sylvia. 90% of our clients have social security disability income or maybe have $700, $800 a month. If they're really lucky and if they can find a $400 apartment and then have $400 for food and transportation, there is no money for the monthly data plan.

There is no money to replace the mobile phone when it is lost. And it is lost. So we have to figure out not just the cultural divide but the class divide that is limiting access to many of the future modalities that we know work and will change the course of people's lives. >> penny: one more question before I hand it off to the audience. It's a long one. If you could make your answers short for this. I know both Dr. Armstrong and Dr. Ben-zeev, your systems existed outside of the electronic health record. So that potentially could pose a barrier for adoption of both of your systems. Can you talk a little bit about kind of any changes that you see, movement in this area, or potentially what needs to be done to address that? >> Dr. Dror ben-zeev: I have a thought rather than a solution. I think the question of interoperability and the ability to connect with electronic records is a crucial one. And I think that many stakeholders have incentives to keep things the way that they are and have electronic health records that are siloed by nature and that require a fair amounts of resources to make changes even in the context. Same healthcare system. So along with the theme of patient-centered outcomes and patient advocacy, I think the pressure needs to come from outside to create conditions where it's mandated to have the flow of certain forms of data that are likely going to be collected whether through one particular mobile platform or another, things like self-reported patient data, things like accelemetry, geo location. Devices have a host of sensors that cutting across health conditions will need to flow into electronic health records. So I find that to probably be the number one barrier to meaningful integration even within our healthcare system where we might have pockets it's not directly fee-for-service. There might be contractual agreements and other ways of reimbursement that we don't necessarily need toe bill for the particular deployment of a device or mobile health specialist. But still, when you need to provide an external laptop that works outside of an electronic health record to a system in order for a clinician to access a dashboard that creates an enormous barrier to meaningful use. >> Dr. April armstrong: I think the technology is there to integrate. I think the bigger issue is getting stakeholders to talk together. There's a lot of issues around proprietariness of each technology. So I think it's thrill not a technological issue even though it seems like a technological issue. It's really about playing together in the sandbox. I agree that perhaps outside pressure is needed. Another way to think about it is that for the larger ehr companies that are out there, within that system, how do we buff up the telehealth arm of that? >> penny: thanks.

I'm going to open it up for questions now. I'm hoping since this is the end of the day, if people could stay a little bit later, we can stay a little bit later, too. >> hi. [indiscernible] university of rochester. Quick question for the panel about machine learning. Obviously when clinicians are limited, budgets are limited, it can be hard to reach all of the patients in the time that you need to reach them. For something a ynchronous like dermatology photos or interactions with conversation-based app, can we use machine learning to help screen, you know, the least complicated cases, let's say, and what are the clinical and legal considerations that we should take into account if we want to do something like that? >> penny: go ahead. >> sylvia trujillo: so the released -- you're talking about one of the systems, a subsystem within artificial intelligence. So, I think there are tremendous promises related to not just machine learning but other ai systems. You mentioned actually not just machine learning but computer vision which are two systems married together. We are delving into what are quite potentially very significant opportunities to scale using these types of systems. I'm also an fda lawyer. And I am up to my neck in the fda's work around regulation oversight of software as a medical device. That is challenging even without machine learning. And just to be clear for everyone here, what people are really talking about when they talk about machine learning is that -- I think you're talking about continuously learning systems potentially that are allowed to change themselves. They're called learner algorithms. As they attain more information, they take that information in. And then they modify themselves. And then they have a different output. The short answer to that is it sounds great. The only machine learning that's ready for primetime, and I am not unique in this regard -- every responsible researchers from harvard partners to duke to stanford and from major manufacturers and technology companies will tell you that unlocked continuous learning systems that use machine learning, that's not responsible. You cannot turn those loose on clinical patient population, not know what it's doing, and basically have some pretty pernicious outcomes. Now, having said that, you can use machine learning systems for some tremendously powerful algorithms that are locked and then tested, validated, and then used for scale for things like interactions through a very predictable set -- when I say very predictable, they're not that predictable because the machine learning figured it out and we didn't but through a machine-learning validated set of interactions or in sequencing they find variants for cancer that then have diagnostics that can find things that we couldn't find. So there's some interesting questions but if you can't explain why the algorithm is doing

what it's doing, you have to convince a physician if there's been little to no validation or research to demonstrate that in a head-to-head way it's as effective or appropriate as the average clinician. Then you're going to have both liability problem, coverage problem, and an adoption problem. >> penny: there's a question in the middle -- >> if I could just add one comment. >> penny: yeah. >> Dr. April armstrong: so maybe we can think about machine learning as a part of augmented intelligence. That is the clinician is still at the helm and using tools of artificial intelligence and machine learning to have augmented intelligence of the clinician. I do think, like sylvia said, when left by itself, unfettered, there could be problems. I think there are a few technology already that's really taking off. In dermatology image capturing -- how many of you use online banking? That take the check -- so I use bank of america, right? You put the check there and now I don't even have to -- it says move closer, move and boom, it snaps the image. So it's getting better and better. So a lot of these are being adopted into telehealth and telederm where we are optimizing the image capturing from the patient's end. When patients take a picture, what's hard is how do you take a good, quality picture? So a lot of that is incorporated with regards to just the program itself, try to figure out optimal way of capturing the image to its diagnosis. There are a number of apps -- not apps. There are programs that are out there that have shown, for example, a detection of melanoma, some of them are scary, is the same between some of the melanoma detection programs versus a dermatologist. There were no statistical differences. So that scares most dermatologists. But I think it's actually good. It's augmented intelligence. So I think if a tool that's going to perform better than me, I would want to go to that tool. And there's obviously a lot of variabilities I think there's promise. It needs to be augmented intelligence to complement what we do. And i'm glad to see there are a lot of evolving research in the area. >> sylvia trujillo: : if I may, I really have to respond. I really do. I spend a tremendous amount of time on ai issues. So, the image does call it august meant intelligence, just for the record, as do a lot of, I think, very responsible developers and manufacturers and technology companies. The fda has done an authorization of a program called idxdr which provides for screening of diabetic retinopathy. Tremendous. This is great. And the reason why is because we don't have enough referrals.

So I think the number is ridiculous, like 60,000 people go blind because they can't have access to screening. This software does not augment physician decision making. It is in lieu of physician decision making. And what's fascinating about it is that when the fda actually approved it, it's for deployment in the primary care physician's office and it has been evaluated to make sure not only that it's as good as or better than an ophthalmologist. The fda, if you look at the memorandum, which I urge you all to do with the 510k de novo evaluation, evaluated the conditions under which it should be deployed. So the skill set of the staff that would be setting up -- excuse me --setting up the machine and evaluating the likely unexpected adverse consequences that could flow if it was not properly deployed. I bring that up not to disparage or -- healthcare ai but to underscore that that particular system, which is called autonomous ai is a locked system. So I emphasize that it's very important when people talk about machine learning and deep learning they make clear that they are talking about a locked system being deployed into healthcare. This is not banking. If my check doesn't show up and my deposit doesn't show up, it's a major inconvenience. If I have more than a nominal amount of diabetic retinopathy and the machine says I have less than, that could have very serious consequences for my care. So this is a really important area. In fact, it's a great area for PCORI and is a whole session that should be dedicated to it. >> penny: I apologize. We are at time. But I will take one more question. But before I take that one more question, I just wanted to encourage people. This morning we had an absolutely stupendous facebook live event that dror ben-zeev and mark ishaug were at. It's fantastic. It's on our website. So if you're interested in that, very dynamic and great way to see some of the work that they are doing. Yes, one more -- oh, I guess i'll take two more questions because i've been looking here. Go ahead. >> my name is katina lindsey. I am a social worker. I am a professor alabama state university. And one of the things that we're doing is a colleague and I are working in regarding to telehealth in a rural area called the bible belt -- the black belt, i'm sorry, of alabama, where there are socioeconomic disparities in that particular area. But my question is in regarding the gaps of mental health services and the consideration or if you have had clinical social workers who also -- can also help in regarding the psychosocial issues as it relates to the population that you're serving with

mental health. We are beginning to do the beginning of the work as it relates to bridging some of the gap. But like I said, we're at the beginning of that. And one of the things I wanted to know is whether or not you've done that. >> mark ishaug: great question. So we have psychiatry and apns and now integrated co-located primary healthcare because people with mental illnesses die earlier than everybody else in part because they don't have access basic primary care. I would say 90% of our work -- dror, help me here, you know -- is really meeting the social determinants of people's mental health and behavioral healthcare. So our staff are focused almost exclusively on ensuring that people have basic income, insuring that people have access to food, insuring that people have access to shelter. So, you know, 80% of health outcomes are related to social determinants. And that's what we focus on. I do not believe you can do community behavioral health with poor people anywhere unless your focus is principally on meeting the social determinants of their health and behavioral health. >> penny: everyone, feel free to leave if you want. We are at the end of the session. Can the panel stay for a little bit longer? Are you ok? You need to go? Yeah. So one more question. I know you've been waiting patiently. >> thanks. My name's julianna. I'm at johns hopkins in the school of education. I have a question about the platforms for telehealth or especially mhealth and app production. I wonder if your experience in creating the apps or your creation at the policy level leads you to be hopeful or not about possibly creating a uniform initial launch space when I open up my phone for apps related to mental health or other health areas for apps so that I can open up my phone and no matter where i'm going to get to, there is a basic platform opening that then drives me to a module, modularized, app that's specific to me. So from a design perspective, i've got mhealth-related apps that I would like to create, but I would like to not reinvent a wheel and create a design interface that's different from the one you created from the one you've created, etc. But have an easy entree point that is universal from a software design perspective. >> Dr. Dror ben-zeev: I think the vision of having a platform that hosts multiple apps whether it's complete, stand-alone apps or modular elements like you're describing is really the focus of industry right now. So there are multiple companies that, for various reasons I won't name out loud but you can find by googling mental health platform.

And they are creating many of these. Apps that focus -- that focus on things like relaxation strategies through management of complex disorders, comorbid with diabetes and other things, developed to sit on these platforms and I think the platform rather than the specific apps that are being hosted is what they are intending to market and are already trying to do that. Some of them start with a v. Some of them start with a p. >> penny: thanks very much to our fantastic panel. and thank you for coming this evening. Page