So it’s not often that a
meeting, a workshop runs so well that we’re actually
ahead of schedule, but we are. So thank you, and sorry to
those of you who are wandering for another 15 minutes
while we wait to reconvene, because we’re
reconvening now because we recognize this
has been a long day. We want to hopefully
amuse you, hopefully, you know, have more discussion but
then give you a breather because we’re going to
come back in the morning and do this all over again. So I’m David
Chambers. I’m here at the NCI, Deputy Director for
Implementation Science. I get to introduce myself
first, so I’m going to give you a little bit of an intro
around implementation science. This session, as you can see,
is around Health Care Delivery: Approaches to Development,
Implementation, and Evaluation. It should have a
part one at the end of it because tomorrow
morning we’re going to get into even more of this. But to whet your appetite we’re
going to have a few talks, a few discussions. So
I’ll kick things off, turn things over to Juan again, who deserves gold stars
for being willing to address and play multiple
roles here from the CDC, and then Joy Larsen Haidle,
who is the past president of the National
Society of Genetic Counselors, focusing on workforce, on
capacity, and her experience in thinking about
how do we actually scale up different services. And then we’ll have two
discussants: Muin Khoury, who many folks know
is the founding director of CDC’s Office of
Public Health Genomics, and then David
Ransohoff from UMC, all of whom have wonderful bios
that I’m not going to take as much time to read, but we’re grateful for
their comments in advance and then grateful for the audience
discussion that will follow. So there’s that. Okay. So, what I wanted to do, and it’s been
really helpful listening to such tremendous
expertise over the day so far, is in effect say
a lot of the things that you’ve been
talking about already but put a slightly
different spin on it, because I think many of
the implementation challenges that come pretty
much in every single talk are the things that we
focus on on the scientific end as we’re trying to develop and
then deliver the knowledge base around implementation. And so hopefully
you’ll take this ride with me as mood lighting
prevails, and then hopefully we can think together
about what are the ways in which some of these decisions
that are often confronting us in real time are
things that we can plan for, things that we can learn from,
and ways that we can build the knowledge base
toward a better optimization of what we know to
be high-quality care. So what I’m briefly
going to go through is just a couple thoughts about
implementation science overall, at least some thoughts
about how it might relate to what we’re
talking about today, give you a bit of a sense
of some of the activities and resources that
we’re in the midst of in the
implementation science world, push a little bit around
a couple of the assumptions that I think too
often limit our thinking about how to make
progress in terms of the value of
biomedical research, and then end with
some discussion questions that other folks
will take on as well. So just by show of hands — this is the audience
participation portion of this — how many have seen this slide?
Okay. A few but not many. What this slide is
showing, this pulls together actually 17 years ago,
and this is a key time point as you’ll see in a
moment, a study that was saying let’s assume that the
end product of our research is the publication, it’s that wonderful
high-impact publication. What happens if our
goal is to try and see the findings
implemented within practice? On the left side what
you see are all of the ways in which we’re losing
that valuable information. Maybe there’s a negative result that made it hard
to get published. Maybe it never makes its way
into databases, into reviews, into guidelines and textbooks, and maybe it doesn’t get
its way toward implementation. On the right side it basically
just tries to talk about on average how
long does this take. The reason why this
slide is important is because it walks through the
rationale behind a factoid that often gets mentioned without consideration
of where it came from. Here’s the factoid. And I’m guessing that more
of you have seen this mentioned many times as we
talk about the challenge of getting our
research into practice, that it takes 17
years to turn 14 percent of original research to
the benefit of patient care. This is the reason why many of
us and many of you in the room are turning your attention
more to these challenges of how do we
take these findings, how do we take
these effective tests, these effective
diagnostic procedures, these effective treatments
and preventive strategies and turn them
into standard care. This is the reason
why we can’t just assume that the endpoint of our
science is the publication. And this is one good reason why
we have this kind of challenge, because we’re often at this — with this mentality I’m
guessing you can fill in the rest of the sentence,
1989, Field of Dreams, the voice above Kevin Costner’s
head saying if you build it — ALL: They will come. DAVID C:
They will come, right, but they
don’t in many cases, right? They do not come. You can see this lovely
baseball diamond right here. Okay. It’s February, so
maybe that’s one reason, although spring
training is starting, but there are many, many reasons why people don’t
take the intervention, the innovations that we create. Maybe it’s a place where
baseball is not all that common. Maybe it’s just
other preferences. We really need to think
beyond the science push, which we’ve been
very good at to a point. But implementation science
is really trying to make sure that we don’t end
with things on shelves, that we don’t end with
these wonderful innovations that nobody
actually is able to use. So when we think
about genetic testing — and here’s just a
way in which sometimes these wonderful tests break down if we fail to think
about different stages related to implementation. So even if genetic
testing gave us the ability to identify optimal treatment
for a specific illness, reduce risk for
health populations, as we start to look at
what happens as that tries to make its way to
become standard care, let’s assume of all
those who are insured and all those
insurance companies, say the scenario
is that half of them are choosing to
provide this genetic testing, choosing to reimburse for it. Let’s assume that of the systems that are working
with those insurers, they’re only the half of
the systems in the world. By the way, this is
actually an optimistic look. So sorry to say that sometimes the actual is
even less than this. But let’s say half
of the health systems are willing to go that next step and think about how
do we train clinicians to prescribe testing,
how do we train folks to be aware in
trying to identify when testing would be indicated. Now let’s say they go
that next step, right, and those
clinicians are trained. Let’s say only half
of those clinicians, because we know the
continuing medical education and other continuing
education has its flaws. Let’s say half of those
clinicians go that next step and say, “We’re now
going to take this training and we’re going to apply it and we’re going to
start delivering the tests.” And let’s say, you
know, again a busy group, we only get to half of those
patients within, you know, who could potentially benefit who are seen by those
clinicians to get the tests. What this basically shows, and even the scenario we have
perfect access, perfect testing, perfect follow-up, no
problems in the system, just as you see
going through those steps we’re down to just 6 percent
of this wonderful test — the benefit of
this wonderful test that we thought was
going to change the game for the given illness
or the health problem. Now, when folks
have studied this, and this comes from the
[inaudible] framework — Russ Glasgow, who was
at the NCI before me, with others had
developed this — what we basically
come to is it’s not just the effectiveness
of our interventions. We’ve made a lot of
progress in moving beyond the pristine efficacy trials as we’ve seen some of the
messiness of the real world, but it’s not just effectiveness. We need to think about
reach, we need to think about the adoption of our
different innovations. We need to think
about implementation. And we need to
think about the end as maintenance
or sustainability. Can we sustain
the things in care that we know to be effective? When Russ had looked at
this kind of a scenario for certain health conditions, 6
percent would have been great. It was like .03 when it was looking at the
community penetration of some of our
evidence-based interventions. So we need to do better,
and we start by focusing on these not just as
decisions that we need to make but maybe questions that we need
to ask in a research context. So the key terms
that we deal with are both implementation
science, which is the name of our team here at NCI, and dissemination and
implementation research, which is how we
framed this for a while in a number of
our NIH activities. So implementation
science, as you can see, is really about
the study of methods to try and integrate
research findings, evidence, effective interventions, into
health care policy and practice. We define health
care pretty broadly, so we have clinical as well
as community settings anywhere where someone’s health
could potentially be benefited. We also talk about dissemination
and implementation research. And that’s because on
the dissemination side, remember our classical efforts
of disseminating evidence through the publication
don’t necessarily meet all those who we think might
benefit from the research. And dissemination research is
really about changing the game, making it better,
making it more likely that evidence gets to those
who could benefit from it and helps them in a way so that
they can apply that evidence to better health
and better health care. And then implementation
research is really about how do we
develop these strategies to adopt and integrate a whole
range of different interventions in the specific settings
where they can best be used. So as we think
about Lynch Syndrome, again this isn’t
saying anything different from what I’ve been hearing from
a lot of the talks thus far. It’s not just
about identifying that we have high-risk individuals. Would we be able
to capture them? But it really is on the
implementation science side of trying to
identify those strategies that are going to help
us with the identification of Lynch Syndrome
within the broader population of those with colorectal cancer and, as was
mentioned even beyond that, it’s thinking about what
are the right strategies to improve scale-up among
different family members. What are the right
ways to not assume that the endpoint was
that we identify people, but what do we
do in the long term in terms of implementing ongoing
screening, ongoing monitoring, treatment, et cetera? And importantly, as was
mentioned several times, it’s about the
workforce capacity, it’s about the training needs. How do we figure out
how best to staff up our broader community of providers,
of families, et cetera, of all those who might be
able to make this a reality? And these aren’t
trivial considerations, but they’re the things that
we want to develop strategies on the implementation side
to hope for or to advance. When we think just briefly about
precision medicine as a whole, it becomes really
interesting because we have this
opportunity to see — and I think it was mentioned
certainly any number of times of how fast our
evidence base is changing. We have an opportunity
to really think about what does it mean to try
and create a health care system where clinical
practice can incorporate all of these ongoing findings. There’s any number of
iterations of guidelines. There’s any number of new
studies that are coming out, and we haven’t really thought
about the dynamism here. And we haven’t really
thought about the opportunity to implement a
health care system that’s able to take advantage of all
of these different findings. How do we think about
implementing evidence that isn’t static but
that’s evolving over time, and how does the
system that we develop actually expect that
there are going to be changes on the horizon, and the decisions that we
have to make may need to be remade again and
again and again, and not assuming that
the latest and greatest is the last that
we will ever expect? Again, we really need to be
thinking about the workforce, about the financing, ways
in which this kind of care can be provided. It’s not trivial, of course,
as all of you have known and you’ve said much
more eloquently than I am. It’s incredibly important
not to lose sight of what it is that needs to be implemented. Throughout the course of
the day I would say that we’ve heard a whole
range of different things that are part of
this intervention package that we need to think about if
we’re trying to optimize care. Is it the testing on one level? Is it ways in which we need
to educate various stakeholders so that we can pave
the way for more knowledge and ideally more
awareness of sort of what’s available to
identify and then improve the lives of folks
with Lynch Syndrome and other hereditary cancers? We need to be thinking not just
about that initial test but about monitoring and follow-up.
What is preventive care? What is treatment? Is our “what”
in this case all of the above? And if it is, that of course
is a pretty significant task. But starting with
what helps us then to say how do we get that into place, and it’s very important
not to jump over what. So when we think about
studying implementation, this comes from
2009, a paper that Enola Proctor and
others had pulled together. Basically what we’re
trying to do in implementation is contrast or go that next
step beyond just in the what, but it is important
to start with the what, the quality improvement efforts or the empirically
supported treatments, evidence-based practices
that you see on the left, and the cascade that gets
us to better health outcomes at a population level. What we are
used to dealing with in a lot of our
research, certainly interventions research,
is the left and the right. What do we do when there’s a
specific patient or a person in front of us and
we’re trying to maximize their health outcome? Most of our trials,
or many of our trials, are really focusing on
that sort of outer end, right? The what is it that we deliver if all of the
circumstances are in place so that that person
has access and can benefit from that
particular intervention, and then what are
the health outcomes that they’re likely to generate,
improvement in health status, reduction in symptoms, improvement in
satisfaction and functioning. With implementation,
we’re really thinking about not just the what
but the how, right? How do we get
that care delivered to everyone who
can benefit from it? And we’ve identified a set —
and you’ll see it right here — a set of specific outcomes
related to implementation that says have
we done a good job, have we developed
those effective strategies that are going to enable
these evidence-based practices, these tests, the follow-up,
et cetera to be delivered? The idea through
implementation science is that by focusing on the
how, by focusing on the implementation strategies, and by succeeding in getting
higher rates of adoption, higher rates of implementation,
higher rates of sustainability, we’re going to
improve the service systems which are caring for
patients, families, et cetera, and then at a knock-on benefit if we can improve the
service systems, we’re going to improve health outcomes
not one person at a time but at a population level. So if we’re successful,
it’s moving beyond the usual to really think about the core
of implementation research, as how are these
strategies getting us to effective implementation, but then if we’re successful,
seeing that it cascades outward. So another type of
cascade, really thinking about what do we do at that system
level, at that population level so that we’re not just
relying on what was achievable through our earlier research, but it truly becomes
standard care for all. Over the course of years, and folks in the room have
been responsible for this, we’ve seen further
sort of refinements to how we think about
translational research. Fifteen years-ish ago
we were very much focused on these two big
building blocks of translation. Translation one from basic
to clinical and then everything else was trying to say how do we
take this clinical application and get it into practice. Over time — and you’ll see Muin
Khoury’s and others’ graphic in the bottom right —
there’s been a recognition of additional
translational hurdles. Also over time I think
we’ve been a little bit better at not seeing this as
strictly a linear process but saying what can we
learn at each stage of research that’s going to help
us with that next step. Can we be thinking — and
this would be the challenge hopefully for others
to take on as well — can we be thinking more
and more about implementation earlier and
earlier in the process? So the good news out
of a lot of the efforts of people who have
been engaged in this area for the last 10,
20, 30, 40 years is that we have a lot of models. Now George Box is attributed — this quote is attributed to him, that essentially
all models are wrong because they’re
simplistic, right. They’re simplistic
representations of a much more complicated
universe, but some are useful. In the implementation science
space we have at least — this is, you know, and
counting — 61 and then some conceptual models that we think are probably wrong
but definitely useful. So we did a review in
2002 and just started out with trying to capture all of the
different conceptual frameworks that have been
developed to try and say what goes wrong
and what goes right if we’re taking evidence and
integrating it within practice. And then we tried to say is that more about
information dissemination? Is it more about trying to
take particular interventions and get them implemented? And we tried within this
paper to give some guidance as to how does
one choose a model. I’m going to very quickly
mention a couple of them. One of them is an
oldie but a goody. Everett Roger’s
Diffusion of Innovations, which came out of agriculture,
basically started out by looking at farming practices and how farmers taught one
another the different ways in which they could optimize I guess it’s
productivity with seed corn. And it was very quickly
applied to health intervention. What’s very nice about
this is that it focuses on, as we said before, the what,
what are the characteristics of the intervention
and how does that make it more easy or harder
for it to be implemented, and then it recognizes that
there’s a cascade of decisions that need to be made, decisions
to adopt the intervention, decisions that will result in
more effective implementation of that intervention
within the system of care, and then importantly what are
the outcomes at different levels that are going to be
generated from that. It also importantly focused on whether we’re
talking about a hospital, we’re talking about a clinic,
we’re talking about a community. There are
characteristics of that setting that are going to be
very important in determining whether we’re
successful in this process. And importantly
as well that there’s this whole broader environment
that we have to keep in mind. Context being important
came up again and again today. And just to let you
know, from the beginning, the theorists, the folks who are
thinking about implementation, have that front and center
because it is so very important. An even more
complicated slide that you probably won’t
be able to read — at least I
probably can’t from here, but you may not from the back — was a framework that Greg
Aarons and others developed in terms of
mental health services, where they talked
about four major stages that if people are
thinking about implementation, they need to go through
and they need to plan for. From exploring the idea
that a particular intervention should be implemented
through to preparation for it, the act of implementation stage, and then the sustainment phase. And again, very
important that we think not just about
initial implementation but about sustainment over time. And again, these
frameworks are helpful because they point to
many of the same concerns that have been raised
over the course of the day and likely into tomorrow. So just to let you
know, this is our chance. If you want to ask and
answer these questions, have we got the
funding announcements for you. Across NIH we have
program announcements that are specifically devoted to dissemination
implementation research studies that most of our institutes
and centers at the NIH are involved in. And genomic medicine, methods
development system science, a whole range of
different priority areas are there for the taking. And I think with pretty
much each and every talk I heard
questions, I heard comments that very much fit
what we’re looking for. Namely, how can we think
more about the sustainability in this case of
different genetic testings, follow-up cascade
screening approaches? How do we think about how some
of these tests are adapting or evolving over time?
More on that in a second. Not just thinking about
interventions one at a time, but what is our
evidence-based system of care that gets someone from
the starting lines, you know, the starting point to a long-term evidence-based
care being provided? What are the ways in
which we can learn more from innovative
dissemination strategies? And how do we not just
think about implementation at that local level but exactly what many of
you have been talking about, how do we scale up
effective practices across health plan systems,
networks, states, nations? Importantly as
well we want people to not just be thinking about
bringing more and more in, but how do we
take out those things that are no longer effective? How do we
de-implement, exnovate? There are other
terms that have been used. But how do we
think about all sides of this implementation picture? We have a number
of growing resources. We have training programs.
We have hundreds of trainees — some of them are in the room — who have said this is the type
of science that I want to do, and it would be
great to figure out ways to foster those
partnerships so that again many of the questions that
have come up can have answers, can have solid expertise
around them. We have journals. We have a whole set
of measurement tools. We have annual D&I
science conferences. It would be great to think
more about how do we build the space of implementation
science for Lynch Syndrome, for hereditary cancers, et cetera
more than we have already. So I’m just going
to finish up with a couple of maybe challenges moving beyond the
traditional assumptions that I would love to
see some consideration so that we don’t continue
to reapply some of the things that haven’t worked in the past. Traditional
assumptions as I see it are the evidence-based practices
that interventions are static. Once we’ve captured
them, we’ve manualized them, they’re done, we’re moving on. The systems, we
know the systems. We know the systems
within which we work. Once we know them,
we’ll always know them. But implementation
goes sort of one practice, one test at a time. That consumers and
patients are homogeneous and that choosing to not
implement is irrational. I’ll say that most of
the people in the room, and myself as well, would not
necessarily assume these things, but I will tell you
that most of the research that we have seen, at least even in the implementation
science space, has suggested that these assumptions
are still pretty true. We’ve got our
manualized intervention and we know our systems so we’re going to go forward
with this one particular test or this intervention. So I think we need
to ask that question, particularly as we’re
thinking about scale-up, is might it be
rational not to implement a given test or a given
cascade screening, et cetera within a certain population
or a certain setting? Does it fit or does it not fit? Again, I think we
should relax the assumption to assume that everyone,
that every circumstance is appropriate for
every single intervention that we have created or
innovation that we’ve created. It’s really
important not to assume that we’ve got a fit but actually
to try and assess that fit. Does anyone quickly know
who this is a picture of or what this is a picture of? MALE: The Beatles.
DAVID C: The Beatles. Thank you. This is the Beatles, if you
just look at their hairdos, their haircuts
from 1963 to 1970. The point that I
want to make here is that often when we’re
thinking about sustainability, we’re thinking about
things remaining exactly in the same state
as it was initially. I think what people who follow
the Beatles, like myself, would say that evolution
in this case was good, that the music evolved,
that the hairstyles evolved, that they adjusted
to their circumstances, and they learned and
they got better over time. We don’t
necessarily ascribe that to a lot of our
health interventions, and we may be limited by that. Last one. On the left
side, those of you in the front should be able to
see, it is Spock, right? So this is Spock when
he’s happy, when he’s lonely, when he’s jealous, when
he’s sad, when he’s angry, when he’s joyous,
envious, et cetera, right? On the right side, anyone know
who that is? Johnny Depp, right? So Johnny Depp, right? So
this is the contrast, right? So the Spock idea is that no
matter what the circumstance is, you have pretty much
the same intervention. In the other case what you
see is ongoing adaptation. You see for every
different circumstance there’s a slightly different
way in which that performance, that intervention is delivered. We need to be thinking about, rather than saying that
it’s either one or the other, we need to take onboard
much more understanding of what can be adapted,
what should be adapted, what needs to stay the same. And so this fidelity versus
adaption, which plagues I think at least our field at
times, needs to be moved beyond and really think about
what are optimal adaptations that need to happen. So you’ll see these
questions as we go through the other presentations,
but just to sort of prime you, these are the
kind of things that we would love to
engage you in discussing, and we would love to
brainstorm with you. How can we take what
we’ve already learned in implementation
science and move it forward, and how can this be
a really nice example to learn as much
as we possibly can? So that’s my pitch. Thank you
for strolling with me on it. And now I will turn things over
to Juan. So thanks very much. [Applause] JUAN: Afternoon. So I’ll try
to — I’ll ask you to recall a slide Lisa Richardson
presented this morning. I know that was at like 8:30
this morning so my apologies. But she showed a
little timeline of kind of the genomics program that
the CDC has funded over time. And so I’m going to
talk a little bit about one of the programs
we’re currently funding and one of the things
we’re currently doing or some of our states that
we fund are currently doing. So we have a
cooperative agreement where we fund state health departments
in five different states — Michigan, Oregon, Connecticut,
Utah, and Colorado — to implement strategies
in cancer genomics so that we can kind of do exactly
what David was talking about. How do we actually
put these guidelines and these
recommendations into practice and so then that we
look at our state programs as kind of
laboratories for implementation so that we can try to figure out
what works, what doesn’t work, and then how that could work in other settings
or other situations. So our states do a
lot of different things. They do public and
provider education. They do sort of
basic surveillance work. But one thing they do
have in common across all five of our programs is
that they all try to use state cancer registries in ways that go kind of
beyond basic surveillance or basic ways of
assessing cancer burden. And that mostly came up
because all of our programs are trying to
really answer the question how do we identify
high-risk individuals on a population-based level. And so, you know, they’re not
necessarily working in clinics, they’re not necessarily kind of
providing direct patient care, as Deb kind of mentioned
during her talk today, but they are trying to kind
of work at a population level to identify
high-risk individuals. And that’s kind of where work with our state
cancer registry started. And so I’m going to give
a very brief introduction to cancer registries
and cancer registration. I’m going to do a
massive disservice to both the
programs NCI and CDC fund, but just in case there’s
someone who doesn’t have a very strong kind of background or even a general
background in cancer registries just to kind of get everyone
kind of general lingo terms. So there are two federally
funded cancer systems. At the CDC the program is called the National Program of
Cancer Registries, or NPCR. And at NCI there is a Surveillance, Epidemiology, and
End Results Program, or SEER. And together these two programs
collect data on cancer cases for the entire nation and
several U.S. territories. So the state enables our
agencies and the public, since most of this
data is publicly available, to monitor the cancer burden, evaluate the
success of our programs, and identify needs for cancer
prevention and control efforts. Now our registries, they
collect different information, but there are
minimum data standards that our registries collect
across each state program. Generally every
registry collects data on patient demographics,
primary tumor sites, tumor morphology and stage,
and first course of treatment. And so having this kind of data, this kind of population-based
data at the state level really presents an opportunity for public health
cancer genomics programs to leverage registry
data to identify individuals diagnosed with
cancers that may have the hereditary
[inaudible] component. And so this is usually done by
states kind of making decisions on what would be classified
as hereditary cancers. So examples are — so they
can assess all breast cancers diagnosed under
age 45, for example, all contralateral
breast cancer cases, anyone with a history of
both breast and ovarian cancer, any colorectal cancers
diagnosed under age 50, or all colorectal
cancer cases diagnosed in the case of Lynch Syndrome, endometrial
cancers under age 50. All endometrial
cancers or programs have kind of different cutoffs they
use for some of these things. And so for, in the
case of Lynch Syndrome, some of our registries also
collect information on IHC, testing [inaudible],
but that information is not necessarily
collected as routinely across all state
registry programs. That information is
also a little harder to find in electric medical records because all registry
information ends up — is extracted out
of medical records and it would be a
little harder to find. So routinely finding
some of those details in registry records are
a little more complicated. Similarly for BRCA testing
status, for mutation status, some of that information, some registries
collect it, some don’t. So we don’t necessarily
have kind of national data on some of that. But we have two — our
states use two primary methods for
identification of individuals with suspected
hereditary cancers. They either work through
the physician or institution or they kind of go
directly to the patient. And so I’ll kind of
break those down individually. What I mean by physician
or institution reporting, a lot of our states call
that bi-directional reporting, and what they essentially do — I created kind of
these rough flowcharts of how this kind of
usually works — is that the physician
or reporting institution send the diagnosing information
or the tumor information to the state cancer registry. The state cancer genomics
program extracts relevant cases based on the definitions
they’ve set from that registry, and then they create
reports that they send back to either the reporting
physician or the institution. Now these reports can
kind of vary depending on state policies related
to privacy and confidentiality of registry data, but
usually on a very general sense they at the very least
include aggregate information on the number of individuals diagnosed by a
particular provider or by a particular
health center or system with the suspected
hereditary cancer. So you’ve diagnosed in the
last year X number of patients with these kinds of cancers. They usually let them
know at the state level how many were
diagnosed with those diseases, with that same disease,
and then provide information on in terms of genetics how
that patient should have — that these patients would
have been ideal candidates for referral to
genetic counseling and testing solely based on their
diagnosis information. Family history
is usually a field not available in
cancer registries. So any kind of
determination of whether someone may or may not be
high risk for hereditary cancer is solely based on the
primary diagnosis information. And so these reports tend to go
back to the reporting physician or the institution to
kind of let them know, hey, in a given calendar year these
are the cases you diagnosed. And also with the packets are
usually included information on referral to
genetic counseling services that providers may find useful. They also provide
educational information to both the physician and for physicians
to share with patients. Sometimes
depending on state policies, some of our physicians can
receive identified data reports where they’re
actually able to put the name, birth date, and potentially
some contact information for the patient so the
doctor has the ability to refer back to a
specific patient case. So the benefits of
this kind of method are that it identifies patients who are potential candidates for
genetic counseling and testing at a population-based
level within a given state and it kind of does
provide a new opportunity for implementation of
cascade screening efforts. However, it does
present some challenges, as we discussed before. Only relying on individuals
already diagnosed with cancer will leave a lot of
relevant cases left out. The provider or
institution in the registry may not have been
involved in the treatment. Since the extraction of
medical records for registries don’t necessarily
happen at every state and several times over
the patient’s process, you could have been
diagnosed somewhere but then received care somewhere or you could have
moved to a different state so your information would be in a different
state cancer registry. There’s also a lag time
usually between diagnosis, registry entry, and the time
the program can extract data. Those lags could
cause a potential provider to not want to reach
back out to a patient. So if they find
out two years later they should have
recommended genetic counseling, they may be hesitant to
reach back out to that patient. And if you’re reporting
back to a health system level, recipients of reports
and dissemination plans may vary by institution. So who at Kaiser
Permanente, for example, should receive this report? You have to really make sure
you’ve identified the champion in that health care system or have a partnership of some sort
with that health care system to know who to really
send that information to. Some of our states
anecdotally have reported that as a system intervention
this hasn’t necessarily been the most effective, but
that it has been effective as an educational
intervention for physicians. A lot of times our
programs have been invited to present the results of
kind of some of this work at tumor boards or grand rounds so they kind of have an
opportunity to educate doctors when there’s a captive audience on some of these cancer
cases that might have benefited from genetic
counseling and testing. But we do also have some
examples where health systems, just by being
educated or physicians just by being educated on the cancer
cases that they should have probably referred
to cancer genetics, that they question their systems associated with this sometimes. And Heather
alluded to this earlier, it’s really about realizing
the burden that you might have or how popular
those services might be that might make you
actually stop and think, well, maybe this is something
that we should be looking at in a different way
than we have been before. It’s especially been true for
some of our community hospitals. The other system is
really patient-level reporting. And this is not something
necessarily — this is a system that many of us
are kind of used to. Especially anyone who’s
done patient contact research through a registry or anyone
who’s done caregiver research is probably pretty
familiar with this, where we instead of kind of the
state cancer genomics program reporting back to the
reporting hospital institution, it kind of goes
directly to the cancer survivor and then work with
the cancer survivor to kind of contact relatives
who may also benefit from cancer genetic services or from
increased screening services. Depending on state
policies, there might be a step in between the state
cancer genomics program reaching out to the
cancer survivors themselves where they might have to
get a physician consent process kind of put into the system in
order to contact the patient. And so the way this
usually works is where the state genomics program
counsels the cancer survivor, works with them, does a
pretty intensive interview to get a sense of
family history of cancer, if any cancer
genetic services were part of their diagnosis process
or their treatment process, and then work with them to
kind of identify family members based on the pedigree
information that was collected in order to really see if
there are certain key relatives who might be beneficial
to kind of bring in together and then kind of
follow protocol, similar to what
Heather talked about where you kind of try to get
that captive audience together, and the patient
and their relatives, in order to kind of talk through
the family history of cancer, what the risk to
the family might be, and who in the family
might be ideal candidates for receiving genetic
counseling [inaudible]. Similarly, this
also identifies patients who are potential candidates for
genetic counseling and testing at a population
level, and they really have significant opportunities for improving
testing screening efforts since you have a captive
audience of the survivor and their family. But also similarly
there are several challenges. It is hard, like I said before,
to identify all relevant cases. The lag time and availability
of data in the registry becomes potentially
more of a concern now because you’re trying
to reach out to patients two years
potentially post-diagnosis, and they might have a
lot of questions or issues associated with why
this wasn’t talked about when they were in the
diagnosis and treatment process of their cancer if
it wasn’t brought up. This does require much
more significant resources. Especially a lot of our
state health departments that have engaged
in these activities [inaudible]
additional funding though where we’re
normally able to provide and it’s really
necessitated partnership with large academic institutions
in order to kind of have the infrastructure to kind
of manage a study of this size. IRB-approved research
protocols are needed, especially for programs
like ours that are mostly based on implementation. A lot of the IRB needs
of our programs are often kind of put under
public health evaluation so it’s got a stringent research
protocol approval process, but for studies like
this you can’t just send out health education
materials to a list of people who have been
diagnosed with cancer. Most states do not allow that; you need to have some sort
of research protocol involved. So the sustainability of an
activity like this over time, and you have to wonder
how many research protocols the state department or
university is kind of going to go through doing the same
thing over and over again to try to keep this moving. And also the burden of
communicating risk to relatives is kind of solely left
to the cancer survivor, as we’ve
discussed earlier today. They might be able to
receive some coaching from a genetic counselor or
someone on the research staff before kind of
reaching out to the relatives [inaudible] the
ability to get materials. But it may be difficult. These are difficult
conversations potentially to have for patients
with their family members, especially if there
are family issues or family communication problems. So this kind of leaving the onus
on the patient can backfire and can make [inaudible]
not as productive potentially. And also, if anyone
else has ever been involved in contacting people
from a cancer registry, there can be alarm at the
existence of a cancer registry. I, myself, have been
involved in several studies where we contacted patients
from cancer registries, and not everyone is excited to
know that the federal government knows that they were
diagnosed with cancer. So in summary, cancer registries
present an opportunity for cancer genomics
programs to identify high-risk cancer survivors
either through physicians, health care systems, or the
patient and the family members. While these methods
have the potential to increase appropriate utilization of genetic counseling and
testing, further work is needed to better
understand how to implement and evaluate these activities. Evaluation of this work has
been significantly challenging because it’s difficult to
figure out who you survey at a health system, and
doctors aren’t the greatest at returning surveys. So we have a lot of
challenges implementing this. Challenges and
limitations of these methods may make it difficult
for timely implementation during the
diagnostic treatment process. That’s this lag I’ve been
talking about from the data when the patient gets diagnosed until you see the
data in the registry. But hopefully through the
work of our five grantees we’ll be able to
clarify some of these questions or be able to push the
envelope a little further in this over the last
two years we have left in our funding cycle.
All right. Thank you. [Applause] JOY: Good afternoon, everyone. Thank you for the invitation
for allowing me to share some time with
you this afternoon. So I wanted to focus
a little bit more on the workforce potential
for genetic counselors and talk through
some strategies in which we might begin to
meet the future demand. Let’s go through a
little bit of background. This data is coming from the
2016 professional status survey through the National
Society of Genetic Counselors, and the picture is showing you the wide variety
of subspecialties that genetic counselors work in and the various
work settings as well. The number I do want to
highlight on this slide is that actually in
2016 there’s 48 percent of the genetic counselors
that are specializing in cancer genetics, which
is up from 29 percent in 2014. We worked hard to develop
multiple service delivery models because we
understand the need for access in various geographic areas. We’ve always used traditional
face-to-face consultation, but we’ve expanded into using
telephone and telegenetic, which is a combination of
telephone and computer, as well as group sessions and using
some interactive technology to help fit the
different educational needs that the patients have. Early data suggesting that
there’s comparable satisfaction regardless of which
model it is that we’re using. So this is a nice way to
help reach a further audience. When it comes to
framing the workforce issue, this is a difficult
concept to try to tackle, and we have
multiple facets that NSGC and our sister
organizations had worked on to come to this conclusion. So let’s start with
improving the utilization of current workforce. Right now there’s 4,140 board
certified genetic counselors in the U.S. and there’s underutilization
of their services. We have existing capacity
that could be improved upon. Part of that is
lack of education about how to find us and
where we’re located, both on the part of the public
as well as referring providers. So there’s two
tools that can be used. One is through
NSGC.org and the other one is the brand-new
consumer website that is
aboutgeneticounselors.com. But if we have
access then to utilize these alternate
service delivery models, genetic counselors tend to be in
the larger metropolitan areas. That means that people
who are in rural areas have the ability today to
access genetic counselors if we can get them connected. But we’ve also
learned in this process that availability
alone does not lead to appropriate access
to genetic services. Part of what we’re dealing with
is still under-recognition, under-referral of the patients, and that’s a big
contributor to the problem. When it comes to accessibility,
I thought it would be good for us to start historically. In the 10 years
between 2006, 2016 the profession had
an 88 percent growth in the number of
genetic counselors. That’s really good growth. Most genetic
counselors are reporting that they could see a new
patient within a week and if there are stat
consults that are needed for immediate
surgical management, the vast majority of
genetic counselors were saying that they could accommodate that
request in one to two days. We’ve also focused
on efficiency efforts because it’s
very important for us to take and maximize efficiency
of our current workforce to increase
those patient volumes, yet maintaining high-quality
care at the same time. We’ve explored electronic tools and creating some
interoperability so that we’re not doing redundant
data entry to be able to complete the risk assessment
in that consultation. And a systematic way
to collect family history would be important because
oftentimes those pedigrees on an easy short pedigree
will take about 15 minutes. Easily that can go to 30 minutes depending on the
size of that family tree. Having that information
up-front can save a lot of time because we’re only updating
and dedicating our time to the conversation instead. We also need to
improve administrative support so that genetic counselors
are focusing on the work that is specific to
their unique skill set. We need to move away
from doing scheduling and secretarial or
administrative tasks, because there are other
people that are very skilled at being able to
accomplish those tasks. And then
streamline the EHR function so that we can have
the genetic information, do the queries and again
create that interoperability. There are access
barriers, and I’ll focus on lack of Medicare
recognition and state licensure more in a minute, but reimbursement for services
is a significant problem. There are areas
throughout the country where reimbursement for the
services is a covered benefit, but it’s not
consistent across the country. So there’s work
that needs to be done so that patients can
have access to that service. And also working
with the payers to create an accurate provider registry. Many times the provider
networks are too narrow. The payers are contracting with one given
telephone company, for example. They may not be
allowing their policyholders to utilize the genetic
counselors that are local. So we’re not
able to capitalize on all the different
service delivery models that are available to
reach those patients. There’s also an
access barrier because many of these registries do
not have genetic counselors as a provider
category, so the policyholders have a difficult
time identifying who would be
appropriate for them to go see. When it comes to the
barriers to referral, I mean we’re all busy.
We’re busy clinicians. We know for a while that
providers have a lack of time to obtain and
assess the family history, and there’s variability
in the comfort level in the knowledge in
genetics to be able to fulfill that task
once they’ve obtained it. There’s also
substitution effects because traditionally there has
been — it’s been noted that there’s a
shortage of genetic counselors, which actually
hasn’t been true until about the last three years. Now we’ve got work
that we’ve got to do. But the substitution
effect, we’ve got providers who are desperately
wanting to help provide access to their patients
and offering that. Sometimes that’s being done very
well, other times not so much. We think about what’s happening
in that clinical landscape. There’s about 10
new genetic tests that are coming to the
market on a daily basis, and the kind of complexity that
goes along with those tests has gone up dramatically. So from a provider
standpoint, unless they’re able to input concentrated
effort on continuing education in genetics, it
will be very easy for the best-intended
provider to fall behind. The lack of CMS
recognition is a barrier issue. We’re working with
Representative Paulsen, who happens to be my
representative from Minnesota. We do have a bill
drafted that will recognize certified genetic counselors
as providers under Medicare. And we’ve spent a
lot of time working with other organizations
gaining support for this, and this is a place
where the NIH and the NCI could also be of
benefit for us in this process. The goal is to introduce the bill sometime
during this session. State licensure
has been another area where we’ve
placed concerted effort. There’s now 22
states that are actively — there are 22 states
that have licensure. Nineteen are actively issuing,
three are in rule-making. And licensure
efforts are ongoing in an additional 26 states,
from having bills introduced to just starting
that conversation. So at least across the
country there’s work on licensure in 48
states so that we can be recognized providers
in that capacity too. Then we have to
think about the training. Right now there’s 41 genetic
counseling training programs that are available. And we worked with
the current programs to increase the
number of students that are admitted
to those programs. On average that
class size is currently about eight
students per program. The program directors are
commenting that they’ve had a very significant increase in
the number of unique applicants that are
applying to the programs, and these are
well-qualified people, but there isn’t
enough training slots to be able to invite
them into the profession. So we have a robust
pipeline of more people to help build the programs. And there’s also
work to ensure that accreditation and certification
trends are aligned with the best
practice in supporting the growth of the workforce. We need to think
about new programs. There’s been active interest in
starting new training programs. There’s 21 schools that
have expressed interest in exploring the
development of a program, and the new programs coming
on are intending to admit much larger class sizes than the current
availability of the programs. So if we’re spreading the
entry of those programs out over a couple years,
that gives us the opportunity to see a bigger increase in the
number of genetic counselors but also not overshoot
when we’re trying to estimate what is that
demand for services. One of the bigger issues
that we’re starting to see now is clinical genetic
counselors are having more and more
demands on their time. So we need to be working
with the employers so that the employers are
supporting the ability of the clinical genetic
counselors to train the students and be a rotation site. This is the part that I’m sure
you were all really waiting for, and that is really how
can we project the growth to meet this demand? So we’ve hired
Dobson DaVanzo, which is a
consulting firm here in D.C. And taking some of this
data that we’ve talked about, we’re trying to project
what is the future growth of the genetic counseling
profession in the next 10 years. They’ve
estimated that that growth would have been 72 percent. But in the first year
since that report was written we’re already
outpacing that by 6 percent. So this number is
going to be pretty fluid because it is
difficult to try to calculate. They’ve also looked at
a couple different models of trying to
determine how many — for one genetic
counselor how many people do they need to
be available for, and that’s
another difficult model. If we take the model
that one genetic counselor per 100,000 people
in the United States, we should reach
equilibrium in five years. If we take the model that one genetic
counselor per 75,000 people, the estimate is that the
equilibrium will be reached in 10 years or less. So this is a gap,
but it’s a closeable gap with concerted effort. I also wanted to provide
some clinical perspectives on meeting some of the
needs, and some of which I’m going to echo
some of the things that were already said today
so I’ll gloss over some of it, but we’ve all said
that Lynch Syndrome is not the only explanation for
the cancers that we’re seeing. And while people have
talked about this before, I’m a clinician. A good family
history to me is critical to interpret these test results. Without it we’ve got
an indeterminate negative and we run the risk
of false reassurance and the lack of
utilization of risk reduction or surveillance options. And we know we’ve
talked about the panel testing to test for other things. But this to me is where the pre- and
post-test genetic counseling is more important
now than it has been. We talk a little bit
about the cultural differences because not all cultures value
genetic testing in the same way, so we do have to be
aware of the potential for stigmatization of
results, transmitter guilt, and some cultures feel that this
is going to be a detriment, not a benefit. So if patients aren’t
ready and they’re not choosing to find out this information
on their own time frame, they may not be ready to
utilize the risk reduction or the surveillance
options that are available. And we still have to
deal with the pervasive racial and ethnic
disparities between access to genetic counselors as well as the other specialists
that might be needed and the access to
quality care and follow-up. So if we identify the
mutation, we need to make sure those patients
have access to people to take care of them
and reduce the disparity in their ability
to pay for the care. When it comes to the
public and the providers, we do have a risk of harm from misinterpretation
of the results. We can end up with
unnecessary surgeries and, again, that missed
opportunity to take action based on that DNA test result. We’ve talked already today about the importance of cascade
testing, but as a clinician, I’d like to see family history
as another screening tool that needs to be revisited
on a once-a-year basis because those
family histories change. We need to go back periodically
and re-evaluate that risk. And people need to understand
that there is a potential that we will identify
unanticipated findings. And particularly lately when
we’re talking about concerns on behalf of our patients between
health, life, disability, and long-term care insurances. We need to be able to
address those concerns, and until we can do that, it will limit the
uptake of the genetic testing. The systematic need, part of
— one of the biggest impacts across the board, regardless
of subspecialty in medicine, is being able to systematically
take a family history because the vast majority
of time that’s not happening or if it is happening,
it’s not even to the point that we’d be able to triage it,
let alone do a risk assessment. So collecting that base
family history across the board will be a really
good first step. Having a centralized database,
because we can prevent the repeat testing and a
method to re-contact people because over time the
interpretation of that result is going to mature
as the data matures. And as people move through
different phases of their life, what that result means to
them will change over time, and we need to be
able to keep them in contact with
genetic services. And we’ve already talked about the clinical
infrastructure needs. Earlier today we
had a brief talk about what is the process
for genetic counseling, and I think there’s
several key questions that we need to be
asking as we determine how would we
implement such a project. Who would get access to care? Is it everybody or is it only
those that would test positive? And what happens
to the participants that don’t schedule the visit? Have we generated a liability
on the part of the provider that we don’t have the ability
to connect with the patient? Who’s going to provide
the genetic counseling and how can we ensure that they
have the appropriate training to do that and be able to
ensure that the interpretation of that result is in the
context of the family tree? So when you think about the
impact of universal screening on the genetic
counselor workforce, if we’re going to estimate
that the care frequency is at about 1 in 279, I
think we do need to think about large-scale ways in
which we can provide education, because at the time
of a cancer diagnosis or potential of an identification
of an inherited risk, it becomes a crisis moment. So having real-time
access, whether that be through a recorded video
or electronic tools or different education materials that are quick and easy for
the providers and the patients to be able to take home until they have the ability to
have a further discussion with the genetics professional
would be important. And we need to consider
again the needs of the programs because at the time of diagnosis
they’re also very concerned about their relatives, and being able to get
that cascade testing going, as Heather had alluded to
earlier, is very valuable. Bringing down
the access barriers. And this is something
that I think we can all be working on together because
part of the access barrier is being able to accept that these alternate service
delivery models have value, that they are a way, while it’s not the
traditional face to face, they still can
connect a person one to one with a trained
provider and be able to communicate that
information in a timely way. And it adds flexibility and
convenience to the patient — frankly to the counselor
and the referring provider — because we’re able
to see more people and bring down
costs at the same time. So we’ve learned from talking with our patients
and the consumers that we can’t rely on
just a one-size-fits-all model of how to
disseminate the information. Not everybody
learns in the same way. Some people are really going to
prefer that one-to-one visit, others will prefer the privacy
that’s afforded by telephone, and then others will
— especially nowadays they’re much more comfortable
with the interactive technology. They’d rather learn it that way than trying to go to
another appointment. So when we’re
thinking about collaboration, this is something
that’s going to be important because our current
system is not working. From multiple different data
sources that we’ve approached to collect information
over the past several years we’ve learned that about
30 percent of genetic tests that are ordered by
non-genetics providers are inappropriately ordered. And what I mean by that is it could be the wrong
technology, the wrong gene, that person is not
a testing candidate, or they’re blending both
sides of the family together in order to meet criteria. So that’s suggesting
that we’re needing — that providers
are needing some help in being able to
execute this process well. And we need to stop
thinking about genetics as a boutique service and get that integrated
into routine patient care because it’s
something that impacts every subspecialty of medicine. So by working together we can be providing and
maintaining education either from us providing the
genetics but also looking at the other subspecialties,
you informing us, and so that
we’re working together as a good cohesive health
care team for better care. So again, collaboration to
me is going to be key in this and it fits well with
the Moonshot initiative because by working together
we can make a difference to benefit our patients,
the health care system, and the population. Genetic counselors would be
happy to take a leading role in the development,
implementation, and the delivery of these services. We can enhance the team approach with our health care colleagues. And it’s imperative that we’re
collectively doing a better job. Thank you. [Applause] MUIN: All right. Good afternoon. Do you want to take
a one-minute stretch? It’s been a long day for
all of us. Stand up. Thank you. All right. First of all, I’d like to
thank all of you for staying, because it’s been a long
day and throughout the day I almost changed my
slides at least five times, but the good news is
that after such a long day everything that I wanted
to say has already been said. But here are our
discussion questions. What can we learn
from other conditions and other pre-cancers, although
the focus is on Lynch Syndrome, workforce issues, implementation
beyond genetic testing, and system level? So what I decided to do
in my five to seven minutes is to take a
bigger-picture approach. You’ve heard about implementation
science from David. You’ve heard about
public health efforts from Juan and earlier from
Lisa Richardson. And Lynch Syndrome
represents really the very tip of the iceberg for
precision medicine. Think about genomic medicine with all the many, many
genetic tests, panel testing, even [inaudible]
whole-genome sequencing. What this
intersection represents is sort of what we need. I’d like to describe
to you that intersection. It’s very small right
now, but it’s growing. And in order for
precision medicine and Lynch Syndrome in
particular to succeed, we need that
intersection to grow. As was alluded before, and
actually Heather mentioned tier one genomic tests
and applications earlier, I don’t think anyone
knows what tier one is. Part of what we do in our
office at CDC is to try to have CDC programs focus on
where the evidence is because there is a lot
of hype about genetics, and people sometimes are —
in public health in particular are willing to throw
the baby with the bathwater because they don’t think
that it’s any baby there. And so for the last few
years we focused our energy on trying to find
those applications that are ready
for implementation, and Lynch
Syndrome is one of them. There is a lot of
stuff in the red. Actually, as a matter of fact, most genomics are
still in tier three, not ready for application, but there is things
bubbling up to the surface. And a few years ago
we had this workshop that identified the top
three tier applications. Lynch Syndrome is
one of them, HBOC, and I want to use an example
from the non-cancer world, familial hypercholesterolemia. No one today has
talked about FH, but there is a lot of
parallelism between FH and these other two cancers. All three conditions
are autosomal dominant with high
penetrance, and they’re common. In the U.S., actually
that number around 2 million may be closer to 3 million, depending on what
numbers you quote. The problem is most people
with these three conditions don’t know they have it. So one of the questions is why
not screen the whole population. We screen newborns, don’t
we? We’re not going to do that. I don’t think there’s
any evidence to do that yet, and Barry Kramer
always has the right question about the penetrance
and the natural history because there’s a lot
of things we don’t know about these conditions. But I wanted to
kind of revisit with you the three recommendations
that are the basis of these three
tier applications. EGAPP was mentioned. EGAPP has been really
sort of a pioneer effort. [Inaudible], who is
a speaker tomorrow, was the EGAPP
chair for a few years. And as a matter of fact,
EGAPP was the first group to recommend universal screening for Lynch Syndrome
for colorectal cancer. NICE in England today
released the recommendation for universal Lynch
Syndrome screening. Up to this point
they had an age cutoff. So we are eight
years through the journey from the
recommendation to practice. I don’t think we
are at 6 percent yet in terms of that benefit. So what we have to do, and even if we
implement this recommendation, it will miss
more than two-thirds of Lynch Syndrome
patients in the U.S. That number is a back-
of-the-envelope calculation. When Doug Lowrie [ph.] was
here this morning, he made his own back-of-
the-envelope calculation, and he landed at 20
percent, which is not that far unless you start doing cascading
beyond the nuclear family to cascade to first, second,
third, fourth, fifth generation and, you know, Heather
is shaking her head there because the more
relatives you get, the more cases you will find. And that experience we
have in the Netherlands for cascade screening for FH, which we don’t
have in this country. Just a word about BRCA. We’ve seen the recommendations
and they’re all family-based. There is no recommendation
for universal screening in the population, although
people like Mary-Claire King would love all
women to be screened, and Joy and I had
the chance to debate her at the last National Society
for Genetic Counselors meeting. But Mary-Claire is right. If you implement the U.S.
Preventative Services Task Force recommendation,
you’re going to miss more than half of
people with hereditary breast and ovarian cancer due
to BRCA 1 and 2 mutations. NICE in the U.K.
was ahead of the U.S. with respect to FH because we don’t have
U.S.-based recommendations around familial
hypercholesterolemia no matter how much we tried
to influence the task force. As a matter of fact, they recommended cascade
screening in 2008. The last task force in 2016 around familial
hypercholesterolemia concluded with the
following statement, and I want to read it
because it applies to cancer. “Cascade screening was
excluded from this review because of its limited relevance to the current U.S.
primary care practice, but it may be a promising
strategy for FH-case finding, especially as
genetic testing evolves.” Cascade screening is
a wonderful concept, but it’s very hard to implement,
but we shouldn’t give up. So what did we do
from 2009 to 2017? How do we
accelerate this process? So the first thing we did
was to convene that meeting back in 2011 which
[inaudible] had mentioned, as a result of which the Lynch
Syndrome screening network was created, and you’ve seen
some of the results from that. The other meeting
which the NCI put together was this multilevel
intervention approach for the cancer care continuum. That was a special
supplement to JNCI that was put
together back in 2012 based on a series of
conversations and meetings, and there is
[inaudible] right now that shows that in
order to implement anything in cancer care and prevention, you need to look at all
these levels and you need to look at them together,
not just one at a time. At that time one of
the papers was presented and I had the chance
to work with many people to write about the
multilevel research, and the challenges of
implementing genomic medicine was part of that conversation. And we used Lynch Syndrome as
an example even back in 2011. You can’t see all of this stuff, but you can go back
to the paper and see how the factors impede or
facilitate implementation of finding cases and cascading
to relatives and implementation. And you’ve heard about all of these
examples of factors earlier. And some of the
studies that were presented and are in the
literature look at that. But let’s explore
this intersection between implementation
science and genomic medicine. David and I had the chance
to work with Megan Roberts in the back,
[inaudible] and Amy Kennedy, who I’m not sure if
she’s still here anymore, to actually review the
literature on that intersection. We took one year
of data from PubMed, which is about
100,000 genomic publications. And we found only 283 articles that meet the definition
of implementation science. That’s about .28 percent
of the genetic literature, is implementation science. Half of it is most
relevant to cancer. There were 12 articles
relevant to Lynch Syndrome. So Lynch Syndrome is an
active area of research, but some of the
common features of the current published implementation
science in genomics really has room for improvement. Only less than 2
percent of these studies integrated these implementation
science frameworks. Most of the studies are
based on the individual other than systems or providers. As you heard the
whole day, you know, in order to do proper
implementation and dissemination you have to
include all these levels. The majority of
published studies were done in academic medical centers. You’re going to hear from
Greg Field [ph.] tomorrow about sort of the real world
out there outside of academia. Fewer than half of the
studies included information on race and ethnicity,
challenging our issues around disparities
and generalizability. The majority of
studies were in oncology, which is fine in this case, but there’s plenty that’s
going on outside oncology. Most studies are observational. There is very little in terms of
strategies for experimentation with implementation A
versus B, for example. And few studies reported
on the use of collaboration, which is a key
metric in order to succeed. So I want to end up with
sort of a shout-out here. I don’t know why this is coming
like this, but let’s — okay. So for the last year we’ve
been exploring through the National Academies of Sciences
Engineering and Medicine. It’s a mouthful to say. I
used to love the [inaudible]. It was much easier. We created a new
action collaborative, genomics and population
health, which is what I call an example of
precision public health. We are focusing on
these three conditions: finding ways to try to
scale up the implementation using the public
health health care interface. You heard from Juan that
five states are being funded to do Lynch Syndrome work, but the other 45 are the
states working with, you know, all the partners. You heard from Michigan earlier. So we have two working groups, and both chairs of these
working groups are here. [Inaudible], who
you’ll hear from tomorrow, is in charge of the evidence
working group reviewing what kind of evidence does public health and population
health planners need for action. And the one
thing that, you know, coming from a world
of public health, people always tell me, line up. There is a whole
slew of activities we need to do in public health
before we get to genomics. And I have to fight my way
to the table telling them, but look, there is a million
people that need to be found, and we can save X
numbers of lives every year, and it’s cost
effective, you know, with this kind of metric or not. And the second working group
is chaired by Deb Duquette, and Heather is also part of it, and there’s other
people involved here. It’s to try to, you
know, do the landscape of finding who’s
implementing, developing the tools and
metrics for success. And what we’re
going to do this year, I think one of the
things that is really ripe for studying using
implementation science framework is study the science of
cascading because cascading — and we heard from
Patrick Lynch this morning — is not easy and there
has to be a way to study it, study the best approaches to it because the more
we find [inaudible], the more we’re going to
get to that one million person who have Lynch
Syndrome in the population. And last but not least, I
think we are going to have to study the feasibility
of population screening, whether actual
and opportunistic. And what’s the difference
between the two of them? The opportunistic
population screening is the precision
medicine initiative. All of a sudden we’re going
to have millions of people with the sequence in hand. So for these people we can
study by returning the results of at least some of these genes
to figure out who’s part — who has these diseases,
are they part of the system, how can we reach them
with the proper services? So with that I’m going to turn
it to my friend David Ransohoff for further
discussion. Thank you. [Applause] DAVID R: Well, thank you. This
has been a fascinating day. It has been very intense. Luckily, as Muin said, we’ve
heard just about everything that anybody can say, so
I’m going to be very brief. I have just about
six slides total. And I wanted to
select, let’s see, next. You know, if it’s not a Mac, I
just don’t know how to do this. Good. Now I can do the
arrow, the down arrow. Thank you. All right. Okay. So I want to talk
just very, very briefly about one topic that
we’ve heard some about today. We’ve talked mainly
about cascade screening, and I want to talk a little bit
about beyond cascade screening and why this might be important to keep on the
radar of this field. This is not the main topic
of the Blue Ribbon Report, if you’ll look at the 74 pages. Stuff about non-cascade
screening or population-based screening does appear in
the supplemental material. So it’s on the radar somewhere. I just want to make
a case for having this somewhere on the radar. And the topic —
right, down arrow. Thank you. Where is it, the down arrow?
Okay. All right. There we go. So this relates to
our discussion questions about
implementation very broadly. Here’s the
discussion questions here. And what I want to
focus on just for a minute, again this is
very, very brief, is can we go beyond
cascade screening for Lynch and other things like BRCA and other things where
there’s strong mutations that relate to risk? And the question is,
what’s the colon cancer risk associated with Lynch
mutations in the population not in families with
Lynch cancer or BRCA? And the
importance of this is that precision testing by genetics,
this is what we’ve promised to the public and
what they expect. Now that doesn’t mean that we
can deliver this to the public, but when they think about
genetics and precision medicine, this is basically what
we have held out to them, it’s deterministic. And 50 percent of Lynch cancers, or 80 percent of Lynch cancers, or 20 percent of
Lynch cancers are missed. A lot of Lynch cancers are
missed by cascade screening. Mutations found in primary
testing if they’re acted on, because we get them from
23 and Me or whatever it is, these things can cause mischief,
meaning major operations, major worry, major worry when people with the gene
for long QT and sudden death go to their cardiologist
to say I’m going to die. Do I need a defibrillator? There’s serious stuff
that’s going on here, and we’re messing
with people’s lives. And if we had
reliable risk data, that could allow
population screening. And I was fascinated this
morning when it was Barry Kramer asked during one of the
panel discussions, he said, “Okay, let’s pretend
we’ve got a person who’s got a Lynch mutation but they
don’t have a family member with cancer. What’s their
absolute risk over a lifetime?” It took several hours
to get to that point where we’re talking about numbers.
What is somebody’s risk. And the answer was we
basically have no idea. And that by
itself is fascinating. Rick Boland then said, “Well, I think I can
speculate about what some of the other
mutations or problems are or features about why there’s this huge
spectrum of penetrance.” But this is a really,
really serious problem that gets in the way of
our having reliable risk data, not for people with families
but for people without. Is there a way to address that? And there was some talk
about that this morning. This is not a new problem. This is being studied for
BRCA in the U.K. and so forth. It’s on people’s
radars. It’s hard to study. But is Lynch one of the things that would let us
study this more easily? Should I press the down
button? Yeah. Okay. Yeah. See. All right. We’re nearly done. Implementation strategies would
differ from cascade screening for hereditary cancer,
and this is a future topic. We’re talking about
implementation now. The
implementation scheme would be substantially
different if we did this. And this is now the last
two slides and then I’m done. I just want to put this
more on people’s radars. Again, I know
it’s already there. The importance of putting
this problem on our radar and the conclusion of this
tiny little few comments is Lynch, if anything,
is a poster child — Lynch should be the
poster child example for population screening.
It’s an autosomal dominant. The colon cancer risk
can get very, very high, and it’s easy to intervene. We’re not doing
radical operations on people. This is a fairly easy problem. And then this is the last slide. If we can’t figure out
population screening for Lynch, then how can we
implement testing and follow-up for more complicated
or more subtle risk? And I’m wondering, you know, if
somebody was king of the world, you could suggest to them,
let’s really study the heck out of this problem
and BRCA and figure out, like for
Mary-Claire King or whoever, what can we really do in
population-based screening, because if we can’t figure
it out for these problems, we’re going to
have a real hard time figuring it out for others. I was struck too by Dr. Brody’s
comment today earlier saying, you know, maybe these things
are really extreme one-offs and we’ve even got
problems using them. What does that say
for the whole field? The last point is
if we can study this, we’re going to learn
interesting things about how hard it is to do,
how easy it is to do, whether it’s feasible. That’s going to be
useful clinically. We’re also going to
learn, like Rick was saying, really interesting
biology if we can understand why some people have bad
outcomes and others don’t. Again, this is not the
main focus of the Moonshot immediately in the 74-page
report, but it’s on the radar because it’s in the
supplemental material. Just something to keep in mind. Again, this has
been a fascinating day and I’ve learned a
lot and want to thank you for being invited and
participating. Thanks. [Applause] MALE: Okay. So I am going
to put you back to the past. And the past looked a
little something like this. So you have the panelists here. It is opportune time for
questions, for comments. As you can see, a
number of discussion questions were touched upon by
the different panelists. Feel free to weigh in but
also to gain more insights from their expertise. So, Barry? MALE: So for those
out in cyberspace, we are working on the
microphone becoming live. And there we go. BARRY: You made a very good
point, but it was a brief point about the
implementation, because the whole day we basically
focused on implementation. And the question
is if we accidentally and with all good intentions
overshoot the mark, usually it takes years if not decades
to get the pendulum back. And so I would just
like your thoughts on what you do to
anticipate, you know, in case you overshoot the mark. If you overdo it
and you learn something that you’ve
incurred [inaudible] harm, if you haven’t
already thought about it and done research on it, then you’re starting
from a standing start when the null
hypothesis has already been set and to stay in place
for years and years. So do you have any thoughts on should we be anticipating that and planning
research in that direction? MALE: Yeah. So
thanks, Barry, for that. Yes, I think we
should be far more willing to take advantage of
ongoing natural opportunities to see and to try and
actually encourage and assess the need for de-implementation. Most any time that
we’re implementing something we’re often
needing to substitute or need to take something out. And I don’t think we really — at least a lot of our studies
are kind of ignoring that. It’s almost like we’re
dealing with a vacuum. So I think the first
thing that we can do is think a
little bit more through the collection of efforts trying
to implement something new to think about that
other side of the equation. The other thing, and I
think folks within the VA have done this very
nicely, is thought about ways to continue to
expand the evidence base while considering
implementation around the effectiveness of
different interventions. And again, our evidence
base for most interventions is incomplete just given the
heterogeneity of the population of systems, et cetera. And so the folks in the
VA who have talked about hybrid studies around
effectiveness and implementation can help as well,
because it means that while we’re
focusing on implementation, we’re continuing to gather that what is the impact at
the individual level. And so I don’t think we
do either all that much. I think the other thing is just encouraging more people to focus
exactly on de-implementation. There was a provocative question
over the last few years that basically was trying to
encourage more researchers to think of, you know, NCI studies
around de-implementation, around, you know,
abandoning things that don’t seem to have value. We didn’t get a lot I
would say, as much as we could. And so it
certainly is an opportunity to continue to add
to the evidence base. But thanks for
accentuating the point. MALE: Barry, in response,
I think de-implementation is a lot harder to
do than implementation, and we need to
think about these things with prostate cancer screening, with aggressive breast
cancer screening where the wife of the secretary of
HHS concludes I got screened, I got treated, I’m alive,
therefore cause and effect. And it’s much harder to
take things away from people than to give them something. And I think that now
we’re in this market and sort of free-for-all and everybody can have
information about genetics. I think we risk not learning and
using the lessons in the past and it can be
hard to de-implement and we may need to take a more
aggressive stance about that. MALE: I’ll just posit that
not only is it harder to do, but the strategies are
not simply going in reverse. The strategies are very
different and understudied compared to implementation. MALE: Yeah. Good point.
MALE: [Inaudible] more. One of the
challenges for genomics, the rapid
changes in the science. I mean since the EGAPP
recommendations in 2009 the field has
moved very quickly. Now we’re dealing with panels, we’re dealing
with whole, you know, different markers,
different sequences. So, you know, part of
appropriate implementation is to figure out what to implement
and what not to implement, but we can’t wait too long. I mean even the
guidelines will go stale after five years
even without genomics. And genomics is much more
quickly moving and changing year to year than other areas. MARK: Mark Jenkins,
University of Melbourne. I just want to pick
up on David’s question. How do we know what
the risk of cancer is in those mutation carriers who
haven’t got a family history? I’m trying to think of a
design that you could enact to answer that question. The only thing I
can think of is, would you have to
sequence unaffected people in the population chosen at random,
then wait 20 or 30 years, 40 years until they
did or did not get cancer? And given that 1 in
280 in the population are carrying a mutation, you’d have to maybe sequence
hundreds of thousands of people and then wait 20 or 30 years. So essentially it’s
impossible to [inaudible]. MALE: No. It’s a
big HMO, the U.K. MARK: Yeah, but
they’re going to have to wait many years for this to happen. MALE: Yeah. Yeah. You would. MARK: The alternative
is to ask the question, why would you think
it would be different? In the absence of information, the null hypothesis would be
that it would be no different. We do this all
the time in science. You could keep
looking in cases different or you could just assume
it’s there, take the safe side, screen them if you find them. MALE: Yeah, with respect to
cohorts, I mean Larry Brody may have the
answer for BRCA as well. I mean people do
that all the time. The difference in genetics, people with family
history share other genes, they share other factors. So it’s a not
unreasonable hypothesis that the penetrance
should be higher in people with family history than
somebody from the population. So it can be studied and it
can be studied retrospectively. I mean, if you get
all these cohorts from nurses’ studies
[ph.] and other cohorts that NCI and others
have funded over time, I think we have a good
idea of the natural history. MALE: Just to disagree
with Mark. I really do. But actually the Manchester
[inaudible] did some nice work where they looked at
people who were BRCA-tested who didn’t have the mutation, and they still had a higher
incidence of breast cancer. They’re arguing that the
people that they were seeing were families. There
were family-related factors. But even after you
took the BRCA out, it was still higher than
the background population. So I think there will be
some family-related factors that bring that
family to attention. MALE: We know in
Lynch Syndrome that once you identify the
non-carriers in most families, their risk is a population risk. FEMALE: I think just
to this last point that isn’t that what the purpose of
all of this is supposed to be, to enroll a
million people in research and people who are,
quote, “very average,” so that we can
assess the prevalence of these genetic predispositions
in the general population? Whether or not we actually are
able to achieve that, we’ll see. All of you gave tremendous
presentations and so thoughtful. I want to — I thank everybody for everything you’ve
shared with us today. I’ve had the
opportunity to interact with Juan and Muin previously. Joy, I think you hit
the nail on the head with everything
you talked about, and I think that goes back to what needs to be implemented
beyond genetic testing. We talk about ethics. I think there’s an
ethical responsibility that if we’re going to test
people, we have to give them the means to get the
follow-up care that they need. And that’s something that
our system here in the U.S. is not necessarily set
up to do at the moment. We have our
insurers can pick or choose what they want to cover. One insurer will cover,
you know, the colonoscopy for the 35-year-old because
they say they’re high-risk. Another one will fight them. The same thing with
BRCA mutation carriers. They’re supposed to
start getting breast MRIs according to NCCN. Some insurers are saying well, you have to meet
your deductible. So basically these women
are paying $1,500 out of pocket every year to get these,
you know, interventions. So, and this is a
very challenging thing and it’s not something
that a group of scientists can necessarily address,
but I want it to be something that’s on the
radar as we move forward that maybe our system
needs to be tweaked somehow to make sure that when
we do identify individuals who are high risk, whether
it be through cascade screening or any other, you know,
population-wide screening, that we have an
ethical responsibility to help them get the
follow-up care if we do in fact provide them with those
genetic testing results. JOHN: Three things briefly. First of all, a little
pitch from our little island across the water.
John Burn from Newcastle. The UK Biobank is here. We got the 500,000 people there, so we’re now at the point where
we’re getting genomic data from them, so we will get some genuinely population-level
information from that. The second point
I want to make is we keep using
the word screening, and I know the public
health people in the room and the epidemiologists
will get a bit anxious. It gets used rather loosely. And we need to be
very careful to know when we’re talking about
truly screening populations and when we’re simply
talking about surveillance or whether we’re
looking for mutations in a gene and people often say
we’re screening the gene. So I think we need to
use the word screening a little more carefully
because it confuses the world. And the final point
I want to make, just we were developing a course
for our counselors in genomics, and I read a very
interesting article in the [inaudible]
where the writer constantly flipped across different
peas in a pod, I call it. So it was the person,
the patient, the public, and the politics. And sometimes these
conversations pass in the night because you’ve got the
genetic counselor worried about the person’s life being wrecked
by being given [inaudible]. You’ve got the
doctor who feels like they want to do a test like
they do all the other tests. The public health
doctors are wondering how we deploy this
into a population, and the head of NCI is
wondering how you pay for it. And sometimes they
are not synced together and we just need to
be clear in our minds when we’re talking
about patient or person, patient, public, or politics. I think that sometimes
helps clarify it a bit. FEMALE: So just to tackle the
penetrance question again — because I think
it’s interesting; I’m going to agree with Mark. And the best analogy I
think I’ve seen lately is we’ve been finding incidental
[inaudible] mutations on panels, and that causes hereditary
diffuse gastric cancer. But sometimes we’re
finding them on patients with no family
history of gastric cancer. And then it’s a bit of a panic because the intervention for
these people is not trivial. It’s removing their stomach. And are you going to take
a stomach out of a person with an [inaudible] mutation who has no family history
of stomach cancer or not? And the evidence that
has been presented lately, [inaudible] group at Michigan,
would imply that in fact yes, they do have the same risk for
gastric cancer as anybody else. I guess my question is, what’s
a negative family history? Is it a small family where they don’t know
their family history? Is it a de novo mutation and that’s why
there’s no family history? Is it
non-paternity and that’s why they don’t know about
their family history? So I think
negative family history is not always
negative family history, and I would tend to
err on the side of Mark as the default
assuming the penetrance. And I do know it is different. I mean obviously the
penetrance is different in our population-based cohort than in a high-risk
clinic-based cohort, but I can’t imagine
it going down to zero or to the
general population risk. FEMALE: I guess I just
wanted to say something again similar with the UK Biobank
around the population screening. So at Geisinger we
have our MyCode biobank where our patients are consented to also have sequencing
and have results returned. So, and we have 100,000 so far. We’ve hit our target for the
100,000. We are pushing upwards. Our next goal is 250,000. And we are starting to see what are these population
prevalences, how much [inaudible] for the BRCAs
and the Lynch Syndrome gene, what are we seeing in a
general population sample compared to what we
expect with the family history. So we’re hoping we
will get some of that. Again, as you
pointed out though, it’s going to take a
very long time for us to see that in the
general population of is it any different
with the cancer incidence, things like that. But there’s lots of
us working on that, or at least some of
us are working on that. MALE: Well, thanks. MALE: I was going to
address the impossibility, because too many things that are felt to be
impossible are quite doable. And there are study
designs to get at — and I think it’s
already been mentioned — retrospectively get a
prospective assessment of risk of genes, and so we shouldn’t declare that
it’s impossible at this point. But more to the
point of the assumption, it’s a strong assumption
that family history adds zero information to, you know, to a so-called deterministic
mutation in a gene. That’s a pretty
strong assumption because it assumes no modifiers. We already know
that amongst families the absolute risk differs, so
I don’t think it’s a far leap. So this gets to the point,
supposing we overshoot the mark. Whenever you are intervening
with a preventive intervention or I’ll use the
word screening loosely, screening the population,
it’s not a matter of whether the risk drops
down to the absolute lowest. It’s a matter —
or to the average — it’s how far it drops, because
there are risks and benefits. And of course you’re
always measuring them and you want to be sure if the risk drops
below a certain level that you still are not maintaining
the same level of harm assuming that the risk
hasn’t dropped at all. So I think it’s a
worthwhile thing to know before we start
preventing diseases surgically, because surgical prevention is not always an ideal
preventive intervention. RICK: Rick Boland, San Diego. One way to perhaps get around
this issue of finding out how many trivial
sequence variations there are that don’t cause disease is to do a lot of
sequencing on old people. I mean, you know, take
people who are 70 and 80 and if the mutations are
there in that population, there’s your answer. MALE: So one more here, and then we have a
couple I think online. FEMALE: But then my question
is why didn’t they get cancer? So what are their modifiers as to what made
them not get cancer? So a lot of what we’ve
heard about this morning in terms of
screening for cancer, I will try to be
specific for colorectal cancer, has been colonoscopy,
colonoscopy, colonoscopy, and obviously that
is the gold standard. But in an effort to deal with
this de-implementation up front, should we think of a
trial, a screening trial in which to enroll all
of these Lynch patients that we’re going to
find into a screening trial in which we’re looking at
different screening methods to reduce the risk
of the cancer screening as we’ve heard
colonoscopy creates harm? So thoughts on
other screening methods for colorectal
cancer in this population. MALE: You know, what you would
want from a screening test in Lynch is for it
to be very sensitive. So if it was negative, say the
person doesn’t have a lesion. If we look at the stool
DNA combined with FIT test, part of the problem
is that the DNA markers, they’re methylated things that aren’t
strongly related to Lynch. Something else
might be developed. And the FIT component
does a lot of diagnosing, but Lynch things are, you
know, they may not bleed. I don’t know if it’s
been studied directly, but that’s one of
the potential problems, but some empirical
testing could help. The problem is too though
that if the risk is very high, you really want a
test that’s very sensitive so that if it’s negative —
so it’s a reasonable question, but I don’t know empirical
studies. Rick, do you? RICK: Yeah. Actually I think
Exact [ph.] at one time put mutated microsatellite sequences
into their fecal DNA test. They were going to
test it in Lynch Syndrome. They stopped doing it. And I have asked Barry Berger
I don’t know how many times [inaudible] when are
you going to resurrect — MALE: In the first
generation it was there, but it had a
small positive effect, and they took it
out of the second. The thing that’s
available now is the — MALE: No, it’s not in there now. MALE: It was
BAT26 or something. MALE: But it’s
technically feasible, and they could well design a
study and, say, alternate with, let’s say you did colonoscopies
every two or three years, and do that on the other years
and see if that would help them. It actually works, but they didn’t see a
big enough market in it. MALE: Okay. So I think we
have a couple things online. FEMALE: We have
one question for Joy concerning assessing the demands for genetic counseling
services into the future. If you’ve considered demand
related to precision medicine where incidental findings from
tumor DNA [inaudible] germline mutation and need for counseling
and testing, if that’s something that you thought
about in those calculations? JOY: So, Heather talked about trying to project that
kind of need is very difficult because we can’t — as a
genetic counselor workforce we can’t only focus
just on cancer itself. It’s what other kinds of
conditions might the public have that might require the
utility of a genetic counselor for that conversation. From our perspective if people
are doing the tumor testing, you know, that
should be a population that we’re already
working with because we already
anticipated that if someone is being
diagnosed with a cancer, that they have a
potential to come in and see a genetic counselor. So when we
[inaudible] population that one genetic
counselor per 100,000 people, that can be healthy people and all other health
conditions included in that, so that was part of the
facet that was calculated in. So I would count the
tumor testing as included. TRAVIS: So this
is the second one, and I’m aiming
this to Dr. Ransohoff. I’m Travis Bray of the Hereditary Colon
Cancer Foundation. It’s a comment mostly, but
you can comment on the comment. Thank you for your insights. “Beyond cascade testing”
population screening for Lynch would be great from a
colorectal cancer perspective, but we need to look beyond
colon-based intervention. For example, interventions
for women with Lynch Syndrome are rather extreme, as they include
hysterectomy and oophorectomy. So I think it would be great if we knew everyone
who had Lynch Syndrome, but we must not minimize the
effects of population screening in a broader context. It would have to be concomitant
with an increased knowledge of absolute cancer risk, stratified
by age and gene mutation, and an educated
workforce of physicians capable of individualizing
care plans for their patients. And whatever your comments
are, I do think this relates to what Barry Kramer talked
about before with balancing. DAVID R: That’s a very
fair and important comment. I was focusing
mainly on colon cancer, and the other
cancers get harder because we don’t have any strong
imaging or treatment modalities that are easy; like colonoscopy
if you were to go in and try to take [inaudible]. So that is a very fair concern
about possible [inaudible]. FEMALE: Can I ask Karen
to make a comment about it? Do you have anything to add, since that’s your
area of expertise? KAREN: No. I mean I think
the comment has to do with the harm part of it.
And so as we get into — I appreciated the comment
about really having to study cascade testing because
we’ve been at it for decades and we’re not that successful. That’s the brute
force part of it. So kind of the attractiveness
of more population testing at such a low cost.
It’s very attractive. But the likelihood of
harm is huge, increased cost but really at the end
of the day patient harm. And so I had a conversation with
Laney today about, you know, endometrial and ovarian
screening. It’s not very good. It’s fairly burdensome. Even prophylactic
surgery in someone who’s had a low
interior resection, radiation, it’s not that easy to
do kind of a hysterectomy. So all of those, I
think, issues again for us to think in a
multidisciplinary standpoint, it’s more than just colon. MALE: Okay. So let me
say on behalf of those who deserve far
more credit than I do, thank you very much
for your attendance and all of the
conversation today. We will be starting
tomorrow at 8 a.m. sharp. So feel free to do
whatever it is you need to do, and we’ll look forward to seeing
you back, so thanks very much.

Health Care Delivery Day 1 of 2: Approaches to Development, Implementation and Evaluation

Leave a Reply

Your email address will not be published. Required fields are marked *