[This Transcript is Unedited]
Hubert H. Humphrey Building
Room 505-A
200 Independence Avenue, S.W.
Washington, DC
Agenda Item: Call to Order and Introductions - Simon Cohn, M.D.
DR. COHN: Why don't we get started? Good morning, I want to call this meeting to order. This is the first day of two days of hearings of the Subcommittee on Standards and Security of the National Committee on Vital and Health Statistics. The committee is the main public advisory committee to the US Department of Health and Human Services on national health information policy.
I am Simon Cohn, chairman of the subcommittee, and National Director of Health Information Policy at Kaiser Permanente, as well as a practicing physician. I want to welcome fellow subcommittee members, HHS staff, and others here in person.
Actually, I want to take a minute out to thank the subcommittee members for traveling during these uncertain times, and joining us in person for what I think will be a very important session. I also obviously want to acknowledge, because we all are these days of international strive, the valiant efforts and sacrifices of our military men and women in Iraq.
Obviously, I also want to welcome all of those who are joining us on the Internet, and remind everyone to speak clearly and into the microphone.
Before I discuss the day's activities, I do want mention, as you will be hearing more about tomorrow, that the secretary of Health and Human Services has announced the first set of uniform standards for the electronic exchange of clinical data to be used within federal health care agencies, including HHS, DoD, and the VA. These are based, as you all know, on our previous recommendations, and of course the very good work of the combined health care initiative.
This is of course, very good news for all of us, and for all of our work. And I think it reinforces the importance of the work of this subcommittee, as well as the integrity of our process.
Today, we continue our work on PMRI terminologies. I have asked Jeff Blair to lead this session. I want to thank him and staff for helping to organize this. I should mention we go from 9:00 am - 12:30 pm tomorrow, we will continue our discussion into ways to improve and stabilize the HIPAA process, as well as some contingency planning for the administrative and financial transactions final rule.
Following that discussion, we will have an update from Jared Adair and the CHI, which are touching on some of the comments I just made. Then we'll discuss the results of the ASCA database review, which Steve Steindel and Mike Fitzmaurice are going to lead that discussion, with a view towards what we are learning, as well as what sort of best practices we ought to be publishing. Finally, we will discuss the status of the ICD-10 cost impact study.
With that, let's have introductions around the table and around the room. If there are any issues coming before the committee today, subcommittee members, for which you need to recuse yourself, please so publicly state in your introduction.
Kepa, would you like to start?
DR. ZUBELDIA: Thank you, Simon. I'm Kepa Zubeldia with Claredi Corporation, a member of the committee and subcommittee.
MS. GREENBERG: I'm Marjorie Greenberg with the National Center for Health Statistics, CDC, and executive secretary for the committee.
DR. FERRER: I'm Jose Ferrer, medical officer, Centers for Medicare and Medicaid Services.
MS. AULD: Vivian Auld with the National Library of Medicine, attached to the subcommittee.
DR. SUJANSKY: I'm Walter Sujansky. I'm an independent consultant and advisor to the subcommittee on the selection of terminologies for patient medical record information.
MS. HUMPHREYS: Betsy Humphreys, National Library of Medicine, staff to the subcommittee.
MS. BURKE-BEBEE: Suzie Bebee, NCHS, staff to the subcommittee.
DR. STEINDEL: Steven Steindel, Centers for Disease Control and Prevention, liaison to the full committee and staff to the subcommittee.
DR. FITZMAURICE: Michael Fitzmaurice, Agency for Healthcare Research and Quality, liaison to the full committee and staff to the Subcommittee on Standards and Security.
DR. HUFF: Stan Huff with Intermountain Health Care and the University of Utah in Salt Lake City. I'm a member of the committee. I have a potential conflict of interest if we talk about ICD-9-PCS codes. I'm a co-chair of LOINC committee, and former chair of the HL7 board of directors. And so, I have no financial interest, but possible biases in those two terminologies and standards activities.
DR. COHN: Stan, weren't you a previously a member and involved in various terminology activities?
DR. HUFF: Yes, also formerly a member of the SNOMED editorial board, so I'm not sure I can participate at all.
MR. BLAIR: Jeff Blair, Medical Records Institute, vice chair of the Subcommittee on Standards and Security. I am a member of HL7, ASTM, COMIA(?), and HMSIS(?). And there is nothing that I can think of that would be a potential conflict of interest that I would need to recuse myself from.
MS. EDESQUIRE(?): Mary Edesquire. I'm with NCHS, and staff to the subcommittee.
DR. SALVA(?): I'm Virginia Salva. I'm with Georgetown University and the developer of the home health care classification system.
DR. BICKFORD: Carol Bickford, American Nurses Association.
DR. KENNEDY: Mary Kennedy, College of American Pathology.
MS. ROBB: Dominica Robb, HHS staff.
DR. ASTON: Phillip Aston, Institute of Medicine.
DR. WILLIAMSON: Michelle Williamson, National Center for Health Statistics, CDC.
DR. COHN: And finally, this is the chair. Actually, since I sit on the CPT editorial panel, while I don't think that there are any issues coming up today that directly relate to CPT, I do need to recuse myself from those conversations.
With that, Jeff, would you like to lead off the discussion?
Agenda Item: Review of Agenda - Jeff Blair
MR. BLAIR: Yes, thank you.
Just a little bit of background, so we can put our day's activity into context. In August of 1996, when the Health Insurance Portability and Accountability Act was passed there were directives to the NCVHS to study issues related to the adoption of uniform data standards for patient medical record information, and the electronic exchange of that information, and to make a report to the secretary four years later, by August 2000.
The committee did so. That particular report set forth the framework that we would use to go forward in selecting and making recommendations for PMRI standards. It set forth an initial set of guiding principles for the selection of specific PMRI standards. And it had a set of recommendations which I'm going to repeat, but the report is available at the NCVHS Web site if somebody wishes to review those recommendations.
The report was the foundation for the first set of specific PMRI recommendations which were set forth February 27, 2002, for PMRI message format standards. And we began the process of studying, evaluating, selecting, and recommending PMRI terminologies last August.
Last August we wound up considering issues about the scope, and refining the criteria of selection for PMRI terminologies. And we wound up having additional discussions in October. And by the December timeframe, we had evolved to the point where we could agree on a set of questions that we would be asking PMRI terminology developers to answer that would reflect the criteria for selection that we have.
So, those questionnaires were sent out on January 6, with the deadline of February 14 for responses. There were some that came somewhat later. We have been able to analyze -- and Walter Sujansky, our consultant, will be able to take you through this -- we have been able to analyze 38 of 41 questionnaires that have been returned to us. Today is the day when we will receiving a report from Walter that reflects the compilation and analysis of the information from the terminology developers.
In May, May 22, we will be looking for critiques and comments on the terminologies that we are focusing on by users of these terminologies. And then our process goes on into June. If we need additional expert industry testimony to clarify any issues, we can wind up doing that in June. We also hope to be able to have our first draft of our recommendations to the secretary in July. We would refine them in August, and we hope to bring them to the full committee for approval by September of this year.
With respect to the agenda for today, Dr. Sujansky is going to be presenting his compilation and analysis, and a number of issues that he has identified.
I would indicate that in addition to addressing issues such as what should be a core set of PMRI terminologies that come close to meeting the criteria that the NCVHS has set for these terminologies, we still have open issues as to whether or not we should consider other domain terminologies or not, and having a little bit tighter agreement with respect to the infrastructure and mapping between the core and other terminologies.
Other issues that we look forward to discussing today would be what types of areas do we need additional information, either from the PMRI terminology developers, or that would be a beginning of the questions that we would ask the testifiers on May 21, who are the users of the terminologies.
And then finally, there is one other issue that has come up that I look forward to discussing later today, and that is there has been a couple of different federal agencies and other professional associations that have expressed interest in the body of information that we have collected so far, and we need to come to agreement as to what are the policies and practices to address that issue.
With that, let me turn it over to Dr. Walter Sujansky, and he will take us through the presentation.
Agenda Item: Discussion - Statement of Objectives - Walter Sujansky, M.D.
DR. SUJANSKY: Thank you, Jeff. Again, it's nice to be back here before the subcommittee, and I look forward to today's activities. I see today is very much a working meeting for us, so I hope that there will be lots of participation by everyone, and it will be very interactive.
In terms of the objectives that I see for today, of course we will be reviewing, as Jeff mentioned, the compilation and analysis of the questionnaires that we received. And this is the self-reported information by a number of developers of the terminologies that are under consideration for the standards.
And specifically, we will be looking at those data in the context of the scope and criteria that we previous specified for selecting PMRI terminology standards. So, we will be returning to that scope and criteria document and content a bit. And as we go through the results, points and issues will come up with respect to that scope and criteria that we'll need to revisit and resolve in some cases.
I will be presenting some early and tentative in some cases, conclusions regarding the characteristics of the terminology with respect to the scope and criteria. And I will be in a sense, bouncing that off of the subcommittee, and looking for feedback and validation of my conclusions based on the data that I have seen.
And at the end of the meeting, again, as Jeff mentioned, hopefully, we'll feel commonly comfortable with the conclusions that we have at this point based on the data we have. And we'll also be able to identify where there is missing information, maybe where we need to discuss issues a little bit more, and what kinds of information we will need to seek from the users of the terminologies, and other parties in the subsequent steps to this process.
Agenda Item: Brief Review of Scope and Criteria for Selection of PMRI Terminologies
DR. SUJANSKY: So, before I launch into the actually data and results and so forth, I also wanted to briefly review the scope and criteria document that we prepared together before, because that is really the context for the analysis that I did.
And I'm sure most of you remember this, but just for the record, and it's been a few months, and for those of you who don't, in that document we specified that we would be selecting what we called a core terminology group for the standard. A set of terminologies that covered the domains of interest, and were technically sound, and internally consistent, and would meet the requirements for terminology standards for patient medical record information. Our goal was to identify a group of terminologies, as opposed to one terminology, that would meet those goals.
We also discussed the need to identify certain important related terminologies that would not be part of the core terminology group, but that were report because they were in widespread use, perhaps not even for clinical purposes, but including for clinical purposes. And some comment or relationship of the core terminology group with respect to those important related terminologies would be or important in order to help facilitate the adoption of the terminology standards that the subcommittee was going to recommend.
So, that's just a little history of what we had decided before. And I should tell you that that document, and those decisions are really the point of departure for the analysis that I did that we are going to be talking about today. If we need to revisit that, revise that, that's of course something we can do, but that should be explicitly discussed.
Agenda Item: Review of Questionnaire Response Data
DR. SUJANSKY: So, without any further ado, let's launch into the data. The contents of the presentation I'm going to go through on the data -- by the way, I know everyone received a copy of the printed report that I also prepared, but because of the late date that that was sent out, I'm sure most of you haven't had a chance to certainly review that in detail.
So, most of the information in there will be repeated in a sense, in this presentation, and we'll be able to go over that, and I'll be able to take your comments, and we'll be able to discuss it interactively. So, apologies to those of you who have looked at it in detail, and have digested the entire document, because a lot of this will be boring and a recap for you then.
Nevertheless, the contents of that document that we are going to be talking about today, we'll start with a brief summary of the questionnaire, and the response rate to the questionnaire that was sent out in January. A categorization of the respondents, just a way to organize the terminologies for which we got information.
I'll review the methodology I went through to compile and evaluate the data, and talk a little bit about the quality of the data. And then we'll spend a lot of time talking about the actual numerical scoring that was applied to the responses that were received. And that scoring was applied with respect to what we earlier specified in the scope and criteria document as the essential technical criteria, as well as additional desired criteria that weren't essential, and then some desired organizational and process criteria.
I'll put forth some of my conclusions based on that scoring and that data. And then we can talk about the next steps.
So, I know again, most of you have seen the questionnaire, but just to recap, it was a fairly lengthy questionnaire that we sent out. I have to give a lot of credit to everyone who responded to the questionnaire, and took the time to respond to it, all the terminology developers. There were over 100 questions.
For comparison, I was looking back at the questionnaire that was sent out when message format standards were being evaluated, and there were 40 questions in that questionnaire, so this was over twice as long, 13 pages to fill out.
Of those 100 questions, 60 of them were what I termed unconditional questions. In other words, they should have been answered. Regardless of whether they answered any other questions, a respondent could have, and should have answered 60 of these. The remaining questions are conditional in the sense that if a particular question was answered in the affirmative, then perhaps examples or further information were required in a subsequent question.
Multiple choice and free text responses were sought. And there were generally seven sections. I won't go through all of these, but they spanned the gamut of the kinds of information that one could seek from the terminology developers. And they were targeted at the kinds of information we needed in order to evaluate the terminology with respect to the criteria that we had previously identified. I'll go through that relationship a little bit as well.
Jeff Blair sent the questionnaire out to 45 terminology developers. Now, that number, just to be specific about that, I believe we sent the questionnaire out to more like 60 people, but they represented what turned out to be 45 terminologies. So, we didn't know that at the time.
The way the invitation was phrased was send us information about any medical or clinical terminologies that you have developed. And when it got it back, it turned out that there were 45 such terminologies that we were trying to get information for.
We received information for 38 of those terminologies in the time frame that we could analyze them and include them in this analysis. Information about three terminologies was submitted too late for analysis. We didn't get these until the first week of March or so, and the analysis was already underway at that point.
And then four of the recipients didn't respond. I should clarify that the ADA actually submitted information for four other terminologies, but we were apparently seeking information about some additional terminologies, which may or may not have been required or relevant to our goal. And then there were a couple of others that we didn't get information for at all.
When I got all this data, I was a little daunted. As you can imagine, we got 38 responses to a 13 page questionnaire consisting of 100 questions. So, it was quite a bit of data. So, the first step was to try to partition this mass of information in some way. So, I categorized the terminologies that came in based on some of the information in the responses of course, and from my prior knowledge of them.
I would emphasize this was really just for organizational and purposes of comparing and organizing, and just trying to get my arms around the problem. So, there are other ways to categorize the terminologies than what I did, but I hope you will find this reasonable at least. And we can discuss it if another categorization might be useful for the purpose. But I would just keep in mind the ultimate goals here, and not worry too much about the exact categorization.
So, I'm going to go through the categories and the terms, just so we get familiar a little bit with what specifically which terminologies we got data for. So, in a category that I call general, which was characterized by content that spanned a lot of different domain areas -- diagnoses, signs, symptoms, medications, procedures, and so forth -- there were four terminologies that had data across a lot of different domain areas -- MEPS, MedDRA, NCI Thesaurus, and SNOMED-CT. We got data for all of these.
And in the lab category we got ISBT, International Society for Blood Transfusions, we got their terminology as well as the LONIC terminology. Now, clearly LOINC has observations in addition to lab, but about four-fifths of the concepts in LOINC are lab concepts, so I put LOINC into the lab category.
The FDA sent information about NDC codes, as well as other dose forms and so forth, data they have, NDDF Plus from First Data Bank. And then these two related drug terminologies, NDFRT from the VA, and RX Norm. That's information we got specific to drugs.
MR. BLAIR: What does NDDF stand for from First Data Bank? Because that was one of the acronyms I remember when I was reading through, I had to double check again, and I thought that some other folks might have looked at that and said what is NDDF?
DR. SUJANSKY: You know, they didn't really specify exactly in their responses, but I believe it's National something Drug File.
MR. BLAIR: Okay, but it's from First Data Bank?
DR. SUJANSKY: It's from First Data Bank, yes. It's the one they have had for a long time. Essentially, the core of most of their projects.
And there was a nursing category, and this included the terminologies I received that either had nursing in the name or the terminology, or nursing specified in the response to the question of the primary use or purpose of the terminology. There were nine in this category. I'll just show them all here.
We see the ABC codes in the alternative medicine category. That's the only one there.
The ADA sent four different terminologies that spanned kind of a large part of the domain in terms of diseases and procedures and so forth, very specific areas such as standardized identification coding for teeth and areas of the oval cavity.
Devices, UMDNS, Universal Medical Device Nomenclature System.
And then I created a class for the messaging terminologies. Obviously, the actual concepts in terms in these terminologies spans a lot of different domain areas, and it is arguable whether this was appropriate to put all of these into one category.
But I chose to do so, so that we can focus a little bit on whether the messaging terminologies that we received are suitable and needed for inclusion in a terminology standard for PMRI, given that messaging standard have already been selected. And this is something that we should discuss later. For that reason, I put all of these into one category.
The other distinction is that the concepts and terms here are really specifically designed to support messaging, as opposed to general purpose uses of controlled medical terminologies and medical records, and for other purposes. And all of the respondents stated that as such.
And then there was a class I called statistical abstraction and administrative. Again, these are categories in many cases that have been used for clinical, for medical records, and clinical purposes, and clinical data analysis, and so forth. But they are really designed for statistical abstraction, in many cases for administrative purposes, for billing and so forth. And for that reason, I categorized them all together.
And then there was the miscellaneous, in a sense where not enough information was provided about the terminologies to classify them accurately in one or the other categories, or there was no other category that really was appropriate, like the provider taxonomy code set. It's really a category unto its own.
So, that's it, that's the 38 terminologies that we received information for. And just a review of the categories then, because these will come up again in the analysis and in our discussion. There are 10 categories then. I looked at the terminologies within and across these 10 categories.
DR. COHN: As you go through the list here, obviously one observes that there is a lot of overlap between all of this stuff. That certain codes have other code sets integrated in, or parts of other code sets integrated in. Is that good? Bad? An issue? Obviously, when we get to UMLS it's going to be a lot of everything, maybe not everything that we have listed so far. Is that something that you are going to be observing as you go along?
DR. SUJANSKY: Well, I think it's important in the sense of -- in two respects. It's favorable in the sense that if a terminology is otherwise not suitable, for various technical and other reasons, but it has been incorporated into a terminology that is suitable, and that terminology covers a domain area, that's important. And that may be favorable. An example of that may be the inclusion of some of the nursing content into SNOMED-CT.
In other cases there is overlap of content, not necessarily inclusion of one terminology within another. But there are certainly a lot of areas of overlap of content, and that can be a problem. So, it kind of cuts both ways.
Did you want to comment on that?
MS. HUMPRHEYS: Yes. I just wanted to say that it seems to me that in terms of the designation of the core set, that the chief issue with UMLS, since we will include everything in the core set in the UMLS, the chief issue is whether there would be anything in the UMLS itself or in its structure, or in the way it is put together that would somehow subtract value from any of the components of the core data set.
In other words, if UMLS could not accurately represent the high qualities that led you to put the thing in there to begin with, then that would of course be a bad thing. I think that it would be our contention, and I think we could prove it that in fact that's not going to happen. We will accurately represent the characteristics of the terminologies, while adding what we consider to be certain desirable features like a common set of identifiers on top of the ones they already have, semantic types, links, synonymy between, and so forth.
So, I don't really see this as a same issue when you are talking about the UMLS or elements of the core set.
DR. COHN: I was not singling out the UMLS. I was actually thinking more of the many of the other terminologies that Walter had mentioned. I thought he offered sort of an extreme example of an inter-lingua.
MS. HUMPRHEYS: I didn't regard it as any type of a criticism. I just thought I would clarify that.
MR. BLAIR: Walter, I think it would be helpful, especially for the people on the Internet if they don't have the document, for you to indicate the names of the additional terminologies, even though we didn't have time to analyze them and compile them for comparison, which ones they were.
And in particular, since UMLS was one of them, I think Betsy really made an effort to try to accommodate the questionnaire to be able to give us meaningful information. I think that would be good for people to hear about.
DR. SUJANSKY: I assume you are specifically referring to three terminologies that came in after -- came in too late really to be included in the analysis? And as you mentioned, that included the UMLS, the response given by the National Library of Medicine for the UMLS. And Cerner submitted a response for the Multum lexicon. And also the World Health Organization submitted a response for the ICF, the International Classification of Functioning Disability and Health. Those were the three that were submitted, but could not be included in the analysis.
MR. BLAIR: And Betsy, maybe you could help us understand the way you accommodated the questionnaire, so that it would be appropriate to reflect the role that UMLS plays.
MS. HUMPRHEYS: Well, I just had to adjust some of the questions, because clearly, the orientation was around a single terminology developer. So, the issue was what are the characteristics of your terminology. And within the hundred vocabularies that are in the Unified Medical Language System Metathesaurus, only two are appropriately considered NLM's terminology. One is Medical - Settings, and the other one RxNorm. So, there were those issues.
And then there were also issues like there were questions that were clearly devised again, for a single terminology -- well, they all were really. But for example, do you map between two other terminologies? Well, since the whole purpose of the Metathesaurus is to map among the hundred terminologies, the questions don't quite fit.
But the structural questions about the identifiers and the retirement of codes and so forth, those I think are of interest to people to see how we do that, and how we represent relationships and so forth. So, we were able to put that stuff in, and cut and paste pieces of tables and stuff, so that people could see what the native UMLS format was for some of these data.
MR. BLAIR: Thank you for doing that.
MS. GREENBERG: I just wondered if the ones that were submitted too late to make this analysis will be included in the final analysis, or if it's just too late to be included in this whole process?
DR. SUJANSKY: They will not be included in the final analysis. And it is too late.
DR. COHN: Well, I think that's probably something we need to talk about.
Clem, welcome. Would you like to introduce yourself, and let us know whether you needed to recuse yourself from any of the issues today.
DR. MC DONALD: Yes, I'm Clem McDonald from Indiana University, Regenstrief Institute. I'm co-chairman of the LOINC Development Committee.
But I really was going to bring up that same thing about the three. I think I would encourage any ability to include other large vocabularies, because that will make the document more valuable as a resource in general, than were it just because of some cut off date. I should add, LOINC made the cut off.
But I just think that these are big vocabularies, and if they did the work of putting the stuff together, it would be useful. I don't know what the limitations are, but if there is a way to find a way, I think I would encourage that.
DR. COHN: Well, why don't we hold that thought, and just reflect on it towards the end of all of our discussions to see where we think the value is, and what we might be able to persuade our consult and co-lead on this project.
DR. MC DONALD: I had a second question or issue. It's whether we could consider alternative hierarchies, more than one hierarchy for classifying these things? Because these are complicated beasts. I know that doesn't sound like fun, but at least there would be some possibility of having another --
DR. SUJANSKY: It's sort of a meta-terminology problem. So, I can see why you bring that up. I would comment to that, Clem, that that's certainly possible, and may be very valuable, but I think we should discuss what the value of that would be specifically before we embark on that. And that's something we can talk about today.
DR. FITZMAURICE: In looking over the report, I'm impressed with the large number and the comprehensiveness of the field that reported. I guess the one I see missing most glaringly, not counting those who did submit, are HCPCS. The good news is that parts of HCPCS are included in the terminologies.
The bad news is that the learning that you get from filling out the questionnaire, as well as being able to describe accurately your coding system, for something that is being used for 40 percent, 50 percent of the spending in the United States, it seems to be quite a loss. But yet, this is a voluntary effort for the survey form.
I would say too that with regard to those who submitted late, it would seem to be a question of the logic of fitting them in, the time that it would take, and the resources available to doing it, and those should probably be subjects of additional conversations, as Simon said.
MR. BLAIR: Could I do this just to encourage our discussion, and I was the first one to violate this rule by asking questions myself, but we do have three of the four time slots for today to address issues. I think if we give Walter a little bit of time here to take us through the rest of his presentation, then we'll really have plenty of time. I don't mean to completely shut off specific questions for clarifications as he is going through, but I think we are going to have a lot of time to be able to dig into these issues during the day.
DR. SUJANSKY: Okay, that was sort of the boring bookkeeping part of the presentation, kind of going through the list of respondents, and the categories and so forth. So, I think it will get more interesting after that point.
Briefly, just to review the types of organizations that submitted. And again, I'm sure there are different ways to categorize these as well, but by far, almost half of the responses were from terminologies maintained by professional societies and trade groups, whether it's the AMA, ADA, or HL7 and so forth.
The US government comprised eight of the respondents. And then the remaining ones were split pretty evenly between academic, non-profit, private, and for-profit, private entities.
So, let me review quickly how I went about evaluating this data. The first thing I did was took all the data out of the document format and transcribed it more into a spreadsheet format so that it was easier to review and compare across terminologies.
Then I assessed the quality of the data collected from the point of view of completeness, and I could do that quantitatively. And then accuracy, which was more subjective.
And then I numerically scored the responses with respect to a set of four essential technical criteria that we identified before. This was again, to help scope the task, and carve out those terminologies that met these essential criteria. And the ones that did not meet those criteria were excluded from the remaining scoring and evaluation at this stage of the selection process.
The ones that did meet those criteria, and I actually applied those criteria generously, to be as inclusive as possible, since the ones that were excluded, that were not going to be considered in the rest of the evaluation.
So, the ones that met the criteria were also scored with respect to the other desired technical criteria, desired organizational process criteria. And then the results of that were analyzed to determine what information may be missing and still needed, and what the next steps were.
So, just quickly in terms of transforming this into a spreadsheet format, the only reason I'm going to go through this is because the spreadsheet format I'm going to make available to anyone on the subcommittee who might be interested in really reviewing the primary data. So, just to familiarize you a little bit with that format in case you are interested in looking at it, this is how the information came back in these documents that we sent out.
And then we took it, and basically transcribed it into spreadsheet form here, where the same questions are now down this column, and the responses -- here you see two different terminologies side-by-side -- they are all spread out this way.
And in many cases, the information in the Word document really couldn't be put into a cell of a spreadsheet. In that case, a link was provided back to the actual source document. So again, I mention this in case there are those of you who are interested in reviewing this information.
I'll just quickly show you how that works. So, for example, here in providing three examples of how the concepts are represented. This allowed me to quickly go back and review the information without having to shoe horn everything into this spreadsheet. So, again, for those of you who are interested in doing this yourself in your spare time, this data will be available. It may be an interesting resource for lots of other purposes besides just the selection process.
Are there those of you who are interested in this? Can I just sort of see if there anyone? Okay. So, I'll make this available. I'll work with Suzie and so forth, to make this available to everyone. Actually, I can put it just in a Zip file that you can download, because all the terminologies have to be in there too, about two megabytes, and then we can make it available to everyone.
MR. BLAIR: I think that that is also pending the discussion that we will have later today in terms of whether some of the information may be proprietary or confidential, and what policies we'll have for dealing with that.
DR. SUJANSKY: Okay, that's a good point, yes. Certainly, that information shouldn't be passed on further than the subcommittee. And then we can discuss even within the subcommittee, in what format it should be distributed.
So, then in terms of the data quality, the completeness of the responses. So, recall that I mentioned that there were 60 what I called unconditional questions that everyone should have and could have answered. So, with respect to those, across all 38 respondents, about 83 percent of those 60 were answered on average, which is not bad for a lengthy questionnaire. And there were some outliers on the low end, so the median number is actually 88 percent, almost 90 percent.
There were two respondents who answered fewer than 50 percent, but that was really rare, with the low point being 32 percent. And one respondent answered 100 percent. But there were many who were in the 95 plus area. So, the responses were very good, and again, I was impressed with the time and care that people took to answer the questionnaire.
Most of the follow-up questions were answered appropriately and fully, but there were some areas where important information was omitted. This happened especially in terms of some of the licensing terms, where there cases where respondents answered that they didn't have standard licensing terms, but they didn't go on really to describe their non-standard licensing terms at all. So, there was really no information.
And there may have been confusion. Maybe what they were trying to say is that their terminology is free, and there are really no licensing terms, and you can just download and use it. But that wasn't explicitly stated, so further information may be required in some cases, as we'll discuss later. But again, in general, the responses were quite complete.
Accuracy -- now, you can't tell in general how accurate the responses were, but sometimes you can tell when they aren't accurate, when they are inconsistent, internally inconsistent or otherwise a disparity between that point and what I was otherwise aware of regarding that terminology.
And there a few questions where this cropped up more than once, so in response to whether they distinguish in terms in their terminology, a number of respondents said they did, but then in fact in the examples it was clear that they didn't.
How is the meaning of each coded concept represented? This was a multiple choice question. One of the answers was with formal definitions. And several of the respondents answered that they did use formal definitions. But again, in the examples it was clear that they didn't, and text descriptions were used alone.
And then there were a lot of vague answers to for example, describing how the proposed content changes are prioritized and so forth. But again, in general, the responses were quite complete and quite accurate, and certainly sufficient for scoring and making determinations. So, that was good. That was very favorable.
So, as I mentioned, the next step was actually applying some numerical scoring to get a handle on these 38 terminologies across this host of criteria that we had previously established. So, the way the scoring worked was first I identified the selection criteria that were amenable to numeric scoring. And they weren't all, but most of them were amenable to that.
Then we identified the questions in the questionnaire related to those criteria, and then developed usually a three level scoring metric for those criteria, and then applied that to each terminology where it was possible. And then computed the totals, and looked at the results.
So, for example -- again, this is in the paper report -- for one of the desired technical criteria is meaningless identifiers. So, three levels of compliance with this were identified, the first being identifies represents -- the terminology got a zero if the identifiers represented the concept term position in the hierarchy, or the meaning of the concept term in any way, that was not a meaningless identifier.
Terminology received a two if all the identifiers were meaningless, and an intermediate score of one if some of the identifiers carried meaning, although other perhaps alternative identifiers were meaningless. So, there were some terminologies that had that characteristic of actual two different identifiers for the terms and concepts.
And then the questions were identified. This is just kind of going through the minutia of how I did this. This is all in the report as well, for those of you who are interested. And the responses to these questions were reviewed in order to derive the score.
So, the first set of criteria that this was done with was the essential technical criteria. And I will just review these actually here, maybe in a little more detail. Then I'll go through some of the others, because these were used to exclude terminologies from further consideration.
So, the first of these that we decided to apply was concept orientation, namely, the elements of the terminology or coded concepts, possibly multiple synonymous text representations and hierarchical or definitional relationships to other concepts. There are no redundant, ambiguous or vague concepts in terminology, or none that are not recognized. The alternative to that is that the terminology is not concept-oriented, and those features don't apply.
Concept permanence was also an important distinguishing criterion. And this of course means the meaning of each coded concept in the terminology remains forever unchanged. And if the meaning of a concept needs to be changed or refined a new coded concept is introduced, and no retired codes are deleted or reused. The absence of that, of course, is scored as zero on that.
Non-ambiguity, each coded concept in the terminology has a clear, unique meaning; that's also very important. And explicit version IDs. Each version of the terminology is designated with a unique identifier so parties exchanging data can readily determine if they are using the same set of terms, and the absence of that is scored as zero.
So, without going through all the details of exactly how and why each score for criterion was applied to each terminology, let me go through the results of applying the essential criteria. What this resulted in was really a partitioning of these 38 terminologies into three groups.
The first group is those terminologies that, with high degree of confidence, met the essential technical criteria. There were 10 terminologies in that group, and we'll see these again and again, so I'll just name them here: MEDCIN, MedDRA, SNOMED-CT, NCI Thesaurus, LOINC, NDDF, RxNorm, SNODENT, UMDNS codes, and the HL7 Version 3 codes.
The second group were ones that failed to meet the essential technical criteria, again, with high degree of confidence. You will notice that there are a lot of question marks in the concept orientation column here. That criterion was the most difficult one to apply, because it is somewhat subjective. And I'll go on and discuss that a little bit more, but the end result was that I assigned a lot of question marks there, so as not to unreasonably exclude certain terminologies from further consideration based on a simple application of concept orientation.
But if you will notice among these, most of these were excluded because they failed to meet the concept permanence criterion. Based on the responses given, codes and concepts were removed, their meanings were changed. I think there was one where they even reused concepts. So, for the most part, that was the really exclusionary criterion for these. I believe there are 15 terminologies here that were excluded on that basis alone.
MR. BLAIR: For the benefit of the folks on the Internet, and for the blind guy in the audience here, will you just read those?
DR. SUJANSKY: Yes, I'll read those. So, the FDA's NDC codes, the International Classification of Nursing Practices, MANDA, MIC, NOC, Omaha System, Perioperative Nursing Data Set, the ABC codes, codes on dental procedures and nomenclature, Universal National Tooth Designation System, Core Health Data Elements, CPT, DSM-4, ICD-10-CM, and the Provider Taxonomy Code Set were excluded. I believe there are 15 there.
And then the third category were ones that were indeterminant, again, primarily based on the concept orientation. Now, at this point I'll talk a little bit more about this criterion, because it may or not be something we want to discuss a little bit more.
Initially, I went through, and terminologies that explicitly stated that they did not distinguish concepts and terms, and they didn't have synonyms, which comes hand-in-hand with that, I initially excluded those, and said those were not concept-oriented.
I later decided to expand that definition, looking back through some of the literature and so forth. The distinction between concepts and terms isn't necessarily required. What's required is a clear definition of a concept, with no redundancy and ambiguity. And so, theoretically you could imagine that there is a code set where there is just coded terms with no synonyms, but that could be very clear and unambiguous and so forth.
And so, I basically gave these terminologies the benefit of the doubt with respect to concept orientation. Now, if any of you have any strong feelings about that, I'm certainly open to that. We may see as we go on and look at the other criteria, that it doesn't really matter that much, or may not.
If we need to revisit this, and look into these specific terminologies that may meet the criteria, but don't distinguish concepts and terms, we can go back and do that based on the specific contents of the terminologies. But I didn't want to do that in the first pass, because it's very time consuming, and it may not be necessary.
So, again, because this is so important, this part, I actually invite question at this point, since we are excluding 15 terminologies here from further consideration on this slide.
DR. COHN: As I look at this one, the one that -- unfortunately, I'm not truly knowledgeable about this particular code set for the DSM-4. I was observing that it seemed to be solely not considered because of the concept permanence issue.
DR. SUJANSKY: That's right.
DR. COHN: And I'm just trying to think in my own mind, is the concept permanence issue -- are you sure that there isn't concept permanence there, I guess is the question.
DR. SUJANSKY: Well, they stated that they changed the definitions of their concepts based as new information comes out about the various disorders and so forth that are coded there.
DR. FITZMAURICE: Depression, for example. I think the APA testified that with new scientific evidence, they have changed the definition of depression from the 1950s to the current time. And they complained that CLU(?) wouldn't change the definition. So, they felt the descriptions in the current DSM codes used by HCPCS were not accurate, and there was a big discussion about that here.
DR. STEINDEL: Simon, this also relates a little bit to the earlier discussion that we had. I think a lot of this is put into the context of what is the purpose of this report. And what I think Walter is saying, these terminologies are excluded for the purpose of this report. And I think we might discuss a little bit later what is the purpose of this report, and what are the purposes of excluding and including terminologies. And once that becomes clear, exclusion for the purpose of this report may not be such a major barrier.
DR. COHN: Okay, thank you.
MS. HUMPRHEYS: I just want to comment that from my understanding of ICD-10-PCS, I believe that that is a concept granted vocabulary. They have only one way of representing each concept within that scheme. And they have an outside index that points into it. So, the index is not actually giving alternative names of the things.
But I think there is only one way within that scheme, of representing a specific procedure. If I understand it correctly, it's by putting a bunch of things together in a coded format, which in some case you could say was almost analogous to LOINC. That is, you have to have this piece, this piece, and this piece linked together.
So, I don't think there is more than one way to say the same thing in that. So, that was your point about whether the mere existence of alternate names or not really is exactly the same thing as concept orientation. And I think you are right, I don't think it is.
DR. SUJANSKY: Yes. One of the other criteria I applied, for example, the ICD-10-CM was that that terminology includes terms that have the not elsewhere classified suffix if you will. And in my opinion, that unto itself is contrary to concept orientation. That's a vague term.
MS. HUMPRHEYS: I would agree with that.
DR. SUJANSKY: And I didn't know if ICD-10-PCS also included that, and that's why it was indeterminant.
MS. HUMPRHEYS: Well, we can look into that. I know at one point it did not have it. And I don't know whether it was later introduced or not.
DR. SUJANSKY: Okay, so, again, these are follow-up points. And I should add that we may or may not choose to pursue this further when we look at the other criteria and distinguishing criteria, and the scoring for those. So, we'll keep that in the back of our minds.
MS. GREENBERG: On the point that you just made, I guess I was reading this over last night, and I'm sort of, of two minds on this. On the one hand, I think it's useful to have these detailed questionnaires completed even for terminologies, classifications that are, as you suggested, maybe ruled out of scope for this report, because I think there is a lot of information that is included in them. And this gives us a rather rich database.
But on the other hand, I think the concepts or the principle of say not elsewhere classified is a characteristic of classifications that must cover everything, but their classifications -- and we have had this discussion versus vocabularies. You've got to put everything somewhere.
And if that was a reason to completely eliminate a candidate from consideration, maybe that should have been made clear in the first place, because it was, as you pointed out, very time consuming to complete these questionnaires. So, if the fact that you function as a classification eliminated you from consideration, maybe that should have been clear, and it would have been up to the developer as to whether they wanted to take the time to fill this out. If you're a classification, then you have that principle.
MS. HUMPRHEYS: There is another way to do it, though. You don't have to have an elsewhere classified. There is another way to do it.
DR. STEINDEL: Marjorie's comment just led to a second comment. I have two comments to make. The first one was the thing that impressed me about this table that we are showing about the terminologies that were included and excluded should be a very clear message to those people who are developing terminologies about the importance of concept permanence, and not reusing terms. And I think this is a very strong statement in that area. One thing that comes out of this report is that statement.
And with respect to what Marjorie was saying, one thing that did disturb me about this report is that we didn't go ahead and still analyze whatever number -- I think it was 13, that were excluded -- and provide information on their responses to the other questions.
And the reason why I say that is not because we should look at them from an inclusion point of view for the purpose of this report, but as Marjorie said, these people did a lot of work. And it's very beneficial to the profession to see the breakdown of the information for those other terminologies as well in similar tables.
MR. BLAIR: Let me just mention something, I think to put in perspective as I'm listening to this discussion a little bit here. Just like everything else when you go through a process, you try to strike a balance. One was to what degree we try to stay on schedule, and to what degree do we assume some flexibility or get to precise definitions.
And for those folks that might not recall or be familiar with our discussions in October and December, we really were not able to get to a point where as a subcommittee, we had a consensus on what would be the definition of a core. And so, we decided to go forward with this, without having a preconsidered agreement on our subcommittee as to what would be the core of PMRI terminologies.
And so, the analysis that Walter has given to us, I think gives us additional data that will help us today to maybe get closer, if not actually define saying this is the core, and then you have domains, and then you have other terminologies to which we map to.
And I think your point, Steve, what is the purpose of this report. I think one of the purposes of the report is to help us understand these issues well enough to be able to make those types of decisions.
The other piece that I just wish to indicate, that of the three terminologies that Walter was not able to analyze, it wasn't that there was an arbitrary deadline. It was because Walter accepted terminologies up to two and a half weeks after the deadline, if he had the capability of doing the analysis. It was only those that came in at a date where it just wasn't possible for him to do the analysis. So, just to be fair to Walter, I just thought I'd mention that.
DR. SUJANSKY: Yes, I would say, just in brief response to some of the comments, I would say that we did actually make some statements about what are essential criteria and desired criteria for the core terminology group. And those were in the report that we published earlier. So, that's not to say we can't change our minds now about some of those things, but the analysis that was done, was based on those previous decisions in that report.
The other comment is that I agree with Steve, that ideally we would have rich data regarding all the features of all these terminologies. But again, the process here was very goal-oriented in terms of moving the terminology selection process along, and focusing the resources and the time best to do that. And so, that's why not all of that data is included.
DR. STEINDEL: But the breakdown data for these excluded terminologies exists in the spreadsheet.
DR. SUJANSKY: Yes, so it would be possible.
DR. STEINDEL: And that's all that we really need.
DR. SUJANSKY: Yes, I didn't shred anything.
MS. GREENBERG: I should add here too that I think you did an amazing job with very voluminous information.
DR. SUJANSKY: Thank you.
DR. ZUBELDIA: Walter, how did you address things that don't quite fit as a terminology? I had a question on the question marks that you have like on the two designation numbers, because the concept orientation doesn't quite apply. It's just a number.
DR. SUJANSKY: Right.
DR. ZUBELDIA: How do you address those as far as concept orientation. I know you put a question mark.
DR. SUJANSKY: Well, I was uncertain as to how to address that, because you're right exactly. It's a number that represents a tooth or an area of the oral cavity. But there could be synonyms for example, for an area of the oral cavity or for a tooth. And this terminology doesn't allow for that, because it doesn't have the distinction between concepts and terms. But again, is that sufficient to exclude it? I don't know. That's why it was included in subsequent analysis.
We are actually doing okay in terms of time, because we are exploring some of the issues slated really for the second session this morning, now. And so, we can kind of mix and match those parts of the agenda, and that's okay.
DR. STEINDEL: Just in quick response to Kepa's comment. I heard a fascinating discussion by a dentist working in the area of terminology at Clem's clinical LOINC meeting. And there is a lot of ambiguity in teeth, so I could see where Walter might have had some problems.
DR. MC DONALD: There are more than 32 teeth.
DR. SUJANSKY: Okay, good.
Just based on the inclusion, exclusion, and so forth that we just discussed, these three groups. One thing to note is that among those terminologies that unambiguously or with certainty confidence meet the essential technical criteria, and I named those earlier, if you look at the categories and the general contents of those terminologies, at least at the high level, they span a lot of the domain area.
All four of the terminologies in the general area, in the general category met those criteria. The LOINC codes for lab met those. Two of the four drug terminologies met those, SNODENT, the devices, and so forth. So, there is a rich and representative set of terminologies that could be included in a core, even in the absence of the 28 others that were excluded, or may be excluded. So, that's something to note.
So, then I went on, with the 23 terminologies that met the criteria, or may meet the criteria, I went on to apply the desired technical criteria, non-essential, but desired technical criteria and score those. Again, those criteria were specified in the previous document. I'll just list them here.
There were nine of them. Seven of them were amenable to scoring. I think in the report I actually said eight, which is wrong. Seven of them were amenable to scoring. The first one was comprehensive domain coverage, and this actually was not amenable to sort of a consistent, straightforward scoring. I'll discuss that a little bit more. The others included: meaningless identifiers, multi-hierarchies, non-redundancy, formal concept definitions, infrastructure tools for collaborative development, changed sets, mapping to other terminologies.
The ninth one was support for local customization. And I believe we didn't actually ask any questions about this in the questionnaire, and if that turns out to be important, it may be something we want to go back and review, if it's an important distinguishing feature. But we didn't have data for that, as it turned out. So, I won't go through all the scoring criteria for those. We can come back to that if it's relevant.
Now, domain coverage, before we go onto the actual scores, I didn't feel this was really -- even though we had data as to the number of concepts and terms in the terminologies and the various subparts of the terminologies, and so forth, I think there is a lot of context that needs to be applied to domain coverage.
And the adequacy will vary from terminology to terminology, depending on the domain area that is intended to be covered by the terminology, as well as the domain area we may choose, the subcommittee may choose to be covered by that terminology if it is included in the core terminology group.
So, the other important thing is that sheer numbers aren't the whole story in terms of domain coverage. It's important to consider the level of abstraction that a terminology may include, and whether that is appropriate to patient medical record information. And I'll talk a little bit about that in a second.
So, some of the things to note here, in this general category, there is a large range of just in terms of numerical, number of concepts and terms, and so forth. I got an e-mail from Frank Hartel(?), mentioning that he didn't feel NCI Thesaurus was appropriately categorized in general, because it is specific to oncology and so forth.
It was categorized in general, because it includes concepts across much of the domain of medicine, but clearly it is used in the oncology domain primarily. There are symptoms and findings and procedures and diagnoses and so forth that are applicable beyond oncology. But its scope is one reason why it has relative fewer concepts and terms. So, that's one point I wanted to make based on this slide.
The other one, apropos of my earlier comment about really the purpose of a terminology being important in terms of assessing its domain coverage, an example of LOINC. As I mentioned, LOINC includes about 80 percent of the concepts are specific to lab, and 20 percent are other types of observations.
If we decide LOINC is a good terminology for laboratory tests specifically, we might say that the domain coverage is excellent. If we decide it's also an appropriate single terminology for representing all other observations, then we may decide the domain coverage is not adequate and not appropriate. So, it really depends on the context and the intended use.
The other point I was making was about level of abstraction. If you compare the first two here, NDDF Plus and RxNorm, the numbers are different by a factor of 12 or so. However, NDDF includes all the NDC codes, for example, as well as various other levels of abstraction. NDC may not be appropriate to patient medical record information. It may be the wrong level of abstraction there.
And RxNorm in fact, with only 40,000 concepts, may include all the important concepts needed for medical records. Again, all this remains to be determined, but the point is that it's impossible to make a simple statement about domain coverage from the numbers.
Now, as I mentioned, there were seven criteria that could be scored. I won't show a table like I did earlier for all the scores for the 23 terminologies. I don't think that will be meaningful, but we'll go to the totals here. And I just want to point out a few things.
So, there is a maximum score of 14 for those 7 criteria. So, a few things to note --
DR. COHN: Walter, could you go to the previous slide for just a second? I just had a question about it. This had to do with the concepts versus terms. And you can appreciate that people have more terms than concepts. But it seems to be reversed here for a number of the terminologies. I was going to ask if you could maybe explain that.
DR. SUJANSKY: What's up with that?
DR. COHN: Yes.
DR. SUJANSKY: I don't know. That's the data they gave me. When I first made this slide I put sic next to those, just to note that that's what they gave us. Either they just switched them accidently, or their definition of concepts and terms may not be quite right, or there is something else going on. So, that might bear further examination.
MR. BLAIR: Walter, do you have a separate listing of items like that, where we will be able to review them for trying to get additional information or clarification from the developers?
DR. SUJANSKY: Yes.
DR. MC DONALD: The lab 25,000 terms, that's the set of LOINC that is a lab test, not the total set of terms in LOINC.
DR. SUJANSKY: I'm sorry, say that again.
DR. MC DONALD: There are 25,000. I think you said it right. But that's the 80 percent. The 25,000 number is the 80 percent.
DR. SUJANSKY: That's right. Did I say something different?
DR. MC DONALD: No.
DR. SUJANSKY: That's a lot.
Again, I mentioned just showing the aggregate scoring here for the desired technical criteria with the maximum of 14. The first thing to note is that the scores in these other criteria -- again, these are not the essential technical criteria that we use to designate the members of these two groups.
But the scoring on those other criteria is also quite different between the two groups, possibly meeting the essential criteria than not. You can see the average being 10 for those that definitely meet them, and 4 for those that don't.
Again, I didn't rank these, because it doesn't really make sense to rank them with respect to each other, because they cover different domain areas generally. But you can see some of the general scores there.
So, the third area that was numerically scored was the desired organizational and process criteria. And these really include things that were amenable to numerical scoring, and then a lot of things that weren't amenable, that require more of a qualitative analysis.
As we stated in the earlier report, this includes things like governance structure, as well as responsiveness, and is it consensus-based or not consensus-based, and what's the funding mechanism. Is it viable in the long run, et cetera, et cetera? So, I distilled these into six really criteria that could be numerically evaluated.
The first three have to do with licensing costs and intellectual property restrictions. So, obviously, low licensing cost is good. Few intellectual property restrictions are good. And not requirements for third party platforms and tools are good. And with respect to each of these there could be a zero, one, or two score, again, with a maximum of six.
So, these 23 terminologies were again, scored there. Let me just point out a few highlights here. The first one is SNOMED-CT. I again, gave it an indeterminant score in terms of low licensing costs and intellectual property issues, because it's TBD basically.
Obviously, we are aware of some of the activity in this area with respect to providing low licensing costs and few intellectual property restrictions, but that's not a done deal yet to my knowledge. So, until it is, I have kept that as indeterminant, and we'll see what happens. So, that's the first point.
The second point is on these general terminologies, which I think will be very important to the core terminology set. SNOMED of course is in there. But two of these MEDCIN and MedDRA really don't have very favorable licensing and IP features. MEDCIN of course is a proprietary, for-profit -- the developer is a for-profit company. And MedDRA has some licensing costs, as well as intellectual property restrictions.
In addition to SNOMED though, the NCI Thesaurus actually has very few restrictions and low costs, but as we saw, the domain coverage maybe is not really comparable to SNOMED. So, if things with SNOMED don't work out, I think we'll be hard pressed to find an alternative in the sort of general category.
Yes, Stan?
DR. HUFF: It's true that MEDCIN is a for-profit company. But the last I knew, you could use their terminology basically without restriction for $25.
DR. SUJANSKY: I think they have per site -- that's the per site fee. You can use it at IHC in your work for $25, but if I'm a vendor and I have 500 customers --
DR. HUFF: Each one of them would buy it for $25.
DR. SUJANSKY: Each one of them would buy it for $25.
DR. HUFF: That seems pretty low to me.
DR. SUJANSKY: Well, to get a two, you had to be nominal. We can decide if that's nominal or not, and what the licensing terms -- I thought the cost was actually more than that. We can actually look at their response right now, if we want to clarify this on the spot. And you may well be right.
DR. HUFF: But there are different things too. That is, there is the terminology. And then if you want their knowledge base, that's an entirely different thing.
DR. SUJANSKY: Yes, $25 per site per year.
DR. HUFF: That seems pretty nominal.
DR. COHN: And a site is defined as IHC? Who knows?
DR. MC DONALD: There are three issues I would like to bring up here. One of them is it would be useful when you are talking about cost, to have another category of no cost, but that's unarguable. If it's no cost, it's no cost. And then the nominal we can get in these arguments, but I think it's still a good category. I think it's still important to distinguish the nominals from the big deals. But having one that is no cost. And there are a number of them now that are, is a nice highlight.
The second thing is when you say intellectual property, there is a distinction in vocabularies, and you may have been being nicer to some of these codes than you meant, but I think it should be there. Derivative works, if you are saying you can use it in a program, a vocabulary, in some sense that's a derivative, and that should absolutely -- you would want that. You would want that to be without constraint.
But if you mean by derivative work a third variant of the code, it is a little more subtle deal. In open source, you allow that. But if everyone can make derivatives, where is the standard? So, there is a subtle line there about holding back on derivatives of the code system itself. Whereas, you want them to be able to incorporate it in software freely. People may get confused about that.
DR. SUJANSKY: You're saying that may be a bad thing, rather than a good thing?
DR. MC DONALD: Well, I think you want to make a distinction between what you mean by derivative. When you said you could do derivative works, that's not true for some of the case systems that you listed there. If you are meaning by that you can make derivative code systems, that is, you can make version 7, 9, 10, 57, so it would have 500 versions of the code system. That is a derivative work thing, which I think in many contexts you don't want to encourage.
It's tricky, because there may be extensions you would want to encourage. But certainly, derivative works in the sense they incorporate in software, you would want to have no restrictions on that.
DR. SUJANSKY: So, the point you are making is that the question was perhaps -- that we asked the question that was not specific enough.
DR. MC DONALD: I think you probably answered it with some interpretation. I think you probably categorized things in a rational way, but I think you've just got to get clear in your own head.
DR. SUJANSKY: We may not have the data. The question we asked was may licensees make derivative works of the terminology without further compensation?
MS. HUMPRHEYS: Yes, and the question is -- I believe that Clem is correct. There can be a wide variation on what is a derivative work. For example, you may say that if I produce a table that uses one of these terminologies on one side, and then maps it to something else, which may be used in my software or something, and I distribute this to everyone, then in some cases people would say that was a derivative work.
Or if I am creating a drug information knowledge base, and in my drug information knowledge base I wish to include for each drug that is in there, a standard name and code set from X, Y, Z, then somebody could say well, that's a derivative work, because it is built around my vocabulary. And in some cases, you would say that's great. And other people would say no, you couldn't do that without a license.
And all of those would not be a problem. In fact, I suspect we would like to promote them. But the issue that I should suddenly decide that I was going to put out version 1.6 of their vocabulary, with slight modifications of the meanings, that's the kind of derivative work you don't want.
MR. BLAIR: Stan, with your comment it made me think of an issue, and I don't have an answer to this, but I would like to raise it as one of the issues we begin to consider later on, which is there are some non-government, non-professional association entities that have developed terminologies.
And there is some degree of acceptance in the marketplace for those, even if they may charge a modest price. Or if they could be free or very low cost, and still require the purchase of a terminology server, which is an additional expense. And I don't recall whether our questions really got to that.
And I'm kind of going down a thread here is an issue, because I don't know that this is necessarily a problem when you balance it off with other things. But even if there is a particular company that might require that you purchase their terminology server in order to be able to use their terminology, if the value of their terminology is very great, and there is acceptance in the marketplace, then I don't think that they should be disqualified either because they charge a license fee per site, or because they necessarily charge for using their terminology server. I just think we ought to be aware of these, and put that into the balance as we begin to look at what's there.
DR. COHN: I had a request from one of our staff and members here to have a break. Is there a final question before we break at this point?
DR. MC DONALD: What we are leading to is some recommendation in terms of a national recommendation for one way to do something in some domains. And in that context, and I have said this many times before, you won't be surprised, I think we ought to push as hard as possible on no cost, or low enough that these things would be not be constrained, and could be used by many vendors throughout, and there would be no risk of inflational costs, and all kinds of other traps and hooks.
That's the reason why we worry it in the cost. If it's just let the marketplace decide, there is no problem at all.
DR. COHN: Okay, well, with that thought, why don't we take a 15 minute break, and we will continue after.
[Brief recess.]
DR. COHN: Okay, Walter, I think that you have the floor at this point.
DR. SUJANSKY: So, I'll just pick up where we left off. When we left off we were talking about licensing costs and IP restrictions. And we just reviewed the relative scores with respect to these in the general category. There is some other information that I think is useful that comes out of this.
If we look the Universal Medical Device Nomenclature System, there are some significant licensing costs and intellectual property restrictions associated with this one. So, we need to think about the role of that in the core terminology group. If those costs and restrictions are acceptable, as we were talking about earlier. As Jeff was mentioning earlier, there may be some role for that. And also if the relevant aspects of that terminology are covered in other terminologies with different provisions.
DR. STEINDEL: I believe the FDA has been trying for a while, to negotiate the use of that.
DR. SUJANSKY: Oh, okay. How is that going?
DR. STEINDEL: Slowly.
DR. SUJANSKY: Okay. Let me put it another way. What is the likelihood that any progress will be made there in the timeframe of the subcommittee's recommendations?
DR. STEINDEL: I think within the timeframe of the subcommittee's recommendations, none.
DR. SUJANSKY: Okay.
DR. COHN: I want to just reflect on this one, because I think this is an important area. But there is an underlying issue here, which is that developing and maintaining good terminologies is not a no cost activity. So, I think probably what we are seeing underlying this is a couple of different financing methodologies.
One way is to have the government provide the funds, either through grants or other things to create and maintain a code set. And I think my understanding is LOINC is maintained in that fashion. And so therefore, yes, it can be free, but obviously other people are paying money in other ways, and there are other financing models.
Obviously, this is probably not a killer here, but clearly if we are going to be recommending anything where there are major licensing costs involved, part of the recommendation clearly needs to be government involvement in ways to underwrite the costs of maintenance of that terminology. And I think we are already seeing that in some other vehicles. So, I don't know if this is a show stopper, but it's certainly part of that underlying financial issue.
DR. HUFF: I would agree. And I think the other principle there is that you don't want this licensing to be done in a way that gives a government-mandated monopoly to a company, which can then take undo advantage of the constituencies.
DR. SUJANSKY: Right. I think the important point is these are criteria among many criteria. They are not of that exclusionary nature that those other ones we talked about earlier are. And everything in whole needs to be taken into consideration. And there may be a role for terminologies in the core terminology group that do have significant licensing costs and restrictions, if it worth it, and if funding for those can be arranged in some way that makes it acceptable.
Then on the drugs, if you will, terminologies, there we see again, NDDF, as we know is a proprietary terminology. It does entail significant costs. RxNorm scores very well in the licensing costs and IP restriction area, and the NDF terminology, again, which I believe is one that will pass the essential criteria. So, it's a technical reason that it's not included at this time. It also scores fairly well. But of course, RxNorm and the NDF are interrelated. So, there are some distinctions that are worth recognizing in terms of licensing costs.
And then SNODENT -- again, I think we should discuss the whole dental area a little bit area -- but SNODENT, it wasn't clear from the responses whether SNODENT is or will be part of SNOMED as far as licensing is concerned. I don't know if anyone on the subcommittee has any information about that. But again, if the dental domain is crucial, that's something we may want to look into.
Another point here in this data is that there are really very few terminologies that have requirements for third party platforms and tools. So, that really wasn't a distinguishing criterion of any note. Really, it's licensing costs and restrictions that are more important.
DR. COHN: Walter, can you help me with this one? I see no requirements. I'm trying to think of what a requirement would look like here. If you actually have to buy a server to actually be able to license terminology? What are you referring to?
DR. SUJANSKY: Yes, to be able to use it. For example, if it were only distributed in a binary format, that you needed access through the API of a terminology server. That's a restriction. I think the NCI Thesaurus said -- without going back and looking at the specific data -- that there was a requirement to use a terminology server, though it mentioned there were some public domain ones that were available.
So, it's a cost issue in the sense of not a monetary cost, but one does need to go out and get a specific piece of software, kind of software in order to use the terminology, which is a cost of sorts.
DR. COHN: Well, the reason I'm asking is I'm just sort of reflecting that some of the terminologies that we have identified are so large, you sort of need something to be able to use them very effectively. So, that's an issue number one.
Number two is that in some of these terminologies, really their value comes out because of their knowledge base capabilities. So, I'm having a little trouble with the universe on this one.
MS. HUMPRHEYS: As someone who recently filled out the questionnaire -- too recently -- the issue was, was it mandatory for you to use somebody else's proprietary product, or did you have to get another tool in order to use this. And I think in general, the answer to this is no, which is a good thing.
Because it's not that anyone can just use the terminology lying on the table without software, but if the question is obviously the terminology were free, but it required a $1 million piece of software, and a particular one, so you couldn't even develop your own or shop around, then that would be a negative. And that's you were getting at.
DR. MC DONALD: There was actually a good example of that kind of licensing entrapment, if you will, from the REED(?) codes. That they were free. You could get them from the queen. But you couldn't use them without an installation fee, which was fairly substantial. So, I think it's a good thing to be looking for these.
DR. SUJANSKY: And you're right, you can't just get a text file of these and start using them. There are very few terminologies that that be possible with. But I think the important point is whether you could for example, build your own system to work with it, or whether that's possible even, versus having to use a third party tool. That's really what this gets at.
MR. BLAIR: Could you clarify to what degree MEDCIN might fall into this category?
DR. SUJANSKY: I think MEDCIN's database provides the terminology in a form that doesn't require any kind of server or browser, but they also provide a browser, which has an additional fee as an enhanced product offering, a convenience, a service, if you will. But it's not required.
MR. BLAIR: It's not required, okay.
DR. ZUBELDIA: So, in third party tools you include tools from the terminology developer?
DR. SUJANSKY: Yes. But again, MEDCIN doesn't have that as a requirement. I guess I wouldn't -- well, technically speaking, yes. A better way to state that would be a tool that is acquired or licensed aside from the acquisition or licensing of the terminology content itself, whether it is provided by the same source, or a third party. That might be a better way to state it. In other words, that entails additional costs of some sort.
DR. ZUBELDIA: What if doesn't entail additional costs? What if the terminology developer gives you the terminology viewable with their own browser, and it's included for free, but that's the only way to see the terminology?
DR. SUJANSKY: Then it scores a one, an intermediate score here from the zero to two scale. Because that's a limitation in some respect, but it's not as onerous as something very expensive or costly. Again, you could argue if that's the right way to score it or not, but it's obviously not the same as no requirement at all.
The other point here is that a lot of the terminologies that didn't meet the essential criteria scored quite well in terms of licensing and IP restrictions. I guess this may not be a surprise. You could hypothesize that a lot of those terminologies are -- well, I don't know if that's really true, looking at the terminologies. But they are maintained as parts of messaging code sets and so forth, so they are less costly. They are already included in those other resources, and so forth.
But in any case, with respect to licensing costs, and IP restrictions, there isn't a significant cost between those that met the essential criteria and those that didn't. The responses were kind of all across the board. Which is different, as we saw, than the desired technical criteria, where there was sort of a systematic difference in the terminologies that met the essential criteria, also did well with respect to the other technical criteria. I don't know if that point makes sense, or is worth making, but there it is.
The next thing is something we have already touched on. These are sort of question marked points. Should one, can one, I think we already discussed this, that we shouldn't exclude terminologies based on their licensing costs at this point, but that was just a question I raised for these specific three terminologies that are part of the group that meets the essential criteria, and obviously we need to discuss that more.
MR. BLAIR: Which are the three?
DR. SUJANSKY: It's MEDCIN, NDDF, and the UMDNS, the device codes.
And then the other aspect of the desired organizational process criteria -- it wasn't phrased this way in the questionnaire in our previous discussions, but it's something I called responsiveness to constituents. This could be scored by looking at the criteria, the update frequency, varied sources of update requests, and availability of training.
And the update frequency, again, this depends on the kind of terminology it is, the kind of data in there. For example, a drug terminology might really require more frequent update than a lab test terminology. All other things being equal, I'm sure lab tests also need to be updated in a timely fashion. But just as a coarse scoring, any terminology that was updated less frequently than annually got a zero, and any terminology that was updated at least annually got a two. So, that was very coarse.
I won't go through all the details of the scoring criteria, but let me just show the results here for this responsive to constituents, and highlight a few points here. So, first of all, almost all the terminologies that met the essential criteria did very well in terms of update frequency, and varied sources of update requests.
Only SNODENT and the HL7 Version 3 codes are updated less frequently than annually, as far as a formal release goes. So, those really weren't distinguishing criteria though, date frequency and the varied sources of requests.
In this category, the distinguishing criteria was availability of training. But you might argue, I would posit that maybe that may not be as important a consideration in the context of what we are doing here, because terminologies that were selected for a national standard, there would very possibly be third party sources of training that became available for something that was going to be used very widely.
Obviously, free training is better than paid training, but all in all the issue here is availability of training. And even the terminology developers provide various levels of support for that, third parties would likely provide support where the developers didn't.
DR. ZUBELDIA: Walter, could you clarify for me on the update frequency? Is it updated more frequently than once a year, or less than every year?
DR. SUJANSKY: A good score is at least once a year. More frequently than once a year, or once a year is a good score.
DR. ZUBELDIA: If it is updated every three months, that's a great score?
DR. SUJANSKY: Right.
DR. FITZMAURICE: I must say, it was a little confusing too when I saw the frequency listed annually. I thought well, every six months, every three months. Every day might be bad.
DR. SUJANSKY: If you want to get technical about it, the higher a frequency, the more often something happens. So, greater than something frequency is good. But yes, I understand why it's confusing. I think it was scored appropriately.
DR. FITZMAURICE: Maybe frequency of greater than once per year.
DR. SUJANSKY: Oh, okay.
DR. HUFF: Or greater than or equal. But that's what you did though.
DR. SUJANSKY: Right. The terminologies that didn't meet the essential criteria were highly variable in terms of this responsiveness to constituent score; more variable than the terminologies that did meet the criteria. So, if you look at all these, they are all over the board. The update frequency is highly variable. The sources of date requests are variable. The availability of training is highly variable. So, there is less consistency there.
Now, for these organizational and process criteria, this was the area where there are a lot of criteria I think that couldn't be evaluated quantitatively in a straightforward fashion. And so, there are some qualitative attributes that need to be considered.
In the report and in the analysis that I have done so far, they are not fully addressed, and I'll just tell you that right now. So, for those terminologies that we feel are important and viable candidates, and we're trying to distinguish among and so forth, we need to look at these things more.
So, the first one is funding mechanism of the terminology developers themselves. It is hard to decide what is a good funding mechanism of the choices that we gave them. Which the responses included things like license fees and membership fees, the federal budget, federal grants or private grants, subsidies from a parent organization, and so forth.
I think we'll need to, in a subjective way, evaluate those responses for a smaller set, and make some decisions about which terminologies we feel have long-term viability in some sense. And of course, it will depend on, as we discussed before, whether government funding will be extended to terminologies that are in the group of recommendations, and so forth. So, there is sort of a chicken and egg issue there as well, to a certain extent.
Another one is what we talked about as governance structure earlier. Here, I think there are some trade offs between very formal balloting-based governance structure, consensus-based, balloting-based, which is great for making sure everyone's views are heard, and there is consensus in the update to terminologies. And the timeliness of those updates.
And so, again, it's hard to say what the right answer there is in terms of governance structure. It's something we need to evaluate and discuss more.
And lastly, the issue of market acceptance. We asked questions and we got a lot of data about how many licensees do you have, how many users do you have, who are they, how many are government organizations, et cetera, et cetera. Again, I found it difficult to score this on a zero to two or zero to three score as to what are good answers there.
More is better, but if you have a large number of end user licensees and no vendors, is that better than having eight of the ten top vendors and no end user licensees, and so forth, and no government licensees? So, again, that's something that we will have to look at on a case-by-case basis, and we will.
So, the grand totals, and again, this is based on the data that we have so far, an also solely on the quantitative assessments, the scoring that I have described. Now, with respect to the totals, what I did was I actually created two totals.
Because there were a lot of question marks, as you recall in that area where we were talking about licensing costs and IP and so forth, I excluded that whole area from one of the totals, so that we would have some number for at least all of the terminologies. And that is this first column, total score without licensing, IP. The maximum score there is 20.
And then, I included also another score that included the licensing and IP. And if there is a question mark with respect to that, then this whole score received a question mark. So, you can see up there how that looks.
So, a few things to consider. One is SNOMED-CT. Again, as we discussed before, the licensing and so forth provisions are a little bit up in the air. If SNOMED received the highest scores, if you will, with respect to licensing costs and IP restrictions, then it would score a 26 out of 26 with regard to the numerical scoring at least. So, that's one thing to note. It scores very well, basically. That goes without saying.
The other point is that among these general terminologies -- what was the point I wanted to make here? There is some -- again, SNOMED scores the best among those. And the NCI Thesaurus also scores well, but MEDCIN, which we talked about a little bit earlier, scores 17 out of a possible 26.
So, I think that is worth considering when talking about would the $25 per site license be justifiable or worth it or so forth if that was to be selected as a standard?
MS. HUMPRHEYS: If we disagreed with your scoring, and then their total score would be 19, in other words, right?
DR. SUJANSKY: That's a good point.
MS. HUMPRHEYS: If we said that was a nominal cost.
DR. SUJANSKY: If we said that was a nominal cost, their score would be 19, that's right. It would be the same ad MedDRA versus possibly a 25 or 26 for SNOMED.
DR. MC DONALD: You said if SNOMED -- it would be that high. It's hard to know where the scoring comes out. So, that if everyone satisfied that thing, would they be this high? How did they get to 26?
DR. SUJANSKY: Remember back here --
DR. MC DONALD: You left it as a question mark.
DR. SUJANSKY: Yes, the reason I did that -- I'm sorry, let me back up a little bit. When we were back here, we couldn't --
DR. MC DONALD: How many points did that give it?
DR. SUJANSKY: Each of these could be two, so it would get a six here.
DR. MC DONALD: I mean if you did the same thing with MEDCIN, how high would it score; if you said it was perfect too?
DR. SUJANSKY: Well, but we know MEDCIN has some intellectual property restrictions. If we change this from a zero to two, as Betsy was suggesting, it would score 19.
DR. MC DONALD: No, I'm still saying if you just gave it the full six points.
DR. SUJANSKY: If somehow they changed their licensing and intellectual property restrictions?
DR. MC DONALD: If they made a deal with NMM(?).
DR. SUJANSKY: Right. Then they would have three more points than they have now, so they would 20.
DR. MC DONALD: Only 20, okay. So, where did they fail compared to SNOMED? What keeps their score down?
DR. SUJANSKY: It's in the report, actually, Clem.
DR. COHN: Do you have a copy of the report? Okay.
DR. SUJANSKY: So, the next point is that LOINC scored very well also, 23 out of 26. That's certainly viable, at least in the lab domain, and pending further analysis of the domain coverage, other parts of LOINC as well.
DR. HUFF: Could you say more about that? That other part of LOINC isn't an unknown quantity.
DR. SUJANSKY: No, I just didn't go through as part of this initial analysis, and look at the specific terms and content at a specific level. For example, is the coverage of the various observations in LOINC better or worse than coverage in SNOMED of similar domain areas? I's and O's and EKGs and blood pressure and so forth. Did the contribution that the read codes made to SNOMED enhance it in that area sufficiently to be as good or better than LOINC? I don't know that yet.
DR. COHN: Maybe I should comment as one of the few people on this subcommittee who can talk about LOINC without having to recuse themselves. My understanding is that LOINC is really sort of three different things. One is lab LOINC. One is the sort of physiologic and other observations. And then there is a whole set of like questions that we may eventually see in a claims attachment standard.
And so, in some ways I think my understanding is the analysis here -- and maybe I'm wrong about this one, but I sense that the way you describe this as lab is that at this moment we have been talking specifically about the lab LOINC, and that there is no intention here -- somehow we haven't really been able to get our hands around the physiologic LOINC.
And obviously, the claims attachment LOINC is whole other animal that totally needs to come out somewhere else. It's some way for us to think about that one. I don't know.
DR. STEINDEL: Simon, my feeling on the analysis that is presented in this report looks at the responses -- and I haven't seen the LOINC questionnaire responses, so, I'm hypothesizing somewhat -- but I don't think they submitted a set of responses for lab LOINC, a set of responses for the clinical portion of LOINC. They submitted responses for LOINC.
But the areas that we are looking at, the processes that LOINC goes through for its other domains is very similar to the process it goes through for the lab domains. Now, the coverage may not be adequate, but as Walter pointed out, he had problems determining the adequacy of coverage from just the responses to the questionnaires.
And if we determine that adequacy of coverage is going to be a very important criteria, then he has to look in more detail at those particular terminologies that we wish him to look in more detail at. And at that point, the answer to the specific question that you are asking may come out.
DR. COHN: Okay.
DR. HUFF: I would agree exactly with what Steve said. I guess the one clarification I would make is that if I were to characterize it in the way you did, I would say that the other part of LOINC would be survey questions, not restricting that to "attachment." It's survey questions in general, which is turning out to be a very big area of interest, because there has been no standardization of those kind of things across domains and content.
DR. COHN: Okay, well, thank you for the correction. And obviously, I think Steve is probably right, that this is an area that right now for simplicity, we are obviously calling lab. But as we peel away the onion a little bit, we are sort of seeing obviously, it's a little more complex, and probably will bear further analysis, as well as input.
MS. BURKE-BEBEE: I just wanted to make a comment that that the categorization of MedDRA as having a concept orientation, I'm not real sure that that is accurate, based on the fact that there are a small number of terms, relatively speaking, when you go and look, if they say how many concepts do they have versus how many terms. And that seems to be accurate, that MedDRA is in the UMLS. So, I'm familiar with it.
But what I believe is the case is that some of the terms are in fact synonyms for the primary concepts. Some of the terms are in fact what I would call entry terms, and they have narrower means, and they are pointing to the smaller number of preferred terms. And it is not, as far as I know -- there is no distinction within MedDRA as to which of all of the terms that are pointing to their preferred term, mean the same thing, and which have other meanings.
So, to me, that is not concept orientation. I think what they have is preferred terms, with bunches of terms pointing to them, and some of these terms that are pointing to them are synonyms, and some of them are not. And within their structure, this distinction is not made.
DR. SUJANSKY: Okay.
MS. BURKE-BEBEE: I think I'm correct about that, because I believe -- and I can go back and verify it, but I think that we had to impose concept orientation by linking these various entry terms to various concepts in the meta-thesaurus in order to accurately get MedDRA into the meta-thesaurus. I was a little surprised by that one.
DR. SUJANSKY: Maybe we can go back and look at their responses to the questionnaire.
MS. BURKE-BEBEE: It could of course again be the fact that not everybody keeps the same definitions of these terms in mind. As you pointed out yourself, what people really understand by that when they were filling out that question. And I will go back also and verify that I'm correct. But at least in terms of what we have seen, and we have been incorporating updates to MedDRA in the UMLS, and I think I'm right about that. I'll check on that.
DR. SUJANSKY: Yes, let's follow-up on that.
DR. STEINDEL: Just picking on what Betsy said, I think when I was looking through the details of this, there are a few responses from specific vendors that I felt I think that the terminology deserves a different point than you gave it. But as I was going through the whole report, my final conclusion was if we did change that, would it really make any difference in the final conclusions? And in most cases, my reaction was that it probably wouldn't in the aggregate. So, it may shift the total score like one or two points, or something like that.
But when we look at the way the point spread came out in the end, that one or two points is really not going to make that significant a difference I think in our deliberations.
MS. HUMPRHEYS: I would agree with that, Steve. I would think we probably, since there were these four things that put people either in or out, that if someone looking at the report would say, well, I know that vocabulary, and they are wrong about that, it might sort of call the whole thing into question. So, I think we probably want to be as accurate as we can. Even though it won't affect the bottom line, it will give greater credibility to the report.
DR. STEINDEL: And I think we need to pick on some of these things' credibility.
DR. SUJANSKY: Well, I think again, if you -- I will make all the data available. And there is a way to kind of trace back the scores through the questionnaire that I can discuss with you in these areas where you have doubts or questions about how a particular score was arrived at, and whether the questionnaire responses are accurate that caused me to arrive at that score. Then I would welcome that input.
Again, obviously, the more people that can validate the accuracy of a response, the better, because to a certain extent, we just have that data to go on. Okay, very good.
Then the other point, then again, is that recall that -- the other thing I want to say is that these scores exclude the scoring of the essential technical criteria, because to a certain extent, those all have to be four out of four in order to even get to this phase, even though we know some of them have question marks.
So, these scores are all other features and criteria, aside from those four essential ones. So, the reason I mention that is when we are looking at the total score of the terminologies that may not meet the essential criteria, we can see that they are generally lower than those that do meet the essential criteria, with some exceptions. Only three of the ones that don't are above 10 even out of 20. And only one of the ones that do meet the essential criteria are below 10.
Is that point clear?
DR. COHN: Let me comment on this, and Suzie you are next. I guess we will talk this afternoon about triage and criteria and sort of place in the sun. It looks like some of these things are very different sorts of things.
DR. SUJANSKY: In terms of domain coverage?
DR. COHN: Domain coverage and sort of what they are. I mean to try to compare SNOMED-CT to HL7 Version 2 codes is --
DR. SUJANSKY: This is with respect to the technical criteria, not the purpose of the terminology, or anything like that. Just the technical and organizational process criteria that we talked about -- non-redundancy, ambiguity, multi-hierarchy, et cetera, et cetera. Just with those respects, in terms of those criteria.
DR. COHN: I know what you're saying, but I'm saying the purposes are so radically different. We need to look maybe this afternoon towards sort of the purposes and all of this, and see how they fit. One would observe that probably some more in the HL7 Version 2 is demographics, and demographic codes.
And one would observe that maybe those don't change on a yearly basis, or have to be updated on an every six month basis, just for example. So, the purposes may be so different, it may be that the technical criteria aren't quite as durable in those areas. It's just something we need to think about.
Suzie, do you have a comment?
MS. BURKE-BEBEE: Just an observation. When I was reading this, looking at the Table 10, I understood in your text.4 that you said that it did not include the essential technical criteria, but is it not inherent in the way the table is set up?
DR. SUJANSKY: Yes, it is, it's inherent in the way the table is set up. It's not reflected in the score. That's right. Okay, that's a good point.
Again, a general point is first of all, there is no weighting on any of the criteria. So, it's just a straight line thing, and that may not be appropriate in all cases.
And secondly, the weighting itself may vary from terminology to terminology, depending on the purpose. Like you said, it may be more important for something like a drug terminology to be updated frequently than HL7 message codes and so forth. So, those are all valid points.
I should put conclusions in quotation marks. But here are things that we may conclude from what we have seen so far. A large and diverse sample of medical terminologies was represented in the return terminology questionnaires. I can attest to that. I hope that was clear and not controversial.
The completeness and accuracy of the data was sufficient to score the terminologies against many of the important selection criteria. Given Betsy's comments a few moments ago, I think that is largely true, but there may be some cases where maybe there are some problems with the accuracy. But, that's largely true.
Now, kind of the meat of it, the same set of terminologies scored consistently well against all quantitative measures as a group. These included the essential technical criteria, the desired technical criteria, and the desired organizational process criteria. I think we saw that in the results.
And this set of terminologies -- and again, mainly these are the ones that definitely, notwithstanding Betsy's comments, met the essential technical criteria. These terminologies cover much of the clinical domain required for PMRI standards. Again, that's a tentative conclusion based on the further analysis of the clinical domains that indeed are covered by these terminologies. But I think that's a good working hypothesis.
And additionally, there may be overlapping gaps in certain domain areas. Overlaps I think definitely occur in terms of diagnoses, symptoms, findings, observations, procedures, and others probably. Gaps may occur in terms of nursing observations. We saw that none of the nursing terminologies definitely met the essential technical criteria. If that turns out to be the case, are there gaps in those observations? Or are the critical observations already part of SNOMED-CT, for example?
MR. BLAIR: I thought there was one nursing terminology that --
MS. HUMPRHEYS: There were some that were in the questionable area.
DR. SUJANSKY: Right, there three that were on the questionable list, Jeff. But there were none among the ten that were not questionable.
And we may find gaps among devices, if we conclude that the UMDNS's licensing provisions are exclusionary in some respect, and there are no other sources of that comprehensive terminology for device representation. Again, the important devices may be represented in other terminologies already, and we need to look at that.
Lastly, additional analysis is required to resolve these and other issues. Does SNOMED-CT subsume the other general terminologies already? If we had SNOMED-CT, would we need MedDRA or MEDCIN or NCI Thesaurus? As I mentioned are nursing observations included in SNOMED?
And can overlapping terminologies be combined in the core terminology group to support a good PMRI terminology standard? More importantly, how can they be combined? There probably will be some overlap almost certainly among the terminologies required to cover the domains, but how can we accommodate the overlaps in that group?
That's basically it for the data presentation and conclusions and so forth. Some of the next steps we'll discuss more this afternoon. I think the next thing we are going to do is talk about some of the scope and criteria issues, and review a little bit. But later today we will talk about some of the next steps.
These include validating our assumptions regarding scope and process. What I just mentioned for example, is the dental domain and scope. We need evaluate the excluded, probably excluded terminologies. Do we need to spend the time doing that?
Also, another step is review the specific content of remaining terminologies. So, further evaluation of domain coverage, validation of non-ambiguity and concept orientation, at least in some cases. And we know that we also need to seek input from IT vendors and other terminology users to get their take on the usability value of some of these terminologies, the acceptability of existing licensing and IT provisions, and the responsiveness to their specific business needs of these terminology developers.
Agenda Item: Discussion - Triage Based on Scope - Walter Sujansky, M.D.
DR. SUJANSKY: And that's it for the presentation. So, we have over half a half an hour before the scheduled lunch break. So, keeping with the agenda, one important thing I think to discuss is some of the issues of scope. I mentioned one of them in the slides, and there are some specific questions that we address now.
If there are some comments or general comments, we can take those before we move onto that.
DR. FITZMAURICE: Just a quick question. Do we have a copy of your conclusions in the paper, the ones you presented up there?
DR. SUJANSKY: Some of them I think are in there; not all by design. It's a draft, and some of those I wanted to bounce off of all of you here today.
DR. FITZMAURICE: Can we get a copy of your slides?
DR. SUJANSKY: Sure.
DR. STEINDEL: Just a quick comment on scope question. I'm concerned about the phraseology as the dental domain in scope. I think dentistry is definitely a scope of clinical medicine. Dentists do that all the time. It's very important. The question is, is the dental domain adequately covered somewhere else? And that's a different question than in scope.
DR. SUJANSKY: Okay, that's fine. I did want to ask that question, and you have now answered it. If everyone is in agreement --
DR. COHN: Is everyone in agreement? I think that's another question.
DR. MC DONALD: Agreement with what?
DR. COHN: With whether dentistry is in scope for the initial core terminology set.
DR. MC DONALD: I'm nodding my head yes. It's part of the body.
DR. COHN: Okay. Marjorie?
MS. GREENBERG: Just regarding the gaps, and I realize this is a work in progress, and that you haven't completed the gap analysis, but just you mentioned already that none of the nursing classifications at least made the top category. Also, I think we should try to make sure that there is adequate coverage in the core of issues related to alternative medicine, since I think the only submission on that also did not meet the essential criteria.
And also issues related to functional status, because it's my understand that those are not well covered by the ones that met the essential categories. And we have eliminated ICF from consideration.
DR. COHN: Well, actually, I think we decided that we were going to discuss whether things that were late, those three items needed further analysis. So, I don't know that it's necessarily absolutely off.
MS. GREENBERG: Okay, but either way, I think the whole range of functionality and functional status, et cetera, needs to be considered, as well as the alternative.
DR. COHN: Sure. Betsy, do you have a comment?
MS. HUMPRHEYS: I guess I share Marjorie's view that probably it would be very useful to include ICF. I do think UMLS is a different animal. And while we might want to have something in this report somewhere about it, I'm not sure that putting it in the same table with all of its constituent vocabularies, or many of them actually presents an accurate picture.
So, I don't think that Walter necessarily has to reduce the UMLS thing to the same report, but I do think ICF is something that probably ought to be looked at.
The other thing I was going to say, in looking at this and thinking about it, it occurred to me that it is possible that some of the sections of NLM's medical subject headings, which would score extremely high on the technical criteria, and on the process, and on the licensing, and all of that stuff, it may be interesting to look at whether it does not fill deficiencies in some areas, if there are some.
The medical subject headings has really a very huge component of various chemicals and substances, both those which are useful to human beings, and those which are not. So, that's something that could be looked at in some sense, after this.
The other thing would be in areas like proteins, there are some areas where based on what is currently in the UMLS, the medical subject headings is the strongest vocabulary there in some of these areas. Not obviously in clinical conditions, but in some of the other areas.
And so, it didn't occur to me until I was reading your report, that even though in general it isn't a clinical terminology, it might actually fill a gap here and there if we discovered one after we looked at the set.
MR. BLAIR: As I listened to Marjorie's question, and Betsy's, which raised some thoughts that I had myself, it sounds to me as if we need to begin to address a couple of issues, because Walter did an outstanding job of going through the information that was made available to us by the terminology developers, and doing an assessment, especially a technical assessment for clinical specificity, the first four criteria, which gives us a good foundation.
Marjorie, I think raised a question that I very much agree with here, the other areas, such as coverage. Now, if we start to wind up having that as an issue where we wind up saying we need to have something to begin to cover disabilities, alternative medicine, nursing, that they are important parts of the overall picture, then I think there are a couple of different ways we could proceed. I don't know really know what the best way is to proceed.
We could proceed on the basis that we define the core narrowly, and then wind up saying there are other domains that refer to the core. Or we could wind up saying that we define the core broadly, in which case we wind up maybe making it in terms of those terminologies that meet the technical criteria of the core, and those that have work to do, to be able to be recognized as core members in the future. Or we could do a combination of those, or maybe some other approach.
But I do think we need to begin to flesh out this concept of coverage and related domains, and whether we have a narrow or broad core. I guess I'm asking for help.
DR. STEINDEL: I think Jeff just asked the same question that I was going to ask, maybe slightly differently, but that net effect is going to be the same. I think in the area of what is the scope of what we are doing, I think Jeff very succinctly described that very well. What constitutes the core? And that will have a lot to do with how we take a look at the wonderful job Walter did about analyzing this large amount of information, and trying to fit it together.
My response on reading it was as Walter noted, there are a few terminologies that come out very, very high in all criteria. But if you look at the content of those terminologies, you find that in some specific areas they are deficit. And just as an example, as we were talking, and I was thinking about the question of dental terminology, I took a look at the latest version of SNOMED to see how it covers teeth.
And it does a very good job of covering teeth, but it doesn't have tooth number. That concept is not in SNOMED. Well, is that important or not? Well, I know it's very important to dentists. So, there's an area where we might want to so that the ISO tooth designation is very important, because it's not adequately covered in SNOMED, even though it in itself scored very, very low.
And I think there are a lot of things in this report where you can build examples similar to the one that I just built. And that's what we are going to be discussing this afternoon, what is the scope of the report.
DR. COHN: Clem, it looked like you were getting ready to make a comment.
DR. MC DONALD: Well, I think that the problem of scope and coverage is a lifetime job for the rest of history. That is, we should expect none of these to be very good when they really connect totally with life; to be perfect when you connect totally with life. And that is not bad. Language has changed a lot in 100 years, English, so we are going to have that.
On the other hand, those tooth numbers are in HL7. So, that we'll have to worry about keeping congruence as we submit them.
MS. HUMPRHEYS: That brings up a good point. Were SNOMED to be designated as part of this, then the issue would be if in fact teeth are well represented, and there is one common synonym that is used by one group that has to use this that isn't in there, can we just say let's just add the synonym? Or with ISO-2 into the UMLS and we'll map it to SNOMED, and that is the end of the problem there. That would be a very simple one to solve.
Some of these others will be very difficult, and I think that part of it is if there are good terminologies that cover a fair amount of what we are interested in, perhaps the movement of those along as recommended things, so that we can then start looking at what are the real gaps and so forth, and how to fill them, or whether there is something available that fills them. You've got to start somewhere. I don't know that the first set of recommendations has to cover the entire waterfront.
MS. BURKE-BEBEE: I was interested in two specific questions on the survey in reference to mappings. On page 35 of your report that we have today, K talks about does the terminology content include mappings to any other terminologies? And E, on page 41, does your organization participate in any cross-terminology standards organization bodies, giving an example of ANSI HSB for HL7.
I was just curious, as part of the criteria that we have from December, and your report talks about that kind of thing, what were the type of responses that you got from the questionnaires?
DR. SUJANSKY: Well, a lot of the terminologies did map to other terminologies. And the extent of that mapping varied widely. So, if you are interested in the details of that, it's available in the spreadsheet.
The other question, as far as participating in the cross-terminology, ANSI HSB and so forth, I didn't include that in my analysis, because participate can mean anything from showing up at the meetings to being the chair, and working hard to make sure everything is consistent, and changing your terminology to make sure it's consistent.
So, I didn't feel that was very informative from that point of view, from the point of view of mapping and relationships to other terminologies. It may be useful just to know who is participating and who is not.
MR. BLAIR: My though is that maybe the data wasn't there. Maybe we didn't ask the question in the way to bring forth meaningful answers. But in the broader picture, I think it is important that we have terminologies that relate well to HL7 and the RIM, and to the standards we have already developed, as well as to the HIPAA transaction standards and code sets. But maybe we need to come up with subsequent ways of collecting that data to identify that. Although it's part of the broader picture.
DR. SUJANSKY: As far as the existence of mappings and so forth, that of course was included and scored. But you're right, there wasn't specific consideration for which mappings are more important than others, and the scoring based on that, and all the complexities of that. Again, that data is available.
So, to a certain extent, we have the data as far as the mappings is concerned. I think, Jeff, as far as that is concerned, we may have asked the right question there, and gotten the data we need. But the analysis of that bears further examination, because it is complex and varies from terminology to terminology.
DR. COHN: The issue of mappings is not the existence or non-existence. It's the quality of the mapping for intended purpose.
DR. SUJANSKY: Exactly.
DR. COHN: And I think that really is the critical issue in all of this. So, it's good to know that there is a mapping, but would you use it as a crosswalk for a HIPAA transaction? I don't know.
DR. ZUBELDIA: I'm wondering to what extent it would be useful to run this draft of the report by the terminology developers themselves that answered the questionnaire, and let them perhaps see their own evaluation, and provide feedback. They are probably not going to agree, but at least if they provide some feedback that says well, you haven't seen this aspect of our terminology. And something that is a positive feedback, it could help refine the product.
DR. HUFF: A few things have occurred to me as the discussion has progressed here. One is that some of the questions we are asking about overlap and gap and other things require further context of use to answer the question.
And what I mean by that is you can imagine for instance, that there might be a lot of apparently overlap between RxNorm and one of either NDFRT or NDDF Plus. But in fact, if you look at the context of drug orders versus dispensed drugs, then that overlap may be more apparent than real, because you dispense things that you bought from somebody that is a brand name or generic manufactured thing, and RxNorm in general is talking about clinical drugs, which are named at the generic level that you would use for ordering, brand-independent of the thing. So, I think we have to consider that aspect of this in the strategy.
The second thing is sort of one of process, and comes back to some things that we talked about before, which is what is the strategy of implementation of these things? One model that I have thought about, and there could be more than the two that I have thought about. One is that these core terminologies become part of the Metathesaurus, and then we identify out of the total concepts in the Metathesaurus, those things that participate in messages and in the EHR and that sort of thing.
Another way is that we identify them as core terminologies, but in fact they are implemented by people essentially buying that terminology, or getting that terminology from the individual vendor. So, they are never actually in a central part, and that is more troublesome in that scenario, because they are not a part of a single, consistent core, then the overlap is more troublesome, because you can't make the correlation between the various terminologies.
And there are also differences in those two strategies in that for instance, if there is an area that's a gap, the choice may not be to include a whole new classification system or something else, but say what are the individual concepts that we add to the Metathesaurus to cover that gap?
Because if our option is only to include or not include a terminology, it may be for instance that we are including a lot of things we don't want to include, because we want to get these eight concepts that are important to cover a given domain area.
MS. HUMPRHEYS: You could imagine a variation on the strategy, where okay, there are the end clinical vocabularies that have been designed, and they are part of the designed clinical subset of the UMLS. Then just let's take the medical subject headings as an example, somebody says none of these have this type of concept.
But although the medical subject headings has 150,000 concepts, and we don't want them all, actually we do want these 500 that are well modeled there, so we'll just take those 500 as being part, and we'll relate them to the others in some way in the Metathesaurus. And therefore, we can merrily go on our way without somebody who is interested in that clinical view of what is in the UMLS, taking the entire medical subject headings, including many things which have very little relevance to direct kinds of care.
DR. SUJANSKY: We're getting into a discussion on how to combine all of these things, which is an important, difficult discussion.
DR. HUFF: Just a couple of quick other points. One is I think it's important that we not necessarily think of these things as static. And what I mean by that is that there could be correctable deficiencies in these things, and that that's immutable. We can go back to the people and say, well, what if you did this from now on? And that would turn the thing into a very useful terminology if they made that correction.
And I suspect that especially when they see that they are either going to get through the gate or not through the gate, they would be highly motivated in fact to try and meet the requirements. So, I don't think we want to treat these things as immutable; that in fact we could correct those things.
And then in terms of scope, it has come out a couple of times that the difference between classifications and primary findings or observations and that sort of thing. And in a sense that's scope I think in some sense. Some of the things that we're thinking of as terminologies are because they are intended to report statistics and aggregate findings, rather than primary findings. And I guess we need to make sure where our boundary and scope is between those things.
DR. COHN: Yes, I'm going to sort of chime in on that one. I would be curious to see where our boundary and scope is. I think the good news is that there have been a number of classifications that have already been accepted as part of the initial HIPAA process. So, I don't think we have to go back and redo those. That's probably good.
DR. HUFF: One of the areas where I was thinking about most is in the nursing. The nursing terminologies generally are referred not as terminologies, but as classifications for that reason. And you can't paint them all with one color, but a lot of them were intended to record was this thing done, which is very different from what was the value of the thing that you saw. And that's a very different use of the terminologies, both of which might be very valid.
DR. COHN: I think you really bring up the issue of how we put all the domains together, and how we even look at these things. I mean certainly from my view, the thing that I most worry about in all of this stuff is not broad or narrow scope. I sort of presume that by the time we are done, because of the size of many of these terminologies, we're likely to get relatively broad.
My bigger issue is really the overlap, and trying to figure out what is overlap, which I think you are bringing up with the drug issue.
DR. HUFF: I just come back to my first point that a lot of these things you could talk about interminably, unless you say, oh, my intent was to be able to send this kind of information from a health care institution to the CDC. Or my intent was that this terminology would be used to send an order message from a medication ordering application to a pharmacy.
Unless you get specific in where the terminologies are going to be used, the questions are going to be interminable about scope and about usage, and we won't achieve the goal that we have, which is to be able to communicate this information, and share this information across health care enterprises, government agencies, et cetera, et cetera.
DR. MC DONALD: To continue in the same kind of complex discussion as the scope is that first, whether you think of it as sort of classification or a base vocabulary changes and mutates. And it changes and mutates over time. And I would argue that everything is a fractal, and you can always dig another level down, or it will evolve into a fractal. And so maybe we shouldn't make such hard lines between them.
Secondly is when we think of vocabularies historically, we have thought about them sort of as word dictionaries, word phrase dictionaries. And where that stops -- a survey instrument, in the last two weeks, have you had chest pain more than twice, that might be a survey instrument. That is clearly not a word dictionary, but it might be a principal data collection item in every hospital admission or something, the SF-36, which may not be a good example, because it's proprietary.
But they always have this qualification of time, some amount of thing, and then there are some variables, some answers at the end. And those things are probably just as important as the glucose measurement, at least some of those things are.
So, I think we aren't anywhere near done, I guess the short answer is. When we start thinking, yes, we've got a pretty good dictionary of words and short phrases, but we haven't developed all the concepts that are actually passed around as identifiable things.
And that is what you see when you start getting into the specifics of a reporting document, or a specific claims attachment. It goes on and on. And it's good news, because we'll have a lot of work for a long time.
MS. HUMPRHEYS: I feel certain of that. But I just wanted to comment that on the overlap issue, the relative ease or complexity of resolving it is entirely dependent on the set of things where you are trying to minimize the overlap. And so, I don't know that you can establish a principle in the absence of knowing the structure and coverage of the four or six or ten things that you somehow want to get into a coherent set.
We deal with 100 vocabularies, classifications, whatever you want to call them in the UMLS. And integrating some of them into any type of a coherent set would be a fool's errand. And integrating some of them would actually work pretty well. So, you can't sort of establish a principle of how we would integrate any vocabulary.
You have to talk about how you are going to integrate the five that we actually care about, because that might turn out to be either an easy job or a hard job. But it would be a job that we could specify, once we knew what they were.
DR. SUJANSKY: Well, said. I think a lot of important points have been raised. I would like to highlight two of them that are paramount in my mind. I think they were both raised by Stan, and then others added to them.
One is the issue of intended use and purpose and so forth of what we are doing. It gets back to something we talk about earlier or months ago about requirements and use cases, and so forth. And the need for an understanding of what the goals of PMRI terminology standards that will be recommended are, what are the goals of that work, those recommendations?
If the recommendations were to be adopted 100 percent, what would be the goals of that thing that would result? And I don't know that we have a good sense for that yet. And to some extent we're a little handicapped in moving forth because of that, I would propose. And it may be worthwhile to try to articulate. Maybe there are those on the subcommittee who already have a good sense for that, but I don't know that we have a statement of that, or that we are necessarily in agreement on that.
The second point is what we were just talking about, the need to integrate terminologies, and a model for doing that, and the role of the UMLS, and so forth. I think Betsy's right, the right model for integrating to some extent depends on what you are integrating. But I think perhaps certain principles can be drawn from it.
For example, will a third party be doing all the integration, and then the developers of the source terminologies, if you will, will be completely uninvolved in that, which I think would be a much more difficult task, as the folks at the NLM know. You can correct me if I'm wrong. I don't know how frequently the NLM contacts a source terminology developer and says if you just change this, it would be easier to map to that other terminology.
But I think that's the kind of stuff that is going to have to happen to really get a very good, tight, functional, usable integration of multiple terminologies in a core terminology group. Or could you just add some synonyms that have the tooth numbers, because then it will be really easy to map to the ISO tooth designation codes, which are a requirement for billing for all dentists, et cetera, et cetera, or maybe -- I don't recall.
But I think a discussion of models that include that kind of process, and whether that is feasible and desirable and so forth, I think are important discussions to have as we move forward.
DR. COHN: Are there any final comments before we break for lunch?
DR. MC DONALD: A worrisome perspective. In our institution we have at least seven doctor files. These are all identifiable, distinct human beings. You can go out and grab a hold of them. There is no ambiguity to who this human being is.
Despite the fact that the institute has one whole doctor file, none of the other ones are the same. There are all differences. They are not completely different. And it has to do with sort of accidental limitations like we don't put nurse practitioners in our file, and we have to. Or we only have room for 5,000. Or we don't want to keep the old ones in old, because it's harder to look up.
But those kind of things will come up in this mapping process too. And I'll use this as an excuse to ask a question, if I might, about say if RxNorm is going to be integrated with say SNOMED. And RxNorm would probably want to be internationally freely available. I'm not pushing those issues.
But then if it turns out that it might be natural to take the pieces from SNOMED and put them into the RxNorm thing, which is bigger. It's got a sort of pre-coordinated thing. But then would that then preclude the RxNorm being used internationally, which would inhibit this kind of approach? There is another analogy I won't use.
Is there anything you can say about that this early?
MS. HUMPRHEYS: I can tell you that the way the structure of RxNorm is defined, the pieces it is pointing to in all cases are coming from also sources that could be distributable. I mean you could imagine it doing it a different way. That happens to be the way we are doing it.
But your issue, there definitely gets to be international issues, whether we pick that example, where that might not apply. Or pick another one.
DR. COHN: Jeff, do you have a final comment before we break for lunch?
MR. BLAIR: No, I think I'll save it for after.
DR. COHN: After we have had some food. Okay, it's now 12:10 pm. We will get together at 1:10 pm.
Thank you all.
[Whereupon, the meeting was recessed for lunch at 12:10 pm, to reconvene at 1:10 pm.]
DR. COHN: This is the afternoon session. Jeff, do you have some comments?
MR. BLAIR: Well, I think I can begin the session, unless you have some other --
DR. COHN: No, go ahead.
MR. BLAIR: I'm assuming everybody is back. Is Walter back?
DR. SUJANSKY: Yes.
MR. BLAIR: Walter, do you have the capability to capture ideas and information from the subcommittee as we begin to identify purposes, goals, scenarios, and then domain coverage, things like that, that might be --
DR. SUJANSKY: I can take notes. Of course, I'll have the transcript later to go by as well.
MR. BLAIR: The transcript won't be until quite a bit later.
DR. SUJANSKY: I'll also take notes. So, yes I can take notes.
MR. BLAIR: I'm wondering. Sometimes we used to be able to have a screen where you could do it on the computer.
DR. SUJANSKY: Oh, that's what you were referring to. Got you.
MR. BLAIR: Do we have that?
DR. SUJANSKY: Yes, we can do that.
MR. BLAIR: Over lunch Stan Huff had some ideas for the way we might go through this process. I'm just going to do a high level piece of it. I think you are going to put much more flesh on these bones, but the idea, the thought here was that to be pragmatic, we had talked about what would be the purposes. Folks were winding up mentioning what are the purposes of the terminologies that we are having, whether they are in scenarios, we use cases, or whatever the phrases you want to use to be as a driving entity.
From that, you could wind up indicating what kind of domain coverage you might need. And then from that, there is a third level down, which might be refinement, which is in what context the information is being used. And Stan, I think is much more capable of maybe stepping us through this than my simple description.
DR. COHN: Jeff, did you want Walter to capture any of your comments?
DR. SUJANSKY: Can you give me the three levels again? I just kind of got this up and running here.
MR. BLAIR: Okay, and then when I do, I think Stan may refine them quite a bit. But I would say the top level is that essentially we are defining maybe 10 or 15 major categories or purposes for terminology. Not a hard number, just something that is manageable for us to be able to deal with today. That would be drivers for identifying what are the important terminology domain coverage areas in order to support those purposes.
And I don't know if we will have time today to do it, but I hope that we could go down to the third level, which would be some refinement of that. You might have the domain coverages, and you might have terminologies that appear to have overlap, but when you look at them in the context of how they are being used, they are more complementary than overlapping. And that that would be the third level of reconciliation.
DR. COHN: Say that again.
MR. BLAIR: Sometimes you have terminology domains, like for drugs, which appear to be overlaps, but they are used in a different context. They are used for different purposes. So, there is less overlap than might appear at first.
DR. COHN: I guess that's part of the final analysis, right?
MR. BLAIR: Yes. Stan, could I turn this over to you?
DR. COHN: Actually, Steve Steindel raised his hand.
DR. STEINDEL: Jeff, how much of this was covered in the first PMRI report where you had that chart on terminologies?
MR. BLAIR: To be honest with you, I don't think we gave it quite as much thought. I think we had tried to identify the terminologies that were out there, and group them together. I don't remember. There might have been something were we wound up indicating you need terminologies for statistical and reimbursement purposes and others. But I don't think we did quite in the same context.
DR. COHN: I'm going to jump in here. I guess I sort of thought that the stuff from the original report was more along the lines of diagnosis procedures. Comprehensive was another terminology. Nursing was another. I don't know that that is the same.
But a similar question I had was I was reflecting on a document, Steve, that you had shared with the subcommittee a couple of months ago that had been developed by CHI addressing -- once again, I don't know that it's the same thing as uses of standard terminologies, but I think that we all sort of grapple for how to slice the domains up. I actually thought that for lack of a better domain map, that that was not a bad way to describe it.
DR. STEINDEL: The questionnaire itself has a set of domains where we asked for -- and it's not the same as what the CHI one was, but it's very similar.
MS. GREENBERG: The domains are really different.
DR. STEINDEL: I know.
MR. BLAIR: Well, let me see. I think the domains that you had, and that we thought of in the original report were more in that second level. But I think that what Stan is mentioning is to be more directed by objectives or a purpose for what we are trying to achieve.
DR. COHN: Well, Stan, I'm very interested in what you are going to say. When I come up with the drivers, they are typically at such a high level and of such purity, that it's hard to know what to do with them once you have them.
DR. HUFF: Maybe this won't work at all, but the idea is that in order for us to make decisions about which terminologies ultimately we want to be part of the core, et cetera, it seems to me that we have to be somewhat explicit about what purpose we have for having these. So, the kind of things that I'm thinking, and it might become a long list, which is okay. We probably don't want to make a long list in the meeting here today.
But the idea was to say from the start, my mental idea of sort of the big picture was that we are going to use these terminologies to share patient-specific information between interested parties or appropriate parties. And then the drill down on that major concept was to say, okay, we want to enable public health data to be passed from health care providers to the CDC.
We want to be able to send health care data to CMS, so that CMS can determine the quality or appropriateness of patient care based on these things. We want to enable data to be sent to cancer or other kinds of disease-specific registries so that we can make that easier and more cost effective to contribute data to registries.
That's the sort of thing that I'm thinking about. So that when we think about including or not including the terminology, we can well, is that a use case we are trying to cover, or is that something else?
DR. STEINDEL: Well, I would love to see everything that you mentioned being the drivers for what we are doing, given that I'm from CDC, and I think you mentioned just about everything that we do, except for what Jorge does across the way in CMS.
But I understand that this process is for selection for PMRI terminology. And I think we have to look at our use cases with respect to the primary use of a patient medical record, and the electronic medical record systems that this terminology will drive. So, I'm looking at the primary use cases as being something internal to within the medical institution. And if they can help cast the information outside of the medical institution, so much the better. But I don't think that should be a principal driver.
DR. HUFF: Well, I think that should be on the list.
DR. MC DONALD: I think I can solve it. I don't think the first list is contradictory. But I think what you said, we should put on some other items like -- and this is between institutions or within an institution, is to order a diagnostic study, order a medication or prescription, send results of diagnostic studies, send results of clinical measurements, all the things that we know are necessary within an institution, across institutions, between an office practice and service providers. So, it keeps on going, but maybe we should put those up at the top.
DR. SUJANSKY: But I think a common point maybe is we are talking about exchange of data, not necessarily just the use of data in a PMRI to display on a screen and delivery of care. It's to deliver it to some other system. And my understanding from the beginning, and I'll read here from what we have been working with, what's been in all these documents from the beginning is what the NCVHS task is in this regard, is to study the issues related to the adoption of uniform data standards for PMRI, and the electronic exchange of such information.
So, the exchange is an explicit aspect, and I would argue is the driver for standardization. I think the reason to standardize is to be able to exchange. If you are just using it in on place, then it doesn't need to be standard.
DR. MC DONALD: Well, the beauty of this strategy, and actually if you carry it further -- I don't know that we can carry it to the next step -- but if you take a real message, whatever the darn message is, let's say it's an HL7 message or an X12 message, and we've got to have something for the scale. It becomes real clear what you need. It's just clearer than heck.
DR. HUFF: And that's exactly the motivation for saying this is that if you take it to that level, and say we want to be able to share drug prescription information, then you can talk about the set of drug codes that you need to enable that behavior.
And until you do that, there are so many ambiguities about what you want with drug codes, that you just can't -- and you can say, we want to enable people to record accurately, drug allergies, and then determine whether the drug being ordered is one of the drugs they are allergic to. And then that's a different set of drug terminology that you need to record allergies, than it is to order drugs.
So, I take all those as friendly amendments. That is, I would start with what Steve Steindel said, and say we want to enable the exchange of this information within systems in an institution or between institutions for providing patient care, for taking care of patients. And I don't see that as contradictory with any of these other sharings that we talked about.
MR. BLAIR: Now, is that broad enough for example, to include ambulatory care and alternative health care and long-term care?
DR. HUFF: I was being inclusive of those, but if you want to be explicit, I'm happy with that too.
DR. MC DONALD: The truth is, if you really look at it, you're going to collect data. You're going to write orders or get services provided. You're going to send reports. I don't think they differ a whole lot, except in terms of the codes that they might talk about.
MR. BLAIR: I'm assuming you've been capturing this, Walter?
DR. SUJANSKY: Yes.
MS. BURKE-BEBEE: Stan, is what you said to begin with too broad? You said share patient information between interested parties.
MR. BLAIR: Can I go back to one other thing. I know I'm jumping in here. One of the concepts, if it proves to be useful here, that we had in our report, August 2000, was a vision, something that we aspired to, may never fully achieve, was the idea that clinically specific standards for patient medical record information would get us closer to an environment where we could capture data once at the point of care for patient care purposes, in a manner where we could use derivatives of that information for reimbursement, for public health, and for clinical research.
Is that a useful place to start? Do we still believe that? Is that still our vision?
DR. HUFF: I think it is. I'm trying to figure out how to -- somehow that is like a motivation, rather than --
DR. COHN: I was just going to add something here that I think we have sort of talked about obviously the patient care aspect of all this. The other piece which I know many of us are concerned about is the whole issue of leveraging data for the development of evidence-based best practices. And I think that somehow also needs to be reflected in whatever endpoint.
There are two things that we talked about. One is interoperability, and the other is comparability. And the interoperability is the sharing, and the comparability is can you really lump it together and use it?
DR. HUFF: But Clem said a bunch of things that we didn't capture, which I think I would like to get on the list. And that is that we would like to have these terminologies be used to represent things that were ordered, represent things that were measured or observed in the patient, basically results if you will, of the things that were done, consultations.
DR. SUJANSKY: Are you referring to the content of the medical record? Are we enumerating what that is?
DR. MC DONALD: We are enumerating messages that would have to be sent. We could go through messages. You've got to order stuff. And there is a field that tells you what you are going to order, and there are probably other fields. I haven't gone back and dissected it. If you are going to get results back, if you are going to do a consult, if you are writing a note, you have to put labels on top of things. Those are vocabularies that essential.
And then you have values for the results that come back, which is another set of codes. They are not the same codes. And the endpoint on it, if you say I'm describing drugs I heard of, you get a different code system than when you're saying I want to order a drug, because you just don't say penicillin. You end up with a different code. It binds the codes to the use case in way that makes it much easier to know what kind of codes you need.
MR. BLAIR: How does nursing activities --
DR. MC DONALD: There are goals. They record goals. They record measurements. They have nursing plans, and there are actually some fields for those things too that one could go to and get better context.
DR. HUFF: So, I would include in your list there consultations, goals, plans. Those are very important.
DR. STEINDEL: As I listen more and more to this, I'm not sure how this is driving us closer to deciding on what type of recommendations to make for core terminologies. We are basically saying that these are the attributes that should be in an electronic medical record, or any type of system where enumerating delivery of care, and I think we already know that.
DR. HUFF: No, because I think the question comes in, aggregate data analysis. Now, we are trying to enable that. Are we in fact choosing code sets that represent names of aggregate classes? See, I think to me, that wasn't what we were trying to do.
DR. SUJANSKY: That's not what this implies, if you are equating those.
DR. HUFF: Well, whether you implied it or not, that's why I think it forms a boundary. I mean I didn't see it within scope for instance, for us to create a data element that said number of patients that received -- percent of patients that had adverse drug events during their hospitalization. That's a question. Maybe it is or maybe it isn't.
DR. MC DONALD: Or percent of patients with beta blocker treatment at post-acute myocardial infarction, which is a field that is required by HEDIS.
DR. HUFF: Is that the scope of PMRI terminologies, or is that more administrative terminology? Because I think this ultimately does either put some things in or out of scope.
DR. STEINDEL: In answer to that, I think that the PMRI terminology should be able to determine -- we should be able to use it to determine who is on beta blockers. But I think what percent of your patients are on beta blockers that are getting whatever, no, I don't think that we are looking at that. We are looking for a way to derive that from PMRI information, so that the base terminology should contain that type of information.
Likewise, the base terminology should contain information that we could derive public health statistics from, or public health reporting information from. And hopefully, it will be the same terminology that will be used by public health to reduce burden. But I don't think we should look at the requirement for public health as being the deciding factor for the PMRI terminology.
DR. HUFF: Part of this is to try and create expectations too. And what I mean by that is so often people sit in the room, and we're talking about these terminologies, and they are thinking that everybody is going to use them but them.
And so for instance, I would like to include on here that these terminologies should also be used in clinical trials, and to share clinical trial data, because so often people say, yes, that's great, but their behavior is to go on creating ad hoc terminologies for purposes of each clinical trial.
And so, I would like to say that we are trying to do this so that we can in fact tell people about adverse drug events, and have it be done the same way everywhere. And that we use these terminologies for the data elements that are used in clinical trials. And that clinical trials would actually start using the same data elements across trials, instead of making them up every time in each clinical trial.
And I think it serves a purpose of saying, okay, if we decide we're going to include a certain set of things in here, we could say this ties back to the use case that we thought, because we thought we wanted to be able to send prescriptions, or we thought we wanted to send results, or patient measurements or observations.
DR. STEINDEL: I think you bring up a very good point, and I think this gets to somewhat of the scope of what we are recommending. And I think we are also looking at a difference between a standard or standards, which could be more than one core sets of core terminologies. And we are also looking at as part of that terminology set, specialized terminologies that can augment it.
And the clinical trial is a good example, because clinical trial information sometimes requires a finer level of granularity than would be required of the average hospital system. And it may or may not exist in a standard core terminology. If you talk to the people from NCI, they are going to tell you that it probably shouldn't exist in a standard core terminology, because it's highly specialized. They are collecting much more information, and some of it may or may not be useful in the future.
I think a very good example from an area that I have just been looking at from a CHI point of view, where we were looking at laboratory result reporting, and one area that you need to report laboratory results for is organisms. And NCBI maintains an extremely good taxonomy of organisms. It has about 100,000 plus organisms in it.
SNOMED also maintains a useful terminology for organisms that CDC has adopted. It has about 17,000 organisms. Which one is better? Is it the more the merrier? No, because the NCBI taxonomy is designed for researchers. It's a different purpose. And yes, there are overlaps. The organisms that exist in SNOMED, exist in the NCBI terminology.
But if I was going out to a clinical institution, I would tell them to use the one that fits your use case best, which is the one that is more clinically suited. But I will also tell people that the other one is satisfactory, because it does cover the clinical cases, and can be used for research purposes. We might use that internally in CDC.
MR. BLAIR: Steve, do you feel like there is a better way for us to approach this organization?
DR. STEINDEL: One of the things, I was looking back at the documentation that we put together at the last meeting, where we talked about the core terminology, and hanging off of it the administrative terminologies and the legacy terminologies.
And the more I thought about that, the more I was looking at it as there is a core terminology that everybody can find use for. But then there are needs of specialty terminologies that augment the core terminology. And whether we say these are part of the core terminology, and we expand the list from beyond what Jim Campbell originally talked about when he presented the concept of a core terminology, where there were four terminologies in the core as a first approximation number, but generally a small number in the core.
In which case we would be looking at your very basic sets of terminologies that were pretty inclusive, that has very little overlap. But if we said we wanted to expand the core, and the specialized terminologies now come into the core, we are looking at a core that may have 20 terminologies.
Or we can say the core terminology is still the same definition we conceived of before. That these are our basic encompassing terminologies, but the specialized terminologies now are essential parts of the PMRI record. They are just not part of the core. They are used for specialized purposes.
And as an example, the ICO tooth number designation, we could include it or not include it, but that's an example of something that might be a very necessary specialized terminology.
MR. BLAIR: Could you give us a framework for how we might pursue the discussion.
DR. MC DONALD: A point of order. The data types weren't meant to be data types. Those were messages. Those should be done on the same share-specific data. At least that's what I was trying to say. They are not data types. Those are clinical messages.
DR. ZUBELDIA: Steve, the tooth example is a very clean one. There is no overlap. What do you do with the overlap between the specialized terminologies and the core terminology when there is a substantial overlap?
DR. STEINDEL: And this is an issue that I think we have brought out a little bit. And we have skirted and we have addressed both, and that's the need for mappings. It's the need for identification where there is overlap. And I think the NCBI terminology is a very good example of that, because we know there is overlap.
I'm glad Betsy just walked in, because I was about to invoke UMLS in a complimentary fashion even. It's one of the purposes of the Metathesaurus, to allow us to identify the areas where there is overlap, and where there is extension. If we take a look at something like the NCBI thesaurus, if we go into the UMLS and look at organisms, in the area of UMLS where there are organisms, we'll find multiple terminologies listed there with an organism.
But three come very distinctly to mind. One is MESH. The other is SNOMED. And the third is the NCBI taxonomy. In many cases, all three of those have the same organism. They will have different identifications within their own terminologies, and will map to the same UMLS GUI. So we can the UMLS GUIs to identify the areas where the concepts are the same.
But then also we know from the fact that the NCBI terminology may have an organism that is used for research purposes, and we find that it's listed in no other terminology. And yet we can say both can be used for organisms where they overlap. You can use any of the three. Or you use the UMLS GUI. It's the same.
And if we designate a terminology as being part of the standard core terminology, what we are saying is the code that is represented in there is the one that we consider to be the primary code.
DR. SUJANSKY: I don't know that we are getting at the use cases, and so forth.
DR. STEINDEL: No, we are not.
DR. ZUBELDIA: The problem is that if you say you can use any of the three, then you are going to run into problems of electronic exchange of data. You can use any of the three for your internal purposes, but as soon as you try to send data to another provider --
DR. STEINDEL: We're running into a difficulty. The issue here is that what we are relying on is the ability for UMLS to interchange those various code sets. So, if you did send one that wasn't used, then someone could do a look-up in the UMLS to translate it into the code set that you do use.
So, we are adding a step to it, but we are not necessarily making the data semantically -- we can still exchange the data, and exchange the data in a semantically understandable fashion. And ideally, people would use what was designated as the primary terminology, because they would want to avoid the extra step.
PARTICIPANT: Well, that begs the question. We already have the UMLS, so why specify -- why make any further recommendation?
DR. HUFF: I would have also not in philosophy, but in principle agreed with your characterized of specialized terminologies versus the core. And what I mean by that is that seems to lead to a principle that says we are going to include or not include things based on the frequency of use.
And so, the difference between the 100,000 organisms that are at CDC, and the 17,000 that are in SNOMED is not in any way qualitatively different. It's just how often you would see those in clinical medicine, versus in a clinical trial or some other thing. So, what I would say is we want a principle that says all of those things in fact are core.
I would have included all of those as core, but I might have said that we use frequency of use as a way of prioritizing what we include first. And so, if we wanted to do 17,000 now, because we know those are the most frequently used, and the most important in clinical medicine, great. But I would have said our goal was to be all 100,000 in there, because they are ultimately going to be pertinent in some domain that we want to be consistent, reusable, comparable in the exchange of patient data.
DR. STEINDEL: I agree with you. If you remember, my original statement is do we want to limit the core to the concept that Jim Campbell had originally of core terminology, or do we want to expand it to include these? And you are making the statement that we should expand it to include it. And I have no problem with that. I basically agree with that.
And the examples that I used, were I think very clean examples. And I think some other terminologies that become very important, that you can talk of in the domain of specialized that are very frequently used are like the various nursing terminologies.
So, they have a very high frequency of use, and yet if we go by the criteria that we have established, we probably would not think of them as basically core terminologies, based on the report that we have seen. And yet, they are very critical for any type of reporting system that we use.
DR. SUJANSKY: Well, there you are raising an issue though of domain coverage versus the technical criteria that we have decided are required or important for these terminologies.
DR. STEINDEL: One thing that was mentioned earlier today was is there a way that we can use this report to get some terminologies to change some things, so that they would fit better into the technical criteria. And I think a lot of the nursing terminologies may be a good example of that.
DR. SUJANSKY: Right. So, first, we should look at how much of that content is already in SNOMED as part of the initiative of what they have done already to incorporate a lot of that work. And those terminologies that have content that is unique in some way, but they don't fulfill the technical requirements, what can we do to move that along?
DR. STEINDEL: Yes.
DR. SUJANSKY: But circling around to our point of departure, I think the thing we were discussing earlier is, is this process of identifying these use cases or scenarios or specific requirements for exchanging, specific reasons we want to standardize PMRI terminologies, is that a useful thing to do right now or ever? Personally, I think it is. I think it's an important thing to do. The question is should we do it now?
DR. COHN: Walter, I actually think it is. I was just wondering, isn't that what your previous document scope and criteria is all about?
DR. SUJANSKY: It's suggesting that we need to do that.
DR. MC DONALD: What's it?
DR. SUJANSKY: It is defining the requirements for the standard terminology work.
DR. MC DONALD: In general?
DR. SUJANSKY: Yes.
DR. HUFF: This document basically said how would you know a good terminology if you saw one? It doesn't say terminology good for what? And that's all I'm trying to do.
The other thing that I think we could say here maybe specifically is to say something that are not within scope. So again, I would say for instance we are not trying to adopt terminologies that would necessarily be good for indexing medical literature. I didn't think that was the scope of what this particular thing was about.
I didn't think that we were trying to adopt terminologies for reporting of sort of the classical use of classifications. Admitting the fuzzy boundary between what is a classification, and what's an other thing. Basically, I'm trying to distinguish between what we are doing here, versus the terminologies like ICD-9 codes that are used for reporting of aggregate public statistics.
And I'm thinking that we have gone as far as we can usefully in this meeting. And I would be glad to take it as a homework assignment, to try and add some more details in here. But we could also do it as a group, as we went along, and we think about a particular thing, say we don't think that one's good because we don't think that's in scope.
And we could add that to our list of what's not in scope and/or we could come to things where we think here is a particular scenario where that would be used, so, we want to include that one. And we would just extend our list to make sure that we included that as something that was part of our scope.
DR. MC DONALD: I would actually support giving him work.
But I want to come back to the importance of really looking at the concrete realities. And I can give you a lot of examples of where we make up -- it's sort of the experts at the airport syndrome. And we have been doing this in medicine for as long as I have been in the field. So, you get 10 experts. They come and write up some thing. It's a little bit like a papal statement, it's just the official truth.
I don't mean to be hard on any particular religion, but it's just official truth, and people kowtow around after it. And I don't know about the papal statements, but whenever you actually take these things and you confront them with reality, they are never right. They just about are never right. And I can give you specific examples.
And what you get into in sort of the theoretical vocabulary development is sometimes the same problem. So, we could say, okay, we've got all the codes we need for reporting cervical pap smears. But look at what happened at the Bethesda statement. They have defined a sentence for each of these findings. That's what you've got to report. That's what people report.
So, when you get into the real life, it doesn't work out the same. And there is the great beauty of saying we are going to actually send this thing. You can kind of test it against what really goes in those things, and see if your vocabulary is good or not, it needs to be extended, or you're even close.
DR. SUJANSKY: Right, I agree. I think we need these cases. The thing we want to avoid is at the end of all of this, that someone says well, that's great. You guys spent all this time and money, and we integrated all these terminologies, and spent millions of dollars. Does it do X? And X turns out to be something that is really important, and because it wasn't recognized as a criteria that it needed to do X, but it doesn't do X.
DR. MC DONALD: Variably we have to count on having some Xs. If we set out to eliminate all Xs, we're here for another ten years.
DR. SUJANSKY: Well, yes, but --
DR. MC DONALD: We just have to say does it do the Xs that are sort of on everybody's minds.
DR. SUJANSKY: Exactly. But we need to have some statement of what are going to be the Xs that it supports.
DR. MC DONALD: Right. I think this is what we are trying to do, is have use cases of what can be sent.
MS. HUMPRHEYS: As a person who walked into the room when it was all done, the only thing that seemed -- in the first rough draft you better mention public health in there somewhere, other than just registries.
DR. MC DONALD: What specifically?
MS. HUMPRHEYS: Reporting of cases to the public health department.
DR. MC DONALD: Reporting of reportable diseases. It's kind of redundant, isn't it? Report of communicable diseases sounds better, but it's not all there is. It's more than communicable disease reporting. There is a better way to say that though.
DR. HUFF: That was the very first thing I said.
DR. SUJANSKY: And I didn't write it down.
DR. STEINDEL: If you want to put that in there, it's identifying diseases to report to public health.
DR. MC DONALD: The exchange is the whole thing.
MS. HUMPRHEYS: The clinical system still has to be able to send everything that needs to be in that message.
DR. STEINDEL: But it's identifying what needs --
MS. HUMPRHEYS: That's a piece of it.
DR. MC DONALD: You still want to get it there.
DR. STEINDEL: But actually putting it together and sending it, that may be a differently defined function.
MS. HUMPRHEYS: If it needs vocabulary, we want it covered in this, don't we, both of them?
DR. STEINDEL: Ideally, I think it's public health's responsibility to pick up the vocabularies that are being used by PMRI. Not PMRI to pick up the vocabularies that are being used by public health.
MS. HUMPRHEYS: I agree with you, but I don't think that statement up there says anything other than that.
DR. SUJANSKY: It just saying let's test our solution against this activity.
DR. HUFF: Again, if people are kind of comfortable with the direction, I would take homework and try and fill this in, in more detail. I would start by looking at the kinds of messages that exist in HL7, that exist in DICOM, the kinds of things that I'm aware of in clinical trials reporting, in tumor registries, and disease-specific registries, et cetera.
And just say we think those are in scope. And use that as sort of a test when we come to a given terminology. Does that fit in one of these things that we're trying to do? And is it consistent with things? And then probably even more valuable is every time we come to stuff that we think is out of scope, and we can for now that's out of scope, and that puts a boundary on what we are trying to do.
MR. BLAIR: I think that is a good thing to do. I have mixed feelings about it. I kind of expressed this to you, but I'm speaking to everybody else right now, so they can hear my comments on it. Which is that pragmatically I feel like we get real benefits out of using HL7 messages as a driver for what we need for semantic clinical specificity of terminologies.
So, yes, I concur with that. I just get a little bit nervous that there are areas of patient care, and expanded views of care such as alternative medicine, such as other ambulatory functions that I just hope that --
DR. HUFF: You haven't convinced me at all, Jeff. And what I mean by that is I'm not talking about politically whether -- because HL7 messages are fully capable of sending alternative medical observations or orders. Okay, there is absolutely nothing there that prohibits that.
And anything that you want to "share" has to have this combination of syntactic and semantic content. And you can't establish one without the other. If you don't say something about the field, and how this is represented, you can't do it.
And a case in point is that patient safety taxonomy that was shared, that's wonderful work. And then I asked myself, how is it supposed to be used? How would I use that to communicate some information from myself to somebody else about patient data? And they don't say that. And for it to be useful, somebody has to say that.
And so, I'm not trying to exclude anything. And I'm not trying to make the scope of this by HL7 messages, because we know there are lots of things that don't exist as HL7 messages. But what I'm saying is that everything that you talk about has to be, I think, talked about in the sense of data sharing. And data sharing always involves a format of exchange, as well as the codes and semantics of data that are involved in that exchange.
And this will ultimately lead us to other things that need to be done. And when we get that list done, we'll say oh. And if we adopt SNOMED, there are at least three ways you can say the same thing. And so, the next body of work is going to be saying which one way would we like to say that, instead of adopting N ways to say that. Or do we want to allow N ways? That's an open question.
But I just don't think you can frame the problem, nor get to a solution which ultimately is interoperability and shared data without putting it into the context of semantics and structure of that exchange.
DR. MC DONALD: The feds just specified that HL7 and DICOM would be used within the federal government for their appropriate uses. So, we might as well take advantage of that.
MR. BLAIR: That was one of the results of our recommendation.
DR. MC DONALD: It's a nice harmony.
MR. BLAIR: My concern, and I think Stan said it very well, he said is not saying that he wouldn't expand this beyond existing HL7 messages to these other areas. He said he was starting with that, because that's fairly well defined. And that was my concern, is that it doesn't end at the boundary of current HL7 messages.
Do we have an agreement on that?
[Several people agree.]
DR. SUJANSKY: Well, Stan, I know you are taking this upon yourself to at least begin this process.
DR. HUFF: I'm just hoping somebody would say, no, no, don't do that.
DR. STEINDEL: Is anyone opposed to this extra work?
DR. SUJANSKY: But I wonder how much time we should take today?
DR. HUFF: I think we're at the end of what we could do as a group here, unless somebody wants to say three more big areas where we haven't thought.
DR. SUJANSKY: Well, we could brainstorm as a group if we think this is important. I'm not saying you are not able to do this adequately, but I think there are a lot of different perspectives, and there will be different kinds of ideas from different perspectives. I don't know how much time we want to spend on it today. Obviously, work that can be done outside of this meeting should take second place to work that should be done inside of the meeting. But we do have the rest of the day.
DR. ZUBELDIA: Will the recommendations include a specific vote for UMLS as the bridge to the multiple terminologies?
DR. SUJANSKY: Could be.
DR. HUFF: Well, I'm in agreement with that. I wasn't seeing that as part of this work. But that's more like I would think sort of an implementation strategy that the committee ought to decide on, which I'm totally in favor of.
DR. MC DONALD: This is very useful, I think though, to have a white paper. It's so much easier to attack a document than it is to talk free for all. So, that if Stan is willing to volunteer to write something, I think that we would end up with more progress, faster.
MR. BLAIR: There are a couple of other issues that we wanted to deal with this afternoon. Are we ready to go on to the other ones? Are you saying you feel the issue of defining the scope was something that we can proceed further on at this point?
DR. HUFF: Are people comfortable with this direction, or do you think this is not useful? In which case, I would be glad not to do it.
MR. BLAIR: Steven, do you feel --
MS. BURKE-BEBEE: Steve had to go to a conference call. He'll be back shortly.
MS. HUMPRHEYS: But he was in favor of it.
Agenda Item: Review of Issues and Actions to Address Them - Jeff Blair
MR. BLAIR: Okay. Anyone else?
Two other issues. One was missing information. And whether we needed to have Walter go back to get additional information for clarification, or how we should get the additional information for clarification on the open-ended issues.
Walter, did you have a suggestion on what we do with those, like where the question marks existed? How do we want to reconcile those?
DR. SUJANSKY: Well, not all the question marks are the same. I think there are some that that I should definitely pursue, specifically in the area of licensing. I think the question marks for those terminologies among those ten that we have identified as certainly viable candidates, we should pursue licensing.
I think we should pursue some of these issues of concept orientation. There are nine terminologies in that intermediate group that would be ruled in or ruled out just on the basis of concept orientation. So, I think it would be pursuing those. And some of those have good scores on some of the other characteristics as well.
MR. BLAIR: Are those things that you can do? Do you need any help from the subcommittee to do that?
DR. SUJANSKY: No, I can do that. And part of that may entail actually getting copies of the terminology. So, if I can get the endorsement of the subcommittee to seek copies of those actual terminologies as needed, then I can try to do that.
MS. HUMPRHEYS: I would think that it would be a good idea to get clear -- and some people in the room may know this -- but I think it would be a good idea to get clear exactly to what extent and which of the nursing terminologies that fell into the lower two categories are actually being incorporated, or are incorporated in SNOMED CT to some extent or not. You would already be pursuing perhaps with others, the situation about those questions of concept orientation.
DR. ZUBELDIA: Walter, are you interested in market penetration, market acceptance, how much use there is today?
DR. SUJANSKY: Yes, we have data about that in the questionnaire, self-reported data. Do you have access to other data? So, we are interested. We have data to support it, but I haven't incorporated that into the analysis yet. What I would like to do is first -- there are a lot of different factors that contribute to which terminologies you want to look at more closely. And some of them we need to look at more closely to decide if we want to look at them even more closely.
So, there is a sequencing issue. Do we want to get into the subjective area of looking at market acceptance of one versus another before we get more information to determine if they are concept oriented or not? I would like to do the later part first, and just kind of decide on the technical criteria, just because to some extent it will take less time I think, than getting into the more subjective areas, just as an issue of efficiency.
MR. BLAIR: That makes sense.
DR. SUJANSKY: And then once we are looking at a short list, if you will, a final short list, we can then look at those specifically all three of those that I mentioned, the governance structure, the funding mechanism, and the market acceptance in a more subjective way.
MS. BURKE-BEBEE: When I was calling to the different terminology developers about the survey before it was sent out, I know there was a person, and Carol Bickford might help me with this, from the nursing perspective, to answer your question, that in fact is responsible for the mapping of different nursing terminologies, not all of them, to SNOMED. And I looked at my documentation, and I can't find her name, but she in fact was going to participate in answering the terminology questionnaire.
Carol, can you remember?
MS. BICKFORD: This is Carol Bickford. Deb Konicek is the nurse at SNOMED who has been integrally involved in the incorporation of nursing terminology within SNOMED. We just had a telephone conference call with her, and she identified that they do have a table which maps all of the nursing terminologies to each other as part of that work. And they do indeed make that available to individual licensees who have already licensed one of the nursing terminologies. So, so that is available information that might be helpful in determining that work already.
DR. SUJANSKY: I guess the more important question in my mind is has SNOMED actually incorporated content, brought it into SNOMED under the SNOMED authority, and the SNOMED editorial process?
MS. BICKFORD: They have the diagnoses in.
DR. SUJANSKY: For which terminologies?
MS. BICKFORD: NANDA is in. Home health, OMAHA. But some of the nursing terminologies already have NANDA in them, and have added to that. They have done the interventions, and they will have the outcomes. So, essentially, all the nursing terminologies that they have been able to secure, will be in by January 2004, was her statement.
DR. SUJANSKY: That SNOMED has been able to secure will be in SNOMED?
MS. BICKFORD: Yes.
DR. SUJANSKY: Will actually be incorporated in SNOMED?
MS. BICKFORD: That's my understanding.
DR. SUJANSKY: Okay.
MS. BICKFORD: But you would have to confirm with them.
DR. SUJANSKY: Okay, we'll follow-up. What did you say the name of the lady was?
MS. BICKFORD: Deb Konicek. I can provide the contact information. Are you here tomorrow?
DR. SUJANSKY: Yes.
MS. BICKFORD: I'll bring it tomorrow.
DR. SUJANSKY: I think that's an important question. I think there are also some other content issues I would like to look at. For example, in the drug area I think I would like to learn a little bit more, and I can just do this by talking to Stuart.
MR. BLAIR: Are these topics things that you are envisioning would be in the revised report for April 17?
DR. SUJANSKY: Yes.
MR. BLAIR: I don't know, I failed to mention when I discussed the process earlier this morning that Walter will be providing to us a revised report based on the information from today on April 17.
Other areas of additional information that Walter, you felt you needed to pull together, or anybody else on the subcommittee felt that we need a supplement?
Okay, let me go on to the next topic. Am I going too fast?
DR. COHN: No, you're doing fine, Jeff.
MR. BLAIR: The next topic is on May 21, is the day we have put aside to hear from users of the terminologies, so that we could identify strengths and weaknesses and limitations and constraints. And in part to verify or validate the information that we have received from the terminology developers, and also to also understand other issues related to their use and as somebody said, their use and abuse.
So, I think that there are a number of things that have come out of what you have told us today, where we could begin to craft questions. I know it's something that in the past I have often worked with Michael Fitzmaurice in coming up with questions for the testifiers, but in this case before I do that, or Walter does that, I would really like to solicit from the subcommittee as a whole, issues that you would like to explore with users of these terminologies, that you feel would be helpful.
DR. FITZMAURICE: Jeff, this is Mike. One thing I think that would be important, and Simon would say it if he were here, is that in the business case we are using vocabularies. If these are the users, then it would be wise to have them give us a couple of words, give us a paragraph on the business case. How does it save them money, or give them additional value than what it would cost to use terminologies?
MR. BLAIR: I think that that's an appropriate question. I think a lot of people struggle with an answer, because they haven't made an analysis. But, yes, I'm agreeing that we should include it.
DR. FITZMAURICE: I guess I would also ask them in what areas are they currently using terminologies, and what do they think the three next fruitful areas are? I can think of computerized physician order entry for drugs, for laboratory tests, maybe for reporting radiology results. But I wonder if they would agree? They are right on the ground floor doing this.
MR. BLAIR: Right.
DR. FITZMAURICE: What information would they like to have that they can't get because there is no hook to that information to pull it down? It may be some particular bits of medical knowledge that they could obtain if only their system talked the same language, and could pull it out of the National Library of Medicine, for example, or some other database such as a database of decision rules that is being developed by HL7 or by IMKI.
MR. BLAIR: The other thing was that, Suzie, there are a number of folks that commented to you that they felt they would be candidates to testify?
MS. BURKE-BEBEE: Well, actually what I'm getting, Jeff, oh, yes, there are two things here -- I'm remembering now what you said. Some of the work that I'm doing with CHI and I'm trying to bring that to the table here, and maybe Jared will mention some of this tomorrow.
But it was suggested not just the vendors putting the vocabulary into their products, but also there was a suggestion that we might want to pursue vendors that are doing work with servers, databases, and implementing the terminologies in that way.
The second part of this, in phone calls that I'm getting, it's still from the terminologies developers that want to expand a little bit further. So, it made me wonder when was it you Kepa, or someone that referred to earlier sharing this document. Someone mentioned sharing this with the terminology developers that responded.
That would probably pique even further, their interest in wanting to participate. But when you and I talked about that too, Jeff, we talked about the time element. Because we had talked about the three different panels that we might have, which could be the vendors of the software, could be the vendors of the servers, but also the actual users that we wanted to bring in.
Do we have time? Should we make the time to bring to answer some of these issues, the terminology developers as well?
DR. SUJANSKY: Terminology developers? The ones who already submitted the questionnaires?
MS. BURKE-BEBEE: Yes. I'm just throwing out as a possibility of being here to answer those questions if that would be effective or not. Or are we just going to possibly put together the questions? Walter will handle that?
MR. BLAIR: The inclination that I have is that we stick with the decision that we made I guess back in December that we not invite terminology developers for testimony, because if we do, we've got 38 developers that we would have to invite for equity. And I think it would be more appropriate for Walter to get clarifications directly for the questions that we do have remaining.
We had discussed this at that time. It seemed as if everybody agreed with that viewpoint. Has that changed?
DR. HUFF: I would agree. I think I would like to see it done exactly as you described. Send Walter's draft document out to the folks and say, do you disagree with the characterization we have of you in these areas? And give them a chance to respond, and correct as appropriate, not necessarily do everything they say.
Because if there is no basis in fact for what they are saying, or in fact contradictory information based on looking at the contents of their terminology, then we leave it the same. But they have an opportunity to respond. But yes, I don't think we want the 38 people to come and give us a sales pitch about their terminology. I think we want users of the terminologies to tell us which terminologies they are using and why, and why they are useful, and what's the business case.
DR. SUJANSKY: Stan, I would like to respond to that. I'm not sure it would be useful to essentially invite the developers, the people who have submitted the questionnaires to comment on the evaluation, because they will take issue with a number of things, undoubtedly. And the evaluation process and the scoring process was necessarily subjective to a certain point, and done with best effort, and least bias as possible.
And I think inviting 38 terminology developers to make comments on a very large set of assessments against those responses will create a tremendous amount of additional data, and additional work that will slow down the process significantly.
MR. BLAIR: I would agree. I also feel as if changing an evaluation by one or two or three points really doesn't change the overall balance that you have. I do think that for you to solicit additional specific information, or get clarification in terms of coverage or concept, I think that's appropriate.
Stan, will you accept that? Or do you feel very, very strongly that we have to go back to all these developers and have them critique our critique? We didn't really have that in mind. That's kind of a significant expansion of the scope of --
DR. HUFF: I'm happy to do what the group wants. If we have time, I think it's a better way to do it, because again, I actually think it's just a matter of whether we have time to do it. I think there are people who would argue that it or isn't this thing or that. And that's actually good. And if we have done it wrong, because it adds to the credibility of the report.
If we are saying things in this report that somebody could make a convincing case is inaccurate, and we are basing our judgment about these terminologies based on that content, I think it brings in the question of the credibility of the whole report.
MS. HUMPRHEYS: I would strongly agree with what Stan says. It seems to me that I appreciate everything that has been said about the effort, and so forth. It does seem to me that if you are going to take a lot of information from people and summarize it, and then this group is going to make recommendations, which as we know, actually have a major impact -- congratulations, they had a major impact. Last Friday, they had a major impact.
And then somebody is going to come along, this was all based on this original report, where somebody totally misinterpreted something, or I didn't understand the question, and I didn't provide the right data. Even if the actual probability of that making a big difference is low, I think you've got a perception problem which is major coming out a national committee like this.
So, I don't think that you have to say hey, critique this thing from start to finish, and object to every single thing we have said. I think you just have to send it out and say, here is the first rough draft.
And if you want to be able to distribute the data, like people said they would like to be able to look at the source data, because they thought it would be valuable, you are going to have to ask them their permission to allow that anyway. So, if you well, okay, can we share this data, and here is the draft report. And we've got a lot of questions in here which we are going to follow-up if we have questions with you.
I just think that the NCVHS is going to look very bad the first time somebody starts writing letters to the secretary saying, you know, they made this recommendation based on this report. They never allowed me to read the report. I never had an opportunity to object to the characterization of my thing gets to be a major problem.
So, I see the work, but I see more of a downside later on.
DR. SUJANSKY: Was this done with the message format questionnaire and report?
MR. BLAIR: No. If we do that, I think we really need to go back and re-negotiate and expand the consulting agreement. This is beyond the scope of work that we had envisioned. It would be a significant expansion.
DR. FITZMAURICE: I'm convinced with what Betsy and Stan have to say, that we should go back, and we should look to see what resources will it take, what time does Walter have, assuming that he has only 24 hours in the day, like the rest of us. And what resources can we come up with. We don't have enough information to make that decision now, but I think it is a pretty strong effort. It will make a national difference.
And since we have done it the right way by getting the criteria, assessing things against the criteria, I think the right way is also to go back to the companies and say, here is how you shake out. Do you want to tell us anything more? It could sizably expand the scope.
DR. SUJANSKY: Well, it will also extend the duration of the project.
DR. FITZMAURICE: You mean we are likely to miss the September deadline?
DR. SUJANSKY: Yes. But if it sounds like it's very important. And if it's that important, then it sounds like it is worth it.
DR. HUFF: Again, especially for instance where we have got the four essential criteria. And if one of the companies that didn't meet one of the four essential criteria, fell below the line and said, we are concept-based. You marked me as not. I think they need to have the opportunity to present their case. And you're right, it takes more time.
MR. BLAIR: Didn't Walter already identify those with question marks to get clarified.
MS. HUMPRHEYS: Yes, but I think that the issue that -- and I think this is a nominal piece of work, and I actually don't think we'll see much changed based on this, but it is sort of like I have a question whether the answer to one of the questions was correct, or whether the people interpreted it correctly.
I look at this and I say, I don't think so. One of the ones that met them all, I'm not sure it really does. So, if there is somebody down at the bottom who apparently gave information which clearly excluded them, but some reason gave a screwy answer to the question, I really don't expect this to be a lot of people.
The other question is whether this in some sense is something where additional resources are necessary, but they don't necessarily mean transmorgafying Walter into somebody with a 36 hour day. Because somebody comes in and they say, we've got the whole spreadsheet laying out in front of us, and they come in and they either provide additional information, or they don't.
And it may be that somebody else can at least do the first cuts and see what these responses are, really changes anything or not.
MR. BLAIR: Is Marjorie in the room?
MS. GREENBERG: I'm here. I was thinking about this. I don't know about the message format project, although I do remember we discussed it in a number of open meetings. So, there was an opportunity, I guess for that. And I'm thinking that's the case here as well, that some of the terminology developers probably are here today who submitted responses. Others may be listening.
So, you are likely to hear from some of them. Certainly, those that are following this right now, you are likely to hear from them. And the document is available to certainly anyone who is here, and anyone who requests it, for that matter. It's a draft document, but we make available anything that is being discussed by the committee.
So, at a minimum, we can't avoid that type of feedback, that I would think we would at least need to evaluate the legitimacy of it, or whether it does provide information to Walter and his analysis, other than what he already had.
We certainly have in the past -- I'm thinking back to the core health data elements report, and that goes back quite a few years, but I remember sending that out, and uniform data sets, and all of our work in that. We always sent them out, and we collected a lot of information. Similarly, we sent them out and gave people an opportunity to comment.
Generally, we didn't get that many comments. As Betsy suggested, the stakes might be a little higher here in that specific terminologies are being evaluated, as opposed to sets of data elements. It certainly would be good, I think, to give people some opportunity to look at the assessment and comment on it, though I don't think we would want to give people a lot of time, because I don't think we have a lot of time.
And again, it might be that the responses could be reviewed first by Jeff and the staff or something. And then those that seemed to really need addressing, pass them on. I don't know. I hear what Walter is saying too. That it could end up adding quite a bit of time.
DR. SUJANSKY: The thing I'm concerned about is really getting into protracted discussions with people about subjective issues, and having to explain principles of terminology merits and so forth to 10 or 20 different people.
DR. HUFF: I was envisioning somebody calling up and saying I'm concept-based because of this. And you saying, okay, I'll look at that.
DR. SUJANSKY: But that's the kind of discussion I just described, Stan. And then I'll say, well, here's why I don't think you are. And then they will say, yes, but that's not quite right.
DR. HUFF: It's not their vote. It's their opportunity to say to you this is why I think you misunderstood. It's your determination whether you misunderstood or not, and how you score that category.
MS. HUMPRHEYS: And my feeling is that I believe there are probably are -- I agree with Marjorie. I think that you will not be inundated with information. And I think that there are other people who, given the information they provide you, can be in some sense, the group that says, yes, it is concept-based, or no, it isn't.
I'm just thinking that the committee has put in a huge amount of work. You have put in a huge amount of work. I would really not like us to get to the end, and for some reason, have the work carry less weight because somebody said, well, it really -- I think this process was flawed in a major way. They never asked us if he misinterpreted.
I'm just thinking that it would be too bad to get too far along the pike and then get that. And I actually don't think it's analogous to the message standards, because I think that the consideration of the message format standards, there were fewer of them that were being considered. There were always people in the room that were part of the people that wrote them. And here, we just have this large array of people, and it's hard to get them around, to give them their two cents.
DR. ZUBELDIA: The way I would approach it, I would summon this draft, not the spreadsheet, just this draft, and say this work in progress. This is where we are going. If you want to submit any supplemental information, you have until such a date to submit supplemental information.
DR. SUJANSKY: In writing. That's the other thing, can we insist on comments in writing?
DR. ZUBELDIA: And make it unidirectional, where they submit supplemental information in writing, not to expect any response or discussion.
DR. SUJANSKY: Right. And I would ask that those be sent to the NCVH staff, not to myself. And then that can be compiled there. Yes, that would be great. And a deadline, yes, that would be good too.
DR. ZUBELDIA: And they know what they submitted in the first place. I don't think there is a need for them to see the spreadsheet, to see how they compare with others. I think that's just too much.
Send this draft report, March 25 from Walter, that we have today. You send them that and say this is where we are today. It's a draft. It's work in progress. If you want to help us by submitting supplemental information, and help us with cleaning up some of the question marks, feel free to send your information. But we have a deadline of whatever the deadline is.
MR. BLAIR: Okay.
MS. GREENBERG: You'll send it out, Jeff? Or Suzie will send it out? We'll send it out, because if Walter sends it out, then he gets the responses.
DR. SUJANSKY: And then Jeff, you and Michael and I will talk about modifications and so forth when these come in.
DR. FITZMAURICE: Marjorie, shouldn't it be sent out under Jeff's signature?
MS. GREENBERG: That's fine. I think an e-mail.
DR. FITZMAURICE: It would be less than a survey then.
MS. GREENBERG: Yes. That's not why I don't want Walter to send it out. I feel like no matter how often you send out a message and say please respond to Joe, they respond to the person who sent out the message. It's just extra work for him. So, I think we should send it, either you Jeff, or us.
DR. SUJANSKY: But if we engage in this, and it sounds like we are, we should recognize that this may delay the entire process past September or whenever, and that may well happen.
MS. GREENBERG: What we need to do is give them a date. And when we see what we have at that point, and then we'll have to reassess how this might change the scope or the time.
MS. BURKE-BEBEE: I agree with this. I think it's a great idea. I think what we are assuming though is we are going to get the right responses that we want. There are some questions that we have, that we wanted to get answers for. What happens if we don't get the answers?
DR. ZUBELDIA: I think it's a separate and independent process. Walter still needs to investigate the questions that he has.
MS. HUMPRHEYS: Yes, and I think that in some sense, can you decide whether you are going to refer to that separate process in this message or not.
DR. SUJANSKY: I prefer not to, because at the time that I follow-up to get that additional information, I don't want to engage in those conversations at that time about the scoring. All that needs to be submitted in a written form.
Any comments related to disagreements about scoring and so forth, I think need to be submitted in a written form. And then the issues where additional information is needed, the question marks, I will address directly with them. But that's all that I will address directly with them at that time.
Does that sound reasonable?
MR. BLAIR: Okay, getting back to our crafting of an agenda, and testifiers for May 21. Suzie alluded to this, but let me indicate it again. We thought that the folks that we would have that could give us useful testimony would be the users of the terminologies. And we thought of them kind of in three groups.
One would be vendors that are using these terminologies as part of their application software offerings. A second would be terminology servers that similarly are winding up either providing enablers or browsers or other support functions for the use of these terminologies. And the third would be end users such as providers themselves that might be using these.
And specifically, we are looking for users that have experience with these terminologies, and can give useful critiques in terms of their strengths and their weaknesses.
Any comments, suggestions? Are we proceeding the right way? Do we have the right target audiences?
DR. FITZMAURICE: Jeff, this is Mike. I'm not sure what the terminology servers are, the enablers and the browsers.
MR. BLAIR: Well, there are two of them that I think of but there may be more. I think of Apalon(?) as a terminology server. There used to be Cyber Plus, but I think it became Health Language Center. And I don't know whether the software that enables the MEDCIN product would be also considered a terminology server or not.
Walter, can you help clarify this?
DR. SUJANSKY: I believe that application is just a browser. That's my recollection.
DR. HUFF: I don't think you would think of it as a terminology server. It's very application-specific.
DR. FITZMAURICE: These are the people who are the go-betweens, between the things you want to connect with, and your application in front of you. The Apalons go out and say, here is how you use the vocabulary to build that bridge. What you need and where it's located. How you find it, like terminology consultants might be another way to say it, at least in my mind.
MR. BLAIR: Okay.
DR. FITZMAURICE: I would also include hospital CIOs, because they have to make the hard decisions about is it worth putting a vocabulary into their systems. And they may have some insights that the vendors don't have in terms of trade offs that they have to make.
MR. BLAIR: I would really like to solicit either the names of entities, organizations, or individuals that you feel would be good testifiers on May 21.
MS. HUMPRHEYS: We have been in discussions with Dave Riley, who is talking to us on behalf of the DoD, because they have implemented the use of MEDCIN in conjunction with other terminology, and are talking to us about how they might -- essentially how we work together to migrate forward in that environment if the SNOMED deal is in place.
So, I think that he or someone that he works with in the DoD used to be in the DoD, and now he is not, but he still works with them, that might be a place to find someone who would be a good really user of MEDCIN to have talk.
MS. BURKE-BEBEE: Jeff, I think I'm remembering what you were referring to with the CHI. There were some suggestions that they may have some interest to come forward with some questions too. And maybe even some other people that might be appropriate to testify. Did you want me to pursue that?
MR. BLAIR: Sure.
MS. HUMPRHEYS: There are plenty of LOINC users that Clem knows.
DR. SUJANSKY: I don't Stan, how long it may take you to come up with a first draft or so of the use case, the requirements and so forth that we talked about. But if that may be available soon, it might help to inform the decision about whom to invite to comment. For example, we talked about clinical trials maybe being an important constituent. So, then the vendors or users of clinical trial systems might be important.
DR. HUFF: First, I was wondering who was going to speak, and now I'm wondering how we're going to pare down the list.
DR. SUJANSKY: That's why it may be good to have targeted invitations. I don't know how this was done before, if sort of a broad cast to a great many people was done, or it should be more targeted.
DR. HUFF: I wasn't planning to do a hugely elaborate thing. So, my list of stuff I could do in a week or so.
DR. MC DONALD: This I should maybe ask Marjorie if I can talk about it. I wanted to use the word LOINC.
MS. GREENBERG: Are you trying to sell it someone?
DR. MC DONALD: No, I just thought what I'm forbidden to say in this context.
MS. GREENBERG: Yes, go ahead. You want to provide some information?
DR. MC DONALD: It's actually just a question. LOINC was also supported under the CHI proposal. And strategically, do we want to pretend like that didn't happen, or that did happen?
MR. BLAIR: We've had some conversations about that. And essentially, what I indicated, and I think Steve Steindel felt the same way, and I discussed it with Simon as well, we're out of synch with respect to lab LOINC. There was a need that CHI had to be able to designate that for their own purposes. But at the same time, they felt somewhat uncomfortable, because they would like to move in parallel with us, and be able to use this as a forum for the selection of terminologies.
The thought that I had was as of September, we will be making our recommendations. And there is a very high probability that LOINC will be part of those recommendations. And that as far as I was concerned, we should tolerate the fact that we are out of synch.
The other alternative would be for us to grossly accelerate our terminology selection process. And I think we all felt as if the process that we have has value. There is a certain amount of credibility and integrity to the process that we have, and people seem to feel that it has value. And therefore, we are choosing to retain the integrity of our selection process, which still brings us to September, and tolerate the fact that we are out of synch with lab LOINC.
Okay, does that answer it?
DR. MC DONALD: Yes.
MS. GREENBERG: Can I ask a question. I'm about ten minutes before here. We're talking about sending out the current draft, right?
MR. BLAIR: Current draft of what?
MS. GREENBERG: Of this report.
MR. BLAIR: The one that Walter just did?
MS. GREENBERG: The one that we had the discussion about.
MR. BLAIR: Yes.
MS. GREENBERG: But at the same time, Walter is going to be seeking clarification. So, some one will get this report, and they will see these question marks. So, at a minimum we should say our contractor may be following up with you with questions he has, or something.
MR. BLAIR: It is awkward.
MS. GREENBERG: I can see this as ships crossing in the night here. On the other hand, if you wait until after you gather your information --
MS. HUMPRHEYS: Well, let's look at it this way. Walter, how long did you think it was going to take you to ask your questions?
MS. GREENBERG: Yes, what's your timeframe for that?
MS. HUMPRHEYS: Are you going to start making phone calls tomorrow? We can send this out after you are finished.
DR. SUJANSKY: Well, no, it won't be tomorrow. The revised report is due on April 17. So, maybe the best time would be the beginning of next month.
MS. GREENBERG: Do we want to send it out after?
But the problem is after Walter gathers this additional information and then changes question marks to whatever, you would have to redraft the report.
DR. SUJANSKY: We are essentially asking for a third report.
MS. GREENBERG: Yes, which we don't want to do. But then you would have to be asking for another draft.
MS. HUMPRHEYS: I guess my feeling is that you have raised a good point. It seems to me that if Walter is supposed to come back with another draft on the 17th, then that's the draft that you should send out to the people. And in the meantime, any of us who see things in here that are not questions, but we question them, we can try to pursue verification of that from our perspective, and maybe verify this against whatever your spreadsheet said for these people.
Then the chances are whatever happens, whatever is available on April 17 will be a more accurate characterization of everybody. It's less likely to have comments anyway. And I think you are onto it. I think that's the one.
MR. BLAIR: That may cause much less of an impact.
DR. ZUBELDIA: Walter, by the time the questionnaire is sent back to the vendors, you make sure you change your e-mail address.
MS. HUMPRHEYS: No, what he's going to do is he's going to have a forward. And guess where it's going?
MS. BURKE-BEBEE: I'm just thinking through this process. If it goes out the mid part of April, Jeff, then that would put the beginning maybe only two weeks of turnaround time. So that we then have three weeks going into the May meeting? Does that work?
MR. BLAIR: Let's see, if Walter provides us with the report by the 17th, and we send it out on the 18th or 19th of April and ask folks to return that by mid-May, then --
MS. GREENBERG: Two weeks maximum.
MR. BLAIR: -- for any additional comments they may have on the report, any corrections they wish to make or supplementary information.
Walter, do you feel comfortable with that?
DR. SUJANSKY: I guess I would like to ask, before we go on with that, a question comes to mind, I would just like to ask the subcommittee. If the terminology does not distinguish between concepts and terms, what is our take on its concept orientation? How would you decide if it's concept-oriented?
MS. HUMPRHEYS: I would say that if the terminology has only one name for every concept, then it can't very easily distinguish between concepts and terms, at least at that level. So, it isn't the fact that there are multiple names for concepts that makes it concept-oriented. That, in and of itself, doesn't do it.
DR. SUJANSKY: So, what makes it concept-oriented? What questions would you ask if it were indeterminant? What information would you seek to decide if something is concept-oriented or not, to help me when I do my follow-up?
MS. HUMPRHEYS: Yes, I think it's very tough. The real issue of course is are there multiple codes that cover the same concept, but they have different terms. And you this is a matter of study to determine that. I don't think you can ask a question and get an answer to that.
You can of course -- whether something is concept-oriented is a lot clearer if they have multiple names pointing at the same identifier, because then you can look at all the names, or ask them, do you consider all of these things to be synonyms or not? Then it actually does often take an independent evaluation to find out whether they mean the same thing by synonym as you did when you asked the question. So, it's a non-trivial problem.
DR. SUJANSKY: Yes, so that's why one of the things I was hoping to do is to try to exclude some of the terminologies based on other features, ones that were out of scope, et cetera, et cetera before the time was spent to go back and do the non-trivial, as you put it, study of the question marks with respect to concept orientation.
So, I would like to ask the subcommittee's guidance on that. For example, HL7 version 2 -- well, I don't remember if that was a indeterminant or not.
DR. HUFF: It's not concept-oriented, I can tell you.
DR. SUJANSKY: I think that one was not in doubt. First of all, there is the whole class that didn't have concept permanence, but for some of those, concept orientation was in doubt. I would like to propose that I don't go back and get any more information about those.
MR. BLAIR: Does anybody object? I don't hear any objections.
MS. GREENBERG: I think what concerns me, and I will honestly acknowledge that I don't fully understand some of these distinctions, but if we are going to have an essential criterion which basically puts the terminology in or out of consideration, we should really be able to explain it very clearly. It's sort of problematic if we can't.
DR. FITZMAURICE: And to develop on what Marjorie is saying, suppose that there is one that doesn't seem to make it, but by jolly, the market has adopted it. And there is pressure on us to consider it strongly, because it is less expensive to adopt something that is already used widely in the market. Getting them some feedback on their concept orientation and other attributes could make that terminology better over the course of the next two years.
So, there are some pluses to investigating it, and feeding back that information. On other hand, some things can be done later, and not disturb our critical path toward getting a report out in September.
DR. HUFF: I think we should come up with clear criteria.
DR. SUJANSKY: I mean, it's listed. Let me read to you what we have. The problem is part of this definition requires an exhaustive in the extreme, examination of the terminology itself. The terminology are coded concepts with possibly multiple synonymous text representations and hierarchical or definitional relationships to other coded concepts. No redundant, ambiguous, or vague concepts.
Now, for some of these, you can make clear determinations. If you have a not elsewhere classified, that a vague concept. That is not concept oriented. In other cases, again, short of examining every single code, you cannot conclude with 100 percent confidence that there are no redundant concepts and no vague concepts. But clearly, surely that is outside the scope of what is possible to do as part of this evaluation.
MS. GREENBERG: Well, how were you able to answer yes for some of them without examining the code? Some people were able to provide you with information that --
DR. SUJANSKY: I answered that based on their statement that they represent synonymy explicitly, and distinguish concepts and terms. But there could be -- actually, some of them said that there are redundant concepts in there. But when they find them, they remedy that. So, really the intent is important there too. That that's considered an error, versus not.
But at the extreme, even for the ones that indicated they were concept-oriented -- that's why concept orientation really, I don't know that you can make that an absolute requirement. It's concept orientation. It's not that the terminology is free of errors. No terminology is free of errors. It's really that the intent is to represent concepts that have the same meaning, to explicitly represent that, to make that clear, to make those two different terms for the same concept, et cetera.
So, terminology that doesn't distinguish concepts and terms doesn't even have the ability to do that. So, that means if you have two concepts that mean the same thing, then that maybe is part of the terminology, in which case it's not concept-oriented. But to make that distinction -- that would be a difficult and time consuming undertaking.
MS. HUMPRHEYS: Yes, I was trying to look down here to see of those where you have question marks, I think that I can probably get the information to answer those things for ICD-10-PCS. So, I will see about doing that.
DR. SUJANSKY: But how will you get it? Will you look at every single term?
MS. HUMPRHEYS: I mean Simon, who is no longer here at the moment -- I was on an outside advisory committee for the development of this thing. I have seen copies of it, and I understand I believe, how it is meant to be applied. And the way it is meant to be applied, I believe gives it concept orientation.
DR. SUJANSKY: So, its intent is to be concept-oriented.
MS. HUMPRHEYS: Yes. I think concept orientation is an interesting issue for any terminology that doesn't have a large installed base of users, because while you are sitting there developing this terminology, and you haven't actually really released it for major use, you can have an intent of concept permanence, but you could have cleaned up errors and not done it, because no one is actually using it.
So, those that have a greater installed base, you can at least go back and see what they have actually done with that.
DR. SUJANSKY: Right. Well, let me ask this. There are four messaging terminologies in here, ISO, the device communication one, DICOM, HL7 V.2, which Dan has confirmed is not concept-oriented, and the NCPDP. Do we need to pursue those -- besides HL7 -- do we need to pursue those further? Is there any reason or chance that we would include the code sets from those messaging standards among the core terminology group?
DR. ZUBELDIA: They are not portable from one messaging system to another. So, I don't think we need to pursue those any more. In addition to that list, I had listed X12, UBX MAL, and UN EDIFACT. But then I realized that there is no sense in pursuing those. They have code sets that are unique to their messaging standard.
DR. SUJANSKY: They don't live independent of a message.
MS. HUMPRHEYS: But the question that many of us ask, and I think I have heard Stan and maybe Clem on this is that if there are in fact sets of values, tables of values like we, in developing RxNorm were basing the dosage forms that we were using in RxNorm on dosage forms that were part of some list that was a value set within HL7. And maybe the work that we are doing with RxNorm will feedback in, so that maybe the dosage forms need to be expanded, or whatever.
But we were using that value set, because you couldn't find anywhere else, in any of the standard vocabularies that we were dealing with, a set of dosage forms that seemed to work for this purpose. And we didn't want to invent a new one. So, we took them from HL7.
I think that the problem is embedded in these smaller code sets that are part of all of our messaging standards, including the HIPAA standards, are things that you actually need to have a controlled form of. They are a short list of controlled vocabulary that you actually need to run a health information system.
DR. SUJANSKY: Right. But should the complete code set of DICOM -- is there any chance that that would be included in core terminology? You may need those concepts.
DR. MC DONALD: What is that? What is the complete code set?
DR. SUJANSKY: I don't know. It's radiology, devices and findings relevant to transmitting radiology information, including settings of machines and so forth. But I haven't seen the whole terminology. They have about 400 radiology-related things, and then a handful that cross over into other categories for their responses.
MS. HUMPRHEYS: I'm envisioning a piece of work or a task activity that has to be done at some point, where we go into all of these messaging things that have been adopted, and we find out what all of these codes sets are.
And we decide which of them have to go into the UMLS, because we've got to have a way of sharing them, and we've got to have a way of mapping them to everything else, because everybody's got to use them if they are going to send a message from A to B. And in many cases, although not all, they have a bunch of values in there that aren't anywhere else.
Without studying it, my guess, Walter, would be we are going to find out that some of these things don't actually need to be there. It would be far better if the message at that point in time referred off to one of these larger code sets. And then we are going to find the reverse is true too, that we are going to need them. I don't know if that helps you at all. I suspect some are going to be needed.
DR. SUJANSKY: Well, but is that terminology in its whole going to be included? Or are elements of that going to be more likely placed into one of the other terminologies as needed that are part of this core? And I think it's much more likely that the latter will happen.
MS. HUMPRHEYS: Not necessarily. I think that we may just have another terminology, which is the HL7 X terminology that has all the stuff that isn't in the other, that we still in an HL7 message. It might not be as big as it is now.
DR. SUJANSKY: But clearly, there are a lot of things in HL7 you -- that aren't really --
MS. HUMPRHEYS: The stuff that is suitable.
DR. ZUBELDIA: Betsy, those code sets are going to have to be mapped at some point, because even if you select the dosage forms or HL7, when you send the pharmacy message under HIPAA, it has to be NCPDP, which has a different list of dosage forms. And when you send it as an 835 for the payment, you are going to have again, a different list. At some point there has to be a mapping of those, and that's your role, right?
MS. HUMPRHEYS: You but somebody is going to have to sift through and say what of this makes sense to leave the way it is, and what of it is that we just have to fix this and go on.
I guess from your perspective, is this where we are going to start with this process? Are we going to come up with the recommendations that are included in all these? That's where you are really coming from. Is this in some sense, a second order issue after we have established what these others are, so that we then can find out how many of these things are actually in them or not?
DR. SUJANSKY: Yes, the short-term question is how much time should I spend investigating whether DICOM's codes are concept-oriented or not? That's the immediate question.
MS. HUMPRHEYS: Yes.
DR. SUJANSKY: So, I'm trying to get some guidance on that. That's why I'm asking these questions. So, one thing I have suggested is that I'm not going to pursue these ones that do not meet the essential technical criteria, not because of concept orientation, but because of concept permanence, and perhaps other things, that I don't pursue those any more at all. And we'll just remain forever ignorant whether they are concept-oriented or not, because it really doesn't matter. We are not going to select them anyway.
DR. COHN: We are not going to select them as core. You see, I could imagine that we could -- I don't want to get into wordsmithing here, but they not be core, but they may be essential. And then the question gets to be, the recommendation comes more like, well, if we have got different ways of describing the NCPDP, which is different than X12, which is different than HL7, are they all right, and is it a mapping issue, or is it really more fundamental that people made arbitrary choices when they did all of this? And maybe that needs to be somehow put together.
And it reminds actually of work that Dr. McDonald did in something called HISTP. He actually got himself quite in some stew around all of those discussions.
DR. MC DONALD: We almost had that done too, but that's another story.
I think there are two kinds of things. There are mapping issues, but the problem is almost always there are slight model differences. It's not arbitrary. There is a different set of use cases. X12, for example, has one great, big code system which the subfields refer to. That's not a bad model difference, actually. Other systems will have designated code tables.
But usually, if you just look at the race and ethnicity codes, that you see the challenge. It depends on what the deep model was, what they were really trying to achieve. Guam is not exactly the biggest population center in the US, but it's got its own code in the US, Guamian, because it's a political sort of an entity. So, anyway, things end up different for reasons other than arbitrary, but they just come out the same, I guess.
DR. COHN: Well, Clem, do you have a recommendation? I'm sort of trying to think of what direction.
DR. MC DONALD: I would suggest that he's got enough work. If he's at the edge of something, don't get into it. But on the other side, I think that the codes within the message standards do need some work -- maybe it shouldn't be on his plate -- to see what we really have, and what are the issues.
There will be codes in some of these systems that are just beautiful, separate, pristine, perfect codes for some uses. They just don't overlap with anything.
DR. COHN: And I think what we are trying to do is to avoid having you have do work. But let's see if we can come to a conclusion on this so that it can find its way into a report or recommendation.
DR. MC DONALD: But DICOM, it's all built into the standards, very built in. You don't have this nice separate table that X12 has. You can just find all the terms.
DR. SUJANSKY: Okay, so I won't look at the ones that we have already excluded because of concept permanence, but I'll look at the ones that are in that intermediate category, and determine their concept orientation.
DR. STEINDEL: I thought we were discussing a separate issue, and that was with regard to the messaging.
DR. SUJANSKY: We were discussing that earlier, yes. Do you have a comment on that?
DR. STEINDEL: I think I missed that one. I don't know what the conclusion was, but we are looking for things not to spend time on. I think that's an area that we really don't need to spend any amount of time on, and more than a few minutes. Because like for instance, I suspect the ISO 11073 codes are the HL7 codes. And the DICOM codes are embedded in SNOMED.
DR. SUJANSKY: Okay, so you propose not pursuing those?
DR. STEINDEL: Yes, I just think the messaging ones, like we said before, are used and developed for a different purpose.
DR. MC DONALD: Steve, just to clarify, I said the micro glossary. But is that true for all those other lists of various kinds of doses and radiation and slices? I don't think it is.
DR. STEINDEL: I think they put the slices in too, but I'm not 100 percent certain. But they were working very closely together for a long time. But I think we will find that a lot with the messaging code sets.
DR. SUJANSKY: I'll defer any further investigation on that.
MS. HUMPRHEYS: NLM has done a lot of work with ICPC, so I think that we probably can get back to you in the report with whether they are concept-oriented or not.
MS. GREENBERG: I guess I'm still uncomfortable having an essential criterion that we aren't quite sure how to define or determine. And so, it's an eliminating factor. And just leaving it unknown without some effort to clarify it just because it's another essential criterion that also isn't met.
DR. SUJANSKY: But that's the point. Because there is another one that we feel confident about, the concept orientation in essence doesn't matter as far as excluding it.
MR. BLAIR: Marjorie is making a different point.
DR. SUJANSKY: Oh, I'm sorry. Did I misunderstand you?
MS. GREENBERG: I understand the practical reason. It doesn't meet one of other ones anyway, so we know it's not going to qualify. That doesn't eliminate the fact that you have an essential criterion that we have not adequately defined.
DR. SUJANSKY: I think earlier I returned to the definition that we had here. I think we are comfortable with that definition. The issue is that to make the determination with 100 percent confidence, you need to exclude any redundancy, and that's the thing that is very difficult to do exhaustively.
MS. HUMPRHEYS: The real issue is whether there is planned redundancy, which is the thing that would be a definite -- you would definitely say that any system that essentially had planned redundancy at the concept level, was not concept-oriented.
And any system that basically took a bunch of terms that didn't mean anything, and linked them all to the same code with no differentiation between the ones that meant the same thing or not, I mean those are two clear indications that this thing is not concept-oriented. An error is another matter. Errors exist.
So I think that very often it is the examination of how they treat alternate names, and what those really are that really could give you the tip off.
DR. SUJANSKY: I just want to clarify, is there anyone on the subcommittee who is uncomfortable with going ahead and excluding the ones we have excluded because of concept permanence, and leaving it at that? Okay, so then Steve suggested deferring further investigation of the messaging ones. Betsy is going to help me with ICD-10-PCS and ICPC.
MS. HUMPRHEYS: And I'm also going to check on MedDRA.
DR. SUJANSKY: Great. Actually, I have the data right here. Maybe we can sift through that afterwards, and have a brief --
MS. HUMPRHEYS: I think from the point of view of once you verify which of the nursing terminologies are in fact incorporated into SNOMED CT, then I think that probably where they are down here in this first table, excluded or whatever, some of them, then they should be annotated, because I think many of these say that irrespective of their characteristics as a stand alone, once they have completed integration into SNOMED CT, theoretically they would be covered there.
And then of course, they would be put in there with the basic structural characteristics of SNOMED. So, that might make that look a little less alarming, that we had eliminated nursing.
MR. BLAIR: Are there other issues and concerns? Otherwise, I could summarize the things we have agreed to. Or, Walter, do you want to do that?
DR. SUJANSKY: Sure. I think we have agreed that I would pursue a number of items of missing information or incomplete information in the data that we have, that are important to furthering the work of the subcommittee and making recommendations. This includes some of the licensing cost information, uncertainty about concept orientation of a number of terminologies, as we have discovered. Determining which nursing terminologies are in SNOMED CT, and a few other points.
And subsequent to that, I will produce a revised version of this by April 17, and that version will be distributed to the terminology developers who submitted questionnaires for their comment. And those comments will solicited in writing, and returned to someone on the staff, I guess, Suzie perhaps, who is graciously volunteering.
And then some initial assessment of those comments will be made, and that information will be forwarded to me. And there will be a subsequent version of the draft report, third version of this report that will then prepared at some point after that, to be determined by just how many comments we get, I suppose.
MR. BLAIR: Any other things?
DR. SUJANSKY: Tell me if I forgot to summarize anything.
DR. COHN: Suzie, do you have a comment about that?
MS. BURKE-BEBEE: In addition to that, I was wondering about whether the three surveys fell as far as not having been incorporated. That might be something that needs to be discussed, ICF, and there were two others that weren't incorporated into the overall picture.
MR. BLAIR: The Cerner Multum and the UMLS.
MS. BURKE-BEBEE: That's it.
MR. BLAIR: And thank you for the reminder. We are kind of at the edge of the day, and Walter, you were going to try to get a feeling for the other tasks that you would have to do whereby in order to pull that into the document, whether that would run you over what you expected to do by April 17.
DR. SUJANSKY: Right, well, I think what we decided was that other activity that might run me over April 17, that we were going to defer until after April 17 anyway, namely the incorporation and consideration of the comments from terminology developers. Or are you referring to these three? Oh, I see, never mind.
Now I understand what you are saying, that based on the other things that I need to do in the scope of work, I think the best way to proceed is to look at those three terminologies individually.
MR. BLAIR: Does the UMLS really need to be put into the matrix?
MS. HUMPRHEYS: No, I don't really think it does.
MR. BLAIR: So, it's only the two.
MS. HUMPRHEYS: I think that Walter and Jeff, you have that, so you could send it around to the committee if they are interested in looking at it. And I think it has more to do with looking at the UMLS structurally and some of the characteristics of it.
It has more to do with what or doesn't happen -- I hope does -- for the recommendation about the distribution of these things in the UMLS. And the fact that the UMLS structure is appropriate for this. And that will give back that information. Rather than including it on a table where it's a bowl of fruit with a bunch of apples and oranges.
DR. COHN: Actually, let me make s comment here, because Betsy, at the 80 percent level, I agree with you. But I think earlier today you had discussed two terminologies that are really unique to the UMLS, Metathesaurus and --
MS. HUMPRHEYS: They are certainly not unique to the Metathesaurus. MESH is distributed independent of the UMLS any time by itself, or you can sign up and also get the Metathesaurus.
DR. COHN: Obviously, I'm using the wrong term. What I was describing was otherwise UMLS is really an interlingua of other terminologies.
MS. HUMPRHEYS: It is. Absolutely it is. Stuart already submitted a separate one for RxNorm. And RxNorm will appear in here, and that's very sensible.
DR. COHN: And what about MESH?
MS. HUMPRHEYS: The issue about MESH is I think that we want to look at MESH when we have identified the clinical, or what you see as this core set, and we are looking at doing a gap analysis. I think it's conceivable that in some of these areas, MESH might fill the gap. And it's also conceivable -- and we can look at this in the drug area -- it is also conceivable that MESH, you might actually want to describe MESH separately for that. But I don't think we need to put that in the middle of Walter's critical tasks.
DR. SUJANSKY: And then ICF one, I'll take a look at that one. And without going through that whole process that I went through here for all the other ones, I will see if there are some things that would exclude it, like we saw with some of the other ones. And if there is, then I will let you all know that, and stop there. If there is nothing to exclude it, then I will go ahead and score it like I did everything else for ICF.
The Cerner Multum one, I'm not going to -- that came in very late, and I don't think it is likely to be -- there are a lot of other vendors who have terminologies that didn't submit anything. So, I don't think that we need to have all the vendors' input. And given that it came in so late, I'm just treating it as if it hadn't been really submitted frankly, because it just came in too late for consideration.
So, I won't treat that one at all. So, the questionnaire will be available, the responses will be available among the others. So, that's what I propose. That's how I propose to proceed there.
MR. BLAIR: Is there anyone that feels uncomfortable with that?
DR. FITZMAURICE: Maybe just a little, Jeff, because I'm thinking in terms of the people we invite, the terminology users to testify. And some of them probably will be vendors. And if we look at some of the larger vendors, I suppose some consideration ought to be given to SMS, McKesson, Cerner per se.
And suppose you had somebody come who said, well, we sent somebody up, and we didn't get any feedback. Does that argue for not inviting Cerner to testify?
DR. MC DONALD: I have a comment. Actually, Cerner, there has been three major drug vocabularies, commercial. There is Medastat(?) and I don't know where it is. It got split back off. It got bought by First Data, and then it got split back off. So, I think it's still alive.
There is Multum, which is by Cerner. And the one that did submit. It would probably be useful to get all three of them if you are going to discuss drug vocabularies and solicit it. I think it's a matter of big commercial vendors though, because only Cerner has bought one of those big vendors.
Alternatively, you have done a valid process. You gave them a chance, and they got it in so darn late, you couldn't do it.
What do you think? Is there a fourth one, a drug vocabulary?
DR. SUJANSKY: There is Gold Multimedia, which is another vendor.
MR. BLAIR: The only thing that makes me a little nervous, Multum I think appealed to us very strongly some time back for consideration, and indicated that they were willing to make their --
DR. MC DONALD: That was while they were an independent company. I don't know if that rumbling is still there. Stan, can you speak to it?
DR. COHN: Without trying to prejudge this one, I guess I feel sort of funny about this, just on a process issue. It has nothing to do with the quality of the Multum terminology one way or another, but it seems like we are talking about ICF. We are talking about -- we have somehow dealt with the UMLS.
This one, unlike everything else that we have received, we are choosing to ignore. And I just feel that we need to, just because of process issues, be consistent. I'm not saying you go from here and start getting other terminologies to evaluate.
DR. SUJANSKY: I think we should do ICF then. I agree with you.
DR. ZUBELDIA: You are saying not ignore ICF.
DR. MC DONALD: Did it come late?
DR. SUJANSKY: It came very late, yes.
I shouldn't make these decisions. We need to decide if the deadline meant anything at all.
DR. MC DONALD: If life was good, we've got a resource constraint. And I don't know how to factor that in, but it would be grand to have whatever the three major ones are shown, just as a research statement.
DR. SUJANSKY: Well, you know what, here's one thing we can do. I don't have the time to do this. You're right, we have a resource constraint. And given the other follow-up information and the other new aspects of the project that we may be adding.
But I think someone on the staff could put their responses into the spreadsheet form, and work to apply the scoring criteria to those responses. The questions that are related to each scoring metric are clearly identified. The responses that correspond to each numerical score are clearly identified in the document. So, I think all that could be applied to those terminologies by someone else.
DR. FITZMAURICE: And you would review it and adopt it as your own if the process had been followed?
DR. SUJANSKY: Yes, I would review it, yes.
DR. FITZMAURICE: Well, that leaves one question, doesn't it?
DR. MC DONALD: Just to get the beautiful review of these things that we don't have in the field, which this is the beginning of, this sounds even worse. You are just missing Medastat, right?
DR. COHN: And the Cerner Multum.
DR. MC DONALD: We got a response.
DR. SUJANSKY: They never replied.
DR. STEINDEL: We can't do anything about that.
DR. COHN: Well, I guess I would defer to both Walter and Jeff to sort of figure out a process of how to handle these three. As long as it's a consistent process.
DR. SUJANSKY: You know I hate to say what I just said, but the reality is I wouldn't have the time to do it, given other commitments that I have already made. But I do think it would be good to have done. I agree with you all there.
MR. BLAIR: Okay, at this point, can Marjorie and Michael, can I leave it with the fact that there would be somebody on the staff that could go ahead and process the information for Cerner's Multum?
MS. GREENBERG: I don't know that there is somebody who can do that. You are talking about getting the matrix I guess, and then taking the questionnaire and coding the questionnaire to the matrix? Is that what you are talking about?
DR. SUJANSKY: Yes, exactly. You have right the report, right? There are criteria that are specified numerically. For example, meaningless identifiers get a 0 score, a one corresponds to this, a two corresponds to this. And also for meaningless identifier there is a set of the specific questions in the questionnaire that relate to that.
MS. GREENBERG: But this is primarily a data entry task?
DR. SUJANSKY: Well, no, more than that.
DR. STEINDEL: It has to be done by somebody who is knowledgeable.
MS. GREENBERG: That's what I'm trying to clarify here, whether it's taking things from the questionnaire and putting them into the matrix based on clear criteria, or whether we need a little Walter. And I don't think we have one.
DR. FITZMAURICE: It's identifying the person.
MS. HUMPRHEYS: You can put the things in the matrix for scoring.
DR. SUJANSKY: That requires some judgment, yes.
MS. HUMPRHEYS: The scoring is what requires the judgment. Putting things in the right spot --
DR. SUJANSKY: Right, that's straightforward enough.
MS. GREENBERG: We have a few questions here. One is time. There are two types of resources. There is money and there is time. And time is worth money, of course. So, if Walter is saying he doesn't care what we pay him, he does not have the time to do it, that is, we have to find another way to do it or not have them done, it sounds like.
If it's a question of renegotiating contracts because there is additional tasks, which actually may happen anyway because of the decision to send these out to all the developers and give them an opportunity to comment, which was not part of the original scope.
MS. HUMPRHEYS: If somebody will take the stuff and put it in the spreadsheet with his things across the top, and send it over to NLM, we will have somebody who could code it.
MR. BLAIR: Great, thank you.
MS. HUMPRHEYS: And I suspect that I will have to look into what is the currency of our version of Multum, which we also include in the UMLS, to see whether we actually know if not everything, some of the key technical criteria, because we would have had to fix them if they weren't there.
DR. SUJANSKY: Multum or Medastat? Because Multum we got responses to.
MS. HUMPRHEYS: Medastat I think we have as well. SO, we will just have to look to see what is the currency of our data, because if we got it from them several years ago, I wouldn't want to comment on their current technical organization based on the version that we have. If we have one that is relatively recent, then we would at least be able to say something about the identifiers and concept orientation.
MS. GREENBERG: So, we'll have someone do the data entry for Multum and ICF, not UMLS. Then we'll send that to NLM, and they will score it.
MR. BLAIR: Great. The other items are that Stan has volunteered to help us wrestle down a set of issues that were considered to be too time consuming for us to deal with in today's session. And Stan, do you have a feeling for when you might be able to send that to us?
DR. HUFF: I'll try and send it by a week from Monday.
DR. FITZMAURICE: Do we want to discuss what the paper will contain, so it doesn't expand beyond Stan's capability?
MR. BLAIR: Stan, why don't you give your description of what you feel you can do.
DR. HUFF: I was just going to expand the list that we started to include the kinds of data exchange, and the information sharing that I thought were within the scope of this thing. And it's mostly just going to be an expanded list. It wasn't going to be a huge treatise, and also include anything else that I could think of that would be excluded from the scope of this project.
MR. BLAIR: It would serve as a tool to help us identify the terminology domains that we would consider to be in scope or not in scope, is that correct?
DR. HUFF: That's correct.
MR. BLAIR: And the other item we have to discuss is the area of May 21. And I think that we sort of understand how we will be proceeding. If anybody has names of folks that you feel should testify, that would be very helpful. If you could send that to Michael or Suzie or myself, that will help us.
I don't think we have anything else for me to say thank you.
DR. COHN: There are some hands around the room. Steve?
DR. STEINDEL: Just a quick comment. The question was asked what nursing terminologies are in SNOMED. The January 2000 release of SNOMED added nursing intervention concepts from NIC, the OMAHA system and the Georgetown Home Health Care Classification. NANDA is in there now. They are working on in the future adding the perioperative nursing data set, and NOC.
DR. SUJANSKY: Before we adjourn or get ready to, Jeff, we talked about discussing how and if we should distribute the questionnaire responses.
MR. BLAIR: Yes.
DR. SUJANSKY: Did you want to address that today?
MR. BLAIR: Yes, Marjorie and I discussed that. She has advised me that with the Freedom of Information, when documents are submitted to us, they really are and should be public domain. I think, Walter, you and I are aware of the fact that there were at least, if not two or three folks that with respect to the optional questions for getting to the issue of market acceptance, went to the point very nicely of indicating the names of the users, which they indicated that they did consider proprietary.
And at least my thinking is that that should be respected, if they consider that proprietary. Maybe after the meeting at some point, if you have a feeling for how many folks specified something, a response that they felt was proprietary, maybe you and I could figure out how we could respect that.
DR. COHN: I was going to say, I think Steve had a comment on this one.
DR. STEINDEL: The Freedom of Information Act allows us to exclude data that is business confidential.
MS. GREENBERG: Right, but I think what the process would be is whatever we could distribute could exclude that proprietary information. If somebody still wanted to make a request for it, then they would have to make a Freedom of Information request, and that could meet one of the exceptions because it was proprietary information.
MS. HUMPRHEYS: But the thing is that if you are distributing the spreadsheet, rather than the original things, then it will just have to be edited before it gets distributed.
DR. SUJANSKY: Well, that's another decision, do we want to distribute the spreadsheet, or just the original questionnaires widely. I was thinking internally here, we will distribute the spreadsheet.
MR. BLAIR: Yes, at this stage I think the spreadsheet should only go to committee members and the immediate staff, and not be distributed to anyone outside.
DR. SUJANSKY: There are some editorial comments in the spreadsheet.
MR. BLAIR: Now, Marjorie, it is my understanding that you didn't feel it was necessary for us to send an e-mail to all of the terminology developers, inquiring again whether they have anything proprietary, unless they marked it and identified something as proprietary, confidential. We would just go with what they have marked. Was that correct?
MS. GREENBERG: I think so. I think it's known that the NCVHS is a federal advisory committee, and the materials you work with are in the public domain.
MR. BLAIR: Okay.
DR. COHN: Let me just sort of finish up here for just a second. One thing past May that we may need to think about is that I think we are going to need to brief the full committee at the June meeting about sort of our draft recommendations at that point, as well as directions. Now, I guess one could hope optimistically that we will have such insightful input in May, that we will all come to a consensus, and have recommendations done by June.
But I think that at least according to the work plan, it's likely that that will be a September deliverable. But I do think we all need to think about what it is that we need to share and brief the full committee about.
DR. MC DONALD: One of the tactic we used in the first round, which might still be useful in this round is to clearly announce that this is step one. Pick whatever we pick, and not indicate that anything that is not picked, won't be picked honestly. So that it doesn't become a desperate race to get in under the deadline forever and always.
It may simplify the process to say we are just going to do the ones that are whatever X is, and then have another round as we sort out the details of what is necessary for different kinds of messages, et cetera, different kinds of communications.
DR. COHN: Good point. Just another phase of our recommendations. Okay.
DR. MC DONALD: There is a lot more parts in this. We ended up with looking at only -- there is actually more codes than there are message standards altogether, right?
DR. COHN: Yes.
DR. MC DONALD: So, it's a bigger space, and likely to grow bigger yet.
MS. GREENBERG: If we are about to end, find, but otherwise I think we should take a break.
DR. COHN: Yes, I think we are about ready to end, actually, if I understand it. I was actually just going to comment that tomorrow morning starts at 9:00 am, upstairs in 705-A, as opposed to this room. And we will be adjourning by 12:30 pm tomorrow.
Are there any final comments? Obviously, Marjorie doesn't want these comments to last long. Okay.
Thank you, everybody. It's been a very productive day, and I think we are moving forward.
The meeting is adjourned.
[Whereupon, the meeting was recessed at 3:45 pm, to reconvene the following day, Wednesday, March 26, 2003, at 9:00 am.]