Reliability Studies
NCI SEER conducts a number of web-based reliability studies as a vital part of the quality improvement process. The studies are designed to test the skills of central and hospital registry personnel and to measure consistency in the application of codes and coding rules across a program. In a reliability study, all participants code information from the same medical records using the same references under similar conditions.
The quality improvement process of SEER reliability studies features a reconciliation component which focuses on uncovering the reasons for discrepant answers. Results of reliability studies are used to develop educational materials and to improve data collection manuals.
- Real cases
The cases used in a reliability study are actual de-identified cases submitted to SEER by central and hospital registrars. The cases are not manipulated or altered. - Reconciliation
At the completion of a study, results are reconciled. The reconciliation meetings are held to review discrepant answers to identify the need for objective-based revisions and clarifications, and identify educational needs.
Each SEER registry receives a report of their registry's results overall and their results by individual participant. A facilitator at each SEER registry is charged with organizing the registry into a group discussion of all those data elements for which agreement with the preferred answer was less than a certain percentage; for example, less than 85%.
If the group's final answer is different than the provisional/preferred answer, the facilitator submits a form to NCI SEER stating their group's final answer, with justification and citation of supporting sources.
All facilitators come together for one final meeting with NCI SEER to discuss the discrepant answers. Recommendations for changes to the manuals and other documentation, along with educational needs are derived from the reconciliation discussions. Final answers are determined based on the reconciliation.
