3D Visualization in Medicine

(A Course Proposal for SIGGRAPH 98)

Summary:

This course presents current techniques in 3D medical visualization. We describe the problem as a pipeline from acquisition to display, examining new data acquisition technologies, outlining algorithms and optimization strategies, and pointing out inherent problems along the way. Case studies will be presented periodically throughout the course, illuminating the motivations, benefits, and potential pitfalls of computer graphics research in medicine.

Course Organizer	Speakers
Terry S. Yoo	Henry Fuchs Ron Kikinis, M.D. Bill Lorensen Michael W. Vannier, M.D.

Index

3D Visualization In Medicine

Summary Statement:

This course presents current techniques for 3D medical visualization. We describe the problem as a pipeline from acquisition to display, examining new data acquisition technologies, outlining algorithms and optimization strategies, and pointing out inherent problems along the way. Case studies will be presented periodically throughout the course, illuminating the motivations, benefits, and potential pitfalls of computer graphics research in medicine.

Expanded Statement:

The growing healthcare industry is providing new opportunities for applied research in computer graphics. Although volume rendering and mesh generation techniques will be briefly covered, the course will concentrate on areas of the visualization pipeline not traditionally covered: acquisition and medical evaluation. This course will present not only how to approach 3D visualization in medicine, but also, through case studies, will discuss the motivations and limitations of such methods. Participants interested in getting started in this area will learn about the sources of volume medical data, and clinicians will present their view of visualization and what their requirements are for effective and safe applications of computer graphics in their field.

Course Vital Statistics

Duration: Full Day

Format: Lectures, Case Studies, and a final half-hour panel session on future research directions.

Special Notes Requirements:

Special Presentation Requirements:

Course History

The proposed course revisits and updates material presented at SIGGRAPH '93 and SIGGRAPH '94. This material has not been covered in four years, and while the basic techniques have not changed substantially, there have been important advances in data acquisition and the growth of interactive graphics in the clinic. This proposal emphasizes the advances in interactive volume ultrasound, interventional MRI, and image guided therapy. Case studies will be presented showing where and why computer visualization has been effective with the hope of motivating members of the audience to participate in this growing field. Slides, video, and perhaps interactive demonstrations of visualization systems will be used to present the material.

Prerequisites

Basic knowledge of 3D computer graphics and an understanding of the basic principles of image processing. Some familiarity with medical terminology or experience working on a clinical project would be useful, but not necessary.

Topics Beyond the Prerequisites

Participants will gain insights into what makes an effective medical visualization and the processes by which they are created. Other topics include the sources of 3D clinical data and their characteristics, the future of advanced displays (virtual worlds) in medical settings, and the uses of interactive computer graphics for surgery.

Course Description:

We are designing the course around the presentation of a simplified medical visualization pipeline. Like a graphics pipeline, there are steps throughout the procedure that are familiar; however, the beginning is image acquisition rather than geometry/modeling. The segmentation/classification stages are new to graphics people, but not to people familiar with image processing techniques. The later half of the pipeline will cover rendering techniques commonly used in medical visualization, though perhaps in less detail than normally covered in either the introductory course or the advanced course in volume visualization. A section on display systems will follow, including the utilization of head-mounted display technology in medical applications.

The remainder of the course will be dedicated to familiarizing attendees with current directions of computer graphics in medicine, including current and developing applications for this technology, and existing problems regarding accuracy, robustness, and interaction with the medical community.

At all times during the course, through case studies we will emphasize the role that visualization plays in diagnosis and treatment. The presentation will be driven toward application. Analysis of error or noise will be discussed relative to its impact upon the medical task to be performed. Examples of rendering techniques will typically be presented using medical data, to continue to familiarize the computer graphics portion of the audience with the medical aspect, and to keep the medical attendees involved in the process.

Target Audience

It is our intention to address those members of the graphics community who are interested in expanding their research directions toward medicine. It is essential that computer scientists who apply their knowledge to medicine understand the requirements that are particular to clinical applications, specifically: robustness, limiting artifactual error, and maximizing comprehension and retention (targeting clinicians).

We hope this course will be an enlightening presentation of this material for all concerned. Speakers from both the medical as well as the computer graphics community will be present to provide their particular view of the field. We have selected speakers familiar with each stage of the pipeline, giving particular insight into each of the areas where error or improvement can be introduced. The whole collection of proposed speakers represents one of the finest cross-sections of the use of computer graphics in medicine.

Medical Image Acquisition:

Although people working in the field of volume visualization regularly manage large volumetric datasets, they often do not have a full command of the imaging modality that creates the data. To provide for optimal visualization of information within medical volume data, it is essential that the process be suited to the modality used to create the data itself.

There are three imaging modalities that we will be considering in this course: X-ray computed tomography, nuclear medicine imaging (including both PET (positron emission tomography) and SPECT (single photon emission computed tomography), and magnetic resonance imaging. We intend to briefly cover the mechanisms underlying each of these modalities, their strengths and their weakness. A treatment of the noise properties, linear/geometric distortions, and contrast sensitivity of each of these modalities will be discussed, not only in light of their uses in medicine, but also regarding their impact on the visualization process.

Segmentation and Classification

An often serious weakness in volume visualization systems is the lack of a robust classifier. Segmentation is the division of an image into coherent regions. Classification is the labeling of those regions, often with the aid of a user. Classifiers can be based on many different mechanisms; however, the simplest are intensity windows, useful primarily with X-ray CT data.

More advanced mechanisms based upon the geometry of images or of statistical measurements made of images are available, though often underutilized in our community. There is a large body of literature covering statistical and structural pattern recognition techniques that is often overlooked by volume visualization specialists. We will present some approaches to segmentation and classification that are directed toward medicine. These methods are either automated or semi-automatic, requiring interaction with a medical expert (but not necessarily a computer expert).

Rendering

An entire hour of the course will be dedicated to rendering. This may be redundant to people who are already familiar with display techniques for volume data; however, it is considered an essential element of the course. Moreover, even though much of this material may overlap with course materials presented in courses on volume visualization, we will retain our focus on specifically medical applications. For example, curvilinear grids do not often exist in medical data; on the contrary, regularly sampled rectilinear grids with an abundance of information usually flood visualization systems. The issue is not in how to display the data, but how best to extract pertinent information and present it for maximum effect.

Volume rendering topics to be covered will include raycasting volume renderers, splatting (hierarchical splatting, and attempts at interactive splatting), and Fourier rendering. The surface rendering discussion will cover both the marching cubes and the dividing cubes algorithms, as well as the recent work in automated mesh refinement.

Display

Emerging technology in immersive display (virtual reality) is showing remarkable promise in medical applications. The challenges in overcoming the technological limitations are difficult, but the opportunities seem endless. We will present work in viewing ultrasound data through head-mounted displays as well as postulate some new directions for this technology in medicine.

Beyond the Pipeline: a computer view of medicine vs. a medical view of computers

Examples: Existing Systems, Clinical Applications, and Futures

Future direction for the field will be covered at the end with questions from the audience about any aspect of the course. The final session is expected to be a panel, with the audience participating, directing questions at any of the speakers and addressing the whole range of issues from acquisition to display.

Final Comments

It is our contention that the focus of this course is no more narrow than other sub-fields also presented as courses at SIGGRAPH. The growth of biomedical imaging conferences and symposia such as Virtual Reality in Medicine as well as a significant number of recent SIGGRAPH papers related to this topic seem to indicate growing interest in this area.

Speaker Information

Speaker Biosketches

Terry S. Yoo is an Assistant Professor of Radiology at the University of Mississippi Medical Center and Research Assistant Professor of Engineering at the University of Mississippi. His research interests include visualization of 3D medical data, interactive computer graphics and medical image processing. He is currently developing a research program in Interventional Magnetic Resonance Imaging. He worked for BBN Laboratories, Incorporated, AT&T Technologies, and MCNC before beginning his doctoral studies. During an interim in his doctoral research, he served as site coordinator for the NSF/ARPA Science and Technology Center for Computer Graphics and Scientific Visualization. In 1993, he was co-chair of a SIGGRAPH Course on Three Dimensional Visualization Using Medical Data.

Yoo received his A.B. in Biology from Harvard in 1985 and his M.S. and Ph.D. in Computer Science from the University of North Carolina at Chapel Hill in 1990 and in 1996, respectively.

Henry Fuchs is Federico Gil Professor of Computer Science and Adjunct Professor of Radiation Oncology at the University of North Carolina at Chapel Hill. He received a BA in Information and Computer Science from the University of California at Santa Cruz in 1970 and a Ph.D. in computer science from the University of Utah in 1975. He has been an associate editor of ACM Transactions on Graphics (1983-1988) and the guest editor of its first issue (Jan. 1982). He was the chairman of the first of the Symposia on Interactive 3D Graphics (1986), co-director of the NATO Advanced Research Workshop on 3D Imaging in Medicine (1990), and co-chair of the National Science Foundation Workshop on the Future of Virtual Environments Research (1992). He received the 1992 Computer Graphics Achievement Award from ACM/SIGGRAPH and the National Computer Graphics Association Academic Award (1992). His current research interests include the application of head-mounted display technologies to problems in medicine.

Ron Kikinis is the Director of the Surgical Planning Laboratory of the Department of Radiology, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, and an Associate Professor of Radiology at Harvard Medical School, as well as an Adjoint Professor of Biomedical Engineering at Boston University. His interests include the development of clinical applications for image processing, computer vision and interactive rendering methods. He is currently concentrating on introducing interactive computer graphics into the operating room. He is the author and co-author of more than 52 peer-reviewed articles. Before joining Brigham and Women's Hospital in 1988, he worked as a researcher at the ETH in Zurich and as a resident at the University Hospital in Zurich, Switzerland. He received his M.D. from the University of Zurich, Switzerland in 1982.

Bill Lorensen is a Graphics Engineer in the Electronic Systems Laboratory at GE's Corporate Research and Development Center in Schenectady, NY. He has over 25 years of experience in computer graphics and software engineering. Bill is currently working on algorithms for 3D medical graphics and scientific visualization. He is a co-developer of the marching cubes and dividing cubes surface extraction algorithms, two popular isosurface extraction algorithms. Bill is one of the chief architects of LYMB, an object-oriented software development environment written in C. His other interests include computer animation, color graphics systems for data presentation, and object-oriented software tools. Bill is the author or co-author of over 60 technical articles on topics ranging from finite element pre/postprocessing, 3D medical imaging, computer animation and object-oriented design. He is a co-author of "Object-Oriented Modeling and Design" published by Prentice Hall, 1991. He is also co-author with Will Schroeder and Ken Martin of the book "The Visualization Toolkit: An Object-Oriented Approach to 3D Graphics" published by Prentice Hall in February 1996. He gives frequent tutorials at the annual SIGGRAPH and IEEE Visualization conferences.

Bill holds 24 US Patents on medical and visualization algorithms. In 1991, he was named a Coolidge Fellow, the highest scientific honor at GE's Corporate R&D.

Prior to joining GE in 1978, he was a Mathematician at the US Army Benet Weapons Laboratory where he worked on computer graphics software for structural analysis. He has a BS in Mathematics and an MS in Computer Science from Rensselaer Polytechnic Institute.

Michael W. Vannier is a Professor and Head of the Department of Radiology at the University of Iowa College of Medicine. He received a B.S.M.E in Mechanical Engineering from the University of Kentucky in 1971, a B.S. in Engineering Sciences from Colorado State University in 1971, and an M.D. in 1976 at the University of Kentucky. He then did his residency in Radiology at the Mallinckrodt Institute of Radiology, Washington University, St. Louis, where he later stayed to become Full Professor of Radiology and Affiliate Professor of System Science and Mathematics in the College of Engineering. Before taking his current position, from 1994 through 1996, Dr. Vannier was Georgia Eminent Scholar in Medical Imaging, Emory University School of Medicine.

Dr. Vannier is a Fellow of the American College of Radiology. He is a recipient of the Lindberg Award from the American Institute of Aeronautics and Astronautics. He is Editor-in-Chief of IEEE Transactions on Medical Imaging, and serves on the editorial boards of several other medical journals. His interests span the wide field of computer graphics and image processing to improve the diagnostic value of radiologic images.

Course Syllabus:

Unlike the presentation strategy in previous years, applications and case studies are not being held until the end of the course. Rather, a greater emphasis is being placed on case studies, affording them more time. The intent is to provide real examples for the lessons being taught and to illuminate motivating and limiting factors while teaching the techniques.

A Preliminary Course Syllabus

1st morning session - (90 mins total)

INTRO.[Yoo]        (15 mins)
why visualize?
        medical image pipeline
        topics to be covered
        topics not to be covered
        what we hope to accomplish today.

CASE STUDY:  Computer Assisted Neurosurgery [Kikinis]        (30 mins)

SEGMENTATION AND CLASSIFICATION [Kikinis]        	(30 mins)
        Bayesian Statistical Segmentation
	Deformable Surfaces/Volumes for Segmentation
	Atlas based segmentation

MEDICAL IMAGE ACQUISITION (CT) [Yoo]        (15 mins)
        Intro: CT, MRI, PET/SPECT, Ultrasound
	CT Physics -
        Parameters: slice thickness, tissue window
        Artifacts:  partial voluming, motion

---BREAK---

2nd morning session - (105 Mins total)

MEDICAL IMAGE ACQUISITION (MRI) [Yoo]        (15 mins)
        MRI Physics - 
        Parameters:  slice thickness, gap, noise
        Artifacts:  partial voluming, motion
                    geometric distortion, non-stationary intensity distortion

VOLUME RENDERING  [Yoo]        (30 mins)
        raycasting, splatting, Fourier rendering
        acceleration, parallel algorithms
        integrated segmentation and rendering (e.g., Volume Seedlings)
	hardware based volume rendering

CASE STUDY:  Marching Through the Virtual Human [Lorensen]        (30 mins)

SURFACE RENDERING  [Lorensen]        (30 mins)
        Marching Cubes, Dividing Cubes
        Textures and visualization
        Polygon Decimation
	Experiments in Smoothing

---LUNCH---

1st afternoon session - (90 Mins total)

CASE STUDY:  Optical Surface Scanning in Medicine [Vannier]        (30 mins)

CLINICAL PROBLEMS [Vannier]        (30 mins)
	Applications of 3D CT/MRI and Optical Surface Scanning
	Craniofacial reconstruction for surgical planning
	Prosthetic design for lower limbs

PROBLEMS SPECIFIC TO MEDICINE II [Kikinis]        (30 mins)
        Clinical performance:  rapid, robust tool development
        Interactive graphics in surgery:  do no harm
        Comparison: registration
	Multiple Sclerosis

---BREAK---

2nd afternoon session - (105 Mins total)

CASE STUDY:  Clinical Uses for Head Mounted Displays [Fuchs]        (30 mins)
	Volume ultrasound

DISPLAY  [Fuchs]        (30 mins)
        Perceptual issues: stereo, kinetic depth, head motion parallax
        Proprioception
        HMD
        video see through
        optical see-through
        challenges to head mounted display technology: 
                tracking, real-time rendering, registration.

PROBLEMS SPECIFIC TO MEDICINE  [Vannier]        (30 mins)
        Quantitative vs. Qualitative Measurement
        Noise: understanding the source of the data
        Robustness (in medicine)
	Volume and mass measurement

FUTURE/ RESEARCH DIRECTIONS [All speakers]        (15 mins)
        Open Floor Q&A