Transmission of Medical Images over Wide Area Networks
L. Rodney Long
Lister Hill National Center for Biomedical Communications
National Library of Medicine
Abstract: Rapid transmission of medical image
data across wide area networks remains a technical challenge,
due to the high data volume of the images and the transmission
rates typically achievable. We are investigating two methods for
potentially achieving better transmission rates when using the
TCP protocol: (1) multisocket transmission, an application-level
alternative to conventional single- channel TCP transmission; and
(2) implementations of Internet RFC 1323, which specifies TCP
extensions for high performance. This paper discusses
transmission rate limitations in current TCP and presents results
to date in using these alternative methods to send large image
files across the Internet and across dedicated high-speed links
with long signal path delays.
Keywords: medical images, wide area networks,
transmission rate, TCP/IP, multisocket, RFC 1323, bandwidth,
delay
1.0 Introduction
Wide area network delivery of large amounts of data is a topic of
increasing interest, particularly for the transmission of
biomedical images, such as the Visible Human digital color
slices, because of the large sizes of the image files and the
need to disseminate them on a national or international scale.
Network protocols have a direct impact on the transmission rate
achievable. At NLM's Communications Engineering Branch (CEB), we
are focusing on the transport protocol and its relationship to
link performance. We are investigating performance on the
critical communications links of today (Internet links) and also
on emerging high-speed optical fiber and satellite links. While
much attention has been given to the emplacement of links with
high physical capacity in building the National Information
Infrastructure, the role of the transport protocol is also
critical, since this protocol incorporates data flow control.
The most widely-used transport protocol for Internet
communication is TCP.
2.0 TCP characteristics affecting transmission rate
performance
TCP windows and data flow. To control data flow, TCP
uses windows on the sequence number spaces
corresponding to the byte streams being sent and received. The
TCP send window is illustrated in Figure 1.
The send window slides to the right in the stream of bytes to be
transmitted (send sequence space) as acknowledgments
are received for sequence numbers within the window, allowing new
packets to be sent to the network. Note that TCP sends data in
units of packets, but acknowledges data by using the
sequence numbers of bytes in the data stream being sent.
For example, in Figure 1, the receipt of
the packet containing bytes 512-1023 would be acknowledged by
transmitting a "1024" (next expected byte) back to the
sender. On the receive end, the receive window
incorporates a similar sliding window scheme. The receive window
size is a measure of the amount of buffer area that the receiver
has currently available to receive data. At this end of the
communication, there is a conceptual byte stream corresponding to
all of the bytes to be received. The receive window slides to
the right in this conceptual stream (receive window
space) as data is received into and emptied from its
available buffer area.
The TCP receiver transmits its current window size to the sender
in the TCP header. The sender uses this receive window size as
one of its key parameters in controlling data flow.
The sending TCP maintains a limit called the congestion
window which, absent any packet loss, is the same size as
the receive window. TCP maintains a dynamically-computed
estimate of the round-trip time (RTT) for data transfer; when the
RTT decreases, it assumes less congestion on the network, and
will increase the congestion window size; similarly, when the RTT
increases, TCP decreases the congestion window size.
At any time, the send window behaves as if its size S is given by
[1]:
S = minimum(receive window, congestion window).
The larger the send window, the more data can be sent to the
network, hence the greater throughput potential.
We are analyzing TCP performance in two network environments:
Case 1: Bandwidth-delay limited environment. If the
network environment is contention-free (such as a dedicated
point-to-point satellite link), the congestion window will be
sized the same as the receive window. Hence, the limiting factor
for S becomes only the receive window size. In order to maintain
full use of a TCP connection at a specified bandwidth, the
following relationship must hold:
receive window >= bandwidth * delay
where delay is the round-trip time mentioned above. If a
network's throughput is determined by the above relationship, we
refer to the network as bandwidth-delay limited. For
example, in order to achieve a throughput of 50,000 bytes/second
over a link with a 200 millisecond delay, the receive window must
be at least 10,000 bytes in size. In current TCP, the receive
window size is limited to 2**16-1. This limitation, commonly
called the "64K window limit", results from the fact that the
receive window size is passed to the sender in the TCP header,
where the data field allocated to hold the window size is only
two bytes in length. This has serious implications for network
transmissions where the transmission distance, and hence the
delay is great , and the desired throughput is high. These
so-called "long, fat networks" (LFNs) are directly impacted by
the receive window limitation in current TCP [2]. The achievable
bandwidth in current TCP may be calculated by
bandwidth <= (2**16-1)/ delay.
For example a transmission with a network delay of 50
milliseconds will have a throughput limited by about 1.3 million
bytes/second, regardless of the underlying physical capacity of
the link. Institutions such as NLM desire to transmit many
megabytes of data rapidly across long distances; these are
precisely the class of users suited to LFNs, and also those for
which the TCP window limitation becomes a potential problem. Figure 2 illustrates the bandwidth-delay
limits under current TCP for a range of network delays, by
graphing the relationship
bandwidth-delay limit = ( 2**16-1) / ( network delay) .
The significance of this relationship for emerging T-3 networks
is illustrated in Figure 3, where published
network delays [3] were used to graph the bandwidth-delay limits
for transmissions between College Park, Maryland and various
sites within the United States over ANSNET. As the figure shows,
the limitations are severe for all except the Boston
connection.
Case 2: Internet environment. Bandwidth-delay
limitations are not yet a major factor on Internet communications
within the continental United States. The reason: most Internet
users are connected to the Internet with at most a T-1 rate
connection. Current TCP can tolerate a delay of (Max window
size)/(T-1 rate) = 64 Kbytes/192,000 bytes/sec = about 340 ms
without bandwidth- delay affecting the achievable throughput.
In other words, unless the Internet delay is at least 340 ms,
the transmission will not be bandwidth-delay limited under
current TCP. On the Internet, delays within the continental
United States typically do not approach the 340 ms figure.
However, Internet transmission rates, while not typically
bandwidth-delay limited, frequently fall very short of the
physical link capacity. Figure 4 shows
the results of a three-week Internet data collection for which 5
megabyte digitized x-ray images were transmitted from the
University of Arizona (Tucson) to NLM, using the standard File
Transfer Protocol (FTP) tool, which uses one TCP connection for
data transfer. The files were sent at 15-minute intervals. The
graph shows the mean transmission time observed at each hour of
the day. The horizontal line shows the transmission time
expected based on the physical throughput capacity (192,000
bytes/sec) of the link. If the transmission achieved full link
capacity, the file-to-file transmission time should have been 27
seconds for link transmission, plus approximately 6 seconds for
file I/O time, giving a total of 33 seconds, as compared to FTP
overall mean file-to-file transmission time of 150 seconds.
Accounting for the lost performance, both in the dedicated link
case and in the Internet case, and developing technical methods
to regain it is the crux of our research. Obviously, we may not
expect to regain performance losses due to real congestion on the
Internet; but, surprisingly, as we shall explain, it appears that
much of the degraded performance in our Internet tests is not due
to actual congestion, and is in fact recoverable using
transmission alternatives to FTP.
3.0 Methods and results
Bandwidth-delay limited environment. The
bandwidth-delay limit has been known for years in the Internet
research community and was addressed in a set of proposals which
were incorporated into RFC 1323, "TCP Extensions for High
Performance" [2]. This RFC includes specifications for allowing
TCP to operate with a receive window up to size 230. The RFC
1323 specifications have been incorporated into an experimental
TCP which was tested on a DS-3 (44.7 Mbps) wide area network with
delay comparable to a transmission across the continental United
States; researchers reported achieving 89% of link capacity,
using a window size of 512 kilobytes [4]. In addition, Skibo
[5] reported achieving a 31 Mbit/sec rate on a DS-3 25
ms link between Urbana, Illinois and Murray Hill, New Jersey; the
bandwidth-delay limit for this link using conventional TCP is 21
Mbit/sec. The RFC 1323 specifications have also been
incorporated into commercially-available TCP software, for
example, a Sun Microsystems product called TCPLFN [6]. Using
this software, one researcher has reported obtaining 90% of
theoretical link throughput using a 100 kilobyte receive window
when transmitting 20 megabyte digital mammography images over a
T-1 geosynchronous satellite link [7]. TCP modifications for
performance other than those specified in RFC 1323 have been
proposed and experimentally tested, one of the most notable being
selective acknowledgment [8] of packets as opposed to
the cumulative acknowledgment scheme incorporated into
current TCP. We have focused on RFC 1323 because it addresses
the receive window problem and because commercial implementations
are readily available.
At NLM we are testing the performance of this software for
transmitting Visible Human and other large medical images. We
have also developed an alternative, application-level method
called multisocket transmission which has proven
effective in tests run over a bandwidth-delay limited link. (See
[9] for a description of the multisocket algorithm as implemented
by NLM.) We have conducted experiments on a T-1 satellite link
using this method. Visible Human digital color images (7
megabyte file size) have been transmitted between NLM and the
Laboratory for Radiological Informatics at the University of
California at San Francisco. A 100-transmission run using
Visible Human images yielded multisocket performance several
times faster than FTP. The results of this run, which used 12 TCP
connections is shown is Figure 5. In
addition, multisocket transmissions across the link have been
measured to perform faster than the (single TCP channel) limit
predicted by the bandwidth-delay product. In this particular
test, illustrated in Figure 6, a 5 megabyte
file was transmitted repeatedly using an increasing number of TCP
connections; as shown in the figure, when the number of
connections exceeded 9, the multisocket rate exceeded the limit
predicted by the bandwidth-delay product. (The bandwidth-delay
limit is a limiting factor for a single TCP connection, not for
the aggregate rate achievable by using multiple connections.)
For all data collected to date on the geosynchronous link, the
NLM tests indicate achieving approximately 80% of the T-1 link
capacity.
A multisocket approach has been independently implemented by
researchers at Ohio University, who have modified SunOS FTP
source code to build multisocket capability directly into FTP.
They have reported achieving 91% of T-1 link capacity over a
geosynchronous satellite link [10].
Internet environment. Transmission limiting factors in
the complex Internet environment are less well understood, but
tests conducted at NLM have demonstrated the existence of
available, unutilized bandwidth when TCP transmissions were made
on three different Internet links. Multisocket transmission
achieved significantly better performance by factors of 1.6 to
3.5 on these links, as compared to conventional TCP
transmission.
Our empirical data collections on the Internet show that (1)
throughput falls well below the barrier imposed by the TCP window
limit; and (2) there is available bandwidth not being used; this
bandwidth may be exploited by methods such as multisocket
transmission. The question of whether this degraded throughput
is correlated to TCP congestion control algorithms remains
unanswered: this would require observing TCP dynamically-changing
internal parameter values, such as the setting of the congestion
window.
What we have ascertained is that the our multisocket transmission
method outperforms conventional TCP transfer methods, in
particular FTP, on several Internet links, in three series of
tests. This data was taken on transmissions between NLM and the
University of Arizona (Tucson), Texas Tech University (Lubbock),
and the NASA Lewis Research Center (Cleveland). Data collections
of durations ranging from 24 hours to 29 days were made. For
Test Series 1-2 a 5 megabyte digitized cervical x-ray was
transmitted; for Test Series 3, a 7 megabyte Visible Human
digital color slice image was transmitted. For each collection,
the image file was repeatedly transmitted from the remote site
to NLM at small (10-15 minute) time intervals, using the
multisocket method. Also, at the same time intervals, but offset
from the multisocket transmission times, the same image file was
transmitted using FTP or a similar 1-socket-pair transfer.
(Conventional TCP transmission uses one pair of communication
end-points--"sockets"--to establish one TCP channel.)
Transmission times were logged, and statistics of the results
were computed, including mean, median, and standard deviation.
In every case, the average transmission time using the
multisocket method was significantly shorter than for FTP. The
speed-up ratios of multisocket transmission as compared to FTP
for the first test series ranged from 1.7 to 3.5, depending on
the particular link and data collection interval [9]. For the
second test series, the speed-ups ranged from 1.6 to 2.7 [11];
and in the single 1996 Internet data collection, the speed-up
ratio was 2.1. For Test Series 1, which included the
longest-duration collection, confidence intervals for the mean
transmission time values were computed, and the multisocket
intervals showed clear separation from the FTP intervals at a
95% level of confidence. (For example, the first data collection
from Tucson had a multisocket confidence interval of 6 seconds,
centered on a mean of 107; for comparison, the single-socket
confidence interval was 12 seconds, centered on a mean of 280.)
Figure 7 shows the performance of the
multisocket method for the same test setup as Figure 4; we concluded that in this case the
degraded FTP througput was not due to actual network congestion,
since the multisocket method achieved significantly greater
througput over the same link and test period.
4.0 Conclusion
Wide area network transmission of medical image data under TCP
protocol continues to be an area where conventional transfer
methods, such as FTP using current 64K-receive window TCP, are
frequently inefficient. High performance TCP as defined by RFC
1323 has been reported in the technical literature to be an
effective method for getting more efficient transmission in the
dedicated link, bandwidth-delay limited environment. NLM is
researching the use of this method not only for possible use on
dedicated links, but for use on the Internet, where its
performance is less well understood. In addition, NLM is
comparing the performance of an inhouse-developed method,
multisocket transmission, to the peformance of the RFC 1323
implementation. Multisocket transmission has been shown to have
improved transmission rate performance, as compared to FTP, by
factors ranging from 1.7 to 3.5 in a series of NLM tests on the
Internet. In addition, multisocket transmission has been shown
to surpass the single TCP connection bandwidth-delay limit in one
test conducted on a dedicated link, geosynchrous satellite
environment.
References
1. Comer DE. Internetworking with TCP/IP, Volume I, Principles,
Protocols, and Architecture. Prentice-Hall, Englewood Cliffs,
New Jersey, 1991.
2. Jacobson V, Braden R, Borman D. TCP extensions for high
performance. Internet RFC 1323, May 1992.
3. ANSNET T3 delay matrix report for April 1995. Available from
the Merit Network, Inc., Ann Arbor, Michigan as
ftp://ftp.merit.net/statistics/nsfnet/1995/nsf-9504.delay.gz.
4. Villamizar C, Song C. High performance TCP in ANSNET.
Computer Communications Review, October 1994; vol. 24, no. 5;
45-60.
5. Skibo T. Experiences with TCP extensions for
high-performance networks. Department of Computer Science,
University of Illinois at Urbana-Champaign. May 14, 1992.
6. Sun Microsystems. CONSULT-TCPLFN data sheet. Sun
Consulting, Mountain View, California, February 1996.
7. Barnett BG, Dudding KE, Abdel-Malek A, Mitchell RJ.
Satellite teleradiology testbed for digital mammography. Proc.
SPIE Medical Imaging '96, Newport Beach, California, February 10-
15, 1996; vol. 2217; 308-317.
8. Jacobson V, Braden R. TCP extensions for long-delay paths.
Internet RFC 1072, October 1988.
9. Long LR, Berman LE, Neve L, Roy G, Thoma GR. An
application-level technique for faster transmission of large
images on the Internet. Proc. SPIE Multimedia Computing and
Networking 1995, San Jose, CA, February 6-8, 1995; vol. 2417;
116-129.
10. Allman M, Ostermann S, Kruse H. Data transfer efficiency
over satellite circuits using a multi- socket extension to the
File Transfer Protocol (FTP). Proceeedings of ACTS Results
Conference; NASA Lewis Research Center, Cleveland, OH, September
11-13, 1995; Network section.
11. Long LR, Berman LE, Thoma GR. Client/Server design for fast
retrieval of large images on the Internet. Proc. 8th IEEE
Symposium of Computer-Based Medical Systems (CBMS'95), Lubbock
TX, June 9-10, 1995; 284-291.
Last Modified: 08:11am EDT, October 10, 1996