Concerns Over High Performance Computer
Exporters' Ability to Review End-Users in the PRC Prompted the
Requirement for Prior Notification
The January 1996 revisions to the Export Administration Regulations
governing HPCs made several other important changes. Most importantly,
they made exporters responsible for determining whether an export license
is required, based on the MTOPS level of the computer, and for screening
end users and end uses for military or proliferation concerns.134
Thus, U.S. companies that wish to export HPCs are now authorized to
determine their own eligibility for a license exception.135
Prior to this change, only U.S. HPC exports to Japan were allowed
without an individual license. At that time, a violation of the Export
Administration Regulations could be identified by an export of an HPC that
occurred without a license.
Since the change, in order to prove a violation of the regulations, the
Commerce Department must demonstrate that an exporter improperly used the
Composite Theoretical Performance license exception and knew or had reason
to know that the intended end user would be engaged in military or
proliferation activities.136
Also, the revised Export Administration Regulations required that
exporters keep records and report to the Commerce Department on exports of
computers with performance levels at or above 2,000 MTOPS. In addition to
existing record-keeping requirements, the regulations added requirements
for the date of the shipment, the name and address of the end user and of
each intermediate consignee, and the end use of each exported computer.
Although these records have been reported to the Commerce Department on a
quarterly basis for the past two years, some companies have reported
inconsistent and incomplete data for resellers or distributors as end
users.137
Since U.S. HPCs obtained by countries of proliferation concern could be
used in weapons-related activities, the Congress enacted a provision in
the Fiscal Year 1998 National Defense Authorization Act138 that required
exporters to notify the Commerce Department of all proposed HPC sales over
2,000 MTOPS to Tier 3 countries. The Act gives the U.S. Government an
opportunity to assess these exports within 10 days and determine the need
for a license. Following such notification, the Departments of Commerce,
State, Defense, and Energy, and the U.S. Arms Control and Disarmament
Agency, can review a proposed HPC sale and object to its proceeding
without an export license. The Commerce Department announced regulations
implementing the law on February 3, 1998.139
A November 1998 Defense
Department study, however, identified potential problems with the 10-day
notification procedure. The study noted that the Defense Department
provides comments on export notices referred to it regarding those end
users for which the Defense Department has information. The study also
noted that:
The operating assumption is that, if there is no information on
the end-user, then the end-user is assumed to be legitimate. This is
probably true in most cases; however, there is no means to verify that
high performance computers are not making their way to end-users of
concern to the United States.140
Furthermore, the Defense Department study expressed concern that
foreign buyers might circumvent current Export Administration Regulations
provisions requiring attestation to the buyer's knowledge that the export
will have no military or proliferation end user or end use.141 By
designating a company in the United States to act on its behalf, the
foreign company could have its U.S. designee submit the HPC notification
to the Commerce Department; the U.S. designee and not the foreign buyer
would then be responsible for all compliance with notification
procedures.142 The U.S. designee would be responsible only for shipping
the item and would not take title of the item.143
Under the Export Administration Regulations, the U.S. designee could
complete the notification to its knowledge, which might be useless if the
U.S. designee is in fact ignorant of the actual end use. The Defense
Department study noted the obvious problems with this system.
The study also observed that the 10-day notification period was
insufficient to ensure that U.S. designees and foreign buyers are
providing accurate and complete information.144
Finally, the Defense Department study warned that foreign buyers of
U.S. computer technology might circumvent the notification procedure by
notifying the Commerce Department that they are purchasing a system that
is not above the 7,000 MTOPS threshold, but later upgrading the system
with processors that are below the 2,000 MTOPS level. There would be no
requirement to notify the Commerce Department of the acquisition of the
lower than 2,000 MTOPS upgrades to the previously-notified system.145
The U.S.
Government Has Conducted Only One End-Use Check for High Performance
Computers in the PRC
The Fiscal 1998 National Defense Authorization Act now requires the
Commerce Department to perform post-shipment verifications on all HPC
exports of HPCs to Tier 3 countries with performance levels over 2,000
MTOPS.146
Post-shipment verifications are important for detecting and deterring
physical diversions of HPCs, but they do not always verify the end use of
HPCs.147
The PRC traditionally has not allowed the United States to conduct
post-shipment verifications, based on claims of national sovereignty,
despite U.S. Government efforts since the early 1980s.148 This obduracy
has had little consequence for the PRC, since HPC exports have continued
to be approved and, in fact, have increased in recent years.
In June 1998, the PRC agreed with the United States to cooperate and
allow post-shipment verifications for all exports, including HPCs.149 PRC
conditions on the implementation of post-shipment verifications for HPCs,
however, render the agreement useless.150 Specifically:
· The PRC considers requests
from the U.S. Commerce Department to verify the actual end-use of a U.S.
HPC to be non-binding
· The PRC insists that any
end-use verification, if it agrees to one, be conducted by one of its
own ministries, not by U.S. representatives
· The PRC takes the view that
U.S. Embassy and Consulate commercial service personnel may not attend
an end-use verification, unless they are invited by the PRC
· The PRC argues scheduling of
any end-use verification - or indeed, whether to permit it at all - is
at the PRC's discretion
· The PRC will not permit any
end-use verification of a U.S. HPC at any time after the first six
months of the computer's arrival in the PRC
The Select Committee has reviewed the terms of the U.S.-PRC agreement
and found them wholly inadequate. The Clinton administration has, however,
advised the Select Committee that the PRC would object to making the terms
of the agreement public. As a result, the Clinton administration has
determined that no further description of the agreement may be included in
this report.
According to Iain S. Baird, Deputy Assistant Secretary of Commerce for
Export Administration within the Bureau of Export Administration,
post-shipment verifications are conducted by the PRC's Ministry of Foreign
Trade and Economic Cooperation for U.S. computers having over 2,000 MTOPS
that are exported to the PRC. He says such verifications are done in the
presence of the U.S. commercial attaché.151
Commerce reported on November 17, 1998, that no post-shipment
verifications would be performed on HPCs that were exported to the PRC
from November 18, 1997 through June 25, 1998 because the PRC/U.S.
agreement applies only prospectively from June 26.
Since June 26, the Commerce
Department reported, only one post-shipment verification has been
completed and one was pending as of November 12, 1998. Commerce also
stated that "Post shipment verifications were not done on most of the
others [HPCs] because the transactions do not conform to our arrangement
with the PRC for end use checks."152
Thus, post-shipment verifications will not be done on any HPCs exported
to the PRC prior to the agreement, nor on any HPCs shipped that are
exported in the future under the Composite Theoretical Performance license
exception (that is, those between 2,000 and 7,000 MTOPS) to civilian end
users.
According to Commerce Department Under Secretary for Export Enforcement
William Reinsch, a pending regulatory change will instruct HPC exporters
to seek end-use certificates from the PRC Government. Where PRC end-use
certificates are obtained, this regulation purportedly would allow more
post-shipment verifications to be requested consistent with the PRC-U.S.
agreement.153
Reinsch stated that the PRC has indicated that it would be willing to
issue end-use certificates. However, the PRC office in question reportedly
has a staff of five, which would severely limit the number of
post-shipment verifications it could implement.154
According to a September 1998 report from the General Accounting
Office, U.S. Government officials agreed that the manner in which
post-shipment verifications for computers traditionally have been
conducted has limited their value because they establish only the physical
presence of an HPC, not its actual use. In any event, according to
national weapons laboratory officials within the Energy Department, it is
easy to conceal how a computer is being used.155
Even when U.S. Government officials perform the post-shipment
verification, the verifying officials have received no specific computer
training and are capable of doing little more than verifying the
computer's location. It is possible to verify an HPC's use by reviewing
internal computer data, but this is costly and intrusive, and requires
sophisticated computer analysis.156
The General Accounting Office report also noted that the U.S.
Government makes limited efforts to monitor exporter and end-user
compliance with explicit conditions that are often attached to HPC export
licenses for sensitive end users. The U.S. Government relies largely on
the HPC exporters to monitor end use, and may require them or the end
users to safeguard the exports by limiting access to the computers or
inspecting computer logs and outputs.157
The end user may also be required to agree to on-site inspections, even
on short notice, by the U.S. Government or exporter. These inspections
would include review of the programs and software that are being used on
the computer, or remote electronic monitoring of the computer.158
Commerce officials stated to GAO that they may have reviewed computer
logs in the past, but do not do so anymore, and that they have not
conducted any short-notice visits. They also acknowledged that they
currently do not do any remote monitoring of HPC use anywhere and that,
ultimately, monitoring compliance with safeguards plans and their
conditions is the HPC exporter's responsibility.159
Some U.S. High
Performance Computer Exports to the PRC Have Violated U.S.
Restrictions
During the 1990s, there have been several cases of export control
violations involving computer technology shipments to the PRC. One ongoing
case concerns the diversion of a Sun Microsystems HPC from Hong Kong to
the PRC.160
On December 26, 1996, a Hong Kong reseller for Sun Microsystems,
Automated Systems Ltd., sold an HPC to the PRC Scientific Institute, a
technical institute under the Chinese Academy of Sciences - a State
laboratory specializing in parallel and distributed processing. At some
point after the sale but before delivery, the computer was sold to
Changsha Science and Technology Institute in Changsha, Hunan Province. The
machine was delivered directly to that Institute in March 1997.161
Automated Systems of Hong Kong claimed to Sun officials in June 1997
that it had understood that the Changsha Institute was "an educational
institute in Wuhan Province providing technological studies under the
Ministry of Education." The end use there, according to Automated Systems,
was to be for "education and research studies in the college and sometimes
for application development for outside projects." Sun was recommended to
contact the end user, the Changsha Institute, for more specific end-use
information.162
The HPC sale came to the attention of the Deputy Assistant Secretary
for Export Enforcement, Frank Deliberti. He queried the U.S. Embassy in
Beijing about the Changsha Institute. Deliberti gave the information he
obtained to Sun Microsystems, which then initiated efforts to have its
computer returned.163
During the same period, the Foreign Commercial Officer at the U.S.
Embassy in Beijing consulted his contacts at the PRC's Ministry of Foreign
Trade and Economic Cooperation. The Ministry denied that the Changsha
Institute was affiliated with the PRC military.164
Subsequently, the Ministry called
the FCO to inform him that the actual buyer of the computer was an entity
called the Yuanwang Corporation, and that Sun Microsystems had been aware
of this corporation's PRC military ties. Reportedly, Yuanwang is an
entity of the Commission on Science, Technology, and Industry for National
Defense (COSTIND). So far as the PRC's Ministry of Foreign Trade and
Economic Cooperation reportedly could determine, the end-use statements
that had been provided to Sun through Automated Systems of Hong Kong were
totally fictitious. The Changsha Science and Technology Institute,
according to the Ministry, did not exist.165
The official position of the Ministry of Foreign Trade and Economic
Cooperation was that the PRC Government would not help to obtain the
return of the computer. The role of the PRC Government, the Ministry
asserted, had been merely to help two private parties rectify a
misunderstanding. In any event, the computer was returned to the United
States on November 6, 1997.166 The Commerce Department investigation
reportedly is continuing.167
A number of other violations of U.S. laws and regulations concerning
computers exported to the PRC have been investigated by the Commerce
Department:
New World Transtechnology
On December 20, 1996, New World Transtechnology of Galveston,
Texas, pled guilty to charges that it violated the export control laws and
engaged in false statements by illegally exporting controlled computers to
a nuclear equipment factory in the PRC in August 1992. The company was
also charged with attempting to illegally export an additional computer to
the PRC through Hong Kong in October 1992. The company was sentenced to
pay a $10,000 criminal fine and a $600 special assessment fee.168
Compaq Computer Corporation
On April 18, 1997, the Commerce Department imposed a $55,000
civil penalty on Compaq Computer Corporation of Houston, Texas, for
alleged violations of the Export Administration Regulations. The Commerce
Department alleged that, on three separate occasions between September 17,
1992 and June 11, 1993, Compaq exported computer equipment from the United
States to several countries, including the PRC, without obtaining required
export licenses. Compaq agreed to pay the civil penalty to settle the
allegations.169
Digital Creations On June
12, 1997, Digital Creations Corporation of Closter, New Jersey, was
sentenced to pay an $800,000 criminal fine for violating the Export
Administration Act and Regulations in connection with exports of computers
to the PRC. Digital had previously pled guilty in December 1994 to charges
that it had violated the Export Administration Regulations by illegally
exporting a Digital Equipment Corporation computer to the PRC without
obtaining the required export license.170
Lansing Technologies Corporation
On June 17, 1997, Lansing Technologies Corporation, of
Flushing, New York, pled guilty to charges that it violated the Export
Administration Regulations in 1992 by exporting a Digital Equipment
Corporation computer vector processor and a data acquisition control
system to the PRC without obtaining the required export licenses from the
Commerce Department.171
Other serious violations of HPC export control laws and regulations
have occurred in recent years, but these concerned Russia. On July 31,
1998, for example, the Department of Justice announced that IBM East
Europe/Asia Ltd. entered a guilty plea. IBM received the maximum allowable
fine of $8.5 million for 17 counts of violating U.S. export laws through
the sale of HPCs to a Russian nuclear weapons laboratory known as
Arzamus-16. In another example, an ongoing U.S. Government investigation
of Silicon Graphics Incorporated/Convex is examining whether a violation
of law occurred in a sale of HPCs to another Russian nuclear weapons
laboratory, Chelyabinsk-70.172
High Performance Computers at U.S.
National Weapons Laboratories Are Targets for PRC Espionage
No other place in the world exceeds the computational power found
within the U.S. national weapons laboratories. For this reason, both the
computational power and the data it can generate have been the focus of
the PRC's and other countries' intelligence collection efforts.
The desire for access to this computing power and data, in turn, is one
of the reasons so many foreign nationals want to visit the
laboratories.
According to David Nokes, the network administrator at Los Alamos
National Laboratory, all operating systems have vulnerabilities that can
be exploited by a knowledgeable, valid user.173 Nokes also says that there
are a few solutions to issues of HPC network security. These include:
· Allowing only U.S. students
to use the networks
· Limiting physical access to
high performance computer networks at universities
· Enhancing physical security
and security education at universities174
U.S. National
Weapons Laboratories Have Failed to Obtain Required Export Licenses
for Foreign High Performance Computer Use
When foreign nationals use the U.S. national weapons laboratories'
HPCs, their activities should generally be considered "deemed exports."
The "deemed export" rule [15 CFR 734.2 (b) (ii)] covers those situations
in which an export-controlled technology or software-source code
information is released to a visiting foreign national, for which a
license would have been required. In such situations, an "export" is
"deemed" to have occurred.
The Select Committee is concerned that HPC system managers in the U.S.
national weapons laboratories lack an essential understanding of the
deemed export rule. This lack of understanding was substantiated by
interviews with representatives from the Department of Commerce who had no
recollection of ever having seen an application for a deemed export from
any of the U.S. national weapons laboratories.
When PRC nationals visit and use the HPCs at a U.S. national weapons
laboratory, their access should be limited to the same computing
capabilities to which the PRC itself is restricted, especially for
military uses.175 The Select Committee discovered, however, that the
laboratories do not even measure the computational power of their HPCs in
MTOPS. Moreover, many of the laboratories have difficulty in converting to
MTOPS from the units they use to measure the power of an HPC.
The Department of Commerce could
not recall a laboratory ever having sought guidance on how to compute an
HPC's MTOPS rating. Significantly, the Select Committee discovered
that a rather modest HPC (by Department of Energy standards) in a U.S.
National Laboratory used by foreign nationals had a substantially higher
MTOPS rating than the controlled threshold. No licenses, however, had ever
been obtained.
The "deemed export" rule also applies in those instances in which a PRC
national or entity accesses an HPC remotely via the Internet.
In the absence of an effective audit system, which monitors the codes
being run by the PRC user, the U.S. national weapons laboratories cannot
verify that they are in compliance with the law, or that PLA or PRC
intelligence is not using the HPCs for the design or testing of nuclear or
other weapons.
PRC Students
Have U.S. Citizen-Like Access To High Performance Computers at the
National Weapons Laboratories
The U.S. national weapons laboratories rely upon nuclear weapons test
simulation software and computers provided by the Accelerated Strategic
Computer Initiative (ASCI). Five major U.S. universities support ASCI
through the Academic Strategic Alliances Program (ASAP).
As a result, hundreds of research students and staff at these
universities have access to the HPCs used by the national weapons
laboratories for U.S. nuclear weapons research and testing. As many as 50
percent of these research students and staff are foreign nationals, some
of whom may have foreign intelligence affiliations.
Holders of Immigration and Naturalization Service "green cards" - PRC
nationals who have declared their intent to remain permanently in the U.S.
- are treated as U.S. citizens for export control purposes. They are then
given U.S. citizen-like HPC access, free to return to the PRC once their
objectives are fulfilled.
In November 1998, the Secretary of Energy issued an Action Plan that
includes a task force to review HPC usage by foreign nationals and provide
a report to the Secretary within six months. The Department of Energy is
currently preparing an implementation plan to address counterintelligence
issues identified in a July 1998 report, entitled "Mapping the Future of
the Department of Energy's Counterintelligence Program," including HPC
usage by foreign nationals.
Many Types of Computer Technology Have
Been Made Available to the PRC That Could Facilitate Running Programs
Of National Security Importance
One of the bases for the 1996 increase in export control thresholds was
that individual PCs were widely available on the open market in the United
States, but not able to be exported to the potentially huge PRC market.176
What was an HPC in 1993 (those capable of 195 or more MTOPS) was no longer
even considered necessary to control for weapons proliferation
concerns.177
By 1997, PCs and workstations assembled in the PRC captured
approximately 60 percent of the PRC's domestic market.178 All of these
locally-assembled computers used imported parts - over 70 percent
contained United States-produced Pentium microprocessors.179
Three of the largest manufacturers in the PRC were affiliates of IBM,
Hewlett Packard, and Compaq, with a combined market share of approximately
21 percent.180 A large share (but probably not more than 20 percent) of
the PC assembly in the PRC was done by small, independent assembly
shops.181
The largest individual producer of PCs and workstations in the PRC is
the Legend enterprise, a spin-off of the Chinese Academy of Sciences.182
This domestic computer assembly industry dovetails well with Beijing's
overall plans for economic modernization. Beijing reportedly desires an
independent PRC source of most high-technology items to avoid reliance on
foreign providers for these goods.
To participate more fully in the
PRC market, United States firms have been pressured by the PRC government
to relinquish technological advantage for short-term market opportunities.
The PRC requires that foreign firms be granted access to the PRC
market only in exchange for transferring technology that would enable the
state-run enterprises to eventually capture the home market and begin to
compete internationally.
However, the PRC's strategy of coercing technology from foreign firms
has not enabled state-run industries to close the technology gap with more
developed nations. In the context of establishing domestic production of
computers for sale in the PRC, this PRC "technology coercion" policy
appears to have worked.183 The PRC now has a growing industrial base of
small computer assemblers. For the most part, these companies are not
State-run. The technology that was "coerced" from U.S. computer
manufacturers as a cost of entering the PRC market apparently better
serves the expansion needs of small, relatively independent enterprises
and not the intended needs of central planners in Beijing.
90 percent of PRC consumers of PCs and workstations are business,
government, and educational entities, with individual purchases accounting
for only 10 percent of the PRC's PC market.184 To illustrate the size of
the individual purchaser segment of the PRC's market, it is estimated that
only 5 million individuals out of the PRC's 1.2 billion have the
expendable funds required to purchase a low-end PC in the PRC.185
Despite the limited number of individual purchasers, the actual size of
the PRC PC and workstation market was 2.18 million units in 1996; 3
million units in 1997; and 4.5 million units in 1998. It is anticipated
the PRC PC and workstation market will grow at the rate of 1.5 million to
2 million units per year through the year 2000. According to figures
provided by the Asia Technology Information Project, an independent
research foundation, non-PRC manufacturers of PCs and workstations,
including U.S. manufacturers, could expect to partake of a portion of the
almost 2 million units expected to be imported for sale in the PRC in
1998.186
The PRC Has a Limited Capability to Produce
High Performance Computers
The PRC has demonstrated the capability to produce an HPC using
U.S.-origin microprocessors over the current threshold of 7,000 MTOPS. The
PRC "unveiled" a 10,000 MTOPS HPC - the Galaxy III - in 1997 based on
Western microprocessors.
But PRC HPC application software lags farther behind world levels than
its HPC systems. Also, despite the existence of a few PRC-produced HPCs
based on Western components, the PRC cannot cost-effectively mass-produce
HPCs currently. There really is no domestic HPC industry in the PRC
today.
While it is difficult to ascertain the full measure of HPC resources
that have been made available to the PRC from all sources, available data
indicates that U.S. HPCs dominate the market in the PRC.187
Although the PRC has a large market for workstations and high-end
servers, there is a smaller market for parallel computers which is
entirely dominated by non-PRC companies such as IBM, Silicon
Graphics/Cray, and the Japanese NEC. However, there continues to be
significant market resistance to Japanese HPC products in Asia, especially
as U.S. products are beginning to have significant market penetration.188
U.S. High Performance Computer Exports
To the PRC Are Increasing Dramatically
A review of Commerce Department information regarding the total of HPC
license applications that were received for the time frame January 1, 1992
to September 23, 1997, revealed the following:
· Only one HPC export license
to Hong Kong (with a value of $300,000) was rejected
· 100 HPC export licenses to
the PRC (with a total value of $11,831,140) were rejected by
Commerce
· 37 HPC export licenses to
Hong Kong (with a total value of $55,879,177) were approved
· 23 HPC export licenses to the
PRC for HPCs within the 2,000 to 7,000 MTOPS range (with a total value
of $28,067,626) were approved
· Two of the 23 HPC export
licenses to the PRC for HPCs within the 11,000 to 12,800 MTOPS range
(with a total value of $2,550,000) were approved in
1998189
The approximate total value of the HPCs exported, of whatever
description, to both Hong Kong and the PRC, for the six-year period ending
September 23, 1997, was only $86 million.190
The nine-month period between January 1998 and September 1998, however,
saw U.S. exporters notify the Commerce Department of their intention to
export 434 HPCs (in the 2,000 to 7,000 MTOPS range) to the PRC (total
value $96,882,799).191 Nine times the number of HPCs were exported in
one-ninth the time.192
During approximately the same time frame (calendar year 1998) it is
estimated that 9,680,000 individual PCs and workstations were sold in the
PRC. The market share that U.S. exporters could reasonably expect to
benefit from was approximately 3,872,000 units, worth approximately $1.8
billion.193
Apparently, the proximate cause of U.S. computer manufacturers
aggressively lobbying for the raising and maintaining of export thresholds
above the PC level was to capture this $1.8 billion per year market
share.
The United States dominates the PRC's HPC market, but U.S. exports
clearly do not dominate the PRC's personal computer and workstation
market.194 The difference between the 460-unit, $100 million HPC market
described above, stretched over a six-year period, and the yearly 3.8
million-unit PC and workstation market, with a value of $1.8 billion, is
dramatic.
The performance levels of U.S. HPCs reported to be exported to the PRC
over the past year continued to be predominantly in lower-end machines, as
shown in the following table. For example, 77 percent of U.S. HPCs (a
total of 388 machines) have performance levels below 4,000 MTOPS.
The PRC Is Obtaining Software From U.S.
and Domestic Sources
In June 1997, it was estimated that 96 percent of software programs
sold in the PRC were pirated versions of commercially available U.S.
programs. These programs were designed for use on PCs and workstations,
and are not considered useful for the very sophisticated programming done
on HPCs.
Some major U.S. software producers have begun contracting with PRC
programming firms. These PRC software firms are comprised of
recently-graduated PRC university students. They are attempting to write
programs in Chinese to capitalize on a huge domestic market.196
Two factors mitigate against the success of the PRC developing its
domestic programming industry.
The first factor is that street-level "software pirates" sell dozens of
U.S. computer programs at a time on one CD-ROM for a small fee (reportedly
$20). In other words, one can meet most or all of one's programming needs
in the PRC for a nominal fee. It is anticipated that it will be difficult,
if not impossible, for a domestic software industry to recoup the start up
costs associated with just one software program, let alone the dozens
needed to compete with the street level dealers.
The second factor is that these pirated U.S.-produced, English language
programs are more mature, widespread, and robust than PRC programs.197 It
is axiomatic that any new product will have "bugs in the system." It is
considered unlikely that new, unproven, and possibly weak software
programs will effectively compete with cheap, proven, and robust software
that is widely available at such nominal fees. It is conceivable that the
PRC will abandon instituting a domestic programming industry
altogether.198
Potential Methods of Improving End-Use
Verification
According to a 1996 RAND study, there are non-intrusive and intrusive
approaches to assessing the manner in which a buyer is actually applying
dual-use technologies. Among the non-intrusive methods are:
· Memoranda of understanding
and agreements
· National technical means of
verification
· Limitations designed into the
transferred technologies
· Transparency
measures
Among the intrusive methods are:
· Inspections
· Tagging199
Tagging
Tagging is achieved by attaching an active system to the item that is
to be exported, rather than just a passive tag for identification during
an inspection. The active system would both monitor the object tagged and
communicate that information back to the United States. The RAND study
noted that in practice, this means the objects to be tagged must be
physically large systems, such as a machine-tool cell, or a major
component of some larger system, such as a turbine engine in a
helicopter.200
According to the RAND study, the tag should be capable of at least
communicating information about the item's physical location. Some sensors
may provide other kinds of information, as well. The information could be
communicated to a satellite or over a data link. Early versions of such
devices were already in use in 1996 to monitor nuclear materials and
technologies.201
These "smart" tags exploit the potential of several technologies,
according to the RAND study. They combine encryption, the Global
Positioning System, and emerging global wireless communications systems,
such as Iridium or Orbcomm. These technologies would allow the tags to
report back on the status and location of the tagged object. In principle,
such tags could report the position of an object at any given time in
order to verify limitations on their location. Such tags could also report
on the activities of a "smart" system to which they are attached. For
example, a machine-tool cell could report whether the machine had been
used to make parts resembling aircraft components.202
Such tags could have many applications in a cooperative regime. Their
application and use in a prohibited environment would be more difficult
and consequential.203
The RAND study cautioned that all sellers of a particular technology
must participate in the tagging and that this would probably also require
cooperation of the buyers. Otherwise, buyers would gravitate to untagged
items, if they were available. Attempts to conceal system location or
deviate from a pattern of cooperation would be considered evidence of a
potential failure of performance by the buyer. The study concluded that
tagging may become an important oversight method for controlling
technology transfers, but that it should never become the sole means of
oversight.204
Technical
Safeguards
In 1994 several types of technical safeguards were in advanced
development in the United States. The technologies required for these
safeguards were expected to enter testing within the next two years. They
included:
· Controlled-execution UNIX
- a modified computer operating system that could run only certain
pre-approved programs; likely to be most useful for computers sold to
facilities such as weather-forecasting centers, oil companies,
automobile manufacturers, and banks
· "Black box" monitoring
hardware - inexpensive, secure, long-term audit recording devices,
possibly based on write-once optical storage units that could be
embedded in mass-produced workstations; analogous to the black box
flight-data recorders that are installed in aircraft and used for
post-crash accident analysis
· "Meltdown" software -
modified operating system programs designed to require updating by the
manufacturer at fixed times; if not updated, the computer refuses to
run
· Automated auditing tools
- pattern-recognition or rule-based software; would assist monitoring
agencies to more effectively inspect huge collections of data from
system activity logs and detect the (presumably few) incidents worth
detailed analysis
Although these technical safeguards seem feasible, none had been proved
to be inexpensive, sensitive enough to detect most illegal activity, and
difficult to circumvent by determined adversaries. The auditing tools
under development showed great promise, however. Authorities were
pessimistic about the likelihood that technical high-performance computer
safeguards would be widely adopted and able to succeed in the near
future.
Other
Possibilities
Officials of the Mitre Corporation made several suggestions to
strengthen U.S. national security in the context of HPC export controls.
These included:
· Improving and enforcing
end-use and end-user verification
· Controlling embedded HPC
systems that are useful in military applications
· Monitoring or precluding the
expansion capability of computer hardware
· Marketing aggressively all
generic computing capabilities, such as scanning, to the PRC to
maximize profits and to keep the PRC market-dependent on the United
States
· Focusing on control of any
hardware, software, tools, and services that uniquely support PRC
military applications that are strategic in nature or could
facilitate the tactical turning point in a conflict205
Chapter 3 Technical
Afterward
CHANGING HIGH PERFORMANCE COMPUTER
TECHNOLOGY IS MAKING EXPORT CONTROL MORE DIFFICULT
ew designs in HPCs and systems of computers, as
well as availability of more advanced and less costly processors,
software, and peripheral equipment, is rendering the challenge of applying
export controls to HPCs more difficult.
For certain types of computer designs, the ability to add processors or
boards could increase the machine's performance beyond authorized levels.
In addition, advances in computer processor communications technology have
facilitated the clustering of personal computers and workstations into
effective parallel computers.
The usefulness of clustered computers is application-dependent. Some
U.S. Government and computer industry experts have concluded that for many
problems, networks of workstations could not compete with appropriately
designed high performance computers.206 Most traditional HPCs achieve far
greater efficiency than parallel machines, due to their use of custom-made
components.
Foreign access to high performance computers through networks is
possible because of inadequate security measures.
Vector Architectures
Vector architecture relies on custom-designed processors to move a
complex problem through computer processing units in sequential stages.
This type of machine is designed to handle arithmetic operations
efficiently on elements of arrays, called vectors.207
Vector systems are especially useful in high-performance scientific
computing.208 Vector systems, also called "pipeline" architectures, work
like an assembly line. They work best with many similar tasks that can be
broken down into steps.
The memory interface in vector machines is custom-made, and subject to
export controls.
Vector machines are useful for
cryptography, modeling fluids, and in the design of weapons. In
particular, vector systems are suited to problems in which data at one
point influence other variables in the problem, a common situation in
national security applications.209
It is more straightforward for a programmer to use a vector system than
a system comprised of parallel processors (discussed below), since it is
easier to obtain maximum performance with one or a few high-power
processors than with a collection of many lower capability
processors.210
Since one of the main concerns with any HPC system is the rate of speed
with which data can be retrieved from memory, another advantage is that a
vector machine has a very fast memory.211
Still further advantages of vector systems are that they feature high
memory bandwidth and low memory latency - that is, very large amounts of
data can travel to and from memory very efficiently. A related advantage
is that vector systems have the ability to seek multiple memory locations
at the same time. This translates into very fast computational speed.
A disadvantage of a vector machine is that vector system software is
not really portable. It cannot be readily transported to other vector
machines.212
The main disadvantage of vector systems, however, is their high cost.
Significant improvements in software and hardware allow the purchase of a
parallel processing system for $40,000, as opposed to $1 million for a
comparable vector computer.213
At the Defense Department's High Performance Computer Management
Office, vector systems are being phased out in favor of parallel
processing systems. Out of a total of 40 HPCs in the High Performance
Computer Management Office inventory, fewer than 10 are now vector
systems.214
Parallel Processing: The Connection of Computers Into a Powerful
Central Resource
A parallel processing computer is a collection of processors that are
connected through a communications network.215 The type of processor, the
network configuration, and the operating system that coordinates the
activities distinguish parallel processing systems.
Many national security applications involve problems that can be
separated into independent variables, and it is for these types of
problems that parallel processing is best suited.216
The fastest parallel machines are all based on commodity processors -
that is, processors that are commercially available on the market.217 This
approach has been applied to virtually every area of theoretical and
applied physics.218
Massively Parallel Processors
A massively parallel processor is a collection of computers, or central
processing units, linked together.219 Each computer that is part of the
whole massively parallel processor has its own memory, input/output
system, and central processing unit.220 Massively parallel processors now
use commodity processors, and can utilize commodity interconnects to
communicate between the individual computers that make up the system.221
Some massively parallel processors use custom-made, very fast interconnect
switches that are not commodities and are subject to export
control.222
An advantage of a massively parallel processor is that an unlimited
quantity of processors can be incorporated into the design of the machine.
In a massively parallel processor, the more processors, the greater the
computing speed of the machine.223
Because each processor is equipped with its own memory, massively
parallel processors have much more memory than traditional supercomputers.
The extra memory, in turn, suits these machines to data-intensive
applications, such as imaging or comparing observational data with the
predictions of models.224
A disadvantage of massively
parallel processors is that memory latency is a bigger problem because the
processors have to share the available memory. Another disadvantage is
that each one of the computers that is part of the system has to be
instructed what to do individually.225 This phenomenon requires
specialized, extremely proficient programmers to create efficient
communications between the individual computers.
The commercial availability of inexpensive, powerful microprocessors
has given massively parallel processors a boost in their competition with
vector machines for the supercomputer market. IBM, for example, more than
doubled the number of its computers in the Top 500 list (discussed below)
between November 1997 and June 1998 by introducing the SP2, which strings
together up to 512 of the company's RSI6000 workstation
microprocessors.226
If optimum speed is desired, this massively parallel configuration is
the best of all HPC designs.227 The fastest high performance computer now
available is the ASCI Blue Pacific.228 That machine is part of the
Department of Energy's Accelerated Strategic Computing Initiative (ASCI)
program and is located at Lawrence Livermore National Laboratory.
Developed in conjunction with IBM, it is a 5,856-processor machine,
boasting a top speed of 3.8 teraflops229 (Tflops) with 2.6 terabytes
(Tbytes) of memory.230 In the next phase of the ASCI initiative, IBM will
deliver a 10-Tflops machine to the Department of Energy in
mid-2000.231
Symmetrical Multiprocessor Systems
Symmetrical multiprocessor systems use multiple commodity central
processing units (CPUs) that are tightly coupled via shared memory. The
number of processors can be as low as two and as many as about 128.232
Symmetrical multiprocessor systems treat their multiple CPUs as one
very fast CPU.233 The CPUs in a symmetrical multiprocessor system are
arranged on a single motherboard and share the same memory, input/output
devices, operating system, and communications path.
Although symmetrical multiprocessor systems use multiple CPUs, they
still perform sequential processing,234 and allow multiple concurrent
processes to be executed in parallel within different processors.235
An advantage of symmetrical multiprocessor systems is that the
programming required to control the CPUs is simplified because of the
sharing of common components.236
Another major advantage is cost. A Silicon Graphics symmetrical
multiprocessor system, for example, with 18 microprocessors, each rated at
300 megaflops (MFLOPS)237 or more, and a peak speed of more than 5
gigaflops (GFLOPS), costs about $1 million, whereas a Cray C90 costs about
$30 million.238
Even though the Silicon Graphics machine is about a third as fast as
the Cray machine, it is still very popular with consumers of these types
of machines. The University of Illinois Supercomputing Center reportedly
likes the price, flexibility, and future promise of symmetrical
multiprocessor systems so much that it plans to use them exclusively
within two years. Its older Crays were "cut up for scrap" at the beginning
of this year, and its massively parallel computers will be phased out by
1997.239
One disadvantage of a symmetrical
multiprocessor system is that all the CPUs on a single board share the
resources of that board. This sharing limits the number of CPUs that
can be placed on a single board.240
Although the programming model that a symmetrical multiprocessor system
provides has proved to be user-friendly, the programmer must exercise care
to produce efficient and correct parallel programs. To limit latency in
individual jobs, most software requires enhancement - for example,
employing special programming techniques to prevent components of the
computer program from competing for system resources - thereby increasing
inefficiency.
For this reason, symmetrical multiprocessor systems are not good
platforms for high-performance real-time applications.241
In a symmetrical multiprocessor system design, as is true with a
massively parallel processor system, the number of CPUs determines how
fast a machine potentially will operate. This fact causes a problem for
export controls because it is possible to add CPUs to the boards of a
symmetrical multiprocessor system, or boards to a massively parallel
processor system, and push the machine over export control thresholds
after the original export-licensed purchase.242
Clusters of Commercial Off-the-Shelf Computers and Networks
Recent advances in the process of computer-to-computer communication,
or networking, allow computers to be linked together, or "clustered."
Networking has allowed the clustering of personal computers and
workstations into well-balanced effective parallel computers, with much
higher computing capabilities than any one of the clustered
computers.243
Four thresholds have been crossed in connecting
commercial-off-the-shelf components to create parallel computers:
· Using
commercial-off-the-shelf components to create parallel computers is
simple because of the ease of hardware configuration and the
availability of all necessary system software from market vendors
· It is versatile because a
wide range of possible network designs with excellent communication
characteristics and scalability to large sizes is now available
· Clustered systems performance
has now matured to the point that network communication speed is
within 50 percent of that in vendor-assembled parallel computers244
· Commercial-off-the-shelf
clusters are now affordable
According to officials at the Lawrence Livermore National Laboratory,
networking represents only a 10 percent additional cost over the cost of
the computing hardware for large systems. Thus, up to approximately 50,000
MTOPS, the computing capability available to any country today is limited
only by the amount of money that is available to be spent on
commercial-off-the-shelf networking.245
A typical commercial-off-the-shelf networking technology contains five
essential elements. They are all inexpensive and widely available. The
three hardware elements are switches (approximate cost: $2,000), cables
(approximate cost: $100), and interface cards (approximate cost: $1,500).
The two software elements are low-level network drivers for common
operating systems, and industry standard communication libraries. The
hardware and software technology necessary to successfully cluster
commercial-off-the-shelf CPUs into effective parallel computers is well
developed and disseminated in open, international collaborations
worldwide.246
The concept of clustering commercial-off-the-shelf computers has been a
subject of open academic study for over a decade. Today, the Beowulf
Consortium acts as a focal point for information on clustering technology
and has links to many projects. One Beowulf project is the Avalon computer
at Los Alamos National Laboratory. Avalon can operate at 37,905 MTOPS247
and was built in four days in April 1998 entirely from commodity personal
computer technology (70 DEC Alpha CPUs) for $150,000.
Although commercial-off-the-shelf networking technology has only
recently become effective, it has been adopted rapidly. There currently
are at least seven competing high-performance network technologies (over
100 megabytes per second or higher): Myrinet, HIPPI, FiberChannel, Gigabit
Ethernet, SCI, ATM, and VIA. One network vendor reported over 150
installations in the United States and 17 foreign countries including
Australia, Brazil, Canada, the Netherlands, England, France, India,
Israel, Italy, Japan, the Republic of Korea, and the PRC.248
Gigabit Ethernet is of particular interest because it is being
developed by a cooperative, worldwide industry effort called the Gigabit
Ethernet Alliance. 74 companies have pledged to develop products for the
open standard - that is, the source software is available openly to
software developers. Foreign companies are alliance members and also
participate as members of the steering committee and the certification
process for compliance. Gigabit Ethernet is projected to be a $3 billion
market by the year 2000, which at today's prices translates into
approximately 300,000 network switches per year.249
On October 15, 1997, a group of
experts met to discuss computer performance metrics for export control
purposes. The computer and high-tech industries were represented by
Hewlett-Packard, Silicon Graphics/Cray Research, IBM, Digital Equipment
Corporation, Intel, Sun Microsystems, the Center for Computing Sciences,
the Institute for Defense Analyses, and Centerpoint Ventures. The U.S.
Government was represented by the National Institute of Standards and
Technology, the Naval Research Laboratory, the Defense Advanced Research
Projects Agency, the National Security Agency, Lawrence Livermore National
Laboratory, the Defense Technology Security Administration, and the
Department of Commerce Bureau of Export Administration.250
The consensus of the discussion was that commercial-off-the-shelf
networking is not so significant a threat to replace HPCs as might at
first appear to be the case:
Networks of workstations using [commercial-off-the-shelf]
networking technology differ from supercomputers. Some problems will run
easily and effectively on such networks, while other classes of problems
important to national security concerns will not run effectively without
a major software redesign effort. For many problems no amount
of software redesign will allow networks of workstations to compete with
appropriately designed high performance computers.
Even if a "rogue state" assembled such a large network of
workstations by legitimately acquiring large numbers of commodity
processors, the actual effort to produce the software necessary to
realize the full potential of such an aggregate system would take
several years. During this time, the state of the art of
computational technology would have increased by approximately an order
of magnitude.
After considerable discussion, most of the participants were in
agreement that there was a fundamental difference between a system
designed by a single vendor that was built as an aggregate of many
commodity processors and included the software to enable these
processors to cooperatively work on solving single problems of national
concern, and a large collection of commodity processors not subject to
export control that are externally networked
together.251
According to one expert, many universities have clustered systems, as
they are easy to establish. For $70,000, a 12-node system with two Pentium
II processors at 300 megahertz (MHz) each would produce a system with
7,200 GFLOPS.. However, the system must be properly structured to perform
well, and performance will vary depending on the application, the
programmer's ability, and the connection of the machines. An integrated
system from Silicon Graphics/Cray will achieve between 10-20 percent of
peak performance at best.252
An example of a powerful commercial-off-the-shelf network can be found
at the Illinois Supercomputing Center. Four eight-processor and two
16-processor machines from Silicon Graphics are connected in a cluster
with a peak speed of nearly 20 GFLOPS.253
According to one expert, it does not require any special expertise to
network workstations using commercial-off-the-shelf technology. The
software engineering techniques are being taught to undergraduates as part
of standard courses in advanced computing, but anyone with programming
knowledge should be able to create a network as well.254
The parallel supercomputers of today have peak speeds of over 100
billion floating point operations per second (100 GFLOPS). This is roughly
100 times the peak speed of a Cray YMP class machine, which was the
standard for high-performance computing of just five years ago.255
However, it is difficult to achieve a high percentage of this peak
performance on a parallel machine.
Whereas a tuned code running on a
Cray might reach 80-90 percent of peak speed, codes running on parallel
computers typically execute at only 10-20 percent of peak.256 There
are two reasons for this:
· The first is that Cray-class
computers incorporate extremely expensive, custom-designed processors
with vector-processing hardware. These processors are designed to
stream large amounts of data through a highly efficient calculational
pipeline. Codes that have been tuned to take advantage of this hardware
("vectorized" codes) tend to run at high percentages of peak
speed.257
Parallel machines, on the other hand, are generally built from much
simpler building blocks. For example, they may use the same processors
that are used in stand-alone computer workstations. Individually, these
processors are not nearly so sophisticated or so efficient as the vector
processors. Thus, it is not possible to achieve so high a percentage of
peak speed.258
Some parallel machines contain custom processors (TMC CM-5 vector
units) or custom modifications of off-the-shelf processors (Cray T-3D
modified DEC alpha chips). Even in those cases, however, the percent of
peak achievable on a single node is still on the order of 50 percent or
less. In parallel computer design, there is constant tension between the
need to use commodity parts as the computational building blocks in
order to achieve economies of scale, and the desire to achieve
ever-higher percentages of peak performance through the implementation
of custom hardware.259
· The second reason that
parallel computers run at lower percentages of peak speeds than vector
supercomputers is communications overhead. On parallel computers,
the extraordinary peak speeds of 100 GFLOPS or more are achieved by
linking hundreds or even thousands of processors with a fast
communications network.
Virtually all parallel computers today are "distributed memory"
computers. This means that the random access memory (RAM) is spread
though the machine, typically 32 megabytes at each node. When a
calculation is performed on a parallel machine, access is frequently
needed to pieces of data on different nodes.
It may be possible to overlap this communication with another
computation in a different part of the program in order not to delay the
entire program while waiting for the communication, but this is not
always the case. Since the timing clock continues while the
communication is taking place, even though no calculational work is
being performed, the measured performance of the code goes down and a
lower percentage of peak performance is recorded.260
Domain Decomposition
"Domain decomposition" involves partitioning the data to be processed
by a parallel program across the machine's processors.261
In distributed memory architectures, each processor has direct access
only to the portion of main memory that is physically located on its node.
In order to access other memory on the machine, it must communicate with
the node on which that memory is located and send explicit requests to
that node for data.262 Figuring out the optimal domain decomposition for a
problem is one of the most basic and important tasks in parallel
computing, since it determines the balance between communication and
computation in a program and, ultimately, how fast that program will
run.263
Memory access constitutes an inherent bottleneck in
shared-memory systems.264
Highly Parallel Technology
Microprocessor-based supercomputing has brought about a major change in
accessibility and affordability. Massively parallel processors continue to
account for more than half of all installed supercomputers worldwide, but
there is a move toward shared memory, including the use of more
symmetrical multiprocessor systems and of distributed-shared memory. There
is also a tendency to promote scalability through the clustering of shared
memory machines because of the increased efficiency of message passing
this offers. The task of data parallel programming has been helped by
standardization efforts such as Message Passing Interface and
High-Performance Fortran.265
Highly parallel technology is becoming popular for the following
reasons. First, affordable parallel systems now out-perform the best
conventional supercomputers. Cost is, of course, a strong factor, and the
performance per dollar of parallel systems is particularly favorable.266
The reliability of these systems has greatly improved. Both third-party
scientific and engineering applications, as well as business applications,
are now appearing. Thus, commercial customers, not just research labs, are
acquiring parallel systems.267
Twice a year the "Top 500 list," a compendium of the 500 most powerful
computer systems, is published.268 On the previous page is an example of
the numbers and types of systems in the biannual list of the top 500
fastest computers. As this chart points out, massively parallel processors
and symmetrical multiprocessor systems are on the rise, while vector
systems are losing ground.269
Microprocessor Technology
While vector and massively parallel computers have been contending for
the supercomputing market, an important new factor has become the
availability of extremely powerful commodity microprocessors, the
mass-produced chips at the heart of computer workstations.
Ten years ago, workstation microprocessors were far slower than the
processors in supercomputers. The fastest microprocessor in 1988, for
example, was rated at one million floating point operations per second
(MFLOPS) while Cray's processors were rated at 200 MFLOPS.270 A
floating-point operation is the equivalent of multiplying two 15-digit
numbers. Today, Cray's processors have improved by a factor of ten, to two
gigaflops in the brand-new T90; but the fastest microprocessor runs at 600
MFLOPS, an improvement by a factor of 600.
Commercial off-the-shelf microprocessor power is available for a
fraction of the cost of a traditional vector processor. Unlike vector
processors, which consist of complex collections of chips and are only
fabricated by the hundreds each year, commercial off-the-shelf
microprocessors are designed for mass production based on two decades of
experience making integrated circuits. Research and development costs for
each commercial off-the-shelf microprocessor are spread over hundreds of
thousands of chips.271
Microprocessors, also known as CPUs, are integrated circuits. They can
be divided into broad categories of logic family technologies. The
selection of a certain logic technology in the design of an integrated
circuit is made after determining an application and weighing the
advantages of each type of logic family. Among these are:
· Emitter-Coupled Logic
(ECL) is used for circuits that will operate in a high-speed
environment, as it offers the fastest switching speeds of all logic
families; it is the first type HPC chip. ECL, however, is power-hungry,
requires complex cooling techniques, and is expensive.272
· Complementary Metal-Oxide
Semiconductor Logic (CMOS) is relatively inexpensive, compact and
requires small amounts of power. CMOS off-the-shelf is the standard PC
or workstation chip; proprietary CMOS is custom-built, specially
designed for the particular HPC and incompatible with PCs and
workstations.
Realizing the differences between logic technologies gives a
perspective to understanding where CPU technology is headed, and the
reasons that the market is driving one technology faster than another. As
the following chart illustrates, commercial off-the-shelf, inexpensive
CPUs are coming to dominate the high performance computing world.273
Interconnect Technology
In multiprocessor systems, actual performance is strongly influenced by
the quality of the "interconnect" that moves data among processors and
memory subsystems.274
Traditionally, interconnects could be grouped into two categories:
proprietary high-performance interconnects that were used within the
products of individual vendors, and industry standard interconnects that
were more readily available on the market, such as local area networks.275
The two categories featured different capabilities, measured in bandwidth
and latency.
Recently, a new class of interconnect has emerged: clustering
interconnects. These offer much higher bandwidth and lower latency than
local area networks. Their shortcomings are comparable to proprietary
high-performance interconnects, including lower bandwidth, higher latency,
and greater performance degradation in large configurations or immature
system software environments.276
Message Passing Interface
Message Passing Interface (MPI) is a program containing a set of
sub-routines that provide a method of communication that enables various
components of a parallel computer system to act in concert. The
communications protocol that MPI uses is the same utilized by the
Internet. According to Dr. Jeff Hollingsworth of the University of
Maryland Computer Science Department, an example of how each of the
different software applications interact with the hardware would be as
follows:277
Application (Code)
MPI
TCP/IP
Linux
Windows NT
(Operating system)
Hardware
Some software, says Hollingsworth, is sold in a version that is
compatible with MPI. One example is automobile crash simulation software.
This software, which is essentially code to simulate a physical system in
three dimensions, is adaptable to other scientific applications such as
fluid dynamics, according to Hollingsworth.278
Hollingsworth states that software that is not already "MPI ready" can
be modified into code that can be run in an MPI, or parallel, environment.
Modifying this software to enable it to run in an MPI environment can be
very difficult, or quite easy, says Hollingsworth, depending on "data
decomposition." 279
The ease of converting software that is not "MPI ready" into an "MPI
ready" version is dependent on the expertise of the software engineers and
scientists working on the problem. For a single application and a single
computer program, the level of expertise required to convert a computer
program in this way is attainable in graduate level, and some
undergraduate level, college courses, according to Hollingsworth.280
It has not been possible to determine which, if any, commercially
available software is both MPI ready and applicable to defense-related
scientific work.
|