Hardware
GM/CA @ APS computing environment has identical structure
for all three beamlines: 23-ID-D, 23-ID-B and 23-BM. In order to provide better
computing reliability and avoid heavy network traffic, each beamline is setup
to have independent computing cluster running on a separated subnet. The
structure of single cluster is shown on the picture below.
The core of each cluster is constituted by the
fiber-disk storage array with 80MB/s data writing speed. The capacity of the
array is 16TB at the ID beamlines and 8TB at the BM beamline. This capacity
allows us to keep users data for two weeks after the experiment. This storage
is placed on a 2Gb Storage Area Network. Users who log on the beamline
workstations have their home directories on the storage array and thus all
the workstations access the same home directory for a given user account.
Users are provided with two groups of
workstations. One group is allocated for collecting and processing data on
the day of experiment (day-1 workstations) and the other one for those users
who wish to continue processing or backing up their data after the experiment
is over (day-2 workstations). Almost all computers operate under CentOS-5, a
freeware clone of Redhat Enterprise Linux operating system and have a number
of crystallographic data processing software installed including HKL200,
PyMol, and others. For the details on crystallographic software please check
User Program pages. Typically one Windows XP computer per beamline is provided
mostly for backing up data to windows-formatted external drives.
Most of the day-1 workstations can access the
storage array directly via Global File System (GFS). The names of these
workstations are blXws1, blXws2, blXws6, and marX where X stands for the
beamline number ('1' for ID-in, '2' for ID-out, and '3' for BM). These
workstations have the fastest possible data access and also a faster
processor. As so, they are recommended for data processing and backups (except
for marX dedicated to the data acquisition). The other workstations access
the storage array via NFS exports from the server named blXdl385. While the
data access rate for them is ~2x slower, these are still very fast
dual-processor computers with 2GB of memory. They can serve any computing
needs including data processing and backups while not affecting data collection
and processing rates on day-1 workstations.
The following computing policies are implemented:
- The account management is centralized and all workstations access the same
home directories that actually reside on the storage array.
- No disk quota is enforced on user account.
- All workstations are provided with firewire or USB connectivity and users
are encouraged to bring external firewire or USB-2 disks for making data backups.
Typical firewire/USB-2 speeds are ~16MB/s. One Linux and one Windows computer
per beamline is also provided with eSATA connectivity which is 2x-3x faster
than firewire or USB-2.
- Users can also FTP/SFTP out their data. The speed rate may vary on the
route to user's institution. The best expected rate is about 7-8MB/s. Due to
tight ANL security restrictions, there is no option to FTP/SFTP in, i.e. to
access data from user's home institution.
- Users laptops are provided with 1Gb access to GM/CA network via RJ45 cable
connection. The IP setup is automatically served by the DHCP servers.
From this network users can access any workstation via SSH/SFTP. They can also
mount their home directory on the storage array using SAMBA connections
to the blXdl385 servers. To connect, look for the hubs labeled ID-IN/OUT
NETWORK LINK in the users area.
Besides the wired access, APS provides 54g wireless network in the GM/CA area.
This wireless network allows to access outside Internet resources like web
pages and e-mail, but not the GM/CA computers.
- GM/CA @ APS stores users data for two weeks from their experiment start date.
During this period users are expected to verify that their backup was
successful and data were safely delivered to their home institution. After
two weeks an e-mail is sent to remind users about scheduled deleting of their
data and then in two days the data is automatically deleted from the GM/CA
storage array.
- User accounts will be automatically disabled two weeks after their
experiment start date.
- Remote access using NoMachine technology is possible. For additional
details click here.
|
