HPSS Mass StorageRelated InformationHPSS Software DownloadsHPSS Collaboration |
NERSC HPSS ChargingNERSC uses Storage Resource Units (SRUs) to help manage HPSS storage. The goal is to provide a balanced computing environment with appropriate amounts of storage and adequate bandwidth to keep the compute engines fed with data. Performance and usage tracking allows NERSC to anticipate demand and maintain a responsive storage environment. Storage management also recognizes storage as a distinct resource, in support of an increasing amount of data intensive computing. Finally, storage management and the quota system are intended to encourage efficient usage by the user community. SRUs are reported and managed via the NERSC Information Management (NIM) system. If a user is out of Storage Resource Units in all their HPSS repositories that user will be restricted so that they can no longer write data to HPSS (although they will continue to be able to read data). See: What happens if a repo or user SRU balance is negative?. Users can check their HPSS SRU balances by logging into the NERSC Information Management System and looking at the resource "HPSS" in your account usage summary. A SRU Calculator is available for estimating SRU usage.
Calculating a User's Storage Resource UnitsThree measures of use are included in computing SRUs:
The formula used to compute the number of SRUs incurred by a user each day is: daily user SRUs = 0.0000393 x files + 0.0131147 x space (GB) + 4.0 x I/O (GB) Where 1GB is 10243 bytes. Yearly usage is the sum of daily usage; the yearly formula is: yearly user SRUs = 0.01436 x Avg files + 4.787 x Avg space (GB) + 4.0 x I/O (GB) For an explanation on how the formula was derived see SRU Formula Coefficients. Apportioning User SRUs to Repositories: Project PercentsDOE's Office of Science awards Storage Resource Units to each NERSC project every year. The SRUs are deposited into the project's HPSS group account; this group account is called the HPSS repository (or repo). Users charge their HPSS SRU usage to the HPSS repos of which they are members. If a login name belongs to only one HPSS repo all of its usage is charged to that repo. If a login name belongs to multiple repos its daily charge is apportioned among the repos using the project percents for that login name. Default project percents are assigned based on the size of each repo's storage allocation. The user (only the user, not the project managers) can change her or his project percents by selecting Change SRU Proj Pct from the Actions pull-down list in NIM's Main Menu. Users should try to set project percents to reflect their actual use of HPSS for each of the projects of which they are a member. View "Change Project Percentages"Note that this is quite different from the way that computational resources are charged.
If a user changes her or his project percents this change will apply to all days in the month the change is made, but not to days prior to the month in which the change is made. If a login name is added to a new repo or removed from an existing repoIf a login name is added to a new repo or removed from an existing repo the project percents for that user are adjusted based on the size of the SRU allocations of the repos the login name currently belongs to. However, if the user has previously changed the default project percents the relative ratios of these previously set project percents are respected. For example: say that user u1 belongs to repos r1 and r2 and has changed the default project percents from 50% for each repo to 40% for r1 and to 60% for r2: Login Repo Repo Allocation Proj% u1 r1 50,000 SRUs 40 u1 r2 50,000 SRUs 60 Now assume that u1 becomes a new member of repo r3 which has a storage allocation of 100,000 SRUs. The project percents will be adjusted as follows (to preserve the old ratio of 40:60 between r1 and r2 while adding r3 which has the same SRU allocation as r1+r2): Login Repo Repo Allocation Proj% u1 r1 50,000 SRUs 20 u1 r2 50,000 SRUs 30 u1 r3 100,000 SRUs 50 If SRUs are added to or taken from an HPSS repoIf SRUs are added to or taken from an HPSS repo the project percents for the users in that repo are adjusted as needed to reflect the new sizes of each repo's storage allocation unless the user has changed the project percents from their default values (in this case the project percents are not changed). For example: say that user u2 belongs to repos r1 and r2 and has not changed the default project percents. Repo r2 gets a new infusion of SRUS: Login Repo Old Repo Alloc Old Proj% New Repo Alloc New Proj% u2 r1 50,000 SRUs 50 50,000 SRUs 25 u2 r2 50,000 SRUs 50 150,000 SRUs 75 User Quotas or Allowed PercentsPrincipal Investigators, PI Proxies and Project Managers can assign Allowed Percents (or user quotas) to each user in their repo. These allowed percents have been operational for MPP and PVP repos for a long time, but have only recently been available for HPSS (with the integration of SRUS in NIM). The default Allowed Percent is 100% for each user; Project managers can change these as appropriate. A user's HPSS allowed and used percentages as well as SRU balances are shown in NIM's Account Usage display:
User Statuses for HPSSWithin NIM the term User Status is used to display two sorts of statuses:
What happens if a repo or user SRU balance is negative?Accounting information is sent from HPSS to NIM once daily (in the early morning, Pacific Time). At this time actions are taken if a repo or user SRU balance is negative. If a repo runs out of SRUs all login names associated with it are marked as restricted for that repository (see repository user statuses). Login names are "HPSS restricted" if all of the repos associated with this login name are restricted (see machine user statuses). HPSS restricted login names are able to read data from HPSS but cannot write any data to HPSS. Likewise, when a login name goes over its individual "allowed percent" in a given repo, that (login, repo) pair is marked as restricted. The login name is HPSS restricted only if the (login, repo) repository user status is restricted for each repo associated with this login name. HPSS repos that are negative continue to incur SRU charges every day for each member that has HPSS files or I/O activity. This is because there is a daily charge for files stored within HPSS and for I/O activity. Note that restricted users can still incur I/O charges by reading files. Also, project percents are not adjusted when a repo goes negative. See Calculating a User's SRUs and Apportioning User SRUs to Repos. Likewise, a user who has gone over her or his allowed percent in a given repo will continue to incur charges in that repo. Project percents are not automatically adjusted when a login name exceeds its allowed percent, although a Project Manger can ask the user to adjust them. SRU Usage ReportsThe following SRU Usage Reports are available in NIM:
|
Page last modified: Wed, 18 Jun 2008 00:28:24 GMT Page URL: http://www.nersc.gov/nusers/systems/hpss/hpss-charging.php Web contact: webmaster@nersc.gov Computing questions: consult@nersc.gov Privacy and Security Notice |