Matching Level of Work

Under the former OCSP surveys, BLS field economists used a set list of survey job descriptions, each of which summarized work in a specific occupation at a single GS grade level. In the NCS surveys, BLS field economists cannot use a set list of survey job descriptions because BLS uses a random sampling method and any non-Federal job can be selected in an establishment for leveling (i.e., grading). In addition, it is not feasible for BLS field economists to consult and use the entire GS position classification system to level survey jobs because it would simply take too long to gather all the information needed to level surveyed jobs. This would also place an undue burden on survey participants. Therefore, in its original NCS methodology, BLS adopted the primary standard of the GS Factor Evaluation System (FES) for use in leveling jobs that are selected randomly in the survey. The primary standard is a framework that guides OPM when developing detailed standards for occupations under the FES. However, when the FES was designed and tested in the 1970s, OPM's predecessor, the Civil Service Commission, found a high error rate when only the primary standard was used in leveling jobs. The Federal Salary Council and OPM staff concluded that tests of the NCS program methods revealed similar problems.

To improve grade leveling under the NCS program, OPM developed a simplified 4-factor grade leveling system with 20 job family guides. These guides were designed to provide occupational-specific leveling instructions for the BLS field economists. The four factors were derived and validated by combining the nine factors under the existing FES. The factors were validated against a wide variety of GS positions and proved to replicate current grade levels.

The 20 job family guides cover the complete spectrum of white-collar work found in the Government. BLS and OPM have completed work on the guides, and BLS is now using the guides in its ongoing surveys. It will take 5 years to fully implement the conversion to the new leveling system because of BLS' data collection cycle. See Appendix IV of the 2002 Pay Agent's report for a summary of the BLS data collection cycle. Appendix VI of the 2002 Pay Agent's report contains the 20 job family leveling guides.

Jobs above GS-15

Under the former OCSP program, the occupationally-specific survey job descriptions also included instructions for excluding non-Federal jobs that, if classified under the GS position classification system, would be graded above GS-15. For the NCS program, it was necessary to develop generic instructions for identifying white-collar jobs in the random surveys that would be graded above GS-15 if they existed in the Federal Government. BLS developed and tested the guidance with assistance from OPM. Appendix V of the 2002 Pay Agent's report explains the process for identifying these jobs in the NCS program.

Grading Supervisory Positions

The former OCSP survey job descriptions also included instructions on how to grade or whether to exclude non-Federal supervisory jobs. This presented another problem for the NCS program because the Government does not use the same FES approach to grade supervisory jobs. BLS' original NCS methodology included an experimental approach in which BLS first applied the FES to sampled supervisory positions and then added additional factor points for the level of supervision. OPM classifiers believed this experimental approach would not yield the correct grade level and suggested a new approach based on the highest level of work supervised. Under the new approach, BLS would grade the work supervised using the appropriate four-factor leveling guide, not the supervisory job itself, and then add one grade for a first-level supervisor, two grades for a second-level supervisor, and three grades for a third-level supervisor. BLS and OPM have completed work on developing this procedure, and BLS is now using the new procedures in its ongoing surveys. However, the data available for this report were not processed using the new approach.

As in 2002 and 2003, BLS excluded second- and third-level supervisors entirely from the NCS data this year. BLS graded first-level supervisors by using existing NCS grade leveling procedures. The Pay Agent issued these instructions to BLS because the grades of second- and third-level supervisors are more likely affected by their supervisory duties, while first-level supervisors are more likely graded based on other factors, such as technical expertise. This modification allowed us to use some of the data from supervisory positions.^[5]

Missing Data

While BLS surveys all white-collar jobs under the NCS program, it does not find all jobs at all work levels in each survey area. This is a serious problem with the NCS program and was also a serious problem with OCSP surveys because survey results and pay disparity measures can vary considerably based on which jobs are included. Pay Agent staff developed a model to estimate missing OCSP jobs, and the Pay Agent instructed BLS to develop an econometric model to provide estimates for jobs not found in NCS. The models are described later in this report and in Appendices II and III.

Differences in Results

In 2002, NCS pay gaps were about 4 points below those using the OCSP results, on average. In 2003, the results were about 2.6 points apart, and this year, the averages are within 1.71 points. However, the results vary significantly for a number of locality pay areas. As noted in 2002 and 2003, many factors could cause pay measures under the NCS program to be different from those under OCSP. OPM staff identified a number of possible reasons for this outcome, including the following:

• OCSP data are out of date, and the nationwide rate of change measures used to age the data (i.e., the Employment Cost Index) probably overestimate or underestimate pay on a locality or occupational basis.^[6]
• OCSP used a fixed job list that may have been biased toward higher-paying jobs.^[7]
• Certain key OCSP results are based on small samples and may overstate pay levels.^[8]
• The FSC and OPM staff believe test surveys indicated problems in assigning grades under the NCS program.
• NCS random samples may miss key high-paying jobs that are not common in non-Federal establishments, and modeled values may not fully compensate.
• Between 33 and 84 percent of the GS weighted data in the NCS program are modeled (71 percent modeled, on average). A review of the BLS model in 2002 indicated that it tends to underestimate pay for high-paying jobs. (The OCSP model also tended to underestimate pay for certain jobs.) We had actual survey data for about 70 percent of the OCSP jobs and modeled about 30 percent of that data, but OCSP survey jobs directly represented only about 30 percent of the Federal workforce, so actual data under OCSP represented only about 21 percent of the Federal workforce-a little less than the 29 percent of jobs represented by actual data under the NCS program.
• Job definitions under OCSP were written to match specific Federal jobs, while the SOC crosswalk used in the NCS program has some more generic matches.

We also note that the pay gaps measured with NCS data increased by more than 3 percentage points in three locality pay areas and changed by more than 2 points (up or down) in nine locality pay areas (including the three). These changes could reflect sizable increases in non-Federal pay over the last year or could be due to changes in Federal employment weights, but are most likely due to small samples, BLS sample rotation, or applying large Federal employment weights to small BLS samples. BLS replaced one-fifth of the establishments sampled in several of these areas. If sample rotation does affect the results to this magnitude, a significant increase in survey sample size may be desirable to achieve more stable results.