Methods and Models Using Facility Data

The third step in the process involved analyses of the data compiled previously to test different methods for which pertinent data currently exist. Part of this process involved experimentation with different equations and computational methods to determine which specific formulas are most appropriate for each of the four types of facilities. These activities revealed a number of interesting and important insights about nursing shortages, which are summarized below.

A. Preliminary Analyses

Figure 8 presents the distribution of the indicator of difficulty recruiting RNs based on all facilities in North Carolina. The figure shows the number of facilities that experienced difficulty recruiting RNs (indicator >3) was more than double the number of the facilities with no difficulty recruiting RNs (indicator <3). In this case, 68 facilities (20.9%) reported not having difficulty recruiting RNs compared to 155 facilities (47.7%) that reported having difficulty recruiting RNs. The figure also shows that only 17 facilities (5.2%) reported that it was very easy to recruit RNs, in contrast to 56 facilities (17.2%) that reported it was very difficult to recruit RNs.

Figure 8. Distribution of RN Recruitment Difficulty Indicator, Based on Four Types of Health Facilities in North Carolina in 2004

[D]

Figure 9 presents the distribution of difficulty indicator by facility type. From this figure we can see that the distributions of difficulty to recruit RNs were different among all four types of facilities. For example, 4.6% of hospitals reported it was very difficult to recruit RNs, in contrast to 26.4% of public health facilities reported very difficult to recruit RNs.

Figure 9. Nursing Recruitment Difficulty Indicators in North Carolina, by Facility Type, 2004

[D]

Figure 10 compares the distributions of the predicted recruiting difficulty scores for the four types of facilities in North Carolina, based on the Ordered Probit model estimated using data for 2004. The figure shows clearly that the variation in recruiting difficulty is greatest for public health agencies and least for hospitals. It also shows that on average both public health agencies and long-term care facilities have statistically significantly greater difficulty recruiting RNs than hospitals (p≤0.05, since the 95% confidence intervals do not overlap).

Figure 10. Distribution of Predicted Difficulty Recruiting RNs in North Carolina by Type of Facility, 2004

[D]

Table 2 presents the distribution of facility type by difficulty indicator and Chi-Square statistic to test the null hypothesis that there is no association between type of facility and the difficulty recruiting RNs. Based on the Chi-square statistic, the null hypothesis was rejected (p = 0.011) because different types of facilities had different levels of difficulty recruiting RNs. The implication was that different types of facilities have different behaviors in term of modeling nursing shortages.

Chi-Square = 25.9 (df = 12)
Test of H₀: No association between type of facility and difficulty to recruit
H₀ is rejected with p-value = 0.011

Table 3 presents the distribution of difficulty indicator by number of adverse consequences of shortages and the Spearman correlation coefficient to test the null hypothesis that there is no relationship between difficulty indicator and number of consequences. From the Spearman correlation statistic, the null hypothesis was rejected (p<0.0005), meaning that on average facilities that experienced greater difficulty recruiting RNs had more bad consequences.

 Spearman correlation coefficient = 0.343
 Test of H₀: Correlation = 0
 H₀ is rejected with p-value < 0.0005

A number of models were estimated for hospitals in North Carolina. The steps followed are summarized below.

The indicator of nursing shortage used as a dependent variable was the number of reported negative effects on operations revealed by a facility. Most facilities indicated no effects or only one effect. The mean value for all facilities was 0.89, with a standard deviation of 1.07. Based on this, we defined facilities as being needy (for test purposes only), if they presented two or more effects on operations. Under this definition, 15.5% of hospitals were needy.

The population was adjusted by gender and age based on average use of primary care. This weighted older adults and infants more heavily than younger people and weighted women more heavily than men. The resulting variable was an estimate of how many primary care visits the population would require in a year’s time. Although the relationship between use of primary care and need for services, such as home health or long-term care, is open to debate, this variable was simply a way of standardizing the population based on characteristics known to affect medical need.

The following variables were selected for use in the North Carolina analyses:

1. Active RNs Employed in the County per 100,000 Adjusted Population
2. Students Enrolled in RN Programs in the County per 100,000 Adjusted Population
3. Number of Short-Term General Hospitals
4. Number of Short-Term General Hospital Beds
5. Ratio of Average RN Salary to Median Income
6. Number of Nursing and Personal Care Facilities
7. Percent of Population with Income Below Poverty Level
8. Population per Square Mile
9. Ratio of RNs to Hospital Beds
10. Number of Hours per Week Paid for Agency RNs
11. Number of Overtime RN Hours per Week
12. RN Vacancy Rate
13. RN Turnover Rate
14. Ratio of LPNs to RNs
15. Total Number of Budgeted RN Positions
16. Percent Non-Hispanic White

Average values for these variables are shown in Table 4 for three groups of hospitals in North Carolina.

Population per square mile was very highly correlated with several other variables, so a natural log transformation was applied to reduce problems of multicollinearity. There was also potential multicollinearity between the number of RNs per 100,000 adjusted population and number of general hospital beds per 100,000 adjusted population. Number of hospital beds was dropped in favor of number of hospitals.

4. Run OLS Regression Model, Full and Abbreviated

Two different OLS models were estimated to predict the number of adverse effects in hospitals in North Carolina, one for the full model that included both community and facility data and one that included only community data. These models are summarized below.

Full model

Table 5. Coefficients for Full OLS Regression Model to Predict Number of Adverse Effects of Nursing Shortages in Hospitals in North Carolina

Explanatory (Independent) Variable	Unstandardized Coefficients		Standardized Coefficients	t	p Value
	B	Std Err
Constant	-0.683	3.020	-	-0.226	0.822
RNs per 100,000 Adjusted Need	-0.0035	0.002	-0.353	-2.002	0.052
RN Salary to Average Salary	0.518	0.707	0.132	0.732	0.468
# Nursing and Personal Care Facilities	0.032	0.015	0.663	2.176	0.035
% Population Below Poverty, 2000	0.078	0.065	0.308	1.202	0.236
RNs per Hospital Bed	0.265	0.445	0.082	0.596	0.555
Hours of Agency RNs	0.0025	0.043	0.008	0.058	0.954
Hours of RN Overtime	-0.0008	0.016	-0.007	-0.052	0.959
RN Vacancy Rate	0.032	0.032	0.142	0.985	0.330
RN Turnover Rate	0.011	0.021	0.077	0.505	0.616
Persons per Square Mile (natural ln)	0.156	0.358	0.146	0.436	0.665
# Short-term Community Hospitals, ‘01	-0.359	0.134	-0.610	-2.690	0.010
RN Students per 100K Adjusted Need	-0.010	0.004	-0.392	-2.828	0.007
% Population Non-Hispanic White, 2004	-0.011	0.012	-0.167	-0.902	0.372

Dependent Variable: NUM_CONS
Selecting only cases for which FAC_TYPE = hospital
R² = 0.429
Abbreviated model

Because most of the variables that appeared most critical were community variables rather than facility variables, an abbreviated model was also run using only community information. Due to the constraints of data availability, the abbreviated model is one that can be used more easily in practice. The R², however, dropped substantially, from 0.429 in the full model to only 0.177 in the abbreviated model.

Table 6. Coefficients for Abbreviated OLS Regression Model to Predict Number of Adverse Effects of Nursing Shortages in Hospitals in North Carolina

Independent Variables	Unstandardized Coefficients		Standardized Coefficients	t	p Value
	Coefficient	Std Err
Constant	1.295	2.374	-	0.546	0.587
RNs per 100,000 Adjusted Need	-0.0012	0.001	-0.132	-0.880	0.382
RN Salary to Average Salary	0.281	0.582	0.081	0.482	0.631
# Nursing/Personal Care Facilities	0.023	0.012	0.494	1.905	0.061
% Population Below Poverty, 2000	0.033	0.053	0.136	0.622	0.536
RNs per Hospital Bed	0.044	0.402	0.013	0.108	0.914
Persons per Square Mile (natural ln)	-0.158	0.271	-0.159	-0.582	0.563
# Short-term Community Hospitals, ‘01	-0.227	0.109	-0.414	-2.076	0.041
RN Students per 100K Adjusted Need	-0.0054	0.003	-0.226	-1.967	0.053
% Population White Non-Hispanic, ‘04	-0.0053	0.010	-0.086	-0.511	0.611

Dependent Variable: NUM_CONS
Selecting only cases for which FAC_TYPE = hospital
R² = 0.177

Coefficients from the full and abbreviated regression models were used to estimate predicted number of problems in each facility. The top 16% of facilities in regard to predicted number of problems were considered to have made the test “cut” of 15.5% chosen arbitrarily based on earlier analysis (see Step 1). The facilities selected by the full model and the abbreviated model were compared to the facilities whose actual problem scores were in the top 15.5%.

Using the abbreviated model, 84% of hospitals were classified correctly based on the arbitrary value chosen earlier. Eight percent of facilities were misclassified as not needy by the abbreviated model when their actual scores qualified them as needy, while 7% were misclassified as being needy when their actual scores did not qualify them as such.

Using all the information in the full model would have increased the accuracy of prediction to 89%, with 5% of facilities erroneously classified as needy and 5% erroneously classified as not needy.

Using the information from the testing in Step 5, we conclude that using an abbreviated model with widely available community level data to assign facilities need scores would result in approximately 84% of facilities being correctly classified. Supplementing this with an appeals process requiring the additional information needed for the full model would correctly classify an additional 5% of facilities.

C. Empirical Models for North Dakota Hospitals

The coefficients estimated for North Carolina hospitals were applied to hospitals in North Dakota. The results are summarized below.

When the coefficients for the abbreviated model obtained from the empirical models developed for North Carolina were applied to hospitals in North Dakota, not surprisingly the classifications were less accurate. Seventy-nine percent of North Dakota hospitals were correctly classified by this application of North Carolina data, while 10% were erroneously classified as needy and 10% were erroneously classified as not needy.

This analysis suggests that using coefficients based on models estimated in one state achieves lower accuracy when applied to facilities in another state. Additional research would be required to determine whether the decline in accuracy might be related to the extent to which general characteristics of the states are similar or different.

D. Empirical Models for North Carolina Nursing Homes

The empirical models for nursing homes in North Carolina are summarized below.

The indicator of nursing shortage used as a dependent variable was the number of reported effects on operations reported by a facility. Most facilities reported no effects or only one effect. The mean value for all facilities was 1.0, with a standard deviation of 1.1. Based on this, we defined facilities as being needy (for test purposes only) if they reported two or more effects on operations. Under this definition, 31.3% of nursing homes were needy.

The population was adjusted by gender and age based on average use of primary care. This weighted older adults and infants more heavily than younger people and women more heavily than men. The resulting variable was an estimate of how many primary care visits the population would require in a year’s time. Although the relationship between use of primary care and need for services such as home health or long-term care is open to debate, this variable was simply a way of standardizing the population based on characteristics known to affect medical need.

1. Active RNs employed in the county per 100,000 adjusted population
2. Students enrolled in RN programs in the county per 100,000 adjusted population
3. Number of short-term general hospitals
4. Number of short-term general hospital beds
5. Ratio of average RN salary to median income
6. Number of nursing and personal care facilities
7. Percent of the population with income below poverty level
8. Population per square mile
9. Ratio of RNs to hospital beds
10. Number of hours per week paid for agency RNs
11. Number of overtime RN hours per week
12. RN vacancy rate
13. RN turnover rate
14. Ratio of LPNs to RNs
15. Total number of budgeted RN positions
16. Percent non-Hispanic white

Table 7. Means and Standard Deviations of Selected Independent Variables Related to Nursing Shortages in North Carolina Nursing Homes

Independent Variables	All Nursing Homes		Nursing Homes Reporting No Nurse Staffing Problems		Nursing Homes Reporting Two or More Nurse Staffing Problems
	Mean	S.D.	Mean	S.D.	Mean	S.D.
Active RNs Employed in County per 100K Medical Need	189.6	101.1	204.9	111.2	207.3	87.5
Students in RN Programs per 100K Medical Need	36.0	139.9	29.6	65.8	18.1	30.8
Number of Short-Term Community Hospitals	1.7	1.6	2.1	2.0	1.5	1.2
Number of Short-Term Community Hospital Beds	597.9	689.3	740.5	780.3	704.5	654.9
Ratio of average RN salary to median income	1.5	0.3	1.5	0.3	1.4	0.3
Number of Nursing and Personal Care Facilities	18.4	19.6	22.9	23.0	22.3	18.8
Percent of Population w/ Income Below Poverty Level	13.0	4.1	12.9	4.1	12.5	3.9
Population per Square Mile	300.0	315.0	357.4	373.2	351.6	278.9
Ratio of RNs to Hospital Beds	0.5	0.3	0.4	0.2	0.5	0.2
Hours per Week Paid for Agency RNs	2.1	5.9	2.5	7.9	3.8	9.4
Number of Overtime RN Hours per Week	6.5	9.7	12.4	11.1	14.1	13.3
RN Vacancy Rate	9.5	13.6	8.5	12.2	9.7	13.1
RN Turnover Rate	29.6	43.6	40.7	69.3	38.8	32.8
Ratio of LPNs to RNs	1.3	2.2	2.2	1.5	3.1	4.3
Total Number of Budgeted RN Positions	79.1	240.8	7.2	4.6	7.0	4.9
Percent Non-Hispanic White	70.9	16.4	70.4	17.5	70.2	15.1

Population per square mile was very highly correlated with several other variables, and so a log transformation was applied to avoid problems with multicollinearity. There was also potential multicollinearity between the number of RNs per 100,000 adjusted population, and number of general hospital beds per 100,000 adjusted population. Number of hospital beds was dropped in favor of number of hospitals.

The following regression was run for nursing homes in North Carolina:

Table 8. Coefficients for OLS Regression Model to Predict Number of Adverse Effects of Nursing Shortages in Nursing Homes in North Carolina

Independent Variable	Unstandardized Coefficients		Standardized Coefficients	t	p Value
	B	Std Err	Beta
(Constant)	-2.395	2.872	-	-0.834	0.407
RNs per 100,000 Adjusted Need	-0.0007	0.002	-0.063	-0.379	0.706
RN Salary to Average Salary	-1.307	0.779	-0.338	-1.677	0.098
# Nursing/Personal Care Facilities	-0.00338	0.012	-0.060	-0.271	0.787
% Population Below Poverty, 2000	0.114	0.067	0.393	1.690	0.095
RNs per Hospital Bed	0.585	0.637	0.110	0.919	0.361
Hours of Agency RNs	0.0051	0.016	0.040	0.314	0.754
Hours of RN Overtime	0.0073	0.012	0.070	0.599	0.551
RN Vacancy Rate	-0.0014	0.011	-0.014	-0.131	0.896
RN Turnover Rate	0.0002	0.002	0.012	0.095	0.925
Persons per Square Mile (Natural ln)	0.632	0.330	0.494	1.914	0.059
# Short-Term Commun Hospitals, ‘01	-0.344	0.119	-0.485	-2.881	0.005
RN Students per 100,000 Adjusted Need	0.0010	0.003	0.047	0.385	0.701
% Population White Non-Hispanic, 2004	0.014	0.012	0.199	1.160	0.250

Dependent Variable: NUM_CONS
Selecting only cases for which FAC_TYPE = long-term care
R² = 0.20

This model had little predictive value, perhaps because the chosen dependent measure of nursing shortage was inappropriate for nursing homes, which rely heavily on LPNs. The question about the effects of a nursing shortage on facility operations did not specify RN shortages, and so it seemed plausible that significant relationships were not emerging based on RN variables because respondents answered this question primarily thinking of LPNs.

Therefore, in estimating this model, the decision was made to revert to RN vacancy rates, acknowledging that the facilities reporting the highest vacancy rates are not necessarily the facilities suffering the most from the RN shortage. Several variables relating to the LPN job market were also included in this second version of the model. The mean RN vacancy rate for nursing homes was 10.6, with a standard deviation of 15.8. On this basis, we classified any facility with a RN vacancy rate of more than 26.4 as “needy” as a test value (11.9% of facilities).

An alternate OLS regression model was estimated for RN Vacancy Rates in nursing homes in North Carolina (Table 9). It focused more on LPNs and less on RNs, which better reflects the actual staffing patterns at nursing homes.

Dependent Variable: RNVacRate
Selecting only cases for which FAC_TYPE = long-term care
R² = 0.35

Coefficients from the regression model were used to estimate predicted number of problems in each facility. The top 31.5% of facilities in regard to predicted number of problems were considered to have made the test “cut” of 31.3% chosen arbitrarily based on earlier analysis (see Step 1). The facilities selected by the full model were compared to the facilities whose actual problem scores were in the top 31.3%.

Using the full model, only 73% of nursing homes were classified correctly based on the arbitrary value chosen earlier. Fourteen percent of facilities were misclassified as not needy by the model when their actual scores qualified them as needy, while 12% were misclassified as being needy when their actual scores did not qualify them as such.

The alternate model, however, proved very effective in identifying facilities with the highest RN vacancy rates. Eighty-eight percent of facilities were correctly classified as “needy” based on the arbitrary value chosen earlier. Seven percent were misclassified as not needy by the model when their actual scores qualified them as needy, while 6% were misclassified as being needy when their actual scores did not qualify them as such.

Although there are several reliable indicators of high RN vacancy rates in nursing homes, there is little that predicts need in terms of the problems facilities report in their operations as a result of the nursing shortage. This is problematic because the facilities reporting the highest vacancy rates are not necessarily the facilities suffering the most from nursing shortages. Indeed, RN vacancy rates were unrelated to reports of shortage problems. The facilities the majority of facilities defined as needy on the basis of reported problems were not the same facilities defined as needy on the basis of RN vacancy rates. This may be due to the prominence of LPNs in long-term care, however, causing most people to answer the question about problems based on LPN shortages rather than RN shortages. Given this ambiguity, RN vacancy rates may be the better indicator of long-term care shortages.

Another shortcoming of the analyses is that population is standardized based on primary care utilization rates estimated by age and gender. This formula may be inappropriate for estimating long-term care need in the population, and perhaps a new formula for standardization based on long-term care utilization rates should be introduced. A standardization of the population that is tailored to long-term care might produce more useful models and more reliable estimates of community need. Number of long-term care beds and beds per older adults would also be useful information to include in future attempts to model.

E. Tailoring for Long-Term Care

As stated in the Conclusion section of Part I, the initial analyses were based on a general model tested for four types of facilities: hospitals, home health agencies, public health agencies, and long-term care facilities. For the former three types of facilities, indicators of general medical need and availability of general medical services may be relevant indicators in judging adequacy of the RN supply. Long-term care, however, is a more specific type of care provided to a narrower segment of the population.

When the alternate model obtained in Part I of the pilot testing was applied to nursing homes in North Dakota , the classifications were considerably less accurate. This is the same result as observed for hospitals.

As was the case for hospitals, this analysis suggests that using coefficients based on long-term care models estimated in one state achieves lower accuracy when applied to facilities in another state. Additional research would be required to determine whether the decline in accuracy might be related to the extent to which general characteristics of the states are similar or different.

F. Empirical Models for North Carolina Home Health Agencies

The steps used to estimate the empirical models for home health agencies in North Carolina are summarized below.

The indicator of nursing shortage used as a dependent variable was the number of reported effects on operations reported by an agency. Most agencies reported no effects or only one effect. The mean value for all agencies was 0.8, with a standard deviation of 1.0. Based on this, we defined agencies as being needy (for test purposes only) if they reported two or more effects on operations. Under this definition, 19.4% of home health agencies were needy.

The population was adjusted by gender and age based on average use of primary care. This weighted older adults and infants more heavily than younger people and weighted women more heavily than men. The resulting variable was an estimate of how many primary care visits the population would require in a year’s time. Although the relationship between use of primary care and need for services such as home health or long-term care is open to debate, this variable was simply a way of standardizing the population based on characteristics known to affect medical need.

1. Active RNs employed in the county per 100,000 adjusted population
2. Students enrolled in RN programs in the county per 100,000 adjusted population
3. Number of short-term general hospitals
4. Number of short-term general hospital beds
5. Ratio of average RN salary to median income
6. Number of nursing and personal care facilities
7. Percent of the population with income below poverty level
8. Population per square mile
9. Ratio of RNs to hospital beds
10. Number of hours per week paid for agency RNs
11. Number of overtime RN hours per week
12. RN vacancy rate
13. RN turnover rate
14. Ratio of LPNs to RNs
15. Total number of budgeted RN positions
16. Percent non-Hispanic white

Table 10. Means and Standard Deviations of Selected Independent Variables Related to Nursing Shortages in North Carolina Home Health Agencies

Independent Variable	All Home Health Agencies		Agencies Reporting No Nurse Staffing Problems		Agencies Reporting Two or More Nurse Staffing Problems
	Mean	S.D.	Mean	S.D.	Mean	S.D.
Active RNs in County per 100K Medical Need	184.6	98.0	187.2	102.4	187.5	95.4
Students in RN Programs per 100K Medical Need	48.7	195.8	54.6	230.8	39.5	75.6
Number of Short-Term Community Hospitals	1.7	1.4	1.7	1.4	1.8	1.7
Number of Short-Term Community Hospital Beds	555.8	655.4	590.2	693.9	617.2	679.9
Ratio of Average RN Salary to Median Income	1.5	1.4	1.5	0.3	1.5	0.3
Number of Nursing and Personal Care Facilities	16.8	17.3	17.3	17.8	20.1	20.4
Percent of Population w/ Income Below Poverty Level	12.6	4.1	12.6	4.0	13.5	4.8
Population per Square Mile	287.6	283.5	292.3	287.8	312.6	331.8
Ratio of RNs to Hospital Beds	0.5	0.3	0.4	0.3	0.5	0.2
Number of Hours per Week Paid for Agency RNs	1.7	4.9	1.2	3.4	4.7	9.2
Number of Overtime RN Hours per Week	2.7	4.3	2.3	3.7	5.0	6.6
RN Vacancy Rate	10.1	15.9	7.6	14.1	21.5	21.2
RN Turnover Rate	28.3	37.4	19.1	24.9	60.4	52.9
Ratio of LPNs to RNs	0.3	0.3	0.3	0.3	0.3	0.2
Total Number of Budgeted RN Positions	12.0	10.5	12.5	11.5	10.7	7.9
Percent Non-Hispanic White	72.7	16.4	73.5	15.8	63.2	19.0

Population per square mile was very highly correlated with several other variables, and so a log transformation was applied to avoid problems with multicollinearity. There was also potential multicollinearity between the number of RNs per 100,000 adjusted population, and number of general hospital beds per 100,000 adjusted population. Number of hospital beds was dropped in favor of number of hospitals.

The following regression was run for home health agencies in North Carolinas:

Table 11. Coefficients for OLS Regression Model to Predict Number of Adverse Effects of Nursing Shortages in Home Health Agencies in NC

Independent Variable	Unstandardized Coefficients		Standardized Coefficients	t	p Value
	B	Std Err	Beta
(Constant)	2.270	2.216	-	1.024	0.310
RNs per 100,000 Adjusted Need	0.0022	0.002	0.214	1.412	0.163
RN salary to Average Salary	1.570	0.607	0.480	2.587	0.012
# Nursing/Personal Care Facilities	0.014	0.013	0.255	1.137	0.260
% Population Below Poverty, 2000	-0.118	0.052	-0.519	-2.266	0.027
RNs per Hospital Bed	-0.200	0.337	-0.062	-0.594	0.555
Hours of Agency RNs	0.046	0.022	0.232	2.069	0.043
Hours of RN overtime	-0.011	0.030	-0.041	-0.369	0.713
RN Vacancy Rate	0.024	0.008	0.374	3.078	0.003
RN Turnover Rate	0.0069	0.003	0.265	2.339	0.023
Persons per Square Mile (natural log)	-0.436	0.290	-0.392	-1.502	0.139
# Short-Term Community Hospitals, ‘01	-0.020	0.116	-0.027	-0.170	0.865
RN Students per 100K Adjusted Need	-0.00088	0.001	-0.202	-1.605	0.114
% Population White Non-Hispanic, 2004	-0.0136	0.010	-0.230	-1.340	0.185

Dependent Variable: NUM_CONS
Selecting only cases for which FAC_TYPE = home health
R² = 0.44

An abbreviated model was also estimated. It appeared to have little value for home health agencies because most of the variables that appeared most critical were facility variables rather than community variables, and would have to be collected directly from facilities. Variables that were “optional,” and were able to be dropped for an abbreviated model were the variables most widely available.

Coefficients from the full regression model were used to estimate predicted number of problems in each agency. The top 19.2% of agencies in regard to predicted number of problems were considered to have made the test “cut” of 19.4% chosen arbitrarily based on earlier analysis (see Step 1). The agencies selected by the full model were compared to the agencies whose actual problem scores were in the top 19.4%.

Using the full model, 85% of home health agencies were classified correctly based on the arbitrary value chosen earlier. Seven percent of agencies were misclassified as not needy by the model when their actual scores qualified them as needy, while 8% were misclassified as being needy when their actual scores did not qualify them as such.

Using the information from the testing in Step 5, we can conclude that using the full model with both widely available community level data and data collected directly from agencies to assign need scores would result in approximately 85% of agencies being correctly classified. The importance of the facility-level variables in the model, however, means that any effective strategy for classifying home health agencies will require the collection of data on factors such as turnover and vacancy rates.

As with long-term care facilities, however, there was an issue in using a model designed to incorporate measures of general medical need. Home health is not primary care, and patients tend to be predominantly older while both the oldest and the youngest segments of the population disproportionately consume primary care. A standardization of the population that is tailored to long-term care utilization might produce more useful models and more reliable estimates of community need. While reliable community-level data on home health capacity will not be obtainable, number of long-term care beds and beds per older adult might also be useful information to include in future attempts to model, both because long-term care serves similar populations to home health, and because long-term care and home health may compete for the same pool of RNs. Incorporation of such variables may make community-level indicators more useful in evaluating home health shortages, possible enabling the construction of a reliable abbreviated model as was done for hospitals.

G. Empirical Models for North Dakota Home Health Agencies

The coefficients estimated for North Carolina home health agencies were applied to home health agencies in North Dakota. The results are summarized below.

When the model obtained in Part I of the pilot testing was applied to home health agencies in North Dakota , the classifications were considerably less accurate. This is the same result as observed for hospitals and long-term care facilities.

As was the case for hospitals, this analysis suggests that using coefficients based on home health agencies models estimated in one state achieves lower accuracy when applied to facilities in another state. Additional research would be required to determine whether the decline in accuracy might be related to the extent to which general characteristics of the states are similar or different.

H. Empirical Models for North Carolina Public Health Agencies

The steps used to estimate the empirical models for public health agencies in North Carolina are summarized below.

The indicator of nursing shortage used as a dependent variable was the number of reported effects on operations reported by an agency. Most agencies reported no effects or only one effect. The mean value for all agencies was 1.09, with a standard deviation of 1.03. Based on this, we defined agencies as being needy (for test purposes only) if they reported two or more effects on operations, or more than one standard deviation above the mean. Under this definition, 26.5% of public health agencies were needy.

The population was adjusted by gender and age based on average use of primary care. This weighted older adults and infants more heavily than younger people and weighted women more heavily than men. The resulting variable was an estimate of how many primary care visits the population would require in a year’s time. Although the relationship between use of primary care and need for services such as home health or long-term care is open to debate, this variable was simply a way of standardizing the population based on characteristics known to affect medical need.

1. Active RNs employed in the county per 100,000 adjusted population
2. Students enrolled in RN programs in the county per 100,000 adjusted population
3. Number of short-term general hospitals
4. Number of short-term general hospital beds
5. Ratio of average RN salary to median income
6. Number of nursing and personal care facilities
7. Percent of the population with income below poverty level
8. Population per square mile
9. Ratio of RNs to hospital beds
10. Number of hours per week paid for agency RNs
11. Number of overtime RN hours per week
12. RN vacancy rate
13. RN turnover rate
14. Ratio of LPNs to RNs
15. Total number of budgeted RN positions
16. Percent non-Hispanic white

Table 12. Means and Standard Deviations of Selected Independent Variables Related to Nursing Shortages in North Carolina Public Health Agencies

Independent Variables	All Public Health Agencies		Agencies Reporting No Nurse Staffing Problems		Agencies Reporting Two or More Nurse Staffing Problems
	Mean	S.D.	Mean	S.D.	Mean	S.D.
Active RNs in County per 100K Medical Need	155.6	92.2	148.5	82.5	162.6	109.6
Students in RN Programs per 100K Medical Need	61.9	220.1	90.9	277.8	16.8	26.24
Number of Short-Term Community Hospitals	1.2	1.1	1.2	1.2	1.1	0.6
Number of Short-Term Community Hospital Beds	343.1	488.3	242.2	378.8	380.0	386.1
Ratio of Average RN Salary to Median Income	1.6	0.3	1.6	0.3	1.6	0.3
Number of Nursing and Personal Care Facilities	10.9	13.3	8.8	11.3	12.3	10.4
Percent of Population w/ Income Below Poverty	14.0	4.1	14.1	3.9	14.8	4.5
Population per Square Mile	184.2	218.2	158.8	210.8	200.3	198.3
Ratio of RNs to Hospital Beds	0.5	0.4	0.4	0.3	0.5	0.5
Number of Hours per Week Paid for Agency RNs	0.9	2.9	0.9	3.2	1.7	3.4
Number of Overtime RN Hours per Week	0.8	2.4	0.7	2.2	1.6	3.4
RN Vacancy Rate	9.0	11.6	8.6	12.2	10.2	10.2
RN Turnover Rate	15.5	18.7	16.9	22.3	15.2	11.4
Ratio of LPNs to RNs	0.1	0.2	0.1	0.2	0.1	0.1
Total Number of Budgeted RN Positions	26.6	27.6	22.9	26.4	26.4	17.1
Percent Non-Hispanic White	71.1	17.1	73.3	63.9	63.9	16.8

Population per square mile was very highly correlated with several other variables, and so a log transformation was applied to avoid problems with multicollinearity. There was also potential multicollinearity between the number of RNs per 100,000 adjusted population, and number of general hospital beds per 100,000 adjusted population. Number of hospital beds was dropped in favor of number of hospitals.

The following regression was run for public health agencies.

Dependent Variable: NUM_CONS
Selecting only cases for which FAC_TYPE = public health
R² of 0.34

Because most of the variables that appeared most critical were community variables rather than facility variables, an abbreviated model was also run using only community information. Due to general constraints of data availability, the abbreviated model is one that can be used more realistically in practice. The following regression was run for public health agencies, with an R² of 0.30, which is only slightly smaller than the R² for the full model.

Dependent Variable: NUM_CONS
Selecting only cases for which FAC_TYPE = public health
R² of 0.30

Coefficients from the full regression model were used to estimate predicted numbers of problems in each agency. The top 27.2% of agencies in regard to predicted numbers of problems were considered to have made the test cut of 26.5%, chosen arbitrarily, based on earlier analysis (see Step 1). The agencies selected by the full model were compared to the agencies whose actual problem scores were in the top 26.5%.

Using the full model, 25% of public health agencies were not classified correctly based on the arbitrary value chosen earlier. About 14% of agencies were misclassified as not needy by the model when their actual scores qualified them as needy, while about 12% were misclassified as being needy when their actual scores did not qualify them as such.

The full model provided relatively poor predictive value, suggesting that an abbreviated version of the full model was not worth pursuing for public health agencies.

Although there are significant predictors of problems related to nursing shortages in public health agencies, the full regression model has a high degree of error in predicting which agencies report the greatest problems. This model does not seem effective to estimate RN shortages in public health agencies. More information may be needed to assess the roles of RNs in public health and the consequences of inabilities to fill RN positions.

I. Empirical Models for North Dakota Public Health Agencies

The coefficients estimated for North Carolina public health agencies were applied to public health agencies in North Dakota. The results are summarized below.

When the abbreviated model obtained in Part I of the pilot testing was applied to public health agencies in North Dakota, the classifications were considerably less accurate. This was the same result observed for hospitals, long-term care facilities, and home health agencies.

As was the case for hospitals, long-term care facilities, and home health agencies, this analysis suggested that using coefficients based on long-term care models estimated in one state achieves lower accuracy when applied to facilities in another state. Additional research would be required to determine whether the decline in accuracy might be related to the extent to which general characteristics of the states are similar or different.

J. Ordered Probit Models for North Carolina

Although it is possible (as demonstrated in the analyses in the previous section) to use OLS regression to estimate the relationships between a set of independent explanatory variables and an ordinal dependent variable like “difficulty recruiting RNs,” the fact that the dependent variable was ordinal and not Gaussian violates one of the underlying assumptions of OLS regression. One way to address this violation is to use an alternate regression technique, ordered probit analysis. This technique is similar in concept to OLS regression, but uses very different computational procedures. Most important, however, it is designed to work effectively with ordinal dependent variables.

Two different ordered probit models were developed to identify the factors related to difficulty recruiting RNs in the four types of facilities in North Carolina. The first analyzes all four types of facilities simultaneously. The second analyzes the four different types of facilities separately; i.e., hospitals, home health facilities, long-term care facilities, and public health facilities. Both models included variables that represent community characteristics and facility characteristics. The community variables were divided into three groups – demographic, economic, and nursing variables. For each type of facility, the variables included in the model were based on p-values. The lower the p-value of a variable, the stronger the influence the variable had on the nurse recruiting. In other words, lower p-values meant better prediction of difficulty recruiting RNs; therefore variables with lower p-value were included in the model. If p-value was lower than 0.10 then the variable was statistically significant in explaining the shortage at the 10% level of significance.

In this technique, dummy variables for the types of facilities reflect the effects of facility type. By creating interaction variables (which are the products of the dummy variables with other independent variables), this technique provides coefficient estimates for all four types of facilities (hospitals, home health care, long-term care, and public health). The coefficient for an independent variable for one type of facility may be different from the coefficients for the other facility types. In addition, an independent variable may be statistically significant in explaining the recruiting difficulty for one type of facility, but not for another.

The advantage of estimating the model based on all facilities together was that the predicted recruiting difficulty scores were comparable not only within the same type of facility, but also across facilities of different types. The variables included in the model are shown in Table 15. Each variable in the table was statistically significant for at least one type of facility.

Table 16 presents the coefficient estimates for the simultaneous model. The table shows that different types of facilities had different sets of independent variables and therefore different sets of coefficient estimates. For example, the variables selected for hospitals were: metropolitan area, proportion of American Indian and Alaska Native (AIAN), income per capita, number of hospices per 10,000 individuals, a dummy if the county had a hospital with a nursing school, number of hospital full time persons per 10 individuals, facility type, total number of budgeted RN positions, RN vacancy rate, total number of budgeted LPN positions, and RN turnover rate.

The coefficient estimates were used to calculate a predicted nursing recruitment difficulty score for each facility type (as similar to the OLS models). These predicted nursing recruitment difficulty scores were used to create groups of facilities with different predicted levels of difficulty (Table 17).

At least three indicators can be used to measure the goodness of fit of the estimated model in explaining the difficulty recruiting RNs. The first is based on the significance levels of the independent variables included in the model. Lower p-values mean a better estimated model. The fact that many of the p-values for many of the variables in the model are less than 0.10 (bolded values) means the model is a good one (Table 16).

A second indicator of goodness of fit is based on a cross tabulation of the actual recruiting difficulty indicator for facilities obtained from the original survey data by the recruiting difficulty indicator based predicted by the model (Table 17). If all off-diagonal values in this table were zero, the model would perfectly explain the difficulty in recruiting RNs. A statistical test of goodness of fit can be computed based on this cross tabulation based on the Spearman Rank Order Correlation. This tests the null hypothesis that there is no correlation between actual recruitment difficulty and the predicted recruitment difficulty. Table 17 shows that the Spearman Correlation coefficient is 0.53, which is statistically significantly different from 0 (p < 0.0005). This is a second reason to trust this model, although a higher correlation coefficient would make the model even stronger.

The third goodness of fit indicator is pseudo-R², the McKelvey-Zavoina R². The higher the value of this pseudo-R², the better the accuracy of the model. The value of 0.71 for this statistic shown in Table 16 is high for this kind of model, another indicator that this model is a good one.

Figure 11 presents the distribution of predicted shortage scores for all facilities. The range of the nursing shortage scores for facilities facing difficulty in recruiting RNs was much higher than those for facilities not facing difficulty in recruiting RNs. These predicted values showed that the number of facilities facing difficulty in recruiting RNs was 141 (43.4%), and the number of facilities not facing difficulty in recruiting RNs was 30 (9.2%).

[D]

Table 16. Coefficient Estimates of the Ordered Probit Nursing Shortage Model Based on All Facilities in North Carolina

Variable	Hospital		Home Health		Long-Term Care		Public Health
	Coeff	p	Coeff	p	Coeff	p	Coeff	p
Demographic Variables
Dummy for metropolitan area	-0.343	0.323			-0.750	0.016	-0.474	0.289
Proportion of population < 5 years					-7.032	0.009
Proportion of population age 20 - 65 years			25.836	0.001
Proportion of population >65 years			8.543	0.145	-20.231	0.001	27.654	0.001
Proportion of White population							-59.011	0.005
Proportion of Black population			2.270	0.121			-50.752	0.014
Proportion of Hispanic population			1.207	0.039	-1.844	0.000	-4.511	0.033
Proportion of AIAN population	1.202	0.150			0.586	0.020
Income per capita ($10,000)	0.692	0.099			-0.593	0.296	-2.144	0.066
Percentage of population in poverty			-0.232	0.004	-0.110	0.099	-0.262	0.014
Proportion of population using Medicare					1.5818	0.040
Proportion of population using Medicaid							2.177	0.052
Nursing Variables
# of RNs per 100 individuals					-1.103	0.009
# of Med Records & Health Info Techs per 1,000 individuals					1.942	0.008
# of hospitals per 10,000 individuals			2.242	0.039			-4.656	0.000
# of Hospices per 10,000 individuals	-1.035	0.454	0.696	0.450			2.457	0.048
Dummy for county having hospital with nursing school	-1.210	0.061			0.399	0.427	2.457	0.048
# of hospital full time personals per 10 individuals	1.176	0.469			-2.89	0.101
# of nursing home full time personals per 1,000 individuals			-0.550	0.038
Ratio of average RN salary to median income			2.530	0.010	-1.877	0.018	-4.023	0.004
Facility Variables
Facility type	-5.384	0.078	-22.06	<0.0005	9.801	0.022	63.513	0.001
Total number of budgeted RN positions	-0.130	0.092	-1.946	0.121	1.834	0.438	-2.491	0.012
RN vacancy rate	1.936	0.046	50.736	<0.0005	35.816	0.010
Total number of budgeted LPN positions	-0.854	0.115
LPN vacation rate					14.321	0.114
RN turnover rate	1.729	0.322			0.1987	0.291	6.396	0.005

Recruiting Difficulty Thresholds
Very easy (1) to recruit if score < -5.494008
Easy (2) to recruit if score < -4.429288
Not difficult (3) to recruit if score < -3.348048
Difficult (4) to recruit if score < -2.158602
Very difficult (5) to recruit if score > -2.158602
McKelvey-Zavoina R² = 0.71

Table 17. Cross Tabulation of Actual Nursing Recruitment Difficulty Indicator by Predicted Nursing Recruitment Difficulty Indicator

Actual	Predicted
	1	2	3	4	5	Total
1	1	9	7	0	0	17
2	0	10	32	9	0	51
3	0	3	66	31	2	102
4	0	7	40	47	5	99
5	0	0	9	26	21	56
Total	1	29	154	113	28	325

Note: Values on the diagonal are shaded
Spearman correlation coefficient = 0.53
Test of H₀: Correlation = 0
 H₀ is rejected with p-value < 0.0005

The descriptive statistics of predicted nursing shortage scores by type of facility are presented in Table 18, which shows that on average the shortage was highest for public health and lowest for hospitals. This means that on average public health facilities faced the most nursing recruitment difficulty and hospitals faced the least.

Table 18. Descriptive Statistics of For Predicted Nursing Recruitment Difficulty Score Based on Ordered Probit Model Using North Carolina Data for 2004

Facility Type	Predicted Shortage Score
	Mean	SD	Minimum	Maximum
Hospital	-3.668	0.466	-5.0797	-2.4393
Home health	-3.391	0.901	-5.1829	0.0163
Long term care	-3.342	0.848	-5.5617	0.5345
Public health	-2.432	3.298	-5.1676	14.8448

Figure 12 shows more clearly the differences in recruiting difficulty among the four types of facilities. The figure presents the distribution of the predicted nursing recruitment difficulty by type of facility. From the figure we can see that a relatively high proportion of public health agencies have high scores (the right side of the figure). This confirms the finding presented above in Table 18.

[D]

Tables 19 to 22 present the coefficient estimates for hospitals, home health facilities, long-term care facilities, and public health, respectively, based on separate ordered probit models for each type of facility. Similar to Technique 1, using these coefficients one can calculate predicted nursing recruitment difficulty scores for each facility. The key difference is that one cannot compare the predicted nursing shortage scores of different types of facilities. For example, a score for a hospital cannot be compared to a score for a nursing home.

In general, the p-values obtained from separate models were lower than those obtained from the simultaneous model (i.e., the results were more significant statistically). This implied that the number of significant variables obtained from the separate models was greater than the number of significant variables obtained from the simultaneous model. These lower p-values also tell us that the estimation using separate models gave more efficient results.

Both techniques provided very similar patterns of predicted nursing recruitment difficulty scores for each type of facility. The strength of the relationship between predicted nursing shortage scores obtained from the two models can be measured using a correlation coefficient. The Spearman correlation coefficient between the two predicted scores was 0.9985 for hospitals; 0.9911 for home health agencies; 0.9991 for long-term care facilities; and 0.9853 for public health agencies. This meant both techniques gave very similar ranks of predicted nursing recruitment difficulty scores across facilities in North Carolina.

McKelvey-Zavoina R² = 0.362

McKelvey-Zavoina R² = 0.406

McKelvey-Zavoina R² = 0.364

McKelvey-Zavoina R² = 0.830

One of the objectives of this study was to assess the importance of facility-specific variables for predicting the difficulty of recruiting RNs and other measures of nursing shortages. Figures 13 through 16 present the results of a series of four comparisons of models, one for each of the four facility types. The figures revealed, based on OLS analysis of data from North Carolina in 2004, that predictions of nurse recruiting difficulty with and without facility data were positively and significantly correlated for all four types of facilities. Similar results were obtained using ordered probit models based on the same data. This result was encouraging for subsequent studies of nursing shortages and related topics because it suggested that, although some predictive accuracy was lost when facility data were not available, at least some helpful insights could be obtained from community data alone.

[D]

[D]

[D]

As part of the process of developing and refining the North Carolina ordered probit models, a special validation process was devised to confirm that the values of the nursing recruitment difficulty index predicted by the statistical models were realistic. This process was made more difficult by the requirement of anonymity of the facilities by the NCCN.

The validation procedure used involved sending the anonymous facility ID Codes back to the NCCN for the 10 facilities of each type that had the highest and lowest nursing recruitment difficulty index scores. NCCN staff then attached to each ID Code the name and contact information for each facility on the list. These individuals were then surveyed over the telephone (see Appendix C) asking for insights about the difficulty experienced by the facility in recruiting RNs at the time, six months earlier, and in 2004 (when the original survey data were collected).

When the survey responses were returned, the data were entered into a separate file for analysis. The primary analysis used in this validation was based on a Spearman Rank-Order Correlation coefficient between the variable indicating that the facility was in the top 10 or bottom 10 for its type, and the 5-point scale from the questionnaire rating difficulty of recruiting RNs in 2004. Based on the 48 (out of a possible 80) facilities that responded to the survey questionnaire, the Spearman’s Rho was 0.347, p = 0.016. Although the correlation between the original rating of recruiting difficulty and the retrospective rating obtained in the validation process was statistically significant, the low value of the correlation coefficient gave little support for the use of these kinds of subjective measures in a formal shortage designation process.

Although this statistical test (that the correlation coefficient = zero) was not particularly stringent, it did provide an indication that the independent variables in the ordered probit model helped to explain variations in nursing recruitment difficulty. Based on this conclusion, project staff moved forward with plans to examine the possibility of using a model estimated in one state to predict nursing recruitment difficulty in another state (in this case, North Dakota).

Data were shared with project staff by two states (North Carolina and North Dakota). Although the North Dakota (ND) data were based on a survey instrument identical in many respects with the North Carolina (NC) questionnaire, the ND survey did not ask the same question about difficulty recruiting RNs that was asked in NC. ND did ask a question about vacancy rates for RNs in the facilities, but unfortunately, the question was answered by only 20% of the respondents. The net result was that the ND data did not provide a sound dependent variable to use in an independent modeling effort similar to that conducted for the NC data.

K. Models for North Carolina and North Dakota Combined

The characteristics of counties in North Carolina and counties in North Dakota differ considerably. For example, using the ARF database, 40% of counties in North Carolina were metropolitan compared to only 8% of counties in North Dakota; 83% of counties in North Carolina had a hospital compared to 64% of counties in North Dakota. In addition, 63% of counties in North Carolina had a hospice compared to 25% of counties in North Dakota.

The averages of community variables of counties in North Carolina and in North Dakota are presented in Table 23. The average percentage of Whites in the population was higher in North Dakota than in North Carolina. The average per capita number of hospital beds in North Dakota was more than three times higher than in North Carolina. Although the average per capita number of hospital beds was much higher in North Dakota than in North Carolina, the average per capita number of full time RNs was slightly lower in North Dakota . Moreover, the average percentage of the population in poverty in North Carolina was slightly higher than that of North Dakota, while average per capita income was slightly higher in North Carolina.

The differences between North Carolina and North Dakota were not only in terms of the community characteristics, but also in terms of facility characteristics. For example, 76% of facilities in North Dakota reported zero vacancy rates compared to 37% in North Carolina; 53% of facilities in North Dakota reported zero turnover rates compared to 13% in North Carolina. Table 24 presents the averages of facility variables in North Carolina and North Dakota and shows that characteristics of the states’ facilities differ considerably. The average number of budgeted RN positions of facilities in North Carolina is almost four times the average in North Dakota. In addition, the average RN vacancy rate of facilities in North Carolina is almost three times the average in North Dakota.

Using data from both North Carolina and North Dakota together, OLS regression was run for RN vacancy rates on a combination of community variables and facility variables. Vacancy rate was chosen because it was the dependent variable collected using the same definitions in both states. A state dummy variable was included, defined as 1 if a facility was located in North Dakota and 0 if it was located in North Carolina. The model was estimated separately for each type of facility. The coefficient estimates are presented in Tables 25 to 28 for hospitals, home health agencies, long-term care facilities, and public health agencies, respectively. Variables included in the model were selected based on their p-values. Variables with smaller p-values can explain variation in dependent variable better than variables with higher p-values. In addition, adjusted-R² was also considered when selecting variables to be included in the model. The higher the adjusted-R², the better the model.

The table shows that each type of facility yielded different sets of independent variables that were statistically significant. For example, dummy for North Dakota was not significant in both hospital and long-term care models, while it was significant in both home health and public health models.

The dependent variable in the models was RN vacancy rate. The higher the value of RN vacancy rate the bigger was the shortage. Thus, a positive coefficient revealed that a facility with a higher value of the corresponding independent variable faced a bigger shortage compared to a facility with a lower value of the variable. A negative coefficient revealed that a facility with a higher value of the corresponding independent variable faced less shortage compared to a facility with a lower value. For example, the coefficient estimate of dummy for North Dakota in the home health model was negative. This indicated that on average home health facilities in North Dakota faced less shortage than home health facilities in North Carolina.

R² = 0.400

R² = 0.346

R² = 0.238

R² = 0.389