Quick Response Freight Manual II

8.0 Model Validation

8.1 Introduction

Model validation involves testing the model’s capability to predict current travel demand so that it can be used effectively to predict future travel demand. In other words, freight travel models need to be able to replicate observed conditions within a reasonable range before they can be used to produce future year forecasts. As metropolitan areas continue to refine and improve their travel demand forecasting processes, the credibility of the process with decision-makers will depend largely on the ability of analysts to properly validate procedures and models used. Also, the travel demand models have become more complex, resulting in complex procedures needed to validate them. Often there are tradeoffs between increasing confidence in the level of accuracy of the models and the cost of data collection and effort required to validate models. Tests or checks used to evaluate the reliability of models can range from a simple assessment of the reasonableness of model outputs to sophisticated statistical techniques.

The model validation and reasonableness tests involve a two-part procedure – calibration and validation. The term “model calibration” is the process of adjusting parameter values until predicted travel matches observed travel demand levels in the given region. The term “model validation” is the process of comparing the model predictions with information other than that used in estimating the model. Model calibration and validation data should be obtained from different sources than the data used in estimating model parameters. As a result, one needs to identify unique sources of data that can support model calibration and validation. For the purpose of this report, calibration and validation data are those data that can be used to compare with model predictions to determine the reasonableness of the model parameters. Model calibration and validation data also are used as a means to adjust the model parameter values so that model predicted travel match observed travel in the region. This is especially important when applying nationally derived model parameters to a specific region.

The focus of this section is to provide various model calibration and validation techniques necessary for each of the freight and truck modeling components. The methodology and the data required are described in detail in the following sections. This section comprises numerous data sources with case studies relevant to the validation of various freight and truck modeling techniques.

8.2 Trip Generation Validation

The trip generation model estimates the number of truck trips to and from each TAZ in the study area. In this step of the travel forecasting process, socioeconomic data are used to estimate the daily truck trips within the study area, i.e., internal-internal, and with origins or destinations outside the study area, i.e., external-internal or internal-external. The trip generation model estimates trip productions and trip attractions.

Trip production and attraction models have been based primarily on one of two basic structures – linear regression models and land use-based trip rate models. The regression models for trip generation are generally developed when origin-destination surveys are conducted for relatively large sample sizes. The regression equations explain the variation in the truck trips based on one or more independent or explanatory variables such as employment and households. Two sets of regression equations are required, one each for the production end and the attraction end. The production models predict the truck trips produced based on variables at the production end, while the attraction models predict the trips being attracted based on variables at the attraction end. Therefore, these models estimate the coefficients for each explanatory variable and the robustness of the models are determined based on a range of statistics. These include the t‑statistic associated with the standard error of the coefficient estimate for each variable and the R‑square for the model that indicates how well it fits the data.

A major drawback with the linear regression models is that the explanatory variables are often interrelated and correlated with each other. It also assumes that the relationship between the explanatory variables, typically employment for freight and truck models, and the truck trips generated are linear.

The land use-based trip rate models are developed based on information on land uses at trip ends and the household and employment data at the zonal level. The trip rates are computed as a ratio between the total study area truck trips to a particular land use and the total study area employment for that particular land use. This approach is used for the trip ends, both production and attraction. The performance of this approach lies in the ability to stratify the employment into many categories and correlate it with the correct land use types. One or more types of employment can influence the truck trip generation on a particular land use type. The main disadvantage of this approach is the need to forecast the various types of employment. This approach also necessitates the data collection effort to be more informative in terms of land use types at trip ends.

There are a number of sources of error in the development of freight/truck trip generation models. The sampling error and bias in the travel survey affect the trip generation rates. Also, the models may not be specified correctly with the relevant explanatory variables. So, the validation procedure must include tests that involve the examination of total and land use-specific trip rates. These tests can be aggregated as well as disaggregated as described below.

8.2.1 Total Truck Trip Productions and Attractions per Employee

Objective – Compare estimated trip rates against national average rates.

One of the biggest issues in the internal model methodology is the data availability and methods of estimating trip generation. In 2002, the National Cooperative Highway Research Project (NCHRP) published Synthesis Report #298 on truck trip generation. This report critiques all of the available methods for estimating truck trip generation models and rates and presents data from numerous studies in North America. This report can be used to evaluate the reasonableness of the trip rates estimated from the trip generation model and can provide guidance on techniques for improving estimation of these rates.

The QRFM that provides truck trip rates by land use category from a number of different studies also can be used as a means to validate and adjust the trip rates computed from the trip diaries during model calibration. Other sources of validation data for trip generation can include previous version of travel models, local studies on truck trips at various business facilities, and dispatch logs of truckers that some motor carriers maintain.

8.2.2 Total Truck Trips by Purpose or Business Sector

Objective – Compare estimated truck trips by business sector against other local models/reports/data and studies from other regions and agencies.

Due to the varying trip-making behavior of trucks across different business sectors, different trip generation models are usually estimated by business sector that are analogous to trip purpose in a passenger travel model. The typical sectors include manufacturing, warehouses/distribution centers, retail, local pickup and delivery, and service industry. The model results from other regions and agencies can be used to compare the estimated trips or percentage of total trips by business sector to do a reasonableness check.

8.2.3 Observed versus Estimated Truck Trips

Objective – Compare estimated truck trips against observed data by sector, geography, and truck type.

The best validation test is to compare the estimate truck trips against observed data. This test should usually be done by different sectors, geographical area, and truck type. Just like the travel characteristics of truck trips are different across sectors, they also vary by geographical location and truck type. The distribution of land uses and employment in a region drives the variability of truck travel behavior by geography, and the nature of freight flows and the commodity being shipped influences the travel behavior of trucks of different weight classes.

The observed data is gathered either by traditional truck surveys for the entire region which are then expanded to the entire truck population. This expanded data gives the O‑D truck trip data at the zonal level for the region or study area. Another source of data is truck intercept surveys at certain key locations in the region. Vehicle classification counts also can be used to develop validation targets for truck trip generation models at those locations where counts are collected.

The differences between observed and estimated trip totals may be due to either error in the trip generation model, or the sampling error in the truck travel survey. These differences need to be reduced to an acceptable range during model calibration that could involve many steps. These include adjusting the trip rates, re-estimating models with different set of explanatory variables, regrouping sectors, and reclassifying truck types.

8.2.4 Coefficient of Determination (R‑Square)

Objective – Check the model for its predictive power.

The coefficient of determination, or R‑square, measures the proportion of variability in the survey data that is accounted for in the trip generation model. If the value is closer to 1.0, then the model is considered statistically a good model with good predictive power. If the R‑square is low, then it could be either the variables specified are not the right kind, or they are correlated to one another. This also is attributed to low sample sizes with large variances.

8.2.5 Plot of Observed versus Estimated Trips (or Trip Rates)

Objective – Check the model for geographical biases.

This validation test is usually done at the district level to see how well the estimated trip or trip rates compare with the observed data. This is a good indicator of model performance and also can help in detecting in any geographical biases, which will need specific attention during calibration.

8.2.6 Disaggregate Validation – Observed versus Estimated

Objective – Apply model to survey records.

A simple and common method of validating the model estimated is to apply it to the survey data that was used in the estimation. That is, applying the trip generation model to the survey records to estimate the productions and attractions. The comparison of the estimated trip end totals to that of the survey totals can be done at any desired level, from very disaggregate to aggregate.

8.3 Trip Distribution Validation

Trip distribution links the trip productions in the region with the trip attractions to create matrices of interzonal and intrazonal travel, called trip tables. The critical outputs of trip distribution are trip length and travel orientation (suburb to CBD, CBD to suburb, etc.), and the resulting magnitude of traffic and passenger volumes. The most common form of model used for trip distribution is the gravity model. The inputs for gravity model-based trip distribution models are productions and attractions for each zone and a matrix of interzonal and intrazonal travel impedances. The productions and attractions are derived from the trip generation model while the travel impedances are obtained from determining the path of least resistance between each pair of zones.

Travel impedances reflect the spatial separation of the zones based on shortest travel-time paths for each zone-to-zone interchange. Some models use a generalized cost approach which converts highway travel time to cost and combines the time cost with other highway costs, including operating expenses (i.e., gas, wear-and-tear), parking, and tolls. Regardless of the procedure used to estimate travel impedances, several types of reasonableness checks can be performed to ensure that the highway skims contain realistic values. The first is a simple determination of implied speeds for each interchange. The second might be a simple frequency distribution of speeds on all interchanges. Another aggregate network-level check is of terminal times. These represent the time spent traveling to/from a vehicle to/from the final origin or destination within the TAZ. Terminal times are generally determined using the area type of the TAZ. The terminal times may be adjusted as part of the trip distribution model calibration process in order to make the average trip lengths produced by the model more closely match the observed average trip lengths. If terminal times are used to adjust impedances, then these will tend to shift the friction factor curve to the right making the distribution of trips from that zone less sensitive to impedance.

During calibration of trip distribution models, the observed and estimated trip lengths are both calculated using network-based impedance. Most travel demand modeling packages automatically calculate average trip length for all trip interchanges. In effect, it is finding the average travel time from the skims matrix weighted by the trip matrix. Some of the truck travel surveys like the trip diary approach do ask truckers to report travel times for their trips. However, these times are not considered as reliable as the origin and destination information obtained from the survey. The reported times are used only to provide an approximate estimate of truck trip lengths in the model validation.

Another source of trip length data is the 2002 VIUS, which consists of trip summaries of commercial vehicles in the entire country. The VIUS data also can be summarized by state or metropolitan areas within a state and also can be done by industry sector and truck type.

8.3.1 Compare Average Trip Lengths

The most standard validation checks of trip distribution models used as part of the calibration process are comparisons of observed and estimated truck trip lengths. The modeled average trip lengths should generally be within five percent of observed average trip lengths. This is typically done by truck type or weight class, and also can be extended to include the sector type stratification, if data permits.

If a generalized cost is used as the measure of impedance, average trip lengths and trip length frequency distributions should be checked using the individual components of generalized cost (e.g., time and distance).

8.3.2 Compare Trip Lengths for Trips Produced versus Trips Attracted

Another way of calibrating the gravity model is by comparing truck trip lengths for trips produced against the trip attracted by sector and area type. This will indicate if the model is performing well at both trip ends and if it is reflecting the observed distribution of truck flows by sector and area type. The average trip lengths sent (produced) and received (attracted) by district also could be mapped using GIS to examine the model performance.

8.3.3 Plot Trip Length Frequency Distributions

The plot of trip length frequency distribution shows how well the model can replicate observed trip lengths over the range of time. The visual comparison of distributions such as shown in Figure 8.1 is an effective method for calibration and validation. A quantitative measure which can be used to evaluate distribution validation is the coincidence ratio.

Figure 8.1 Trip Length Frequency Distribution

Coincidence Ratio

The coincidence ratio is used to compare two distributions. In using the coincidence ratio, the ratio in common between two distributions is measured as a percentage of the total area of those distributions. Mathematically, the sum of the lower value of the two distributions at each increment of X is divided by the sum of the higher value of the two distributions at each increment of X. Generally, the coincidence ratio measures the percent of area that “coincides” for the two curves.

The procedure to calculate the coincidence of distributions is as follows:

• Coincidence = sum {min(count_+X/count₊, count_-X/count_-)}
• Total = sum {max(count_+X/count₊, count_-X/count_-)}
• Calculate for X = 1, maxX
• Coincidence Ratio = coincidence/total

where,

• count_+T = value of estimated distribution at time T
• count₊ = total count of estimated distribution
• count_-T = value of observed distribution at time T
• count_- = total count of observed distribution

The coincidence ratio lies between zero and one, where zero indicates two disjoint distributions and one indicates identical distributions. An example is presented in Figure 8.2, where the shaded areas represent how well the distributions match or coincide. The top chart has a much smaller shaded portion than the bottom chart, which indicates a better match in the distributions in the bottom chart. Thus, the coincidence ratio will be higher for the bottom chart.

Figure 8.2 Coincidence Ratio for Trip Distribution

8.3.4 Plot Normalized Friction Factors

If a gravity model is used for trip distribution, then it also is worthwhile to plot the calibrated friction factors (scaled to a common value at the lowest impedance value). Such a plot provides a picture of the average trucker’s sensitivity to impedance by truck type or sector, and can be compared to friction factors from other regions. For example, certain types of truck trips might be less sensitive to travel time since these trips must be made every day and can usually not be shifted to off-peak conditions or to different locations. This phenomenon can be observed from the plot where the friction factors show gradual change as travel time increases.

If there are significant differences between observed and estimated trip lengths, then this may be due to either inadequate closure on production/attraction balancing or travel impedances may be too high or too low. After validating the trip distribution model at a regional level, the model results should be checked for subgroups of trips and segments of the region.

8.3.5 Compare Observed and Estimated District-to-District Trip Interchanges and Major Trip Movements

Although comparing trip lengths provides a good regional check of trip distribution, the model can match trip lengths without distributing trips between the correct locations. In order to permit easier review of the truck trip tables, zonal interchanges can be summarized into counties, districts, or groups of zones. Trips to the major employment area in the region (i.e., CBD) should be reviewed. Major trip movements to various special generators such as ports, airports, and intermodal facilities should be summarized as well.

K-Factors

K‑factors are usually district-to-district factors that correct for major discrepancies in trip interchanges. These factors are computed as the ratio between observed and estimated trip interchanges. K‑factors are typically justified as representing economic activity that affect truck trip making but are not otherwise represented in the gravity model.

The use of K‑factors is, however, generally discouraged and seen as a major weakness of traditional gravity models when used to correct for intangible factors. Since K‑factors represent characteristics of the economic activity and population which change over time, the assumption that K‑factors stay constant in the future can introduce a significant amount of error in predictions of future trip distributions.

8.4 Mode Split Validation

Mode split models are unique to studies that deal with multimodal facilities; that is, besides roadways for trucks, they involve railways for trains carrying freight, water transportation at ports, and by air at airports. These studies often deal with commodity-based freight flows (in tonnage) that are split into different transportation modes based on the characteristics of the shipment being carried, destination, cost, and delivery time associated with the shipments. The treatment of modal choice can vary a great deal by region and the availability of various facilities. For regions with limited or no rail, water and air transportation, it may be sufficient to apply a fixed mode split factors to determine the percentage of freight moved other than by trucks.

Mode split is usually based on a logit mode choice model and historical mode split percentages. Most statewide models utilize qualitative estimates varying observed mode shares. The reason behind this is because mode choice for freight does not follow strict probabilistic rules because the magnitude of freight flows in tons is determined by only a few shipper or carrier decision-makers.

8.4.1 Comparison of Mode Split Model Coefficients with Other Studies

The basic validation test for a mode split model is to compare the estimated model coefficients with other studies that deal with similar modes and under similar conditions. This gives an indication if the model is performing within reasonable expectations. The important things to look for are the signs and magnitudes of the level of service variables such as cost and time. These should always be negative and within acceptable range of values. The values of time associated with each mode also should be computed to determine the reasonableness of the coefficients.

The comparison of model coefficients and derived variables can be considered both a validation check and a sensitivity check. Typically, when mode choice models are estimated, not only the model coefficients, but also derived ratios and model elasticities are compared to those from other regions. If model coefficients (and constants) and derived ratios are in the range of what has been reported elsewhere, the model sensitivity should be similar to models used in other regions. Tests on sensitivity are described in the following section.

8.4.2 Sensitivity Tests – Elasticity of Demand to Supply Relationship

A common sensitivity test for mode choice models is the direct or cross elasticities of the model. Elasticities can be used to estimate the percent change in demand given a percent change in supply. As with the values of the model coefficients and derived ratios, elasticities can be considered as both validation and sensitivity tests. Sensitivity tests can be made on model elasticities for costs and travel-time attributes. Sensitivity tests are performed by applying the model with unit changes in variables, e.g., a $0.25 increase in cost or a 10 percent increase in travel time.

8.4.3 Observed versus Estimated Shares of Freight Flows

An aggregate validation test is to compare the estimated shares to that of observed shares of freight flows by different modes of travel such as trucks, rail, ship, and air. This test can be used as a calibration procedure which involves adjusting the modal constants until the shares match well within acceptable ranges.

8.5 Assignment Validation

Trip assignment is the fourth and last step of the traditional four-step process. This includes both passenger and commercial vehicle assignment carrying people and goods respectively. The assignment of trips to the network is the final output of the modeling process and becomes the basis for validating the model set’s ability to replicate observed travel in the base year as well as to evaluate the transportation improvements in the future years. Depending on the level of analysis being done, the assignment can be to a regional highway network for systemwide planning or to a detailed network for a subarea or corridor study.

The calibrated commercial vehicle trip tables are assigned to a network along with passenger vehicle trip tables to produce estimates of total traffic on network links. There are, however, some special considerations that may affect the assignment of commercial vehicle trips. These include the larger impact of trucks on congestion than passenger vehicles on a per VMT basis, existence of truck only lanes, and the prohibition of trucks on certain corridors in a region.

The Highway Capacity Manual provides “passenger car equivalence” (PCE) factors that can be used to quantify the relative impact of different types of vehicles on congestion. For example, a PCE value of 2.0 indicates that the vehicle in question has the same effect on congestion as 2.0 passenger cars. Specifically, the HCM recommends a PCE value of 1.5 for trucks and buses on level terrain, with trucks defined as commercial vehicles with six or more tires. Hence, to reflect the effect of heavier vehicles on congestion, the trip tables for single-unit trucks with six or more tires and combinations can be multiplied by 1.5 and 2.0, respectively, before being assigned to the network. The resulting assignment volumes will then be expressed in PCEs, not number of vehicles. No adjustments to PCE values are needed for four-tire commercial vehicles, since these vehicles are generally similar to passenger cars in terms of acceleration and deceleration capabilities. Though usually not considered as a major factor, the grade of roadway facilities also plays a significant role in determining the actual PCE value of a truck. In some applications, a variable PCE methodology also is adopted to increase the performance of the assignment model. Adjusting the PCE values also is considered a calibration techniques to better match the observed truck traffic counts by weight class.

If trucks are prohibited from using key network links, then the truck prohibitions must be enforced in the basic network description. Usually, four-tire commercial vehicles such as pickup trucks and vans are not considered to be trucks for the purpose of enforcing truck bans, so that such vehicles would be combined with passenger cars in the assignment process. In some areas, truck-only lanes also are considered to better regulate the traffic flows during congested time periods and areas. So this also should be properly accounted for when developing the roadway network attributes.

The validation tests for truck assignments are presented at three levels: systemwide, corridor, and link specific. There are several systemwide or aggregate validation checks of the assignment process. The checks are generally made on daily volumes, but it is prudent to make the checks on volumes by time of day as well. Systemwide checks include VMT, cordon volume summaries, and screenline summaries.

8.5.1 Vehicle Miles Traveled

Objective – Compare model VMT against HPMS VMT estimates by functional classification and area type.

The validation of a truck model using VMT addresses all major steps in the travel model system, including trip generation (the number of trips), trip distribution (the trip lengths), and assignment (the paths taken). Using observed data such as traffic counts and roadway mile, a regionwide estimate of VMT can be developed to be used for validating the base year assignment of commercial vehicles produced by a travel demand model. These traffic counts are collected in most urban areas as part of the ongoing transportation planning process and are used to validate the passenger portion of urban travel demand models. In addition to any counts that might be undertaken for planning purposes, state DOTs are required to include Annualized Average Daily Traffic Counts and mileage for all roadways, based on a statistical sample, for each urban area as part of their annual Highway Performance Monitoring System (HPMS) submittal. The HPMS VMT can be summarized by functional classification of highways and by area type and compared to the urban area model volumes by functional classification and area type. When using HPMS estimates of VMT, it is important to understand that VMT is for all roadways, including local roads. Travel demand models, in contrast, generally do not include these local roads so this comparison should consider an adjustment for them to allow for a comparison of the total observed and estimated VMT.

Generally, traffic counts are collected and VMT is calculated either for all vehicles or for vehicles classified by axle configuration. Traffic count information is predominately collected by Automatic Traffic Recorders (ATR) and, thus will rarely include any other classification of commercial vehicles. This information will typically be based on a visual identification of commercial markings on the vehicle or a visual observation of the commercial registration plate.

HPMS estimates of percentages of single unit and combination trucks, based on ATRs, can be used to develop VMT for these types of trucks. Not all commercial vehicles are included in these classes and intercity freight trucks that are excluded from the definition of urban commercial vehicles are responsible for a considerable portion of the truck travel on higher functional classes. Nevertheless, HPMS estimates of truck VMT can be used to validate commercial vehicle models. It should be noted, however, that the HPMS values for trucks are based on statistical samples. Thus, the “observed” truck VMT is in reality an estimate.

Based on accepted standards for model validation, modeled regional VMT should generally be within 5 percent of observed VMT [Barton-Aschman Associates and Cambridge Systematics, Inc., Model Validation and Reasonableness Checking Manual, Travel Model Improvement Program, FHWA, 1997]. When the regional models are used to track VMT for air quality purposes, the Environmental Protection Agency requires that estimates be within 3 percent. However, these estimates are for the total of all vehicles irrespective of vehicle type. If commercial vehicles generally represent 13 percent of total VMT, and if a travel demand model’s estimate of commercial VMT is within 5 percent of that value, it would be consistent with the overall validation standards.

The mix of commercial vehicles by functional class will, however, vary considerably by vehicle category. For example, school buses travel almost exclusively on local or collector roads, while urban freight vehicles travel principally on the arterial system. Thus, commercial vehicles cannot be expected to have the same distribution by functional classification as that of other vehicles. However, the variability of usage of the functionally classified roads by urban area size can be expected to occur for commercial vehicles.

In addition to validating modeled VMT to observed VMT by functional class, it is customary to use measures such as VMT per person or per household to assess the reasonableness of urban models. Reasonable ranges of total VMT per household are 40 to 60 miles per day for large urban areas and 30 to 40 miles per day for small urban areas (Barton-Aschman, 1997). If one applies the 13 percent of total VMT that is estimated for commercial VMT to these household ranges, then the VMT per household for commercial vehicle demand would represent 5 to 8 miles per day for large urban areas and 4 to 5 miles per day for small urban areas.

There are many useful statistics that can be calculated for the systemwide-level validation of VMT. These include both the absolute and relative (percent) difference. Another measure is to compare current estimates of regionwide VMT with the historical trend and rate of growth from HPMS. The absolute difference is the simple difference between observed and modeled VMT. The difference is typically large for high-volume links and low for low-volume links, so the size of the numerical difference does not reliably reflect the true significance of error. Percent difference is often preferred to absolute difference since its magnitude indicates the relative significance of error. Modeled regional VMT should generally be within five percent of observed regional VMT. This five percent difference is particularly important in light of the accepted error that EPA allows for VMT tracking using the HPMS data.

8.5.2 Vehicle Classification Counts

Objective – Compare modeled truck traffic volumes against observed truck traffic counts – screenlines, area type, volume group, and facility type.

Travel demand models are validated by comparing observed versus estimated traffic volume on the highway network and by comparing summations of volumes at both cordons and screenlines. Vehicle classification counts have been used to validate automobile and truck volumes, but this is not directly useful to validate commercial vehicles by category, since many categories contain both automobiles and trucks. Nonetheless, it is one of the only sources to verify the reasonableness of traffic volumes based on the inclusion of commercial vehicles into the transportation planning models.

The vehicle classification count data, which classifies vehicles according to the 13-axle-based classes of the FHWA, is generally available from state departments of transportation for sampled sets of streets and highways. For the 13 classes, the information includes counts by location, hour of the day, and date. In summary format, this information generally presents truck volumes (defined as FHWA Classes 5 through 13, six tires and above) and occasionally includes buses (FHWA Class 4). Four-tire pickup trucks, vans, and sport utility vehicles (FHWA Class 3), are almost always included with passenger cars.

After assignments of commercial vehicles by type (automobile and truck at a minimum), the vehicle classification counts can be used to compare the observed automobile and truck counts (and shares by vehicle type) with the estimated automobile and truck volumes (and shares) produced by the travel demand model. These vehicle assignments will include both personal and commercial vehicles, derived from both personal and commercial models, so calibration adjustments deemed necessary from these comparisons may be required for either the personal or commercial models or both. The validation summaries also are usually summarized by functional class, area type, and screenlines.

Compare Observed versus Estimated Volumes by Screenline

The validation targets can vary for screenlines depending upon the importance of the screenline locations in the study area. Figure 8.3 shows the maximum desirable deviation in total screenline volumes according to the observed screenline volume.

Figure 8.3 Maximum Desirable Deviation in Total Screenline Volumes

Compare Observed versus Estimated Volumes for All Links with Counts

With the use of the on‑screen network editors and plots of network attributes, the checking of link level counts visually is relatively simple. In addition to visually checking the correlation of the counts to volumes, as shown in Figure 8.4, it also is useful to compute aggregate statistics on the validity of the traffic assignment. There are two measures computed widely during model validation, and these are the correlation coefficient and the percent root mean square of the error (RMSE).

Figure 8.4 Assigned versus Observed Average Daily Traffic Volumes

R2 (Coefficient of Determination)

R2 is computed to determine the performance of the model predictability. It is used to compare the regionwide observed traffic counts to that of the estimated volumes regionwide. A value closer to 1.0 indicates a better model. Another useful validation tool is to plot a scattergram of the counts versus the assigned volumes (as shown in Figure 8.4). Any data points (links) that lie outside of a reasonable boundary of the 45-degree line should be reviewed.

RMSE

The RMSE is computed using the following formula:

The acceptable RMSE ranges vary based on the facility type; it should be as small as 5 percent for freeways and expressways, and as large as 40 to 50 percent for local and minor arterials.

Model Parameters

There are a number of parameters in an assignment model that are potential sources of error. Although the actual parameters and calculation options involved depend on the modeling software and assignment methodology being used, other possibilities include:

• Assignment procedures, including number of iterations, convergence criterion, expansion of incremental loads, and damping factors;
• Volume-delay parameters such as the BPR coefficient and exponent;
• Peak-hour conversion factors used to adjust hourly capacity and/or daily volumes in volume-delay function;
• PCE factors for commercial vehicles;
• Scaling or conversion factors to change units of time, distance, or speed (miles/hour or kilometer/hour);
• Maximum/minimum speed constraints;
• Preload purposes (HOV, through trips, trucks, long/short trips); and
• Toll queuing parameters (diversion, shift constant, etc.).

8.5.3 Registration Records

Objective – Comparison of model fleet sizes against observed fleet sizes by sector and district/county/region.

State vehicle registration databases often indicate whether registered vehicles are used for commercial purposes. These databases typically show vehicle weight classes, but not service use. Service use can be inferred based on vehicle make/model, weight class, owner, and possibly other data. However, this requires considerable data processing. Many states databases also do not include odometer readings. It also should be recognized that motor carriers and private fleet operators may register their trucks in states with based not on operations but on consideration of state taxes and regulations and adjustments and thus state truck registrations may underestimate or overestimate the actual size of a state’s active truck fleet.

In a recent Federal research effort, Cambridge Systematics came up with an approach to compute and compare the modeled truck fleet sizes against the observed fleet sizes derived from local DMV registration data. Vehicle registration databases that are maintained by a state can yield useful information on the number of commercial vehicles existing within a particular geographic area. For example, the California Energy Commission has been working with the California DMV and other agencies since the late 1990s in an effort to clean, organize, and analyze the State’s vehicle data. The California DMV employed all key words from the 120-character owner field of each record in the database that reveal any potential business use information. The Energy Commission divided the DMV data into two main groups: 1) light vehicles and 2) medium and heavy vehicles. It further divided the light vehicle category by use, and the medium and heavy vehicle category by body type.

Based on use and body-type subcategories, the registration data was mapped to the 12 categories of commercial vehicles, as shown in Table 8.1. No vehicle types in the California DMV database correlate to the following commercial vehicle categories in this study: Shuttle Service: Airports, Stations; Private Transportation: Taxi, Limos, Shuttles and Paratransit: Social Services, Church Buses.

Table 8.1 California DMV Vehicle Types by Commercial Vehicle Category

Commercial Vehicle Category	California Light-Duty Vehicles	California Medium- and Heavy-Duty Vehicles
1 School Bus	[no data]	Bus
5 Rental Cars	Daily Rental	[no data]
6 Package, Product and Mail Delivery: USPS, UPS, FedEx, etc.	[no data]	Parcel Delivery
7 Urban Freight Distribution, Warehouse Deliveries	[no data]	Automobile Carrier, Beverage, Cargo Cutaway, Dromedary, Logger, Multiple Bodies, Refrigerated, Stake or Rack, Tandem, Tank, Tractor Truck, Tractor Truck Gas
8 Construction Transport	[no data]	Boom, Concrete Mixer, Crane, Cutaway, Dump, Flat Bed/Platform, Motorized Cataway
9 Safety Vehicles: Police, Fire, Building Inspections, Tow Trucks	Government – District – Fire Government – District – Police	Ambulance, Fire Truck, Tow Truck Wrecker
10 Utilities Vehicles (Trash, Meter Readers, Maintenance, Plumbers, Electricians, etc.)	Government – District – Utility Government – District – Water/Irrigation	Garbage, Utility
11 Public Service (Federal, State, City, Local Government)	Government – City Government – County Government – State Government – Federal Government – District – School Government – District – College Government – District – Transit Government – District – Other	[no data]
12 Business and Personal Services (Personal Transportation, Realtors, Door-to-Door Sales, Public Relations)	Other Commercial	Armored Truck, Panel, Pickup, Step Van, Van
Not Categorized	Personal	Chassis and Cab, Conventional Cab, Forward Control, Gliders, Incomplete Chassis, Tilt Cab, Tilt Tandem, Unknown, Motorized Home

Source: California Department of Motor Vehicles registration data processed by the California Energy Commission, 2002.

The California DMV data has a large category of “other commercial” light duty vehicles that was assigned to the business and personal services categories. Since not all “other commercial” vehicles are being used for commercial purposes, this category can be factored to exclude the business and personal service vehicles used for personal activities, based on the VIUS estimates of the use of these vehicles. The VIUS Business and Personal Services category vehicles was then cross-tabulated by business use and personal use, and was determined that in California, 22 percent of total vehicles (both personal and commercial) are used for commercial purposes. Accordingly, “other commercial” vehicles in the California DMV data were multiplied by 0.22 to obtain the numbers of Business and Personal Services vehicles as shown in Table 8.2.

Table 8.2 Business and Personal Services Vehicles in California Cities

Vehicle	San Francisco	Los Angeles	San Diego	Sacramento
“Other Vehicles” from California DMV Database	687,169	1,474,911	242,156	210,271
Factors from VIUS Database	0.22	0.22	0.22	0.22
Business and Personal Services Vehicles	152,263	321,445	50,488	43,984

Source: Accounting for Commercial Vehicles in Urban Transportation Models, FHWA, 2003.

Registration data, such as that collected by the California DMV, is the best source of fleet size statistics. Table 8.3 presents the California DMV data on fleet size for four California urban areas that could be used for calibration or validation of urban area commercial vehicle models.

Table 8.3 Fleet Sizes across Select Cities in California

Commercial Vehicle (CV) Category	San Francisco: Number of CV	San Francisco: Percent	Los Angeles: Number of CV	Los Angeles: Percent	San Diego: Number of CV	San Diego: Percent	Sacramento: Number of CV	Sacramento: Percent
School Bus	1,510	0.03%	5,259	0.05%	1,267	0.06%	1,011	0.07%
Rental Car	89,805	1.78%	88,217	0.83%	12,107	0.61%	9,913	0.69%
Package, Product, Mail	470	0.01%	449	0.00%	41	0.00%	42	0.00%
Urban Freight	22,484	0.44%	69,617	0.65%	8,510	0.43%	10,651	0.74%
Construction	22,561	0.45%	36,318	0.34%	6,939	0.35%	8,798	0.61%
Safety Vehicles	5,090	0.10%	11,149	0.10%	3,364	0.17%	7,090	0.49%
Utility Vehicle	7,552	0.15%	19,488	0.18%	2,729	0.14%	5,108	0.36%
Public Service	38,094	0.75%	83,219	0.78%	13,111	0.66%	36,710	2.56%
Business and Personal Services	152,263	3.01%	321,445	3.01%	50,488	2.55%	43,984	3.07%
Total Commercial Vehicles	885,120	17.50%	1,806,460	16.90%	292,652	14.80%	291,849	20.34%
Total Vehicles	5,057,355	100%	10,688,810	100%	1,977,794	100%	1,434,670	100%

Source: Accounting for Commercial Vehicles in Urban Transportation Models, FHWA, 2003.

Vehicle registration data for New York State are available at their web site [New York State Department of Motor Vehicles, 2001]. These data are not as detailed as the California DMV data. Vehicle registration and new vehicle data also may be purchased from R.L. Polk & Co., a privately owned consumer marketing information company. Polk develops custom reports for customers, providing data by ZIP code, Metropolitan Statistical Area, county, state, or for the entire United States. The numbers of vehicles by type are summarized for five cities in Table 8.4.

Table 8.4 Fleet Sizes across Select Cities

Commercial Vehicle (CV) Category	New York City (Bronx Only): Number of CV	New York City (Bronx Only): Percent	San Francisco: Number of CV	San Francisco: Percent	Los Angeles: Number of CV	Los Angeles: Percent	San Diego: Number of CV	San Diego: Percent	Sacramento: Number of CV	Sacramento: Percent
Bus	624	0.2%	2,101	0.4%	19	0.3%	230	0.1%	72	0.0%
Taxi	5,394	2.0%	11,844	2.5%	175	2.5%	6,720	2.6%	325	0.2%
Trailer	1,561	0.6%	2,424	0.5%	57	0.8%	932	0.4%	8,981	4.2%
Ambulance	63	0.0%	642	0.1%	2	0.0%	135	0.1%	42	0.0%
Motorcycle	2,395	0.9%	4,831	1.0%	77	1.1%	5,374	2.1%	4,465	2.1%
Moped	80	0.0%	253	0.1%	4	0.1%	887	0.3%	146	0.1%
Rental Vehicles	334	0.1%	2,246	0.5%	78	1.1%	207	0.1%	2,236	1.0%
Total Commercial Vehicles	17,317	6.4%	38,420	8.2%	662	9.3%	21,885	8.5%	39,430	18.2%
Total Vehicles	269,577	100%	470,290	100%	7,086	100%	257,531	100%	216,133	100%