How to Analyze Your Data with the ISIR Analysis Tool

Over the last several years Federal Student Aid has delivered multiple training sessions and produced a number of documents that provide guidance on how to use the ISIR Analysis Tool. We have come to realize, however, that many schools have a hard time applying this training to the analysis of their own data. To help schools get started, we have prepared this analytic “menu.”

The analysis you should do depends on what you hope to accomplish. There are at least three distinct reasons to analyze changes to ISIR data in the Tool. The right analysis will help you:

Start verifying records that need of correction that current verification efforts are missing
Stop verifying records that do not experience a change in eligibility for need-based aid when verified
Improve the accuracy of initial FAFSA information though consumer information or educational outreach

Below we identify the reports to run and provide brief pointers about what to look for in seeking to accomplish each of the goals listed above. We have used excerpts from previously released documents and other Word documents to provide detailed instructions. We are using the analogy of a menu, because the flexibility of the Tool allows each user to load up his or her analytic plate in the way that best works for them. We hope that this document makes it easier for schools to order a satisfying meal.

Start verifying records in need of correction that current verification efforts are missing

Possible only in alternating years when verifying a random sample!

You have two basic options

Use the Field Increment Report

Run the Field Increment Report from the Domain Tree (Click here and view #1 for complete instructions)
The chef recommends:
Using the filter screen to focus exclusively on either dependent or independent applicants
Selecting increments based on a field you use (or are interested in using) to select records for verification
What to look for:
High percentages of increases and decreases in EFC not just corrections
Value ranges with more than their share of increases or decreases to EFC. For best results compare the distribution of corrections observed in the Field Increment Report to the distribution of ALL of your school’s financial aid applicants. Ask your systems people to produce counts of your population within the applicable ranges or the Northern Colorado shared report described in the Word document below to find the number of all records uploaded into the Tool within each applicable range.

Twist on the Schools Verification Report
Words of Caution:
Be wary of results based on a small number of records - multiply the percentages by the report sample size to know how many relevant cases you’re dealing with. If this number is less than five, take results with a grain of salt.
Consider whether the observed changes to EFC are likely to be above or below your school’s cost of attendance.
Optional 1st course - to focus exclusively on changes to a single ISIR Field; drill down to the Field Increment Report from the Field Change Report. (Click here and view #2 for complete instructions)

Use the School Verification Analysis (Shared) Report

Access and Run the School Verification Analysis Report Link to training materials and Excel templates (Click here and view #3 for complete instructions)
Link to training materials and Excel templates
www.ifap.ed.gov/qahome/training.html
The chef recommends:
Adjusting the value of AAA_COST_OF_ATTENDANCE to reflect your schools price. By the default this value is set to $17,336. Any record with an initial and paid on EFC above this threshold will be excluded from the report. (Click here and view #4 for complete instructions)
Adjusting the value of AAA_MAJOR_CHANGE_DEF to the desired level. By default any change to Pell or a change to EFC in excess of 400 is deemed a major change. Modifying this value adjusts the EFC threshold. (Click here and view #5 for complete instructions)
What to look for:
Categories with the largest “Selected without major change” (light blue) bands in their stacked column graphs
The categories with the largest “Selected without major change” bands that also have relatively large “Not selected without major change” (white) bands
Use the “Unnecessary Diagnostics“ tab in the School Verification Graphs template to see the results of all the results in a single place
See the 2008 QA Pre-Conference Materials for details. (Click here and view #6 for complete instructions)
Words of caution:
Do not use results based on the analysis of five or fewer cases.
Make sure that you have at least some records loaded in the Tool that were NOT selected for school verification and that some of these records did not experience a “major change.”
Make sure that you have some dependent records with and without a “major change” loaded in the Tool.
If you failed to meet either of the two conditions above, make the necessary adjustments to the output of the shared report as specified in the 2008 QA Pre-Conference Materials. (Click here and view #7 for complete instructions)

Stop Verifying Records that DON'T CHANGE

Possible every year! Very similar in terms of what reports to run as above, but by changing focus you can use the same report to look for types of students to exclude from verification.

Use the Field Increment Report

Run the Field Increment Report from the Domain Tree. (Click here and view #1 for complete instructions)
The chef recommends:
Using the filter screen to focus exclusively on either dependent or independent applicants.
Selecting increments based on a field you use to select records for verification
What to look for:
Value ranges with less than their share of increases or decreases to EFC. For best results compare the distribution of corrections observed in the Field Increment Report to the distribution of ALL of your school’s financial aid applicants.
When looking for under-represented ranges consider the percentages in increases and decreases in EFC columns not just the values in the corrected column
Words of caution:
Be wary of results based on a small number of records - multiply the percentages by the report sample size to know how many relevant cases you’re dealing with. If this number is less than five, take results with a grain of salt.
Consider whether the observed changes to EFC are likely to be above or below your school’s cost of attendance.

Use the School Verification Analysis (Shared) Report

Access and Run the School Verification Analysis Report. (Click here and view #3 for complete instructions)
Link to training materials and Excel Templates
http://www.ifap.ed.gov/qahome/training.html
The chef recommends:
Adjusting the value of AAA_COST_OF_ATTENDANCE to reflect your schools price. By the default this value is set to $17,336. Any record with an initial and paid on EFC above this threshold will be excluded from the report. (Click here and view #4 for complete instructions)
Adjusting the value of AAA_MAJOR_CHANGE_DEF to the desired level. By default any change to Pell or a change to EFC in excess of 400 is deemed a major change. Modifying this value adjusts the EFC threshold. (Click here and view #5 for complete instructions)
What to look for:
Categories with the largest “Selected without major change” (light blue) bands in their stacked column graphs.
The categories with the largest “Selected without major change” bands that also have relatively large “Not selected without major change” (white) bands
Use the “Unnecessary Diagnostics“ tab in the School Verification Graphs template to see the results of all the results in a single place
See the 2008 QA Pre-Conference Materials for details. (Click here and view #6 for complete instructions)
Words of caution:
Do not use results based on the analysis of five or fewer cases.
Make sure that you have at least some records loaded in the Tool that were NOT selected for school verification and that some of these records did not experience a “major change.”
Make sure that you have some dependent records with and without a “major change” loaded in the Tool.
If you failed to meet either of the two conditions above, make the necessary adjustments to the output of the shared report as specified in the 2008 QA Pre-Conference Materials. (Click here and view #7 for complete instructions)

Improve the accuracy of initial FAFSA information though consumer information or educational outreach

All the main dishes available here feature the same main ingredient - the Field Change Report - but there are multiple ways that this “dish” can serve as the basis for a satisfying meal.

Use the Field Change Report

Preparation options
Run from the domain tree.(Click here and view #8 for complete instructions)
Drill down from the Verification Summary Report to focus on a specific subset of applicants.
The chef recommends:
Using the filter screen or Verification Summary Report to focus exclusively on either dependent or independent applicants(Click here and view #9 for complete instructions)
Focusing on critical ISIR fields - the fields that enter into the calculation of EFC
Focusing on relatively few (five or less) ISIR fields per dependency model in your analysis
What to look for:
High percentages of records corrected associated with increases and decreases in EFC, especially if they are associated with Pell changes
Problematic ISIR fields with low percentages currently selected for verification.
Selected side dishes:
Drill down to the Field Increment Report to see if there are value ranges of specific ISIR elements where you can target outreach efforts.(Click here and view #10 for complete instructions)
Drill down to the Pell Eligibility by Dependency Status Report to see if Pell and dependency status are related to applicants needing help in accurately reporting a specific ISIR field.(Click here and view #11 for complete instructions)
From the Pell Eligibility by Dependency Status Report drill down further to the Student Listing and Student Detail Report to better understand the full context of corrections to specific ISIR fields.(Click here and view #12 for complete instructions)
Words of caution:
Focus on corrections that are not just changes. Restrict analysis to records that have been verified, either because they are normally selected for verification or included in random sample subjected to 100 percent verification.
Be wary of results based on a small number of records - multiply the percent corrected by the report’s sample size to know how many cases you’re dealing with. The larger this number is the greater your confidence in the results should be.