Upload Guidance for DFIRM Submissions August 2008 OVERVIEW GENERAL GUIDELINES UPLOADING DATA OPTIONS 1. MIP Studies Workflow - Upload 2. Citrix 3. MIP Studies Workflow – Mail Data 4. DFIRM Tools FOLDER STRUCTURE AND INCLUDED FILES TIPS FOR FASTER PROCESSING OTHER TOOLS RESUBMISSIONS TRACKING SUBMISSIONS Changes since last version: Recommended file size limit for zipped submissions uploaded through the MIP (Submit Data Files portlet) increased from 300MB to 1GB (with a maximum on 8000 individual files). Users are still recommended to upload only in 300MB chunks if uploading through Citrix. Added bullet 2 to the General Guidelines section to clarify the definition of submission package Added bullet 10 to the General Guidelines section recommending to users to in the Final Submission step to first submit the DFIRM database for auto (without images), and when you receive a passing report for auto, then submit for visual (with the DFIRM data and the images). To do this, separate your submission and submit just your DFIRM data for Auto review (through the MIP or Citrix) and once you receive the passing auto report, then submit your passed DFIRM database AND panels (plot images) for visual validation (through the MIP, Citrix or to Data Depot, depending on the size of the submission). Additional reminder: Review the Folder Structure and Included Files section of this document as it may help reduce additional submissions to FAFS. OVERVIEW This document is intended to provide guidance on the options for uploading DFIRM submission packages to the MIP and for First American Federal Solutions (FAFS) review, including metadata profile and DFIRM database QA validation. Following these guidelines may lead to increased efficiency in receiving results from FAFS and improved MIP system performance. FAFS will no longer accept submission packages through eFTP or mailed directly to their office as the data in these submissions does not get stored on the MIP. There are four options for uploading submissions – 1. Upload completed submission package through the MIP portlet 2. Upload files to the J drive through Citrix 3. Mail data to the Data Depot 4. Use DFIRM Tools Each process is applicable for all DFIRM database-related tasks, including Develop DFIRM DB, Produce Preliminary Map Products, and Produce Final Map Products. Included in this document are the correct options for submitting packages, the process and timeframe for each, information on how to resubmit data and tracking your submissions. Also included is the correct folder structure and which files to include in each step of the workflow process. If you encounter a scenario that is not covered in the documented options, please contact your MIP Champion and/or Black Belt for assistance who will then open a MIP Help ticket if necessary. GENERAL GUIDELINES 1. Recommended file size limits for each option: Upload data through the MIP portlet (Submit Data Files): complete zipped submission packages smaller than 1GB, with fewer than 8000 individual files Upload through Citrix: upload individual files and/or zip files of no more than 300MB (at one time, in one upload submission), with the total submission package placed on the J drive less than 2GB Mail data to Data Depot: complete submission package over 2GB - Reminder: allow two weeks for mail and upload process DFIRM Tools: depends on the upload process, see DFIRM Tools section 2. A submission package is defined as all required files included in the zipped submission for submission to FAFS. 3. Plan ahead! Take into consideration your file submission size (to determine whether the submission can be submitted through the MIP or mailed to the Data Depot) and the turnaround times for FAFS review and plan time into your schedule to allow ample time for the upload and FAFS process. 4. Do not submit your data for upload until your submission package is complete, whether you are using the MIP Submit Data Files portlet or clicking Validate Contents when uploading through Citrix or the Data Depot. And do not resubmit any files, through any submission process, until results of the previous submission are returned. 5. Average turnaround times: Draft and Prelim Submissions: During certain periods, FAFS experiences a high volume of submissions for review and the turnaround time for an auto validation submission during the draft and prelim stages may be up to two full business days. During this time the MIP will read “In-Progress” in the Submission Status section. If the MIP still reads “In-Progress” after two business days, contact your MIP Black Belt. If necessary, they will contact MIP Help. Do not upload another submission package or click Validate Contents while the Submission Status section reads In-Progress. Final Submission: During a final submission, a visual inspection takes places after the auto validation. Due to the additional visual inspection, the turnaround time for a final submission may take up to 10 business days. During this time the MIP will read “In-Progress”. If the MIP still reads “In-Progress” after the 10th business day, contact your MIP Black Belt. If necessary, they will contact MIP Help. Do not upload another submission package or click Validate Contents while the Submission Status section reads In-Progress. 6. Do not upload individual files using the MIP Submit Data Files portlet. Each time you add a file through the MIP, the file(s) is automatically sent to FAFS for review. Incomplete submission packages result in a failure to pass automated validation. This also causes version control problems at FAFS as their system does not necessarily process items in the order they were submitted. Multiple submissions can overwhelm the FAFS system, resulting in delays for all mapping partners. Recommendation: wait until the data submission package is complete before submitting for FAFS review. Zip the files into one submission package using the .zip file extension. 7. Each time you upload/re-upload a file through the MIP portlet (Submit Data Files), it kicks off the validation process. Validation is not started in the other options (DFIRM Tools, Citrix or mail data) until the user returns to the MIP and clicks Validate Contents. 8. Once files are placed on the MIP, by any of the four process options, and validation is started, DFIRM DB QA validation will be performed, as well as the automatic metadata validation (the two validations are performed concurrently). 9. In certain circumstances, metadata can pass validation but fail on QA (the two validations use the metadata file for different purposes). The user must update the metadata file which requires another metadata validation and QA review, as any change in the submission requires the entire submission package to be resubmitted for review. For faster processing, wait until both Metadata and DFIRM DB QA to change from In-Progress to pass/fail before resubmitting. 10. To speed processing in the Final Submission step, first submit for auto (without your images), and when you receive a passing report for auto, then submit for visual (with the DFIRM data and the images). To do this, separate your submission and submit just your DFIRM data for Auto review (through the MIP or Citrix) and once you receive the passing auto report, then submit your submission AND panels (plot images) for visual validation (through the MIP, Citrix or to Data Depot, depending on the size of the submission). 11. Any changes to the J drive will change the submission from the current status (In-Progress, Passed, etc) to Data Revised. This indicates to the MIP that a file(s) has been altered and requires the user to click Validate Contents to send the submission to FAFS for review. 12. If you have corrections to make to only certain files, it is not necessary to upload all files again, only the files that have the corrections. Make sure these files have the same file name as the incorrect submissions and when you upload the new files, they will overwrite the old files. 13. The user who performed the upload or clicked Validate Contents to start the validation process will receive an email notification when the DFIRM QA validation has been completed. 14. If you have a submission currently in FAFS review, do not submit another submission package until the previous one is returned, as it may have errors that need to be corrected. Once you have corrected those errors, resubmit the package, including any new files you may have created/modified. 15. Both the Metadata Profile Validation and DFIRM DB QA Validation fields in the Submission Status section of the task must read Passed before the user can complete the task. 16. Click on the links in the Submission Status section (i.e., Passed, Failed) to view the validation report. 17. Data is copied from the J to K drive for final storage when the user completes the task on the MIP (i.e, clicks Complete Task on the Review screen). UPLOADING DATA OPTIONS 1. MIP Studies Workflow - Upload Highlights: Use the MIP Submit Data Files portlet to upload: - Once your complete submission package (all files to be submitted to FAFS) is assembled and ready for validation - If your zipped file submission package is 1GB or smaller, with fewer than 8000 individual files The validation process automatically begins when you click the Upload button from the Submit Data Files portlet in the MIP workflow. Do not submit your data for upload until your submission package is complete. Incomplete submission packages result in a failure to pass automated validation. This also causes version control problems at FAFS as their system does not necessarily process items in the order they were submitted. Multiple submissions can overwhelm the FAFS system, resulting in delays for all mapping partners. Do not add or updates files to the submission package until you receive the results from FAFS regarding your previous submission. Uploading new and/or modified files through the MIP Submit Data Files portlet triggers a new submission to FAFS. If resubmitting files, only upload the files that have any edits, not the entire submission package. If named the same, the new files will overwrite the existing files and automatically be sent to FAFS for review. Process: - Use the Submit Data Files button on the MIP to upload your completed submission package. This button is found by opening the Submit Data twisty. - When the project is created, the MIP automatically creates a path to where your files will go on the J drive, as indicated in the Submit Data area. - This path is not a link to anything, but indicates where your files will be uploaded. - If the submission you are uploading through the MIP is already on the J drive (in a different location), the system will move your files to this new folder on the J drive. - The Internet Connection Speed and File Size fields on the data submission window are optional; used to calculate the approximate amount of time the files will take to upload. - Users can navigate/browse to any drive (including the J or K drive) to find files to upload the submission. - Do not upload your submission until you have assembled the entire submission package (all files to be submitted to FAFS). - The MIP automatically unzips your files once you upload through this process. Depending on the size of the submission and the number of files, this may take a few minutes. - Automatic validation is performed after contents are uploaded and unzipped. - Do not make any changes until you receive the results from the previous submission since making changes triggers a new submission. - Once the user receives a passing report for both Metadata Profile Validation and DFIRM DB QA Validation, click Complete Task. This will copy the files from the J drive to the K drive and allow the user to move forward in the MIP Study workflow. 2. Citrix Highlights: Use Citrix to upload files individually and/or parts of the submission package as they become ready. Upload individual files files of no more than 300MB (at one time, in one upload submission), with the total submission package placed on the J drive less than 2GB. Do not add subfolders in the directory structure. Unlike the MIP workflow process for uploading, validation does not start automatically. The user must return to the MIP workflow step and click Validate Contents to kick off the validation. Do not click Validate Contents in the MIP workflow until your complete submission is ready for review. Process: - Upload files (in submissions of no more than 300 MB) from your local drive through Citrix to the correct tasksysid folder on the J drive - Once ALL files are ready for review (i.e, the submission package is complete), return to the MIP workflow step and click Validate Contents to start the validation process. - Even if you have modified your submission, do not click Validate Contents again until you receive a result for the previous submission. - Once the user receives a passing report for both Metadata Profile Validation and DFIRM DB QA Validation, click Complete Task. This will copy the files from the J drive to the K drive and allow the user to move forward in the MIP Study workflow. 3. MIP Studies Workflow – Mail Data Highlights: Mail submissions that are larger than 2GB to the Data Depot. It typically takes two weeks from the time you mail the data to the Data Depot to the time that data is uploaded into the MIP. If you have not received notification of the upload within 10 business days of mailing your submission, please contact MIP Help to determine the progress on your submittal. Allow sufficient time in your schedule to accommodate the mail and upload process for these large file submissions. Process: - Users who mail submissions to the Data Depot must still complete the task in the workflow. - To indicate that files are being mailed, click on the Submit Data Files button from the Submit Data section, click the Mail the Data radio button and click Submit to automatically create a MIP Help ticket and cc the user indicating the data will be mailed. The Data Depot address and all required information are displayed on this screen. Users should never send to the Data Depot without completing this information on the MIP. - Don’t forget to mail the submission! - Once the mailed data is received and uploaded to the MIP, MIP Help will send an email notification to the user to inform them that their data has been uploaded. - Unlike uploading through the MIP, the validation process for mailed submissions is not automatic. Users must click the Validate Contents button to start the validation process. - Once the user receives a passing report for both Metadata Profile Validation and DFIRM DB QA Validation, click Complete Task to copy the files from the J drive to the K drive. 4. DFIRM Tools Highlights: Use DFIRM tools to upload your submission when you have used the DFIRM Tools to create your files. Remember, after you upload through DFIRM Tools, you must return to the MIP workflow to start the validation process. The DFIRM Tools do not link directly to the DFIRM QA database. Process: - Once the files are uploaded in DFIRM Tools, click Export. This places the submission on the J drive. - Users must convert the .txt metadata files into .xml to be included in the submission package. - The user must return to the MIP workflow step and click Validate Contents to kick off the validation. - Do not click Validate Contents in the MIP workflow until your complete submission is ready for review. - Once the user receives a passing report for both Metadata Profile Validation and DFIRM DB QA Validation, click Complete Task to copy the files from the J drive to the K drive. FOLDER STRUCTURE AND INCLUDED FILES When submitting for FAFS auto and/or visual review only submit the required files and in the proper file structure, as outlined below. Do not include subfolders within the zipped file that you submit. This will reduce excess overload on the FAFS system and speed processing. DRAFT SUBMISSION: Required files: .e00 files and DFIRM database metadata files (*.xml and *txt) - Do not include other metadata files - The .xml file is required, but .txt can be included as may be used if the .xml file is corrupt When preparing a valid submission to FAFS, verify that your submission is located in the following folder: J:\FEMA\[region]\[state]\[county]\[county or community]\[FEMA Case Number]\SubmissionUpload\Mapping.Draft_DFIRM_DB\[task system ID] PRELIM SUBMISSION: Required files: e00 files and metadata files (*.xml and *txt) When preparing a valid submission to FAFS, verify that your submission is located in the following folder: J:\FEMA\[region]\[state]\[county]\[county or community]\[FEMA Case Number]\SubmissionUpload\Mapping.Prelimanry_DFIRM_DB\[task system ID] FINAL SUBMISSION: Required files for Task Sys ID folder: - SHP, SHX, DBF - e00 - MIF, MID Required files for RFIRM folder: - Ortho-base: PNG, PGW - Vector: TIF, TFW When preparing a valid submission to FAFS, verify that your submission is located in the following folder: J:\FEMA\[region]\[state]\[county]\[county or community]\[FEMA Case Number]\SubmissionUpload\Mapping.Final_DFIRM_DB\[task system ID]. TIPS FOR FASTER PROCESSING The following are some examples as to how to improve processing time for your submission. Before you upload your submission, remember the following: The following are some examples as to how to improve processing time for your submission. Before you upload your submission, remember the following: 1. Do not upload individual files using the MIP Submit Data Files portlet. Each time you add a file through the MIP, the file(s) is automatically sent to FAFS for review. Adding individual files fills up the FAFS queue, thereby taking longer to retrieve and review the data. Recommendation: wait until the data submission package (all files to be submitted to FAFS) is complete before uploading through the MIP portlet. 2. If you are more comfortable uploading files individually, use Citrix to upload. But wait until your entire package is ready for FAFS review before returning to the MIP to click Validate Contents to start the validation process. 3. To speed processing in the Final Submission step, first submit your DFIRM database for auto (without your images), and when you receive a passing report for auto, then submit for visual (with the DFIRM data and the images). To do this, separate your submission and submit just your DFIRM data for Auto review (through the MIP or Citrix) and once you receive the passing auto report, then submit your DFIRM database AND panels (plot images) for visual validation (through the MIP, Citrix or to Data Depot, depending on the size of the submission). 4. Confirm that your XML metadata files are readable. FAFS cannot review unreadable XML metadata files and the user must be contacted to fix the file and resend it, thereby creating a delay in receiving results. 5. Only include the DFIRM database metadata file; do not include the base map metadata file or metadata for the individual layers. If included, FAFS must remove the extra files and load it again manually which may delay the review process. 6. Format projection information correctly. State plane submissions are supposed to define the grid system as “State Plane Coordinate System 1983”. Often users will use some other phrasing, which causes the FAFS system to stop processing the study. Users must be notified to correct the issue which may delay the review process. 7. Check for typos in included tables. The FAFS system checks the metadata to determine which DFIRM layers are supposed to be in the submission. It looks for the string “The following tables are included in this data set:” or “The following tables may be included in this data set:”. If this phrase has a typo, the system will halt. 8. Metadata files should be included in the data submission as a separate file in the root directory. An .xml Metadata file is still required. However, text files can be included in the submission folder. Metaman will ignore the .txt file, and validate the .xml file. 9. The FIS folder is no longer created. FIS submissions should be loaded into the appropriate folder within the DFIRM submission. Tools & Links – Tools and Links should only be used as a tool for an initial, “practice” DFIRM DB QA validation for auto review. This allows Mapping Partners to see what errors they need to fix, if any, prior to submitting the DFIRM to FAFS for review. – Uploading DFIRM submissions through Tools and Links does not place data on the J drive, it does not create any records in the MIP, nor does it validate metadata. – The user only receives an email notification of a passing or failed report. – FAFS is not funded to perform “practice” visual reviews. The user must perform submit for this review at the appropriate workflow step. Mail Data or eFTP Directly to FAFS - Users should not submit any files directly to FAFS. – There is no link between FAFS and the MIP if files are submitted outside of the MIP; no data is stored on the MIP, nor can a user progress through the MIP workflow. – This guidance supersedes any other guidance for providing data directly to FAFS. RESUBMISSIONS - Any change to the submission package requires the submission to be validated again. - If you have a submission currently in FAFS review, do not submit another submission package until the previous one is returned, as it may have errors that need to be corrected. Once you have corrected those errors, resubmit the package, including any new files you may have created/modified. - You can use any of the four process options to resubmit your submission, depending on the size of your submission package. - If you have corrections to make to only certain files, it is not necessary to upload the entire submission package again, only the files that have the corrections. Make sure these files have the same file name as the incorrect submissions and when you upload the new files, they will overwrite the old files. TRACKING SUBMISSIONS The MIP displays the progress of the submission (Uploading, Unzipping, In-Progress, Passed, Failed) in the Submission Status section of the MIP workflow task. Uploading: Applicable for submissions uploaded through the MIP Submit Data Files portlet - Indicates the submission is being uploaded and will appear as long as the blue status submission bar is displayed. Unzipping: Applicable for submissions uploaded through the MIP Submit Data Files portlet - Indicates the MIP is unzipping the submission package. - Depending on the size of the submission, this process may take up to 30 minutes. In-Progress: Applicable for all submission options - Indicates the submission package has been sent to FAFS for review - Depending on FAFS’ queue, this may take up to two business days. Passed: Applicable for all submission options - Indicates a passing report from FAFS Failed: Applicable for all submission options - Indicates a failed report from FAFS - Users may click on the link provided for more information on the reasons for failure. Data Revised: Applicable for all submission options - Indicates some part of the submission has been changed and needs to be resubmitted for FAFS review - Click Validate Contents to start the validation process The user who started the validation process in the MIP will receive an email notification when the DFIRM validation is complete. In addition, the status of the DFIRM review in the MIP workflow will be updated with the pass/fail result and a link to the review report.