We're getting ready to implement some internal changes that will effect the information we provide in files on our FTP server. Before we throw the switch we want to explain the biggest differences you'll see initially and invite you to let us know if these will cause real problems for you and any ideas you have to overcome those problems.
What we're doing first is changing the source for information in the individual contributions file(s) (e.g. indiv10.zip; add10.zip; chg10.zip; delete10.zip), the PAC/Party contributions file (e.g. pas210.zip); and the miscellaneous transactions file (e.g. oth10.zip). These are created now from the old database system we have maintained for MANY years. The model for creating this file was based on our receipt of paper reports (amazingly, this still happens for Senate campaign committees. . . ). When committees filed amendments to paper reports they did so in several different ways - sometimes just giving us information that was new or changed from the original report. As a result, the transactions in the data file would continue to come from the original filing but any changes from an amendment would be manually identified by our staff and the original records changed to reflect the amendment. This is a very time consuming manual process, and with the advent of electronic filing where any amendment must completely replace what came before, we were able to develop a different process that could be automated much more easily.
So, in the new data structures which assume electronic filing, the files that are comparable to the FTP files contain the latest version of all of the transactions - i.e. those entries that appear on the most recent amendment to any report. This change has a couple of consequences that might be important to you, depending on how you use the data.
First, the image numbers for many of the transactions will be different in the new file from those appearing in the old file. This is because the old file often points to the image of the original report and the new file points to the most recent amendment.
Second, the FEC record number that appears in the old file can't be reproduced in the new. There will be a different unique identifier in the new file but since many of the new records are different from the old version (in most cases the only difference is the filing from which they are taken, not the substantive information about the transaction) there isn't any way to directly connect the old identifier to the new.
This may mean that any matching of old and new records will require comparisons of all of the fields except the record number and image number.
We're going to have a meeting to talk about these changes with our developers and anyone from the outside who's interested in participating. It is scheduled for Tuesday, February 15 at 4 pm here at the FEC. If you'd like to attend send me an email or leave a comment here. I'll report back on how we decide to proceed.
Posted by Coulter Jones on February 10, 2011 at 01:01 PM EST #