Data Download Usage Guide
We often receive requests from users interested in downloading larger segments of the data than are supported by the API. NREL has no means of sharing the data other than via these API endpoints. It is important to understand that these datasets are large. Currently our storage archive contains in excess of 7 terabytes of data! That number would be much larger if all of the data were extracted into CSV format. In order to reliably serve dynamic segments of a dataset of this size to a growing community of users we have calculated the maximum capacity our server hardware can sustain. We use this physical capacity to determine our API rate limits.
The API is restricted in both the number of requests a single user can make in a 24 hour period, as well as in the maximum size of a single request. The API rate limits are set at:
300 requests per day
1 request every 2 seconds
For downloading directly via CSV there are separate rate limits. This is due to the fact that it is only possible to download a single site for a single year via the CSV endpoint, therefor these requests are always quite small by comparison:
2000 requests per day
1 request per second
The size limit per each single request is determined by the number of total attributes in each request. The maximum weight of each request is 63072000. The calculation for determining the weight of each request is:
site-count*attribute-count*year-count*data-intervals-per-year
- site-count is derived from the WKT value submitted and can be retrieved using the site_count API endpoint.
- attribute-count is equal to the number of attributes requested
- year-count is equal to the number of years requested
- data-intervals-per-year is ((60/interval)*24*365) where interval is the interval requested
To maximize a single request simply minimize the variables wherever possible. For example by requesting half as many attributes one can request twice as many years (or sites) worth of data.
Imagine a use case where one wanted to download all Wind Toolkit data for the state of Texas. The first step would be to refine the request down to the least number of attributes, years, and intervals possible. From that one could determine the number of sites that could be requested in a single request. By experimenting with the site_count endpoint a polygon size could be identified that intersects close to that many sites. Using that polygon size create a grid that covers all of Texas. The final step would be to write a script that invokes the API once for every required grid cell at a rate of no more than 1 every 2 seconds. If more than 300 requests are required the script will have to be continued across multiple days.