Quantitative Analysis of NYC Open Data: Every data set that the city releases tells a story. This blog is all about telling those stories, one data set at a time.
If there is one thing I can get behind, it is using data to make better policy decisions here in NYC. So when I saw the following tweet by the 5th precinct in Manhattan, I was intrigued:
The 5th Pct. had a 115% increase in bicycle collisions this year Police Officers will be enforcing violations by bicyclists.#bike safety
Now we could argue all day if ticketing cyclist in response to more collisions is the right thing to do here, and you will find people on both sides of the argument. A quick browse through the Twitter responses shows what many people thought. And of course the efficacy of data-driven decisions like this are a difficult thing to measure. That’s not a problem in itself because good policy does not always need to have data to back it up. But as I teach my students at Pratt, when data is being used to support a decision, it should be done with caution.
One thing that is not hard to measure is whether there really was a 115% increase in bicycle collisions. That number seemed astoundingly high to me. More than double the number of collisions from last year would indicate something has drastically changed in the neighborhood.
I am a big fan of posting raw data and methodology along with statistical claims, as “reproducibility” is a central core to the scientific method. Unfortunately, we don’t see that same philosophy being adhered to by the precinct, so I am left guessing. Even so, I took the liberty of dumping the raw collision data out of the city’s Open Data portal to put the claim to the test.
It turns out that VisionZero collision data provided by the city lacks precinct information, so I intersected that data with a police precinct spatial layer using QGIS. The following map shows the 5th precincts 2013 cycling collisions in red and 2014 cycling collisions in orange . (Both data sets are through November 21st of that year and 2013 circles are slightly large.)
At first glance, I see a lot of red and a lot of orange, but no obvious surge. So with the data in hand, I compared the cumulative monthly counts of all collisions involving bicycles in the 5th precinct from January 1st - November 21st, for both 2013 and 2014, in search of this massive uptick this year. The results:
It turns out that 2013 had more collisions Year-to-date than 2014, so that must not have been what they were talking about.
2013: 86, 2014: 83
Maybe they were talking only about collisions where pedestrians were injured, since that is usually the cycle ticketing rationale:
2013: 1, 2014: 0
Maybe cyclist accidents that caused deaths, since the aim of VisionZero is to drop that number to 0?
2013: 0, 2014: 0
OK, next try. Cyclist accidents that injured drivers:
2013: 8, 2014: 2
Last one. Number of cyclist collisions where no cars were involved, since this tweet focusses on cyclists, not drivers:
2013: 0, 2014: 1
Ah ha! One reported collision that did not involve an automobile in 2014, vs zero in 2013. A number where 2014 shows an uptick. And in fact it’s an infinite percent increase.
But seriously, what gives? Well, I have absolutely no idea. Perhaps I’m missing something here and that there really is some underlying data that supports the NYPD’s assertion. (I’d be able to measure the ticket uptick as well if summons data was released publicly on the portal, but we are still waiting on that…) Data nerds: if you run the numbers yourself, and reach different conclusions, please let me know.
My ask here from the NYPD? Please, when using open data to make policy decisions, give us a hint about the underlying data and methods. Data driven enforcement decisions can have a very big impact on the citizens of this city, and if you are going to use data (which is great by the way), at the very least we deserve an explanation of where the statistic came from.
If you are a New York City local, there is a reasonably good chance that you have not picked up a payphone in the last decade or so. And for those of you who have, you probably knew there was an even better chance that the phone you picked up was going to be out of service.
Useless, right? Well, not for long. This week the city announced that all of those pay phones (with the exception of two) will become free wifi hot spots! Yes, for real.
Now I know what your next thought is. ”But wait, there are not a lot of pay phones left in NYC.” Believe it or not though, there are a LOT of pay phones left in NYC. A lot lot. Don’t believe me? Check out this map courtesy of data from the NYC Open Data portal:
So lots of phones that will soon be free wifi hotspots. We also learned that each of these hotspots will broadcast wifi for 150 feet. So, I took all of the phones in the data set, added 150 feet buffers around them and made the cities very first free wifi coverage map:
The 150 foot radius covers an impressive amount of many neighborhoods.
While this represents about 6,000 phone locations, the NY Times reported that they will be adding at least 1,600 additional locations. So where should those locations be? I decided to explore which neighborhoods have the best and worst coverage given the current phones to identify where additional locations would add the most value. To do that, I calculated the total area with wifi coverage in each neighborhood as a fraction of the total area of the neighborhood. The top findings are below. (You can search any neighborhood in the table using the filter on the upper left):
The winner? Kips Bay with a whopping 65% coverage. In fact, 9 manhattan neighborhoods have so many pay phones that they have over 50% of their area covered. As the map shows, the coverage is definitely strongest in Manhattan. In fact, the first 38 neighborhoods on the list are all from the borough. Its not till 39 and 40 on the list where we see the outer boroughs: Downtown Brooklyn with 16% coverage and , and Ridgewood with 15%.
The borough breakdown is pretty interesting as well: 55% of phones are in Manhattan, 18% are in Brooklyn, 17% Queens, 10% Bronx and less than 1% for Staten Island.
If these kiosks are mainly aiming to help tourists, there are some notable tourist destinations that could use a boost. Battery Park has only 12% coverage from 14 phones. Central park fares the worst in Manhattan, with less than 1% coverage from only 7 phones in the whole park. In Brooklyn, one up-and-coming tourist destination fares particularly poorly: Red Hood with only 0.2% coverage form a single phone.
If they are hoping to offer internet for lower income communities with this program, once again Red Hook notably scores pretty poorly. Cypress Hills gets 1% in Brooklyn, and Windsor Terrace has a single phone providing < 1% coverage. In fact there were 50 neighborhoods with out any phones in the list, giving 0% coverage. Those might be a good place to start when placing additional units.
Still want to make a phone call though? Don’t worry, you’ll be able to do that too… for FREE again. The catch? It will be through a speaker phone. So just don’t say anything you wouldn’t want a whole city block to hear. Though in this city, that’s probably not very limiting.
—-
This post was done in response to a Twitter Request by @BrianYoung88.
There was a time, not that long ago, when you had to actually think to find an address. Yes, a time before iPhones or even MapQuest would send you on your way without any thinking.
So given an address, how do know what cross street or avenue an address is on?
There is of course a method to address numbers in Manhattan. Some things most New Yorkers know: Numbers get higher as you head towards the waterways. Numbers start from 1 at 5th Avenue. Numbers get higher as you head North.
Still, I was curious how this all looked if you mapped it out. Could a map help add some clarity to the system?
So I turned once again to the wonderful NYC Open Data set PLUTO to map out these phenomena.
The map below allows you to make a bit more sense out of the system. Higher address numbers are darker and lower address numbers are lighter. The lightest color are for addresses 1-100, the next is 101-200, etc. The darkest color (of the 7) is for 601 and greater.
Some findings:
The street with the highest address number (5365) is no surprise; it is also the longest street: Broadway. And it shows in the map, as the lots around it get darker and darker.
Zooming in on neighborhoods like the Upper East side, one can see the buildings that are addressed on the avenues versus those on the side streets.
The Upper West Side avenues all change names and reset at 59th Street, which is shown in the map by the sudden lightening of the neighborhood.
Frederick Douglas and Adam C Powell Blvd seem to start at 2040 and 1800 respectively, right at the top of Central Park, as opposed to 1.
Once above 14th street, most blocks are split into 100s. It’s especially easy to see on the Upper East Side.
I also wondered what the most common address numbers were, so I made a histogram counting the number of addresses 1 - 100:
It turns out that 15 is the most common address number in Manhattan, with 179 distinct addresses. The plot also shows that numbers cluster around 5s and 10s more often then other numbers. There is also a sudden drop in unlucky 13 on the chart! I knew it affected elevators, but never knew it affected addresses.
Zooming out a bit, we can see addresses 1 - 2000 counts, bucketed by intervals of size 10:
We see here that numbers spike up at the 100s and work their way down. So an address ending in 99 is more rare.
In the end, I imagine that this map is in most NYC Taxi Drivers’ heads. So it’s probably nothing new. (Though they don’t seem to have to know this kind of thing for the TLC taxi test.) Nonetheless, I feel I can find my way around Manhattan just a little bit better now that I’ve seen it. Thanks Open Data!
—
-Lot Data Provided by NYC Open Data here -Maps Made in QGIS and exported to cartodb via GeoJSON. -Histograms made in IPython
Last week I found the farthest apartment from a subway in Manhattan. It was a mere 0.7 miles from the nearest subway entrance.
After that post, I got several requests that I do the same in my very own home borough of Brooklyn. So I joined building lot data, neighborhood name data and subway entrance data to create a subway distance map of Brooklyn. The result is below:
Darker areas are farther from subway entrances. The ranking of all Brooklyn neighborhoods based on subway distance is below:
The five neighborhoods whose centers are farthest from the subway are Marine Park, Sea Gate, Bergen Beach, Mill Basin, and Mill Island. (Note that they all have nautical terms.)
And the home that is farthest from a subway in Brooklyn is… 2336 National Drive in Mill Basin:
Google Street View
The property, which is on the left side of this picture, was described in a real estate listing as a “Fisherman’s dream” and sold for $860,000 in 2012. It is about 2.2 miles from the nearest subway as the pigeon flies. (It’s a bit longer if you have to stick to the road).
Google Maps
The best part? The property has its own dock. So while you may be far from the subway, you might just be a quick ride from the Statue of Liberty.
——
Lot data comes from Pluto Neighborhood Location/Name data comes from Bytes of Big Apple Subway Entrance data comes form the Open Data Portal Graph made using QGIS.
Huge Shameless Plug: Hi Internet. In addition to this work, I help direct and perform in improv comedy shows for people undergoing chemo with a non-profit called Cherub Improv. We also do improv-based job training programs. We have a gala coming up. If you want to support our cause, it would mean the world to me: Donate any amount here!
The phrase public housing often conjures up images of the large complexes built throughout many of the countries most urban areas. Indeed, public housing is a big part of the social fabric of large cities like New York and Chicago, but it actually appears in all 50 states, Washington D.C., and even three American Territories. And as it turns out, about 20% of our nation’s public housing stock is located outside of urban areas. Regardless of where the housing is located or the local cost of living, tenants pay rent capped at about 30% of their income, making housing more affordable for the over two million American’s residing in these properties.
This got me thinking: are there areas of the country with significantly better/worse public housing options?
Mailing List
