The Opportunity Gap: learn about the public schools in your neighborhood with @ProPublica and Foursquare
As our recent hackathon showed, there are tons of ways that developers can use Foursquare to power amazing apps – from ones that shame you into going to the gym, to others that alert you to restaurants with health code violations.
Today, ProPublica, an award-winning investigative journalism site, is relaunching its Opportunity Gap news application, which helps people find and compare statistics about public schools across the nation. With their new Foursquare integration, you can connect your Foursquare account to instantly see statistics for schools you’ve checked in to before. And when you’re out, you can instantly get stats about a school on your phone whenever you check in to one. It’s a great example of how news organizations can use Foursquare to reach their readers with relevant information when they’re out in the real world.
Learn more about The Opportunity Gap and connect your Foursquare account here.
A Foursquare Hackathon for the new year – sign up and build the next amazing hack!
And take two! Although Sandy foiled our hackathon plans in November, we’re back and ready to hack it up with the best and brightest. On January 5, we’re inviting developers and designers in NYC, SF, and everywhere around the world to sign up and build some more amazing hacks using the Foursquare API. We’ll have prizes, swag, and (naturally) global glory for the best Foursquare hacks, no matter where in the world they originate.
Head over to our Meetup page and keep an eye on hackathon.foursquare.com for more details as the event draws nearer. If you’re in SF or NYC, sign up to work from our HQ, otherwise, you can use Meetup to connect with hackers in your area.
(Fun fact: back in 2010, two designer/developer friends got together and entered the first Foursquare hackathon. They built a snazzy little hack that took people’s Foursquare check-in histories and resurfaced them in the form of a daily email that showcased people’s check-ins from exactly one year ago. They dubbed their hack, “4SquareAnd7YearsAgo,” and today you might know it as Timehop.)
Now, go sign up for the Foursquare Hackathon 2013!
MongoDB at Foursquare: From the the cloud to bare metal
We’ve been running Mongo as our main storage engine for almost 3 years. For most of that time the Mongo servers and all the rest of our infrastructure were hosted on Amazon’s EC2. We recently migrated the Mongo servers onto our own hardware hosted in a datacenter and now have a hybrid environment with everything else still on EC2. I’ll be talking about why and how we did this at the upcoming MongoSV conference on Tuesday, December 4th, 2012.
The name of the talk is “MongoDB at Foursquare: From the the cloud to bare metal.” Come check it out!
For more information about the event, check out the agenda or 10gen’s blog post Get Ready for MongoSV. Register now with the discount code “foursquare20” and receive 20% off.
- Jon Hoffman (@hoffrocket)
Even more ways to personalize Foursquare – check out the latest additions to our app gallery
Want to know if there’s a teaser after the movie credits, or the healthiest dish to order at a restaurant? There are a bunch of apps you can connect to your Foursquare account to help you make the most of your check-ins. Our gallery keeps growing – check out some of the latest additions:
- The Winester Square – it’s like Untappd for wine! Learn about the reds and whites on the menu after you check in.
- YOLO tells you the most expensive item on the menu.
- #mom helps you quickly let your mom know that you’re safe and sound with a call or text when you check in.
- GeoPollster will tell you which political party the businesses you check in at support.
To connect these and other apps to your Foursquare account, go to our app gallery at Foursquare.com/apps or the settings screen in the Foursquare app on your phone. Got an idea for an app? Build one (and win prizes!) at our upcoming Global Hackathon on November 3.
Open-sourcing our dashboard for Apache Oozie
At Foursquare, we use Apache Oozie to manage the scheduling and control of our offline data processing workflows. We’ve had great success with the project, and we run upwards of 1000 Oozie workflows per day.
Despite the quality of Oozie’s core workflow engine, the web UI is a little clunky, and is franky unusable in a lot of circumstances, especially when you’re using it at a moderate scale.
- Unique URL’s for coordinators, and workflows
- Proper ordering of coordinator / workflow actions
- Syntax highlighting of job definition and configuration files
- Coordinator actions link to their corresponding workflows
- Workflow actions link to their corresponding hadoop jobs
- Re-run failed coordinator actions with a single click.
- A better search implementation that matches substrings in workflow names
We’ve been using Oozie Web internally for a couple of months now, so we figured it was about time to make the project open-source and give back to the community. We’re releasing the project under the Apache 2.0 license, and it’s available right now on github: http://github.com/foursquare/oozie-web
- Joe Crobak (@joecrobak), Joe Ennever (@TDJoe), and Matthew Rathbone (@rathboma)
Big Data and the Big Apple: Understanding New York City using Millions of Check-ins
Last Friday, I spoke at DataGotham about how foursquare data can provide an unprecedented view into the behavior of cities. In this talk, I focus on what we can learn about New York City from aggregating the check-ins of millions of New Yorkers, and demonstrate tools that we are building to help make cities, like New York, easier to use.
Connect all the apps! Dig into our new connected apps platform at our 2012 hackathon
You guys! The foursquare hackathon is back! And this year, we’re inviting you to hack foursquare itself. Our all-new Connected Apps platform enables you to wire new functionality directly into the core foursquare experience and make your app discoverable to the over 20 million people already on foursquare.
On November 3, we’ll open up our NYC & SF offices for an all-day Connected Apps hackathon. And like last year, we’re inviting developers all around the world to band together and hack up some awesome connected apps. We’ll have great prizes (last year one of the winning teams had their hack running on a giant screen in Times Square and even rang the closing bell for NASDAQ! Oh, and there’s a new title belt!), swag, and (naturally) global glory for the best foursquare hacks, no matter where in the world they come from.
To sign up, head over to our Meetup page and keep an eye on hackathon.foursquare.com for more details as the event draws nearer.
Ready? Set? Connect all the apps!
Developers, grow your numbers! Connected app success stories from our pilot partners
We’ve seen some amazing new apps built since we released the developer preview of our new connected apps framework two months ago. We’re continuing to make improvements, and we can’t wait for the official launch of our App Gallery to our entire community in the coming months.
Connected apps are great for both developers and the foursquare community. For people checking in, connected apps make foursquare more interesting and customized. For developers, connected apps provide an opportunity for more people to discover and engage with your timely and targeted content.
The connected apps effect
Our connected apps launch partners have seen significant jumps in sign-ups and downloads, increased engagement, greater exposure to new audiences, and spikes in web traffic. Here’s what some of them had to say:
Snoball
The Snoball connected app lets people set-up a donation to trigger when they check in to a specific place on foursquare. For example, a person can pledge $1 to charity:water when they check in to their favorite coffee house.
“Since the app’s July launch, the number of check in-based donations within Snoball has increased by 500%, while nonprofits within the platform saw a 40% increase in donations through Snoball with an average donation of $2 per check-in.” – Todd Graves, Software Developer at Snoball
Untappd
Untappd gives beer recommendations to people who check in to a bar and makes it easy to track the beers they drink.
“Through the foursquare connected app, we saw a 10% increase in sign-ups for Untappd and that 10% of all replies sent to the user turned into a beer check-in. Most importantly, our users have told us how much they love getting great recommendations about beer choices while at their favorite establishments, without leaving the foursquare app.” – Greg Avola, Co-Founder of Untappd
Eat This, Not That
Eat This, Not That recommends the healthiest dishes to order when people check in to restaurants.
“Our goal in this partnership was to deliver quality information that is relevant to our viewers when they need it most. We provided the editorial content and foursquare provided the technology… Viewers are literally sitting at these restaurants, checking in, and looking at a list of menu options (usually without calorie counts); serving them helpful tips on what to order (and what to stay away from) is not only timely, but extremely useful.
We plan on launching a second connected app this year with Women’s Health, which will deliver content to its health-conscious readers in the nutrition/fitness category.” – Jackie Lebowitz, Special Projects Editor at Men’s Health
Sonar
Sonar lets users know when their friends and other interesting people who share interests and connections are nearby.
“The impact of being a part of foursquare’s connected apps pilot has been fantastic. We’ve been able to attribute increases across all the standard metrics we track daily to our presence within foursquare, including downloads and engagement. On some days we’ve seen downloads from foursquare as much as doubling our daily download numbers, and we’ve seen a 6000% increase in website traffic sent from foursquare. We’ve also noticed a jump in downloads in new markets where foursquare has a high penetration, including Turkey, Russia and Brazil.” – Katie Smith-Adair, Vice President at Sonar
Want to build a connected app, or have one you’d like us to feature? Learn more or submit your app to be featured in our Connected Apps Gallery.
Checking in at the movies? Here’s how to never miss an after-the-credits teaser again.
The latest addition to the connected apps gallery is a perfect example of how a little bit of information at the right time can make all the difference. Now when you check in to a movie on foursquare, Mantas Vidutis’s After Credits can let you know if there’s something worth sticking around for after the credits roll.
Here’s how to never miss a teaser, blooper reel, or Avengers shawarma-eating-sesh ever again:
1. Click here to connect the After Credits app to your foursquare account. You can also find it in our app gallery online or from the settings page in the foursquare app.
2. The next time you’re at the movies, be sure to check in on foursquare with the flick you’re seeing.
3. After you check in, scroll down to see messages from your connected apps. After Credits will let you know if you should sit tight for something awesome! You can tap on the message for more details.
To connect more apps that add information to your check-ins (like the healthiest dish to order, The Weather Channel forecast, or interesting people you may know nearby), check out our app gallery. We’ll be adding our favorite new apps here as we lead up to our full release in a few months.
Developers, want to build a connected app, or have one you’d like us to feature? Learn more or submit your app.
foursquare now uses Elastic Search (and on a related note: Slashem also works with Elastic Search)!
Historically venue search used to be done in Mongo at foursquare; we later migrated it to Solr and now we are using Elastic Search. While Solr has been great (and we continue to use it for users and to-dos) we want to make sure that we can keep up with the growth of our venues database as our wonderful users keep adding more places.
Elastic Search provides a really nice way to shard search data and manage the related problems around this. By sharding search data we are able to split it up the documents into groups which can easily fit on a single machine and then get the results back from a collection of machines. Elastic Search and Solr have quite a few similarities that come from both being built on top of lucene. The Elastic Search team has been super helpful during the migration process, providing ideas on how to tweak our queries to get better performance out of Elastic Search. We got a non-trivial performance improvement by switching the field types for our geohash to allow us to skip the analyzer. One of the biggest improvements for performance we found was paramaterazing our ranking scripts to avoid the per-search compilation over-head. We also saw an improvement when using a custom scoring plugin (written in scala.)
By far the largest improvement came from switching the type of search we were asking Elastic Search to do. When you are doing a search for “blue bottle coffee shop” you probably care more about the terms “blue bottle” matching then the term “coffee shop” matching, since blue bottle doesn’t show up very often in comparison to “coffee shop”. By switching the search type from DFS_QUERY_THEN_FETCH to QUERY_THEN_FETCH we use the term frequencies in each shard rather than getting a global set of term frequencies. This works really well for us since our venues are fairly uniformly distributed between the shards, but might not work so well if that isn’t the case for your data.
The migration from Solr to Elastic Search has really emphasized the usefulness of having easy to update throttles. The throttles let us switch code paths in our application for different groups of users without having to do deploy. As with deploying any large piece of infrastructure, there were a few hiccups during the roll-out* of Elastic Search and throttles helped us minimize the impact on end users.
As we’ve previously written about, the majority of our search queries are written using Slashem, an in-house DSL for querying search backends. Rather than re-write all of our queries for Elastic Search we instead updated Slashem to generate queries for Elastic Search as well as Solr. The support for Elastic Search in Slashem is not as comprehensive as the Solr support, although its pretty good for basic searching at this point. A lot of the query generation is still fairly naive, but we are working on improving this. As always patches are welcome 
.
Cheers & Hugs,
- @holdenkarau & @adamalix
* Although #sfsearch has normally listened to levels for our deploys.
2 Comments