I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out. Like this young buck, trying to get a potato to cry.
We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.
Here are a few of the more interesting experiments OkCupid has run.
Experiment 1: LOVE IS BLIND, OR SHOULD BE
OkCupid’s ten-year history has been the epitome of the old saying: two steps forward, one total fiasco. A while ago, we had the genius idea of an app that set up blind dates; we spent a year and a half on it, and it was gone from the app store in six months.
Of course, being geniuses, we chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013.
All our site metrics were way down during the “celebration”, for example:
But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos:
And it wasn’t that “looks weren’t important” to the users who’d chosen to stick around. When the photos were restored at 4PM, 2,200 people were in the middle of conversations that had started “blind”. Those conversations melted away. The goodness was gone, in fact worse than gone. It was like we’d turned on the bright lights at the bar at midnight.
This whole episode made me curious, so I went and looked up the data for the people who had actually used the blind date app. I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).
Oddly, it appears that having a better-looking blind date made women slightly less happy—my operating theory is that hotter guys were assholes more often. Anyhow, the fascinating thing is the online reaction of those exact same women was just as judgmental as everyone else’s:
Basically, people are exactly as shallow as their technology allows them to be.
Experiment 2: SO WHAT’S A PICTURE WORTH?
All dating sites let users rate profiles, and OkCupid’s original system gave people two separate scales for judging each other, “personality” and “looks.”
I found this old screenshot. The “loading” icon over the picture pretty much sums up our first four years. Anyhow, here’s the vote system:
Our thinking was that a person might not be classically gorgeous or handsome but could still be cool, and we wanted to recognize that, which just goes to show that when OkCupid started out, the only thing with more bugs than our HTML was our understanding of human nature.
Here’s some data I dug up from the backup tapes. Each dot here is a person. The two scores are within a half point of each other for 92% of the sample after just 25 votes (and that percentage approaches 100% as vote totals get higher).
In short, according to our users, “looks” and “personality” were the same thing, which of course makes perfect sense because, you know, this young female account holder, with a 99th percentile personality:
…and whose profile, by the way, contained no text, is just so obviously a really cool person to hang out and talk to and clutch driftwood with.
After we got rid of the two scales, and replaced it with just one, we ran a direct experiment to confirm our hunch—that people just look at the picture. We took a small sample of users and half the time we showed them, we hid their profile text. That generated two independent sets of scores for each profile, one score for “the picture and the text together” and one for “the picture alone.” Here’s how they compare. Again, each dot is a user. Essentially, the text is less than 10% of what people think of you.
So, your picture is worth that fabled thousand words, but your actual words are worth…almost nothing.
Experiment 3: THE POWER OF SUGGESTION
The ultimate question at OkCupid is, does this thing even work? By all our internal measures, the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on. But in the back of our minds, there’s always been the possibility: maybe it works just because we tell people it does. Maybe people just like each other because they think they’re supposed to? Like how Jay-Z still sells albums?
To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.
But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.
When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.
The four-message threshold is our internal measure for a real conversation. And though the data is noisier, this same “higher display means more success” pattern seems to hold when you look at contact information exchanges, too.
This got us worried—maybe our matching algorithm was just garbage and it’s only the power of suggestion that brings people together. So we tested things the other way, too: we told people who were actually good for each other, that they were bad, and watched what happened.
Here’s the whole scope of results (I’m using the odds of exchanging four messages number here):
As you can see, the ideal situation is the lower right: to both be told you’re a good match, and at the same time actually be one. OkCupid definitely works, but that’s not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth. Thus the career of someone like Doctor Oz, in a nutshell. And, of course, to some degree, mine.
Journalists, please write pressokcupid.com for more information.
The first two experiments are certainly A/B testing, but the third experiment is something entirely different. You’re no longer displaying something or not displaying something. You’re not even tweaking the weights of your regression equation. You’re just lying to people. Admittedly, this is pretty benign stuff, but I don’t think you should be so casual about it.
As someone who’s conducted benign psychology experiments with human participants, I always found the Institutional Review Board to be a necessary, though occasionally annoying, evil. You may want to look into establishing an independent group to review your more “out there” experiment plans. You need someone to push back on your “wouldn’t it be cool if we…” ideas. You need someone to say, “What controls do you have in place safeguard users…” At the very least you should have those involved in these experiments take a human experimentation ethics course. Hopefully that will help them better distinguish between A/B testing and experimentation that can actually have an impact in people’s lives.
Hello,
Found out about your site in 2006, Definitely one of your guinea pigs. Met 2 in person in that time. Still single and happy till the right one comes. Have hidden my profile a few times because I was tired of the games. Really you should consider astrology birth chart compatibility. Gemini and Pisces do not mix well for example.
Thanks for your great work in relationship compatibility!
Hi,
After not reading this entire article, I thought you might be interested to know about a system concocted by Robert Lee Camp. His books “Love Cards” and “Destiny Cards” are fairly accurate when trying to figure out how people match up energetically. All you need to know is the birth day . . not year. I have been using his “Love Cards” book for years just for fun and not always for romantic connections. I think you will enjoy it and maybe add it to your ways. I know I would pay attention to it.
Too much text to read ….
I’m locked out with no explanation and I like to know why. I feel a reply is necessary and would like one very soon like today. Ziggy90909
Now this is some interesting “Social engineering/experimentation”. I would like to know why Christian/okc decided to make a public statement about the matter! As if they have complete control of the situation. Then to confirm the misdeeds of the website and have a somewhat arrogant cavalier response to the multi nuanced admission. However my take on this streaming reality of the Interwebs is coming into deep focus. Guilt is not the preffered correlation but by association. NSA/Snowden seems a right fitted glove. Don’t hate the messenger OK now you C++ and the middle Finger is the new national bird.
And women on OKC complain when a guy’s profile isn’t a 2000 word essay.
Just as I always suspected – the content of your profile means nothing. Utterly worthless. Thank you OKC for clearing this up.
It’s really cool that OkCupid draws conclusions from its data and tries to improve itself through experiments like this. If people think this is some kind of horrible violation, that just goes to show how much faith people put in the match percentage algorithm in the first place…
…which brings me to my main beef with this article. It seems like there’s a logical oversight in your selection of 4 messages as your measure of success. After all, the match percentage is a kind of advertisement. So this data just shows me the degree to which people are willing to give each other a chance because they trust the site’s technology/marketing and want the algorithm to work, instead of proving the disempowering idea that “the mere suggestion causes people to actually like each other,” which is actually a very different statement from “When we tell people they are a good match, they act as if they are” (a more accurate statement). I guess the key is the “as far as we can measure” part, but I think you could have measured things better if you actually asked the people what they thought of each other at least a few weeks later. When I used the site I went on my fair share of “nice” dates with >90% matches who I talked to for two hours even though they bored me in the hopes that things would get better (after all, they’re >90%, right?), and exchanged messages with four times only to meet and feel zero connection with. I might even send them a few polite messages afterward even though it was going nowhere, or to tell them it was going nowhere. So yeah, sure, people are vulnerable to the power of suggestion. But that doesn’t answer the question you initially asked: “does this thing even work?” You need to reconsider what “working” means if you think 4 messages is all there is to this. Try these questions instead: What percentage matches are people who eventually say they fall in love? How many people does a given person in that group go on dates with in each percentage category before that? Did they go on a lot of dates with people with higher percentages that didn’t pan out? That data may be harder to gather, but it would be a hell of a lot more informative. Or maybe Cupid’s just lazy?
Did OkCupid inform user involved in experiment 4 after the experiment that the match score was systematically manipulated and that there might have been information which was not according to the original algorithmic result?
nice article, worth reading, thanks for such good efforts and research you, okcupid, has done.
Hope you would remain as leader of this kind of platform, connecting people around. All thebest.
OKC IS NOT COOL! It’s unethical to screw with people’s lives, lead them to believe that others are a good match when they are not, that they are a bad match when they could be a good one, to change the content of what a user has put in his or her profile, to read a user’s personal messages for the sake of experiments. Feel free to add this paragraph to your profile if you agree.
Wow I feel betrayed. I could’ve bin with the one but because I put my faith in the percentage I turned down lots of guys thank u so freakn much >¿<
Hi,
I just want to let you know that I don’t exactly have much faith on general in the whole matchup thing here. I’m currently seeing someone I met here but not because of the whole system, only because they messaged me and were wiling to do so until I decided to meet them. Maybe I’m just too untrusting. The whole numbers thing makes no sense here really because there just aren’t enough topics that are important involved in learning about someone in real life.
people’s shallowness is an incredible bias as expected….. i’m losing more faith in humanity -_-
and this is why actual intelligent peoples are not concerned at all by the experiment: empty profiles can go suck a lemon, and personnal appreciation matters instead of match percentage……
okcupid is nice enough to do some vague sorting with the percentage, but after reading this article, i’m quite proud to be one of those who would NEVER deny an answer to a low percentage user, and who have initiated conversations with an actual number of those…………………
but i’m pretty sure that user with “hot” pictures get messages from people whom they have 15%compatibility with, yes, i’m talking about you, shocked people in the comments, claiming that the experiment made you refuse to talk to people…… i’m pretty sure you just skipped average-looking people, and not “hot” ones !
I don’t mind the testing. As a matter of fact I am curious about the following stats:
a)How many people left your site after being told that Cupid plays with their minds.
b)Some guys ruin it for everybody by acting and being pervs and asking for sexual pictures the first time they get in touch with somebody: what percentage of men do this?
c)Some profiles are short lived. What is the average life span of a profile?
d)Some girls, to avoid pervs change their names and profiles. How many times they do this?
e)How many replies does a girl get in average? A guy?
f)Incomes by percentage.
g)Race by percentage.
h)Ages by percentage
Also, you should prohibit people that join from other countries. I noticed a lot of Philippines and Asians that live in their respective Countries.
I’m not entirely sure you, as OkCupid, read your own graphs right. Your “Personality VS. Looks” graph were never all the same levels. For example, one of the dots far from the 1,000s of dots were no way near the same level. It was close to “3” on the “Personality” portion and like a “4.5” on the “Looks” portion. That’s nowhere near the same thing. Besides that, how exactly do you know if you were apart of this study?
Honestly, I could forgive all this if you just reverse all the damage of changing profiles and percentages. I got a lot of creeps and sexual invites which should be banned. I reputed bad/sexual conduct and sexual entrapment. They never removed him or any of the other’s that were highly inappropriate. You tell me that should be fair to other girls? They shouldn’t have to pay for OkCupid not doing what they said they would do. Btw, FYI LADIES…. ZOOSK IS EASY WORSE AND DOES THE SAME BULL CRAP. Whose to say that we didn’t recieve messages from guys that seemed to have an interest but, OkCupid interfered with us recieving their messages?
As an OKCupid member, I’m not too offended by this because I’ve got my own standards for who I reply to and who I don’t. I eliminate a lot of clutter by …
(1) paying no attention to who I’m told will be compatible,
(2) responding only to real messages from guys who sound like they’ve read my profile (ignoring winks/likes/etc), and
(3) responding only to guys who post pictures (figuring that *no picture* means they’re hiding something, and partly as a safety precaution, like, how do I recognize them when I show up at the coffee shop?).
These online experiments don’t surprise me. I’ve seen manipulation from sites before–like when your paid membership is just about up, suddenly you have 12 responses to your profile, mostly *winks* from guys without profile pictures. **DELETE** I’ve also seen the news that Match advertisements show profile pictures for people who don’t exist. I pretty much knew that all along.
Really this is so interesting to surf such kind of blogs which provides valuable information. Today the blogs and the websites are the best way to spread the knowledge. One of the website i know to get knowledge is web9cloud, where the visitors will be provided with the full fleshed knowledge about php, about development etc. For more information must visit this website.