Ben Radford
Volume 38.6, November/December 2014
Original link: http://www.csicop.org/si/show/sweet_science_of_seduction_or_scam_evaluating_eharmony
The popular online dating site eHarmony claims that its
matching methods are both successful and scientific. But a closer look
at the evidence suggests otherwise.
As long as there are lonely hearts there will be a market for matchmakers. The emergence of the Internet has of course revolutionized many things, and the search for compatible mates is no exception. Online dating is a huge business, with dozens of websites offering people the chance to find love in cyberspace.
Like many services for which there is competition, online dating sites struggle to distinguish their services from all the others. After all, the premise is pretty basic: men and women searching for each other based upon their interests and profiles. There are only so many ways to “sex up” the process (so to speak); there’s really not a lot of room for “new and improved” ways of connecting people.
However there is one important area in which online dating services try to compete: better ways of matching clients. Like many other businesses, online dating sites like to adopt the veneer of scientific validity. Websites such as eHarmony.com, True.com, and Chemistry.com claim to use science to help people find compatible partners and, eventually, love. But how good is the evidence for their claims?
eHarmony’s Scientific Claims
Perhaps the best-known dating service claiming to mix science with seduction is eHarmony. According to the company’s website, its marriage profile, “developed by a team of clinical experts . . . is rooted in classical psychometric theory—which uses well-established standards to measure mental abilities and traits in a reliable way.” The site makes direct and explicit claims about the scientific validity of its matching program, which it calls a “Compatibility Matching System.” On the website under a section titled “Compatibility Science,” eHarmony states:Based on his thirty-five years of marriage counseling and studies of thousands of married couples, eHarmony founder Dr. Neil Clark Warren exhaustively researched what makes marriages succeed and fail. His findings? Chemistry is not enough. . . . By studying the difference between happy and unhappy married couples, Dr. Warren’s team of researchers found 29 Dimensions [a registered trademark] that help predict great relationships. These compatibility dimensions include: Core traits like emotional temperament, adaptability, curiosity, and intellect; values and beliefs, such as spirituality and feelings about children; and relationship skills, such as conflict resolution. (eHarmony 2011a)Again appealing to the authority of science, the following paragraph explains how the “Science of Compatibility” is used by eHarmony: “Our Compatibility Matching System matches you by taking into account the 29 Dimensions of Compatibility that help predict the potential for relationship success. The results are single matches unlike anything you will find anywhere else.”
In all, the words science or scientific appear five times within three short paragraphs on that web page. On a page that lists the eHarmony research team (“mandated to advance the scientific understanding of human relationships”), many of the members have titles like “Senior Director of Research and Development,” “Senior Research Scientist,” and “Research Scientist” (eHarmony 2011b). In short, eHarmony’s claims to science and scientific validity are both explicit and implicit throughout the company’s literature. Furthermore, a representative from eHarmony (Breton 2011) told me explicitly that “eHarmony’s matching system was developed using an empirical, scientific process.”
Other eHarmony Claims
All this seems very impressive, but there are questions about the validity of eHarmony’s much-vaunted “scientific, 29-dimension” tests. Does its “science” greatly improve the quality (or odds) of a match? How well do its tests construct validity? What were the control measures? What is the scientific evidence for these claims?Steven Carter, director of research at eHarmony, wrote an article in the APS Observer (published by the Association for Psychological Science) about his work. Instead of offering data or empirical evidence that eHarmony’s matching system is valid (or superior to other dating sites’ systems), Carter (2005) boasted about how helpful his work was to lonely singles and that “working at eHarmony has, in many ways, been a dream job.”
While Carter offered little support for his claims, he did state that “to date, we estimate that over 9,000 eHarmony couples have married.” That statistic, if true, clearly doesn’t tell the whole story, as it cherry-picks the successes and omits the failures. How many of the eHarmony matches were incompatible? If, by one count, there are more than twenty million eHarmony members looking for matches or marriage, 9,000 may not seem like that impressive a success ratio. Furthermore, the real question would seem to be how many of those 9,000 marriages lasted longer than average; for all we know, most of the eHarmony couples may have since divorced.
Evaluating the Claims
Oddly, no references to studies scientifically validating eHarmony’s methods appear on the company’s website. Nor was I able to find any published research in an online literature review.There is actually very little available information about eHarmony’s methods, which raised a troubling question in the mind of Robert Epstein, a contributing editor to Scientific American Mind: “Why would a major company such as eHarmony, which claims to have 12 million members, not subject its ‘scientific, 29-dimension’ test to a scientific validation process?” (Epstein 2007). He noted that eHarmony representatives presented a paper in 2004 that claimed eHarmony couples were happier than couples who met through other means, “But this paper still has not been published, possibly because of its obvious flaws.” Four years after Epstein wrote that, there still appeared to be little or no data on eHarmony’s methods.
I contacted Paul Breton, the director of communications for eHarmony, to ask for research: “I am specifically interested in any published studies or research that has tested the construct validity of the compatibility measures that eHarmony uses, and any research that demonstrates that eHarmony relationships last longer than average. Also, please provide references to any published studies or research by Neil Clark Warren on the subject of relationships, marriage, or compatibility.”
Breton (2011) responded with a lengthy (1,400-word) email that read like boilerplate copy lifted from a press release. Breton mostly addressed questions I had not asked, including whether eHarmony’s tests improved the chances of a match; whether eHarmony offers services to same-sex partners; how eHarmony measures the quality of their services; whether eHarmony clients were satisfied with their services and matches, and so on.
In the final paragraph Breton finally addressed my query: “Two peer-reviewed papers have been published or are in press that have used data from eHarmony couples and are relevant to the efficacy of eHarmony’s matching system. They are summarized below and attached for reference.” Breton’s email ended with a list of nearly twenty articles and pieces by eHarmony Senior Director of Research and Development Dr. Gian Gonzaga, though none of them were relevant to my query.
A Closer Look
I was slightly surprised that eHarmony only provided two studies, and as I read through the studies I immediately found problems.The first study, “Assortative Mating, Convergence, and Satisfaction in Married Couples,” (Gonzaga, Carter, and Buckwalter 2010) had nothing to do with the scientific validity of eHarmony’s tests or measures but instead discussed marital satisfaction in general.
The abstract noted, “This work investigates assortative mating and convergence in personality and their effect on marital satisfaction. Measures of personality were collected from a sample of married couples before they met and twice after they were married. Results showed evidence for assortative mating but not for convergence in an average couple.” As far as I could tell, the only relevance to eHarmony was that all three coauthors were employed there. This was clearly a red herring.
I then reviewed the second study (Carter and Buckwalter 2009). This piece was coauthored by Steven R. Carter and J. Galen Buckwalter (both of eHarmony; the following conflict of interest disclosure appears at the bottom of the title page: “The reader should be aware that the authors are employed by, and have a financial interest in the success of, eHarmony.com, Inc.”). This piece
investigates the effects of a broadly adopted online matchmaking site on the nature and quality of married couples formed. Measures of personality, emotion, interests, values, and marital adjustment were collected from a sample of married couples who had been introduced by an online matchmaking service, and from a sample of married couples who had met through unfettered choice. Results showed that couples introduced by the online matchmaking site were more similar. . . . We conclude that online matchmaking services based on predictive inference and proscribed selection can be observed to have a significant and meaningful impact on marital quality.In other words the eHarmony team concluded, unsurprisingly, that couples introduced through an online dating service (say, for example, eHarmony) tended to be more similar than couples who met randomly—and that this similarity would likely lead to a better or longer-lasting marriage. That was it; I read the entire study, and there was not a single word about any evidence for the claim that eHarmony is based on scientific principles, or that its much-hyped, carefully constructed “29 Dimension” questionnaires were scientifically valid.
I then became curious about the journal in which this piece appeared, Interpersona. Why were eHarmony’s top scientists publishing their research in a journal I’d never heard of? I did some research and discovered that it’s published by something called the Brazilian Association for Interpersonal Relationship Research, which I had also never heard of. I did an online search (in late 2011) and found a grand total of six hits for this organization on the World Wide Web (the Journal of the American Medical Association, by way of comparison, has nearly ten million hits). Clearly this was not a prestigious or well-known professional organization; in fact, by all appearances the Brazilian Association for Interpersonal Relationship Research is a dummy organization, and its journal Interpersona is basically a vanity press for interpersonal dating researchers to get work e-published. The fact that eHarmony’s top researchers published here raises serious questions.
Dr. Warren’s Research
I also searched for evidence that Neil Clark Warren, the marriage and relationship expert behind eHarmony, has created important work in, and published on the subjects of, marriage or relationships. A 2011 search of several academic databases (including PsycInfo and MedLine) did not reveal a single article published by Neil Warren on the subject of marriage, compatibility, or relationships. If Warren has been researching the subject for the past thirty-five years as is claimed (“based on his 35 years of marriage counseling and studies of thousands of married couples, eHarmony founder Dr. Neil Clark Warren exhaustively researched what makes marriages succeed and fail”), why hasn’t he published it? You would expect that a scientist or researcher who has spent decades studying a topic would have published on it. (Warren is author of several books on relationships, though anyone can write a book of relationship advice.)Breton (2011) admitted that Warren had little or no role in creating eHarmony’s current matching system: “While Dr. Warren founded eHarmony and has played an important role within the company, he is not the sole (nor even a primary) person involved in conducting academic-style relationship research or developing our matching algorithms.” This is probably just as well, since it appears that despite his thirty- five years of “exhaustive” research, Warren never published a single study on the topic.
More Questions about Validity
I responded to eHarmony’s Breton with a few simple questions:You reference the journal Interpersona, which neither I nor any of my colleagues have heard of. . . . Is this the highest profile publication that eHarmony’s research has appeared in? Have any of your team published in any mainstream, peer-reviewed psychology or sociology journals . . . such as American Psychologist, or International Journal of Applied Psychology, or Journal of Experimental Psychology, or British Journal of Psychology, or Psychological Bulletin, or Psychological Review, or Social Psychology, and so on?Breton responded, saying:
The critiques you raise seem to confuse “scientific” with “fully transparent.” One can still use a scientific method, which eHarmony does, without exposing every last detail to the public. . . . eHarmony’s matching system was developed using an empirical, scientific process. The founding research team administered a standard measure of marital satisfaction (the Dyadic Adjustment Scale) to thousands of married couples and rank-ordered the couples on the basis of their relationship satisfaction. Then a separate survey was administered to assess the personality traits, values, attitudes, and beliefs of the couples to determine what qualities distinguished the happiest couples (those in the top quartile) from all the others. Using those insights, a final survey instrument was developed to predict which pairings of single individuals would share the most commonalities with the happiest married couples in the survey.The next statement from eHarmony’s director of communications was particularly telling:
While many of the details of those studies have been published, not everything has been disclosed in a peer-reviewed journal. As a rule, academic journals in the behavioral sciences do not publish articles that validate or promote the efficacy of for-profit technologies. There are also intellectual property considerations eHarmony has to balance. None of this, by default, makes eHarmony’s scientific process any less valid. Furthermore, while there are many merits to peer-review, it is not the only measure of scientific validity.Here we have several interesting admissions, including that “not everything” (in fact, nearly nothing) about eHarmony’s research has appeared in peer-reviewed journals. Breton’s claim that peer-reviewed journals do not publish articles that validate or promote for-profit technologies is specious: respected scientific journals routinely publish studies and articles about the validity and efficacy of for-profit technologies, including pharmaceutical drug studies, medical implants (new artificial hips or joints, for example), experimental therapies, and so on. Furthermore, scientists employed by for-profit organizations regularly publish research that does not reveal trade secrets or proprietary intellectual property.
I concluded my correspondence with Breton:
I don’t think I’m confusing “scientific” with “fully transparent” at all. I’m simply requesting references to published studies in peer-reviewed journals regarding the scientific validity of eHarmony’s tests, which you claim have been scientifically validated. I specifically requested published studies “that investigate and test the validity of eHarmony’s compatibility tests (not relationship advice in general).” The two references you provided, from Interpersona and Personal Relationships, do not contain anything about the validity of eHarmony’s testing methods, scientific validity, or matching system. Unless you can provide me with references, I will assume that you cannot provide published, peer-reviewed research that specifically tested and validated eHarmony’s compatibility tests. I know how to read journal articles [and] so far you have not shown me anything resembling good science behind eHarmony’s methodologies. If you have the studies I’m asking for, give me the references. If you don’t, then just admit it.I never heard any more from eHarmony.
My search for the science behind eHarmony had come to an end. I had searched the eHarmony website in vain for any reference to studies or research validating eHarmony’s tests, measures, or matching criteria. I searched the psychological and medical literature, both for scientific research done by the company’s founder, Neil Warren, or by any current employees, and found nothing.
Finally, I contacted a representative for eHarmony, who was ultimately unable to provide a single reference (published or unpublished, peer-reviewed or not) to any research confirming the scientific validity of eHarmony’s methods. Surely if eHarmony had conducted solid, scientific research validating its methods it would be happy to publicize them. In fact, eHarmony has several hallmarks of pseudoscience, including a reluctance to subject its claims and data to peer review.
In 2013, eHarmony’s senior research scientist, Gian Gonzaga, gave a presentation at the annual Society for Personality and Social Psychology conference in which he tried in vain to convince his colleagues of the scientific validity of eHarmony’s methods. Gonzaga, while reiterating that eHarmony’s formulas are secret and proprietary, offered evidence from unpublished, nonpeer- reviewed studies that he claimed demonstrated that a couple’s questionnaire scores correlated with relationship satisfaction years later. Critics such as the University of Rochester’s Harry Reis, who coauthored a study of eHarmony’s methods (Finkel et al. 2012), saw the matter differently. According to John Tierney (2013) of The New York Times:
He and his coauthors argued that eHarmony’s results could merely reflect the well-known “person effect”: an agreeable, non-neurotic, optimistic person will tend to fare better in any relationship. But the research demonstrating this effect also showed that it’s hard to make predictions based on what’s called a dyadic effect—how similar the partners are to each other. . . . “If dyadic effects are real, and if eHarmony can establish this point validly, then this would be a major advance to our science,” Reis said. But he and his colleagues said that eHarmony hadn’t yet carried out, let alone published, the sort of rigorous study necessary to prove that its algorithm worked. . . . To verify the algorithm’s effectiveness, the critics said, would require a randomized controlled clinical trial like the ones run by pharmaceutical companies. Randomly assign some individuals to be matched by eHarmony’s algorithm, and some in a control group to be matched arbitrarily; then track the resulting relationships to see who’s more satisfied.Reis’s coauthor, Northwestern University’s Eli Finkel, added, “Nobody in the world has the treasure chest of resources for relationships research that eHarmony has, so we can’t figure out why they haven’t done the study.”
Of course, plenty of people can find compatible matches on eHarmony (or any other dating service) without the help of scientifically validated tests; wonderful, lasting relationships sprout all the time by random chance. But if people are choosing eHarmony over a competitor because they believe that there is some validated science behind the matching, they are likely being misled. Caveat emptor.
