What's notable about travel site rankings?
Our research team found three surprises in the credibility rankings for travel sites. First of all, Yahoo! Travel received the highest credibility score in this category. This top placement and high rankings for Yahoo! in other categories suggests that this brand-name company excels at establishing credibility online.
The next surprise is the mediocre ranking of Orbitz, a Web offering backed by major airlines. This site ended up being arguably less credible than GoNomad, a site focusing on alternative travel. The takeaway message here is that an unknown yet well executed Web Site can win more credibility points than a site by established players in the industry.
The third outcome of interest is how poorly Priceline scored in this study, ranking almost at the bottom. Although Priceline has lots of name recognition and a celebrity personality, the site failed to convey credibility to people in this study, suggesting that something significant is damaging Priceline's credibility.
CONCLUDING DISCUSSION FOCUSING ON Web Site CATEGORIES
We conclude this final part of the Results & Discussion section by addressing two issues. First, we take a critical look at the rankings presented above. Next, we draw a general conclusion about how site purpose affects credibility evaluations.
Concluding discussion about Web Site rankings
Because the research methodology we use in this study is new, one question to ask when viewing the site rankings is this: Do the Web credibility rankings produced by this study make sense? Our answer is yes. As a research team we find that the rankings portray a believable picture of how people view the credibility of the Web Sites under study. Of course, there are surprises in the data, which makes a study like this interesting. But for the most part, the ranking results are in line with what our team (experienced Web users and researchers) would expect to find.
The next question to ask is: How much would these credibility rankings change if our participant demographics were different? As described earlier, our participants are mostly female (58% compared to the U.S. average of about 51%) (6) and older than the U.S. average (39.9 years compared to U.S. average of 35.5) . Yet the most unusual difference about our participants is their reported use of the Internet, which averaged almost 20 hours per week. This is more than five times the weekly average reported by Nielsen/NetRatings in August of 2002.
So how would the credibility rankings differ if our demographics were not skewed toward slightly older females with considerable Web experience? The answer is impossible to determine exactly. However, one can get a sense for the impact of our demographic skew — skewed toward experienced older females — by imagining this study as having the opposite skew — younger males with little Web experience. Which type of participant is likely to give better credibility evaluations and contribute quality comments? For our research team, this is an easy choice: We'd opt for the demographic we have in this study — slightly older females with considerable Web experience. While the rankings may change with a younger, more male, or less experienced set of participants, we believe the rankings produced in this study are likely to be more accurate.
One could ask other questions about these rankings, but our last one focuses on a practical issue: What good are these rankings?
In our view, the best practical use of these rankings — at least for Web Site designers — is to compare the sites with the highest credibility to the sites with lowest credibility. These extreme credibility differences seem undeniable, regardless of shortcomings in the study method or participants. From a credibility standpoint, the user experience of the sites that ranked #1 or #2 in a category is different from user experience of sites that ranked at the bottom. As the comments make clear, sometimes these differences have to do with established reputation, such as in the case of MayoClinic.com or Schwab. But in other cases, most notably with McMaster.com and Sharebuilder.com, the credibility was apparently won by Web Site performance alone. In sum, a practical use of the rankings is to examine the Web Sites at the extreme ends of the rankings to determine how the highly credible sites differ from sites ranked at the bottom. This approach can lead to insights about what to do — and what not to do — when designing for credibility.
While the rankings are interesting to view, we have downplayed their importance in this report because our other data type — the comments gathered during this research — provide richer insight as to how people evaluate the credibility of Web Sites today.
How Web Site topic influences credibility assessments
One overall conclusion from this study, something that has been hypothesized but never clearly shown, is that people assess credibility differently for different types of Web Sites. For example, our data showed that at times the identity of the site operator mattered very much in determining credibility (such as with nonprofit sites), and at other times identity mattered very little (as with entertainment sites). Our data also showed that at times information bias figured prominently into the credibility evaluations (as with news sites), but it didn't seem to be prominent in evaluations of other types of sites (such as travel). Although universal factors that affect Web Site credibility exist — such as design look and information structure/design — people in this study showed that differences exist as well.
We need more research to understand these differences, as well as research to fully understand the range of credibility variables that relates to a single category, such as travel Web Sites or search engine sites. The findings from these new studies will have implications not only for people who design Web Sites, but also for the mission of Consumer WebWatch, as it sets forth additional guidelines for improving people's experiences on the Web.
The fact that people notice different issues when evaluating site credibility, depending on the type of site, leads into the next section of the report, which aims to provide readers with a deeper understanding of how people evaluate credibility.
How to View This Study & Other Web Credibility Research
To better understand the contributions of the research described in this paper and how these findings complement previous work in this area, one must have a basic understanding of Prominence-Interpretation Theory (P-I Theory) (Fogg 2002a).
In brief, P-I Theory posits that two things happen when people assess credibility: a person (1) notices something (Prominence) and (2) makes a judgment about it (Interpretation). If one or the other does not happen, then there is no credibility assessment. The process of noticing prominent elements and interpreting them will typically happen more than once when a person evaluates a Web Site, with new aspects of the site being noticed and interpreted until the person reaches satisfaction with an overall credibility assessment or reaches a constraint, such as running out of time.
A summary of P-I Theory can be described as follows:
Figure 24: A summary of Prominence-Interpretation Theory.
Prominence |
X |
Interpretation |
= |
Credibility Impact |
An element's likelihood of being noticed when people evaluate credibility.
|
|
What value or meaning people assign to element, good or bad |
|
The impact that element has on credibility assessment |
Affected by involvement (motivation & ability) content, task, experience, individual differences & more. |
|
Affected by user's assumptions (culture, experience & heuristics), skill, knowledge, goals & more |
|
|
Previous research on Web credibility has investigated the Interpretation component of this theory (Finberg, Stone & Lynch, 2001; Fogg, 2000b; Fogg & Tseng, 1999; Fogg et al., 2002; Fogg, Lee, & Marshall, 2002; Fogg, Marshall, Kameda et al., 2001; Fogg et al, 2000; Fogg et al. 2001; Kim & Moon, 1998; Princeton, 2002). For example, the study by Princeton Survey Research Associates conducted for Consumer WebWatch in April 2002 (Princeton, 2002) is a study about Interpretation. In this study, researchers contacted people via the phone and asked them to assess the importance (their "interpretations") of various elements of a Web Site, such as knowing who owns the Web Site and having a privacy policy. Participants responded to each item, making a value judgment of those items. The Stanford Persuasive Technology Lab has also done research that addresses the Interpretation component in P-I Theory, using online questionnaires (Fogg et al., 2001; Fogg et al. 2002). In these studies, conducted in 1999 and 2002, people were asked how different aspects of a Web Site would affect the credibility of the site. The questions probed how people viewed sites that "looked professionally designed," that had "a broken link," that "gave a quick response to a customer service question," and over 50 other items. These previous studies were about Interpretation. None of these studies examined the Prominence part of the equation.
In contrast to previous work, the study described in this paper focuses on Prominence. It investigates what people notice when asked to evaluate the credibility of a Web Site. (ix) And what do people notice? What is prominent? This research has given us some preliminary answers. For all 10 categories of Web Sites, people noticed the design look. After that, people noticed different things most often, depending on the content category. For news Web Sites, people noticed bias of information. For nonprofit Web Sites, people noticed who was behind the site — the identity of the site operator. As Prominence-Interpretation Theory suggests, the content and purpose of a Web Site affects what people notice when they evaluate the Web Site Credibility.
The release of this report is a step forward in the study of Web credibility, because to gain a rich understanding of credibility impact, one must have information about both Prominence and Interpretation. Studies that focus on these separate components can be woven together to create a rich warp-and-woof understanding of Web credibility, an approach that is far richer and — in our view — more compelling than previous explanations of how people assess the credibility of Web Sites.
Consider how having a privacy-policy statement affects the perceived credibility of a Web Site. Previous research (the various studies that focused on Interpretation) found that people claim to assign more credibility to sites that have a privacy policy. This makes sense. But what if people don't notice the privacy policy? Prominence-Interpretation Theory suggests that if people don't notice an element, such as a privacy policy, then it will not have any impact on the overall credibility assessment. Again, this makes sense: Any site with a privacy policy that does not get noticed gets no credibility boost from having the privacy policy. Our research shows how this plays out in real Web Sites: Fewer than 1 percent of the comments about the 100 Web Sites mentioned anything about a privacy policy. This element was rarely noticed and, as a result, had almost no real impact on the credibility assessments people made. The same is true for any other element, such as a broken link buried deep inside the site. Although previous studies show that a single broken link will hurt the credibility of a Web Site — at least that's what people reported — the broken link will have no effect on the credibility assessment if people don't notice it.
An additional example helps show how previous studies and the current research work together in understanding Web Site credibility evaluations. The Stanford studies on Web credibility elements found, both in 1999 and 2002, that people claim to assign more credibility to a site that "looks professionally designed" — that's an issue of Interpretation. Our current study suggests that people frequently notice the design look of the site — which is an issue of Prominence. As P-I Theory suggests, the combination of high Prominence and favorable Interpretation make "professional-looking design" a Web Site quality that will significantly boost a site's overall perceived credibility. An appealing visual design is a pivotal issue in assessing Web Site credibility, since this aspect of a site is likely to be both noticed and interpreted positively.
Even though our current study makes significant steps forward in understanding Prominence, our research has shortcomings. Because the specific percentages in this study are the result of variables that can change — the coding categories, the study context, the users who chose to participate, the 100 Web Sites selected for this study — we caution readers against becoming too attached to these particular values. Although we performed our calculations with care, readers should view the resulting percentages as approximations, since this study is an early attempt to measure Prominence. We hope future studies can draw on what we have done in order to enhance the research method and the data analysis. For example, creating a more precise coding system will be an ongoing process that will require multiple studies and many debates. Perhaps the biggest contribution from our study will be to provide an initial set of results that future research can refine or refute. In other words, we view our study as the opening statement in a new conversation, not as the final word.
While various questions remain unanswered, one thing is clear: Our collective understanding of Web credibility assessments will become richer as research continues to give insight in two areas: (1) what people notice when evaluating Web Site credibility, and (2) how people evaluate different Web Site elements or features. Both paths are worthy directions for future research.
Implications for Consumer WebWatch
One purpose of this study was to help Consumer WebWatch fulfill its mission of making the World Wide Web a place where ordinary people can safely and efficiently find information and make transactions. Consumer WebWatch has outlined five guidelines involving transparency and fairness. In brief, these guidelines are as follows (see http://consumerwebwatch.org or Appendix C for the wording of these guidelines):
- Identity: Making clear who owns the site and how people can contact them
- Advertising and Sponsorships: Distinguishing between ads and content and disclosing relevant business relationships
- Customer Service: Disclosing costs and policies relevant to consumers
- Corrections: Correcting past information that was false or misleading
- Privacy: Disclosing how personal information will be used
The results of our study suggest that ordinary people do not often use the above criteria in evaluating the credibility of a Web Site. None of these categories appeared in more than 15 percent of the comments from our participants; some of them appeared rarely, if ever. Our results do not mean that the five guidelines are irrelevant to consumers. When asked whether these issues matter, people say yes (Princeton, 2002). Indeed, if people looked for these features on the Web Sites they visit, they would get a clearer picture about what they can and cannot trust online. But our data suggest that when actually using the Web, people likely don't think about these five key issues. They don't probe very deeply into issues of identity, sponsorship, corrections, customer service, or privacy.
The disconnect between what people actually do and what they should do creates two important opportunities for Consumer WebWatch, one dealing with education, the other with evaluation.
THE NEED FOR EDUCATION
One role Consumer WebWatch can play in the future is that of an educator about what people should be looking for as they assess how safe it is to believe information or transact business with a Web Site. If consumers knew they should look for the five markers of transparency and fairness, they would be more likely to seek and notice those things. And as people start looking for those things, Web Site designers will be under pressure to provide them. Right now there is no standard way to inspect a site to figure out whether what the site says or does is worthwhile. Consumer WebWatch is in a good position to create this standard.
THE NEED FOR EVALUATION ASSISTANCE
The second role Consumer WebWatch is perhaps more realistic than the previous role. (Yes, changing the way people evaluate Web Sites is a worthy goal, but this may be too ambitious a task.) A more tractable endeavor would be to provide people with assistance in evaluating Web Sites — i.e., to do most of the work in identifying which Web Sites are worthy of people's time and attention and which sites are not.
This is not a new role for Consumers Union. In fact, it's the core-value proposition of its leading publication, Consumer Reports. Ordinary people turn to this publication to find out which products and services are most reliable and which products and services they should probably avoid. It's the trusted guide for people buying washing machines, insurance policies, and mobile-phone plans. This established role for Consumers Union could — and probably should — be extended into the online world.
The consumer need for evaluation assistance in the online world may be greater than that need in the physical world. At least in the physical world, people have years of experience to draw on; in addition, hundreds of regulations have developed over the years to weed out the worst players. This is not so in the online world. When we use the Web today, all of us are entering territory that's new and constantly changing (consider how dramatically online ads have changed in the last six months, for example). Because the Web is new and dynamic, even those of us who make a profession of studying the Web would benefit from the evaluation assistance that Consumer WebWatch could provide. An organization dedicated to this purpose not only could save people time and energy, but it also could do the job better than virtually any individual working alone.
END
Back to start of the article: A Good Web Site Design Brings Trust and Success |
