 |
Internet Marketing · eMarketing · Internet Advertising · Online Branding |
|
Internet Marketing · eMarketing · Internet Advertising · Online Branding |
|
|
|
||
| ADVERTISING BY ICONOCAST |
||
COLLEGE TEST PREP TAKES A TEST
A Review of Ten Online SAT Test Preparation Services
COLLEGE TEST PREP TAKES A TEST
A Review of Ten Online SAT Test Preparation Services
by Warren Buckleitner, Ph.D.
Mediatech Foundation
May 1, 2006
Consumer Reports WebWatch
101 Truman Avenue
Yonkers, NY 107031057
Tel.: 914.378.2600
www.ConsumerWebWatch.org
Page 2
TABLE OF CONTENTS
EXECUTIVE SUMMARY________________________________________________ 3
KEY FINDINGS _________________________________________________________________ 5
CREDITS_______________________________________________________________________ 6
ABOUT CONSUMER REPORTS WEBWATCH _____________________________ 7
INTRODUCTION AND METHODOLOGY _________________________________ 8
A New Way to Prep: Online ________________________________________________________ 8
METHODOLOGY__________________________________________________________ 9
Selection of Online Test Prep Services ________________________________________________ 9
Selection of Student Testers________________________________________________________ 10
The Testing Process ______________________________________________________________ 11
Data Collection _________________________________________________________________ 11
MAJOR FINDINGS AND CONCERNS ___________________________________ 13
Online Services Come with Aggressive Advertising_____________________________________ 13
Higher Price and Reputation Don’t Necessarily Make for Better Products ____________________ 14
Students’ Privacy Protection Questioned______________________________________________ 14
Site Functionality Influences Perception of Trust _______________________________________ 15
Interactivity Distinguishes Online Services, Needs Improvement___________________________ 15
Errors Abound __________________________________________________________________ 16
Site Identity Often Confusing ______________________________________________________ 16
Technical Glitches Hamper Site Performance __________________________________________ 16
Customer Service Generally Good___________________________________________________ 17
RESULTS BY SERVICE________________________________________________ 18
Barron's Test Prep _______________________________________________________________ 18
Boston Test Prep ________________________________________________________________ 18
Kaplan's SAT Online Prep _________________________________________________________ 19
Number2.com___________________________________________________________________ 21
Peterson's Online SAT Course______________________________________________________ 23
PrepMe________________________________________________________________________ 23
SAT Secrets ____________________________________________________________________ 25
Test Preparation Program Online Test Prep____________________________________________ 25
The Official SAT Online Course ____________________________________________________ 26
The Princeton Review ____________________________________________________________ 26
TIPS FOR STUDENTS AND FAMILIES __________________________________ 29
RECOMMENDATIONS FOR PUBLISHERS ______________________________ 31
Appendix 1: Online SAT Test Preparation Evaluation Instrument ______________ 32
Appendix II: Student Responses __________________________________________ 38
Appendix III: Academic Profile of Student Testers ___________________________ 41
Appendix IV: Student Tester SAT Scores, Before and After Study, by Service _____ 43
Page 3
EXECUTIVE SUMMARY
Taking the SAT exam has become a rite of passage for more than 1 million
collegebound
American high school students each year, a number that has
steadily increased since the SAT was first administered by the nonprofit College
Board in 1926.
Not surprisingly, demand for SAT test preparation services has grown in tandem,
fueled by the competitive nature of college admissions and the ability to take the
SAT repeatedly to try for a better score. There are also more students with
college plans, according to statistics provided by The College Board. The number
of college students rose 15 percent in the decade ending in 2003, while the
number of students taking the SAT grew 35 percent, to almost 1.5 million this
year – the 15th consecutive year of growth. 1
Additionally, Americans’ appetite for test preparation assistance grew noticeably
this year, when the “new” SAT format was unveiled. For the first time, the test
featured an openended,
essaywriting
component in addition to the traditional
multiplechoice
test format. The scoring system also was revised, with a new
high score of 2400 rather than the traditional 1600.
As a result of this confluence of factors, the market for test preparation services
in general, not just for the SAT, has ballooned to more than $4 billion, according
to Eduventures Inc., a Bostonbased
market research firm. A student facing a
forthcoming SAT exam can purchase a variety of products that promise a better
score, including Webbased
programs, books, flash cards, handheld
electronic
devices, software, personal tutors and classes. There are also a variety of free or
lowcost
options, such high school courses, and free resources provided at the
site of the test maker, The College Board and others. Internetdelivered
test
preparation products are among the newest form of test prep to emerge in recent
years, representing a growing slice of the market estimated at $50 million,
according to Eduventures.
Due to the expanding test preparation services market and the increasing role
the Internet is playing in delivering these services to students and families,
Consumer Reports WebWatch spent five weeks in the summer of 2005
examining 10 Web sites providing online SAT testprep
services. WebWatch
wanted to learn how the market’s largest, bestknown
and, often, most expensive
options – such as services from Kaplan and The Princeton Review – fared
against smaller, lesserknown
services, such as the free site Number2.com. The
field of 10 sites, therefore, was selected with a focus on cost variation and market
share, to assure a mix of sites was included. This report reviewed only those
1 The College Board. Available online at
http://www.collegeboard.com/press/article/0,,46851,00.html
Page 4
services offered online, not those featuring live tutors, books, classes or other
offline study aids.
The 10 services evaluated were: Barron’s Test Prep, Boston Test Prep, Kaplan's
SAT Online Prep, Number2.com, Peterson's SAT Online Course, PrepMe, SAT
Secrets, Test Preparation Program, The Official SAT Online Course, and The
Princeton Review.
The evaluation process consisted of both expert and student testing. WebWatch,
in conjunction with Mediatech Foundation, recruited 20 high school juniors who
had already taken the SAT test in the spring of 2005 and planned to take the
exam again in the fall. Each of the 10 sites was reviewed by two students during
a minimum of five, fourhour
sessions in a controlled lab setting under the
supervision of the study authors, both experienced educators. Testers spent
more than 400 combined hours reviewing these sites,
While testers deemed seven of the 10 sites generally effective in their product
delivery and the overall quality of their services, WebWatch noted some troubling
trends, particularly regarding the blending of advertising and educational content
and aggressive marketing of promotional materials to collegebound
students. In
one case, a student who expressed interest in free scholarship information
received an email
containing a link to an Air Force Reserve Officer Training
Corps enrollment form. Students using these services are most often legal
minors, and greater care should be taken to not pressure them, either directly or
indirectly, to share their personal information or to prompt them to purchase
additional products with the promise of a higher score.
Moreover, marketing practices of some sites can be considered questionable,
such as the implied promise of access to Ivy League educators, or combining
test reminders with addon
products. Peterson's SAT Online Course, for
example, promised access to a Harvardeducated
essay reader whose identity
was never revealed, and whose credentials, therefore, could not be verified.
Some testers were surprised to learn The Princeton Review had no actual
connection to Princeton University or the Educational Testing Service, which is
located in Princeton, N.J. More troubling was the practice of using test anxiety as
a motivator to purchase more books or products by combining email
test
preparation reminders with product advertisements.
In addition, testers found typos and grammatical errors on some sites, prompting
concern about the inconsistent quality of purchased online materials. These
discoveries not only chipped away at the perceived credibility of online products
designed to help students excel on a scholastic aptitude test, but also prompted
questions regarding the credentials of those providing – and charging for – these
services.
Page 5
Overall, however, the testers appear to have benefited from the process of
spending hours on these test preparation sites. The 20 high school juniors who
served as testers took the SAT for the second time in the fall of 2005, after
participating in this study, and saw their test scores improve by an average of 38
points per student – a 1.6% gain. While this scoring increase may be the result of
many factors, including greater familiarity with the SAT test, the students did
appear to gain greater testing confidence after dedicating hours to test
preparation.
KEY FINDINGS
Online Services Come with Aggressive Advertising. By registering
for services, including Kaplan's, The College Board, Boston Test Prep
and The Princeton Review, students were suffused with messages
from college and military recruiters or offers for financial aid or study
aids. One email
message from The Princeton Review contained a link
to a United States Air Force recruiting form. Additionally, The College
Board marketed its Webbased
online test prep service in advertising
space not clearly distinguished from free test resources.
Errors Abound. Mistakes in online sample tests – including
grammatical problems, questions with no answers, missing sections of
text, font problems, or poorly constructed questions – were consistently
present in 6 of the 10 services evaluated, but were particularly evident
in Barron's Test Prep, Peterson's SAT Online Course, PrepMe and
Test Preparation Program sites. Only one of the 10 services tested —
Boston Test Prep — offered a means for reporting errors.
Higher Price and Reputation Don’t Necessarily Make for Better
Products. Number2.com was the only freeofcharge
service tested,
and it performed exceptionally well against expensive, betterknown
products, such as The Princeton Review and Kaplan's. The prep
service offered by The College Board, the creator of the SAT, was
buggy and lacked interactive features common on other sites. Testers
also found The College Board’s computerized essay grading service
too general to be useful, compared with The Princeton Review’s more
specific feedback provided by human editors.
Technical Issues Hampered Performance. Kaplan's SAT Online
Prep, The Princeton Review and Boston Test Prep may not work on
older browsers with virus protection or popup
blocking features
activated, testers discovered. New versions of browsers are
recommended or required, with current plugins and the ability to use
cookies turned on. Unfortunately, this also makes a browser more
vulnerable to popup
advertising and adware.
Page 6
Students’ Privacy Protection Questioned. Some sites contained
only vague descriptions about what student information would be
shared, and with whom. The Princeton Review, The College Board,
and Kaplan's SAT Online Prep were not forthcoming about who their
“partners” were and what “opt in” actually means, definitions that
should be clearly provided and easy to understand, given the target
audience for these products is teenagers.
CREDITS
This project was written and directed by Warren Buckleitner, Ph.D., a consultant
and adviser to Consumer Reports WebWatch. Buckleitner is editor of Children's
Technology Review, a periodical covering children’s interactive media. He also is
the founder and interim director of the Mediatech Foundation, a nonprofit
public
community technology center based in Flemington, N.J., where the research was
conducted. This study was coordinated by Linda Bradway, a retired public school
principal and researcher at Mediatech.
The project was funded at a cost of approximately $33,000 by Consumer Reports
WebWatch, with funding and staff contribution from Consumers Union’s Statistics
& Quality Management Department. The research report was edited by Tracy
Ziemer, WebWatch’s researcher and Web site producer at the time of the testing,
and Beau Brendler, WebWatch’s director. Consumers Union and Mediatech are
not affiliated with any SAT testprep
service reviewed in this study.
Page 7
ABOUT CONSUMER REPORTS WEBWATCH
Consumer Reports WebWatch is a project of Consumers Union, the nonprofit
publisher of Consumer Reports magazine and ConsumerReports.org. No
Consumers Union publication accepts any funds whatsoever from advertising.
WebWatch is supported by The Pew Charitable Trusts, which invests in ideas
that fuel timely action and result; the John S. and James L. Knight Foundation,
which promotes excellence in journalism worldwide and invests in the vitality of
26 U.S. communities; and the Open Society Institute, which encourages debate
in areas in which one view of an issue dominates all others.
Consumer Reports WebWatch's mission is to improve the credibility of Web
sites, through research, news investigations, and articulation of best practices
guidelines in specific sectors of Web publishing. WebWatch's research,
investigations, guidelines, conference transcripts, a list of sites that comply with
WebWatch's best practices guidelines, and other materials are available for free
at http://www.consumerwebwatch.org.
Page 8
INTRODUCTION AND METHODOLOGY
What costs $41.50, lasts almost four hours, and gets almost 1.5 million
teenagers – actually, 1,475,623 in 2005 out
of bed early on a Saturday
morning? The answer is the SAT, an often mindnumbing
riteofpassage
for
almost every U.S. student who wants to attend college.
The SAT Reasoning Test was commissioned in 1926 by an association of
colleges, known today as The College Board, with the intention of measuring a
student’s critical reading, math and writing skills needed for academic success in
college. In 1947, The College Board began working with the Educational Testing
Service (ETS), a large, independent, nonprofit
organization in Princeton, N.J., to
create and administer the SAT. Today, ETS is a thriving hub of standardized
testing, creator of more than 30 wellknown
tests that include the GRE, AP and
TOEFL. The College Board remains ETS’s largest client. While groups such as
the National Center for Fair & Open Testing debate the value of standardized
testing, there is no denying the SAT is entrenched in American academic culture.
The SAT, given seven times per year across the country, is designed to
challenge even the brightest student, especially those who have not come facetoface
with a quadratic equation for some time. The test is almost entirely
multiple choice, though the format of the SAT changed rather significantly in
2005 to include an openended
essaywriting
section. In addition, there are now
three sections instead of two and a more difficult math section. Students can
achieve a top score of 2400 instead of 1600.
A common strategy for many students, including the 20 that participated in this
study, is to take the test at least twice once
during their junior year, typically the
spring, then again the first semester of their senior year to try for a better score.
Students can take the SAT as many times as they are willing to pay for it,
tormenting some students with the notion that, as with a golf swing, one can
always do better.
A New Way to Prep: Online
The increasingly competitive nature of college admissions, combined with the
ability to take the test more than once, has created a thriving SAT testprep
business. In the past, testprep
products have consisted of books, personal tutors
and classes. More recently, subscriptionbased
online testprep
courses have
become available. Requiring a computer, credit card and Internet connection,
these services differ considerably in price and service.
While the exact size of the online SAT market is difficult to define, it has been
estimated at $50 million by Eduventures Inc., a Bostonbased
education market
research firm. One of the market leaders, The Princeton Review, provides prep
Page 9
services for a variety of tests, including the SAT, and reported a 15% increase in
testprep
revenue, from $61.2 million in 2004 to $70.3 million in 2005.
It’s significant to recognize the lure of 1.5 million SAT takers and future U.S.
college students as a market demographic group to commercial and, potentially,
military interests. This is a population on the verge of spending tens of thousands
dollars on college tuition, housing, books, computers and other electronics.
Moreover, some families are making longterm
decisions about loans, mobile
phone services, scholarships and firsttime
credit card ownership.
Given the potential financial significance of this demographic, WebWatch also
examined relationships between these business interests and test preparation
sites and discovered troubling issues regarding the handling of students’ private
information and advertising to minors. This matter is discussed at greater length
in the Findings and Concerns section of this report.
METHODOLOGY
Selection of Online Test Prep Services
In an effort to better understand the nature of the online test preparation
business, WebWatch surveyed both commercial, forprofit
services and nonprofit
options, and selected 10 sites that represented a mix of prices and market
share to assure the full spectrum of options available to students was covered.
The sites selected included both householdname
programs, such as The
College Board, Kaplan's and Princeton Review, as well as lesserknown
services, such as PrepMe and SAT Secrets.
In order to be considered, a site had to meet the following criteria:
• Actively advertise testprep
services for the new SAT format on their Internet
sites;
• Be available during a time period of July 15Aug
15, 2005, when WebWatch
conducted testing;
• Be delivered primarily via the Internet rather than through books, classes or
personal tutors.
The following services were selected for this study, listed here with the
subscription prices paid by WebWatch on or around July 15, 2005:
1. Barron's Test Prep from Barron's Educational Series, Inc., $10.95 for 6
months; www.barronstestprep.com.
2. Boston Test Prep from Boston Test Prep, $99.00 for 3 months;
http://sat.bostontestprep.com.
3. Kaplan's SAT Online Prep from Kaplan, Inc., $399.00 for 6 months;
www.kaptest.com.
Page 10
4. Number2.com from Xap Corp., free, with no time limit; www.number2.com.
5. Peterson's SAT Online Course from The Thompson Corporation, $169.95
for 4 months; www.petersons.com.
6. PrepMe from PrepMe Corporation, $499.99 for 4 months;
www.prepme.com.
7. SAT Secrets from Morrison Media LLC, $49.95;
www.secretsstudyguide.com/sat/
8. Test Preparation Program from Online Test Prep, $29.95 for 3 months;
www.onlinetestprep.com.
9. The Official SAT Online Course from The College Board, $69.95 for 4
months; http://store.collegeboard.com.
10. The Princeton Review SAT Online from the Princeton Review, $399.00 for
4 months; www.review.com
Selection of Student Testers
These 10 online SAT test preparation services were evaluated by 20 high school
juniors in New Jersey, where Mediatech, a nonprofit
community technology
center, is located. The students, recruited through a local newspaper article,
received free access to two randomly selected services in exchange for their site
evaluations and future SAT test scores. The testprep
sessions took place every
weekday morning from 9am to noon on Internetenabled
computer stationsover
the course of four weeks.
This field of 20 students (11 male, 9 female):
· Was willing to share past and future SAT scores, and high school grade
point averages;
· Was from four different New Jersey high schools (three public, one
private);
· Had already taken the SAT test, with the new essay format, in the spring
of 2005 and planned to take it again in the fall;
· Was doing no other test preparation over the summer in addition to the
work at Mediatech for this study.
It should be noted 20 testers is not large a large enough sample to claim
statistical significance with any of the measures. No statistical validity should be
assumed by the reader with regard to the quantitative measures in this report.
Instead, the student responses provide qualitative indicators of how a small
group of randomly assigned student testers felt about a particular service during
a specific time period.
Page 11
The Testing Process
Students were randomly assigned a testprep
service to evaluate, then were
registered to that service using real names and home mailing addresses. They
were also given an operational email
address created specifically for this project.
This email
address was continually monitored by one of the adult researchers to
track ongoing correspondence from the sites, such as promotional offers,
reminders or billing issues.
While students were encouraged to stay with their assigned programs, four of the
20 students requested permission to change to another service after alerting
adult researchers as to significant site performance and/or content issues with
their assigned sites. Researchers made note of the issues at hand and, if agreed
the student’s test preparation needs were not being met, granted permission for
students to evaluate another service.
Eight of the 10 services were evaluated by two or more students for a minimum
of 10, fourhour
sessions for a total of 40 hours per service. The two exceptions
were PrepMe and SAT Secrets; in these cases, students requested permission
to terminate participation in these services. (For more information, see Results by
Service). In all, students logged more than 400 hours at Mediatech evaluating
services, with additional hours logged at home.
Students were instructed to deny permission to share personal information with
outside partners, or opt out of this process, when registering. This occurred for all
but two services: Kaplan’s and The Princeton Review. In the case with Kaplan’s,
a student accidentally opted in after becoming confused by a registration form.
With The Princeton Review registrations, a student who needed to have
materials mailed to a home address was required to opt in, in order to receive the
booklets. In both cases, students could later adjust their personal preferences to
opt out of the information exchange.
All registration fees were paid by WebWatch, and each student was given a $5
cash incentive at the end of each test preparation session.
Data Collection
Both quantitative and qualitative data were collected. Students took screen
captures of each site during the registration process, then used a personal log to
document any site errors, events or comments during the testing process. At the
end of the 20hour
period, researchers used a checklist designed to capture
strengths and weaknesses about each test preparation service to interview each
student. These checklists were tallied and scored, providing more specific
information about each service’s strengths and weaknesses (see Appendix 1).
Page 12
In addition, sections 611
of the instrument were designed specifically to examine
factors that have been associated with the trustworthiness of an Internet service.
These include such issues as identity, disclosure, payment and order
confirmation, privacy and the use of advertising.
Page 13
MAJOR FINDINGS AND CONCERNS
The online test preparation services evaluated in this study varied significantly in
price, features and quality. While, in most cases, student testers were pleased
with their online testprep
experience and would recommend this course of study
to others, the research also cast light on serious issues of aggressive advertising
and site performance users of such online services should be aware of.
Before detailing these and other observations, the limitations of this research
should be noted. First, Internetdelivered
services, including the 10 SAT testprep
services considered in this study, are objects in motion. The observations and
details described here occurred between July 21 and August 15, 2005.
Second, the group of 20 testers was not large enough to provide valid statistical
evidence. Instead, this survey information provides qualitative indicators of how a
small group of randomly selected student testers felt about a particular service.
Third, when it comes to the issue of SAT test performance, there are many
potentially confounding variables that can emerge, and sometimes did. These
can include test anxiety, exposure to outside testprep
materials that may not
have been disclosed, and learning disabilities, to name a few.
Despite these statistical limitations, this research provides a thorough qualitative
assessment of online testprep
site services and functionality, when the target
audience is young adults under age 18 making major decisions about their
futures.
Researchers interviewed each student to capture detailed individual reactions to
each testprep
site reviewed. Researchers used testers’ perceptions, as well as
culled observations during the testing process, to assess a site’s trustworthiness
and to arrive at the following findings:
Online Services Come with Aggressive Advertising
By registering for services, including Kaplan's, The College Board, Boston Test
Prep and The Princeton Review, students often received messages from banks,
college and military recruiters, or offers for financial aid or study aids. One email
message from The Princeton Review contained a link to a United States Air
Force recruiting form.
Additionally, The College Board, the maker of the SAT, mixed free “official” test
information, such as sample tests and parental guidance, with advertisements for
"The Official SAT Online Course," a $69.95 product, without disclosing the price
upfront.
Page 14
If students were careful to opt out of promotions when registering, nearly all email
solicitations were avoided. However, this process was not always obvious.
In some cases, such as with The Princeton Review and Kaplan’s, opting out was
not a forthright process. In the case of Boston Test Prep, promotional offers
arrived via email
despite students’ requests not to receive such offers.
Higher Price and Reputation Don’t Necessarily Make Better Products
Number2.com was the only completely free service tested, and it performed
exceptionally well against expensive, betterknown
products, such as The
Princeton Review and Kaplan's, which cost $399 each.
The College Board, the creator of the SAT, was a betterknown
brand than
Number2.com, but was buggy during testing and lacked interactive features
common on other sites. Testers also found The College Board’s computerized
essaygrading
service to be too general to be useful, compared with The
Princeton Review’s more specific human feedback.
Students’ Privacy Protection Questioned
Some sites contained only vague descriptions about what student information
would be shared, and with whom. Student testers expressed difficulty finding
information on business and marketing partnerships and what “opt in” actually
means when using online products for The Princeton Review, The College
Board, and Kaplan's SAT Online Prep. These definitions should be clearly
provided and easy to understand, considering the target audience is teenagers.
In all but one case – SAT Secrets – the process for storing or handling student
information, including credit card information, was not clearly defined, nor for a
student's personal mailing address, phone, email,
passwords and personal
interests.
In fact, with most sites evaluated, understanding who has access to students’
personal information is vague at best. Take, for example, this text from The
Princeton Review site:
"From time to time, we obtain lists of people who may be interested in our
products and services from thirdparty
marketing list suppliers. We use
these lists to send direct marketing promotions by mail or by email. We
do our best to only obtain these lists from suppliers who have optin
policies for thirdparty
marketing, but we are not responsible for the
conduct and policies of these suppliers. Please note that even if you are
a registered customer of ours and have asked us not to use the
information you provided to us to send you promotional material, a thirdparty
marketing list supplier could still provide us with your name and
Page 15
address, in which case you will receive the promotional material sent by
us to others on the supplier's list. If you have received a promotion from
us and do not wish to receive future promotions, please contact us at:
privacy@review.com."
(http://www.princetonreview.com/footer/privacy_policy.asp)
Given the price of the services being sold – some approaching $500 – and the
confidential nature of student information being collected, it is clear online testprep
services need to do a better job disclosing who is behind the service, what
information is collected, and how student information is used.
Site Functionality Influences Perception of Trust
The overall design and functionality of a site contributed not only to higher
student satisfaction, but a higher level of trust in that service, testers said. When
asked to react to designrelated
site elements, such as ease of use, interactive
features and technical robustness (see Appendix 1), students gave high marks to
Kaplan's, Number2.com, Boston Test Prep, Petersons and The Princeton
Review. Of these, Kaplan’s and The Princeton Review were singled out by
testers for providing the best combination of content, coaching methods and
individualized record keeping. The College Board’s Official SAT Online Course
was deemed trustworthy despite having design problems, because, as student
testers noted, The College Board also publishes the SAT test.
Three of the services – SAT Secrets, PrepMe and the Test Preparation Program
– were found by students to have significant design problems, including a
frustratingly complex signin
process and poorly designed user interface. These
issues interfered with site functionality and, therefore, user trust.
Interactivity Distinguishes Online Services, Needs Improvement
Making online test prep more than just a book on a computer was best achieved
by Kaplan's SAT Online Prep and Number2.com, according to students. Testers
particularly appreciated these sites’ immediate feedback functions and efforts to
organize content into shorter sections, which made material easier to digest.
Testers also liked being able to track their improvement via recordkeeping
features, tools that heightened the online experience over traditional testprep
books. A site’s use of audio, Flashbased
video and other multimedia
components rather than merely listing long sections of questions – as one might
find in a book – was valued by student testers.
However, all sites tested lagged behind other electronic educational and
entertainment materials in terms of dynamic interactive features. When asked
about each site’s use of interactive techniques – including contextsensitive
help,
blogs by other students, rollover hints, word definitions, Flashbased
games,
Page 16
autocorrecting and autoleveling
– students said all services tested could
improve.
Errors Abound
Mistakes in online sample tests – including grammatical problems, questions with
no answers, missing sections of text, font problems, or poorly constructed
questions – were consistently present in 6 of the 10 services evaluated, but were
particularly problematic with Barron's Test Prep, Peterson's Online SAT Course,
PrepMe and Online Test Prep. These errors diminished students’ trust in these
sites as scholarly resources.
Only one of the 10 services tested — Boston Test Prep — offered a means for
reporting errors.
Site Identity Often Confusing
Three of the services tested – Number2.com, Boston Test Prep and Barron's –
provided more than one company name during the registration and payment
process, raising questions about which organization was actually behind these
services and the relationship between the firms.
In the most troubling case, researchers discovered the Test Preparation Program
was registered to “Nicole Smith” in San Antonio, Texas, with an Internet service
provider located overseas (ZoomBangkok.com). The site did not respond to, nor
did it comply with, testers’ requests for refunds.
In addition to confirming the identity of the bricksandmortar
institutions that
publish each testprep
site, researchers reviewed whether sites provided the
credentials or identities of human essay readers, particularly since at least one
site touted the use of “Harvardeducated”
readers. In only one case, PrepMe,
were readers’ qualifications and expertise detailed.
Technical Glitches Hamper Site Performance
Over the course of four weeks of testing, students logged hundreds of glitches
and errors. These could be categorized in two ways – errors due to the service
provider (service outages, problems logging in, sluggish performance as a result
of the publisher's hardware); and those caused by the student’s computer or
browser (font errors, missing plugins,
popup
blockers).
In general, most of the services performed fairly reliably on a daytoday
basis,
as long as a newer version of Microsoft Internet Explorer browser was used on
aWindows XP computer system. When sites were browsed using an older
Macintosh computer or a more obscure browser – such as Mozilla Firefox,
Netscape or Apple Safari – there were more problems, including screen layout
Page 17
issues, missing fonts or sections of text, or an inability to log in. New versions of
browsers are therefore recommended or required, with current plugins and the
ability to use cookies turned on. Unfortunately, this also makes a browser more
vulnerable to popup
advertising and adware from other Web sites. (Note that
systematic testing with different browsers was not part of this study.)
Students rated Boston Test Prep, Kaplan's SAT Online Prep and Number2.com
as generally reliable. Also good, but with minor glitches or occasional server
outages, were Barron's Test Prep, The Princeton Review and Peterson's SAT
Online Course. Students reported a pattern of both server and browser errors
with The Official SAT Online Course, PrepMe and Test Preparation Program.
Customer Service Generally Good
Researchers reviewed each site’s payment and order confirmation process to
assess the proper handling of confidential credit card information. For three of
the 10 services reviewed – Test Preparation Program, PrepMe and Peterson's
SAT Online Course – testers experienced either improper credit card charges or
didn’t receive services that were promised. In two cases, researchers were able
to resolve the problems by calling sites directly. Because there was no phone or
email
contact information for the Test Preparation Program, researchers were
unable to obtain a refund.
Page 18
RESULTS BY SERVICE
Barron's Test Prep
Barron's Educational Series, Inc.
$10.95 for 6 months
http://www.barronstestprep.com
Basic, easy to use and affordable ($10.95 for six months of access), testers
found Barron's Test Prep to be lacking in features but effective for taking selfscoring
practice exams. Barron's trustworthiness was eroded by the use of five
different publisher names on the site. Barron’s, Barron’s Test Prep,
Barronseduc.com, Barron’s Educational Series, Inc. and Plug & Play
Technologies, Inc. were all used at various points in either the registration
process or in the initial pitch screen, leaving the consumer to wonder, “Will the
real Barron's please stand up?”
After a straightforward signin
process, students were presented with a summary
of their activities, including prior SAT practice test scores. Site content included
long lists of dry SAT hints and techniques and graded practice exams. One tester
found the math section to be harder than the actual SAT test. In addition, some
of the same questions were used on different practice tests. One tester wanted
better help on some of the questions and found the essayscoring
feature to be
"useless” because essays were returned with a computergenerated
score.
The vocabulary section was nothing more than lists of highfrequency
SAT
words, with standard dictionary definitions. Testers noted that their trust in
Barron's was eroded by the low price ("what's the catch?"), an “unprofessional”
look to the graphics and presentation, and the confusing publisher identity.
Boston Test Prep
Boston Test Prep
$99 for 3 months
http://sat.bostontestprep.com
Wellorganized
and easy to use, Boston Test Prep (BTP) offered a
comprehensive service with some innovative options, like the ability to interact
with other students online. While testers found the site to be dry and lacking in
interactive features, they liked the clean design, with the math, writing and
reading sections broken into easily digested subsections. They also liked having
the ability to custom build custom quizzes around specific weaknesses.
No essaygrading
features were noticed, just general writing tips and sample
essays for students to compare. Testers were disappointed to find that one of the
essaywriting
options consisted of nothing more than a PDF file of a blank sheet
Page 19
of essay paper. One tester submitted a question using the "contact us" form
asking if an essay grading option was available, but did not get a reply.
To its credit, BTP is one of the only sites to ask for suggestions on how to
improve the site, by way of a "Bug Submission Form" to report errors using email.
Trust in BTP was diminished both by unfamiliarity with the brand name, and
the appearance of more than one organization’s name in the purchasing process.
This was noticed on the Mediatech's credit card statement, when a company
called "Acadient, Inc." was listed instead of Boston Test Prep, as the company
that processed the registration fees. During registration, testers, who had
difficulty finding a phone number and confirming the site’s identity, were relieved
to find a Boston, Mass., mailing address.
BTP was the only online testprep
service tested to use a potentially confusing
"free trial" registration system, with an automatically renewable subscription plan.
If the user selected the monthly plan, BTP would automatically bill the credit card
for $99, even if the service was not used. Researchers were able to cancel the
service after the testing period, by calling technical support.
The first attempt to register resulted in a cryptic error message, requiring a
successful call to technical support, which blamed the problem on an "upgraded
server."
Advertising appeared on the home page for additional BTP products, but not
inside the testprep
areas. While there was a stated privacy policy, we could not
find a way to opt out of email
correspondence. For example, on September 1,
2005, both testers received the following email
promotion:
"Do you have an MP3 player? (if not, here is a new reason to get one.)
Boston Test Prep has partnered with Audible.com to bring you BTP TO
GO:SAT Prep Talk. Part motivation, part preparation, SAT Prep Talk gives
you the tools and techniques to conquer the new SAT. SAT Prep Talk is a
great compliment to our online SAT prep program. When you can't be
online, put your headphones on and keep preping. Download our SAT
Prep Talk MP3 today."
Kaplan's SAT Online Prep
Kaplan, Inc.
$399.00 for 6 months
http://www.kaptest.com
Part of The Washington Post Company, Kaplan's SAT Online Test Preparation
provided a welldesigned
but expensive service that was easy to use and full of
stateoftheart
tutorial features. SAT Online Prep was just one of a variety of
services for sale at kaptest.com.
Page 20
After some initial trouble finding the login page, students were able to sign in and
take a "course introduction" section of the site. The features were clearly
explained in both text and audio, which students liked. After a diagnostic exam,
Kaplan's presented a personal study plan based on the user’s stated anticipated
SAT test date. For example, one student was encouraged to begin with her
weaker subject area and proceed to her stronger subjects, math and writing.
The service provided comprehensive progress tracking, using rollover
graphics
and colorcoded
graphs. Students liked the extra handholding
when they
needed it, and Kaplan's provided it, by way of remedial lessons, immediate
feedback, and stepbystep
assistance. These features made Kaplan's the most
liked of any of the services used by our testers.
Kaplan's SAT Online Test Prep was run by a known company, with functional email
and telephone contact numbers. Refund policies were defined, and Kaplan's
used a "higher score or moneyback
guarantee" that promised additional testprep
help at no cost, if a higher score was not provided. This guarantee was not
tested as part of this study.
However, of all the online testprep
services, Kaplan's used a troubling technique
to harvest and share student information. Trying to avoid giving permission to
release personal information on a Kaplan's registration form is like a game of
whackamole,
due to “springloaded”
optin
fields that repopulate themselves as
a default condition, unless a vigilant student notices and reunchecks
them. For
example:
Kaplan's Registration, Screen 1: Note that the three "opt in" boxes are not
checked, and one of the more obscure fields ("How did you hear about Kaplan's
Test Prep and Admissions") has been overlooked, which is easy to do, because
it looks full.
Page 21
Kaplan's registration, Screen 2: The checkout form was rejected, with a request
to fill in the missing field (top arrow). Kaplan's also populates, or checks two of
the three optin
boxes, so that they have to be manually unchecked each time
the form is viewed. Because these are located away from the CONTINUE box, a
student may overlook the boxes.
Number2.com
Xap Corp.
Free no
time limit
http://www.number2.com
This welldesigned,
free service combined depth of content, record keeping, and
individualized features with SAT test practice. Number2 matches up with sites
costing $500, and in some cases does things better. So, what's the catch?
Number2.com was created by Xap, Corp., a company that sells software and
services to college admissions personnel. Xap is no stranger to the college
admissions business, and access to the eyes and ears of thousands of soontobe
college students obviously is of key interest to Xap's clients. While it is
Page 22
possible to opt out of email
solicitations, this is not the default condition. We
could find no evidence of direct selling to students. In addition, Number2.com
was the only service to include banner advertising (for a bank) throughout the
test preparation portion of the service.
Because of a lack of need for credit card and subscription verification steps, the
registration process for Number2.com was the easiest of any other site, requiring
only a name, address and valid email
account. It was possible to register using
fictional information, although a valid email
address was required to access the
login information.
Number2.com generated weekly email
messages to remind students of their
impending date with the SAT, as follows:
"It's been over five weeks since you logged into your account at
Number2.com for your SAT prep course. Remember, practicing at regular
intervals (at least once a week) is the best way to prepare for the SAT.
Since you have not visited Number2.com for so long, we're going to
assume now that you don't want any more reminders. We won't send you
any more tutorial emails, unless you log in again, in which case they will
automatically restart. If you want to continue receiving emails, don't worry.
You can restart the emails at any time by just logging into your account.
And we'll keep your account active for at least a year from signup. If you
have forgotten your password, you can get a new password by going to
the lost password page at <link to signin.> Or you can go straight in by
following this link <link to student account> For your convenience,
following this link will automatically log you in; (but to protect your security,
it will only work once). With most email readers you can just click on the
link, but for some you need to cut and paste the link into the web browser.
Good luck with your test prep!
The service maintained a student page with bookmarks for where the user left
off, as well as hints, test tips and daily questions. Number2.com used email
to
reach students, sending out a daily tip and a progress report, and researchers
encountered no evidence of using email
to tempt students with upgrades, a
chance to win a laptop, surveys or ROTC scholarships, as with other sites tested.
Testers reported that using the service was straightforward, with tips, quizzes
and plentiful hints. Students reported that Number2.com was especially helpful
with the math section of the SAT by providing instant feedback for questions,
along with hints if needed.
The site explained that revenue is generated from sponsorships and licensing,
but does not disclose sources or provide details.
Page 23
Peterson's SAT Online Course
The Thomson Corporation
$169.95 for 4 months
http://www.petersons.com
This testprep
service costs about half of Kaplan's and The Princeton Review and
promised customized lessons, practice tests and expert strategies, including
essay help from "a Harvardeducated
editor." Both testers were happy with their
Peterson's testprep
experience, recommending it to others.
After a straightforward signin
and registration process, the program started with
a diagnostic test. Based on the test results, each was assigned a "Customized
Learning Path," with lessons based on weaknesses. Testers were able to choose
to take lessons and tests as often as needed in each area, finding the site flexible
and easy to read, with helpful graphics.
The essaygrading
section was not as good as promised. There was no
coaching, other than general tips. After submitting their essays, testers
immediately received an autogenerated
score from 1 to 6, on each of the five
sections of a computergenerated
scoring rubric, along with a message stating
that their essay would be scored by a Harvardeducated
editor after one to two
days. Neither tester received a response. Below is a screen grab documenting
the process for submitting an essay, along with a sample question.
Despite being a solid and useful testprep
service, several factors decreased
student trust in Petersons. It was difficult to pin down the exact name of the
company (Petersons, Thomson Petersons and EssayEdge were each used at
various points), and questions about the essaygrading
policy were not
answered. The site was highly commercialized, selling college consulting and
help finding scholarships through banner advertising and email
correspondence.
During registration, testers were able to successfully optout
of receiving
promotional messages.
PrepMe
PrepMe Corporation
$499.99 for 4 months
Page 24
http://www.prepme.com
PrepMe combined online advice, a box of books worth about $38 shipped directly
from Amazon.com, and phone/email
access to a live tutor who, according to
promotional materials, had already "aced" the SAT.
The online registration process for PrepMe was primitive at best. After filling in
the required information, including a valid creditcard
number, an error message
appeared: "Transaction processed. However we have encountered a problem.
Click to retry post to merchant." After two retries, testers tried a third time using a
different computerbrowser
combination, resulting in the same error. We then
contacted the telephone number given as technical support, which turned out to
be a mobile phone. We were told that our credit card had been billed $1,499.97.
The problem, according to the company, was, "we just switched servers to
Yahoo." Test administrators were relieved to reach help, and were able to
successfully get two registrations refunded.
After the cumbersome online registration process, both testers received a
welcome email
from their “live tutors,” ready to help. The testers were interested
in the online component of PrepMe, which consisted of a series of slides of
poorly written hints, tips and suggested strategies for "acing" the SAT. There was
no way to practice what was learned from the slides, and no writing exercises. In
addition, grammar and spelling mistakes were noted throughout the testprep
portion of the service, which lowered student trust in the program as a scholarly
resource. Finally, both students noted feeling uncomfortable sharing
correspondence with a stranger online, even though this person was a hired
tutor.
Because of these issues, both students eventually asked to be put on another
program, one within four hours and the other after five days. Test administrators
were successful in receiving a partial refund for one of the students ($499.97
minus $38, the cost of the Amazon.com books).
PrepMe provided a legal business name, physical address and background
information online, and gave every appearance of being an established
company. However, information about the site’s length of service was not
obvious, and there was no defined privacy policy.
The service offered a 100point
increase guarantee, or the student would be
allowed to take the course again at no cost, for as many as three times. The
guarantee only applied to a student who scored below 2260.
Both students found the online component of PrepMe to be crudely done, "like
reading a really bad book on the computer screen." Both PrepMe tutors made
diligent efforts to contact the testers by email.
As a way to connect with a real
Page 25
tutor, PrepMe may have been effective, but as an SAT online testprep
resource,
it did not meet testers’ needs.
SAT Secrets
Morrison Media LLC
$49.95
http://www.secretsstudyguide.com/sat/
Our two testers dropped out of this $49.95 service, as it actually was more of an
elaborate order form for a 145page
book previously published by Morrison
Media to "teach the test." This book is available in either electronic (pdf or Word
document) or print form. There were no interactive features on the site.
Unfortunately, the only way to learn the true nature of the program was to
register and pay the $49.95.
As part of the order, we received a special offer in the form of a report titled "How
to Overcome Test Taking Anxiety" and "Contact Information for Major Colleges
and Universities."
The publisher, Morrison Media, sold a variety of other publications, such as
"What your test score can tell you about your IQ" and "Scholarship Secrets."
While the sales pitch of the site was misleading, the company provides sufficient
background and working contact information, and answered the tech support
line, offering a "100% cash refund within one year if not satisfied.” Researchers
tested this moneyback
guarantee in January 2006, six months after testing, and
promptly received a full refund from Morrison Media within 10 days.
Test Preparation Program Online Test Prep
$29.95 for 3 months
http://www.onlinetestprep.com
Poorly designed, full of errors, and hard to use, both testers dropped out of this
testprep
service, one after one day, and the other after five days. The site was
functional enough to take a $29.95 registration fee, twice. The testprep
experience was ridden with errors, and not interactive.
The program progress and practice tests contained frequent spelling and
grammatical errors, including words like "whore" instead of "where." Content
included noninteractive
pages of techniques, tips and strategies that students
were told to memorize before going on to the practice tests.
This service lost trust quickly, with 20 nonfunctional
links on the home page, and
no information about the publisher. No technical support number or address was
Page 26
listed, and payment was handled by PayPal, a thirdparty
service. Unfortunately,
one of the registered students could not sign back into the service and there was
no recourse due to the site’s lack of technical support.
Using Network Solutions’ “WHOIS” function to learn more about who the site is
registered to, we discovered the listing “Nicole Smith” in San Antonio, Texas, with
an Internet service provider located overseas (ZoomBangkok.com). Email
requests for a refund or for help signing in were not answered.
The Official SAT Online Course
The College Board
$69.95 for 4 months
http://store.collegeboard.com
Promoted as "created by the same experts who created the SAT," The Official
SAT Online Course was one of hundreds of items sold at The College Board's
online store, along with books, reports, and teaching materials that deal with the
SAT Reasoning Test. The fact that this service was published by The College
Board – a detail highlighted in marketing efforts – helped boost its perceived
trustworthiness among testers. One student said, "Considering that they make
the SAT and I registered with them already to take the actual SAT, I trust them."
The service was easy to use, providing plenty of testtaking
practice, especially
for math, but was lacking in interactive features. The service was also buggy,
with more server down time than the other services. In one case, a section of the
test appeared twice. Another page was missing an essay, making it impossible
for the student to answer a set of questions. Testers complained that the
presentation was dry ("like a book on the computer"), with tips and "lessons"
consisting of nothing more than noninteractive
slide shows.
The ordering process was straightforward, but the claim "students receive
detailed personalized feedback on all practice tests and quizzes" did not square
with the actual student experience, according to testers.
Researchers noted the promotion of "official" books and testprep
services, listed
alongside free testprep
resources, with no indication which were free, and which
were for sale. Items promoted included print books on the SAT, and a series of
subjectspecific
workbooks. Testers were able to effectively opt out of receiving
a monthly newsletter and being contacted by colleges.
The Princeton Review
The Princeton Review, Inc.
$399.00 for 4 months
http://www.review.com
Page 27
The Princeton Review's Online SAT testprep
program was available in three
formats: SAT ExpressOnline ($99), SAT Online ($399) and SAT LiveOnline
($599). Each provided identical online presentations, with different lengths of
access time, varying sets of printed materials shipped by mail. The SAT
LiveOnline service, which we did not test, included realtime,
online class
sessions with a live instructor. Researchers tested the $99 and the $399 services
and found both to be well designed and highly commercialized. The Princeton
Review offered the best essaygrading
service of any other service tested,
according to students. Other strong site characteristics included interactive
features, student recordkeeping
tools and diagnostic features.
WebWatch found The Princeton Review’s registration process to be timeconsuming.
Moreover, in one case, the site strongly compelled a student into
opting in so that books that were part of the $399 service would be mailed to his
home: "You have requested testprep
information but have not granted us
permission to send it to you. Please check the optin
box regarding Princeton
Review Test Prep products and services." Afterwards, the student received a
steady stream of promotional offers, including these:
"Upgrade your computer. Upgrade your life. Complete this form to receive
more information from our partner institutions about distance learning—
and all that it could offer you—and you’ll automatically be entered to win
an Inspiron 6000 laptop computer from Dell. Maybe it’s your time to
upgrade!"
"Air Force Reserve Officer Training Corps (ROTC) Scholarship program
for high school students is a great way to cover the costs of school so you
can get the most out of your college experience. Air Force ROTC
has scholarships that can cover up to 100% of your college tuition. ... Plus,
a scholarship from Air Force ROTC offers something traditional
scholarships don't—a guaranteed job immediately after graduation where
you can make a difference."
Note: This latter email promotion asked for the student's Social Security
Number on the first screen.
Students were asked for their prior SAT scores, high school grades and future
goals. The service then used this information to prescribe a complete study plan,
using frequent email
notices to remind students of their activity level or suggest
additional services. Testers noted the math questions were harder than those on
the actual SAT, and some of the multimedia features seemed off topic. But the
overall presentation was deemed useful for holding interest. In addition, students
liked the testtaking
advice and noted the math tips section was especially
helpful.
Like the other services tested, The Princeton Review was not without technical
glitches. Students experienced a significant amount of server down time. One
Page 28
day (August 9) in particular, an entire morning was lost due to an unexpected
server problem. Calls to technical support were always answered promptly,
though accurate information regarding the status of the server was not provided.
The Princeton Review's $399 testprep
offering was the service tested to provide
live, human feedback electronically on one of the submitted essays. Feedback
included personal comments and recommendations, as follows.
Hi <student name>, Your results are in! You earned a score of 12 on your
essay. Here’s what our graders had to say about your essay and how to
improve it to maximize your score on test day.
Content: Interesting introductory paragraph! Your argument is well
reasoned nicely
done. You go into a good level of detail with your
examples. Your examples are appropriate for the topic.
Other: This typewritten essay was very long. Please aware of the fact that
you will not be able to write this much on the real essay, which, as you
know, must be handwritten. You don’t have to give three examples. It’s
fine to go with one or two examples, as long as they are well developed
and you still reach two pages. Your conclusion is more important than a
third example. Your essay displayed a great historical awareness and a
talent for writing.
Overall, students rated The Princeton Review high in terms of trust. However,
testers did not know that The Princeton Review was located in New York City
and had no affiliation with Princeton University. While this fact was disclosed on
the books and print materials sent from The Princeton Review, researchers were
not able to find this clearly disclosed online – either on the home page or in the
"Who We Are" section of the Web site, perhaps explaining why testers were
confused.
While there were no advertisements in the password sections of the actual testprep
service, the student's signin
screen featured advertisements for additional
testprep
tools, mediatv.com, banks, loans, and colleges, as well as frequent
exposure to advertisements for service upgrades.
As with the other testprep
sites reviewed, The Princeton Review was vague at
best in describing how it collected student information, who its business partners
were, and what information was shared. The Princeton Review “guarantee” was
vague: “Your score is guaranteed to improve by at least 200 points combined, or
we will work with you again for free.”
Page 29
TIPS FOR STUDENTS AND FAMILIES
The SAT is designed to be challenging, and despite marketing claims made in
the promotional materials of many online SAT testprep
sites, there are no
secrets or tricks that can substitute for years of elementary, middle and highschool
education. Just as you can't grow a plant faster by watering it more, you
can't get a higher SAT score with lastminute
cramming.
When it comes time to the take the test, valid online testprep
services are one of
several ways students can familiarize themselves with the SAT format, and this
familiarity can reduce anxiety, according to Ken Hartman, the former Director of
Admissions and Guidance Services for The College Board. Other testprep
options, generally priced much lower than many online testprep
services,
include books, CDROMs,
or high school courses.
Based on this report, WebWatch makes these following recommendations to
students and parents:
* Costlier options are not always best. Consider free or affordable services, such
as Number2.com and others. Take advantage of free testprep
resources
available at The College Board (www.collegeboard.com).
* Make sure your browser, computer and Internet connection are uptodate
prior
to enrolling in online programs. Older browsers and slow Internet connections
can result in sluggish performance or errors in the way information is displayed
on the screen. In addition, be aware filters and popup
blocking software may
need to be disabled in order to use a commercial testprep
service.
* Before purchasing, contact the service by phone to confirm who is running it,
and that it will be possible to receive a refund if you cannot successfully log in.
Also during this call, ask the customer service representative about the site’s
privacy policy and whether a student’s private information will be shared with
companies and advertisers.
* Don't purchase services that won't be used. We found that once the mandatory
testing period was over, students’ use of the services quickly diminished. Unless
your student is selfmotivated
or can follow a schedule established prior to the
SAT test, online test prep may not be a viable option.
* Be vigilant and present during the registration process, to make sure your child
does not accidentally authorize the release of personal information, unless you
want him or her to.
Page 30
* Know that information sent to your child regarding college scholarships, student
loans, or additional testprep
services most likely is advertising, with business
motives.
Page 31
RECOMMENDATIONS FOR PUBLISHERS
Consumer Reports WebWatch recommends that online testprep
service
companies take these steps to improve their products:
· Clearly label advertising and establish more firm boundaries between
editorial, testrelated
content and promotions for test materials.
· Make it easier for users to sign in and begin using the service immediately
after registering and paying for the service.
· Clearly identify your business name and location. Provide a working
phone number and/or email
address so students and parents can follow
up if necessary.
· Make it easy for users to report errors.
· Assure your service functions on a variety of browsers. If it doesn’t, state
so clearly, upfront,
before students pay for services they won’t be able to
use.
· Be more diligent in checking your site for accuracy, from correcting
spelling and grammar errors to testing the validity and effectiveness of
sample tests.
· Keep in mind people purchasing online SAT testprep
services are
students and, therefore, legal minors. As a result, care should be taken to
guard student information and adhere to stated or bestpractice
privacy
policies..
· Clearly disclose business relationships and other affiliations.
Page 32
Appendix 1: Online SAT Test Preparation Evaluation
Instrument
Title of Service:
Browser & OS Used for Review:
Name of Reviewer:
Tester Code:
Date of Review:
Number of Hours Using the Service:
Start and end date that the service was evaluated: Start___ End___
KEY
A= Always (1 point) SE= Some Extent (.5 point) N = Never 0 points NA = Not
Applicable (not calculated)
PART 1: DESIGN
I. Ease of Use
The service is easy to install, register for, and use over an extended time.
A SE N NA
1.___ ___ ___ ___ The registration process is simple
2. ___ ___ ___ ___ Starting a session is easy
3. ___ ___ ___ ___ Ending a session is easy
4. ___ ___ ___ ___ It is possible to get in and out of menus and activities at any
point
5. ___ ___ ___ ___ The experience is responsive and easy to control
6. ___ ___ ___ ___ Instructions are clear and easy to follow
7. ___ ___ ___ ___ A clear, consistent menu system is used
NOTES:
II. Educational Validity
What do you walk away from the experience with, that you didn't have when you
first came to the service? The service provides an effective test preparation
service.
1. ___ ___ ___ ___Question sets match the actual SAT test
2. ___ ___ ___ ___The service considers the strengths and weaknesses of the
student
3. ___ ___ ___ ___The service creates an individual profile, including prior test
scores
4. ___ ___ ___ ___The service employs a varied presentation with a degree of
novelty
5. ___ ___ ___ ___Test Taking Skills: includes strategies, techniques & facts
about the test, before test day, stress techniques, last minute
advice
Page 33
6. ___ ___ ___ ___Critical Reading: The practice exams simulate the actual SAT
(pacing in critical reading, comprehension, short passages,
long passages, paired passages, word branches)
7. ___ ___ ___ ___Writing: Examples are given to judge that simulate the actual
SAT (pacing, style, common errors)
8. ___ ___ ___ ___Math: The practice exams simulate the actual SAT (gridins,
working backwards, calculator strategy, trigonometry, word
problems)
9. ___ ___ ___ ___Questions are well designed
10. ___ ___ ___ ___ Provides valid test taking advice and techniques
11. ___ ___ ___ ___Questions reflect the new SAT format
NOTES:
III. Motivational Techniques
The site uses state of the art graphic and programming techniques to provide an
engaging interactive media experience.
1. ___ ___ ___ ___ Allows the student to vary the order of studying
2. ___ ___ ___ ___ Uses audio feedback
3. ___ ___ ___ ___ Uses drag and drop features (e.g., move parts of sentences
around)
4. ___ ___ ___ ___ Uses rollover features (e.g., roll over words to see them
highlighted)
5. ___ ___ ___ ___ Uses flash and shockwave features
NOTES:
IV. Features
The service adapts and adjusts, using a students time wisely
1. ___ ___ ___ ___Remembers allow
bookmarking of student progress
2. ___ ___ ___ ___Individualizes Adjusts
in difficulty according to student level
3. ___ ___ ___ ___Is Diagnostic Recommends
the best path to success based
on survey or pretest information.
4. ___ ___ ___ ___Listens Allows
for student input into the process
NOTES:
V. Technical Robustness
The service works as promised, at different times of the day and on different
computers.
1. ___ ___ ___ ___Provides system requirements up front
2. ___ ___ ___ ___Provides an initial pass/fail system check
3. ___ ___ ___ ___Provides links to needed plugins
4. ___ ___ ___ ___Has reasonable system requirements
5. ___ ___ ___ ___Works from different computers
(e.g.,home/laptop/school/library)
Page 34
6. ___ ___ ___ ___Works with more than one browser
7. ___ ___ ___ ___ Does not have server down time
8. ___ ___ ___ ___ Can "limp" with an older system, and still function
NOTES:
PART 2: TRUST
VI. Identity
1. ___ ___ ___Provides legal name of the business
2. ___ ___ ___Provides phone contact
3. ___ ___ ___Provides email contact
4. ___ ___ ___Provides location of business (geographical address)
5. ___ ___ ___ ___A consistent company name is used throughout the
transaction
NOTES:
VII. Disclosure prior to purchase.
1. ___ ___ ___ ___ Features are clearly described
2. ___ ___ ___ The term or length of service is provided
3. ___ ___ ___ Notice of additional costs is provided
4. ___ ___ ___ Terms, condition and methods of payment are provided
5. ___ ___ ___ Restrictions, limitations or conditions of purchase are defined
6. ___ ___ ___ ___ Other limits are defined, e.g., geography, language or parent
approval
7. ___ ___ ___ ___ Refund policy information is provided
8. ___ ___ ___ ___ Guarantees are defined
9. ___ ___ ___ ___ Sponsors or affiliations are disclosed, and any other motive
other than test prep is identified
NOTES:
VII. Payment and order confirmation
1. ___ ___ ___ ___Can the consumer confirm the item/items selected before
finalizing the purchase?
2. ___ ___ ___ ___Are consumers provided with easytouse,
secure payment
mechanisms and information on the level of security such
mechanisms afford?
3. ___ ___ ___ ___Does a dispute resolution process exists?
NOTES:
VIII. Privacy
How does the service handle information that could be considered confidential?
This includes credit card information, physical address, phone, email, passwords.
Page 35
Information related to the test process includes test preferences, test scores,
performance data and essay questions.
1. ___ ___ ___Is it possible to request that your email information not be
distributed?
2. ___ ___ ___ ___Is there a privacy policy in place before the purchase,
describing how personal information (such as scores on
tests or essay content) will be handled?
3. ___ ___ ___Is the default condition one that does not distribute or share
personal information? (The student must indicate in some
way (e.g., check a box) to request additional information
and services during the registration process)
4. ___ ___ ___ ___ Does the service use student correspondence to market its
own products?
5. ___ ___ ___ ___ Does the service release student data to outside firms? Is
there evidence that the email address has been given to
third party groups?)
NOTES:
IX. Technical Support?
Does the service provide technical support?
1. ___ ___ ___ ___ By telephone (check listed numbers to see if they are
working)
2. ___ ___ ___ ___ By email contact (check for a response)
Rate your experience with customer service
not helpful very helpful
1 2 3 4 5 6 7 8 9 10
X. Advertising
1. Were external products advertised behind the sign in screen? Y N
(external products are products not produced by the publisher)
2. Were internal products advertised behind the sign in screen? Y N
(internal products are additional books, tapes or services offered by the service
provider)
XI. Trustworthiness
Rate this site on a scale of 1 to 10 on its general trustworthiness
not trustworthy very trustworthy
1 2 3 4 5 6 7 8 9 10
Page 36
Do you suspect that there is any other motive other than test prep? (yes, no) If
so, please define.
NOTES:
Page 37
Appendix II: Student Responses
Note: A minimum of two students used each service, independently of one another, for five fourhour
days. In some
cases, more than 2 students used the same service. Students were surveyed on only the services they used.
After 20 hours, each student was interviewed, guided by the checklist (Appendix 1) with results recorded both on paper
and on videotape. The scores were checked for validity in
other words, if two students gave different answers about the
Princeton Review's use of advertising or had not yet tried the essay grading feature on Kaplan’s, an adult research would
explore further. In some cases, as many as four students tested the same service.
The mean survey score is reported on this chart. Our goal was to eliminate variability among tester answers, or to
understand why there was variability, until we understood each product.
Barron's
Test Prep
Boston
Test Prep
Kaplan's
SAT
Online
Prep
Number2.
com
Peterson's
Online SAT
Course
PrepMe SAT
Secrets
Test
Preparation
Program
The
Official
SAT
Online
Course
The
Princeton
Review
DESIGN 63 76 86 83 74 31 43 31 64 72
Ease of Use 82 93 89 100 93 64 NA 64 96 79
Educational
Validity
70 85 95 82 82 30 43 23 93 91
Motivating 51 40 90 90 75 5 NA 10 40 75
Features 31 75 69 57 50 0 NA 25 31 38
Technical
Robustness
81 89 88 88 72 58 NA 31 59 80
TRUST 75 77 92 82 86 72 92 13 92 91
Identity 80 80 100 70 100 50 100 20 100 100
Disclosure 50 100 94 75 83 67 100 22 100 83
Payment &
Order
Confirmation
100 100 100 NA 67 60 100 0 83 100
Privacy 40 40 75 80 80 80 100 13 80 80
Technical
Support
Y N Y Y Y Y Y N Y Y
Noticed
advertising
outside prep
sections?
N N N Y N N N N N N
Page 39
Notice
advertising
inside the prep
sections?
N N Y Y Y Y Y N Y Y
Overall
trustworthiness
80 90 85 85 85 75 60 10 96 83
Page 40
Appendix III: Academic Profile of Student Testers
SAT Online Test Preparation Study Participant Data
Student # Gender Age (yrs) Prior Tutoring H.S. SAT Math
SAT
Reading SAT Writing SAT Essay HS GPS Total SAT
Birth
Order
1013 F 18 Y 1 510 530 600 8 3.8778 1640 2
1014 M 18 N 2 360 240 240 3 21,370 840 1
1016 F 17.4 N 3 420 450 460 8 3.2785 1330 3
1011 M 17.1 Y 3 490 440 550 8 3.7122 1480 3
1019 M 18 N 3 630 530 590 10 3.2929 1750 1
1015 M 17 N 4 480 430 530 8 n/a 1440 1
1021 M 17.6 N 3 650 700 650 10 3.9043 2000 1
1008 F 17.6 N 2 410 500 470 9 3.2484 1380 3
1012 F 17.7 Y 1 690 630 680 n/a 3.9253 2000 2
1003 M 17.7 N 2 670 500 450 7 3.1734 1620 1
1002 M 17.2 N 4 520 500 430 6 2.9475 1450 1
1020 M 17.6 N 5 690 630 640 9 n/a 1960 1
1001 M 17.2 N 2 520 550 550 8 2.5737 1620 1
1017 F 17.1 N 3 570 580 800 12 2.8265 1950 3
1005 F 17.6 N 2 470 380 370 6 4.0696 1220 twin
1004 F 17.6 N 2 520 400 340 6 4.1366 1260 twin
1006 M 16.8 N 3 470 470 420 7 2.7969 1360 3
1010 M 17.3 N 3 580 490 480 8 3.3231 1550 1
1022 F 17.4 N 2 620 590 590 10 4.444 1800 1
1009 F 17.4 N 2 530 680 580 7 3.9219 1790 1
540 511 521 7.5 1572
Page 42
Appendix IV: Student Tester SAT Scores, Before and
After Study, by Service
Gender Reading Math Writing Total Reading Math Writing Total Change
BTP F 530 510 600 1640 530 540 640 1710 70
BTP M 470 470 420 1360 * * * * *
Barrons M 500 520 430 1430 520 530 440 1490 60
Barrons F 680 530 580
PR
Express
M 490 580 540 1610 570 510 520 1600 10
PR
Express
F 400 520 340 1260 470 440 410 1320 60
Princeton
Review
M 500 670 450 1620 680 500 510 1690 70
Princeton
Review
F 590 620 590 1800 670 620 560 1850 50
Petersons F 500 410 470 1380 410 500 530 1440 60
Petersons M 700 650 650 2000 670 630 650 1950 50
College
Board
F 630 690 680 2000 College
Board
M 630 690 640 1960 640 680 710 2030 70
College
Board
M 550 520 550 1620 570 510 480 1560 60
Number 2 M 240 360 240 840 320 440 260 1020 180
Number 2 F 450 420 460 1330 440 480 470 1390 60
Number 2 M 530 630 590 1750 * * * * *
Number 2 F 580 570 800 1950 640 560 710 1910 40
Kaplan's F 380 470 370 1220 420 340 370 1130 90
Kaplan's M 470 510 450 1430 470 570 530 1570 140
Prepme M 430 480 530 1440 400 580 510 1490 50
Percent Total 620
Mean 38.75
Percent change 1.61%
* = did not take second exam due to a change in plans.
=
scores not yet turned in.
Page 44
1640 1710
Mean 1526 1562.666667
Variance 100940 84663.80952
Observations 15 15
Pearson Correlation 0.974145509
Hypothesized Mean
Difference 0.05
df 14
t Stat 1.918295991
P(T<=t) onetail
0.037849819
t Critical onetail
1.761310115
P(T<=t) twotail
0.075699638
t Critical twotail
2.144786681