Baidu Inc. recently crowed about besting rivals such as Google
Inc. and Microsoft Corp. in a closely watched standard test of
artificial intelligence. On Tuesday, the Chinese technology giant
apologized for misleading the public about its accomplishment.
Baidu's gaffe is a sign of the increasingly high stakes as the
world's top tech companies race to develop technologies that allow
computers to recognize photographs, control robots, understand
spoken language and perform other tasks that until recently
required human attention. Big Internet companies, which have
amassed immense computing resources, are under pressure to assert
leadership in the field, both for prestige and the potential
commercial payoff.
In addition to Baidu, Google, and Microsoft, tech powerhouses
including Apple Inc., Facebook Inc. and International Business
Machines Corp. have been heavily recruiting AI researchers and
vying for bragging rights.
In mid-May, Baidu said it scored a record low 4.58% error rate
on the ImageNet test of image recognition. Microsoft's software had
a 4.94% error rate, and Google achieved 4.8% in their latest tests.
With practice, humans can achieve an error rate of about 5%.
On Tuesday, however, the volunteer computer scientists who
administer the test reported that Baidu had stacked the deck by
taking the test far more frequently than allowed. ImageNet allows
contestants to submit two sets of test results a week. Baidu made
more than 40 submissions over a five-day period in March, ImageNet
organizers said in a blog post Tuesday. They said the company set
up 30 accounts to submit about 200 test results over a six-month
period.
The ImageNet test is based on a database of about one million
photographs sorted into about 1,000 categories. Researchers
typically use one portion of the database to tune their
image-recognition software and another to test it. By taking the
test so many times, Baidu's engineers could have gained an
advantage by tuning their software to information that was supposed
to be unfamiliar.
"This is pretty bad, and it is exactly why there is a held-out
test set for the competition that is hosted on a separate server
with limited access," said Matthew Zeiler, chief executive of AI
company Clarifai Inc. and a past winner of the ImageNet
competition. "If you know the test set, then you can tweak your
parameters of your model however you want to optimize the test
set."
The organizers have asked Baidu to stop submitting ImageNet
results for the next year.
Baidu declined to answer questions about the incident, but in a
statement posted on the ImageNet website, Baidu scientist Ren Wu
apologized for what he called a mistake. "We have added a note to
our research paper...and will continue to provide relevant updates
as we learn more," he wrote. The statement offered no further
explanation.
Access Investor Kit for Baidu, Inc.
Visit
http://www.companyspotlight.com/partner?cp_code=P479&isin=US0567521085
Access Investor Kit for Google, Inc.
Visit
http://www.companyspotlight.com/partner?cp_code=P479&isin=US38259P5089
Access Investor Kit for Google, Inc.
Visit
http://www.companyspotlight.com/partner?cp_code=P479&isin=US38259P7069
Access Investor Kit for International Business Machines
Corp.
Visit
http://www.companyspotlight.com/partner?cp_code=P479&isin=US4592001014
Subscribe to WSJ: http://online.wsj.com?mod=djnwires