Statistics experts label ISP filtering trials unscientific

Trials for mandatory filtering would never be accepted in an academic statistics journal

The Federal Government’s ISP filter trials lack proper methodology and are not representative, according to experts in statistics and testing from two of Australia’s leading universities.

The criticisms come after two of the nine ISPs participating revealed only 15 of their customers, which in one case was 1 per cent of the total, chose to have their Internet filtered.

The vast majority of ISPs also used an opt-in system that requires users wanting to be filtered to request it.

“I would not have confidence in any of the results they find because of the way the sample has been constructed,” expert in statistics and senior lecturer at the Queensland University of Technology, Dr Daniel Johnson, said.

The professor contends a major reason why an opt-in system was selected was the difficulty in getting ISPs to force filtering on their users and said the result was a system without a strong basis in science.

“It’s probable that the results will be favourable for the survey because of the way they’ve collected data,” he said.

A good mandatory filter trial, Johnson said, would need a larger random sample set that couldn’t opt-out, rather than the opt-in system currently in place for most participating ISPs.

“Make people trial it for a week and then let them pull out if they need to because then at least you start with a more representative sample,” he said.

While the study was not easy to run it didn’t make the data gathered any more valid, Johnson said.

“I would expect that any professional in this area would state the shortcomings of their data in their findings,” he said.

Professor of statistics at Monash University and editor-in-chief of the International Journal of Forecasting, Professor Rob Hyndman, described an opt-in system as “about the worst way you can do it”.

“It would not be published in a statistical journal. It would never be of interest to statisticians because of the way it’s been conducted,” he said.

Hyndman went on to say while compulsory trials on randomised sample groups would be statistically ideal, the ethics of such an act would be questionable.

The Minister for the Department of Broadband, Communications and the Digital Economy (DBCDE), Senator Stephen Conroy, has previously stated the current trial’s success or failure will help form much of the government’s stance on mandatory nation-wide ISP filtering.

“We'll be guided by that trial. We've always said, consistently, we'll be guided by the trial,” the Minister said on the ABC’s Q&A program.

The company running the evaluation of the ISP filtering trial, Enex, was contacted for comment, but ARN was told due to a non-disclosure agreement between the company and the Federal Government, all queries on the ISP filtering trial had to be directed to the latter.

ARN contacted Senator Conroy’s office but did not receive a response by time of publication.

The Australian Bureau of Statistics was also contacted but the organisation declined to comment on the validity of the trial’s methodology.

Other leading tech experts have criticised the small number of participants and stated that while larger ISPs may be able to handle a national ISP filter, smaller providers may not.

The majority of the participating ISPs have revealed they are using Marshal8e6 filtering solutions as part of the trial.

Sponsored Content: Collaboration has become the new movement in IT. Servers will become an integral part of this industry transition. Click here to learn more.

Join the ARN newsletter!

Error: Please check your email address.

Tags ISP filter trials

More about ABC NetworksABC NetworksARNAustralian Bureau of StatisticsMarshal8e6Monash UniversityMonash UniversityQueensland University of TechnologyQueensland University of Technology

 

Latest News

07:24AM
Worst security breaches of the year 2014: Sony tops the list
06:23AM
Messaging app Line buys Microsoft's MixRadio music-streaming app
06:16AM
Vulnerability in embedded Web server exposes millions of routers to hacking
04:40AM
Top 4 things that changed networking in 2014
More News
05 May
CeBIT Australia 2015
27 May
World Business Forum Sydney
View all events