Statistics experts label ISP filtering trials unscientific

Statistics experts label ISP filtering trials unscientific

Trials for mandatory filtering would never be accepted in an academic statistics journal

The Federal Government’s ISP filter trials lack proper methodology and are not representative, according to experts in statistics and testing from two of Australia’s leading universities.

The criticisms come after two of the nine ISPs participating revealed only 15 of their customers, which in one case was 1 per cent of the total, chose to have their Internet filtered.

The vast majority of ISPs also used an opt-in system that requires users wanting to be filtered to request it.

“I would not have confidence in any of the results they find because of the way the sample has been constructed,” expert in statistics and senior lecturer at the Queensland University of Technology, Dr Daniel Johnson, said.

The professor contends a major reason why an opt-in system was selected was the difficulty in getting ISPs to force filtering on their users and said the result was a system without a strong basis in science.

“It’s probable that the results will be favourable for the survey because of the way they’ve collected data,” he said.

A good mandatory filter trial, Johnson said, would need a larger random sample set that couldn’t opt-out, rather than the opt-in system currently in place for most participating ISPs.

“Make people trial it for a week and then let them pull out if they need to because then at least you start with a more representative sample,” he said.

While the study was not easy to run it didn’t make the data gathered any more valid, Johnson said.

“I would expect that any professional in this area would state the shortcomings of their data in their findings,” he said.

Professor of statistics at Monash University and editor-in-chief of the International Journal of Forecasting, Professor Rob Hyndman, described an opt-in system as “about the worst way you can do it”.

“It would not be published in a statistical journal. It would never be of interest to statisticians because of the way it’s been conducted,” he said.

Hyndman went on to say while compulsory trials on randomised sample groups would be statistically ideal, the ethics of such an act would be questionable.

The Minister for the Department of Broadband, Communications and the Digital Economy (DBCDE), Senator Stephen Conroy, has previously stated the current trial’s success or failure will help form much of the government’s stance on mandatory nation-wide ISP filtering.

“We'll be guided by that trial. We've always said, consistently, we'll be guided by the trial,” the Minister said on the ABC’s Q&A program.

The company running the evaluation of the ISP filtering trial, Enex, was contacted for comment, but ARN was told due to a non-disclosure agreement between the company and the Federal Government, all queries on the ISP filtering trial had to be directed to the latter.

ARN contacted Senator Conroy’s office but did not receive a response by time of publication.

The Australian Bureau of Statistics was also contacted but the organisation declined to comment on the validity of the trial’s methodology.

Other leading tech experts have criticised the small number of participants and stated that while larger ISPs may be able to handle a national ISP filter, smaller providers may not.

The majority of the participating ISPs have revealed they are using Marshal8e6 filtering solutions as part of the trial.

Follow Us

Join the ARN newsletter!

Error: Please check your email address.

Tags ISP filter trials



 IN PICTURES: VeeamON Tour in Sydney and Melbourne (+ 17 photos)

IN PICTURES: VeeamON Tour in Sydney and Melbourne (+ 17 photos)

The VeeamON tour showcased company's upcoming release of the Veeam Availability Suite v9. Veeam product strategist, Rick Vanover led the discussions in Sydney and Melbourne about the upcoming v9 and other Veeam innovations for the new year.​ VeeamON was held on September 23 at the Museum of Sydney, Warrane Theatre, and at Melbourne Museum in Melbourne on September 24.

IN PICTURES: VeeamON Tour in Sydney and Melbourne (+ 17 photos)
 IN PICTURES: Greentree A/NZ Partners Explore Japan (+ 13 PHOTOS)

IN PICTURES: Greentree A/NZ Partners Explore Japan (+ 13 PHOTOS)

As part of a regular strategic planning program for select Greentree A/NZ partners, they enjoyed a week in Japan combining strategic planning sessions with sightseeing. Principals and their partners from Star Business Solutions, Endeavour Solutions, Addax Business Solutions, GT Business Solutions, bizlinkIT and Verde Group joined Greentree chief executive, Peter Dickinson, channel director, Graham Hill, and R&D director, Stephen Sims in Tokyo and took in Hiroshima, Kyoto and Hakone, including travel on the famed bullet train which hit a top speed of 296 KPH.

IN PICTURES: Greentree A/NZ Partners Explore Japan (+ 13 PHOTOS) is a channel management ecosystem that automates all major aspects of the entire sales, marketing and service process, including data tracking, integrated learning, knowledge management and product lifecycle management.

Show Comments