The Federal Government’s ISP filter trials lack proper methodology and are not representative, according to experts in statistics and testing from two of Australia’s leading universities.
The criticisms come after two of the nine ISPs participating revealed only 15 of their customers, which in one case was 1 per cent of the total, chose to have their Internet filtered.
The vast majority of ISPs also used an opt-in system that requires users wanting to be filtered to request it.
“I would not have confidence in any of the results they find because of the way the sample has been constructed,” expert in statistics and senior lecturer at the Queensland University of Technology, Dr Daniel Johnson, said.
The professor contends a major reason why an opt-in system was selected was the difficulty in getting ISPs to force filtering on their users and said the result was a system without a strong basis in science.
“It’s probable that the results will be favourable for the survey because of the way they’ve collected data,” he said.
A good mandatory filter trial, Johnson said, would need a larger random sample set that couldn’t opt-out, rather than the opt-in system currently in place for most participating ISPs.
“Make people trial it for a week and then let them pull out if they need to because then at least you start with a more representative sample,” he said.
While the study was not easy to run it didn’t make the data gathered any more valid, Johnson said.
“I would expect that any professional in this area would state the shortcomings of their data in their findings,” he said.
Professor of statistics at Monash University and editor-in-chief of the International Journal of Forecasting, Professor Rob Hyndman, described an opt-in system as “about the worst way you can do it”.
“It would not be published in a statistical journal. It would never be of interest to statisticians because of the way it’s been conducted,” he said.
Hyndman went on to say while compulsory trials on randomised sample groups would be statistically ideal, the ethics of such an act would be questionable.
The Minister for the Department of Broadband, Communications and the Digital Economy (DBCDE), Senator Stephen Conroy, has previously stated the current trial’s success or failure will help form much of the government’s stance on mandatory nation-wide ISP filtering.
“We'll be guided by that trial. We've always said, consistently, we'll be guided by the trial,” the Minister said on the ABC’s Q&A program.
The company running the evaluation of the ISP filtering trial, Enex, was contacted for comment, but ARN was told due to a non-disclosure agreement between the company and the Federal Government, all queries on the ISP filtering trial had to be directed to the latter.
ARN contacted Senator Conroy’s office but did not receive a response by time of publication.
The Australian Bureau of Statistics was also contacted but the organisation declined to comment on the validity of the trial’s methodology.
Other leading tech experts have criticised the small number of participants and stated that while larger ISPs may be able to handle a national ISP filter, smaller providers may not.
The majority of the participating ISPs have revealed they are using Marshal8e6 filtering solutions as part of the trial.