Statistics experts label ISP filtering trials unscientific
The Federal Government’s ISP filter trials lack proper methodology and are not representative, according to experts in statistics and testing from two of Australia’s leading universities.
The criticisms come after two of the nine ISPs participating revealed only 15 of their customers, which in one case was 1 per cent of the total, chose to have their Internet filtered.
The vast majority of ISPs also used an opt-in system that requires users wanting to be filtered to request it.
“I would not have confidence in any of the results they find because of the way the sample has been constructed,” expert in statistics and senior lecturer at the Queensland University of Technology, Dr Daniel Johnson, said.
The professor contends a major reason why an opt-in system was selected was the difficulty in getting ISPs to force filtering on their users and said the result was a system without a strong basis in science.
“It’s probable that the results will be favourable for the survey because of the way they’ve collected data,” he said.
A good mandatory filter trial, Johnson said, would need a larger random sample set that couldn’t opt-out, rather than the opt-in system currently in place for most participating ISPs.
“Make people trial it for a week and then let them pull out if they need to because then at least you start with a more representative sample,” he said.
While the study was not easy to run it didn’t make the data gathered any more valid, Johnson said.
“I would expect that any professional in this area would state the shortcomings of their data in their findings,” he said.
Professor of statistics at Monash University and editor-in-chief of the International Journal of Forecasting, Professor Rob Hyndman, described an opt-in system as “about the worst way you can do it”.
“It would not be published in a statistical journal. It would never be of interest to statisticians because of the way it’s been conducted,” he said.
Hyndman went on to say while compulsory trials on randomised sample groups would be statistically ideal, the ethics of such an act would be questionable.
The Minister for the Department of Broadband, Communications and the Digital Economy (DBCDE), Senator Stephen Conroy, has previously stated the current trial’s success or failure will help form much of the government’s stance on mandatory nation-wide ISP filtering.
“We'll be guided by that trial. We've always said, consistently, we'll be guided by the trial,” the Minister said on the ABC’s Q&A program.
The company running the evaluation of the ISP filtering trial, Enex, was contacted for comment, but ARN was told due to a non-disclosure agreement between the company and the Federal Government, all queries on the ISP filtering trial had to be directed to the latter.
ARN contacted Senator Conroy’s office but did not receive a response by time of publication.
The Australian Bureau of Statistics was also contacted but the organisation declined to comment on the validity of the trial’s methodology.
Other leading tech experts have criticised the small number of participants and stated that while larger ISPs may be able to handle a national ISP filter, smaller providers may not.
The majority of the participating ISPs have revealed they are using Marshal8e6 filtering solutions as part of the trial.
Comments
global isp
Soon, the entire planet will be accessible with these global ISPs. They are also very competitive when it comes to price.
That bloody filter
Surely, surely, surely this "trial" will go on quietly for a decade now ? This has to be dead, doesn't it ? I have never seen a policy this unpopular. The smug dishonesty of it all is what gets me. The only thing consistent about Conroy's utterances has been the refusal to come out and admit what the "banned" list is going to entail, even though we know.
Show me yours first.
What you're essentially saying is that your (Enex') response to this debacle is "Since our testing methodologies are hidden behind Non-Disclosure Agreements and Commerical In Confidence, we're immune to peer review; ergo these academics can't possibly be talking about us; ergo our excrement still smells like roses."
Choose your bedfellows carefully, Enex: you never know what you'll catch from them.
Show me the Methodology
We are a large network engineering services group and we architect some of the most secure networks around and we test them too. I know Enex has been well respected for testing here for a long time.
Where are the methodologies that were analysed and commented on by these guys for this story? Can't see any reference to them in this article. Can someone from ARN post a link or point us in the right direction.
Thanks.
Propaganda Minister
Senator Stephen 'Goebbels' Conroy
Apparently not.
Apparently not.
Real results...
The methodology is useless for the analysis that Conroy's actually trying to do, but from my point it does show something useful. Surely the fact that only one percent of potential testers actually chose to do so clearly demonstrates that the number of Australians who want this filter imposed on them is also about one percent? I thought democratic government was supposed to represent the majority of voters?
Okay, let's first say what
Okay, let's first say what everyone who has been watching the trials with great interest has been saying - the trials are definitely rigged. The ISPs were handpicked by Stephen Conroy, excluding one of the largest in the country who wanted to perform a scientifically valid test - iiNet. No parameters for success have been defined and the test is clearly scientifically invalid.
Let's be generous and assume that the tests are a scientifically valid success, what would that mean?
Well they tested an opt-in filter of the full ACMA blacklist (including "adult nudity" and other R rated content). I don't think anyone would seriously object to such a filter being available to those who want it if it were implemented properly and not a significant drain on the taxpayers. Such a plan would have my thumbs up.
But that's not what the plan is. The plan is for a top-secret mandatory Iranian-style filter. You just can't test a mandatory filter on an opt-in basis.
Most of these ISPs selected
Most of these ISPs selected by conroy have what, less than 1% of the internet traffic? You can't tell how much the service speed degradation will be by selecting some of the smallest ISPs.
Someone as mindless as conroy shouldn't be allowed near the internet, we've all seen the kind of things the blacklist flags. MMOs and MMORPGs like Second Life, World of Warcraft etc, will be banned because there is no way to control what the online community says, so it all becomes RC.
Conroy takes a serious subject like child pornography and basically mocks it by accusing people who are anti-filter of being a pedophile, he is trying to create false protection, even the children protection groups agree it wont help!
He is being fueled by various religious communities, as proof when he said he was going to drop the filter, the religious communities were the ones to become angry.
The best result one can hope for
“We'll be guided by that trial. We've always said, consistently, we'll be guided by the trial,”
In many ways, that's the best news one could hope for. Even if the trail is a "success", then the most Conroy can do, is push for an ISP filtering system that is opt-in ONLY and limited to a small number of ISP customers. Since lets face it, that's exactly what the trail has tested, so in order to be consistent, the trial can't provide any more guidance past that.
The results prove something, but not what Conroy wants
At best, the trial results will prove that the filter does not slow internet access provided it is used as an opt-in service and that only 1% of an ISP's customer base chooses to use it.
No matter where Senator Conroy chooses to position the goal posts, he cannot use the results to make any assertions as to the effectiveness of the filter on an unwilling participant (i.e. someone who does want to view banned material), the impact on internet speed for more than 1% of an ISPs customers, or anything to do with the opt-in element for 'kiddie-safe' internet access. None of these aspects were tested at all.
I look after a team of engineers. If Conroy was working for me and presented these as his test results, I'd sack him for gross incompetence. Incomplete test coverage. No success criteria. Dodgy methodology. Fail!
Hardly surprising.
Strongly religious idiots like Conroy have a natural aversion to science and reality. They're consistently flat-out wrong about recognised scientific phenomena like global warming, evolution, gravity, etc.
Pity how they often seem to end up in easily-abused positions of power.
Disgraceful
Evidence based policy? What if the evidence isn't worth a damn to begin with?
Government transparency? ("Sen. Conroy's office did not respond before deadline", or "a spokeswoman for Sen Conroy refused to comment" are starting to become all too common sights)
Scientific process? Bwahahaha, I don't think Conroy even knows what this is. If he did, he would have been more careful about fudging his test. Seriously, this guy isn't even smart enough to fake a good test result...
What a joke.
Enex
I think Enex should be dragged through the dirt for being party to such a poor study, as should any academic (especially from RMIT) involved. I always knew that the business of securing government grants was a bit mercenary, but this takes the biscuit!
Ok, don't know what happened
Ok, don't know what happened there, anyway...
Don't forget that Conroy is waiting to see what the results of the trial are before setting the pass or fail criteria.
So no matter what the result may be he can now just move the goal posts to where ever he wants to make it seem favourable.
It's not only obvious that the trial has been a farce (15 user sample size!) from the beginning but the Government also knew that it would not succeed in providing the result they want so they have in effect rigged it to guaranty the outcome they want.
Say good bye to your credibility Enex, hope it was worth it.
Don't forget...
... that Conroy is waiting to see what the results of the trial are before setting the pass or fail criteria.
So no matter what the result may be he can now just move the goal posts to where ever he wants to make it seem favourable.
It's not only obvious that the trial has been a farce (15 user sample size...!) from the beginning but the Government also knew that it would not succeed in providing the result they want so they have in effect rigged it to guaranty the outcome they want.
Apparently "Proper" Lap tests, Industry Experts, ISPs and Internet Users saying the censorship won't work and is unwanted doesn't matter when you have a fixed trial paid for by the Government who want to implement this.
That's right, the filtering plan failed the first round of lab tests yet $100,000's of dollars have been spent to buy the result they want.
Conroy would be fired and never be hired in his industry again if he did this in any Corporate environment, but I guess it's ok to waste Tax payers money...
Say good bye to your credibility Enex, hope it was worth it.
Also...
The majority of the participating ISPs have revealed they are using Marshal8e6 filtering solutions as part of the trial.
I wonder how much Marshal8e6 stand to gain financially if ISP-level filtering is indeed made mandatory for all 22 million Australian citizens.
But I'm sure they only have our best interests at heart...
A good mandatory filter trial
A good mandatory filter trial, Johnson said, would need a larger random sample set that couldn’t opt-out, rather than the opt-in system currently in place for most participating ISPs.
Funny, that's exactly what iiNet wanted to do, but Conroy wouldn't let them. I wonder why...?
The company running the evaluation of the ISP filtering trial, Enex, was contacted for comment, but ARN was told due to a non-disclosure agreement between the company and the Federal Government, all queries on the ISP filtering trial had to be directed to the latter.
I hope Enex enjoy the $800,000 they were awarded for that contract.
ARN contacted Senator Conroy’s office but did not receive a response by time of publication.
Wow, there's a shock!
Post new comment