Statistics experts label ISP filtering trials unscientific

Trials for mandatory filtering would never be accepted in an academic statistics journal

The Federal Government’s ISP filter trials lack proper methodology and are not representative, according to experts in statistics and testing from two of Australia’s leading universities.

The criticisms come after two of the nine ISPs participating revealed only 15 of their customers, which in one case was 1 per cent of the total, chose to have their Internet filtered.

The vast majority of ISPs also used an opt-in system that requires users wanting to be filtered to request it.

“I would not have confidence in any of the results they find because of the way the sample has been constructed,” expert in statistics and senior lecturer at the Queensland University of Technology, Dr Daniel Johnson, said.

The professor contends a major reason why an opt-in system was selected was the difficulty in getting ISPs to force filtering on their users and said the result was a system without a strong basis in science.

“It’s probable that the results will be favourable for the survey because of the way they’ve collected data,” he said.

A good mandatory filter trial, Johnson said, would need a larger random sample set that couldn’t opt-out, rather than the opt-in system currently in place for most participating ISPs.

“Make people trial it for a week and then let them pull out if they need to because then at least you start with a more representative sample,” he said.

While the study was not easy to run it didn’t make the data gathered any more valid, Johnson said.

“I would expect that any professional in this area would state the shortcomings of their data in their findings,” he said.

Professor of statistics at Monash University and editor-in-chief of the International Journal of Forecasting, Professor Rob Hyndman, described an opt-in system as “about the worst way you can do it”.

“It would not be published in a statistical journal. It would never be of interest to statisticians because of the way it’s been conducted,” he said.

Hyndman went on to say while compulsory trials on randomised sample groups would be statistically ideal, the ethics of such an act would be questionable.

The Minister for the Department of Broadband, Communications and the Digital Economy (DBCDE), Senator Stephen Conroy, has previously stated the current trial’s success or failure will help form much of the government’s stance on mandatory nation-wide ISP filtering.

“We'll be guided by that trial. We've always said, consistently, we'll be guided by the trial,” the Minister said on the ABC’s Q&A program.

The company running the evaluation of the ISP filtering trial, Enex, was contacted for comment, but ARN was told due to a non-disclosure agreement between the company and the Federal Government, all queries on the ISP filtering trial had to be directed to the latter.

ARN contacted Senator Conroy’s office but did not receive a response by time of publication.

The Australian Bureau of Statistics was also contacted but the organisation declined to comment on the validity of the trial’s methodology.

Other leading tech experts have criticised the small number of participants and stated that while larger ISPs may be able to handle a national ISP filter, smaller providers may not.

The majority of the participating ISPs have revealed they are using Marshal8e6 filtering solutions as part of the trial.

2014 ARN Women in ICT Awards - Nominate Now!: Nominations have opened for WIICTA 2014 and will stay open until October 22. But don't be late, be among the first in and NOMINATE NOW!!!

Tags ISP filter trials

More about ABC NetworksABC NetworksARNAustralian Bureau of StatisticsMarshal8e6Monash UniversityMonash UniversityQueensland University of TechnologyQueensland University of Technology

20 Comments

Anonymous

1

A good mandatory filter trial

<em>A good mandatory filter trial, Johnson said, would need a larger random sample set that couldn&#8217;t opt-out, rather than the opt-in system currently in place for most participating ISPs.</em>

Funny, that's exactly what iiNet wanted to do, but Conroy wouldn't let them. I wonder why...?

<em>The company running the evaluation of the ISP filtering trial, Enex, was contacted for comment, but ARN was told due to a non-disclosure agreement between the company and the Federal Government, all queries on the ISP filtering trial had to be directed to the latter.</em>

I hope Enex enjoy the $800,000 they were awarded for that contract.

<em>ARN contacted Senator Conroy&#8217;s office but did not receive a response by time of publication.</em>

Wow, there's a shock!

Anonymous

2

Also...

<em>The majority of the participating ISPs have revealed they are using Marshal8e6 filtering solutions as part of the trial.</em>

I wonder how much Marshal8e6 stand to gain financially if ISP-level filtering is indeed made mandatory for all 22 million Australian citizens.

But I'm sure they only have our best interests at heart...

Akira Doe

3

Don't forget...

... that Conroy is waiting to see what the results of the trial are before setting the pass or fail criteria.

So no matter what the result may be he can now just move the goal posts to where ever he wants to make it seem favourable.

It's not only obvious that the trial has been a farce (15 user sample size...!) from the beginning but the Government also knew that it would not succeed in providing the result they want so they have in effect rigged it to guaranty the outcome they want.

Apparently "Proper" Lap tests, Industry Experts, ISPs and Internet Users saying the censorship won't work and is unwanted doesn't matter when you have a fixed trial paid for by the Government who want to implement this.

That's right, the filtering plan failed the first round of lab tests yet $100,000's of dollars have been spent to buy the result they want.

Conroy would be fired and never be hired in his industry again if he did this in any Corporate environment, but I guess it's ok to waste Tax payers money...

Say good bye to your credibility Enex, hope it was worth it.

Akira Doe

4

Ok, don't know what happened there, anyway...

Don't forget that Conroy is waiting to see what the results of the trial are before setting the pass or fail criteria.

So no matter what the result may be he can now just move the goal posts to where ever he wants to make it seem favourable.

It's not only obvious that the trial has been a farce (15 user sample size!) from the beginning but the Government also knew that it would not succeed in providing the result they want so they have in effect rigged it to guaranty the outcome they want.

Say good bye to your credibility Enex, hope it was worth it.

Fictitious1

5

Enex

I think Enex should be dragged through the dirt for being party to such a poor study, as should any academic (especially from RMIT) involved. I always knew that the business of securing government grants was a bit mercenary, but this takes the biscuit!

Anonymous

6

Disgraceful

Evidence based policy? What if the evidence isn't worth a damn to begin with?

Government transparency? ("Sen. Conroy's office did not respond before deadline", or "a spokeswoman for Sen Conroy refused to comment" are starting to become all too common sights)

Scientific process? Bwahahaha, I don't think Conroy even knows what this is. If he did, he would have been more careful about fudging his test. Seriously, this guy isn't even smart enough to fake a good test result...

What a joke.

Bill

7

Hardly surprising.

Strongly religious idiots like Conroy have a natural aversion to science and reality. They're consistently flat-out wrong about recognised scientific phenomena like global warming, evolution, gravity, etc.

Pity how they often seem to end up in easily-abused positions of power.

Anonymous

8

The results prove something, but not what Conroy wants

At best, the trial results will prove that the filter does not slow internet access provided it is used as an opt-in service and that only 1% of an ISP's customer base chooses to use it.

No matter where Senator Conroy chooses to position the goal posts, he cannot use the results to make any assertions as to the effectiveness of the filter on an unwilling participant (i.e. someone who does want to view banned material), the impact on internet speed for more than 1% of an ISPs customers, or anything to do with the opt-in element for 'kiddie-safe' internet access. None of these aspects were tested at all.

I look after a team of engineers. If Conroy was working for me and presented these as his test results, I'd sack him for gross incompetence. Incomplete test coverage. No success criteria. Dodgy methodology. Fail!

Anthony

9

The best result one can hope for

“We'll be guided by that trial. We've always said, consistently, we'll be guided by the trial,”

In many ways, that's the best news one could hope for. Even if the trail is a "success", then the most Conroy can do, is push for an ISP filtering system that is opt-in ONLY and limited to a small number of ISP customers. Since lets face it, that's exactly what the trail has tested, so in order to be consistent, the trial can't provide any more guidance past that.

Holly

10

Most of these ISPs selected by conroy have what, less than 1% of the internet traffic? You can't tell how much the service speed degradation will be by selecting some of the smallest ISPs.

Someone as mindless as conroy shouldn't be allowed near the internet, we've all seen the kind of things the blacklist flags. MMOs and MMORPGs like Second Life, World of Warcraft etc, will be banned because there is no way to control what the online community says, so it all becomes RC.

Conroy takes a serious subject like child pornography and basically mocks it by accusing people who are anti-filter of being a pedophile, he is trying to create false protection, even the children protection groups agree it wont help!

He is being fueled by various religious communities, as proof when he said he was going to drop the filter, the religious communities were the ones to become angry.

Anonymous

11

Okay, let's first say what everyone who has been watching the trials with great interest has been saying - the trials are definitely rigged. The ISPs were handpicked by Stephen Conroy, excluding one of the largest in the country who wanted to perform a scientifically valid test - iiNet. No parameters for success have been defined and the test is clearly scientifically invalid.

Let's be generous and assume that the tests are a scientifically valid success, what would that mean?

Well they tested an opt-in filter of the full ACMA blacklist (including "adult nudity" and other R rated content). I don't think anyone would seriously object to such a filter being available to those who want it if it were implemented properly and not a significant drain on the taxpayers. Such a plan would have my thumbs up.

But that's not what the plan is. The plan is for a top-secret mandatory Iranian-style filter. You just can't test a mandatory filter on an opt-in basis.

Sandmann

12

Real results...

The methodology is useless for the analysis that Conroy's actually trying to do, but from my point it does show something useful. Surely the fact that only one percent of potential testers actually chose to do so clearly demonstrates that the number of Australians who want this filter imposed on them is also about one percent? I thought democratic government was supposed to represent the majority of voters?

Anonymous

13

Re: Okay, let's first say what

Sorry mate, but you are misleading the reading public:

"excluding one of the largest in the country who wanted to perform a scientifically valid test - iiNet"

Let's quote the truth:

"iiNet Managing Director Michael Malone (pictured) said the ISP "only agreed to participate in the trial to demonstrate that the policy was fundamentally flawed, a waste of taxpayers' money and would not work."

That does not sound scientific, that sounds far more like someone determined to not remain objective... Science needs objectivity, not a clear and stated target to prove soemthing wrong...

Second: "But that's not what the plan is. The plan is for a top-secret mandatory Iranian-style filter. "

You have no idea in the world mate, just no idea what is going on in Iran and what style of filter they have in place. Comparisons of Iran and Australia are just so over the top and unrealistic.

Lastly: Conroy always stated that the trial was not there to decide if the gov introduces a mandatory filter, never. It was there to decide how they go about it and to which degree the ISPs could choose and manage their own filter choice...

Oh yeah, just like in Iran, right???

Anonymous

14

Re: Enex ( Fictitious1)

Nice subjective bit of vitriol here...

Enex had a multi-year contract to do federal government testing projects. It would be a breach of contract for them to not do this testing, and their numbers in the past have assisted in many attacks on government policies or intentions.

In my opinion it is a gutless dirtbag that so cheaply attacks these guys for just doing their jobs and fulfilling their legal contract obligations.

You should have a good think about yourself, you are not winning any support against the filter plans with this cheap shot at Enex...

Holly

15

Apparently not.

Dane

16

Propaganda Minister

Senator Stephen 'Goebbels' Conroy

Anonymous

17

Show me the Methodology

We are a large network engineering services group and we architect some of the most secure networks around and we test them too. I know Enex has been well respected for testing here for a long time.

Where are the methodologies that were analysed and commented on by these guys for this story? Can't see any reference to them in this article. Can someone from ARN post a link or point us in the right direction.

Thanks.

Rant, Rot & Ruin

18

Show me yours first.

What you're essentially saying is that your (Enex') response to this debacle is "Since our testing methodologies are hidden behind Non-Disclosure Agreements and Commerical In Confidence, we're immune to peer review; ergo these academics can't possibly be talking about us; ergo our excrement still smells like roses."

Choose your bedfellows carefully, Enex: you never know what you'll catch from them.

Dave

19

That bloody filter

Surely, surely, surely this "trial" will go on quietly for a decade now ? This has to be dead, doesn't it ? I have never seen a policy this unpopular. The smug dishonesty of it all is what gets me. The only thing consistent about Conroy's utterances has been the refusal to come out and admit what the "banned" list is going to entail, even though we know.

Wolfgang muller

20

global isp

Soon, the entire planet will be accessible with these global ISPs. They are also very competitive when it comes to price.

Comments are now closed

 

Latest News

03:26PM
Gigamon signs VARs to support Arrow distribution partnership
02:45PM
Revel forms tech alliance with Deputy
02:07PM
Avnet clinches Lenovo global distribution deal
01:46PM
Steve Wozniak to teach at UTS
More News
21 Oct
NewLease & Red Hat Breakfast Briefing (Melbourne)
21 Oct
DCIM Solutions Architect
22 Oct
NewLease & Microsoft Technical Sessions
23 Oct
NewLease & Red Hat Breakfast Briefing (Sydney)
View all events