A Funny Thing Happened on the Way to the Forum

One Sunday morning, a bit over two weeks ago, Kat, busy putting the finishing touches to her article on transparency for the Centre for Policy Development‘s group submission to the government 2.0 taskforce “Upgrading Democracy” leans over to me and says:

“How about we submit a Perl app to the taskforce as an essay in transparency?”

I say, “Okay… but I’ll write it in Ruby

A perfect half serious, half tongue-in-cheek example of what we’re about. If you have the choice between talking about something and doing something, “do” wins every time.

So, I started writing a little Ruby script to screen scrape data from the Federal Register of Lobbyists which is one of those things that I’d been meaning to take a look at for a long time. Now you might reasonably ask, why do that?

The Federal Register of Lobbyists “… is a public document that contains information about lobbyists who make representations to Government on behalf of their clients”

Look at how the Federal Register of Lobbyists is organised. You can get a list of all the lobbyists on the register and for each of those you can follow the link which leads you to the list of clients they represent.

This is useful if you know the name of a lobbying firm or individual but not a whole lot of use if you know the name of a client and want to find out who lobbies for them on their behalf.

So the purpose of writing the little Ruby script was simply to download the information from the whole register by downloading 270 or so web pages, extract all the relevant information from the web pages and save it as a local file in a format that can then be easily loaded into any spreadsheet program for easy analysis.

This program demonstrates the value of transparency on a dataset which is about transparency. It essentially “unlocks” the data that is trapped on the web pages in a form which makes it hard to analyse. Many people have talked about this as the importance of data and API’s. I won’t try to repeat or even summarise what they’ve said.

Anyway, I submit the program to the taskforce with a small explanatory email. In the email I say that I had originally intended to submit the resulting data with the submission but did not because the website that hosts the register crashed and the copyright on the website did not allow for republishing without permission.

So, all I was submitting was the source code for a small Ruby script which is copyright OpenAustralia Foundation and I licensed under the GPL, so that anyone can use and alter the script freely within the terms of license.

And then I mostly forgot about it…

Some time passed…

Lots of submissions were appearing on the taskforce website but mine wasn’t there. What was going on? I emailed the taskforce to find out, received a reply from the secretariat explaining that the submission would be up early next week as they were processing many submissions.

Perfectly fair. The following week rolls round, more submissions appear on the taskforce website, but mine is still not there. That’s strange. On the Thursday I email again, asking what’s happened, I even mention it on Twitter, but no luck.

Finally, yesterday, more than two weeks after I emailed the taskforce my submission I send another email outlining the dates of all my correspondence and asking again when my submission would appear.

Then, today I receive this email

Good afternoon Matthew,

Apologies for the delay in dealing with your submission and correspondence. Putting submissions online is an ongoing process that is still being worked through, particularly after the redesign of the Submissions page. We are working to have the remaining submissions online by the end of this week.

We have been analysing security issues about putting your submission online. The Taskforce blog is hosted on a Department of Finance and Deregulation server, and as such is subject to security policies which prevent us from offering your program as a direct download to users.

However we have placed the emailed text of your submission on the Submissions page in whole, including the link to the page where users will still be able to access the program. Please see http://gov2.net.au/submissions/matthew-landauer-openaustralia-foundation/

We are also asking all submission authors if they would like the WordPress comment function switched on for their submission. Should we turn it on for yours?

Regards,

Government 2.0 Taskforce Secretariat
Australian Government Information Management Office
Department of Finance and Deregulation

“Security issues” eh? What does that mean?

Thanks for responding. I greatly appreciate it.

I’m a little surprised, to put it mildly, to hear you’re saying that
there are security issues with putting my submission online. What
exactly are those security concerns? Maybe I can help put any fear to
rest by answering some questions?

As for comments on my submission please go ahead and switch them on.

A small point of clarification on my end – the emailed text (without
the program) is not my submission. That you have put the text up I
think is a good first step. The emailed text was a little further
explanation but the core of my submission is still the Ruby program.
So, I hope that we can work through whatever security concerns you
might have so that my submission in full can be put up on the
taskforce blog along with those of everyone else.

All the best,
Matthew

And shortly thereafter, I get the following response

Hi Matthew,

To be more specific – the security policies that stop us from putting up your program for download aren’t about the nature of the program. They are restrictions about offering externally-created code for download on a departmentally-hosted website.

We acknowledge that you didn’t consider the email to be your submission, and are able to replace/update the text we have put up on the Submissions page if you want to make changes to what is online now. And of course we will retain the link to your Github page so that blog users will still have access to your program, albeit indirectly.

Regards,

Government 2.0 Taskforce Secretariat

And then my response is

I obviously don’t know what the departments guidelines are
specifically, but I would imagine (applying common-sense to this) that
there are restrictions on the distribution of program binaries via the
website.

However, what I’m supplying is source code which by itself is of no
security threat. It is not a binary program where someone clicks on a
link and then it starts up a nasty program which does something
terrible. This is a bit of source code that by itself does nothing. It
is no different then a piece of text, of which there is an enormous
amount on the taskforce website.

In light of this, I would ask you to reconsider publishing the program
in full on your website.

All the best,
Matthew

Let’s hope that this can get resolved quickly. I’m sure that nobody could have anticipated that someone would come along and do an apparently silly thing like submit a Ruby program as a submission to a government taskforce, but in this instance it made good sense, as a practical way of demonstrating many facets of what it takes to do government 2.0 in a practical way.

I took care not to infringe any copyright of the original data, licensed the script under a liberal license so that anyone could reuse it and still there was a problem…

This entry was posted in Uncategorized and tagged , , , , , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

6 Comments

  1. Andrew
    Posted September 9, 2009 at 9:48 pm | Permalink

    A fantastic tale! It confirms my long held view that, “If Innovators [like the OA crew] and Early Adopters do not acquaint government and the community at large with the civic potential for broadband services, then the medium may quickly become captured by interest groups wishing to control it for their own ends[eg maintaining the status quo].” (1995)

    The great power that OA brings to public debate in Australia is your ability to both identify/analyse the problem/policy issues and develop/implement solutions.

    It is inspirational to see thinking and doing happening in very short order – rather than talking, thinking, talking, wondering, then perhaps doing.

    Quick and accurate risk assessment is vital for the fostering of innovation – so we need to keep encouraging kids to learn the power of coding and to get stuck into it!

  2. Posted September 9, 2009 at 11:03 pm | Permalink

    It would be very interesting to get a definition of “program”. Let’s see…

    * Microsoft Office macros are embedded in some of the documents commonly found on government agency websites (ABS in particular). Are these programs?

    * PDF formats are executable programs (especially if you embed javascript or postscript). There are even PDF functions to sense the host operating system and decide which command line string you want to “shell out”. Are these programs?

    * Flash? I seem to remember programming manuals for flash.

    * Javascript? Ditto.

    And some of the above have their code executed on the client.

    Evil persons (and I DO NOT advocate this) could still obfuscate languages like perl. You can even write Latin perl (http://www.csse.monash.edu.au/~damian/papers/HTML/Perligata.html)

    I can see the problem the agencies are trying to address, which is why Xena from naa.gov.au got shifted to sourceforge (see http://www.naa.gov.au/records-management/secure-and-store/e-preservation/at-NAA/software.aspx#section3)

    However, it is /extremely/ rare for tarballs to be considered “executable programs”.

    Mind you, how about the downloadable java applet from http://www.ipaustralia.gov.au/patents/search_software.shtml

    or
    http://www.abs.gov.au/AUSSTATS/abs@.nsf//web+pages/Download+Install+SuperTABLE+Software

    or

    http://www.nhmrc.gov.au/grants/apply/projects/software/_files/filsetup.exe

    ftp://ftp.ips.gov.au/users/asaps/ASAPS5Demo.exe (click on the link in http://www.ips.gov.au/Products_and_Services/1/1/3)

    and of course, the tax office’s tool via
    http://www.ato.gov.au/individuals/content.asp?doc=/content/32234.htm

    http://www.environment.gov.au/settlements/energyefficiency/motors/motorselector/pubs/installmsv31.exe

    And there’s REALM, but that’s Victoria
    http://www.ourwater.vic.gov.au/__data/assets/file/0017/52460/Install_Realm.exe

  3. Posted September 10, 2009 at 8:58 am | Permalink

    While I get the risk control issues the Taskforce Secretariat (note it’s not the Taskforce members) are trying to manage, it’s this sort of inability to deal with something outside the carefully framed bounds of “the way we do things” that gets us all to realise that parts of the public sector aren’t yet where the rest of the world are.

    I wonder what the Kiwis would have done with code from say, their hackfest a couple of weeks back…

  4. Posted September 10, 2009 at 10:39 am | Permalink

    Matthew’s submission has now been posted as originally proposed. We have learnt a useful lesson – which is a good thing, isn’t it?

  5. Posted September 10, 2009 at 11:26 am | Permalink

    John, well done! The ability for you folk to do this and manage it in spite of it being somewhat unfamiliar shows some light and forward thinking.

    Thanks for letting us know and nice work.

  6. Matthew Landauer
    Posted September 10, 2009 at 1:35 pm | Permalink

    @John – thank you from me as well!

2 Trackbacks

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*
*

Subscribe without commenting