DPS, where the bloody hell are you?

A fairly typical email exchange with the Department of Parliamentary Services (DPS), the administrators for the aph.gov.au website, goes something like this:

from: Matthew Landauer
to: webmanager@aph.gov.au
date: Thu, Apr 2, 2009 at 11:39 AM
subject: Hansard 23 September 2008 probably error in division

According to the Hansard, Mark Vaile voted in a division on 23 September 2008. See
http://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;query=Id%3A%22chamber%2Fhansardr%2F2008-09-23%2F0060%22

However, according to his biography he resigned several months earlier on 30 July 2008. See
http://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;query=Id%3A%22handbook%2Fallmps%2FSU5%22

There appears to be an inconsistency between these two facts. I would guess that the Hansard is probably mistaken. Is that correct? If so, it would be great if it could be fixed.

Many thanks!

All the best,
Matthew Landauer
Founder
http://www.openaustralia.org/

It’s looking good. I receive this very quick response.

from : Web, Manager (DPS)
to: Matthew Landauer
date: Thu, Apr 2, 2009 at 12:47 PM
subject: RE: Hansard 23 September 2008 probably error in division

Thanks for your email Matthew

I will get this discrepancy checked and get back to you as soon as possible.

Kind regards
Margaret

Margaret Cazabon
Parliamentary Web Manager
Department of Parliamentary Services
Parliament House
Canberra ACT 2600

I reply a few hours later thanking her.

More than a week passes with no further news. Hmmm… Naively I would expect that they would take this issue fairly seriously. It appears from the official Hansard record that someone has voted who wasn’t even a member of parliament at the time.

Anyway, I send the following email to follow up:

from: Matthew Landauer
to: webmanager@aph.gov.au
date: Sat, Apr 11, 2009 at 1:13 PM

Any news on this?

Any possibility that OpenAustralia.org could be assigned a technical contact for the Hansard? Our software does lots of cross-checking of
the Hansard XML data and regularly finds spelling mistakes, formatting errors, logical inconsistencies and the like which I’m sure you would
like us to alert you to. However, this is currently very difficult as we have no technical contact.

It would also lighten your load as a go-between.

All the best,
Matthew

I’ve asked for an ongoing contact for OpenAustralia at the Department of Parliamentary Services before but those emails were never replied to. I wonder if I’ll hear any more this time.

Stony silence.

from: Matthew Landauer
to: webmanager@aph.gov.au
date: Mon, Apr 20, 2009 at 12:37 PM

Hello, is anybody there??

So, it’s been two weeks and I’ve heard nothing more.

In the greater scheme of things it’s no big deal that a couple of emails from openaustralia.org are left unanswered. Hell, we’ve been known to not reply to all the emails we get.

The unfortunate things is this email exchange is fairly typical of all our dealings with the DPS over the last year. We’ll report a problem. If it’s something really glaring (like today’s Hansard is missing!) it will get fixed in half a day or so and I’ll receive a response to my email. If it’s something more subtle but just as important, I’ll sometimes get a response saying they’ll look into it but then I’ll never hear anything again.

Does it get relegated to the “too hard” basket?

There should be two-way communication between DPS and OpenAustralia. The DPS is providing a service to the general public by publishing the Hansard. We’re finding faults with this vital document of national importance and we don’t expect our reports of errors to be ignored.

We’re also providing a public service (entirely run by volunteers) and we have a responsibility to make it as good as it can be.

We’re providing all the results of our work (our software, our data) for free for anyone to use. We have software that we’ve written in the course of building OpenAustralia that finds many problems in the Hansard completely automatically. This software we’ve made freely available to everyone (including the DPS). However, they are apparently not interested in hearing from us.

DPS, where the bloody hell are you? We can help you. Ask us how.

Posted in Development | 5 Responses

Radio National – Future Tense: e-Australia

http://www.abc.net.au/radionational/programs/futuretense/e-australia/3181162

Posted in Media, OpenAustralia Foundation, OpenAustralia.org | Tagged , | Comments closed

eGov AU: OpenAustralia becomes first to provide the Australian Federal register of Senators’ Interests online

http://egovau.blogspot.com.au/2009/01/openaustralia-becomes-first-to-provide.html

Posted in Media, OpenAustralia.org | Comments closed

The Register of Senators’ Interests is now online

Today is a big milestone. We are the first website to make the Register of Senators’ Interests available online. This important public document until now has only been available to the small number of people who were able to visit the office in Canberra where the documents are held. In the Register each Senator declares information of financial interests, stocks and shares held, gifts received over a certain value, and memberships of Clubs and Associations.

The register is available on each Senator’s page. For example, have a look at the Register for Senator Judith Adams.

As soon as we can we will also make the Register of Members’ Interests available online, the equivalent document for the Representatives. Our main obstacle right now is getting our paper copy of the register scanned. If you have access to a bulk scanner and can help please contact us.

Posted in Announcement, OpenAustralia.org | 4 Responses

The Register of Senators' Interests is now online

Today is a big milestone. We are the first website to make the Register of Senators’ Interests available online. This important public document until now has only been available to the small number of people who were able to visit the office in Canberra where the documents are held. In the Register each Senator declares information of financial interests, stocks and shares held, gifts received over a certain value, and memberships of Clubs and Associations.

The register is available on each Senator’s page. For example, have a look at the Register for Senator Judith Adams.

As soon as we can we will also make the Register of Members’ Interests available online, the equivalent document for the Representatives. Our main obstacle right now is getting our paper copy of the register scanned. If you have access to a bulk scanner and can help please contact us.

Posted in Announcement | 4 Responses

Sydney Morning Herald: Interests of MPs to go online – it’s about time

http://www.smh.com.au/news/opinion/interests-of-mps-to-go-online–its-about-time/2008/12/19/1229189883754.html

Posted in Media, OpenAustralia.org | Comments closed

Australian – Wires & Lights: Interview – OpenAustralia.org

Posted in Media | Tagged , | Comments closed

Happy talk

Last week I gave a talk at the Open Source Developers’ Conference in Sydney. It was great. The room was full, people were excited, asked great questions and even applauded. To round it all of, at the end of the conference I was voted best speaker of the conference. What a wonderful surprise.

I met many great people at the conference who told me how happy they were to see what OpenAustralia is doing and want to get involved. So, expect many new things to come. Stay tuned.

There were not any recordings made of any of the talks but do have a look at the slides.

Katherine Szuminska and I will also be talking at the “Free as in Freedom” miniconf as part of the Linux Conference Australia (LCA) which is happening between 19-24 January in Hobart. Please come along!

Posted in Presentation | Tagged , , , , | Leave a comment

Why is OpenAustralia not getting updated?

It’s only a temporary affair but OpenAustralia is not getting updated with the latest speeches in the House of Representatives and the Senate.

Why? Well, let me explain. It’s been a tumultuous few weeks behind the scenes here. If you use OpenAustralia you’re probably blissfully unaware of some changes that have taken place at the official online home of the Hansard at aph.gov.au which have caused us a great deal of grief.

Several weeks ago, a new system for accessing the Hansard at aph.gov.au was made live and the old system was immediately switched off. We had some warning that this was going to happen. Also, we were told by a person at the Department of Parliamentary Services (DPS) that the old system would be kept online for about a month after the switchover. Unfortunately, this isn’t what actually happened.

After the switchover nothing worked for us. Our parser that scrapes all the Hansard information depended very tightly on how the information was structured and everything had changed! So, nothing worked.

Many conversation ensued with the DPS imploring them to turn on the old system again and at least give us some grace period to try to rewrite our parser to work with the new parlinfo search. Thankfully after a few days they agreed to put the old system back up for a short period of time.

That allowed OpenAustralia to keep on working for a little while.

Then, for me, the fun truly started. I was faced with a new system that bore only a passing resemblance to the old one. The way that the Hansard was split into multiple pages had changed; The structure of the HTML markup had changed; the metadata associated with the pages had changed – everything had changed! Worse still, I soon discovered that there were some absolutely fundamental problems. Information was missing, such as whether a particular page is “procedural text”, most pages are not valid XHTML – a typical page when put through an HTML validator comes up with over 600 errors; I discovered some instances where the text was in the wrong order, even where several different sections of text from different places had been combined into one section.

Somehow I tried to work my way around each of these problems. I battled away at this for a few weeks making very slow and painful progress.

Then, I heard murmurings from the DPS that another solution might be coming. What might this be?

Three days ago, Friday last week, they added a new link to Hansard pages that allow you to download an XML file. This XML file is the underlying data that until now has only been used internally within the DPS. It is what comes out of the “Hansard Production System” which are the people and systems that annotate and record the Hansard and is what goes into the web system. So, it has all the information required to truly make sense of the Hansard.

I had asked for access to the XML data in November of last year when I started working on what became OpenAustralia. I never heard anything back. Also, during phone calls with DPS I brought it up again but I never expected it to get anywhere. It turned out that at the same time Jason Wilson from GetUp‘s Project Democracy had been asking for the same thing. So, huge thanks goes to Jason Wilson and his team at GetUp for helping getting DPS to give us the Hansard XML data.

I dropped everything and have spent every waking moment since then working on rewriting the parser to work from the XML file. I’ve made good progress. Now, it’s Monday, but I don’t realistically think that it’s going to be anywhere near ready by tomorrow when the first of the Hansard from this most recent parliamentary day will appear.

So, please be patient while we fix this. We’ll do everything we can to make it as quick as possible.

And, of course, we’ll keep you posted.

Posted in Announcement, Development | 5 Responses

New theme/design has gone live

I came up with the new design several months ago and developed about 80% of it – in fact before OpenAustralia was even launched. However getting the last 20% done took a little while, back and forth, implementing some of the trickier stuff which required getting into the code … and when I looked at setting up my own development instance of OpenAustralia it all came to a standstill.

However last weekend after the Web Directions South 2008 conference I dropped in at Matthew and Kat’s place for about 4 hours and we just ripped through the last few little styling issues and CSS bugs and now a week later the new theme has gone live!

As charming as the TWFY green & red design was, we decided we wanted to stand apart with our own unique look – something a bit more representative of Australia (no we didn’t go green & gold). It’s a rather dry theme, beiges, browns, greys and oranges.

Hope you like it!

Posted in Announcement | Tagged , , , , , | 1 Response