The story of a tiny, almost invisible change

By Matthew Landauer | Published: September 29, 2020

I don’t really like technical blog posts.

Instead this is a story of a tiny, almost invisible change we made to PlanningAlerts, what led us to make this change and the surprising amount of under-the-hood work it took to get there.

We made a “tiny” change to how comments are shown on PlanningAlerts. “delivered to the planning authority” is now a link (it’s underlined).

screenshot of comment

When you click on the link it gives you some more information

screenshot of comment after clicking

This should give you extra confidence that the comment was definitely delivered to the local authority.

How it all started

A kind and considerate soul emailed us to let us know of a strange comment that had been posted on PlanningAlerts.

Does this post have any truth in it?

Is this post damaging the Planning Alerts site?

Was the CAPS really necessary?

Here is the comment in full

”DONT” NOT LEAVE COMMENT HERE IF IN FAVOUR OR AGAINST APPLICATION. MANY IF NOT ALL DO NOT MAKE IT TO COUNCIL MOST COMMENTS HERE NEVER MAKE IT TO THE RELEVANT DEPARTMENTAL AUTHORITY!!

Ive been let down by this page thinking mine and many others are being heard here. It makes a difference. It does not. After contacting council in past, they had no clue about my rejection of development or from others on this page. It seems Acting authorities never know about what written here. This site should be taken down as everyone presumes it has direct or indirect authority to distribute to relevant departments.

*** PLEASE CONTACT COUNCIL OR RELEVANT AUTHORITY DIRECTLY ***

There were similar comments made by the same person on three other applications.

My first emotional response was “how dare this person say this, how rude!”.

That person didn’t even contact us directly to ever let us know there was a problem. We don’t read all the comments. There’s just too many for a tiny team doing too much. And then to throw around those blanket accusations.

Ouch! It really hurt to hear that. We have put so much time and effort over the years into ensuring that emails from PlanningAlerts get successfully delivered. We went as far as developing a new open-source transactional email server, Cuttlefish, to give us visibility and improve deliverability across the board. We made PlanningAlerts to give people access to the democratic process and getting their comment delivered is pretty core to that objective.

Then to my horror I realised that what the rude comment was saying was partly true. For the last couple of months 15 or so comments were not delivered to that one particular council. This was terrible!

We are human and our software is written by humans.

In 10 years PlanningAlerts had an amazing track record. We actually only know of one instance when emails were not delivered to a council In that case, a council changed their email address and didn’t forward emails to their old address. As soon as we were notified, we manually readdressed the emails and worked with the council to get them delivered in time.

So, what was going on? Down the rabbit hole of email sending

(Here’s where we get technical)

What was super weird at first was that only some emails to that one council were bouncing. The bounce message mentioned “DMARC” which was not something I knew about at that stage. So why were only some messages getting bounced and not others when all the comment emails are generated the same way? Nothing in the content of the comments looked like spam, the formatting of each email is identical. I found it really confusing.

Then, I noticed a pattern. All the comment emails that were bouncing were made by people with “yahoo.com” or “yahoo.com.au” email addresses. A quick search of “yahoo DMARC” led me to an explanation.

Yahoo had an unusually strict DMARC policy which meant that any problems with emails that look to originate from yahoo domains would get hard bounced. So, we had a DMARC problem with our comment emails but it had been unknown to us up to that point because almost nobody was enforcing it so strictly.

At this point I’ll just quickly backtrack a little. For a long time we’ve been ensuring that our emails pass SPF and DKIM tests. Cuttlefish helps enforce this and does it all pretty much automatically. However DMARC, does two main things. It allows an owner of a domain (e.g. yahoo.com) to say how email servers should handle mail from that domain when SPF and DKIM tests fail. So, in our case SPF and DKIM was fine, or so we thought, so why were the yahoo.com emails failing to get delivered?

Well it turned out it was to do with the second part of what DMARC does. It also enforces domain alignment; the domain in the “from” address of the email needs to be the same domain as used in SPF and DKIM. This is the problem that got us. It turned out that our comment emails had headers of the following form:

From: <email address of the person making the comment>
Sender: contact@planningalerts.org.au
To: <email address of council>

The Sender header is used in circumstances where emails are being sent on behalf of someone else. It’s defined in the email RFCs but generally has not been particularly well supported.

The purpose of all that was to ensure that when someone makes a comment via email that they receive replies from the council rather than us.

It turns out that DMARC essentially stops the Sender from ever being used. It appears that this was a conscious choice made by the DMARC designers because the Sender field is not shown to users in email clients.

So, we switched the email headers to:

From: contact@planningalerts.org.au
Reply-to: <email address of the person making the comment>
To: <email address of council>

And this magically solved our problem.

We resent all the recent comments that had previously bounced because of this, and our job was done.

Actually no. It would have been easy to stop there but actually we wanted to make sure that if something similar to this happens in the future we don’t find out about it because someone emails us asking about a comment on the site. We should be the first ones to know there is a problem.

After we fixed the way that comment emails are sent we started working on a whole host of other changes to improve the visibility and the impact of any potential future hard bounces for comment emails.

Improving visibility and reducing the impact of problems

In explaining earlier that only emails from the “yahoo.com” didn’t get delivered I was over-simplifying. What actually happened was that a yahoo.com email would hard bounce and then the email address of the council would get added to a “deny list” in Cuttlefish. Once an address gets added to the “deny list” any further emails sent to that address will not be sent. Addresses then get cleared out from the “deny list” after one week. This is an automatic feature of cuttlefish designed to reduce the chances that we send out bad emails which improves the deliverability of our emails across the board.

So the case of sending emails to local authorities is different from sending transactional emails (email alerts, etc..) to “random” people on the internet. We should be really confident that the email address of the council is correct and that emails to them generally should not bounce.

So, next we added a feature to cuttlefish.io to allow PlanningAlerts when it’s sending the comment emails to say “please ignore the deny list when sending this email”. This means that if an email hard bounces to a council, that cuttlefish will still attempt delivery for any subsequent emails to the same address. So now if hard bounces do happen the impact will be reduced.

Then we started working on the “visibility” part. If a hard bounce happens for a comment email how can we get notified of this urgent problem? One option would be that we get an email. In recent years we’ve been trying hard to limit the number of emails generated automatically for notifications and instead send those notifications to Slack. So, why not just try sending a notification directly to Slack instead?

How to do this? We don’t want Cuttlefish to have the knowledge to notify us of certain problems with certain emails but not others. It makes much more sense for PlanningAlerts itself to do that because it’s the one that knows the difference between the different kinds of emails. This is where we need to get some of the delivery information from Cuttlefish back into PlanningAlerts.

This finally gave me the reason and motivation to add a feature to Cuttlefish that I’ve been wanting to do since I first built it – which is to add webhooks where every time there is a successful or failed email delivery a custom URL (on another site of your choosing) is hit with information about the email. In our case we’ll be making a new URL endpoint on PlanningAlerts to accept those webhook callbacks.

So how do we identify if the email corresponds to a comment email or an alert email and which comment email? I think the most elegant solution to that is to allow the sender of an email to attach custom metadata.

Here’s an example of a comment email as seen in Cuttlefish showing the attached metadata (in this case comment-id):

screenshot of cuttlefish metadata

So then it was a relatively simple matter of hooking up a bit of logic in PlanningAlerts where on every “delivery event” it receives from cuttlefish, it checks whether it’s from a comment and whether it was a hard bounce and if so it sends a notification to our slack channel.

screenshot of slack notification

At the same time as we put this in we noticed another council was bouncing emails! This time for some reason they had just decided to specifically block our email server. Thanks for that. Thankfully they quickly fixed it when we asked them to and we again resent the emails.

Making it visible to people using PlanningAlerts

We’re confident now that all the changes we made to improve the delivery of emails to councils and to notify us of problems makes it unlikely that similar problems will reoccur in future.

However all this stuff is under the hood and invisible to users. Perhaps that’s how it should be?

However, it’s not uncommon that we get emails from people who just want to make sure that the council did in fact receive the comment they made through PlanningAlerts. The council tells them they never received it, so we check in Cuttlefish to find that the council absolutely received the email.

What if we could make this more clear for people?

Why don’t we do something similar to what is in RightToKnow.org.au where the message says whether it was successfully delivered?

screenshot of RightToKnow request

So that’s what we did. The final piece in a long chain.

Posted in Planning, PlanningAlerts.org.au | Tagged | 1 Response

Last week in Parliament

By Mackay Ash | Published: April 17, 2020

You may have been reading about the unsuccessful amendments to the Coronavirus Economic Response Package that opposition MPs and Senators attempted to pass last week.

The amendments related to expanding the Government’s JobKeeper payment to include more workers, such as casual workers, workers with major charities and working-visa holders who are still in the country. They also called on the Government to create a specific emergency response package for the Arts and Entertainment sector, as many workers in that sector are ineligible for the JobKeeper payment. (You can see how your representatives voted on expanding the JobKeeper payment and supporting the Arts sector on They Vote For You.)

The majority of these amendments were proposed by Labor Party MPs and, while there was broad agreement between Labor, the Greens and other opposition parties and independents, there was also disagreement. For example, Labor Party senators refused to support several substantive Greens amendments in the Senate, arguing that to do so would only delay the passage of the bills. In the words of ACT Senator Katy Gallagher (Labor):

We will not engage in a situation where the legislation could get blocked between chambers. That is not an option.

Though this raises the question: if the Labor Party were unwilling to risk any delay to the legislation, why did they propose any amendments at all?

The answer relates to the difference between substantive amendments – like what the Greens senators proposed – and what could be called unsubstantial amendments.

Substantive versus ‘unsubstantial’ amendments

A substantive amendment would be one that is introduced in order to change the actual wording of the bill. This can mean replacing words, deleting words or adding new words (for example, adding a new clause). On the other hand, an ‘unsubstantial’ amendment wouldn’t make any changes to the bill itself but would instead change the wording of, for example, one of the usual procedural motions.

What are the usual procedural motions?

When a bill goes through Parliament, there are certain motions that must be agreed to. One of these is the second reading motion, which is:

That this bill be now read a second time.

Reading a bill for a second time is parliamentary jargon for agreeing with the main idea of the bill. If this motion is successful, the bill will then be considered in greater detail. If it is unsuccessful, then the bill is said to have failed and will no longer be considered by either house of parliament.

Amendments to procedural motions don’t affect the content of the bill. Instead, they can be used to express an opinion towards the bill. For example, an MP may propose to add the text: “but the House believes the Government is doing a very bad job.” Even if this proposed amendment is successful, it doesn’t actually change anything: the bill will still be read a second time and will continue its passage through the House. The only change is that future readers of the parliamentary record will be able to see that the majority of the House was unhappy with the Government at that time.

Last week’s amendments

With the exception of one substantive amendment in the House, all of the Labor Party’s proposed amendments were of the unsubstantial kind. That is, they proposed to add text to the end of the usual second reading motion but didn’t actually make any changes to the bills.  For example, Rankin MP Jim Chalmers (Labor) introduced the following unsubstantial motion:

That all words after “That” be omitted with a view to substituting the following words:

” whilst not declining to give these bills a second reading, the House:

(1) notes that this legislation gives the Treasurer extraordinary powers to include those not currently eligible for the JobKeeper Payment; and

(2) calls on the Treasurer to use his power under this legislation to ensure more jobs are protected and that struggling, otherwise viable businesses and organisations are able to access the JobKeeper Payment”.

This unsubstantial amendment was unsuccessful, but even if it had passed, it would not have made any actual change to the bills. Instead, future readers of the parliamentary record would simply be able to read about how the majority of the House wanted the Treasurer to use his powers – the Treasurer himself wouldn’t be bound to follow their advice.

The Labor Party’s one substantive amendment was moved by MP for Watson Tony Burke (Labor) during the Consideration in Detail stage, which is when the House debates the bill in greater detail. That amendment proposed to add new words to a clause in the bill but was unsuccessful because it did not have the support of the Coalition MPs.

In the Senate, the opposition senators often have the numbers to pass amendments even when all the government senators vote against them. However, if the amendment is substantive and changes the wording of a bill, it needs to go back to the House of Representatives for our MPs’ approval before it can become part of the bill.

This is where bills can become stuck: the government majority in the House will refuse to agree with the Senate’s amendments and the opposition majority in the Senate will refuse to accept the bill without those amendments.

Because the Government had already made clear that they would not be agreeing with any amendments in the House, the Labor Party senators decided not to attempt any substantive amendments in the Senate. They also chose not to support the Greens Party senators’ substantive amendments, which only received the support of senators from the Greens and Centre Alliance parties.

What next for Parliament?

Currently, Parliament will not sit again until 11 August 2020, due to concerns about spreading COVID-19. But according to ABC News:

… Mr Albanese [Leader of the Opposition] doesn’t accept that argument. He maintains that parliaments in other nations affected by coronavirus continue to meet and scrutinise their governments’ policies.

Whether the Opposition will succeed in convincing the Government to re-open Parliament remains to be seen.

UPDATE: The Prime Minister Scott Morrison has announced that there may be a trial week of sitting in May.

Posted in OpenAustralia.org, They Vote For You, Uncategorized | Tagged , , , , , | Leave a comment

Coronavirus and our Federal Parliament

By Mackay Ash | Published: March 20, 2020
Sitting calendar for the first half of 2020

Our federal parliament is scheduled to sit next week but, due to the Coronavirus pandemic, it’s going to look a bit different. On Wednesday, the Prime Minister released a media release explaining the type of changes we should expect. In a nut shell, our parliament is battening down the hatches and reducing its business to the bare minimum required to keep the country going.

What does this mean for representation?

For those of us with representatives who belong to the two major parties – the Coalition and the Australian Labor Party – there won’t be any significant changes in terms of how you are being represented in parliament. This is because your representatives are already voting according to how their party whip tells them to, except on those rare occasions when they cross the floor. So whether they are in parliament next week, or simply paired with an opposition member, the result will most likely remain the same.

For those of us with representatives who are independent or belong to a minor party, the question is a little trickier. The main reference to these representatives in the Prime Minister’s media release was that “All crossbench and independent Members and Senators will be able to attend [parliament], should they want to.” In other words, they will not be prevented from attending parliament, but the voting arrangements being made are mostly between the two major parties, leaving the independent and minor parties to their own devices. We will have to wait and see how many of these representatives are able to attend parliament next week.

What does this mean for bills that have not yet been passed?

The media release makes clear that parliament is only meeting to consider the bills relevant to their planned stimulus package and “any other immediate business”. Once that business is dealt with, the parliament will adjourn and our representatives will return home. This means that all bills that are not immediately relevant to the current crisis or required for the continued running of the country will be put off until a later date.

How can I keep up to date with our parliament?

The most up to date information is generally available via the House of Representative and Senate twitter accounts. Follow them for the latest news.

Posted in They Vote For You, Uncategorized | Tagged , , , , , , | Leave a comment

Who will you vote for this election? A party or a person?

By Mackay Ash | Published: May 3, 2019

They Vote For You is based on the principle of holding individual representatives to account. You can find your representative’s report card and see how your electorate’s voting power is being used in Parliament. If you’re unhappy with what you find, you can contact your representative and ask for an explanation. And if you’re very unhappy, you can vote against them in the upcoming federal election.

That’s how They Vote For You works: you look up your representative and you assess their individual performance. But does this person-based approach reflect reality?

Party politics

The vast majority of our MPs and senators belong to a party. And most belong to either the Australian Labor Party or to the Coalition (which is actually an alliance of parties). This means that when our representatives vote in Parliament, they vote along party lines – a process that is helped along by party whips who make sure their members vote the way their party wants them to vote.

On very rare occasions, our politicians rebel and cross the floor to vote with the other side. But this usually leads to the rebellious representative leaving the party, as was the case with former Liberal and now Independent MP for Chisolm Julia Banks.

This all begs the question: when we go to the polls on 18 May 2019, who are we voting for? The person? Or the party?

The answer no doubt depends on each voter themselves, as many of us have strong party affiliations whether or not we’re actually card-carrying members. Only you can say whether you’re ticking that box because you think Person X will best represent your electorate or because you like their party and want it to have the most voting power in Parliament.

And thanks to our party-based political system, both of these approaches to voting are valid.

But the important thing to remember is this: your representative is only in Parliament because your electorate put them there. And that means you outrank the party whip when it comes to deciding how they should vote on your behalf.

So after this election is over and the business of government resumes, remember to keep an eye on how your representative is voting for you on the issues you care about and, if you don’t like what you see, tell them about it. Because ultimately, they answer to you.

Posted in They Vote For You | Tagged , , , , , , , | Leave a comment

Do They Vote For You?

By Mackay Ash | Published: April 23, 2019

Comparing words and actions

The federal election is less than a month away and, with some exceptions, our MPs are hot on the campaign trail seeking re-election. Now’s the time to decide whether you’re happy with your local MP’s work in representing your electorate, or whether you think it’s time for a change.

How can They Vote For You help?

MPs seeking re-election will be making many claims about how they’ve stood up for their electorate during their time in parliament. They Vote For You lets you check their claims against their actual voting record.

For example, to check Barnaby Joyce’s voting records on water rights, you can enter his name into the search field on theyvoteforyou.org.au and be taken to his profile page.

From here you can search within the page for “water” and see that Barnaby Joyce has voted very strongly for “Making more water from Murray-Darling Basin available to use” and very strongly against “Increasing protection of Australia’s fresh water“. Clicking on those policy names takes you to a detailed list of all the times when this MP is on record as voting on divisions (or formal votes) which relate to this policy. We also include a detailed breakdown of how we calculate the scoring to determine a stance.

How accurate is this voting record?

They Vote For You is as accurate as the available parliamentary data allows us to be. Unfortunately, our parliament does not keep a record of how our representatives vote ‘on the voices,’ which is when our MPs stay in their seats and shout ‘Aye!’ and No!’ so that the loudest side wins. This is unfortunate, because most votes in parliament take place on the voices, leading to an incomplete voting record.

It is also important to keep in mind that politics is a messy beast. Even if the way an MP has voted in parliament doesn’t line up with how their electorate would want them to vote on one particular issue, that doesn’t mean they aren’t doing their job and representing their electorate. It’s often the case that politics involves compromise and tradeoffs – voting against the electorate’s desires on a smaller issue in order to be able to deliver a bigger win for the electorate on a more important issue. Those trade-offs and bigger pictures aren’t captured in the raw voting data, but we’d expect that any MP should be able to explain those things to their electorate.

Posted in They Vote For You | Leave a comment

A new era for morph.io

By James Polley | Published: April 10, 2019

When morph.io was built, we leveraged a lot of awesome existing work already in the open-source ecosystem. In particular, we were able to heavily lean on work down by the Herokuish project had done to create a platform that can run scripts written in a variety of languages. The Herokuish project, in turn, built on the cedar-14 stack which had been open sourced by Heroku; and that in turn was built on top of Ubuntu 14.04, which builds on countless contributions from people all over the world.

Five years later, the cedar-14 stack is nearing the end of its life. As of this month (April 2019) Canonical will no longer support Ubuntu 14.04; Heroku will no longer support cedar-14 – and morph.io will no longer be able to support the platform we’ve relied on for so long.

Instead, we’re going to be rolling out a new platform. We aren’t changing too much though – the components we’ve been using are working well – we’re simply upgrading them. The morph.io scraper platform will now be based on the current version of Herokuish, which is based on the Heroku-18 stack, which is based on Ubuntu 18.04, which is based on even more contributions from even more people around the world.

To help smooth the transition, we’ve added multi-platform support to Morph. The new platform is available now for you to start testing, and to start using if you find that it works for you.

We will be changing the default platform in early May. At that time, any scrapers which haven’t explicitly chosen to stay on the existing platform will start to run on the new platform. We’ll still keep the old platform around for a while longer to help anyone who didn’t manage to update their scraper before the change.

What is changing?

Supported language versions

Heroku-18 supports  more recent language versions than were usable on Cedar-14. However, this means that many older versions aren’t supported any more. If you were relying on an older version of your language, this is likely to mean you need to check for compatibility with the new version.

Heroku-18 currently supports Ruby 2.4.6, 2.5.5, and 2.6.2, as well as a wider list of Ruby versions availabe through JRuby. Python versions 2.7.16, 3.6.8, and 3.7.3 are currently supported. PHP is currently supported for version 7.1, 7.2, and 7.3. A full list of the supported versions for every language is available here.

If your scraper specifies a different language version, it’s possible that Herokuish may still be able to install and run that version for you. However, we suggest that you upgrade to the supported versions wherever possible.

Operating system packages

The shift from Ubuntu 14.04 to Ubuntu 18.04 is a large one. Many of the packages on the base platform have been upgraded to a much newer version. However, in some cases, packages have been replaced by a newer equivalent package, or have gone away completely. In some cases, new packages have been added that add capabilities that weren’t present before.

For the most part, we expect that you probably won’t need to pay attention to the OS package versions. However, the full list of packages on both cedar-14 and heroku-18 is available from Heroku’s support site.

What do I need to do?

Choosing a platform

You can choose which platform you want to use by including a file named platform in the top level of your scraper’s repository. The contents of the file should match one of the tags in our buildstep container registry. Currently, the available tags are early_release for testing the new platform, and cedar-14 to keep your scraper using the old platform. If no platform file is found, we’ll use the default release.

You can see an example of a scraper being configured for the new platform here.

Learning more and reporting problems

We’ll be posting updates about the new platform on help.morph.io as we get closer to making the new platform the default.

If you find bugs or run into problems on the new platform, please let us know on help.morph.io/.

Posted in Announcement, Morph | Leave a comment

Freedom of Information Amendments proposed

By Kat Szuminska | Published: November 23, 2018

On 23 August 2018 the Senate referred the Freedom of Information Legislation Amendment (Improving Access and Transparency) Bill 2018 to the Legal and Constitutional Affairs Legislation Committee for inquiry and report by 30 November 2018.

Here is OpenAustralia Foundation’s submission to the Senate in response.

Prepared by Ben Fairless, Katherine Szuminska, James Polley, Matthew Landauer

Summary
Amends the: Archives Act 1983 to require the reporting of external legal expenses incurred by the National Archives of Australia; Australian Information Commissioner Act 2010 to: ensure that the Information Commissioner holds specified qualifications; and require the separate appointment of the Australian Information Commissioner, the Privacy Commissioner and the Freedom of Information (FOI) Commissioner; and Freedom of Information Act 1982 to: enable the transfer of Information Commissioner reviewable decisions to the Administrative Appeals Tribunal (AAT); require the consistent application of exemptions by decision makers in the context of a review by the Information Commissioner; prevent the Information Commissioner from making FOI decisions if he or she does not hold specified qualifications; prevent agencies from publishing FOI information until at least 10 days after the applicant has received his or her copy of the information; and require the reporting of external legal expenses for each Information Commission or AAT FOI matter that has concluded.

Bills of the current Parliament

The OpenAustralia Foundation would like to thank the Senate Standing Committee on Legal and Constitutional Affairs for extending this opportunity to make a submission to the Committee’s inquiry into the Freedom of Information Legislation Amendment (Improving Access and Transparency) Bill 2018 (the Bill).

Over the last few years the government has acted to undermine and disempower the OAIC, reducing scrutiny of its actions and limiting access of citizens to government information. We broadly support the intentions of the amendments as they serve to reduce some of the ways in which the OAIC has been undermined.

We broadly support the intentions of these amendments, as laid out in the explanatory memorandum, and comment on a number of the amendments.

We recommend that the committee support many of the following proposed amendments

The OpenAustralia Foundation (OAF) recommends that the committee support many of the the amendments proposed in the Freedom of Information Legislation Amendment (Improving Access and Transparency) Bill 2018 (the Bill).

The OpenAustralia Foundation recommends the committee support proposed amendment 1 –  the Archives Act 1983, 55B, reporting on legal expenses we support this amendment.We further encourage the committee to consider if, in 2018, it isn’t unnecessary to wait up to 12 months for the public to know how much the government spent on external lawyers to challenge Freedom of Information (FOI) requests. We therefore respectfully further suggest that this information be required to be ongoingly reported. For example, this information could be declared publicly online within 30 days, in a machine readable format, on their own website, and uploaded to a public repository such as data.gov.au. Summaries would then easily available and downloadable, for the Annual Report, as well as reuse by others who report this information independently, and for external researchers use.

Amendments to Australian Information Commissioner Act 2010 and Amendment 2  At the end of section 10, Amendment 3 Subsection 12(2)

The OpenAustralia Foundation supports qualified assessment of FOI reviews by a sufficiently resourced, fully Independent FOI Commissioner who has appropriate qualifications and who has demonstrated support of FOI as a vehicle for giving people the right to access information.

To the extent that this amendment removes a loophole under which an unqualified Privacy or Information commissioner could exercise the FOI functions, we support this amendment.

Amendment 4  At the end of section 14 Separate commissioners to be appointed

We support the intention that each Commissioner’s appointment be a separate person, so that one person is not doing the work of three people.

The OpenAustralia Foundation broadly support amendment that the office not be left open for more than 3 months is useful in the short term. It’s a defensive maneuver to ensure that the role be not effectively ceased by leaving it open.

However, we have a question about this. Is it possible to compel the executive to make an appointment? Or give the power to the Senate or the House to appoint someone if the position remains vacant for a period of time?

While this comment is beyond the scope of consideration by the committee, the OpenAustralia Foundation would like to suggest a more sustainable way to ensure the role be usefully maintained. Introduce a more robustly Independent process, where the appointment of Information Commissioners is not dependent on the support of the executive that they are there, ultimately, to hold to account.

It’s time for a full review of Freedom of Information

While OpenAustralia Foundation is recommending the committee support this amendments bill, we do so as it is the only thing we are aware of before Parliament that seeks to improve access to information.

We note the recommendations of Dr. Alan Hawke in the 2012-2013 review and the suite of recommendations that yet lie unexplored and untapped in the provision of better FOI service for everyone.

In Australia’s first Open Government Partnership National Action Plan, the Government committed to a range of reforms including Information management and access laws for the 21st century. This too remains stalled or incomplete.

Freedom of Information is essential to a secure democracy

The Government holds information gathered on behalf of the Australian people. The Government is required to make documents in the public interest freely available to the public. Governments also need to give open access to people wanting specific information.

Australia was an early adopter of securing the rights of everyone to access information, but the rest of the world is moving quickly and we’re now falling behind.

Only half a dozen countries secured the rights of everyone to access information when Australia enacted the Freedom of Information Act 1982. As of September 2013, at least 95 countries around the world now have Freedom of Information laws. These are also known as Access to Information and Right to Information laws. [1]  A number of countries have enacted FOI legislation since this report, taking the number to well over 100. This access is a key component of transparency, accountability and civic participation. It’s a way for everyone to scrutinize Government and public sector information, to become better informed, and to take full part in our democratic system. FOI is increasingly strengthened as part of Open Government initiatives, in which Governments are expected to be open and promote openness.

The last time we wrote to support retaining the Office of the Information Commissioner (OAIC), Australia ranked 47th in the world, and today that ranking has further slipped to 65th in the global index of right to information standards. [2]

We can do much, much better.

RightToKnow.org.au

Freedom Of Information (FOI) is a crucial part of the checks and balances in any democracy, and key part of transparency and accountability of Government.

The process of making a freedom of information request is not very straight forward. OAF created the RightToKnow.org.au site with the aim of demystifying and simplifying this process and helping more Australians make FOI requests.

The site not only shows all requests but the paper trail of correspondence in pursuit of the request from those requesting documents and those holding them. Read more below at About RightToKnow.org.au

The site provides an unparalleled public view of the workings of the Australian Federal FOI system.

RightToKnow collects evidence

OAF facilitated 1627 requests in 2017, and 1787 so far in 2018 FOI requests and their related correspondence made to local, state and Australian Federal authorities through RightToKnow.org.au

OpenAustralia Foundation sees how many requests were unsuccessful (refused or did not turn up any documents), and just how many remain unresolved. These await reply, await classification, are overdue, or long overdue for a response. RightToKnow doesn’t provide a way to follow  them through to Information Commissioner review currently.

The OAF has previously provided information relating to unsuccessful requests to earlier inquiries. We are currently compiling the latest statistics and will publish them here as soon as they are available.

Australia needs a fully functional and fully independent OAIC

For accountability, everyone need access to a free merit review system administered by an effective independent office.  We already have that office, in the Office of the Australian Information Commissioner (OAIC). Underfunding the OAIC, then complaining that IC reviews take too long and using public money to hand appeals over to the Administrative Appeals Tribunal (AAT), a system that is inherently more expensive to administer each decision, and which requires legal representation on both sides, looks like a heavy handed, inefficient move at increased cost to the Australian public.

Delay release by the Agency or Minister to the public, by default

Amendment 8  Subsection 11C(6) where the  agency or Minister delay public release of information released to the applicant.

In response to amendment 8, the OpenAustralia Foundation understands that FOI is applicant blind. 10 days is a fairly lengthy delay by default, however delaying release by default can also be a safer option. We’ve seen examples on RightToKnow just in the last couple of months where private information was erroneously published; therefore it makes more sense to default to a delayed release to give both sides a chance to review what’s about to be released.

We note that this would be possible to implement across the board in RightToKnow, and is a provision we have been actively considering.

We note the importance that this provision only binds the agency or Minister; it doesn’t prevent the applicant from making the information they received public immediately.

*Time limit for publication
(6) The agency or Minister must comply with this section within 10 working days after the day the person is given access to the document.”

Further to this provision, there is an opportunity to clarify that documents should be published at this time. Currently what’s required to be published in disclosure logs is a way to access the documents, and not necessarily the documents themselves. This is interpreted by some to not publish the documents online.

Keep Freedom Of Information Free for Everyone

Amendments 9  Subsection 29(1)(d) and 10  After subsection 29(5) suggests granting an exemption of fees if the application processing costs are under $1,000.

The OpenAustralia Foundation opposes amendments giving different rights of access to Members of the House of Representatives and Senators. We believe that the existing 5(b) public interest test would be sufficient here – that is, if the cost is low and a Senator or Member is requesting the information, we believe that it’s likely to be in the public interest that charges are waived..

We believe that making exemptions for politicians makes it more difficult for politicians to notice problems with the legislation that affect everybody else using it.

If this amendment has been suggested because of an understanding that Parliament is currently paying too much in order to have FOI applications approved, we believe this indicates that costs are too high for everybody and would prefer to see them dropped overall rather than an exemption being made for Senators and Members.


Fund the OAIC, and let it serve the Australian Community. Keep Appeals Timely and Free, for Everyone

Amendment 12:  At the end of Division 6 of Part VI, 55JA   Procedure in IC review—notice requirement if lengthy review and 55JB   Procedure in IC review—transfer to Tribunal

To give ordinary users of FOI access recourse when their requests are denied, refused or avoided, then FOI law gives the a right of appeal to an independent office. An external review system which is accessible, free, and appropriately resourced to enable independent and timely assessment of whether the everyone’s rights of access were upheld is essential.

The OpenAustralia Foundation does not support amendment 12 as it stands, it’s a long time to wait for a review. The applicant has probably gone through a 30 day initial, 30 day internal review, maybe some consultation, even where the authority is straightforward in their dealings. It’s possible for the request to be outstanding for 60+ days when the matter gets to the Information Commissioner (IC)  – The IC should be sufficiently funded to be able to make decisions in the normal course of events within 30 days, and allow them to be referred to the AAT.

We do however, fully support waiving the costs associated with referral to the Administrative Appeals Tribunal.

Having been significantly hampered in its operations since the Government attempted to abolish the OAIC, agencies continue to routine refusing requests knowing this will go on unchecked for significant periods of time, and as a result, far fewer cases have be investigated. If fewer people trust the FOI system, fewer requests will be made. Determinations will lead straight to a more costly review process needing expensive legal assistance. That would be a terrible outcome for FOI and for Australia’s democratic health.

There are those who do wish to go straight to the Administrative Appeals Tribunal (AAT), including media reporters following a time critical story. While this bill provides options to allow people to refer a matter to the AAT without waiting a minimum of 6 months, we are concerned that this provides an incentive to not increase the funding of the OAIC to provide an acceptable service to the Australian community, but rather divert funds to a body that necessarily formalises the appeals process.

Framing FOI as a legal problem, not a civic right is plain wrong. FOI decision makers and officers use the language of legal exemptions in interpreting the Act, rather than emphasise the need for openness and of ‘maximum disclosure’ made more explicit in 2010 reforms to the FOI Act.[4] Having learned all the standard tricks of the refusal trade, agencies have become very adept at refusing FOI requests as a matter of course.

Everyone Has a Right To Information

Switching from one argument to refuse a request to another can be a lack of clarity in the initial assessment, a defensive move, a delay tactic, or a blindness to everyone’s rights to know.

As we have noted before, we’ve had an opportunity running RightToKnow, to see how agencies behave. Publicly available FOI requests and their correspondence have given us all an opportunity to see first hand how agencies handle requests.

The evidence we present shows that there are big differences across agencies’ handling of FOI requests. Some agencies handle requests professionally and courteously. Thank you to those agencies.

Some agencies show a systemic culture of secrecy and a disrespect for FOI requests at work; they’re the ones gaming the system.

A systemic culture of secrecy

We support amendment 11.

Australia has unbalanced laws about releasing information. The Government already appears acutely aware of the risks associated with releasing information but much less aware of the risks of not releasing enough information. We have a system which severely punishes those public servants who release information which in it’s view ought not be public, and consciously or not, systematically encourages and protects those who avoid publishing information they could easily share. Thus it becomes safer for every public servant to hold documents close, and release as little as possible by default. This all helps to create the culture of secrecy.

We’re not surprised when we see agencies interpret FOI law to the most minute detail with the purpose of   releasing information to the public by default. They argue against the release of even mundane documents where the material is uncontentious or even publicly available in another form already. At the same time, obstructive agencies also display what might be wilful misinterpretation or incompetence in their failure or inability to give the documents requested.

Such responses show that there is a culture of working harder to refuse rather than share their documents, Whether due to failing inefficient old information storage and retrieval systems, fear or lack of leadership, they’re acting in flat contradiction to their responsibilities under the Freedom of Information Act 1982(FOI Act).

Some are knowingly gaming the system, and we have observed that switching reasons for refusal can fall into that pattern of delaying tactics. For this reason, we support amendment 11.

Amendment 11  Procedure in IC review—consistent application of exemptions by decision-maker  that includes relying on any exemptions in Divisions 2 and 3 of Part IV that were not relied upon in making the IC reviewable decision.

About RightToKnow.org.au

Right To Know aims to make it easier for everyone to make Freedom of Information requests in a few different ways.

  • Helping you make your successful request is the main focus of the site
  • Bringing all the authorities together saves time, you don’t have to trawl the web for the right authority first
  • To make the process easier to understand, it uses plain english
  • To get a feeling for the scope, wording of successful requests, it offers you dynamic search for related requests so you can see how others have done it
  • Clearly communicates your rights of access
  • Guidance appears as and when you need it to keep the request flowing through Right to Know and the Public Authority to whom your request is being made

RightToKnow does not provide help for individuals accessing private or personal information held by government.

About OpenAustralia Foundation

The OpenAustralia Foundation encourages and enables people to participate directly in the political process on a local, community and national level. We believe that we can help to reinvigorate Australia’s civic culture by using powerful and exciting new technologies to inform and empower people, to address the growing disconnect between the Government and the people who elect it.

We currently do this through our five online projects TheyVoteForYou.org.au RightToKnow.org.au, OpenAustralia.org, PlanningAlerts.org.au and previously ElectionLeaflets.org.au. Additionally we provide infrastructure with morph.io These services aim at finding better ways of making government, the public sector and political information freely and easily available and usable by all Australians. We aim to inform people so they can make a positive difference.

The OpenAustralia Foundation is a strictly non-partisan organisation. We are not affiliated with any political party. We are simply passionate about making our democracy work.

Notes and References

*Not included here or at RightToKnow.org.au are requests which were hidden from the website because they contained inappropriately personal requests for information (not what the site is for). This is made clear on the site’s help page.

[1] Right to Information Index: http://right2info.org/access-to-information-laws

[2] https://www.rti-rating.org/country-data/

Posted in Announcement, RightToKnow.org.au | 1 Response

Electronic voting is coming to the House of Representatives!

By Mackay Ash | Published: June 15, 2018

Two years after the House of Representatives Standing Committee on Procedure published a report supporting introducing electronic voting in the House of Representatives, Coalition MP Christopher Pyne has announced that the Government will be doing just that by 2019!

While this is very good news and a step in the right direction, there is no indication that the Government plans to change its voting practices. Instead, they’ll simply replace how the House currently conducts formal votes – known as divisions – with a new electronic voting mechanism.

How are divisions currently done?

There are two kinds of votes: voting ‘on the voices’ and divisions.

Votes that take place ‘on the voices’ are considered less important and the House doesn’t record how our Members of Parliament (MPs) vote in them. Instead, the vote is won by shouting ‘AYE!’ or ‘NO!’ more loudly than the other side.

Divisions, on the other hand, are recorded. During a division, our MPs stand up and walk to one side of the chamber or the other depending on whether they wish to vote ‘Aye’ or ‘No’. Their names are then recorded and the side with the most names wins the vote.

How will electronic voting change things?

As things currently stand, electronic voting won’t change very much. Instead of standing up and walking around in order to vote in a division, our MPs will instead input their vote electronically.

This will speed things up – “freeing up more time for important parliamentary business”, according to MP Pyne – but won’t otherwise change parliamentary procedure. Voting ‘on the voices’ will continue to take place and we will still not have any data on how our MPs are voting at those times.

Why is this disappointing?

Most votes in Parliament take place ‘on the voices’. This means that we have no data on how our individual representatives are voting most of the time – a serious problem in terms of keeping our representatives accountable.

As we submitted to the inquiry back in 2016, we believe that, thanks to the time saved by electronic voting, it would allow the House to “produce a fuller voting records” by ensuring that all – or, at least, most – votes are officially recorded:

This record will be used by Australians to hold their MPs accountable, and will become an important historical record of MPs’ contribution to our democracy. Without this level of accountability, our democracy remains open to abuse by politicians who say one thing to their electorate but vote quite differently within Parliament.

Where to from here?

Introducing electronic voting into the House is an excellent first step towards true accountability. Although the Standing Committee on Procedure disagreed with our proposal back in 2016, it is possible that once electronic voting is up and running well, they will be more inclined to extend its use to all forms of voting in the House.

What about the Senate?

There are currently no plans to introduce electronic voting into the Senate. But the good news is that there are far more divisions taking place in the Senate thanks to our noisy independent and small party senators, who regularly demand that a formal vote takes place rather than simply resolving the matter ‘on the voices’. This means our senators’ voting records already tend to give a more accurate picture of how they are voting.

 

 

Posted in They Vote For You | Tagged , , , , , , | Leave a comment

My Outreachy Internship with Open Australia Foundation

By Hisayo Horie | Published: October 16, 2017

Cross-posted from Hisayo Horie’s blog, our wonderful Outreachy intern for the last 3 months.

Officially I finished my internship with Open Australia Foundation, and I already miss working with OAF so much. I learned so much about programing and development, as well as community centred technology, challenges, and most importantly where I stand and where I want to go.

The project I worked on (and will most definitely keep working as a contributor) is councillor data contribution project (we did not find a cooler name for the project in the period of my internship, but I believe it can still happen). The project is about creating a user-friendly system and interface to enable volunteer contributions to keep an Australian local councillor database so that other apps can be fed by the database.

Screenshot showing an example of adding councillors to PlanningAlerts

I really like this project because it is about expanding and diversifying the participant base of civic tech. One of the challenges in civic tech is the critical lack of community participation and consultation in the process. It is interesting in some way (not in a good way) when a group of social justice minded developers create apps to solve social issues without consulting the area experts or people who are the most affected by the issue, nor attempting to create a process that lets those people actively participate to be a part of the solution. Deconstructing this divide between tech people and non-tech people is a key to create community-centered technology. And I believe the best way for deconstruction is the construction of something to actually address the challenge better. In the case of our project, to create a system that makes the contribution possible without technical knowledge.

Another thing that is important about this project is the process. If we create something to challenge the issue of exclusion, our development process needs to reflect the value and the goal. Before start coding, we spent about 2 weeks to figure out our design principles and the goal. Although I felt an itch to rush the process to get on coding, I am glad we spent the time to build the solid foundation of the project.

It is truly great for me to work on the project that addresses an issue I personally care for, and even more so because the team I worked with was just unbelievably amazing. Luke and Henare are both very patient, willing to teach, and VERY particular in how we write code. And that‘s what I want from mentors. They definitely exceeded the expectation. I learned a lot not only about rails (the framework we use for the app), but also about communication and version control using git, and reason why it is so important.

I missed the official application deadline because I spent too much time questioning if I should apply or not. I thought I wouldn’t get it anyways because I am not experienced enough. It was a fortune that OAF had an extended deadline, because they joined later. I saw they are using Ruby on Rails, and decided to give it a shot. I am so happy I did as I ended up working with great people for a great project, and I learned tons.

At that time, I was on the fence about giving up a career in programming because no matter what I try, the door doesn’t seem to open for me. And probably I was not settled with the decision myself. The leap I made from community work, art-based education and storytelling was very big, and in previous work I mostly worked with black, indigenous, and people of color who do not shy away from being political because we have to to survive. I was struggling to see myself in the corporate world of tech.

And to be very honest, it is not easy for people like us out there. Even within the carefully worded eligible candidates of Outreachy, (“You identify as a woman (cis or trans), trans man, or genderqueer person (including genderfluid or genderfree)”), we need to acknowledge there are big gaps among these group in terms of access, level of surveillance, and even the resources (i.e. other privileges) to capitalize on these oppressed identity markers (and this happens all the time in the time of diversity industrial complex). There are not that many opportunities out there for us, and these internships are a few of them. This is the first time I felt seen in my experience of tech industry.

Through this internship, I came to realize that technology doesn’t and shouldn’t only belong in the corporate industry, because technology also is a culture. We need to build the culture of technology that is truly inclusive and welcoming, and to work to build technology for social good. I had long forgotten why I started to learn code, and through this internship I remembered. I started to learn code because I wanted to advance my art-based education practice. I wanted to write code to contribute in creating technologies in hands of people who are at the margin. Although there is a lot that needs to be done and change, I also started to see the ways to contribute to nurture the culture of social justice from inside of the industry, and started to meet people who are working for it too.

This internship was life changing. The biggest thank you for Luke Bacon, Henare Degan and Outreachy for making this possible. I will be around to make more contributions in OAF projects!

Posted in Development, Outreachy, PlanningAlerts.org.au | Tagged , , , , , , , , , , , , , | Leave a comment

People Have Problems Accessing Information

By Kat Szuminska | Published: September 26, 2017

‘Australia will ensure our information access laws, policies and practices are modern and appropriate for the digital information age.’ That’s the ambitious task that Attorney General’s Department took on with Open Government commitment 3.1 Information management and access laws for the 21st century, in Australia’s first Open Government National Action Plan.

Attorney General’s Department saw a chance to use a different approach to understanding this problem, and they started by asking people who use Freedom of Information and Archives about their experiences.

Attorney General’s Department staff held consultations with people inside Government, Civil Society Organisation members and people who access information through FOI and archives. Peter Timmins (AOGPN) and me (Kat Szuminska – AOGPN/OpenAustralia Foundation) met with them to reflect on experiences that people have with the current FOI and archives systems. By June 30 this year, all together, AGD had consulted with 32 government agencies, 17 civil society members and nine end users.

To bring real world experiences to the table I asked our valued administrators at Right To Know to share what they’ve seen when people make public freedom of information requests in Australia. 

Here’s what they shared:

 

You must request documents, not information, with your Freedom of Information request

Citizens don’t think in terms of documents, bureaucracies, do. People don’t know what types of documents will have the information they want because they don’t work inside the agency they’re making a request to. They mostly don’t want the whole document—they just want some information. This rule confuse people and makes FOI more legalistic, bureaucratic, and resource intensive for agencies and requesters than need be.

FOI Officers are able to use this rule as a way to push back on requests they would otherwise fulfil if they had said they wanted ‘documents that include’ the information they want.

FOI Officers can actually just fulfil the information request as an administrative/information request, and we see the system working at it’s best when this occurs.

Here’s an example of an FOI officer refusing the request for clear reasons, but then providing the basic information the requester wanted anyway:

Document: https://www.righttoknow.org.au/request/3005/response/8446/attach/2/Jonathan%20Ref%20009%2017.pdf

Request: https://www.righttoknow.org.au/request/retention_period#incoming-8446

Here’s another example of a local council providing lots of information to help a requester:

https://www.righttoknow.org.au/request/description_and_purpose_of_works_2

 

Fees

More on fees. Big fees, little fees etc.

A big issue with fees is where massive, spurious fees are imposed for something that shouldn’t take long. Also fees of any kind being imposed to access docs with a clear public interest is bad

Imposing fees used to slow down requests and impose more process on people.

For instance, estimating a large fee and waiting for the applicant to ask that it be lowered stops the clock on the 30 day timelimit https://oaic.gov.au/freedom-of-information/foi-guidelines/part-3-processing-requests-for-access#ftnref44

 

  1. “it took 33 days from when I made a request to waive charges for a decision to be made.” https://www.righttoknow.org.au/request/mygov_privacy_impact_assessment#comment-527

 

Requests for small amounts, e.g. $15, seem counter intuitive. The cost of administering the small payments if often is more than the amount charged for both agencies and requesters. When fees like this seem arbitrary they kill trust in the process and the good will of FOI officers.

At the state jurisdictions the imposition of fees and forms is the major reason requests are cut short.

On the other hand, many FOI Officers make their own public interest assessment and waive fees. Once again, this is the system working at it’s best. People should be supported in making public interest information public, not charged. Where charges are imposed on public interest requests, income and wealth shape people’s ability to help our government agencies increase their transparency.

Here’s a great example of an officer waiving fees:

https://www.righttoknow.org.au/request/point_bolingbroke_and_ardrossan#incoming-6379

Often the person making the request is more of an expert in the topic than the FOI officer. Accordingly they often know more efficient methods to extract the information they need from the standard systems used by the agency. A more collaborative approach would promote these knowledge exchanges to extract information from government systems more efficiently. In situations where the officer ignores advice from the requester, imposed fees often seem absurd or obstructive.

Forms

Like the ‘documents, not information’ rule, FOI request forms seems to purely benefit agencies and deter people from making requests.

We see lots of requests ending when forms and fees are imposed.

Agencies impose forms when there’s no clear reason to, often going against their Act and Information Commissioner recommendations.

These forms are mostly poorly designed and request information not required by the Act for a request to be processed. This creates a bad experience and deters requesters.

Like small fees, requiring people to fill out forms with information they’ve just emailed the agency seems arbitrary and destories good will between the requester and the FOI officer.

Here’s some examples:

NSW Police requiring forms where the GIPA Act says you don’t need them:

https://www.righttoknow.org.au/request/latest_staff_training_manuals_fo

NT Department of Justice requiring forms against their Act and advice from their Information Commissioner:

https://www.righttoknow.org.au/request/list_of_information_requests_to#incoming-8732

Legalistic culture that intimidates people out of requesting

Inconsistent/nonexistent disclosure log practice

Initially under 2010 reforms many agencies were placing full copies documents on their disclosure log but by 2013 this was ceased by several agencies including a decision of the Executive Board of the Attorney General’s Department to “reduce resources spent on preparing documents for publication”. Other agencies have interpreted the OAIC guidance on disclosure logs which includes a template to be followed in a way that reduces the usability for example the ACCC merely publishes each release labeled with a internal code that doesn’t suggest what the FOI request regards.

At the same time in 2014 the Department of Immigration and Border Protection began increasing the number of documents placed online such that the vast majority are now available.

 

Cumbersome and expensive arbitrary payment methods

No option to pay by Credit Card or Electronic Funds Transfer from many large agencies despite fact that charges could be placed in the Consolidated Revenue Fund centrally

 

 

OAIC doesn’t come into bat for people

Shouldn’t they be making sure agencies that clearly don’t comply with the spirit and letter of the Act are made to comply? e.g. Department of Human Services, Australian Tax Office, Department of Immigration and Border Protection

Read more about experiences with RightToKnow, and some of the work that the OpenAustralia Foundation does to help people get more, and expect more, from FOI. 

What next?

The Government recently published their Mid-term self assessment on OGP including this update:

..the Attorney-General’s Department consulted with government and non-government stakeholders, they  held a workshop in conjunction with the Department of Human Services Design Hub. The aim of the workshop was for AGD to present its findings of the initial consultation (the discovery phase) and for government, non-government and civil society stakeholders to come together to develop and co-design reform options to make access to, and management of, government information easier in the 21st century.

I participated in this workshop, and was asked not to share content from this event. If not before, I’d expect to see an update that includes a round up at the next meeting of the Open Government Forum in October.

Updates about this reform and find out more about how Australia’s Open Government National Action Plan is going at PM&C’s OGP Dashboard

 

Posted in Open Government Partnership, RightToKnow.org.au | Tagged , , , , , , | Leave a comment