They Vote For You

By Matthew Landauer | Published: October 21, 2014

We believe everyone should know what our politicians do on our behalf in parliament. If we don’t, how can we hold them to account meaningfully? How can we begin to talk to them about what we want them to do?

Trust in politicians is at an all time low. Yet we live in a time when technology has the potential to bring us ever closer to our representatives. It’s time we started using it.

What we see on television or read about online or in newspapers about parliament focuses on arguments in Question Time. We see adversarial battles, emotional rhetoric and partisan spin. It can be a distraction. It’s not an accurate view of what goes on in parliament.

We need to look behind what they say, and examine how they vote, because the most important things that politicians do in parliament is vote. They vote to make new laws, and update and amend the existing ones. Changes to the law can and do have a profound effect on our society, and the way we go about our daily lives. Yet if you want find out how they voted, you have to roll your sleeves up, trawl through pages and pages of Hansard, Votes and Proceedings and Journals of the Senate and try to make sense of it all. This is not something anyone can do easily.

Parliament should be easy, and accessible. That is why today the OpenAustralia Foundation is launching a new site They Vote for You, so you can find out how your representatives in parliament vote on issues you care about.

They_Vote_For_You_—_How_does_your_MP_vote_on_the_issues_that_matter_to_you_

Our political researchers have been working to accurately summarise votes and connect these with easy to understand policy positions. So, now you can see where everyone in parliament stands on issues that you could talk to them about, issues that your MP or Senators might vote on again in the near future. Do you agree with them, or not?

How, for instance, has your representative voted on “a minerals resource rent tax“, “decreasing availability of welfare payments”, “increasing surveillance powers”, “decreasing availability of abortion drugs”, or “increasing restrictions on gambling”?

For_decreasing_availability_of_abortion_drugs_—_They_Vote_For_You

They Vote for You is a small but vital piece of “civic infrastructure” which help build the roads and bridges that 21st century citizens need to help them get to where they want to go.

You might ask if parliament should be putting all this together? Well, yes perhaps, but the machinery of parliament is slow, and we are not. In many ways, it’s easier for us to do it than it is for them. That’s why we are filling the gaps and hopefully in the process, change everyones’ expectations of what democracy looks like in the internet age.

And you can help.

If there is a policy missing that you would like to see you can do your own research and add it, much like Wikipedia. See a mistake? You can fix it yourself. A mistake could be as simple as a spelling error, or as subtle as one word in a description which changes how you interpret the result of a vote.

In our short life as a charity, the OpenAustralia Foundation, has already created several other non-partisan pieces of civic infrastructure. OpenAustralia allows you to keep tabs on what politicians say in Federal Parliament. With PlanningAlerts you receive an email any time something new is planned to be built or knocked down in your local area. With RightToKnow you can access inside information on what your government is doing by making a Freedom of Information request really easily. With ElectionLeaflets we monitor the leaflets that people receive in the mail during elections. If you live in Victoria, you can upload election leaflets you get in the mail for the upcoming state election.

The right to know what politicians say in parliament and how they vote on our behalf is one that has been fought for and won by citizens. It is a right that we must not take for granted.

Now it’s time to bring it bang up to date.

(An edited version of this was published at Guardian Australia)

Posted in Announcement, They Vote For You | Tagged , , , , , , | Leave a comment

And what a hackfest it was

By Henare Degan | Published: October 3, 2014

September 15th is The International Day of Democracy and this year also marked the start of the Global Legislative Openness Week (GLOW). GLOW is a week of openness activities around the world, with events hosted by the Legislative Openness Working Group of the Open Government Partnership and members of the parliamentary openness community. To celebrate GLOW, the OpenAustralia Foundation ran one of our legendary hackfests on the weekend of the 20th and 21st.

And what a hackfest it was. Our most successful ever – if the number of code contributions is anything to go by. 18 pull requests from over half a dozen different authors in just one weekend is a excellent achievement. These contributions have made valuable and tangible improvements to a number of our civic projects.

We started the weekend with an introduction to parliamentary and legislative openness and an exciting new project we’re soon to launch. This progressed into a discussion with participants about what challenges they want to use civic tech to address and what everyone wanted to work on over the weekend. Matthew set the scene for the weekend:

“Let’s make stuff. What will you make?”

Over the weekend participants were encouraged to run breakout sessions on topic or projects that interested them. Nick explored political donation data, Luke gave us a detailed look at our soon-to-be-released parliamentary vote tracker project and Matthew provided an introduction to web scraping on our morph.io platform.

We were very pleased to have representatives from the Australian Federal Parliamentary Library attend our hackfest. To our knowledge this was the first time staff from any of Australia’s parliaments have attended a hackfest in an official capacity. They quickly discovered that we share many of the same interests in parliamentary data and Hannah generously shared her knowledge of parliamentary procedure in another exciting and informative breakout session.

When the dust settled Sunday afternoon our show and tell revealed the impressive results of a fun weekend hacking parliamentary, legislative and democratic data. In addition to the many pull requests, Sash had created a beautiful new rich email template for OpenAustralia.org, Matthew had used MapIt to finally solve the problem of easily mapping your location to electorates, Alex added great new Parliamentary Bills data to the OpenAustralia.org scraper, and Tracy developed much improved forms with Luke on our soon to be released parliamentary voting project.

Thank you to everyone that participated, you made it the amazing weekend that it was. Thanks to Googlers Tim and Jack for volunteering all weekend and thanks once again to our hosts Google for their continued support.

See you at the next pub meet or hackfest,

Henare

P.S. Check out more great photos of the hackfest by Lisa Cross.

Posted in Event | Tagged , , , , , , , , , , , , , , | Leave a comment

Time for another hackfest – Global Legislative Openness Week (GLOW) edition

By Henare Degan | Published: August 13, 2014

A photo from a previous OpenAustralia Hackfest

The OpenAustralia Foundation is pleased to invite you to another one of our legendary hackfests. In some of our previous hackfests we’ve opened PlanningAlerts to over 1.8 million more Australians and brought hacks and hackers together to launch our Freedom of Information project Right To Know.

Places are limited so RSVP for free now or read on for more details below. You can RSVP on our Meetup group.

This time we’re going back to the old school and will be concentrating on opening parliaments, politicians and elections. Our event is timed to coincide with the Global Legislative Openness Week (GLOW). This week sees events hosted by the Legislative Openness Working Group of the Open Government Partnership and events like ours hosted by members of the parliamentary openness community around the world.

GLOW Logo

As usual we’re inviting anyone interested in open government, not just developers, along to our event that will run on the weekend of the 20th and 21st of September 2014. Once again we’re being generously hosted by Google Sydney.

While you’re welcome to come along and work on anything you’d like, here’s some ideas to get you started:

  • Spruce up OpenAustralia.org or add new features
  • Features like expenses data for our federal politicians
  • Or maybe transcribing their financial interests into a database
  • Visualising party preferences
  • Writing scrapers on morph.io to liberate more parliamentary data

Oh, we’ll also be providing special access to our latest project that allows you to analyse votes in our federal parliament!

This is roughly what the weekend will look like:

Saturday
 10:30 – Doors open
11:00 – Introduction and talk(s)
12:00 – (Free!) Lunch, chat and meet people to work with
13:00 – People have eaten and found something and someone to work with
17:00 – Pack up and have a drink somewhere
Sunday
 10:30 – Doors open and hacking starts
12:00 – Lunch
15:00 – Show and tell
16:00 – Grab another well deserved beer

Grab your free RSVP now – we hope to see you there :)

Posted in Event, Morph, OpenAustralia Foundation, OpenAustralia.org, Projects | Tagged , , , , , , , , , , , | Leave a comment

Sydney Pub Meet Now A Regular Feature

By Henare Degan | Published: July 14, 2014

Photo of the Woolpack Hotel

Before the night was even over at our last pub meet people were asking when the next one would be so we’re making it a regular thing!

From now on, the last Tuesday of every month will feature the OpenAustralia Sydney Pub Meet. We’ve now got a fancy Meetup group set up so please join and RSVP if you can make our next meet on the 29th of July.

Once again we’ll be cosy by the fire of the Library room at the Woolpack Hotel, a short walk from Central station, where you can enjoy good pub food and craft beer.

See you there!

Don’t forget to join our Meetup group and RSVP: http://www.meetup.com/OpenAustralia-Foundation/events/194788272/

Posted in Announcement, Event | Tagged , , , , , , , , | Leave a comment

Sydney Pub Meet – 24th June 2014

By Henare Degan | Published: June 17, 2014

Photo of The Library room

We’re having a Sydney pub meet on Tuesday the 24th of June 2014 and you’re invited.

Come along, say hi, and have a chat about open government, transparency, FOI, and civic hacking with us and friends. You can even get a sneak peek at our upcoming project.

We’ll be sitting by the fire in the cosy Library room of the Woolpack Hotel, which is a short walk from Central Station. There’s good pub food and craft beers on tap – join us at 6:30 PM (for 7 PM).

Please RSVP so we can make sure they have room for us. You can find all the details on the Eventbrite page.

See you Tuesday :)

Posted in Announcement, Event | Tagged , , , | Leave a comment

Parliamentary vote tracker project: A Case Study – Asylum Seeker Policy

By Mackay Ash | Published: April 28, 2014

This post is part of a series on our parliamentary vote tracker project. Read our previous post to learn more about the concept of policies and policy selection. You can also read our previous posts: a technical update and an introduction to what we’re doing and why.

Co-authored by Natasha Burrows.

Linda Planing Wood by Lester Public Library on Flickr

Specific wordings of policies have been an issue throughout the process of policy development. The one that springs to mind is our policy relating to asylum seekers. The problem here is exacerbated given the heightened politics surrounding the issue. Narrowing this policy down to something that is objective and can be measured has been quite difficult.

In a draft, the policy was written like this: For “a stronger system for asylum seekers arriving by boat”. This wording was borrowed from the UK Public Whip site. However, this way of phrasing the policy was flawed because both of the two main parties, Labor and the Coalition, broadly agree on the need to implement a “stronger system” with regards to asylum seekers arriving by boat. However in the last session of Parliament, the Coalition often voted against Labor’s measures for a stricter system, specifically in regards to the so-called Malaysia Solution. As a result of the phrasing of our policy, our data suggested that the Coalition was voting against a stricter system for asylum seekers arriving by boat. However in reality the Coalition was really just voting against Labor’s policies for a stricter system for asylum seekers, while at the same time proposing their own strict system.

The language has also been an issue. What exactly does a “strict system” entail? Our reasoning for keeping the language broad (and, admittedly undefined) with a word like strict was because asylum seeker policy for boat arrivals in Australia is quite broad. It comprises of issues relating to visas, mandatory detention and offshore processing. To narrow this down to only one specific component of asylum seeker policy risked making the policy too specific for the legislation that was presented to parliament. However broad language risked presenting issues and policies with a pseudo-political-correctness. In reality, developing a succinct, understandable and objective policy is much harder than we first imagined.

Ultimately, we decided to split the broader policy into four parts: “For greater scrutiny of detention centres”; “For offshore/regional processing of asylum seekers“; “For temporary protection visas“; and “For the Refugees Convention and Refugees Protocol“. So far this phrasing seems to be working.

Screen shot of example policy

Because our whole modus operandi is to increase openness in Parliament, it only makes sense for us to open up the processes by which we are doing this. We really do want as much feedback as possible.

Posted in Development, Projects, They Vote For You | Tagged , , , , , , , , , | Leave a comment

Parliamentary vote tracker project: The concept of policies and policy selection

By Natasha Burrows | Published: April 11, 2014

This post is part of a series on our parliamentary vote tracker project. You can also read our previous posts: a technical update and an introduction to what we’re doing and why.

Apples and Oranges by Automania on Flickr

Advice from Richard Taylor has been a great anchor throughout the process of developing policies. This is that:

The first priority when creating new policies is that they should attempt to provide the public with information on voting behaviour that is intelligible and interesting to a wide range of users. The second priority should be that a reasonable, neutral observer would conclude that the language and topic contained in a the policy description was a fair representation of an MP’s voting behaviour, whatever side of the policy they end up on. The title and description of the policy can always be improved later.

For now we are continuing along, aware that the flaws that we have can indeed be improved later. We acknowledge the current flaws in our project and are looking for advice on how to improve them.

For now we have put together a patchwork of factors that we believe are important when defining a policy. These factors have been a checklist for us when deciding on a policy focus. These have included:

  • Issues that rank highly as concerns to Australians in polling data. For example Ipsos consistently ranks crime, cost of living, healthcare, education and asylum seekers as the top five concerns to Australians, so policies we have chosen engage with these concerns.
  • The focus on the issue in the media. While this has not been a deciding factor in policy selection, we have decided policies with an understanding of the coverage it is getting in the media. Often, however, we have been inspired to focus on issues that are not necessarily getting a whole lot of media coverage, for example ‘For Stronger Unions’.
  • Any potential impacts on the Constitution. For example, legislation introduced in the last sitting of the 43rd Parliament (2010-2013) had the potential to impact the Constitution, specifically how the relationship between the federal, state and local governments interact.
  • The turnout in Parliament, meaning how many representatives are present in Parliament during a division.
  • The number of rebellions in Parliament. A rebellion occurs when a representative votes crosses the floor to vote against the rest of their party.
Posted in Development, Projects, They Vote For You | Tagged , , , , , , , , , | Leave a comment

Parliamentary vote tracker project: what we are doing and why

By Natasha Burrows | Published: March 31, 2014

Chamber Red and Notice Paper

Hello! Welcome to a series of posts about the OpenAustralia Foundation’s new conception, a parliamentary vote tracker. This project is based on the Public Whip, a site developed in the UK by Julian Todd and Francis Irving. The point of this project is to make the processes and procedures of parliament more accessible to the general public. Our project takes voting data from Hansard to see exactly how our MPs are voting on legislation.

In Australia we are truly lucky to have an established democracy. However, this system is not as open and accessible as it could and should be. Information about our politicians and how they represent us is freely available but the data is convoluted and too difficult to understand.  The language used in Parliament and copied into Hansard is often complicated, and the processes are too far removed from what is easily understood. One of our main goals in this project to make what happens in parliament – the debates, the votes and the proposed and passed legislation – more easily available to the public eye.

This is why we have been working on this project. Our end goal is for any member of the public to access our site and be able to see how a Member of Parliament or Senator votes on any number of issues. We are dissecting the votes in Parliament to make each division (or vote on a piece of legislation) easily understood. Our conception focuses on the context surrounding a piece of legislation, what the legislation actually proposes, the impact it has and the debate surrounding it in Parliament. We aim to be able to distil these votes down so that anybody can come to our site and understand exactly what each vote in Parliament really means.

We are narrowing these issues down to specific policies. Under this notion, a policy is a statement that you can objectively measure votes against. Most basically, policies are just a group of divisions (votes) in parliament. As a Research Assistant, my role is to look at these votes and develop policies by which to measure a Member of Parliament’s or Senator’s stance on an issue. A key aspect of this is keeping the language plain, which is something that Hansard is not.

Firstly, let’s get up to date on the lingo. A division is simply an official vote in parliament. For example, representatives may vote on a motion to read a bill for a second time (or, put another way, vote on a motion to agree with the main idea of a bill). Divisions are only called when there is not a clear indication of how many representatives either vote aye (yes) or no in relation to motions. This in and of itself is a problem with parliament – we only have data on controversial votes. Most of the legislation passed in Parliament does so without a division. This is because when there is bipartisan support on an issue it is clear that it will be passed and a division is not called.

A screenshot of divisions

Along the way we have spitballed, debated and drafted the best way to select and articulate what exactly our policies should be. Defining these policies and our method of selection has been one of the largest hurdles in the project so far. We are still deciding how best to address these concerns, and are really open to any feedback and advice you have, dear users of the internet.

Posted in Development, Projects, They Vote For You | Tagged , , , , , , , , , , , , , , , | Leave a comment

A long overdue tech update about Australia’s first ever parliamentary vote tracker

By Henare Degan | Published: March 23, 2014

Photo of working on the project

Back in July last year Matthew told you that we’d started work on a new project and it’s high time for an update on the progress of that project. I want to share with you what we’ve been doing, where we’re at right now, and some exciting new development that we’re undertaking before we launch this pioneering project for all Australians.

The open source code for our vote tracking project is once again inherited from the UK. The great thing about this is that our parliamentary systems have a lot in common, at least on the surface. Dig a little deeper (and writing code to analyse parliamentary voting is certainly doing that!) and you find many subtle but crucially important differences.

These are the kinds of questions we’ve needed to answer and then represent in code:

  • Can MPs vote both aye and no as in the UK?
  • Can the speakers and their deputies vote? If so, under what circumstances? Are they representing their party when they vote?
  • Is there a way to actively abstain in the Federal Parliament?
  • Should tellers be included when tallying up votes?

A key part of this project is interpreting the data we’re analysing and making it easy to understand  – without a PhD in political science. To this end, we’ve been excited to have some political researchers on our project team. They too will be sharing some posts about the process they’ve gone through in the coming weeks.

Like our work on OpenAustralia.org, this work has uncovered a number of issues in the official record. Things like people voting when they aren’t MPs any more. As always we’ve sent these fixes to Parliament House to be corrected. By the way, this would be so much easier if they just had an open bug tracker :)

The code we inherited for this project was written by some brilliantly smart people but some of it is over a decade old and it wasn’t being actively developed. It’s a simple fact that over time complexity creeps in and technical debt builds up. This has lead to making some difficult decisions about how to move forward.

We’ve started work on a Rails port in an effort to strip back complexity in both the codebase and usability. We’re not rebuilders by nature so this was a decision that was not made lightly but we’re confident this will mean the project will have a brighter future.

We’re approaching this port carefully, in the same way Matthew did with PlanningAlerts. The PHP code still works and it’s built in such a way that the still active UK site could take advantage of our work and “upgrade” to this version.

To be able to present our best work possible there are still a small number of bugs to resolve. Once that’s done we’ll focus on getting the Rails port ready for launch. We don’t have a date right now but it should be within the next few months.

If you’d like to find our more you can see all the development live on GitHub (including those issues I imported recently, yay!). Please drop me an email if you’re interesting in getting early access and helping out. Also, don’t forget to keep an eye out for some really interesting posts from our political researchers in the coming weeks.

Posted in Development, Projects, They Vote For You | Tagged , , , , , , , , , , , , , , , | Leave a comment

Morph: Announcing A New Scraping Platform

By Matthew Landauer | Published: February 27, 2014

Web scrapers could be called the unsung heroes of data liberation. They invisibly power so many websites including real estate search sites, product comparison sites and the ever familiar Google search engine.

Scraping is where you run a program to extract structured data from web pages.

Web scraping powers PlanningAlerts, which allows you to find out what’s being built and knocked down in your area. It works by running lots of scrapers to collect the development applications from nearly one hundred planning authority websites.

In fact our other projects OpenAustralia, ElectionLeaflets and RightToKnow depend in one way or another on scrapers too.

For some while we’ve been using a scraping platform, ScraperWiki Classic, to host most of our scrapers. The system took care of running them regularly and giving PlanningAlerts a nice API with which we can get the data.

Scrapers are a little different than other kinds of code. They are dependent on the page layout of a web page remaining consistent. If a website that is being scraped changes then the scraper needs adjusting. To make it easy to maintain scrapers we need an easy way to find out when things need adjusting and we need easy tools for multiple people to help fix things.

ScraperWiki would send us email alerts if any of our scrapers would break. It’s a tool that really served us well.

Unfortunately for the OpenAustralia Foundation, late last year ScraperWiki announced that they would be shutting down their ScraperWiki Classic service. For a while now most of their effort has gone into developing their new platform, which has gone hand in hand with a refocusing of the company’s efforts away from providing free services and towards its paid data science work. This is completely understandable but left me wondering what to do next.

This is what I wrote to Francis Irving, the CEO of ScraperWiki, about our dilemma.

As I’m sure you’re well aware the demise of ScraperWiki Classic has been a thing of great sadness for us. For us at least it hit a real sweet spot. The collaboration was great. Not having to host things was fabulous. The scraping happening whenever and the result being stored in an API accessible db was magical. Also, not having to trust that someone’s code edit will not nuke your server another bonus.

Thank you for everything!

And then on the question of what to do next.

We had a bit of panic at the end of last year when we thought it was shutting down in September and we had to move a heap of scrapers somewhere.

We had an internal scraping thing for PlanningAlerts which still has a bunch of scrapers running on it. Initially we were just going to move everything back there and figure out what to do next. It was a scary prospect having to go back to that because in the days when that was the only thing we used we never had a single scraper code contribution and that’s not pleasant when you have over 80 scrapers to maintain.

Introducing Morph

So just before Christmas I made the difficult decision to take on writing a new scraping platform. I wanted to make something which would be a replacement for ScraperWiki Classic but with a twist that puts a much greater emphasis on collaboration. I started building Morph.

Morph logo

It’s really exciting to be able to share this with you now!

Morph is built around GitHub. GitHub has become the de-facto way that open source software developers collaborate. They truly have nailed the solution with pull requests, their issue system, commenting on code, and social discovery through starring repositories and more.

It’s important to me to leverage those tools with Morph and integrate them very tightly so that Morph becomes a way of using Github but that’s specialised towards scrapers. I’ve made a start on that now and already we can see that it’s working.

The idea with Morph is to borrow the best of ScraperWiki Classic, merge it with the best of GitHub, and then build on top of that to really streamline the process of writing and running scrapers.

Writing and running a scraper yourself is something that any capable developer can do without Morph. With Morph it’s fun and easy.

How Does Morph Work?

A scraper on Morph is a Python, Ruby or PHP script that collects information from web pages and writes it to an SQLite database in its working directory. That’s it. What information the scraper collects, how it does that and how it writes to the database, that’s up to you.

The scraper code is kept in a repository on GitHub which is linked to the scraper on Morph.

You develop the scraper on your local machine and when you’re ready to run it you can test it on Morph with a special command line tool. When you’re happy you commit your change to git. The next time the scraper runs on Morph it picks up the latest code.

The resulting data is stored in a database that you can query through an API. Or you can just download all the data as JSON or as a CSV (you can import into a spreadsheet) or just download a whole binary SQLite database.

Data

JSON output

 

 

 

 

 

Morph will run your scrapers for you automatically if you want. It will email you if there are problems with your scrapers or someone else’s if you choose.

You get great visibility on what a scraper is doing

Melbourne scraper

Morph supports organisations. You can group a whole set of scrapers together in an organisation which multiple people have access to.

Organisations

If you’re a user of ScraperWiki Classic it couldn’t be easier to bring your scraper over to Morph.

Forking from ScraperWiki

You just put in the name of your scraper on ScraperWiki Classic and if the defaults are fine you just click the fork button. The code is put on Github (automatically tweaked for a few naming changes), the data is imported into Morph and in most cases the scraper will run fine without any changes at all! When we transferred 69 PlanningAlerts scrapers from ScraperWiki to Morph only 3 or 4 scrapers needed any changes.

Morph supports scrapers written Ruby, PHP and Python with the plan to support Javascript (and anything else enough people want)

So, please go ahead start using Morph!

https://morph.io

 

Morph Landing Page

 

And here’s your part of the deal. Please let us know what you think. Did it work well for you? Something that could be clearer? Let us know.

Keeping Morph going

OpenAustralia Foundation will be letting anyone run arbitrary code on our servers. This will take computing power and generate traffic. Because we’re paying for those servers and this is likely to need a lot of computing power this is not sustainable.

It is our intention to keep Morph free for most people; the casual user who has a few scrapers that are pretty lightweight. Morph should be an incredibly useful tool that they can depend on.

For the heavy users we will be bringing in paid plans that pay for continued development and the serious server infrastructure that will be needed.

If there is need for it we also intend to support private scrapers which will be under a paid plan as well.

What next for Morph?

There are of course many possibilities.

We’d like to make scraping more accessible so it doesn’t even require writing code. One approach to that would be to write a very high level scraping library that can cover 90% of situations but isn’t really like programming at all. It describes the fields, how multiple pages are navigated and it does the rest.

Then, we create a user-interface that allows you to pick fields from any website that you would like to extract. Behind the scenes it turns that into some code that uses our high level scraping library. If something more complicated needs to get done the existing generated code can be used as a starting point.

The vision there is that Morph maintains all its power and flexibility for the experienced programmer and allows a non-programmer to scrape just as well as the programmer for the common situations.

We’re particularly excited about the possibilities here for making data journalism more accessible to all journalists not just the ones that know how to program. In fact we’ve applied to the Walkley Grants for Innovation in Journalism for funding to pursue this idea.

We’d also like to continue to relentlessly pursue ease of use and design features to increase collaboration.

Enough of what we want it to do. The most important thing is what do you want for the future?

Let us know either by emailing us or even better if you put it up at  https://github.com/openaustralia/morph/issues

Posted in Announcement, Morph | Tagged , , , , , , | 1 Response