Turking the Healthcare Bill

As some of you have no doubt heard, the 2,000+ page healthcare bill is making its way through the U.S. Senate and is due for a preliminary vote on Saturday evening. There's a big challenge ahead for legislators trying to make sense of this complex and lengthy legislation.

We'd hate to pass up the opportunity to perform a public service, so we took a few minutes to use our technology to ask Internet denizens (via Amazon's Mechanical Turk) to collectively read the bill in a lot less time than the 36 hours it would take to read it on the Senate floor.

We know that the eyes of the average citizen tend to glaze over looking at the bill, so we opted out of asking turkers for a deep analysis of each page -- they wouldn't have context of neighboring pages, for instance. We instead asked for a visceral response; we asked each turker to type in their most favorite and least favorite words (from their point of view, of course) that they could find in their particular page of legislation.

First we converted each page of the extremely-long PDF into individual bitmaps -- each one looking something like this (though larger):





Then we used our command-line client (discussed previously) to submit each page individually to Mechanical Turk.

The best (most preferred) words can be visualized in the following word cloud:



The worst (least preferred) words are here:



So what does this all mean? Well, if you read the images literally, citizens want quality affordable health benefits, information, and respect. They are fearful of abuse and fiscal limitations . They also loathe the bill's document structure ("subparagraphs" and "subsections") and legal mumbo-jumbo ("striking" and "amended").

The final costs were as follows:

Mechanical Turk fees: $20
Time until entire bill was read: 8 hours

The $20 fee results from each worker's payout of 1 cent for their analysis of a single page. Certainly you'd have to pay a well-heeled K-Street analysis firm a bit more for a similar result. ;)

Batching Mechanical Turk Jobs at the Command Line

Here at Voxilate, we use Amazon Mechanical Turk for a lot of things - from crowdsourcing translations to gathering market research to deciding where to go for lunch! Some of the jobs are one-time deals that we don't want to spend a lot of time on.

Typically, you can submit, monitor, and approve Mechanical Turk jobs (HITs) using the API or the web interface. The web interface requires a lot of mouse-clicking and .csv file generation and it really doesn't allow for full control. The API is complex and doesn't lend itself to quick experimentation. There are also a set of command-line tools provided by Amazon, but these just wrap the API and can be a little clunky.

Wouldn't it be nice if you could just type a single line and watch Workers' responses come back in real-time like this:

% mturk -j 3 -t "What is your favorite color?"
blue
blue
red

Or this:

% mturk -t "Who is in this picture?" albert.jpg bea.jpg
Albert Einstein
Bea Arthur

We thought it would be pretty cool, so we created a quick script using Python and the boto library.

How does it work?

Given a command like:

./mturk.py -p .50 -j 10 -t "Is there text in this graphic?" -D "If you see text in this graphic, type \"text.\" If not, type \"no text.\"" -w 300 -A IsThereText01.jpg IsThereText02.jpg

We'd create two distinct HITs - one for each file (JPEGs, in this case). The -j parameter controls the number of assignments we submit for each HIT; in this case, we'd run 10 iterations of each HIT, for a total of 20 assignments. Each assignment would pay out 50 cents (specified by -p .50), and because we used -A for auto-approval, you'd be out $10 if workers accepted and completed all of your hits!

By default, the script uses the Mechanical Turk sandbox so that you ensure that your usage is correct before spending real money. The -w option ensures that the script waits the specified number of seconds for HIT results to print to standard out, then exits.

In our example, we used JPEG files, but the input files themselves can be:

• Blank - You don't want to pass any data, you just want to ask a question with a single answer.
  
• Image, Audio, Video - show the user a picture, video, or sound and have them return a response.
  
• HTML - This allows you to have full control over the HIT. Whatever <form> tags you use in the HTML file now posts results to Mechanical Turk. They are hosted on Amazon S3 as an "external question" and appear in the IFRAME of the HIT. (More on this in a future blog post.)

We've found this really kind of useful - for instance, we've used this script as part of a Makefile to implement a complicated workflow. Let's run through an example.

First, let's check our status:

$ ./mturk.py
You are in test mode.
Funds remaining: [$10,000.00]
There are 0 jobs active.

Note that we see $10,000 because we're using the sandbox. If you had active jobs, you'd see the jobs listed by HIT ID and filename. Running mturk.py in live mode (./mturk.py -l) will show you how much money you really have:

$ ./mturk.py -l
You are in LIVE MODE.
Funds remaining: [$41.38]
There are 0 jobs active.


Next, let's submit a job. In this batch, we're using the sandbox (no -l!), and "paying" 4 cents for each assignment, with two iterations of two graphics for tagging. We should auto-approve submissions 360 seconds after completion and wait 400 seconds before exiting the script.

Here's what we run and see on the command line:

$ ./mturk.py -p .04 -j 2 -t "Tag this graphic" -D "Type keywords, separated by spaces, that you think best describe the image. Please enter at least 2 and no more than 5 keywords." -k "voxilate tagging" -d 120 -e 300 -a 360 -w 400 Namazu.jpg SydneyOperaHouse.jpg
You are in test mode.
Uploaded to http://com.voxilate.mturk.s3.amazonaws.com/home/jen/Turkpipe/Namazu.jpg
/home/jen/Turkpipe/Namazu.jpg: Created HIT 936MY1Y9KY4ZTJH4ZXTZ.
Uploaded to http://com.voxilate.mturk.s3.amazonaws.com/home/jen/Turkpipe/SydneyOperaHouse.jpg
/home/jen/Turkpipe/SydneyOperaHouse.jpg: Created HIT M1N2Y04MX34RW21CGY6Z.
/home/jen/Turkpipe/Namazu.jpg: 0/2 assignments completed.
/home/jen/Turkpipe/SydneyOperaHouse.jpg: 0/2 assignments completed.


And here's what we see in the Mechanical Turk Sandbox:



Next, we'll play worker and grab some of these HITs:



And the other:



We'll then see changes at the console, as it polls for completion:

/home/jen/Turkpipe/Namazu.jpg: 1/2 assignments completed.
/home/jen/Turkpipe/SydneyOperaHouse.jpg: 1/2 assignments completed.

Because we submitted two iterations, the Turk won't let me do *all* of the tasks myself, so we wait for other workers to jump in to complete them (on the Sandbox, bug a friend or use another account). When complete, the script exits, dumping the following to the console:

/home/jen/workspace/Turkpipe/Namazu.jpg: 2/2 assignments completed.
/home/jen/workspace/Turkpipe/SydneyOperaHouse.jpg: 2/2 assignments completed.
catfish japanese art
japanese catfish painting
sydney australia opera house
Sydney opera hall Australia water

And that's it - Turking at the command line successful! Add a -l to the command (and an -o output_file to save results to file) and you're ready to go live!

Mobile-Enabled Silent Auction

We've been recently playing with Twilio, and looking for other niche IVR applications to build. The perfect opportunity arose as we were involved with helping out a local charity event.

During the event, there's a silent auction. Attendees bid by entering their name and the amount of their bid on a bid sheet next to the item, and at the end of the event the winners collect their bid sheets and pay for their item.

Cash and check are obvious -- but using a credit card requires the winner to transfer his information to a volunteer who manually enters it into a payment system. There are always more people looking to check out than volunteers to take their payment, so this process is ripe for automation.

Here's a quick walk-through of how it works:

1. All bid items are listed in a Google spreadsheet


2. Users at the silent auction enter their name, bid amount, and phone number on the auction sheets.


3. When the auction completes, on-site staff enter the winners into the Google Spreadsheet.


4. Bidders dial the Twilio-provided number and enter their phone number as a PIN (automatic if they send Caller ID). They are told whether they have won, and if so, what their total is (important for those who bid on multiple items!). They are then instructed to say their names, addresses, and other pertinent information and are of course thanked for participating.


5. This information is then transcribed and posted into the Google Spreadsheet (this is fun to watch in real-time!) The Twilio link to audio is also included in case the information needs to be double-checked.


6. On-site staff can then complete the checkout at their leisure.

We're using Google App Engine for serving the TwiML responses to Twilio, and the Google Data APIs for connecting to Google Spreadsheet.

There are a couple of issues right now, like that of credit card numbers traveling throughout the cloud -- even though all communciation is encrypted between Twilio <-> Google, some of us have a security background and we like to be extra careful ;) We're also looking @ payment integration to pre-authorize cards, for instance, and a SMS back to the winner to tell them their payment has been verified.

The ultimate addition would be to have the entire bidding process phone-enabled, and a demo of such a system is what we've demonstrated with our Twilio contest entry! You can call (877) 277-9839 to try it from your cell phone. Click here beforehand to see the auction items. (Don't worry, it's only a demo .. bid away!)

But this does show how a mashup that takes just a few hours to build might save a ton of headaches and help out a good cause!