Sutton « Adrian Short

Sutton

Building a local news mashup with Twitter, TwitterFeed, Delicious, Yahoo! Pipes, Ruby and RSS

(Click on the image to download the PDF, 19KB, opens in new window/tab.)

Like this? Follow me on Twitter: http://twitter.com/adrianshort

I’m a self-confessed and unashamed news junkie and this is how I’m starting to mash up news in my local area. For those that aren’t local, Sutton is a London borough with a population of approximately 180,000. Stonecot Hill is a neighbourhood within Sutton with a population of a few thousand.

Here’s how it all works.

Sources (green boxes)

I write Stonecot Hill News which is a local news blog running as a standalone WordPress installation on its own server. It produces an RSS 2.0 feed which here is treated as an outbound API.

Paul Burstow is the local member of parliament (constituency: Sutton & Cheam). Paul posts news regularly to his website and for many years that site has been serving an RSS 1.0 (RDF) feed. Whether he realises it or not, Paul laid one of the first foundations for news mashability in the borough.

The Sutton Guardian is the local newspaper, published by Newsquest. Together with its sister titles in other areas, they publish several dozen RSS 2.0 feeds for a wide variety of content.

Sutton Council is the local authority for the borough. Despite a recent £270,000 revamp to their website they haven’t yet managed to step into the Twenty-First and produce any RSS feeds. However, they do publish a variety of content regularly on their website, including their press releases.

APIs (grey boxes)

For the non-technical: API stands for Application Programming Interface, but that doesn’t tell you very much. Think of APIs like connectors or adapters that allow one program to plug into another in the same way that our household appliances can all connect to the electrical network because they share common plugs and sockets.

An API may be inbound (allowing data to be put into an application), outbound (allowing data to be extracted) or both.

As we can see in the diagram, applications which use APIs can be daisy-chained together, with the output of one application being fed into another.

RSS and Atom feeds are also APIs in that they provide a structured way for a program to get data out of an application. These feed formats are simple to implement (many applications produce them automatically) and are the first thing to consider when implementing a simple outbound API for an application.

Mashers (pink boxes)

Mashers are small programs that connect otherwise incompatible inbound and outbound APIs together. TwitterFeed is a simple example. Say you want to automatically post the new items from your blog to your Twitter account. Your blog serves an RSS feed but Twitter, while it has an inbound API, cannot accept RSS directly as input. TwitterFeed links the two, allowing the user to define any number of RSS feeds as inputs and any number of Twitter accounts as outputs, via the Twitter API. In this way, TwitterFeed plugs blogs into Twitter.

Yahoo! Pipes is a much more sophisticated and flexible masher. It can take inputs from a variety of sources (RSS, Atom, CSV, Flickr API, Google Base or even raw web pages), sort, filter and combine them in every conceivable way, and output the results as a single stream in various formats (RSS, JSON, and KML, the geo-format used by Google Earth). For my mashup I created this pipe to filter Paul Burstow’s, the Sutton Guardian’s and Sutton Council’s news and only pass through items containing the word “stonecot” to the stream that eventually ends in the @stonecothill Twitter feed, which is just for Stonecot Hill residents. The number of items coming through these sources about Stonecot Hill is very low, but when something appears residents will want to see it. (By way of example, only a single press release from Sutton Council in the last 227 concerns the Stonecot Hill area specifically.)

As mentioned above, Sutton Council doesn’t provide an RSS feed or any other kind of outbound API for its press release. I wrote a screen scraper in Ruby (using Hpricot) that grabs the press releases directly from the council website, dumps them into a MySQL database and pushes new items into the Delicious API. I’ve used Delicious here for two reasons. Firstly, because it generates an RSS feed automatically from all the items posted to it, so I can easily connect this output to other mashers and APIs further downstream without having to generate and host an RSS feed myself. Also, Delicious provides a useful search facility on its website allowing me to easily search just the press releases from Sutton Council. This isn’t possible with the council’s own website, where searches are scoped to the entire site.

Destinations (orange boxes)

In my diagram, the destinations are sites and services which represent new ways of consuming information coming from the original sources. Don’t want to read Sutton Council’s press releases on their own website? You can folllow them in Delicious or on Twitter. Want to keep up with the latest news about Stonecot Hill? Again, the @stonecothill Twitter account can find this for you from various sources. I also add my own items to @stonecothill, making it a unique mashup of original and syndicated content that’s highly targeted and very local.

The information stream doesn’t need to end with these destinations. Any destination that provides an outbound API can simply be another link in the chain to downstream services. In my diagram, the RSS feed from Delicious is used to do just that, pushing all its content on to the @suttonboro Twitter account, and just the Stonecot Hill-related content on to the @stonecothill account via the Yahoo! Pipes filter. Twitter has its own specific outbound API and also serves RSS feeds. There’s nothing to stop anyone else building on these destinations by combining and filtering them with other sources to produce their own unique, relevant information streams that they find useful.

What next?

If you run a website, it’s time to start thinking of mashability with the same degree of seriousness as you treat human visitors. Your website needs to serve up feeds and APIs so that other programs can connect to your content and deliver it to people in ways and contexts that they find useful. Some of these may have an audience of thousands or even millions. Others may have an audience of one. Regardless, by providing an API to your content you enable others to build things that you haven’t imagined, don’t have the resources or desire to build yourself, and won’t have to maintain. Businesses like newspapers that survive by selling their content (or selling advertising around their content) are thinking very carefully about the challenges and opportunities for the future of their industries. For government and voluntary organisations, it’s time to start thinking more like evangelists than economists. Spread the word like the free Bibles in hotel bedrooms and take every opportunity to get your message out there.

Sutton Council have been encouraged in various ways to implement feeds on their own website and the song will remain the same until they do. I don’t want to maintain my scraper for ever and I certainly don’t want to build any more of them.

The whole API and mashability agenda is far bigger than simple web feed formats like RSS and Atom. It’s time for technologists to stop flogging the line that “RSS is an easy way for people who follow lots of websites to read all their news in one place”. Direct human consumption of RSS feeds is never going to hit the mainstream in that way. If you’re reading this, you’re far more likely that average to use an RSS reader. (I’ve got 86 feeds in my Google Reader right now). The average web user has barely heard of the concept and most definitely don’t do it. I suspect they never will. But it’s likely they’re already benefiting from syndicated content through sites and applications that they use. If they never have to see or care about the underlying technology that’s really no more a problem than worrying that the average web user doesn’t understand HTTP or DNS. It’s just plumbing that can stay out of sight and out of mind as long as it works.

For the minority that do use personal RSS readers, I’d like to see more of them with built-in filtering features. Setting a simple keyword filter on a feed makes RSS reading considerably more powerful.

For those serving up feeds, I’d like to see Atom more widely used. Without wanting to open a can of Wineresque worms, RSS 2.0 fudges a number of important issues around content semantics and provides no support whatsoever for correctly attributing items in feeds mashed from several sources. Atom was designed to solve these problems and it does. Let’s use it.

Lastly, mashability is about every conceivable kind of content and content type. It’s not just about news and text. Every stream of information should have its own machine-readable feed. Every system that can accept data from human input should implement an inbound API to do likewise. To take one example, FixMyStreet is a website for people to report street faults to local authorities and currently takes around 1000 reports a week. It even has its own iPhone application so people can report faults complete with GPS locations and photos directly from the street. Only a single local authority in over 400 has implemented an inbound API to receive these reports. The rest get them by email, which must be manually copied into their own databases with all the effort, expense, possibility for error and opportunity costs that represents. Third-parties building extensions to other people’s systems is no longer unusual, so organisations need to embrace the possibilities rather than fighting against it or standing around looking bemused.

It’s time to open the doors and windows and get the web joined up, mashed up and moving.

Technosocial scenarios for Sutton: 3: Starting a chess club

Brian loves chess but finds it hard to get a decent match with an opponent at his level. He’d love to start a local chess club but doesn’t want to take the risk of setting something up and having too few people attend. He asks on a local chat forum whether anyone would like to start a club. He gets a couple of tentative offers and a suggestion that he posts a new pledge to Sutton Council’s own local version of PledgeBank.

PledgeBank is new to Brian but he soon creates a pledge, saying, “I will join a new local chess club and pay a membership fee of up to £20 a year but only if 15 other people from Sutton will too.” A month later, Brian has found 18 members for his new club. A few people found out about it through the chat forum (to which Brian posted a link to the pledge). Some more found it just by browsing Sutton’s PledgeBank. Items from the RSS feed of new pledges appear on the council’s website home page one week in every four, which brought in five new people. One person even had a filtered subscription to the new pledges feed for “chess”.

Brian easily finds a meeting room for his group using Sutton FreeSpace, which allows people to book halls and function rooms across the borough by finding free space from the various venues’ iCalendar feeds. But Brian doesn’t need to know anything about the technology, he just asks for a room for up to 25 people any weekday evening within his price range and he gets a few options nicely plotted on a map for him with availability and pricing. He books a room at a local community centre and as the event is a public meeting, an entry is automatically created in the centre’s public calendar feed which is then syndicated to the Sutton Guardian (where it appears in print as well as on their website), Upcoming, the council’s main borough calendar and a couple of local blogs. The first meeting of the new chess club is a great success. A year later, Brian is back on Sutton FreeSpace looking for a bigger venue for their club nights.

Sutton / Web design — 2 Comments
10
Mar 09

Technosocial scenarios for Sutton: 2: Street faults

Early one Sunday morning, Mark walks to the newsagent for his paper. He almost literally trips over a metal street bollard that has been uprooted from somewhere and dumped a couple of doors down from his house. He scans the bollard with the RFID reader in his phone and learns that the problem has already been reported to the council. Mark rolls the bollard to the side of the pavement and with a tap adds this street fault’s progress RSS feed to his phone’s reader so he can make sure it gets fixed soon. The problem was reported by a neighbour of Mark’s who has waived his privacy and allowed his name to appear on the fault report. Later that day, Mark meets his neighbour and they discuss various incidents of vandalism which they think might be related to drinkers from a local pub. The neighbour invites Mark to a Neighbourhood Watch meeting and sends the event details to his phone, which Mark then forwards to another couple of neighbours. Two weeks later the police representative at the Neighbourhood Watch meeting agrees to increase pro-active patrols around the pub area at weekends. They’re also pleased to see (with a couple of taps on a phone) that the bollard has been cemented back in place. With another tap, Mark rates this transaction as “positive” on the council’s feedback system.

Sutton / Uncategorized — 2 Comments
10
Mar 09

Technosocial scenarios for Sutton: 1: The library

Susan is a bookworm and regular library user. She filters the RSS feed of new acquisitions at her local library for the names of authors which she likes and reads it on her phone. One morning a new book by one of her favourite authors appears on the list. She reserves it with a single tap. At lunchtime she walks to the library and picks the book off the shelf. Susan scans the book’s RFID tag with her phone and with another tap she checks it out. (Anyone can check out or renew any item with an RFID-enabled phone but they must use the library’s own scanners to check things back in.) Her phone also shows two local events: the first for the library’s book club and the second for a reading by that author at a nearby bookshop in two months’ time. She adds the book reading to her calendar with a single tap. Two months later, Susan’s openly-licenced, tagged and geotagged photos of the author that she takes at the book signing appear automatically within minutes on the book club’s website, with a credit to her, a link back to her own profile page on the photo sharing website and a link to the author’s page on the local library’s website. Nearly all his books are out.

Sutton — Comments Off
28
Oct 08

Introducing Sutton Chat, a new forum for the borough

Today I’m launching Sutton Chat, a new discussion forum for the borough. If you’re local, I hope you’ll take a look and join in. Don’t be put off by the silence — we’ve only just started.

It’s good to see that the amount of online discussion in the borough is increasing in quantity and to a degree, in quality. Several local bloggers have built up a good readership, produced great material and have hosted some worthwhile discussions. Bloggers: I’m right behind you. May you go from strength to strength.

What’s missing is a good neutral forum where anyone can raise a topic for discussion. Running a blog takes a fair bit of commitment. Sutton Chat aims to fill a gap and encourage more people to get involved with local issues online and hopefully in real life, too.

While Sutton Chat is politically and commercially independent, it’s designed intentionally to reflect my own ideas of how a site like this should work. Members must register using their real names. My aim here is to encourage people to be accountable for what they say and to enable people that know each other in real life to recognise each other and carry those relationships forward on the forums.

As an advocate of simple design, I’ve done my utmost to ensure that the site is clear and straightforward to use. Most of the usual cruft found on online forums like signatures, post counts and smileys is absent. The idea is to allow members to concentrate on pure discussion, making it both easier to read and to write.

Forums don’t run themselves. If you’re keen to debate the hot local issues and get to know more people in the area I hope to see you online soon at Sutton Chat.

Citizenship / Sutton — 2 Comments
14
Oct 08

The real problem with “Your High”

Carshalton and Wallington MP Tom Brake has been leading a tenacious and spirited campaign against Your High, a “head shop” in Wallington which sells pipes, bongs, rolling papers and all the various paraphernalia used to consume cannabis. Mr Brake is unhappy that there are currently no legal restrictions on selling such materials and has been working hard to encourage the relevant authorities to make some. His campaign has the support of many local residents. According to Mr Brake’s own account, over 100 people signed a petition to put a stop this sort of thing on just one Saturday morning while others were encouraged by local nominal liberals to gather to protest outside this entirely legal private business.

Cannabis is pretty unpleasant stuff. Some people it makes sad, others dull, and we are told that still others are made quite mad by it. I can’t recommend it. The law against cannabis itself, while probably unnecessary and ineffective, is at least no more ridiculous than many of the other arbitrary limitations on personal freedom which we endure.

Yet to seek a ban on the sale of “cannabis paraphernalia” is a whole different kettle of fish. I employ quotation marks here because the devices and supplies to which I refer are neither necessary for consuming cannabis nor exclusively useful for doing so. The most common adjuncts to cannabis consumption (so I’m told) are tobacco and rolling papers, both of which are freely and mostly uncontroversially sold by every respectable newsagent and supermarket in the country. Tesco supply more dope fiends in an hour than Your High’s proprietor could ever hope to in a year in his most brain-addled and resinous pipe dream. Yet never do we hear of campaigns against Tesco’s lucrative dope-enabling business or enjoy the opportunity to sign righteous petitions against it.

Petitions are curious things. While they are a useful tool for campaigners (and having organised a few of my own) I’m certain that the things themselves have absolutely no meaning. It would be a straightforward matter to gain a substantial number of signatures in Wallington High Street of a morning demanding Mr Brake’s immediate removal from office, or conversely pledges of support for his continuation. A petition to call for people to be allowed to smoke cannabis in the privacy of their own homes and be left entirely alone could easily attract significant support in a short period of time, there being roughly as many Brits who believe other people’s business is their own as those that don’t. The numbers count for little either way.

So it’s pretty clear that the real problem with Your High isn’t that it helps people to smoke cannabis. In as much as it may do, it’s a mere amateur against the real pros in the big retailers and tobacco companies.

Perhaps the problem is that it encourages people to smoke cannabis that might otherwise not have contemplated the matter. Is this small shop with its jaunty green leaf insignia a siren call to the suburban stoner lifestyle? Might the doughty and otherwise blameless citizens of Wallington on passing its portals one time too many be tipped into packing in their sober habits for something a little more turned on, tuned in and dropped out? Are children particularly susceptible to its dubious countercultural charms?

Sadly, neither you nor I nor Mr Brake have any idea. Perhaps a survey would illuminate matters.

Q. Does this shop which we’re standing outside, “Your High”, make you:

Much more likely to use cannabis?
Somewhat more likely to use cannabis?
Neither more nor less likely to use cannabis?
Somewhat less likely to use cannabis?
Much less likely to use cannabis?

If the full weight of the law is to be brought upon people running “head shops”, it seems only proper to attempt to define and where possible quantify the harm they supposedly cause. Where children are suspected to be potential or actual victims, suitable similar research could be conducted in nearby schools, where apparently 95% of parents are in favour of action against Your High. If there is a case that these shops lure the vulnerable and unwary into the clutches of the pungent leaf, it has yet to be made.

The real problem with Your High isn’t that it helps people to smoke cannabis or encourages them to do so. The problem is that it’s tacky, tawdry, provocative and calculatedly effective in pricking suburban sensibilities. It’s out of place and in highly bad taste, like farting loudly and proudly in church. Similar shops in more urban areas operate without a hint of fuss, their neighbours being far too busy earning themselves a living while often dodging the more crazed and deadly effects of the sharper end of the illicit drugs trade. Your High raises hackles not because it has made Wallington a magnet for drug-fuelled anarchy (it hasn’t). Despite the worst fears and predictions, there hasn’t been an outbreak of reefer madness among the local youth. I doubt many local people are even particularly bothered that some of their neighbours smoke cannabis in a private and entirely innocuous way. They just don’t want it in their faces and would rather use the law to punish those who revolt them than endure such an unpleasant sight. The longer this shop remains, the weaker the case for outlawing it appears.

As all we hold dear looks increasingly precarious in the face of seemingly impending economic and environmental collapse, it’s a great pity that our political representatives have chosen to spend so much of their time railing against minor offences to the spirit rather than the major threats to our lives, limbs and property. If there’s one thing I find in particularly bad taste it’s laws against bad taste.

Citizenship / Sutton — 8 Comments
13
Oct 08

Sutton’s (oxy)moronic “voluntary smoking ban”

What a great day for liberalism and local democracy. The good burghers of Beddington and Wallington have enacted a “voluntary smoking ban” in the area’s public playgrounds. £3200 of honest, hard-working local taxpayers’ money has been allocated to the scheme which local councillors now hope to roll out across the rest of the borough.

This radical and presumably unique project has just one minor flaw: it has no legal basis whatsoever. Sutton’s LibDem councillors see no legal or linguistic impediment to the idea of a “voluntary ban” which in plain English would probably look more like a request. Deviant playground smokers flouting the voluntary ban run the risk of muted social disapproval. Serious and repeat offenders may find themselves being tutted at by council wardens.

Fortunately, the council has science on its side. The new “voluntary ban” is the permanent establishment of a pilot scheme that was enacted after Councillor Bruce Glithero complained that passive smoking left his daughter spluttering. His research findings can be read in more detail in the latest edition of the Journal of Anecdotal Evidence.

Sutton is to be congratulated for this bold experiment in local democracy. The beauty of a “voluntary ban” is that you can “ban” anything, just as long as you don’t ban anything. Everyone can have a go. It’s democratic, accessible and incredibly liberal.

I’ll be writing to my local committee to address the following issues with voluntary “bans”:

Men not wearing shirts in public. “Ban” them.
The word “whatever” used as a sentence substitute. “Ban” it instantly.
People wearing hoods when the weather is fine. A “ban” is the only solution.
Groups that cross the road in a haphazard and slovenly fashion. If a “ban” won’t make them cross brisky, simultaneously and perpendicularly to the carriageway, nothing will.
Rainy weekends followed by fine Monday mornings. Surely a “ban” would be a step forward?
Children sitting on steps and talking. “Ban” it immediately! Oh, they already did, for real.

What would you “ban” today?

–

Updated on 19 October with a photo by Cllr Terry Faulds of a sign in Beddington Park.

Adrian ShortDesign, citizenship and the city