rss « Adrian Short

Posts Tagged: rss

Hillingdon Council creates an RSS feed for every page

Ever wanted to track the changes to a webpage but found there was no easy way of doing it? The Hillingdon Council website makes this easy by generating an RSS feed for every page.

At the bottom of each page there’s a “Subscribe to this page” link and feed icon. The site also makes these feeds easy to find by putting them in autodiscovery tags, providing a consistent way to subscribe in each browser.

Using RSS as a mechanism for receiving page status updates makes much more sense than writing a custom subscription system and requiring user registration such as on Sutton Council’s website.

There’s plenty of scope for Hillingdon to produce more comprehensive specific feeds for other uses but this is a very useful feature in its own right. Combined with a feed-to-email link on every page to a service like Feed My Inbox it could see a lot of usage.

Let’s also remember that there’s more to life than RSS. Other feed formats and APIs are more appropriate for different types of data such as iCalendar for events.

Top RSS tips for councils (and everyone else)

1. Validate your feeds

It only takes a moment to validate a feed. Invalid feeds can cause all kinds of unexpected weirdness in feed readers and other applications. Find any errors and fix them.

2. Use autodiscovery

People that use feeds a lot love autodiscovery. It provides a consistent way of finding and subscribing to feeds from any website. Put an autodiscovery <link> tag on your home page for every feed you’ve got and a tag on every interior page that’s got its own feed, eg. a tag for the news feed on the news page.

The standard tag format for RSS autodiscovery is:

<link rel="alternate" type="application/rss+xml" href="http://www.anytown.gov.uk/news.rss" title="Anytown Council News" />

and for Atom autodiscovery, use:

<link rel="alternate" type="application/atom+xml" href="http://www.anytown.gov.uk/news.atom" title="Anytown Council News" />

3. Use standard feed icons but only as direct subscription links

Whether you’re using RSS or Atom feeds, the standard feed icons are the common “beacon” style ones from Feed Icons. Get rid of the old orange rectangular “RSS” and “XML” text icons if you have them — they’re obsolete.

But don’t use the feed icons as illustrations. They should be clickable links directly to the feeds themselves so that people can subscribe. Avoid using them to link to RSS help pages or anything else.

Think: When I see this icon, I can click on it to subscribe.

4. Put your feed icons at the top of the related content

In-page feed subscription icons should be placed as near to the top of the related content as possible. Don’t bury the icon at the bottom of a list of news headlines, or even worse, in your page footer or a sidebar. Try to reinforce in your readers’ minds that the feed is an alternative way of viewing the same content.

5. Check your feeds in an RSS reader

This will often show up odd things in your feed that a validator won’t catch — like all your item dates being the same, for example.

6. Don’t lose your subscribers when your feed moves

If your CMS will let you, publish your feeds at permanent URLs. If it won’t, you’ll lose all your subscribers when you move to a new CMS and the URLs change.

If this happens, use an HTTP 301 redirect to tell your readers’ clients that the feed URL has changed permanently. Here’s how you do it.

Alternatively, proxy your feed through FeedBurner which will give it a permanent URL (and a few other toys to play with like usage stats, too).

7. Don’t offer the same feed in more than one format

People have got better things to do than to try to decide whether they want to read a feed in RSS 2.0, RSS 1.1 or Atom 1.0 format. Choose a single format for your whole site and stick with it. In practice, all formats work in all applications anyway. It gives you less to maintain and one less thing to worry about.

8. Shorten your item links if they’re longer than 255 characters

Some RSS reader applications use 255-character long fields to store item links. Some links are longer than that and will be truncated (and therefore, broken). If your CMS serves up item URLs longer than 255 characters try to run them through a URL shortening API like TinyURL first so they’ll always work.

9. Write a sensible feed title

The bit in your feed’s <title> element is what gets displayed as your feed’s title in RSS readers and other applications. Usually this should be your organisation’s name and a short indication of the content, eg:

Anytown Council News
Anytown Council – what’s on
Anytown Council Job Vacancies

Far, far too many feeds just have either the “Anytown Council” or the “news” bit, forcing users to rename those subscriptions in their readers if they’re going to make any sense at all.

Here are some bad examples from the Mash the State database:

Council Website Updates (but which council?)
Events (for who?)
Jobs Vacancies (sic)
Latest Online Consultations
Travel Information
www.anytown.gov.uk::Latest News (get rid of all that clutter)

10. Create an easy subscribe-by-email service

Many councils provide email alert services to keep their residents up to date. This is a lot of hassle, having to deal with bounces, unsubscribes and maintaining the whole mechanism. If you’ve got RSS feeds it only takes a moment to let people subscribe to them by email using third-party services like FeedMyInbox.

You can create a direct subscription link like this:

http://www.feedmyinbox.com/?feed=http://www.anytown.gov.uk/news.rss

This is how we do the black subscribe-by-email links on Mash the State’s council pages. You get a whole email service for nothing without having to pay a penny or do any more work at all. Just like with RSS in general.

Got any more good tips? Leave a comment and I’ll work them into the article here.

Enjoy.

How-To Guides / Local Government / Mash the State — 5 Comments
20
Apr 09

Comments not allowed at your council website? Here’s how to answer back

UPDATE 27 Feb 2010: The Boris Backchat blog mentioned in this post has served its demonstration purpose and has now been deleted.

A few people have raised the objection that what Mash the State is currently doing with council RSS feeds is really just helping councils to deliver their PR (or as those critics often like put it, “propaganda”).

In one sense, they’re right. A council’s press releases or “news” are just their own side of the story. You’d have to be pretty naive to think otherwise.

But getting any kind of information out into the open where it can be scrutinised, compared, cross-referenced and easily discussed is always an advantage. Here’s how to build a discussion blog around your local council’s news. Of course, if they don’t have an RSS feed this isn’t possible, which is why Mash the State exists in the first place.

Time required: Around 15 minutes.

Skills required: Just basic web use stuff. No programming or HTML. Anyone online should be able to do this.

Here’s one I made earlier: Boris Backchat. Got something to say to the London mayor? Just leave a comment.

Apologies to those outside London — I had to choose something!

Here’s how I did it:

1. I registered a new blog on WordPress.com. This is free and only took a moment.

2. I found the URL (web address) of Boris’s RSS feed. Visit your local council or other government website and hover your mouse over the RSS feed link or icon. Right-click and choose “Copy shortcut” (Internet Explorer) or “Copy link location” (Firefox) or whatever your browser gives you in the right-click menu.

In this particular case it was easier to grab the feed URL from the Greater London Authority Mash the State page.

3. I signed up at xFruits which has a whole set of free tools to do things with RSS feeds. This is free.

4. I used the “RSS to my blog” tool on xFruits which automatically copies the contents of an RSS feed into a blog, making a new blog post for each item in the feed.

First I typed a title for the new blog site and a few tags.

Then I pasted in the feed URL that I’d copied in step 2.

To configure this I also needed the URL of my new blog’s “API endpoint”. This is the address which other programs can use to push data into your blog.

The API endpoint URL for this blog is:

http://borisbackchat.wordpress.com/xmlrpc.php

The format is the same for all blogs on WordPress.com:

http://yourblogname.wordpress.com/xmlrpc.php

I also had to type in my WordPress username and password, and as I’ve got several blogs on WordPress, had to choose the right one to send the RSS feed to from the drop-down menu.

5. And that’s pretty much it. I went into the WordPress settings and set the time zone correctly and edited the site description. Now we’re ready to go.

To close the loop, if you want to keep up with the latest posts on Boris Backchat you can subscribe to both the new posts (articles) and comments in your RSS reader.

xFruits will work with most popular blog systems including Blogger, TypePad, Movable Type and WordPress hosted on your own server.

Welcome to open government. :)

Thanks to Jon Bounds on Twitter for tipping me off about xFruits. It’s a great set of tools. Jon has just set up a similar site for Birmingham City Council.

… and as I always like to say about these things, it’s taken longer to write about it than to do it!

Citizenship / Web design — Comments Off
14
Apr 09

Why I’m throwing down the gauntlet to our councils over RSS feeds

You’re free to republish this article under the Creative Commons Attribution 2.0 UK licence with credit and a link to Adrian Short / Mash the State

Today I connected 66 councils to their citizens by making it easy to subscribe to their news by email. It took me around ten minutes. I’d say this was a fairly good use of my time in terms of the ratio of effort to value produced, but I can’t claim to have done it single handed. What made it possible is that all 66 of these councils serve an RSS feed from their websites — and they’re the only ones in the country that do. Hooking those feeds up to FeedMyInbox through the council pages at Mash the State was a simple matter of dropping a single web link into a template and pushing it to the live site. Job done.

RSS is a simple way of getting data out of a website and into another program. The technology is ten years old and RSS feeds are ubiquitous on blogs, on mainstream news media websites and in Web 2.0 applications. The three leading web browsers — Internet Explorer, Firefox and Safari — all contain built-in RSS readers. Yet despite running websites costing tens of thousands of pounds annually each, only 15% of UK councils bother with RSS. Nothing could be more symbolic of large parts of government’s unwillingness to think beyond the confines of their own websites than making it practically impossible to receive basic local council information like news and events except by taking a trip to anytown.gov.uk to do it on the council’s own terms.

The ten minutes it took to emailify those 66 councils compare quite unfavourably with probably a similar number of hours I’ve spent trying to scrape Sutton Council’s news into a database, and from there through Delicious into RSS and Twitter. Writing screen scrapers — programs which extract text from web pages and turn them into structured, reusable data — is sometimes tricky but Sutton’s news is trickier than most. The news archive serves inconsistent page structures and even dynamically changing URLs to compete with. I vowed never to write another scraper, though as we’ll see, that’s a promise I soon had to break.

Screen scraping and copyright infringement are the dirty not-so-secrets of the civic hacking world. Show me a useful, innovative third-party civic website and I’ll most probably be able to show you the terms and conditions that were ignored and the data that was taken and repurposed without permission or legal licence. Similar behaviour is not unknown in the public sector itself, in some cases because government organisations are recycling that very same stolen data from third party applications into their own websites. The recent Rewired State National Hack the Government Day saw some incredibly inspiring, innovative and useful projects produced in very short order. How many of these projects didn’t involve citizens jailbreaking their own government to get the data they’ve paid for? What kind of society not only massively impedes but actually criminalises — in principle if not in practice — citizens devoting their own time, skills and money to write software to improve democracy and public services? Our society, it seems.

This has to stop. Hackers have shown their ability and willingness to surmount technical obstacles and run legal risks to get the data they need but less technical citizens simply cannot. No-one should have to. A rich, technologically-advanced and supposedly forward-thinking society such as ours should make citizens’ access to government data so commonplace that it doesn’t deserve comment. No technical wizardry required. No legal minefields to navigate. Just all the data served through common protocols with open licences that permit, well, anything. Then we can focus our time and energy on the considerably more interesting higher-order opportunities that come from actually using government data, not just getting hold of it.

Last week I launched Mash the State, a national campaign to get government data to the people. It’s not a new idea but our method is. We’ll be setting up a series of challenges to the public sector, asking one group of public bodies at a time to release one specific set of data. Our first challenge asks all local councils to serve up an RSS news feed by Christmas. I wouldn’t have bet good money in 2003 that by 2009 370 councils would still be without RSS, but here we are. I’ve thrown the gauntlet down and I’m pleased to see that a couple of hundred people have signed up to our website or followed us on Twitter to help make this happen. The councils have got over eight months to do what in most cases will not be more than half a day’s work to serve RSS from their websites. Others less fortunate will have to persuade their content management system suppliers to enable this feature for them. All have got plenty of time to perform this technically trivial task in time to give the public a small but highly symbolic Christmas present that shows that government in this country is prepared to trust its citizens with their own data.

As for my promise never to write another scraper, it didn’t last long. The very first task to build Mash the State was an hour spent writing a scraper to tease a list of councils from a government website. Join us and help to hasten the day when no-one will ever have to do anything like that again.

Web design — 1 Comment
30
Mar 09

DDAM: Don’t discriminate against machines

I’d like to emphasise a point that Emma Mulqueeny has alluded to in her seven principles for digital engagement and which I also made in passing in my previous article on building local news mashups.

The web is rife with discrimination of the most insidious and socially-destructive kind. It largely goes unnoticed as those that are well-served by the web care little for the plight of those that are not.

I’m talking about the web’s widespread discrimination against machines.

Conventional thinking about “websites” focuses almost wholly on human users. In the best-case scenario, people turn up to a website, find the information they want or do the thing they want to do, then go away again. If the website is useful and provides that information or service, and if it’s usable and accessible, people can do what they want to do with a minimum of fuss and effort and be satisfied.

Many websites are a long, long way from being able to provide a good experience for human users but I doubt many don’t have it as their goal, however ineptly they may deliver on the details.

By contrast, providing for machine users mostly happens either as an afterthought or not at all.

Machine users are other programs, websites or software systems that could interact with your website by extracting data, inputting data, or both.

The simplest example is an RSS feed fetching a list of news articles or job vacancies. By providing an RSS feed, a website makes it easy for other programs to capture that information and re-use it in any way possible. They might republish that information largely as-is, combine it with information from other sources in a mashup or even derive statistical information from it. Or all of these things.

More generally an API allows information to be read from or posted into another system. Can humans search your website? Then provide a search API for machines. Can people place orders and make payments online? Then provide an ordering and payment API. Every form on your website should have an API that makes provision for machine users to post in data and get a machine-readable response. Every sequential collection of information should be available as a feed.

Providing for machine users by building APIs and serving feeds is ultimately about serving human needs. Until machines achieve some kind of consciousness they will neither know nor care about accessing information on other systems. Every program is written by and provides information for people.

APIs make it possible for people to write programs to use your information and services in ways that suit them, in ways that you can’t anticipate, in ways that you don’t have the resources to provide, through systems that you won’t have to maintain. Many organisations won’t like this. It means the creation of infinite new layers of intermediaries, third-party services that provide new interfaces to your information and services or whole new applications that combine your services with others’. If you think you can provide the best interface to your services for every conceivable context you’re a fool. If you think you should be the sole gatekeeper of your services then prepare to lose your customers to other businesses or see your citizens disappear to third-party services implemented through any means necessary.

Let’s snap out of abstractions for a moment and look at an example.

As an activist with Living Streets I’m always on the lookout for maintenance issues and faults on the street, particularly where they affect pedestrians. I can report street faults directly to my local council or I can use FixMyStreet, a national service that ostensibly does the same thing.

I always use FixMyStreet. Why?

It’s easier. The user interface is better.
It’s geographically agnostic. I can report a fault anywhere in the country without having to know which authority is responsible for it.
It’s public. There’s a lot of value for me in being able to see other people’s street reports as I’m interested in looking at the wider issue of urban design and maintenance, not just getting a specific fault that bothers me fixed. I can browse other fault reports and see statistics for each borough.
It’s flexible. I can get reports sent to me by email or through an RSS feed. I can file a report on an iPhone and soon through many other mobile clients.

Overall it’s just better, and better in so many ways that my council and most other councils will not be able to emulate.

Smart councils would realise this and most probably abandon their local street fault reporting systems. They could put their resources into developing a clean API between their own faults database and FixMyStreet (or any other similar application). They could actually invest in FixMyStreet itself. It’s open source, so why not? It’s not going to disappear, and if it gets superseded by another, better, open-source system, no-one loses.

It’s APIs (and often, ersatz, hacky APIs) that make this kind of thing possible. It leads to better services, greater participation, and more flexibility, diversity. We need to put machine users on parity with human users so that people can be best served.

If you’re building any kind of website or online service, serve a feed for every stream and an API for every form. Let this be your mantra: DDAM, DDAM, DDAM. Don’t discriminate against machines.

Like this? Follow me on Twitter or subscribe to my RSS feed.

Software design / Sutton — 19 Comments
15
Mar 09

Building a local news mashup with Twitter, TwitterFeed, Delicious, Yahoo! Pipes, Ruby and RSS

(Click on the image to download the PDF, 19KB, opens in new window/tab.)

Like this? Follow me on Twitter: http://twitter.com/adrianshort

I’m a self-confessed and unashamed news junkie and this is how I’m starting to mash up news in my local area. For those that aren’t local, Sutton is a London borough with a population of approximately 180,000. Stonecot Hill is a neighbourhood within Sutton with a population of a few thousand.

Here’s how it all works.

Sources (green boxes)

I write Stonecot Hill News which is a local news blog running as a standalone WordPress installation on its own server. It produces an RSS 2.0 feed which here is treated as an outbound API.

Paul Burstow is the local member of parliament (constituency: Sutton & Cheam). Paul posts news regularly to his website and for many years that site has been serving an RSS 1.0 (RDF) feed. Whether he realises it or not, Paul laid one of the first foundations for news mashability in the borough.

The Sutton Guardian is the local newspaper, published by Newsquest. Together with its sister titles in other areas, they publish several dozen RSS 2.0 feeds for a wide variety of content.

Sutton Council is the local authority for the borough. Despite a recent £270,000 revamp to their website they haven’t yet managed to step into the Twenty-First and produce any RSS feeds. However, they do publish a variety of content regularly on their website, including their press releases.

APIs (grey boxes)

For the non-technical: API stands for Application Programming Interface, but that doesn’t tell you very much. Think of APIs like connectors or adapters that allow one program to plug into another in the same way that our household appliances can all connect to the electrical network because they share common plugs and sockets.

An API may be inbound (allowing data to be put into an application), outbound (allowing data to be extracted) or both.

As we can see in the diagram, applications which use APIs can be daisy-chained together, with the output of one application being fed into another.

RSS and Atom feeds are also APIs in that they provide a structured way for a program to get data out of an application. These feed formats are simple to implement (many applications produce them automatically) and are the first thing to consider when implementing a simple outbound API for an application.

Mashers (pink boxes)

Mashers are small programs that connect otherwise incompatible inbound and outbound APIs together. TwitterFeed is a simple example. Say you want to automatically post the new items from your blog to your Twitter account. Your blog serves an RSS feed but Twitter, while it has an inbound API, cannot accept RSS directly as input. TwitterFeed links the two, allowing the user to define any number of RSS feeds as inputs and any number of Twitter accounts as outputs, via the Twitter API. In this way, TwitterFeed plugs blogs into Twitter.

Yahoo! Pipes is a much more sophisticated and flexible masher. It can take inputs from a variety of sources (RSS, Atom, CSV, Flickr API, Google Base or even raw web pages), sort, filter and combine them in every conceivable way, and output the results as a single stream in various formats (RSS, JSON, and KML, the geo-format used by Google Earth). For my mashup I created this pipe to filter Paul Burstow’s, the Sutton Guardian’s and Sutton Council’s news and only pass through items containing the word “stonecot” to the stream that eventually ends in the @stonecothill Twitter feed, which is just for Stonecot Hill residents. The number of items coming through these sources about Stonecot Hill is very low, but when something appears residents will want to see it. (By way of example, only a single press release from Sutton Council in the last 227 concerns the Stonecot Hill area specifically.)

As mentioned above, Sutton Council doesn’t provide an RSS feed or any other kind of outbound API for its press release. I wrote a screen scraper in Ruby (using Hpricot) that grabs the press releases directly from the council website, dumps them into a MySQL database and pushes new items into the Delicious API. I’ve used Delicious here for two reasons. Firstly, because it generates an RSS feed automatically from all the items posted to it, so I can easily connect this output to other mashers and APIs further downstream without having to generate and host an RSS feed myself. Also, Delicious provides a useful search facility on its website allowing me to easily search just the press releases from Sutton Council. This isn’t possible with the council’s own website, where searches are scoped to the entire site.

Destinations (orange boxes)

In my diagram, the destinations are sites and services which represent new ways of consuming information coming from the original sources. Don’t want to read Sutton Council’s press releases on their own website? You can folllow them in Delicious or on Twitter. Want to keep up with the latest news about Stonecot Hill? Again, the @stonecothill Twitter account can find this for you from various sources. I also add my own items to @stonecothill, making it a unique mashup of original and syndicated content that’s highly targeted and very local.

The information stream doesn’t need to end with these destinations. Any destination that provides an outbound API can simply be another link in the chain to downstream services. In my diagram, the RSS feed from Delicious is used to do just that, pushing all its content on to the @suttonboro Twitter account, and just the Stonecot Hill-related content on to the @stonecothill account via the Yahoo! Pipes filter. Twitter has its own specific outbound API and also serves RSS feeds. There’s nothing to stop anyone else building on these destinations by combining and filtering them with other sources to produce their own unique, relevant information streams that they find useful.

What next?

If you run a website, it’s time to start thinking of mashability with the same degree of seriousness as you treat human visitors. Your website needs to serve up feeds and APIs so that other programs can connect to your content and deliver it to people in ways and contexts that they find useful. Some of these may have an audience of thousands or even millions. Others may have an audience of one. Regardless, by providing an API to your content you enable others to build things that you haven’t imagined, don’t have the resources or desire to build yourself, and won’t have to maintain. Businesses like newspapers that survive by selling their content (or selling advertising around their content) are thinking very carefully about the challenges and opportunities for the future of their industries. For government and voluntary organisations, it’s time to start thinking more like evangelists than economists. Spread the word like the free Bibles in hotel bedrooms and take every opportunity to get your message out there.

Sutton Council have been encouraged in various ways to implement feeds on their own website and the song will remain the same until they do. I don’t want to maintain my scraper for ever and I certainly don’t want to build any more of them.

The whole API and mashability agenda is far bigger than simple web feed formats like RSS and Atom. It’s time for technologists to stop flogging the line that “RSS is an easy way for people who follow lots of websites to read all their news in one place”. Direct human consumption of RSS feeds is never going to hit the mainstream in that way. If you’re reading this, you’re far more likely that average to use an RSS reader. (I’ve got 86 feeds in my Google Reader right now). The average web user has barely heard of the concept and most definitely don’t do it. I suspect they never will. But it’s likely they’re already benefiting from syndicated content through sites and applications that they use. If they never have to see or care about the underlying technology that’s really no more a problem than worrying that the average web user doesn’t understand HTTP or DNS. It’s just plumbing that can stay out of sight and out of mind as long as it works.

For the minority that do use personal RSS readers, I’d like to see more of them with built-in filtering features. Setting a simple keyword filter on a feed makes RSS reading considerably more powerful.

For those serving up feeds, I’d like to see Atom more widely used. Without wanting to open a can of Wineresque worms, RSS 2.0 fudges a number of important issues around content semantics and provides no support whatsoever for correctly attributing items in feeds mashed from several sources. Atom was designed to solve these problems and it does. Let’s use it.

Lastly, mashability is about every conceivable kind of content and content type. It’s not just about news and text. Every stream of information should have its own machine-readable feed. Every system that can accept data from human input should implement an inbound API to do likewise. To take one example, FixMyStreet is a website for people to report street faults to local authorities and currently takes around 1000 reports a week. It even has its own iPhone application so people can report faults complete with GPS locations and photos directly from the street. Only a single local authority in over 400 has implemented an inbound API to receive these reports. The rest get them by email, which must be manually copied into their own databases with all the effort, expense, possibility for error and opportunity costs that represents. Third-parties building extensions to other people’s systems is no longer unusual, so organisations need to embrace the possibilities rather than fighting against it or standing around looking bemused.

It’s time to open the doors and windows and get the web joined up, mashed up and moving.

Sutton / Web design — 5 Comments
10
Mar 09

Technosocial scenarios for Sutton: 3: Starting a chess club

Brian loves chess but finds it hard to get a decent match with an opponent at his level. He’d love to start a local chess club but doesn’t want to take the risk of setting something up and having too few people attend. He asks on a local chat forum whether anyone would like to start a club. He gets a couple of tentative offers and a suggestion that he posts a new pledge to Sutton Council’s own local version of PledgeBank.

PledgeBank is new to Brian but he soon creates a pledge, saying, “I will join a new local chess club and pay a membership fee of up to £20 a year but only if 15 other people from Sutton will too.” A month later, Brian has found 18 members for his new club. A few people found out about it through the chat forum (to which Brian posted a link to the pledge). Some more found it just by browsing Sutton’s PledgeBank. Items from the RSS feed of new pledges appear on the council’s website home page one week in every four, which brought in five new people. One person even had a filtered subscription to the new pledges feed for “chess”.

Brian easily finds a meeting room for his group using Sutton FreeSpace, which allows people to book halls and function rooms across the borough by finding free space from the various venues’ iCalendar feeds. But Brian doesn’t need to know anything about the technology, he just asks for a room for up to 25 people any weekday evening within his price range and he gets a few options nicely plotted on a map for him with availability and pricing. He books a room at a local community centre and as the event is a public meeting, an entry is automatically created in the centre’s public calendar feed which is then syndicated to the Sutton Guardian (where it appears in print as well as on their website), Upcoming, the council’s main borough calendar and a couple of local blogs. The first meeting of the new chess club is a great success. A year later, Brian is back on Sutton FreeSpace looking for a bigger venue for their club nights.

Adrian ShortDesign, citizenship and the city