Using YQL to load and convert RSS feeds really, really fast.

My esteemed colleague Stoyan Stefanov is currently running an advent calendar (blog post a day) on performance. Today I have a guest slot on his blog showing how you can use YQL to retrieve five RSS feeds much faster than with any other technology.

As stated at the end of the article, you could use a YQL open table with embedded JavaScript to move all of the hard conversion work to the YQL server, too.

This table does exactly that. The speed of the retrieval slows down a bit with this (as YQL needs to do another request to pull the table definition):

However, using this table to retrieve multiple feeds as HTML is dead easy:

$data = array(
  'http://code.flickr.com/blog/feed/rss/',
  'http://feeds.delicious.com/v2/rss/codepo8?count=15',
  'http://www.stevesouders.com/blog/feed/rss',
  'http://www.yqlblog.net/blog/feed/',
  'http://www.quirksmode.org/blog/index.xml'
);
$url ='http://query.yahooapis.com/v1/public/yql?q=';
$query = "use 'http://github.com/codepo8/yql-rss-speed-comparison/raw/master/rss.multi.list.xml' as m;select * from m where feeds=\"'".implode("','",$data)."'\" and html='true' and compact='true'";
$url.=urlencode($query).'&format=xml&diagnostics=false';
$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, $url); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
$content = curl_exec($ch); 
curl_close($ch);
$content = preg_replace('/.*<results><div/','<results><div',$content);
$content = preg_replace('/div><\/results>.*/','div>',$content);
echo $content;

To use the open table you simply need to give it the list of RSS feeds as the feeds parameter:

use 'http://github.com/codepo8/yql-rss-speed-comparison/raw/master/rss.multi.list.xml' as m;
select * from m where feeds="
  'http://code.flickr.com/blog/feed/rss/',
  'http://feeds.delicious.com/v2/rss/codepo8?count=15',
  'http://www.stevesouders.com/blog/feed/rss',
  'http://www.yqlblog.net/blog/feed/',
  'http://www.quirksmode.org/blog/index.xml'
" and html='true' and compact='true'

Try it out in the YQL console.

The html parameter defines if you want to get HTML back from the table. Take it out to get a list of feeds instead.

See the results as HTML (with an HTML parameter) or as feeds (without the HTML parameter).

The compact parameter defines if you want to get descriptions back for each entry or not.

See the results as HTML (without descriptions) or as HTML with descriptions.

By using the JSON-P-X output format (xml with callback) you could easily use this in JavaScript:

<div id="feeds"></div>
<script type="text/javascript" charset="utf-8">
function feeds(o){
  var f = document.getElementById('feeds');
  f.innerHTML = o.results;
}  
</script>
<script type="text/javascript" src="http://query.yahooapis.com/v1/public/yql?q=use%20%27http%3A%2F%2Fgithub.com%2Fcodepo8%2Fyql-rss-speed-comparison%2Fraw%2Fmaster%2Frss.multi.list.xml%27%20as%20m%3Bselect%20*%20from%20m%20where%20feeds%3D%22%27http%3A%2F%2Fcode.flickr.com%2Fblog%2Ffeed%2Frss%2F%27%2C%27http%3A%2F%2Ffeeds.delicious.com%2Fv2%2Frss%2Fcodepo8%3Fcount%3D15%27%2C%27http%3A%2F%2Fwww.stevesouders.com%2Fblog%2Ffeed%2Frss%27%2C%27http%3A%2F%2Fwww.yqlblog.net%2Fblog%2Ffeed%2F%27%2C%27http%3A%2F%2Fwww.quirksmode.org%2Fblog%2Findex.xml%27%22%20and%20html%3D%27true%27&format=xml&callback=feeds&diagnostics=false"></script>

If you want to compare yourself, get the source code of all the examples from GitHub.

Tags: conversion, curl, loading, multicurl, open tables, performance, rss, yql

This entry was posted on Tuesday, December 8th, 2009 at 10:45 am and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.