Posts Tagged ‘yql’

The Hackday Toolbox – getting you started faster

Thursday, July 29th, 2010

Just having spent a lot of time at the amazing open hack day in Bangalore, India I found that most of the questions about starting a hack using Yahoo technology revolved around a few issues:

  • How do I access data on the web/from web services?
  • How do I use YQL from JavaScript or PHP?
  • How do I display information I received from YQL with PHP or JavaScript?
  • How do I get the location of the user and how do I analyse content for geographical locations?
  • How do I access oAuth authenticated information of Yahoo?
  • How do I set up PHP and where can I see errors?

So, to avoid having to repeat myself again I put together The Hackday Toolbox which contains sample code that deals with all these issues.

The Hackday Toolbox

The hackday toolbox contains:

  • An introduction to installing and using PHP with MAMP/XAMPP and debugging it
  • YQLGeo for all your geo and location needs
  • Demos of querying YQL in JavaScript, YUI3 and PHP
  • Demos to display YQL data
  • Authenticated example to access the Yahoo Firehose
  • Rendering Yahoo Geoplanet data as a map

You can download the Hackday Toolbox on GitHub or try the examples.

The toolbox is BSD licensed, so if you want to add Java/Ruby/Python/Heskell/Pascal/Logo/Fortran/6502 Assembly code examples, please do so.

I put my hack in a box…

Geo this! A Chrome plug-in to turn any web site into a map

Wednesday, July 21st, 2010

I’ve just started playing with Google Chrome Extensions heavily inspired by Mark Wubben’s talk at SWDC and of course the first thing I build is something to do with geographical locations.

Geo This! adds a little Earth icon to Chrome that, when pressed, analyses the page and shows a map of the locations it found in the text. If you highlight a part of the page first you only get the locations in this section. Click the screenshot below to go to the download site:

Geo This! A chrome plugin to turn any web site (or highlighted text) into a map

You can also see the extension in action in the following screencast:

Under the hood the extension is more or less a port of my addmap.js hack which in turn works most of its magic by using Yahoo Placemaker and YQL. The source code of the extension is available on GitHub.

Things that will come in future versions are:

  • Fixing the problem that Google Maps only numbers the first 10 markers on the static maps APIv1.0
  • More details on the found locationsv1.0
  • Ability to save the map and locations as an image
  • Better icons (anyone can paint me one?) – right now I am using one of the Project Icons by Mihaiciuc Bogdan and cruelly resized it in Skitch

What do you think?

Analyzing the FIFA2010 worldcup with Guardian Data and YQL

Wednesday, July 14th, 2010

Breaking news: The Guardian once again involved in committing a data awesome! As before, the UK newspaper graced developers with a really cool piece of information published on the web: all the World Cup 2010 statistics as an Excel Sheet.

Now, the easiest way to play with this data is to use YQL, so I simply took a copy of the information and shared it as a CSV document on Google Docs. That way I can use it in YQL:

select * from csv where url="http://spreadsheets.google.com/pub?
key=0AhphLklK1Ve4dEdrWC1YcjVKN0ZRbTlHQUhaWXBKdGc&single=true&gid=1&x=1&
output=csv" and columns="surname,team,position,time,shots,passes,tackles,saves"

You can Try this out in the console and see the results here.

Using YQL to filter and sort this, you can do some interesting searches on that information. For example:

What were the German mid field Players?

select * from csv where url="http://spreadsheets.google.com/pub?
key=0AhphLklK1Ve4dEdrWC1YcjVKN0ZRbTlHQUhaWXBKdGc&single=true&gid=1&x=1&
output=csv" and columns="surname,team,position,time,shots,passes,tackles,saves"
and team="Germany" and position="Midfielder"

You can Try this out in the console and see the results here.

Using sort() and reverse() you can do rankings. For example:

Who was the goalkeeper with the most saves?

(Neuner of Germany, Kingson of Ghana and Enyeama of Nigeria in case you wonder)

select * from csv where url="http://spreadsheets.google.com/pub?
key=0AhphLklK1Ve4dEdrWC1YcjVKN0ZRbTlHQUhaWXBKdGc&single=true&gid=1&x=1&
output=csv" and
columns="surname,team,position,time,shots,passes,tackles,saves" 
and position="Goalkeeper" | sort(field="saves") | reverse()

You can Try this out in the console and see the results here.

Which was the player that spent most time on the pitch?

select * from csv where url="http://spreadsheets.google.com/pub?
key=0AhphLklK1Ve4dEdrWC1YcjVKN0ZRbTlHQUhaWXBKdGc&single=true&gid=1&
output=csv" and
columns="surname,team,position,time,shots,passes,tackles,saves" 
| sort(field="time") | reverse()

You can Try this out in the console and see the results here.

Who were the players who were the least on the pitch in the German and Brazilian teams?

select * from csv where url="http://spreadsheets.google.com/pub?
key=0AhphLklK1Ve4dEdrWC1YcjVKN0ZRbTlHQUhaWXBKdGc&single=true&gid=1&x=1&
output=csv" and columns="surname,team,position,time,shots,passes,tackles,saves" 
and team in ("Germany","Brazil") | sort(field="time")

You can Try this out in the console and see the results here.

Using the CSV output and YQL you can do all kind of cool things with that data – as YQL also releases it as JSON it makes it easy to create interactive interfaces and visualizations, too – why don’t you have a go?

Decreasing the server load of Keywordfinder with YQL

Sunday, July 4th, 2010

Back in the days when “Yahoo BOSS“:http://developer.yahoo.com/search/boss/ came out, I built “Keywordfinder”:http://keywordfinder.org to show off the “keywords” functionality of BOSS. BOSS returns you keywords people entered for a certain web site in its results and I used that to list the most popular search terms for a term you enter.

Now, yesterday I got an alert by my host that Keywordfinder is exceeding my GPU limit and checking the logs I realized that it is heavily hit by users from South America. So I decided to add a JavaScript layer to the app that would take away some of the traffic and a lot of the computation from the server.

I could have just replicated the PHP functionality in JavaScript but I thought it is more beneficial for everybody to turn the functionality into a YQL table and use Yahoo’s servers to do the heavy lifting for me.

You can “see the keywordfinder open table”:http://github.com/yql/yql-tables/blob/master/keywordfinder/keywordfinder.xml on GitHub and it will be soon replicated over to datatables and show up in the YQL console. For now you can apply it using the “use” command:

use "http://github.com/yql/yql-tables/raw/master/keywordfinder/keywordfinder.xml" as keywordfinder;
 select * from keywordfinder where term="oil spill";

try it in the console or see the results as XML

The results right now are: oil, oil spill, spill, the Gulf of Mexico, BP, the Gulf, BP oil, Oil Spill, Deepwater, gallons, Gulf of Mexico, Gulf oil, Horizon oil, crude oil, Gulf, Gulf Coast, Gulf Oil, oil rig, birds, Mexico oil.

The table returns the search term, the region, the language and the terms. It sanitizes all of them for HTML display. By default, the table uses English as the language and USA as the region. If you provide a different language and region you get other results:

use "http://github.com/yql/yql-tables/raw/master/keywordfinder/keywordfinder.xml" as keywordfinder; 
select * from keywordfinder where term="fussball weltmeister" 
and language="de" and region="de";

try it out in the console or see the result as XML.

I think it is a nice example how you can speed up an app and help others by adding the functionality to YQL. Enjoy.

La Red Innova – Yahoo – opening the web for you

Tuesday, June 15th, 2010

I am currently in Madrid, Spain for the “La Red Innova”:http://www.laredinnova.com/ conference. My talk was a hybrid of explaining the social graph and “Yahoo Firehose”:http://developer.yahoo.com/social/updates/firehose.html and showing off some of the free technologies Yahoo has for publishers and bloggers.

I’ve recorded the audio of the talk and will publish it once I checked the quality (the stage was very noisy so I am not sure if it worked out). I’ll also upload the official Yahoo slide deck once the connection here is a bit more stable.

Here are my slides and links:

The slide deck

Yahoo – opening the web for you

Notes and links of the conference

Blogs I write for:

  • “http://wait-till-i.com”:http://wait-till-i.com
  • “http://ajaxian.com”:http://ajaxian.com
  • “http://smashingmagazine.com”:http://smashingmagazine.com
  • “http://thinkvitamin.com”:http://thinkvitamin.com
  • “Microsoft Script Junkie”:http://http://msdn.microsoft.com/en-us/scriptjunkie/default.aspx

Links:

Wait till I come! is the blog of Christian Heilmann , a developer evangelist living and working in London, England. Download vcard.

Feed me, Seymour: Entries (RSS) and Comments (RSS).