August 4, 2009

Reuse Yahoo! Pipes with parameters

Yahoo! Pipes are useful to get and manipulate data from the web. We can provide user defined values to a pipe to configure modules in the pipe. The user input adds flexibility to a pipe at runtime. The user value can be passed to the pipe URL (really useful) or asked when we use the Run Pipe link.

We create a new pipe where we want to do a Twitter search and convert the results to a RSS feed. The parameters for the Twitter search are provided by the user and can change. The User inputs section contains several input modules. We use the Text Input module to get the search query. This module has 5 fields. The Name field is important now, because that is the name of the parameter we can use in the pipe URL. We can even provide a default value if the input is not given with the Default field. The Debug field is for testing the pipe: the value we fill in is used in the Pipe editor and the Debugger. The Prompt and Position fields are used when we use the Run Pipe link, but we don't use them for our example. We use search for the Name field and type Yahoo Pipes as a Debug value.

Next we use this input module to build a correct Twitter search string. We drop the URL Builder builder module in the editor. The Base URL is, the Path element is search.json and we have one query parameter named q. The value of the q parameter is the value of our input module. So we connect the input module to the grey dot to the right of the value field of the q parameter. When the pipe is executed the value of the q parameter will be the value of the pipe's input module. We can check the value of the complete Twitter search URL in the Debugger window:

We have our Twitter search URL and pass it on to a Fetch Data module. The results of the Twitter search are in JSON format. To convert these into a RSS feed we use the Create RSS module from the Sources section. We simply map fields from JSON to RSS fields.

Here is the complete pipe:

To search for "Yahoo Pipes" we invoke the following URL: But we can also use this pipe to search for "Grails":, or we search for Java: