If you're new here, you may want to subscribe to my RSS feed. So that you can read the latest updates about Web2.0 tools, Making Money Online, Tips in SEO, Ajax and many more. Thanks for visiting ProgramimiCOM!

In February 2007, Yahoo quietly went live with Yahoo Pipes. Tim O’Reilly described it as “a milestone in the history of the Internet.” It’s the ultimate web mashup tool; keep reading to find out what it can do for you.If you’re at all familiar with UNIX, you understand the concept behind Yahoo Pipes. A “pipe” in UNIX connects two programs, taking the output from one program and using it as the input of the next program. You can even connect a series of programs with a whole set of pipes, performing several different operations until you get the result you need.

For the sake of an example, assume that you are looking for an apartment close to a new job. You might create a program that visits web sites with classified listings, extracts the apartment listings, runs the addresses through a locator program to check the distance, discards all the ones over a certain price range, then sorts the remainder by some factor (say number of bedrooms or closeness to schools). That would be five steps run through four pipes.

Potentially, that would also be a lot of work if you’re actually writing a UNIX program to do this for you. Full disclosure: I’m not a programmer myself, so I wouldn’t know. But Yahoo Pipes lets users do exactly that, and more. It lets you combine many different data feeds (such as RSS) into a single feed. Once you’ve combined all the feeds, you can use “modules,” which are like mini-applications, to sort, filter, remix, and generally pick out whatever is important to you while leaving the rest.

I’m sure I don’t have to tell you that there are a ton of possibilities here. You can consolidate all of your favorite news feeds together into one feed and have it filter for specific words; you can put your entire online life together in one place (LiveJournal, Flickr, Facebook, what have you) to make it easier to tell your friends and family what you’ve been doing; you can even custom build an aggregated feed that covers your web site’s topic. Let’s take a closer look at how it all works.

Getting Started 

It’s easy enough to get started. You go to the Yahoo Pipes page and click on My Pipes. Before I take you there, let me show you a screen shot of the page:

It doesn’t really do the page justice, but I think you can see that it lends itself to poking around. When I checked the page, the left column (“About Pipes”) made a number of interesting suggestions as to what you can do with Yahoo Pipes; the middle column was devoted to popular pipes; and the right column talked about updates. One recent update I’m sure many of you will be interested in explained how to use Yahoo Maps with Yahoo Pipes.

If you’re ready to build a pipe, just click on My Pipes. You’ll need to log in to Yahoo; it should come as no surprise that you can’t use the service unless you’re registered. Once you log in, you reach a page that says “Looks like you don’t have any Pipes. Why not create one?” It links to an editing page for creating your first Pipe. You’ll also find links to example pipes. You should be warned that example pipes seem to take quite a while to load. I didn’t want to wait, so I thought I’d better just try putting one together myself.

Thankfully, Yahoo provides a simple overview so I had some idea of what I’d be doing, at least in principle. I’m not a programmer by any stretch of the imagination, so those of you who are more tech savvy can laugh at my halting steps. You can also find a complete list of Pipes modules and other useful information in Yahoo’s documentation for Yahoo Pipes.

Building the First Pipe 

Before we start building a pipe, you need to be aware that Yahoo Pipes does not work on IE 6. I found this out the hard way. You’ll see the page, but when you try to drag and drop modules onto the editing area, they’ll just disappear. It works in FireFox as long as you’re using the latest version, otherwise it acts flaky.

Anyway, here’s the first page you’re confronted with when you start building your first pipe:

I’ve cut and reduced the image to fit. Now the menu column on the left is what Yahoo refers to as the Library. It lists all of the available modules as well as any pipes you have saved. Yahoo separates the modules into different categories depending on their functions:

• Sources are data sources that return an RSS feed.
• User inputs are input fields that your Pipe’s users fill in at runtime.
• Operators are basic functions like foreach, sort, count, and filter.
• Url modules build and manipulate URLs.
• String modules handle strings.
• Date modules manipulate dates.

Fortunately you get a helpful tool tip when you hover over each module that tells you in a sentence what it does. You’ll usually want to start a pipe by specifying where it will be getting the data. That means you’ll want to grab one of the feed modules, which you find under “Sources,” and drag it onto the “canvas,” like so:

As you can see, the module changed; it’s bigger, it changed color, and it now has a “pipe output” section. It’s waiting for you to put in the URL of the RSS feed you want to use. Yahoo helpfully provides a search box for feeds. You can add more feeds using the same module by clicking the plus sign next to the URL. Below, I’ve added five feeds:

The feeds I’ve chosen aren’t particularly large, but there are plenty of feeds online that are nothing short of huge. Of course, if you don’t think you have enough feeds (and since the point of Yahoo Pipes is to connect lots of different feeds), you may want to drag another feed box onto your editing canvas and connect the two modules. You’ll want to use a “Union” box for this, which you’ll find under “Operators.” You click and drag your mouse from the URL modules to the Union module to hook them up. You can really see how pretty the interface is at this point; you’re dragging some lovely blue connectors. Take a look:

At this point you’ll want to filter your pipe. Drag a Filter box over from the Operators. The Filter box lets you block (or permit) items that match certain rules. Here’s an example, hooked up to my pipe:

You can see how easy it would be for someone who is interested in a particular kind of news or trying to stay up to date in a particular field to pick out the stories that would be of most interest. If you have ever searched for something using key words, you can easily set up a filter.

Once you have finished filtering your pipe, you want to connect the filter to the “pipe output” box that appeared when you dragged your first module onto the editing surface. You do that in the same way that you did it with the other connections.

Finishing Your Pipe 

At the upper right hand corner of your working area are three buttons: New, Save, and Publish. When you have finished your pipe, click Save. You’ll be prompted to name your pipe. Type a name into the text box and hit Save.

After it saves, you’ll see a box at the top of the screen that says “Pipe Saved” and you’ll be able to click on a link labeled “Run Pipe.” That’s always a good idea. Does the pipe grab too much information? Or perhaps it grabs too little (as mine did)? You can always go back and modify your pipe until it gives you what you’re looking for.

You can build much more complicated pipes than the example I gave here. Yahoo points to one called “Apartment Near Something” as a possible inspiration. It starts with a search on Craigslist for apartments as the input and uses the Location Input module to narrow things down by city/state and zip code. Then the pipe accepts text input from the user specifying what they want to find an apartment near (i.e. parks, walking trails, schools, etc). Finally a sort module is added, so the apartments can be sorted by their distance from the desirable item.

As you can see, it’s pretty easy for even a non-programmer to start using this lovely graphical interface. If there is interest, I’ll cover the other areas of the Yahoo Pipes site: the many pipes that are already published, how to publish a pipe so the whole world can see it, the discussion forums, how to clone a pipe, how to build a more complicated pipe, and more. Meanwhile, I think I’ve given you enough to play with for now!

—

by Terri Wells