How the Web Bug Simulator works

Background 

Over the weekend we deployed a new feature for the Radio 8 community called the Web Bug Simulator for XML. This is a technical explanation of the feature. It's very important for XML geeks to know how this works, but not necessary for people who just want to use the feature to find new stuff to subscribe to.

In the HTML feeds we generate (otherwise known as weblogs), we insert a 1-pixel gif so we can track referers and hits and rank the sites in a Top 100 list. We applied the same idea to XML feeds that our users read through Radio's news aggregator.

How it works 

When Radio starts a scan, it creates an object database table to record the sources that had new items. For each source, it records the name of the source and the number of new items (this bit of info is not used in the rankings).

When the scan is complete it sends the results in an XML-RPC struct to the community server. For every feed that had at least one new item, it bumps the hit counter for the feed by one, and records the user's weblog in the referers table for the resource.

On the HTML readout of this tabulation, if you click on the name of the source, it takes you to the list of referers. The number next to each referer is the number of times their aggregator got something new from the resource.

It's a little tricky, so read the text carefully and stare at the readouts a bit. The result is a measure of how much new stuff was delivered to who by who.

The XML-RPC call 

Using SCNS notation, here's what the call looks like:

[server].weblogStats.ping ("rss", myWeblogTitle, myWeblogUrl, theStruct)

Where server is "xmlrpc://subhonker7.userland.com/RPC2".

myWeblogTitle is the title of the Radio user's weblog, and myWeblogUrl is its URL. This is used in the referer ranking table for each XML resource. Example.

Obviously all the juice is in theStruct, which you'll see in detail in the example request and response below.

The Radio 8 implementation 

Radio 8 users can see the workstation side of this interface in this table:

radio.aggregator.webBugSimulator.postScanCallback

We didn't use SCNS because we wanted it to work with versions of Radio that don't have it.

Request and response 

Here's an example of a call and a response.

POST /RPC2 HTTP/1.0
User-Agent: Radio UserLand/8.0.5 (WinNT)
Host: subhonker7.userland.com
Content-Type: text/xml
Content-length: 3395

<?xml version="1.0"?>
<methodCall>

<methodName>weblogStats.ping</methodName>

<params>

<param>

<value>rss</value>

</param>

<param>

<value>Dave's Handsome Radio Blog!</value>

</param>

<param>

<value>http://radio.weblogs.com/0001015/</value>

</param>

<param>

<value><struct>

<member>

<name>http://export.cnet.com/export/feeds/news/rss/1,11176,,00.xml</name>

<value>

<struct>

<member>

<name>ctnewstories</name>

<value><i4>2</i4></value>

</member>

<member>

<name>title</name>

<value>CNET News.com</value>

</member>

</struct>

</value>

</member>

<member>

<name>http://jrobb.userland.com/rss.xml</name>

<value>

<struct>

<member>

<name>ctnewstories</name>

<value><i4>1</i4></value>

</member>

<member>

<name>title</name>

<value>John Robb's Radio Weblog</value>

</member>

</struct>

</value>

</member>

<member>

<name>http://radio.weblogs.com/0001013/rss.xml</name>

<value>

<struct>

<member>

<name>ctnewstories</name>

<value><i4>1</i4></value>

</member>

<member>

<name>title</name>

<value>lawrence's notebook</value>

</member>

</struct>

</value>

</member>

<member>

<name>http://radio.weblogs.com/0001015/rss.xml</name>

<value>

<struct>

<member>

<name>ctnewstories</name>

<value><i4>1</i4></value>

</member>

<member>

<name>title</name>

<value>Dave's Handsome Radio Blog!</value>

</member>

</struct>

</value>

</member>

<member>

<name>http://scriptingnews.userland.com/xml/scriptingNews2.xml</name>

<value>

<struct>

<member>

<name>ctnewstories</name>

<value><i4>11</i4></value>

</member>

<member>

<name>title</name>

<value>Scripting News</value>

</member>

</struct>

</value>

</member>

<member>

<name>http://static.userland.com/tomalak/links2.xml</name>

<value>

<struct>

<member>

<name>ctnewstories</name>

<value><i4>1</i4></value>

</member>

<member>

<name>title</name>

<value>Tomalak's Realm</value>

</member>

</struct>

</value>

</member>

<member>

<name>http://www.redmonk.net/redmonk.rss</name>

<value>

<struct>

<member>

<name>ctnewstories</name>

<value><i4>2</i4></value>

</member>

<member>

<name>title</name>

<value>Redmonk News Feed</value>

</member>

</struct>

</value>

</member>

<member>

<name>http://www.theshiftedlibrarian.com/rss.xml</name>

<value>

<struct>

<member>

<name>ctnewstories</name>

<value><i4>1</i4></value>

</member>

<member>

<name>title</name>

<value>The Shifted Librarian</value>

</member>

</struct>

</value>

</member>

</struct></value>

</param>

</params>

</methodCall>

HTTP/1.1 200 OK
Connection: close
Content-Length: 149
Content-Type: text/xml
Date: Mon, 04 Mar 2002 18:39:03 GMT
Server: UserLand Frontier/8.0b46-WinNT

<?xml version="1.0"?>
<methodResponse>

<params>

<param>

<value><boolean>1</boolean></value>

</param>

</params>

</methodResponse>