I was asked today how to host datafeed images on your local server. Starting with the script given in Step 2 of the datafeed series, we’re just going to add some code after the MySQL query has been executed without errors: Read more
Linkshare Web ServicesBuilding a product site using the LinkShare Merchandiser Web Service is one of the easiest and most straight-forward tasks you can do. If you’re just getting started with datafeeds and product-based sites, you may want to start here.
If you don’t have an account, you first need to signup with LinkShare. Then login, and click on Web Services under Links. Next, generate or update your token. You’ll need this when making requests. Read more
I received an email last week asking how to add Commission Junction products to datafeed websites. To do this, take a look at the CJ Web Services. Particularly, the Product Catalog Search Service (REST).
What this does is allows you to specify a merchant’s ID, and get all of the product links for that merchant. In my example, I was grabbing the products from Mighty Leaf Tea.
So this PHP script will sit in your site’s root, like the other admin scripts, and when you pull it up, you’ll be presented with a form, asking for the Advertiser ID. That’s the AID when you’re pulling links, or advertiserId in the URL when you’re looking at the merchant in CJ. For Mighty Leaf Tea, it’s 2346375.
The biggest problem is that the CJ Web Service for Product Catalog Search doesn’t return the product ID! So I had to come up with a way to create an ID for a product. In this case, I based it on the SKU and ended up with:
If you don’t know what this is doing, don’t worry about it. If you’re bored, look-up those functions & let me know if you can think of a better way to make an ID from a sku. There has to be a simpler solution.
Anyway, because of this, I had to change the ProductID field to a bigint(20), since these item IDs were quite large (ex: 22229748315941). In addition, whenever the front-end scripts type-juggled the itemID into an (int), I had to remove that, as int’s are not big enough to hold this data (browse.php and item.php).
Getting back to the script, when I enter 2346375 in the form, then submit it, this script will query CJ for all product links for that merchant, insert or update them in the database, and finally remove any old products.
<?php
include('./vars.php');
include('./admin-password.php');
$bNavHome = true;
include('./header.php');
$cTitle = 'Get CJ Datafeed';
?>
<div class="post">
<h2 class="title"><?= $cTitle ?></h2>
<div class="entry">
<?php
if (!empty($_GET['advertiser-id'])) {
$cDevKey = 'youridhere';
$nWebSiteID = "yourwebsiteid";
$nMax = 1;
// We'll use this later to determine which products can be deleted
mysql_query("update products set bActiveProduct=0 where MerchantID=" . (int)$_GET['advertiser-id'] . "");
function addCJProduct($oCJProduct) {
list($nProductID, ) = explode('E', round(hexdec(md5($oCJProduct->{'sku'}))/1000000000000000000000000));
$nProductID = str_replace('.', '', $nProductID);
$nProductID = abs($nProductID);
if (mysql_num_rows(mysql_query("select * from products where ProductID=" . $nProductID . " and MerchantID=" . (int)$oCJProduct->{'advertiser-id'} . " limit 1")) == 0) {
// This is a new record
$cQuery = "insert into products (ProductID, Name, MerchantID, Merchant, Link, Thumbnail, BigImage, Price, RetailPrice, Description, Lastupdated, Manufacturer, PartNumber, ISBN, UPC, SKU, bActiveProduct) values (" . $nProductID . ", '" . myres($oCJProduct->{'name'}) . "', '" . myres($oCJProduct->{'advertiser-id'}) . "', '" . myres($oCJProduct->{'advertiser-name'}) . "', '" . myres($oCJProduct->{'buy-url'}) . "', '" . myres($oCJProduct->{'image-url'}) . "', '" . myres($oCJProduct->{'image-url'}) . "', '" . myres($oCJProduct->{'price'}) . "', '" . myres($oCJProduct->{'retail-price'}) . "', '" . myres($oCJProduct->{'description'}) . "', now(), '" . myres($oCJProduct->{'manufacturer-name'}) . "', '" . myres($oCJProduct->{'sku'}) . "', '" . myres($oCJProduct->{'isbn'}) . "', '" . myres($oCJProduct->{'upc'}) . "', '" . myres($oCJProduct->{'sku'}) . "', 1)";
} // ends <insert new record>
else {
// This is an existing record
$cQuery = "update products set Name='" . myres($oCJProduct->{'name'}) . "', MerchantID=" . (int)$oCJProduct->{'advertiser-id'} . ", Merchant='" . myres($oCJProduct->{'advertiser-name'}) . "', Link='" . myres($oCJProduct->{'buy-url'}) . "', Thumbnail='" . myres($oCJProduct->{'image-url'}) . "', BigImage='" . myres($oCJProduct->{'image-url'}) . "', Price='" . myres($oCJProduct->{'price'}) . "', RetailPrice='" . myres($oCJProduct->{'retail-price'}) . "', Description='" . myres($oCJProduct->{'description'}) . "', Lastupdated=now(), Manufacturer='" . myres($oCJProduct->{'manufacturer-name'}) . "', PartNumber='" . myres($oCJProduct->{'sku'}) . "', ISBN='" . myres($oCJProduct->{'isbn'}) . "', UPC='" . myres($oCJProduct->{'upc'}) . "', SKU='" . myres($oCJProduct->{'sku'}) . "', bActiveProduct=1 where ProductID=" . $nProductID . " and MerchantID=" . (int)$oCJProduct->{'advertiser-id'} . " limit 1";
} // ends else from <updating existing record>
mysql_query($cQuery);
if (mysql_error()) {
echo("<p><b>MySQL Error: " . mysql_error() . "<br />\n");
echo("$cQuery</p>");
echo("<pre>");
print_r($rsItem);
echo("</pre>");
exit();
} // ends if (mysql_error())
} // ends function addCJProduct($oCJProduct)
for ($nPageNumber = 1; $nPageNumber <= $nMax; $nPageNumber++) {
$cURL = 'https://product-search.api.cj.com/v2/product-search?';
$cURL .= 'advertiser-ids=' . $_GET['advertiser-id'] . '&';
$cURL .= 'records-per-page=1000&';
$cURL .= 'page-number=' . $nPageNumber . '&';
$cURL .= 'website-id=' . $nWebSiteID;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $cURL);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Authorization: ' . $cDevKey,
'User-Agent: "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.15) Gecko/2009101601 Firefox/3.0.15 GTB6 (.NET CLR 3.5.30729)"'
));
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$cHTML = curl_exec($ch);
if (curl_error($ch)) {
echo "Curl error: " . curl_error($ch);
} // ends if (curl_error($ch))
else {
$cXML = simplexml_load_string($cHTML);
// Update nMax
if ($cXML->products->attributes()->{'total-matched'} > 1000) {
$nMax = floor($cXML->products->attributes()->{'total-matched'}/$cXML->products->attributes()->{'records-returned'});
} // ends if ($cXML->products->attributes()->{'total-matched'} > 1000)
for ($i = 0; $i < count($cXML->products->product); $i++) {
addCJProduct($cXML->products->product[$i]);
} // ends
} // ends else from if (curl_error($ch))
} // ends for loop
mysql_query("delete from products where bActiveProduct=0 and MerchantID=" . (int)$_GET['advertiser-id'] . "");
echo("<p><strong>" . $cXML->products->attributes()->{'total-matched'} . "</strong> products have been loaded.</p>");
echo("<p>There are " . mysql_num_rows(mysql_query("select * from products where MerchantID=" . (int)$_GET['advertiser-id'] . "")) . " products.</p>");
} // ends if (!empty($_GET['advertiser-id']))
else {
// Show form
?>
<form method="get" action="<?= $_SERVER['PHP_SELF'] ?>">
CJ Advertiser ID: <input type="text" name="advertiser-id" /><br />
<input type="submit" name="cAction" value="Get Datafeed" />
</form>
<?php
} // ends else from if (!empty($_GET['advertiser-id']))
?>
</div>
</div><!-- ends class="post" -->
<p align="center">[ <a href="<?= $_SERVER['PHP_SELF'] ?>"><?= $cTitle ?> Home</a> | <a href="./admin.php" target="_top">Admin Home</a> ]</p>
<?php
include('./footer.php');
?>
Because CJ products do not include categories, I had to change my browse.php to search the product title for the category (green, black, white). To create a quality site, however, I recommend you put products in their appropriate category using the custom category script.
You’ll notice I output 2 numbers at the end – the first is how many products were returned, and the second is how many were loaded into the database. For Mighty Leaf Tea, it loaded 413 products, but only 410 were in the database when I was done. Either 3 products had duplicate SKUs, or 3 products ended up with duplicate ProductIDs after my process was done to generate a ProductID from SKU. I’m not sure, but 410/413 is pretty good, so I left it alone.
This script was really hacked together – let me know if you have questions / problems.
I received an email the other day from Garrett, and he asked:
I’m just curious how you might deal with pages/urls on your site that have been indexed (and may be getting good traffic/links), but all of a sudden they’ll disappear if the product happens to drop off the merchant’s feed.
After a couple emails back & forth, I think the best solution would be to find a similar product, and 301 redirect to that product. First, I’m going to change my .htaccess file to pass in the “name” of the item that was requested:
Then edit item.php, and after $rsData = mysql_fetch_array($oResult); (about line 6), add:
if (!$rsData) {
// We don't have this item anymore!
// Get the parts of the product name
// See http://www.ericnagel.com/2010/03/follow-up-on-datafeeds-podcast.html
$aNameParts = explode('-', $_GET['cName']);
// Look up similar products in the database
$cQuery = "select * from products where";
foreach ($aNameParts as $cNamePart) {
$cQuery .= " Name like '%" . mysql_real_escape_string($cNamePart) . "%' or ";
} // ends foreach ($aNameParts as $cNamePart)
$cQuery = ereg_replace(" or $", "",$cQuery ); // remove the last "or"
$oResult = mysql_query($cQuery);
$rsData = mysql_fetch_array($oResult);
if (!$rsData) {
// No clue
header("Location: /");
exit();
} // ends if (!$rsData)
else {
$p = 0;
$nItemID = $cName = '';
$cOrig = str_replace('-', ' ', $_GET['cName']);
do {
similar_text ($cOrig, $rsData['Name'], $fMatch);
if ($fMatch > $p) {
$nItemID = $rsData['ProductID'];
$cName = $rsData['Name'];
$p = $fMatch;
} // ends if ($fMatch > $p)
} while ($rsData = mysql_fetch_array($oResult));
if ($p > 0) {
// Best match
header("Location: /" . ereg_replace("[^A-Za-z0-9]", "-", $cName) . "-p" . (int)$nItemID . ".php", TRUE, 301);
exit();
} // ends if ($p > 0)
else {
// No good match
header("Location: /");
exit();
} // ends else from if ($p > 0)
} // ends else from
} // ends if (!$rsData)
By default, the redirect is “HTTP/1.1 302 Found”, so we add “, TRUE, 301″ to the header call to change it to a permanent redirect.
Of course, for all this to work, you have to be using .htaccess rewrites, otherwise there’s no way to get the name of the missing (and requested) product.
Thanks for the inspiration to this post, Garrett! Hope this answered your question for you!
Google just announced an image sitemap format which will help get images from your site indexed. I thought this was a perfect extension to the datafeed series.
The first step is to make Google think we have the images on our server. So inside an “images” folder, create “image.php” like so:
If you were slick, you’d add some error checking and caching to these scripts to make things go quicker.
The tea niche may not be ideal for someone searching for images (think shoes!) but this at least gives you an idea of how to take advantage of this new tool by Google.
I’m so happy to hear from affiliates who have followed in the datafeed series and created websites of their own! The other day, Jen Goode showed me a site she was working on, and brought up a common problem with datafeeds: how to categorize the products.
I was able to do this using simple like queries:
select * from products where MerchantSubcategory like '%green tea%' or MerchantCategory like '%green tea%' order by Name;
But sometimes this isn’t the case. Sometimes you have to (gasp!) manually categorize the products. To do this, start with some changes to the products table:
ALTER TABLE `products` ADD `MyCategory` VARCHAR( 10 ) NOT NULL ;
ALTER TABLE `products` ADD INDEX ( `MyCategory` ) ;
Then create an admin page for you to categorize the products. Use the admin-sas-datafeed.php as your template, but change the inside code to:
<?php
if (!empty($_POST['ProductID']) && !empty($_POST['MyCategory'])) {
mysql_query("update products set MyCategory='" . myres($_POST['MyCategory']) . "' where ProductID=" . (int)$_POST['ProductID'] . " limit 1");
echo("<p>The product has been updated.</p>");
} // ends
$cQuery = "select * from products where MyCategory='' limit 1";
$oResult = mysql_query($cQuery);
if ($rsData = mysql_fetch_array($oResult)) {
?>
<form name="productForm" method="post" action="<?= $_SERVER['PHP_SELF'] ?>">
<input type="hidden" name="ProductID" value="<?= (int)$rsData['ProductID'] ?>" />
<table width="500">
<tr>
<td><a href="/item.php?ProductID=<?= (int)$rsData['ProductID'] ?>" target="_blank"><?= $rsData['Name'] ?></a>: </td>
<td><select name="MyCategory" onChange="document.forms['productForm'].submit();">
<option value="">Choose</option>
<option value="green-tea">Green Tea</option>
<option value="white-tea">White Tea</option>
<option value="black-tea">Black Tea</option>
<option value="oolong-tea">Oolong Tea</option>
<option value="tea-pot">Tea Pot</option>
<option value="tea-cup">Tea Cup</option>
<option value="junk">Do Not Use</option>
</select></td>
</tr>
<tr>
<td></td>
<td><input type="submit" name="cAction" value="Submit" /></td>
</tr>
</table>
</form>
<?php
} // ends if ($rsData = mysql_fetch_array($oResult))
else {
echo("<p><strong>Congratulations</strong>! All products have been categorized!</p>");
} // ends else from if ($rsData = mysql_fetch_array($oResult))
?>
What this does is update the “MyCategory” field for each product. Then, your SQL query on the browse page is simply:
select * from products where MyCategory='green-tea' order by Name;
Don’t worry – when you update the datafeed (by downloading a new zip file and uploading it in the admin tool you built before), the MyCategory field isn’t touched!
Now you can categorize products however you’d like – enjoy!
In step 3, we’re going to actually display the data saved in the database! So far I’ve held your hand along the way, but now I’m just going to point you in the right direction, because there isn’t one way to do this. Merchants handle categories in their datafeeds differently, so you’re going to have to look at what the merchants you’ve chosen to work with are doing, and adjust accordingly.
Looking at my tea example, I wanted pages for Green Tea, White Tea, Black Tea and Oolong Tea. So (in theory, I don’t do it this way) we could make the Green Tea page /browse.php?cCategory=green+tea, and then pull in all the green teas. How do we do that? This is where you have to think.
What I’m doing is the following:
select * from products where MerchantSubcategory like '%green tea%' or MerchantCategory like '%green tea%' order by Name;
That works nicely for my dataset. However, if you were using the Baghaus datafeed, your query may be
select * from products where Name like '%Tylie Malibu%' order by Price desc;
Once you have your data, loop through it & display the products on the page, with a link to the product details page. You can use a list, or a table… whatever you’d like. In the end, you’ll be linking to individual products, like: /item.php?ProductID=466820102
And on item.php, you’ll do a quick lookup and display the product:
$cQuery = "select * from products where ProductID=" . (int)$_GET['ProductID'] . " limit 1";
$oResult = mysql_query($cQuery);
$rsData = mysql_fetch_array($oResult);
If you look at my tea site, you’ll notice I’m not using links like those in the blog post. Instead, I’m using .htaccess to rewrite the URLs, to make them “pretty”. You can learn about this in my Follow-up on Datafeeds Podcast blog post.
Finally, one problem you’ll be sure to run into is that merchants often have broken images in their datafeeds. This sucks, but there isn’t much you can do about it.
At first, I had a PHP script that made sure the image existed, and if so displayed it and if not, displayed another one. But, that slowed down the page display considerably. So instead, I use a bit of Javascript to replace broken images with a pre-defined image:
Warren Buffett once commented, “I want to give my kids just enough so that they would feel that they could do anything, but not so much that they would feel like doing nothing”.
In this datafeed series, I want to think the same thing. If I give you a finished site, you haven’t learned anything. However, if I give you the framework, and set you in the right direction, the finished product will be something you developed, not something you copied.
If you’ve followed from Step 1, and I know some of you have, you should be well on your way to a finished site. But if you’re waiting for a template to work off of, here it is:
The first step, if you haven’t done so already, is to download the datafeed from the ShareASale website. While many merchants offer FTP access, and we could automate this a bit more, it’s easier when getting started to upload the data manually (yeah, uploading a zip of the datafeed is “manual” to me).
You’ll need to create a “temp” folder, chmod it to 777 so the script can write to it, and maybe change where your unzip program sits on your server (line 17)
The script to load the products table isn’t terribly long, but I’m not going to paste the whole thing in this blog post. So…
Then, upload the file to your website, go to yourdomain.com/admin-sas-datafeed.php and click on the “Choose File” button, then select the zip file from your computer, and click “Upload”.
What this does is unzips the .zip file, reads the .txt file and if there’s a new product, adds it to the database. If the product already exists, it gets updated. Finally, any product that wasn’t just added or updated, gets removed.
Repeat as necessary for each merchant that you’re working with.
This step isn’t very long, or difficult (considering I give you everything you need!) but it’s exciting as now you have a database full of products, ready to display on your website!
Next up… we’ll work on the display of the products.
The next step in building a datafeed website is getting the data into the database, but to do this, we need to build an administrative section to your website, and add some helper functions & files.
vars.php was a file seen in Step 1, which I use to store the mysql connection and other variables. Here’s a simplified version of the one I’m using:
<?php
// MySQL
$DB_NAME = 'your_db_name';
$DB_USER = 'your_db_username';
$DB_PASSWORD = 'your_db_password';
$DB_HOST = 'your_db_host';
mysql_connect($DB_HOST, $DB_USER, $DB_PASSWORD);
mysql_select_db($DB_NAME);
// Your ShareASale ID
$nSASID = 132296;
// Shortcut / Alias
function myres($cString) {
return mysql_real_escape_string(stripslashes($cString));
} // ends function mres($cString)
?>
Change lines 3-6 and 12.
Next, you’ll need to set-up an admin.php page, like so:
Set your username & password that you’d like on lines 8 & 9
Finally, in your MySQL database, create the products table:
CREATE TABLE `products` (
`ProductID` int(10) unsigned NOT NULL,
`Name` varchar(255) NOT NULL,
`MerchantID` int(10) unsigned NOT NULL,
`Merchant` varchar(255) NOT NULL,
`Link` varchar(255) NOT NULL,
`Thumbnail` varchar(255) NOT NULL,
`BigImage` varchar(255) NOT NULL,
`Price` float(8,2) unsigned NOT NULL,
`RetailPrice` float(8,2) unsigned NOT NULL,
`Category` varchar(50) NOT NULL,
`Subcategory` varchar(50) NOT NULL,
`Description` text NOT NULL,
`Custom1` varchar(255) NOT NULL,
`Custom2` varchar(255) NOT NULL,
`Custom3` varchar(255) NOT NULL,
`Custom4` varchar(255) NOT NULL,
`Custom5` varchar(255) NOT NULL,
`Lastupdated` datetime NOT NULL,
`Status` varchar(50) NOT NULL,
`Manufacturer` varchar(255) NOT NULL,
`PartNumber` varchar(255) NOT NULL,
`MerchantCategory` varchar(255) NOT NULL,
`MerchantSubcategory` varchar(255) NOT NULL,
`ShortDescription` varchar(255) NOT NULL,
`ISBN` varchar(25) NOT NULL,
`UPC` varchar(25) NOT NULL,
`SKU` varchar(255) NOT NULL,
`CrossSell` varchar(255) NOT NULL,
`MerchantGroup` varchar(255) NOT NULL,
`MerchantSubgroup` varchar(255) NOT NULL,
`CompatibleWith` varchar(255) NOT NULL,
`CompareTo` varchar(255) NOT NULL,
`QuantityDiscount` varchar(255) NOT NULL,
`Bestseller` tinyint(1) unsigned NOT NULL,
`AddToCartURL` varchar(255) NOT NULL,
`ReviewsRSSURL` varchar(255) NOT NULL,
`Option1` varchar(255) NOT NULL,
`Option2` varchar(255) NOT NULL,
`Option3` varchar(255) NOT NULL,
`Option4` varchar(255) NOT NULL,
`Option5` varchar(255) NOT NULL,
`bActiveProduct` tinyint(1) unsigned NOT NULL,
UNIQUE KEY `ProductID` (`ProductID`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
I’m wrapping up the script which loads the data, which will be posted tomorrow.