SEO - Duplicate content

oh and a tip for the categories page...  the ?PageIndex query string represents a real page which we want to keep..

So...


            HtmlLink canonical = new HtmlLink();

            string PageIndex = CommonHelper.QueryString("PageIndex");

            if (currentURL.Contains("PageIndex"))
                canonical.Href = string.Format("{0}{1}{2}", catURL, "?PageIndex=", PageIndex);

            else
                canonical.Href = catURL;
                
            canonical.Attributes["rel"] = "canonical";
            Page.Header.Controls.Add(canonical);

In version 1.5 the URL was /Product/99-xxx.aspx
in version 1.7 the URL is /product/99-xxx.aspx
in 1.7 All URL's are now lower case.

I now get a lot og duplicate content problems, because Google indexed the upper case A's Well A's the lower case version.

If I have external links to an upper case version,
How should I implement the proposed solution?

If I use the canonial link to make the lower case the original page,
Will I then loose the "link juice" from the the link (that is pointing to the upper case page)

Hope some of you clever SEO guys are Reading this and can give some explanation.

Best regards
Thomas
http://www.slikfabrikken.dk

you should be able to change it back to how it originally did it mate..

either using the re-write rules or in the pages SEO friendly url setting

that will continue your juice..



I eventually got my nopCom site running with 0 duplicate content link possibilities, and every page with a canonical URL... using variations on the code earlier in this thread :)

Hi guys

does your code look anything like this? I think it is working but maybe I have missed something?

And a related question:
When I type http://www.slikfabrikken.dk/products/177-gummi-sko.aspx I get the correct page
When I type http://www.slikfabrikken.dk/PRODUCTS/177-gummi-sko.aspx I get the correct page with the canonial link rel added
When I type http://www.slikfabrikken.dk/products/177-gxxxxxxxummi-sko.aspx I get my own 404 errorpage witch is what I want

BUT

When I type http://www.slikfabrikken.dk/produ/177-gummi-sko.aspx I get the standard errorpage fron IIS

Do you know why there is a difference between typing the filename wrong and typing the folder name wrong?

            //thn - canonial test
            if (product == null || product.Deleted)
            {
                Response.StatusCode = 404;
                string url = string.Format(CultureInfo.InvariantCulture, "{0}FileNotFound.htm", CommonHelper.GetStoreAdminLocation());
                Response.Redirect(url);
            }

            string prodURL = SEOHelper.GetProductUrl(product);
            string currentURL = Request.Url.AbsoluteUri;

            if (!currentURL.ToLowerInvariant().Contains(prodURL))
            {
                Response.StatusCode = 404;
                string url = string.Format(CultureInfo.InvariantCulture, "{0}FileNotFound.htm", CommonHelper.GetStoreAdminLocation());
                Response.Redirect(url);
            }

            string title = string.Empty;
            if (!string.IsNullOrEmpty(product.LocalizedMetaTitle))
                title = product.LocalizedMetaTitle;
            else
                title = product.LocalizedName;
            SEOHelper.RenderTitle(this, title, true);
            SEOHelper.RenderMetaTag(this, "description", product.LocalizedMetaDescription, true);
            SEOHelper.RenderMetaTag(this, "keywords", product.LocalizedMetaKeywords, true);

            HtmlLink canonical = new HtmlLink();
            canonical.Href = prodURL;
            canonical.Attributes["rel"] = "canonical";
            if (!currentURL.Contains(prodURL))
            {
                Page.Header.Controls.Add(canonical);
            }

Regards
Thomas

Hi Thomas,

Your code throws a 404 THEN goes to a 404 page which isn't what you want... just throw the 404, and set IIS to display the appropriate error message.


Also...  Another thing to do is this...  In the URL re-writes, they are processed in order..

So add a *.aspx at the end, and 404 it..    that way ALL remaining .aspx requests which havent been re-written already will result in a 404...   I re-wrote mine to /notfound

Which won't display in the browser address bar, it will just throw a genuine 404 instead!

That's all you wanna do at the end of the day, make anything invalid throw a 404... get IIS to deal with that effectively by displaying your custom page...   And include canonical URLs in all your pages!

I have done all this :D

Hi Andy,

I have removed the  Response.Redirect(url) and checked the UrlRewriting.config where I have added

<add name="CatchRemaining" virtualUrl="^(.*).aspx"
         rewriteUrlParameter="ExcludeFromClientQueryString"
         destinationUrl="~/FileNotFound.htm"
         ignoreCase="true" />

and it look's as if I'm almost there, but the address http://www.slikfabrikken.dk/products/177-gxxxxxxxummi-sko.aspx still shows the original page. I checked the response using liveheaders and it is a 404. It is just not showing the errorpage. Where am I going wrong?  

/Thomas

Instead of 404 or the canonical header tag, why not just permanent redirect the pages to your preferred url.  Then you are definitely getting your link juice transfered to the permanently redirected destination.  The canonical header tag is more of a piece of advice for the search engines to consider.  With a 301 redirect, you can't possibly choice the unprefered url.

Using nopCommerce 1.90 I have made the following modifications to accomplish this.

File: NopCommerceStore\Controls\BaseNopFrontendPage.cs

    Added string declaration in #region Fields
    
        

protected string strRedirectURL;

    OnPreRender sub replaced with:

        

protected override void OnPreRender(EventArgs e)
        {

            //************************************************MOD
            if (strRedirectURL != null && strRedirectURL != string.Empty)
                Response.RedirectPermanent(strRedirectURL);
            //************************************************

            //java-script
            string publicJS = CommonHelper.GetStoreLocation() + "Scripts/public.js";
            Page.ClientScript.RegisterClientScriptInclude(publicJS, publicJS);

            base.OnPreRender(e);
        }

    OnPreInit sub replaced with:

        

protected override void OnPreInit(EventArgs e)
        {
            //store is closed
            if (this.SettingManager.GetSettingValueBoolean("Common.StoreClosed"))
            {
                if (NopContext.Current.User != null &&
                    NopContext.Current.User.IsAdmin &&
                    this.SettingManager.GetSettingValueBoolean("Common.StoreClosed.AllowAdminAccess"))
                {
                    //do nothing - allow admin access
                }
                else
                {
                    Response.Redirect("~/StoreClosed.htm");
                }
            }

            //SSL
            switch (this.SslProtected)
            {
                case PageSslProtectionEnum.Yes:
                    {
                        CommonHelper.EnsureSsl();
                    }
                    break;
                case PageSslProtectionEnum.No:
                    {
                        CommonHelper.EnsureNonSsl();

                        //***************************************************MOD
                        string strURL = Request.Url.AbsoluteUri;
                        string strDomain = IoC.Resolve<NopSolutions.NopCommerce.BusinessLogic.Configuration.Settings.ISettingManager>().StoreUrl;
                        if (strURL.IndexOf("http://localhost") == -1 && strURL.IndexOf(strDomain) == -1)
                        {
                            string strRedirectPath = Request.RawUrl.Substring(1);
                            strRedirectURL = strDomain + strRedirectPath;
                        }
                        //*****************************************************

                    }
                    break;
                case PageSslProtectionEnum.DoesntMatter:
                    {
                        //do nothing in this case
                    }
                    break;
            }

            //allow navigation only for registered customers
            if (this.CustomerService.AllowNavigationOnlyRegisteredCustomers)
            {
                if (NopContext.Current.User == null || NopContext.Current.User.IsGuest)
                {
                    if (!this.AllowGuestNavigation)
                    {
                        //it's not login/logout/passwordrecovery/captchaimage/register/accountactivation page (be default)
                        string loginURL = SEOHelper.GetLoginPageUrl(false);
                        Response.Redirect(loginURL);
                    }
                }
            }

            //theme
            if (!String.IsNullOrEmpty(NopContext.Current.WorkingTheme))
            {
                this.Theme = NopContext.Current.WorkingTheme;
            }
            base.OnPreInit(e);
        }

File: Libraries\Nop.BusinessLogic\SEO\SEOHelper.cs

    GetProductUrl(Product product) replaced with:

        

public static string GetProductUrl(Product product)
        {
            if (product == null)
                throw new ArgumentNullException("product");
            string seName = GetProductSEName(product);

            string url2 = SEOHelper.EnableUrlRewriting ? IoC.Resolve<ISettingManager>().GetSettingValue("SEO.Product.UrlRewriteFormat") : "{0}Product.aspx?ProductID={1}";
            string url = string.Format(url2, IoC.Resolve<Configuration.Settings.ISettingManager>().StoreUrl, product.ProductId, seName);
            return url.ToLowerInvariant();
        }

    GetCategoryUrl(Category category) replaced with:

        

public static string GetCategoryUrl(Category category)
        {
            if (category == null)
                throw new ArgumentNullException("category");
            string seName = GetCategorySEName(category);

            string url2 = SEOHelper.EnableUrlRewriting ? IoC.Resolve<ISettingManager>().GetSettingValue("SEO.Category.UrlRewriteFormat") : "{0}Category.aspx?CategoryID={1}";
            string url = string.Format(url2, IoC.Resolve<Configuration.Settings.ISettingManager>().StoreUrl, category.CategoryId, seName);
            return url.ToLowerInvariant();
        }

File: NopCommerceStore\Category.aspx.cs

    PageLoad replaced with:

        

protected void Page_Load(object sender, EventArgs e)
        {
            if (category == null || category.Deleted || !category.Published)
                Response.Redirect(CommonHelper.GetStoreLocation());

            //title, meta
            string title = string.Empty;
            if (!string.IsNullOrEmpty(category.LocalizedMetaTitle))
                title = category.LocalizedMetaTitle;
            else
                title = category.LocalizedName;
            SEOHelper.RenderTitle(this, title, true);
            SEOHelper.RenderMetaTag(this, "description", category.LocalizedMetaDescription, true);
            SEOHelper.RenderMetaTag(this, "keywords", category.LocalizedMetaKeywords, true);

            //canonical URL
            if (SEOHelper.EnableUrlRewriting &&
                this.SettingManager.GetSettingValueBoolean("SEO.CanonicalURLs.Category.Enabled"))
            {
                if (!this.SEName.Equals(SEOHelper.GetCategorySEName(category)) || Request.RawUrl.ToLower().IndexOf("/category.aspx") != -1)
                {
                    string canonicalUrl = SEOHelper.GetCategoryUrl(category);
                    if (this.Request.QueryString != null)
                    {
                        for (int i = 0; i < this.Request.QueryString.Count; i++)
                        {
                            string key = Request.QueryString.GetKey(i);
                            if (!String.IsNullOrEmpty(key) &&
                                (key.ToLowerInvariant() != "categoryid") && 
                                (key.ToLowerInvariant() != "sename"))
                            {
                                canonicalUrl = CommonHelper.ModifyQueryString(canonicalUrl, key + "=" + Request.QueryString[i], null);
                            }
                        }
                    }

                    
                    strRedirectURL = canonicalUrl;

                     //SEOHelper.RenderCanonicalTag(Page, canonicalUrl);
                }
            }

            if (!Page.IsPostBack)
            {
                NopContext.Current.LastContinueShoppingPage = CommonHelper.GetThisPageUrl(true);
            }
        }

File:  NopCommerceStore\Product.aspx.cs

    PageLoad replaced with:

        

protected void Page_Load(object sender, EventArgs e)
        {
            if (product == null || product.Deleted)
                Response.Redirect(CommonHelper.GetStoreLocation());
            
            //title, meta
            string title = string.Empty;
            if (!string.IsNullOrEmpty(product.LocalizedMetaTitle))
                title = product.LocalizedMetaTitle;
            else
                title = product.LocalizedName;
            SEOHelper.RenderTitle(this, title, true);
            SEOHelper.RenderMetaTag(this, "description", product.LocalizedMetaDescription, true);
            SEOHelper.RenderMetaTag(this, "keywords", product.LocalizedMetaKeywords, true);

            //canonical URL
            if (SEOHelper.EnableUrlRewriting &&
                this.SettingManager.GetSettingValueBoolean("SEO.CanonicalURLs.Products.Enabled"))
            {

                if (!this.SEName.Equals(SEOHelper.GetProductSEName(product)) || Request.RawUrl.ToLower().IndexOf("/product.aspx") != -1)
                {
                    string canonicalUrl = SEOHelper.GetProductUrl(product);

                    strRedirectURL = canonicalUrl;

                    //SEOHelper.RenderCanonicalTag(Page, canonicalUrl);
                }
            }

            if (!Page.IsPostBack)
            {
                this.ProductService.AddProductToRecentlyViewedList(product.ProductId);
            }
        }

This now also handles when product or category pages are requested via actual path and query (eg. /product.aspx?productid=2292&sename=example-product-name&)

This will also ensure that your domain is always the same as the value supplied in Store Url of administration/globalsettings.aspx  (eg. with or without www.).   For all front end pages that ensure non-ssl.

I have done this in a way that ensures there won't be chained redirects between the preferred domain and preferred rewritten url.

Let me know if anyone has any thoughts or concerns.

Instead of just using Response.Redirect (which from memory returns a 302 response - Object Moved). You could use a 301 (Permanently moved) which is better for SEO:

Response.Status = "301 Moved Permanently"
Response.AddHeader "Location", "[newUrl]"
Response.End();

(Code is untested, but should work - let me know if it doesn't!)

HTH,
Chris

Yeah!

Eventually what I did was to 301 any incorrect requests to the correct page :)

Sorry forgot to update this thread!

chris_rowtcliff wrote:


This is unnecessary, just use

Response.RedirectPermanent(strRedirectURL);

like in my code above . . this will automatically set the status code to 301