l'indicible blog » SEO

How to automatically add a trailing slash to an URL

Matt — Mon, 03 May 2010 20:55:23 +0000

After long and boring researchs on how to automatically add a trailing slash ( ‘/’ ) after my web sites’ addresses, I thought some people might enjoy sharing the results of this quest…

For those who would wonder the purpose in that, it’s once again about search engine optimization.
Indeed, for a search engine, http://www.webagency.com/name and http://www.webagency.com/name/ are not the same. This is inherited from a time where /name would be a file and /name/ a directory. Nothing by then was preventing a file and folder in the same place to share a name.

Today, we took the habit to use more clear and user-friendly URIs, and most of the time we make it so that http://www.webagency.com/name and http://www.webagency.com/name/ point toward the same page.
Once again, the risk is to end up with two pages being considered as dupes by the search engines, diluting the global weight of the page.
To avoid this, we will try to have a canonical page, and to have the other one point onto this one.

There is several ways to so so, the easiest probably being the use of the “canonical” attribute recently introduced by Google.
For reference, it is about using a like tag in the section of the page.
This way, several URLs will be allowed to point to this page, but Google will know that the address given in the href is the one to take into account.

Another method is to always redirect the visitor to the address we want to have as the main one.
This can be done at the page level (for example with php), but it is probably more fit to do it from the web server (Apache) or a .htaccess file.
We then have several tools at our disposal, related to mod_alias and mod_rewrite (make sur they are activated on your hosting platform before you begin experimenting! This is not always the case)

My experiments lead me from RewriteRule to RedirectMatch, and back to RewriteRule for a result that at last seemed satisfactory.
As a documentation, I suggest you visit Apache’s doc, particularily the following links :
http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteRule
http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewritelog

With no further ado, here’s my solution :

RewriteEngine On

RewriteCond %{REQUEST_FILENAME} -s [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d [OR]
RewriteCond %{REQUEST_URI} (^/blog/.*)$  [NC]
RewriteRule ^.*$ - [NC,L]

RewriteRule ^(.+[^/])$  %{REQUEST_URI}/  [R=301,L]

RewriteRule ^.*$ index.php [NC,L]

Let’s take a closer look :

RewriteEngine On

We make sure the URL rewriting engine is on. If it is not available, we will usually get a 500 error.

We will then test various cases :

RewriteCond %{REQUEST_FILENAME} -s [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d [OR]
RewriteCond %{REQUEST_URI} (^/blog/.*)$  [NC]
RewriteRule ^.*$ - [NC,L]

If the link points to a non empty file (-s = sized file)
or
the link points to a symbolic link (-l = link)
or
the link points to a directory (-d = directory)
or the URI begins with /blog/ with no regard to case (NC = Not Case-sensitive)
then we will replace the URL with itself (-) and ignore the following rules (L = Last rule)

RewriteRule ^.*[^/]$  %{REQUEST_URI}/  [R=301,L]

If we are sill here, it means the previous conditions didn’t apply.
So we are not facing a file/link/directory nor a blog entry (that we put here as an example of a branch to exclude… Note that blogs like WordPress have their own URL rewriting engine).
We are therefore going to replace everything that doesn’t finish with a / ( described as ^.*[^/]$ using regular expressions ) with the requested address followed by a / ( %{REQUEST_URI}/ ) and make an external redirection on this address (R = Redirect, 301 = the error code we want to return ).
This will be our last rule (L = Last), in order for it not to be replaced by another one before it gets applied (this seems trivial, but I must have lost some hairs before I realised it!)

At last, within the frame of our customised URLs projects, we will usually send everything else toward index.php that will interpret it :

RewriteRule ^.*$ index.php [NC,L]

There may be better ways to proceeed. If that is the case, I’d be glad to find out about them.
Anyhow, feel free to leave a comment, it’s always appreciated

DOMDocument and UTF-8, a php charset problem

Matt — Sat, 17 Oct 2009 23:44:11 +0000

Today we’ll see how to manipulate DOM elements with php.
We’ll take for example a very useful function which is oddly kind of hard to find : how to add a specific attribute to some HTML elements.
This can be useful for example to add a rel=”nofollow” attribute to some links to let the search engines know they don’t need to follow them, while leaving those links available for your users.
In an SEO point of view, it can be quite useful as it prevents your PageRank from leaking to all the links on your pages.

To achieve that goal, we’ll meet some vicious problems. Let’s start with a ready function, to preserve the time of the smartests among you :

function addAttribute($context, $tag, $attribute, $value)
{
	$initialEncoding = mb_detect_encoding($context);
	if( $initialEncoding != 'UTF-8' ){
		$context = utf8_encode($context);
	}

	$doc = new DOMDocument("4.01", "utf-8");

	$contentPrefix = 'required meta for utf-8 handling!';
	$contentSuffix = '';

	$doc->loadHTML($contentPrefix . $context . $contentSuffix);

	$elements = $doc->getElementsByTagName($tag);

	if(!is_array($value)){
		$value = array($value);
	}

	foreach($elements as $element)
	{

		foreach($value as $currentValue)
		{
			$alreadySet = false;

			if($element->hasAttribute($attribute))
			{

				$attributeCurrentValue = $element->getAttribute($attribute);

				$attributeCurrentValues = explode(' ', $attributeCurrentValue);

				foreach( $attributeCurrentValues as $attributeCurrentValue )
				{
					if($attributeCurrentValue == $currentValue){
						$alreadySet = true;
					}
				}
				if(!$alreadySet){
					$element->setAttribute($attribute, implode(' ', $attributeCurrentValues) . ' ' . $currentValue);
				}
			} else {
				$element->setAttribute($attribute, $currentValue);
			}
		}
	}

	$output = mb_substr($doc->saveHTML(), 236, -16);

	if( $initialEncoding != 'UTF-8' ){
		mb_convert_encoding($output, $initialEncoding, 'UTF-8');
	}

	return $output;
}

Explanations

$initialEncoding = mb_detect_encoding($context);
	if( $initialEncoding != 'UTF-8' ){
		$context = utf8_encode($context);
	}

We start by detecting the encoding format currently used in the given context.
We store it in order to give the feedback in the same format, and we convert it to UTF-8.
Why UTF-8 ? This format as the (huge) advantage to handle all characters, including the accented or special ones from various languages.

	$doc = new DOMDocument("4.01", "utf-8");

Then, we create a new DOM object, to whose constructor we pass two parameters : the version of the document we’re going to use (typically “1.0″ for XML and “4.01″ for HTML), and the charset of this document.

	$contentPrefix = 'required meta for utf-8 handling!';
	$contentSuffix = '';

	$doc->loadHTML($contentPrefix . $context . $contentSuffix);

Next, we’re going to add a header to the given context. Indeed, even if we have defined the expected values for the given document, those will be overrideen by the document if these headers aren’t declared. And I can assure you that when we don’t know this, it’s a hair pulling scenario !! It is undoubtfully the most tricky part of the manipulation.
We can then load the content in that DOM object.

	if(!is_array($value)){
		$value = array($value);
	}

This function allows us to add several values to a given attribute. Therefore if the argument passed to the function is a string, we convert it to an array.

I won’t spend much time ont the actual function, which is quite explicit.
Just note that we preserve the previous values by storing them in an array, and that we check if the attribute value already exists before adding the new value.

	$output = mb_substr($doc->saveHTML(), 236, -16);

We’ll then save the result, and remove the prefix and suffix we’ve added with a function handling multibyte characters. Indeed, a classic substr wouldn’t wotk well with some characters, as we are here using UTF-8 which uses several bytes to store some of them.

	if( $initialEncoding != 'UTF-8' ){
		mb_convert_encoding($output, $initialEncoding, 'UTF-8');
	}

	return $output;

It’s time to set the result back to its original charset and to return the result.

And… Voila!

URL-Rewriting and SEO

Matt — Mon, 08 Dec 2008 21:39:26 +0000

You’ve decided to use url-rewriting to have more readable and ‘user-friendly’ ones. That is probably a good thing, given the fact that search engines too prefer them this way. But careful, there are some pitfalls to avoid !

Indeed, with this type of configuration, we don’t make a good distinction between content and directory… and neither do robots! Therefore, http://www.mydomain.com/example and http://www.mydomain.com/example/ usually point to the same content, but appear as two different pages by search engines.

We then have a referencing dilution, and duplicates are coming to the party. How horrid !

We already cured a similar problem regarding multiple domain names in a previous article. The solution here will be of the same kind.

To avoid any duplicate annoyance, you’ll have to make sure the addresses without a trailing slash (as http://www.mydomain.com/example) point to the same address with a trailing slash (as http://www.mydomain.com/example/) with a 301error (page moved permanently).

To do so, you’ll proceed as in the following example, using a .htaccess file or modifying the configuration of the web server :


	RewriteEngine On
	RewriteBase /
	RewriteCond %{REQUEST_FILENAME} !-f
	RewriteCond %{REQUEST_URI} !-d
	RewriteCond %{REQUEST_URI} !(.*)/$
	RewriteRule ^(.*)$ http://www.mondomaine.com/$1/ [L,R=301]

The first line states that we’re gonna use a rewrite rule, and the second that we’ll use the current directory as root.
RewriteCond %{REQUEST_FILENAME} !-f excludes files that really exist from this rule.
RewriteCond %{REQUEST_FILENAME} !-d does the same with directories really existing on the site.
RewriteCond %{REQUEST_URI} !(.*)/$ takes care of excluding URLs already ending with a slash.
Last, RewriteRule ^(.*)$ http://www.mydomain.com/$1/ [L,R=301] adds a trailing slash to URLs thereby selected, returning at the same time the appropriate error code.

Note that you have to place those rules before the pre-existing ones to make sure you get the wanted behaviour.

Multiples domain names and SEO

Matt — Sun, 23 Nov 2008 21:24:45 +0000

What to do when we have several domain names pointing to the same site to handle best the referencing ?

Be it to protect the closest names (ex: mysite.com and my-site.com) or to take advantage of the various TLD (Top Level Domain) (ex: mysite.com and mysite.org), there are good reasons to use several domain names for a single site.

If we don’t pay attention, the search engines that matter (exhaustive list of search engines that matter:Google) can see those sites as duplicates and/or arbitrarily attribute the pages to one domain or another , diluting your results in the SERP (Search Engine Result Pages). In any case, the quality of your referencing suffers.

To avoid this problem, we have to choose a main domain name that will be used as a reference. This implies using that name everywhere you may need to write a URL, and also to alter the behaviour of the secondary domains.

We’ll make the secondary domains return the same content as the main domain, while warning the robots they’re not on the main domain. Once again, the HTTP error codes will get useful : we will return a 301 code (moved permanently) along with the content.

Note that a 302 error code (moved temporarily) would be way less efficient about what we’re trying to achieve… and that’s the one that gets sent by default when a redirection occurs !

To return that code, several methods exist. The simplest is probably the use of the ‘redirect’ instruction in the web server (Apache) configuration or in a .htaccess file at the server root.

Here is the instruction to use to redirect all requests to the main address :

Redirect 301 / http://mysite.com/

Redirect permanent / http://mysite.com/

You could for example include a virtual hostname for every secondary domain this way :


    ServerName www.mysite.org
    ServerAlias mysite.org
    ServerAlias www.my-site.com
    ServerAlias my-site.com
    
        Redirect permanent / http://www.my-main-site.com/

Another way of achieving this without altering Apache’s conf would be to apply RewriteRules through a .htaccess file at the root of your site. For example :

RewriteEngine On
RewriteCond %{HTTP_HOST} ^(.*)my-site.com$
RewriteRule ^(.*)$ http://www.my-main-site.com/$1 [R=301,L]

or with :

RedirectMatch 301 /blog(.*) /$1

With a well handled error code, you make sure the main domain achieves all its potential. The search engines that matter are happy, so are you !

404 and SEO

Matt — Wed, 29 Oct 2008 20:02:17 +0000

When trying to optimize a site for a better referencement on search engines, one quickly dives into the underground world of SEO (Search Engine Optimization).

There is in that world a lot of common sense and a few tricks to know, along with some concepts flirting with the sacred, like the well known Google’s PageRank.

Globally, you have to understand how the robots used by search engines see your site, i.e. exclusively in text. You can have a glance at what it’d look like in a text browser like lynx.

Anything that can help those robots work is an advantage for you. Using w3c validated html and valid stylesheets won’t change your pagerank, but will give a better visibility of the structure of your site to the robots than formatting it with tables and divs everywhere.

The html code semantics allows robots to distinguish the content types, and use them in a pertinent way.

For example, the

tag should containt first level titles,

tag second level titles,
text paragraphs, and so on.

The most important is obviously the tag of your pages, which has to contain the most important keywords as much as possible. The specificity of robots versus users doesn’t stop here. Along with the pages content, error codes are emitted. Those codes rarely are visible to the user, except the famous 404 (page not found), but they are important. As a reference, here is a <a rel="external" href="http://www.google.com/support/webmasters/bin/answer.py?answer=40132&hl=en">list of error codes</a> a web server might return. Let’s imagine you have a dynamic content… Some pages can disappear from your sites, although some external links still point toward them. Typically, we want to redirect the user to a page close to his request, rather than confront him with an error page that may stall him. But we also want to warn the search engines that this link is obsolete and has to be removed. Keep in mind that when a robot crosses path with a 404 too often, it doesn’t like it and that can have consequences on your referencement ! How to solve that, will you ask ? Simply, by emitting an appropriate error code, while displaying content for the user ! The 404 code is the most known, but this error only states that the page could not be reached, for whatever reason. For a page purposefully deactivated, we will prefer the 410 error that states that the page is not available anymore purposefully and not because of a technical problem. The choice is yours on which <a rel="external" href="http://webmaster.iu.edu/tool_guide_info/errorcodes.shtml">error code</a> seems to fit the page status the more accurately… In order to generate an error code, you will need to change the ‘header’ of the page. Using php, this means using the following command : <pre>header('HTTP/1.0 410 Gone');</pre> Note that the header is the very first thing being emitted when the page is sent. Therefore, it has to be located at the very beginning of the document, before even the doctype declaration(<a rel="external" href="http://en.wikipedia.org/wiki/Document_Type_Definition">DTD</a>). Not very handy when the content handling system is structured in such a way that it separates the static elements of the page (like the logo or menu) from the actual content. Indeed, if the script sends the top of the page before handling the content related query and noticing it doesn’t exist, it’s too late to change the header ! A simple way around this problem is to raise the size of the php buffer. In php.ini, change the output_buffering line this way: <pre>output_buffering = 65535</pre> This value is a limit, but usually enough (if your page don’t weight an undecent amount!) You can now return any kind of error when you want, without disturbing your user’s visit. </article> </main></body></html>