Inevitably (unless you are very lucky) you will come across a site that needs to be completely changed. Nothing on the site is right, awkward navigation, disorientating site structure, unweildy urls and an almost unfathomable buying process.
The part of the site that I’m going to concentrate on today is the url structure. Overly complicated variable stuffed urls have several problems. Firstly they do not indicate the structure of the site (modern users are more commonly using the url as a means of orientation and navigation), secondly they do not contain the keywords that are relevant to the page and thirdly, depending on the number of variables, they may not be adequately crawled by the search engine spiders.
Effectively your only option is to change the url structure. But this can be fraught with dangers. If they have worked hard to get their site ranking (even with the url structure as it is) you really don’t want ot lose this.
There really is only one option that is open to you as far as mapping your sites is concerned and that is the use of the 301 redirect. This tells the browser/search engine that the page they are looking for no longer exists and that it has moved permanently to another address. I won’t go into why search engines love this but trust me they do.
Depending on the technology your site uses please be cautious who you use to implement these redirects. Problems can and do occur
http://www.mattcutts.com/blog/asp-net-2-and-url-rewriting-sometimes-harmful/
http://dev.communityserver.com/forums/p/476049/536640.aspx
Now the process of mapping your new sites urls to your old sites urls is always going to be tedious and time consuming. In an ideal world you would find the equivalent pages on your current site and your new site, redirect the old to the new and thats it job done. However things are never that simple. You will probably find that every page you currently have does not have an equivelent on the new site.
The site I am currently working on has around 10,000 pages indexed. I am fortunate enough to have three years worth of analytics and stats data with which to work. Without this data your job is going to essentially be guesswork until the site goes live.
So i run a report for the landing pages over the last 3 years and get a list of around 8,000. I then cross reference this with a report on the most popular pages and what pages have referers over the same period and get an excel sheet with the url, landings, refered pages and visits in columns. This is just to check that i am not missing any pages that may have significance but have never been an entrance page. As i expected, over the course of 8,000 urls the two data sets virtually merge into one and no important urls are missed. I am then left with several thousand urls that i wish to work with.
Now some of these urls are no longer going to exist and some of them had had less than 10 visits. I prioritise the pages based on number of entrances and then write a short PHP script to crawl each url and pull out the title of the page. So I now have a list of all the urls on the old site that i want to have equivalents for on the new site. I also have information about the contents of that url to help whoever is doing the mapping.
You guessed it, i have no intention of doing this myself, but my web monkey is more than happy to oblige. Once he is used to the practice of finding the relevant page on the test version of the new site and copying that into the spreadsheet he can do a lot in a day! Within a week or so we have a full list of all the urls and their equivalents ready for the 301 magic.
What magic? …well i will elaborate in a future post but please do not migrate 8,000 urls all at once!!
notes
From experience Google will cope with largish numbers of redirects but it may take time for the new site to settle down to the same (and hopefully improved) state of the old site.
Where there is no equivalent page I simply redirect to a page that has similar content and matches the perceived intenet of a visitor to that page (a bit of thought required from the web monkey).
This is not the only thing you should be doing to ensure your new site suffers as little as possible (think site map and notifying sites that link to you amoungst other things)
You may suffer in the rankings in the short term due to your pages losing trust rank but this should sort itself out over time if your SEO efforts are up to scratch
Excellent discussion on this here