Customized instance upgrade guide

I wanted to see if we could park a thread where we can discuss/share/define the best approach to upgrading/migrating a customized instance to current functionality. Many like me have invested heavily in customized processes and functionality that we or our clients can't lose but we would also like to share in the new functionality of later releases.

This is a daunting task with many different approaches that could benefit from the collective experience of the Nop user and developer base. For example, the number of SQL diff tools out there is impressive. There are free open source tools, there are tools that cost $39 and tools that cost $899. Most of these tools are great at synching two databases but what most of us need is a schema merge tool.

Merging the application code base is somewhat more straight forward with tools like WinMerge or any of the many diff tools out there but the workload can be massive as every change in every file must be scrutinized.

I would love to here how you guys are approaching this and the tools you are using.

Chris

PS Admins, please consider making this a sticky

I have been using the trial release of http://www.dbsolo.com/ to create the database DIFFs between my modified database schema and the 1.5 database. Making these changes has been relatively simple.

First, I created a diff of my version to 1.5, then I created a diff of 1.5 and my version. These were both saved as text files. I then used WinMerge to diff the two files and create a merge script for my version.

DB Solo also does creates data scripts so I used to synch some of the tables. Nice little tool.

Then I spent 2 days using WinMerge on the project files. This is a bit more daunting. Usually, I work by date so you can automatically update the files that you haven't touched. Then I move all of my unique files into the merge solution, then all the 1.5 unique files in. Finally, I was left with about 50 files that needed to be scrutinized. About half only had some formatting and syntax differences like typecasting variables off "var" (this is done everywhere). Then came the 30 or so files that needed heavy scrutiny. This is painfully detailed work and there is probably no way around it. You simply have to go through each diff and decide what to do. After you have finished you end up with a merge solution that needs to be built.

There is still a lot of work to be done like including the new projects in this solution and adding the relevant references. At then end of it all you will end up rebuilding your solution dozens of times and fixing probably thousands of lines of broken code. I'm still fixing. Its not fun.

There must be a better way?

C.

I think upgrading has quite a few issues. I know you're talking a lot about the database here but moving from v1.4 to v1.5 was a bit more work than I expected mainly due to the CSS changes and the single style sheet, the subject of which I'll post a thread on cos I don't think a single sheet is the best way to go.

But the whole issue of upgrading with each new release needs some thought and ideas. I think the nopCommerce team need to take this into account a lot more for future releases. At the very least a list of amended or replaced files and changes to code, etc. with the release would help.

I'm sort of looking forward to v1.6 but at the same time wondering how much work I've got to put in and whether it's worth it in some cases.

I guess this all goes with the territory of open source, but like you've said there must be a better way.

I like the single style.css  - makes finding the classes much faster and easier (only 1 file to open and search)

Anyway, upgrading (remember there is a screencast for this)

1) backup everything - including the database - and put a copy of it somewhere safe

upgrading the database should really be no problem at all, just run the upgrade script over it and it should be done in a very short while automatically without any pain. I guess any changes you made to the database should be retained but if you made changes to the database, scrutinise the upgrade script before you run it

2) make organised notes of the changes you make - do this from the minute you start to play, it will make upgrading much less painful.

I didn't use any tools for comparing changes, maybe its fast, but is it worth doing when maybe in one file all you changed was  'repeatcolumns="2" to  repeatcolumns="3" ?

okay, i know some of the sites built around nop have been very highly customised and this list could be enormous and maybe not the right solution but it will allways be a solid point of reference also, I'm sure a lot of people only make a handful of changes - and forget where - so keeping a note will still save time.

i put a template below to demonstrate how i note code changes.

you don't really need to note the actual changes, just the idea of what you did - when you look at the old file, you will probably be able to see fairly clearly how you will integrate it into the new one - it might be as simple as copy and paste - on top of that, you should be able to quickly examine the changes bit by bit in a browser.

the sylesheet:
like i said, i like the single stylesheet, i see only pros, no cons

making notes of changes here? not to difficult really (though the big change in class names made it a little awkward)

my solution is to mark the stylesheet    with     /**/   above  any classes you change  - when you upgrade, open the old stylesheet on one side of your screen and the new one on the other, you can then easily search down the page for  '/*' and then find the matching class on the new stylesheet  - really, it is very straight forward

c#
allways comment the code and make a note of which files you edit

the site which is online is basically a copy of what is on my local machine - any change is made there first then uploaded so i always have a backup of the site. in a separate folder, the newest version sits waiting for changes to be made - once thats done and tested, i upload the new site to a separate folder on the server. i copy the database and create a new, replica database  on my local machine, i run the upgrade script over it and test it - i then copy this and write it over the hosted database. then repoint the domain to the new folder - you might want to close the store while you are deploying the database so that no one can access the store while you are upgrading.



below is typical (but pared down)  list - its just a .txt document but i find it effective and useful

keep it alphabetical - it makes working through the files faster

==========================================================================
v1.5
          
storename       1.5
==================

notes
~~~~


ADMIN CHANGES
~~~~~~~~~~~

       MODULES  .ascx
       ~~~~~~~~~~~~~~

CategoryProductAdd.ascx
  PageSize="100">      from "10"        line 52          

CategoryProduct.ascx
  PageSize="50"   from 15     line 62

products.ascx
  PageSize="50"> from 15     line 61


STOREFRONT
~~~~~~~~~~~

       MODULES - .ascx
       ~~~~~~~~~~~~~~


CheckoutConfirm.ascx
  added checkbox
  altered .cs

HomePageCategories.ascx
  repeatcolums to "4"


ProductBox1.ascx.cs     hide add to cart buttons   changed at line 106
ProductBox2.ascx.cs    hide add to cart buttons   changed at line 102



       WEBFORMS- .aspx
       ~~~~~~~~~~~~~~

AboutUs.aspx
       changed masterpage to TwoColumn.master




C# CODE
~~~~~~~

pdfhelper.cs
  customised deliverynote



NEW PAGES
~~~~~~~~~~

news.aspx
press.aspx


DATABASE
~~~~~~~~~


       STORED PROCEDURES
       ~~~~~~~~~~~~~~~~~~~

* In the stored procedure Nop_ProductLoadAllPaged,
    after line "  SET @Keywords = '%' + rtrim(ltrim(@Keywords)) + '%'  ", insert the following:
    SET @Keywords = REPLACE (@Keywords, ' ', '%')



OTHER
~~~~~~

==========================================================================

like i said, not everyone will want to do it this way and a huge amount of code changes would make a massive list, but consider this, if you add 3 lines of c# to one codebehind file in the whole store, and you run a piece of comparison software, it's going to highlight every code change made to all the c# files in the project by the developers - whould you rather trawl through the generated data or be able to go directly to the file, open it and navigate to the 3 lines you changed ?

There are over 4,500 files in 650 folders  in v1.5  do you need to make a comparison between them all when you upgrade ?

.

Yes I think a good organised approach is the only way.

I backup everything twice and to separate hardware. I also tinker on one version and develop to go live on another so I need to track my changes.

I log most of my changes by pen and paper and try to give myself an idea of whether it's work in progress or finished on every file I work on, it's a bit onerous but otherwise I wouldn't have a clue what I've done come an upgrade or data loss.

Putting comments are useful, if not essential, but even these can seem a bit vague over time (or a bottle of wine).

There's some software used where I work called Visual Storesafe that you have to check in and out all files for editing and it keeps an series of backups based on your settings. I think Clearquest is another they use. Not sure if it's practical for a single developer tho'.

I'm using VS Express - does VS Std or Pro have some sort of logging system for files edited in a project so you can see what you've done over a certain period ?

haydie wrote:

I have added fields to several tables and changed some of the logic in some of the procs. With the kinds of changes in this past upgrade (every proc that returned a row ID was modified) almost all of my procs would have been overwritten.

As for using source control, before I began, I added the whole solution to Google Code as an SVN because I work on it from multiple computers and because I like to have rollback on everything. This saved me some time since I can generate change logs but without a diff-ing program, it would have been almost impossible to preserve my changes and get the new changes in. For example, I prefer the look and feel and functionality of my single page chechout to the accordian in 1.5. The code files are very different and the way the order totals are calculated are different (I had changed those as well as the previous paradigm of SubTotal with discount taken out followed by the discount amount that was already subtracted was very confusing).

I completely scrapped the styles.css for my migration. It was too much for me to rollback out of my theme and incorporate all the new class names. So i fixed what was broken only and preserved everything else as it was. This is going to be a limitation later on. Hopefully, Team Nop will come up with a better themeing scheme so this kind of adventure can be avoided.

Having spent years doing these sorts of upgrades, I can state with confidence that:

1. It's never as easy as it seems
2. Automatic upgrades, no matter how well programmed, almost never work properly
3. Whatever can go wrong, will go wrong.

Therefore, for those who have made any code and/or theme changes, I suggest the following, somewhat low-tech approach to upgrading from 1.4 to 1.5.   It may save you many headaches and get you to the goal sooner.

1. Backup everything!

2. Make a complete inventory of all changes made - code, database, and data related.  (Note: http://www.red-gate.com/ makes the best tools for comparing SQL Server databases.  In this case, such a tool may be best used to identify changes, not to actually make the changes.)

3. Create a separate web site (under a different URL, for example new.yoursite.com), and separate nopCommerce database, for staging your "new" v1.5 installation, keeping the old site running and in place.

4. Migrate your changes, one by one, from the old site (still running) to the new site (running but not publicly visible).  
You're making ALL changes on the new site, which is not public at the moment.

A possible order might be:

    a) SQL server structural or procedure changes

    b) C# code changes

    c) Theme changes

    d) Images folders

    d) Data changes - consider doing this the "hard" way, through the interface, with the "old" site open on your screen, as well as the "new" site open.  It might take longer but your database integrity will be preserved.  (This is not the time to try out a database compare tool.  It will not go as well as hoped.)

5. Test, test, and retest.  

6. Once you are sure the "new" site is ready to go live, make it live.  Keep the "old" site online indefinitely (under a different URL, for example old.yoursite.com), so you can easily grab anything you forgot.

I've used the above approach many times, with great success.  It's low tech, but allows you to make the changes one by one and make sure that things are not broken along the way.  Give it a try and post your results here!  I need to do a 1.4 - 1.5 upgrade soon, so I'll be following my own advice.

We launched a new site recently, and I'm posting here some thoughts that might be helpful to others.

Since the beginning of our project, it has been a top priority to ensure that our customizations are made in way that minimizes the complexity of upgrading to new versions of NopCommerce. We built the site on version 1.4 and recently upgraded to version 1.5. The upgrade effort went smoothly. Here are some strategies we employed:

Documentation
We documented every file that was modified and every configuration change that was made. We did not document the detail of those modifications; preferring instead to add comments with within the modified files. Here is our document if you’d like to see what we modified (http://dl.dropbox.com/u/4026885/NopCommerce/SolutionCustomization.doc).

Comments
Within each modified file a comment with the prefix //HACK: was added to identify each customization. We used the word “HACK” because it’s recognized by Visual Studio in the task window. Within ASPX and ASCX files the comment was surrounded by <% %>. When used this way, they’re not picked up as tasks, but the main benefit to us is the ability to search the solution for “//HACK” to see every customization point.

Source Control
We used the Subversion “svn:external” property to share source files from other projects into the NopCommerce solution. That way we can continue to maintain one set of shared code where applicable. On a side note, we had too much difficulty performing a merge to update our source control system with the 1.5 code base, so instead we tagged our final 1.4 code, then deleted everything in the trunk, and re-added the customized 1.5 code.

Website Additions
All new web files were added to a new root folder called “Additions”. The content of this folder mimics the NopCommerce folder organization convention. For example, we created sub-folders Administration, Controls, Modules and Templates to contain the applicable files. One caveat of this approach is that “/Additions” is included in the URL of new ASPX files, but for us the benefit of isolating the customization outweighed this drawback.

Website Modifications
If a file required modification, we considered whether to put a copy of the file into the Additions folder and update existing references to the file, or modify the file in place. The chosen approach was the one that caused the lesser impact to the existing code base. Usually, this was to put a copy in the Additions folder, but some user controls were referenced in many places so it made more sense to modify in place.

Removing Unwanted Features
We first looked in the admin area for a method to turn off unwanted screen elements. Btw, if you don’t find anything, look for conditional display logic in the code behind of the applicable user control to make sure you didn’t miss it. In cases where there wasn’t an available setting to turn off the item, we targeted it via CSS with “display: none”. In general, the NopCommerce markup provides sufficient classes to target applicable elements this way. One exception was the Shipping & Returns link in InfoBlock.ascx, so we addressed that one by putting a modified copy in our /Additions/Modules folder.

Administration Area
Inclusion of new modules in the admin area was as simple as updating Admin.sitemap (we put a copy in /Additions/Administration and modified that instead). One thing to be aware of when creating admin functionality in a folder other than /Administration is that you will need to copy the Web.config from the Administration folder to your Additions/Administration folder. This will apply various admin area configurations to your files, such as selection of the right app theme and inclusion of namespace references. Only in one case did we add functionality to an existing module in the Admin area. We added a new tab (called “Extra”) to the Product Variant screen. To integrate the new user control for this tab required an in-place modification to an existing NopCommerce user control.

Page Inheritance
We updated BaseNopMasterPage and BaseNopPage to inherit from new classes rather than the standard .NET ones. This allowed us to customize page processing and rendering without interfering with existing NopCommerce code. Our new classes were put in /Additions/Controls.

Master Pages and App Theme
We made an early decision to stylize the site as much as possible without modifying the standard markup because we wanted to minimize rework after upgrades. However, if you do choose to make more extensive modifications to the rendered markup, then the impact should not be too great if you can confine the changes to the master pages. We made a copy of the default darkOrange theme and started customizing. The upgrade to 1.5 had an unexpected impact to our custom theme because the CSS classes had all been renamed, but this was fairly easy to fix.

Business Logic
The NopCommerce team has thoughtfully used the partial keyword on classes, so we were able to create new classes with the same name to add functionality to existing objects. The new class files were put in a folder Nop.BusinessLogic/Additions with sub-folders mimicking the main folder structure. The namespace of these classes should not include “.Additions” because they need to match the existing partial classes exactly.

DAL
We used LINQ-to-SQL to get to the database when existing business logic objects didn’t provide the functionality we needed. It was a lot faster than making additions to the existing BLL, DAL interface, and physical DAL classes. This was fine for us, but we’re not necessarily recommending this approach ;)

Solution Modifications
We created a solution folder called “External” (in retrospect I would call this “Additions” instead for consistency instead). New projects were added here. We had to add several references to existing NopCommerce projects to incorporate the new dependencies.

Multi-Lingual Entities
Adding multiple language support for products and categories was the most invasive customization we had to make. Quite a few stored procedures needed modification, and in cases where a new parameter was added (the locale), we had to modify the DAL and BLL code as well. We didn’t modify the admin UI, preferring instead to export the data to an existing localization tool, and reimport the translations.

Database Tables
It’s worth noting that we never modified any existing NopCommerce tables. Instead, we always added new tables to support new functionality (using a different table prefix than “Nop_”). This means that we had tables like Napp_ProductVariant to store the additional product variant attributes we needed. In general, I don’t like “add-on” tables, but in this particular case, the benefit of isolation outweighed the ugliness of this approach.

Those are the main points that I can think of. As we worked on the implementation we treated each potential impact to the existing code base as a major issue to be avoided if at all possible, and otherwise, to be documented. I think it’s important to have this mindset if you want to be able to keep pace with future upgrades without facing major challenges.

Regards,
Gary

Hey Gary !

I read ur Post and documents, that is   nice one !

I want to ask certain things from you ,

?  have you Add new Field in Existing Page of NOP Commerce Project
  ? In your  document you have written that File Modified  "  /Modules/CustomerRegister.cs (modify Page_Load and CreatedUser event)"
After this Modofication,  Is  your code running withour error  

If you have done this  change with succes , Please tell me the steps

I want to add one field in  CheckoutShippingAddress.aspx Page

Looking for your early reply !!!

NopCommerce$ wrote:

If you want to add a field for an existing object, you should define that field at each layer: database, DAL, BOL, and UI. Making modifications to the NopCommerce source code is no different than working with any other c# project.

Regards,
Gary