I know this post is a bit old - but I thought I would contribute a finding I had.
I originally created a sub folder under my DNN site and pointed my server's home directory to the sub directory to set up the store - The thought was just to see how the store worked. But I really like DNN's other features so I pointed the home directory back to the DNN site and worked on getting noCommerce working in a sub directory.
This is a first attempt hack - But this is what I did on IIS 6 to get the store to work in a sub directory of DNN. nopCommerce version 1.80 using SQLexpress running locally.
1. I created a sub folder called store.
2. This is where I put nopcommerce
3. In the DNN web config I added the following.
<location path="." inheritInChildApplications="false">
<system.web>
...
</system.web>
</location>
4. In IIS 6 - I turned the subfolder into an application named "store"
5. I also created a separate application pool called "storepool" and put my new store application in that pool
6. In the nopcommerce web config I removed targetFramework="4.0" attribute from the compilation tag (I am running .NET 4.0 on IIS)
7. In the nopcommerce connections.config I removed <remove name="LocalSqlServer" />
It appears to work for me. Preliminary test appear good.
I am very interested in hearing if someone else takes this approach and does some better testing. This was my first attempt, so I am not sure if all the steps listed above are needed or if I should include some other steps in my solution. If I run into anything interesting I'll post it.