I've also seen the same. Try going to admin area > configuration > general and misc settings > SEO settings and disabling all JS and CSS bundling. Is memory usage dropped?
So I did some further testing today. As suggested I disabled all JS and CSS bundling in the settings, then I removed the WebGrease.dll from the bin folder and removed the dependentAssembly entry for WebGrease in the web.config. After that I restarted IIS and ran .NET Memory Profiler again.
This time the virtual memory usage was 220 MB. So switching off WebGrease seems to have freed up around 190 MB (compared to my previous test).