I have solved it. You simply add the admin project as reference to the Nop.Web project.
This does, however, not solve that plugins are not build and installed.
I am not 100% sure what to do here. It seems that the default behavior of Azure is to only build the startup project and any project that are referenced through this.
Not sure how to go about it without making a custom deploy script.... (or not using git, instead using vs2013 deploy which works fine)...