I have had the exact same problem. The App_Data, Theme and Plugin folders were not removed by the ClearPluginAssembly Task.
It seems to be an issue with the VS 2019 MSBuild.exe.
Unfortunately, I haven't figured out what the exact issue is. I was using VS 2019 Pro. If I build the plugin with VS 2017 it works properly.
I am using JetBrains' Rider, as well. It used the most recent version of the MSBuild.exe which was the one from VS 2019. I switched it to the 2017 version in the settings and builds properly.
Hope this help.