Could remove NuGet 'packages' folder from Git source repo to reduce repo size

I was just wondering why is the 'Packages' folder included in the source repository?
Anyways Visual Studio will downloaded the missing NuGet packages when we build the solution/project for the fist time, so there is no need to include 'Packages' folder. And most of the developer would have the packages in their local cache so packages will not download from internet as well.

Including 'Packages' folder increases the size of repo and in repo like Git it could become difficult to clone by time.

Generally in our projects we do not include the packages folder and it works fine for us and hope it could work over here as well. Only thing to be taken care is 'packages.config' file is updated (again this is updated by NuGet package manger).

Please let know if I am missing something in understanding the use of Packages over here.

Thanks a lot for suggestion! This work item already exists