You can only remove packages which are not required by another package.
Note: This only removes the reference to the package in the manifest. The package itself, including its contents, remains intact, even for local packages.
When you remove a package from a project, any Editor or run-time functionality that it implemented is no longer available in that project.
To remove an installed package:
Open the Package Manager window and select In Project from the Packages drop-down menu.
Select the package you want to remove from the list of packages. The details view now displays that package’s information.
Click the Remove button.
When the progress bar finishes, the package disappears from the list.
If you want to restore a removed package, follow the instructions for installing a package.