make clean removes any intermediate or output files from your source / build tree. However, it only affects the source / build tree; it does not touch the rest of the filesystem and so will not remove previously installed software.
If you’re lucky, running make uninstall will work. It’s up to the library’s authors to provide that, however; some authors provide an uninstall target, others don’t.
If you’re not lucky, you’ll have to manually uninstall it. Running make -n install can be helpful, since it will show the steps that the software would take to install itself but won’t actually do anything. You can then manually reverse those steps.