Today I learned that if you need to update (or delete) credentials that were previously saved with the git-credential-manager, then you have to go the Windows Credential Manager.
Find it under Control Panel -> Credential Manager -> Generic Credentials.