When I open a ssh session in the terminal, it asks for my passphrase in a graphical prompt window. That would be ok in theory. But I don’t know my passphrase. So I need to copy it from my password manager. And unfortunately the stupid window doesn’t allow me to access anything else. So, I wanted to disable it.

The usual way is with the environment variable SSH_ASKPASS. To disable the graphical prompt, just remove the value of this variable:

unset SSH_ASKPASS

Unfortunately, in my case this did not work and I needed to remove also another variable:

unset SSH_AUTH_SOCK