Yubikey and ssh issue after timeout

I use Yubikey for storing ssh keys and very frequently I face following issue when I forget or delay to press button on Yubikey device:

sign_and_send_pubkey: signing failed for RSA "PIV AUTH pubkey" from agent: agent refused operation

If you try to add key again it will not help:

ssh_add

Enter passphrase for PKCS#11: 

Could not add card "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so": agent refused operation

This happens on lattest Ubuntu 25.04.

What is the fix?

First we need to restarts pcscd daemon:

sudo systemctl restart pcscd.service

Then we need to review ssh agent processes and helpers:

ps aux|grep ssh

pavel       3539  0.0  0.0  97720  6556 ?        Ssl  19:01   0:00 /usr/libexec/gcr-ssh-agent --base-dir /run/user/1000/gcr

pavel       6349  0.0  0.0  10576  6164 ?        S    19:02   0:00 /usr/bin/ssh-agent -D -a /run/user/1000/keyring/.ssh

pavel       6353  0.0  0.0 190560 14212 ?        SLl  19:02   0:00 /usr/lib/openssh/ssh-pkcs11-helper

pavel       8765  0.0  0.0  18052  2264 pts/0    S+   19:16   0:00 grep --color=auto ssh

Then kill both ssh-pkcs11-helper and agent:

kill 6353

kill 6349

After that run ssh_add again:

ssh_add

ssh_add is an alias to ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so