My experience is the same as Lucy’s. The only time I ever lost a wallet was as a teenager and it contained my SS card. I have never needed to have a physical card since.
I have thought I should get a replacement card, particularly before I become eligible for SS retirement benefits.
ETA: I just requested a replacement SS card online. I figured no better time to do it than while I was thinking about it.