The sideways force on the connector while in my pocket has always been a real concern to me. It's the main thing keeping me from regularly using a dongle.
Get a USB-C magnetic breakaway adapter and you shouldn't have concerns damaging the port. Mildly annoyingly will probably breakaway while you're jogging occasionally but at least it shouldn't damage the port.
