be careful with - be careful of

Let's start with objects.
Be careful with something that you are touching or carrying - something that is precious or, perhaps, dangerous if you mishandle it.
Be careful of something more distant. Something precious that you might bump into and damage, or something that might hurt you if you are not careful.

People?
Be careful with ... someone who is emotionally fragile, someone you might hurt. OR ... someone who might hurt you.
Be careful of ... - that doesn't sound right - I might say Beware of ... though.