Let's start with objects. Be carefulwith something that you are touching or carrying - something that is precious or, perhaps, dangerous if you mishandle it. Be carefulof something more distant. Something precious that you might bump into and damage, or something that might hurt you if you are not careful.
People? Be careful with ... someone who is emotionally fragile, someone you might hurt. OR ... someone who might hurt you. Be careful of ... - that doesn't sound right - I might say Beware of ... though.