Don't pay attention to people who are hostile to you and your activities.

Part of it is your imagination; for the rest, try to negate it by being nice to them, but do not let it disrupt your holy work. To deal with your fear at night, be careful with the nighttime Shema, recite several lines of Tanya in bed, and study Shaar HaBitachon several times. I'm confident you have a picture of the previous Rebbe with you. https://www.torahrecordings.com/rebbe/igroskodesh/011/009/3555