What do you call the phobia or is there a for people who are scared to let in?

There is actually a term for this kind of phobia. It's called "autophobia". Generally speaking, autophobia is a phobia characterized by an intense, persistent fear of oneself or being alone. It is a specific type of anxiety disorder that can cause significant distress and impairment in functioning. People with autophobia may avoid social situations, withdraw from relationships, or experience difficulty sleeping or eating due to their fear.