This is such a hard idea to grasp, but once you understand it, and understand the energy around it, it really works. It isn't easy, and it takes time and patience to master how to use it;
*What you think about you bring about.*
At times, you may not even know it, or have any awareness.
I am guessing that something is going on with you that you attract these types. You say you don't know why. But you do! That is the key. Find out that answer.
It took me YEARS to understand how energy works, and how I can master my own thoughts and mind, and bring into my life exactly what I am thinking, wanting and doing. I still don't have it 100%, but I am darn close.
Another hard concept to grasp is love. You can heal more and protect yourself more when you send out a loving, for the greater good for mankind than any other vibrational frequency.
You can grid your house to protect you from unwanted energy. I think the crystals for that are black tourmaline, black obsidian, snowflake obsidian and Shungite. You can print out a grid on line, and then program your crystals. There is tons of free information on gridding. Using quartz crystals will amplify working with these crystals.
Also, you may want to read Judy Hall's book:
"Crystals for Psychic Self-Protection."
I am in the process of reading that now. She also has a new book out.