/cgl/ - Cosplay & EGL

.I just don't know what to do with myself.
I feel like in a way that wearing Japanese fashion is fucking up my life. It used to not be so bad when I was 16-21 but now at 22 and 23 it seems like it has been a lot worse.
I wear fairy kei and other colorful clothes pretty much daily and before I would get a lot of compliments and some judgmental people but a lot more positives than negatives but now it seems like no matter what I am wearing as long as it is "cute" in any way or fairy kei looking it has been drawing all sorts of nasty attention.
Anywhere from people I work with calling me a kid and saying I look like a pedophile's dream to people I meet that I think are friendly turning out being two faced and between my natural peppy demeaner and my clothes thinking I am "fake" or superficial or that I look like a 7 year old and that they would be embarassed to hang out with me alone. This is even when I wear really toned down outfits like normal jean shorts but some colorful bracelets and a pastel t shirt!
I just don't know what to do. I love other forms of alternative fashion too but have always LOVED pastels so much and they really make me feel the most comfortable and really truly myself. Even my boyfriend finds me not physically appealing in any of my colorful clothes but deals with it because he loves me for what is inside instead.
I could stop wearing pastels but then I would feel like I would be hiding away what to me is a big part of who I am and a truly ingrained part of myself. I don't want so many nasty experiences with people and to embarass everyone but I also know if I changed my style for good and threw out my pastels it would be losing and hiding a part of who I am just to appeal to other people.
I am so conflicted and lost.