But on a segment of HuffPost Live on Tuesday, the Jewish reggae singer called leaving the Hasidic community “one of the hardest things [he] had to go through.”

From 2001 to 2007, Matisyahu (born Matthew Miller) was associated with the Chabad-Lubavitch movement based in Crown Heights, Brooklyn. In 2011, he shaved the bushy beard that had become a signature part of his image.

“I’m aware of who I am and what I represent to different people and that I came out as a Hasidic Orthodox Jewish artist … and when you have such a strong identity, you know, most people are not able to break from that type of thing,” Matisyahu said. “I didn’t really think about what the reaction would be, honestly, how it would affect people.”

Matisyahu for years performed wearing a yarmulke or black hat and other traditional Hasidic garb. After shedding that image and becoming less religious, he told the Huffington Post that he got “hit with a wave of very upset or confused people.”

“A lot of people made the automatic assumption that it was sort of the rock and roll lifestyle playing its due course and it was just a matter of time before that lifestyle would take its toll and sort of take me into the dark side,” he said. “In fact, the decision for me to let go of the board and to make those changes was an incredibly spiritual and honest experience.”

When he shaved his beard, he issued a statement that said: “And for those concerned with my naked face, don’t worry … you haven’t seen the last of my facial hair.” However, he has not been seen with a significant beard since.

Matisyahu’s latest album was released in June 2014, but he has been back in the media spotlight in recent months after being disinvited and then invited back to a Spanish festival.

Since the August festival, he has played in Jerusalem multiple times.