left to a fairly large degree and went to other much smaller parues, or formed new ones like Alternative for Sweden (Alternativ för Sverige) ie AfS.
SD (NOT AfS though) itself even has some support from non-whites and/or immigrants in higher numbers than some would assume.
https://sv.wikipedia.org/wiki/Alternativ_f%C3%B6r_Sverige
It was founded in March 2018. The party was founded by Gustav Kasselstrand and William Hahne.
Alternative for Sweden was founded by members of the Sweden Democrat Youth, who were collectively expelled from the Sweden Democrats in 2015. As its primary questions, it advocates forced repatriation of one million immigrants and Swedish withdrawal from the European Union.
AfS ran in the Swedish general election in 2018, but failed to enter the Riksdag. With 0.31% of the vote, AfS is the second largest party without representation in the Riksdag. In the 2019 European Parliament election in Sweden the party ran on an anti-EU platform, receiving 0.49% of the vote. In November 2020 AfS announced their intention to run in the 2021 election to the Church of Sweden council. They won 1.24% of the votes giving them 3 seats in the Church council.
Also, many of the most hardcore, actual neo nazis (like Nordfront, Nordic Resistance Movement, etc) detest the Sweden Democrats. because under Jimmie Åkesson's leadership, SD has forged many ties with RW Israeli parties and groups (united in their common Islamophobia and anti-Arab sentiments). The SD leadership has gone over to Israel on multiple occasions.