Why do people criticize the EDL so much?

I am not English but it seems that all of Europe is getting a dramatic influx of immigrants, its making the English culture less distinct. I am not trying to be racist but I see the same thing here in America where the nation is developing a significant Hispanic population that is making parts of the Southwestern US look like Mexico. So,I understan that the EDL can be extrem sometimes, but why do people downplay their ideology as simply racist rather than trying to save their own culture? I mean its not like theyre the Nazis, theyre not exterminating people.