Freedom of Speech for Whom?

Recent events have prompted discussion about freedom of speech and its limits and, whilst I'd rather not weigh in on the case of Geert Wilders specifically, I would like to raise the more general and philosophical issue of freedom of speech - an issue that obviously affects the specific case in hand.
As far as those people whose views I find objectionable - such as Holocaust Deniers, Fascists, Theocrats et al - I'm generally of the opinion expressed by Evelyn Beatrice Hall (and mis-attributed to Voltaire) - "I disapprove of what you say, but I will defend to the death your right to say it." However, if I may play Devil's Advocate, I would like to put forward a counter argument that is specific to our time.
I will phrase this argument in the form of a question (one which, given the present climate, I cannot fully and satisfactorily answer myself):

Is it logical to grant freedom of speech to someone whose views - if accepted by a large part of the population - would put an end to freedom of speech?

This question is hinting at a clause within the principle of freedom of speech, almost a self-continuing clause, if you will, that freedom of speech should not be granted in those few times where it is possible that the result of that freedom would be the end of that freedom.
Now, generally I would disagree with this argument whole-heartedly, and I will bring up some counter arguments shortly. However, I think that it is important, when considering this question, to look at the state of the world's economy. In times of economic crisis, when the vast majority of people are doing very badly financially, extremism tends to do very well. William Cobbett's old saying, "I defy you to agitate a man on a full stomach", rings well in reverse too. Thus, it might be argued, that in the short term, we are at greater risk from political extremism. Therefore, would it not be sensible to limit the freedoms of only those few extremists whose freedom might result in the curtailment of the freedom of others, and create a very serious threat to public order. Should we not curtail a few freedoms in the short term, in order to maintain the principle of freedom in the long term? You might take Geert Wilders to be one of those people - although I probably would not, given the very small number of people for which Wilders' own brand of Islamophobia and prejudice is attractive (some might say that the electoral success of the BNP in some areas shows Wilders' threat, although I would say that the great mobiliser for the BNP is not Islam, but immigration, and alienation - although Nick Griffin may well come within the bounds of someone whose freedoms should be curtailed). You might also take Yusuf al-Qaradawi to be such an individual, and he probably has a larger audience than Geert Wilders.
There are multiple possible counter arguments (or counter counter arguments - if you will):
- Who will decide who constitutes such a great threat to our freedom? Judges? Politicians? Civil Servants? The system would be open to misuse for political reasons.
- If freedom of speech is an absolute then there can be no situation in which it should be curtailed. If you accept freedom of speech for a particular individual in most cases, why not in this case? It seems illogical that an individual who would be free to say what he pleased twelve months ago should now be prevented from doing so. The counter argument to this is the example of Adolf Hitler and the NAZI party, who gained support during the Great Depression quite rapidly, moving from being a fringe party of lunatics to a major player in the Reichstag.
- The suggestion is not applicable to modern life, as there is no extremist group that truly threatens to gain power and thus destroy our basic freedoms. I will stress that the argument is hypothetical, it is a 'what if?' not a specific point about a specific individual. However, I will once again raise the specter of Hitler and fascism to demonstrate how easy it can be in a time of crisis for people to turn to the extremes. Another example from (arguably) the other end of the spectrum, would be the Bolsheviks in 1917. In February, after the Tsar was deposed, a liberal(ish) Provisional Government was set up that was forced to work in tandem with the socialist(ish) Petrograd Soviet. The Bolsheviks, who were always a fringe organisation - even within the RSDLP - were not involved in either. They were only able to take over the Soviet - through elections, mind - due to the ongoing war and the coalescence of the more moderate Mensheviks and Socialist Revolutionaries behind the PG's war effort. Once again, in October, they were only able to take full power due to popular unrest, greatly influenced by the PG's weakness (and by the war and the economy etc).
- One should be confident that, through argument and dialogue, one can argue succesfully for the liberal point of view. The problem with this is that it might be sound in normal prosperity, but times of crisis result in a paradigm shift that may not allow for it.
I think that I can see the case that I have put forward here - I can see that it has some validity. In my head, it sounds sensible. However, in my heart, I cannot accept it. I can also see the pervading authoritarianism of it, and the problems that entails. All-in-all, I cannot see a resolution to this problem that will overcome all the issues - it seems to be an ultimate catch-22.
In case you're wondering why this is so important:
For an example of what happens in a country where freedom of speech is curtailed to the point of non-existence, one need look no further than here.