I think it's worse with Libertarians. People usually become Libertarian after studying both the Democratic and Republican parties and finding that they agree with portions of each party, leading to the mixed views of Libertarianism, whether it be Social, Fiscal, Objective, Socialist, or whatever.

The average person knows nothing of Libertarianism except for what he hears about the Libertarian Party, which is so far right wing that even many right-wingers refuse to associate with it. Thus we always get a bad name from everyone. Not just one side, but all sides.

__________________
"When in the course of human events it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation... We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness...."
- The Declaration of Independence