I believe football is strongly connected with working class culture in UK, while sports like rugby or cricket are considerd more sophisticated and attracting more upper class people.
Here in Japan, there isn't such a class divide for sports. Like football and baseball are our 2 biggest sports but preference is by no means affected by one's social status.
However, hooliganism seems rather common and notorious in many european countries and I wonder if football and its fans tend to be looked down on by “educated” people widely, not just UK.