It is one of the most cliché elitist replies when someone asks you if you like a band. Typically the speaker is trying to show off that they've listened to the band longer than newfound fans. In some instances, however, it's a genuine opinion void of hubris.
I confess that I've used the phrase for both reasons. Recently a friend asked me to go to see The National with him and I found myself saying this exact thing. I love their albums Alligator and Boxer, but found myself bored with their two latest. They weren't bad albums, just disappointing.