Your assumption is that a la carte can be added just as an extra option without affecting the rest of us. I believe that's a wrong assumption! The moment you introduce a la carte, you change the balance: some subscribers will stop paying for certain channels, which means that others will have to pay more, or the channels will have to be killed.
And as everyone else, I do have a list of channels that I never watch and don't mind to be killed. The problem is, that everyone's list is different!

I don't want a channel to be killed just because I don't watch it. That would be selfish. And of course, I don't want my favorite channels to be killed if they are not popular enough.
Letting the market decide and allowing just the most popular channels survive is a dangerous proposition, which would in the end significantly limit our choices. There needs to be a balance that allows new and niche channels to exist alongside with the mainstream channels.