Can I cancel my direct debit before cancelling my phone contract?

I want to cancel my O2 month-to-month contract but I hate ringing these people as they're always so pushy and I'm not good with confrontation! So what if I just cancel my direct debit and leave it at that? (Obviously after paying my last month though) Legally speaking, can they do anything? Thanks in advance!