Remove the roster delegate if it gets deallocated.

Maybe it's not a good idea altogether to add/remove the roster
automatically?