Gujarat Congress working president Hardik Patel on Thursday alleged that functionaries of the party's state unit were harassing him as they wanted him to leave. Patel also said that although he had taken up the issue with Congress leader Rahul Gandhi many times, no action has been taken to address it.