"I once stopped these very wharf workers from unionizing"