I'm pretty sure you weren't planning to bring him along anyway, just ask him to teleport you to the caverns.
...and that's not the best way to make him think you forgive him for anything he thinks he may have done.