There was something utterly nerve-wracking about not being seen. It made the Doctor feel like he was doing something wrong, like he was spying on the people around him. Even Jack, who was aware he was there, still couldn't see what the Doctor was up to. It should've felt freeing, and sometimes it did, but mostly it just felt wrong. Like he suddenly had no one to account for.
Which, considering his ability to touch anything had also vanished, wasn't too terrible, he supposed. He couldn't exactly do anything to be accounted for. Except watch and make the odd comment.
Also, not piloting the TARDIS was up there on the unpleasant scale. Not because Jack was a bad pilot (quite the contrary), but because it was just another sign of how utterly out of control he was now. No idea where his physical form was, no idea what that storm could be, and unable to even touch his ship.
As a whole, he'd had better days.
"Well, you couldn't get rid of me," he told Jack, teasingly. "Too much to do just yet."