Why wont the narrator wait for doodle?

The narrator does not wait for Doodle because he is ashamed of him. He is embarrassed by his brother's slowness and disability, and he does not want to be associated with him. He also feels that Doodle is holding him back from his own life and dreams, and he resents him for that.