The speaker said it himself! His "mind's not right" and it becomes more and more clear as "Skunk Hour" progresses. Because we don't find out this little tidbit until the last half of the poem, it sheds a whole new light on things for your second read, and gives us an explanation for the scattered way the speaker reports what's happening on Nautilus Island. He becomes more disturbed at night – things start to appear more threatening than they are, he's restless, and seems in search of something he never finds. So, there you have it – our speaker, our guide through this poem, is a little off kilter, and we think it makes the ride all the more interesting.
The speaker isn't mentally ill. He's just feeling a little down in the dumps and is tired as he's driving around at night.
The speaker was driven mad by the "you" (implied in the "our" in the final stanza), and that's who he's addressing all this to the whole time.