It's getting personal here, but not too personal. That said, it's important to note that we have a first-person voice in "Some Trees." Just like any first-person speaker, we want to be careful not to confuse the art with the artist. Sometimes writers and poets just like to use "I" for a specific effect, even if the speaker is nothing like the artist. Usually that effect is intended to make us feel the poem more deeply, as if the speaker is revealing his secrets to us.
Here the effect seems to be somewhat similar, but since we don't get any specifics in the poem, we can't say for sure that we know any secrets by the end. So, in using first-person, Ashbery seems to create a speaker that's getting personal with us without saying too much. But we can assume the speaker is thinking about a romantic relationship since he speaks about loving and touching in the third stanza.
This kind of voice also lends to the casual conversational sound that we hear, even if the words can be quite formal and "painterly." There's nothing sensational about it or over-the-top. In keeping with those trees, our speaker just is. He's cool, calm, and collected, unlike the rest of us who would like to shake those trees until we get some answers.