What IS It About This Fellow Named Brahms?

Wagner  once described Brahms as that "prophylactic composer."
Tchaikovsky called him at one time a "mediocre" composer; at another time a  "scoundrel" - finally, in a state of a  more highly stylized   invective "that artless bastard!"
After their first meeting in 1888 in Germany, the Russian composer remarked that the primary impression that Brahms left upon him was that  "he reminded me of  a benign Russian priest."
The legendary pianist Vladimir  Horowitz remarked more than once about the discomfort experienced whenever he played Brahms, and projected the sense that he was not sure that he  really understood the music of one of the giants of the latter half of the nineteenth century.
And yet Horowitz has left us with several powerful  readings  of the music of Brahms; especially the 2nd Concerto, with his father-in-law Toscanini.
Is it true that, indeed, Brahms burned a number of his works that he felt disappointed in? Some of his contemporaries claimed  just that.
His First Symphony took about fourteen years to complete; actually, about twenty one years if one counts the sketches prior to final writing.
Why is it that whenever I go through the opus 117 Intermezzi, I feel that rather than play the notes, I should rather just gaze at the superbly burnished surface of each note, as if it were some painting? Though brief, these pieces took months to receive final commitment to paper.
Of all the major composers, Brahms, and only Brahms, sometimes glares at me, chiding me to "go ahead; try to figure me out!"
Am I the only one who thinks THIS way about the legendary composer?