Referent of “one”

That's generally true, but a noun isn't always just one word. "Melody" is a noun, and so is "form," but "form of melody" is also a noun. In this case, it's the referent of the pronoun "one."

There's also the musical context; a melody only ever appearing in one composer's work would be unremarkable.

Maybe this is why the Germans like to bunch up all their nounphrases into newwords.

Although I agree generally with dave's response, to be more accurate I would say "form of melody" is a noun (not prepositional) phrase to which "one" refers.

"One" refers to the real subject - "a (highly unusual) form of melody", which is a noun phrase. Take away the pronoun and you have - "It is a highly unusual form of melody that occurs only in this composer’s work".

"I frequently hear the rule that the referent has to be the prior proximate noun. " - perhaps "the prior proximate noun phrase" would be better, especially where 'of' id concerned, although perhaps without the modifiers:

"Can you bring me that box of rather delicious chocolates, the one on the kitchen table?"

"Just look at those friends of David's, the ones hanging around the school gates."