Okay, this may have nothing to do with the expression, but I came across a rather interesting parallel story regarding the expression, "Talking out of your hat."

It seems when the prophet and founder of the Mormon faith, Joseph Smith, was composing "THE BOOK OF MORMON" (and I am now quoting from the book by Fawn M. Brodie, "NO MAN KNOWS MY HISTORY THE LIFE OF JOSEPH SMITH" {Pg.61, Vintage Books Addition, August, 1995} ), as follows:

"David Whitmer, a young farmer from Fayette, New York, and a friend of Cowdrey, paid a visit and watched the process of translation with great wonder. "Joseph Smith", he said, "would put the seer stone into a hat, and put his face in the hat, drawing it closely around his face to exclude the light; and in darkness the spiritual light would shine"... "Thus the Book of Mormon was translated by the gift and power of God"...

It seems that Joseph Smith would not allow anyone to lay eyes on the mysterious gold plates from which he supposedly translated the entire Book of Mormon. Instead, he would place the metal plates into a hat, and translate into prophecy (by the use of a "seer stone"), the images from the plates.

As I read these and other passages from the book, the expression "talking out of your hat" suddenly popped into my mind. Since so many people seem to disbelieve in the validity or legitimacy of the Mormon faith, I wondered if somehow this phrase had long ago risen in reference to the arguably dubious nature of Joseph Smith's translations of his golden plates.