Afraid not

I always understood "I'm afraid not" to mean something like, "I'm sorry to have to disappoint you, but no."

I suppose the "fear" represented by the word "afraid" is the same sort of thing expressed by, "The train is going to be twenty minutes late, I fear." In the latter case, the speaker is apprehensive of some unwelcome future event. In your example, the speaker is apprehensive of giving an unwelcome emotion to the person who asked for the newspaper.

I suppose I could try answering your question. :)

"I'm afraid not" and similar phrases can be used where there is some uncertainty as to whether the unwelcome event will actually happen. You might say, for example, "I'm afraid I shall not be able to attend your dinner party this evening," or, "I'm afraid that we may have to lay off several people from this department." If you were sure that the unwelcome event must happen, you would instead say, "I'm sorry; I will not be able to attend your dinner party this evening," or, "We are going to have to lay off several people from this department."

The context in which you are most likely to use the "afraid" construction is when you are politely expressing regret in a somewhat formal social context, or in a business context ("I'm afraid I will not be able to have the report ready by Friday.").

In a very formal social context, I advise you to consult a book of formal writing etiquette. For example, you would not use "afraid" to answer a wedding invitation if you cannot go; you would say something more like, "Mr. and Mrs. Nizou deeply regret that they will not be able to attend."

To more directly answer your question, yes, the phrase in question is idiomatic.

The phrase is more commonly used in a conversation rather than being written. A semi-formal letter might accept its use; in an informal letter, you would just say no; and in a formal letter you would choose a less conversational form, such as 'I regret to inform you'.

Hearing the phrase conjures the image of the upper class English for me. Stiff upper lip and all that. Although speedwell's interpretation is correct, I percieve the phrase more strongly as a softened or polite way of saying that you regret something, e.g.

Is it going to be sunny today?
I'm afraid not.

The implication of this statement is that you would like it to be sunny but can't do anything about it.

If you are writing a formal paper, news story, business report , or any written form that is supposed to be devoid of opinion, then the use of the first person (I, we), and thus the phrase 'I'm afraid not', is not appropriate, unless you are explicitly expressing opinion.

obligatory rope joke ...
http://whatknot.tripod.com/knots/knotfun.htm

Actually, your analysis about dropping the "can" is correct, although I would think of it as dropping the following:
"I'm afraid (that you may) not (have the newspaper)."
Why do you think this doesn't make sense? All the dropped words are unnecessary as they are already supplied by the question. It's really no different than saying:
"May I please have the newspaper?"
"No."
Is "No" a nonsensical answer? Is it really necessary to repeat the question with:
"No, you may not have the newspaper"?
Of course not. No, or yes, by themselves can be used to answer a question.
In the affirmative, it's usually:
"I'm afraid so", with so meaning true.
As was mentioned, afraid in this context means filled with regret, concern, or worry, not fear. Any dictionary will give you these alternate definitions.
(By the way, the sentence I used above, "of course not" is structurally similar to "I'm afraid not". it means the same as "of course it is not")

The most common phrase in spoken English to show that the speaker recognises that his/her reaction is in some way unhelpful or unwelcome is "I'm afraid" or "I'm afraid no". It may warn of disagreement, but its general meaning is wider and indicates the speakter sees his/her reaction as unavoidably unhelpful.

English allows for the omission of the relative particles in some instances: "The dog that I walked" means exactly the same thing as "The dog I walked." I feel like this is a similar kind of omission where "I'm afraid not" is another way to say "I'm afraid that the answer is 'not.'"

I agree with porsche here. "Not," in this context, like "so," is what's called a "prosentence" (IIRC)--it stands in for an entire sentence in the way a pronoun stands in for a noun. There are lots of other constructions like this:

"I think not/so."
"I hope not/so."
"If not/so, ..."

...and so on.

The "I'm afraid" is indeed an idiom, coming from the ettiquete convention of acknowledging or pretending that any inconvenience to the other party causes you distress.

The phrase "I'm afraid not" employs an ellipsis, defined elsewhere as an omission of words for brevity and compression. Actually, two ellipses.

I'm afraid (ellipsis: THAT YOU MAY) not (ellipsis: HAVE THE NEWSPAPER).

Hi, gosh, not to be negative, but you all generally make it worse. It seems that even though you all have a good grasp on English language, it is clear that you are mostly not English or American.
The term "afraid" can be looked at as "I realize" or "I suppose" Fear can, but does not necessarily have much to do with it. Furthermore, American English is often evolving unfortunately becoming more simplified thanks to black culture, I'm afraid. Perhaps the writer or speaker is trying to convey that they deem the outcome unfortunate. In other words, you wont say, "I'm afraid you made me happy" but you can say " The people are angry I'm afraid" The more proper way is to use it at the end of sentence to be less serious, or coy, sarcastic as well. Perhaps a way of saying not certain; to not commit fully to your statement, but the context must be taken into account. I am only on this blog because of a misunderstanding on Facebook with my woman friend from Hong Kong. The subject being "Cheating sexually", she wrote a little scenario. I was being cute to say..."it happens all the time, I'm afraid"
But she took it as to mean that i thought the subject matter was inappropriate for my facebook. That i was afraid! So, I'm sending a few links...amazing how much written on this expression. "afraid not" kind of an appologetic "no" or, saying it may be better if things were different. "Is it going to rain?" "I'm afraid not" that response would indicate that the person replying would rather have it rain, but not fearful of no rain. It is also a way of being rude in American style..."May I have a cigarette?" "I'm afraid not" that is a sarcastic and annoying way to look down upon someone while denying them something. Good question....as far as proper writing? go ahead and use it, writing is more art than formality. It is a good way to open discussion on a subject if you want the reader to inquire as to clarification as to why you are "afraid not"
Lastly, there is a funny joke about "a frayed Knot" but thats for another time...google it. thanks for reading

nappidesignm offers this:

"Furthermore, American English is often evolving unfortunately becoming more simplified thanks to black culture, I’m afraid."

Please explain this comment.