Genre - these are primarily plot-driven stories, caring little for the development of a character, but telling a gripping story instead. It's more about will the good guy catch the bad guy, army "A" be victorious over army "B," or will he/she fall in love, etc.
Mainstream - to me, while having a plot, is normally more character driven. My novel "The Bridge Beckons" is a young adult/mainstream novel. It is character driven, but still has a plot and a theme. Character driven stories are NOT the exclusive property of literary fiction, if at all. Mainstream fiction is concerned with the human condition, and explains how a character (i.e. character driven) copes with his circumstances.
Literary - is usually character driven but even more, language and style driven, to paint a picture or make a commentary on some world condition. It is the darling of the elite, or ivy-towered group. In my opinion, literary fiction is little more than an essay dressed-up in a fictional style. That may explain why literary “novels,” if one can really classify a literary work as a novel, do not usually sell well. Who wants to read an essay or commentary, when what the general public wants is a story with characters they can curl up in bed with?
Read "Lilies of the Field" or "Red Badge of Courage" for examples of what I mean by literary fiction. I may be wrong, but I had a hard time figuring out a plot line in either of them. Oh, I suppose they had some modicum of a story, but neither of them gripped me, but had wonderful descriptions of the conditions at hand. If they had told a gripping story, rather than repetitious descriptions of conditions, I might have liked them.
There are varying degrees in each category of course. However, most genre novels are primarily plot-driven, whereas, mainstream are primarily character driven with a plot line and theme.
There are many novels "labeled" mainstream that are plot-driven, but become classified "mainstream" by their appeal to a wide audience. However, in my opinion many of them are still genre, but with a wide appeal.