Tone in fiction is a strange thing, not often talked about these days. We hear a lot about voice - does the writer have an original voice? Does the narrator have a unique voice? Does each character have their own voice? And then editors say voice is what they look for - a weak plot can always be fixed but a voice that's not working? Forget it. And sometimes you hear people discussing style. The use of language. The cadence of sentences. Imagery and metaphor. Literary novels are more likely to have style than genre fiction.
But tone? What is it? How do you define it? I used to know when my mother was mad - her tone changed to snappy and short. And if I was mad, she'd say, "Don't use that tone with me." (Hardly fair, if you ask me.) I'll start by saying tone is embedded in the decision about whether you are writing comedy or tragedy. A comedy signals right from the start that it's meant to be funny. The narrator jokes, his or her view of the world is humorous, we see funny things happening. Yes, there'll be some tragedy in there somewhere, but overall we expect that tone to stay light, and we expect a good laugh.
Tragedy, of course, is the mirror side. Lots of drama, serious relationship stuff, catastrophes and character change and growth. Some humour in there works well to leaven the darkness, but the tone will be clearly straight-down-the-line and the work will provide angst/or and page-turning material. Nothing fluffy.
What happens when the tone jars? One example is the YA whiner, the narrator who spends the whole book complaining in a hilarious way that isn't very hilarious by page 30. In fact, by page 40 you want to reach into the book and give the character a good slap. Another is the novel that at heart is about a serious issue, but the writer decides to try and make the issue more palatable by pretending the novel is a comedy. Lots of snappy lines from the main character, and some slapstick here and there, heavily dosed with dialogue that "tells us what we need to know about this issue". That's a book that fails to engage, for sure. No one likes being lectured to, even via dialogue.
Another kind of novel where tone fails (but not completely) is a story which is patently unbelievable, so therefore reader expectation is that we'll go along for the ride here and have a good laugh, with lots of humorous wordplay and situations. Except when you look at the plot, most of it is about serious relationship stuff - boy/girl, mother/daughter - and also stuff about peer pressure. And all the serious stuff is treated very seriously. So all the funny stuff seems a bit odd.
My last example is a movie I saw years ago with Nick Nolte in it (sorry, can't remember the title) which started out very clearly as a comedy. Funny, witty, etc. Then 20 minutes in, it turned into a serious drama. What happened? Did the director bail out and the new director change his mind? How did that happen and no one noticed? That's the thing about tone. It has to be consistent. What the reader picks up in the first 20 minutes or 20 pages is like a blueprint for the rest of the book, especially in terms of tone. You deviate or play with it at your peril. And really, what would be the point? Don't you want your reader to be so inside your story that they wouldn't notice if the roof fell in? Inconsistent tone might not be as immediately noticeable as a character whose hair changes from blonde to brunette on page 50, but it will make the reader uneasy and doubtful about whether you can really tell an engaging story.
(If you can supply any examples of books where you think tone was an issue, I'd love to hear about them.)