scalzi has a theory, and I agree with him (mostly) that part of what's going on is that SF is too much of an in-club; that you need to come in with way too much background, currently, to be able to just pick up your average Iain Banks novel and read it. Which is an interesting problem, and actually one I'm kind of trying to address, and I know John is as well. But I think there's another thing going on.
I think we're not having enough fun. People complain of lack of sensawunda, as they say--big scope, big ideas, big spinning wheels in the sky. Some of that may be because the field is pretty heavily mined--the Dyson Sphere stories have been written--but it's not like it's impossible to do. In just the last couple of years, I can point to books by Charlie Stross (Accelerando), Peter Watts (Starfish), Robert Charles Wilson--pretty much everything he writes, let's be honest here--and so on, that are just full of shiny shiny ideas and plain neat weird. ("Neat weird" is not a bad definition of "sensawunda," now that I think about it.)
Now, big ideas are not really my baliwick as a writer. I mean, I can come up with 'em sometimes (There's a couple in each of my SF books that I'm more or less proud of.)? But I tend to bury them under plot and character, because that's what I love. So of course I'm going to come up with opposing theories.
And my current one is that we've gotten awfully fucking serious over here in SFland, and we're not all that damned much fun to read anymore. Now, I don't mean that we should abandon all pretense of social relevance, or real characterization, or Lit'ry Merit. because frankly, I like that stuff. And find the uncomplicated whiz-bangs pretty damned dull.
On the other hand, there is something to be said for packing in the cool, the fun, the adventury, and the zingy dialogue. One does not have to be staid and cryptic to discuss things of import. Irony and sarcasm and humor are useful tools for opposing the establishment. (Ask any feminist. Or, you know, any of the majority of feminists who do have a sense of humor.)
Sure, we're all over here sawing away, hammering together coffins. But I don't see why we can't nail some gingerbread and paint onto the damned things while we're at it. I mean, "fun" is pretty much de rigeur in a fantasy novel. And those sell a bit better. And I understand that there's this romance genre that's doing alright, and that might have some fun in it too. Or maybe some smut. But it can't be smut alone, because SF has accepted smut.
I guess what I mean to say is, yanno: if we can't dance, nobody's going to want to be a part of our revolution.