It's a phrase I loathe, as well. But there you have it. (I am of color: I am a pinky beige.)
The consensus by those attending the panel was that it was a self-replicating process. If people do not see people like them doing something, they assume that it is not something they "can" do. (Fiona Patton called it a "you-shaped door.")
And yeah, being exceedingly white is one of the reasons I am not the person to head it up. (The other is being a sucktastic administrator.)
It's not incumbent upon those who are discriminated against to do all the work of ending discrimination, of course.
One of the things that *we* can do is increase the diversity of our writing.