The stuff I need at any given moment pushes out the stuff I was using last time, crams it into corners and pushes a lot of it clean out of my brain. A lot of it flushes during the writing process, too--don't ask me to explain, extemporaneously, the stuff I exposit in Carnival. It's gone; it's on the page.
Once the book has been born, it doesn't need the writer any more.
Also, a brain full of local Las Vegas history or string theory or alien biology makes it hard to remember to pick up your drycleaning, too.