They actually bother me when they fall out if I'm looking at the manual. Back in the day, I used to just put them back in the manual like you but it wasn't long before I got fed up with picking spine cards up off the floor every time I opened a manual.
That's understandable, but reading's for suckers. When I first get my grubby mitts on a title, I'll look through the manual once then never again (unless it's a fighting game).