I'm a new mom of an 8 month old so I can't say that I have any expertise in the matter at hand, but I'll offer my opinion. I think it's really important that kids feel like they have access to books and choices in books so having some that are "theirs" is great-- then any time they want a book they can have one. Having nicer ones set aside is also great, because some books (in the world of adults and kids both) are more fragile/special and need to be treated differently. After all, when they go to college they'll need to feel such ownership of books to make notes in the margins or highlight pages, etc.-- provided that we're not all completely digital by then ;-)
Anyway, they are old enough to be learning that they need to be gentle with animals and people, so why not books too? When you sit down to read one of the special books or get books from the library, you can talk to them about what makes this book different (maybe it could get ripped easily or the library books don't belong to them). When you notice that a book is being damaged or is being used in a way that is not appropriate, you could calmly have a conversation with them about what books are for and that treating books with respect will help make them last longer and increase the enjoyment (who wants to read a book that has been scribbled in with crayon?). If you haven't already, maybe you could arrange things in your home where there is a special reading area, so the books are less likely to be mixed in with toys and such? Maybe they are bored with some of those books and need new ones (thrift stores actually my have some in good condition) and then if they get news ones you can tell them that these are books, not toys, so if they need building blocks or whatever, they need to use what they have, not the books (and put the old books away for a while). Also, setting an example yourself of reading is always recommended-- ok, you might not have time to sit around reading your own book :-D but when you read to/with them, use a bookmark, maybe talk about how great the illustrations are, etc. (some people even like the way some books smell!) so they get another message about how special books are. I personally don't think a "check out" system at home is such a great idea because I think it might make them feel like books are not for them. But you can get them their own library cards and then go regularly (we went once a week when I was a kid) and have them check out books (5 each?). Tell them that they are responsible for them, just like they are with clothes or toys.
OK, I feel like I am starting to repeat myself a bit. Anyway, it's great that you want them to have books and enjoy them and want to teach them respect for them.