The best way to have a local up-to-date copy of all files is to setup a private mirror: See: the Mirroring How-To
wget is free software and available for Linux, Windows, and Mac OS X at www.gnu.org/software/wget/
This is an example of how to get files using wget:
wget -w 2 -m -H "http://www.gutenberg.org/robot/harvest?filetypes[]=html"
Replace html with the file type you are interested in.
If you want only files in a given language say:
wget -w 2 -m -H "http://www.gutenberg.org/robot/harvest?filetypes[]=html&langs[]=de"
Replace ‘de’ with the ISO language code you are interested in. Tip: you can learn the language code of any language in the Project Gutenberg catalog by looking at the status window of your browser while moving your cursor over the language at this page
If you want to setup a mirror of Project Gutenberg, read the Mirroring How-To
You can extract the whole Project Gutenberg catalog data from the Project Gutenberg catalog in machine-readable format. The catalog data are granted to the public domain.
Project Gutenberg publishes hundreds of new eBooks every month, and we also fix hundreds of books when issues like typos are reported. In addition, the ‘generated’ formats (under the ‘cache’ top-level directory) are re-built monthly and sometimes have changes due to improvements in the software that builds them.
When you save a copy of a Project Gutenberg eBook, consider periodically checking for any changes.