Today I was looking to generate a list of all files under remote site directory, including sub directories. I found no built-in options for this in wget. This is how I did it:
wget http://example.com/dir/ --spider -r -np 2>&1 | grep http:// | tr -s ' ' | cut -f3 -d' '
I managed to retrieve 12212 entries from the URL I was exploring.
Comments !