But they did:
"A web crawler, also called a spider, automatically moves from website to website, following links embedded in each document, and can be programmed to copy everything in its path. "
An explanation is necessary (I agree) but that one is sufficient. The article is about why he had access to tools which could so easily be used towards these ends, not the tools themselves.
