shel/crawler

Last activity 19 May 2025 7 1828

composer require shel/crawler

Allows crawling of sitemaps and node-trees

0.9.0

Requires

Shel.Crawler for Neos CMS

Crawler for Neos CMS nodes and sites. It can be used to warmup the caches after a release.

Installation

Run the following command in your project

composer require shel/crawler Usage

To crawl all pages based on a single sitemap run

./flow crawler:crawlsitemap --url=http://huve.de.test/sitemap.xml --simultaneousLimit=10 --delay=0

To crawl all pages based on all sitemaps listed in a robots.txt file

./flow crawler:crawlrobotstxt --url=http://huve.de.test/robots.txt --simultaneousLimit=10 --delay=0 Experimental node based crawling

This command will try to generate all page html without using actual requests and only renders them …
Read more