shel/crawler

Last activity 19 May 2025 7 1828
composer require shel/crawler

Allows crawling of sitemaps and node-trees

2.2.0

Version
2.2.0
Type
neos-plugin
Release Date
Mar 10, 2022 14:54
Source
git
Distribution
zip
Requires
  • neos/neos: ^5.0 || ^7.0 || dev-master
  • chuyskywalker/rolling-curl: ~3.1
  • ext-curl: *
  • ext-simplexml: *
  • ext-libxml: *
GPL-3.0 389426404b00990ff96431b42f9340b9dbb5f12c

Readme excerpt

Shel.Crawler for Neos CMS

Crawler for Neos CMS nodes and sites. It can be used to warmup the caches after a release.

Installation

Run the following command in your project

composer require shel/crawler Usage

To crawl all pages based on a single sitemap run

./flow crawler:crawlsitemap --url=http://huve.de.test/sitemap.xml --simultaneousLimit=10 --delay=0

To crawl all pages based on all sitemaps listed in a robots.txt file

./flow crawler:crawlrobotstxt --url=http://huve.de.test/robots.txt --simultaneousLimit=10 --delay=0 Experimental node based crawling

This command will try to generate all page html without using actual requests and only renders them …
Read more

The content of the readme is provided by an external source, which we have no control over. Check out the actual readme at its source.
The same vendor provides 27 package(s).