shel/crawler

Last activity 19 May 2025 7 1828
composer require shel/crawler

Allows crawling of sitemaps and node-trees

2.4.2

Version
2.4.2
Type
neos-plugin
Release Date
May 19, 2025 10:07
Source
git
Distribution
zip
Requires
  • neos/neos: ^8.3
  • php: >=8.1
  • chuyskywalker/rolling-curl: ~3.1
  • ext-curl: *
  • ext-simplexml: *
  • ext-libxml: *
GPL-3.0 c34afba40a9ce342da69f13d542ca5e765e59140

Readme excerpt

Shel.Crawler for Neos CMS

Crawler for Neos CMS nodes and sites. It can be used to warmup the caches after a release.

Installation

Run the following command in your project

composer require shel/crawler Usage

To crawl all pages based on a single sitemap run

./flow crawler:crawlsitemap --url=http://huve.de.test/sitemap.xml --simultaneousLimit=10 --delay=0

To crawl all pages based on all sitemaps listed in a robots.txt file

./flow crawler:crawlrobotstxt --url=http://huve.de.test/robots.txt --simultaneousLimit=10 --delay=0 Experimental node based crawling

This command will try to generate all page html without using actual requests and only renders them …
Read more

The content of the readme is provided by an external source, which we have no control over. Check out the actual readme at its source.
The same vendor provides 27 package(s).