Set pages included and excluded
Objective: focus on useful content and avoid pages that should not be used as a reference.
Pages to include
Section titled “Pages to include”The inclusion list is used to specify the URLs you want to process. Add one URL per line.
Examples:
https://exemple.com/serviceshttps://exemple.com/tarifshttps://exemple.com/faqPages to be excluded
The exclusion list is used to ignore URLs without knowledge value or which should not be used.
Common exclusions:
- connection and private space;
- basket and payment;
- internal search results;
- test pages;
- obsolete archives;
- parameters containing private information.
Save before a classical exploration
Section titled “Save before a classical exploration”The launch of classical exploration uses the rules recorded in the site settings. Edit and save these rules before resuming exploration.
When importing a sitemap, the lists visible in this path are transmitted with the file.
Edit rules after first exploration
Section titled “Edit rules after first exploration”- Update lists in site settings.
- Record.
- Relaunch exploration or sitemap according to your source.
- Open Pages and Resources to control the result.
- Exclude or remove any remaining content manually if necessary.
Verification
Section titled “Verification”In the Pages tab, check that:
- important pages are present;
- excluded pages are not used as knowledge;
- treatment states do not report error;
- the number of pages reasonably corresponds to the expected perimeter.
Example: site with client space
Add in inclusion:
https://exemple.com/serviceshttps://exemple.com/tarifshttps://exemple.com/faqAdd exclusion:
https://exemple.com/connexionhttps://exemple.com/mon-comptehttps://exemple.com/panierhttps://exemple.com/rechercheInclusions target public responses; exclusions protect private pages and variable results. After recording, restart the exploration and verify in Pages that an important URL is well processed and that an excluded URL does not become a knowledge.