Skip to content
English

Set pages included and excluded

Objective: focus on useful content and avoid pages that should not be used as a reference.

The inclusion list is used to specify the URLs you want to process. Add one URL per line.

Examples:

https://exemple.com/services
https://exemple.com/tarifs
https://exemple.com/faq

Pages to be excluded

The exclusion list is used to ignore URLs without knowledge value or which should not be used.

Common exclusions:

  • connection and private space;
  • basket and payment;
  • internal search results;
  • test pages;
  • obsolete archives;
  • parameters containing private information.

The launch of classical exploration uses the rules recorded in the site settings. Edit and save these rules before resuming exploration.

When importing a sitemap, the lists visible in this path are transmitted with the file.

  1. Update lists in site settings.
  2. Record.
  3. Relaunch exploration or sitemap according to your source.
  4. Open Pages and Resources to control the result.
  5. Exclude or remove any remaining content manually if necessary.

In the Pages tab, check that:

  • important pages are present;
  • excluded pages are not used as knowledge;
  • treatment states do not report error;
  • the number of pages reasonably corresponds to the expected perimeter.

Example: site with client space

Add in inclusion:

https://exemple.com/services
https://exemple.com/tarifs
https://exemple.com/faq

Add exclusion:

https://exemple.com/connexion
https://exemple.com/mon-compte
https://exemple.com/panier
https://exemple.com/recherche

Inclusions target public responses; exclusions protect private pages and variable results. After recording, restart the exploration and verify in Pages that an important URL is well processed and that an excluded URL does not become a knowledge.