Configuring the Discovery Service

This document is for:

Invicti Enterprise On-Demand, Invicti Enterprise On-Premises

You can configure multiple components of the discovery service. Suggestions are dynamically added to your list of discovered web applications through:

Email matching
Website matching
Reverse IP Address lookup
Organization Name matching

You can also restrict results to only web applications that have a publicly available DNS record.

Additionally, you can make manual adjustments to refine the search criteria employed by the discovery service:

Second-Level Domain Names Additions and Exclusions
Top-Level Domains Exclusions
Organization Names Additions and Exclusions
IP Addresses Additions and Exclusions

Finally, you can use the Knowledge Base component of your Scan Reports as a hint to customizing your discovery service settings.

Configuring the Discovery Service - Dynamic Resources

Email Matching

By default, the discovery service will use the domain name of your email account to suggest websites that might belong to you.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Match Settings panel

The Email Matching feature is enabled by default, but can be disabled if necessary
Click the Save & Recrawl button at the bottom of the page

Website Matching

By default, the discovery service will use the domain name of any assets already listed in the Websites & APIs page as a search term to suggest websites that might belong to you.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Match Settings panel

The Website Matching feature is enabled by default, but can be disabled if necessary
The discovery service can use a maximum of 32 websites in your website list to build the search query

there is no practical limit to the number of results that can be discovered
if the number of websites in your list is more than 32, this feature is transparently disabled to avoid massive queries from overloading the service

Click the Save & Recrawl button at the bottom of the page

Reverse IP Address Lookup

By default, the discovery service will use the IP Addresses of any assets already listed in the Websites & APIs page as a search term to suggest other websites that are known to be hosted on the same web host.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Match Settings panel; the Reverse IP Lookup feature is enabled by default, but can be disabled if necessary

Click the Save & Recrawl button at the bottom of the page

Organization Name Matching

By default, the discovery service will use the Organization Name in the SSL Certificate of any assets already listed in the Websites & APIs page to search for websites that have the same Organization Name in their SSL Certificate.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Match Settings panel; the Organization Name Matching feature is enabled by default, but can be disabled if necessary

For this example, we shall assume that you have the website https://www.example.org https://web.mit.edu in your list of websites
Consider the certificate for https://www.example.org:

Once the Organization Name Matching feature is enabled, the discovery service will look for websites that have a TLS Certificate with a matching Organization Name

Click the Save & Recrawl button at the bottom of the page

Configuring the Discovery Service - Only Registered Domains

By default, the discovery service is limited to returning only websites that have a publicly available DNS record. To adjust this setting:

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Match Settings panel; the Only Registered Domains feature is enabled by default, but can be disabled if necessary

Click the Save & Recrawl button at the bottom of the page

Configuring the Discovery Service - Manual Adjustments

Second-Level Domain Names - Additions and Exclusions

Additional Second-Level Domain Names

You may wish to configure additional second-level domain names to amplify the results generated by the discovery service. For example, if your main company domain name is example.com, but you wish to discover any additional assets that may be using the word alternative in the name, you can configure the discovery service to also search for websites that match this search parameter.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Second Level Domains panel

The Enable Second Level Domain checkbox:

is only available for the Cloud version of Invicti Enterprise
is disabled by default in the Cloud version of Invicti Enterprise, but can be enabled if necessary
is not visible in the On-Premises version of Invicti Enterprise, and the feature is therefore always enabled

Add the second-level domain names you wish the discovery service to query for
Click the Save & Recrawl button

Excluded Second-Level Domain Names

Once you have enabled additional second-level domain names, you may also refine your search to explicitly exclude second-level domain names to reduce any unnecessary results generated by the discovery service. For example, you may have a defunct number of domains such as revoked.com, revoked.eu, revoked.us, and possibly other combinations with the word revoked in the second-level domain name. You can configure the discovery service to exclude any results that match this search parameter.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Excluded Second Level Domains panel

Add the second-level domain names you wish the discovery service to exclude from its results
Click the Save & Recrawl button

Excluded Top-Level Domain Names

You may also refine your search to explicitly exclude certain top-level domain names to reduce any unnecessary results generated by the discovery service. For example, you may want to ensure that you exclude all results in the .gov, .mil, and .gov.uk top-level domains. You can configure the discovery service to exclude any results that match this search parameter.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Excluded Top Level Domains panel

Add the top-level domain names you wish the discovery service to exclude from its results
Click the Save & Recrawl button

Organization Names - Additions and Exclusions

Additional Organization Names

You may wish to configure additional SSL Certificate organization names to amplify the results generated by the discovery service. For example, if your company has a policy to include the organization name Example Inc in all its SSL Certificates, and you wish to discover any additional assets that may be using an SSL Certificate with this organization name, you can configure the discovery service to also search for websites that match this search parameter.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Organizations panel

Add the organization names you wish the discovery service to query for
Click the Save & Recrawl button

Excluded Organization Names

You may also refine your search to explicitly exclude organization names to reduce any unnecessary results generated by the discovery service. For example, if your company has sold off a business unit that used SpinOff Inc as the organization name in its SSL Certificates, you can configure the discovery service to exclude any results that match this search parameter.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Excluded Organization Names panel

Add the organization names you wish the discovery service to exclude from its results
Click the Save & Recrawl button

IP Addresses - Additions and Exclusions

Additional IP Addresses

If your web setup uses one or more specific IP Addresses to host multiple web assets, you can expand the scope of the discovery service by including one or more IP Addresses or IP Address ranges so the discovery service can also search for websites that are hosted on web servers on the configured list of IP Addresses.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the IP Addresses panel

Add the IP Address and IP Address ranges you wish the discovery service to query for
Click the Save & Recrawl button

Excluded IP Addresses

You may also refine your search to explicitly exclude IP Addresses to reduce any unnecessary results generated by the discovery service. For example, if your company has a web server that only hosts test websites that you do NOT intend to scan, you can configure the discovery service to exclude any results that match this search parameter.

Select the Discovery -> Settings option in the sidebar

In the Application and Service Discovery Settings page, navigate to the Excluded IP Addresses panel

Add the IP Addresses and IP Address ranges you wish the discovery service to exclude from its results
Click the Save & Recrawl button

Configuring the Discovery Service - Scan Reports Knowledge Base

The Scan Report Knowledge Base can be a powerful ally to help you track down web assets which may be missing from your inventory. Whenever a scan is made, the scan report will list paths which are out of the scope of the scan, and were therefore not crawled or tested.

You can use this list as a source of information; for example, the first few links in the Uncrawled section of this scan report's knowledge base indicates that you might want to add testinvicti and testsparker into your list of additional second-level domains.