> For the complete documentation index, see [llms.txt](https://help.readymage.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.readymage.com/application-management/search-engine-bots-discovery.md).

# Search Engine Bots Discovery

By default, this setting will **block** Search Engine Bots from crawling and indexing your website.

Crawling & indexing your website by Search Engine Bots should be done only on production instances so we recommend leaving this setting in a **prevented** state for all other instances.

## Block Search Engine Bots from discovering your website

To prevent Search Engine Bots from discovering your website you must navigate to **Portal > Selected Project > Selected Environment > Application Management > Search Engine Bots Discovery** tab and click **Block Search Engine Bots**.

<figure><img src="/files/DeElTmX3pJaTpodEHgZ2" alt=""><figcaption><p>Search Engine Discovery not allowed</p></figcaption></figure>

Confirm that you want to prevent Search Engine Bots Discovery.

<figure><img src="/files/1f7HJA1ArjFiEUOIzMS2" alt=""><figcaption><p>Allow Search Engine Discovery prompt</p></figcaption></figure>

Now **Search Engine Bots Discovery** setting tab will have an updated description.

<figure><img src="/files/2GQxW9Kgk6jojjHZFMeN" alt=""><figcaption></figcaption></figure>

In the [Deployments](/application-management/deploy-changes-to-your-instance.md) tab, you will be required to run the next deployment without a cache. Automatic deployment will also happen without a cache.

<figure><img src="/files/NewwaScYjfuQqTlpszd8" alt=""><figcaption></figcaption></figure>

After successful deployment, the **Search Engine Bots** will be blocked from crawling, discovering and indexing your website.

## Allow Search Engine Bots to discover your website

To allow Search Engines to crawl, discover, and index your website you must navigate to **Portal > Selected Project > Selected Environment > Application Management > Search Engine Bots Discovery** tab and click **Allow Search Engine Bots**.

<figure><img src="/files/lqOIuf5ptdhfhCais8rA" alt=""><figcaption></figcaption></figure>

Confirm that you want to allow Search Engine Bots Discovery.

<figure><img src="/files/LSbKVeldY64L7jmrsz5d" alt=""><figcaption></figcaption></figure>

Now **Search Engine Bots Discovery** setting tab will have an updated description.

<figure><img src="/files/dK1xoSpTxQ26FTQfyosD" alt=""><figcaption></figcaption></figure>

In the [Deployments](/application-management/deploy-changes-to-your-instance.md) tab, you will be required to run the next deployment without a cache. Automatic deployment will also happen without a cache.

<figure><img src="/files/5FRyVIdpQ7KAkHu3Ul0q" alt=""><figcaption></figcaption></figure>

After successful deployment, navigate to **Magento Admin Panel > Content > Design > Configuration > Default Website > Search Engine Robots** and confirm that **Default Robots** is set to **INDEX, FOLLOW** and **Custom instructions for robots.txt** are either **empty** or contain correct instructions for Search Engine Bots.

<figure><img src="/files/EJqozpbYI0uCTfgrywqX" alt=""><figcaption></figcaption></figure>

Click **Save Configuration** and flush the invalidated cache type! That is it!