# Search Engine Bots Discovery

By default, this setting will **block** Search Engine Bots from crawling and indexing your website.

Crawling & indexing your website by Search Engine Bots should be done only on production instances so we recommend leaving this setting in a **prevented** state for all other instances.

## Block Search Engine Bots from discovering your website

To prevent Search Engine Bots from discovering your website you must navigate to **Portal > Selected Project > Selected Environment > Application Management > Search Engine Bots Discovery** tab and click **Block Search Engine Bots**.

<figure><img src="https://2103450200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MKew-XtJKAPabdeTxXg%2Fuploads%2FrL4HpBQUiw8qTRYopOby%2FScreenshot_20240411_114157.png?alt=media&#x26;token=2e2646b4-abea-49bd-a658-376ae4c1c1a8" alt=""><figcaption><p>Search Engine Discovery not allowed</p></figcaption></figure>

Confirm that you want to prevent Search Engine Bots Discovery.

<figure><img src="https://2103450200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MKew-XtJKAPabdeTxXg%2Fuploads%2F03rbmi70A8B1AofiYS9v%2FScreenshot_20240411_114246.png?alt=media&#x26;token=5758977f-ed87-475f-a0fa-0ab28e736bec" alt=""><figcaption><p>Allow Search Engine Discovery prompt</p></figcaption></figure>

Now **Search Engine Bots Discovery** setting tab will have an updated description.

<figure><img src="https://2103450200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MKew-XtJKAPabdeTxXg%2Fuploads%2F2sShmvirwjFj5oUwC57L%2FScreenshot_20240411_114310.png?alt=media&#x26;token=688fd23c-8136-490e-8cb5-978161d35923" alt=""><figcaption></figcaption></figure>

In the [Deployments](https://help.readymage.com/application-management/deploy-changes-to-your-instance) tab, you will be required to run the next deployment without a cache. Automatic deployment will also happen without a cache.

<figure><img src="https://2103450200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MKew-XtJKAPabdeTxXg%2Fuploads%2FBt81lbrNlwYR39wq9tUH%2FScreenshot_20240411_114542.png?alt=media&#x26;token=85b49873-9a70-4ffc-9e91-01e0d47f094e" alt=""><figcaption></figcaption></figure>

After successful deployment, the **Search Engine Bots** will be blocked from crawling, discovering and indexing your website.

## Allow Search Engine Bots to discover your website

To allow Search Engines to crawl, discover, and index your website you must navigate to **Portal > Selected Project > Selected Environment > Application Management > Search Engine Bots Discovery** tab and click **Allow Search Engine Bots**.

<figure><img src="https://2103450200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MKew-XtJKAPabdeTxXg%2Fuploads%2Fm4PN8S2Lgistr71yTzZF%2FScreenshot_20240411_114651.png?alt=media&#x26;token=7bb28648-0c1a-45c7-b911-e9d33642c528" alt=""><figcaption></figcaption></figure>

Confirm that you want to allow Search Engine Bots Discovery.

<figure><img src="https://2103450200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MKew-XtJKAPabdeTxXg%2Fuploads%2FFX3kT87GIFpTc8iqgRSV%2FScreenshot_20240411_114756.png?alt=media&#x26;token=a58e30d2-e517-4a76-8c94-2f8c22232b21" alt=""><figcaption></figcaption></figure>

Now **Search Engine Bots Discovery** setting tab will have an updated description.

<figure><img src="https://2103450200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MKew-XtJKAPabdeTxXg%2Fuploads%2Fjor0YE3bwipFmhq1AR3O%2FScreenshot_20240411_114840.png?alt=media&#x26;token=67f66882-be72-429a-bf54-520c50577691" alt=""><figcaption></figcaption></figure>

In the [Deployments](https://help.readymage.com/application-management/deploy-changes-to-your-instance) tab, you will be required to run the next deployment without a cache. Automatic deployment will also happen without a cache.

<figure><img src="https://2103450200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MKew-XtJKAPabdeTxXg%2Fuploads%2F6dxlKqBQ54MlOpIgZHmQ%2FScreenshot_20240411_114927.png?alt=media&#x26;token=e2f4ba96-4290-4f95-9d4e-3fec43cfe37a" alt=""><figcaption></figcaption></figure>

After successful deployment, navigate to **Magento Admin Panel > Content > Design > Configuration > Default Website > Search Engine Robots** and confirm that **Default Robots** is set to **INDEX, FOLLOW** and **Custom instructions for robots.txt** are either **empty** or contain correct instructions for Search Engine Bots.

<figure><img src="https://2103450200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MKew-XtJKAPabdeTxXg%2Fuploads%2F3O0y37m5QB4fvgar6wXT%2FScreenshot_20240327_171956-1.png?alt=media&#x26;token=7ce2542c-2367-4cbf-84ac-c67084db7a21" alt=""><figcaption></figcaption></figure>

Click **Save Configuration** and flush the invalidated cache type! That is it!