# Search Engine Bots Discovery

By default, this setting will **block** Search Engine Bots from crawling and indexing your website.

Crawling & indexing your website by Search Engine Bots should be done only on production instances so we recommend leaving this setting in a **prevented** state for all other instances.

## Block Search Engine Bots from discovering your website

To prevent Search Engine Bots from discovering your website you must navigate to **Portal > Selected Project > Selected Environment > Application Management > Search Engine Bots Discovery** tab and click **Block Search Engine Bots**.

<figure><img src="/files/DeElTmX3pJaTpodEHgZ2" alt=""><figcaption><p>Search Engine Discovery not allowed</p></figcaption></figure>

Confirm that you want to prevent Search Engine Bots Discovery.

<figure><img src="/files/1f7HJA1ArjFiEUOIzMS2" alt=""><figcaption><p>Allow Search Engine Discovery prompt</p></figcaption></figure>

Now **Search Engine Bots Discovery** setting tab will have an updated description.

<figure><img src="/files/2GQxW9Kgk6jojjHZFMeN" alt=""><figcaption></figcaption></figure>

In the [Deployments](/application-management/deploy-changes-to-your-instance.md) tab, you will be required to run the next deployment without a cache. Automatic deployment will also happen without a cache.

<figure><img src="/files/NewwaScYjfuQqTlpszd8" alt=""><figcaption></figcaption></figure>

After successful deployment, the **Search Engine Bots** will be blocked from crawling, discovering and indexing your website.

## Allow Search Engine Bots to discover your website

To allow Search Engines to crawl, discover, and index your website you must navigate to **Portal > Selected Project > Selected Environment > Application Management > Search Engine Bots Discovery** tab and click **Allow Search Engine Bots**.

<figure><img src="/files/lqOIuf5ptdhfhCais8rA" alt=""><figcaption></figcaption></figure>

Confirm that you want to allow Search Engine Bots Discovery.

<figure><img src="/files/LSbKVeldY64L7jmrsz5d" alt=""><figcaption></figcaption></figure>

Now **Search Engine Bots Discovery** setting tab will have an updated description.

<figure><img src="/files/dK1xoSpTxQ26FTQfyosD" alt=""><figcaption></figcaption></figure>

In the [Deployments](/application-management/deploy-changes-to-your-instance.md) tab, you will be required to run the next deployment without a cache. Automatic deployment will also happen without a cache.

<figure><img src="/files/5FRyVIdpQ7KAkHu3Ul0q" alt=""><figcaption></figcaption></figure>

After successful deployment, navigate to **Magento Admin Panel > Content > Design > Configuration > Default Website > Search Engine Robots** and confirm that **Default Robots** is set to **INDEX, FOLLOW** and **Custom instructions for robots.txt** are either **empty** or contain correct instructions for Search Engine Bots.

<figure><img src="/files/EJqozpbYI0uCTfgrywqX" alt=""><figcaption></figcaption></figure>

Click **Save Configuration** and flush the invalidated cache type! That is it!


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://help.readymage.com/application-management/search-engine-bots-discovery.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.