A report failure often occurs when our crawlers are blocked by search engines. This happens when search engines mistakenly identify us as automated bots rather than human searchers, resulting in a temporary halt to data collection.
Although we employ various methods to bypass these blocks, certain reports are more prone to triggering them, particularly those with many similar search terms. This pattern is due to the ease with which search engines can detect repetitive or similar queries.
To address this, we automatically re-run any failed reports within 15 minutes. This brief interval allows us to avoid continuous blocking by giving the search engines a break before attempting another collection.
Additionally, we cache search results for 12 hours. If you re-run a report within this period, you'll receive the same results, and only the terms that failed initially will need to be retried. This approach minimizes the need for a complete re-run and focuses only on the unresolved search terms.
Comments
0 comments
Please sign in to leave a comment.