For site owners

PraykBot — the Prayk.com web crawler

PraykBot reads public listings on Azerbaijani classifieds and online-store websites so that buyers can see the whole market in one search. Every listing is shown with a link to its original source. This page explains how the crawler works, how to recognise it and how to control it.

  • robots.txt name: PraykBot
  • At least 0.7 s between requests
  • PraykBot rules are applied automatically

How to recognise the crawler

PraykBot identifies itself openly in every request: the User-Agent string contains the crawler name, the address of this page and a contact e-mail.

User-Agent

PraykBot/1.0 (+https://prayk.com/bot; [email protected])
  • The crawler’s name in robots.txt is PraykBot (case-insensitive).
  • Other software may also call itself PraykBot. If you are unsure a request really came from us, send us the IP address and time — we will check and reply.

What PraykBot does

  • Reads public listing and product pages — visible to anyone without signing in — as well as catalogue feeds and sitemap files.
  • Collects the listing title, price, description, photos, category, location and the contact details shown publicly in the listing.
  • On Prayk every such listing carries a “Listing source” block: the source site’s name and a direct link to the original.
  • Re-checks listings regularly: a listing removed at the source is deactivated on Prayk as well.

What it does not do

  • Does not sign in, use logins or passwords, or enter personal account areas.
  • Does not collect paid content or content available only to registered users.
  • Does not fill in forms, post listings or message your site’s users.

Load on your site

  • Requests are sent sequentially, with a pause of at least 0.7 seconds between them.
  • On sites that specify Crawl-delay in robots.txt the pause is increased accordingly.
  • On a server error (5xx) a request is retried at most once; 4xx responses are not retried.
  • If your site applies rate limits or you notice load, write to us — we will slow down.

Control via robots.txt

Rules addressed to PraykBot are read automatically at the start of every check and applied — disallowed URLs are never requested.

To block the crawler from your whole site:

User-agent: PraykBot
Disallow: /

To block only certain sections (example):

User-agent: PraykBot
Disallow: /profile/
Disallow: /*?sort=
Allow: /

Supported: Allow, Disallow, * and $ (the longest matching rule wins; on a tie, Allow wins). General “User-agent: *” rules are reviewed separately whenever we connect a source. If you want listings already collected from your site removed from Prayk, write to us — we will stop the source and remove those listings from the catalogue.

Official partnership

Our goal is to show listings with the consent of the source sites. This is how we formalise cooperation with site owners:

  • Consent — written permission to show your site’s listings on Prayk with a link to the source.
  • Feed or API — instead of reading HTML pages we pull from an XML/JSON feed you provide; load on your site drops to almost zero.
  • Rate and scope — we agree on check frequency and which sections are included together.
  • Traffic — every listing links to your original page, so buyers come back to your site.
  • Stop at any time — if you withdraw consent, the source is stopped and its listings are removed from the catalogue.
Write to us by e-mail · [email protected]Contact form

Last updated: 27 September 2026