User-agent: * Disallow: /vendor/ Disallow: /storage/ Disallow: /resources/ Disallow: /config/ Disallow: /node_modules/ Disallow: /admin/ Disallow: /login/ Disallow: /register/ Disallow: /private/ Disallow: /tmp/ Disallow: /cache/ # Explicitly allow Facebook crawler User-agent: facebookexternalhit Allow: / # Optional: allow major bots User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # Sitemap Sitemap: https://yourdomain.com/sitemap.xml