News about Aimy Sitemap

Aimy Sitemap v3.17.0 allows to disguise as a browser while crawling in the PRO version and adds a workaround for forced GZIP compression.

Disguise as Browser (PRO Version)

Some websites deliver different content based on the User-Agent or even block clients that are not listed in a whitelist based on this criterion. To work around this, Aimy Sitemap’s crawler now allows to disguise as a common user browser instead of using its own User-Agent.

Normally, Aimy Sitemap’s crawler identifies itself as:

Mozilla/5.0 (compatible; AimySitemapCrawler/3.17.0)

If the “Disguise as Browser” option is enabled, Aimy Sitemap’s crawler uses a Firefox identification string as User-Agent:

Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0

Workaround for Forced GZIP Compression

Compression is supposed to be enabled only if both client and server support it. However, in some setups compression is used by the server side even if the client does not indicate support.

Aimy Sitemap now detects those setups and contains a workaround to handle the forced GZIP compression (i.e. Apache’s force-gzip environment setting) as expected.