10. Gzip Compression: Enable Gzip compression on the server to decrease the dimension of files transferred concerning the server plus the user’s browser. txt file is then parsed and can instruct the robotic concerning which internet pages aren't to get crawled. To be a search engine crawler could hold a https://www.youtube.com/watch?v=7cBHAtTOxoI