My site  suddenly disappeared from Google search and Webmaster Tools says it can't access robots.txt. Fetch as Google says "temporarily unreachable" for robots.txt and even for the home page. The robots.txt tester throws the below error:     You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file.   My website comes up fine in browser and even robots.txt comes up fine. What could be the reason?   My Robots.txt content:  User-agent: * Allow: /  User-agent: Googlebot Disallow: /Admin/ Allow: /   I have not moved hosts recently. It was working fine and it happened suddenly.
123690
When looking into it in WebmasterTools, I saw that there is a "Fetch as Google - Temporarily Unreachable Issue". It is for an image, and font.  How do I resolve this issue?
14699
I have done major updates to my website, and renamed all my URLs.  How can I tell Google that I have renamed these URLs and let Google refresh its index?  My website URL: http://www.pndmasr.com/  My sitemap: http://www.pndmasr.com/sitemap.xml  I have uploaded my sitemap via Google Webmaster Tools many times.  However, every time I search Google for "pndmasr", I still get results with old pages. I have waited more than 3 days but the problem remains. Any suggestions?  Is there a problem with my sitemap?
65117
If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?  Or can the lack of one be harmful?
49157
Before we launched our website, we had set up a "coming soon" page and google picked up the title and description from its contents. So the description in the search results said:     Coming soon! Visit example.org for updates.   It's been a few weeks since we launched our website. We've even created a sitemap and submitted it to Google. In Google Search Console, the pages have been crawled and all the pages are appearing as expected on Google, EXCEPT the homepage which is still not updated! The title and description of the homepage in google search results still says "coming soon.".***  I checked on bing too, but it appears to be fine there.  All other pages seem to be indexed fine on Google. I don't even have any crawl errors in my reports. So what seems to be the problem? I've already waited for 2 weeks.
101853
I recently received the following message from Google Webmaster Tools:     Dear site owner or webmaster of ,      [...]      Below are one or more example URLs on your site which may be part of a   phishing attack:            [...]   What I don't understand is that I never had a subdomain repair.gotgenes.com, but visiting it in the web browser gives an actual  My DNS is , which does not list a repair subdomain. My domain name is registered with GoDaddy, and the nameservers are correctly set to NS1.AFRAID.ORG, NS2.AFRAID.ORG, NS3.AFRAID.ORG, and NS4.AFRAID.ORG.  I have the following questions:   Where is repair.gotgenes.com actually registered? How was it registered? What action can I take to have it removed from DNSs? How can I prevent this from happening in the future?   This is pretty disconcerting; I feel like my domain has been hijacked. Any help would be much appreciated.
97466
This is kind of a weird request. For some reason whoever owns safeandbuy.com has pointed their domain at my IP address. The reason it's a problem is that I'm having all kinds of crawlers that are trying to crawl my site with that domain name.   Is there anything I can do about this?
20395
From what I learned, there's one way I could cause all URLs on a domain strictly serving images not to be indexed and that is with the x-robots-tag HTTP header. Now I check my logs and find out that google and even baidu are downloading the entire contents of the image URLs. I was hoping they'd stop downloading when they came across this line:  X-Robots-Tag: noindex, noimageindex   Either I formatted that line wrong (used wrong casing or wrong order of values or something), or search engines are just plain dumb and they just decide to download everything just to waste customer's money.  I looked into robots.txt and thought of using the noindex line but when I did, google complained about having no access to what they call an "important url" when it isn't important.  I don't want to block their IP's because I have text-based content on another domain running on the same server that I want them to index.  I'm tempted to offer search engines the equivalent of what users get if they requested the URL via the HEAD method (full headers but no actual content), but I might get penalized for content cloaking.   Is there something I can do to rectify this?
46548
I made a fairly simple .txt file I am getting the following errors   I find it very odd. The 2.txt is just the .txt with fewer than 50,000 results. Somehow it seems like Google can read the longer .txt file but not the shorter 2.txt file.
56185
I tried to connect to  and got an error message:      ERR_ADDRESS_UNREACHABLE.   Is this a known problem? Do you expect that the problem will be solved soon?
67730
I am a member of a writing website called HubPages. We are each given a subdomain on which to write articles and are also given access to Google Webmaster Tools for our subdomain.  Upon checking out my stats in Webmaster Tools, I see that Google is running into some 404 errors for two reasons:   I took down an article Google (or someone) got the url totally wrong.   I am essentially getting view-only access for webmaster tools (HubPages must have set it up to allow view only so that a user doesn't mess things up) so I can't submit a url removal request to Google.  Could this be hurting the SEO for my subdomain? If not, are there other problems that could occur for me or for HubPages by leaving these URLs as is and not submitting a removal request? Should HubPages be submitting url removal requests to Google for dead or incorrect urls?
65021
I just opened a URL to a question on programmers.SE and got the message:     But the question was locked, so I couldn't do what it asked me to do.  I consider this to be a very minor edge case bug and it's likely not worth investing developer time into it. But since this  I guess it's worth reporting even if it's just for making you aware of it.
49948
I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this: User-Agent: * Disallow: /email  Yet I recently noticed that Google still sometimes returns links to those pages in their search results.  Why does this happen, and how can I stop it? Background: Several years ago, I made a simple web site for a club a relative of mine was involved in.  They wanted to have e-mail links on their pages, so, to try and keep those e-mail addresses from ending up on too many spam lists, instead of using direct mailto: links I made those links point to a simple redirector /  script running on my own site.  This script would return either a 301 redirect to the actual mailto: URL, or, if it detected a suspicious access pattern, a page containing lots of random fake e-mail addresses and links to more such pages.  To keep legitimate search bots away from the trap, I set up the robots.txt rule shown above, disallowing the entire space of both legit redirector links and trap pages. Just recently, however, one of the people in the club searched Google for their own name and was quite surprised when one of the results on the first page was a link to the redirector script, with a title consisting of their e-mail address followed by my name.  Of course, they immediately e-mailed me and wanted to know how to get their address out of Google's index.  I was quite surprised too, since I had no idea that Google would index such URLs at all, seemingly in violation of my robots.txt rule. I did manage to submit a removal request to Google, and it seems to have worked, but I'd like to know why and how Google is circumventing my robots.txt like that and how to make sure that none of the disallowed pages will show up in their search results. Ps. I actually found out a possible explanation and solution, which I'll post below, while preparing this question, but  in case someone else might have the same problem.  Please do feel free to post your own answers.  I'd also be interested in knowing if other search engines do this too, and whether the same solutions work for them also.
63060
I have a bit of a problem. My client is running a Joomla 1.0 site and some content (text) was lost. There was no backup configured of the MySQL database. The content was lost approx. around the 16:th of December 2012. I tried finding the old site by using the Web Archive, but no cigar. The site is not even indexed.  I tried to retrieve it via Google cache instead. The cached page itself didn't have the content, but the preview screenshot of the website had the old content. I wrote everything I could read off the screenshot. However, there was content below the screenshot that could not be retrieved.  Here is what I typed into Google Search to find the old page:  site:gerami.se previous ass   Ass = assignments for those wondering ;)  I would really like to get the rest of the text that was below the screenshot when it was taken.  P.S: Whether this thread gets a solution or not, the moral of the story is still the same: Backup is your friend!
108765
I searched  for "Halting problem" and got redirected to a captcha. I'm not even modifying anything, just searching. What is prompting this?
116538
I am running a website but for some days i stopped it and put the under-construction page because the Index of admin page is visible to the outside world through the Google search. One of my friend told me that your websites index is visible and its one step away to access the password file and he shows me that very simply using the Google search.  How can i prevent this and i am hosting my site with a hosting company and i report about this to them but they simply replied to me still its secure so you no need to worry...  am i really don need to worry and continue my site with the visible index of admin page?
78217
I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act as an CDN and serve static content in order to improve speed. However, all of my content can be accessed via subdomain aswell, so Google has indexed it all and now I am dealing with duplicated content.  How could I deny access to crawlers for the subdomain baring in mind that I do not have different subfolder for subdomain, so I can't create a separate robots.txt file?
87596
How to disallow one domain path in robots.txt? (there is a module which allows to do that but it is better to avoid it because of performance issues) For example, onedomain.com/path1/* must be disallowed. Is it right to type like this in robots.txt (there is another domain twodomain.com for the same website):  Disallow: onedomain.com/path1/*  Twodomain path must be allowed to be crawled and Onedomain path must be disallowed to be crawled (both of them are pointing to the same path).