I own a website .  We did not have proper content before so to prevent search engine from crawling our site  we added a robots.txt with code to prevent crawling.  Now we had updated our sites content and want to get it crawled, but Google is not crawling it yet. We removed the code from robots.txt to prevent Googlebot to crawl, but still it is not getting crawled.   We almost did this change before 20-21 days but still Google is not crawling our site and if search I search on Google it displays:     A description for this result is not available because of this site's   robots.txt – learn more.   Kindly guide me, what I need to do for this?
122853
(Note that I am going to use screenshots here because I suspect writing about this will change the behavior over time.)  If you do a Google search for     uiviewcontroller best practices   either with or without the quotes, you end up with results like this:      Note that none of these pages resolve to the actual Stack Overflow question containing those words in the title. They resolve to either a) sites that are mirroring our creative commons data and correctly pointing back to the source question without nofollow, as properly specified by  or b) our own internal links to the question, but not the actual question itself.   The actual page with the title ...     Custom UIView and UIViewController best practices?   ... does exist at this URL ...  http://stackoverflow.com/questions/3300183/custom-uiview-and-uiviewcontroller-best-practices  ... and apparently it is present in Google's index!    But why does it not appear when we search for      uiviewcontroller best practices   ?   We know that   Google contains this page in its index Our search terms match the title of the question Stack Overflow has much higher pagerank than the other sites that are    I don't get it. What are we doing wrong here?
109807
There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not Found errors despite the fact that the pages are up and running. I didn't do any changes in header's crawler related meta tags and robot.txt neither.  Tried to submit a sitemap.xml but  for that too.  What causes this problem?
80898
I have a friend who had around 2000 visitors per day to his website before a web designer with no SEO knowledge changed all his URLs. I've used the Wayback Machine and Google Analytics to determine the old URLs and I've done a 301 redirect and checked that it redirects properly.  My question is how long does it take before Google notices the redirects and passes link juice? And will those long-lost backlinks which where pointing to 404 pages count in Google Webmaster Tools - will I see a higher number of backlinks?   And last question, will I get a higher PageRank?
71088
Considering upgrade of CMS to latest version- the process may take anywhere from 2 to 4 weeks.  May suggest - we need to work on live server, what we have planned:   take exact replica of site and move to a test domain, but on live server Block Google, Bing, Yahoo - User-agent: Google Disallow: /, User-agent: Bing Disallow: /, User-agent: Yahoo Disallow: / in robots.txt Will upgrade CMS and add functionality - will test the entire structure, check URLusing xenu and move on to configure the site on original domain   The process upgradation and new tools may take 2 to 4 weeks....  Concern is that despite blocking Google, Bing &amp; Yahoo through      User agent disallow    can still the URLof test site can be crawled by the search engines - if yes - it may hurt the original site as will read on as entire duplicate or is there any alternate way around.
102
I have a very stupid question. I developed a website that has only one page at the public html directory(index.php).  all content are dynamically loaded as different pages.   How can i SEO my pages for google?  i mean like the what we type in this site are .............i believe stored in database and search engine find these pages like any static pages...  Just a little tip would be appreciable.......!
77295
I will be building a (wordpress) web site. While I am developing, other team members will be pushing content.  I'd like to have it hidden from google while under development.  It will be hosted on godaddy.   I have thought of not pointing the domain name to it until live and using "",   or buying a  during development.    Or hosting dev site in a sub-directory ("/dev/") until ready and then moving it up a level.  If in the dev directory I'd add htaccess or robots.txt to not crawl.   Is any of this a bad idea?  Will google penalize for any of this - like search by IP and then associate that with the domain later on?  Any better ideas?
73777
Is there any way to remove a website from appearing in Google search results under an unwanted keyword?   About a year ago, I added a featured project to a company website. This page included the client's name - she requested we keep the project anonymous so we removed any mention of her name and any identifying details.  Today - when you Google her name, our website homepage comes up regardless of her name not being on our website. I've even checked Google's cached page.  I'm stumped as to why our site, without any matching keywords would appear above other relevant results.
27403
All of a sudden, without any change in the index.html, our site is showing up in Google search results (on multiple browsers and devices) with Google Preview text that, instead of giving a summary of our site, now says:     Your browser does not currently recognize any of the video formats available. Click here to visit our frequently asked questions about HTML5 video.     We have a YouTube embed on our homepage that does include an html5 parameter in the media URL link in the embed, but there are no issues playing the video with any browser and device we can find.  What would suddenly cause this text to appear in Google search results?   We are using the  tags as the metatags in the &lt;head&gt; of our webpage and do indeed have title and description tags.    To mimic the behavior of Googlebot, I installed Windows on a VM and got Chromium 41 running. I get the same Google Preview text with the html5 video message when the site comes up in Google search results using this browser.    When I view the site in Chromium 41, I see the YouTube embed and it renders on the page and plays with no issue, just as with all other browser and devices we've tested on. The only time we see this message is in the Google search results, not on our actual site.
20395
From what I learned, there's one way I could cause all URLs on a domain strictly serving images not to be indexed and that is with the x-robots-tag HTTP header. Now I check my logs and find out that google and even baidu are downloading the entire contents of the image URLs. I was hoping they'd stop downloading when they came across this line:  X-Robots-Tag: noindex, noimageindex   Either I formatted that line wrong (used wrong casing or wrong order of values or something), or search engines are just plain dumb and they just decide to download everything just to waste customer's money.  I looked into robots.txt and thought of using the noindex line but when I did, google complained about having no access to what they call an "important url" when it isn't important.  I don't want to block their IP's because I have text-based content on another domain running on the same server that I want them to index.  I'm tempted to offer search engines the equivalent of what users get if they requested the URL via the HEAD method (full headers but no actual content), but I might get penalized for content cloaking.   Is there something I can do to rectify this?
105712
I have a site where Google has already indexed about 1 million pages, and I want to change all the URL paths for SEO purposes, ensuring that we always have our keywords in the path.  Technically I know how to complete it (301 redirects, new canonical URL, update sitemap, etc.) but I'm wondering if Google will penalize the site because of it, or if links will disappear from search results until they are re-crawled?
103368
How does Stack Overflow work so well with search engines? It's really amazing; if I post a question, it'll show up in Google's search results within 15-30 minutes!  What are the tricks? sitemap.xml? Or, any other good methods?  Help me, and others too, if anyone knows a good technique/method.
299
This is a general, community wiki  question and answer pair intended to address any questions   concerning the reasons a site or specific site contents do not appear   in search engine results.      If your question was closed as a duplicate of this question and you   feel that the information provided here does not provide a sufficient   answer, please open a discussion on .   My site (or specific pages on my site) is not appearing in search engine results.  Why isn't my content indexed and what can I do about it?
81995
Sometimes when I search for something, there are plenty of links to sites with no content (those who only have the premade templates which the search terms), spammers, and leechers (as in copying content from other sites, like B r o t h e r s o f t who leeches on SoftPedia.com).  How come they have such a high ranking without any original content (furthermore, why are they not simply banned from the results)?  And also, is there some way of filtering them out? Like for example completely exclude a domain from search results?
104994
During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden decrease a sign there is a problem? Can this be just a temporary thing?    Update: The Time spent downloading a page (in milliseconds) graph remained steady.
48337
I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else.  I just discovered tonight that the page is already indexed in Google! There are no links to the site (I haven't even told anyone about the domain since I haven't done anything with it yet).  Is Google trawling WHOIS records or something?
88379
We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site has appeared on Google and is affecting our search results.  I'm trying to figure out how the pages got there and how to remove them. The pages have always had &lt;meta name="robots" content="noindex, nofollow"&gt; in the head.  I am kinda new at this but was under the impression this should prevent Google from showing my site in results. I am pretty sure the results appeared in google after I accidently copy pasted some codes from the staging site to the live site that contained links to pages on the staging site. If anyone can point me in the right direction to figure out what happened and prevent it from happening again would be much appreciated.  robots.txt looks like this:  User-agent: * Disallow: /calendar-2/action~posterboard/ Disallow: /calendar-2/action~agenda/ Disallow: /calendar-2/action~oneday/ Disallow: /calendar-2/action~month/ Disallow: /calendar-2/action~week/ Disallow: /calendar-2/action~stream/ #Begin Attracta SEO Tools Sitemap. Do not remove sitemap: http://cdn.attracta.com/sitemap/4035112.xml.gz #End Attracta SEO Tools Sitemap. Do not remove   I also tried adding an X-Robots-Tag header and submited the site to be re-crawled. Did that a few days ago and I still see no changes. Here are the HTTP headers according to "Fetch as Google":  HTTP/1.1 200 OK X-Robots-Tag: noindex,nofollow Vary: Accept-Encoding Transfer-Encoding: chunked Date: Sun, 24 May 2015 16:26:49 GMT Server: LiteSpeed Connection: close X-Pingback: http://trailheadpaddleshack.ca/staging1/xmlrpc.php Content-Type: text/html; charset=UTF-8 Link: &lt;http://trailheadpaddleshack.ca/staging1/?p=170&gt;; rel=shortlink   I am now faced with a bunch of results I need to remove from Google asap as they contain out of date information and are affecting our search results. Webmaster Tools has something for removal of a single URL but I am looking to remove the entire /staging1/ subfolder. Any tips?
22398
I recently switched my domain from  to , and added my new domain to my Webmaster Tools account. After a few days, I went there and checked the data, and I was a bit worried when I saw this:    How can I interpret this? Is Google really not indexing my site at all? I'm afraid that's the case, because when I google for it, only my old blogspot domain shows up!