I am releasing a website which has 30 million pages of content. (This isn't spam material). It has a detail page for each of the 30 million places around the World.  e.g /mywebsite.com/id/country/city/street etc..  I assumed submitting a sitemap would get Google to index these pages, but I was mistaken, how can I achieve this?
444
We are currently developing a site that currently has 8 million unique pages that will grow to about 20 million right away, and eventually to about 50 million or more.  Before you criticize... Yes, it provides unique, useful content. We continually process raw data from public records and by doing some data scrubbing, entity rollups, and relationship mapping, we've been able to generate quality content, developing a site that's quite useful and also unique, in part due to the breadth of the data.  It's PR is 0 (new domain, no links), and we're getting spidered at a rate of about 500 pages per day, putting us at about 30,000 pages indexed thus far. At this rate, it would take over 400 years to index all of our data.  I have two questions:   Is the rate of the indexing directly correlated to PR, and by that I mean is it correlated enough that by purchasing an old domain with good PR will get us to a workable indexing rate (in the neighborhood of 100,000 pages per day). Are there any SEO consultants who specialize in aiding the indexing process itself. We're otherwise doing very well with SEO, on-page especially, besides, the competition for our "long-tail" keyword phrases is pretty low, so our success hinges mostly on the number of pages indexed.   Our main competitor has achieved approx 20MM pages indexed in just over one year's time, along with an Alexa 2000-ish ranking.  Noteworthy qualities we have in place:   page download speed is pretty good (250-500 ms) no errors (no 404 or 500 errors when getting spidered) we use Google webmaster tools and login daily friendly URLs in place I'm afraid to submit sitemaps. Some SEO community postings suggest a new site with millions of pages and no PR is suspicious. There is a , too, in order to avoid increased scrutiny (at approx 2:30 in the video).      Clickable site links deliver all pages, no more than four pages deep and typically no more than 250(-ish) internal links on a page. Anchor text for internal links is logical and adds relevance hierarchically to the data on the detail pages. We had previously set the crawl rate to the highest on webmaster tools (only about a page every two seconds, max). I recently turned it back to "let Google decide" which is what is advised.
29745
In Google webmaster tools it shows that I have 309 pages indexed, but when I search using "site:site.com" appear about 180 only (On the first page appears "about 608 results," something near the total number of posts).  I have also noticed that some pages that were indexed before have simply disappeared from Google. In Google webmaster tools there is no error message or anything to indicate any punishment by Google.  According to Google webmaster tools, daily new pages are indexed, but for some reason they are not displayed in the search.  Am I really affected by some punishment by Google or this is some kind of bug?  How can I solve this problem and have all pages indexed normally?
112581
We're launching new website with new url structure and new content. Domain is the same. Many old pages are indexed by google. Should I use 404 or 301 and redirect to '/' for these old resources? I also want to remove these pages from google index.
67528
Just out of curiosity, I have developed a small app analysing and generating sitemaps for a given domain.   Can I tell search engines to look for the sitemap of a given domain on my app using a dynamic url i.e. can you use cross-domain sitemaps.xml without validating you are owner of the destination domain?  For example :   http://example.com/mydomain.com/sitemap.xml
105604
How do get my sitelinks to appear on Google search results?
65364
I'm a web developer from Myanmar (Burma). Google delivers a ccTLD  to the people searching from our country.   I have a website which is a single language site (English) and it targets the global audience. I have also Facebook page and Twitter page related to my website. When I search my website using some related keywords in  (without ), my site including Facebook and Twitter pages are listed in the first SERP. However, when I search it with the same keywords in , I could not entirely find the site.  So far I have learned about Google SEO, there is an option  which can be enabled/flagged from Google Webmasters Tool. And I know that it is only for geographic data and it is not suitable for my case.     This setting is only for geographic data. If you're targeting users in   different locations—for example, if you have a site in French that you   want users in France, Canada, and Mali to read—don't use this tool to   set France as a geographic target. A good example of where it would be   useful is for a restaurant website: if the restaurant is in Canada,   it's probably not of interest to folks in France. But if your content   is in French and is of interest to people in multiple   countries/regions, it's probably better not to restrict it.   What are the best possible ways to solve this issue to list my site in google.com.mm as well?
70059
I've heard Google's algorithm is based on PageRank, and that it has something to do with the links to your site. Can someone explain it, and more importantly, how I can increase my site's PageRank?
63828
My website is javatechig.com and is around 6 months old. I have used All-in-One SEO WordPress plugin for SEO tricks. Definitely after using SEO plugin, my website page view is increased, but still it is not giving one good site-link in Google Search.   I have followed the following steps:   Created sitemap.xml and submitted to Google Webmaster Tools Edited all posts with meta tags Header tags are carefully taken care   I want to make a sitelink similar to the attached image:
115582
So I have just built a site and all is well... but I want to control what appears on google search results.  Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then underneath is a list of pages on that site with preview and  a link "More results from thefwa.com". How can I tell google to display my chosen content/pages? Do I use robots.txt or sitemap.xml? Because now the pages that are associated with my site are displayed kind of random.  Is it possible to tell google what to display?
83457
My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found).   But when I look at the source pages that are accused of having these broken links, there are no such links in them.  As of now, I have 6 404 errors in sitemaps and 28 in the rest of my site.  E.g. the address  does not exist, yet Google indicates that I have 3 pages that link to this address:  http://www.webdesk.co.il/sitemapgen.php http://www.webdesk.co.il/articles.php?t=css http://www.webdesk.co.il/articles.php?id=17 //this address dont even exist   If you open each of these pages, you'll see that they don't have these links in them.  I thought this may be a result of an old crawling that was not yet updated, but those pages haven't existed for over than 6 months and Google indicates that all the crawling results are up to date (Dec 22 2012).  Why do I keep getting these 404 errors by Google?
23174
I need to add Google Map with about 400 markers into WordPress site (into one page). Every marker needs to have info window with photo, description and link. It is really high number of markers, so I'm not sure what is the best way to implement that kind of map.   What do you suggest?
115038
I installed the  module and would like to customize the sitemap.  For example, I would like to exclude some nodes that have a certain term reference field. I would also like to be able to create separate sitemaps for each content type.  Is there a way to do that?  Otherwise I could probably generate the xml sitemaps with  module.
55048
Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked to from anywhere then it shouldn't get crawled.  For example, let's say I own example.com and I make a new subdomain with a weird  random name such as adayinthewoods.example.com and I throw a wordpress installation on there that I plan to use for testing purposes.  What would have to happen for google to start crawling and indexing this? Is google going to look at the whois records and see that I've added a subdomain in my DNS table and then start crawling it as a result? Is the fact I've installed wordpress on it making my installation "ping search engines"? How does that work? How do top level domains get crawled when it's a brand new domain? I assume the mechanism there is different from random subdomains.  What if I add a new page in my root folder called "noOneWillEverSeeThis.html", could that ever get crawled/indexed if it wasn't included in any sitemap and wasn't linked to from anywhere?  Would really appreciate a solid answer from someone that understands what's going on with this.  Thanks much
55169
I built a site with Angular 10. When the home page is visited it reroutes to this /welcome. However There's no link to that from anywhere within the app, so I don't think Google has any way of indexing that page. What would be the best way of telling Google to index the page.  From my research it seems I could do something like this: &lt;body&gt;   &lt;a class=&quot;hidden&quot; href=&quot;/welcome&quot; title=&quot;Welcome to example.com&quot;&gt;Welcome to Example!&lt;/a&gt;   &lt;app-root&gt;&lt;/app-root&gt; &lt;/body&gt;  What is the max number of links that this would work with and any other approaches?
27697
This is probably wishful thinking but here goes...  Now obviously I don't expect some magical app to crawl a website, summarise each page and detail all the assets within each page (if only!) however is there a crawler of some sort that will trawl a given website and produce a full set of links and respective hierarchy would be invaluable. That would ensure that no pages get missed out as they invariably would through a purely manual inventory.  From there I would be able to see the existing site hierarchies and taxonomy decisions that have been made when building the site, and could use this as the basis for creating a detailed website content inventory.  Whether this be an online application, script or even a useful manual technique other than just manually sitting at the site and following links would be great.
6476
It's been sometime and Google has yet to index my site. How can I get Google to index my website?
84712
I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has found these pages on my site and indexed them.  Further, I updated the site with a robots.txt file that WMT reports as being read, and testing the blocked pages using the WMT test tool indicates that the pages will not be indexed. Apparently this does not remove the pages from existing results, though I submitted a request to do so.  How did Google find these pages to begin with? Is it because I used a common subdirectory name and Google guesses at common subdirectories?