Every Friday, Remora's Chief Marketing Officer (CMO), Steve Seeley, is here to answer any questions you might have related to your dealership's digital presence. From tracking data in Google Analytics to website optimization and running campaigns in Google Ads, no question will go unanswered. With this in mind, let's go ahead jump in to this week's edition of FAQ Fridays.
Here's a question for your dealership: Can you tell the difference between website crawling, indexing, and ranking? If you're drawing a blank or feeling stumped, don't worry; most dealerships (and plenty of vendors and service providers) aren't able to properly explain the nuance found between these tools of the trade employed by Google and the rest of the search engine world either.
However, just because plenty of your dealership's peers are in the dark regarding this topic, that doesn't mean that you should be too. With this in mind, feel free to join me as I break down all of the intricacies of how crawling, indexing, and ranking work — and how this information can change how your dealership handles its website and greater digital presence moving forward
For this explanation, I will use Google as an example. Google uses what they call a "spider" to scour the web and follow every link it can find. This is also referred to as Googlebot because that is the user-agent that is shown in your server log files so you know when and what they were looking at on your domain. When the spider crawls a page, it analyzes all of the code to find any other possible links. When any link is found on the page, the spider will then go crawl those pages as well. The search engine spider does not actually click on anything on the page. It renders the page through the DOM, otherwise known as the Document Object Model. This is how websites get found online, and once a single page is crawled (as long as the site structure is set up correctly), Google will crawl every page it finds and potentially add it to its index.
If you want your dealership site crawled and indexed by Google, the spiders need to be able to view your site correctly. In order for any page to rank in Google (remember, they rank pages and not sites) it must be crawled first. Some websites may hit their crawl budget, which is the allocated amount of pages that Google’s spider will review on your website. If you have too many pages or URLs that are simply repetitive (think duplicate content), you may hit your crawl budget.
I will discuss the Google crawl budget another time, as most car dealership websites will not have to worry about this scenario — unless you use one particular approved original equipment manufacturer (OEM) vendor.
After the spider has crawled your pages, it will then add them to Google’s database to possibly use later. Each page is cataloged and organized with the full text being stored in a database. This is how the search engines are able to show you the cached version of a page.
Other information stored on your pages include the topic of the content and its relevance. It also has a full trail of all of the pages that are interlinked to that specific page, as well as where certain information is located on the page.
For the most part, Google — and other search engines like Bing, Yahoo, and DuckDuckGo — will spider and index your web pages on their own. Almost all pages are within a search engine's index, and you can check to see if your individual pages have been indexed by doing a site command like this one:
You can also do this for any other page you want to check as well. Simply put in the full URL to the page that you want to check the index status of within your preferred search engine's search bar.
It is important to note that not all pages will be indexed. Here are a few examples of how a page, or even an entire website, can be removed from Google’s index:
- Disallowed within the robots.txt file (Not all search engines honor this)
- "Noindex" meta tag on a given page
- Page has a different canonical URL set
- Manual or algorithmic penalty
As you know, this database is massive in size and many, many years ago Google had separate data centers in the US that would show different search results based on which data center it was pulling from to fulfill the search query in question.
Before Google decides which pages to show from its index, it will put those pages through its algorithms to determine where those pages rank. This allows Google to figure out what each page is about, as well as what types of queries it will show for going forward. A lot of people think that indexing and ranking are one and the same. However, after getting this far, your dealership now knows that these processes are actually completely separate functions of Google's search engine operations.
How Does It All Come Together?
The first thing that happens is the algorithm will determine the intent of the search being performed. It will then identify all of the pages within its index that are related to the query and the user's intent. Lastly, Google ranks those indexed pages based on their relevance and overall authority. This is all leads to a user interacting with a search engine result page, or what is also commonly called the SERP.
There are a lot of varying factors that go into where a particular web page ranks, and that is why search engine optimization (SEO) is so important. Many people do not realize just how much work is going on behind the scenes when they perform a search query on a search engine. You are not searching the Internet, you are searching that search engine's index of data. This is why each search engine is different; they have different algorithms and they display their indexed data differently when ranking the results.
Naturally, this is a lot to take in all at once. Even so, arming your dealership with a true understanding of how the search engines crawl, index, and rank web pages is essential to ensuring that you truly get the best return from your car dealer website provider and other vendor services. From here, you can move forward with confidence knowing that the nuts and bolts of things like how a SERP is generated or having your new pages indexed doesn't have to be so mysterious or confusing.
Do You Have Specific Website or Marketing Related Questions You Would like Answered?
Have a burning question about your dealership's website or online marketing endeavors that you would love to see answered in this weekly feature? Then be sure to send in this query via our contact form and there is a good chance it will be featured on an upcoming FAQ Friday post.