SEO 101: Crawling, Indexing, and Ranking

Ever wonder how Google's algorithm works? In essence, Google's algorithm is a system for crawling and indexing, and ranking websites. Websites must be optimized to satisfy each aspect of this system.

Viewing these three functions separately offers a helpful view of exactly why algorithms--and SEO firms--exist.

Crawling

The SEO world often calls web crawlers "spiders,"and for good reason: This is the simplest way to envision the work of crawling.

Spiders crawl the Internet collecting information from websites. If a certain website links to another page on the same site, or a to a different site, the spider follows the link, collecting more information. In this way, the spider creates an information web.

This web is then sent back to Google for indexing.

However, not all websites are "crawled." Spiders cannot crawl webstes with certain development or design errors. This is why it's vitally important to follow today's web development standards, and why, I believe, it's equally important for your developer to understand SEO basics.

Read: "Website Development: The Perfect Job for Spock." 

Beyond assuring your website is crawled, a website developer with knowledge of SEO may make "granular choices" to refine how it is crawled. As Google writes here:

"Most websites don’t need to set up restrictions for crawling, indexing or serving, so their pages are eligible to appear in search results without having to do any extra work. That said, site owners have many choices about how Google crawls and indexes their sites through Webmaster Tools and a file called “robots.txt”. With the robots.txt file, site owners can choose not to be crawled by Googlebot, or they can provide more specific instructions about how to process pages on their sites." 

At the least, your website developer should know about these choices.

Indexing

The information the crawlers collect needs to be organized. This is the function of indexing. This process is self-evident: Once Google receives information from the crawler, it creates an index, much like the back of a non-fiction book, so that it can easily retrieve the information for future use. 

As Google says:

"The web is like an ever-growing public library with billions of books and no central filing system. Google essentially gathers the pages during the crawl process and then creates an index, so we know exactly how to look things up. Much like the index in the back of a book, the Google index includes information about words and their locations. When you search, at the most basic level, our algorithms look up your search terms in the index to find the appropriate pages."

SEO can have a profound influence on how a website is discovered by the crawlers, but indexing is purely a a Google function.

Ranking

Ranking is the end-result of crawling and indexing. Once the information is gathered, and sorted, it is ranked to the benefit of Google's users. In ranking, Google places special emphasis on the browser's experience--known in the SEO world as "user experience."

Attentiveness to crawling and indexing is an important function of SEO, but attentiveness to Google's many ranking singles sets the best SEO specialists apart.

Brian Dean from Backlinko has a very helpful list of the "complete list" of Google ranking factors. Some of this list is speculative (Google is stingy about offering ranking single information), but much of it is spot-on. Some factors are more important than others; and some matter little, although it's good to be attentive to every one.

Read our post on Google's confirmed top three ranking signals: "Links, Content, and RankBrain."

Optimize User Experience With Stepman's SEO! 

To build an effective, fully-optimized website, you need a web design and development company that understands both UX and SEO.

Stepman's SEO is the rare company that offers a host of SEO and online marketing professionals to optimize your website.

Contact Stepman's SEO today to learn how you can improve your website's performance: 215-900-9398.