- Crawling
- Index
- Algorithm
- Displaying results
CRAWLING
Crawling is the technical process that tracks down the latest information. In the SEO terminology the term “crawling” means “following your links” using the file called as robots.txt. Crawling is simply defined as following all the links of the web page to fetch information from web pages using the software called “crawl”, which is also called as “spider”, “bot”. This software developed by Google fetches information from web pages. The information that the crawl collects like domain names, URL, descriptions of the data, keywords, images, videos, audios etc. There is any number of sites and pages being generated, is it possible for a human to collect, record and organize them, absolutely no right? these crawlers had made this job easier. I suppose you have created the new page or website and links to your earlier website main menu, then that work of linking is of crawls. As soon as the bots get the signal about this new page, it visits your page and indexes it. There are few platforms such as WordPress that will alert the search engine automatically above your creating the new page. The crawlers are in the form of “Document Object Model” (DOM) which is written in the HTML and JAVA code. The thing to remember here is, every website you create will not be crawled by the Google, instead, it only crawls the trusted websites that serve proper content.- As the crawling process involves of interacting with thousands of web serves, it is considered as the most fragile application.
- In this application, the spiders or crawlers are used to create a copy of the visited websites and their data links for later processing by a search engine.
- The crawlers can also be used to maintain the tasks on a website such as, checking links, validating HTML codes etc.
There are few features that the crawler provides
Robustness
There are few websites that create the “crawler traps”, which are generated that misleads the crawlers in getting trapped fetching any number of pages in a particular domain. So the crawlers are to designed in such a way that they are not affected by such traps.Politeness
The web servers have some policies regulating the rate at which a crawler must visit them. So the politeness policies must be respected.Quality
The crawler must fetch the web pages first that are useful without consisting any poor information.Extensible
The crawler must be extensible so that it could be adapted to new data formats and protocols.Efficiency
The performance of the crawler must be efficient and speed.INDEXING
The indexing process is a process of creating an index for all the data that is been collected through crawling and keeping them into a giant database from which when the users search for anything, the data from this data is retrieved and shown in the results that are suited for the keyword searched. The index of the google is similar to the index as the book. The google’s index has about hundreds of billions of web pages.- Once the webpage is detected by the crawlers, that particular data is stored like said above and indexes in such a way that makes easier when a user searches.
- Accordingly, that is related to the user’s question
- Ranked according to the relevancy
For example, let’s us consider for “Latest songs”
The index(storing of data like in list) is made for separate keywords, it means the index is made for “latest” in documents and “songs” in other documents. When both keywords are searched as one, the results have shown consists of data from both the documents. Not only storing, the data must also be sorted based on ranking or scoring according to the relevancy of the user’s searched keyword. Here are some of the factors on which the ranking is based,- The page rank
- Based on authority a trust of the webpage.
- The recurrence of the keywords, sentences or may be synonyms of the keyword that occurs on the page.
RANKING
- The ranking is considered as one of the major areas of the search engine optimization.
- The next step which is the last step is the displaying the results. Once the crawling and index processes are over, the results are shown on the basis of the user’s search. It retrieves the data from above processes and displays the best-matched results in the browser as per the search query by determining what each website is about and how it should rank. But, there is something you need to do to make your website show in the search engine results. So, for this, you have to make sure search engines can crawl and index your site correctly, otherwise your website content will not be shown in the search results appropriately.
