An Industry Built upon Search (part 1)
The past decade has seen the greatest marketing phenomenon of all time: the Web. It has forever altered the way people communicate and deliver information, products and services, not just locally but across international borders. It is truly a communications superhighway with some 14.6% of the 6.5 billion world population, almost a billion souls, now using the Web - currently holding some 8 billion identifiable and indexed pages.
During this time the Web has blossomed from isolated static pages of scientific papers to billion pound revenue dynamic marketing behemoths driven by teams of professional web developers and marketers whose collective duties culminate in a web presence made visible in the search engines.
At the time of writing, Autumn 2005, three search engines have stood the test of time - Google, Yahoo! And MSN - and offer millions of webmasters the world over unprecedented marketing opportunities to promote their clients’ products and services. Others have either withered and died or now dance to new masters - such as Inktomi and Overture, absorbed by Yahoo!. Certainly, the likes of Alta Vista, Lycos, AllTheWeb, Ask Jeeves and numerous market-specific search engines remain active but their market shares are miniscule compared to the heavyweights.
Website marketing is huge online business, accounting for multi-billion pound annual sales, maturing along with the Internet to embrace not only the mechanics of search engine optimization but evolving into an eminently viable professional marketing medium.
Web Searching - The Early Years
By the early ‘90s millions of documents populated the Internet. Most were largely academic papers submitted by learning institutions. It was becoming evident the Web was growing fast and some method of identifying, archiving, cataloguing and categorising the ever growing content was required. The World Wide Web, as we now know it, was yet to be born but the proliferation of disparate information was soon to be corralled as first Archie and Veronica then Gopher indexed and transmitted requests for specific information from the first Internet document database. 1993 saw the first spider, World Wide Web Wanderer, actively seeking and cataloguing web pages.
Much of the initial research and development was undertaken by Universities and other academic establishments but big business was scented and by 1994 Excite bought WebCrawler, used by AOL for its NetFind search system. The Web was being indexed.
There were problems, though. The proliferation of documents and disparate naming conventions meant indexing was unreliable and uninformative. In an effort to combat this the first human editable directory, Yahoo!, was born, supplementing a file name with a text description. But despite much innovation in categorisation and with the advent of newer, faster web delivery technologies the problem of returning search relevancy remained.
Then along came Google in 1998 with it’s PageRank (PR) algorithm for calculating and weighting web content. Developed by Sergey Brin and Larry Page while attending Stanford University, PageRank is a measure of popularity based upon web page linking, an attempt to harness the apparent democracy of the Web; the more sites pointing to a particular site, the higher its PR. The practicalities are more complex but the principle remains: a quality voting system for website content.
PageRank alone did not resolve the relevancy challenges but it did prove a strong indicator of content authority, or topic accuracy. Research into data structures helped Google to grow and more efficiently and accurately index information through use of hundreds of thousands of PCs until today it sits at the pinnacle of web search technology, delivering fast, up-to-date, relevant results in its SERPs (search engine results pages).
To this day, great emphasis is placed upon authority sites whose information content is deemed accurate, of import or generally informative within its peer structure or market arena.
Big Business
The largely ungoverned nature of the early Web meant anybody with a connection and space to host a website could deliver Internet content, and it didn’t take long before life’s entrepreneurs realised a nice little earner could be made from examining how search engines operated and indexed web pages; search engine optimization was born – although the techniques used in the mid ‘90s were far less subtle than today’s more elegant efforts, mostly because website cataloguing was rudimentary and the fledgling engines looked for little else than keywords at the head of a document from which to deduce content relevancy.
The engines responded promptly to this manipulation, tightened their algorithms and generally learned more about their own craft of document indexing, making it increasingly more difficult for the unscrupulous to contrive search results. But the web was recognised as a lucrative marketing tool and the intervening years have seen the growth of website optimization as a professional, respected discipline
