Skip to content
You are here:Home arrow Blog arrow Yahoo Webmap: Roughly 1 Trillion Links
Yahoo Webmap: Roughly 1 Trillion Links PDF Print E-mail

On a very related note, we're announcing today that we implemented what we believe is the world's largest commercial application of Apache Hadoop. We are now using Hadoop to process the Webmap -- the application which produces the index from the billions of pages crawled by Yahoo! Search.

Yahoo Implements Apache Hadoop To Process Webmap

More about Hadoop running in production on the Yahoo! Search Webmap

Some Webmap size data:

    Number of links between pages in the index: roughly 1 trillion links
      Size of output: over 300 TB, compressed!
        Number of cores used to run a single Map-Reduce job: over 10,000
          Raw disk used in the production cluster: over 5 Petabytes

          Very interesting interview by Jeremy Zawadony of two of the Y! engineers (since Inktomi days) about the Y! search infrastructure on that second linked page.
          1 Trillion links and the best Yahoo can return is:

          select top 10 *
          from links
          where id = newId()

          lol
          Any time Yahoo tries to boast about how good it is, I go and do a search for some basic stuff -- disappointed yet again with nothing but crap. Hey Yahoo guys, you are in trouble either way. All by yourself you are useless and with Microsoft you are marrying another loser in the search business.
          I think this would double the quality and split the Trillion in half though:

          select *
          from links
          where id not in(select id from links where link is not like '%.info%')

          .
          Yes, Maybe they can now clear old Inktomi penalties that they seem lost on how to go about doing as well.

          Featured Links:

          Google Payload
          Proven Money Maker At $ 9,547 A Day.
          Burn The Fat Feed The Muscle
          Burn The Fat - Top Selling Fat Loss Ebook Since 2003.
          Save My Marriage Today!
          Have You Ever Stayed Awake at Night Stressing About Whether or Not Your Marriage Will Last ... And What You Can Possibly Do to Save It?
          Ultra Hot* SpywareBot: #1 AntiSpyware
          Scan your computer for hidden AdWare and Spyware, Remove them permanently.
          Top Movie Downloads
          Join the Internet revolution and start downloading free movies and more!
          Get Google Pay-Per-Click Ads Free!
          Internet Marketer Gets Million in Google Pay-Per-Click Ads FREE! ... And Makes Over 4 Million as a Result! ...And Now He's Going to Give You This Same Secret for Next to Nothing!
           
          < Prev   Next >

          Main Menu

          Home
          Blog
          Search
          Links

          Webmaster

          Webmaster

          Latest News

          Popular