US9152678B1 - Time based ranking - Google Patents

Time based ranking Download PDF

Info

Publication number
US9152678B1
US9152678B1 US14/563,911 US201414563911A US9152678B1 US 9152678 B1 US9152678 B1 US 9152678B1 US 201414563911 A US201414563911 A US 201414563911A US 9152678 B1 US9152678 B1 US 9152678B1
Authority
US
United States
Prior art keywords
time period
click
query
search result
content items
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US14/563,911
Inventor
Philip A. McDonnell
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Google LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google LLC filed Critical Google LLC
Priority to US14/563,911 priority Critical patent/US9152678B1/en
Assigned to GOOGLE INC. reassignment GOOGLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MCDONNELL, PHILIP
Application granted granted Critical
Publication of US9152678B1 publication Critical patent/US9152678B1/en
Assigned to GOOGLE LLC reassignment GOOGLE LLC CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: GOOGLE INC.
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2457Query processing with adaptation to user needs
    • G06F16/24578Query processing with adaptation to user needs using ranking
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/951Indexing; Web crawling techniques
    • G06F17/3053
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2458Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2477Temporal data queries
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/29Geographical information databases
    • G06F17/30241
    • G06F17/30551
    • G06F17/30864

Definitions

  • This document relates to ranking content items responsive to a query and based on when the query is received.
  • the rise of the Internet has enabled access to a wide variety of content items, e.g., video and/or audio files, web pages for particular subjects, news articles, etc.
  • Content items of particular interest to a user can be identified by a search engine in response to a user query.
  • search engine is the Google search engine provided by Google Inc. of Mountain View, Calif., U.S.A.
  • the query can include one or more search terms or phrases, and the search engine can identify and, optionally, rank the content items based on the search terms or phrases in the query and present the content items to the user (e.g., in order according to the rank).
  • Queries are sometimes ambiguous with respect to the interests of the users. For example, queries for “turkey” can return results related to the country of Turkey and results related to turkey recipes for Thanksgiving holidays in Canada and the United States. Accordingly, users may have to revise their queries one or more times until the search engine returns the content items that are of interest to the user.
  • Apparatus, systems and methods for content item ranking are disclosed.
  • click logs and query logs related to a query are analyzed.
  • Historical click-through rates of content items are determined based on the analysis of the click logs and query logs.
  • a temporal change during a historical time period for an historical click-through rate of a first content item relative to an historical click-through rate of a second content item is identified.
  • a query can be received during another time period. If the another time period is related to the historical time period, the query can be disambiguated based on the temporal change.
  • click logs and query logs related to a query are analyzed.
  • Historical click-through rates of content items based on the click logs and query logs are determined.
  • Temporal changes for historical click-through rates of first content items relative to historical click-through rates of second content items for a historical time period are identified.
  • Search results received from a search engine for the query during a related time period and that include least some of the content items are adjusted according to the temporal change.
  • a system in another implementation, includes a click-through engine and a temporal analyzer engine.
  • the click-through engine accesses query logs related to queries and click logs related to content items and determines historical click-through rates for content items related to a query.
  • the temporal analyzer engine identifies a temporal change in the historical click-through rate of a first content item relative to the historical click-through rate of a second content item for a first time period.
  • the first and second content items are relevant to the query.
  • the temporal analyzer causes the rankings of search results for the query that include at least the first content item to be adjusted during a second time period according to the temporal change, wherein the second time period is related to the first time period.
  • FIG. 1 is a block diagram of an example online environment.
  • FIG. 2 is an example of a temporal change for click-through rates graph.
  • FIG. 3 is another example of a temporal change for click-through rates graph.
  • FIG. 4 is an example screen shot of search results.
  • FIG. 5 is another example screen shot of search results.
  • FIG. 6 a is a block diagram of an example content identification system.
  • FIG. 6 b is a block diagram of another example content identification system.
  • FIG. 7 is a flow diagram of a content ranking process.
  • FIG. 8 is a flow diagram of another content ranking process.
  • FIG. 9 is a flow diagram of another content ranking process.
  • FIG. 1 is a block diagram of an example online environment 100 .
  • the online environment 100 can facilitate the identification and serving of content items, e.g., web pages, advertisements, etc., to users.
  • a computer network 110 such as a local area network (LAN), wide area network (WAN), the Internet, or a combination thereof, connects advertisers 102 , an advertisement management system 104 , publishers 106 , and user devices 108 .
  • LAN local area network
  • WAN wide area network
  • the Internet or a combination thereof, connects advertisers 102 , an advertisement management system 104 , publishers 106 , and user devices 108 .
  • the advertisers 102 can directly, or indirectly, enter, maintain, and track advertisement (“ad”) information in an advertising management system 104 .
  • the ads may be in the form of graphical ads, such as banner ads, text only ads, image ads, audio ads, video ads, ads combining one or more of any of such components, etc.
  • the ads may also include embedded information, such as links, meta-information, and/or machine executable instructions.
  • One or more publishers 106 may submit requests for ads to the system 104 .
  • the system 104 responds by sending ads to a requesting publisher 106 for placement on or association with one or more of the publisher's content items (e.g., web properties).
  • Example web properties can include web pages, television and radio advertising slots, and even print media space.
  • One example publisher 106 is general content server 106 a that receives requests for content (e.g., articles, discussion threads, music, video, graphics, search results, web page listings, information feeds, etc.), and retrieves the requested content in response to the request.
  • the content server 106 a may submit a request for ads to an advertisement server in the system 104 .
  • the ad request may also include content request information.
  • This information can include the content itself (e.g., page, video broadcast, radio show, or other type of content), a category corresponding to the content or the content request (e.g., arts, business, computers, arts-movies, arts-music, etc.), part or all of the content request, content age, content type (e.g., text, graphics, video, audio, mixed media, etc.), geo-location information of a requesting device 108 , etc.
  • the content server 106 a or a client browser can combine the requested content with one or more of the ads provided by the system 104 .
  • the combined content and ads can be sent/rendered on the user device 108 that requested the content for presentation in a viewer (e.g., a browser or other content display system).
  • Another example publisher 106 b is a search service 106 b .
  • the search service 106 b can receive queries from the user devices 108 for search results.
  • the search service 106 b can derive relevant search results from an index of documents (e.g., from an index of web pages).
  • An exemplary search service 106 b is described in the article S. Brin and L. Page, “The Anatomy of a Large-Scale Hypertextual Search Engine,” Seventh International World Wide Web Conference, Brisbane, Australia and in U.S. Pat. No. 6,285,999.
  • Search results can include, for example, lists of web page titles, snippets of text extracted from those web pages, and hypertext links to those web pages, and may be grouped into a predetermined number of (e.g., ten) search results.
  • the search results can be provided to the requesting user device 108 .
  • the search service 106 b can submit a request for ads to the advertising management system 104 .
  • the request for ads may also include the query (as entered or parsed), information based on the query (such as geo-location information, whether the query came from an affiliate and an identifier of such an affiliate), and/or information associated with, or based on, the search results.
  • information may include, for example, identifiers related to the search results (e.g., document identifiers or “docIDs”), scores related to the search results (e.g., information retrieval (“IR”) scores, page rank scores, etc.), snippets of text extracted from identified documents (e.g., web pages), full text of identified documents, feature vectors of identified documents, etc.
  • IR scores are computed from, for example, dot products of feature vectors corresponding to a query and a document, page rank scores, and/or combinations of IR scores and page rank scores, etc.
  • the search service 106 b can combine the search results with one or more of the ads provided by the system 104 . This combined information can then be forwarded to the user device 108 that requested the content.
  • the search results can be maintained as distinct from the ads, so as not to confuse the user between paid ads and presumably neutral search results.
  • the search service 106 b may rank the search results based on keywords in the user's query. The search service 106 b may also consider user preferences and/or actions when determining the rankings. User preferences may be determined by different usage information provided to the search service.
  • the user devices 108 can provide usage information to the search service, such as, for example, whether a click-through related to a content item has occurred, whether a conversion has occurred, or whether some other defined event has occurred. This usage information can include measured or observed user behavior related to content items that have been served by the publisher. Such usage information can be processed to measure performance metrics, such as a click-through rate (“CTR”), conversion rate, etc.
  • CTR click-through rate
  • a click-through can occur, for example, when a user of a user device, selects or “clicks” on a link to a content item returned by the publisher or the advertising management system.
  • the CTR is a performance metric that is obtained by dividing the number of users that clicked on the content item, e.g., a link to a landing page, an advertisement, or a search result, by the number of times the content item was delivered. For example, if a link to a content item is delivered 100 times, and three persons click on the content item, then the CTR for that content item is 3%. Other usage information and/or performance metrics can also be used.
  • Users generally desire that the search results identify the content items that are most responsive to the user's query. However, what users actually consider to be responsive may fluctuate over time. For example, a particular content item may be determined to be highly responsive to users submitting queries on weekends, moderately responsive to users submitting queries on weekday evenings, but less responsive to users submitting queries on weekday mornings and afternoons, even when the same users are making the same queries at different times.
  • the search service 106 b can identify a temporal change indicative of changes in users' interests for one content item over another content item that are responsive to the same query.
  • a temporal change may be associated with a time period and/or a region.
  • a temporal change is identified by analyzing the CTRs of content items related to the query.
  • the temporal change may be a change in the CTR of a content item responsive to a query relative to the CTR of another content item responsive to the same query over a time period, and for which the change exceeds a threshold.
  • the ranking of the search results can be based in part on the temporal change.
  • FIG. 2 is an example of a temporal change for click-through rates graph 200 .
  • the CTR graph 200 represents the historical weekly CTRs over a one year time period of two different content items that are responsive to the same query. By comparing the CTRs of the two content items and determining whether a value generated by the comparison exceeds a threshold, temporal changes in the CTRs of the content items during the one year time period can be identified.
  • the CTR 202 line represents the CTR of the first content item responsive to a query over a one year period
  • CTR 204 line represents the CTR of the second content item responsive to the same query over the same one year period.
  • a query for “turkey” may return content items associated with vacation resorts in the country Turkey and content items associated with Thanksgiving turkey recipes. Although both sets of content items are responsive to a query for “turkey,” users may click on the content items associated with the vacation resorts in Turkey more frequently than the content items associated with the Thanksgiving turkey recipes.
  • historical time period 210 is representative of the majority of historical time periods shown on the graph 200 , with the exception of the historical time period 206 and the historical time period 208 .
  • the CTR 202 indicates the CTR of a content item associated with the country Turkey as averaging 0.36
  • the CTR 204 indicates the CTR of content items associated with Thanksgiving turkey recipes as averaging 0.05.
  • the CTRs of content items may change over time.
  • the change in the CTR may be a daily change, weekly change, seasonal change, one time change, or change during any other periodic or aperiodic time period for which the CTR is measured.
  • the CTR 204 line indicates that the CTR of a content item associated with turkey recipes increases significantly during the historical time period 206 and the historical time period 208 . This increase indicates that relatively more users are clicking on the content item associated with Thanksgiving turkey recipes during the time periods 206 and 208 , likely a result of the increased demand for turkey recipes during Thanksgiving and Christmas holidays.
  • the CTR 202 line indicates that the CTR of a content item associated with the country Turkey decreases significantly during the same time period 206 and period 208 , likely a result of the increased percentage of users clicking turkey recipes in the period 206 and period 208 .
  • the change in the CTR 202 and the CTR 204 is an example of a temporal change during the period 206 and the period 208 .
  • the temporal change may be identified by recognizing a change in a ratio of the CTRs of the content items that exceeds a threshold.
  • a click-fraction is used to quantify the ratio of the CTR of one content item relative to another content item. For example, a click-fraction quantifying the ratio of the CTR 202 to the CTR 204 during the historical time period 210 is:
  • a change in a click-fraction for a time period must exceed a minimum change threshold to be considered a temporal change.
  • the minimum change threshold may be calculated by a formula or may be provided by a person or another system.
  • the standard deviation of the click-fractions of the time periods may be the minimum change threshold. If the standard deviation for the click-fractions of all the monthly periods of FIG. 2 is 2.56, any change in the click-fraction during a time period that deviates from the average CTR by more than the standard deviation can be considered a temporal change.
  • the time period 206 is 2.56, any change in the click-fraction during a time period that deviates from the average CTR by more than the standard deviation can be considered a temporal change.
  • time period 212 the change in the click-fraction does not exceed the minimum change threshold of 2.56. Accordingly, the time period 212 is not considered to have a temporal change.
  • the difference between the change in the click-fraction and the minimum change threshold can also be used to determine the degree of the temporal change; the larger the difference, the more significant the change.
  • the graph 200 indicates that the CTR 202 and CTR 204 changed considerably during period 206 compared to during period 210 , while the change was less significant during period 212 compared to during period 210 .
  • the difference between the minimum change threshold and the change in the click fraction during period 206 , 6 . 5 is much greater than the difference between the minimum change threshold and the change in the click fraction during period 212 , which is 1.53.
  • the larger difference during period 206 is indicative of a larger temporal change in the CTRs of the content items during period 206 .
  • determining a threshold value can also be used. For example, an average of CTR's over multiple historical time periods can be determined, and an average of the click fraction (or some other central tendency of the click fraction) over the periods can likewise be determined. If the click fraction for a monthly period deviates from the average click fraction by a certain percentage, e.g., 20% of the average click fraction, then that time period can be determined to have a temporal change.
  • a certain percentage e.g. 20% of the average click fraction
  • a temporal change can be identified for a change in the click fraction comparing the CTR of one content item during a time period relative to the CTR of another content item during the same time period
  • a temporal change may also be identified for a change in the CTR of a content item during a time period relative to the CTR of the same content item during other time periods.
  • the CTR 204 changes significantly relative to the CTR 204 during period 210 .
  • This variation in the CTR 204 may be determined to be a temporal change during the period 214 .
  • FIG. 3 is another example of a temporal change for click-through rates graph 300 .
  • the graph 300 represents the daily CTRs over a historical three week time period of two different groups of content items that are responsive to the same query. By comparing the CTRs of the two groups of content items, temporal changes in the CTRs during the three week time period can be identified.
  • the graph 300 is similar to the graph 200 of FIG. 2 , except that the graph 300 is identified through multiple weekly periods, and the lines for CTR 302 and 304 correspond to groups of content items.
  • the CTR 302 line represents the CTR of the first group of content items
  • CTR 304 represents the CTR of the second group of content items.
  • a query for “bill” may return content items associated with a draft statute, called a “bill,” and content items associated with a cartoon character named Bill.
  • both sets of content items are related to a query for “bill,” during certain time periods users may click on the content items associated with the draft statute more frequently than the content items associated with Bill the cartoon character.
  • the CTR of content items associated with the draft statute, CTR 302 averages 0.33 and the CTR of content items associated with Bill the cartoon character, CTR 304 , averages 0.07.
  • the CTRs of the content items may change hourly, weekly, biweekly, monthly, etc.
  • a weekly change in the CTR 302 and the CTR 304 during the Monday time period 306 is of substantial enough magnitude to be considered a temporal change.
  • the temporal change during period 306 may arise because the cartoon is shown on television on Monday, which results in an increased interest in Bill the cartoon character during the period 306 .
  • This temporal change repeats weekly, and may be used to predict the next period of a temporal change, e.g. the time period 306 ′.
  • Identification of a temporal change during a historical time period can facilitate modification of rankings of search results responsive to a query during other time periods that are related to the historical time periods for which a temporal change is identified.
  • Examples of such other time periods are time periods that are of the same or substantial period as the historical time period, e.g., same days of week, same annual holidays such as July 4th holiday, Thanksgiving holiday, Christmas holiday, etc.
  • FIGS. 4 and 5 illustrate such a modification.
  • FIG. 4 is an example screen shot of search results 400 .
  • the search results 400 can, for example, be search results from a search engine in time periods not affected by a temporal change, e.g. a time period related to the historical time period 210 of FIG. 2 .
  • a search engine may rank content items related to vacation resorts in Turkey higher than content items related to Thanksgiving turkey recipes when the queries for “turkey” during the month of May.
  • the search engine may rank Country Turkey Web Page 404 as the highest content item in a query 402 for “turkey.” Additionally, the search engine may rank the next seven content items related to vacation resorts in Turkey, Country Turkey Web Pages 406 , 408 , 410 and 412 , and Turkey Country Ads 414 , 416 and 418 , higher than any content related to Thanksgiving turkey recipes.
  • FIG. 5 is another example screen shot of search results 500 .
  • the search results 500 can, for example, be search results from a search engine during a time period when the search results are ranked in response to a temporal change, e.g., during a time period that is related to the historical time period 206 of FIG. 2 , such as the month of November, or a week in November.
  • the search engine may rank Thanksgiving turkey recipe content items higher than content items associated with vacation resorts in Turkey.
  • Thanksgiving Turkey Web Page 504 may rank as the highest content item in a query 502 for “turkey.”
  • the search engine may rank the five of the next seven content items related to Thanksgiving turkey recipes, i.e., Thanksgiving Turkey Web Pages 506 , 508 , and 512 relatively higher than the rankings of the same content items during periods of no temporal change.
  • Such ranking adjustments can be implemented within a search engine.
  • one or more search engine variables related to ranking a content item in response to a query can be adjusted based on an identified temporal change.
  • the search engine generates rankings based on a product of a page rank and an information retrieval score
  • one or more of the IR scores and page ranks scores can be adjusted to increase or decrease the ranking of a search result during the period of the temporal change.
  • Other search algorithms can likewise be adjusted in a similar manner.
  • advertisement ranking may also be adjusted based on a temporal change.
  • Turkey Bird Ads 514 and 516 are ranked ahead of the Turkey Country Ad 518 .
  • Such adjustments can be based on relevance, e.g., if the advertisements are ranked, in part, on a relevance of the advertisement to a query, then the relevance (and thus the ranking) of an advertisement can be adjusted in a similar manner that the search results are adjusted.
  • Other methods of adjusting advertisement rankings can also be used, e.g., adjusting bid requirements, adjusting quality requirements, etc.
  • the search results can be received from the search engine and adjusted based on an identified temporal change. For example, the search engine may rank the search results without considering the time period during which the query was submitted.
  • the search results can be processed by a ranking filter that can re-rank some or all of the search results accordingly.
  • the ranking adjustments can correspond to the magnitude of the temporal change. For example, if the temporal change indicates a significant change in the CTRs of the first content items to second content items such that the CTRs of the first content items are higher than the CTRs of the second content items, such as during the time period 206 of FIG. 2 , then the first content items can be ranked higher than the second content items. Similarly, if the temporal change is less significant, e.g., the CTRs of the first content items relative to the CTRs second content items increase but are still less than the CTRs of the second content items, such as during the time period 208 of FIG. 2 , then the rankings of the first content items can be increased, but still remain less than the rankings of the second content items.
  • FIG. 6 a is a block diagram of an example content identification system 600 .
  • the system 600 can, for example, be used to distinguish content items responsive to the same query, based on identified temporal changes.
  • the system may be implemented as part of a search engine, or as a separate component or system that communicates with a search engine.
  • the system 600 can select a query from a query log and identify content items responsive to the query.
  • the system 600 can process click logs related to the identified content items served in response to the query and determine CTRs for the respective content items over a series of time periods, e.g., daily CTRs over a one-year period; weekly CTRs over a two year period, etc.
  • the CTRs of each of the content items during the time periods can be compared for each time period. If the change of a CTR for a content item relative to the CTR of another content item for a period exceeds a threshold, then a temporal change for those content items can be identified for the period.
  • the system 600 can then adjust the ranking of the content items for search results for the same query submitted during another time period that is related to the time period of the temporal change.
  • a CTR engine 610 accesses a query log 606 and a click log 608 to generate historical click-through data 612 for content items related to a query 602 .
  • the query log 606 stores the keywords used in the query 602 and in past queries.
  • the click log 608 stores the number of clicks different content items have received when identified in response to different queries and the number of times the content items have been presented as search results relevant to the queries.
  • the CTR engine 610 uses the data from the query log 606 and the click log 608 to generate the historical CTR data 612 for content items related to the query.
  • the query log 606 may store queries with the keyword “turkey”
  • the click log 608 may store the number of presentations to and the number of clicks received from the users for content items, e.g., content items that are associated with Thanksgiving turkey recipes and content items that are associated with vacation resorts in Turkey.
  • the CTR engine 610 may access the query log 606 and click logs 612 and generate historical CTR data 612 related to the query for the keyword “turkey.”
  • a temporal analyzer engine 614 analyzes the historical CTR data 612 for the different content items responsive to the query 602 and identifies temporal changes in the CTRs of the content items responsive to the same query during a time period. For example, the temporal analyzer engine 614 may identify an increase in the CTR for the content items associated with Thanksgiving turkey recipes during the month of November over several years. Similarly, the temporal analyzer engine 614 may identify a decrease in the CTR for the content items associated with resort locations in the country of Turkey during November over the same several years.
  • the temporal analyzer engine 614 may identify the month of November as a time period during which a temporal change for the two content items responsive to a query for “turkey.” Because the time period is substantially periodic, i.e., occurring during the month of November, the temporal analyzer engine 614 can determine that the temporal change is to be used to adjust the ranking of content items during the same period in the future, e.g., during each month of November.
  • the search engine 604 identifies content items responsive to a query, and then ranks the content items in response to the temporal change identified by the temporal analyzer engine 614 to provide the most relevant search results in ranked results 616 .
  • results related to vacation resorts in the country Turkey may generally be ranked higher than results related to Thanksgiving turkey recipes.
  • rankings of the content items can be modified.
  • the content items associated with Thanksgiving turkey recipes may be ranked relatively higher in the ranked results 616 during November to reflect the increased CTR of Thanksgiving related content items.
  • the adjustment can be implemented within the search engine 604 or external to the search engine 604 as described with reference to FIG. 5 .
  • the identification of a temporal change for a query and responsive content items and the adjustment of the search results for the time period of the temporal change can thus disambiguate the query, i.e., provide adjusted search results that are more likely to be related to the users' current interests.
  • Other methods of disambiguating a query can also be used.
  • the search engine 604 may rely on the CTR engine 610 and/or the temporal analyzer engine 614 to identify keywords to disambiguate the query, and provide the keywords to the search engine 604 with the query 602 .
  • the keywords may be identified by the temporal analyzer engine 614 or the search engine 604 as keywords related to the content items subject to a temporal change.
  • the temporal analyzer engine 614 determines that there is a temporal change in CTRs favoring a first set of content items, the first set of content items can be evaluated, e.g., text mined, to determine corresponding keywords related to the content items.
  • the temporal analyzer engine 614 can provide the search engine with additional keywords to include in the query, such as “recipe,” “Thanksgiving,” etc., when the query is received during the Thanksgiving time period.
  • an alternative query directed to the second set of content items can be optionally shown with the search results during the time period of the temporal change. For example, as illustrated by the dashed user interface element 520 , an alternate query related to the country of Turkey can be submitted to the search engine upon selection by a user, the processing of which would generate the search results of FIG. 4 .
  • the historical CTR 612 data can be processed for time periods of different durations.
  • the historical CTR data 612 may identify the CTR of a content item over a first time period, e.g., hourly, and can be aggregated into additional time periods, e.g., daily, weekly, monthly, etc.
  • the CTR engine 610 and/or the temporal analyzer engine 614 can identify temporal changes based on a lunar calendar, a Gregorian calendar, or other periods.
  • the historical CTR data 612 may identify a decrease in the CTR of content items associated with food for a particular query during the month of Ramadan, a Muslim month of fasting based on the lunar calendar, and likewise identify an increase in the CTR of content items for the query associated fasting during Ramadan.
  • the temporal changes that occur only on the day of Thanksgiving in the United States, i.e., the fourth Thursday of November in the United States, can also be identified and correlated to the occurrence of the holiday.
  • the historical CTR data 612 may also be stored for different geographical regions as historical regional CTR data.
  • the historical regional CTR data may identify the CTR during a time period of a content item in a city, state, country, or other regional categorization. If the historical CTR data 612 identifies different regions, the temporal analyzer engine 614 may identify temporal changes on a regional basis during a time period. For example, the historical CTR data 612 may identify an increased CTR for content items associated with Thanksgiving turkey recipes during November for queries received from the United States. Conversely, such a temporal change may not be identified for queries received from Japan in November.
  • the historical CTR data 612 may also include time zone data to facilitate the identification of temporal changes during a time period in a time zone of a content item.
  • a temporal change for content items associated with the query “eggs” might be identified between 7:00 AM and 8:00 AM for a time zone, and may occur across each time zone at the same respective time.
  • content items for egg recipes and restaurants with breakfast menus may be ranked higher than content items that include other information about eggs.
  • content items for egg recipes and restaurants with breakfast menus may be ranked lower than content items that include other information about eggs.
  • the CTR engine 610 and/or the temporal analyzer 614 can group content items with similar CTR behavior during a time period according to a category.
  • a category There are a number of ways that the category of a content item may be identified. For example, in some implementations, content items may be mapped to predefined categories by the search engine 604 , and if the majority of content items exhibiting a similar CTR behavior during a time period belong to a predefined category, then the content items can be identified as belonging to the predefined category. In other implementations, categories can also be determined by empirical data. A group of content items that have similar CTR patterns for the query “turkey,” can be categorized together by the CTR engine 610 and/or the temporal analyzer 614 .
  • rankings for all content items belonging to a category can be adjusted during a time period of a temporal change for a query.
  • the temporal analyzer engine 614 may identify a temporal change associated with all content items in a category of content items, where those content items are responsive to a query for “turkey” during November. For example, if a content item associated with deep fried turkey recipes is identified as being in the same category as a content item associated with a turkey stuffing recipes, both of those content items can be rank-adjusted based on a temporal change identified for the category.
  • the search engine 604 , CTR engine 610 and temporal analyzer engine 614 can be realized by instructions that upon execution cause one or more processing devices to carry out the processes and functions described above. Such instructions can, for example, comprise interpreted instructions, such as script instructions, e.g., JavaScript or ECMAScript instructions, or executable code, or other instructions stored in a computer readable medium.
  • the engines 604 , 610 and 614 can be implemented separately, or can be implemented as a single software entity.
  • the engines 604 , 610 and 614 can also be distributively implemented by multiple computers communicating with each other over a network, such as in a server farm, or can be implemented in a single computer device.
  • FIG. 6 b is a block diagram of another example content identification system 650 .
  • the system 650 is similar to the system 600 from FIG. 6 a , except the system 650 includes a ranking filter 618 that may be used to adjust the results of the search engine 604 to generate the ranked results 616 .
  • the search results can be received from the search engine 604 and adjusted by the rank filter 618 based on an identified temporal change.
  • the temporal analyzer engine 615 uses the historical CTR data 612 to identify a temporal change, which the temporal analyzer engine 615 communicates to the filter 618 . For example, when the temporal analyzer engine 615 identifies a temporal change in the CTR during November of content items related to Thanksgiving turkey recipes, the temporal change is communicated to the filter 618 .
  • the search engine 604 may continue to rank content items related to vacation resorts in Turkey higher than content items related to Thanksgiving turkey recipes, even though the temporal change identifies a higher CTR for Thanksgiving related content items during November.
  • the filter 618 may then modify the search results from the search engine 604 by adjusting the ranking the search results from the search engine 604 to reflect the temporal change in CTRs of the content items during the time period of the temporal change.
  • the filter 618 can analyze the search results received from the search engine 604 during a period related to temporal change to identify the content items having CTRs that define the temporal change.
  • the filter can then adjust the rankings during the time period related to temporal change so that content items that have a CTR increase are ranked higher than during a period of no temporal change, and content items that experience a CTR decrease are ranked lower than during a period of no temporal change.
  • the modified results can be ranked results 616 .
  • the search engine 604 may rank content items related to vacation resorts in Turkey higher than content items related to Thanksgiving turkey recipes.
  • a temporal change of the two content items responsive to a query for “turkey” during November may be identified by the temporal analyzer engine 615 .
  • the temporal change is communicated to the filter 618 , and the filter 618 can re-rank the search results from the search engine 604 during the time period of the temporal change so that content items for Thanksgiving turkey recipes are ranked higher than content items related to vacation resorts in the country Turkey.
  • the search engine 604 , CTR engine 610 , temporal analyzer engine 615 and the filter 618 can be realized by instructions that upon execution cause one or more processing devices to carry out the processes and functions described above. Such instructions can, for example, comprise interpreted instructions, such as script instructions, e.g., JavaScript or ECMAScript instructions, or executable code, or other instructions stored in a computer readable medium.
  • the components 604 , 610 , 615 and 618 can be implemented separately, or can be implemented as a single software entity.
  • the components 604 , 610 , 615 and 618 can also be distributively implemented over a network, such as a server farm, or can be implemented in a single computer device.
  • FIG. 7 is a flow diagram of a content ranking process 700 .
  • the process 700 can, for example, be implemented by the system 600 of FIG. 6 or the system 650 of FIG. 6 b.
  • Stage 702 analyzes click logs and query logs related to a query.
  • a server device such as the CTR engine 610 may access the query logs 606 and click logs 608 related to a query 602 for “turkey.”
  • all received queries are selected for evaluation.
  • only queries that exceed a submission threshold e.g., queries that are most frequently submitted, are selected for evaluation.
  • Stage 704 determines historical click-through rates of content items.
  • the CTR engine 610 may determine the historical CTR of content items responsive to the query 602 for “turkey,” such as content items associated with vacation resorts in Turkey and content items associated with Thanksgiving turkey recipes.
  • the historical CTR data 612 may be stored by different time periods, by category, by region, or by any other grouping specified by the temporal analyzer engine 614 or 615 .
  • Stage 706 identifies a temporal change for an historical click-through rate of a first content item.
  • the historical CTR of the first content item may be compared to the historical CTR of a second content item for a historical time period. If the historical CTR of the first content item has substantially changed relative to the historical CTR of the second content item during the historical time period, there is a temporal change in the CTRs during the historical time period.
  • the temporal analyzer engine 614 or 615 may identify the time periods when the historical CTR data 612 for “turkey” queries is indicative of a higher CTR for content items associated with Thanksgiving turkey recipes and a lower CTR for content items associated with vacation resorts in Turkey. Any such time periods are the period of a temporal change in the CTRs of the content items.
  • the historical CTRs of content items can be compared for all identified content items. In other implementations, the historical CTRs of content items having CTR values that exceed a minimum threshold can be compared to exclude underperforming content items. In other implementations, the historical CTRs of content items that have been served in excess of a minimum number of times over a period can be compared to exclude very unique content items. Other selection processes can also be used.
  • Stage 708 disambiguates the query for the first content item, i.e., provides adjusted search results that are more likely to be related to the users' current interests.
  • the temporal analyzer engine 614 or the filter 618 may adjust the results of a search engine 604 to rank content items related to Thanksgiving turkey recipes higher than content items related to vacation resorts in Turkey during another time period that is related to the historical time period having an identified temporal change.
  • the temporal analyzer engine 614 may alternatively provide or augment a query to the search engine that is directed to the content items related to the temporal change.
  • the temporal analyzer engine 614 may provide a query for “Thanksgiving turkey” for an initial query of “turkey” during another period associated with a temporal change, e.g., a month of November.
  • FIG. 8 is a flow diagram of another content ranking process 800 .
  • the process can, for example, be implemented by the content ranking system 600 of FIG. 6 a or the content ranking system 650 of FIG. 6 b.
  • Stage 802 selects a time period for analysis.
  • the temporal analyzer engine 614 or 615 can select historical hourly, daily, weekly, or monthly calendar time periods, or any other periodic or substantially periodic time periods. Multiple time periods can be processed, e.g., 52 time weekly time periods for a year of query-log data and click-log data.
  • Stage 804 determines click-through rates of the first content item responsive to a query.
  • the CTR engine 610 can determine the CTRs of a first content item during the time periods selected by the temporal analyzer engine 614 or 615 .
  • the CTR engine 610 may determine the weekly CTR of a content item related to Thanksgiving turkey recipes during every week for which there is historical CTR data.
  • Stage 806 determines click-through rates of the second content item.
  • the CTR engine 610 can determine the CTRs of a second content item during the same time periods selected by the temporal analyzer engine 614 or 615 , e.g., the CTR engine 610 may determine the weekly CTR of a content item related to vacation resorts in Turkey during every week for which there is historical CTR data.
  • Stage 808 generates content item click-fractions for each of the time periods.
  • the click-fraction may represent a ratio of the CTR of a content item associated with Thanksgiving turkey recipes to the CTR of a content item associated with vacation resorts in Turkey.
  • Stage 810 determines whether the content item click-fractions exceed a minimum change threshold. For example, the temporal analyzer 614 or 615 can identify if the weekly click-fraction comparing two content items responsive to the query for “turkey” is greater than the average click-fraction for the two content items stored in the historical CTR data 608 by an amount exceeding the minimum change threshold.
  • stage 812 identifies the temporal change for the historical time period. For example, if the temporal analyzer engine 614 or 615 determines that the click-fraction during November for the two content items responsive to a query for “turkey” exceeds the average click-fraction by more than the minimum change threshold, then a temporal change has been for the content items during November.
  • the CTR engine 610 identifies another content item to compare the first content item against. For example, if no temporal change occurred for any of the Thanksgiving related content items when compared to the content items related to vacation resorts in Turkey, the CTR engine may restart the process at stage 806 by comparing the CTR of vacation resorts in Turkey to the CTR of a clothing brand that uses a turkey as the logo and is also responsive to a query 602 for “turkey.”
  • FIG. 9 is a flow diagram of another content ranking process 900 .
  • the process can, for example, be implemented by the content ranking system 650 of FIG. 6 b.
  • Stage 902 analyzes click logs and query logs related to a query.
  • a server device such as the CTR engine 610 may access the query logs 606 and click logs 608 related to a query 602 for “turkey.”
  • Stage 904 determines historical click-through rates of content items.
  • the CTR engine 610 may determine the historical CTR of content items related to the query 602 for “turkey”, such as content items associated with vacation resorts in Turkey and content items associated with Thanksgiving turkey recipes.
  • the historical CTR data 612 may be stored by different time periods, by category, by region, or by any other grouping specified by the temporal analyzer engine 615 .
  • Stage 906 identifies temporal changes for historical click-through rates of the content items.
  • the historical CTR of different content items can be compared to identify any temporal changes in the CTRs. If the historical CTR of one content item has changed relative to the historical CTR of another content item during a time period, there is a temporal change in the CTRs of the content items during the time period.
  • the temporal analyzer engine 615 may identify the time periods when the CTR for content items associated with Thanksgiving turkey recipes is higher than average, or the CTR for content items associated with vacation resorts in Turkey is lower than average. Based on this comparison, the temporal analyzer engine 615 may identify the month of November as the time period of a temporal change.
  • Stage 908 receives search results from a search engine for the query.
  • the filter 618 may receive search results from the search engine 604 from a query for “turkey.”
  • the search results may include content items associated with Thanksgiving turkey recipes and content items associated with vacation resorts in Turkey.
  • Stage 910 adjusts rankings of the content items. For example, during a time period that is related to a historical time period of an identified temporal change for content items, the filter 618 can adjust the rankings of content according to the temporal change. As previously described, the filter 618 can facilitate a higher ranking to content items associated with Thanksgiving turkey recipes during November, the period identified by stage 906 as the period of a temporal change. Similarly, the filter 618 can facilitate a lower ranking to a second content item related to vacation resorts in Turkey during the same time period to reflect the lower CTR of the second content item during November.
  • CTRs are used to identify a temporal changes during a time period for content items responsive to a query
  • other methods of identifying temporal changes can be also used.
  • one alternative method of identifying a temporal change includes human identification and evaluation of queries and responsive content items and identification of the temporal changes that become apparent during time periods.
  • a map search service is used in addition to or instead of a search service.
  • the map search service identifies locations on the map that are more responsive to a map query during a time period. For example, the map search service may identify responsive restaurant locations as those that are open during a time period.
  • Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
  • Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible program carrier for execution by, or to control the operation of, data processing apparatus.
  • the tangible program carrier can be a propagated signal or a computer readable medium.
  • the propagated signal is an artificially generated signal, e.g., a machine generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a computer.
  • the computer readable medium can be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of matter effecting a machine readable propagated signal, or a combination of one or more of them.
  • a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • a computer program does not necessarily correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
  • processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer.
  • a processor will receive instructions and data from a read only memory or a random access memory or both.
  • the essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • a computer need not have such devices.
  • Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD ROM disks.
  • semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
  • magnetic disks e.g., internal hard disks or removable disks
  • magneto optical disks e.g., CD ROM and DVD ROM disks.
  • the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
  • embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • keyboard and a pointing device e.g., a mouse or a trackball
  • Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
  • Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described is this specification, or any combination of one or more such back end, middleware, or front end components.
  • the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.
  • LAN local area network
  • WAN wide area network
  • the computing system can include clients and servers.
  • a client and server are generally remote from each other and typically interact through a communication network.
  • the relationship of client and server arises by virtue of computer programs running on the respective computers and having a client server relationship to each other.

Abstract

Apparatus, systems and methods for content ranking are disclosed. Historical click-through rates of first and second content items for a query are determined from an analysis of click logs and query logs. Temporal changes in the historical click-through rate of the first and second content items are identified. The rankings of the first and second content items are adjusted based on the temporal change.

Description

CROSS REFERENCE TO RELATED APPLICATIONS
This application is a continuation of and claims priority to U.S. patent application Ser. No. 11/870,893, filed on Oct. 11, 2007, the entire contents of which are hereby incorporated by reference.
BACKGROUND
This document relates to ranking content items responsive to a query and based on when the query is received.
The rise of the Internet has enabled access to a wide variety of content items, e.g., video and/or audio files, web pages for particular subjects, news articles, etc. Content items of particular interest to a user can be identified by a search engine in response to a user query. One example search engine is the Google search engine provided by Google Inc. of Mountain View, Calif., U.S.A. The query can include one or more search terms or phrases, and the search engine can identify and, optionally, rank the content items based on the search terms or phrases in the query and present the content items to the user (e.g., in order according to the rank).
Queries are sometimes ambiguous with respect to the interests of the users. For example, queries for “turkey” can return results related to the country of Turkey and results related to turkey recipes for Thanksgiving holidays in Canada and the United States. Accordingly, users may have to revise their queries one or more times until the search engine returns the content items that are of interest to the user.
SUMMARY
Apparatus, systems and methods for content item ranking are disclosed. In one implementation, click logs and query logs related to a query are analyzed. Historical click-through rates of content items are determined based on the analysis of the click logs and query logs. A temporal change during a historical time period for an historical click-through rate of a first content item relative to an historical click-through rate of a second content item is identified. A query can be received during another time period. If the another time period is related to the historical time period, the query can be disambiguated based on the temporal change.
In another implementation, click logs and query logs related to a query are analyzed. Historical click-through rates of content items based on the click logs and query logs are determined. Temporal changes for historical click-through rates of first content items relative to historical click-through rates of second content items for a historical time period are identified. Search results received from a search engine for the query during a related time period and that include least some of the content items are adjusted according to the temporal change.
In another implementation, a system includes a click-through engine and a temporal analyzer engine. The click-through engine accesses query logs related to queries and click logs related to content items and determines historical click-through rates for content items related to a query. The temporal analyzer engine identifies a temporal change in the historical click-through rate of a first content item relative to the historical click-through rate of a second content item for a first time period. The first and second content items are relevant to the query. The temporal analyzer causes the rankings of search results for the query that include at least the first content item to be adjusted during a second time period according to the temporal change, wherein the second time period is related to the first time period.
The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of an example online environment.
FIG. 2 is an example of a temporal change for click-through rates graph.
FIG. 3 is another example of a temporal change for click-through rates graph.
FIG. 4 is an example screen shot of search results.
FIG. 5 is another example screen shot of search results.
FIG. 6 a is a block diagram of an example content identification system.
FIG. 6 b is a block diagram of another example content identification system.
FIG. 7 is a flow diagram of a content ranking process.
FIG. 8 is a flow diagram of another content ranking process.
FIG. 9 is a flow diagram of another content ranking process.
Like reference numbers and designations in the various drawings indicate like elements.
DETAILED DESCRIPTION
FIG. 1 is a block diagram of an example online environment 100. The online environment 100 can facilitate the identification and serving of content items, e.g., web pages, advertisements, etc., to users. A computer network 110, such as a local area network (LAN), wide area network (WAN), the Internet, or a combination thereof, connects advertisers 102, an advertisement management system 104, publishers 106, and user devices 108.
In some implementations, the advertisers 102 can directly, or indirectly, enter, maintain, and track advertisement (“ad”) information in an advertising management system 104. Though reference is made to advertising, other forms of content, including other forms of sponsored content, can be delivered by the system 100. The ads may be in the form of graphical ads, such as banner ads, text only ads, image ads, audio ads, video ads, ads combining one or more of any of such components, etc. The ads may also include embedded information, such as links, meta-information, and/or machine executable instructions. One or more publishers 106 may submit requests for ads to the system 104. The system 104 responds by sending ads to a requesting publisher 106 for placement on or association with one or more of the publisher's content items (e.g., web properties). Example web properties can include web pages, television and radio advertising slots, and even print media space.
One example publisher 106 is general content server 106 a that receives requests for content (e.g., articles, discussion threads, music, video, graphics, search results, web page listings, information feeds, etc.), and retrieves the requested content in response to the request. The content server 106 a may submit a request for ads to an advertisement server in the system 104. The ad request may also include content request information. This information can include the content itself (e.g., page, video broadcast, radio show, or other type of content), a category corresponding to the content or the content request (e.g., arts, business, computers, arts-movies, arts-music, etc.), part or all of the content request, content age, content type (e.g., text, graphics, video, audio, mixed media, etc.), geo-location information of a requesting device 108, etc. In some implementations, the content server 106 a or a client browser can combine the requested content with one or more of the ads provided by the system 104. The combined content and ads can be sent/rendered on the user device 108 that requested the content for presentation in a viewer (e.g., a browser or other content display system).
Another example publisher 106 b is a search service 106 b. The search service 106 b can receive queries from the user devices 108 for search results. In response, the search service 106 b can derive relevant search results from an index of documents (e.g., from an index of web pages). An exemplary search service 106 b is described in the article S. Brin and L. Page, “The Anatomy of a Large-Scale Hypertextual Search Engine,” Seventh International World Wide Web Conference, Brisbane, Australia and in U.S. Pat. No. 6,285,999. Search results can include, for example, lists of web page titles, snippets of text extracted from those web pages, and hypertext links to those web pages, and may be grouped into a predetermined number of (e.g., ten) search results. The search results can be provided to the requesting user device 108.
In some implementations, the search service 106 b can submit a request for ads to the advertising management system 104. The request for ads may also include the query (as entered or parsed), information based on the query (such as geo-location information, whether the query came from an affiliate and an identifier of such an affiliate), and/or information associated with, or based on, the search results. Such information may include, for example, identifiers related to the search results (e.g., document identifiers or “docIDs”), scores related to the search results (e.g., information retrieval (“IR”) scores, page rank scores, etc.), snippets of text extracted from identified documents (e.g., web pages), full text of identified documents, feature vectors of identified documents, etc. In some implementations, IR scores are computed from, for example, dot products of feature vectors corresponding to a query and a document, page rank scores, and/or combinations of IR scores and page rank scores, etc.
The search service 106 b can combine the search results with one or more of the ads provided by the system 104. This combined information can then be forwarded to the user device 108 that requested the content. The search results can be maintained as distinct from the ads, so as not to confuse the user between paid ads and presumably neutral search results.
In some implementations, the search service 106 b may rank the search results based on keywords in the user's query. The search service 106 b may also consider user preferences and/or actions when determining the rankings. User preferences may be determined by different usage information provided to the search service. In some implementations, the user devices 108 can provide usage information to the search service, such as, for example, whether a click-through related to a content item has occurred, whether a conversion has occurred, or whether some other defined event has occurred. This usage information can include measured or observed user behavior related to content items that have been served by the publisher. Such usage information can be processed to measure performance metrics, such as a click-through rate (“CTR”), conversion rate, etc.
A click-through can occur, for example, when a user of a user device, selects or “clicks” on a link to a content item returned by the publisher or the advertising management system. The CTR is a performance metric that is obtained by dividing the number of users that clicked on the content item, e.g., a link to a landing page, an advertisement, or a search result, by the number of times the content item was delivered. For example, if a link to a content item is delivered 100 times, and three persons click on the content item, then the CTR for that content item is 3%. Other usage information and/or performance metrics can also be used.
Users generally desire that the search results identify the content items that are most responsive to the user's query. However, what users actually consider to be responsive may fluctuate over time. For example, a particular content item may be determined to be highly responsive to users submitting queries on weekends, moderately responsive to users submitting queries on weekday evenings, but less responsive to users submitting queries on weekday mornings and afternoons, even when the same users are making the same queries at different times.
In some implementations, the search service 106 b can identify a temporal change indicative of changes in users' interests for one content item over another content item that are responsive to the same query. A temporal change may be associated with a time period and/or a region. In some implementations, a temporal change is identified by analyzing the CTRs of content items related to the query. For example, the temporal change may be a change in the CTR of a content item responsive to a query relative to the CTR of another content item responsive to the same query over a time period, and for which the change exceeds a threshold. In some implementations, the ranking of the search results can be based in part on the temporal change.
FIG. 2 is an example of a temporal change for click-through rates graph 200. The CTR graph 200 represents the historical weekly CTRs over a one year time period of two different content items that are responsive to the same query. By comparing the CTRs of the two content items and determining whether a value generated by the comparison exceeds a threshold, temporal changes in the CTRs of the content items during the one year time period can be identified.
The CTR 202 line represents the CTR of the first content item responsive to a query over a one year period, and CTR 204 line represents the CTR of the second content item responsive to the same query over the same one year period. For example, a query for “turkey” may return content items associated with vacation resorts in the country Turkey and content items associated with Thanksgiving turkey recipes. Although both sets of content items are responsive to a query for “turkey,” users may click on the content items associated with the vacation resorts in Turkey more frequently than the content items associated with the Thanksgiving turkey recipes. For example, historical time period 210 is representative of the majority of historical time periods shown on the graph 200, with the exception of the historical time period 206 and the historical time period 208. During the historical time period 210, the CTR 202 indicates the CTR of a content item associated with the country Turkey as averaging 0.36, and the CTR 204 indicates the CTR of content items associated with Thanksgiving turkey recipes as averaging 0.05.
However, the CTRs of content items may change over time. The change in the CTR may be a daily change, weekly change, seasonal change, one time change, or change during any other periodic or aperiodic time period for which the CTR is measured. For example, the CTR 204 line indicates that the CTR of a content item associated with turkey recipes increases significantly during the historical time period 206 and the historical time period 208. This increase indicates that relatively more users are clicking on the content item associated with Thanksgiving turkey recipes during the time periods 206 and 208, likely a result of the increased demand for turkey recipes during Thanksgiving and Christmas holidays. Similarly, the CTR 202 line indicates that the CTR of a content item associated with the country Turkey decreases significantly during the same time period 206 and period 208, likely a result of the increased percentage of users clicking turkey recipes in the period 206 and period 208. The change in the CTR 202 and the CTR 204 is an example of a temporal change during the period 206 and the period 208.
In some implementations, the temporal change may be identified by recognizing a change in a ratio of the CTRs of the content items that exceeds a threshold. In some implementations, a click-fraction is used to quantify the ratio of the CTR of one content item relative to another content item. For example, a click-fraction quantifying the ratio of the CTR 202 to the CTR 204 during the historical time period 210 is:
C T R 202 C T R 204 = 0.36 0.05 = 7.2 = Click-fraction for the time period 210
This click-fraction suggests that during time period 210, from a query for “turkey,” users are 7.2 times more likely to click on the content item associated with vacation resorts in Turkey than with content item associated with Thanksgiving turkey recipes. In the graph 200, the click-fraction values are substantially the same for every month, except for the period 206 and the period 208. The click-fraction for the period 206 changes significantly from the click-fraction for the historical time period 210:
C T R 202 C T R 204 = 0.14 0.20 = 7.0 = Click-fraction for the time period 206
This click-fraction suggests that during time period 206, from a query for “turkey,” users are 1.4 times more likely to click on the content item associated with Thanksgiving turkey recipes than with the content item associated with vacation resorts in Turkey. The change in the click-fractions of the historical time periods 206 and 210 indicates that there may be a temporal change for the query for “turkey” during the period 206 for the content items associated with the CTR 202 and the CTR 204
In some implementations, a change in a click-fraction for a time period must exceed a minimum change threshold to be considered a temporal change. The minimum change threshold may be calculated by a formula or may be provided by a person or another system. For example, the standard deviation of the click-fractions of the time periods may be the minimum change threshold. If the standard deviation for the click-fractions of all the monthly periods of FIG. 2 is 2.56, any change in the click-fraction during a time period that deviates from the average CTR by more than the standard deviation can be considered a temporal change. Thus for the time period 206:
Average click-fraction=7.2
Minimum change threshold=2.56
Change in the click-fraction for the time period 206=7.2−0.7=6.5
Because the change in the click-fraction during the time period 206 compared to the baseline average, 6.5, exceeds the minimum change threshold, 2.56, it may be considered a temporal change. Conversely, during period 212, both the CTR 202 and the CTR 204 fluctuate but the change in the click-fraction during the period 212 is not significant enough:
0.34 0.06 = 5.67 = Click-fraction for the time period 212
Average click-fraction=7.2
Minimum change threshold=2.56
Change in the click-fraction=7.2−5.67=1.53
Thus, during time period 212, the change in the click-fraction does not exceed the minimum change threshold of 2.56. Accordingly, the time period 212 is not considered to have a temporal change.
The difference between the change in the click-fraction and the minimum change threshold can also be used to determine the degree of the temporal change; the larger the difference, the more significant the change. For example, the graph 200 indicates that the CTR 202 and CTR 204 changed considerably during period 206 compared to during period 210, while the change was less significant during period 212 compared to during period 210. Accordingly, the difference between the minimum change threshold and the change in the click fraction during period 206, 6.5, is much greater than the difference between the minimum change threshold and the change in the click fraction during period 212, which is 1.53. The larger difference during period 206 is indicative of a larger temporal change in the CTRs of the content items during period 206.
Other methods of determining a threshold value can also be used. For example, an average of CTR's over multiple historical time periods can be determined, and an average of the click fraction (or some other central tendency of the click fraction) over the periods can likewise be determined. If the click fraction for a monthly period deviates from the average click fraction by a certain percentage, e.g., 20% of the average click fraction, then that time period can be determined to have a temporal change.
Although a temporal change can be identified for a change in the click fraction comparing the CTR of one content item during a time period relative to the CTR of another content item during the same time period, a temporal change may also be identified for a change in the CTR of a content item during a time period relative to the CTR of the same content item during other time periods. For example, during time period 214 the CTR 204 changes significantly relative to the CTR 204 during period 210. This variation in the CTR 204 may be determined to be a temporal change during the period 214.
FIG. 3 is another example of a temporal change for click-through rates graph 300. The graph 300 represents the daily CTRs over a historical three week time period of two different groups of content items that are responsive to the same query. By comparing the CTRs of the two groups of content items, temporal changes in the CTRs during the three week time period can be identified. The graph 300 is similar to the graph 200 of FIG. 2, except that the graph 300 is identified through multiple weekly periods, and the lines for CTR 302 and 304 correspond to groups of content items.
The CTR 302 line represents the CTR of the first group of content items, and CTR 304 represents the CTR of the second group of content items. For example, a query for “bill” may return content items associated with a draft statute, called a “bill,” and content items associated with a cartoon character named Bill. Although both sets of content items are related to a query for “bill,” during certain time periods users may click on the content items associated with the draft statute more frequently than the content items associated with Bill the cartoon character. For example, from a query for “bill,” during daily period 308, the CTR of content items associated with the draft statute, CTR 302, averages 0.33 and the CTR of content items associated with Bill the cartoon character, CTR 304, averages 0.07.
The CTRs of the content items may change hourly, weekly, biweekly, monthly, etc. In graph 300, a weekly change in the CTR 302 and the CTR 304 during the Monday time period 306 is of substantial enough magnitude to be considered a temporal change. The temporal change during period 306 may arise because the cartoon is shown on television on Monday, which results in an increased interest in Bill the cartoon character during the period 306. This temporal change repeats weekly, and may be used to predict the next period of a temporal change, e.g. the time period 306′.
Identification of a temporal change during a historical time period can facilitate modification of rankings of search results responsive to a query during other time periods that are related to the historical time periods for which a temporal change is identified. Examples of such other time periods are time periods that are of the same or substantial period as the historical time period, e.g., same days of week, same annual holidays such as July 4th holiday, Thanksgiving holiday, Christmas holiday, etc. FIGS. 4 and 5 illustrate such a modification.
FIG. 4 is an example screen shot of search results 400. The search results 400 can, for example, be search results from a search engine in time periods not affected by a temporal change, e.g. a time period related to the historical time period 210 of FIG. 2.
For example, a search engine may rank content items related to vacation resorts in Turkey higher than content items related to Thanksgiving turkey recipes when the queries for “turkey” during the month of May. The search engine may rank Country Turkey Web Page 404 as the highest content item in a query 402 for “turkey.” Additionally, the search engine may rank the next seven content items related to vacation resorts in Turkey, Country Turkey Web Pages 406, 408, 410 and 412, and Turkey Country Ads 414, 416 and 418, higher than any content related to Thanksgiving turkey recipes.
FIG. 5 is another example screen shot of search results 500. The search results 500 can, for example, be search results from a search engine during a time period when the search results are ranked in response to a temporal change, e.g., during a time period that is related to the historical time period 206 of FIG. 2, such as the month of November, or a week in November.
For example, when there is a temporal change with an increase in the CTR of content items associated with Thanksgiving turkey recipes, and a query that is relevant to the content items is submitted during the time period defined by the temporal change, i.e., any November month, the search engine may rank Thanksgiving turkey recipe content items higher than content items associated with vacation resorts in Turkey. Thus, in the search results 500, generated during the time period of the temporal change, Thanksgiving Turkey Web Page 504 may rank as the highest content item in a query 502 for “turkey.” The search engine may rank the five of the next seven content items related to Thanksgiving turkey recipes, i.e., Thanksgiving Turkey Web Pages 506, 508, and 512 relatively higher than the rankings of the same content items during periods of no temporal change.
Such ranking adjustments can be implemented within a search engine. In some implementations, one or more search engine variables related to ranking a content item in response to a query can be adjusted based on an identified temporal change. Thus, if the search engine generates rankings based on a product of a page rank and an information retrieval score, one or more of the IR scores and page ranks scores can be adjusted to increase or decrease the ranking of a search result during the period of the temporal change. Other search algorithms can likewise be adjusted in a similar manner.
Other content items can also be adjusted based on a temporal change. For example, advertisement ranking may also be adjusted based on a temporal change. As shown in FIG. 5, Turkey Bird Ads 514 and 516 are ranked ahead of the Turkey Country Ad 518. Such adjustments can be based on relevance, e.g., if the advertisements are ranked, in part, on a relevance of the advertisement to a query, then the relevance (and thus the ranking) of an advertisement can be adjusted in a similar manner that the search results are adjusted. Other methods of adjusting advertisement rankings can also be used, e.g., adjusting bid requirements, adjusting quality requirements, etc.
In some implementations, the search results can be received from the search engine and adjusted based on an identified temporal change. For example, the search engine may rank the search results without considering the time period during which the query was submitted. The search results can be processed by a ranking filter that can re-rank some or all of the search results accordingly.
In some implementations, the ranking adjustments can correspond to the magnitude of the temporal change. For example, if the temporal change indicates a significant change in the CTRs of the first content items to second content items such that the CTRs of the first content items are higher than the CTRs of the second content items, such as during the time period 206 of FIG. 2, then the first content items can be ranked higher than the second content items. Similarly, if the temporal change is less significant, e.g., the CTRs of the first content items relative to the CTRs second content items increase but are still less than the CTRs of the second content items, such as during the time period 208 of FIG. 2, then the rankings of the first content items can be increased, but still remain less than the rankings of the second content items.
FIG. 6 a is a block diagram of an example content identification system 600. The system 600 can, for example, be used to distinguish content items responsive to the same query, based on identified temporal changes. The system may be implemented as part of a search engine, or as a separate component or system that communicates with a search engine.
As an overview, the system 600 can select a query from a query log and identify content items responsive to the query. The system 600 can process click logs related to the identified content items served in response to the query and determine CTRs for the respective content items over a series of time periods, e.g., daily CTRs over a one-year period; weekly CTRs over a two year period, etc. The CTRs of each of the content items during the time periods can be compared for each time period. If the change of a CTR for a content item relative to the CTR of another content item for a period exceeds a threshold, then a temporal change for those content items can be identified for the period. The system 600 can then adjust the ranking of the content items for search results for the same query submitted during another time period that is related to the time period of the temporal change.
In an implementation, a CTR engine 610 accesses a query log 606 and a click log 608 to generate historical click-through data 612 for content items related to a query 602. The query log 606 stores the keywords used in the query 602 and in past queries. The click log 608 stores the number of clicks different content items have received when identified in response to different queries and the number of times the content items have been presented as search results relevant to the queries. The CTR engine 610 uses the data from the query log 606 and the click log 608 to generate the historical CTR data 612 for content items related to the query. For example, the query log 606 may store queries with the keyword “turkey”, and the click log 608 may store the number of presentations to and the number of clicks received from the users for content items, e.g., content items that are associated with Thanksgiving turkey recipes and content items that are associated with vacation resorts in Turkey. The CTR engine 610 may access the query log 606 and click logs 612 and generate historical CTR data 612 related to the query for the keyword “turkey.”
In some implementations, a temporal analyzer engine 614 analyzes the historical CTR data 612 for the different content items responsive to the query 602 and identifies temporal changes in the CTRs of the content items responsive to the same query during a time period. For example, the temporal analyzer engine 614 may identify an increase in the CTR for the content items associated with Thanksgiving turkey recipes during the month of November over several years. Similarly, the temporal analyzer engine 614 may identify a decrease in the CTR for the content items associated with resort locations in the country of Turkey during November over the same several years. Thus The temporal analyzer engine 614 may identify the month of November as a time period during which a temporal change for the two content items responsive to a query for “turkey.” Because the time period is substantially periodic, i.e., occurring during the month of November, the temporal analyzer engine 614 can determine that the temporal change is to be used to adjust the ranking of content items during the same period in the future, e.g., during each month of November.
In some implementations, the search engine 604 identifies content items responsive to a query, and then ranks the content items in response to the temporal change identified by the temporal analyzer engine 614 to provide the most relevant search results in ranked results 616. For example, for the single-word query “turkey,” results related to vacation resorts in the country Turkey may generally be ranked higher than results related to Thanksgiving turkey recipes. However, because there may be a temporal change in the CTR for content items containing Thanksgiving turkey recipes during November, rankings of the content items can be modified. For a query for “turkey,” the content items associated with Thanksgiving turkey recipes may be ranked relatively higher in the ranked results 616 during November to reflect the increased CTR of Thanksgiving related content items. The adjustment can be implemented within the search engine 604 or external to the search engine 604 as described with reference to FIG. 5.
The identification of a temporal change for a query and responsive content items and the adjustment of the search results for the time period of the temporal change can thus disambiguate the query, i.e., provide adjusted search results that are more likely to be related to the users' current interests. Other methods of disambiguating a query can also be used. For example, in some implementations, the search engine 604 may rely on the CTR engine 610 and/or the temporal analyzer engine 614 to identify keywords to disambiguate the query, and provide the keywords to the search engine 604 with the query 602. The keywords may be identified by the temporal analyzer engine 614 or the search engine 604 as keywords related to the content items subject to a temporal change.
For example, if the temporal analyzer engine 614 determines that there is a temporal change in CTRs favoring a first set of content items, the first set of content items can be evaluated, e.g., text mined, to determine corresponding keywords related to the content items. Thus, if the first set of content items are related to Thanksgiving turkey recipes, the temporal analyzer engine 614 can provide the search engine with additional keywords to include in the query, such as “recipe,” “Thanksgiving,” etc., when the query is received during the Thanksgiving time period. In a variation of this implementation, an alternative query directed to the second set of content items can be optionally shown with the search results during the time period of the temporal change. For example, as illustrated by the dashed user interface element 520, an alternate query related to the country of Turkey can be submitted to the search engine upon selection by a user, the processing of which would generate the search results of FIG. 4.
In some implementations, the historical CTR 612 data can be processed for time periods of different durations. The historical CTR data 612 may identify the CTR of a content item over a first time period, e.g., hourly, and can be aggregated into additional time periods, e.g., daily, weekly, monthly, etc.
In some implementations, the CTR engine 610 and/or the temporal analyzer engine 614 can identify temporal changes based on a lunar calendar, a Gregorian calendar, or other periods. For example, the historical CTR data 612 may identify a decrease in the CTR of content items associated with food for a particular query during the month of Ramadan, a Muslim month of fasting based on the lunar calendar, and likewise identify an increase in the CTR of content items for the query associated fasting during Ramadan. Likewise, the temporal changes that occur only on the day of Thanksgiving in the United States, i.e., the fourth Thursday of November in the United States, can also be identified and correlated to the occurrence of the holiday.
In some implementations, the historical CTR data 612 may also be stored for different geographical regions as historical regional CTR data. The historical regional CTR data may identify the CTR during a time period of a content item in a city, state, country, or other regional categorization. If the historical CTR data 612 identifies different regions, the temporal analyzer engine 614 may identify temporal changes on a regional basis during a time period. For example, the historical CTR data 612 may identify an increased CTR for content items associated with Thanksgiving turkey recipes during November for queries received from the United States. Conversely, such a temporal change may not be identified for queries received from Japan in November.
In some implementations, the historical CTR data 612 may also include time zone data to facilitate the identification of temporal changes during a time period in a time zone of a content item. For example, a temporal change for content items associated with the query “eggs” might be identified between 7:00 AM and 8:00 AM for a time zone, and may occur across each time zone at the same respective time. Thus, for queries submitted between the time of 7:00 AM and 8:00 AM in each time zone, content items for egg recipes and restaurants with breakfast menus may be ranked higher than content items that include other information about eggs. Conversely, for queries submitted after 8:00 AM in each time zone, content items for egg recipes and restaurants with breakfast menus may be ranked lower than content items that include other information about eggs.
In some implementations, the CTR engine 610 and/or the temporal analyzer 614 can group content items with similar CTR behavior during a time period according to a category. There are a number of ways that the category of a content item may be identified. For example, in some implementations, content items may be mapped to predefined categories by the search engine 604, and if the majority of content items exhibiting a similar CTR behavior during a time period belong to a predefined category, then the content items can be identified as belonging to the predefined category. In other implementations, categories can also be determined by empirical data. A group of content items that have similar CTR patterns for the query “turkey,” can be categorized together by the CTR engine 610 and/or the temporal analyzer 614.
In some implementations, rankings for all content items belonging to a category can be adjusted during a time period of a temporal change for a query. For example, the temporal analyzer engine 614 may identify a temporal change associated with all content items in a category of content items, where those content items are responsive to a query for “turkey” during November. For example, if a content item associated with deep fried turkey recipes is identified as being in the same category as a content item associated with a turkey stuffing recipes, both of those content items can be rank-adjusted based on a temporal change identified for the category.
The search engine 604, CTR engine 610 and temporal analyzer engine 614 can be realized by instructions that upon execution cause one or more processing devices to carry out the processes and functions described above. Such instructions can, for example, comprise interpreted instructions, such as script instructions, e.g., JavaScript or ECMAScript instructions, or executable code, or other instructions stored in a computer readable medium. The engines 604, 610 and 614 can be implemented separately, or can be implemented as a single software entity. The engines 604, 610 and 614 can also be distributively implemented by multiple computers communicating with each other over a network, such as in a server farm, or can be implemented in a single computer device.
FIG. 6 b is a block diagram of another example content identification system 650. The system 650 is similar to the system 600 from FIG. 6 a, except the system 650 includes a ranking filter 618 that may be used to adjust the results of the search engine 604 to generate the ranked results 616. The search results can be received from the search engine 604 and adjusted by the rank filter 618 based on an identified temporal change.
In some implementations, the temporal analyzer engine 615 uses the historical CTR data 612 to identify a temporal change, which the temporal analyzer engine 615 communicates to the filter 618. For example, when the temporal analyzer engine 615 identifies a temporal change in the CTR during November of content items related to Thanksgiving turkey recipes, the temporal change is communicated to the filter 618. The search engine 604 may continue to rank content items related to vacation resorts in Turkey higher than content items related to Thanksgiving turkey recipes, even though the temporal change identifies a higher CTR for Thanksgiving related content items during November.
The filter 618 may then modify the search results from the search engine 604 by adjusting the ranking the search results from the search engine 604 to reflect the temporal change in CTRs of the content items during the time period of the temporal change. For example, the filter 618 can analyze the search results received from the search engine 604 during a period related to temporal change to identify the content items having CTRs that define the temporal change. The filter can then adjust the rankings during the time period related to temporal change so that content items that have a CTR increase are ranked higher than during a period of no temporal change, and content items that experience a CTR decrease are ranked lower than during a period of no temporal change. The modified results can be ranked results 616.
For example, as a result of a query for “turkey,” the search engine 604 may rank content items related to vacation resorts in Turkey higher than content items related to Thanksgiving turkey recipes. As in system 600, a temporal change of the two content items responsive to a query for “turkey” during November may be identified by the temporal analyzer engine 615. However, in system 650, the temporal change is communicated to the filter 618, and the filter 618 can re-rank the search results from the search engine 604 during the time period of the temporal change so that content items for Thanksgiving turkey recipes are ranked higher than content items related to vacation resorts in the country Turkey.
The search engine 604, CTR engine 610, temporal analyzer engine 615 and the filter 618 can be realized by instructions that upon execution cause one or more processing devices to carry out the processes and functions described above. Such instructions can, for example, comprise interpreted instructions, such as script instructions, e.g., JavaScript or ECMAScript instructions, or executable code, or other instructions stored in a computer readable medium. The components 604, 610, 615 and 618 can be implemented separately, or can be implemented as a single software entity. The components 604, 610, 615 and 618 can also be distributively implemented over a network, such as a server farm, or can be implemented in a single computer device.
FIG. 7 is a flow diagram of a content ranking process 700. The process 700 can, for example, be implemented by the system 600 of FIG. 6 or the system 650 of FIG. 6 b.
Stage 702 analyzes click logs and query logs related to a query. For example, a server device such as the CTR engine 610 may access the query logs 606 and click logs 608 related to a query 602 for “turkey.” In some implementations, all received queries are selected for evaluation. In other implementations, only queries that exceed a submission threshold, e.g., queries that are most frequently submitted, are selected for evaluation.
Stage 704 determines historical click-through rates of content items. For example, the CTR engine 610 may determine the historical CTR of content items responsive to the query 602 for “turkey,” such as content items associated with vacation resorts in Turkey and content items associated with Thanksgiving turkey recipes. The historical CTR data 612 may be stored by different time periods, by category, by region, or by any other grouping specified by the temporal analyzer engine 614 or 615.
Stage 706 identifies a temporal change for an historical click-through rate of a first content item. For example, the historical CTR of the first content item may be compared to the historical CTR of a second content item for a historical time period. If the historical CTR of the first content item has substantially changed relative to the historical CTR of the second content item during the historical time period, there is a temporal change in the CTRs during the historical time period. For example, the temporal analyzer engine 614 or 615 may identify the time periods when the historical CTR data 612 for “turkey” queries is indicative of a higher CTR for content items associated with Thanksgiving turkey recipes and a lower CTR for content items associated with vacation resorts in Turkey. Any such time periods are the period of a temporal change in the CTRs of the content items.
In some implementations, the historical CTRs of content items can be compared for all identified content items. In other implementations, the historical CTRs of content items having CTR values that exceed a minimum threshold can be compared to exclude underperforming content items. In other implementations, the historical CTRs of content items that have been served in excess of a minimum number of times over a period can be compared to exclude very unique content items. Other selection processes can also be used.
Stage 708 disambiguates the query for the first content item, i.e., provides adjusted search results that are more likely to be related to the users' current interests. For example, the temporal analyzer engine 614 or the filter 618 may adjust the results of a search engine 604 to rank content items related to Thanksgiving turkey recipes higher than content items related to vacation resorts in Turkey during another time period that is related to the historical time period having an identified temporal change. The temporal analyzer engine 614 may alternatively provide or augment a query to the search engine that is directed to the content items related to the temporal change. For example, the temporal analyzer engine 614 may provide a query for “Thanksgiving turkey” for an initial query of “turkey” during another period associated with a temporal change, e.g., a month of November.
FIG. 8 is a flow diagram of another content ranking process 800. The process can, for example, be implemented by the content ranking system 600 of FIG. 6 a or the content ranking system 650 of FIG. 6 b.
Stage 802 selects a time period for analysis. For example, the temporal analyzer engine 614 or 615 can select historical hourly, daily, weekly, or monthly calendar time periods, or any other periodic or substantially periodic time periods. Multiple time periods can be processed, e.g., 52 time weekly time periods for a year of query-log data and click-log data.
Stage 804 determines click-through rates of the first content item responsive to a query. For example, the CTR engine 610 can determine the CTRs of a first content item during the time periods selected by the temporal analyzer engine 614 or 615. For example, the CTR engine 610 may determine the weekly CTR of a content item related to Thanksgiving turkey recipes during every week for which there is historical CTR data.
Stage 806 determines click-through rates of the second content item. For example, the CTR engine 610 can determine the CTRs of a second content item during the same time periods selected by the temporal analyzer engine 614 or 615, e.g., the CTR engine 610 may determine the weekly CTR of a content item related to vacation resorts in Turkey during every week for which there is historical CTR data.
Stage 808 generates content item click-fractions for each of the time periods. For example, the click-fraction may represent a ratio of the CTR of a content item associated with Thanksgiving turkey recipes to the CTR of a content item associated with vacation resorts in Turkey.
Stage 810 determines whether the content item click-fractions exceed a minimum change threshold. For example, the temporal analyzer 614 or 615 can identify if the weekly click-fraction comparing two content items responsive to the query for “turkey” is greater than the average click-fraction for the two content items stored in the historical CTR data 608 by an amount exceeding the minimum change threshold.
If stage 810 determines that any of the content item click-fractions exceed a minimum change threshold, stage 812 identifies the temporal change for the historical time period. For example, if the temporal analyzer engine 614 or 615 determines that the click-fraction during November for the two content items responsive to a query for “turkey” exceeds the average click-fraction by more than the minimum change threshold, then a temporal change has been for the content items during November.
If the temporal change does not exceed the minimum change threshold, the CTR engine 610 identifies another content item to compare the first content item against. For example, if no temporal change occurred for any of the Thanksgiving related content items when compared to the content items related to vacation resorts in Turkey, the CTR engine may restart the process at stage 806 by comparing the CTR of vacation resorts in Turkey to the CTR of a clothing brand that uses a turkey as the logo and is also responsive to a query 602 for “turkey.”
FIG. 9 is a flow diagram of another content ranking process 900. The process can, for example, be implemented by the content ranking system 650 of FIG. 6 b.
Stage 902 analyzes click logs and query logs related to a query. For example, a server device such as the CTR engine 610 may access the query logs 606 and click logs 608 related to a query 602 for “turkey.”
Stage 904 determines historical click-through rates of content items. For example, the CTR engine 610 may determine the historical CTR of content items related to the query 602 for “turkey”, such as content items associated with vacation resorts in Turkey and content items associated with Thanksgiving turkey recipes. The historical CTR data 612 may be stored by different time periods, by category, by region, or by any other grouping specified by the temporal analyzer engine 615.
Stage 906 identifies temporal changes for historical click-through rates of the content items. The historical CTR of different content items can be compared to identify any temporal changes in the CTRs. If the historical CTR of one content item has changed relative to the historical CTR of another content item during a time period, there is a temporal change in the CTRs of the content items during the time period. For example, the temporal analyzer engine 615 may identify the time periods when the CTR for content items associated with Thanksgiving turkey recipes is higher than average, or the CTR for content items associated with vacation resorts in Turkey is lower than average. Based on this comparison, the temporal analyzer engine 615 may identify the month of November as the time period of a temporal change.
Stage 908 receives search results from a search engine for the query. For example, the filter 618 may receive search results from the search engine 604 from a query for “turkey.” The search results may include content items associated with Thanksgiving turkey recipes and content items associated with vacation resorts in Turkey.
Stage 910 adjusts rankings of the content items. For example, during a time period that is related to a historical time period of an identified temporal change for content items, the filter 618 can adjust the rankings of content according to the temporal change. As previously described, the filter 618 can facilitate a higher ranking to content items associated with Thanksgiving turkey recipes during November, the period identified by stage 906 as the period of a temporal change. Similarly, the filter 618 can facilitate a lower ranking to a second content item related to vacation resorts in Turkey during the same time period to reflect the lower CTR of the second content item during November.
While implementations have been described in which CTRs are used to identify a temporal changes during a time period for content items responsive to a query, other methods of identifying temporal changes can be also used. For example, one alternative method of identifying a temporal change includes human identification and evaluation of queries and responsive content items and identification of the temporal changes that become apparent during time periods.
Additionally, in some implementations a map search service is used in addition to or instead of a search service. The map search service identifies locations on the map that are more responsive to a map query during a time period. For example, the map search service may identify responsive restaurant locations as those that are open during a time period.
Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible program carrier for execution by, or to control the operation of, data processing apparatus. The tangible program carrier can be a propagated signal or a computer readable medium. The propagated signal is an artificially generated signal, e.g., a machine generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a computer. The computer readable medium can be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of matter effecting a machine readable propagated signal, or a combination of one or more of them.
A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
Additionally, the logic flows and structure block diagrams described in this patent document, which describe particular methods and/or corresponding acts in support of steps and corresponding functions in support of disclosed structural means, may also be utilized to implement corresponding software structures and algorithms, and equivalents thereof. The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices.
Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described is this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client server relationship to each other.
While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Particular embodiments of the subject matter described in this specification have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims (30)

What is claimed is:
1. A method comprising:
determining historical click-through rates over a plurality of time periods for a first search result responsive to a query and for a second search result responsive to the query wherein the first and second search results refer to different respective web pages;
calculating click-fractions for one or more of the plurality of time periods based on the determined historical click-through rates of the first and second search results;
determining that a particular click-fraction of the calculated click-fractions in a first time period of the plurality of time periods exceeds a minimum change threshold;
receiving the query from a user during a second time period that chronologically follows the plurality of time periods;
obtaining search results responsive to the query;
adjusting a ranking of the first search result in the obtained search results during the second time period; and
providing the search results including the adjusted ranking of the first search result to the user.
2. The method of claim 1, wherein the adjustment of the ranking of the first search result is in proportion to a magnitude of a change of the historical click-through rate of the first search result relative to a change of the historical click-through rate of the second search result during the first time period.
3. The method of claim 1, further comprising:
identifying a keyword related to the first search result; and
adding the keyword to the query when the query is submitted to a search engine during the second time period.
4. The method of claim 1, wherein the determined historical click-through rates are for a first geographic region.
5. The method of claim 4, wherein adjusting the ranking of the first search result further comprises determining that a region from which the query is received is the first geographic region.
6. The method of claim 1, wherein a click-fraction of the calculated click fractions is a ratio of the determined historical click-through rate of the first search result in the time period to the determined historical click-through rate of the second search result in the time period.
7. The method of claim 1, wherein the first time period and the second time period are corresponding periods based on a lunar calendar or a Gregorian calendar.
8. The method of claim 1, wherein the first time period and the second time period are each a same day of a week, same days of a week, same annual holidays, same weeks of a month, same weeks of a year, same biweekly periods of a year, same months of a year, or same seasons of a year.
9. The method of claim 1, wherein the minimum change threshold is a standard deviation of the calculated click-fractions.
10. The method of claim 1, wherein the obtained search results include the first search result.
11. A system comprising at least one:
data processing apparatus programmed to perform operations comprising:
determining historical click-through rates over a plurality of time periods for a first search result responsive to a query and for a second search result responsive to the query wherein the first and second search results refer to different respective web pages;
calculating click-fractions for one or more of the plurality of time periods based on the determined historical click-through rates of the first and second search results;
determining that a particular click-fraction of the calculated click-fractions in a first time period of the plurality of time periods exceeds a minimum change threshold;
receiving the query from a user during a second time period that chronologically follows the plurality of time periods;
obtaining search results responsive to the query;
adjusting a ranking of the first search result in the obtained search results during the second time period; and
providing the search results including the adjusted ranking of the first search result to the user.
12. The system of claim 11, wherein the adjustment of the ranking of the first search result is in proportion to a magnitude of a change of the historical click-through rate of the first search result relative to a change of the historical click-through rate of the second search result during the first time period.
13. The system of claim 11, wherein the operations further comprise:
identifying a keyword related to the first search result; and
adding the keyword to the query when the query is submitted to a search engine during the second time period.
14. The system of claim 11, wherein the determined historical click-through rates are for a first geographic region.
15. The system of claim 14, wherein adjusting the ranking of the first search result further comprises determining that a region from which the query is received is the first geographic region.
16. The system of claim 11, wherein a click-fraction of the calculated click fractions is a ratio of the determined historical click-through rate of the first search result in the time period to the determined historical click-through rate of the second search result in the time period.
17. The system of claim 11, wherein the first time period and the second time period are corresponding periods based on a lunar calendar or a Gregorian calendar.
18. The system of claim 11, wherein the first time period and the second time period are each a same day of a week, same days of a week, same annual holidays, same weeks of a month, same weeks of a year, same biweekly periods of a year, same months of a year, or same seasons of a year.
19. The system of claim 11, wherein the minimum change threshold is a standard deviation of the calculated click-fractions.
20. The system of claim 11, wherein the obtained search results include the first search result.
21. A non-transitory machine-readable storage device having instructions stored thereon which, when executed by at least one data processing apparatus, cause the at least one data processing apparatus to perform operations comprising:
determining historical click-through rates over a plurality of time periods for a first search result responsive to a query and for a second search result responsive to the query wherein the first and second search results refer to different respective web pages;
calculating click-fractions for one or more of the plurality of time periods based on the determined historical click-through rates of the first and second search results;
determining that a particular click-fraction of the calculated click-fractions in a first time period of the plurality of time periods exceeds a minimum change threshold;
receiving the query from a user during a second time period that chronologically follows the plurality of time periods;
obtaining search results responsive to the query;
adjusting a ranking of the first search result in the obtained search results during the second time period; and
providing the search results including the adjusted ranking of the first search result to the user.
22. The storage device of claim 21, wherein the adjustment of the ranking of the first search result is in proportion to a magnitude of a change of the historical click-through rate of the first search result relative to a change of the historical click-through rate of the second search result during the first time period.
23. The storage device of claim 21, wherein the operations further comprise:
identifying a keyword related to the first search result; and
adding the keyword to the query when the query is submitted to a search engine during the second time period.
24. The storage device of claim 21, wherein the determined historical click-through rates are for a first geographic region.
25. The storage device of claim 24, wherein adjusting the ranking of the first search result further comprises determining that a region from which the query is received is the first geographic region.
26. The storage device of claim 21, wherein a click-fraction of the calculated click fractions is a ratio of the determined historical click-through rate of the first search result in the time period to the determined historical click-through rate of the second search result in the time period.
27. The storage device of claim 21, wherein the first time period and the second time period are corresponding periods based on a lunar calendar or a Gregorian calendar.
28. The storage device of claim 21, wherein the first time period and the second time period are each a same day of a week, same days of a week, same annual holidays, same weeks of a month, same weeks of a year, same biweekly periods of a year, same months of a year, or same seasons of a year.
29. The storage device of claim 21, wherein the minimum change threshold is a standard deviation of the calculated click-fractions.
30. The storage device of claim 21, wherein the obtained search results include the first search result.
US14/563,911 2007-10-11 2014-12-08 Time based ranking Active US9152678B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US14/563,911 US9152678B1 (en) 2007-10-11 2014-12-08 Time based ranking

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US11/870,893 US8909655B1 (en) 2007-10-11 2007-10-11 Time based ranking
US14/563,911 US9152678B1 (en) 2007-10-11 2014-12-08 Time based ranking

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US11/870,893 Continuation US8909655B1 (en) 2007-10-11 2007-10-11 Time based ranking

Publications (1)

Publication Number Publication Date
US9152678B1 true US9152678B1 (en) 2015-10-06

Family

ID=52001786

Family Applications (2)

Application Number Title Priority Date Filing Date
US11/870,893 Expired - Fee Related US8909655B1 (en) 2007-10-11 2007-10-11 Time based ranking
US14/563,911 Active US9152678B1 (en) 2007-10-11 2014-12-08 Time based ranking

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US11/870,893 Expired - Fee Related US8909655B1 (en) 2007-10-11 2007-10-11 Time based ranking

Country Status (1)

Country Link
US (2) US8909655B1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130067364A1 (en) * 2011-09-08 2013-03-14 Microsoft Corporation Presenting search result items having varied prominence

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080307339A1 (en) 2006-03-20 2008-12-11 Kidzui, Inc. Child-oriented computing system
US8661029B1 (en) 2006-11-02 2014-02-25 Google Inc. Modifying search result ranking based on implicit user feedback
US9348912B2 (en) 2007-10-18 2016-05-24 Microsoft Technology Licensing, Llc Document length as a static relevance feature for ranking search results
US8868741B2 (en) * 2008-03-03 2014-10-21 Leapfrog Enterprises, Inc. Method and apparatus for custodial monitoring, filtering, and approving of content
US8713016B2 (en) 2008-12-24 2014-04-29 Comcast Interactive Media, Llc Method and apparatus for organizing segments of media assets and determining relevance of segments to a query
US11531668B2 (en) 2008-12-29 2022-12-20 Comcast Interactive Media, Llc Merging of multiple data sets
US9396258B2 (en) * 2009-01-22 2016-07-19 Google Inc. Recommending video programs
US8176043B2 (en) * 2009-03-12 2012-05-08 Comcast Interactive Media, Llc Ranking search results
US9892730B2 (en) 2009-07-01 2018-02-13 Comcast Interactive Media, Llc Generating topic-specific language models
US8972391B1 (en) 2009-10-02 2015-03-03 Google Inc. Recent interest based relevance scoring
US9262532B2 (en) * 2010-07-30 2016-02-16 Yahoo! Inc. Ranking entity facets using user-click feedback
WO2012044305A1 (en) * 2010-09-30 2012-04-05 Hewlett-Packard Development Company, L.P. Identification of events of interest
US8527483B2 (en) * 2011-02-04 2013-09-03 Mikko VÄÄNÄNEN Method and means for browsing by walking
US9495462B2 (en) * 2012-01-27 2016-11-15 Microsoft Technology Licensing, Llc Re-ranking search results
US9183499B1 (en) 2013-04-19 2015-11-10 Google Inc. Evaluating quality based on neighbor features
JP5633944B1 (en) * 2013-06-02 2014-12-03 データ・サイエンティスト株式会社 Evaluation method, evaluation apparatus, and program
JP5997738B2 (en) * 2014-09-11 2016-09-28 ヤフー株式会社 Information providing system, information providing server, information providing method and program
US10509790B2 (en) * 2015-06-07 2019-12-17 Apple Inc. Electronic message searching
US10380187B2 (en) * 2015-10-30 2019-08-13 International Business Machines Corporation System, method, and recording medium for knowledge graph augmentation through schema extension
CN108009170B (en) * 2016-11-02 2020-11-06 北京酷我科技有限公司 Searching method and device
CN108073794B (en) * 2016-11-18 2022-03-15 上海优扬新媒信息技术有限公司 Authentication method and device
RU2692045C1 (en) * 2018-05-18 2019-06-19 Общество С Ограниченной Ответственностью "Яндекс" Method and system for recommending fresh suggest search requests in a search engine
CN112148700A (en) * 2020-10-12 2020-12-29 平安科技(深圳)有限公司 Log data processing method and device, computer equipment and storage medium

Citations (268)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5265065A (en) 1991-10-08 1993-11-23 West Publishing Company Method and apparatus for information retrieval from a database by replacing domain specific stemmed phases in a natural language to create a search query
US5488725A (en) 1991-10-08 1996-01-30 West Publishing Company System of document representation retrieval by successive iterated probability sampling
US5696962A (en) 1993-06-24 1997-12-09 Xerox Corporation Method for computerized information retrieval using shallow linguistic analysis
US5920854A (en) 1996-08-14 1999-07-06 Infoseek Corporation Real-time document collection search engine with phrase indexing
US5963940A (en) 1995-08-16 1999-10-05 Syracuse University Natural language information retrieval system and method
US6006222A (en) 1997-04-25 1999-12-21 Culliss; Gary Method for organizing information
US6014665A (en) 1997-08-01 2000-01-11 Culliss; Gary Method for organizing information
US6026388A (en) 1995-08-16 2000-02-15 Textwise, Llc User interface and other enhancements for natural language information retrieval system and method
US6067565A (en) 1998-01-15 2000-05-23 Microsoft Corporation Technique for prefetching a web page of potential future interest in lieu of continuing a current information download
US6076051A (en) 1997-03-07 2000-06-13 Microsoft Corporation Information retrieval utilizing semantic representation of text
US6078916A (en) 1997-08-01 2000-06-20 Culliss; Gary Method for organizing information
US6078917A (en) 1997-12-18 2000-06-20 International Business Machines Corporation System for searching internet using automatic relevance feedback
US6088692A (en) 1994-12-06 2000-07-11 University Of Central Florida Natural language method and system for searching for and ranking relevant documents from a computer database
US6134532A (en) 1997-11-14 2000-10-17 Aptex Software, Inc. System and method for optimal adaptive matching of users to most relevant entity and information in real-time
WO2000077689A1 (en) 1999-06-16 2000-12-21 Triogo, Inc. A process for improving search engine efficiency using user feedback
US6182068B1 (en) 1997-08-01 2001-01-30 Ask Jeeves, Inc. Personalized search methods
US6182066B1 (en) 1997-11-26 2001-01-30 International Business Machines Corp. Category processing of query topics and electronic document content topics
US6185559B1 (en) 1997-05-09 2001-02-06 Hitachi America, Ltd. Method and apparatus for dynamically counting large itemsets
WO2001016807A1 (en) 1999-08-31 2001-03-08 Mindcrossing, Inc. An internet search system for tracking and ranking selected records from a previous search
US20010000356A1 (en) 1995-07-07 2001-04-19 Woods William A. Method and apparatus for generating query responses in a computer-based document retrieval system
US6249252B1 (en) 1996-09-09 2001-06-19 Tracbeam Llc Wireless location using multiple location estimators
US6285999B1 (en) 1997-01-10 2001-09-04 The Board Of Trustees Of The Leland Stanford Junior University Method for node ranking in a linked database
WO2001067297A1 (en) 2000-03-07 2001-09-13 Tzunami Inc. System and method for computer searching
US6321228B1 (en) 1999-08-31 2001-11-20 Powercast Media, Inc. Internet search system for retrieving selected results from a previous search
US6327590B1 (en) 1999-05-05 2001-12-04 Xerox Corporation System and method for collaborative ranking of search results employing user and group profiles derived from document collection content analysis
US6341283B1 (en) 1998-05-21 2002-01-22 Fujitsu Limited Apparatus for data decomposition and method and storage medium therefor
US6353849B1 (en) 1996-12-20 2002-03-05 Intel Corporation System and server for providing customized web information based on attributes of the requestor
US20020034292A1 (en) 2000-08-22 2002-03-21 Tuoriniemi Veijo M. System and a method to match demand and supply based on geographical location derived from a positioning system
US6363378B1 (en) 1998-10-13 2002-03-26 Oracle Corporation Ranking of query feedback terms in an information retrieval system
US6370526B1 (en) 1999-05-18 2002-04-09 International Business Machines Corporation Self-adaptive method and system for providing a user-preferred ranking order of object sets
US20020042791A1 (en) 2000-07-06 2002-04-11 Google, Inc. Methods and apparatus for using a modified index to provide search results in response to an ambiguous search query
US20020049752A1 (en) 1998-03-03 2002-04-25 Dwayne Bowman Identifying the items most relevant to a current query based on items selected in connection with similar queries
US6421675B1 (en) 1998-03-16 2002-07-16 S. L. I. Systems, Inc. Search engine
US20020103790A1 (en) 2001-01-30 2002-08-01 Wang Shirley S. Utility for cross platform database query
US20020123988A1 (en) 2001-03-02 2002-09-05 Google, Inc. Methods and apparatus for employing usage statistics in document retrieval
US20020133481A1 (en) 2000-07-06 2002-09-19 Google, Inc. Methods and apparatus for providing search results in response to an ambiguous search query
US20020165849A1 (en) 1999-05-28 2002-11-07 Singh Narinder Pal Automatic advertiser notification for a system for providing place and price protection in a search result list generated by a computer network search engine
US6480843B2 (en) 1998-11-03 2002-11-12 Nec Usa, Inc. Supporting web-query expansion efficiently using multi-granularity indexing and query processing
US6490575B1 (en) 1999-12-06 2002-12-03 International Business Machines Corporation Distributed network search engine
US20030009399A1 (en) 2001-03-22 2003-01-09 Boerner Sean T. Method and system to identify discrete trends in time series
US20030018707A1 (en) 2001-07-20 2003-01-23 Flocken Philip Andrew Server-side filter for corrupt web-browser cookies
US20030028529A1 (en) 2001-08-03 2003-02-06 Cheung Dominic Dough-Ming Search engine account monitoring
US20030037074A1 (en) 2001-05-01 2003-02-20 Ibm Corporation System and method for aggregating ranking results from various sources to improve the results of web searching
US6526440B1 (en) 2001-01-30 2003-02-25 Google, Inc. Ranking search results by reranking the results based on local inter-connectivity
US20030078914A1 (en) 2001-10-18 2003-04-24 Witbrock Michael J. Search results using editor feedback
US6560590B1 (en) 2000-02-14 2003-05-06 Kana Software, Inc. Method and apparatus for multiple tiered matching of natural language queries to positions in a text corpus
US6567103B1 (en) 2000-08-02 2003-05-20 Verity, Inc. Graphical search results system and method
US20030120654A1 (en) 2000-01-14 2003-06-26 International Business Machines Corporation Metadata search results ranking system
US6587848B1 (en) 2000-03-08 2003-07-01 International Business Machines Corporation Methods and apparatus for performing an affinity based similarity search
US20030135490A1 (en) 2002-01-15 2003-07-17 Barrett Michael E. Enhanced popularity ranking
US20030149704A1 (en) 2002-02-05 2003-08-07 Hitachi, Inc. Similarity-based search method by relevance feedback
US6615209B1 (en) 2000-02-22 2003-09-02 Google, Inc. Detecting query-specific duplicate documents
US20030167252A1 (en) 2002-02-26 2003-09-04 Pliant Technologies, Inc. Topic identification and use thereof in information retrieval systems
US6623529B1 (en) 1998-02-23 2003-09-23 David Lakritz Multilingual electronic document translation, management, and delivery system
US20030195877A1 (en) 1999-12-08 2003-10-16 Ford James L. Search query processing to provide category-ranked presentation of search results
US20030204495A1 (en) 2002-04-30 2003-10-30 Lehnert Bernd R. Data gathering
US20030220913A1 (en) 2002-05-24 2003-11-27 International Business Machines Corporation Techniques for personalized and adaptive search services
US6658423B1 (en) 2001-01-24 2003-12-02 Google, Inc. Detecting duplicate and near-duplicate files
US20030229640A1 (en) 2002-06-07 2003-12-11 International Business Machines Corporation Parallel database query processing for non-uniform data sources via buffered access
US6671681B1 (en) 2000-05-31 2003-12-30 International Business Machines Corporation System and technique for suggesting alternate query expressions based on prior user selections and their query strings
US20040006456A1 (en) 1998-11-30 2004-01-08 Wayne Loofbourrow Multi-language document search and retrieval system
US20040006740A1 (en) 2000-09-29 2004-01-08 Uwe Krohn Information access
US6678681B1 (en) 1999-03-10 2004-01-13 Google Inc. Information extraction from a database
US20040034632A1 (en) 2002-07-31 2004-02-19 International Business Machines Corporation Automatic query refinement
US6701309B1 (en) 2000-04-21 2004-03-02 Lycos, Inc. Method and system for collecting related queries
US20040049486A1 (en) 2000-04-18 2004-03-11 Scanlon Henry R. Image relationships derived from thresholding of historically tracked user data for facilitating image based searching
US20040059708A1 (en) 2002-09-24 2004-03-25 Google, Inc. Methods and apparatus for serving relevant advertisements
US20040083205A1 (en) 2002-10-29 2004-04-29 Steve Yeager Continuous knowledgebase access improvement systems and methods
US20040093325A1 (en) 2002-11-07 2004-05-13 International Business Machines Corporation System and method for location influenced network search
US6738764B2 (en) 2001-05-08 2004-05-18 Verity, Inc. Apparatus and method for adaptively ranking search results
US6754873B1 (en) 1999-09-20 2004-06-22 Google Inc. Techniques for finding related hyperlinked documents using link-based analysis
US20040122811A1 (en) 1997-01-10 2004-06-24 Google, Inc. Method for searching media
US20040119740A1 (en) 2002-12-24 2004-06-24 Google, Inc., A Corporation Of The State Of California Methods and apparatus for displaying and replying to electronic messages
WO2004059514A1 (en) 2002-12-24 2004-07-15 American Type Culture Collection Systems and methods for enabling a user to find information of interest to the user
US20040153472A1 (en) 2003-01-31 2004-08-05 Rieffanaugh Neal King Human resource networking system and method thereof
US20040158560A1 (en) 2003-02-12 2004-08-12 Ji-Rong Wen Systems and methods for query expansion
US6792416B2 (en) 1999-09-21 2004-09-14 International Business Machines Corporation Managing results of federated searches across heterogeneous datastores with a federated result set cursor object
US6795820B2 (en) 2001-06-20 2004-09-21 Nextpage, Inc. Metasearch technique that ranks documents obtained from multiple collections
US20040186996A1 (en) 2000-03-29 2004-09-23 Gibbs Benjamin K. Unique digital signature
US20040199419A1 (en) 2001-11-13 2004-10-07 International Business Machines Corporation Promoting strategic documents by bias ranking of search results on a web browser
US20040215607A1 (en) 2003-04-25 2004-10-28 Travis Robert L. Method and system fo blending search engine results from disparate sources into one search result
US20050015366A1 (en) 2003-07-18 2005-01-20 Carrasco John Joseph M. Disambiguation of search phrases using interpretation clusters
US20050027691A1 (en) 2003-07-28 2005-02-03 Sergey Brin System and method for providing a user interface with search query broadening
US6853993B2 (en) 1998-07-15 2005-02-08 A9.Com, Inc. System and methods for predicting correct spellings of terms in multiple-term search queries
US20050033803A1 (en) 2003-07-02 2005-02-10 Vleet Taylor N. Van Server architecture and methods for persistently storing and serving event data
US20050050014A1 (en) 2003-08-29 2005-03-03 Gosse David B. Method, device and software for querying and presenting search results
US20050055342A1 (en) 2000-11-08 2005-03-10 Bharat Krishna Asur Method for estimating coverage of Web search engines
US20050055345A1 (en) 2002-02-14 2005-03-10 Infoglide Software Corporation Similarity search engine for use with relational databases
US20050060311A1 (en) 2003-09-12 2005-03-17 Simon Tong Methods and systems for improving a search ranking using related queries
US20050060290A1 (en) 2003-09-15 2005-03-17 International Business Machines Corporation Automatic query routing and rank configuration for search queries in an information retrieval system
US20050060310A1 (en) 2003-09-12 2005-03-17 Simon Tong Methods and systems for improving a search ranking using population information
US6873982B1 (en) 1999-07-16 2005-03-29 International Business Machines Corporation Ordering of database search results based on user feedback
US20050071741A1 (en) 2003-09-30 2005-03-31 Anurag Acharya Information retrieval based on historical data
US6877002B2 (en) 2000-11-21 2005-04-05 America Online, Inc. Fuzzy database retrieval
US6882999B2 (en) 1999-05-03 2005-04-19 Microsoft Corporation URL mapping methods and systems
US20050102282A1 (en) 2003-11-07 2005-05-12 Greg Linden Method for personalized search
US6901402B1 (en) 1999-06-18 2005-05-31 Microsoft Corporation System for improving the performance of information retrieval-type tasks by identifying the relations of constituents
US20050125376A1 (en) 2003-12-08 2005-06-09 Andy Curtis Methods and systems for providing a response to a query
US6912505B2 (en) 1998-09-18 2005-06-28 Amazon.Com, Inc. Use of product viewing histories of users to identify related products
US20050160083A1 (en) 2004-01-16 2005-07-21 Yahoo! Inc. User-specific vertical search
US20050192946A1 (en) 2003-12-29 2005-09-01 Yahoo! Inc. Lateral search
US20050198026A1 (en) 2004-02-03 2005-09-08 Dehlinger Peter J. Code, system, and method for generating concepts
US6944612B2 (en) 2002-11-13 2005-09-13 Xerox Corporation Structured contextual clustering method and system in a federated search engine
US6944611B2 (en) 2000-08-28 2005-09-13 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
US20050222987A1 (en) 2004-04-02 2005-10-06 Vadon Eric R Automated detection of associations between search criteria and item categories based on collective analysis of user activity data
US20050222998A1 (en) 2004-03-31 2005-10-06 Oce-Technologies B.V. Apparatus and computerised method for determining constituent words of a compound word
US6954750B2 (en) 2000-10-10 2005-10-11 Content Analyst Company, Llc Method and system for facilitating the refinement of data queries
US20050240580A1 (en) 2003-09-30 2005-10-27 Zamir Oren E Personalization of placed content ordering in search results
US20050240576A1 (en) 2003-06-10 2005-10-27 John Piscitello Named URL entry
US20050256848A1 (en) 2004-05-13 2005-11-17 International Business Machines Corporation System and method for user rank search
US6990453B2 (en) 2000-07-31 2006-01-24 Landmark Digital Services Llc System and methods for recognizing sound and music signals in high noise and distortion
US20060047643A1 (en) 2004-08-31 2006-03-02 Chirag Chaman Method and system for a personalized search engine
US7016939B1 (en) 2001-07-26 2006-03-21 Mcafee, Inc. Intelligent SPAM detection system using statistical analysis
US20060069667A1 (en) 2004-09-30 2006-03-30 Microsoft Corporation Content evaluation
US20060074903A1 (en) 2004-09-30 2006-04-06 Microsoft Corporation System and method for ranking search results using click distance
US7028027B1 (en) 2002-09-17 2006-04-11 Yahoo! Inc. Associating documents with classifications and ranking documents based on classification weights
US20060089926A1 (en) 2004-10-27 2006-04-27 Harris Corporation, Corporation Of The State Of Delaware Method for re-ranking documents retrieved from a document database
US20060095421A1 (en) 2004-10-22 2006-05-04 Canon Kabushiki Kaisha Method, apparatus, and program for searching for data
US20060106793A1 (en) 2003-12-29 2006-05-18 Ping Liang Internet and computer information retrieval and mining with intelligent conceptual filtering, visualization and automation
US7072886B2 (en) 2001-05-15 2006-07-04 Nokia Corporation Method and business process to maintain privacy in distributed recommendation systems
US7085761B2 (en) 2002-06-28 2006-08-01 Fujitsu Limited Program for changing search results rank, recording medium for recording such a program, and content search processing method
US20060173830A1 (en) 2003-07-23 2006-08-03 Barry Smyth Information retrieval
US20060195443A1 (en) 2005-02-11 2006-08-31 Franklin Gary L Information prioritisation system and method
US20060200476A1 (en) 2005-03-03 2006-09-07 Microsoft Corporation Creating, storing and viewing process models
US20060200556A1 (en) 2004-12-29 2006-09-07 Scott Brave Method and apparatus for identifying, extracting, capturing, and leveraging expertise and knowledge
US7113939B2 (en) 1999-09-21 2006-09-26 International Business Machines Corporation Architecture to enable search gateways as part of federated search
US7117206B1 (en) 1998-01-15 2006-10-03 Overture Services, Inc. Method for ranking hyperlinked pages using content and connectivity analysis
US20060227992A1 (en) 2005-04-08 2006-10-12 Rathus Spencer A System and method for accessing electronic data via an image search engine
US20060230040A1 (en) 2003-12-08 2006-10-12 Andy Curtis Methods and systems for providing a response to a query
US7136849B2 (en) 2001-08-10 2006-11-14 International Business Machines Corporation Method systems and computer program products for indicating links to external URLs
US7146361B2 (en) 2003-05-30 2006-12-05 International Business Machines Corporation System, method and computer program product for performing unstructured information management and automatic text analysis, including a search operator functioning as a Weighted AND (WAND)
US20060293950A1 (en) 2005-06-28 2006-12-28 Microsoft Corporation Automatic ad placement
US20070005588A1 (en) 2005-07-01 2007-01-04 Microsoft Corporation Determining relevance using queries as surrogate content
US20070005575A1 (en) 2005-06-30 2007-01-04 Microsoft Corporation Prioritizing search results by client search satisfaction
US20070038659A1 (en) 2005-08-15 2007-02-15 Google, Inc. Scalable user clustering based on set similarity
US20070050339A1 (en) 2005-08-24 2007-03-01 Richard Kasperski Biasing queries to determine suggested queries
US20070061211A1 (en) 2005-09-14 2007-03-15 Jorey Ramer Preventing mobile communication facility click fraud
US20070061195A1 (en) 2005-09-13 2007-03-15 Yahoo! Inc. Framework for selecting and delivering advertisements over a network based on combined short-term and long-term user behavioral interests
US20070081197A1 (en) 2001-06-22 2007-04-12 Nosa Omoigui System and method for semantic knowledge retrieval, management, capture, sharing, discovery, delivery and presentation
US20070106659A1 (en) 2005-03-18 2007-05-10 Yunshan Lu Search engine that applies feedback from users to improve search results
US20070112730A1 (en) 2005-11-07 2007-05-17 Antonino Gulli Sampling Internet user traffic to improve search results
US7222127B1 (en) 2003-11-14 2007-05-22 Google Inc. Large scale machine learning systems and methods
US20070130370A1 (en) 2005-12-06 2007-06-07 Emeka Akaezuwa Portable search engine
US20070156677A1 (en) 1999-07-21 2007-07-05 Alberti Anemometer Llc Database access system
US7243102B1 (en) 2004-07-01 2007-07-10 Microsoft Corporation Machine directed improvement of ranking algorithms
US20070172155A1 (en) 2006-01-21 2007-07-26 Elizabeth Guckenberger Photo Automatic Linking System and method for accessing, linking, and visualizing "key-face" and/or multiple similar facial images along with associated electronic data via a facial image recognition search engine
US20070180355A1 (en) 2006-02-01 2007-08-02 Ricoh Co., Ltd. Enhancing accuracy of jumping by incorporating interestingness estimates
US20070192190A1 (en) 2005-12-06 2007-08-16 Authenticlick Method and system for scoring quality of traffic to network sites
US7266765B2 (en) 2001-08-31 2007-09-04 Fuji Xerox Co., Ltd. Detection and processing of annotated anchors
US20070208730A1 (en) 2006-03-02 2007-09-06 Microsoft Corporation Mining web search user behavior to enhance web search relevance
US20070214131A1 (en) 2006-03-13 2007-09-13 Microsoft Corporation Re-ranking search results based on query log
US20070233653A1 (en) 2006-03-31 2007-10-04 Microsoft Corporation Selecting directly bid upon advertisements for display
US20070255689A1 (en) 2006-04-28 2007-11-01 Gordon Sun System and method for indexing web content using click-through features
US7293016B1 (en) 2004-01-22 2007-11-06 Microsoft Corporation Index partitioning based on document relevance for document indexes
US20070260597A1 (en) 2006-05-02 2007-11-08 Mark Cramer Dynamic search engine results employing user behavior
US20070260596A1 (en) 2006-03-29 2007-11-08 Koran Joshua M Behavioral targeting system
US20070266439A1 (en) 2005-11-30 2007-11-15 Harold Kraft Privacy management and transaction system
US20070266021A1 (en) 2006-03-06 2007-11-15 Murali Aravamudan Methods and systems for selecting and presenting content based on dynamically identifying microgenres associated with the content
US20070288450A1 (en) 2006-04-19 2007-12-13 Datta Ruchira S Query language determination using query terms and interface language
US20080010143A1 (en) 2006-06-22 2008-01-10 Rob Kniaz Secure and extensible pay per action online advertising
US20080027913A1 (en) 2006-07-25 2008-01-31 Yahoo! Inc. System and method of information retrieval engine evaluation using human judgment input
US20080052219A1 (en) 2006-03-31 2008-02-28 Combinenet, Inc. System for and method of expressive auctions of user events
US20080052273A1 (en) 2006-08-22 2008-02-28 Fuji Xerox Co., Ltd. Apparatus and method for term context modeling for information retrieval
US20080059453A1 (en) 2006-08-29 2008-03-06 Raphael Laderman System and method for enhancing the result of a query
US20080077570A1 (en) 2004-10-25 2008-03-27 Infovell, Inc. Full Text Query and Search Systems and Method of Use
US20080082518A1 (en) 2006-09-29 2008-04-03 Loftesness David E Strategy for Providing Query Results Based on Analysis of User Intent
US20080091650A1 (en) 2006-10-11 2008-04-17 Marcus Felipe Fontoura Augmented Search With Error Detection and Replacement
US20080104043A1 (en) 2006-10-25 2008-05-01 Ashutosh Garg Server-side match
US20080114750A1 (en) 2006-11-14 2008-05-15 Microsoft Corporation Retrieval and ranking of items utilizing similarity
US20080114624A1 (en) 2006-11-13 2008-05-15 Microsoft Corporation Click-fraud protector
US20080114729A1 (en) 2006-11-13 2008-05-15 Google Inc. Computer-implemented interactive, virtual bookshelf system and method
US7379951B2 (en) 2002-05-31 2008-05-27 Microsoft Corporation Support for real-time queries concerning current state, data and history of a process
US7382358B2 (en) 2003-01-16 2008-06-03 Forword Input, Inc. System and method for continuous stroke word-based text input
US20080140699A1 (en) 2005-11-09 2008-06-12 Rosie Jones System and method for generating substitutable queries
US7395222B1 (en) 2000-09-07 2008-07-01 Sotos John G Method and system for identifying expertise
US20080162475A1 (en) 2007-01-03 2008-07-03 Meggs Anthony F Click-fraud detection method
US20080183660A1 (en) 2007-01-30 2008-07-31 Google Inc. Content identification expansion
US20080189269A1 (en) 2006-11-07 2008-08-07 Fast Search & Transfer Asa Relevance-weighted navigation in information access, search and retrieval
US20080208825A1 (en) 2003-12-08 2008-08-28 Andy Curtis Methods and systems for providing a response to a query
US7426507B1 (en) 2004-07-26 2008-09-16 Google, Inc. Automatic taxonomy generation in search results using phrases
US20080228442A1 (en) 2007-03-07 2008-09-18 Lisa Ellen Lippincott Statistical data inspector
US20080256050A1 (en) 2007-04-10 2008-10-16 Ruofei Zhang System and method for modeling user selection feedback in a search result page
US7451487B2 (en) 2003-09-08 2008-11-11 Sonicwall, Inc. Fraudulent message detection
US20080313168A1 (en) 2007-06-18 2008-12-18 Microsoft Corporation Ranking documents based on a series of document graphs
US20080313247A1 (en) 2007-06-12 2008-12-18 Brian Galvin Page Ranking Based on a Behavioral WEB Graph
US20090012969A1 (en) 2007-07-02 2009-01-08 Rail Peter D Systems and processes for evaluating webpages
US20090055392A1 (en) 2003-03-31 2009-02-26 Google Inc. Ordering of search results based on language and/or country of the search results
US20090070194A1 (en) 2007-09-06 2009-03-12 Nhn Corporation Method of providing estimated cost for keyword advertisement
US7516146B2 (en) 2003-05-15 2009-04-07 Microsoft Corporation Fast adaptive document filtering
US7526470B1 (en) 2003-05-28 2009-04-28 Microsoft Corporation System and method for measuring and improving search result relevance based on user satisfaction
US7533130B2 (en) 2006-12-19 2009-05-12 Yahoo! Inc. User behavior reporting based on pre-aggregated activity data
US7533092B2 (en) 2004-10-28 2009-05-12 Yahoo! Inc. Link-based spam detection
US20090157643A1 (en) 2007-12-12 2009-06-18 Microsoft Corporation Semi-supervised part-of-speech tagging
US7552112B2 (en) 2006-09-18 2009-06-23 Yahoo! Inc. Discovering associative intent queries from search web logs
US20090171943A1 (en) 2008-01-02 2009-07-02 Microsoft Corporation Modifying relevance ranking of search result items
US20090182723A1 (en) 2008-01-10 2009-07-16 Microsoft Corporation Ranking search results using author extraction
US7565363B2 (en) 1999-04-01 2009-07-21 Anwar Mohammed S Search engine with user activity memory
US7566363B2 (en) 2007-07-24 2009-07-28 Silverbrook Research Pty Ltd Alternative phthalocyanine dyes suitable for use in offset inks
US7574530B2 (en) 2005-03-10 2009-08-11 Microsoft Corporation Method and system for web resource location classification and detection
US7584181B2 (en) 2003-09-30 2009-09-01 Microsoft Corporation Implicit links search enhancement system and method for search engines using implicit links generated by mining user access patterns
US20090228442A1 (en) 2008-03-10 2009-09-10 Searchme, Inc. Systems and methods for building a document index
US7610282B1 (en) 2007-03-30 2009-10-27 Google Inc. Rank-adjusted content items
US20090287656A1 (en) 2008-05-13 2009-11-19 Bennett James D Network search engine utilizing client browser favorites
US7636714B1 (en) 2005-03-31 2009-12-22 Google Inc. Determining query term synonyms within query context
US7657626B1 (en) 2006-09-19 2010-02-02 Enquisite, Inc. Click fraud detection
US7680775B2 (en) 2005-12-13 2010-03-16 Iac Search & Media, Inc. Methods and systems for generating query and result-based relevance indexes
US7693818B2 (en) 2005-11-15 2010-04-06 Microsoft Corporation UserRank: ranking linked nodes leveraging user logs
US20100106706A1 (en) 2000-05-22 2010-04-29 Yahoo! Inc. Method and apparatus for identifying related searches in a database search system
US7716225B1 (en) 2004-06-17 2010-05-11 Google Inc. Ranking documents based on user behavior and/or feature data
US20100131563A1 (en) 2008-11-25 2010-05-27 Hongfeng Yin System and methods for automatic clustering of ranked and categorized search objects
US7747612B2 (en) 2005-10-31 2010-06-29 Yahoo! Inc. Indication of exclusive items in a result set
US7756887B1 (en) 2004-12-30 2010-07-13 Google Inc. System and method for modulating search relevancy using pointer activity monitoring
US7769740B2 (en) 2007-12-21 2010-08-03 Yahoo! Inc. Systems and methods of ranking attention
US20100205541A1 (en) 2009-02-11 2010-08-12 Jeffrey A. Rapaport social network driven indexing system for instantly clustering people with concurrent focus on same topic into on-topic chat rooms and/or for generating on-topic search results tailored to user preferences regarding topic
US7783632B2 (en) 2005-11-03 2010-08-24 Microsoft Corporation Using popularity data for ranking
US20100228738A1 (en) 2009-03-04 2010-09-09 Mehta Rupesh R Adaptive document sampling for information extraction
US7801885B1 (en) 2007-01-25 2010-09-21 Neal Akash Verma Search engine system and method with user feedback on search results
US20100241472A1 (en) 2009-03-23 2010-09-23 David Antonio Hernandez Illness specific diagnostic system
US7809716B2 (en) 2006-06-27 2010-10-05 International Business Machines Corporation Method and apparatus for establishing relationship between documents
US7818320B2 (en) 2007-05-31 2010-10-19 Yahoo! Inc. Enhanced search results based on user feedback relating to search result abstracts
US7836058B2 (en) 2008-03-27 2010-11-16 Microsoft Corporation Web searching
US7844589B2 (en) 2003-11-18 2010-11-30 Yahoo! Inc. Method and apparatus for performing a search
US7849089B2 (en) 2005-05-10 2010-12-07 Microsoft Corporation Method and system for adapting search results to personal information needs
US7853557B2 (en) 2002-06-14 2010-12-14 Siebel Systems, Inc. Method and computer for responding to a query according to the language used
US7877404B2 (en) 2008-03-05 2011-01-25 Microsoft Corporation Query classification based on query click logs
US7895177B2 (en) 2007-05-29 2011-02-22 Yahoo! Inc. Enabling searching of user ratings and reviews using user profile location, and social networks
US7925498B1 (en) 2006-12-29 2011-04-12 Google Inc. Identifying a synonym with N-gram agreement for a query phrase
US20110087656A1 (en) 2009-10-13 2011-04-14 Electronics And Telecommunications Research Institute Apparatus for question answering based on answer trustworthiness and method thereof
US7953740B1 (en) 2006-02-13 2011-05-31 Amazon Technologies, Inc. Detection of behavior-based associations between search strings and items
US7974974B2 (en) 2008-03-20 2011-07-05 Microsoft Corporation Techniques to perform relative ranking for search results
US7987185B1 (en) 2006-12-29 2011-07-26 Google Inc. Ranking custom search results
US8001136B1 (en) 2007-07-10 2011-08-16 Google Inc. Longest-common-subsequence detection for common synonyms
US20110219025A1 (en) 1997-10-27 2011-09-08 Massachusetts Institute Of Technology Image search using images in query
US8019650B2 (en) 2005-01-21 2011-09-13 Amazon Technologies, Inc. Method and system for producing item comparisons
US8024330B1 (en) 2004-05-20 2011-09-20 Hector Franco Collaborative incident alert system for mobile devices
US8027439B2 (en) 2006-09-18 2011-09-27 Fair Isaac Corporation Self-calibrating fraud detection
US8037042B2 (en) 2007-05-10 2011-10-11 Microsoft Corporation Automated analysis of user search behavior
US8037086B1 (en) 2007-07-10 2011-10-11 Google Inc. Identifying common co-occurring elements in lists
US8037043B2 (en) 2008-09-09 2011-10-11 Microsoft Corporation Information retrieval system
US8051061B2 (en) 2007-07-20 2011-11-01 Microsoft Corporation Cross-lingual query suggestion
US8060456B2 (en) 2008-10-01 2011-11-15 Microsoft Corporation Training a search result ranker with automatically-generated samples
US8060497B1 (en) 2009-07-23 2011-11-15 Google Inc. Framework for evaluating web search scoring functions
US20110282906A1 (en) 2010-05-14 2011-11-17 Rovi Technologies Corporation Systems and methods for performing a search based on a media content snapshot image
US8065296B1 (en) 2004-09-29 2011-11-22 Google Inc. Systems and methods for determining a quality of provided items
US8069182B2 (en) 2006-04-24 2011-11-29 Working Research, Inc. Relevancy-based domain classification
US20110295844A1 (en) 2010-05-27 2011-12-01 Microsoft Corporation Enhancing freshness of search results
US20110295879A1 (en) 2010-05-27 2011-12-01 Neuone, Llc Systems and methods for document management
US8073263B2 (en) 2006-07-31 2011-12-06 Ricoh Co., Ltd. Multi-classifier selection and monitoring for MMR-based image recognition
US8073772B2 (en) 1999-11-05 2011-12-06 American Express Travel Related Services Company, Inc. Systems and methods for processing transactions using multiple budgets
US8086599B1 (en) 2006-10-24 2011-12-27 Google Inc. Method and apparatus for automatically identifying compunds
US8086690B1 (en) 2003-09-22 2011-12-27 Google Inc. Determining geographical relevance of web documents
US8090717B1 (en) 2002-09-20 2012-01-03 Google Inc. Methods and apparatus for ranking documents
US8126839B2 (en) 2008-06-19 2012-02-28 Yahoo! Inc. Methods and apparatuses for adapting a ranking function of a search engine for use with a specific domain
US8156111B2 (en) 2008-11-24 2012-04-10 Yahoo! Inc. Identifying and expanding implicitly temporally qualified queries
US8171041B2 (en) 2008-05-15 2012-05-01 Enpulz, L.L.C. Support for international search terms
US8396865B1 (en) 2008-12-10 2013-03-12 Google Inc. Sharing search engine relevance data between corpora
US8412699B1 (en) 2009-06-12 2013-04-02 Google Inc. Fresh related search suggestions
US8447760B1 (en) 2009-07-20 2013-05-21 Google Inc. Generating a related set of documents for an initial set of documents
US8458165B2 (en) 2007-06-28 2013-06-04 Oracle International Corporation System and method for applying ranking SVM in query relaxation
US8498974B1 (en) 2009-08-31 2013-07-30 Google Inc. Refining search results
US8521725B1 (en) 2003-12-03 2013-08-27 Google Inc. Systems and methods for improved searching
US8615514B1 (en) 2010-02-03 2013-12-24 Google Inc. Evaluating website properties by partitioning user feedback
US8661029B1 (en) 2006-11-02 2014-02-25 Google Inc. Modifying search result ranking based on implicit user feedback
US8661012B1 (en) 2006-12-29 2014-02-25 Google Inc. Ensuring that a synonym for a query phrase does not drop information present in the query phrase
US8694374B1 (en) 2007-03-14 2014-04-08 Google Inc. Detecting click spam
US8694511B1 (en) 2007-08-20 2014-04-08 Google Inc. Modifying search result ranking based on populations
US8832083B1 (en) 2010-07-23 2014-09-09 Google Inc. Combining user feedback
US8898153B1 (en) 2009-11-20 2014-11-25 Google Inc. Modifying scoring data based on historical changes

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6636848B1 (en) * 2000-05-31 2003-10-21 International Business Machines Corporation Information search using knowledge agents
KR100458459B1 (en) * 2004-01-27 2004-11-26 엔에이치엔(주) Method for offering a search-word advertisement and generating a search result list in response to the search-demand of a searcher and a system thereof
KR100492075B1 (en) * 2004-06-24 2005-06-01 엔에이치엔(주) Method and system for selecting search list table in an internet search engine in response to search request
US7958115B2 (en) * 2004-07-29 2011-06-07 Yahoo! Inc. Search systems and methods using in-line contextual queries
WO2006036781A2 (en) * 2004-09-22 2006-04-06 Perfect Market Technologies, Inc. Search engine using user intent
US20070027751A1 (en) * 2005-07-29 2007-02-01 Chad Carson Positioning advertisements on the bases of expected revenue
KR100785352B1 (en) * 2006-04-21 2007-12-18 엔에이치엔(주) Method and system for creating search-result-list
KR100834446B1 (en) * 2006-10-10 2008-06-04 엔에이치엔(주) Method and system for extracting weight classified by ranking
EP2115985B1 (en) 2007-02-26 2013-04-10 Telefonaktiebolaget LM Ericsson (publ) Apparatuses and a method for reducing peak power in a transmitter of telecommunications systems
US20080208731A1 (en) * 2007-02-28 2008-08-28 Bellsouth Intellectual Property Corporation Methods and computer program products for auctioning on-line advertisements based on Internet search term query origination location
US7672937B2 (en) * 2007-04-11 2010-03-02 Yahoo, Inc. Temporal targeting of advertisements
US8359309B1 (en) * 2007-05-23 2013-01-22 Google Inc. Modifying search result ranking based on corpus search statistics
US20090076899A1 (en) * 2007-09-14 2009-03-19 Gbodimowo Gbeminiyi A Method for analyzing, searching for, and trading targeted advertisement spaces

Patent Citations (286)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5488725A (en) 1991-10-08 1996-01-30 West Publishing Company System of document representation retrieval by successive iterated probability sampling
US5265065A (en) 1991-10-08 1993-11-23 West Publishing Company Method and apparatus for information retrieval from a database by replacing domain specific stemmed phases in a natural language to create a search query
US5696962A (en) 1993-06-24 1997-12-09 Xerox Corporation Method for computerized information retrieval using shallow linguistic analysis
US6088692A (en) 1994-12-06 2000-07-11 University Of Central Florida Natural language method and system for searching for and ranking relevant documents from a computer database
US20010000356A1 (en) 1995-07-07 2001-04-19 Woods William A. Method and apparatus for generating query responses in a computer-based document retrieval system
US5963940A (en) 1995-08-16 1999-10-05 Syracuse University Natural language information retrieval system and method
US6026388A (en) 1995-08-16 2000-02-15 Textwise, Llc User interface and other enhancements for natural language information retrieval system and method
US5920854A (en) 1996-08-14 1999-07-06 Infoseek Corporation Real-time document collection search engine with phrase indexing
US6249252B1 (en) 1996-09-09 2001-06-19 Tracbeam Llc Wireless location using multiple location estimators
US6353849B1 (en) 1996-12-20 2002-03-05 Intel Corporation System and server for providing customized web information based on attributes of the requestor
US6285999B1 (en) 1997-01-10 2001-09-04 The Board Of Trustees Of The Leland Stanford Junior University Method for node ranking in a linked database
US20040122811A1 (en) 1997-01-10 2004-06-24 Google, Inc. Method for searching media
US6076051A (en) 1997-03-07 2000-06-13 Microsoft Corporation Information retrieval utilizing semantic representation of text
US6006222A (en) 1997-04-25 1999-12-21 Culliss; Gary Method for organizing information
US6185559B1 (en) 1997-05-09 2001-02-06 Hitachi America, Ltd. Method and apparatus for dynamically counting large itemsets
US6078916A (en) 1997-08-01 2000-06-20 Culliss; Gary Method for organizing information
US6539377B1 (en) 1997-08-01 2003-03-25 Ask Jeeves, Inc. Personalized search methods
US6182068B1 (en) 1997-08-01 2001-01-30 Ask Jeeves, Inc. Personalized search methods
US6816850B2 (en) 1997-08-01 2004-11-09 Ask Jeeves, Inc. Personalized search methods including combining index entries for catagories of personal data
US6014665A (en) 1997-08-01 2000-01-11 Culliss; Gary Method for organizing information
US20110219025A1 (en) 1997-10-27 2011-09-08 Massachusetts Institute Of Technology Image search using images in query
US6134532A (en) 1997-11-14 2000-10-17 Aptex Software, Inc. System and method for optimal adaptive matching of users to most relevant entity and information in real-time
US6182066B1 (en) 1997-11-26 2001-01-30 International Business Machines Corp. Category processing of query topics and electronic document content topics
US6078917A (en) 1997-12-18 2000-06-20 International Business Machines Corporation System for searching internet using automatic relevance feedback
US7117206B1 (en) 1998-01-15 2006-10-03 Overture Services, Inc. Method for ranking hyperlinked pages using content and connectivity analysis
US6067565A (en) 1998-01-15 2000-05-23 Microsoft Corporation Technique for prefetching a web page of potential future interest in lieu of continuing a current information download
US6623529B1 (en) 1998-02-23 2003-09-23 David Lakritz Multilingual electronic document translation, management, and delivery system
US20020049752A1 (en) 1998-03-03 2002-04-25 Dwayne Bowman Identifying the items most relevant to a current query based on items selected in connection with similar queries
US6421675B1 (en) 1998-03-16 2002-07-16 S. L. I. Systems, Inc. Search engine
US6341283B1 (en) 1998-05-21 2002-01-22 Fujitsu Limited Apparatus for data decomposition and method and storage medium therefor
US6853993B2 (en) 1998-07-15 2005-02-08 A9.Com, Inc. System and methods for predicting correct spellings of terms in multiple-term search queries
US6912505B2 (en) 1998-09-18 2005-06-28 Amazon.Com, Inc. Use of product viewing histories of users to identify related products
US6363378B1 (en) 1998-10-13 2002-03-26 Oracle Corporation Ranking of query feedback terms in an information retrieval system
US6480843B2 (en) 1998-11-03 2002-11-12 Nec Usa, Inc. Supporting web-query expansion efficiently using multi-granularity indexing and query processing
US20040006456A1 (en) 1998-11-30 2004-01-08 Wayne Loofbourrow Multi-language document search and retrieval system
US6678681B1 (en) 1999-03-10 2004-01-13 Google Inc. Information extraction from a database
US7565363B2 (en) 1999-04-01 2009-07-21 Anwar Mohammed S Search engine with user activity memory
US6882999B2 (en) 1999-05-03 2005-04-19 Microsoft Corporation URL mapping methods and systems
US6327590B1 (en) 1999-05-05 2001-12-04 Xerox Corporation System and method for collaborative ranking of search results employing user and group profiles derived from document collection content analysis
US6370526B1 (en) 1999-05-18 2002-04-09 International Business Machines Corporation Self-adaptive method and system for providing a user-preferred ranking order of object sets
US20020165849A1 (en) 1999-05-28 2002-11-07 Singh Narinder Pal Automatic advertiser notification for a system for providing place and price protection in a search result list generated by a computer network search engine
WO2000077689A1 (en) 1999-06-16 2000-12-21 Triogo, Inc. A process for improving search engine efficiency using user feedback
US6901402B1 (en) 1999-06-18 2005-05-31 Microsoft Corporation System for improving the performance of information retrieval-type tasks by identifying the relations of constituents
US6873982B1 (en) 1999-07-16 2005-03-29 International Business Machines Corporation Ordering of database search results based on user feedback
US20070156677A1 (en) 1999-07-21 2007-07-05 Alberti Anemometer Llc Database access system
US6321228B1 (en) 1999-08-31 2001-11-20 Powercast Media, Inc. Internet search system for retrieving selected results from a previous search
WO2001016807A1 (en) 1999-08-31 2001-03-08 Mindcrossing, Inc. An internet search system for tracking and ranking selected records from a previous search
US6754873B1 (en) 1999-09-20 2004-06-22 Google Inc. Techniques for finding related hyperlinked documents using link-based analysis
US7113939B2 (en) 1999-09-21 2006-09-26 International Business Machines Corporation Architecture to enable search gateways as part of federated search
US6792416B2 (en) 1999-09-21 2004-09-14 International Business Machines Corporation Managing results of federated searches across heterogeneous datastores with a federated result set cursor object
US8073772B2 (en) 1999-11-05 2011-12-06 American Express Travel Related Services Company, Inc. Systems and methods for processing transactions using multiple budgets
US6490575B1 (en) 1999-12-06 2002-12-03 International Business Machines Corporation Distributed network search engine
US20030195877A1 (en) 1999-12-08 2003-10-16 Ford James L. Search query processing to provide category-ranked presentation of search results
US6963867B2 (en) 1999-12-08 2005-11-08 A9.Com, Inc. Search query processing to provide category-ranked presentation of search results
US20030120654A1 (en) 2000-01-14 2003-06-26 International Business Machines Corporation Metadata search results ranking system
US6560590B1 (en) 2000-02-14 2003-05-06 Kana Software, Inc. Method and apparatus for multiple tiered matching of natural language queries to positions in a text corpus
US6615209B1 (en) 2000-02-22 2003-09-02 Google, Inc. Detecting query-specific duplicate documents
WO2001067297A1 (en) 2000-03-07 2001-09-13 Tzunami Inc. System and method for computer searching
US6587848B1 (en) 2000-03-08 2003-07-01 International Business Machines Corporation Methods and apparatus for performing an affinity based similarity search
US20040186996A1 (en) 2000-03-29 2004-09-23 Gibbs Benjamin K. Unique digital signature
US20040049486A1 (en) 2000-04-18 2004-03-11 Scanlon Henry R. Image relationships derived from thresholding of historically tracked user data for facilitating image based searching
US6701309B1 (en) 2000-04-21 2004-03-02 Lycos, Inc. Method and system for collecting related queries
US20100106706A1 (en) 2000-05-22 2010-04-29 Yahoo! Inc. Method and apparatus for identifying related searches in a database search system
US6671681B1 (en) 2000-05-31 2003-12-30 International Business Machines Corporation System and technique for suggesting alternate query expressions based on prior user selections and their query strings
US20020042791A1 (en) 2000-07-06 2002-04-11 Google, Inc. Methods and apparatus for using a modified index to provide search results in response to an ambiguous search query
US6529903B2 (en) 2000-07-06 2003-03-04 Google, Inc. Methods and apparatus for using a modified index to provide search results in response to an ambiguous search query
US20020133481A1 (en) 2000-07-06 2002-09-19 Google, Inc. Methods and apparatus for providing search results in response to an ambiguous search query
US6990453B2 (en) 2000-07-31 2006-01-24 Landmark Digital Services Llc System and methods for recognizing sound and music signals in high noise and distortion
US6567103B1 (en) 2000-08-02 2003-05-20 Verity, Inc. Graphical search results system and method
US20020034292A1 (en) 2000-08-22 2002-03-21 Tuoriniemi Veijo M. System and a method to match demand and supply based on geographical location derived from a positioning system
US6944611B2 (en) 2000-08-28 2005-09-13 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
US7395222B1 (en) 2000-09-07 2008-07-01 Sotos John G Method and system for identifying expertise
US20040006740A1 (en) 2000-09-29 2004-01-08 Uwe Krohn Information access
US6954750B2 (en) 2000-10-10 2005-10-11 Content Analyst Company, Llc Method and system for facilitating the refinement of data queries
US20050055342A1 (en) 2000-11-08 2005-03-10 Bharat Krishna Asur Method for estimating coverage of Web search engines
US6877002B2 (en) 2000-11-21 2005-04-05 America Online, Inc. Fuzzy database retrieval
US6658423B1 (en) 2001-01-24 2003-12-02 Google, Inc. Detecting duplicate and near-duplicate files
US6526440B1 (en) 2001-01-30 2003-02-25 Google, Inc. Ranking search results by reranking the results based on local inter-connectivity
US20020103790A1 (en) 2001-01-30 2002-08-01 Wang Shirley S. Utility for cross platform database query
US6725259B1 (en) 2001-01-30 2004-04-20 Google Inc. Ranking search results by reranking the results based on local inter-connectivity
US20020123988A1 (en) 2001-03-02 2002-09-05 Google, Inc. Methods and apparatus for employing usage statistics in document retrieval
US20030009399A1 (en) 2001-03-22 2003-01-09 Boerner Sean T. Method and system to identify discrete trends in time series
US20030037074A1 (en) 2001-05-01 2003-02-20 Ibm Corporation System and method for aggregating ranking results from various sources to improve the results of web searching
US6738764B2 (en) 2001-05-08 2004-05-18 Verity, Inc. Apparatus and method for adaptively ranking search results
US7072886B2 (en) 2001-05-15 2006-07-04 Nokia Corporation Method and business process to maintain privacy in distributed recommendation systems
US6795820B2 (en) 2001-06-20 2004-09-21 Nextpage, Inc. Metasearch technique that ranks documents obtained from multiple collections
US20070081197A1 (en) 2001-06-22 2007-04-12 Nosa Omoigui System and method for semantic knowledge retrieval, management, capture, sharing, discovery, delivery and presentation
US20030018707A1 (en) 2001-07-20 2003-01-23 Flocken Philip Andrew Server-side filter for corrupt web-browser cookies
US7016939B1 (en) 2001-07-26 2006-03-21 Mcafee, Inc. Intelligent SPAM detection system using statistical analysis
US20030028529A1 (en) 2001-08-03 2003-02-06 Cheung Dominic Dough-Ming Search engine account monitoring
US7136849B2 (en) 2001-08-10 2006-11-14 International Business Machines Corporation Method systems and computer program products for indicating links to external URLs
US7266765B2 (en) 2001-08-31 2007-09-04 Fuji Xerox Co., Ltd. Detection and processing of annotated anchors
US20030078914A1 (en) 2001-10-18 2003-04-24 Witbrock Michael J. Search results using editor feedback
US20040199419A1 (en) 2001-11-13 2004-10-07 International Business Machines Corporation Promoting strategic documents by bias ranking of search results on a web browser
US7565367B2 (en) 2002-01-15 2009-07-21 Iac Search & Media, Inc. Enhanced popularity ranking
US20030135490A1 (en) 2002-01-15 2003-07-17 Barrett Michael E. Enhanced popularity ranking
US20030149704A1 (en) 2002-02-05 2003-08-07 Hitachi, Inc. Similarity-based search method by relevance feedback
US20050055345A1 (en) 2002-02-14 2005-03-10 Infoglide Software Corporation Similarity search engine for use with relational databases
US20030167252A1 (en) 2002-02-26 2003-09-04 Pliant Technologies, Inc. Topic identification and use thereof in information retrieval systems
US20030204495A1 (en) 2002-04-30 2003-10-30 Lehnert Bernd R. Data gathering
US20030220913A1 (en) 2002-05-24 2003-11-27 International Business Machines Corporation Techniques for personalized and adaptive search services
US7379951B2 (en) 2002-05-31 2008-05-27 Microsoft Corporation Support for real-time queries concerning current state, data and history of a process
US20030229640A1 (en) 2002-06-07 2003-12-11 International Business Machines Corporation Parallel database query processing for non-uniform data sources via buffered access
US7853557B2 (en) 2002-06-14 2010-12-14 Siebel Systems, Inc. Method and computer for responding to a query according to the language used
US7085761B2 (en) 2002-06-28 2006-08-01 Fujitsu Limited Program for changing search results rank, recording medium for recording such a program, and content search processing method
US20040034632A1 (en) 2002-07-31 2004-02-19 International Business Machines Corporation Automatic query refinement
US7028027B1 (en) 2002-09-17 2006-04-11 Yahoo! Inc. Associating documents with classifications and ranking documents based on classification weights
US8090717B1 (en) 2002-09-20 2012-01-03 Google Inc. Methods and apparatus for ranking documents
US20040059708A1 (en) 2002-09-24 2004-03-25 Google, Inc. Methods and apparatus for serving relevant advertisements
US20040083205A1 (en) 2002-10-29 2004-04-29 Steve Yeager Continuous knowledgebase access improvement systems and methods
US20040093325A1 (en) 2002-11-07 2004-05-13 International Business Machines Corporation System and method for location influenced network search
US6944612B2 (en) 2002-11-13 2005-09-13 Xerox Corporation Structured contextual clustering method and system in a federated search engine
WO2004059514A1 (en) 2002-12-24 2004-07-15 American Type Culture Collection Systems and methods for enabling a user to find information of interest to the user
US20040119740A1 (en) 2002-12-24 2004-06-24 Google, Inc., A Corporation Of The State Of California Methods and apparatus for displaying and replying to electronic messages
US20040186828A1 (en) 2002-12-24 2004-09-23 Prem Yadav Systems and methods for enabling a user to find information of interest to the user
US7382358B2 (en) 2003-01-16 2008-06-03 Forword Input, Inc. System and method for continuous stroke word-based text input
US20040153472A1 (en) 2003-01-31 2004-08-05 Rieffanaugh Neal King Human resource networking system and method thereof
US20040158560A1 (en) 2003-02-12 2004-08-12 Ji-Rong Wen Systems and methods for query expansion
US20090055392A1 (en) 2003-03-31 2009-02-26 Google Inc. Ordering of search results based on language and/or country of the search results
US20040215607A1 (en) 2003-04-25 2004-10-28 Travis Robert L. Method and system fo blending search engine results from disparate sources into one search result
US7516146B2 (en) 2003-05-15 2009-04-07 Microsoft Corporation Fast adaptive document filtering
US7526470B1 (en) 2003-05-28 2009-04-28 Microsoft Corporation System and method for measuring and improving search result relevance based on user satisfaction
US7146361B2 (en) 2003-05-30 2006-12-05 International Business Machines Corporation System, method and computer program product for performing unstructured information management and automatic text analysis, including a search operator functioning as a Weighted AND (WAND)
US20050240576A1 (en) 2003-06-10 2005-10-27 John Piscitello Named URL entry
US20050033803A1 (en) 2003-07-02 2005-02-10 Vleet Taylor N. Van Server architecture and methods for persistently storing and serving event data
US20050015366A1 (en) 2003-07-18 2005-01-20 Carrasco John Joseph M. Disambiguation of search phrases using interpretation clusters
US20060173830A1 (en) 2003-07-23 2006-08-03 Barry Smyth Information retrieval
US20050027691A1 (en) 2003-07-28 2005-02-03 Sergey Brin System and method for providing a user interface with search query broadening
US20050050014A1 (en) 2003-08-29 2005-03-03 Gosse David B. Method, device and software for querying and presenting search results
US7451487B2 (en) 2003-09-08 2008-11-11 Sonicwall, Inc. Fraudulent message detection
US7505964B2 (en) 2003-09-12 2009-03-17 Google Inc. Methods and systems for improving a search ranking using related queries
US8024326B2 (en) 2003-09-12 2011-09-20 Google Inc. Methods and systems for improving a search ranking using related queries
US20050060310A1 (en) 2003-09-12 2005-03-17 Simon Tong Methods and systems for improving a search ranking using population information
US20120191705A1 (en) 2003-09-12 2012-07-26 Google Inc. Methods and systems for improving a search ranking using related queries
US20050060311A1 (en) 2003-09-12 2005-03-17 Simon Tong Methods and systems for improving a search ranking using related queries
US20050060290A1 (en) 2003-09-15 2005-03-17 International Business Machines Corporation Automatic query routing and rank configuration for search queries in an information retrieval system
US8086690B1 (en) 2003-09-22 2011-12-27 Google Inc. Determining geographical relevance of web documents
US20050240580A1 (en) 2003-09-30 2005-10-27 Zamir Oren E Personalization of placed content ordering in search results
US8224827B2 (en) 2003-09-30 2012-07-17 Google Inc. Document ranking based on document classification
US7584181B2 (en) 2003-09-30 2009-09-01 Microsoft Corporation Implicit links search enhancement system and method for search engines using implicit links generated by mining user access patterns
US20050071741A1 (en) 2003-09-30 2005-03-31 Anurag Acharya Information retrieval based on historical data
US20050102282A1 (en) 2003-11-07 2005-05-12 Greg Linden Method for personalized search
US7231399B1 (en) 2003-11-14 2007-06-12 Google Inc. Ranking documents based on large data sets
US7222127B1 (en) 2003-11-14 2007-05-22 Google Inc. Large scale machine learning systems and methods
US7844589B2 (en) 2003-11-18 2010-11-30 Yahoo! Inc. Method and apparatus for performing a search
US8521725B1 (en) 2003-12-03 2013-08-27 Google Inc. Systems and methods for improved searching
US20060230040A1 (en) 2003-12-08 2006-10-12 Andy Curtis Methods and systems for providing a response to a query
US20050125376A1 (en) 2003-12-08 2005-06-09 Andy Curtis Methods and systems for providing a response to a query
US20080208825A1 (en) 2003-12-08 2008-08-28 Andy Curtis Methods and systems for providing a response to a query
US20050192946A1 (en) 2003-12-29 2005-09-01 Yahoo! Inc. Lateral search
US20060106793A1 (en) 2003-12-29 2006-05-18 Ping Liang Internet and computer information retrieval and mining with intelligent conceptual filtering, visualization and automation
US20050160083A1 (en) 2004-01-16 2005-07-21 Yahoo! Inc. User-specific vertical search
US7293016B1 (en) 2004-01-22 2007-11-06 Microsoft Corporation Index partitioning based on document relevance for document indexes
US20050198026A1 (en) 2004-02-03 2005-09-08 Dehlinger Peter J. Code, system, and method for generating concepts
US20050222998A1 (en) 2004-03-31 2005-10-06 Oce-Technologies B.V. Apparatus and computerised method for determining constituent words of a compound word
US20050222987A1 (en) 2004-04-02 2005-10-06 Vadon Eric R Automated detection of associations between search criteria and item categories based on collective analysis of user activity data
US20050256848A1 (en) 2004-05-13 2005-11-17 International Business Machines Corporation System and method for user rank search
US8024330B1 (en) 2004-05-20 2011-09-20 Hector Franco Collaborative incident alert system for mobile devices
US7716225B1 (en) 2004-06-17 2010-05-11 Google Inc. Ranking documents based on user behavior and/or feature data
US7243102B1 (en) 2004-07-01 2007-07-10 Microsoft Corporation Machine directed improvement of ranking algorithms
US7426507B1 (en) 2004-07-26 2008-09-16 Google, Inc. Automatic taxonomy generation in search results using phrases
US20060047643A1 (en) 2004-08-31 2006-03-02 Chirag Chaman Method and system for a personalized search engine
US8583636B1 (en) 2004-09-29 2013-11-12 Google Inc. Systems and methods for determining a quality of provided items
US8065296B1 (en) 2004-09-29 2011-11-22 Google Inc. Systems and methods for determining a quality of provided items
US20060069667A1 (en) 2004-09-30 2006-03-30 Microsoft Corporation Content evaluation
US20060074903A1 (en) 2004-09-30 2006-04-06 Microsoft Corporation System and method for ranking search results using click distance
US20060095421A1 (en) 2004-10-22 2006-05-04 Canon Kabushiki Kaisha Method, apparatus, and program for searching for data
US20080077570A1 (en) 2004-10-25 2008-03-27 Infovell, Inc. Full Text Query and Search Systems and Method of Use
US20060089926A1 (en) 2004-10-27 2006-04-27 Harris Corporation, Corporation Of The State Of Delaware Method for re-ranking documents retrieved from a document database
US7533092B2 (en) 2004-10-28 2009-05-12 Yahoo! Inc. Link-based spam detection
US20060200556A1 (en) 2004-12-29 2006-09-07 Scott Brave Method and apparatus for identifying, extracting, capturing, and leveraging expertise and knowledge
US7756887B1 (en) 2004-12-30 2010-07-13 Google Inc. System and method for modulating search relevancy using pointer activity monitoring
US8019650B2 (en) 2005-01-21 2011-09-13 Amazon Technologies, Inc. Method and system for producing item comparisons
US20060195443A1 (en) 2005-02-11 2006-08-31 Franklin Gary L Information prioritisation system and method
US20060200476A1 (en) 2005-03-03 2006-09-07 Microsoft Corporation Creating, storing and viewing process models
US7574530B2 (en) 2005-03-10 2009-08-11 Microsoft Corporation Method and system for web resource location classification and detection
US20070106659A1 (en) 2005-03-18 2007-05-10 Yunshan Lu Search engine that applies feedback from users to improve search results
US7636714B1 (en) 2005-03-31 2009-12-22 Google Inc. Determining query term synonyms within query context
US20060227992A1 (en) 2005-04-08 2006-10-12 Rathus Spencer A System and method for accessing electronic data via an image search engine
US20120011148A1 (en) 2005-04-08 2012-01-12 Rathus Spencer A System and method for accessing electronic data via an image search engine
US7849089B2 (en) 2005-05-10 2010-12-07 Microsoft Corporation Method and system for adapting search results to personal information needs
US20060293950A1 (en) 2005-06-28 2006-12-28 Microsoft Corporation Automatic ad placement
US20070005575A1 (en) 2005-06-30 2007-01-04 Microsoft Corporation Prioritizing search results by client search satisfaction
US20070005588A1 (en) 2005-07-01 2007-01-04 Microsoft Corporation Determining relevance using queries as surrogate content
US20070038659A1 (en) 2005-08-15 2007-02-15 Google, Inc. Scalable user clustering based on set similarity
US20070050339A1 (en) 2005-08-24 2007-03-01 Richard Kasperski Biasing queries to determine suggested queries
US20070061195A1 (en) 2005-09-13 2007-03-15 Yahoo! Inc. Framework for selecting and delivering advertisements over a network based on combined short-term and long-term user behavioral interests
US20070061211A1 (en) 2005-09-14 2007-03-15 Jorey Ramer Preventing mobile communication facility click fraud
US7747612B2 (en) 2005-10-31 2010-06-29 Yahoo! Inc. Indication of exclusive items in a result set
US7783632B2 (en) 2005-11-03 2010-08-24 Microsoft Corporation Using popularity data for ranking
US20070112730A1 (en) 2005-11-07 2007-05-17 Antonino Gulli Sampling Internet user traffic to improve search results
US20080140699A1 (en) 2005-11-09 2008-06-12 Rosie Jones System and method for generating substitutable queries
US7693818B2 (en) 2005-11-15 2010-04-06 Microsoft Corporation UserRank: ranking linked nodes leveraging user logs
US20070266439A1 (en) 2005-11-30 2007-11-15 Harold Kraft Privacy management and transaction system
US20070130370A1 (en) 2005-12-06 2007-06-07 Emeka Akaezuwa Portable search engine
US20070192190A1 (en) 2005-12-06 2007-08-16 Authenticlick Method and system for scoring quality of traffic to network sites
US7680775B2 (en) 2005-12-13 2010-03-16 Iac Search & Media, Inc. Methods and systems for generating query and result-based relevance indexes
US20070172155A1 (en) 2006-01-21 2007-07-26 Elizabeth Guckenberger Photo Automatic Linking System and method for accessing, linking, and visualizing "key-face" and/or multiple similar facial images along with associated electronic data via a facial image recognition search engine
US20070180355A1 (en) 2006-02-01 2007-08-02 Ricoh Co., Ltd. Enhancing accuracy of jumping by incorporating interestingness estimates
US7953740B1 (en) 2006-02-13 2011-05-31 Amazon Technologies, Inc. Detection of behavior-based associations between search strings and items
US20070208730A1 (en) 2006-03-02 2007-09-06 Microsoft Corporation Mining web search user behavior to enhance web search relevance
US20070266021A1 (en) 2006-03-06 2007-11-15 Murali Aravamudan Methods and systems for selecting and presenting content based on dynamically identifying microgenres associated with the content
US7818315B2 (en) 2006-03-13 2010-10-19 Microsoft Corporation Re-ranking search results based on query log
US20070214131A1 (en) 2006-03-13 2007-09-13 Microsoft Corporation Re-ranking search results based on query log
US20070260596A1 (en) 2006-03-29 2007-11-08 Koran Joshua M Behavioral targeting system
US20070233653A1 (en) 2006-03-31 2007-10-04 Microsoft Corporation Selecting directly bid upon advertisements for display
US20080052219A1 (en) 2006-03-31 2008-02-28 Combinenet, Inc. System for and method of expressive auctions of user events
US20070288450A1 (en) 2006-04-19 2007-12-13 Datta Ruchira S Query language determination using query terms and interface language
US8069182B2 (en) 2006-04-24 2011-11-29 Working Research, Inc. Relevancy-based domain classification
US20070255689A1 (en) 2006-04-28 2007-11-01 Gordon Sun System and method for indexing web content using click-through features
US20070260597A1 (en) 2006-05-02 2007-11-08 Mark Cramer Dynamic search engine results employing user behavior
US20080010143A1 (en) 2006-06-22 2008-01-10 Rob Kniaz Secure and extensible pay per action online advertising
US7809716B2 (en) 2006-06-27 2010-10-05 International Business Machines Corporation Method and apparatus for establishing relationship between documents
US20080027913A1 (en) 2006-07-25 2008-01-31 Yahoo! Inc. System and method of information retrieval engine evaluation using human judgment input
US8073263B2 (en) 2006-07-31 2011-12-06 Ricoh Co., Ltd. Multi-classifier selection and monitoring for MMR-based image recognition
US20080052273A1 (en) 2006-08-22 2008-02-28 Fuji Xerox Co., Ltd. Apparatus and method for term context modeling for information retrieval
US20080059453A1 (en) 2006-08-29 2008-03-06 Raphael Laderman System and method for enhancing the result of a query
US7552112B2 (en) 2006-09-18 2009-06-23 Yahoo! Inc. Discovering associative intent queries from search web logs
US8027439B2 (en) 2006-09-18 2011-09-27 Fair Isaac Corporation Self-calibrating fraud detection
US7657626B1 (en) 2006-09-19 2010-02-02 Enquisite, Inc. Click fraud detection
US20080082518A1 (en) 2006-09-29 2008-04-03 Loftesness David E Strategy for Providing Query Results Based on Analysis of User Intent
US7860886B2 (en) 2006-09-29 2010-12-28 A9.Com, Inc. Strategy for providing query results based on analysis of user intent
US20080091650A1 (en) 2006-10-11 2008-04-17 Marcus Felipe Fontoura Augmented Search With Error Detection and Replacement
US8086599B1 (en) 2006-10-24 2011-12-27 Google Inc. Method and apparatus for automatically identifying compunds
US20080104043A1 (en) 2006-10-25 2008-05-01 Ashutosh Garg Server-side match
US8661029B1 (en) 2006-11-02 2014-02-25 Google Inc. Modifying search result ranking based on implicit user feedback
US20080189269A1 (en) 2006-11-07 2008-08-07 Fast Search & Transfer Asa Relevance-weighted navigation in information access, search and retrieval
US20080114624A1 (en) 2006-11-13 2008-05-15 Microsoft Corporation Click-fraud protector
US20080114729A1 (en) 2006-11-13 2008-05-15 Google Inc. Computer-implemented interactive, virtual bookshelf system and method
US20080114750A1 (en) 2006-11-14 2008-05-15 Microsoft Corporation Retrieval and ranking of items utilizing similarity
US7533130B2 (en) 2006-12-19 2009-05-12 Yahoo! Inc. User behavior reporting based on pre-aggregated activity data
US8661012B1 (en) 2006-12-29 2014-02-25 Google Inc. Ensuring that a synonym for a query phrase does not drop information present in the query phrase
US7925498B1 (en) 2006-12-29 2011-04-12 Google Inc. Identifying a synonym with N-gram agreement for a query phrase
US7987185B1 (en) 2006-12-29 2011-07-26 Google Inc. Ranking custom search results
US20080162475A1 (en) 2007-01-03 2008-07-03 Meggs Anthony F Click-fraud detection method
US7801885B1 (en) 2007-01-25 2010-09-21 Neal Akash Verma Search engine system and method with user feedback on search results
US20080183660A1 (en) 2007-01-30 2008-07-31 Google Inc. Content identification expansion
US20080228442A1 (en) 2007-03-07 2008-09-18 Lisa Ellen Lippincott Statistical data inspector
US8694374B1 (en) 2007-03-14 2014-04-08 Google Inc. Detecting click spam
US7610282B1 (en) 2007-03-30 2009-10-27 Google Inc. Rank-adjusted content items
US20080256050A1 (en) 2007-04-10 2008-10-16 Ruofei Zhang System and method for modeling user selection feedback in a search result page
US8037042B2 (en) 2007-05-10 2011-10-11 Microsoft Corporation Automated analysis of user search behavior
US7895177B2 (en) 2007-05-29 2011-02-22 Yahoo! Inc. Enabling searching of user ratings and reviews using user profile location, and social networks
US7818320B2 (en) 2007-05-31 2010-10-19 Yahoo! Inc. Enhanced search results based on user feedback relating to search result abstracts
US20080313247A1 (en) 2007-06-12 2008-12-18 Brian Galvin Page Ranking Based on a Behavioral WEB Graph
US20080313168A1 (en) 2007-06-18 2008-12-18 Microsoft Corporation Ranking documents based on a series of document graphs
US8458165B2 (en) 2007-06-28 2013-06-04 Oracle International Corporation System and method for applying ranking SVM in query relaxation
US20090012969A1 (en) 2007-07-02 2009-01-08 Rail Peter D Systems and processes for evaluating webpages
US8001136B1 (en) 2007-07-10 2011-08-16 Google Inc. Longest-common-subsequence detection for common synonyms
US8037086B1 (en) 2007-07-10 2011-10-11 Google Inc. Identifying common co-occurring elements in lists
US8051061B2 (en) 2007-07-20 2011-11-01 Microsoft Corporation Cross-lingual query suggestion
US7566363B2 (en) 2007-07-24 2009-07-28 Silverbrook Research Pty Ltd Alternative phthalocyanine dyes suitable for use in offset inks
US8694511B1 (en) 2007-08-20 2014-04-08 Google Inc. Modifying search result ranking based on populations
US20090070194A1 (en) 2007-09-06 2009-03-12 Nhn Corporation Method of providing estimated cost for keyword advertisement
US20090157643A1 (en) 2007-12-12 2009-06-18 Microsoft Corporation Semi-supervised part-of-speech tagging
US7769740B2 (en) 2007-12-21 2010-08-03 Yahoo! Inc. Systems and methods of ranking attention
US20090171943A1 (en) 2008-01-02 2009-07-02 Microsoft Corporation Modifying relevance ranking of search result items
US20090182723A1 (en) 2008-01-10 2009-07-16 Microsoft Corporation Ranking search results using author extraction
US7877404B2 (en) 2008-03-05 2011-01-25 Microsoft Corporation Query classification based on query click logs
US20090228442A1 (en) 2008-03-10 2009-09-10 Searchme, Inc. Systems and methods for building a document index
US7974974B2 (en) 2008-03-20 2011-07-05 Microsoft Corporation Techniques to perform relative ranking for search results
US7836058B2 (en) 2008-03-27 2010-11-16 Microsoft Corporation Web searching
US20090287656A1 (en) 2008-05-13 2009-11-19 Bennett James D Network search engine utilizing client browser favorites
US8171041B2 (en) 2008-05-15 2012-05-01 Enpulz, L.L.C. Support for international search terms
US8126839B2 (en) 2008-06-19 2012-02-28 Yahoo! Inc. Methods and apparatuses for adapting a ranking function of a search engine for use with a specific domain
US8037043B2 (en) 2008-09-09 2011-10-11 Microsoft Corporation Information retrieval system
US8060456B2 (en) 2008-10-01 2011-11-15 Microsoft Corporation Training a search result ranker with automatically-generated samples
US8156111B2 (en) 2008-11-24 2012-04-10 Yahoo! Inc. Identifying and expanding implicitly temporally qualified queries
US20100131563A1 (en) 2008-11-25 2010-05-27 Hongfeng Yin System and methods for automatic clustering of ranked and categorized search objects
US8898152B1 (en) 2008-12-10 2014-11-25 Google Inc. Sharing search engine relevance data
US8396865B1 (en) 2008-12-10 2013-03-12 Google Inc. Sharing search engine relevance data between corpora
US20100205541A1 (en) 2009-02-11 2010-08-12 Jeffrey A. Rapaport social network driven indexing system for instantly clustering people with concurrent focus on same topic into on-topic chat rooms and/or for generating on-topic search results tailored to user preferences regarding topic
US20100228738A1 (en) 2009-03-04 2010-09-09 Mehta Rupesh R Adaptive document sampling for information extraction
US20100241472A1 (en) 2009-03-23 2010-09-23 David Antonio Hernandez Illness specific diagnostic system
US8412699B1 (en) 2009-06-12 2013-04-02 Google Inc. Fresh related search suggestions
US8447760B1 (en) 2009-07-20 2013-05-21 Google Inc. Generating a related set of documents for an initial set of documents
US8060497B1 (en) 2009-07-23 2011-11-15 Google Inc. Framework for evaluating web search scoring functions
US8738596B1 (en) 2009-08-31 2014-05-27 Google Inc. Refining search results
US8498974B1 (en) 2009-08-31 2013-07-30 Google Inc. Refining search results
US20110087656A1 (en) 2009-10-13 2011-04-14 Electronics And Telecommunications Research Institute Apparatus for question answering based on answer trustworthiness and method thereof
US8898153B1 (en) 2009-11-20 2014-11-25 Google Inc. Modifying scoring data based on historical changes
US8615514B1 (en) 2010-02-03 2013-12-24 Google Inc. Evaluating website properties by partitioning user feedback
US20110282906A1 (en) 2010-05-14 2011-11-17 Rovi Technologies Corporation Systems and methods for performing a search based on a media content snapshot image
US20110295879A1 (en) 2010-05-27 2011-12-01 Neuone, Llc Systems and methods for document management
US20110295844A1 (en) 2010-05-27 2011-12-01 Microsoft Corporation Enhancing freshness of search results
US8832083B1 (en) 2010-07-23 2014-09-09 Google Inc. Combining user feedback

Non-Patent Citations (58)

* Cited by examiner, † Cited by third party
Title
Agichtein, et al; Improving Web Search Ranking by Incorporating User Behavior Information; Aug. 2006; Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 19-26.
Agichtein, et al; Learning User Interaction Models for Predicting Web Search Result Performances; Aug. 2006; Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 3-10.
Australian Patent Office Non-Final Office Action in AU App. Ser. No. 2004275274, mailed Feb. 3, 2010, 2 pages.
Authorized Officer Athina Nickitas-Etienne, International Preliminary Report and Written Opinion for Application No. PCT/US2004/029615, mailed Mar. 23, 2006.
Baeza-Yates, Ricardo, Carlos Hurtado, and Marcelo Mendoza. "Query recommendation using query logs in search engines." Current Trends in Database Technology-EDBT 2004 Workshops. Springer Berlin Heidelberg, 2005.
Bar-Llan et al., "Presentation Bias is Significant in Determining User Preference for Search Results-A User Study"; Journal of the American Society for Information Science and Technology, vol. 60, Issue 1 (p. 135-149), Sep. 2008, 15 pages.
Bar-Llan et al.; ""Methods for comparing rankings of search engine results""; Computer Networks: The International Journal of Computer and Telecommunications Networking, Jul. 2006, vol. 50, Issue 10, 19 pages.
Boldi, et al.; The Query flow Graph: Model and Applications; CKIM '08, Oct. 26-30, Napa Valley, California, USA, pp. 609-617.
Boyan et al.; A Machine Learning Architecture for Optimizing Web Search Engines; Aug. 1996; Internet-based information systems-Workshop Technical Report-American Association for Artificial Intelligence, p. 1-8.
Brin, S. and L. Page, The Anatomy of a Large-Scale Hypertextual Web Search Engine, Computer Science Department, Apr. 1998.
Burke, Robin, Integrating Knowledge-based and Collaborative-filtering Recommender Systems, AAAI Technical Report WS-99-01. Compilation copyright © 1999, AAAI (www.aaai.org), pp. 69-72.
Craswell, et al.; Random Walks on the Click Graph; Jul. 2007; SIGIR '07, Amsterdam, the Netherlands, 8 pages.
Cutrell, et al.; Eye tracking in MSN Search: Investigating snippet length, target position and task types; 2007; Conference on Human Factors in Computing Systems-Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
Dan Olsen et al., "Query-by-critique: Spoken Language Access to Large Lists", ACM, Oct. 2002, pp. 131-140.
Danish Search Report and Written Opinion for Application No. 200601630-7, dated Jun. 21, 2007, 15 pages.
Diligenti, et al., Users, Queries and Documents: A Unified Representation for Web Mining, wi-iat, vol. 1, 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, 2009, pp. 238-244.
Gabriel Somlo et al., "Using Web Hepler Agent Profiles in Query Generation", ACM, Jul. 2003, pp. 812-818.
Google News archive, Jul. 8, 2003, Webmasterworld.com, [online] Retrieved from the Internet http://www.webmasterwolrd.com/forum3/15085.htm [retrieved on Nov. 20, 2009] 3 pages.
Gr{hacek over (c)}ar, Miha, User Profiling: Collaborative Filtering, SIKDD 2004, Oct. 12-15, 2004, Ljubljana, Slovenia, 4 pages.
Hofmann, Thomas, Latent Semantic Models for Collaborative Filtering, ACM Transactions on Information Systems, vol. 22, No. 1, Jan. 2004, pp. 89-115.
Hungarian Patent Office, International Search Report and Written Opinion for Application No. 200806756-3, dated Nov. 19, 2010 12 pages.
Indian Office Action in Indian Application No. 686/KOLNP/2006, mailed Jun. 3, 2008, 2 pages.
International Search Report and Written Opinion for Application No. PCT/US2004/029615, dated Jan. 19, 2005, 8 pages.
Jansen et al., "An Analysis of Web Documents Retrieved and Viewed", School of Information Sciences and Technology, The Pennsylvania State University, the 4th International Conference on Internet Computing, Las Vegas, Nevada, pp. 65-69, Jun. 23-26, 2003, 5 pages.
Ji-Rong Wen et al., "Query Clustering using User Logs", ACM, Jan. 2002, pp. 59-81.
Joachims et al., "Search Engines that Learn from Implicit Feedback"; Aug. 2007, IEEE Computer Society.
Joachims, "Evaluating Search Engines Using Clickthrough Data", Cornell University, Department of Computer Science, Draft, Feb. 19, 2002, 13 pages.
Joachims, T., Evaluating retrieval performance using clickthrough data. Proceedings of the SIGIR Workshop on Mathematical/Formal Methods in Information Retrieval; Aug. 12-15, 2002; Tampere, Finland, 18 pages.
Joachims; Optimizing search engines using clickthrough data; 2002; Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 133-142.
Jones et al., "Pictures of Relevance: A Geometric Analysis of Similarity Measures", Journal of the American Society for Information Science, Nov. 1987, 23 pages.
Kaplan et al., "Adaptive Hypertext Navigation Based on User Goals and Context", User Modeling and User-Adapted Interaction 2, Sep. 1, 1993; pp. 193-220, 28 pages.
Kelly, et al.; Implicit Feedback for Inferring User Preference: A Bibliography; SIGIR Forum, vol. 37, No. 2 (2003), pp. 18-28.
Lemire, Daniel, Scale and Translation Invariant Collaborative Filtering Systems, Published in Information Retrieval, 8(1), pp. 129-150, 2005.
Liddy et al., "A Natural Language Text Retrieval System With Relevance Feedback", 16th National Online, May 2-6, 1995, 3 pages.
Linden, Greg et al., Amazon.com Recommendations: Item-to-Item Collaborative Filtering, [online], http://computer.org/internet/, IEEE Internet Computing, Jan.-Feb. 2003, IEEE Computer Society, pp. 76-80.
Mandala, Rila, Takenobu Tokunaga, and Hozumi Tanaka. "Combining multiple evidence from different types of thesaurus for query expansion." Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 1999.
Nicolas Bruno et al., "Top-K Selection Queries over Relational Databases: Mapping Strategies and Performance Evaluation", ACM, Jun. 2002, pp. 153-187.
Nicole, Kristen, Heeii is StumbleUpon Plus Google Suggestions, [online], Retrieved from the Internet http://mashable.com/2007/05/15/heeii/, 11 pages.
Radlinski, et al., Query Chains: Learning to Rank from Implicit Feedback, KDD '05, Aug. 21-24, 2005, Chicago, Illinois, USA, 10 pages.
Schwab, et al., Adaptivity through Unobstrusive Learning, 2002, 16(3), pp. 5-9.
Soumen Chakrabarti, et al. "Enhanced Topic Distillation using Text, Markup tags, and Hyperlinks" ACM, Sep. 9-12, 2001, pp. 208-216.
Stoilova, Lubomira et al., GiveALink: Mining a Semantic Network of Bookmarks for Web Search and Recommendation, LinkKDD '05, Aug. 21, 2005, Chicago, IL, USA, 8 pages.
Susan Gauch et al., "A Corpus Analysis Approach for Automatic Query Expansion and its Extension to Multiple Databases", ACM, Jul. 1999, pp. 250-269.
U.S. Patent Office, U.S. Appl. No. 11/556,086 filed Nov. 2, 2006, in Office Action mailed Jun. 23, 2010, 21 pages.
U.S. Patent Office, U.S. Appl. No. 11/556,143, filed Nov. 2, 2006, in Office Action mailed Apr. 20, 2011, 18 pages.
U.S. Patent Office, U.S. Appl. No. 11/556,143, filed Nov. 2, 2006, in Office Action mailed Jan. 25, 2010, 14 pages.
U.S. Patent Office, U.S. Appl. No. 11/556,143, filed Nov. 2, 2006, in Office Action mailed Jul. 6, 2010, 20 pages.
U.S. Patent Office, U.S. Appl. No. 11/685,095, filed Mar. 12, 2007, in Office Action mailed Apr. 13, 2011, 31 pages.
U.S. Patent Office, U.S. Appl. No. 11/685,095, filed Mar. 12, 2007, in Office Action mailed Feb. 25, 2009, 21 pages.
U.S. Patent Office, U.S. Appl. No. 11/685,095, filed Mar. 12, 2007, in Office Action mailed Feb. 8, 2010, 31 pages.
U.S. Patent Office, U.S. Appl. No. 11/685,095, filed Mar. 12, 2007, in Office Action mailed Sep. 10, 2009, 23 pages.
Velez, Bienvenido, et al. "Fast and effective query refinement." ACM SIGIR Forum. vol. 31. No. SI. ACM, 1997.
W3C, URIs, URLs and URNs: Classification and Recommendations 1.0, Report from the joint W3C/IETF URI Planning Interest Group, Sep. 21, 2001, 8 pages.
Xiao, et al., Measuring Similarity of Interests for Clustering Web-Users, ADC, 2001, pp. 107-114.
Xie et al., Web User Clustering from Access Log Using Belief Function, K-CAP '01, Oct. 22-23, 2001, Victoria, British Columbia, Canada, pp. 202-208.
Yu et al., Selecting Relevant Instances for Efficient and Accurate Collaborative Filtering, CIKM '01, Nov. 5-10, 2001, Atlanta, Georgia, pp. 239-246.
Zeng et al., Similarity Measure and Instance Selection for Collaborative Filtering, WWW '03, May 20-24, 2003, Budapest, Hungary, pp. 652-658.
Zeng, et al., "Learning to Cluster Web Search Results", SIGIR '04, Proceedings of the 27th Annual International ACM SIGIR conference on research and development in information retrieval, 2004.

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130067364A1 (en) * 2011-09-08 2013-03-14 Microsoft Corporation Presenting search result items having varied prominence
US9335883B2 (en) * 2011-09-08 2016-05-10 Microsoft Technology Licensing, Llc Presenting search result items having varied prominence

Also Published As

Publication number Publication date
US8909655B1 (en) 2014-12-09

Similar Documents

Publication Publication Date Title
US9152678B1 (en) Time based ranking
JP6600703B2 (en) Dynamic content item creation
US11694228B1 (en) System and method for optimizing online marketing based upon relative advertisement placement
US7840438B2 (en) System and method for discounting of historical click through data for multiple versions of an advertisement
US9495442B2 (en) System and method for automatically publishing data items associated with an event
US9058613B2 (en) Hybrid advertising campaign
US20170262899A1 (en) Computing Mathematically-Optimized Properties for Paid Search
US20090106087A1 (en) Contextual auction bidding
US10810275B2 (en) Query-dependent and content-class based ranking
US20100241516A1 (en) Editing interface
US20090248513A1 (en) Allocation of presentation positions
US8799061B1 (en) Classifying users for ad targeting
US20140214883A1 (en) Keyword trending data
US20150100423A1 (en) Delivering and pricing sponsored content items
US8700465B1 (en) Determining online advertisement statistics
US20130262218A1 (en) Incorporating Delayed Feedback In Performance-Based Content Distribution
CA2721647C (en) Interactive placement ordering
US9087339B2 (en) Content performance estimation
US8635542B2 (en) Campaign monitor
US8473838B2 (en) Website advertising inventory
GB2556970A (en) Method and system for providing content
AU2013205758B2 (en) Hybrid advertising campaign
US20170024775A1 (en) Valuing distribution data

Legal Events

Date Code Title Description
AS Assignment

Owner name: GOOGLE INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MCDONNELL, PHILIP;REEL/FRAME:034814/0646

Effective date: 20071011

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044334/0466

Effective date: 20170929

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8