US7603350B1 - Search result ranking based on trust - Google Patents

Search result ranking based on trust Download PDF

Info

Publication number
US7603350B1
US7603350B1 US11/382,449 US38244906A US7603350B1 US 7603350 B1 US7603350 B1 US 7603350B1 US 38244906 A US38244906 A US 38244906A US 7603350 B1 US7603350 B1 US 7603350B1
Authority
US
United States
Prior art keywords
resource
query
term
trust
entity
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US11/382,449
Inventor
Ramanathan Guha
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Google LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google LLC filed Critical Google LLC
Priority to US11/382,449 priority Critical patent/US7603350B1/en
Assigned to GOOGLE INC. reassignment GOOGLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GUHA, RAMANATHAN
Priority to US12/552,956 priority patent/US8352467B1/en
Application granted granted Critical
Publication of US7603350B1 publication Critical patent/US7603350B1/en
Priority to US13/614,339 priority patent/US8818995B1/en
Priority to US14/445,209 priority patent/US10268641B1/en
Assigned to GOOGLE LLC reassignment GOOGLE LLC CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: GOOGLE INC.
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2457Query processing with adaptation to user needs
    • G06F16/24578Query processing with adaptation to user needs using ranking
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/951Indexing; Web crawling techniques
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • Y10S707/99935Query augmenting and refining, e.g. inexact access

Definitions

  • the present invention relates to search engines, and more specifically to search engines that use information indicative of trust relationship between users to rank search results.
  • search engines in the context of the Internet and World Wide Web, use a wide variety of techniques to improve the quality and usefulness of the search results. These techniques address every possible aspect of search engine design, from the basic indexing algorithms and document representation, through query analysis and modification, to relevance ranking and result presentation, methodologies too numerous to fully catalog here.
  • vertical knowledge sites typically community sites for users of shared interests, allow users to link to content on the Internet and provide labels or tags describing the content.
  • a site may enable a user to link to the website of an automobile manufacturer, and post comment or description about a particular car being offered by the manufacturer; similarly, such a site could enable a user to link to a news report on the website of a news organization and post comment about the report.
  • These and other vertical knowledge sites may also host the analysis and comments of experts or others with knowledge, expertise, or a point of view in particular fields, who again can comment on content found on the Internet.
  • a website operated by a digital camera expert and devoted to digital cameras typically includes product reviews, guidance on how to purchase a digital camera, as well as links to camera manufacturer's sites, new products announcements, technical articles, additional reviews, or other sources of content.
  • the expert may include comments on the linked content, such as labeling a particular technical article as “expert level,” or a particular review as “negative professional review,” or a new product announcement as “new 10 MP digital SLR”.
  • a user interested in a particular point of view, type of information, or the like then search within the domain of such a site for articles or links that have certain associated labels or comments. For example, a user could search the aforementioned digital camera site for all camera reviews labeled “digital SLR”.
  • a particular user's (or author's) views may not be trusted by others.
  • identifying which other members of a community are trustworthy and which are not can be a difficult and time consuming process.
  • Some vertical knowledge sites now provide various types of indicators or proxies for the trustworthiness of particular individuals who participate at the site.
  • Auction sites use rating systems to identify trusted buyers and sellers.
  • Forum sites use membership criteria and other factors to distinguish between posters. But many sites simply rely on general reputation of their experts to instill a sense of trust in users who visit the site. Thus, at best a user can currently search within the context or domain of a particular website for comments, opinions or the like made by individuals who are trusted by the user or by others.
  • a search engine system provides search results that are ranked according to a measure of the trust associated with entities that have provided labels for the documents in the search results.
  • the system maintains information describing annotations provided by any of the entities.
  • An entity creates an annotation for a particular item (or collection) of web content as a way of associating a label with a particular item of web content, such as a web page.
  • an entity such as a digital camera expert operating a website devoted to digital cameras, may create an annotation associating the label “professional review” with a particular review of a digital camera on some third party site (e.g., on the site of a news publication).
  • the system maintains information about trust relationships between entities, such as individual users, indicating whether (or the degree to which) one entity trusts another entity.
  • entities such as individual users, indicating whether (or the degree to which) one entity trusts another entity.
  • users can indicate whether or the extent to which they trust the digital camera expert.
  • the system can determine a trust rank for any particular entity, based on the trust relationships between the various entities.
  • a user provides a query to the system; the query contains at least one query term and optionally includes one or more labels of interest to the user.
  • the system retrieves a set of search results comprising documents that are relevant to the query term(s).
  • the system determines which query labels are applicable to which of the search result documents.
  • the system determines for each document an overall trust factor to apply to the document based on the trust ranks of those entities that provided the labels that match the query labels. Applying the trust factor to the document adjusts the document's information retrieval score, to provide a trust adjusted information retrieval score.
  • the system reranks the search result documents based at on the trust adjusted information retrieval scores.
  • the present invention has various embodiments, including as a computer implemented process, as computer apparatuses, as integrated circuits, and as computer program products that execute on general or special purpose processors.
  • FIG. 1 illustrates a generalized system architecture for a search engine system in accordance with one embodiment.
  • FIG. 2 is a flowchart of a method for collecting trust information.
  • FIG. 3 is a flowchart of a method for collecting annotation information.
  • FIG. 4 is a flowchart of a method for processing search results with trust ranks.
  • FIG. 5 is an example of a search query including a label and corresponding search results.
  • FIG. 1 presents a diagram illustrating a system that ranks search results using trust in accordance with an embodiment of the present invention.
  • a user operates a browser 104 located on a client 103 .
  • Client 103 can generally include any node on a network including computational capability and including a mechanism for communicating across the network.
  • the client 103 can be any type of client, including any type of computer (e.g., desktop computer, workstation, notebook, mainframe, terminal, etc.), handheld device (personal digital assistant, cellular phone, etc.), or the like.
  • the client device 103 need only have the capability to communicate over a network (e.g. Internet, telephony, LAN, WAN, or combination thereof) with the search engine system 100 .
  • a client device 103 will also support appropriate networking applications and components, all of which are known to those of skill in the art.
  • Browser 104 can generally include any type of web browser capable of viewing a web site, such as the INTERNET EXPLORERTM browser distributed by the Microsoft Corporation of Redmond, Wash.
  • Search engine system 100 can generally include any computational node including a mechanism for servicing queries from a client for computational and/or data storage resources. Note that the code and data involved in processing queries typically resides within a memory within search engine system 100 . Copies of the code and data can also reside within non-volatile storage which is included in the system 100 . The figure does not show a number of conventional components (e.g. network, firewalls, routers, domain name servers, load balancers, etc.) in order to not obscure the relevant details of the embodiment.
  • conventional components e.g. network, firewalls, routers, domain name servers, load balancers, etc.
  • a user's client 103 can communicate with search engine system 100 to search for documents relevant to a query.
  • a query general comprises one or more query terms and optionally includes one or more labels.
  • Query terms are words that the user believes are relevant to the user's information need; labels are words, phrases, markers or other indicia that have been associated with certain web content (pages, sites, documents, media, etc.) by others as descriptive or categorical identifiers.
  • a query “cancer label:symptoms” includes the query term cancer and a query label “symptoms”, and thus is a request for documents relevant to cancer, and that has been labeled as relating to “symptoms.” Labels such as these can be associated with documents from any entity, whether the entity created the document, or is a third party. The entity that has labeled a document has some degree of trust, as further described below.
  • the browser 104 can also access any web site available on the network (not shown) belonging to a person, or any other type of entity such as a company, enterprise, community group, and so forth.
  • browser 104 accesses Expert's web site 105 , which is hosted by a third-party server, or alternatively by the server associated with the search engine system 100 .
  • Expert's web site 105 can generally be any type of web content provided by Expert.
  • Expert's web site 105 can include Expert's personal home page, along with pages directed towards Expert's interests, as well as Expert's blog, an e-commerce storefront for goods that Expert sells, and so forth.
  • Expert's site 105 can include pages containing links to other sites (including entire sites, portions of a site, or individual pages) along with annotations 106 associated with such linked content.
  • An annotation 106 includes a label 107 and a URL pattern associated with the label; the URL pattern can be specific to an individual web page or to any portion of a web site or pages therein.
  • the term “document” is used herein to refer to any and all types of content that can be accessed by a client over a network.
  • Expert may create an annotation 106 including the label 107 “Professional review” for a review 114 of Canon digital SLR camera on a web site “www.digitalcameraworld.com”, a label 107 of “Jazz music” for a CD 115 on the site “www.jazzworld.com”, a label 107 of “Classic Drama” for the movie 116 “North by Northwest” listed on website “www.movierental.com”, and a label 107 of “Symptoms” for a group of pages describing the symptoms of colon cancer on a website 117 “www.yourhealth.com”.
  • labels 107 can also include numerical values (not shown), indicating a rating or degree of significance that the entity attaches to the labeled document.
  • Expert's web site 105 can also include trust information. More specifically, Expert's web site 105 can include a trust list 109 of entities whom Expert trusts. This list may be in the form of a list of entity names, the URLs of such entities' web pages, or by other identifying information. Expert's web site 105 may also include a vanity list 111 listing entities who trust Expert; again this may be in the form of a list of entity names, URLs, or other identifying information. The trust list 109 and vanity list 111 may be encoded in any computer readable form supported by Expert's website. An “entity” may be a specific person, group, organization, website, business, institution, government agency or the like.
  • a web crawler (not shown) obtains labels and trust information and sends it to search engine system 100 to facilitate subsequent usage in search result ranking. This process is described in more detail below with reference to FIGS. 2-4 .
  • Expert's web site 105 can also include a “trust button” 112 .
  • a user visiting Expert's web site can click on trust button 112 to cause a corresponding trust relationship to be recorded at the search engine system 100 .
  • This trust relationship indicates that the user trusts Expert, and may be understood as well to indicate that the user trusts the labels that Expert has provided for various documents and site.
  • the trust button 112 can be implemented as an inline frame (iframe) provided by such server, which examines a cookie to identify user and then provides an identifier for the user and an identifier for Expert to the search engine system 100 . When accessed, the search engine system 100 receives information indicative of the trust relationship between the user and Expert.
  • This information may take the form of a tuple of data ⁇ entity 1 , entity 2 , trust_value> identifying the trusting entity, the trusted entity, and a trust value.
  • the tuple ⁇ 3365, 1230, 1> indicates that the entity (e.g., a user) with the identifier 3365 trusts the entity with the identifier 1230 (e.g., Expert) with a trust value of 1.
  • the trust information is stored in a trust database 190 .
  • a user can have different levels of trust in an entity for different topics. For example, a user may trust an entity with respect to politics and economics, but not with respect to sports and entertainment.
  • the trust information tuple can thus be extended as ⁇ entity 1 , entity 2 , topic, trust_value>, where topic identifies the particular topic for which entity 1 trusts entity 2 (the topic can be identified by a text string, a topic identifier, or other means).
  • the user manifestation of topic based trust can vary.
  • the trust button 112 can be linked to a further selection or list of topics.
  • the user can include a list of the trusted topics for each entity he or she trusts.
  • the trust information provided by the users with respect to others is used to determine a trust rank for each user, which is measure of the overall degree of trust that users have in the particular entity.
  • user's client 103 sends a search query the search engine system 100 , which is received by a front end server 110 .
  • the front end server 110 is adapted to receive a query from the client 102 and pass it to the query processor 120 , and then subsequently receive the search results back from the query processor 120 and provide them to the client 102 .
  • the front end server 110 is further adapted to handle the queries from multiple different clients 102 concurrently.
  • the front end server 110 passes the query to the search engine 180 , which processes a user's query (using a content server and index 170 ) to obtain a set of search results that are relevant to the query term(s). Where the query includes labels (as these are optional), the search engine 180 accesses an annotation database 140 to determine which of the documents in the search results have one or more of the query labels associated with them. Using the annotation information and trust information from the trust database 190 , the search engine 180 determines a trust factor for each document. A document's trust factor is a function of the trust ranks associated with the entities have labeled the document with labels that match the query labels. The search engine 180 adjusts each document's underlying information retrieval score using the document's trust factor, and then reranks the search results using the adjusted scores. The trust adjusted search results are returned to the front end server 110 , which passes them to the client 103 .
  • FIG. 5 illustrates an example of search query including a label and corresponding search results.
  • Search query field 506 is shown including a search query 500 that include the query term “cancer” and the indicated label “symptoms.” This query has been provided to the search engine system 100 , and some of the search results 502 (a set of documents) as shown in trust-adjusted ranked order.
  • each search result 502 (document) is the indicia of the matching label 504 “symptom” for each shown search result, as well as other labels 504 such as “Tests/diagnosis,”, and “Treatment.” Also shown next to each result is the name 508 of the entity that provided the matching label, here the user named Guha; the name 508 of another user Roni, is also shown on one of the results. The results are listed in an order based on the trust ranks of the entities 508 which have provided the labels 504 that match the query label of “symptoms.” Thus, the user can determine the quality and significance of each search result in part by seeing which entities have provided the labels that match the query labels.
  • FIG. 2 presents a flowchart illustrating the process of obtaining and storing trust information in accordance with an embodiment of the present invention
  • the system 100 receives 202 information indicating that the user trusts labels provided by the entity. This can involve receiving this information directly from the user. For example, the user can click on a “trust button” on a web page belonging to the entity, which causes a corresponding record for a trust relationship to be recorded in the trust database 190 . In general any type of input from the user indicating that such as trust relationship exists can be used.
  • the system can also use a web crawler to examine web pages to locate information indicating that which user trusts a particular entity. While examining web pages, the web crawler can look for a number of relationships, including: (1) links from the user's web page to web pages belonging to trusted entities; (2) a trust list that identifies entities that the user trusts; or (3) a vanity list which identifies users who trust the owner of the vanity page.
  • the system can also examine web visitation patterns of the user and can infer from the web visitation patterns which entities the user trusts. For example, the system can infer that a particular user trust a particular entity when the user visits the entity's web page with a certain frequency.
  • the search engine system 100 can also examine computer-based records belonging to the user, such as an email contact list or an instant-messaging chat list, and can infer from these computer-based records that the user trusts entities included in such contact or chat lists. More particular, for each entity included in the user's contact/chat list, a trust record indicating a trust relationship between the user and entity can be entered in the trust database 190 . Alternatively, a user may specifically mark or identify entities in the user's contact or instant messaging list as being trusted, or with a particular degree of trust, or trusted topics.
  • the search engine system 100 can also infer trust transitively, by assigning a trust value between a user and first entity, where the user directly trusts a second entity, and the second entity in turn trusts the first entity.
  • the search engine system 100 stores 204 a corresponding record for the trust relationship between the user and the entity in the trust database 190 to facilitate subsequent data retrieval operations.
  • the trust database 190 can include any type of data structure or database system that can facilitate data retrieval of trust information.
  • the trust information in the trust database 190 is stored in a square trust matrix M, where each matrix value M ij stores a value indicative of entity i's trust of entity j.
  • the trust value is 0 or 1, though other values or ranges of values can be used.
  • the matrix information can be stored in any suitable storage structure, including tables, arrays, bit maps, trees, or the like.
  • a trust rank TR i can be computed 206 from the trust matrix M.
  • the TR i of entity i is the i th component of the eigenvector of TR that is associated with the eigenvalue of 1.
  • a separate trust matrix M is constructed for each topic t and a topic-specific trust rank TR it is computed.
  • the topic matrices can aggregated at various levels according to topical hierarchies, and aggregated topic-specific trust ranks may also be computed therefrom. While FIG. 2 shows the computation 206 of trust rank following storing 204 , these steps need not occur in this order, and in one embodiment, the computation 206 of trust rank is executed in an independent process.
  • trust relationships can change.
  • the system can increase (or decrease) the strength of a trust relationship for a trusted entity.
  • the search engine system 100 can also cause the strength of a trust relationship to decay over time if the trust relationship is not affirmed by the user, for example by visiting the entity's web site and activating the trust button 112 .
  • the search engine system 100 may also expose a user interface to the trust database 190 by which the user can edit the user trust relationships, including adding or removing trust relationships with selected entities.
  • the trust information in the trust database 190 is also periodically updated by crawling of web sites, including sites of entities with trust information (e.g., trust lists, vanity lists); trust ranks are recomputed based on the updated trust information.
  • trust information e.g., trust lists, vanity lists
  • FIG. 3 presents a flowchart illustrating the process of obtaining and storing annotations (including labels) on web content in accordance with an embodiment of the present invention
  • the system gathers 302 annotations 106 associated with an entity. This can involve using a web crawler to examine web pages associated with the entity to identify annotations 106 created by the entity.
  • a web crawler to examine web pages associated with the entity to identify annotations 106 created by the entity.
  • the system can also receive annotations 106 from an entity via an annotation interface 160 that is configured receive annotation files.
  • An annotation file contains annotations.
  • an annotation includes a pattern for a uniform resource locator (URL) for the URLs of documents, and a label to be applied to a document whose URL matches the URL pattern.
  • URL uniform resource locator
  • an annotation may take the form:
  • URL_pattern is a specification of a pattern for a URL.
  • the URL pattern can include wildcards as well as regular expressions.
  • the annotation database 104 includes thousands, even millions of such annotations. There is a many-to-many relationship between labels and URL patterns, in that a given label may be applied to any document matching multiple different URL patterns, and a given URL pattern may be associated with many different labels.
  • the annotation database 140 includes an inverted index (or equivalent) that indexes each annotation to whatever URL patterns have been defined for it (whether by the same or different entities).
  • an entity may simply create a link from its site to a particular item of web content (e.g., a document) and provide a label 107 as the anchor text of the link.
  • a crawler extracts the label from the link and generates the appropriate annotation by using the label and the associated URL of the linked content.
  • Each annotation is associated with the entity that provided the annotation.
  • the trust rank TR for the associated entity can thereby be accessed as follows.
  • the annotations 106 associated with the document can be determined from the annotation database 140 .
  • the entity that provided the annotation can be determined, and that entity's trust rank TR retrieved from the trust database 190 .
  • the search engine system 100 periodically updates the annotation database 140 by crawling web sites and extracting annotations therein, as well as receiving annotation files via the annotation interface 160 .
  • FIG. 4 presents a flowchart illustrating one process of generating trust ranked search results in accordance with an embodiment of the present invention
  • the front end server 110 receives 402 a query from a user.
  • the query includes at least one query term, and optionally includes one or more labels, as described above.
  • the query is provided to the search engine 180 .
  • a query may be
  • the terms “digital camera” are the query terms, and the token “label:” indicates that the following term is a label of interest.
  • the user is searching for documents that have been labeled (by one or more individuals, such as other users, experts, etc.) as being professional reviews of digital cameras.
  • the user would be searching for documents that relevant to abortion and had been labeled as either statistics or pro-choice related.
  • the particular syntax used to indicate labels is not limited to the use of “label:” and other tokens and formats can be used as well.
  • the query can take the form such as
  • the front end server 110 (or other module) includes a parser that performs the initial parsing of the query into query term and labels.
  • the search engine 180 retrieves 404 a set of documents that are relevant to the query term(s).
  • the search engine 180 can use any type of information retrieval model for selecting and scoring relevant documents.
  • One suitable information retrieval model is a link-based model, such as PageRank which is described in U.S. Pat. No. 6,285,999, and incorporated by reference herein.
  • the retrieved documents are ranked ordered in terms of an underlying (base) information retrieval score used by the search engine 180 .
  • the labels associated with the document are determined 406 from the annotation database 140 by matching the URL of the document to the URL patterns in the annotations. Thus, if the URL of a particular search result document matches the three different annotations for three different labels, each of these labels will be determined as associated with the document.
  • the query labels are then matched against the retrieved annotation labels, to identify which query labels apply to the document; this can be done by matching the query labels to the document's annotated labels.
  • the trust rank TR of entity that provided the annotation label is retrieved 408 .
  • a document may have been labeled with the same label by several different entities. For example, three different entities may have labeled a particular review of digital camera with the label “professional review” while a fourth labeled the same review with the label “negative review.” If the query label included “professional review” then only the first three annotation labels would be deemed matched. It should be noted that steps 406 and 408 can performed in batch against all the search results, or iteratively for each search result, depending the desired programming model.
  • the trust ranks of the entities associated with the matching labels are aggregated 410 to create an aggregated trust rank.
  • the trust ranks of the three different experts would be aggregated into a single trust rank associated with the particular label “professional review” for the review document. This aggregation is performed for each label associated with the document.
  • the trust ranks of the entities could be aggregated with respect to these labels.
  • the trust ranks can be aggregated using a weighting function. Aggregation using a linear weighting function would be:
  • This approach basically applies a fixed trust weight (e.g., 1) to the trust ranks prior to summing them.
  • a fixed trust weight e.g. 1, 1
  • the trust ranks TR can be aggregated in a number of different ways other than linear weighting.
  • One variation is an asymptotic weighting function.
  • one approach here would be to sum the log of the trust ranks to form the aggregated trust rank.
  • Another variation of the weighting function is to apply a weight that decays with the increasing number of instances of a particular label.
  • the trust ranks can be ordered by age of the associated annotation, and weighted to decrease (alternatively increase) the weight applied to the trust rank for the oldest annotation.
  • Another way in which the trust ranks can be aggregated is by use of a sigmoid weighting function.
  • the aggregated trust ranks are applied 412 to the document's base information retrieval score.
  • the aggregated trust ranks can be combined into a trust factor, using any type of aggregation function.
  • trust factor can be the sum total of the aggregated trust ranks.
  • the aggregated trust ranks can be aggregated asymptotically or via a sigmoid function to form the trust factor.
  • the trust factor is determined from the aggregated trust ranks, it is then applied to the base information retrieval score, for example by multiplying with the trust factor. Other ways of applying the trust factor to the base information retrieval score as also possible.
  • the resulting score is said to be trust-adjusted information retrieval score.
  • the search engine 180 After (or as) the trust-adjusted information retrieval scores have been determined, the search engine 180 reranks 414 the search results by these adjusted scores.
  • the result set is returned to the front end server 110 , which in turn passes it to the client 103 . The user can then review the search results.
  • the search engine system 100 is adapted to receive queries that do not include labels, and still provide trust based ranking. In this case, if certain annotations are applicable to a search result document, then the trust rank for the entities providing these annotations is retrieved, aggregated, and applied to the base information retrieval score of the document as well, and the documents are reranked accordingly.
  • Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.
  • the present invention also relates to an apparatus for performing the operations herein.
  • This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer.
  • a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.

Abstract

A search engine system provides search results that are ranked according to a measure of the trust associated with entities that have provided labels for the documents in the search results. A search engine receives a query and selects documents relevant to the query. The search engine also determines labels associated with selected documents, and the trust ranks of the entities that provided the labels. The trust ranks are used to determine trust factors for the respective documents. The trust factors are used to adjust information retrieval scores of the documents. The search results are then ranked based on the adjusted information retrieval scores.

Description

FIELD OF THE INVENTION
The present invention relates to search engines, and more specifically to search engines that use information indicative of trust relationship between users to rank search results.
BACKGROUND OF THE INVENTION
The development of information retrieval systems has predominantly focused on improving the overall quality of the search results presented to the user. The quality of the results has typically been measured in terms of precision, recall, or other quantifiable measures of performance. Information retrieval systems, or ‘search engines’ in the context of the Internet and World Wide Web, use a wide variety of techniques to improve the quality and usefulness of the search results. These techniques address every possible aspect of search engine design, from the basic indexing algorithms and document representation, through query analysis and modification, to relevance ranking and result presentation, methodologies too numerous to fully catalog here.
An inherent problem in the design of search engines is that the relevance of search results to a particular user depends on factors that are highly dependent on the user's intent in conducting the search—that is why they are conducting the search—as well as the user's circumstances, the facts pertaining to the user's information need. Thus, given the same query by two different users, a given set of search results can be relevant to one user and irrelevant to another, entirely because of the different intent and information needs. Most attempts at solving the problem of inferring a user's intent typically depend on relatively weak indicators, such as static user preferences, or predefined methods of query reformulation that are nothing more than educated guesses about what the user is interested in based on the query terms. Approaches such as these cannot fully capture user intent because such intent is itself highly variable and dependent on numerous situational facts that cannot be extrapolated from typical query terms.
In part because of the inability of contemporary search engines to consistently find information that satisfies the user's information need, and not merely the user's query terms, users frequently turn to websites that offer additional analysis or understanding of content available on the Internet. For the purposes of discussion these sites are called vertical knowledge sites. Some vertical knowledge websites, typically community sites for users of shared interests, allow users to link to content on the Internet and provide labels or tags describing the content. For example, a site may enable a user to link to the website of an automobile manufacturer, and post comment or description about a particular car being offered by the manufacturer; similarly, such a site could enable a user to link to a news report on the website of a news organization and post comment about the report. These and other vertical knowledge sites may also host the analysis and comments of experts or others with knowledge, expertise, or a point of view in particular fields, who again can comment on content found on the Internet. For example, a website operated by a digital camera expert and devoted to digital cameras typically includes product reviews, guidance on how to purchase a digital camera, as well as links to camera manufacturer's sites, new products announcements, technical articles, additional reviews, or other sources of content. To assist the user, the expert may include comments on the linked content, such as labeling a particular technical article as “expert level,” or a particular review as “negative professional review,” or a new product announcement as “new 10 MP digital SLR”. A user interested in a particular point of view, type of information, or the like then search within the domain of such a site for articles or links that have certain associated labels or comments. For example, a user could search the aforementioned digital camera site for all camera reviews labeled “digital SLR”.
One of the underlying aspects of vertical knowledge sites that makes them appealing to users is that some of the user who participate on them come to be perceived as being trustworthy in their comments, analysis, opinions and recommendations. This degree of trust is valuable to users as a way of evaluating the often bewildering array of information that is available on the Internet. Indeed, many popular vertical knowledge sites, blogs, news outlets, so forth, are primarily devoted to facilitating dissemination of the opinions of individual experts or commentators, while other vertical knowledge sites such as forums, rating sites, and community sites operate to share and disseminate the opinions of many users in a community. In either case, many users access these sites because of an underlying sense of trust in at least some of the others users who are providing their opinions. Of course, in some cases a particular user's (or author's) views may not be trusted by others. For example, on most forums there will be one or more users who are viewed by others as being uninformed, biased, hostile or otherwise not trustworthy. Particularly for new users, identifying which other members of a community are trustworthy and which are not can be a difficult and time consuming process.
Some vertical knowledge sites now provide various types of indicators or proxies for the trustworthiness of particular individuals who participate at the site. Auction sites use rating systems to identify trusted buyers and sellers. Forum sites use membership criteria and other factors to distinguish between posters. But many sites simply rely on general reputation of their experts to instill a sense of trust in users who visit the site. Thus, at best a user can currently search within the context or domain of a particular website for comments, opinions or the like made by individuals who are trusted by the user or by others.
The problem remains that when the user returns to a general search engine, outside of the vertical knowledge site, the user is unable to obtain search results that reflect the trustworthiness of the documents themselves or the trustworthiness of any commentary or opinions that may be associated with the search result documents. Thus, none of the additional reputation based information that is associated with users in the vertical knowledge site is available to the general search engine in order to provide more meaningful search results to other users.
SUMMARY
A search engine system provides search results that are ranked according to a measure of the trust associated with entities that have provided labels for the documents in the search results. The system maintains information describing annotations provided by any of the entities. An entity creates an annotation for a particular item (or collection) of web content as a way of associating a label with a particular item of web content, such as a web page. For example, an entity such as a digital camera expert operating a website devoted to digital cameras, may create an annotation associating the label “professional review” with a particular review of a digital camera on some third party site (e.g., on the site of a news publication). In addition, the system maintains information about trust relationships between entities, such as individual users, indicating whether (or the degree to which) one entity trusts another entity. Thus, users can indicate whether or the extent to which they trust the digital camera expert. The system can determine a trust rank for any particular entity, based on the trust relationships between the various entities.
A user provides a query to the system; the query contains at least one query term and optionally includes one or more labels of interest to the user. The system retrieves a set of search results comprising documents that are relevant to the query term(s). The system determines which query labels are applicable to which of the search result documents. The system determines for each document an overall trust factor to apply to the document based on the trust ranks of those entities that provided the labels that match the query labels. Applying the trust factor to the document adjusts the document's information retrieval score, to provide a trust adjusted information retrieval score. The system reranks the search result documents based at on the trust adjusted information retrieval scores.
The present invention has various embodiments, including as a computer implemented process, as computer apparatuses, as integrated circuits, and as computer program products that execute on general or special purpose processors.
The features and advantages described in this summary and the following detailed description are not all-inclusive. Many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims hereof.
BRIEF DESCRIPTION OF THE FIGURES
FIG. 1 illustrates a generalized system architecture for a search engine system in accordance with one embodiment.
FIG. 2 is a flowchart of a method for collecting trust information.
FIG. 3 is a flowchart of a method for collecting annotation information.
FIG. 4 is a flowchart of a method for processing search results with trust ranks.
FIG. 5 is an example of a search query including a label and corresponding search results.
The figures depict various embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the illustrated and described structures, methods, and functions may be employed without departing from the principles of the invention.
DETAILED DESCRIPTION System Overview
FIG. 1 presents a diagram illustrating a system that ranks search results using trust in accordance with an embodiment of the present invention. Referring to FIG. 1, a user operates a browser 104 located on a client 103. Client 103 can generally include any node on a network including computational capability and including a mechanism for communicating across the network. The client 103 can be any type of client, including any type of computer (e.g., desktop computer, workstation, notebook, mainframe, terminal, etc.), handheld device (personal digital assistant, cellular phone, etc.), or the like. The client device 103 need only have the capability to communicate over a network (e.g. Internet, telephony, LAN, WAN, or combination thereof) with the search engine system 100. Typically, a client device 103 will also support appropriate networking applications and components, all of which are known to those of skill in the art.
Browser 104 can generally include any type of web browser capable of viewing a web site, such as the INTERNET EXPLORER™ browser distributed by the Microsoft Corporation of Redmond, Wash. Search engine system 100 can generally include any computational node including a mechanism for servicing queries from a client for computational and/or data storage resources. Note that the code and data involved in processing queries typically resides within a memory within search engine system 100. Copies of the code and data can also reside within non-volatile storage which is included in the system 100. The figure does not show a number of conventional components (e.g. network, firewalls, routers, domain name servers, load balancers, etc.) in order to not obscure the relevant details of the embodiment.
During operation, a user's client 103 can communicate with search engine system 100 to search for documents relevant to a query. A query general comprises one or more query terms and optionally includes one or more labels. Query terms are words that the user believes are relevant to the user's information need; labels are words, phrases, markers or other indicia that have been associated with certain web content (pages, sites, documents, media, etc.) by others as descriptive or categorical identifiers. For example, a query “cancer label:symptoms” includes the query term cancer and a query label “symptoms”, and thus is a request for documents relevant to cancer, and that has been labeled as relating to “symptoms.” Labels such as these can be associated with documents from any entity, whether the entity created the document, or is a third party. The entity that has labeled a document has some degree of trust, as further described below.
The browser 104 can also access any web site available on the network (not shown) belonging to a person, or any other type of entity such as a company, enterprise, community group, and so forth. For example, in FIG. 1 browser 104 accesses Expert's web site 105, which is hosted by a third-party server, or alternatively by the server associated with the search engine system 100. Expert's web site 105 can generally be any type of web content provided by Expert. For example, Expert's web site 105 can include Expert's personal home page, along with pages directed towards Expert's interests, as well as Expert's blog, an e-commerce storefront for goods that Expert sells, and so forth. In addition, Expert's site 105 can include pages containing links to other sites (including entire sites, portions of a site, or individual pages) along with annotations 106 associated with such linked content. An annotation 106 includes a label 107 and a URL pattern associated with the label; the URL pattern can be specific to an individual web page or to any portion of a web site or pages therein. The term “document” is used herein to refer to any and all types of content that can be accessed by a client over a network.
For example, Expert may create an annotation 106 including the label 107 “Professional review” for a review 114 of Canon digital SLR camera on a web site “www.digitalcameraworld.com”, a label 107 of “Jazz music” for a CD 115 on the site “www.jazzworld.com”, a label 107 of “Classic Drama” for the movie 116 “North by Northwest” listed on website “www.movierental.com”, and a label 107 of “Symptoms” for a group of pages describing the symptoms of colon cancer on a website 117 “www.yourhealth.com”. Note that labels 107 can also include numerical values (not shown), indicating a rating or degree of significance that the entity attaches to the labeled document.
Expert's web site 105 can also include trust information. More specifically, Expert's web site 105 can include a trust list 109 of entities whom Expert trusts. This list may be in the form of a list of entity names, the URLs of such entities' web pages, or by other identifying information. Expert's web site 105 may also include a vanity list 111 listing entities who trust Expert; again this may be in the form of a list of entity names, URLs, or other identifying information. The trust list 109 and vanity list 111 may be encoded in any computer readable form supported by Expert's website. An “entity” may be a specific person, group, organization, website, business, institution, government agency or the like.
In one embodiment of the present invention, a web crawler (not shown) obtains labels and trust information and sends it to search engine system 100 to facilitate subsequent usage in search result ranking. This process is described in more detail below with reference to FIGS. 2-4.
Expert's web site 105 can also include a “trust button” 112. A user visiting Expert's web site can click on trust button 112 to cause a corresponding trust relationship to be recorded at the search engine system 100. This trust relationship indicates that the user trusts Expert, and may be understood as well to indicate that the user trusts the labels that Expert has provided for various documents and site. Note that if Expert's web site 105 is provided by a third-party server, the trust button 112 can be implemented as an inline frame (iframe) provided by such server, which examines a cookie to identify user and then provides an identifier for the user and an identifier for Expert to the search engine system 100. When accessed, the search engine system 100 receives information indicative of the trust relationship between the user and Expert. This information may take the form of a tuple of data <entity1, entity2, trust_value> identifying the trusting entity, the trusted entity, and a trust value. For example, the tuple <3365, 1230, 1> indicates that the entity (e.g., a user) with the identifier 3365 trusts the entity with the identifier 1230 (e.g., Expert) with a trust value of 1. The trust information is stored in a trust database 190.
In one embodiment, a user can have different levels of trust in an entity for different topics. For example, a user may trust an entity with respect to politics and economics, but not with respect to sports and entertainment. The trust information tuple can thus be extended as <entity1, entity2, topic, trust_value>, where topic identifies the particular topic for which entity1 trusts entity2 (the topic can be identified by a text string, a topic identifier, or other means). The user manifestation of topic based trust can vary. For example, the trust button 112 can be linked to a further selection or list of topics. For a trust list, the user can include a list of the trusted topics for each entity he or she trusts. The trust information provided by the users with respect to others is used to determine a trust rank for each user, which is measure of the overall degree of trust that users have in the particular entity.
In order to obtain search results, user's client 103 sends a search query the search engine system 100, which is received by a front end server 110. The front end server 110 is adapted to receive a query from the client 102 and pass it to the query processor 120, and then subsequently receive the search results back from the query processor 120 and provide them to the client 102. The front end server 110 is further adapted to handle the queries from multiple different clients 102 concurrently.
The front end server 110 passes the query to the search engine 180, which processes a user's query (using a content server and index 170) to obtain a set of search results that are relevant to the query term(s). Where the query includes labels (as these are optional), the search engine 180 accesses an annotation database 140 to determine which of the documents in the search results have one or more of the query labels associated with them. Using the annotation information and trust information from the trust database 190, the search engine 180 determines a trust factor for each document. A document's trust factor is a function of the trust ranks associated with the entities have labeled the document with labels that match the query labels. The search engine 180 adjusts each document's underlying information retrieval score using the document's trust factor, and then reranks the search results using the adjusted scores. The trust adjusted search results are returned to the front end server 110, which passes them to the client 103.
FIG. 5 illustrates an example of search query including a label and corresponding search results. Search query field 506 is shown including a search query 500 that include the query term “cancer” and the indicated label “symptoms.” This query has been provided to the search engine system 100, and some of the search results 502 (a set of documents) as shown in trust-adjusted ranked order. Of interest in each search result 502 (document) is the indicia of the matching label 504 “symptom” for each shown search result, as well as other labels 504 such as “Tests/diagnosis,”, and “Treatment.” Also shown next to each result is the name 508 of the entity that provided the matching label, here the user named Guha; the name 508 of another user Roni, is also shown on one of the results. The results are listed in an order based on the trust ranks of the entities 508 which have provided the labels 504 that match the query label of “symptoms.” Thus, the user can determine the quality and significance of each search result in part by seeing which entities have provided the labels that match the query labels.
Obtaining and Storing Trust Information
FIG. 2 presents a flowchart illustrating the process of obtaining and storing trust information in accordance with an embodiment of the present invention; other variations of the steps or procedures of this process can be readily developed to achieve the same results.
During this process, the system 100 receives 202 information indicating that the user trusts labels provided by the entity. This can involve receiving this information directly from the user. For example, the user can click on a “trust button” on a web page belonging to the entity, which causes a corresponding record for a trust relationship to be recorded in the trust database 190. In general any type of input from the user indicating that such as trust relationship exists can be used.
As indicated above, the system can also use a web crawler to examine web pages to locate information indicating that which user trusts a particular entity. While examining web pages, the web crawler can look for a number of relationships, including: (1) links from the user's web page to web pages belonging to trusted entities; (2) a trust list that identifies entities that the user trusts; or (3) a vanity list which identifies users who trust the owner of the vanity page.
The system can also examine web visitation patterns of the user and can infer from the web visitation patterns which entities the user trusts. For example, the system can infer that a particular user trust a particular entity when the user visits the entity's web page with a certain frequency.
The search engine system 100 can also examine computer-based records belonging to the user, such as an email contact list or an instant-messaging chat list, and can infer from these computer-based records that the user trusts entities included in such contact or chat lists. More particular, for each entity included in the user's contact/chat list, a trust record indicating a trust relationship between the user and entity can be entered in the trust database 190. Alternatively, a user may specifically mark or identify entities in the user's contact or instant messaging list as being trusted, or with a particular degree of trust, or trusted topics.
The search engine system 100 can also infer trust transitively, by assigning a trust value between a user and first entity, where the user directly trusts a second entity, and the second entity in turn trusts the first entity.
Next, the search engine system 100 stores 204 a corresponding record for the trust relationship between the user and the entity in the trust database 190 to facilitate subsequent data retrieval operations. The trust database 190 can include any type of data structure or database system that can facilitate data retrieval of trust information.
In one embodiment, the trust information in the trust database 190 is stored in a square trust matrix M, where each matrix value Mij stores a value indicative of entity i's trust of entity j. As noted above, in one embodiment, the trust value is 0 or 1, though other values or ranges of values can be used. The matrix information can be stored in any suitable storage structure, including tables, arrays, bit maps, trees, or the like.
For each entity i, a trust rank TRi can be computed 206 from the trust matrix M. The TRi of entity i is the ith component of the eigenvector of TR that is associated with the eigenvalue of 1. Where trust information includes topic information, a separate trust matrix M is constructed for each topic t and a topic-specific trust rank TRit is computed. The topic matrices can aggregated at various levels according to topical hierarchies, and aggregated topic-specific trust ranks may also be computed therefrom. While FIG. 2 shows the computation 206 of trust rank following storing 204, these steps need not occur in this order, and in one embodiment, the computation 206 of trust rank is executed in an independent process.
Note that trust relationships can change. For example, the system can increase (or decrease) the strength of a trust relationship for a trusted entity. The search engine system 100 can also cause the strength of a trust relationship to decay over time if the trust relationship is not affirmed by the user, for example by visiting the entity's web site and activating the trust button 112. The search engine system 100 may also expose a user interface to the trust database 190 by which the user can edit the user trust relationships, including adding or removing trust relationships with selected entities.
The trust information in the trust database 190 is also periodically updated by crawling of web sites, including sites of entities with trust information (e.g., trust lists, vanity lists); trust ranks are recomputed based on the updated trust information.
Obtaining and Storing Label Information
FIG. 3 presents a flowchart illustrating the process of obtaining and storing annotations (including labels) on web content in accordance with an embodiment of the present invention; other variations of the steps or procedures of this process can be readily developed to achieve the same results.
During this process, the system gathers 302 annotations 106 associated with an entity. This can involve using a web crawler to examine web pages associated with the entity to identify annotations 106 created by the entity. One system and methodology for analysis of annotations provided by entities is described in U.S. application Ser. No. 11/202,423, filed on Aug. 10, 2005 and entitled “Programmable Search Engine”, which is incorporated by reference herein.
The system can also receive annotations 106 from an entity via an annotation interface 160 that is configured receive annotation files. An annotation file contains annotations. As indicated previously, an annotation includes a pattern for a uniform resource locator (URL) for the URLs of documents, and a label to be applied to a document whose URL matches the URL pattern. Schematically, an annotation may take the form:
<label, URL_pattern>
where label is a term or phrase, and URL_pattern is a specification of a pattern for a URL.
For example, the annotation
<“professional review”, www.digitalcameraworld.com/review/>
would be used to apply the label “professional review” to any document whose URL includes a URL prefix matching the network location “www.digitalcameraworld.com/review/”; all documents in this particular host's directory are considered by the entity providing the annotation to be “professional reviews” of digital cameras. In one embodiment, the URL pattern can include wildcards as well as regular expressions. In a typical embodiment, the annotation database 104 includes thousands, even millions of such annotations. There is a many-to-many relationship between labels and URL patterns, in that a given label may be applied to any document matching multiple different URL patterns, and a given URL pattern may be associated with many different labels.
From the received annotation information, the system stores the annotations in the annotation database 140. The annotation database 140 includes an inverted index (or equivalent) that indexes each annotation to whatever URL patterns have been defined for it (whether by the same or different entities). In some cases, an entity may simply create a link from its site to a particular item of web content (e.g., a document) and provide a label 107 as the anchor text of the link. In this case, a crawler extracts the label from the link and generates the appropriate annotation by using the label and the associated URL of the linked content. Each annotation is associated with the entity that provided the annotation. The trust rank TR for the associated entity can thereby be accessed as follows. First, given a document identifier (e.g., a URL), the annotations 106 associated with the document can be determined from the annotation database 140. For each identified annotation, the entity that provided the annotation can be determined, and that entity's trust rank TR retrieved from the trust database 190.
The search engine system 100 periodically updates the annotation database 140 by crawling web sites and extracting annotations therein, as well as receiving annotation files via the annotation interface 160.
Generated Trust Ranked Search Results
FIG. 4 presents a flowchart illustrating one process of generating trust ranked search results in accordance with an embodiment of the present invention; other variations of the steps or procedures of this process can be readily developed to achieve the same results.
The front end server 110 receives 402 a query from a user. The query includes at least one query term, and optionally includes one or more labels, as described above. The query is provided to the search engine 180.
For example, a query may be
digital camera more: professional review
where the terms “digital camera” are the query terms, and the token “label:” indicates that the following term is a label of interest. Thus, in this example, the user is searching for documents that have been labeled (by one or more individuals, such as other users, experts, etc.) as being professional reviews of digital cameras.
Another example would be the query
abortion label:statistics label:pro-choice.
In this example, the user would be searching for documents that relevant to abortion and had been labeled as either statistics or pro-choice related. The particular syntax used to indicate labels (as differentiated from query terms) is not limited to the use of “label:” and other tokens and formats can be used as well. For example, the query can take the form such as
    • abortion label:(statistics pro-choice)
      where the terms in the parenthetical label are the labels of interest (as disjuncts).
The front end server 110 (or other module) includes a parser that performs the initial parsing of the query into query term and labels.
The search engine 180 retrieves 404 a set of documents that are relevant to the query term(s). The search engine 180 can use any type of information retrieval model for selecting and scoring relevant documents. One suitable information retrieval model is a link-based model, such as PageRank which is described in U.S. Pat. No. 6,285,999, and incorporated by reference herein.
The retrieved documents are ranked ordered in terms of an underlying (base) information retrieval score used by the search engine 180. For each document, the labels associated with the document are determined 406 from the annotation database 140 by matching the URL of the document to the URL patterns in the annotations. Thus, if the URL of a particular search result document matches the three different annotations for three different labels, each of these labels will be determined as associated with the document.
The query labels are then matched against the retrieved annotation labels, to identify which query labels apply to the document; this can be done by matching the query labels to the document's annotated labels. For each such annotation label matching a query label, the trust rank TR of entity that provided the annotation label is retrieved 408. A document may have been labeled with the same label by several different entities. For example, three different entities may have labeled a particular review of digital camera with the label “professional review” while a fourth labeled the same review with the label “negative review.” If the query label included “professional review” then only the first three annotation labels would be deemed matched. It should be noted that steps 406 and 408 can performed in batch against all the search results, or iteratively for each search result, depending the desired programming model.
The trust ranks of the entities associated with the matching labels are aggregated 410 to create an aggregated trust rank. Thus, in this example, the trust ranks of the three different experts would be aggregated into a single trust rank associated with the particular label “professional review” for the review document. This aggregation is performed for each label associated with the document.
For example, assume that a particular document, such as a review of a Casio digital camera, has the following annotations associated therewith:
Label Entity Trust Rank
“Professional Review” Phil Photo 8
“Professional Review” Earl Expert 6
“Professional Review” Chris Click 7
“Digital SLR” Phil Photo 8
“Digital SLR” Eddy Shooter 2
“Best buy” Betsy Buyer 3
The trust ranks of the entities could be aggregated with respect to these labels. The trust ranks can be aggregated using a weighting function. Aggregation using a linear weighting function would be:
Linear Aggregated
Label Trust Rank
“Professional Review” 21
“Digital SLR” 10
“Best buy” 3
This approach basically applies a fixed trust weight (e.g., 1) to the trust ranks prior to summing them.
The trust ranks TR can be aggregated in a number of different ways other than linear weighting. One variation is an asymptotic weighting function. For example, one approach here would be to sum the log of the trust ranks to form the aggregated trust rank. Another variation of the weighting function is to apply a weight that decays with the increasing number of instances of a particular label. For example, the trust ranks can be ordered by age of the associated annotation, and weighted to decrease (alternatively increase) the weight applied to the trust rank for the oldest annotation. Another way in which the trust ranks can be aggregated is by use of a sigmoid weighting function.
The foregoing are various examples of different ways that trust ranks can be aggregated; other way of combining the trust ranks of individual entities can be readily implemented, and these too are deemed to come within the concept of aggregation.
Once the trust ranks have been aggregated for each label that matches a query label, the aggregated trust ranks are applied 412 to the document's base information retrieval score. The aggregated trust ranks can be combined into a trust factor, using any type of aggregation function. For example, trust factor can be the sum total of the aggregated trust ranks. Second, the aggregated trust ranks can be aggregated asymptotically or via a sigmoid function to form the trust factor.
However the trust factor is determined from the aggregated trust ranks, it is then applied to the base information retrieval score, for example by multiplying with the trust factor. Other ways of applying the trust factor to the base information retrieval score as also possible. The resulting score is said to be trust-adjusted information retrieval score.
After (or as) the trust-adjusted information retrieval scores have been determined, the search engine 180 reranks 414 the search results by these adjusted scores. The result set is returned to the front end server 110, which in turn passes it to the client 103. The user can then review the search results.
The search engine system 100 is adapted to receive queries that do not include labels, and still provide trust based ranking. In this case, if certain annotations are applicable to a search result document, then the trust rank for the entities providing these annotations is retrieved, aggregated, and applied to the base information retrieval score of the document as well, and the documents are reranked accordingly.
In discussion of the various embodiments, examples have been discussed using search queries including a single query term and a single label, as well as examples referencing individual documents. It is understood that the embodiments and invention are not so limited, and should be generally understood as equally operable with queries including a plurality of query terms and a plurality of labels. Thus any reference herein, including in the claims below, to various words in the singular noun form, such as query term, label, annotation, document, and so forth, are not intended to be limited to the singular, but should be read as including at least one or a plurality thereof, unless such a construction is expressly indicated as not intended or appropriate for the circumstances.
The present invention has been described in particular detail with respect to one possible embodiment. Those of skill in the art will appreciate that the invention may be practiced in other embodiments. First, the particular naming of the components, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. Further, the system may be implemented via a combination of hardware and software, as described, or entirely in hardware elements. Also, the particular division of functionality between the various system components described herein is merely exemplary, and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.
Some portions of above description present the features of the present invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described functionally or logically, are understood to be implemented by computer programs. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules or by functional names, without loss of generality.
Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as “calculating” or “determining” or “identifying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Certain aspects of the present invention have been described using commands, mnemonics, tokens, formats, syntax, and other programming conventions. The particular selection of the names, formats, syntax, and the like are merely illustrative, and not limiting. Those of skill in the art can readily construct alternative names, formats, syntax rules, and so forth for defining context files and programming the operations a programmable search engine via context processing.
Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.
The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent to those of skill in the art, along with equivalent variations. In addition, the present invention is not described with reference to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references to specific languages are provided for disclosure of enablement and best mode of the present invention.
Finally, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

Claims (14)

1. A method performed by a data processing apparatus, the method comprising:
receiving a search query, the search query comprising a query term and a query label term, the query label term being a categorical identifier;
identifying one or more resources in a search result set responsive to the search query, wherein each of the one or more resources has an associated matching label term that matches the query label term and an associated non-matching label term that does not match the query label term, each associated label term being the categorical identifier describing each resource and having been associated with the resource by a respective entity;
determining, for each of the one or more resources:
a trust rank of the entity that associated the associated matching label term with the resource, wherein the trust rank indicates whether a user trusts the associated matching label term of the resource, and
increasing a relevance score of each resource that has the associated matching label term based on the respective trust rank, the relevance score indicating a degree of relevance between the respective resource and the query term;
ranking each of the one or more resources in the search result set based on the respective relevance scores;
annotating, with a name of the respective entity that associated the matching label term with each resource, indicia identifying each resource of the ranked search result set; and
providing the indicia in a response to the search query.
2. The method of claim 1, wherein annotating, with the name of the respective entity further comprises:
annotating, with the matching label term associated with each resource, the indicia identifying each resource of the ranked search result set.
3. The method of claim 1, further comprising:
determining, for each of the one or more resources:
a second trust rank of a second entity that associated a second matching label term with the resource; and
aggregating the trust rank and the second trust rank to determine a trust factor.
4. A method executed by one or more processors causing the one or more processors to execute the steps of:
receiving a search query, the search query comprising a query term and a query label term, the query label term being a categorical identifier;
identifying one or more resources in a search result set responsive to the search query, wherein each of the one or more resources has two or more associated matching label terms that match the query label term and that each label term being the categorical identifier describing each resource and been associated with the resource by a different respective entity;
determining, for each of the one or more resources:
a trust rank of each entity that associated a matching label term with the resource, wherein the trust rank indicates whether a user trusts the associated matching label term of the resource, and
increasing a relevance score of each resource that has the two or more associated matching label terms based on the respective trust factors, the relevance score indicating a degree of relevance between the respective resource and the query term;
ranking each of the one or more resources in the search result set based on the respective relevance scores;
annotating, with a name of each entity that associated the matching label terms with each resource, indicia identifying each resource of the ranked search result set; and
providing the indicia in a response to the search query.
5. The method of claim 4, further comprising:
determining, for each of the one or more resources:
a second trust rank of a second entity that associated a second matching label term with the resource; and
aggregating the trust rank and the second trust rank to determine a trust factor.
6. The method of claim 4, wherein annotating, with the name of each entity further comprises:
annotating, with the two matching label terms associated with each resource, the indicia identifying each resource of the ranked search result set.
7. A system comprising:
one or more computers;
a computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
receiving a search query, the search query comprising a query term and a query label term, the query label term being a categorical identifier;
identifying one or more resources in a search result set responsive to the search query, wherein each of the one or more resources has an associated matching label term that matches the query label term and an associated non-matching label term that does not match the query label term, each associated label term being the categorical identifier describing each resource and having been associated with the resource by a respective entity;
determining, for each of the one or more resources:
a trust rank of the entity that associated the associated matching label term with the resource, wherein the trust rank indicates whether a user trusts the associated matching label term of the resource, and
increasing a relevance score of each resource that has the associated matching label term based on the respective trust rank, the relevance score indicating a degree of relevance between the respective resource and the query term;
ranking each of the one or more resources in the search result set based on the respective relevance scores;
annotating, with a name of the respective entity that associated the matching label term with each resource, indicia identifying each resource of the ranked search result set; and
providing the indicia in a response to the search query.
8. The system of claim 7, where in the operations further comprise:
annotating, with the matching label term associated with each resource, the indicia identifying each resource of the ranked search result set.
9. The system of claim 7, wherein the operations further comprise:
determining, for each of the one or more resources:
a second trust rank of a second entity that associated a second matching label term with the resource; and
aggregating the trust rank and the second trust rank to determine a trust factor.
10. A system comprising:
one or more computers;
a computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
receiving a search query, the search query comprising a query term and a query label term, the query label term being a categorical identifier;
identifying one or more resources in a search result set responsive to the search query, wherein each of the one or more resources has two or more associated matching label terms that match the query label term and that each label term being the categorical identifier describing each resource and been associated with the resource by a different respective entity;
determining, for each of the one or more resources:
a trust rank of each entity that associated a matching label term with the resource based on the trust information; and
increasing a relevance score of each resource that has the two or more associated matching label terms based on the respective trust ranks, the relevance score indicating a degree of relevance between the respective resource and the query term;
ranking each of the one or more resources in the search result set based on the respective relevance scores;
annotating, with a name of each entity that associated the matching label term with each resource, indicia identifying each resource of the ranked search result set; and
providing the indicia in a response to the search query.
11. The system of claim 10, further comprising:
determining, for each of the one or more resources:
a second trust rank of a second entity that associated a second matching label term with the resource; and
aggregating the trust rank and the second trust rank to determine a trust factor.
12. The system of claim 10, wherein the operations further comprise:
annotating, with the two matching label terms associated with each resource, the indicia identifying each resource of the ranked search result set.
13. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
receiving a search query, the search query comprising a query term and a query label term, the query label term being a categorical identifier;
identifying one or more resources in a search result set responsive to the search query, wherein each of the one or more resources has an associated matching label term that matches the query label term and an associated non-matching label term that does not match the query label term, each associated label term being the categorical identifier describing each resource and having been associated with the resource by a respective entity;
determining, for each of the one or more resources:
a trust rank of the entity that associated the associated matching label term with the resource based on the trust information, wherein the trust rank indicates whether a user trusts the associated matching label term of the resource, and
increasing a relevance score of each resource that has the associated matching label term based on the respective trust rank, the relevance score indicating a degree of relevance between the respective resource and the query term;
ranking each of the one or more resources in the search result set based on the respective relevance scores;
annotating, with a name of the respective entity that associated the matching label term with each resource, indicia identifying each resource of the ranked search result set; and
providing the indicia in a response to the search query.
14. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
receiving a search query, the search query comprising a query term and a query label term, the query label term being a categorical identifier;
identifying one or more resources in a search result set responsive to the search query, wherein each of the one or more resources has two or more associated matching label terms that match the query label term and that each label term being the categorical identifier describing each resource and been associated with the resource by a different respective entity;
determining, for each of the one or more resources:
a trust rank of each entity that associated a matching label term with the resource, wherein the trust rank indicates whether a user trusts the associated matching label term of the resource, and
increasing a relevance score of each resource that has the two or more associated matching label terms based on the respective trust ranks, the relevance score indicating a degree of relevance between the respective resource and the query term;
ranking each of the one or more resources in the search result set based on the respective relevance scores;
annotating, with a name of each entity that associated the matching label term with each resource, indicia identifying each resource of the ranked search result set; and
providing the indicia in a response to the search query.
US11/382,449 2006-05-09 2006-05-09 Search result ranking based on trust Active 2027-01-20 US7603350B1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US11/382,449 US7603350B1 (en) 2006-05-09 2006-05-09 Search result ranking based on trust
US12/552,956 US8352467B1 (en) 2006-05-09 2009-09-02 Search result ranking based on trust
US13/614,339 US8818995B1 (en) 2006-05-09 2012-09-13 Search result ranking based on trust
US14/445,209 US10268641B1 (en) 2006-05-09 2014-07-29 Search result ranking based on trust

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US11/382,449 US7603350B1 (en) 2006-05-09 2006-05-09 Search result ranking based on trust

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US12/552,956 Continuation US8352467B1 (en) 2006-05-09 2009-09-02 Search result ranking based on trust

Publications (1)

Publication Number Publication Date
US7603350B1 true US7603350B1 (en) 2009-10-13

Family

ID=41138080

Family Applications (4)

Application Number Title Priority Date Filing Date
US11/382,449 Active 2027-01-20 US7603350B1 (en) 2006-05-09 2006-05-09 Search result ranking based on trust
US12/552,956 Active 2027-02-05 US8352467B1 (en) 2006-05-09 2009-09-02 Search result ranking based on trust
US13/614,339 Expired - Fee Related US8818995B1 (en) 2006-05-09 2012-09-13 Search result ranking based on trust
US14/445,209 Expired - Fee Related US10268641B1 (en) 2006-05-09 2014-07-29 Search result ranking based on trust

Family Applications After (3)

Application Number Title Priority Date Filing Date
US12/552,956 Active 2027-02-05 US8352467B1 (en) 2006-05-09 2009-09-02 Search result ranking based on trust
US13/614,339 Expired - Fee Related US8818995B1 (en) 2006-05-09 2012-09-13 Search result ranking based on trust
US14/445,209 Expired - Fee Related US10268641B1 (en) 2006-05-09 2014-07-29 Search result ranking based on trust

Country Status (1)

Country Link
US (4) US7603350B1 (en)

Cited By (90)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070208613A1 (en) * 2006-02-09 2007-09-06 Alejandro Backer Reputation system for web pages and online entities
US20080065646A1 (en) * 2006-09-08 2008-03-13 Microsoft Corporation Enabling access to aggregated software security information
US20080172630A1 (en) * 2006-09-08 2008-07-17 Microsoft Corporation Graphical representation of aggregated data
US20080276177A1 (en) * 2007-05-03 2008-11-06 Microsoft Corporation Tag-sharing and tag-sharing application program interface
US20080281769A1 (en) * 2007-05-10 2008-11-13 Jason Hibbets Systems and methods for community tagging
US20080301091A1 (en) * 2007-05-31 2008-12-04 Hibbets Jason S Systems and methods for improved forums
US20080301115A1 (en) * 2007-05-31 2008-12-04 Mattox John R Systems and methods for directed forums
US20090007272A1 (en) * 2007-06-28 2009-01-01 Microsoft Corporation Identifying data associated with security issue attributes
US20090007271A1 (en) * 2007-06-28 2009-01-01 Microsoft Corporation Identifying attributes of aggregated data
US20090063408A1 (en) * 2007-08-28 2009-03-05 International Business Machines Corporation Managing user ratings in a web services environment
US20090063386A1 (en) * 2007-08-27 2009-03-05 Hibbets Jason S Systems and methods for linking an issue with an entry in a knowledgebase
US20100082685A1 (en) * 2008-09-23 2010-04-01 Barr Hoisman Hila System and method for evaluation and presenting authorization rights in an organization
US20100153354A1 (en) * 2008-12-17 2010-06-17 International Business Machines Corporation Web Search Among Rich Media Objects
US20100169265A1 (en) * 2008-12-30 2010-07-01 Herbert Willi Artur Ristock Scoring Persons and Files for Trust in Digital Communication
US20100169430A1 (en) * 2008-12-30 2010-07-01 Herbert Willi Artur Ristock Trust -Interactive Communication Applications
US20100325105A1 (en) * 2009-06-19 2010-12-23 Alibaba Group Holding Limited Generating ranked search results using linear and nonlinear ranking models
US20110029613A1 (en) * 2007-08-03 2011-02-03 John Norman Hedditch Evaluation of an attribute of an information object
WO2011050495A1 (en) * 2009-10-29 2011-05-05 Google Inc. Ranking user generated web content
US7966319B2 (en) 2007-06-07 2011-06-21 Red Hat, Inc. Systems and methods for a rating system
US20110161317A1 (en) * 2009-12-25 2011-06-30 Fujifilm Corporation Document retrieval system
US20120005183A1 (en) * 2010-06-30 2012-01-05 Emergency24, Inc. System and method for aggregating and interactive ranking of search engine results
WO2012091541A1 (en) * 2010-12-28 2012-07-05 Mimos Berhad A semantic web constructor system and a method thereof
US20120296918A1 (en) * 2011-05-18 2012-11-22 Microsoft Corporation Credibility Information in Returned Web Results
US20120323888A1 (en) * 2011-06-17 2012-12-20 Osann Jr Robert Automatic Webpage Characterization and Search Results Annotation
US8386335B1 (en) 2011-04-04 2013-02-26 Google Inc. Cross-referencing comments
US20130166552A1 (en) * 2011-12-21 2013-06-27 Guy Rozenwald Systems and methods for merging source records in accordance with survivorship rules
US20130297464A1 (en) * 2012-05-01 2013-11-07 Shopsavvy Inc. System, Method, and Computer-Readable Storage Medium For Identifying A Product
US8601059B2 (en) * 2010-05-13 2013-12-03 International Business Machines Corporation Sharing form training result utilizing a social network
US8620136B1 (en) 2011-04-30 2013-12-31 Cisco Technology, Inc. System and method for media intelligent recording in a network environment
US8645394B1 (en) 2011-04-15 2014-02-04 Google Inc. Ranking clusters and resources in a cluster
US8645393B1 (en) 2011-04-15 2014-02-04 Google Inc. Ranking clusters and resources in a cluster
US8667169B2 (en) 2010-12-17 2014-03-04 Cisco Technology, Inc. System and method for providing argument maps based on activity in a network environment
US20140114959A1 (en) * 2010-07-31 2014-04-24 Viralheat, Inc. Discerning human intent based on user-generated metadata
US20140129539A1 (en) * 2007-11-14 2014-05-08 Paul Vincent Hayes System and method for personalized search
US20140201271A1 (en) * 2013-01-13 2014-07-17 Qualcomm Incorporated User generated rating by machine classification of entity
US20140214818A1 (en) * 2011-09-08 2014-07-31 Nokia Corporation Method and apparatus for hybrid social search model
US8831403B2 (en) 2012-02-01 2014-09-09 Cisco Technology, Inc. System and method for creating customized on-demand video reports in a network environment
US20140280011A1 (en) * 2013-03-15 2014-09-18 Google Inc. Predicting Site Quality
WO2014146230A1 (en) * 2013-03-18 2014-09-25 Nokia Corporation Method and apparatus for querying resources through search field
WO2014100790A3 (en) * 2012-12-21 2014-10-09 Microsoft Corporation Search results through interest circles
US8886797B2 (en) 2011-07-14 2014-11-11 Cisco Technology, Inc. System and method for deriving user expertise based on data propagating in a network environment
US8909624B2 (en) 2011-05-31 2014-12-09 Cisco Technology, Inc. System and method for evaluating results of a search query in a network environment
US8935274B1 (en) * 2010-05-12 2015-01-13 Cisco Technology, Inc System and method for deriving user expertise based on data propagating in a network environment
US8972394B1 (en) 2009-07-20 2015-03-03 Google Inc. Generating a related set of documents for an initial set of documents
US8972391B1 (en) 2009-10-02 2015-03-03 Google Inc. Recent interest based relevance scoring
US8990083B1 (en) 2009-09-30 2015-03-24 Cisco Technology, Inc. System and method for generating personal vocabulary from network data
US9002867B1 (en) 2010-12-30 2015-04-07 Google Inc. Modifying ranking data based on document changes
US9009146B1 (en) 2009-04-08 2015-04-14 Google Inc. Ranking search results based on similar queries
US9031929B1 (en) 2012-01-05 2015-05-12 Google Inc. Site quality score
US20150193495A1 (en) * 2014-01-03 2015-07-09 Yahoo! Inc. Systems and methods for quote extraction
US20150195295A1 (en) * 2009-07-06 2015-07-09 Google Inc. Monitoring of Negative Feedback Systems
US20150193440A1 (en) * 2014-01-03 2015-07-09 Yahoo! Inc. Systems and methods for content processing
US9092510B1 (en) 2007-04-30 2015-07-28 Google Inc. Modifying search result ranking based on a temporal element of user feedback
US20150213521A1 (en) * 2014-01-30 2015-07-30 The Toronto-Dominion Bank Adaptive social media scoring model with reviewer influence alignment
US20150261763A1 (en) * 2007-11-30 2015-09-17 Ebay Inc. Using reputation measures to improve search relevance
TWI501096B (en) * 2009-10-30 2015-09-21 Google Inc Ranking user generated web content
US9141772B2 (en) 2007-12-31 2015-09-22 Genesys Telecommunications Laboratories, Inc. Trust conferencing apparatus and methods in digital communication
US20150278210A1 (en) * 2014-03-28 2015-10-01 Microsoft Corporation Building user trust in profile creation and recommendations based on managed interaction with user
US20150302425A1 (en) * 2014-04-22 2015-10-22 International Business Machines Corporation Assigning priority levels to citizen sensor reports
US20150310111A1 (en) * 2006-05-19 2015-10-29 Jorn Lyseggen Source search engine
US20150310022A1 (en) * 2011-07-11 2015-10-29 International Business Machines Corporation Searching documentation across interconnected nodes in a distributed network
US20150332169A1 (en) * 2014-05-15 2015-11-19 International Business Machines Corporation Introducing user trustworthiness in implicit feedback based search result ranking
US9195944B1 (en) 2012-02-24 2015-11-24 Google Inc. Scoring site quality
US9201965B1 (en) 2009-09-30 2015-12-01 Cisco Technology, Inc. System and method for providing speech recognition using personal vocabulary in a network environment
US9235627B1 (en) 2006-11-02 2016-01-12 Google Inc. Modifying search result ranking based on implicit user feedback
US9378065B2 (en) 2013-03-15 2016-06-28 Advanced Elemental Technologies, Inc. Purposeful computing
USD760791S1 (en) 2014-01-03 2016-07-05 Yahoo! Inc. Animated graphical user interface for a display screen or portion thereof
USD760792S1 (en) 2014-01-03 2016-07-05 Yahoo! Inc. Animated graphical user interface for a display screen or portion thereof
USD761833S1 (en) 2014-09-11 2016-07-19 Yahoo! Inc. Display screen with graphical user interface of a menu for a news digest
US9418104B1 (en) 2009-08-31 2016-08-16 Google Inc. Refining search results
US9465795B2 (en) 2010-12-17 2016-10-11 Cisco Technology, Inc. System and method for providing feeds based on activity in a network environment
USD775183S1 (en) 2014-01-03 2016-12-27 Yahoo! Inc. Display screen with transitional graphical user interface for a content digest
US20170032044A1 (en) * 2006-11-14 2017-02-02 Paul Vincent Hayes System and Method for Personalized Search While Maintaining Searcher Privacy
US9623119B1 (en) * 2010-06-29 2017-04-18 Google Inc. Accentuating search results
US20170140059A1 (en) * 2011-12-30 2017-05-18 Microsoft Technology Licensing, Llc Knowledge-based entity detection and disambiguation
US9684697B1 (en) 2012-01-05 2017-06-20 Google Inc. Ranking search results
US9721086B2 (en) 2013-03-15 2017-08-01 Advanced Elemental Technologies, Inc. Methods and systems for secure and reliable identity-based computing
US9742836B2 (en) 2014-01-03 2017-08-22 Yahoo Holdings, Inc. Systems and methods for content delivery
US9904579B2 (en) 2013-03-15 2018-02-27 Advanced Elemental Technologies, Inc. Methods and systems for purposeful computing
US20180113865A1 (en) * 2016-10-26 2018-04-26 Google Inc. Search and retrieval of structured information cards
US9971756B2 (en) 2014-01-03 2018-05-15 Oath Inc. Systems and methods for delivering task-oriented content
US10075384B2 (en) 2013-03-15 2018-09-11 Advanced Elemental Technologies, Inc. Purposeful computing
US10162891B2 (en) 2010-11-29 2018-12-25 Vocus Nm Llc Determining demographics based on user interaction
US10268641B1 (en) 2006-05-09 2019-04-23 Google Llc Search result ranking based on trust
US10296167B2 (en) 2014-01-03 2019-05-21 Oath Inc. Systems and methods for displaying an expanding menu via a user interface
US10839421B2 (en) 2007-07-09 2020-11-17 Groupon, Inc. Implicitly associating metadata using user behavior
US11314746B2 (en) 2013-03-15 2022-04-26 Cision Us Inc. Processing unstructured data streams using continuous queries
US11361076B2 (en) * 2018-10-26 2022-06-14 ThreatWatch Inc. Vulnerability-detection crawler
EP4018337A4 (en) * 2019-08-19 2023-08-23 Yext, Inc. Multiple provider search response verification and integration
US11916960B2 (en) * 2015-12-30 2024-02-27 International Business Machines Corporation Curtailing search engines from obtaining and controlling information

Families Citing this family (43)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8468168B2 (en) 2007-07-25 2013-06-18 Xobni Corporation Display of profile information based on implicit actions
US8990323B2 (en) 2009-07-08 2015-03-24 Yahoo! Inc. Defining a social network model implied by communications data
US9721228B2 (en) 2009-07-08 2017-08-01 Yahoo! Inc. Locally hosting a social network using social data stored on a user's computer
US7930430B2 (en) 2009-07-08 2011-04-19 Xobni Corporation Systems and methods to provide assistance during address input
US20110078188A1 (en) * 2009-09-28 2011-03-31 Microsoft Corporation Mining and Conveying Social Relationships
US9514466B2 (en) 2009-11-16 2016-12-06 Yahoo! Inc. Collecting and presenting data including links from communications sent to or from a user
US9760866B2 (en) 2009-12-15 2017-09-12 Yahoo Holdings, Inc. Systems and methods to provide server side profile information
US9020938B2 (en) 2010-02-03 2015-04-28 Yahoo! Inc. Providing profile information using servers
US8972257B2 (en) 2010-06-02 2015-03-03 Yahoo! Inc. Systems and methods to present voice message information to a user of a computing device
US8620935B2 (en) 2011-06-24 2013-12-31 Yahoo! Inc. Personalizing an online service based on data collected for a user of a computing device
US20120084226A1 (en) * 2010-10-05 2012-04-05 Yahoo! Inc. Measuring or estimating user credibility
US9529822B2 (en) 2010-10-05 2016-12-27 Yahoo! Inc. Media or content tagging determined by user credibility signals
US10078819B2 (en) 2011-06-21 2018-09-18 Oath Inc. Presenting favorite contacts information to a user of a computing device
US9747583B2 (en) 2011-06-30 2017-08-29 Yahoo Holdings, Inc. Presenting entity profile information to a user of a computing device
US11914674B2 (en) 2011-09-24 2024-02-27 Z Advanced Computing, Inc. System and method for extremely efficient image and pattern recognition and artificial intelligence platform
US8873813B2 (en) 2012-09-17 2014-10-28 Z Advanced Computing, Inc. Application of Z-webs and Z-factors to analytics, search engine, learning, recognition, natural language, and other utilities
US11195057B2 (en) 2014-03-18 2021-12-07 Z Advanced Computing, Inc. System and method for extremely efficient image and pattern recognition and artificial intelligence platform
US9916538B2 (en) 2012-09-15 2018-03-13 Z Advanced Computing, Inc. Method and system for feature detection
US11074495B2 (en) 2013-02-28 2021-07-27 Z Advanced Computing, Inc. (Zac) System and method for extremely efficient image and pattern recognition and artificial intelligence platform
US9311650B2 (en) 2012-02-22 2016-04-12 Alibaba Group Holding Limited Determining search result rankings based on trust level values associated with sellers
US10977285B2 (en) 2012-03-28 2021-04-13 Verizon Media Inc. Using observations of a person to determine if data corresponds to the person
US11023520B1 (en) 2012-06-01 2021-06-01 Google Llc Background audio identification for query disambiguation
CN103793388B (en) 2012-10-29 2017-08-25 阿里巴巴集团控股有限公司 The sort method and device of search result
US10013672B2 (en) 2012-11-02 2018-07-03 Oath Inc. Address extraction from a communication
US10192200B2 (en) 2012-12-04 2019-01-29 Oath Inc. Classifying a portion of user contact data into local contacts
US9002837B2 (en) * 2013-03-15 2015-04-07 Ipar, Llc Systems and methods for providing expert thread search results
US11263221B2 (en) 2013-05-29 2022-03-01 Microsoft Technology Licensing, Llc Search result contexts for application launch
US10430418B2 (en) 2013-05-29 2019-10-01 Microsoft Technology Licensing, Llc Context-based actions from a source application
JP6253168B2 (en) * 2013-08-30 2017-12-27 マカフィー, エルエルシー Improved tamper resistance of aggregated data
US9767158B1 (en) * 2013-10-09 2017-09-19 Google Inc. Ranking algorithms using unique bucketing identifiers
US11238114B2 (en) 2013-10-22 2022-02-01 Steven Michael VITTORIO Educational content search and results
US11222084B2 (en) 2013-10-22 2022-01-11 Steven Michael VITTORIO Content search and results
US20150317314A1 (en) * 2014-04-30 2015-11-05 Linkedln Corporation Content search vertical
US9996768B2 (en) * 2014-11-19 2018-06-12 Adobe Systems Incorporated Neural network patch aggregation and statistics
US10187399B2 (en) * 2015-04-07 2019-01-22 Passport Health Communications, Inc. Enriched system for suspicious interaction record detection
WO2016168811A1 (en) 2015-04-17 2016-10-20 Vittorio Steven Michael Content search and results
CN105608139B (en) * 2015-12-16 2019-04-09 携程计算机技术(上海)有限公司 Data matching system and method
US10846387B2 (en) 2017-07-12 2020-11-24 At&T Intellectual Property I, L.P. Managing access based on activities of entities
US11226972B2 (en) 2019-02-19 2022-01-18 International Business Machines Corporation Ranking collections of document passages associated with an entity name by relevance to a query
US11132358B2 (en) 2019-02-19 2021-09-28 International Business Machines Corporation Candidate name generation
US10936819B2 (en) * 2019-02-19 2021-03-02 International Business Machines Corporation Query-directed discovery and alignment of collections of document passages for improving named entity disambiguation precision
EP4118539A4 (en) * 2020-03-10 2024-03-06 Outreach Corp Automatically recognizing and surfacing important moments in multi-party conversations
CN113051487B (en) * 2021-04-20 2022-08-02 安徽师范大学 Travel group recommendation method based on user trust and social influence

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6336117B1 (en) * 1999-04-30 2002-01-01 International Business Machines Corporation Content-indexing search system and method providing search results consistent with content filtering and blocking policies implemented in a blocking engine
US6360215B1 (en) * 1998-11-03 2002-03-19 Inktomi Corporation Method and apparatus for retrieving documents based on information other than document content
US6421675B1 (en) * 1998-03-16 2002-07-16 S. L. I. Systems, Inc. Search engine
US6636854B2 (en) * 2000-12-07 2003-10-21 International Business Machines Corporation Method and system for augmenting web-indexed search engine results with peer-to-peer search results
US7031961B2 (en) * 1999-05-05 2006-04-18 Google, Inc. System and method for searching and recommending objects from a categorically organized information repository
US20060200487A1 (en) * 2004-10-29 2006-09-07 The Go Daddy Group, Inc. Domain name related reputation and secure certificates
US20060294086A1 (en) 2005-06-28 2006-12-28 Yahoo! Inc. Realtime indexing and search in large, rapidly changing document collections

Family Cites Families (93)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AUPQ131399A0 (en) 1999-06-30 1999-07-22 Silverbrook Research Pty Ltd A method and apparatus (NPAGE02)
US5825880A (en) 1994-01-13 1998-10-20 Sudia; Frank W. Multi-step digital signature method and system
EP0771499B1 (en) 1994-07-19 2005-09-28 Certco, Llc Method for securely using digital signatures in a commercial cryptographic system
US5544255A (en) 1994-08-31 1996-08-06 Peripheral Vision Limited Method and system for the capture, storage, transport and authentication of handwritten signatures
US7162635B2 (en) 1995-01-17 2007-01-09 Eoriginal, Inc. System and method for electronic transmission, storage, and retrieval of authenticated electronic original documents
WO1996039765A1 (en) 1995-06-05 1996-12-12 Certco Llc Multi-step digital signature method and system
US7047241B1 (en) 1995-10-13 2006-05-16 Digimarc Corporation System and methods for managing digital creative works
US5892824A (en) 1996-01-12 1999-04-06 International Verifact Inc. Signature capture/verification systems and methods
US7543018B2 (en) 1996-04-11 2009-06-02 Aol Llc, A Delaware Limited Liability Company Caching signatures
US6285999B1 (en) 1997-01-10 2001-09-04 The Board Of Trustees Of The Leland Stanford Junior University Method for node ranking in a linked database
US6742030B1 (en) 1997-11-24 2004-05-25 International Business Machines Corporation Method to keep a persistent trace of weblink use per user
US6356924B2 (en) 1997-11-26 2002-03-12 International Business Machines Corporation Configurable disablement of display objects in a browser
US6167518A (en) 1998-07-28 2000-12-26 Commercial Electronics, Llc Digital signature providing non-repudiation based on biological indicia
US6334131B2 (en) 1998-08-29 2001-12-25 International Business Machines Corporation Method for cataloging, filtering, and relevance ranking frame-based hierarchical information structures
US6260064B1 (en) 1999-01-08 2001-07-10 Paul J. Kurzrok Web site with automatic rating system
US6871191B1 (en) 2000-01-24 2005-03-22 Sam E. Kinney, Jr. Method and system for partial quantity evaluated rank bidding in online auctions
AU4460600A (en) 1999-04-13 2000-11-14 Ilumin Corporation Collaborative creation, editing, reviewing, and signing of electronic documents
US6760752B1 (en) 1999-06-28 2004-07-06 Zix Corporation Secure transmission system
US7886008B2 (en) 1999-07-28 2011-02-08 Rpost International Limited System and method for verifying delivery and integrity of electronic messages
US7433832B1 (en) * 1999-11-19 2008-10-07 Amazon.Com, Inc. Methods and systems for distributing information within a dynamically defined community
US7013323B1 (en) 2000-05-23 2006-03-14 Cyveillance, Inc. System and method for developing and interpreting e-commerce metrics by utilizing a list of rules wherein each rule contain at least one of entity-specific criteria
US7069443B2 (en) 2000-06-06 2006-06-27 Ingeo Systems, Inc. Creating and verifying electronic documents
US6891953B1 (en) 2000-06-27 2005-05-10 Microsoft Corporation Method and system for binding enhanced software features to a persona
US6778986B1 (en) 2000-07-31 2004-08-17 Eliyon Technologies Corporation Computer method and apparatus for determining site type of a web site
US6978369B2 (en) 2000-08-04 2005-12-20 First Data Corporation Person-centric account-based digital signature system
US7185065B1 (en) 2000-10-11 2007-02-27 Buzzmetrics Ltd System and method for scoring electronic messages
US20020062368A1 (en) 2000-10-11 2002-05-23 David Holtzman System and method for establishing and evaluating cross community identities in electronic forums
US7466663B2 (en) 2000-10-26 2008-12-16 Inrotis Technology, Limited Method and apparatus for identifying components of a network having high importance for network integrity
GB0028474D0 (en) 2000-11-22 2001-01-10 Raekanet Ltd Improved computer network architecture and associated method and system
US6961728B2 (en) 2000-11-28 2005-11-01 Centerboard, Inc. System and methods for highly distributed wide-area data management of a network of data sources through a database interface
WO2002057949A1 (en) 2001-01-22 2002-07-25 Contrieve, Inc. Systems and methods for managing and promoting network content
US7437429B2 (en) 2001-02-13 2008-10-14 Microsoft Corporation System and method for providing transparent access to distributed authoring and versioning files including encrypted files
US20020124172A1 (en) 2001-03-05 2002-09-05 Brian Manahan Method and apparatus for signing and validating web pages
US20040205076A1 (en) 2001-03-06 2004-10-14 International Business Machines Corporation System and method to automate the management of hypertext link information in a Web site
US20030051142A1 (en) 2001-05-16 2003-03-13 Hidalgo Lluis Mora Firewalls for providing security in HTTP networks and applications
US6947921B2 (en) 2001-07-03 2005-09-20 Eastman Kodak Company Method and system for capturing memories of deceased individuals
US6892201B2 (en) 2001-09-05 2005-05-10 International Business Machines Corporation Apparatus and method for providing access rights information in a portion of a file
GB0205751D0 (en) 2002-03-12 2002-04-24 James Barry E Improvements relating to memory devices
KR20040101312A (en) 2002-03-14 2004-12-02 콘텐트가드 홀딩즈 인코포레이티드 System and method for exprssing usage rights using modulated signals
US7512649B2 (en) 2002-03-22 2009-03-31 Sun Microsytems, Inc. Distributed identities
US20030221166A1 (en) 2002-05-17 2003-11-27 Xerox Corporation Systems and methods for authoritativeness grading, estimation and sorting of documents in large heterogeneous document collections
US7039631B1 (en) 2002-05-24 2006-05-02 Microsoft Corporation System and method for providing search results with configurable scoring formula
US6892198B2 (en) 2002-06-14 2005-05-10 Entopia, Inc. System and method for personalized information retrieval based on user expertise
US7246306B2 (en) 2002-06-21 2007-07-17 Microsoft Corporation Web information presentation structure for web page authoring
AU2003258037B2 (en) 2002-08-05 2009-11-26 Nokia Corporation Desktop client interaction with a geographic text search system
US8090717B1 (en) 2002-09-20 2012-01-03 Google Inc. Methods and apparatus for ranking documents
US7340503B2 (en) 2003-03-21 2008-03-04 Vocel, Inc. Interactive messaging system
US7216123B2 (en) 2003-03-28 2007-05-08 Board Of Trustees Of The Leland Stanford Junior University Methods for ranking nodes in large directed graphs
US20070174471A1 (en) 2003-04-30 2007-07-26 Cedric Van Rossum Secure, continous, proxy-optimized, device-to-device data download reception system and method of use
DE102004026083A1 (en) 2003-05-25 2005-01-20 Wittkötter, Erland, Dr. Server based linking of information e.g. for finding information on internet, has functional unit having content signature unit, to form document section and or file content which are embedded in electronic document.
US7831693B2 (en) 2003-08-18 2010-11-09 Oracle America, Inc. Structured methodology and design patterns for web services
US7822631B1 (en) * 2003-08-22 2010-10-26 Amazon Technologies, Inc. Assessing content based on assessed trust in users
US8001156B2 (en) 2003-08-29 2011-08-16 Cybertrust Ireland Limited Processing XML node sets
US7577655B2 (en) 2003-09-16 2009-08-18 Google Inc. Systems and methods for improving the ranking of news articles
US20060184452A1 (en) 2003-10-14 2006-08-17 Maccord Mason Pllc, Electronic document management system
US7281005B2 (en) 2003-10-20 2007-10-09 Telenor Asa Backward and forward non-normalized link weight analysis method, system, and computer program product
US7805518B1 (en) * 2003-11-14 2010-09-28 The Board Of Trustees Of The Leland Stanford Junior University Method and system for reputation management in peer-to-peer networks
US7424669B2 (en) 2004-01-16 2008-09-09 International Business Machines Corporation Automatic bibliographical information within electronic documents
US7707039B2 (en) 2004-02-15 2010-04-27 Exbiblio B.V. Automatic modification of web pages
US7822690B2 (en) 2004-02-10 2010-10-26 Paul Rakowicz Paperless process for mortgage closings and other applications
US8788492B2 (en) * 2004-03-15 2014-07-22 Yahoo!, Inc. Search system and methods with integration of user annotations from a trust network
US20060031235A1 (en) 2004-04-07 2006-02-09 Xoopit, Inc. Expression and time-based data creation and creator-controlled organizations
WO2007001247A2 (en) 2004-06-02 2007-01-04 Yahoo! Inc. Content-management system for user behavior targeting
US8494900B2 (en) 2004-06-30 2013-07-23 Google Inc. Adjusting ad costs using document performance or document collection performance
WO2006011819A1 (en) * 2004-07-30 2006-02-02 Eurekster, Inc. Adaptive search engine
US20060036599A1 (en) 2004-08-09 2006-02-16 Glaser Howard J Apparatus, system, and method for identifying the content representation value of a set of terms
US7590589B2 (en) 2004-09-10 2009-09-15 Hoffberg Steven M Game theoretic prioritization scheme for mobile ad hoc networks permitting hierarchal deference
US20060168057A1 (en) 2004-10-06 2006-07-27 Habeas, Inc. Method and system for enhanced electronic mail processing
US8930400B2 (en) 2004-11-22 2015-01-06 Hewlett-Packard Development Company, L. P. System and method for discovering knowledge communities
US7370036B2 (en) 2005-01-28 2008-05-06 Oracle International Corporation Facilitating identification of entire web pages when each web page is rendered from multiple portions and interest is expressed based on content of the portions
US7769724B2 (en) 2005-01-31 2010-08-03 Xerox Corporation System and method for providing S/MIME-based document distribution via electronic mail mechanisms
US20060212931A1 (en) * 2005-03-02 2006-09-21 Markmonitor, Inc. Trust evaluation systems and methods
KR101374651B1 (en) 2005-03-18 2014-03-17 써치 엔진 테크놀로지스, 엘엘씨 Search engine that applies feedback from users to improve search results
US7917761B2 (en) 2005-03-21 2011-03-29 Microsoft Corporation Digitally signing an electronic document with a user-entered signature image
US7577650B2 (en) 2005-04-13 2009-08-18 Microsoft Corporation Method and system for ranking objects of different object types
US8280882B2 (en) 2005-04-21 2012-10-02 Case Western Reserve University Automatic expert identification, ranking and literature search based on authorship in large document collections
US7865399B2 (en) 2005-04-22 2011-01-04 Google Inc. Distributed electronic commerce system with centralized point of purchase
US7822620B2 (en) 2005-05-03 2010-10-26 Mcafee, Inc. Determining website reputations using automatic testing
US8438499B2 (en) 2005-05-03 2013-05-07 Mcafee, Inc. Indicating website reputations during user interactions
US20060253582A1 (en) * 2005-05-03 2006-11-09 Dixon Christopher J Indicating website reputations within search results
US7854007B2 (en) 2005-05-05 2010-12-14 Ironport Systems, Inc. Identifying threats in electronic messages
US20060282396A1 (en) 2005-06-09 2006-12-14 Civil Foundation, Llc Multi-jurisdictional electronic-commerce legal products, methods of production and methods of conducting business therewith
US20080005064A1 (en) 2005-06-28 2008-01-03 Yahoo! Inc. Apparatus and method for content annotation and conditional annotation retrieval in a search context
RU2008107340A (en) 2005-07-27 2009-09-10 Инджениа Текнолоджи Лимитед (Gb) RECIPE AUTHENTICATION USING SPECL STRUCTURES
EP1908028A1 (en) 2005-07-27 2008-04-09 Ingenia Technology Limited Authenticity verification
US7685198B2 (en) 2006-01-25 2010-03-23 Yahoo! Inc. Systems and methods for collaborative tag suggestions
US7603350B1 (en) 2006-05-09 2009-10-13 Google Inc. Search result ranking based on trust
JP4289379B2 (en) 2006-08-24 2009-07-01 コニカミノルタビジネステクノロジーズ株式会社 E-mail processing apparatus, e-mail processing method, and e-mail processing program
US7630972B2 (en) 2007-01-05 2009-12-08 Yahoo! Inc. Clustered search processing
US7636713B2 (en) 2007-01-31 2009-12-22 Yahoo! Inc. Using activation paths to cluster proximity query results
US8676815B2 (en) 2008-05-07 2014-03-18 City University Of Hong Kong Suffix tree similarity measure for document clustering
US8620929B2 (en) 2009-08-14 2013-12-31 Google Inc. Context based resource relevance
US8260789B2 (en) 2010-04-01 2012-09-04 Microsoft Corporation System and method for authority value obtained by defining ranking functions related to weight and confidence value

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6421675B1 (en) * 1998-03-16 2002-07-16 S. L. I. Systems, Inc. Search engine
US6360215B1 (en) * 1998-11-03 2002-03-19 Inktomi Corporation Method and apparatus for retrieving documents based on information other than document content
US6336117B1 (en) * 1999-04-30 2002-01-01 International Business Machines Corporation Content-indexing search system and method providing search results consistent with content filtering and blocking policies implemented in a blocking engine
US7031961B2 (en) * 1999-05-05 2006-04-18 Google, Inc. System and method for searching and recommending objects from a categorically organized information repository
US6636854B2 (en) * 2000-12-07 2003-10-21 International Business Machines Corporation Method and system for augmenting web-indexed search engine results with peer-to-peer search results
US20060200487A1 (en) * 2004-10-29 2006-09-07 The Go Daddy Group, Inc. Domain name related reputation and secure certificates
US20060294086A1 (en) 2005-06-28 2006-12-28 Yahoo! Inc. Realtime indexing and search in large, rapidly changing document collections

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Epinions.com, WayBackMachine-2004 (http://web.archive.org/web/20041013122547/www.epinions.com/Movies), (http://web.archive.org/web/20041211104641/www.epinions.com/mvie-mu-1131678/display-~reviews), http://web.archive.org/web/20041210125754/www.epinions.com/Movies-Foreign-Films). *
Google Directory Search, Feb. 1, 2004 (http://web.archive.org/web/20040201210740/http://www.google.com/dirhp). *
R. Guha, Open Rating Systems, WayBackMachine-Mar. 22, 2005, (http://web.archive.org/web/20050322033341/http://www.w3.org/2001/sw/Europe/events/foaf-galway/papers/fp/open-rating-systems/wot.pdf). *
Zoltan Gyongyi et al. Combating Web Spam with TrustRank, 2004, pp. 576-587, (http://portal.acm.org/citation.cfm?id=1316689.1316740&coll=&dl=). *

Cited By (167)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8015484B2 (en) * 2006-02-09 2011-09-06 Alejandro Backer Reputation system for web pages and online entities
US20120042386A1 (en) * 2006-02-09 2012-02-16 Alejandro Backer Reputation system for web pages and online entities
US20070208613A1 (en) * 2006-02-09 2007-09-06 Alejandro Backer Reputation system for web pages and online entities
US10268641B1 (en) 2006-05-09 2019-04-23 Google Llc Search result ranking based on trust
US20150310111A1 (en) * 2006-05-19 2015-10-29 Jorn Lyseggen Source search engine
US20080065646A1 (en) * 2006-09-08 2008-03-13 Microsoft Corporation Enabling access to aggregated software security information
US20080172630A1 (en) * 2006-09-08 2008-07-17 Microsoft Corporation Graphical representation of aggregated data
US8234706B2 (en) * 2006-09-08 2012-07-31 Microsoft Corporation Enabling access to aggregated software security information
US9147271B2 (en) * 2006-09-08 2015-09-29 Microsoft Technology Licensing, Llc Graphical representation of aggregated data
US11816114B1 (en) 2006-11-02 2023-11-14 Google Llc Modifying search result ranking based on implicit user feedback
US11188544B1 (en) 2006-11-02 2021-11-30 Google Llc Modifying search result ranking based on implicit user feedback
US9811566B1 (en) 2006-11-02 2017-11-07 Google Inc. Modifying search result ranking based on implicit user feedback
US10229166B1 (en) 2006-11-02 2019-03-12 Google Llc Modifying search result ranking based on implicit user feedback
US9235627B1 (en) 2006-11-02 2016-01-12 Google Inc. Modifying search result ranking based on implicit user feedback
US20170032044A1 (en) * 2006-11-14 2017-02-02 Paul Vincent Hayes System and Method for Personalized Search While Maintaining Searcher Privacy
US9092510B1 (en) 2007-04-30 2015-07-28 Google Inc. Modifying search result ranking based on a temporal element of user feedback
US20080276177A1 (en) * 2007-05-03 2008-11-06 Microsoft Corporation Tag-sharing and tag-sharing application program interface
US7898394B2 (en) 2007-05-10 2011-03-01 Red Hat, Inc. Systems and methods for community tagging
US20080281769A1 (en) * 2007-05-10 2008-11-13 Jason Hibbets Systems and methods for community tagging
US20080301091A1 (en) * 2007-05-31 2008-12-04 Hibbets Jason S Systems and methods for improved forums
US20080301115A1 (en) * 2007-05-31 2008-12-04 Mattox John R Systems and methods for directed forums
US8356048B2 (en) * 2007-05-31 2013-01-15 Red Hat, Inc. Systems and methods for improved forums
US8266127B2 (en) 2007-05-31 2012-09-11 Red Hat, Inc. Systems and methods for directed forums
US7966319B2 (en) 2007-06-07 2011-06-21 Red Hat, Inc. Systems and methods for a rating system
US20090007271A1 (en) * 2007-06-28 2009-01-01 Microsoft Corporation Identifying attributes of aggregated data
US8250651B2 (en) 2007-06-28 2012-08-21 Microsoft Corporation Identifying attributes of aggregated data
US8302197B2 (en) 2007-06-28 2012-10-30 Microsoft Corporation Identifying data associated with security issue attributes
US20090007272A1 (en) * 2007-06-28 2009-01-01 Microsoft Corporation Identifying data associated with security issue attributes
US10839421B2 (en) 2007-07-09 2020-11-17 Groupon, Inc. Implicitly associating metadata using user behavior
US11625753B2 (en) 2007-07-09 2023-04-11 Groupon, Inc. Implicitly associating metadata using user behavior
US20110029613A1 (en) * 2007-08-03 2011-02-03 John Norman Hedditch Evaluation of an attribute of an information object
US8037009B2 (en) 2007-08-27 2011-10-11 Red Hat, Inc. Systems and methods for linking an issue with an entry in a knowledgebase
US20090063386A1 (en) * 2007-08-27 2009-03-05 Hibbets Jason S Systems and methods for linking an issue with an entry in a knowledgebase
US9165075B2 (en) 2007-08-28 2015-10-20 International Business Machines Corporation Managing user ratings in a web services environment
US8635222B2 (en) * 2007-08-28 2014-01-21 International Business Machines Corporation Managing user ratings in a web services environment
US9471616B2 (en) 2007-08-28 2016-10-18 International Business Machines Corporation Managing user ratings in a web services environment
US20090063408A1 (en) * 2007-08-28 2009-03-05 International Business Machines Corporation Managing user ratings in a web services environment
US20140129539A1 (en) * 2007-11-14 2014-05-08 Paul Vincent Hayes System and method for personalized search
US20150261763A1 (en) * 2007-11-30 2015-09-17 Ebay Inc. Using reputation measures to improve search relevance
US9141772B2 (en) 2007-12-31 2015-09-22 Genesys Telecommunications Laboratories, Inc. Trust conferencing apparatus and methods in digital communication
US9537890B2 (en) 2007-12-31 2017-01-03 Genesys Telecommunications Laboratories, Inc. Trust in physical networks
US10726112B2 (en) 2007-12-31 2020-07-28 Genesys Telecommunications Laboratories, Inc. Trust in physical networks
US9628463B2 (en) 2007-12-31 2017-04-18 Genesys Telecommunications Laboratories, Inc. Trust-interactive communication applications
US10289817B2 (en) 2007-12-31 2019-05-14 Genesys Telecommunications Laboratories, Inc. Trust conferencing apparatus and methods in digital communication
US20100082685A1 (en) * 2008-09-23 2010-04-01 Barr Hoisman Hila System and method for evaluation and presenting authorization rights in an organization
US8271501B2 (en) * 2008-12-17 2012-09-18 International Business Machines Corporation Web search among rich media objects
US20100153354A1 (en) * 2008-12-17 2010-06-17 International Business Machines Corporation Web Search Among Rich Media Objects
US20100169265A1 (en) * 2008-12-30 2010-07-01 Herbert Willi Artur Ristock Scoring Persons and Files for Trust in Digital Communication
US9979737B2 (en) * 2008-12-30 2018-05-22 Genesys Telecommunications Laboratories, Inc. Scoring persons and files for trust in digital communication
US20100169430A1 (en) * 2008-12-30 2010-07-01 Herbert Willi Artur Ristock Trust -Interactive Communication Applications
US9009146B1 (en) 2009-04-08 2015-04-14 Google Inc. Ranking search results based on similar queries
US8346765B2 (en) 2009-06-19 2013-01-01 Alibaba Group Holding Limited Generating ranked search results using linear and nonlinear ranking models
US9471643B2 (en) 2009-06-19 2016-10-18 Alibaba Group Holding Limited Generating ranked search results using linear and nonlinear ranking models
US20100325105A1 (en) * 2009-06-19 2010-12-23 Alibaba Group Holding Limited Generating ranked search results using linear and nonlinear ranking models
US20150195295A1 (en) * 2009-07-06 2015-07-09 Google Inc. Monitoring of Negative Feedback Systems
US9336310B2 (en) * 2009-07-06 2016-05-10 Google Inc. Monitoring of negative feedback systems
US8977612B1 (en) 2009-07-20 2015-03-10 Google Inc. Generating a related set of documents for an initial set of documents
US8972394B1 (en) 2009-07-20 2015-03-03 Google Inc. Generating a related set of documents for an initial set of documents
US9418104B1 (en) 2009-08-31 2016-08-16 Google Inc. Refining search results
US9697259B1 (en) 2009-08-31 2017-07-04 Google Inc. Refining search results
US8990083B1 (en) 2009-09-30 2015-03-24 Cisco Technology, Inc. System and method for generating personal vocabulary from network data
US9201965B1 (en) 2009-09-30 2015-12-01 Cisco Technology, Inc. System and method for providing speech recognition using personal vocabulary in a network environment
US8972391B1 (en) 2009-10-02 2015-03-03 Google Inc. Recent interest based relevance scoring
US9390143B2 (en) 2009-10-02 2016-07-12 Google Inc. Recent interest based relevance scoring
US8965883B2 (en) 2009-10-29 2015-02-24 Google Inc. Ranking user generated web content
WO2011050495A1 (en) * 2009-10-29 2011-05-05 Google Inc. Ranking user generated web content
TWI501096B (en) * 2009-10-30 2015-09-21 Google Inc Ranking user generated web content
US20110161317A1 (en) * 2009-12-25 2011-06-30 Fujifilm Corporation Document retrieval system
US8935274B1 (en) * 2010-05-12 2015-01-13 Cisco Technology, Inc System and method for deriving user expertise based on data propagating in a network environment
US8788583B2 (en) 2010-05-13 2014-07-22 International Business Machines Corporation Sharing form training result utilizing a social network
US8601059B2 (en) * 2010-05-13 2013-12-03 International Business Machines Corporation Sharing form training result utilizing a social network
US9623119B1 (en) * 2010-06-29 2017-04-18 Google Inc. Accentuating search results
US20120005183A1 (en) * 2010-06-30 2012-01-05 Emergency24, Inc. System and method for aggregating and interactive ranking of search engine results
US20140114959A1 (en) * 2010-07-31 2014-04-24 Viralheat, Inc. Discerning human intent based on user-generated metadata
US10185754B2 (en) * 2010-07-31 2019-01-22 Vocus Nm Llc Discerning human intent based on user-generated metadata
US10162891B2 (en) 2010-11-29 2018-12-25 Vocus Nm Llc Determining demographics based on user interaction
US9465795B2 (en) 2010-12-17 2016-10-11 Cisco Technology, Inc. System and method for providing feeds based on activity in a network environment
US8667169B2 (en) 2010-12-17 2014-03-04 Cisco Technology, Inc. System and method for providing argument maps based on activity in a network environment
WO2012091541A1 (en) * 2010-12-28 2012-07-05 Mimos Berhad A semantic web constructor system and a method thereof
US9002867B1 (en) 2010-12-30 2015-04-07 Google Inc. Modifying ranking data based on document changes
US8386335B1 (en) 2011-04-04 2013-02-26 Google Inc. Cross-referencing comments
US8645394B1 (en) 2011-04-15 2014-02-04 Google Inc. Ranking clusters and resources in a cluster
US8645393B1 (en) 2011-04-15 2014-02-04 Google Inc. Ranking clusters and resources in a cluster
US8620136B1 (en) 2011-04-30 2013-12-31 Cisco Technology, Inc. System and method for media intelligent recording in a network environment
US20120296918A1 (en) * 2011-05-18 2012-11-22 Microsoft Corporation Credibility Information in Returned Web Results
US8909624B2 (en) 2011-05-31 2014-12-09 Cisco Technology, Inc. System and method for evaluating results of a search query in a network environment
US9870405B2 (en) 2011-05-31 2018-01-16 Cisco Technology, Inc. System and method for evaluating results of a search query in a network environment
US9529920B2 (en) * 2011-06-17 2016-12-27 Robert Osann, Jr. Automatic webpage characterization and search results annotation
US20150347601A1 (en) * 2011-06-17 2015-12-03 Robert Osann, Jr. Automatic Webpage Characterization and Search Results Annotation
US10140374B2 (en) * 2011-06-17 2018-11-27 Robert Osann, Jr. Automatic webpage characterization and search results filtering
US20120323888A1 (en) * 2011-06-17 2012-12-20 Osann Jr Robert Automatic Webpage Characterization and Search Results Annotation
US20170177730A1 (en) * 2011-06-17 2017-06-22 Robert JR. Osann Automatic Webpage Characterization and Search Results Filtering
US9104765B2 (en) * 2011-06-17 2015-08-11 Robert Osann, Jr. Automatic webpage characterization and search results annotation
US10437900B2 (en) * 2011-06-17 2019-10-08 Robert Osann, Jr. Internet search results annotation for missing search terms in referenced webpages and descendant webpages
US10324995B2 (en) * 2011-06-17 2019-06-18 Robert Osann, Jr. Internet search results annotation for missing search terms
US20150310022A1 (en) * 2011-07-11 2015-10-29 International Business Machines Corporation Searching documentation across interconnected nodes in a distributed network
US10467232B2 (en) * 2011-07-11 2019-11-05 International Business Machines Corporation Searching documentation across interconnected nodes in a distributed network
US8886797B2 (en) 2011-07-14 2014-11-11 Cisco Technology, Inc. System and method for deriving user expertise based on data propagating in a network environment
US9886509B2 (en) * 2011-09-08 2018-02-06 Nokia Technologies Oy Method and apparatus for processing a query based on associating intent and audience
US20140214818A1 (en) * 2011-09-08 2014-07-31 Nokia Corporation Method and apparatus for hybrid social search model
US20130166552A1 (en) * 2011-12-21 2013-06-27 Guy Rozenwald Systems and methods for merging source records in accordance with survivorship rules
US8943059B2 (en) * 2011-12-21 2015-01-27 Sap Se Systems and methods for merging source records in accordance with survivorship rules
US9864808B2 (en) * 2011-12-30 2018-01-09 Microsoft Technology Licensing, Llc Knowledge-based entity detection and disambiguation
US20170140059A1 (en) * 2011-12-30 2017-05-18 Microsoft Technology Licensing, Llc Knowledge-based entity detection and disambiguation
US10055467B1 (en) 2012-01-05 2018-08-21 Google Llc Ranking search results
US9684697B1 (en) 2012-01-05 2017-06-20 Google Inc. Ranking search results
US9760641B1 (en) 2012-01-05 2017-09-12 Google Inc. Site quality score
US9031929B1 (en) 2012-01-05 2015-05-12 Google Inc. Site quality score
US8831403B2 (en) 2012-02-01 2014-09-09 Cisco Technology, Inc. System and method for creating customized on-demand video reports in a network environment
US9195944B1 (en) 2012-02-24 2015-11-24 Google Inc. Scoring site quality
US20130297464A1 (en) * 2012-05-01 2013-11-07 Shopsavvy Inc. System, Method, and Computer-Readable Storage Medium For Identifying A Product
US9129020B2 (en) 2012-12-21 2015-09-08 Microsoft Technology Licensing, Llc Search results through interest circles
WO2014100790A3 (en) * 2012-12-21 2014-10-09 Microsoft Corporation Search results through interest circles
US20140201271A1 (en) * 2013-01-13 2014-07-17 Qualcomm Incorporated User generated rating by machine classification of entity
WO2014109781A1 (en) * 2013-01-13 2014-07-17 Qualcomm Incorporated Improving user generated rating by machine classification of entity
US11017089B2 (en) 2013-03-15 2021-05-25 Advanced Elemental Technologies, Inc. Methods and systems for secure and reliable identity-based computing
US9904579B2 (en) 2013-03-15 2018-02-27 Advanced Elemental Technologies, Inc. Methods and systems for purposeful computing
US11922215B2 (en) 2013-03-15 2024-03-05 Advanced Elemental Technologies, Inc. Systems and methods for establishing a user purpose class resource information computing environment
US10509907B2 (en) 2013-03-15 2019-12-17 Advanced Elemental Technologies, Inc. Methods and systems for secure and reliable identity-based computing
US10491536B2 (en) 2013-03-15 2019-11-26 Advanced Elemental Technologies, Inc. Methods and systems for enabling identification and/or evaluation of resources for purposeful computing
US11822662B2 (en) 2013-03-15 2023-11-21 Advanced Elemental Technologies, Inc. Methods and systems for secure and reliable identity-based computing
US9721086B2 (en) 2013-03-15 2017-08-01 Advanced Elemental Technologies, Inc. Methods and systems for secure and reliable identity-based computing
US11528233B2 (en) 2013-03-15 2022-12-13 Advanced Elemental Technologies, Inc. Systems and methods for establishing a user purpose fulfillment computing platform
US9971894B2 (en) 2013-03-15 2018-05-15 Advanced Elemental Technologies, Inc. Methods and systems for secure and reliable identity-based computing
US11514164B2 (en) 2013-03-15 2022-11-29 Advanced Elemental Technologies, Inc. Methods and systems for secure and reliable identity-based computing
US11507665B2 (en) 2013-03-15 2022-11-22 Advanced Elemental Technologies, Inc. Methods and systems for secure and reliable identity-based computing
US11314746B2 (en) 2013-03-15 2022-04-26 Cision Us Inc. Processing unstructured data streams using continuous queries
US10075384B2 (en) 2013-03-15 2018-09-11 Advanced Elemental Technologies, Inc. Purposeful computing
US9378065B2 (en) 2013-03-15 2016-06-28 Advanced Elemental Technologies, Inc. Purposeful computing
US11216305B2 (en) 2013-03-15 2022-01-04 Advanced Elemental Technologies, Inc. Systems and methods configured to enable an operating system for connected computing that supports user use of suitable to user purpose resources sourced from one or more resource ecospheres
US9767157B2 (en) * 2013-03-15 2017-09-19 Google Inc. Predicting site quality
US10884803B2 (en) 2013-03-15 2021-01-05 Advanced Elemental Technologies, Inc. Systems and methods for establishing a user purpose class resource information computing environment
US10853136B2 (en) 2013-03-15 2020-12-01 Advanced Elemental Technologies, Inc. Systems and methods configured to enable an operating system for connected computing that supports user use of suitable to user purpose resources sourced from one or more resource ecospheres
US20140280011A1 (en) * 2013-03-15 2014-09-18 Google Inc. Predicting Site Quality
US10834014B2 (en) 2013-03-15 2020-11-10 Advanced Elemental Technologies Systems and methods for establishing a user purpose fulfillment computing platform
US10540205B2 (en) 2013-03-15 2020-01-21 Advanced Elemental Technologies Tamper resistant, identity-based, purposeful networking arrangement
US10523582B2 (en) 2013-03-15 2019-12-31 Advanced Elemental Technologies, Inc. Methods and systems for enabling fact reliability
US10509672B2 (en) 2013-03-15 2019-12-17 Advanced Elemental Technologies, Inc. Systems and methods enabling a resource assertion environment for evaluating the appropriateness of computer resources for user purposes
US9792160B2 (en) 2013-03-15 2017-10-17 Advanced Elemental Technologies, Inc. Methods and systems supporting a resource environment for contextual purpose computing
US11847495B2 (en) 2013-03-15 2023-12-19 Advanced Elemental Technologies, Inc. Systems and methods configured to enable an operating system for connected computing that supports user use of suitable to user purpose resources sourced from one or more resource ecospheres
WO2014146230A1 (en) * 2013-03-18 2014-09-25 Nokia Corporation Method and apparatus for querying resources through search field
US20170199932A1 (en) * 2014-01-03 2017-07-13 Yahoo! Inc. Systems and methods for quote extraction
USD775183S1 (en) 2014-01-03 2016-12-27 Yahoo! Inc. Display screen with transitional graphical user interface for a content digest
US20150193440A1 (en) * 2014-01-03 2015-07-09 Yahoo! Inc. Systems and methods for content processing
US10296167B2 (en) 2014-01-03 2019-05-21 Oath Inc. Systems and methods for displaying an expanding menu via a user interface
US10242095B2 (en) * 2014-01-03 2019-03-26 Oath Inc. Systems and methods for quote extraction
US9940099B2 (en) * 2014-01-03 2018-04-10 Oath Inc. Systems and methods for content processing
US20150193495A1 (en) * 2014-01-03 2015-07-09 Yahoo! Inc. Systems and methods for quote extraction
US9971756B2 (en) 2014-01-03 2018-05-15 Oath Inc. Systems and methods for delivering task-oriented content
US9742836B2 (en) 2014-01-03 2017-08-22 Yahoo Holdings, Inc. Systems and methods for content delivery
USD760791S1 (en) 2014-01-03 2016-07-05 Yahoo! Inc. Animated graphical user interface for a display screen or portion thereof
USD760792S1 (en) 2014-01-03 2016-07-05 Yahoo! Inc. Animated graphical user interface for a display screen or portion thereof
US9558180B2 (en) * 2014-01-03 2017-01-31 Yahoo! Inc. Systems and methods for quote extraction
US10037318B2 (en) 2014-01-03 2018-07-31 Oath Inc. Systems and methods for image processing
US20150213521A1 (en) * 2014-01-30 2015-07-30 The Toronto-Dominion Bank Adaptive social media scoring model with reviewer influence alignment
US9753990B2 (en) * 2014-03-28 2017-09-05 Microsoft Technology Licensing, Llc Building user trust in profile creation and recommendations based on managed interaction with user
US20150278210A1 (en) * 2014-03-28 2015-10-01 Microsoft Corporation Building user trust in profile creation and recommendations based on managed interaction with user
US10503357B2 (en) 2014-04-03 2019-12-10 Oath Inc. Systems and methods for delivering task-oriented content using a desktop widget
US20150302425A1 (en) * 2014-04-22 2015-10-22 International Business Machines Corporation Assigning priority levels to citizen sensor reports
US20150332169A1 (en) * 2014-05-15 2015-11-19 International Business Machines Corporation Introducing user trustworthiness in implicit feedback based search result ranking
USD761833S1 (en) 2014-09-11 2016-07-19 Yahoo! Inc. Display screen with graphical user interface of a menu for a news digest
US11916960B2 (en) * 2015-12-30 2024-02-27 International Business Machines Corporation Curtailing search engines from obtaining and controlling information
US20180113865A1 (en) * 2016-10-26 2018-04-26 Google Inc. Search and retrieval of structured information cards
US11238058B2 (en) 2016-10-26 2022-02-01 Google Llc Search and retrieval of structured information cards
US10824630B2 (en) * 2016-10-26 2020-11-03 Google Llc Search and retrieval of structured information cards
US11361076B2 (en) * 2018-10-26 2022-06-14 ThreatWatch Inc. Vulnerability-detection crawler
EP4018337A4 (en) * 2019-08-19 2023-08-23 Yext, Inc. Multiple provider search response verification and integration

Also Published As

Publication number Publication date
US10268641B1 (en) 2019-04-23
US8818995B1 (en) 2014-08-26
US8352467B1 (en) 2013-01-08

Similar Documents

Publication Publication Date Title
US10268641B1 (en) Search result ranking based on trust
US8577878B1 (en) Filtering search results using annotations
US8452746B2 (en) Detecting spam search results for context processed search queries
US8316040B2 (en) Programmable search engine
US8495211B2 (en) Network service system and mechanism for searching service registries
US8260774B1 (en) Personalization search engine
Price et al. Filtering Web pages for quality indicators: an empirical approach to finding high quality consumer health information on the World Wide Web.
JP5069285B2 (en) Propagating useful information between related web pages, such as web pages on a website
AU2009277143B2 (en) Federated community search
US20070203891A1 (en) Providing and using search index enabling searching based on a targeted content of documents
US20070038603A1 (en) Sharing context data across programmable search engines
US20070150465A1 (en) Method and apparatus for determining expertise based upon observed usage patterns
US20060200556A1 (en) Method and apparatus for identifying, extracting, capturing, and leveraging expertise and knowledge
US9275145B2 (en) Electronic document retrieval system with links to external documents
US20100049762A1 (en) Electronic document retrieval system
Navarro Bullock et al. Accessing information with tags: search and ranking
US20110289081A1 (en) Response relevance determination for a computerized information search and indexing method, software and device
Faba‐Pérez et al. Comparative analysis of webometric measurements in thematic environments
US8131752B2 (en) Breaking documents
White et al. Leveraging popular destinations to enhance web search interaction
Sharma et al. Search engine: a backbone for information extraction in ICT scenario
Upstill Document ranking using web evidence
Pandey Optimising online marketing resources: SEO strategy
Desikan et al. Link Analysis in Web Mining: Techniques and Applications
Petříček Quantitative analysis of networked environments to improve performance of information systems

Legal Events

Date Code Title Description
AS Assignment

Owner name: GOOGLE INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GUHA, RAMANATHAN;REEL/FRAME:017947/0606

Effective date: 20060712

STCF Information on status: patent grant

Free format text: PATENTED CASE

CC Certificate of correction
FPAY Fee payment

Year of fee payment: 4

FPAY Fee payment

Year of fee payment: 8

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044101/0610

Effective date: 20170929

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 12