A web-browser that uses semantic databases to extract typed data from ordinary, un-annotated webpages, and then uses programming-by-example to allow users to create generalized macros to create actions that use the data. For instance, a user could record a macro giving an example of taking a food item (e.g. “diet coke”), and ordering it online. The program infers that “diet coke” is food, and generalizes the macro to ordering any sort of food item online. Then when the user is looking at a recipe on another web page, the program detects all food items present in the page text, realizes that it knows an action that can be performed on food items, and offers the user the action to select one of them and order it online.
excerpt that describes it:
Using the knowledge in ConceptNet? and TAP (note: these are two semantic databases), we have created a toolbar for Microsoft Internet Explorer that matches the semantic context of a Web page to potential user goals. For instance, imagine a user is viewing a Web page that contains a recipe for Blueberry Pudding Cake. The user’s browser will notice a pattern of foods on the page, and present the user with two suggestions: order the foods, or view their nutritional information. When the user selects one of these buttons, all of the foods on the page turn into hyperlinks for the selected action. For instance, by pressing the “Order Food” button, each food in the recipe will be converted into a hyperlink for that food at the user’s favorite online grocery store. Alternatively, the user can view the nutritional information for each of the foods at their favorite Web site for nutritional information:
Figure 1: Automatically associating a user's high-level goals with the content of a Web page
After being presented with this example, a critical reader likely has two significant questions: (1) How does the browser know how to interact with the user’s favorite grocery store? And (2) How does the browser know which of the terms in the recipe are foods? The answer to the first question is by enabling users to train a Web browser to interact with their favorite sites using a Programming by Example system named Creo (Latin, “to create, make”). The answer to the second question is by leveraging the knowledge bases of ConceptNet? and TAP to create a next generation Data Detector named Miro (Latin, “to wonder”). The following two sections discuss both of these topics in detail.
It is important to note that while this “recipe to grocery store” example is used throughout the paper for the purposes of clarity, Creo can automate interactions with other kinds of sites on the Web (not just grocery stores), and Miro can detect any type of data described in ConceptNet? and TAP (not just foods).