University:
Massachusetts Institute of Technology
Course:
15.501/516 | Accounting
Academic year:

2023
Views:

216

Pages:

19
Author:

Armani Calderon

Browser A “raw information” B C Web sites Traditional Use of the Web: For direct human usage Focus: Entertainment Programs for: Examples: Extract just mortgage rate information Enhancing Web Wrapper and Context Mediator Technology A Compare mortgage rates offered by multiple sources Analyzing Build cumulative database of mortgage rates over time Repository Compare cumulative rates with previously stored, alert to new highs and lows Exception reports Consolidating C B Web sites Processing New Use of the Web: Program intermediaries Focus: Productivity Internal data bases MIT Sloan COntext INterchange (COIN) Project Applications Receivers OUTPUT PROCESSING CONTEXT MEDIATION INPUT PROCESSING ODBC Driver * Automatic conflict detection and conversion * Automatic web wrapping - Derived data Web Publishing - Source selection - Source attribution TRUSTED AGENTS Browsers - Semistructured text -Multi-source query plan and execution APPLICATIONS: Financial services, electronic commerce, asset visibility, in-transit visibility. Web Pages Sources Data bases Cameleon Architecture SQL Relational Front End Data Planner Optimizer Executioner Application Output in Desired Format Simple SQL Query & Output Format Core Retrieval HTTP Client Extraction Regular Expression Engine Query Handling Spec File Parsing Authentication, S Web or Database Spec Files Registry CIA Fact Book Example CAMELEON QUERY: Select capital, location, coordinates, totalarea, climate, population, GDP from cia where Country="Singapore" CAMELEON RESULTS: Record 1 CAPITAL Singapore LOCATION Southeastern Asia, islands between Malaysia and Indonesia COORDINATES 1 22 N, 103 48 E TOTALAREA 647.5 sq km CLIMATE tropical; hot, humid, rainy; no pronounced rainy or dry seasons; thunderstorms occur on 40% of all days (67% of days in April) POPULATION 4,151,264 (July 2000 est.) GDP $98 billion (1999 est.) Spec file for CIA Fact Book (partial) #Relation=cia #Source=http://www.odci.gov/cia/publications/factbook/country.html #Attribute=Link#String #Begin=Top\s*of\s*Page #Pattern=< LI>< a href="([^"]*)">#Country#< / a> #End=< / [Bb][oO][dD][yY]> #Source=http://www.odci.gov/cia/publications/factbook/#Link# #Attribute=Telephone#String #Begin=Telephones: #Pattern=\s*([\0-\377]*?)\s* #End=Telephone system: #Attribute=Background#String #Begin=Background: #Pattern=\s*([\0-\377]*?)\s*< #End=Location: #Attribute=Location#String #Begin=Location: #Pattern=\s*([\0-\377]*?)\s* #End=Geographic\s*coordinates: ... Regular Expressions Used in Spec Files * Match 0 or more times (greedy). • *? Match 0 or more times (non-greedy). • + Match 1 or more times (greedy). • ? Match 0 or 1 time (greedy). Greedy quantifiers such as * matches as much as possible, whereas non-greedy quantifiers stop at the minimum match. Example: hello lovely world (.*) would match ‘hello lovely world’ whereas (.*?) would match ‘hello’ and ‘world’ • • • • • • • • • . matches everything except \n [\0-\377] matches everything ^ matches the beginning of a string or line [^ a character] matches everything except the specified character. For instance [^< ] matches anything but < $ matches the end of a string or line \s matches a whitespace character \S matches a non-whitespace character \d matches a digit Expressions within parentheses are saved. Sample Application Research Analyst or Trader Spreadsheet Text Application WWW Legacy Application Manual Data Movement Providing Integrated Data and Analysis Equity prices - TIBCO Real-time feed SEC Filings - EDGAR Web based - Internet News - Reuters, Newswire and Business wire Web based - Internet Merrill Research Reports Text based - Merrill Intranet Market updates - Merrill Lynch home page Web based - Internet Spreadsheet Interface XML – The Silver Bullet ? • XML is (according to press reports …) ¾ “HTML on steroids” ? ¾ “a Rosetta Stone” ? ¾ “a universal way to translate data” ? ¾ “a miraculous way to” … information integration ? ¾ “a silver bullet” ? XML What is it? • • • • • • XML - EXtensible Markup Language Meta language for defining a markup language Based on SGML - Standard Generalized Markup Language Data model for syntax for structuring data Can define tags at will Can nest document structures to arbitrary levels of complexity • Can use Document Type Definition (DTD) • Many other members of “family”: – XSL, XSLT, XLL, XML-Query, etc. XML Does help create structured Web pages Feature XML HTML Extensibility Fixed set of tags Extensible set of tags Tag purpose Presentation Content Views Single Multiple (XSL) Orientation Documents Search Keyword Documents + semistructured data Keywork plus Fieldsensitive queries 15 Example: HTML Compared with XML HTML * < !DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> < html> < head> . . . < BODY topmargin=18 leftmargin=6 bgcolor="#ffffff" link="#0000ee" VLINK="#551A8B" ALINK="#ff0000"> < pre> Palm Pilot V Regular Our Price Price 329.00 236.00 In stock < / pre> < table cellpadding=0 cellspacing=0 border=0> < tr>< td align=left valign=middle width=455 nowrap height=20> < tr>< td align=left valign=top nowrap width=455> . < / BODY> < / HTML> .. * The HTML is normally much messier, with lots more formatting details and < tr> and < td> tags for defining tables and tab positions. XML < XML> < Product info> < Product> Palm Pilot V < \ Product> < Regular price> 329.00 < \ Regular price> < Our price> 236.00 < \ Our price> < InStock> yes < \ InStock> < \ Product info> < \ XML> XML Why do we need it? • W3C wanted to get out of the tag creation business • Separate data from presentation – Use of a style sheet instead of “hardcoded” HTML formatting – Flexibility / scalability / extensibility Netscape browser (default) XML Page • • • • Custom browser / Application Also important for Wireless Applications (WML/ XHTML) Human readability Computer processable Information interchange Sample Applications of Semi-Structured Web Data • Automate Extraction of data from specific Web sites into user tool, like Excel, or own Web browser / consolidator Fidelity Fidelity 500 Bank of Boston 750 Total 500 Bank of Boston 750 1250 Accounts (Web sites) • Automatically Select and Consolidate information across Web sites IBM Evaluations Analyst Evaluation A 5.0 B 4.2 Company: IBM Evaluation: 5.0 A Company: IBM Evaluation: 4.2 B Analyst Reports (Web sources) • Integrate Internet / Intranet / Client Server networks for internal operations Yesterday's shipments (Internal database) Report of shipments not delivered by noon today Delivery Status Program FedEx package tracking (Web site) UPS package tracking (Web site) XML . . . Multiple Standards • What is so great about XML standards – is that there are so many . . . • Should tag for catalog be called “price” or “cost” ? • “The director of electronic trading at Credit Suisse First Boston and chairman of financial services XML working group is struggling with more than a dozen XML protocols … for financial trading applications.” (ComputerWorld, 9 July 2001) XML – The Silver Bullet ? • XML is not quite: 3/4 “a Rosetta Stone” 3/4 “a universal way to translate data” 3/4 “a miraculous way to” … information integration 3/4 “a silver bullet” • It is a helpful tool toward information integration . . . • Some background sources: w3c.org/XML and XML.org • But much more is needed for information integration ¾ Context Interchange and Semantic Web research are promising areas . . . Summary • Tim Berners-Lee, W3C Director: - "The Web is quickly becoming the world's fastest growing repository of data” • In past: Primarily processed by humans. • In future, must be processible by programs (agents for humans) • Tools, such as MIT’s Automatic Web Wrapper and W3C’s XML, are providing these capabilities. Work reported herein has been supported, in part, by the USA Advanced Research Projects Agency, Banco Santander Central Hispano, Citibank, Fleet Bank, First Logic, Merrill Lynch, PricewaterhouseCoopers, MIT Total Data Quality Management Program, MIT Center for eBusiness, Suruga Bank, and USAF/Rome Laboratory.