Semantic Integration (COIN Project)

Motivation • Distributed databases makes many disparate sources available. • The web is making even more semi-structured sources available. -- With XML and Web Wrapping, these can be treated as databases. • Schema integration addresses the problem of syntactic inconsistencies. -- i.e., differing structures. • How do we address semantic inconsistencies. -- i.e., differing meanings. (e.g., what does “price” really mean?) Outline • Story of the "Tower of Babel" • Vision of the Future Ÿ Growing number of information sources accessible via “Information SuperHighway” • Implications and Integration Challenges Ÿ Large-Scale Semantic Heterogeneity (More data, less understanding) • Semantic Integration Approaches Ÿ Context Mediation Metadata Services MIT Sloan COntext INterchange (COIN) Project Applications Receivers OUTPUT PROCESSING CONTEXT MEDIATION INPUT PROCESSING * Automatic web wrapping ODBC Driver * Automatic conflict detection and conversion - Derived data Web Publishing Web Pages Sources - Semistructured text - Source selection - Source attribution TRUSTED AGENTS -Multi-source query plan and execution Browsers APPLICATIONS: Financial services, electronic commerce, asset visibility, in-transit visibility. Data bases Source Contexts Sources Information Extraction Services Information Interpretation & Integration Services Information Quality Control & Sharing Services Information Dissemination Services End-Users Environments & Applications Web Interface 953000 Receivers Contexts Allocated Funds 1000’s of S$ Proj Amount A 5000 B Database Gateway COIN Domain Model Web browser Singapore 1’s of $ ORACLE Database Web-page Specification file Context Mediation Expenses Region1 1’s of $ USA Proj A B Amount 2540000 UK Expenses Region2 Proj Amount A 980000 B ODBC Interface Proj Amount A 635333 USA Excel spreadsheet General Web-Wrapper Semi-structured Web-page 1’s of £ excluding 10% VAT Integrity Analyzer Trusted Agents Optimizer Executioner Database Interface Applications Visual Basic, PowerBuilder, etc. 100’s of £ including 10% VAT UK Authorizations General ODBC-Wrapper Data Warehouse Interface The Context Interchange System Excel Spreadsheet Available Funds Proj Amount A 12430 B 1000’s of S$ Tools for Expenses Region3 100’s of ¥ Japan Project A has spent 45000 and project B --Semi-structured text message General Text-Wrapper Text-page Specification file Declarative Knowledge (context definitions) Question to be answered: How much funding is left for Project A? Data Warehouse • Information Mining • Information Brokering Question: How much funding is left for Project A ? Singapore Role Of Context 02-01-03 Context Context $ 01-02-03 £ ? Context ¥ z 03-02-01 CONTEXT VARIATIONS: - GEOGRAPHIC ( US vs. UK ) - FUNCTIONAL (CASH MGMT vs. LOANS ) - ORGANIZATIONAL ( CITIBANK vs. CHASE ) Data: Databases Web data E-mail Example : Context Differences ( from multiple web sources) Daimler Benz ( DAI ) Key Ratios ABC P/E Ratio EPS 11.6 Dividend 0.29 Bloomberg 5.57 15.32 8.127 DBC 19.19 4.36 0.899 MarketGuide 7.46 10.83 0.47 “What’s the P/E Ratio?” August 21, 2001 – Wall Street Journal -- Page One Feature: What’s the P/E Ratio? Well, Depends on Meaning of Earnings By JONATHAN WEIL, Staff Reporter of THE WALL STREET JOURNAL Few investors know it, but the U.S. stock market today is, by one way of looking at it, the most expensive it has ever been. How could that be, after the numbing slide since the market peaked in early 2000? It turns out that for all the pain, the stock market now is far out of whack with historical norms by one common measure, the price-to-earnings ratio. The P/E ratio measures how companies' share prices compare with their profits, showing how much value the market places on each dollar of a company's earnings. The lower the P/E, as a rough rule of thumb, the cheaper the stock. Though this guide to value has lots of exceptions, it remains a venerable market benchmark. … … Example of FMC: P/E = $1.58 from First Call P/E = -$9.62 according to WSJ calculation The 1999 Overture Unit-of-measure mixup tied to loss of $125Million Mars Orbiter “NASA’s Mars Climate Orbiter was lost because engineers did not make a simple conversion from English units to metric, an embarrassing lapse that sent the $125 million craft off course. . . . . . . The navigators ( JPL ) assumed metric units of force per second, or newtons. In fact, the numbers were in pounds of force per second as supplied by Lockheed Martin ( the contractor ).” The Context Interchange Approach Concept: Length Meters f() meters feet Feet Shared Ontologies Source Context Source Conversion Libraries Context Mediator Receiver Context Context Transformation Receiver Context Management Application COIN Elevation Axioms Primark Web Examples Top 25 US Co. by Net Sales (Disclosure) Rank Company Net Sales (000’s) Date 12/31/95 1 General Motors Corp 168,828,600 ? 137,137,000 12/31/95 2 Ford Motor Co 121,804,000 12/31/95 3 Exxon Corp ? 93,627,000 01/31/96 4 Wal Mart Stores Inc 79,609,000 12/31/95 5 AT&T ? 12/31/95 73,413,000 6 Mobil Corp 12/31/95 7 International Business M71,904,000 70,028 Top 25 International Co. by Net Sales (Worldscope) 8 General Electric Co ... ... ... Rank Company Net Sales (000’s) Date Primark was a company that owned: • Disclosure • Worldscope • Datastream Information services 1 Mitsubishi Corporation 165,848,468 2 General Motors Corp 163,861,100 ... ... ... 8 Exxon Corp 107,893,000 ... ... ... 16 International Business M71,940,000 17 General Electric Co 69,948,000 20 Mobil Corp 64,767,000 ... ... ... 03/31/96 12/31/95 ... 12/31/95 ... 12/31/95 12/31/95 12 12/31/95 ... Another Context Example Company Name DAIMLER-BENZ Net Income 614,995 97,736,992 Sales Context Mediation Services * Datastream Company Name DAIMLER-BENZ AG Net Income 346,577 Sales 56,268,168 * WorldScope Company Name DAIMLER BENZ CORP Net Income 615,000,000 Sales 97,737,000,000 Users & Appl. Systems * Disclosure O&A DEM-USD Exchange Rate 1.00 German Mark= 0.58 US Dollar as 12/31/93 * OANDA Web Server * Wrapper Services Some Context Differences Context Definitions Disclosure Country of Incorporation Money Amount As_Of_Date 3 Letters Currency Used Currency Conversion Currency Symbols Scale Factor 1 Disclosure Names Company Names American with ‘/’ as Date Style separator Worldscope USD Money Amount As_Of_Date 3 Letters DataStream Country of Incorporation Money Amount As_Of_Date 2 Letters 1000 Worldscope Names 1000 DataStream Names American with ‘/’ as separator European with ‘-’ as separator Olsen (OANDA) Web Source uses 3 Letter Currency Symbols and European Date Style with ‘/’ as a separator Domain Model mC ur toC ur fro company format cou ntry I in g d n fyE dateF cy ren cur Inheritance Attribute Modifier mt currencyType companyFinancials countryName ency r r u C l a i c i off ate txnD scaleFactor pe y rT u c string m Sy nco rp exchangeRate number date companyName Some currency context possibilities: • Currency is stated explicitly as part of record • Current not stated, but the same for all (e.g., US $) • Currency not stated or constant, but inferred by country COIN System Architecture SERVER PROCESSES MEDIATOR PROCESSES CLIENT PROCESSES Web Client SQL Query Context Mediator Datalog Query WWW Gateway Optimizer HTTPD-Daemon Wrapper Executioner Mediated Query Optimized Query Plan (cgi-scripts) N HTTPD-Daemon SQL Compiler SQL Query HTTPD-Daemon COIN Repository N ODBC-compliant Apps Results (e.g Microsoft Excel) ODBC-Driver HTTPD-Daemon Web-site Data Store for Intermediate Results Prototype: COIN on World Wide Web INFORMATION CONSUMERS ONTOLOGY PUBLISHERS ... Receiver Contexts ... Ontologies MultiDatabase Browser USER Client S/W (Excel, Powerbuilder) ... Source Contexts GSS Wrapper Context/ Ontology Browser & Editor CONTEXT ADMINISTRATORS INFORMATION PRODUCERS Web pages (e.g., NetWorth) Context Mediator Conflict detection Query reformulation Context conversion & assembly SQL-DB Wrapper Databases NETWORK SERVICES (e.g., Oracle:W orldscope) System Demonstration Single Source Queries with Mediation Q6. Scenario: Using Context Interchange, the financial analyst can look at the Disclosure data using Datastream Context. Query: Find out from Disclosure what Net Income for DAIMLER-BENZ was. Use Datastream Context. Capabilities Demonstrated: Ability to perform Scale Factor Conversion, Date Format Conversion, Company Name Conversion. Demonstration Conflict Detection and Mediation Mediated Query in Datalog Date convert Scale factor convert Name convert Mediated SQL Query & Result Mediated SQL Query Adjust scale factor Date format conversion Name conversion Final results – from Disclosure but in Datastream context Expanded scope of interactions Other Geographic Regions C D A B Other Functions E External Relations ISSUE: How do we capture knowledge needed to make effectuve use of these sources? The 1805 Overture In 1805, the Austrian and Russian Emperors agreed to join forces against Napoleon. The Russians promised that their forces would be in the field in Bavaria by Oct. 20. The Austrian staff planned its campaign based on that date in the Gregorian calendar. Russia, however, still used the ancient Julian calendar, which lagged 10 days behind. The calendar difference allowed Napoleon to surround Austrian General Mack's army at Ulm and force its surrender on Oct. 21, well before the Russian forces could reach him, ultimately setting the stage for Austerlitz. Summary • Tremendous opportunity to gather information • Need to overcome many context challenges • Context-type “metadata” plays a critical role • W3C Resource Description Framework (RDF) is moving in that direction ... • Some joint experiments at MIT Sloan: – Merrill-Lynch: Trader assist & Global risk – Price-Waterhouse: B-to-B e-commerce – BSCH/Suruga/Fleet: Universal Financial Aggregator – Primark: Financial information integration – MITRE: Context Mediation prototype – AF: Logistics and repair