XPORT: eXtensible Profile-Driven Overlay Routing Trees
for Internet-scale stream collection, processing and dissemination



The confluence of ubiquitous, high-performance networking and increased availability of online information has led to the emergence of a new class of large-scale stream processing and dissemination applications. These applications often exhibit diverse logic and performance requirements, yet they all require common facilities, which include construction of an overlay network, routing and processing logic, and membership management. In contrast to existing approaches that provide custom, point solutions to point applications, we introduce a general-purpose infrastructure that provides these core functionalities and can be easily extended for a broad spectrum of target applications.


Extensibility is the central design consideration for our infrastructure, which can be customized to support diverse processing logic, stream types, and performance targets through a set of methods that encapsulate application-specific behavior and a cost model for defining the desired QoS and resource-utilization metrics and constraints. Given these specifications, the system automatically creates and optimizes a data stream acquisition, processing and dissemination overlay network. Its optimization is driven by metric-independent operations, which can refine the structure of the overlay network as well as efficiently distribute processing across the network.


This project subsumes the SemCast project.


Publications

  1. BulletSupporting Generic Cost Models for Wide-Area Stream Processing, O. Papaemmanouil, U. Cetintemel, J. Jannotti, In Proceedings of the 25th International Conference on Data Engineering (ICDE ‘09), April 2009 (to appear).

  2. BulletXFlow: Internet-Scale Distributed Stream Processing. O. Papaemmanouil, U. Cetintemel, J. Jannotti. Brown University, CS Department, Technical Report CS-07-06. [pdf]

  3. BulletExtensible Optimization in Overlay Dissemination Trees. O. Papaemmanouil, Y. Ahmad, U. Cetintemel, J. Jannotti, Y. Yildirim. In Proceeding of the 25th ACM International Conference of Special Interest Group on Management of Data (SIGMOD '06), June 2006. [pdf]


  1. BulletXPORT: Extensible Profile-driven Overlay Routing Trees. O. Papaemmanouil, Y. Ahmad, U. Cetintemel, J. Jannotti, Y. Yildirim. In Proceeding of the 25th ACM International Conference of Special Interest Group on Management of Data (SIGMOD '06), June 2006 [pdf] [poster]


  1. BulletApplication-aware Overlay Networks for Data Dissemination. O. Papaemmanouil, Y. Ahmad, U. Cetintemel, J. Jannotti. In the Proceeding of the International Workshop on Semantics enabled Networks and Services (SeNS '06) (In conjuction with ICDE'06), Atlanta, April 2006. [pdf]

  2. BulletExtensible Profile-driven Data Dissemination. O. Papaemmanouil, Y. Ahmad, U. Cetintemel, J. Jannotti, Y. Yildirim. Brown University, CS Department, Technical Report CS-06-05. (A shorter version appeared in SIGMOD'06). [pdf]

  3. BulletSemCast: Semantic Multicast for Content-based Data Dissemination. O. Papaemmanouil, U. Cetintemel. In Proceeding of the 21st International Conference on Data Engineering (ICDE '05), April 2005. (67/521=12.8% accepted) [pdf]

  4. BulletSemantic Multicast for Content-Based Stream Dissemination. O. Papaemmanouil, U. Cetintemel. In Proceeding of the 7th International Workshop on the Web and Databases (WebDB '04) (In conjuction with SIGMOD'04) , June 2004 (16/61=26.2% accepted) [pdf]

  5. Members


  6. BulletOlga Papaemmanouil,

  7. BulletUgur Cetintemel,

  8. BulletJohn Jannotti,

  9. BulletYanif Ahmad

  10. BulletYenel Yildirim