The UniCats Project
project description
initial project description (May 1998)
project description after first year's work (October 1999)
May 1998
The popularity of the WWW induces a rapidly growing number of service providers to "go online". This is also true, or will become so, for the university libraries which already today are embedded in a network of other libraries, documentation centers, traditional information brokers, publishing houses and individual information producers. These providers differ in their service portfolio, quality of service, and pricing models. All of these could be termed providers of original services. Due to the differences one will come to expect that an open electronic market will develop around digital documents and a host of access and management services for them, e.g. structured bibliographic or full-text searching, traditional lending, online reading, instant printing, subscripiton of jounals, and electronic payment.
Original services aim at a mass market of commonly used basic functionalities. Individual customers, however, have individual problems and needs which require a combination of original services a complete solution. Hence, an intermediary layer of value-added services is needed that selects and combines original services into services of a richer though more specialized nature. At the very least, such services should relieve the customer who is confused by the sheer multitude of available services and hampered in identifying the services for his information needs, that are optimal in terms of their functionality and price/performance ratio.
The aim of our UniCats project is a universal integration of catalogs based on an agent-supported trading and wrapping system to provide a uniform and enriched access mechanism to the variety of existing library services. The target group for the prototype will initially be the members of the University of Karlsruhe.
A three-tier architecture underlies UniCats:
The syntactical and semantical integration of service providers is accomplished by wrappers. Not only will they translate between the market-internal protocol and the native protocols of the providers, but they extract from the providers the information that is needed to install the value-added services. We develop a tool that will make semi-automatical generation of wrappers possible and thereby simplify the integration of new information providers.
Traders play an essential role in any information market. Traders are intermediaries that given the needs of a shopper, identify and locate servers and provide information about them. They are often referred to as the "yellow pages" of a network. Parameters affecting the choice of a provider are the subject areas covered by it, document types, delivery times, prices, and performance.
User support will be provided by elaborate user agents which have or develop an understanding of a user's interest and apply this knowledge to map a user's demands to the available services. Presently we associate with the user agents any value-added service that combines original services. Consequently, their immediate communication partners are the customers and the traders. For the communication with the customers we develop an intuitive graphical user interface on the basis of the metaphor of a virtual library building with individually organizable rooms and bookshelves.
To integrate services liable to charges, e.g. query and delivery services or mediation services of a trader, the system includes components for cost accounting and electronic payment.
October 1999
Background and Motivation
The popularity of the WWW induces a rapidly growing number of service providers to "go online". This is also true, or will become so, for the university libraries which already today are embedded in a network of other libraries, documentation centers, traditional information brokers, publishing houses and individual information producers. These providers differ in their service portfolio, quality of service, and pricing models. All of these could be termed providers of original services. Due to the differences one will come to expect that an open electronic market will develop around digital documents and a host of access and management services for them, e.g., structured bibliographic or fulltext searching, traditional lending, online reading, instant printing, subscription of journals, and electronic payment.
Original services aim at a mass market of commonly used basic functionality. Individual customers, however, have individual problems and needs, which require a combination of original services for a complete solution. Hence, an intermediary layer of value-added services is needed that selects and combines original services into services of a richer though more specialized nature. At the very least, such services should assist a customer who is confused by the sheer multitude of available services and hampered in identifying the services for his information needs that are optimal in terms of their functionality and price/performance ratio.
Objectives
The aim of our UniCats project is to provide a uniform and enriched access mechanism to the variety of existing library services. Our objectives are twofold: to develop - under the premise of an open market for document services - a technical infrastructure that
- allows campus users, under their individually defined, optimal conditions, to locate and retrieve documents from a network, transparently if so desired;
- enables university libraries to compete effectively in the market for a base of academic-oriented customers.
Results
The UniCats environment provides a technical platform that supports the structure and the dynamics of an information market. It is based on independent and communicative UniCats agents. This infrastructure is based on a coarse architecture that encompasses three main components:
- At the one end there are user agents which offer the customer a uniform access point with an appropriate user interface. They develop plans to transform the customer's demand into different queries and contain algorithms to integrate results from different sources.
- At the other end there are wrappers as the proxies for the service providers in our system. Their task is to translate queries into the special format of an information source, and results back from the format of the source into the result format of the UniCats environment.
- In between there are traders. Traders know the available service providers and match incoming user demands with existing service offers to find those providers most appropriate to the user query.
Figure 1 illustrates the communication principle and information flow inside the UniCats environment.
A wrapper has to register with a trader (or more than one trader) to become known within the system and to announce its offers. While doing so, the wrapper gives some metadata about its source, such as number of documents, languages, cost model, or attributes the user can search for.
When given a user query, the user agent contacts a trader to ask for services appropriate for this special query. The trader answers with a list of recommendations for service providers together with available additional information, e.g., the estimated cost for an access to this source.
Thereafter, the user agent addresses the suggested services, actually the wrappers. These transform the query into the native query format of the information source and send the re-transformed query results back to the user agent. The user agent collects the results from the different sources, integrates them, and presents a final result collection to the user.
UniCats agents base on a common communication platform. This
framework overcomes the heterogeneity of the market and establishes
communication and data transfer between the system components. We use
TCP/IP as the universal protocol for data transfer, because this
protocol is widespread used (it is the Internet protocol) and
available for most programming languages and hardware platforms.
We also chose XML (eXtensible Markup Language) as the common data
model. That means, communication inside the UniCats environment
occurs by an exchange of XML-documents. The communication can be
one of two kinds: group communication by multicast, and direct
communication between two agents by exchanges.
For the communication
we adapted OTP (Open Trading Protocol) because of its use in
commerce transactions. The communication between user agent and
customer, and wrapper and service provider, respectively, is based on
HTTP. This is the protocol available to practically all, customers
and providers alike, since customers can use their favorite web
browser to access the UniCats system, and providers can use the
standard HTML site they usually already have.
Interaction with the UniCats environment must be intuitive and
convenient for the user. This implies that for different customer
groups different kinds of user interfaces are most applicable.
We support user-adapted interfaces and currently provide three
different types of user interface:
The user agent coordinates the execution of the search process.
A major task is the integration of result
sets received from different information sources. Duplicates must
be detected and eliminated, results may be grouped and sorted.
Result integration runs incrementally: Results are shown to the user as
soon as they arrive, so the customer can work with the first
results while the user agents receives further data from the
wrappers.
In order to keep the requirements to the providers as simple as
possible (and this is a precondition for an open market), we demand
neither direct access to the provider's database nor the
installation of a special interface. We use the HTML interface
of the provider. Such interfaces are available for practically every
information source.
Not only will wrappers be tailored to the individual information
sources, each change of the source's web presentation forces a
redesign of the wrapper. We implemented wrappers for 20 information
sources manually, investigated the web sites, and detected
regular structures that allow an automation of wrapper generation.
We developed a wrapper generator that facilitates and accelerates
wrapper generation in a great measure, and enables wrapper
generation to persons who usually do not have the
knowledge to build a wrapper manually, e.g., librarians.
Traders have to find the providers most suitable to the customer's
query. They keep metadata about the information providers. The
selection of the appropriate providers is not restricted to
content-based criteria. Aspects such as prices, workload and
response times, locality of the provider, and available
result formats are equally important. The developed selection
algorithm supports a configurable handling of these aspects. To
overcome scalability problems, we developed a model for cooperation
among different traders.
At present there exists a functional prototype of very limited
scale - enough, though, in order to test the validity of our
assumptions and technical approach. What we have left out so far
are all aspects of electronic commerce transactions, security,
payment services, and trust. These will be, over and above
purely technical issues of extending the scale and functionality,
the subject of future work.
The additional components will have to encompass certification
services, notary services, and servers for digital payment. Our goal
is to provide UniCats as a platform for the integration of services
that result in other V3D2 research projects, one of them being
electronic payment. In order to support electronic commerce
transactions and to guarantee confidentiality, we intend to add
modules for encryption and secure data transfer. Additional
functionality such as electronic ordering and delivery improve
the benefit to the customer.
User agents provide the window between customer and system. They
must be able to assist in the entire process consisting of
multiple search and delivery activities. They should maintain
contexts that range across sessions so that users can work
iteratively all the way to the lifetime of a user registration
so that the system can adapt to user profiles. This requires the
development of flexible and robust algorithms for query planning,
because the user agents should work under any condition and should get
over changes in market conditions such as network problems or
price differences.
Wrapper generation must be extended to cope with a much larger
number and diversity of sources. Further, wrappers should be able to
optimize query processing in a particular source, an aspect that
becomes important whenever the providers charge for the use of
the source. A wrapper in an electronic commerce scenario must have
modules for cost planning and payment.
Particular emphasis will be given to the trading algorithms in
order to increase their adaptability and to allow cooperation within
trader federations. Equally important is the development of methods
and algorithms to automatically generate metadata that the providers
may be unable or unwilling to give. We plan to study and experiment
with the role of the trader as a main organizer within the market.
Another point is the acceptance of digital libraries. We plan to
study acceptance issues within the clientele of the university
library in Karlsruhe. This concerns issues such as the behavior
of users, the acceptance of the interface, the cost models and the
billing and payment systems.
Future Steps
