The Knowledge Machine framework has been developed in the Centre for Digital Music, QMUL, over the past few years. It aims to create a workspace for multimedia analysis researchers, which is aware of an open-world knowledge environment.

---

• The Knowledge Machine central architecture
  • Knowledge representation for data analysis
  • Evaluation engines
  • Function Tabling
  • Determinism
• Interaction with a shared knowledge environment
  • Semantic web technologies
    • A web of data...
    • ...which actually means something
    • Open data
  • Interpreting semantic-web knowledge inside the Knowledge Machines
  • Exporting knowledge from the Knowledge Machines to the semantic-web
• How can all this stuff be useful for OMRAS2??
  • Common understanding (domain ontologies)
  • Dynamic creation of results, easy dissemination
  • _Foafing_ the research
  • Feedback, distributed annotations
• Related publications

---

The Knowledge Machine central architecture

Knowledge representation for data analysis

The Knowledge Machine uses a knowledge representation based on predicate calculus. Thus, information such as this spectrogram was computed from this signal using these parameters may be represented as spectrogram(DigitalSignal?,FrameSize,HopSize,Spectrogram). Knowledge consists in facts and composite formulae, built using the logical connectives if, exists, for all, or, and, not, equivalent to.

Evaluation Engines

Such predicates, used in a given mode may hide calls to external evaluation engines:

• Matlab engine
• Java
• C/C++
• Python
• Perl

This can either be done directly, or through an external interpreter. For example, we have an operator evaluating Matlab expressions. A matrix multiplication is represented as:

mtimes(A,B,C) if C===A * B

Function Tabling

We consider tabling of resulting facts. For example, for a particular evaluation of mtimes(a,b,C), I will store the resulting fact mtimes(a,b,c), thus allowing to keep track of all the objects I have in my workspace (what is c? oh yes! it was computed using this particular predicate, and these particular inputs), and also to save similar evaluation (mtimes(a,b,C), will directly bind C to c, without calling the Matlab engine, in this particular example). Everything is computed once.

Determinism of predicates

We recognise several determinism for our predicates, directly influing on the particular tabling mechanism to be used:

• deterministic (1 solution, same mechanism as the one described for mtimes)
• semi-deterministic (no solutions or 1 solution)
• multi-solution (more than 1 solution)
• non-deterministic(zero or more solutions)
• random (involving a random process)

Interaction with a shared knowledge environment

Semantic web technologies

A web of data...

RDF (Resource Description Framework) provides a way to create a web of data. We consider every object as first-class entities, and identify them using an URI (Universal Resource Identifier). We create links between these objects using properties, and we let every actor contribute by making new knowledge available. For example, I might put on the semantic web that me foaf:knows marksandler (foaf:knows is a property from the Friend of a Friend vocabulary, expressing here that, well, I know Mark Sandler:-) ). This statement is a RDF triple.

...which actually means something

OWL (Ontology Web Language) aims to create an understanding of this data by linking it to real-world objects. We create domain ontologies by identifying the important concepts (such as foaf:Person) and relations (such as foaf:knows) in a given domain.

Linked open data

There are more and more open-data, expressed in RDF and following the principles of Linked Data (allowing automatic agents to easily find and discover information on the semantic-web), allowing to create a real machine-understandable cultural environment:

• DBTune, mash-up of several creative commons music repositories
• dbpedia
• geonames
• musicbrainz
• DBLP
• ...

Interpreting semantic-web knowledge inside the Knowledge Machines

We provide a way to access this machine-understandable cultural environment by allowing to interpret available knowledge as predicates, thus available for reuse when bulding composite formulae. For example, I may create a predicate instrument(AudioFile?,Instrument), linking an audio file to a musical instrument pertaining to its creation, which will in fact access knowledge available in the semantic web. Thus, processes become context sensitive.

Exporting knowledge from the Knowledge Machines to the semantic-web

Once we reach a stage where we feel that the results of a particular predicate might be useful for other Knowledge Machines or other agents, we may want to automatically export its results and make them available for reuse. In order to do this, we express what a predicate means through what we call a semantic match, basically linking a predicate to a set of RDF triples.

Once we have done that, we have a simple mechanism allowing to dynamically call predicates in order to fullfill an agent's request. The mechanism used for this moves away from a SOA approach to what we may call a resource-oriented approach. We did several experiments with SOA-based approach (and especially semantic web-services), which basically made us realised it was way too complex and not scalable. Using our approach, an agent only knowing about RDF and HTTP is able to dynamically generate all the results the network of knowledge machines may export. Processes can be automatically combined and preconditions can be checked. The corresponding implementation is still under active development though.

---

How can all this stuff be useful for OMRAS2??

Common understanding (domain ontologies)

Building domain ontologies helps to create a common understanding of RDF graph made available. Some efforts, related to creating ontologies for music-related domain, are:

• The music ontology
• List of multimedia-related vocabularies
• Music recommendation ontology
• Classical music and performances
• MPEG7 ontologies

Efforts are being made to link the editorial, the music production and the music recommentation domain ontologies.

Dynamic creation of results, easy dissemination

By using an approach such as the one described in the previous section, we allow agents only knowing about RDF and HTTP to dynamically access results that our workspaces are able to provide (maybe by combining themselves to other workspaces).

Foafing the research

Using the Friend of a Friend vocabulary, we can define profiles for involved researchers, create networks of researchers, associate a particular result to the profile of the researcher (or group of researchers), automatically look for people interested in the same subject as you... Moreover, using this dynamic creation of results, a researcher might use the results of an other researchers, without noticing it!

Feedback, distributed annotations

Moreover, as any agents can contribute to the semantic-web knowledge environment, it is able to state that a particular result is crap, or that it is really good for a given genre, and let it know.

---

Related publications

• An ontology-based approach to information management for music analysis systems, SamerAbdallah, YvesRaimond, MarkSandler, International AES Convention, 2006
• A scalable framework for multimedia knowledge management, YvesRaimond, SamerAbdallah, MarkSandler, Mounia Lalmas, 1st conference on Semantics and Digital Media, 2006, also available here, video of a presentation by YvesRaimond available here
• Proposal for a common multimedia ontology framework: Information management for music analysis systems, Call for a common multimedia ontology framework, AceMedia^?

-- YvesRaimond - 06 Feb 2007